Lazarus

Programming => LCL => Topic started by: del on October 07, 2021, 01:25:33 pm

Title: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 07, 2021, 01:25:33 pm
Lazarus 2.0.12 / FPC 3.2.2. Linux Manjaro. When I go to open an image file in a directory with 30000 video frames the program stalls, the fan revs up, and the CPU usage hits 100%. I suspect one of the widgets used by Lazarus is becoming unnecessarily busy. My workaround is to change the type of dialog from TOpenDialog to TSaveDialog. This solves the problem. It's weird having to click the "Save" button to open a file. But at least it gets the job done. This problem has occurred often in the past. I never really came up with a good trick until now.
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: Grahame Grieve on October 10, 2021, 05:10:39 am
is this under the debugger? on windows? or other platforms?
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 10, 2021, 05:52:44 am
It happens when both debug and release builds are run from Lazarus. It happens when the release build is run from the command line. I haven't run the debug build from the command line. I don't know the behavior on Windows. On Linux (Manjaro) the "busy" cursor (circling arrows) is shown whenever the mouse is over the dialog. All other applications work fine. The dialog is just maxing out a single core. 100% CPU shown when you run top. 23% CPU shown by Psensor. The temperature of one core is 71C, the other cores approx 30C.
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: dbannon on October 10, 2021, 06:37:44 am
30,000 files in that directory ?  It sounds to me like TOpenDialog is trying to populate its list with all those files. Thats a lot of entries.

In an unrelated exercise, I just established that calling FileExists() uses up a few mS on my pretty fast system. Doing it 30,000 times would take at least a minute. So, did you let it run to completion ? When it finished, finally, did the OpenDialog list make sense ?

On the other hand, wonder why SaveDialog is so much faster ?

Davo
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: Martin_fr on October 10, 2021, 02:09:40 pm
Given the diff between open and save dlg, my *guess* would be it collects some info for "preview".
Not sure, does the Linux open dlg has a preview?

Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: winni on October 10, 2021, 04:41:31 pm
Hi!

TOpenDialog has no preview but TOpenPictureDialog has.

But anyway - this is bad design.

All filesystem slow down with an amount of more than 10.000 files in one directory  - even ext2/ext3/ext4.

Create some subdirectories and sort the file alphabetical oder numerical.

Or copy your directory to a RAM-disk

Winni

Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: winni on October 10, 2021, 05:07:57 pm
Hi

There must be something wrong with the OpenDialog:

Reading 30.000 filenames to a stringlist ,takes ~900 ms - not even a second.
Just tested with ext4 on a harddisk. With cleared buffers it is 1200  ms.

So write your own OpenDialog, because there is not much code to improve:

The most is delegated to the widget set.

Winni

Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: Martin_fr on October 10, 2021, 05:50:50 pm
Well you can "valgrind --tool=callgrind" / kcachegrind it.
Ideally with "--instr-atstart=no".

If you do, do on a folder with less items. Code under valgrind runs a lot slower (like 100 times slower...).

You can then see, if there is any work in the LCL, or if all time is spent in gtk.
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 12, 2021, 07:16:14 pm
I think what is going on is the basic elements shared by file managers and file dialogs jump into some noisy busy housekeeping at each and every opportunity. I see this a lot with "nemo" the file manager. Always updating the counting and indexing of files. Here's a test I did to time the Save and Open dialogs. I move the mouse and pick the file so my "response time" is a factor. I increased the number of frames (jpeg images) from 5000 to 49563 and recorded the time that the dialog was alive (with GetTickCount):

Frames   SaveDialog   OpenDialog
  5000          5 sec            7 sec
10000          4 sec            4 sec
15000          3 sec            4 sec
20000          5 sec            4 sec
25000        10 sec            5 sec
30000        11 sec            8 sec
35000        10 sec          10 sec
40000        11 sec            8 sec
45000        11 sec            8 sec
49563        10 sec            9 sec

Because I used the same directory over and over again, the housekeeping was "up to date" and the lower level processes that freeze up the OpenDialog stayed quiet. That's my theory. I ran "valgrind --tool=callgrind" and looked at it in kcachegrind, but the tiny font in kcachegrind was unusable. I parsed the callgrind.out files with grep but nothing interesting jumped out at me. Thanks for all the responses. I'll keep you posted with any further info / progress.
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: engkin on October 12, 2021, 09:48:46 pm
It seems both SaveDialog and OpenDialog are taking long time. Moreover, OpenDialog seems faster with higher number of images.

Would it be possible to share some hardware specs?
Is this HDD or SSD based folder?
Anti-virus software?
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 13, 2021, 03:32:52 am
It seems both SaveDialog and OpenDialog are taking long time. Moreover, OpenDialog seems faster with higher number of images.

Would it be possible to share some hardware specs?
Is this HDD or SSD based folder?
Anti-virus software?
There's some lag when the dialog opens before it will let me click on it. And there's some variable delay as I move the mouse and make my selection. The drive is hardware. No anti-virus. In Windows I could tell the PC to not index certain directories and that would greatly speed up the response in directories that contained temporary video frames. Maybe I can do something similar with Manjaro.

CPU: Quad Core Intel Core i7-7700 (-MT MCP-) speed/min/max: 3309/800/4200 MHz Kernel:  .4.150-1-MANJARO x86_64 Up: 3d 3h 4m  Mem: 4007.8/15904.7 MiB (25.2%) Storage: 1.82 TiB (41.6% used) Procs: 245 Shell: Bash inxi: 3.3.06
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: engkin on October 13, 2021, 07:29:35 am
What GTK version are you using? Major and minor?
What file filter/s you have set for TOpenDialog?

Screenshot of the OpenDialog might be helpful.

PopulateFileAndDirectory in lcl/interfaces/gtk2/gtk2proc.inc
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 24, 2021, 08:16:36 am
OK here's another clue. The problem seems to only happen when the dialog goes into a directory which has a lot of files AND new files are being dumped into it in real time. If I'm receiving a video stream and converting it into frames, and these frames are being dumped into a directory, that directory will hang up the open file dialog.

I think it gets confused and stalled by the constant pouring in of new frame images. I have a similar app based on wxWidgets and its open file dialog just adds the new filenames to the list as they appear in the directory. So basically an "active" directory with tens of thousands of files will stall the open file dialog and trap the user. OTOH wxWidgets allows the user to escape the "active" directory (with tons of files) by clicking on a sub directory or the parent directory (or Cancel).
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: loaded on October 24, 2021, 08:45:31 am
In the window operating system; If you are dealing with multiple files containing Video and Images and you want speed.
-You should turn off the file indexing service. [I think you did this]
-You should turn off thumbnail caching. Thumbs.db should be disabled.
I haven't tried it, but I think it will be useful for your project.
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 24, 2021, 03:19:04 pm
The OS is Linux Manjaro. My short term workaround is to make my streaming frame directory lower in the tree than anything else I might want to work on. That way I won't have to ever "click through" the streaming directory when I'm navigating to somewhere else. If I actually have to do something in the streaming directory I'll just have to wait until the stream ends, which I would normally do anyway. The problem is having the dialog get hung up when it goes into the streaming directory DURING streaming / downloading.
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: MarkMLl on October 25, 2021, 10:34:32 pm
Oh $deity... I'm reminded of some of the filesystem tuning hacks related to turning off atime updates etc.

There is of course an xkcd for this https://xkcd.com/2531/

I wonder: does this depend on the desktop environment? The CPU might not be going to the open dialog(ue) per se, but to desktop code which is monitoring file creation in the background (using fam) and is unhappy that something is trying to read the directory at the same time as it's being updated... does top show anything useful?

MarkMLl
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: dbannon on October 26, 2021, 02:18:07 am
Oh $deity... I'm reminded of some of the filesystem tuning hacks related to turning off atime updates etc.

There is of course an xkcd for this https://xkcd.com/2531/


Indeed turning off atime can speed up a file system that is handing a lot of files. In a previous life, it was a pretty standard thing to do where where each user had their own file system if they used lots of small files. Image processing, life sciences etc. And people almost never miss the information atime provided.

The degree of benefit did vary with the file system in use but as this was remote access, desktop never came into our equations so I cannot comment on that.

And, yes, there is a relevant XKCD, but there is a relevant XKCD for ANY situation.

Davo

Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 27, 2021, 03:39:53 am
The app is called "CustomOps". Right now I'm using ffmpeg to capture two webcams and save the streams into sequences of jpegs. If I navigate the open file dialog into either of the jpeg directories it will freeze. The first "top" is running a debug build from Lazarus. The second "top" is running the release build from the command line.
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: dbannon on October 27, 2021, 04:09:22 am
well, now that fiddling with FS has been mentioned, its quite safe and does not really take away anything you need, to remove atime.  There are other settings that may also help but I would be less enthusiastic if its not a specialist mount for just one purpose.  But, atime is easy and safe, Windows users for example don't have an atime available any way.

Edit, as root, /etc/fstab, your home dir will probably be like -
Code: Pascal  [Select][+][-]
  1. UUID=a748839c-84fb-40cc-9b5b-caa902ab00ad /home    ext4    defaults     0    2

Add 'noatime', ahead of 'defaults', no space between.  Remount, easiest way to do that is reboot if you can.
Code: Pascal  [Select][+][-]
  1. UUID=a748839c-84fb-40cc-9b5b-caa902ab00ad /home    ext4    noatime,defaults    0    2

It will make a difference, I just don't know how big a difference !

Davo
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 27, 2021, 04:11:01 pm
Thanks Davo. I did a "cat" of the file and got the results shown in the attached image. It looks like noatime is already set, but it's AFTER defaults. Do I need to do anything?
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: dbannon on October 28, 2021, 01:01:53 am
No, nothing more to do there I am afraid, that shot has already been fired.

There are some other FS fine tuning things but none of them are without cost nor really offer a lot of potential benefit.

Sorry !

Davo
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 29, 2021, 02:01:35 pm
No, nothing more to do there I am afraid, that shot has already been fired.

There are some other FS fine tuning things but none of them are without cost nor really offer a lot of potential benefit.

Sorry !

Davo
OK Davo - thanks for the suggestion. It's always great to learn new stuff anyway. The fact that the C++ / wxWidgets file dialog doesn't freeze means that the problem is solvable in software. It's a real corner case (large number of files with new files rapidly flowing in). I'll just accommodate the situation by avoiding the need to enter that directory during downloading.
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: MarkMLl on October 29, 2021, 02:15:51 pm
I've just taken a quick look back through the thread: I don't think you've answered the question of what widget set the app is compiled against which implies that you've probably not looked at different ones.

If you're using GTK try Qt, and vice versa.

MarkMLl
Title: Re: TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 31, 2021, 04:54:46 am
I've just taken a quick look back through the thread: I don't think you've answered the question of what widget set the app is compiled against which implies that you've probably not looked at different ones.

If you're using GTK try Qt, and vice versa.

MarkMLl
Going to QT fixed it. I built the libQt5Pas library (libQt5Pas.so -> libQt5Pas.so.1.2.6) in a non-root directory and put the path in the project options. After rebuilding the app and navigating to a directory with 46111 jpegs and two more being added each second the Open File Dialog did not get hung up. It wasn't snappy but that's because there were 46111 files in the directory. Thanks for the tip. Some of the visual stuff looks a little weird - (font sizes, etc) but I'll play around with that  :)
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: dbannon on October 31, 2021, 07:58:09 am
I am pretty sure you would have found libqt5pas1 and libqt5pas-dev in your distro's repository. But good practice ;-)

Davo
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: MarkMLl on October 31, 2021, 09:18:00 am
I am pretty sure you would have found libqt5pas1 and libqt5pas-dev in your distro's repository. But good practice ;-)

I agree, although I've seen Lazarus incautiously advance beyond what was supported by the version being shipped by Debian "Stable".

OP: I think you have enough detail there for a bug report. Even if it turns out to be a widget set rather than an LCL problem, it still needs to be got into the record.

MarkMLl
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 31, 2021, 11:17:00 am
I am pretty sure you would have found libqt5pas1 and libqt5pas-dev in your distro's repository. But good practice ;-)

Davo
Manjaro doesn't seem to have it. The only thing I could find that was close was in this directory:

/usr/lib/lazarus/lcl/interfaces/qt5/cbindings

the contents of which are:
qt5.pas  Qt5Pas.pro  README.TXT  src  TODO

So I just followed the instructions in README.TXT after I copied everything into a non-root directory. I wanted to avoid using "sudo" as much as possible. Also like you said it was probably better to build it on my machine just for "deeper compatibility" for the lack of a better term.

Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: dbannon on October 31, 2021, 11:37:23 am
Manjaro doesn't seem to have it. The only thing I could find that was close was in this directory:
What you have done is fine but if you want to let someone else use your software, you would need to tell them that it depends on libqt5pas, sadly that seems to have a number of different names in different repositories, libqt5pas, libqt5pas1 and surprisingly, at least one repo capitalises the 'P' (from memory).  On Manjaro, I think it uses pacman ? You might need to do some sort of wildcard search....

> Lazarus incautiously advance beyond what was supported by the version being shipped by Debian "Stable".

Not at present, the one you are thinking of was reverted. But there is a risk of it happening again, hard to see a way around it. Debian has a two year life cycle and they will only take 'release' code.

Davo
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: SymbolicFrank on October 31, 2021, 11:38:39 am
I think the OpenDialog (or a library it depends on) processes an OnChange event every time the contents of the directory change. Like the FindFile results getting invalidated.
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 31, 2021, 03:09:29 pm
This also fixed the problem:

Project Options -> Compiler Options -> Additions and Overrides
LCLWidgetType:=gtk3

When I made the initial change to QT5 I forgot to notice what the default was. So I did a git diff on the .lpi file and it showed the QT5 as an addition:
Quote
+      <SharedMatrixOptions Count="1">
+        <Item1 ID="829919207870" Modes="Release" Type="IDEMacro" MacroName="LCLWidgetType" Value="qt5"/>
+      </SharedMatrixOptions>
But it didn't show any subtractions, i.e., it didn't show what got replaced by the QT5. So I can't tell you what the default was. Right now the test directory has over 51000 jpegs in it with two new ones streaming in per second, and the gtk3 Open File dialog is working fine.
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on October 31, 2021, 09:12:12 pm
Update: I thought I'd try to explicitly specify GTK2 as the widget type
Quote
LCLWidgetType:=gtk2
and the original problem (TOpenDialog - 100% CPU - Fan Revs Up) came back. So it looks like it's a problem in GTK2 somewhere. Which I have no real interest in using. So I'm gonna use GTK3 (libgtk-3.so.0 => /usr/lib/libgtk-3.so.0) for a while (the fonts are bigger than with QT5 - may be a system setting) and see how that goes.
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: dbannon on November 01, 2021, 12:10:14 am
Great, I will be very interested to see how you go using GTK3.  You were, almost certainly using GTK2 originally, its the Lazarus default on Linux.

If you revert back to qt5, there is a utility, qt5ct, that you can install using your distros package manager that will let you alter a number of display parameters for qt5 apps. Things like colors, fonts, sizes, styles etc.  Pretty useful if you like fiddling to get exactly what you want.

Davo
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on November 01, 2021, 02:58:11 am
Thanks Davo. I've used QT in the past with Python and C++ and I like it. It seemed to work great. There was one weird thing: the TProgressBar seemed backwards. I have a vertical one in my app that's supposed to fill up like a thermometer. It was upside down, like gravity was up. So as it showed "progress" the bar descended from its origin at the top. But I can deal with stuff like that.

I was curious about the C++ / wxWidgets app (very similar functions) that I built last week to test / compare. It behaved normally as I described in a previous post:
Quote
ldd $(which CppOps) | grep gtk
   libwx_gtk3u_xrc-3.1.so.5 => /notroot/WXWIDGETS/build/lib/libwx_gtk3u_xrc-3.1.so.5 (0x00007fd3a9570000)
   libwx_gtk3u_html-3.1.so.5 => /notroot/WXWIDGETS/build/lib/libwx_gtk3u_html-3.1.so.5 (0x00007fd3a9496000)
   libwx_gtk3u_qa-3.1.so.5 => /notroot/WXWIDGETS/build/lib/libwx_gtk3u_qa-3.1.so.5 (0x00007fd3a9466000)
   libwx_gtk3u_core-3.1.so.5 => /notroot/WXWIDGETS/build/lib/libwx_gtk3u_core-3.1.so.5 (0x00007fd3a8c05000)
   libgtk-3.so.0 => /usr/lib/libgtk-3.so.0 (0x00007fd3a2d4e000)
It looks like its default was GTK3. Thanks everyone for all the help - I've learned a lot  :)
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: dbannon on November 01, 2021, 05:38:48 am
There was one weird thing: the TProgressBar seemed backwards. I have a vertical one in my app that's supposed to fill up like a thermometer. It was upside down, like gravity was up. So as it showed "progress" the bar descended from its origin at the top.

I probably live on the opposite side of the world than you, so we are used to things being upside down.


Seriously, if you can replicate that issue in a very small app, I'd log it as a bug, zeljko, who looks after Qt5 is very helpful.

Davo
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: del on November 05, 2021, 05:27:24 am
Great, I will be very interested to see how you go using GTK3.
I'm glad you asked. Not so good. GTK3 fixes the original problem. But I can't use TSaveDialog because it segfaults on create

Quote
[FORMS.PP] ExceptionOccurred
  Sender=EAccessViolation
  Exception=Access violation
  Stack trace:
  $00007F95BAF38BC4

That's with release build run from the command line. If I run it from Lazarus in the debug mode I get this:
Quote
Project CustomOps raised exception class 'External: SIGSEGV'.

 In file 'gtk3/gtk3wsdialogs.pp' at line 1279:
CreateOpenDialogHistory(OpenDialog, PGtkWidget(FileSelWidget));
I get the impression that the bindings for GTK3 have been stagnant for several years. Here is the library on my machine
Quote
libgtk-3.so.0 => /usr/lib/libgtk-3.so.0 (0x00007fb1ec884000)
libgdk-3.so.0 => /usr/lib/libgdk-3.so.0 (0x00007fb1ec790000)
Maybe the bindings are mature and GTK3 is stable and this is just an easy bug. Or it may be the first worm in a can of worms. So for now I'm switching to QT5. :)
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: dbannon on November 05, 2021, 05:55:11 am
...... So for now I'm switching to QT5. :)

Good move.

Davo
Title: Re: [SOLVED] TOpenDialog - 100% CPU - Fan Revs Up
Post by: MarkMLl on November 05, 2021, 09:56:00 am
Maybe the bindings are mature and GTK3 is stable and this is just an easy bug. Or it may be the first worm in a can of worms. So for now I'm switching to QT5. :)

I didn't want to comment on this aspect lest it looked like a complaint.

By and large, distreaux are trying to move away from gtk2 onto gtk3.

Lazarus/LCL is still slightly... immature as far as gtk3 is concerned. I don't know why, but in the past the Gnome (etc.) developers have been criticised for gratuitously breaking the compatibility of their APIs and if this is still an issue it is tempting to assume that it makes life extremely difficult for large-scale projects such as the LCL components.

MarkMLl
TinyPortal © 2005-2018