Recent

Author Topic: [UPDATE] TOpenDialog - 100% CPU - Fan Revs Up  (Read 17005 times)

del

  • Sr. Member
  • ****
  • Posts: 258
[UPDATE] TOpenDialog - 100% CPU - Fan Revs Up
« on: October 07, 2021, 01:25:33 pm »
Lazarus 2.0.12 / FPC 3.2.2. Linux Manjaro. When I go to open an image file in a directory with 30000 video frames the program stalls, the fan revs up, and the CPU usage hits 100%. I suspect one of the widgets used by Lazarus is becoming unnecessarily busy. My workaround is to change the type of dialog from TOpenDialog to TSaveDialog. This solves the problem. It's weird having to click the "Save" button to open a file. But at least it gets the job done. This problem has occurred often in the past. I never really came up with a good trick until now.
« Last Edit: June 02, 2022, 09:57:04 pm by del »

Grahame Grieve

  • Sr. Member
  • ****
  • Posts: 365
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #1 on: October 10, 2021, 05:10:39 am »
is this under the debugger? on windows? or other platforms?

del

  • Sr. Member
  • ****
  • Posts: 258
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #2 on: October 10, 2021, 05:52:44 am »
It happens when both debug and release builds are run from Lazarus. It happens when the release build is run from the command line. I haven't run the debug build from the command line. I don't know the behavior on Windows. On Linux (Manjaro) the "busy" cursor (circling arrows) is shown whenever the mouse is over the dialog. All other applications work fine. The dialog is just maxing out a single core. 100% CPU shown when you run top. 23% CPU shown by Psensor. The temperature of one core is 71C, the other cores approx 30C.
« Last Edit: October 10, 2021, 05:54:16 am by del »

dbannon

  • Hero Member
  • *****
  • Posts: 2786
    • tomboy-ng, a rewrite of the classic Tomboy
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #3 on: October 10, 2021, 06:37:44 am »
30,000 files in that directory ?  It sounds to me like TOpenDialog is trying to populate its list with all those files. Thats a lot of entries.

In an unrelated exercise, I just established that calling FileExists() uses up a few mS on my pretty fast system. Doing it 30,000 times would take at least a minute. So, did you let it run to completion ? When it finished, finally, did the OpenDialog list make sense ?

On the other hand, wonder why SaveDialog is so much faster ?

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9792
  • Debugger - SynEdit - and more
    • wiki
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #4 on: October 10, 2021, 02:09:40 pm »
Given the diff between open and save dlg, my *guess* would be it collects some info for "preview".
Not sure, does the Linux open dlg has a preview?


winni

  • Hero Member
  • *****
  • Posts: 3197
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #5 on: October 10, 2021, 04:41:31 pm »
Hi!

TOpenDialog has no preview but TOpenPictureDialog has.

But anyway - this is bad design.

All filesystem slow down with an amount of more than 10.000 files in one directory  - even ext2/ext3/ext4.

Create some subdirectories and sort the file alphabetical oder numerical.

Or copy your directory to a RAM-disk

Winni

« Last Edit: October 10, 2021, 04:43:13 pm by winni »

winni

  • Hero Member
  • *****
  • Posts: 3197
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #6 on: October 10, 2021, 05:07:57 pm »
Hi

There must be something wrong with the OpenDialog:

Reading 30.000 filenames to a stringlist ,takes ~900 ms - not even a second.
Just tested with ext4 on a harddisk. With cleared buffers it is 1200  ms.

So write your own OpenDialog, because there is not much code to improve:

The most is delegated to the widget set.

Winni


Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9792
  • Debugger - SynEdit - and more
    • wiki
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #7 on: October 10, 2021, 05:50:50 pm »
Well you can "valgrind --tool=callgrind" / kcachegrind it.
Ideally with "--instr-atstart=no".

If you do, do on a folder with less items. Code under valgrind runs a lot slower (like 100 times slower...).

You can then see, if there is any work in the LCL, or if all time is spent in gtk.

del

  • Sr. Member
  • ****
  • Posts: 258
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #8 on: October 12, 2021, 07:16:14 pm »
I think what is going on is the basic elements shared by file managers and file dialogs jump into some noisy busy housekeeping at each and every opportunity. I see this a lot with "nemo" the file manager. Always updating the counting and indexing of files. Here's a test I did to time the Save and Open dialogs. I move the mouse and pick the file so my "response time" is a factor. I increased the number of frames (jpeg images) from 5000 to 49563 and recorded the time that the dialog was alive (with GetTickCount):

Frames   SaveDialog   OpenDialog
  5000          5 sec            7 sec
10000          4 sec            4 sec
15000          3 sec            4 sec
20000          5 sec            4 sec
25000        10 sec            5 sec
30000        11 sec            8 sec
35000        10 sec          10 sec
40000        11 sec            8 sec
45000        11 sec            8 sec
49563        10 sec            9 sec

Because I used the same directory over and over again, the housekeeping was "up to date" and the lower level processes that freeze up the OpenDialog stayed quiet. That's my theory. I ran "valgrind --tool=callgrind" and looked at it in kcachegrind, but the tiny font in kcachegrind was unusable. I parsed the callgrind.out files with grep but nothing interesting jumped out at me. Thanks for all the responses. I'll keep you posted with any further info / progress.
« Last Edit: October 12, 2021, 07:18:00 pm by del »

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #9 on: October 12, 2021, 09:48:46 pm »
It seems both SaveDialog and OpenDialog are taking long time. Moreover, OpenDialog seems faster with higher number of images.

Would it be possible to share some hardware specs?
Is this HDD or SSD based folder?
Anti-virus software?

del

  • Sr. Member
  • ****
  • Posts: 258
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #10 on: October 13, 2021, 03:32:52 am »
It seems both SaveDialog and OpenDialog are taking long time. Moreover, OpenDialog seems faster with higher number of images.

Would it be possible to share some hardware specs?
Is this HDD or SSD based folder?
Anti-virus software?
There's some lag when the dialog opens before it will let me click on it. And there's some variable delay as I move the mouse and make my selection. The drive is hardware. No anti-virus. In Windows I could tell the PC to not index certain directories and that would greatly speed up the response in directories that contained temporary video frames. Maybe I can do something similar with Manjaro.

CPU: Quad Core Intel Core i7-7700 (-MT MCP-) speed/min/max: 3309/800/4200 MHz Kernel:  .4.150-1-MANJARO x86_64 Up: 3d 3h 4m  Mem: 4007.8/15904.7 MiB (25.2%) Storage: 1.82 TiB (41.6% used) Procs: 245 Shell: Bash inxi: 3.3.06

engkin

  • Hero Member
  • *****
  • Posts: 3112
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #11 on: October 13, 2021, 07:29:35 am »
What GTK version are you using? Major and minor?
What file filter/s you have set for TOpenDialog?

Screenshot of the OpenDialog might be helpful.

PopulateFileAndDirectory in lcl/interfaces/gtk2/gtk2proc.inc
« Last Edit: October 13, 2021, 09:01:10 am by engkin »

del

  • Sr. Member
  • ****
  • Posts: 258
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #12 on: October 24, 2021, 08:16:36 am »
OK here's another clue. The problem seems to only happen when the dialog goes into a directory which has a lot of files AND new files are being dumped into it in real time. If I'm receiving a video stream and converting it into frames, and these frames are being dumped into a directory, that directory will hang up the open file dialog.

I think it gets confused and stalled by the constant pouring in of new frame images. I have a similar app based on wxWidgets and its open file dialog just adds the new filenames to the list as they appear in the directory. So basically an "active" directory with tens of thousands of files will stall the open file dialog and trap the user. OTOH wxWidgets allows the user to escape the "active" directory (with tons of files) by clicking on a sub directory or the parent directory (or Cancel).

loaded

  • Hero Member
  • *****
  • Posts: 824
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #13 on: October 24, 2021, 08:45:31 am »
In the window operating system; If you are dealing with multiple files containing Video and Images and you want speed.
-You should turn off the file indexing service. [I think you did this]
-You should turn off thumbnail caching. Thumbs.db should be disabled.
I haven't tried it, but I think it will be useful for your project.
Check out  loaded on Strava
https://www.strava.com/athletes/109391137

del

  • Sr. Member
  • ****
  • Posts: 258
Re: TOpenDialog - 100% CPU - Fan Revs Up
« Reply #14 on: October 24, 2021, 03:19:04 pm »
The OS is Linux Manjaro. My short term workaround is to make my streaming frame directory lower in the tree than anything else I might want to work on. That way I won't have to ever "click through" the streaming directory when I'm navigating to somewhere else. If I actually have to do something in the streaming directory I'll just have to wait until the stream ends, which I would normally do anyway. The problem is having the dialog get hung up when it goes into the streaming directory DURING streaming / downloading.

 

TinyPortal © 2005-2018