Recent

Author Topic: TOpenDialog slow with 82000 files  (Read 2637 times)

ChristianundCo

  • New Member
  • *
  • Posts: 11
TOpenDialog slow with 82000 files
« on: August 14, 2020, 12:15:58 pm »
I have a problem with the TOpendialog. I've got a folder with 82000 files. I want to multiselect them all with STRG-A. With 1000 files everything is fine.
But with 82000 files the opendialog is very slow. It takes minutes until it reacts to my STRG+A command and mark all files.
I'm not at IDE. Even when the exe runs without debugger it's very slow. (minutes)
Windows Explorer has nearly no problem to show me the 82k-files-folder. OK, a little bit slower but no minutes to wait. Is it possible to make "opendialog" faster in lazarus or are 82000 files too many?

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9792
  • Debugger - SynEdit - and more
    • wiki
Re: TOpenDialog slow with 82000 files
« Reply #1 on: August 14, 2020, 12:42:57 pm »
If you use Notepad++ (or any app that allows you to select all files in its open dlg), how long does it take?

As I understand this is just selecting them? I.e., waiting till they all are highlighted?  This is NOT actually pressing the "open" button?

Therefore you can try in any app that allows multiselect-open and *.* ?

---
The most files I had in any folder where 10.000.
They can be selected in 4 seconds. And that is the same time in an LCL app and Notepad++.  (On Win64 / i9-8600K)

I have no idea if time increases linear or not, if the file-count increases.




ChristianundCo

  • New Member
  • *
  • Posts: 11
Re: TOpenDialog slow with 82000 files
« Reply #2 on: August 14, 2020, 01:09:01 pm »
Yes, I want just select them and this lasts so long.
But you are right, I tried VLC-Player and MPC-HC-Player both need minutes too and are very slow when I choose this 82k-folder.

OK, I will go another way. I let the user just open one directory (TSelectDirectoryDialog) with all files. No multiselect anymore with TOpendialog.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: TOpenDialog slow with 82000 files
« Reply #3 on: August 14, 2020, 01:23:21 pm »
(is opendialog using the same settings as explorer? You might try to tweak its properties to avoid showing previews, thumbnails etc)

440bx

  • Hero Member
  • *****
  • Posts: 3944
Re: TOpenDialog slow with 82000 files
« Reply #4 on: August 14, 2020, 01:29:28 pm »
I have no idea if time increases linear or not, if the file-count increases.
I can answer that question.  The answer is a definite _yes_ without exceptions.

In addition to the number of files increasing the time it takes to get a list of files, the time _also_ increases with the folder nesting level. 

Listing the same number of files in a folder at level 0 (root) will be faster than listing the files in a folder at level 3 (for instance) and the time required also increases as the nesting level increases.

Regarding Marcov's suggestion, the way the files are listed might make a difference (I don't know if it does or not, I always use "details" for all folders) but, even just asking for the "plain" (details - a simple file list) still results in long wait times when the number of files gets around 10,000

HTH.


(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9792
  • Debugger - SynEdit - and more
    • wiki
Re: TOpenDialog slow with 82000 files
« Reply #5 on: August 14, 2020, 01:36:07 pm »
My guess: When you "select all" displayed files, the OS generates a long string "file1, file2, file3..." and puts it into the filename field of the dialog.

Unless the OS is smart enough to pre-alloc the memory for concatenated list of the 80000 names..., well that is going to take a long time. And each name added takes more time to add, than the name before it.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: TOpenDialog slow with 82000 files
« Reply #6 on: August 14, 2020, 02:07:36 pm »
Regarding Marcov's suggestion, the way the files are listed might make a difference (I don't know if it does or not, I always use "details" for all folders) but, even just asking for the "plain" (details - a simple file list) still results in long wait times when the number of files gets around 10,000

Details, turning off in explorer properties anything related to preview (anything metadata loading related) and thumbnailing.

I make vision systems for a living and have had directories with 1M BMP files in them. Undoable.

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: TOpenDialog slow with 82000 files
« Reply #7 on: August 14, 2020, 03:38:26 pm »
Listing the same number of files in a folder at level 0 (root) will be faster than listing the files in a folder at level 3 (for instance) and the time required also increases as the nesting level increases.

Uhm... why should this be different? Assuming I have a reference to the root directory and some sub directory then listing files inside them will take the same time (if they contain the same amount of files). It would only be different if the root directory is handled differently by the filesystem than other directories (this is for example the case in FAT12).

ChristianundCo

  • New Member
  • *
  • Posts: 11
Re: TOpenDialog slow with 82000 files
« Reply #8 on: August 14, 2020, 04:53:55 pm »
Anyway, I let the user choose only the directory. Then I find all files inside with "FindFirst"-> "FindNext"

and then I sort the filelist with naturalsort.

That's all I wanted...

So the user don't need to wait minutes.

(to disable all thumbnails and previews details didn't help, files are simple too much )

440bx

  • Hero Member
  • *****
  • Posts: 3944
Re: TOpenDialog slow with 82000 files
« Reply #9 on: August 14, 2020, 10:35:23 pm »
Listing the same number of files in a folder at level 0 (root) will be faster than listing the files in a folder at level 3 (for instance) and the time required also increases as the nesting level increases.

Uhm... why should this be different?
I don't know the reason why it's different but, at one time, I tested that because I had performance problems related to accessing files and, directory depth was one of the factors in performance degradation.
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: TOpenDialog slow with 82000 files
« Reply #10 on: August 14, 2020, 10:55:05 pm »
It might have to do with fragmentation avoidance of NTFS, that it initially sorts the rootdirs over various parts of the disk.

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: TOpenDialog slow with 82000 files
« Reply #11 on: August 16, 2020, 02:06:01 pm »
Fragmentation could indeed be a possibility with the contents of the root directory likely being placed sequentially (especially if there isn't much change) compared to a deeper directory. Even on a SSD that will result in multiple read commands while a sequential one could be done with a single read.
But simply from the filesystem's point of view there isn't any difference between the root directory and deeper directories (at least on filesystems commonly used today).

 

TinyPortal © 2005-2018