Recent

Author Topic: FindFirst is very slow  (Read 4030 times)

ttomas

  • Full Member
  • ***
  • Posts: 228
Re: FindFirst is very slow
« Reply #15 on: September 03, 2021, 10:48:11 am »
TStringGrid is usable only for small data sets. In your case I will separate Data (Model) and View. Create TList or array of records with FindFirst/Next and use Virtual String Tree to Display data.
Any change or Adding row in TStringGrid trigger invalidation (RePaint) the control and slow down process. Try to surround loop with BeginUpdate/EndUpdate.
You never mention your network share OS. Windows OS NTFS and FAT32 have a problem with too many files in same folder, better use some subfolder structure for too many files.
For best performance I will create some service on server and move FindFirst/Next on server side.

Zvoni

  • Hero Member
  • *****
  • Posts: 734
Re: FindFirst is very slow
« Reply #16 on: September 03, 2021, 11:36:14 am »
I'm going to take another approach to it:
1) Write a service/app/program, which runs on that File-Server itself (process space), monitoring that share (which on the Server is a simple Directory) writing any changes to a local database (e.g. SQLite, stored in the root of the Share)
See --> https://forum.lazarus.freepascal.org/index.php?topic=37371.0
2) Instead of searching (FindFirst/FindNext/FindAllFiles) you pull the information from the database (sanity-checks optional --> FileExists)

Thoughts?
One System to rule them all, One IDE to find them,
One Code to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
People call me crazy, because i'm jumping out of perfectly fine aircraft

ttomas

  • Full Member
  • ***
  • Posts: 228
Re: FindFirst is very slow
« Reply #17 on: September 03, 2021, 12:28:00 pm »
You can also try Double Commander (Lazarus Open Source Project)
https://sourceforge.net/projects/doublecmd/files/
Just to test where is the problem in your code or in Network share.

Zvoni

  • Hero Member
  • *****
  • Posts: 734
Re: FindFirst is very slow
« Reply #18 on: September 03, 2021, 12:56:50 pm »
Searching/traversing a Folder-Structure for 700K files is definitely a bottleneck, the issues with the StringGrid not withstanding as explained earlier by the others.

So, i'd rather ask myself, if the files on that share are "static" (as in: they don't change a lot).
If yes, i'd rather go for a "static snapshot", in my case a database. The only thing i don't know would be performance pulling information for 700K records from the database through a Network-connection.
OTOH, with that approach i could use that "pulling chunks of 50 records per call"-feature or whatever it's called.
Pushing 700K records into a StringGrid......out of curiousity: how big a screen would you have to have to display all rows at the same time? *ggg*
One System to rule them all, One IDE to find them,
One Code to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
People call me crazy, because i'm jumping out of perfectly fine aircraft

winni

  • Hero Member
  • *****
  • Posts: 2700
Re: FindFirst is very slow
« Reply #19 on: September 03, 2021, 02:31:26 pm »
Hi!

Remember: A harddisk is still a mechanic machine. Sit back in amazment about the speed.

About your idea with a DB and then check with fileExists:

A fileExists needs around 0.0015 seconds on a local external USB HD.
That seems not much.

But if you do a FileExist on one million files it will take 25 minutes.
Linux with clean buffers.

Winni

Zvoni

  • Hero Member
  • *****
  • Posts: 734
Re: FindFirst is very slow
« Reply #20 on: September 03, 2021, 02:46:40 pm »
Hi!

Remember: A harddisk is still a mechanic machine. Sit back in amazment about the speed.

About your idea with a DB and then check with fileExists:

A fileExists needs around 0.0015 seconds on a local external USB HD.
That seems not much.

But if you do a FileExist on one million files it will take 25 minutes.
Linux with clean buffers.

Winni

You misunderstood: The Sanity-Check (FileExists) is optional or do it only (!) if you select a Row/File in the Grid.
The actual House-Keeping of the DB would have to come from that Service/App running on the Server itself monitoring the changes in the Filesystem for that particular Folder
One System to rule them all, One IDE to find them,
One Code to bring them all, and to the Framework bind them,
in the Land of Redmond, where the Windows lie
---------------------------------------------------------------------
People call me crazy, because i'm jumping out of perfectly fine aircraft

BobDog

  • Full Member
  • ***
  • Posts: 140
Re: FindFirst is very slow
« Reply #21 on: September 03, 2021, 04:27:02 pm »

quote Zvoni
"One System to rule them all . . ."
Here is one system:
Code: Pascal  [Select][+][-]
  1.   program finder;
  2.   function  system(s:pchar):integer ; cdecl external 'msvcrt.dll' name 'system';
  3.  
  4.  
  5. procedure find(path:ansistring;mask:ansistring);
  6. begin
  7. system('title  "Done" will show when search is finished');
  8. system (pchar('where /R  '+path + ' '+mask));// *.pas'));
  9. writeln('"Done"');
  10. end;
  11.  
  12. begin
  13. find('C:\windows','notepad');
  14. writeln('Another search');
  15. find('C:\windows','*.bat *.sys');
  16. readln;
  17. end.

 

TinyPortal © 2005-2018