FindFirst is very slow

[ttomas]

TStringGrid is usable only for small data sets. In your case I will separate Data (Model) and View. Create TList or array of records with FindFirst/Next and use Virtual String Tree to Display data.
Any change or Adding row in TStringGrid trigger invalidation (RePaint) the control and slow down process. Try to surround loop with BeginUpdate/EndUpdate.
You never mention your network share OS. Windows OS NTFS and FAT32 have a problem with too many files in same folder, better use some subfolder structure for too many files.
For best performance I will create some service on server and move FindFirst/Next on server side.

[Zvoni]

I'm going to take another approach to it:
1) Write a service/app/program, which runs on that File-Server itself (process space), monitoring that share (which on the Server is a simple Directory) writing any changes to a local database (e.g. SQLite, stored in the root of the Share)
See --> https://forum.lazarus.freepascal.org/index.php?topic=37371.0
2) Instead of searching (FindFirst/FindNext/FindAllFiles) you pull the information from the database (sanity-checks optional --> FileExists)

Thoughts?

[ttomas]

You can also try Double Commander (Lazarus Open Source Project)
https://sourceforge.net/projects/doublecmd/files/
Just to test where is the problem in your code or in Network share.

[Zvoni]

Searching/traversing a Folder-Structure for 700K files is definitely a bottleneck, the issues with the StringGrid not withstanding as explained earlier by the others.

So, i'd rather ask myself, if the files on that share are "static" (as in: they don't change a lot).
If yes, i'd rather go for a "static snapshot", in my case a database. The only thing i don't know would be performance pulling information for 700K records from the database through a Network-connection.
OTOH, with that approach i could use that "pulling chunks of 50 records per call"-feature or whatever it's called.
Pushing 700K records into a StringGrid......out of curiousity: how big a screen would you have to have to display all rows at the same time? *ggg*

[winni]

Hi!

Remember: A harddisk is still a mechanic machine. Sit back in amazment about the speed.

About your idea with a DB and then check with fileExists:

A fileExists needs around 0.0015 seconds on a local external USB HD.
That seems not much.

But if you do a FileExist on one million files it will take 25 minutes.
Linux with clean buffers.

Winni

[Zvoni]

Hi!

Remember: A harddisk is still a mechanic machine. Sit back in amazment about the speed.

About your idea with a DB and then check with fileExists:

A fileExists needs around 0.0015 seconds on a local external USB HD.
That seems not much.

But if you do a FileExist on one million files it will take 25 minutes.
Linux with clean buffers.

Winni

You misunderstood: The Sanity-Check (FileExists) is optional or do it only (!) if you select a Row/File in the Grid.
The actual House-Keeping of the DB would have to come from that Service/App running on the Server itself monitoring the changes in the Filesystem for that particular Folder

[BobDog]

quote Zvoni
"One System to rule them all . . ."
Here is one system:

Code: Pascal  [Select][+][-]

1.   program finder;
2.   function  system(s:pchar):integer ; cdecl external 'msvcrt.dll' name 'system';
3.  
4.  
5. procedure find(path:ansistring;mask:ansistring);
6. begin
7. system('title  "Done" will show when search is finished');
8. system (pchar('where /R  '+path + ' '+mask));// *.pas'));
9. writeln('"Done"');
10. end;
11.  
12. begin
13. find('C:\windows','notepad');
14. writeln('Another search');
15. find('C:\windows','*.bat *.sys');
16. readln;
17. end.