Fast multi threaded file search app

[Tomu]

This find_file app searches for files containing a sub-string. The UI creates a thread to populate a string-list holding all the files which match the mask (.pas|.csv for example). Then a thread pool searches for the target sub-string. Each thread in the pool processes a portion the the string-list holding all the files.

Results a displayed in a grid that can be sorted by

• the number of times the target appears in the file.
• The file name.
• Th File path.
• the sizes of the file.

Double clicking on a file in the results grid loads that file into an editor.

I have only tested it on Linux - would appreciate hearing if anyone complies it on Windows or macOS.
It was developed by myself and a friend named Gemini.

Source at : https://github.com/motregnu/find_file

[LemonParty]

Do you compare a single thread approach with multithread?
Do you use a SIMD code to speed up search?

[Tomu]

I didn't formally profile a single threaded version with the multi-threaded version. However, the difference for searching tens of thousands of files was dramatic enough at a human perception level. I had a UI + single TThread app working and asked Gemini to convert it to be multi-threaded.

I'm not familiar with SIMD code - so I guess I didn't use one.

Below is the code that kicks off the thread pool after the list of target files has been built.

Code: Pascal  [Select][+][-]

1. procedure TFileSearchThread.Execute;
2. var
3.   Searcher: TFileSearcher;
4. begin
5.   Searcher := TFileSearcher.Create;
6.   try
7.     Searcher.FollowSymLink := False;
8.     Searcher.OnFileFound := @FileFoundHandler;
9.  
10.     // 1. Sequentially scan the directory tree to build the list
11.     Searcher.Search(FDirectory, '*', True);
12.  
13.     if Terminated or (FCollectedFiles.Count = 0) then Exit;
14.  
15.     // 2. Distribute the collected files across the CPU thread pool
16.     ProcThreadPool.DoParallel(
17.       @ParallelProcessFile,
18.       0,
19.       FCollectedFiles.Count - 1,FCollectedFiles);
20.  
21.   finally
22.     Searcher.Free;
23.   end;
24. end;          
25.  

[LemonParty]

There is a thought that main limiting factor for this kind of search is the speed of disk you searching in. That is mean that spawning additional threads will not increase total performance a lot. The opposite situation is when you do search on two or more disk at the same time. In this situation you will probably receive a significant speed up from multithread approach. But to check this I recommend to do a benchmark.

[cdbc]

Hi
Benchmarks -- BAH!
Honestly, @Tomu's app is plenty fast, even on my old spinner HD 
...And you should look at the architecture: He's got 1 thread accumulating a list of files and then multiple threads in a pool split up the list and process it in parallel, with only the  necessary locking in place ::
I'll say: Well Done! Tomu \o/\ö/\o/
Regards Benny