FindAllDirectories is slow

[JuhaManninen]

Hi

LazUtils, unit FileUtil has function FindAllDirectories.
It uses the same idea as FindAllFiles, using TFileSearcher internally. Now only it overrides the DoDirectoryFound method.

I realized it must be slow. TFileSearcher searches also all files which it useless if you only want directories.
I am not using the function so this is not "my itch" and I am not planning to fix it now. I have other itches to scratch.

If someone else wants to fix it, I will happily apply the patch. It is a simple and confined task, not many dependencies like in the IDE's code. It can be very rewarding (if you like to optimize code) as the speedup will be significant.
Ideas can be taken from my r39299 in Lazarus trunk in which I implemented a thread to scan and search for FPC sources, or then just by improvising.

Juha

[Bart]

Make DoSearch a protected function and override that?
Then you use faDirectory instead of faAnyfile for search attribute.

Bart

[JuhaManninen]

Make DoSearch a protected function and override that?
Then you use faDirectory instead of faAnyfile for search attribute.

Yes, faDirectory must be used for sure. The rest of the implementation is a matter of taste. It could use an overridden DoSearch or just a simple custom code without classes.

Juha

[Bart]

I'll give it a try when I have time.
I'll post it in Mantis when I'm ready (no commit rights).

Bart

[Bart]

Observations:
1.
Using faDirectory as Attribute instead of faAnyfile, at least on Windows, won't speed up things.
It still lists all files too.

2.
I noticed TMaskList is used.
This is case-insensitive.
So on Linux 'abc*' will also return ABC.txt, which is rather un-linux like, but then again may be usefull when searching somethig like '*.jpg'

It would IMO be a good idea to extend TMask and TMaskList to have a case-sensitive option.

Code: [Select]

  TMask = class
  private
    FMask: TMaskString;
  public
    constructor Create(const AValue: String; const CaseSensitive: Boolean = False);
    destructor Destroy; override;
   
    function Matches(const AFileName: String): Boolean;
  end;

In my own version of my filesearcher I already implemented such.

Bart

[JuhaManninen]

1.
Using faDirectory as Attribute instead of faAnyfile, at least on Windows, won't speed up things.
It still lists all files too.

You are right. Same thing on Linux. Is it a bug?
In fact I introduced a bug when I "optimized" TFileSearcher.Search by removing a test for faDirectory. It worked in my tests but DoDirectoryEnter was apparently called wrongly. I fixed it in r39312.

It means that FindAllDirectories will not gain so big speedup after all.

2.
I noticed TMaskList is used.
This is case-insensitive.
So on Linux 'abc*' will also return ABC.txt, which is rather un-linux like, but then again may be usefull when searching somethig like '*.jpg'
[...]
It would IMO be a good idea to extend TMask and TMaskList to have a case-sensitive option.
In my own version of my filesearcher I already implemented such.

Looks good. It is even backwards compatible.
You didn't get commit rights to LazUtils yet... I will remind other devels, it would make things easier.

Juha

[Bart]

You are right. Same thing on Linux. Is it a bug?

Under Windows it is not. It is just how the underlying API works AFAIK.
Makes sense too. The OS will have to iterate the internal FS structures to find out what is what, I guess.

Looks good. It is even backwards compatible.

I can post a patch in bugtracker.
I think it should be reviewed by other devels before committing.

Bart

[poiuyt555]

In what release i can search for directories usin FindAllFiles or TFileSearcher?
Is it already in Lazarus functionality?

[Leledumbo]

In what release i can search for directories usin FindAllFiles or TFileSearcher?

0.9.X onwards (not sure about the X, probably 24)

Is it already in Lazarus functionality?

Since a long time

wiki:

• http://wiki.lazarus.freepascal.org/findallfiles

API:

• http://lazarus-ccr.sourceforge.net/docs/lcl/fileutil/findallfiles.html
• http://lazarus-ccr.sourceforge.net/docs/lcl/fileutil/tfilesearcher.html

[poiuyt555]

i mean: version with changes that can search for dirs (as i understand in this topic this discussed).?
Or optimized ver, which not check faDirectory?
Now findAllfiles search only for files, but not dirs:
OnDirectoryFound event occurs when any dir found, not applying the filter masks like for files.

Or how correct modify this function that it could search for dirs?