Recent

Author Topic: FindAllDirectories is slow  (Read 14069 times)

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4541
  • I like bugs.
FindAllDirectories is slow
« on: November 18, 2012, 03:28:40 pm »
Hi

LazUtils, unit FileUtil has function FindAllDirectories.
It uses the same idea as FindAllFiles, using TFileSearcher internally. Now only it overrides the DoDirectoryFound method.

I realized it must be slow. TFileSearcher searches also all files which it useless if you only want directories.
I am not using the function so this is not "my itch" and I am not planning to fix it now. I have other itches to scratch.

If someone else wants to fix it, I will happily apply the patch. It is a simple and confined task, not many dependencies like in the IDE's code. It can be very rewarding (if you like to optimize code) as the speedup will be significant.
Ideas can be taken from my r39299 in Lazarus trunk in which I implemented a thread to scan and search for FPC sources, or then just by improvising.

Juha
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Bart

  • Hero Member
  • *****
  • Posts: 5465
    • Bart en Mariska's Webstek
Re: FindAllDirectories is slow
« Reply #1 on: November 18, 2012, 05:52:54 pm »
Make DoSearch a protected function and override that?
Then you use faDirectory instead of faAnyfile for search attribute.

Bart

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4541
  • I like bugs.
Re: FindAllDirectories is slow
« Reply #2 on: November 18, 2012, 06:04:36 pm »
Make DoSearch a protected function and override that?
Then you use faDirectory instead of faAnyfile for search attribute.

Yes, faDirectory must be used for sure. The rest of the implementation is a matter of taste. It could use an overridden DoSearch or just a simple custom code without classes.

Juha
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Bart

  • Hero Member
  • *****
  • Posts: 5465
    • Bart en Mariska's Webstek
Re: FindAllDirectories is slow
« Reply #3 on: November 19, 2012, 01:38:01 pm »
I'll give it a try when I have time.
I'll post it in Mantis when I'm ready (no commit rights).

Bart

Bart

  • Hero Member
  • *****
  • Posts: 5465
    • Bart en Mariska's Webstek
Re: FindAllDirectories is slow
« Reply #4 on: November 19, 2012, 10:13:15 pm »
Observations:
1.
Using faDirectory as Attribute instead of faAnyfile, at least on Windows, won't speed up things.
It still lists all files too.

2.
I noticed TMaskList is used.
This is case-insensitive.
So on Linux 'abc*' will also return ABC.txt, which is rather un-linux like, but then again may be usefull when searching somethig like '*.jpg'

It would IMO be a good idea to extend TMask and TMaskList to have a case-sensitive option.
Code: [Select]
  TMask = class
  private
    FMask: TMaskString;
  public
    constructor Create(const AValue: String; const CaseSensitive: Boolean = False);
    destructor Destroy; override;
   
    function Matches(const AFileName: String): Boolean;
  end;

In my own version of my filesearcher I already implemented such.

Bart

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4541
  • I like bugs.
Re: FindAllDirectories is slow
« Reply #5 on: November 19, 2012, 11:20:09 pm »
1.
Using faDirectory as Attribute instead of faAnyfile, at least on Windows, won't speed up things.
It still lists all files too.

You are right. Same thing on Linux. Is it a bug?
In fact I introduced a bug when I "optimized" TFileSearcher.Search by removing a test for faDirectory. It worked in my tests but DoDirectoryEnter was apparently called wrongly. I fixed it in r39312.

It means that FindAllDirectories will not gain so big speedup after all. :(

Quote
2.
I noticed TMaskList is used.
This is case-insensitive.
So on Linux 'abc*' will also return ABC.txt, which is rather un-linux like, but then again may be usefull when searching somethig like '*.jpg'
[...]
It would IMO be a good idea to extend TMask and TMaskList to have a case-sensitive option.
In my own version of my filesearcher I already implemented such.

Looks good. It is even backwards compatible.
You didn't get commit rights to LazUtils yet... I will remind other devels, it would make things easier.

Juha
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Bart

  • Hero Member
  • *****
  • Posts: 5465
    • Bart en Mariska's Webstek
Re: FindAllDirectories is slow
« Reply #6 on: November 20, 2012, 10:33:25 am »
You are right. Same thing on Linux. Is it a bug?

Under Windows it is not. It is just how the underlying API works AFAIK.
Makes sense too. The OS will have to iterate the internal FS structures to find out what is what, I guess.

Looks good. It is even backwards compatible.

I can post a patch in bugtracker.
I think it should be reviewed by other devels before committing.

Bart

poiuyt555

  • Jr. Member
  • **
  • Posts: 91
Re: FindAllDirectories is slow
« Reply #7 on: February 16, 2013, 01:35:12 pm »
In what release i can search for directories usin FindAllFiles or TFileSearcher?
Is it already in Lazarus functionality?

Leledumbo

  • Hero Member
  • *****
  • Posts: 8774
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: FindAllDirectories is slow
« Reply #8 on: February 16, 2013, 04:45:30 pm »
Quote
In what release i can search for directories usin FindAllFiles or TFileSearcher?
0.9.X onwards (not sure about the X, probably 24)
Quote
Is it already in Lazarus functionality?
Since a long time

wiki:
API:

poiuyt555

  • Jr. Member
  • **
  • Posts: 91
Re: FindAllDirectories is slow
« Reply #9 on: February 18, 2013, 05:46:46 am »
i mean: version with changes that can search for dirs (as i understand in this topic this discussed).?
Or optimized ver, which not check faDirectory?
Now findAllfiles search only for files, but not dirs:
OnDirectoryFound event occurs when any dir found, not applying the filter masks like for files.

Or how correct modify this function that it could search for dirs?

 

TinyPortal © 2005-2018