Recent

Author Topic: File Search is there a way to speed up ?  (Read 5453 times)

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: File Search is there a way to speed up ?
« Reply #15 on: March 16, 2018, 08:42:09 pm »
Getmem: my code is much the same in principle, but I keep an ordered (path sorted) list. This makes it easier to dump a list which can be used which files (in the archive) belong together (same dir)

One more crucial difference performance wise is that I only hash (also md5) if there are multiple files with the same size.

In my case, one of the two dirs is always the same, so the info on files that were md5'ed in that dir are persistent.

The tool is mostly to deduplicate decommissioned harddisks to save time before checking what needs to be saved, and the million files (+/- 500GB) archive dir contains all known files.
« Last Edit: March 16, 2018, 08:55:31 pm by marcov »

balazsszekely

  • Guest
Re: File Search is there a way to speed up ?
« Reply #16 on: March 16, 2018, 09:09:24 pm »
Quote
One more crucial difference performance wise is that I only hash (also md5) if there are multiple files with the same size.
That's a good idea. This way you skip a few unnecessary steps.

ASerge

  • Hero Member
  • *****
  • Posts: 2223
Re: File Search is there a way to speed up ?
« Reply #17 on: March 16, 2018, 09:55:22 pm »
@ASerge All I need is to get file paths
Found that if use WinApi, then the search will be slightly faster. This is achieved by excluding the extraction of short names (FindExInfoBasic) and a large cache (FIND_FIRST_EX_LARGE_FETCH, although it seems to me insignificant).
Test project included.

 

TinyPortal © 2005-2018