Recent

Author Topic: List all files on a DRIVE efficiently  (Read 2127 times)

Gizmo

  • Hero Member
  • *****
  • Posts: 831
List all files on a DRIVE efficiently
« on: October 18, 2018, 09:41:34 pm »
Hi

I am currently using FindAllFiles to generate a StringList of all files in the users chosen folder for OSX. If the user chooses the actual root of the drive (/ in OSX) my program seems to be struggling. It may be that FindAllFiles is caught in an infinite loop and that may be why the SL is over 2Gb of memory used so far. I am guessing the generated StringList that FindAllFiles creates is just getting overwhelmed with the number of path and filename entries in the SL.

Is there a more efficient, memory sparing way of generating an entire list of paths and filenames on a drive?

Thanks

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11351
  • FPC developer.
Re: List all files on a DRIVE efficiently
« Reply #1 on: October 18, 2018, 10:13:56 pm »
Is your tstringlist normal or sorted?

Gizmo

  • Hero Member
  • *****
  • Posts: 831
Re: List all files on a DRIVE efficiently
« Reply #2 on: October 18, 2018, 10:53:18 pm »
Normal (default create action and then FindAllFiles populates it)

440bx

  • Hero Member
  • *****
  • Posts: 3921
Re: List all files on a DRIVE efficiently
« Reply #3 on: October 18, 2018, 11:03:36 pm »
I suggest you first determine with certainty where the problem is taking place.  You are currently assuming that the problem is in how a StringList manages its strings (which is reasonable but, not established as a fact).  It is possible that the problem is in the routine that lists all the files.

I would first test the routine that finds all the files by having it find them but, not add them into any list.  If that works fast and correctly then, I'd add the strings to a StringList and see what happens in that case. 

That way, you'll know for sure the source of the problem and knowing that will help you find a solution that is not based on a guess (even if the guess is a good one.)

HTH.
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

sash

  • Sr. Member
  • ****
  • Posts: 366
Re: List all files on a DRIVE efficiently
« Reply #4 on: October 19, 2018, 12:01:42 am »
Is there a more efficient, memory sparing way of generating an entire list of paths and filenames on a drive?

Probably you need to figure out what goes wrong first (It could be anything: like recursive symlinks, wrong permissions, or even unexpected behavior under the hood of FindAllFiles' defaults). Try TFileSearcher to display a progress of searching to figure out.

p.s. I'm curious why one should query all files from root mount point (/). In the unix-like filesystems there is a concept "everything is a file" - in the sense that everything (incl. disks, devices, memory, ipc, other virtual filesystems) is addressable as filename, but not everything is actually a file. Thus, regarding your title, "/" <> DRIVE.
« Last Edit: October 19, 2018, 12:04:18 am by sash »
Lazarus 2.0.10 FPC 3.2.0 x86_64-linux-gtk2 @ Ubuntu 20.04 XFCE

Coldzer0

  • Jr. Member
  • **
  • Posts: 50
Re: List all files on a DRIVE efficiently
« Reply #5 on: October 21, 2018, 06:33:33 am »
this code was part of a file network share i made in 2016

the code will work in almost any os FPC support

i tested it on Windows , linux , Mac , Android - maybe it will work on other OSs too .

the output is JSON so it can be easily parsed with a lot of tools or programming langs

here's an example (json_pp is json prettier )

FT is the File Type 
0 = Folder
1 = File

FS = File Size
FN = File Name


Code: Pascal  [Select][+][-]
  1. Coldzer0 @ list files $./ls /Users/Coldzer0/FPC/list\ files/ | json_pp
  2. {
  3.    "Count" : 11,
  4.    "MainPath" : "/Users/Coldzer0/FPC/list files/",
  5.    "Files" : [
  6.       {
  7.          "FN" : ".DS_Store",
  8.          "FS" : 6148,
  9.          "FT" : 1
  10.       },
  11.       {
  12.          "FT" : 1,
  13.          "FN" : "superobject.pas",
  14.          "FS" : 184227
  15.       },
  16.       {
  17.          "FN" : "filesfunc.pas",
  18.          "FS" : 2756,
  19.          "FT" : 1
  20.       },
  21.       {
  22.          "FS" : 34118,
  23.          "FN" : "list file json.zip",
  24.          "FT" : 1
  25.       },
  26.       {
  27.          "FS" : 204,
  28.          "FN" : "ls.pas",
  29.          "FT" : 1
  30.       },
  31.       {
  32.          "FT" : 1,
  33.          "FN" : "ls.lpi",
  34.          "FS" : 1525
  35.       },
  36.       {
  37.          "FN" : "lib",
  38.          "FS" : 96,
  39.          "FT" : 0
  40.       },
  41.       {
  42.          "FS" : 746,
  43.          "FN" : "ls.lps",
  44.          "FT" : 1
  45.       },
  46.       {
  47.          "FT" : 1,
  48.          "FN" : "ls",
  49.          "FS" : 1417008
  50.       },
  51.       {
  52.          "FT" : 0,
  53.          "FS" : 96,
  54.          "FN" : "backup"
  55.       },
  56.       {
  57.          "FT" : 1,
  58.          "FS" : 519,
  59.          "FN" : "supertypes.pas"
  60.       }
  61.    ]
  62. }
  63. Coldzer0 @ list files  $
  64.  


hope it will help you  ::)

 

TinyPortal © 2005-2018