Lazarus

Announcements => Third party => Topic started by: bastla on April 30, 2013, 05:37:20 pm

Title: Copying an entire directory (incl. subdirectories)
Post by: bastla on April 30, 2013, 05:37:20 pm
Hello together,

I searched for a possiblity to copy a whole directory on every platform supported by Lazarus. Thanks to Leledumbo (http://www.lazarus.freepascal.org/index.php/topic,18740.msg106047.html#msg106047), I found a way to get this working. Maybe there're some guys outside who still search for this, so I publish my work here :D.

I created an simple and easy class, called "TCopyDir", which uses TFileSearcher to enumerate an entire directory and copy its content file by file.

Example Code:
Code: [Select]
procedure TfrmMain.btCopyClick(Sender: TObject);
var
  CopyDir: TCopyDir;
  Log: TStringList;
begin
  CopyDir := TCopyDir.Create('C:\', 'D:\');
  CopyDir.Start;

  Log := CopyDir.GetLog;

  CopyDir.Free;
end;

As you can see, the class is very simple to use and contains a log (TStringList). In this example, the whole content of "C:\" would be copied to "D:\".

[EDIT 2013-06-23]As of 23.06.2013, there's a improved version of TCopyDir (called TCopyDir v2). For more details see here (http://forum.lazarus.freepascal.org/index.php/topic,20759.msg124308.html#msg124308)[/EDIT]
[EDIT 2013-08-10]onivan found a bug in TCopyDir and has already created a fix. Make sure to read his post (http://forum.lazarus.freepascal.org/index.php/topic,20759.msg126897.html#msg126897).[/EDIT]
[EDIT 2014-01-06]Caladan found another bug (http://forum.lazarus.freepascal.org/index.php/topic,20759.msg137601.html#msg137601) in TCopyDir affecting Unix users only. See this post (http://forum.lazarus.freepascal.org/index.php/topic,20759.msg137737.html#msg137737) for a fix![/EDIT]
[EDIT2014-06-03]I uploaded version 2.1 of TCopyDir which fixes two above mentioned bugs, plus I changed the disclaimer![/EDIT]

Download and example application in the attachments.
My work is Free Domain but remain the terms of LCL!

With the use of this work you agree to the following:
THIS SOFTWARE IS PROVIDED BY BASTLA "AS IS" AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL BASTLA BE LIABLE FOR ANY
DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.


I hope this is useful for some of you!

Greetings,
bastla
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: jmpessoa on April 30, 2013, 09:34:03 pm
Thank you bastla!
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: exdatis on April 30, 2013, 10:36:12 pm
Thank you very much!
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: LA.Center on June 21, 2013, 11:00:34 am
Nice - THX a bunch
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: CM630 on June 21, 2013, 02:00:52 pm
I have two questions:
1. How does it handle hidden and system files?
2. Is it tested with files with nonlatin filenames? Edit- I tested, works fine for me.

Edit: I found a bug- I copied a folder, which contained a hidden file. It copied the file, but it the file in the target directory lost its "Hidden" flag.

Edit: yet another bug- in the folder I had two subfolder. One of them had a name, ending with .txt and it was not copied. The other was was copied fine.

Edit: I made a third folder. It does not copy it, too. It does not copy more than 1 subfolder.

I did the tests using the example app. There is a bug in the example app, too- ledSource.ReadOnly is set to TRUE, this way users cannot paste the folder path there, but they have to browse the whole way to the folders.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: bastla on June 22, 2013, 01:11:28 pm
@paskal:
Thank you for testing my work!

I don't know why file attributes aren't copied. This class uses CopyFile-Method, maybe it doesn't support file attributes. I have to check this and add this.

I just tested it again: I had up to 4 subfolders with files in it, all subfolders and their files were copied.
A folder containing a "." was not copied for me, too. Mabye TFileSearcher declares this to be a file, I have to work on it.

The led.Source.ReadOnly = TRUE is conscious set, I just thought allowing users to paste their path would cause error because of wrong file paths etc.

Greetings,
bastla
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: JuhaManninen on June 22, 2013, 04:02:12 pm
There is function CopyDirTree() in LCL FileUtil unit.
Please test it also. It may still have bugs.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: bastla on June 23, 2013, 08:24:54 pm
As I mentioned above, I worked on TCopyDir and I made some improvements!
I redesigned the whole code, it's so safe and stable now! The main things that changed: When copying a directory, first all files and directories will be cached and stored. Then, the directory structure of the source directory will be transferred to the target directory and only then the files will be copied! File attributes are also supported now!

Let's have a look on a sample code:
Code: [Select]
uses
  CopyDir
[...]
procedure TfrmMain.btCopyClick(Sender: TObject);
var
  CopyDir: TCopyDir;
  Log: TStringList;
begin
  CopyDir := TCopyDir.Create('C:\', 'D:\'); // copy content of "C:" to "D:"
  CopyDir.PrintToTerminal := true;    // print progress information

  CopyDir.CopyReadOnlyFiles := true;  // copy files with attribut "ReadOnly"
  CopyDir.CopyHiddenFiles := false;   // don't copy files with attribut "Hidden"
  CopyDir.CopySystemFiles := false;   // don't copy files with attribut "SystemFile"
  CopyDir.CopyArchiveFiles := true;   // copy files with attribut "Archive"

  CopyDir.PreserverFileDates := true; // preserve dates of copied files
  {$IFNDEF Unix}
  // TCopyDir.PreserveAttributes is not available for Unix systems
  // (Unit "FileUtil" is not able to set attributes on Unix system)
  CopyDir.PreserveAttributes := true; // preserve attributes of copied files
  {$ENDIF}

  CopyDir.Enumerate; // this is optional; files added to directory after enumeration won't be copied
  CopyDir.Start;

  Log := CopyDir.GetLog;
  CopyDir.Free;
end;

I hope this will be enough to satisfy you :)
Improved version will be attachted in the first post of this thread (http://forum.lazarus.freepascal.org/index.php/topic,20759.msg120662.html#msg120662).

Greetings,
bastla

PS: I appreciate feedbacks!

Edit: @JuhaManninen:
I saw Takeda's CopyDirTree(), it's really simple and seems to be effective! But I don't think it can handle file flags like mine class :D
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: CM630 on June 24, 2013, 09:46:18 am
Okay, now it copied all my files and preserved the file modification date. It did not preserve the file creation date, but probably that is how it has to be, since the windows copy function does not preserve it, too.
I suppose that there is some more job to do- can this class handle attributes of *nix file systems?
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: bastla on June 24, 2013, 07:15:02 pm
As far as I remind, there's no method of LCL which modifies the creation date. This has a point, because there isn't any creation flag on Unix systems.

File attributes are caught by FileGetAttr-method (more precisely, its UTF8 equivalent FileGetAttrUTF8) that supports "readonly", "hidden", "sysfile", "volumeid", "directory" and "archive". TCopyDir supports all except "directory" and "volumeid" because directories are handled in a separate way, "volumeid" requires a plain FAT file system (not FAT32!) and only works on Windows.

So, you see I got the most out of that basic LCL stuff :)

Greetings,
bastla
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: CM630 on June 25, 2013, 10:48:42 am
So if I get you right, FileGetAttr method is not useful in Lunux (etc.), or there could be some other method (maybe FileGetPermissions???).
IMHO, currently you functions will cause serious troubles in *nix (http://www.comptechdoc.org/os/linux/usersguide/linux_ugfilesp.html). Just imagine- after copying an executable file it won't be executable anymore.

Also, I came upon this thread (http://forum.lazarus.freepascal.org/index.php?topic=4688.0).
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: bastla on June 29, 2013, 12:28:00 pm
I don't have a Unix system in use actually, so I can't improve things for Unix at the moment.
But I guess that FileGetAttr() can also handle files on Unix systems.

I did a quick research and haven't found any useful things to deal with Unix permissons explicitly.
There seem to be some special Unix methods in unit "BaseUnix", but, as I just said, I can't test them...

Greetings,
bastla
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: onivan on July 29, 2013, 10:27:40 pm
Hello, bastla!
Your code is very cool but I've found one bug: if there is no any second level directories in the source directory then no files are copied if they exist.
Here is my fix:
in procedure TCopyDir._CopyFiles; :
Code: [Select]
if (self._enumerated) and ((self._dirsCreated) or (Length(self._directories) = 0) ) and
    (Length(self._files) > 0) then       
                                               
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: bastla on August 04, 2013, 01:32:53 pm
Hi onivan!

Thank you for reporting this bug! I did really not noticed this while designing the class  :-[
I'd like to mention your solution in my start post, would this be OK with you?

Greetings,
bastla
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: onivan on August 10, 2013, 08:30:04 pm
Surely that's OK!
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: bastla on August 10, 2013, 08:56:38 pm
Thanks!
I added a small note to the start post, hopefully everybody will notice your post.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: Caladan on January 03, 2014, 06:36:36 pm
this is so close to what I been looking for, thank you !

1 error in the source though

line 278
{$IFDEF Unix} self._copyAttributes := false;
    {$ELSE} self._preserveAttributes := true {$ENDIF};

assume that should be _preseriveAttributes on the 1st line.

also it does not copy all file atributes under linux, for example tis not copying the executable flag, will see if I can find a solution for that.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: bastla on January 05, 2014, 01:13:37 pm
Hi Caladan,

yes, your're right, it should be
Code: [Select]
  {$IFDEF Unix} self._preserveAttributes := false; in line 278. I'm sorry for that error! May I link your post to the start post so that everybody will see the bug?

Unfortunately, I'm not an excessive Linux user, so I don't know much about coding on Linux. That's why I can't figure out how to handle file attributes on Linux. It would be great if you could find a solution on how to read and set all those file attributes!

Greetings,
bastla
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: Caladan on January 05, 2014, 01:29:43 pm
feel free to link.

Well with fpchmod you can set the correct attributes but I have yet to find a way to get them from the original file.... so far unlucky to find a command or way to get them.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: bastla on June 03, 2014, 09:50:06 pm
Time to push this once again! :D
I uploaded a new version fixing bugs found by onivan and Caladan! Thank you very much!

Cheers,
bastla
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: aducom on April 05, 2015, 09:26:27 am
Only one remark, the class is using TList as an array of string. If you have a TList in your application than that might interfere. Just change it to TListString and no conflicts any more. I know this is an old post, but a very useful one.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: nogo on July 30, 2015, 01:57:08 pm
For me copying of files in a hidden folder does not work on Windows

kind regards,

nogo
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: AlexTP on August 15, 2015, 01:12:30 pm
Used in CudaText now. Thax... (note: I made error and wrote srcdir==todir, and unit created empty dir. better check SameFilename (Win32 aware) and don't do.)
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: eac on August 23, 2015, 08:53:09 pm
Excellent job here, indeed!  :)

But I have a newbie question.
How could I do to keep the creation date of the files?
Because putting
CopyDir.PreserverFileDates := true;
wasn't helpful for me :(

Thanx!
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: AlexTP on October 30, 2015, 07:26:53 am
Please: feature-request

I want to copy folder A to B but only those files in A, which don't exist in B or newer than files in B-
ie, I need option "CopyOnlyNewerFiles" - can you add, pls?
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: AlexTP on June 09, 2020, 12:57:58 am
My fork, some changes
- some LCL usual type was redefined??? bad. fixed to TDirsArray
- "const" string params
- removed too much logging

https://github.com/Alexey-T/CudaText/blob/master/comp/copydir.pas
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: CM630 on January 20, 2022, 11:19:54 am
...
https://github.com/Alexey-T/CudaText/blob/master/comp/copydir.pas (https://github.com/Alexey-T/CudaText/blob/master/comp/copydir.pas)
This URL is dead.

I tried to use TCopyDir v2-1 by bastla.zip, in a console app, but it does not seem to understand self.....
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: Thaddy on January 20, 2022, 11:52:41 am
https://swissdelphicenter.ch/en/showcode.php?id=152 Windows only. Old but still works.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: AlexTP on January 20, 2022, 01:28:58 pm
Old code from CudaText repo is now stored in own repo. Enjoy.
https://github.com/Alexey-T/CopyDir-Lazarus

Quote
*                               TCOPYDIR
*                               ========
*                                 v2.1
*
*  This unit contains "TCopyDir" class wich copies entire directories (incl. its
*  subdirectories)
*
*  Author: bastla (@ Supernature-Forum / @ Lazarus Forum)
*  License: Free Domain
*
*  How TCopyDir works:
*  -------------------
*  TCopyDir uses TFileSearcher to enumerate a whole directory and copies its
*  content file by file.
*  Because of using LCL-components only, this class should work on all available
*  platforms supported by Lazarus and LCL.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: CM630 on January 24, 2022, 12:29:53 pm
I noticed GetTickCount in the code. This line: self._AddToLog('COPYING DONE (in ' + IntToStr(GetTickCount - __startTime) +  ' ms)');
might show nonsense.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: Zvoni on January 24, 2022, 12:50:08 pm
https://swissdelphicenter.ch/en/showcode.php?id=152 Windows only. Old but still works.
in VBA still one of my favorite Functions, because of the FOF_ALLOWUNDO-Flag for Deleting
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: AlexTP on January 24, 2022, 12:52:29 pm
@CM360
Maybe make a pull-request. I don't know the Copydir code, so cannot fix.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: Josh on January 24, 2022, 05:45:32 pm
@Alextp

Attached a copy of my modified copydir routine that allows

Code: [Select]
* 2021.06.07 - changes by Josh on Lazarus Forum
* Added _copyAllFiles _copyOnlyIfExists,_copyIfNewer,_copyIfSizeChanged,
*    _abortcopy,_copycomparefiles:Boolean
*
*    Ability to AbortCopy Routine by useer setting _Abortcopy to true in GUI.
*
*    Ability To Keep App Responsive
*    _AppProcessMessagesCounterInterval:integer;
*    _keepalive:boolean;
*
*    Configureable Buffer Copy
*    _usebufferedcopy:boolean;
*    _comparebuffersize:LongInt;
*    _CopyBuffer,_CompBuffer1,_CompBuffer2:Array of Byte;
*    _copybuffersize:LongInt;
*
*    Variables to keep Track of Routine for User
*    _CopyDirProcessedFromFile:String;
*    _CopyDirProcessedFromFileSize:int64;
*    _TotalBytesToCopy,_TotalByteProcessed,_ActutalBytesCopied,_appproctickcounter:Qword;

attached Demo project using this routine allowing you to choose all avaiable options to test.

Thought I would post changes I made in-case the changes are suitable for main repository.

Title: Re: Copying an entire directory (incl. subdirectories)
Post by: AlexTP on January 24, 2022, 06:05:02 pm
@josh,
No wish to dig in new additions, sorry, so I trusted you and posted your files to repo
https://github.com/Alexey-T/CopyDir-Lazarus
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: Josh on January 24, 2022, 06:28:51 pm
Cool

Thanks it also give me a place to locate it if I loose my version :)

If I remember it should be pretty much backwards compatible with previous version, obviously the keepAlive routines are gui only and an ifdef lcl is used to add needed units/classes for gui operation.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: Josh on January 24, 2022, 06:54:27 pm
Hi Alextp

Just checked the repo; i noticed the the src folder has an un modified copydir.pas file,
and the Josh_src folder has the modified version.
could the modified version have the following added to the header..

Thanks
Josh

* Added _copyAllFiles _copyOnlyIfExists,_copyIfNewer,_copyIfSizeChanged,
*    _abortcopy,_copycomparefiles:Boolean
*
*    Ability to AbortCopy Routine by useer setting _Abortcopy to true in GUI.
*
*    Ability To Keep App Responsive
*    _AppProcessMessagesCounterInterval:integer;
*    _keepalive:boolean;
*
*    Configureable Buffer Copy
*    _usebufferedcopy:boolean;
*    _comparebuffersize:LongInt;
*    _CopyBuffer,_CompBuffer1,_CompBuffer2:Array of Byte;
*    _copybuffersize:LongInt;
*
*    Variables to keep Track of Routine for User
*    _CopyDirProcessedFromFile:String;
*    _CopyDirProcessedFromFileSize:int64;
*    _TotalBytesToCopy,_TotalByteProcessed,_ActutalBytesCopied,_appproctickcounter:Qword;
*

Title: Re: Copying an entire directory (incl. subdirectories)
Post by: AlexTP on January 24, 2022, 07:41:49 pm
@Josh, I added history.txt from your text.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: CM630 on January 25, 2022, 08:03:02 am


@CM360
Maybe make a pull-request. I don't know the Copydir code, so cannot fix.
The best solution is to use GettTickCount64 instead of GettickCount, but this will break compatibility with WinXP.
So in this particular case, it could be:


Code: Pascal  [Select][+][-]
  1. ...
  2. Function SafeSubtract(Now : Int64; Before : Int64) : Int64;
  3. begin
  4.  Result := (Now - Before) And $FFFFFFFF;
  5. end;
  6. ....
and replace
  self._AddToLog('COPYING DONE (in ' + IntToStr(GetTickCount - __startTime) +   ' ms)');
with
  self._AddToLog('COPYING DONE (in ' + IntToStr(SafeSubtract(GetTickCount - __startTime)) +   ' ms)');
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: PascalDragon on January 25, 2022, 09:06:52 am
@CM360
Maybe make a pull-request. I don't know the Copydir code, so cannot fix.
The best solution is to use GettTickCount64 instead of GettickCount, but this will break compatibility with WinXP.

As long as you use SysUtils.GetTickCount64 it falls back to GetTickCount if GetTickCount64 is not available.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: AlexTP on January 25, 2022, 02:00:42 pm
Replaced GetTickCount->GetTickCount64.
BTW, we use LCLIntf.GetTickCount64, not from SysUtils.

EDIT: it's the same as SysUtils one.
Title: Re: Copying an entire directory (incl. subdirectories)
Post by: Josh on January 25, 2022, 02:54:13 pm
@Alextp,
I should have mentioned I changed to GetTickCount64 in the moded version; changed the necessary vars from DWord to QWord.
TinyPortal © 2005-2018