CopyFile does not copy the file

[dbannon]

Because I wanted to be able to check on how its happening, I split the CopyFile() into a delete/copy process on Windows, something like this -

Code: Pascal  [Select][+][-]

1.    // First write the required file using Stream.SaveToFile(TempName), Stream.Free, no errors reported.
2.         if not DeleteFile(FileName) then begin
3.         ErrorMsg := SysErrorMessage(GetLastOSError);
4.         Debugln('Failed using DeleteFileUTF8 - file name is :' + Filename);
5.         Debugln('OS Error Msg : ' + ErrorMsg);
6.         if not FileExistsUTF8(FileName) then
7.             debugln('But, FileExists says its gone, proceed !');
8.         else begin {do other serious stuff} end;
9.    end;
10.    result := CopyFile(TempName, FileName);

The person experiencing this issue has sent me several logs, in all cases the even though DeleteFile has returned false and the OS error message is "The file can not be found" (in Swedish) the FileExists() says its gone and we proceed to copy over it (or copy to where it was).

However, while this seemed to work, he is now reporting problems that sound suspiciously like a shortage of resource issues, maybe file handles ?

So, does sound like a serious problem. But I cannot see his system. I believe I have a number of Windows users and only one is seeing this problem. A hard one ....

Davo

[winni]

 Hi!

Okay, stumbling in the fog, but I would try this:

If not deleteFile ...

    Sleep  (200) or 300 --- perhaps the encryption/decryption needs time??

   And now the old game with findfirst/findnext/findclose - let you show the whole directory.
   Does the file still exist??
  The has the side effect that the file pointer is heavily moved - if it is a problem with it.

  Which fs? Is it NTFS?

Winni

[winni]

Hi!

Another idea :

Do you just detect just a bug in the M$ file cache?
They do that since some versions.

How are the two flags FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH set in the windows version?

And have a look at

https://docs.microsoft.com/en-us/windows/win32/fileio/file-caching

Ugly problem !

Winni

[jamie]

I don't know why you guys don't use the ShFileOperation API call ?

It allows you just about all you need including a user report so they don't get board while waiting..

 I will tell you this however, files that are currently opened by some process will not get deleted but will be flagged so that the next time the OS views the files or starts up they are remembered along with the operation and once the files become closed and not locked the shell processes them.

 That is, if you are using the Shell functions..

At least that has been my experience..

[dbannon]

Ugly indeed.

Obviously, the Linux file system is massively cached but thats really not the user's business except during a power down event.  And there we have the sync command.

That write up does not mention a standalone command/api call, just parameters to CreateFile. So, difficult to use with copyfile() or deletefile(). 

StackOverflow has quite a lot of (negative) discussion.

I suspect what we are seeing is a conflict between Windows read-cache and write-caches.  One file has been read, is therefore in the read cache, its then either deleted or overwritten via the write-cache and suddenly, the write-cache does not agree with the read-cache.

A possible solution is to abandon the (very useful) process of writing the file out as a tmp file and then moving it onto the real file. While very good practice from a data safety principle, it confuses Window's cache. Yet, its on NTFS that the process is particularly valuable. ext3 and ext4 are well journaled.

Sigh ....
 

[winni]

Hi!

Can't  believe it!

I was seraching for the windows counterpart of sync.

It exists! Have a look at:

https://docs.microsoft.com/en-us/sysinternals/downloads/sync

Perhaps that helps!?

Winni

[440bx]

Code: Pascal  [Select][+][-]

1.    // First write the required file using Stream.SaveToFile(TempName), Stream.Free, no errors reported.
2.         if not DeleteFile(FileName) then begin
3. ...
4.  

... he is now reporting problems that sound suspiciously like a shortage of resource issues, maybe file handles ?

Here is a guess based on the comment in your code and, the fact that there is an apparent shortage of resources:  you use "Stream.SaveToFile(TempName)", the first thing I would do is to ensure the file handle used by that stream function has been closed, because if it hasn't and an attempt to delete the file is made, the behavior will depend on how the O/S will view the attempt to delete an open file. Windows, depending on the situation, could mark the file as "delete on close" and report the file as not existing until it is closed.  That has the potential of accumulated file handles in the system.

HTH.

[dbannon]

I don't know why you guys don't use the ShFileOperation API call ?

Hmm, "This function has been replaced in Windows Vista by IFileOperation." - is that still a useful thing in Windows 10 ? 

I will tell you this however, files that are currently opened by some process will not get deleted but will be flagged so that the next time the OS views the files or starts up they are remembered along with the operation and once the files become closed and not locked the shell processes them.

 That is, if you are using the Shell functions..

Well, I am pretty sure in my case the files are not open, they are opened for reading a little while before this problem but are closed.  And the issue is not so much that the functions don't work, its that the return code is sometimes incorrect.

Jamie, I would love to see a copy n paste code snip-it of that ShFileOperation in pascal ?

Davo

[dbannon]

Here is a guess based on the comment in your code and, the fact that there is an apparent shortage of resources:  you use "Stream.SaveToFile(TempName)", the first thing I would do is to ensure the file handle used by that stream function has been closed.....

What you say makes sense 440bx, but in my case, the reported fail to delete is of a file that has not been written to, was recently read and definitly closed at least several seconds before the attempt to delete.

The file being written to is called TempName and its written then moved to where the deleted file was.

winni, good find but requires admin rights.  No good on an app I distribute.  Wonder why, its always safe to let a user run sync on unix ....

Oh, and winni, I do have code in there that retries after a delay (in {do other serious stuff} in my example above) but so far, its never been executed. We find that the first "if FileExits() says that it does not so we proceed.


Davo

[440bx]

... definitly closed at least several seconds before the attempt to delete.

The file being written to is called TempName and its written then moved to where the deleted file was.

At this time, I can't think of another reason for the problem. 

Are the logs you got from the user something you can share ? there might be a clue in there.

[dbannon]

At this time, I can't think of another reason for the problem. 

Two aspects, first is he is using this File Encryption tool, VeraCrypt but googling for that does not indicate its a problem.
Secondly, is winni's suggestion about Window's caching. It does sound like a badly managed cache to me. 

Are the logs you got from the user something you can share ? there might be a clue in there.

Nothing interesting -

Code: Pascal  [Select][+][-]

1. .....
2. Failed using DeleteFileUTF8 - file name is :C:\Users\UUUUU.DESKTOP-ZZZZZZZ\AppData\Roaming\tomboy-ng\notes\9700D8CD-826C-409D-B180-77EF97F9DF03.note
3. OS: Det går inte att hitta filen.
4. But, FileExists says its gone, proceed !

generated from #695 of https://github.com/tomboy-notes/tomboy-ng/blob/master/tomboy-ng/savenote.pas

The swedish OS error msg translates to "cannot find file", I have replaced the start of the user's user name with 'UUUUU'.

David 

[440bx]

I don't have anything of value to offer but, an opinion...

It does sound like a badly managed cache to me. 

I suppose that is possible but, it seems that if that were the case, that user and just about every Windows user would be experiencing some problems.  That said, maybe the user's file encryption software combined with something else he is using (e.g, real time backup software) might be the cause.  Obviously, all speculation and not even useful.

Nothing interesting -

I agree, not exactly illuminating.   

Here is a shot in the dark, instead of using delete/copy, try using "MoveFile" if you can, just to see what that API says. 

[Zvoni]

Jamie, I would love to see a copy n paste code snip-it of that ShFileOperation in pascal ?

Davo

How good's your german? Note: It's "explicitly" for Delphi, and it's from year 2002
https://www.delphipraxis.net/1260-dateioperationen-mit-shfileoperation.html
The first one uses TStrings, the second one plain strings as FileNames
You need Unit "ShellApi"

[Birger52]

Took me a while.
Sorry about that.
Problem was, that there was a misspelling of the filename, when calling the procedure.
The procedure is actually functioning just fine....

I read all of your ideas, comments and discussions.
Interesting and many idea's.
I use a plain Win7 - with no compression or anything else special running.
Love TC - been using it since Windoze98...

A lot of discussions on deleting files.
I just ran into something....
I have SysUtils in the units uses and FileUtil in the implementations uses.
I have a
DeleteFile(Filename) where Filename is a  UnicodeString (defined simply as Filename:string;)
Compiling I get:
Error: Incompatible type for arg no. 1: Got "AnsiString", expected "PChar"
Using the newly learned trick with [Alt]+[up] I jump to the definition of DeleteFile.
It is in fileutilh.inc line 155:
Function DeleteFile (Const FileName : RawByteString) : Boolean;
In the same file line 135 is another (20 lines up and even marked...):
Function DeleteFile (Const FileName : UnicodeString) : Boolean;

Why does the compiler choose the overwritten function that has the wrong parameter type over the one with the right type?

Looking up the implementation - [Ctrl][Shift][Up] - makes it even more obscure.
The implemtation the compiler chooses to use looks like this:
Function DeleteFile (Const FileName : RawByteString) : Boolean;
begin
  Result:=DeleteFile(UnicodeString(FileName));
end;
Which is basically a call to the one it should have chosen in the first place....
(
The other one implements the same as the one in SysUtil:
Function DeleteFile (Const FileName : UnicodeString) : Boolean;
begin
  Result:=Windows.DeleteFileW(PWidechar(FileName));
end;
)

Using SysUtils.DeleteFile() compiles fine.