Lazarus

Programming => Operating Systems => Windows => Topic started by: dbannon on January 27, 2020, 01:37:44 pm

Title: DeleteFile() fails on Win10
Post by: dbannon on January 27, 2020, 01:37:44 pm
In my app, I move one file over top of another. You know, write out a file in /tmp then, when its all written, use the new file to replace an existing one.

But on Windows, I found that a few users have reported this is not working, I do this -

Code: Pascal  [Select][+][-]
  1.     {$ifdef WINDOWS}                    // Windows cannot 'move' over existing file.
  2.     if not DeleteFileUTF8(FileName) then begin
  3.         if not DeleteFile(FileName) then begin
  4.                 NoteLoc.ErrorStr:='Failed 2nd Delete';
  5.                 exit(False);
  6.         end;
  7.     end;
  8.    {$endif}
  9.     result := RenameFileUTF8(TmpName, FileName);

As you see, I have tried both lazfileutils DeleteFileUTF8 and sysutils DeleteFile(), neither will remove this file.  I cannot replicate this behavour but two of my end users can. They confirm that -
* they see the error about Failed 2nd Delete
* that the file is still there after the exercise
* that they can, manually, remove the file themselves.

My own Win10 system does not exhibit this problem but others do. Can anyone suggest whats happening here please ?

Davo
Title: Re: DeleteFile() fails on Win10
Post by: marcov on January 27, 2020, 01:43:16 pm
Windows can hold short transient locks. Avirus software can also.

If it fails, try to loop and retry a few times with say 10ms or 100ms pauses inbetween. There are several bugreports for fpmake with similar problems

If you use Lazarus in default (utf8) mode, then just the deletefile is enough for FPC3.0+.   The -utf8 versions are mostly a remainder of  before 3.0
Title: Re: DeleteFile() fails on Win10
Post by: Thaddy on January 27, 2020, 03:40:41 pm
"if not DeleteFileUTF8(FileName)"? On Windows? Windows is an UTF16 OS. To me it looks suspicious. DeleteFileW (Which is supported by FPC in the windows unit)
See https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-deletefilew and notice DeleFileTransacted too and this: https://docs.microsoft.com/en-us/windows/win32/fileio/deprecation-of-txf

The crux is that an application other than yours or your application itself may still hold a lock on a file. If that is the case you can not delete it unless you explicitly remove the lock first or wait for the other application to finish: that may be in some cases when such an application is shutdown.
On your own application, make sure you do not have a dangling file handle, make sure it is nilled first, before re-using that handle.
Title: Re: DeleteFile() fails on Win10
Post by: ASBzone on January 27, 2020, 05:32:16 pm
In my app, I move one file over top of another. You know, write out a file in /tmp then, when its all written, use the new file to replace an existing one.

But on Windows, I found that a few users have reported this is not working, I do this -

...

My own Win10 system does not exhibit this problem but others do. Can anyone suggest whats happening here please ?

Davo

What are the file names that this is happening with?
Title: Re: DeleteFile() fails on Win10
Post by: Bart on January 27, 2020, 05:37:29 pm
"if not DeleteFileUTF8(FileName)"? On Windows? Windows is an UTF16 OS.

Why for Pete's sake would that make you suspicious.
Did you even bother to look at the implementation of DeleteFileUTF8?
In old Lazarus versions it did explicit conversion from UTF8 to WideString, then called the W-API.
In trunk it simply calls SysUtils.DeleteFile() (which does the W-API call).

If that makes you suspicious, you might need medication.

Bart
Title: Re: DeleteFile() fails on Win10
Post by: GetMem on January 27, 2020, 05:45:46 pm
@Davo

Perhaps this helps:
Code: Pascal  [Select][+][-]
  1.  if not DeleteFileUTF8(FileName) then
  2.     ShowMessage(SysErrorMessage(GetLastOSError));

For example:
 1. If I try to delete "c:\nothing.txt", the OS returns: "Cannot find the file specified."
 2. If I try to delete the currently running application(DeleteFileUTF8(Application.ExeName)), the OS returns: "Acces denied", for obvious reasons.
Title: Re: DeleteFile() fails on Win10
Post by: 440bx on January 27, 2020, 06:04:40 pm
* that they can, manually, remove the file themselves.
<snip>
Can anyone suggest whats happening here please ?

GetMem's suggestion above is the appropriate first step.  Figure out what it is that Windows didn't like in the attempt to delete the file.

The fact that the users can manually remove the file themselves also opens the possibility Thaddy mentioned that the file might be being read by an antivirus or some other system "protector" getting in the way of getting things done.  As Marcov suggested, it might worth putting the delete calls in a short "auto retry" loop (10 times ?) with a 10ms delay.  If possible, save the "GetLastError" values and incorporate them in the "delete failed" message.  At least that way you will have a better idea of what may be the cause of the problem.

HTH.

Title: Re: DeleteFile() fails on Win10
Post by: marcov on January 27, 2020, 06:30:40 pm
See https://bugs.freepascal.org/view.php?id=21868
Title: Re: DeleteFile() fails on Win10
Post by: dbannon on January 28, 2020, 07:03:32 am
Thanks folks.
I have made a test version of the app that captures the SysErrorMsg  and also waits a bit before trying again. Posted that test version and have to wait for the affected users to try it out.

Good advice from people who know Windows far better than me !

Davo
Title: Re: DeleteFile() fails on Win10
Post by: Thaddy on January 28, 2020, 08:55:36 am
In trunk it simply calls SysUtils.DeleteFile() (which does the W-API call).
Well then. That's what I mean. Nice to see that in trunk, because as you state this has not always been the case.  :D
That still leaves the locking and transacted deletes.
Title: Re: DeleteFile() fails on Win10
Post by: PascalDragon on January 28, 2020, 09:21:08 am
In trunk it simply calls SysUtils.DeleteFile() (which does the W-API call).
Well then. That's what I mean. Nice to see that in trunk, because as you state this has not always been the case.  :D
Even in older versions it called DeleteFileW. Here (https://svn.freepascal.org/cgi-bin/viewvc.cgi/branches/fixes_1_0/components/lazutils/winfileutil.inc?view=markup&revision=38249&root=lazarus#l532) is the code from Lazarus 1.0. If you follow the initialization for the function variable DeleteFile_ you'll notice that it's initialized to a function that calls DeleteFileW except on older Windows versions (older than Windows 2000).
Title: Re: DeleteFile() fails on Win10
Post by: Bart on January 28, 2020, 10:20:06 am
Even in older versions it called DeleteFileW.

As I clearly stated.
Thaddy however seems to have a partial blindness for what other people write.
Most likely that is not of an ocular origin.

Bart
Title: Re: DeleteFile() fails on Win10
Post by: dbannon on January 31, 2020, 12:47:31 am
OK, I have finally sorted this out, superficially at least.
While I cannot reproduce the problem myself, a cooperative end user has been running tests and returning log file to me.

Whats happening I believe is that when you call DeleteFileUTF8() in windows, the file is (almost always?) deleted but sometimes the function returns false.  My guess is that its not having its return value set reliably. There is a path through the function that allows Result to be random.  As the function very rarely fails, few of us check its return value and it has gone unnoticed.

My diagnose is based on calling DeleteFileUTF8() and if it returns false, call FileExists() to really see if its gone. If FileExits() retuns false, allow the programe to proceed, hopefully normally. So far, his tests indicatate that is the case.

I will, in the near future, try and step though the Windows code (bit out of my comfort zone) and will follow up on what I find.

For the record, this end user also has Vera-Crypt running on his windows box and also uses PortableApps.com.  I don't think there is any connection but worth noting.....

Davo
Title: Re: DeleteFile() fails on Win10
Post by: ASBzone on January 31, 2020, 02:16:21 am
OK, I have finally sorted this out, superficially at least.
While I cannot reproduce the problem myself, a cooperative end user has been running tests and returning log file to me.

Whats happening I believe is that when you call DeleteFileUTF8() in windows, the file is (almost always?) deleted but sometimes the function returns false.  My guess is that its not having its return value set reliably. There is a path through the function that allows Result to be random.  As the function very rarely fails, few of us check its return value and it has gone unnoticed.

My diagnose is based on calling DeleteFileUTF8() and if it returns false, call FileExists() to really see if its gone. If FileExits() retuns false, allow the programe to proceed, hopefully normally. So far, his tests indicatate that is the case.

I will, in the near future, try and step though the Windows code (bit out of my comfort zone) and will follow up on what I find.

For the record, this end user also has Vera-Crypt running on his windows box and also uses PortableApps.com.  I don't think there is any connection but worth noting.....

Davo


When AntiMalware tools come into play, they can complicate things.  This can cause delete functions to take more than a normal amount of time, because something else is inspecting the file before finally removing it.


If you get more reports on the issue, we should be able to troubleshoot it some more.
Title: Re: DeleteFile() fails on Win10
Post by: dbannon on January 31, 2020, 02:52:49 am
No ASBzone, I have put quite a lot of code in there  to diagnose whats happening. As soon as I get a false back from DeleteFileUTF8() I read the system error and then check to see if the file is actually there or not.  Fileexists() tells me its not there anymore.   (So does the system error msg) This, on this fellow's machine, is quite reliable.

I also have code in there that dumps details of the file, sleeps for a 10mS and tries again, then for 30mS and, finally for 100mS logging each step carefully.  But its never executed because the file really has been deleted.

The problem really does appear to be that DeleteFileUTF8() can return false even when it correctly deleted the file.

Davo
Title: Re: DeleteFile() fails on Win10
Post by: MoCityMM on March 09, 2020, 03:33:45 pm
Was looking for something else but, ran into this post.

If I am reading this correctly...

Seen similar 'symptoms' with other applications on shared folder (network shares) and local folders.

Could be ACL (access control list) or UAC (which is a pain), if an application writes the file - it's the owner of the file in some circumstances (especially if there is a system account associated with the application) and the user may not have access to delete the file.

I would check to see what 'group' the user belongs to: 'Users', 'Power Users', etc...

If they are part of the Administrators group and it still happens, run iCACLS on file and see if you can delete the file then.

Try the link below, it's worth a shot:

https://theitbros.com/using-icacls-to-list-folder-permissions-and-manage-files/

-Mo



Title: Re: DeleteFile() fails on Win10
Post by: dbannon on March 09, 2020, 11:38:50 pm
Because I cannot, myself replicate this problem, its quite hard to work on.

I have had two users affected, the first reported the issue, I assume he has stopped using my app and does not respond.  The second person was a lot more helpful, but is now convinced that the problem relates to his use of drive encryption. Between us, we spent a lot of time on this problem, I have to assume he, too, has got to the point where he does not want to put in any more.

I have much of my diagnostic code still in there, if it happens to someone else then I can track it much quicker but I suspect it is no longer something that is going to trouble us.

So, thanks for your input MoCityMM but its looking like a cold case.

Davo
Title: Re: DeleteFile() fails on Win10
Post by: avra on March 10, 2020, 09:20:44 am
The problem really does appear to be that DeleteFileUTF8() can return false even when it correctly deleted the file.
So at the end this could be fixed with just additional check for FileExists()?
Title: Re: DeleteFile() fails on Win10
Post by: dbannon on March 10, 2020, 11:18:26 pm
Yes avra, thats what I have done in my code (for windows only). So far, with this users experience, we see only the first log error.
 
Code: Pascal  [Select][+][-]
  1.     if not DeleteFile(FullFileName) then begin
  2.         ErrorMsg := SysErrorMessage(GetLastOSError);
  3.         Debugln('Failed using DeleteFile - file name is :' + FullFilename);
  4.         Debugln('OS Error Msg : ' + ErrorMsg);
  5.         if not FileExistsUTF8(FullFileName) then
  6.             debugln('But, FileExists says its gone, proceed !')
  7.         else begin
  8.                 debugln('I can confirm its still there .');
  9.                 Debugln('Trying a little sleep...');
  10.                 sleep(10);
  11.                 if not DeleteFileUTF8(FullFileName) then begin
  12.                      if not FileExistsUTF8(FullFileName) then
  13.                      debugln('DeleteFileUTF8 says it failed but FileExists says its gone, proceed !')
  14.                 else exit(false);
  15.             end;
  16.         end;
  17.     end;

I'd be a lot happier if I could see it myself ...

Davo
Title: Re: DeleteFile() fails on Win10
Post by: winni on March 10, 2020, 11:50:36 pm
Hello!

What says the OS Error Msg??

Before  that I would insert:

Io := IoResult;

and then write it with DebugLn


Might be at the edge of "Off Topic" but

Reminds me to Win 3.1 and our first Netware Server.
( I know, grandpa talks about the war .....)

You thought all disk operations were executed, but they were not:

It gave the new IOerror 32, but no software new about it.
The error said something about "File Sharing Violation".
You needed some new addon for the Netware Driver.

Maybe it's something similar.

Winni
Title: Re: DeleteFile() fails on Win10
Post by: dbannon on March 11, 2020, 02:04:49 am
Hello!

What says the OS Error Msg??

Before  that I would insert:

Io := IoResult;

and then write it with DebugLn
....

Winni, I do grab SysErrorMessage(GetLastOSError) and drop that out in the debug line. From memory (at end users request, I have closed the issue ticket) it reported that the file could not be found to delete. But, as noted, it appeared it does try to delete it and successes.

Davo
Title: Re: DeleteFile() fails on Win10
Post by: jmnj on June 17, 2020, 05:33:31 pm
This same issue drove me NUTS for some time.  Here's a snippet what I use to reliably delete files in Windows 10...

// Requires Units: LazFileUtils, LAZUTF8

FileName:='String with full path of file to delete.';
DeleteFileUTF8(UTF8ToWinCP(FileName));

Hope this helps!  :D
Title: Re: DeleteFile() fails on Win10
Post by: Remy Lebeau on June 17, 2020, 09:59:04 pm
Here's a snippet what I use to reliably delete files in Windows 10...

One, the FileExists() check is unnecessary, as DeleteFileW() will do the same check and set GetLastError() accordingly.

Two, why would you pass a CP-encoded string to a UTF-8 function?
Title: Re: DeleteFile() fails on Win10
Post by: jmnj on June 18, 2020, 03:05:29 pm
Hi Remy,

Thanks for your comments.  I've removed the FileExists().  Explanations below:

Honestly, I haven't delved into why this works.  I did initially pass UTF8 without success.  Tried racking my head on this for weeks before I was willing to try "anything".

For some unknown reason this does work on Windows 10, post update KB4556799 using Lazarus 2.0.8 r62944.  I just don't know exactly -why- it does. 

I would love to know why this works and I'd love to know if the "normal" way of deleting files gets a permanent fix so that I'm not resorting to a clear hack.  I'm kinda between a rock and a hard place using this hack on a critical project.
Title: Re: DeleteFile() fails on Win10
Post by: dbannon on June 20, 2020, 02:48:37 am
....
One, the FileExists() check is unnecessary, as DeleteFileW() will do the same check and set GetLastError() accordingly.
.....

Just for the record, several variations that I sent the end user apparently received no error message from DeleteFile() and DeleteFileUTF8() but a subsequent call to FileExists() revealed that the file was, in fact still present.  And, on one occasion, I saw a log that indicated, perhaps, the file was deleted but DeleteFile() said it failed !   However, as I later found out, the user was using a disk encryption system and a portable application handler to run it, collectively, we think that was somehow related to the issue. An update to VeraCrypt seems to have helped.

I could not duplicate the end user's problem myself so was dependent on sending him/her a test binary. Fortunately, this was a very cooperative user. One other person did have what might have been a similar problem but with hind sight I don't think that is the case. 

Early stages of this problem was complicated by the fact that Linux/Mac are quite happy to overwrite an existing file with RenameFileUTF8() but Windows is not.  But I recently discovered that Linux is also upset about using RenameFileUTF8() on an existing file if that existing file is on an SMB share. And smb is, of course a Windows protocol ......

I am convinced that this is one very specific, isolated case.  I have left the obsessively careful code in there for Windows in my app, it does no harm and may, perhaps do some good ?

Thanks for input folks !

Davo
Title: Re: DeleteFile() fails on Win10
Post by: Remy Lebeau on June 21, 2020, 12:08:40 am
Just for the record, several variations that I sent the end user apparently received no error message from DeleteFile() and DeleteFileUTF8() but a subsequent call to FileExists() revealed that the file was, in fact still present.

That is certainly a possibility, if the file is in use at the time it is "deleted".  If the file is open, it will be marked for deletion, and won't actually be physically deleted until all open handles to the file are closed.

And, on one occasion, I saw a log that indicated, perhaps, the file was deleted but DeleteFile() said it failed !

That, I don't see being possible, unless something else deleted the file after DeleteFile() was called.  Unless the log was wrong/misleading, for instance if the log showed the file was *about to be* deleted, and then afterwards showed the *actual delete* failing.
TinyPortal © 2005-2018