Recent

Author Topic: DeleteFile() fails on Win10  (Read 1066 times)

dbannon

  • Hero Member
  • *****
  • Posts: 884
    • tomboy-ng, a rewrite of the classic Tomboy
DeleteFile() fails on Win10
« on: January 27, 2020, 01:37:44 pm »
In my app, I move one file over top of another. You know, write out a file in /tmp then, when its all written, use the new file to replace an existing one.

But on Windows, I found that a few users have reported this is not working, I do this -

Code: Pascal  [Select]
  1.     {$ifdef WINDOWS}                    // Windows cannot 'move' over existing file.
  2.     if not DeleteFileUTF8(FileName) then begin
  3.         if not DeleteFile(FileName) then begin
  4.                 NoteLoc.ErrorStr:='Failed 2nd Delete';
  5.                 exit(False);
  6.         end;
  7.     end;
  8.    {$endif}
  9.     result := RenameFileUTF8(TmpName, FileName);

As you see, I have tried both lazfileutils DeleteFileUTF8 and sysutils DeleteFile(), neither will remove this file.  I cannot replicate this behavour but two of my end users can. They confirm that -
* they see the error about Failed 2nd Delete
* that the file is still there after the exercise
* that they can, manually, remove the file themselves.

My own Win10 system does not exhibit this problem but others do. Can anyone suggest whats happening here please ?

Davo
Lazarus 2, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7909
Re: DeleteFile() fails on Win10
« Reply #1 on: January 27, 2020, 01:43:16 pm »
Windows can hold short transient locks. Avirus software can also.

If it fails, try to loop and retry a few times with say 10ms or 100ms pauses inbetween. There are several bugreports for fpmake with similar problems

If you use Lazarus in default (utf8) mode, then just the deletefile is enough for FPC3.0+.   The -utf8 versions are mostly a remainder of  before 3.0
« Last Edit: January 27, 2020, 03:44:00 pm by marcov »

Thaddy

  • Hero Member
  • *****
  • Posts: 9565
Re: DeleteFile() fails on Win10
« Reply #2 on: January 27, 2020, 03:40:41 pm »
"if not DeleteFileUTF8(FileName)"? On Windows? Windows is an UTF16 OS. To me it looks suspicious. DeleteFileW (Which is supported by FPC in the windows unit)
See https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-deletefilew and notice DeleFileTransacted too and this: https://docs.microsoft.com/en-us/windows/win32/fileio/deprecation-of-txf

The crux is that an application other than yours or your application itself may still hold a lock on a file. If that is the case you can not delete it unless you explicitly remove the lock first or wait for the other application to finish: that may be in some cases when such an application is shutdown.
On your own application, make sure you do not have a dangling file handle, make sure it is nilled first, before re-using that handle.
« Last Edit: January 27, 2020, 03:48:21 pm by Thaddy »
also related to equus asinus.

ASBzone

  • Sr. Member
  • ****
  • Posts: 297
  • Automation leads to relaxation...
    • BrainWaveCC Utilities
Re: DeleteFile() fails on Win10
« Reply #3 on: January 27, 2020, 05:32:16 pm »
In my app, I move one file over top of another. You know, write out a file in /tmp then, when its all written, use the new file to replace an existing one.

But on Windows, I found that a few users have reported this is not working, I do this -

...

My own Win10 system does not exhibit this problem but others do. Can anyone suggest whats happening here please ?

Davo

What are the file names that this is happening with?
-ASB: https://www.BrainWaveCC.com

Lazarus v2.0.7 r62618 / FPC v3.2.0-beta-r44160 (via FpcUpDeluxe) -- Windows 64-bit install w/32-bit cross-compile
Primary System: Windows 10 Pro x64, Version 1909 (Build 18363.657)
Other Systems: Windows 10 Pro x64, Version 1909 or greater

Bart

  • Hero Member
  • *****
  • Posts: 3624
    • Bart en Mariska's Webstek
Re: DeleteFile() fails on Win10
« Reply #4 on: January 27, 2020, 05:37:29 pm »
"if not DeleteFileUTF8(FileName)"? On Windows? Windows is an UTF16 OS.

Why for Pete's sake would that make you suspicious.
Did you even bother to look at the implementation of DeleteFileUTF8?
In old Lazarus versions it did explicit conversion from UTF8 to WideString, then called the W-API.
In trunk it simply calls SysUtils.DeleteFile() (which does the W-API call).

If that makes you suspicious, you might need medication.

Bart
« Last Edit: January 27, 2020, 05:52:04 pm by Bart »

GetMem

  • Hero Member
  • *****
  • Posts: 3554
Re: DeleteFile() fails on Win10
« Reply #5 on: January 27, 2020, 05:45:46 pm »
@Davo

Perhaps this helps:
Code: Pascal  [Select]
  1.  if not DeleteFileUTF8(FileName) then
  2.     ShowMessage(SysErrorMessage(GetLastOSError));

For example:
 1. If I try to delete "c:\nothing.txt", the OS returns: "Cannot find the file specified."
 2. If I try to delete the currently running application(DeleteFileUTF8(Application.ExeName)), the OS returns: "Acces denied", for obvious reasons.
« Last Edit: January 27, 2020, 05:53:28 pm by GetMem »

440bx

  • Hero Member
  • *****
  • Posts: 1396
Re: DeleteFile() fails on Win10
« Reply #6 on: January 27, 2020, 06:04:40 pm »
* that they can, manually, remove the file themselves.
<snip>
Can anyone suggest whats happening here please ?

GetMem's suggestion above is the appropriate first step.  Figure out what it is that Windows didn't like in the attempt to delete the file.

The fact that the users can manually remove the file themselves also opens the possibility Thaddy mentioned that the file might be being read by an antivirus or some other system "protector" getting in the way of getting things done.  As Marcov suggested, it might worth putting the delete calls in a short "auto retry" loop (10 times ?) with a 10ms delay.  If possible, save the "GetLastError" values and incorporate them in the "delete failed" message.  At least that way you will have a better idea of what may be the cause of the problem.

HTH.

using FPC v3.0.4 and Lazarus 1.8.2 on Windows 7 64bit.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7909
Re: DeleteFile() fails on Win10
« Reply #7 on: January 27, 2020, 06:30:40 pm »

dbannon

  • Hero Member
  • *****
  • Posts: 884
    • tomboy-ng, a rewrite of the classic Tomboy
Re: DeleteFile() fails on Win10
« Reply #8 on: January 28, 2020, 07:03:32 am »
Thanks folks.
I have made a test version of the app that captures the SysErrorMsg  and also waits a bit before trying again. Posted that test version and have to wait for the affected users to try it out.

Good advice from people who know Windows far better than me !

Davo
Lazarus 2, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng

Thaddy

  • Hero Member
  • *****
  • Posts: 9565
Re: DeleteFile() fails on Win10
« Reply #9 on: January 28, 2020, 08:55:36 am »
In trunk it simply calls SysUtils.DeleteFile() (which does the W-API call).
Well then. That's what I mean. Nice to see that in trunk, because as you state this has not always been the case.  :D
That still leaves the locking and transacted deletes.
« Last Edit: January 28, 2020, 08:57:56 am by Thaddy »
also related to equus asinus.

PascalDragon

  • Hero Member
  • *****
  • Posts: 948
  • Compiler Developer
Re: DeleteFile() fails on Win10
« Reply #10 on: January 28, 2020, 09:21:08 am »
In trunk it simply calls SysUtils.DeleteFile() (which does the W-API call).
Well then. That's what I mean. Nice to see that in trunk, because as you state this has not always been the case.  :D
Even in older versions it called DeleteFileW. Here is the code from Lazarus 1.0. If you follow the initialization for the function variable DeleteFile_ you'll notice that it's initialized to a function that calls DeleteFileW except on older Windows versions (older than Windows 2000).

Bart

  • Hero Member
  • *****
  • Posts: 3624
    • Bart en Mariska's Webstek
Re: DeleteFile() fails on Win10
« Reply #11 on: January 28, 2020, 10:20:06 am »
Even in older versions it called DeleteFileW.

As I clearly stated.
Thaddy however seems to have a partial blindness for what other people write.
Most likely that is not of an ocular origin.

Bart

dbannon

  • Hero Member
  • *****
  • Posts: 884
    • tomboy-ng, a rewrite of the classic Tomboy
Re: DeleteFile() fails on Win10
« Reply #12 on: January 31, 2020, 12:47:31 am »
OK, I have finally sorted this out, superficially at least.
While I cannot reproduce the problem myself, a cooperative end user has been running tests and returning log file to me.

Whats happening I believe is that when you call DeleteFileUTF8() in windows, the file is (almost always?) deleted but sometimes the function returns false.  My guess is that its not having its return value set reliably. There is a path through the function that allows Result to be random.  As the function very rarely fails, few of us check its return value and it has gone unnoticed.

My diagnose is based on calling DeleteFileUTF8() and if it returns false, call FileExists() to really see if its gone. If FileExits() retuns false, allow the programe to proceed, hopefully normally. So far, his tests indicatate that is the case.

I will, in the near future, try and step though the Windows code (bit out of my comfort zone) and will follow up on what I find.

For the record, this end user also has Vera-Crypt running on his windows box and also uses PortableApps.com.  I don't think there is any connection but worth noting.....

Davo
Lazarus 2, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng

ASBzone

  • Sr. Member
  • ****
  • Posts: 297
  • Automation leads to relaxation...
    • BrainWaveCC Utilities
Re: DeleteFile() fails on Win10
« Reply #13 on: January 31, 2020, 02:16:21 am »
OK, I have finally sorted this out, superficially at least.
While I cannot reproduce the problem myself, a cooperative end user has been running tests and returning log file to me.

Whats happening I believe is that when you call DeleteFileUTF8() in windows, the file is (almost always?) deleted but sometimes the function returns false.  My guess is that its not having its return value set reliably. There is a path through the function that allows Result to be random.  As the function very rarely fails, few of us check its return value and it has gone unnoticed.

My diagnose is based on calling DeleteFileUTF8() and if it returns false, call FileExists() to really see if its gone. If FileExits() retuns false, allow the programe to proceed, hopefully normally. So far, his tests indicatate that is the case.

I will, in the near future, try and step though the Windows code (bit out of my comfort zone) and will follow up on what I find.

For the record, this end user also has Vera-Crypt running on his windows box and also uses PortableApps.com.  I don't think there is any connection but worth noting.....

Davo


When AntiMalware tools come into play, they can complicate things.  This can cause delete functions to take more than a normal amount of time, because something else is inspecting the file before finally removing it.


If you get more reports on the issue, we should be able to troubleshoot it some more.
-ASB: https://www.BrainWaveCC.com

Lazarus v2.0.7 r62618 / FPC v3.2.0-beta-r44160 (via FpcUpDeluxe) -- Windows 64-bit install w/32-bit cross-compile
Primary System: Windows 10 Pro x64, Version 1909 (Build 18363.657)
Other Systems: Windows 10 Pro x64, Version 1909 or greater

dbannon

  • Hero Member
  • *****
  • Posts: 884
    • tomboy-ng, a rewrite of the classic Tomboy
Re: DeleteFile() fails on Win10
« Reply #14 on: January 31, 2020, 02:52:49 am »
No ASBzone, I have put quite a lot of code in there  to diagnose whats happening. As soon as I get a false back from DeleteFileUTF8() I read the system error and then check to see if the file is actually there or not.  Fileexists() tells me its not there anymore.   (So does the system error msg) This, on this fellow's machine, is quite reliable.

I also have code in there that dumps details of the file, sleeps for a 10mS and tries again, then for 30mS and, finally for 100mS logging each step carefully.  But its never executed because the file really has been deleted.

The problem really does appear to be that DeleteFileUTF8() can return false even when it correctly deleted the file.

Davo
Lazarus 2, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng