* * *

Author Topic: FileExists doesn't always work in Windows  (Read 4032 times)

ASBzone

  • Jr. Member
  • **
  • Posts: 71
  • Automation leads to relaxation...
    • BrainWaveCC Utilities
FileExists doesn't always work in Windows
« on: August 07, 2018, 08:29:37 pm »
I have a utility that hashes files either by directory, or by individual name.  (It can also provide a hash of text.)

For the directory enumeration, I am using FindAllFiles successfully.
https://www.freepascal.org/docs-html/rtl/sysutils/findfirst.html

For the individual file hashing, I was using if FileExists to determine if it was a legitimate file, and if not, just return the hash of the text of the name.   https://www.freepascal.org/docs-html/rtl/sysutils/fileexists.html

Well, it turns out that for some files -- primarily locked files -- it won't find them at all.   Files such as \pagefile.sys or \swapfile.sys  (which, I agree, we rarely really need to hash those monstrous files) were not working.

Here's the test file:

Code: Pascal  [Select]
  1. Program Lesson2;
  2.  
  3. { This program demonstrates the FileGetAttr function }
  4.  
  5. Uses sysutils;
  6.  
  7. Procedure Testit (Name : String);
  8.  
  9. Var F : Longint;
  10.  
  11. Begin
  12.   if FileExists(Name)
  13.      then Write(Name, ' DOES exist!')
  14.      else Write(Name, ' does NOT exist!');
  15.  
  16.  
  17.   F:=FileGetAttr(Name);
  18.   If F<>-1 then
  19.     begin
  20.     Writeln ('Testing : ',Name);
  21.     If (F and faReadOnly)<>0 then
  22.       Writeln ('File is ReadOnly');
  23.     If (F and faHidden)<>0 then
  24.       Writeln ('File is hidden');
  25.     If (F and faSysFile)<>0 then
  26.       Writeln ('File is a system file');
  27.     If (F and faVolumeID)<>0 then
  28.       Writeln ('File is a disk label');
  29.     If (F and faArchive)<>0 then
  30.       Writeln ('File is archive file');
  31.     If (F and faDirectory)<>0 then
  32.       Writeln ('File is a directory');
  33.     end
  34.   else
  35.    Writeln ('Error reading attributes of ', Name);
  36.    Writeln;
  37. end;
  38.  
  39. begin
  40.   testit ('ex40.pp');
  41.   testit (ParamStr(0));
  42.   testit ('C:\Pagefile.sys');
  43.   testit ('C:\Temp\ThisFileIsHidden.TXT');
  44.   testit ('C:\Temp\TestOutput.TXT');
  45.   testit ('C:\swapfile.sys');
  46.   testit ('C:\bootmgr');
  47.   testit ('C:\boot.ini');
  48.   testit ('.');
  49.   testit ('/');
  50. End.
  51.  

It didn't work with FileGetAttr, either.

I still don't know why, but I went with FindFirst as a replacement, and that has gone rather well.

Should that failure of FileExists be considered a bug? 
-ASB: https://www.BrainWaveCC.com

Lazarus 1.8.4 + FPC 3.0.4 (32-bit w/64-bit cross-compile)
Occasional testing of NewPascal
Windows 10 Pro x64, Version 1803 (Build 17134.228)

(Technically, I logon to these forums from multiple versions of Windows Pro/Enterprise...)

Remy Lebeau

  • Sr. Member
  • ****
  • Posts: 439
    • Lebeau Software
Re: FileExists doesn't always work in Windows
« Reply #1 on: August 07, 2018, 09:07:00 pm »
Well, it turns out that for some files -- primarily locked files -- it won't find them at all.   Files such as \pagefile.sys or \swapfile.sys  (which, I agree, we rarely really need to hash those monstrous files) were not working.

I haven't seen FreePascal's implementation of FileExists(), but hopefully it is more than just a bare-bones call to GetFileAttribute() and nothing else.  That is the "old timer's" way of checking for file existence.  At least Delphi's FileExists() takes extra steps if GetFileAttributes() fails - to handle symbolic links, check for file access errors, even resort to FindFirstFile() - in an attempt to discover if a file *really* exists or not.  Hopefully FreePascal's FileExists() would do something similar.

It didn't work with FileGetAttr, either.

That is a bit odd, as it is very rare, but not unheard of, to not have sufficient rights to at least query a file's attributes.  But I suppose anything is possible with security.  You will have to detect the failure and check the error code to see if it indicate that the file actually exists but could not be accessed, such as access denied, sharing violations, etc.

Should that failure of FileExists be considered a bug?

I would.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

af0815

  • Full Member
  • ***
  • Posts: 153
Re: FileExists doesn't always work in Windows
« Reply #2 on: August 07, 2018, 09:48:48 pm »
Quote
Function FileExists (Const FileName : UnicodeString) : Boolean;
var
  Attr:Dword;
begin

  Attr:=GetFileAttributesW(PWideChar(FileName));
  if Attr <> $ffffffff then
    Result:= (Attr and FILE_ATTRIBUTE_DIRECTORY) = 0
  else
    Result:=False;
end;
This is the full source from SysUtils.
regards
Andreas

ASBzone

  • Jr. Member
  • **
  • Posts: 71
  • Automation leads to relaxation...
    • BrainWaveCC Utilities
Re: FileExists doesn't always work in Windows
« Reply #3 on: August 07, 2018, 10:36:20 pm »
You will have to detect the failure and check the error code to see if it indicate that the file actually exists but could not be accessed, such as access denied, sharing violations, etc.

Thanks, Remy

Actually, it was easier to use FindFirst and check on that error.  Both FindFirst and FindAllFiles provided a valid file handle, which was always accurate for the existence of the file.  If I receive an error trying to use that Handle, then I know that the file has permissions or some other issue, and I can provide a proper "::DENIED::" response.
-ASB: https://www.BrainWaveCC.com

Lazarus 1.8.4 + FPC 3.0.4 (32-bit w/64-bit cross-compile)
Occasional testing of NewPascal
Windows 10 Pro x64, Version 1803 (Build 17134.228)

(Technically, I logon to these forums from multiple versions of Windows Pro/Enterprise...)

ASBzone

  • Jr. Member
  • **
  • Posts: 71
  • Automation leads to relaxation...
    • BrainWaveCC Utilities
Re: FileExists doesn't always work in Windows
« Reply #4 on: August 07, 2018, 10:39:00 pm »
Quote
Function FileExists (Const FileName : UnicodeString) : Boolean;
var
  Attr:Dword;
begin

  Attr:=GetFileAttributesW(PWideChar(FileName));
  if Attr <> $ffffffff then
    Result:= (Attr and FILE_ATTRIBUTE_DIRECTORY) = 0
  else
    Result:=False;
end;
This is the full source from SysUtils.

Thanks for that.  I did not review it when I was testing...
-ASB: https://www.BrainWaveCC.com

Lazarus 1.8.4 + FPC 3.0.4 (32-bit w/64-bit cross-compile)
Occasional testing of NewPascal
Windows 10 Pro x64, Version 1803 (Build 17134.228)

(Technically, I logon to these forums from multiple versions of Windows Pro/Enterprise...)

ASerge

  • Hero Member
  • *****
  • Posts: 865
Re: FileExists doesn't always work in Windows
« Reply #5 on: August 07, 2018, 11:14:37 pm »
This work:
Code: Pascal  [Select]
  1. {$APPTYPE CONSOLE}
  2. {$MODE OBJFPC}
  3.  
  4. uses Windows;
  5.  
  6. function FileExists(const FileName: UnicodeString): Boolean;
  7. const
  8.   INVALID_FILE_ATTRIBUTES = DWORD(-1);
  9.   CNotExistsErrors = [ERROR_FILE_NOT_FOUND, ERROR_PATH_NOT_FOUND, ERROR_INVALID_NAME];
  10. var
  11.   Attr: DWORD;
  12. begin
  13.   Attr := GetFileAttributesW(PWideChar(FileName));
  14.   if Attr = INVALID_FILE_ATTRIBUTES then
  15.     Result := not (GetLastError in CNotExistsErrors)
  16.   else
  17.     Result := (Attr and FILE_ATTRIBUTE_DIRECTORY) <> 0;
  18. end;
  19.  
  20. begin
  21.   if FileExists('c:\pagefile.sys') then
  22.     Writeln('Exists')
  23.   else
  24.     Writeln('Wrong');
  25.   Readln;
  26. end.

ASBzone

  • Jr. Member
  • **
  • Posts: 71
  • Automation leads to relaxation...
    • BrainWaveCC Utilities
Re: FileExists doesn't always work in Windows
« Reply #6 on: August 08, 2018, 01:29:34 am »
This work:

Very nice, ASerge.   Much appreciated.
-ASB: https://www.BrainWaveCC.com

Lazarus 1.8.4 + FPC 3.0.4 (32-bit w/64-bit cross-compile)
Occasional testing of NewPascal
Windows 10 Pro x64, Version 1803 (Build 17134.228)

(Technically, I logon to these forums from multiple versions of Windows Pro/Enterprise...)

Remy Lebeau

  • Sr. Member
  • ****
  • Posts: 439
    • Lebeau Software
Re: FileExists doesn't always work in Windows
« Reply #7 on: August 11, 2018, 03:00:49 am »
Code: [Select]
  if Attr = INVALID_FILE_ATTRIBUTES then
    Result := not (GetLastError in CNotExistsErrors)
  else
    Result := (Attr and FILE_ATTRIBUTE_DIRECTORY) <> 0;

In the case where GetFileAttributes() fails, there are quite a few more error codes that may occur than just the 3 you mention.  However, those 3 are the only ones that Delphi's FileExists() checks for, then it resorts to FindFirstFile().

In the case where GetFileAttribute() succeeds, Delphi has an option to handle FILE_ATTRIBUTE_REPARSE_POINT before checking for FILE_ATTRIBUTE_DIRECTORY.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

af0815

  • Full Member
  • ***
  • Posts: 153
Re: FileExists doesn't always work in Windows
« Reply #8 on: August 11, 2018, 10:34:37 am »
Bug report for this ?
regards
Andreas

ASerge

  • Hero Member
  • *****
  • Posts: 865
Re: FileExists doesn't always work in Windows
« Reply #9 on: August 11, 2018, 12:14:43 pm »
In the case where GetFileAttributes() fails, there are quite a few more error codes that may occur than just the 3 you mention. 
I don't know the full list. Can be add later, if that.

Quote
In the case where GetFileAttribute() succeeds, Delphi has an option to handle FILE_ATTRIBUTE_REPARSE_POINT before checking for FILE_ATTRIBUTE_DIRECTORY.
Delphi has FollowLink parameter. FPC not. I tried to add features with FollowLink support, added patch on mantis, but the administration wasn't interested.

Don't forget that we can't just copy the Delphi sources because of legal restrictions.

New version:
Code: Pascal  [Select]
  1. {$APPTYPE CONSOLE}
  2. {$MODE OBJFPC}
  3.  
  4. uses Windows;
  5.  
  6. function FileExists(const FileName: UnicodeString): Boolean;
  7.  
  8.   function FoundByEnum: Boolean;
  9.   var
  10.     R: TWIN32FINDDATAW;
  11.     H: THandle;
  12.   begin
  13.     H := FindFirstFileExW(PWideChar(FileName), FindExInfoBasic, @R,
  14.       FindExSearchNameMatch, nil, 0); // It's faster, than FindFirstFile
  15.     Result := H <> INVALID_HANDLE_VALUE;
  16.     if Result then
  17.     begin
  18.       FindClose(H);
  19.       Result := (R.dwFileAttributes and FILE_ATTRIBUTE_DIRECTORY) = 0;
  20.     end;
  21.   end;
  22.  
  23. const
  24.   INVALID_FILE_ATTRIBUTES = DWORD(-1);
  25.   CNotExistsErrors = [
  26.     ERROR_FILE_NOT_FOUND,
  27.     ERROR_PATH_NOT_FOUND,
  28.     ERROR_INVALID_NAME,    // Protects from names in the form of masks like '*'
  29.     ERROR_INVALID_DRIVE,
  30.     ERROR_NOT_READY,
  31.     ERROR_INVALID_PARAMETER,
  32.     ERROR_BAD_PATHNAME,
  33.     ERROR_BAD_NETPATH];
  34. var
  35.   Attr: DWORD;
  36. begin
  37.   Attr := GetFileAttributesW(PWideChar(FileName));
  38.   if Attr = INVALID_FILE_ATTRIBUTES then
  39.     Result := not (GetLastError in CNotExistsErrors) and FoundByEnum
  40.   else
  41.     Result := (Attr and FILE_ATTRIBUTE_DIRECTORY) <> 0;
  42. end;
  43.  
  44. begin
  45.   if FileExists('c:\pagefile.sys') then
  46.     Writeln('Exists')
  47.   else
  48.     Writeln('Wrong');
  49.   Readln;
  50. end.

Mr.Madguy

  • Sr. Member
  • ****
  • Posts: 322
Re: FileExists doesn't always work in Windows
« Reply #10 on: August 11, 2018, 03:49:07 pm »
Bug report for this ?
I guess, it's intentional behaviour. Standard programs, like Explorer and CMD, don't show system files. Only special software, like Far Manager, can show system files.
DynamicData 2.0 Released! Many optimizations: 15% smaller sources, 10% smaller code! Improved scalability!
Developing version 2.1.
Crazy unit testing

lucamar

  • Jr. Member
  • **
  • Posts: 61
Re: FileExists doesn't always work in Windows
« Reply #11 on: August 11, 2018, 05:22:32 pm »
Bug report for this ?
I guess, it's intentional behaviour. Standard programs, like Explorer and CMD, don't show system files. Only special software, like Far Manager, can show system files.

Huh? Yes, they do if you configure them that way. Explorer shows them all right in my Windows XP machines, as well as hidden files, (almost) each and every extension, etc. And in CMD you can do DIR /AS and it shows them to you.

It is a bug because the standard FileExists() function is very simplistic, as shown in this same thread. It more or less works well for DOS ... and little else.
Been there, done that ... barely kept the project timelines. ;)

ASBzone

  • Jr. Member
  • **
  • Posts: 71
  • Automation leads to relaxation...
    • BrainWaveCC Utilities
Re: FileExists doesn't always work in Windows
« Reply #12 on: August 14, 2018, 04:49:05 pm »
I guess, it's intentional behaviour. Standard programs, like Explorer and CMD, don't show system files. Only special software, like Far Manager, can show system files.

The issue goes beyond files having SYSTEM or HIDDEN attributes.   I initially thought that was the issue myself, but it turns out that it wasn't.  I tested with a batch of system and hidden files.   It appears to be more related to the issue of files that are opened exclusively by some other process.  I did not test more extensively than that once I realized that FindFirst worked.
-ASB: https://www.BrainWaveCC.com

Lazarus 1.8.4 + FPC 3.0.4 (32-bit w/64-bit cross-compile)
Occasional testing of NewPascal
Windows 10 Pro x64, Version 1803 (Build 17134.228)

(Technically, I logon to these forums from multiple versions of Windows Pro/Enterprise...)

Thaddy

  • Hero Member
  • *****
  • Posts: 6560
Re: FileExists doesn't always work in Windows
« Reply #13 on: August 14, 2018, 05:13:45 pm »
Under windows, just like with unix, find works depending on rights.
Ada's daddy wrote this:"Fools are my theme, let satire be my song."

ASBzone

  • Jr. Member
  • **
  • Posts: 71
  • Automation leads to relaxation...
    • BrainWaveCC Utilities
Re: FileExists doesn't always work in Windows
« Reply #14 on: August 15, 2018, 09:29:07 pm »
Under windows, just like with unix, find works depending on rights.

Yes and no.

In general, this is a true statement, but the problem manifested here is not one of rights.

As shown from the earlier examples, If Exists fails in some scenarios where FindFirst and FindAllFiles are successful.  Same machine, same user credentials, same user context.


-ASB: https://www.BrainWaveCC.com

Lazarus 1.8.4 + FPC 3.0.4 (32-bit w/64-bit cross-compile)
Occasional testing of NewPascal
Windows 10 Pro x64, Version 1803 (Build 17134.228)

(Technically, I logon to these forums from multiple versions of Windows Pro/Enterprise...)

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus