Recent

Author Topic: [SOLVED] FileExists doesn't always work in Windows  (Read 24892 times)

ASBzone

  • Hero Member
  • *****
  • Posts: 678
  • Automation leads to relaxation...
    • Free Console Utilities for Windows (and a few for Linux) from BrainWaveCC
[SOLVED] FileExists doesn't always work in Windows
« on: August 07, 2018, 08:29:37 pm »
I have a utility that hashes files either by directory, or by individual name.  (It can also provide a hash of text.)

For the directory enumeration, I am using FindAllFiles successfully.
https://www.freepascal.org/docs-html/rtl/sysutils/findfirst.html

For the individual file hashing, I was using if FileExists to determine if it was a legitimate file, and if not, just return the hash of the text of the name.   https://www.freepascal.org/docs-html/rtl/sysutils/fileexists.html

Well, it turns out that for some files -- primarily locked files -- it won't find them at all.   Files such as \pagefile.sys or \swapfile.sys  (which, I agree, we rarely really need to hash those monstrous files) were not working.

Here's the test file:

Code: Pascal  [Select][+][-]
  1. Program Lesson2;
  2.  
  3. { This program demonstrates the FileGetAttr function }
  4.  
  5. Uses sysutils;
  6.  
  7. Procedure Testit (Name : String);
  8.  
  9. Var F : Longint;
  10.  
  11. Begin
  12.   if FileExists(Name)
  13.      then Write(Name, ' DOES exist!')
  14.      else Write(Name, ' does NOT exist!');
  15.  
  16.  
  17.   F:=FileGetAttr(Name);
  18.   If F<>-1 then
  19.     begin
  20.     Writeln ('Testing : ',Name);
  21.     If (F and faReadOnly)<>0 then
  22.       Writeln ('File is ReadOnly');
  23.     If (F and faHidden)<>0 then
  24.       Writeln ('File is hidden');
  25.     If (F and faSysFile)<>0 then
  26.       Writeln ('File is a system file');
  27.     If (F and faVolumeID)<>0 then
  28.       Writeln ('File is a disk label');
  29.     If (F and faArchive)<>0 then
  30.       Writeln ('File is archive file');
  31.     If (F and faDirectory)<>0 then
  32.       Writeln ('File is a directory');
  33.     end
  34.   else
  35.    Writeln ('Error reading attributes of ', Name);
  36.    Writeln;
  37. end;
  38.  
  39. begin
  40.   testit ('ex40.pp');
  41.   testit (ParamStr(0));
  42.   testit ('C:\Pagefile.sys');
  43.   testit ('C:\Temp\ThisFileIsHidden.TXT');
  44.   testit ('C:\Temp\TestOutput.TXT');
  45.   testit ('C:\swapfile.sys');
  46.   testit ('C:\bootmgr');
  47.   testit ('C:\boot.ini');
  48.   testit ('.');
  49.   testit ('/');
  50. End.
  51.  

It didn't work with FileGetAttr, either.

I still don't know why, but I went with FindFirst as a replacement, and that has gone rather well.

Should that failure of FileExists be considered a bug? 
« Last Edit: July 25, 2020, 12:16:36 am by ASBzone »
-ASB: https://www.BrainWaveCC.com/

Lazarus v2.2.7-ada7a90186 / FPC v3.2.3-706-gaadb53e72c
(Windows 64-bit install w/Win32 and Linux/Arm cross-compiles via FpcUpDeluxe on both instances)

My Systems: Windows 10/11 Pro x64 (Current)

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: FileExists doesn't always work in Windows
« Reply #1 on: August 07, 2018, 09:07:00 pm »
Well, it turns out that for some files -- primarily locked files -- it won't find them at all.   Files such as \pagefile.sys or \swapfile.sys  (which, I agree, we rarely really need to hash those monstrous files) were not working.

I haven't seen FreePascal's implementation of FileExists(), but hopefully it is more than just a bare-bones call to GetFileAttribute() and nothing else.  That is the "old timer's" way of checking for file existence.  At least Delphi's FileExists() takes extra steps if GetFileAttributes() fails - to handle symbolic links, check for file access errors, even resort to FindFirstFile() - in an attempt to discover if a file *really* exists or not.  Hopefully FreePascal's FileExists() would do something similar.

It didn't work with FileGetAttr, either.

That is a bit odd, as it is very rare, but not unheard of, to not have sufficient rights to at least query a file's attributes.  But I suppose anything is possible with security.  You will have to detect the failure and check the error code to see if it indicate that the file actually exists but could not be accessed, such as access denied, sharing violations, etc.

Should that failure of FileExists be considered a bug?

I would.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

af0815

  • Hero Member
  • *****
  • Posts: 1289
Re: FileExists doesn't always work in Windows
« Reply #2 on: August 07, 2018, 09:48:48 pm »
Quote
Function FileExists (Const FileName : UnicodeString) : Boolean;
var
  Attr:Dword;
begin

  Attr:=GetFileAttributesW(PWideChar(FileName));
  if Attr <> $ffffffff then
    Result:= (Attr and FILE_ATTRIBUTE_DIRECTORY) = 0
  else
    Result:=False;
end;
This is the full source from SysUtils.
regards
Andreas

ASBzone

  • Hero Member
  • *****
  • Posts: 678
  • Automation leads to relaxation...
    • Free Console Utilities for Windows (and a few for Linux) from BrainWaveCC
Re: FileExists doesn't always work in Windows
« Reply #3 on: August 07, 2018, 10:36:20 pm »
You will have to detect the failure and check the error code to see if it indicate that the file actually exists but could not be accessed, such as access denied, sharing violations, etc.

Thanks, Remy

Actually, it was easier to use FindFirst and check on that error.  Both FindFirst and FindAllFiles provided a valid file handle, which was always accurate for the existence of the file.  If I receive an error trying to use that Handle, then I know that the file has permissions or some other issue, and I can provide a proper "::DENIED::" response.
-ASB: https://www.BrainWaveCC.com/

Lazarus v2.2.7-ada7a90186 / FPC v3.2.3-706-gaadb53e72c
(Windows 64-bit install w/Win32 and Linux/Arm cross-compiles via FpcUpDeluxe on both instances)

My Systems: Windows 10/11 Pro x64 (Current)

ASBzone

  • Hero Member
  • *****
  • Posts: 678
  • Automation leads to relaxation...
    • Free Console Utilities for Windows (and a few for Linux) from BrainWaveCC
Re: FileExists doesn't always work in Windows
« Reply #4 on: August 07, 2018, 10:39:00 pm »
Quote
Function FileExists (Const FileName : UnicodeString) : Boolean;
var
  Attr:Dword;
begin

  Attr:=GetFileAttributesW(PWideChar(FileName));
  if Attr <> $ffffffff then
    Result:= (Attr and FILE_ATTRIBUTE_DIRECTORY) = 0
  else
    Result:=False;
end;
This is the full source from SysUtils.

Thanks for that.  I did not review it when I was testing...
-ASB: https://www.BrainWaveCC.com/

Lazarus v2.2.7-ada7a90186 / FPC v3.2.3-706-gaadb53e72c
(Windows 64-bit install w/Win32 and Linux/Arm cross-compiles via FpcUpDeluxe on both instances)

My Systems: Windows 10/11 Pro x64 (Current)

ASerge

  • Hero Member
  • *****
  • Posts: 2223
Re: FileExists doesn't always work in Windows
« Reply #5 on: August 07, 2018, 11:14:37 pm »
This work:
Code: Pascal  [Select][+][-]
  1. {$APPTYPE CONSOLE}
  2. {$MODE OBJFPC}
  3.  
  4. uses Windows;
  5.  
  6. function FileExists(const FileName: UnicodeString): Boolean;
  7. const
  8.   INVALID_FILE_ATTRIBUTES = DWORD(-1);
  9.   CNotExistsErrors = [ERROR_FILE_NOT_FOUND, ERROR_PATH_NOT_FOUND, ERROR_INVALID_NAME];
  10. var
  11.   Attr: DWORD;
  12. begin
  13.   Attr := GetFileAttributesW(PWideChar(FileName));
  14.   if Attr = INVALID_FILE_ATTRIBUTES then
  15.     Result := not (GetLastError in CNotExistsErrors)
  16.   else
  17.     Result := (Attr and FILE_ATTRIBUTE_DIRECTORY) <> 0;
  18. end;
  19.  
  20. begin
  21.   if FileExists('c:\pagefile.sys') then
  22.     Writeln('Exists')
  23.   else
  24.     Writeln('Wrong');
  25.   Readln;
  26. end.

ASBzone

  • Hero Member
  • *****
  • Posts: 678
  • Automation leads to relaxation...
    • Free Console Utilities for Windows (and a few for Linux) from BrainWaveCC
Re: FileExists doesn't always work in Windows
« Reply #6 on: August 08, 2018, 01:29:34 am »
This work:

Very nice, ASerge.   Much appreciated.
-ASB: https://www.BrainWaveCC.com/

Lazarus v2.2.7-ada7a90186 / FPC v3.2.3-706-gaadb53e72c
(Windows 64-bit install w/Win32 and Linux/Arm cross-compiles via FpcUpDeluxe on both instances)

My Systems: Windows 10/11 Pro x64 (Current)

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: FileExists doesn't always work in Windows
« Reply #7 on: August 11, 2018, 03:00:49 am »
Code: [Select]
  if Attr = INVALID_FILE_ATTRIBUTES then
    Result := not (GetLastError in CNotExistsErrors)
  else
    Result := (Attr and FILE_ATTRIBUTE_DIRECTORY) <> 0;

In the case where GetFileAttributes() fails, there are quite a few more error codes that may occur than just the 3 you mention.  However, those 3 are the only ones that Delphi's FileExists() checks for, then it resorts to FindFirstFile().

In the case where GetFileAttribute() succeeds, Delphi has an option to handle FILE_ATTRIBUTE_REPARSE_POINT before checking for FILE_ATTRIBUTE_DIRECTORY.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

af0815

  • Hero Member
  • *****
  • Posts: 1289
Re: FileExists doesn't always work in Windows
« Reply #8 on: August 11, 2018, 10:34:37 am »
Bug report for this ?
regards
Andreas

ASerge

  • Hero Member
  • *****
  • Posts: 2223
Re: FileExists doesn't always work in Windows
« Reply #9 on: August 11, 2018, 12:14:43 pm »
In the case where GetFileAttributes() fails, there are quite a few more error codes that may occur than just the 3 you mention. 
I don't know the full list. Can be add later, if that.

Quote
In the case where GetFileAttribute() succeeds, Delphi has an option to handle FILE_ATTRIBUTE_REPARSE_POINT before checking for FILE_ATTRIBUTE_DIRECTORY.
Delphi has FollowLink parameter. FPC not. I tried to add features with FollowLink support, added patch on mantis, but the administration wasn't interested.

Don't forget that we can't just copy the Delphi sources because of legal restrictions.

New version:
Code: Pascal  [Select][+][-]
  1. {$APPTYPE CONSOLE}
  2. {$MODE OBJFPC}
  3.  
  4. uses Windows;
  5.  
  6. function FileExists(const FileName: UnicodeString): Boolean;
  7.  
  8.   function FoundByEnum: Boolean;
  9.   var
  10.     R: TWIN32FINDDATAW;
  11.     H: THandle;
  12.   begin
  13.     H := FindFirstFileExW(PWideChar(FileName), FindExInfoBasic, @R,
  14.       FindExSearchNameMatch, nil, 0); // It's faster, than FindFirstFile
  15.     Result := H <> INVALID_HANDLE_VALUE;
  16.     if Result then
  17.     begin
  18.       FindClose(H);
  19.       Result := (R.dwFileAttributes and FILE_ATTRIBUTE_DIRECTORY) = 0;
  20.     end;
  21.   end;
  22.  
  23. const
  24.   INVALID_FILE_ATTRIBUTES = DWORD(-1);
  25.   CNotExistsErrors = [
  26.     ERROR_FILE_NOT_FOUND,
  27.     ERROR_PATH_NOT_FOUND,
  28.     ERROR_INVALID_NAME,    // Protects from names in the form of masks like '*'
  29.     ERROR_INVALID_DRIVE,
  30.     ERROR_NOT_READY,
  31.     ERROR_INVALID_PARAMETER,
  32.     ERROR_BAD_PATHNAME,
  33.     ERROR_BAD_NETPATH];
  34. var
  35.   Attr: DWORD;
  36. begin
  37.   Attr := GetFileAttributesW(PWideChar(FileName));
  38.   if Attr = INVALID_FILE_ATTRIBUTES then
  39.     Result := not (GetLastError in CNotExistsErrors) and FoundByEnum
  40.   else
  41.     Result := (Attr and FILE_ATTRIBUTE_DIRECTORY) <> 0;
  42. end;
  43.  
  44. begin
  45.   if FileExists('c:\pagefile.sys') then
  46.     Writeln('Exists')
  47.   else
  48.     Writeln('Wrong');
  49.   Readln;
  50. end.

Mr.Madguy

  • Hero Member
  • *****
  • Posts: 844
Re: FileExists doesn't always work in Windows
« Reply #10 on: August 11, 2018, 03:49:07 pm »
Bug report for this ?
I guess, it's intentional behaviour. Standard programs, like Explorer and CMD, don't show system files. Only special software, like Far Manager, can show system files.
Is it healthy for project not to have regular stable releases?
Just for fun: Code::Blocks, GCC 13 and DOS - is it possible?

lucamar

  • Hero Member
  • *****
  • Posts: 4219
Re: FileExists doesn't always work in Windows
« Reply #11 on: August 11, 2018, 05:22:32 pm »
Bug report for this ?
I guess, it's intentional behaviour. Standard programs, like Explorer and CMD, don't show system files. Only special software, like Far Manager, can show system files.

Huh? Yes, they do if you configure them that way. Explorer shows them all right in my Windows XP machines, as well as hidden files, (almost) each and every extension, etc. And in CMD you can do DIR /AS and it shows them to you.

It is a bug because the standard FileExists() function is very simplistic, as shown in this same thread. It more or less works well for DOS ... and little else.
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.12/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

ASBzone

  • Hero Member
  • *****
  • Posts: 678
  • Automation leads to relaxation...
    • Free Console Utilities for Windows (and a few for Linux) from BrainWaveCC
Re: FileExists doesn't always work in Windows
« Reply #12 on: August 14, 2018, 04:49:05 pm »
I guess, it's intentional behaviour. Standard programs, like Explorer and CMD, don't show system files. Only special software, like Far Manager, can show system files.

The issue goes beyond files having SYSTEM or HIDDEN attributes.   I initially thought that was the issue myself, but it turns out that it wasn't.  I tested with a batch of system and hidden files.   It appears to be more related to the issue of files that are opened exclusively by some other process.  I did not test more extensively than that once I realized that FindFirst worked.
-ASB: https://www.BrainWaveCC.com/

Lazarus v2.2.7-ada7a90186 / FPC v3.2.3-706-gaadb53e72c
(Windows 64-bit install w/Win32 and Linux/Arm cross-compiles via FpcUpDeluxe on both instances)

My Systems: Windows 10/11 Pro x64 (Current)

Thaddy

  • Hero Member
  • *****
  • Posts: 14214
  • Probably until I exterminate Putin.
Re: FileExists doesn't always work in Windows
« Reply #13 on: August 14, 2018, 05:13:45 pm »
Under windows, just like with unix, find works depending on rights.
Specialize a type, not a var.

ASBzone

  • Hero Member
  • *****
  • Posts: 678
  • Automation leads to relaxation...
    • Free Console Utilities for Windows (and a few for Linux) from BrainWaveCC
Re: FileExists doesn't always work in Windows
« Reply #14 on: August 15, 2018, 09:29:07 pm »
Under windows, just like with unix, find works depending on rights.

Yes and no.

In general, this is a true statement, but the problem manifested here is not one of rights.

As shown from the earlier examples, If Exists fails in some scenarios where FindFirst and FindAllFiles are successful.  Same machine, same user credentials, same user context.


-ASB: https://www.BrainWaveCC.com/

Lazarus v2.2.7-ada7a90186 / FPC v3.2.3-706-gaadb53e72c
(Windows 64-bit install w/Win32 and Linux/Arm cross-compiles via FpcUpDeluxe on both instances)

My Systems: Windows 10/11 Pro x64 (Current)

 

TinyPortal © 2005-2018