Recent

Author Topic: [SOLVED] FileExists doesn't always work in Windows  (Read 32227 times)

Thaddy

  • Hero Member
  • *****
  • Posts: 19472
  • Glad to be alive.
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #30 on: September 01, 2020, 05:57:09 pm »
Yes Jamie, but FileExists() is cross platform too... Your Windows only solution isn't.
But you are correct for Windows: if one uses the full canonical name to the file then under Windows it should work without any other means (e.g. setting the second parameter is not necessary, but as per documentation this is not the case on Unix derivatives).
« Last Edit: September 01, 2020, 06:00:21 pm by Thaddy »
Any "programmer" that knows only one programming language is not a programmer

DJMaster

  • New Member
  • *
  • Posts: 44
    • DJMaster on GitHub
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #31 on: September 01, 2020, 06:23:00 pm »
As I guessed, FileGetSymLinkTargetInt implementation is causing the issue with OneDrive for Business files.

Please review attached sysutils_fileexists_cloud_patch.diff and apply if OK.

PBuffer.dat is a dump of TReparseDataBuffer record if anybody needs it.

Thank you.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 12959
  • FPC developer.
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #32 on: September 01, 2020, 06:26:15 pm »
As far as I can see, the issue is generated by FPC 3.2.0+ implementation of FileExists / FileGetSymLinkTargetInt with OneDrive for Business files (with ALP attributes). It is clear that all software developed with Lazarus in combination with FPC 3.2.0+ is affected by this issue. I moved back to FPC 3.0.4 just in time.

Well, as said, it seems that onedrive then simply doesn't handle them in a standard way, so a standard RTL can't access them. Anyway, we'll need more information to why it doesn't work.

DJMaster

  • New Member
  • *
  • Posts: 44
    • DJMaster on GitHub
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #33 on: September 01, 2020, 06:36:59 pm »
Well, as said, it seems that onedrive then simply doesn't handle them in a standard way, so a standard RTL can't access them. Anyway, we'll need more information to why it doesn't work.

What about sysutils_fileexists_cloud_patch.diff patch?

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 12959
  • FPC developer.
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #34 on: September 01, 2020, 09:05:01 pm »
Well, as said, it seems that onedrive then simply doesn't handle them in a standard way, so a standard RTL can't access them. Anyway, we'll need more information to why it doesn't work.

What about sysutils_fileexists_cloud_patch.diff patch?

They crossed it seems. It would be nice to have the urls that were the sources for that patch, and why the exception is needed. Maybe we can prepare more such cases.

DJMaster

  • New Member
  • *
  • Posts: 44
    • DJMaster on GitHub
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #35 on: September 01, 2020, 10:05:55 pm »
They crossed it seems. It would be nice to have the urls that were the sources for that patch, and why the exception is needed. Maybe we can prepare more such cases.

The patch is written by me.

Look at the attached testLinkFileExists.pas program.

As suggested by ASerge (Do it if you think so), I extracted the suspected routine FileGetSymLinkTargetInt and added DEBUG output text to follow the code.

I found that case $9000601A was not handled (case PBuffer^.ReparseTag of).

Saving the whole TReparseDataBuffer record content to PBuffer.dat file helped me to understand that no more info about the link is available within the extracted data.

Googling 0x9000601A I found IO_REPARSE_TAG_CLOUD_6 and other tags related to OneDrive / Cloud in general.

All Reparse Tags are listed here: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fscc/c8e77b37-3909-4fe6-a4ea-2b9d423b1ee4

That's all.


Code: [Select]
program test;

{$MODE objfpc}
{$MODESWITCH OUT}
{ force ansistrings }
{$H+}
{$modeswitch typehelpers}
{$modeswitch advancedrecords}

uses
  Windows, SysUtils;

var
  FindExInfoDefaults : TFINDEX_INFO_LEVELS = FindExInfoStandard;

function MyFileGetSymLinkTargetInt(const FileName: UnicodeString; out SymLinkRec: TUnicodeSymLinkRec; RaiseErrorOnMissing: Boolean): Boolean;
{ reparse point specific declarations from Windows headers }
const
  IO_REPARSE_TAG_MOUNT_POINT = $A0000003;
  IO_REPARSE_TAG_SYMLINK = $A000000C;
  IO_REPARSE_TAG_CLOUD = $9000001A;
  IO_REPARSE_TAG_CLOUD_1 = $9000101A;
  IO_REPARSE_TAG_CLOUD_2 = $9000201A;
  IO_REPARSE_TAG_CLOUD_3 = $9000301A;
  IO_REPARSE_TAG_CLOUD_4 = $9000401A;
  IO_REPARSE_TAG_CLOUD_5 = $9000501A;
  IO_REPARSE_TAG_CLOUD_6 = $9000601A;
  IO_REPARSE_TAG_CLOUD_7 = $9000701A;
  IO_REPARSE_TAG_CLOUD_8 = $9000801A;
  IO_REPARSE_TAG_CLOUD_9 = $9000901A;
  IO_REPARSE_TAG_CLOUD_A = $9000A01A;
  IO_REPARSE_TAG_CLOUD_B = $9000B01A;
  IO_REPARSE_TAG_CLOUD_C = $9000C01A;
  IO_REPARSE_TAG_CLOUD_D = $9000D01A;
  IO_REPARSE_TAG_CLOUD_E = $9000E01A;
  IO_REPARSE_TAG_CLOUD_F = $9000F01A;
  ERROR_REPARSE_TAG_INVALID = 4393;
  FSCTL_GET_REPARSE_POINT = $900A8;
  MAXIMUM_REPARSE_DATA_BUFFER_SIZE = 16 * 1024;
  SYMLINK_FLAG_RELATIVE = 1;
  FILE_FLAG_OPEN_REPARSE_POINT = $200000;
  FILE_READ_EA = $8;
type
  TReparseDataBuffer = record
    ReparseTag: ULONG;
    ReparseDataLength: Word;
    Reserved: Word;
    SubstituteNameOffset: Word;
    SubstituteNameLength: Word;
    PrintNameOffset: Word;
    PrintNameLength: Word;
    case ULONG of
      IO_REPARSE_TAG_MOUNT_POINT: (
        PathBufferMount: array[0..4095] of WCHAR);
      IO_REPARSE_TAG_SYMLINK: (
        Flags: ULONG;
        PathBufferSym: array[0..4095] of WCHAR);
  end;

const
  CShareAny = FILE_SHARE_READ or FILE_SHARE_WRITE or FILE_SHARE_DELETE;
  COpenReparse = FILE_FLAG_OPEN_REPARSE_POINT or FILE_FLAG_BACKUP_SEMANTICS;
var
  HFile, Handle: THandle;
  PBuffer: ^TReparseDataBuffer;
  BytesReturned: DWORD;
 
  f: file;
begin
  SymLinkRec := Default(TUnicodeSymLinkRec);

  HFile := CreateFileW(PUnicodeChar(FileName), FILE_READ_EA, CShareAny, Nil, OPEN_EXISTING, COpenReparse, 0);

  if HFile <> INVALID_HANDLE_VALUE then
    try
      GetMem(PBuffer, MAXIMUM_REPARSE_DATA_BUFFER_SIZE);
      try
        if DeviceIoControl(HFile, FSCTL_GET_REPARSE_POINT, Nil, 0,
             PBuffer, MAXIMUM_REPARSE_DATA_BUFFER_SIZE, @BytesReturned, Nil) then begin

          // Save to buffer.dat TReparseDataBuffer record content
          writeln('BytesReturned ', BytesReturned);
          AssignFile(f,'buffer.dat');
          Rewrite(f, 1);
          BlockWrite(f, PBuffer^, BytesReturned);
          closeFile(f);

          case PBuffer^.ReparseTag of
            IO_REPARSE_TAG_MOUNT_POINT: begin
              // DEBUG
              writeln('case IO_REPARSE_TAG_MOUNT_POINT');
              SymLinkRec.TargetName := WideCharLenToString(
                @PBuffer^.PathBufferMount[4 { skip start '\??\' } +
                  PBuffer^.SubstituteNameOffset div SizeOf(WCHAR)],
                PBuffer^.SubstituteNameLength div SizeOf(WCHAR) - 4);
            end;
            IO_REPARSE_TAG_SYMLINK: begin
              // DEBUG
              writeln('case IO_REPARSE_TAG_SYMLINK');
              SymLinkRec.TargetName := WideCharLenToString(
                @PBuffer^.PathBufferSym[PBuffer^.PrintNameOffset div SizeOf(WCHAR)],
                PBuffer^.PrintNameLength div SizeOf(WCHAR));
              if (PBuffer^.Flags and SYMLINK_FLAG_RELATIVE) <> 0 then
                SymLinkRec.TargetName := ExpandFileName(ExtractFilePath(FileName) + SymLinkRec.TargetName);
            end;
//            IO_REPARSE_TAG_CLOUD,
//            IO_REPARSE_TAG_CLOUD_1,
//            IO_REPARSE_TAG_CLOUD_2,
//            IO_REPARSE_TAG_CLOUD_3,
//            IO_REPARSE_TAG_CLOUD_4,
//            IO_REPARSE_TAG_CLOUD_5,
//            IO_REPARSE_TAG_CLOUD_6,
//            IO_REPARSE_TAG_CLOUD_7,
//            IO_REPARSE_TAG_CLOUD_8,
//            IO_REPARSE_TAG_CLOUD_9,
//            IO_REPARSE_TAG_CLOUD_A,
//            IO_REPARSE_TAG_CLOUD_B,
//            IO_REPARSE_TAG_CLOUD_C,
//            IO_REPARSE_TAG_CLOUD_D,
//            IO_REPARSE_TAG_CLOUD_E,
//            IO_REPARSE_TAG_CLOUD_F: begin
//              SymLinkRec.TargetName := FileName;
//            end;
            else
              // DEBUG
              writeln('case 0x', IntTohex(PBuffer^.ReparseTag,8));
          end;

          Handle := FindFirstFileExW(PUnicodeChar(SymLinkRec.TargetName), FindExInfoDefaults , @SymLinkRec.FindData,
                      FindExSearchNameMatch, Nil, 0);
          if Handle <> INVALID_HANDLE_VALUE then
          begin
            Windows.FindClose(Handle);
            SymLinkRec.Attr := SymLinkRec.FindData.dwFileAttributes;
            SymLinkRec.Size := QWord(SymLinkRec.FindData.nFileSizeHigh) shl 32 + QWord(SymLinkRec.FindData.nFileSizeLow);
          end else if RaiseErrorOnMissing then
            raise EDirectoryNotFoundException.Create(SysErrorMessage(GetLastOSError))
          else
            SymLinkRec.TargetName := '';
        end else
        begin
          SetLastError(ERROR_REPARSE_TAG_INVALID);
        end;
      finally
        FreeMem(PBuffer);
      end;
    finally
      CloseHandle(HFile);
    end;
  Result := SymLinkRec.TargetName <> '';
end;

function MyFileGetSymLinkTarget(const FileName: UnicodeString; out SymLinkRec: TUnicodeSymLinkRec): Boolean;
begin
  Result := MyFileGetSymLinkTargetInt(FileName, SymLinkRec, True);
end;

function MyLinkFileExists(FileOrDirName: String): Boolean;
var
  slr: TUnicodeSymLinkRec;
begin
  Result := MyFileGetSymLinkTargetInt(FileOrDirName, slr, False);
end;

procedure testRoutine(FileName: String);
begin
  writeln('Testing ', FileName);
  if MyLinkFileExists(FileName) then
    writeln('MyLinkFileExists')
  else
    writeln('ERROR');
  writeln;
end;

begin
  // Point to an existing OneDrive for Business file
  testRoutine(ParamStr(1));

  readln;
end.

PascalDragon

  • Hero Member
  • *****
  • Posts: 6403
  • Compiler Developer
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #36 on: September 02, 2020, 09:15:01 am »
It might probably be better to simply handle all Reparse Points that are neither IO_REPARSE_TAG_MOUNT_POINT nor IO_REPARSE_TAG_SYMLINK as non-symlinks...  :-X

ASerge

  • Hero Member
  • *****
  • Posts: 2510
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #37 on: September 02, 2020, 05:16:04 pm »
It might probably be better to simply handle all Reparse Points that are neither IO_REPARSE_TAG_MOUNT_POINT nor IO_REPARSE_TAG_SYMLINK as non-symlinks...  :-X
But then we won't know about the incorrect behavior of the function. IMHO better apply the patch, and also add SetLastError(ERROR_REPARSE_TAG_INVALID); to the block "case Reparse Tag... else".

PascalDragon

  • Hero Member
  • *****
  • Posts: 6403
  • Compiler Developer
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #38 on: September 03, 2020, 09:49:58 am »
It might probably be better to simply handle all Reparse Points that are neither IO_REPARSE_TAG_MOUNT_POINT nor IO_REPARSE_TAG_SYMLINK as non-symlinks...  :-X
But then we won't know about the incorrect behavior of the function. IMHO better apply the patch, and also add SetLastError(ERROR_REPARSE_TAG_INVALID); to the block "case Reparse Tag... else".

Then we'd have to adjust that function each and every time Microsoft adds a new tag and the life of a FPC release is much longer than the span between updates of Windows and so far none appears to really be a symlink (even the one for WSL symlinks would be incorrect to be handled). No. Only those that we know are symlinks will be handled as such. If someone needs more specific handling of reparse points then they need to do it themselves.

jamie

  • Hero Member
  • *****
  • Posts: 7855
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #39 on: September 03, 2020, 09:07:33 pm »
I am a little new to SymLinks but not totally in the dark over it...

So we can create a SymLink using CreateSymLink I believe but that does not make much sense here since this isn't the issue. however, doing the reverse to resolve the full path , would that me CreateSymbolicLink ?

There seems to be little info on going back to the full path name, I do however thing that maybe the FindFIrstFileW could resolve that and show the path name if one was interested in that info.

 Just thinking outload here,.

The only true wisdom is knowing you know nothing

PascalDragon

  • Hero Member
  • *****
  • Posts: 6403
  • Compiler Developer
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #40 on: September 04, 2020, 04:44:46 pm »
@DJMaster: I've now adjusted FileGetSymLinkTarget in 46761 to treat files with unknown tags as non-symlinks. You can check whether this fixes your problem. If so I'll have it merged to 3.2.1.

ASerge

  • Hero Member
  • *****
  • Posts: 2510
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #41 on: September 04, 2020, 05:20:08 pm »
...to treat files with unknown tags as non-symlinks.
After looking at more than 50 tags for the reparse points, I'm inclined to agree with you. Only two options are really links, the rest are only additional information for a server and do not change the file path

DJMaster

  • New Member
  • *
  • Posts: 44
    • DJMaster on GitHub
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #42 on: September 04, 2020, 07:24:12 pm »
@DJMaster: I've now adjusted FileGetSymLinkTarget in 46761 to treat files with unknown tags as non-symlinks. You can check whether this fixes your problem. If so I'll have it merged to 3.2.1.

Revision 46761 is not solving the FileExists issue (SymLinkRec.TargetName = '' with OneDrive for Business files).
« Last Edit: September 04, 2020, 08:36:23 pm by DJMaster »

DJMaster

  • New Member
  • *
  • Posts: 44
    • DJMaster on GitHub
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #43 on: September 04, 2020, 08:53:42 pm »
Please review attached additional patch.

Thank you.


Code: Pascal  [Select][+][-]
  1. Index: rtl/win/sysutils.pp
  2. ===================================================================
  3. --- rtl/win/sysutils.pp (revision 46769)
  4. +++ rtl/win/sysutils.pp (working copy)
  5. @@ -474,6 +474,8 @@
  6.                if (PBuffer^.Flags and SYMLINK_FLAG_RELATIVE) <> 0 then
  7.                  SymLinkRec.TargetName := ExpandFileName(ExtractFilePath(FileName) + SymLinkRec.TargetName);
  8.              end;
  9. +            else
  10. +              SymLinkRec.TargetName := FileName;
  11.            end;
  12.  
  13.            if SymLinkRec.TargetName <> '' then begin
  14.  

PascalDragon

  • Hero Member
  • *****
  • Posts: 6403
  • Compiler Developer
Re: [SOLVED] FileExists doesn't always work in Windows
« Reply #44 on: September 07, 2020, 09:32:33 am »
Please review attached additional patch.

No. Any tag that is not a SYMLINK or MOUNT_POINT is not a symlink, thus FileGetSymLinkTargetInt must return False for them. If FileExists still does not work for your files, then the problem is elsewhere. Probably FileOrDirExists shouldn't assume that a file is a link only because the REPARSE_POINT attribute is set or FileGetSymLinkTargetInt should provide information whether it's really considered a link...

 

TinyPortal © 2005-2018