Recent

Author Topic: FileExists() behaviour changed on Linux  (Read 4265 times)

Bart

  • Hero Member
  • *****
  • Posts: 4169
    • Bart en Mariska's Webstek
Re: FileExists() behaviour changed on Linux
« Reply #15 on: January 24, 2021, 09:56:20 pm »
I will stick to FileAttr I guess for a single function for now..
Try FindFirst('C:\', faDirectory, SR) to see if 'C:\' exists and get a nice surprise.
Some code in fpc uses that to detect if it is a directory and fails.

Root folders on NT systems have the system and hidden bit set...

Bart

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1272
  • Former Delphi 1-7, 10.2 User
Re: FileExists() behaviour changed on Linux
« Reply #16 on: January 24, 2021, 10:36:54 pm »
This statement is just WRONG. On UNIX systems a directory IS a file.

Also, the FPC "RenameFile" function also renames directories, but now the FileExists function no longer works for directories.

This is madness.
Lazarus 2.1 r64368 FPC 3.3.1 r48100 macOS 10.14.6 Xcode 11.3.1
Lazarus 2.1 r64455 3.3.1 r48688  macOS 11.2 aarch64 Xcode 12.4
Lazarus 2.1 r61574 3.3.1 r42318 FreeBSD 12.1 amd64 VMware VM
Lazarus 2.1 r61574 3.0.4 Ubuntu 20.04 Parallels VM
Lazarus 2.0.10 3.2.0 Win10 Parallels VM

BeniBela

  • Hero Member
  • *****
  • Posts: 777
    • homepage
Re: FileExists() behaviour changed on Linux
« Reply #17 on: January 24, 2021, 11:06:06 pm »
I just noticed that this breaks my code as well yesterday



How long has this change been planned? Why aren't the developers using the deprecated/experimental qualifiers or introducing changepending/changedrecently ones?


I do not think there was any planing involved

https://bugs.freepascal.org/view.php?id=8900

 https://bugs.freepascal.org/view.php?id=16938

 https://bugs.freepascal.org/view.php?id=32362

MarkMLl

  • Hero Member
  • *****
  • Posts: 2061
Re: FileExists() behaviour changed on Linux
« Reply #18 on: January 24, 2021, 11:06:42 pm »
This statement is just WRONG. On UNIX systems a directory IS a file.

As is a symlink. This is a can of worms :-(

Quote
Also, the FPC "RenameFile" function also renames directories, but now the FileExists function no longer works for directories.

Oh Lord :-(

OK, I've got a suggestion. "File" in these functions shouldn't be interpreted as "this is a file containing arbitrary data", but as "this is something in the filesystem namespace which is subject to the unix file API".

The change has happened, and there is probably a fair number of "on the ball" users who have moved onto 3.2.0, in most cases probably not realising there is an issue which might bite them when they recompile their code using it. Sooner rather than later we need a 3.2.2 which either reverts this or (and I think this would be my preference) changes FileExists() to make the second parameter mandatory and used to represent what it will match: that will alert anybody who is recompiling their code regularly to the fact that there are problems in this area.

And obviously FindFirst() etc. need to be reviewed for compliance with the spirit of POSIX or whatever MS claims to conform to.

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.

jamie

  • Hero Member
  • *****
  • Posts: 4177
Re: FileExists() behaviour changed on Linux
« Reply #19 on: January 24, 2021, 11:15:28 pm »
I just was playing around with something here..

Code: Pascal  [Select][+][-]
  1. Function PathExists(Name :String):Boolean; inline;
  2.  begin
  3.    Result := FileGetAttr(Name) >=0;
  4. End;
  5.  

This seems to work well on windows... I also believe you can use unc file names within this...

The function returns -1 if there is some error, Meaning if it can't read it then it must not exist!

This was hand typed so forget any errors that maybe in this tag..
The only true wisdom is knowing you know nothing

MarkMLl

  • Hero Member
  • *****
  • Posts: 2061
Re: FileExists() behaviour changed on Linux
« Reply #20 on: January 24, 2021, 11:32:21 pm »
> Result := FileGetAttr(Name) >=0;

But since FileGetAttr() appears to be defined in terms of a DOS-style filesystem, is it really applicable when considering a unix or NTFS filesystem which has different attributes?

The same applies to fpStat() and in particular to the st_mode field that it returns, but I think it would be easier to map Windows attributes onto POSIX than to shoehorn unix/POSIX attributes into DOS-style function results :-/

The bottom line is that most OSes at least play lipservice to POSIX,  so if anybody's out of line here it's Delphi which doesn't go out of its way to consider non-Windows targets. So WTF are we still going out of our way to conform to Delphi's broken ideas?

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.

dsiders

  • Sr. Member
  • ****
  • Posts: 423
Re: FileExists() behaviour changed on Linux
« Reply #21 on: January 25, 2021, 12:41:34 am »
So WTF are we still going out of our way to conform to Delphi's broken ideas?

Because the faithful do not eat their sacred cows... even during a famine. ;)
Lazarus 2.1 (SVN) / FPC 3.0.4 / FPC 3.2.0 / x86-win64 / Windows 8.1

Kays

  • Sr. Member
  • ****
  • Posts: 289
  • Whasup!?
    • KaiBurghardt.de
Re: FileExists() behavior changed on Linux
« Reply #22 on: January 25, 2021, 01:02:09 am »
[…] I'm […] changing my directory walking code to make both calls where necessary in order to emulate the original behaviour since I'm not confident that switching to DirectoryExists() would always be correct […]
Oh dear. You’re changing it by hand? I’d quickly drop in some of my central unit or include file something like:
Code: Pascal  [Select][+][-]
  1. {$if FPC_fullVersion >= 030200}// fileExist `false` on directories
  2. function fileExists(const s: string; const r: Boolean = true): Boolean;
  3. begin
  4.         {$push}
  5.         {$boolEval off} // until `or_else` becomes available
  6.         fileExists := sysUtils.fileExists(s, r) or
  7.                 sysUtils.directoryExists(s, r);
  8.         {$pop}
  9. end;
  10. {$endIf}
Boom! No need change your code. Just verify you never used the FQI somewhere else
Code: Bash  [Select][+][-]
  1. egrep -Ri 'sysutils[[:blank:]]*.[[:blank:]]*fileexists' ./*

[…] if FileExists() fails I'm now making a second call to DirectoryExists(). Since this is Linux-specific I think I could probably improve things by making an fpStat() call instead.
Walking the SysFS directory tree already by itself is a Linux-specific thing. Ergo, there is no benefit in attempting to keep your code cross-platform. Examining an strace(1) of fileExists I see every successful access(2) is (now) followed by a stat(2) just to filter out any matching directories (“false positives”). In fact, fpAccess(…, F_OK) would be sufficient for you.

Somebody needs to fix the documentation […]
It’s already documented since revision 1730.

[…] So a symlink is nothing but a file pointing to another file. […]
A symbolic link is a shortcut for typing out a path. A symbolic link may point to non-existent files and can point across file system boundaries. (For the purposes of this statement “file” also includes directories.)

The reason for the change is consistency across platforms.
The behaviour on Windows is Delphi compatible. […]
Here we go again. Delphi-compatibility is like the holy grail. To quote trev here:
[…] This is madness.
I actually think the change is warranted though, but for the reason of having consistent behavior across all platforms. If Windoze doesn’t deem directories as files, then fileExists should fail on all platforms; there’s directoryExists just for directories.
« Last Edit: January 25, 2021, 03:51:32 am by Kays »
Yours Sincerely
Kai Burghardt

lucamar

  • Hero Member
  • *****
  • Posts: 3591
Re: FileExists() behaviour changed on Linux
« Reply #23 on: January 25, 2021, 01:23:18 am »
So WTF are we still going out of our way to conform to Delphi's broken ideas?

For the same old reason: there is still a lot more code intended for Delphi that might have to behave "the right way" in FreePascal than viceversa. We might like it or not but Delphi compatibility is still important.

IMHO, of course. :-\
Turbo Pascal 3 CP/M - Amstrad PCW 8256 (512 KB !!!) :P
Lazarus/FPC 2.0.8/3.0.4 & 2.0.10/3.2.0 - 32/64 bits on:
(K|L|X)Ubuntu 12..18, Windows XP, 7, 10 and various DOSes.

MarkMLl

  • Hero Member
  • *****
  • Posts: 2061
Re: FileExists() behavior changed on Linux
« Reply #24 on: January 25, 2021, 10:20:05 am »
Oh dear. You’re changing it by hand? I’d quickly drop in some of my central unit or include file something like:

Inserting a function like the one you suggested was no big deal. Unfortunately I don't understand my code well enough to be 100% confident of where I definitely needed the old behaviour or where the new one would be better, so I've ended up with perhaps a dozen edits over three projects.

Quote
Walking the SysFS directory tree already by itself is a Linux-specific thing. Ergo, there is no benefit in attempting to keep your code cross-platform.

It's a general-purpose function which potentially could be used for other things. As such it's worth keeping it portable.

Quote
It’s already documented since revision 1730.

OH NO IT'S EFFING NOT!!!

The live documentation for FPC 3.2.0 is at https://www.freepascal.org/docs-html/current/rtl/sysutils/fileexists.html and it says "On Unixes, passing a directory name will result in True. The rationale is that on UNIX, a directory is a file as well."

Apropos the remainder... my beef is not so much with the change in behaviour, but the fact that this is an undocumented breaking change which has been slipped in with no attempt at getting the compiler to warn the user that his program might no longer work as expected.

As Pascal users we boast that the language's type checking etc. is better- or at least longer-established- than that of the competition, and that robust design and careful implementation helps catch problems early and makes programs more reliable.

But the reputation for caution and consistency that the developers have cultivated over the last 20 years risks being swept away by a single ill-considered change.

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.

Thaddy

  • Hero Member
  • *****
  • Posts: 10704
Re: FileExists() behaviour changed on Linux
« Reply #25 on: January 25, 2021, 10:24:33 am »
The documentation is adapted in 3.3.1 and possibly 3.2.1 as back port. Both are FF'ng (op.cit) not releases. The online docs reflect the latest release which is 3.2.0.
You can build the latest docs yourself, though.

MarkMLl

  • Hero Member
  • *****
  • Posts: 2061
Re: FileExists() behaviour changed on Linux
« Reply #26 on: January 25, 2021, 10:44:25 am »
The documentation is adapted in 3.3.1 and possibly 3.2.1 as back port. Both are FF'ng (op.cit) not releases. The online docs reflect the latest release which is 3.2.0.
You can build the latest docs yourself, though.

Let's try again. If we point a user at the download page https://www.freepascal.org/download.html it explicitly says "The latest release is 3.2.0" and offers him e.g. ftp://ftp.hu.freepascal.org/pub/fpc/dist/3.2.0/x86_64-linux/fpc-3.2.0-x86_64-linux.tar.

If he wants to read the documentation for FileExists() he will go to https://www.freepascal.org/docs-html/current/rtl/sysutils/fileexists.html

That is the public documentation for the FileExists() function being shipped in 3.2.0.

In words of one sound each, IT IS WRONG AND NEEDS TO BE FIXED.

Otherwise the only thing I can conclude is that the core developers have absolutely no interest in anybody other than themselves using the compiler.

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.

dbannon

  • Hero Member
  • *****
  • Posts: 1294
    • tomboy-ng, a rewrite of the classic Tomboy
Re: FileExists() behaviour changed on Linux
« Reply #27 on: January 25, 2021, 10:58:58 am »
Putting aside, momentarily, the process for this change, the change itself, IMHO, makes sense.  As a long time Unix/Linux user, yes I am aware that everything in Unix is a file, thats not just a characteristic, its a key feature. But only at a Operating System level. I know for example, you can open a directory with vi, but I have no idea what you happen if I wrote changes to it ....

For an application, a file is different to a directory, an application does not care how a directory is treated by the OS, it cares that a directory can contain files, it cannot 'open' a directory to read data, it cannot find multiple files in a file.

If, at an application level, I want to know if a directory exists, it seems sensible to me to use DirectoryExists(). Similarly, for a file, FileExists().  I care not that its Delphi compatible, its sensible in its own right. At an application level.

Davo
Lazarus 2, Linux (and reluctantly Win10, OSX)
My Project - https://github.com/tomboy-notes/tomboy-ng

Thaddy

  • Hero Member
  • *****
  • Posts: 10704
Re: FileExists() behaviour changed on Linux
« Reply #28 on: January 25, 2021, 11:54:36 am »
When one uses features of a non-release, you will have to build the docs. As simple as that. If you can build the non-releases, like fixes and trunk, you should also be able to build the docs as well. >:D The above is a non-discussion imnsho.

Note it would be a nice feature for fpcdeluxe to build the docs as well, but e.g. on Windows this is not trivial in my experience. Linux, no problems.
« Last Edit: January 25, 2021, 11:58:20 am by Thaddy »

MarkMLl

  • Hero Member
  • *****
  • Posts: 2061
Re: FileExists() behaviour changed on Linux
« Reply #29 on: January 25, 2021, 11:59:12 am »
When one uses features of a non-release, you will have to build the docs. As simple as that. If you can build the non-releases, like fixes and trunk, you should also be able to build the docs as well. >:D The above is a non-discussion imnsho.

Thaddy. This change is in the released 3.2.0. It should be documented.

MarkMLl
Turbo Pascal v1 on CCP/M-86, multitasking with LAN and graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.

 

TinyPortal © 2005-2018