Recent

Author Topic: FileExists() behaviour changed on Linux  (Read 7391 times)

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
FileExists() behaviour changed on Linux
« on: January 24, 2021, 01:10:49 pm »
On Linux x86_64, there appears to have been a change in FileExists() between FPC 3.0.4 and 3.2.0. This has broken some delicate code (resulting in indelicate language after a long debugging session) which walked the /sys/devices tree trying to determine properties of a serial port... I've not found any way other than brute force to determine things like hardware and driver name since different drivers (kernel modules) store this information differently.

This is the content of the relevant directory:


/sys/devices/pci0000:00/0000:00:1d.2/usb4/4-2/4-2:1.0/ttyUSB0# ls -l
total 0
lrwxrwxrwx 1 root root    0 Jan 24 10:23 driver -> ../../../../../../../bus/usb-serial/drivers/cp210x
-r--r--r-- 1 root root 4096 Jan 24 10:23 port_number
drwxr-xr-x 2 root root    0 Jan 24 10:24 power
lrwxrwxrwx 1 root root    0 Jan 24 10:23 subsystem -> ../../../../../../../bus/usb-serial
drwxr-xr-x 3 root root    0 Jan 24 10:23 tty
-rw-r--r-- 1 root root 4096 Jan 24 10:23 uevent


This is the relevant code fragment:

Code: Pascal  [Select][+][-]
  1. DebugWriteF('Test existence "%s" "%s" / "%s"\n', [dir, candidate, content]);
  2. if FileExists(dir + candidate + '/' + content) then begin
  3.   DebugWriteF('Test predicate "%s" "%s" "%s" "%s"\n', [dir, candidate, content, extra]);
  4.   if wtp2(dir, candidate, content, extra, hint) then
  5.     exit(dir + candidate + '/')
  6. end
  7.  

Code compiled with 3.0.4 gives me this output:


Testing against driver name usb-serial/drivers/cp210x
Test existence "/sys/devices/pci0000:00/0000:00:1d.2/usb4/4-2/4-2:1.0/" "ttyUSB0" / "driver"
Test predicate "/sys/devices/pci0000:00/0000:00:1d.2/usb4/4-2/4-2:1.0/" "ttyUSB0" "driver" "usb-serial/drivers/cp210x"
...


Code compiled with 3.2.0 gives me this output:


Testing against driver name usb-serial/drivers/cp210x
Test existence "/sys/devices/pci0000:00/0000:00:1d.2/usb4/4-2/4-2:1.0/" "ttyUSB0" / "driver"
...


Before I dig in again myself, what is FileExists() objecting to here: the fact that it's looking at a symlink, the fact that it's looking at a symlinked directory, or the fact that it's looking at a directory?

Is this an intentional change, i.e. which will persist in later versions of the compiler, or a regression which I can avoid by telling people not to use that specific version of compiler?

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Kays

  • Hero Member
  • *****
  • Posts: 569
  • Whasup!?
    • KaiBurghardt.de
Re: FileExists() behaviour changed on Linux
« Reply #1 on: January 24, 2021, 02:11:17 pm »
[…] Is this an intentional change […]? […]
Yes, it’s intentional.
Yours Sincerely
Kai Burghardt

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: FileExists() behaviour changed on Linux
« Reply #2 on: January 24, 2021, 02:14:07 pm »
[…] Is this an intentional change […]? […]
Yes, it’s intentional.

Thanks for that. So it's the directory check, not that it's going via a symlink.

The code finds and reads a Mastech MS2115B multimeter, and I'll be reusing it to read a pulse oximeter when the hardware arrives since the serial chip and overall requirement is the same. https://github.com/MarkMLl/Mastech_ms2115b

MarkMLl
« Last Edit: January 24, 2021, 02:18:58 pm by MarkMLl »
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

jamie

  • Hero Member
  • *****
  • Posts: 6090
Re: FileExists() behaviour changed on Linux
« Reply #3 on: January 24, 2021, 02:49:39 pm »
I think someone made a mistake in their logic of thinking..

from day one of my experience of coding in OSes, everything is a file, even in DOS the director "File" had it's own extension using the the "." or ".." extensions. So with that it was easy to copy a whole directory because it was treated as a single file..


 Maybe they may want to revaluate their decision on this ?
The only true wisdom is knowing you know nothing

winni

  • Hero Member
  • *****
  • Posts: 3197
Re: FileExists() behaviour changed on Linux
« Reply #4 on: January 24, 2021, 03:01:49 pm »
I think someone made a mistake in their logic of thinking..

from day one of my experience of coding in OSes, everything is a file, even in DOS the director "File" had it's own extension using the the "." or ".." extensions. So with that it was easy to copy a whole directory because it was treated as a single file..


 Maybe they may want to revaluate their decision on this ?

Yes that is true

It is the old Unix idea "Everything is a file"

So a symlink is nothing but a file pointing to another file.

Winni

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: FileExists() behaviour changed on Linux
« Reply #5 on: January 24, 2021, 03:10:59 pm »
I think someone made a mistake in their logic of thinking..

from day one of my experience of coding in OSes, everything is a file, even in DOS the director "File" had it's own extension using the the "." or ".." extensions. So with that it was easy to copy a whole directory because it was treated as a single file..

 Maybe they may want to revaluate their decision on this ?

From my POV I'm glad to know the status of things and am changing my directory walking code to make both calls where necessary in order to emulate the original behaviour since I'm not confident that switching to DirectoryExists() would always be correct. However I would make two comments.

The first is that in my case there will be a performance hit on what was already a fairly slow brute-force operation since if FileExists() fails I'm now making a second call to DirectoryExists(). Since this is Linux-specific I think I could probably improve things by making an fpStat() call instead.

The second is that IMO (and with all respect to Jamie), "everything is a file" is a mantra that is invoked far too often. "Everything appears in the filesystem namespace" is one thing, but increasingly there are physical and socket-like devices which quite simply don't map well onto the file API... you can open them by name and everything else is a private API via ioctl calls. And then there's e.g. Linux network devices which don't even appear in the filesystem namespace.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: FileExists() behaviour changed on Linux
« Reply #6 on: January 24, 2021, 03:18:17 pm »
So a symlink is nothing but a file pointing to another file.

Hmm. A symlink is an instance of an object which reimplements open() etc. via redirection.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

jamie

  • Hero Member
  • *****
  • Posts: 6090
Re: FileExists() behaviour changed on Linux
« Reply #7 on: January 24, 2021, 03:26:30 pm »
Well I've always liked the methodology of LINUX/UNIX of everything is a file because with devices , the last time I did serial on Linux, you had the device which acted as a file (directory tree Device NAME) and then sub files which were a data channel file and the control file and from that you could simply open two files within the directory (Device name) one being the control and one being the data file. Read or write...

 Like I said its been years since I've done this but that is basically how I did it, get the device name which acted like file or Directory name and then you could in theory enumerate the sub files of that device to see what was available.

some devices maybe read only like a Mouse for example but in any case that device folder/Director held attributes about it..

 By doing work this way you never had to concern yourself with machine design logic from the application level where as with DOS and windows it's still a need to know these things.
The only true wisdom is knowing you know nothing

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: FileExists() behaviour changed on Linux
« Reply #8 on: January 24, 2021, 03:31:53 pm »
Somebody needs to fix the documentation

https://www.freepascal.org/docs-html/current/rtl/sysutils/fileexists.html

"On Unixes, passing a directory name will result in True. The rationale is that on UNIX, a directory is a file as well."

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: FileExists() behaviour changed on Linux
« Reply #9 on: January 24, 2021, 03:34:13 pm »
Well I've always liked the methodology of LINUX/UNIX of everything is a file because with devices , the last time I did serial on Linux, you had the device which acted as a file (directory tree Device NAME) and then sub files which were a data channel file and the control file and from that you could simply open two files within the directory (Device name) one being the control and one being the data file. Read or write...

I've definitely never seen anything like that on Linux or for that matter Solaris, although it could obviously be done like that on e.g. smart multiport cards.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

jamie

  • Hero Member
  • *****
  • Posts: 6090
Re: FileExists() behaviour changed on Linux
« Reply #10 on: January 24, 2021, 07:54:43 pm »
I have serial port code and sound card code here on an redhat machine that did exactly that for those services, They open multiple files against the device for data access and control

 I may have some source code kicking around that I could post, I need to find the archive I back them up on.
The only true wisdom is knowing you know nothing

Bart

  • Hero Member
  • *****
  • Posts: 5275
    • Bart en Mariska's Webstek
Re: FileExists() behaviour changed on Linux
« Reply #11 on: January 24, 2021, 08:20:55 pm »
The reason for the change is consistency across platforms.
The behaviour on Windows is Delphi compatible.
It also avoids code like:
Code: Pascal  [Select][+][-]
  1.   if FileExists(Fn) and not DirectoryExists(Fn) then
  2.     DoSomeThingWithThe File(Fn);
Constructs like that appeared in LCL code IIRC.

Bart

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: FileExists() behaviour changed on Linux
« Reply #12 on: January 24, 2021, 09:29:33 pm »
The reason for the change is consistency across platforms.
The behaviour on Windows is Delphi compatible.

IMO, a gross change which flies in the face of documented behaviour would have merited a parameter change to make sure that people were aware of it, e.g making the 2nd parameter mandatory and using it to specify not just whether a symlink should be followed but what would match.

How long has this change been planned? Why aren't the developers using the deprecated/experimental qualifiers or introducing changepending/changedrecently ones?

Yes, I appreciate that this was in the release notes and that I should have checked earlier. But what's going to be changed next: clBlack swapped with clWhite?

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

jamie

  • Hero Member
  • *****
  • Posts: 6090
Re: FileExists() behaviour changed on Linux
« Reply #13 on: January 24, 2021, 09:40:50 pm »
how about a "PathExist" function...

at least that follows some other language thinking..

I will stick to FileAttr I guess for a single function for now..
The only true wisdom is knowing you know nothing

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: FileExists() behaviour changed on Linux
« Reply #14 on: January 24, 2021, 09:54:38 pm »
Problem there is that "path" will get confused with the directory portion (i.e. with name.extension trimmed). I was looking at FileAttr() earlier, and it looks rather "DOSsy": in my case I do a fair amount of checking of unix-domain sockets etc., and in that case FileAttr()'s behaviour should be undefined.

OI! DEVS! Are you going to change things so that FileExists() doesn't match a unix FIFO or unix-domain socket, in order to conform to Delphi's behaviour? After all, Delphi doesn't run on unix so can't be expected to match them, so it would be entirely reasonable to follow things through to their logical conclusion :-/

Look, I'm sorry if I sound negative about all this. But this was a BREAKING CHANGE and it was my code that got broken.

SHOULD. NOT. HAVE. HAPPENED.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

 

TinyPortal © 2005-2018