Recent

Author Topic: [SOLVED] Can I make 64bit system-calls in 32bit progs for 64bit file timestamps?  (Read 7258 times)

Hartmut

  • Hero Member
  • *****
  • Posts: 742
In unit "baseunix" there are 2 functions fpUtime() and fpStat() to set and get the timestamp of a file. In a 32-bit program both functions use 32-bit variables to pass the timestamps and so they are restricted to the years 1901..2038. For a couple of reasons I want some more.

In a 64-bit program both functions use 64-bit variables to pass the timestamps and so more then enough years are possible. But for a couple of reasons I would like to use this in a 32-bit program, if possible. What I want to avoid is having to call console programs like "touch" and "ls -l" e.g. via "TProcess" each time to accomplish this.

So what I want to know is, if / how it is possible to call the existing 64-bit versions of fpUtime() and fpStat() from a 32-bit program. I use Linux Ubuntu 64-bit. My imagination is, because a 32-bit program obviously can make system calls to a 64-bit OS, that there must exist some kind of "interface" to connect those 2 worlds, but I have no idea how that "works" and if / how it's possible to use this.

I had a look into the sources of unit "baseunix" in FPC 3.2.0 and found:
Code: Pascal  [Select][+][-]
  1. Function fpUtime(path:pchar;times:putimbuf):cint;
  2. begin
  3.   fputime:=do_syscall(syscall_nr_utime,TSysParam(path),TSysParam(times));
  4. end;
  5.  
  6. // and for 32-bit:
  7. function Do_SysCall(sysnr,param1,param2:TSysParam):TSysResult; register; external name 'FPC_SYSCALL2';
  8. // with:
  9. const syscall_nr_utime = 30;
  10. type TSysResult = longint;
  11.      TSysParam  = Longint;
  12.      putimbuf is a pointer to a record of 2 x longint
  13.  
  14. // while for 64-bit:
  15. function Do_SysCall(sysnr,param1,param2:TSysParam):TSysResult;  external name 'FPC_SYSCALL2';
  16. // with:
  17. const syscall_nr_utime = 132;
  18. type TSysResult = int64;
  19.      TSysParam  = int64;
  20.      putimbuf is a pointer to a record of 2 x int64

Function fpStat() also "ends" with a do_syscall(). But from this point I don't know to "follow" what happens in and after the do_syscall() is invoked...

If there is no direct way, maybe it is possible to create a unit or some kind of "library", which does the job? Or that something like this already exists which I can use?

Thanks in advance for your help.

Update: because this Topic got a little longer I added the answer and a short summary in reply #35
« Last Edit: February 07, 2021, 05:12:09 pm by Hartmut »

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #1 on: January 27, 2021, 09:45:43 am »
So what I want to know is, if / how it is possible to call the existing 64-bit versions of fpUtime() and fpStat() from a 32-bit program. I use Linux Ubuntu 64-bit. My imagination is, because a 32-bit program obviously can make system calls to a 64-bit OS, that there must exist some kind of "interface" to connect those 2 worlds, but I have no idea how that "works" and if / how it's possible to use this.

No, from a 32-bit process you only have access to the 32-bit compatible interface of the kernel.

What you need to do instead is to use the correct system call that provides 64-bit timestamps (fpStat should already do that, if it doesn't please report a bug). In case of utime or utimes you should look at the system calls utimensat and utimensat_time64 (and maybe also report a bug so that the y2038 problem is handled correctly).

Function fpStat() also "ends" with a do_syscall(). But from this point I don't know to "follow" what happens in and after the do_syscall() is invoked...

There is nothing to follow, because all do_syscall does is invoke the kernel with the specified arguments and to correctly retrieve the return value. Anything in between is handled directly by the kernel so you'd need to look at the syscall interface of the kernel (been there, done that).

Hartmut

  • Hero Member
  • *****
  • Posts: 742
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #2 on: January 27, 2021, 12:22:24 pm »
Thanks a lot PascalDragon for your reply. But I did not understand all you wrote because I'm not an expert for those things like you.

No, from a 32-bit process you only have access to the 32-bit compatible interface of the kernel.
Does this mean, that a 64-bit Linux provides 2 completely separate "interfaces": one complete set with all system calls only for 64-bit programs and a second complete set with all system calls only for 32-bit programs?

Quote
What you need to do instead is to use the correct system call that provides 64-bit timestamps (fpStat should already do that, if it doesn't please report a bug).
Hmm: when I call fpStat() in a 32-bit program (FPC 3.2.0) it returns a record, where the 3 timestamps are in 32-bit variables, as also shown in https://www.freepascal.org/docs-html/current/rtl/baseunix/stat.html (type Cardinal = LongWord = 32-bit). Is that the bug which I should report?

Quote
In case of utime or utimes you should look at the system calls utimensat and utimensat_time64 (and maybe also report a bug so that the y2038 problem is handled correctly).
I did not find utimensat and utimensat_time64 in unit "baseunix". Then I googled for both and found hard stuff but did not find something what helped me to proceed: it seems that 'utimensat' uses only 32-bit variables (in a 32-bit program) and that 'utimensat_time64' exists only in kernel >= 5.1, but I have 4.15 (Ubuntu 18.04).

So I searched the sources of FPC 3.2.0 for both words and found only 2 (Linux related) matches for 'syscall_nr_utimensat':
1) in file <installdir>/fpcsrc/rtl/linux/sysnr-gen.inc => const syscall_nr_utimensat = 88;
2) in file <installdir>/fpcsrc/rtl/linux/bunxsysc.inc =>
Code: Pascal  [Select][+][-]
  1. {$if defined(generic_linux_syscalls)}
  2. Function fpUtime(path:pchar;times:putimbuf):cint;
  3. var
  4.   tsa: Array[0..1] of timespec;
  5. begin
  6.   tsa[0].tv_sec := times^.actime;
  7.   tsa[0].tv_nsec := 0;
  8.   tsa[1].tv_sec := times^.modtime;
  9.   tsa[1].tv_nsec := 0;
  10.   fputime:=do_syscall(syscall_nr_utimensat,AT_FDCWD,TSysParam(path),
  11.                       TSysParam(@tsa),0);
  12. end;
I had the idea to try to get this running, but type 'timespec' uses only 32-bit variables for the timestamps in a 32-bit program, so I think it can't work.

Please, what did you mean, what should I try exactly? Sorry, I'm a beginner to this stuff. Thanks for your help.

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #3 on: January 27, 2021, 01:43:48 pm »
No, from a 32-bit process you only have access to the 32-bit compatible interface of the kernel.
Does this mean, that a 64-bit Linux provides 2 completely separate "interfaces": one complete set with all system calls only for 64-bit programs and a second complete set with all system calls only for 32-bit programs?

Quote
What you need to do instead is to use the correct system call that provides 64-bit timestamps (fpStat should already do that, if it doesn't please report a bug).
Hmm: when I call fpStat() in a 32-bit program (FPC 3.2.0) it returns a record, where the 3 timestamps are in 32-bit variables, as also shown in https://www.freepascal.org/docs-html/current/rtl/baseunix/stat.html (type Cardinal = LongWord = 32-bit). Is that the bug which I should report?

Ah, that was only for the block counts, not the timestamps... So yes, please report.

Quote
In case of utime or utimes you should look at the system calls utimensat and utimensat_time64 (and maybe also report a bug so that the y2038 problem is handled correctly).
I did not find utimensat and utimensat_time64 in unit "baseunix". Then I googled for both and found hard stuff but did not find something what helped me to proceed: it seems that 'utimensat' uses only 32-bit variables (in a 32-bit program) and that 'utimensat_time64' exists only in kernel >= 5.1, but I have 4.15 (Ubuntu 18.04).

Then that essentially means that a 32-bit program on a kernel older than 5.1 can not get 64-bit timestamps this way. At least if I haven't overlooked anything, but for that I'd have to look deeper for which right now I don't have the time. So best create a bugreport for fpUtime as well so that it won't be forgotten.

Please, what did you mean, what should I try exactly? Sorry, I'm a beginner to this stuff. Thanks for your help.

Best leave it then. Report the bugs and wait for a fix, cause that stuff definitely isn't easy.

Kays

  • Hero Member
  • *****
  • Posts: 569
  • Whasup!?
    • KaiBurghardt.de
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #4 on: January 27, 2021, 04:22:30 pm »
[…] Then that essentially means that a 32-bit program on a kernel older than 5.1 can not get 64-bit timestamps this way. […]
I agree. Linux has a die-hard system call policy: They maintain backward compatibility indefinitely.

Currently, AFAIK all …64 system calls are for file size reasons only. As of yet, there are no legit reasons to provide 32-bit system calls accepting 64-bit time stamps. Your software should deal with that issue nevertheless, but you just do not have a legit reason to, for example, change a file’s mtime to a point beyond 2021 (or even 2037).

I guarantee you we will adopt any necessary changes in our baseUnix in due course. However, I can’t guarantee you 32-bit computers will still be deemed “support-worthy” in 2038.
Yours Sincerely
Kai Burghardt

Lutz Mändle

  • Jr. Member
  • **
  • Posts: 65
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #5 on: January 27, 2021, 04:43:40 pm »
Hi,
the timestamps in the stat record are unsigned values, therefore they go until 2106-02-07 06:28:15 UTC for a 32-bit executable on a 64-bit system.

Here comes a small test program, which can compiled to 32-bit and 64-bit resp. and until the above mentioned date they show no difference.

Code: Pascal  [Select][+][-]
  1. program testfpstat;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   BaseUnix, DateUtils, SysUtils;
  7.  
  8. var
  9.   fname,s:string;
  10.   statbuf:stat;
  11.   dt:TDateTime;
  12.   p:Pointer;
  13.  
  14. begin
  15.   WriteLn('Bitness of executable: ', SizeOf(p) * 8);
  16.  
  17.   fname:=ParamStr(1);
  18.   if not FileExists(fname) then
  19.   begin
  20.     WriteLn('no filename given');
  21.     Exit;
  22.   end;
  23.  
  24.   FpStat(PChar(fname),statbuf);
  25.  
  26.   WriteLn('Filename:   ', fname);
  27.   WriteLn('mtime__sec: ', statbuf.st_mtime);
  28.   WriteLn('mtime_nsec: ', statbuf.st_mtime_nsec);
  29.   dt:=UnixToDateTime(statbuf.st_mtime)+ statbuf.st_mtime_nsec/86400/1000000000;
  30.   s:=FormatDateTime('yyyy-mm-dd hh:nn:ss.zzz',dt);
  31.   WriteLn('Filedate:   ', s);
  32. end.
  33.  

HTH

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #6 on: January 27, 2021, 04:59:21 pm »
the timestamps in the stat record are unsigned values, therefore they go until 2106-02-07 06:28:15 UTC for a 32-bit executable on a 64-bit system.

I'd note at this point that that is highly platform-specific, since that data structure is one that Linux does not attempt to keep consistent across different architectures. The rational here is that when Linux was ported to a new platform it tried to mimic a number of things which were considered "graven in stone" by that platform's dominant unix variant, and that data structure was one of them.

Apart from that I echo PascalDragon's comment about time handling being complex: I've tried to work through the chain to find out what could be done about accuracy being lost as TDateTime moves further from the epoch and not enjoyed it.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #7 on: January 28, 2021, 09:18:57 am »
the timestamps in the stat record are unsigned values, therefore they go until 2106-02-07 06:28:15 UTC for a 32-bit executable on a 64-bit system.

The question is not only whether the stat record can handle that, but whether the system call provides dates beyond 2038.

Hartmut

  • Hero Member
  • *****
  • Posts: 742
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #8 on: January 28, 2021, 11:08:19 am »
@PascalDragon:
Ok, I will create the 2 bug reports and mention them here.

Then that essentially means that a 32-bit program on a kernel older than 5.1 can not get 64-bit timestamps this way.
...
Report the bugs and wait for a fix
If I understand you correctly, such a fix will only work on a kernel >= 5.1 - Then it would make now no sense for me to wait... (then I must decide either to be restricted to 64-bit programs or to give up my idea).

@Kays:
Thanks for that background infos.

@Lutz Mändle:
Thanks a lot for your easy to read demo. Yor are right, that unsigned values allow years up to 2106 (I've made a lot of tests with various functions to get the timestamp of a file on different filesystems and most of them returned signed 32-bit values, so I forgot to mention this detail when creating this topic).

But this does not solve my problem (it would be a longer story) entirely: if I treat timestamps as unsigned, I loose years before 1970. And it would not help me much, if I only can read timestamps > 2038, if I cannot set them (e.g. to preserve the timestamp after I copy a file) and fpUtime() can only set timestamps up to 2038 in a 32-bit program, what I tried.

I had hoped that it would be possible to access 64-bit timestamps in a 32-bit Linux program (in Windows this is easy e.g. via windows.GetFileTime() and windows.SetFileTime() since many years), but I learned, that in Linux this is only possible with a kernel >= 5.1.
So now I know my options and can decide what to do.

@MarkMLl:
Thanks for that warning.

@all:
Thanks to all for your help.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #9 on: January 28, 2021, 12:46:33 pm »
I had hoped that it would be possible to access 64-bit timestamps in a 32-bit Linux program (in Windows this is easy e.g. via windows.GetFileTime() and windows.SetFileTime() since many years), but I learned, that in Linux this is only possible with a kernel >= 5.1.
So now I know my options and can decide what to do.

Have you checked clock_gettime() with the CLOCK_REALTIME parameter? I think that's good for a 32-bit kernel and will give you 64-bit POSIX seconds, /but/ reconciling that with the result of the Now() function will be /decidedly/ non-trivial.

Code: Pascal  [Select][+][-]
  1. (* This is an alternative to the RTL's Now() based on the POSIX clock if
  2.   available otherwise falling back to Now(). It is intended to compensate for
  3.   the fact that as timestamps move further from the epoch they are able to store
  4.   fewer fractional bits, it is anticipated that it will be used primarily for
  5.   interval comparisons and that the operating environment will not change during
  6.   program execution in a way that would cause clock_gettime() to stop working.
  7.  
  8.   Note that this relies on the extended (10-byte real) type for precision, and
  9.   as such will have portability issues.
  10. *)
  11. function PosixSecs(): extended; platform;
  12.  
  13. const
  14.   useGettime= true;
  15.  
  16. var
  17.   ts: timespec;
  18.  
  19. begin
  20.   if useGettime and (clock_gettime(CLOCK_REALTIME, @ts) = 0) then begin
  21.     result := ts.tv_sec;
  22.     result += (ts.tv_nsec / 1E9)
  23.  
  24. (* Since even the extended type doesn't really have enough digits to express *)
  25. (* times relative to the unix epoch in nanoseconds, assume we have to be     *)
  26. (* very careful with the evaluation order.                                   *)
  27.  
  28.   end else begin
  29.     result := Sysutils.Now();
  30.     result -= UnixEpoch;
  31.     result *= SecsPerDay
  32.   end
  33. end { PosixSecs } ;
  34.  

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #10 on: January 28, 2021, 01:29:25 pm »
I had hoped that it would be possible to access 64-bit timestamps in a 32-bit Linux program (in Windows this is easy e.g. via windows.GetFileTime() and windows.SetFileTime() since many years), but I learned, that in Linux this is only possible with a kernel >= 5.1.
So now I know my options and can decide what to do.

Have you checked clock_gettime() with the CLOCK_REALTIME parameter? I think that's good for a 32-bit kernel and will give you 64-bit POSIX seconds, /but/ reconciling that with the result of the Now() function will be /decidedly/ non-trivial.

Hartmut is concerned about the file times (created, modified, access) not the current time.

MarkMLl

  • Hero Member
  • *****
  • Posts: 6676
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #11 on: January 28, 2021, 02:30:56 pm »
Have you checked clock_gettime() with the CLOCK_REALTIME parameter? I think that's good for a 32-bit kernel and will give you 64-bit POSIX seconds, /but/ reconciling that with the result of the Now() function will be /decidedly/ non-trivial.
Hartmut is concerned about the file times (created, modified, access) not the current time.

Ah. My apologies, that's what comes from not re-reading the thread: I was wondering how we'd got there via the stat call.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Hartmut

  • Hero Member
  • *****
  • Posts: 742
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #12 on: January 28, 2021, 03:27:17 pm »
Hi MarkMLl, yes we misunderstood. But I noticed your suggestion for possible future use, thanks.

Hartmut

  • Hero Member
  • *****
  • Posts: 742
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #13 on: January 29, 2021, 09:42:21 am »
I created the requested bug reports:
 - for fpStat() = https://bugs.freepascal.org/view.php?id=38418
 - for fpUtime() = https://bugs.freepascal.org/view.php?id=38419

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: (How) can I make a 64-bit OS-system-call in a 32-bit program (Linux)?
« Reply #14 on: January 29, 2021, 10:24:31 am »
Afaik there is no decision yet about 32-bit Ubuntu packages after the current LTS. (see e.g. https://ubuntu.com/blog/statement-on-32-bit-i386-packages-for-ubuntu-19-10-and-20-04-lts )

I sincerely doubt that practical, out of the box 32-bit binaries support in mainstream distros reach 2025, let alone 2038

 

TinyPortal © 2005-2018