Building Lazarus 3.4 with FreePascal 3.2.2 fails under OpenBSD 7.5

[Fred vS]

Does FPC have any assembly like

Code: [Select]

call do_syscall
Or is it just Pascal code calling the do_syscall function?

I did not find any code like call do_syscall in fpc 3.2.2 source.
There is some assembler code call syscall but seems not concerned by OpenBSD.

[Fred vS]

FreeBSD uses do_syscall too, as well as DragonflyBSD and Linux. And I would presume that the latest Lazarus/FPC builds and runs fine on the latest release of FreeBSD. Which is what led me to initially believe that pinsyscalls() was/is the problem with Lazarus/FPC on 7.5.

But then I dont understand why a app compiled on OpenBSD 7.4. runs ok on OpenBSD 7.5.

[marcov]

Which is what led me to initially believe that pinsyscalls() was/is the problem with Lazarus/FPC on 7.5.

I suspect one of the past OpenBSD contributors saw this coming and switched the RTL from doing syscalls to libc a few years back.

Does FPC have any assembly like

Yes. The above maintainer overlooked this. Search in fpc/rtl/openbsd/x86_64 for syscall, and you see for program termination a syscall is used.  But that is at the end of the program. The program should basically work fine up to the last instruction.

Most general code should respect the switch to libc as that conditional code has already been done for Apple targets. Also when linked on 7.4, the binaries work.   

So that leaves the linker on 7.5 being special, or something scanning binaries for syscall instructions.

But that is all a bit premature, because we haven't exactly established where it crashes. (e.g. if you put a breakpoint on _start, does it hit it?)  Also ktrace/truss or whatever openbsd uses could give info.

[Fred vS]

Also ktrace/truss or whatever openbsd uses could give info.

In attachment ktrace.zip, complete ktrace (converted by kdump -d -H to be human readable) of that empty program compiled and run on 7.5:

Code: Pascal  [Select][+][-]

1. program test1;
2. begin
3. end.

Here resume of ktrace.txt:

Code: Bash  [Select][+][-]

1. obsd$ kdump -d -H
2.  25755/534659  ktrace   RET   ktrace 0
3.  25755/534659  ktrace   CALL  execve(0x775224aca83f,0x775224aca6d0,0x775224aca6e0)
4.  25755/534659  ktrace   NAMI  "/home/fred/fpc_test/test1"
5.  25755/534659  ktrace   ARGS  
6.         [0] = "/home/fred/fpc_test/test1"
7.  25755/534659  test1    NAMI  "/usr/libexec/ld.so"
8.  25755/534659  test1    RET   execve JUSTRETURN
9.  25755/534659  test1    CALL  getentropy(0x741a88170140,40)
10.  25755/534659  test1    RET   getentropy 0
11.  25755/534659  test1    CALL  getentropy(0x741a88170140,40)
12.  25755/534659  test1    RET   getentropy 0
13.  25755/534659  test1    CALL  mmap(0,0x4000,0<PROT_NONE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
14.  25755/534659  test1    RET   mmap 27406487552/0x6618dd000
15.  25755/534659  test1    CALL  mprotect(0x6618de000,0x2000,0x3<PROT_READ|PROT_WRITE>)
16.  25755/534659  test1    RET   mprotect 0
17.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
18.  25755/534659  test1    RET   mmap 29989945344/0x6fb8a4000
19.  25755/534659  test1    CALL  issetugid()
20.  25755/534659  test1    RET   issetugid 0
21.  25755/534659  test1    CALL  mprotect(0x6c4b47000,0x1000,0x1<PROT_READ>)
22.  25755/534659  test1    RET   mprotect 0
23.  25755/534659  test1    CALL  mimmutable(0x6c4b47000,0x1000)
24.  25755/534659  test1    RET   mimmutable 0
25.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
26.  25755/534659  test1    RET   mmap 27950288896/0x681f79000
27.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
28.  25755/534659  test1    RET   mmap 29271437312/0x6d0b6b000
29.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
30.  25755/534659  test1    RET   mmap 25961279488/0x60b69b000
31.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
32.  25755/534659  test1    RET   mmap 26434912256/0x627a4c000
33.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
34.  25755/534659  test1    RET   mmap 28200251392/0x690ddb000
35.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
36.  25755/534659  test1    RET   mmap 25941798912/0x60a407000
37.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
38.  25755/534659  test1    RET   mmap 28025401344/0x68671b000
39.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
40.  25755/534659  test1    RET   mmap 26535165952/0x62d9e8000
41.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
42.  25755/534659  test1    RET   mmap 26564194304/0x62f597000
43.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
44.  25755/534659  test1    RET   mmap 28490244096/0x6a226a000
45.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
46.  25755/534659  test1    RET   mmap 28754604032/0x6b1e87000
47.  25755/534659  test1    CALL  open(0x6c4a47602,0x10000<O_RDONLY|O_CLOEXEC>)
48.  25755/534659  test1    NAMI  "/var/run/ld.so.hints"
49.  25755/534659  test1    RET   open 3
50.  25755/534659  test1    CALL  fstat(3,0x741a8816ff60)
51.  25755/534659  test1    STRU  struct stat { dev=4, ino=103690, mode=-r--r--r-- , nlink=1, uid=0<"root">, gid=0<"wheel">, rdev=418048, atime=1718994445<"Jun 21 20:27:25 2024">.748676392, mtime=1718994444<"Jun 21 20:27:24 2024">.998732632, ctime=1718994444<"Jun 21 20:27:24 2024">.998732632, size=55702, blocks=112, blksize=16384, flags=0x0, gen=0x0 }
52.  25755/534659  test1    RET   fstat 0
53.  25755/534659  test1    CALL  mmap(0,0xd996,0x1<PROT_READ>,0x2<MAP_PRIVATE>,3,0)
54.  25755/534659  test1    RET   mmap 27130327040/0x65117f000
55.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
56.  25755/534659  test1    RET   mmap 27360616448/0x65ed1e000
57.  25755/534659  test1    CALL  mimmutable(0x65117f000,0xd996)
58.  25755/534659  test1    RET   mimmutable 0
59.  25755/534659  test1    CALL  close(3)
60.  25755/534659  test1    RET   close 0
61.  25755/534659  test1    CALL  open(0x65118607d,0x10000<O_RDONLY|O_CLOEXEC>)
62.  25755/534659  test1    NAMI  "/usr/lib/libc.so.99.0"
63.  25755/534659  test1    RET   open 3
64.  25755/534659  test1    CALL  fstat(3,0x741a88170018)
65.  25755/534659  test1    STRU  struct stat { dev=5, ino=51896, mode=-r--r--r-- , nlink=1, uid=0<"root">, gid=7<"bin">, rdev=268552, atime=1719006410<"Jun 21 23:46:50 2024">.208707940, mtime=1718994442<"Jun 21 20:27:22 2024">.733452477, ctime=1718994442<"Jun 21 20:27:22 2024">.733452477, size=3702088, blocks=7264, blksize=16384, flags=0x0, gen=0x0 }
66.  25755/534659  test1    RET   fstat 0
67.  25755/534659  test1    CALL  read(3,0x741a8816eb00,0x1000)
68.  25755/534659  test1    GIO   fd 3 read 4096 bytes
69. ... (removed data of GIO)
70.  25755/534659  test1    RET   read 4096/0x1000
71.  25755/534659  test1    CALL  mmap(0,0xf7000,0<PROT_NONE>,0x2<MAP_PRIVATE>,3,0)
72.  25755/534659  test1    RET   mmap 29379473408/0x6d7273000
73.  25755/534659  test1    CALL  mmap(0x6d7273000,0x37000,0x1<PROT_READ>,0x12<MAP_PRIVATE|MAP_FIXED>,3,0)
74.  25755/534659  test1    RET   mmap 29379473408/0x6d7273000
75.  25755/534659  test1    CALL  mmap(0x6d72aa000,0xa8000,0x4<PROT_EXEC>,0x12<MAP_PRIVATE|MAP_FIXED>,3,0x36000)
76.  25755/534659  test1    RET   mmap 29379698688/0x6d72aa000
77.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
78.  25755/534659  test1    RET   mmap 26728656896/0x63926f000
79.  25755/534659  test1    CALL  mmap(0x6d7352000,0x6000,0x3<PROT_READ|PROT_WRITE>,0x12<MAP_PRIVATE|MAP_FIXED>,3,0xdd000)
80.  25755/534659  test1    RET   mmap 29380386816/0x6d7352000
81.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
82.  25755/534659  test1    RET   mmap 30110879744/0x702bf9000
83.  25755/534659  test1    CALL  mmap(0x6d7358000,0x4000,0x3<PROT_READ|PROT_WRITE>,0x12<MAP_PRIVATE|MAP_FIXED>,3,0xe2000)
84.  25755/534659  test1    RET   mmap 29380411392/0x6d7358000
85.  25755/534659  test1    CALL  mmap(0x6d735c000,0xe000,0x3<PROT_READ|PROT_WRITE>,0x1012<MAP_PRIVATE|MAP_FIXED|MAP_ANON>,-1,0)
86.  25755/534659  test1    RET   mmap 29380427776/0x6d735c000
87.  25755/534659  test1    CALL  mmap(0,0x6e8,0x1<PROT_READ>,0x2<MAP_PRIVATE>,3,0xe6000)
88.  25755/534659  test1    RET   mmap 28794580992/0x6b44a7000
89.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
90.  25755/534659  test1    RET   mmap 28916326400/0x6bb8c2000
91.  25755/534659  test1    CALL  munmap(0x6b44a7000,0x6e8)
92.  25755/534659  test1    RET   munmap 0
93.  25755/534659  test1    CALL  pinsyscalls(29379698688,688128,28916326400,331)
94.  25755/534659  test1    RET   pinsyscalls 0
95.  25755/534659  test1    CALL  msyscall(0x6d72aa000,0xa8000)
96.  25755/534659  test1    RET   msyscall 0
97.  25755/534659  test1    CALL  close(3)
98.  25755/534659  test1    RET   close 0
99.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
100.  25755/534659  test1    RET   mmap 28247621632/0x693b08000
101.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
102.  25755/534659  test1    RET   mmap 27874893824/0x67d792000
103.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
104.  25755/534659  test1    RET   mmap 26432163840/0x6277ad000
105.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
106.  25755/534659  test1    RET   mmap 29160669184/0x6ca1c8000
107.  25755/534659  test1    CALL  getthrid()
108.  25755/534659  test1    RET   getthrid 534659/0x82883
109.  25755/534659  test1    CALL  __set_tcb(0x681f794c0)
110.  25755/534659  test1    RET   __set_tcb 0
111.  25755/534659  test1    CALL  issetugid()
112.  25755/534659  test1    RET   issetugid 0
113.  25755/534659  test1    CALL  mprotect(0x6d7352170,0x5e90,0x1<PROT_READ>)
114.  25755/534659  test1    RET   mprotect 0
115.  25755/534659  test1    CALL  mimmutable(0x6d7352170,0x5e90)
116.  25755/534659  test1    RET   mimmutable 0
117.  25755/534659  test1    CALL  mimmutable(0x6d7273000,0xdf170)
118.  25755/534659  test1    RET   mimmutable 0
119.  25755/534659  test1    CALL  mimmutable(0x6d7358000,0x3000)
120.  25755/534659  test1    RET   mimmutable 0
121.  25755/534659  test1    CALL  mimmutable(0x6d735c000,0xe000)
122.  25755/534659  test1    RET   mimmutable 0
123.  25755/534659  test1    CALL  mprotect(0x406f1a818,0x7e8,0x1<PROT_READ>)
124.  25755/534659  test1    RET   mprotect 0
125.  25755/534659  test1    CALL  mimmutable(0x406f1a818,0x7e8)
126.  25755/534659  test1    RET   mimmutable 0
127.  25755/534659  test1    CALL  kbind(0x741a881702d8,24,0x6b4f62fa8784ccf8)
128.  25755/534659  test1    RET   kbind 0
129.  25755/534659  test1    CALL  mmap(0x6c4a4a000,0x2000,0<PROT_NONE>,0x1012<MAP_PRIVATE|MAP_FIXED|MAP_ANON>,-1,0)
130.  25755/534659  test1    RET   mmap 29068926976/0x6c4a4a000
131.  25755/534659  test1    CALL  mimmutable(0x6c4a4a000,0x2000)
132.  25755/534659  test1    RET   mimmutable 0
133.  25755/534659  test1    CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
134.  25755/534659  test1    RET   mmap 28885450752/0x6b9b50000
135.  25755/534659  test1    CALL  mprotect(0x6b9b50000,0x1000,0x1<PROT_READ>)
136.  25755/534659  test1    RET   mprotect 0
137.  25755/534659  test1    PSIG  SIGSEGV SIG_DFL code=SEGV_ACCERR addr=0x406df5c93 trapno=6
138.  25755/534659  test1    NAMI  "test1.core"
139.  

At first check there is:

Code: Bash  [Select][+][-]

1.  25755/534659  test1    CALL  pinsyscalls(29379698688,688128,28916326400,331)

and this not defined in /rtl/openbsd/sysnr.inc (see https://forum.lazarus.freepascal.org/index.php/topic,67536.msg521228.html#msg521228)

Code: Bash  [Select][+][-]

1.  25755/534659  test1    CALL  msyscall(0x6d72aa000,0xa8000)

[EDIT] In comparaison, in attachment ktrace_74.zip the complete ktrace of the empty program compiled on 7.4 and run on 7.4.
           and ktrace_74_on_75.zip the complete ktrace of the empty program compiled on 7.4 and run OK on 7.5.

[EDIT2] In ktrace_74.txt (ktrace of empty app compiled on 74 and run on 74), there is no CALL pinsyscalls().                 
                  And in  ktrace_74_on_75.txt there is one but not the same as in ktrace.txt (the one compiled+run on 75):

Code: Bash  [Select][+][-]

1.  98986/495930  test1    CALL  pinsyscalls(7303712604160,688128,7306434136064,331)

[duralast]

Thanks for the ktrace!

In the last email link I gave which was from Theo

  1) in the text of a static binary, because the kernel loaded a table for
     *ONLY* the system calls linked into the binary.  It's important to
     realize what this means, by example.  The ping(1) binary does not call
     execve(2) or fork(2).  So now you can't ever call fork() or execve()
     because there is no "syscall" instruction for those two system calls.
     It also cannot call accept(2).

But that is the first thing in the ktrace

Code: [Select]

25755/534659  ktrace   CALL  execve(0x775224aca83f,0x775224aca6d0,0x775224aca6e0)
The binary does call execve though, which means it is linked in the binary. Wouldn't this register the entry location for pinsyscalls()?

[marcov]

I think that is the program loader (ld.so) doing that call, and also some mmaps and mprotects after it to set up the memory area for the new binary.

To me it sounds like ld.so chokes on the binary rather than the binary itself crashing. Running in gdb and setting a breakpoint on the entry point (typically _start) might confirm that. If it doesn't hit it, it never gets to the program.

[DonAlfredo]

Again.
I can successfully build (fixes) compiler ppc1 on 7.5
Running ppc1 goes well and shows all the expected output.
On exit, it crashes.

[marcov]

Again.
I can successfully build (fixes) compiler ppc1 on 7.5
Running ppc1 goes well and shows all the expected output.
On exit, it crashes.

I was wrong. While the shared library still calls exit via a syscall, main programs don't, they call the symbol "exit".

[Fred vS]

Again.
I can successfully build (fixes) compiler ppc1 on 7.5
Running ppc1 goes well and shows all the expected output.
On exit, it crashes.

The same here, ppc1 runs but crash when compiling system.pp.

[EDIT] Hum, no, here ppc1 does not run ok, doing this "nude" to test:

Code: Pascal  [Select][+][-]

1. # /fpc_fixes3.2/compiler/ppc1

--> Crash with segmentation fault error (core dumped).

[EDIT2] and ktrace ppc1 gives the same result as a nude app compiled on 7.5 (see previous post)

[Fred vS]

Hello.

Some other news concerning new ld.so from Changelog 7.5: https://www.openbsd.org/plus75.html

Code: Bash  [Select][+][-]

1. - Made the kernel read pinsyscall tables out of PT_OPENBSD_SYSCALLS in the main program or ld.so,
2.    and accept a submission of that information for libc.so from ld.so via pinsyscalls(2).
3.    At system call invocation, the syscall number is matched to the specific address from which it must come.
4.  
5. - Changed ld.so to only load the first libc version encountered requested and substituting it for all further loads,
6.    ensuring that the libc version requested by an executable itself is the one loaded.
7.  
8. - Ensured the syscall table entries for libc and ld.so are aligned on a 4-byte boundary.
9.  
10. - Populated the non-LOAD openbsd.syscalls section (and PT_OPENBSD_SYSCALL)
11.    with {uint offset, uint syscall#} entries in libc and ld.so.
12.  

I suspect there is something to explore on how to handle their new ld.bfd/ld.so.

[Fred vS]

Hello.

I posted a message to the OpenBSD forum (the only one I found).
https://daemonforums.org/showthread.php?p=75175#post75175

Hopefully we'll get some light from there.
If you see something I should write about our problem, please advise me.

Fre;D

[duralast]

Has anyone contacted, or is anyone going to contact, the OpenBSD developers about this?

[Fred vS]

Ouch, things are getting worse with every new version of OpenBSD.
With the latest version 7.6 all fpc applications, even those compiled on version 7.4, crash.
 

[DonAlfredo]

Strange result indeed.
If I cross-compile fpcupdeluxe towards latest OpenBSD 7.6, all is well. See included screenshot.
If I try to build FPC itself on OpenBSD 7.6, ppc1 crashes due to the well known atexit line 177 failure.

[BSaidus]

Ouch, things are getting worse with every new version of OpenBSD.
With the latest version 7.6 all fpc applications, even those compiled on version 7.4, crash.
 

Hi Fred vS, did you try to compile & run a simple console application with 'fpc from sourceforge' ?
I'm interested about this.