Building Lazarus 3.4 with FreePascal 3.2.2 fails under OpenBSD 7.5

[TCH]

I have three scripts to make FreePascal/Lazarus installation on OpenBSD easy:

install_freepascal_binaries.sh:

Code: Bash  [Select][+][-]

1. #!/bin/sh
2.  
3. if [ "$1" = "" ]
4. then
5.         echo 'Usage: install_freepascal.sh <x86_64/i386> <version> [1=force install]'
6.         exit 0
7. fi
8.  
9. case "$1" in
10.         "i386")
11.                 CARCH="386"
12.         ;;
13.         "x86_64")
14.                 CARCH="x64"
15.         ;;
16.         *)
17.                 echo "Invalid architecture."
18.                 exit 2
19.         ;;
20. esac
21.  
22. if [ -f "/usr/bin/ppc""$CARCH" -a "$3" != "1" ]
23. then
24.         echo "FreePascal binaries seems to be already installed. Force install if you want to reinstall them."
25.         exit 0
26. fi
27.  
28. cd /tmp
29. DNAME="fpc-""$2"".""$1""-openbsd"
30. ANAME="$DNAME"".tar"
31. ftp "https://downloads.sourceforge.net/project/freepascal/OpenBSD/""$2""/""$ANAME"
32. if [ ! -f "$ANAME" ]
33. then
34.         echo "Could not download FreePascal binaries."
35.         exit 1
36. fi
37. tar -xvf "$ANAME"
38. rm "$ANAME"
39. cd "$DNAME"
40. echo "Install to /usr/ instead of /usr/local/"
41. ./install.sh
42. cd ..
43. rm -rf "$DNAME"
44. echo "FreePascal binaries have been successfully installed."
45.  

install_freepascal_sources.sh:

Code: Bash  [Select][+][-]

1. #!/bin/sh
2.  
3. if [ "$1" = "" ]
4. then
5.         echo 'Usage: install_freepascal_sources.sh <version> [1=force install]'
6.         exit 0
7. fi
8.  
9. TNAME="/usr/share/fpcsrc/""$1"
10. if [ -f "$TNAME""/Makefile" -a "$2" != "1" ]
11. then
12.         echo "FreePascal sources seems to be already installed. Force install if you want to reinstall them."
13.         exit 0
14. fi
15.  
16. cd /tmp
17. DNAME="fpc-""$1"
18. ANAME="$DNAME"".source.zip"
19. ftp "https://netcologne.dl.sourceforge.net/project/freepascal/Source/""$1""/""$ANAME" -o "$ANAME" -S dont
20. if [ ! -f "$ANAME" ]
21. then
22.         echo "Could not download FreePascal sources."
23.         exit 1
24. fi
25. unzip "$ANAME"
26. rm "$ANAME"
27. mkdir -p `dirname "$TNAME"`
28. mv "fpc-""$1" "$TNAME"
29. cd "$TNAME"
30. FTOED="packages/fcl-net/fpmake.pp"
31. if [ ! -f "$FTOED" ]
32. then
33.         echo "$FTOED"" is missing."
34.         exit 2
35. fi
36. sed -i "s/'cnetdb\.pp',\[linux,freebsd,solaris,android\]/'cnetdb\.pp',\[linux,freebsd,openbsd,dragonfly,solaris,android\]/" "$FTOED"
37. echo "FreePascal sources have been successfully installed."
38.  

install_lazarus.sh:

Code: Bash  [Select][+][-]

1. #!/bin/sh
2.  
3. if [ "$1" = "" ]
4. then
5.         echo 'Usage: install_lazarus.sh <version> [revision] [1=force install]'
6.         exit 0
7. fi
8.  
9. TNAME="/usr/local/share/lazarus/lazarus"
10. if [ -f "$TNAME" -a "$3" != "1" ]
11. then
12.         echo "Lazarus seems to be already installed. Force install if you want to reinstall it."
13.         exit 0
14. fi
15.  
16. cd /tmp
17. if [ ! -d "lazarus" ];
18. then
19.                 REV=""
20.                 if [ "$2" != "" ];
21.                 then
22.                         REV="-""$2"
23.                 fi
24.                 ANAME="lazarus-""$1""$REV"".zip"
25.                 ftp "https://netix.dl.sourceforge.net/project/lazarus/Lazarus%20Zip%20_%20GZip/Lazarus%20""$1""/""$ANAME" -o "$ANAME" -S dont
26.         if [ ! -f "$ANAME" ];
27.         then
28.                 echo "Could not download Lazarus sources."
29.                 exit 1
30.         fi
31.         unzip "$ANAME"
32.         rm "$ANAME"
33. fi
34. cd lazarus
35. gmake clean bigide
36. if [ -f lazarus ];
37. then
38.         mkdir -p `dirname "$TNAME"`
39.         gmake install
40.         if [ -f "$TNAME" ];
41.         then
42.                 cd ..
43.                 rm -rf lazarus
44.         else
45.                 echo "Install failed."
46.                 exit 2
47.         fi
48. else
49.         echo "Build failed."
50.         exit 3
51. fi
52. echo "Lazarus has been successfully installed."

When i do:

Code: [Select]

install_freepascal_binaries.sh x86_64 3.2.2
install_freepascal_sources.sh 3.2.2
install_lazarus.sh 3.4 0The compiling dies at one point by stating:

Code: [Select]

Can't find unit cNetDB used by netdbThis is the very same message as earlier, however, this time /usr/share/fpcsrc/3.2.2/packages/fcl-net/fpmake.pp is already patched to have

Code: Pascal  [Select][+][-]

1.     T:=P.Targets.AddUnit('cnetdb.pp',[linux,freebsd,openbsd,dragonfly,solaris,android]);

instead of

Code: Pascal  [Select][+][-]

1.     T:=P.Targets.AddUnit('cnetdb.pp',[linux,freebsd,dragonfly,solaris,android]);

yet the compiling dies with this message.

Am i doing something wrong? Should not the sources of FreePascal reside in /usr/share/fpcsrc/3.2.2? Or is it /usr/share/fpcsrc/fpc-3.2.2? AFAIK, it should be the first, but in my current OpenBSD 7.4 system, it is the latter one. Granted, i did not use these scripts last time to build Lazarus, but ibara's port, which worked. Until now.
OpenBSD 7.5 broke something again and after upgrading the system and - more precisely - harfbuzz, Lazarus dies with this error message:

Code: [Select]

(lazarus:40076): GLib-GObject-CRITICAL **: 15:58:36.054: g_object_unref: assertion 'G_IS_OBJECT (object)' failed
[FORMS.PP] ExceptionOccurred
  Sender=EAccessViolation
  Exception=Access violation
  Stack trace:
  $000008A747750768
  $000008A74774E80E
  $000008A74774F02F
  $000008A7664A56BE
  $000008A74773A44D
  $000008A7664A3B16
  $000008A7663D7572
  $000008A747734410
  $000008A747751052
  $000008A74774E80E
  $000008A74774F02F
  $000008A7664A56BE
  $000008A74773A44D
  $000008A7664A3B16
  $000008A4E5F94652  FREESTYLEOBJECT,  line 7715 of gtk2proc.inc
  $000008A4E5F94777  RELEASEALLSTYLES,  line 7742 of gtk2proc.inc
  $000008A4E5E2BFF1  FREEALLSTYLES,  line 1650 of gtk2widgetset.inc
[FORMS.PP] ExceptionOccurredAnd if i try to rebuild Lazarus from the ports (cd /usr/ports/devel/lazarus && make clean='all depends' && make install), then this:

Code: [Select]

gmake -C ide ide
gmake[1]: Entering directory '/usr/ports/pobj/lazarus-2.2.0_0/lazarus/ide'
/bin/mkdir -p ../units/x86_64-openbsd/gtk2
../tools/svn2revisioninc .. revision.inc
Created /usr/ports/pobj/lazarus-2.2.0_0/lazarus/ide/revision.inc for revision: Unversioned directory
An unhandled exception occurred at $00000B4CF72602F2:
EAccessViolation:
  $00000B4CF72602F2 line 177 of /usr/src/lib/libc/stdlib/atexit.c
  $00000B4CF72727E5 line 54 of /usr/src/lib/libc/stdlib/exit.c
  $00000B4A04264983

An unhandled exception occurred at $00000B4A0427FB50:
EAccessViolation:
  $00000B4A0427FB50
  $00000B4CF72727E5 line 54 of /usr/src/lib/libc/stdlib/exit.c
  $00000B4A04264983And then this latter error message repeats forever.

This is why i decided to dust off my old approach and build Lazarus from scratch, but that does not work either.

Any ideas are appreciated.

[marcov]

Make sure you clean the sources thoroughly before you rebuild after a patch.

I usually kill *.ppu *.o and packages/fpmake

Anyway, 3.2.2 is 3 years old at this moment. Please test with the fixes_3_2 branch.

[TCH]

Make sure you clean the sources thoroughly before you rebuild after a patch.

I usually kill *.ppu *.o and packages/fpmake

I am not rebuilding, i am building from scratch. The system has been entirely purged of FreePascal binaries, sources and Lazarus and then the archives have been downloaded and extracted to their place. Then the change was issued via sed. Only then i started the build. There is nothing to be cleaned.

Anyway, 3.2.2 is 3 years old at this moment. Please test with the fixes_3_2 branch.

Where can i download the OpenBSD binaries from that branch? I'd give it a try. (Although i don't think it'll help: if Lazarus 3.4 compiles with 3.2.2 on macOS, Linux, FreeBSD, etc., then why would a newer compiler fix a unit-detection problem which origins from the package and not the compiler?)

[marcov]

Make sure you clean the sources thoroughly before you rebuild after a patch.

I usually kill *.ppu *.o and packages/fpmake

I am not rebuilding, i am building from scratch.

Good.

Anyway, 3.2.2 is 3 years old at this moment. Please test with the fixes_3_2 branch.

Where can i download the OpenBSD binaries from that branch?

You can't. That is work in progress branch, not a release, so source only. But it should build with a 3.2.2 a starting compiler. The rule is generally the next FPC release (and the next release froum fixes_3_2_2 would be 3.2.4) can be build with the previous release.

I'd give it a try. (Although i don't think it'll help: if Lazarus 3.4 compiles with 3.2.2 on macOS, Linux, FreeBSD, etc., then why would a newer compiler fix a unit-detection problem which origins from the package and not the compiler?)

Well, that obviously means that the problem must be in OS specific code!

And your exception messages also seems to point to that, with the limited info given; one is in glib, an external library (a fundament for GTK) and one is in atexit, which is libc initialization/finalization related, which in turn both point to problems in the OpenBSD specific startup code that initializes libc (openbsd/<cpu>/cprt0.as) or its libc headers.

(Note that I haven't compiled OpenBSD (or NetBSD) in over a decade, but I'm still doing FreeBSD  occasionally, and actually that target is a bit modernized in 3.2.4, whenever it lands )

[TCH]

You can't. That is work in progress branch, not a release, so source only. But it should build with a 3.2.2 a starting compiler. The rule is generally the next FPC release (and the next release froum fixes_3_2_2 would be 3.2.4) can be build with the previous release.

http://downloads.freepascal.org/fpc/snapshot/fixes/source/fpc.zip or http://downloads.freepascal.org/fpc/snapshot/fixes/source/fpcbuild.zip?

Well, that obviously means that the problem must be in OS specific code!

Okay, but where? Last time it was caused by a missing unit entry for cnetdb.pp in /usr/share/fpcsrc/3.2.2/packages/fcl-net/fpmake.pp, but this time the entry is there. What else can produce this error?

And your exception messages also seems to point to that, with the limited info given; one is in glib, an external library (a fundament for GTK) and one is in atexit, which is libc initialization/finalization related, which in turn both point to problems in the OpenBSD specific startup code that initializes libc (openbsd/<cpu>/cprt0.as) or its libc headers.

I think those error messages are caused by some kind of ABI-breakage. I've already ran into a similar case when i upgraded OpenBSD 7.2 to 7.3. Lazarus 2.2.6 currently works on OpenBSD 7.4 and i can even upgrade to 7.5, it will work then too. It only dies, when i upgrade the harfbuzz package after the system upgrade. (.libs1-harfbuzz-2.9.1+.libs1-harfbuzz-5.2.0+harfbuzz-8.2.1 updated to harfbuzz-8.3.0, i've reported the issue to ibara.)

[TRon]

http://downloads.freepascal.org/fpc/snapshot/fixes/source/fpc.zip or http://downloads.freepascal.org/fpc/snapshot/fixes/source/fpcbuild.zip?

You can however I personally prefer this as it is directly from the horses mouth. Other archive formats can be downloaded from here (press the code link).

Alternatively use git (the recommended way of doing things)

[TCH]

Okay, thanks guys. In the following days, i'll try to make some time to build the compiler and then try to compile Lazarus with that.

[marcov]

http://downloads.freepascal.org/fpc/snapshot/fixes/source/fpc.zip or http://downloads.freepascal.org/fpc/snapshot/fixes/source/fpcbuild.zip?

I don't know if those are up to date. Very low usage, so a hanging job isn't always detected. Better use git.

I think those error messages are caused by some kind of ABI-breakage. I've already ran into a similar case when i upgraded OpenBSD 7.2 to 7.3. Lazarus 2.2.6 currently works on OpenBSD 7.4 and i can even upgrade to 7.5, it will work then too. It only dies, when i upgrade the harfbuzz package after the system upgrade. (.libs1-harfbuzz-2.9.1+.libs1-harfbuzz-5.2.0+harfbuzz-8.2.1 updated to harfbuzz-8.3.0, i've reported the issue to ibara.)

Is openbsd currently a libc port (like Darwin) or a syscall port like Linux/FreeBSD ?  if it is a libc port, it shouldn't do syscalls, so the abi breakage should be wrapped. 

If you have a working target system, can you run an older compiler on there ? (e.g. a linux one in some linuxator or an older OpenBSD version that selects older binaries based on ELF ident selector like on FreeBSD

Then you can try to build the compiler manually (first RTL, then make all in compiler/) and pass -S to defer linking.

Then run the link step manually (now started as a native program, as it is not executed by a compatibility/linux program)

Similarly, you can try to do this on an older system, and then transfer .o and ppas.sh  and execute the link.

Linking natively uses some startup .o files that come with the system.

If this all fails, it is a bit more involved, and really finding out what is going on is needed, followed by (probably) startup code changes.

If the compiler seems to work, but can't find files, probably some stat or statfs struct changed.   Print size of *nix related records in FPC on your old openbsd, and in a C program in new openbsd, and compare.

[TCH]

Is openbsd currently a libc port (like Darwin) or a syscall port like Linux/FreeBSD ?  if it is a libc port, it shouldn't do syscalls, so the abi breakage should be wrapped.

I have no idea. But speaking of syscalls, OpenBSD 7.5 removed support for syscall() and replaced it with pinsyscalls(). This might be the problem, although if it is, then i don't get it, why the existing Lazarus die only after harfbuzz has been upgraded...

If you have a working target system, can you run an older compiler on there ? (e.g. a linux one in some linuxator or an older OpenBSD version that selects older binaries based on ELF ident selector like on FreeBSD

Then you can try to build the compiler manually (first RTL, then make all in compiler/) and pass -S to defer linking.

Then run the link step manually (now started as a native program, as it is not executed by a compatibility/linux program)

Similarly, you can try to do this on an older system, and then transfer .o and ppas.sh  and execute the link.

Linking natively uses some startup .o files that come with the system.

I can install any older FPC, but should not the sources be built with the latest stable one? Anyway, i tried to build the compiler sources got from the link TRon gave with the existing 3.2.2. It did not go well, the Makefile is erroneous, or i don't know. http://oscomp.hu/depot/make_all.log

Should i try to build it with an older compiler?

If this all fails, it is a bit more involved, and really finding out what is going on is needed, followed by (probably) startup code changes.

If the compiler seems to work, but can't find files, probably some stat or statfs struct changed.   Print size of *nix related records in FPC on your old openbsd, and in a C program in new openbsd, and compare.

I've printed the sizes of struct stat and struct statfs on both 7.4 and 7.5: it is 128 and 568 on both, no changes have occurred.

[duralast]

Security improvements:
Introduce pinsyscalls(2): The kernel and ld.so(1) register the precise entry location of every system call used by a program, as described in the new ELF section .openbsd.syscalls inside ld.so and libc.so. ld.so uses the new syscall pinsyscalls(2) to tell the kernel the precise entry location of system calls in libc.so.
Attempting to use a different system call entry instruction to perform a non-corresponding system call operation will fail and the process will be terminated with signal SIGABRT.
Removed support for syscall(2), the "indirection system call," a dangerous alternative entry point for all system calls.
Together with pinsyscalls(2) this change makes it impossible to perform system call through any other way than the libc system call wrapper functions.
Users of syscall(2), such as Perl and the Go programming language were converted to use the libc functions.

As you noted, pinsyscalls(2) is your problem.

Theo even noted in an email on 27 Oct 2023, "There's going to be some fallout which takes time to fix, especially in the "go" ecosystem."

Obviously, it is more than just Perl and Go.

[TCH]

The whys are still foggy for me. If the removal of syscall() is the cause of the crashing of the Lazarus compiled on older OpenBSD, then Lazarus should die immediately after OpenBSD got upgraded. But it only dies, when the harfbuzz package is upgraded, after the OS/kernel upgrade. Even if Lazarus did not use syscall() directly, the old harfbuzz lib should be incompatible with the new kernel and thus cause Lazarus to crash anyway.
So, for me, it seems, that the problem is with the new harfbuzz package, for instance, because of the changed ABI, Lazarus passes stuff to harfbuzz in the wrong way and crashes because it. Or, it's something else.

It also does not explain the cnetdb problem at compiling Lazarus.

But maybe it could explain the atexit.c and exit.c errors with Lazarus from the ports; the compiler might try to call into syscall() at one point. Or i don't know.

[marcov]

As far as I can see from a comment in rtl/openbsd/makefile.fpc, openbsd is libc based.

Start testing with smaller programs.  Lazarus is too large and unwieldy. If you get glib errors, try to link with glib.

This is one of cases where FPC libc ports are fragile. With syscall ports you can get a minimal part working easier, get the compiler recompiling, and only then tackle dynamical linking and what changed in libc now.

With libc ports that must be done all at once to get a functioning binary.

I can't really help, I haven't worked on net/openbsd in a decade.

[marcov]

I walked through unmerged revisions and found some for openbsd:

commit c249495e2df2c695fe8f97a17cdd5de0a02aa501
Author: Pierre Muller <pierre@freepascal.org>
Date:   Mon Jul 17 10:50:29 2023 +0200

    Add '%note' to .note.openbsd.ident section

M       rtl/openbsd/i386/openbsd_ident.inc
M       rtl/openbsd/i386/prt0.as
M       rtl/openbsd/x86_64/openbsd_ident.inc
M       rtl/openbsd/x86_64/prt0.as

commit d621cef9883623c065ff5dd4d97dbb4cd49a26d4
Author: Pierre Muller <pierre@freepascal.org>
Date:   Wed Nov 29 18:21:08 2023 +0000

    Reenable semaphore support for NetBSD and OpenBSD in syncobjs unit

M       packages/fcl-base/src/syncobjs.pp


The last one might also depend on other revisions. You could try to download the first revision as a patch and have a look and try to apply it against fixes.

[TCH]

Well, at least i figured out a workaround for the cnetdb problem. For testing i intentonally made syntax errors in /usr/share/fpcsrc/3.2.2/packages/fcl-net/fpmake.pp, but Lazarus did not notice, even after a cleanup. It simply does not even touches the file. So, i've checked where is netdb.pp and then

Code: Bash  [Select][+][-]

1. ln -s "/usr/share/fpcsrc/3.2.2/packages/fcl-net/src/cnetdb.pp" "/usr/lib/fpc/3.2.2/units/x86_64-openbsd/fcl-net/cnetdb.pp"

And the cnetdb problem went away.

The compilation still crashes though...
At this time, with different error messages:

Code: [Select]

gmake -C ide bigide
gmake[1]: Entering directory '/tmp/lazarus/ide'
/bin/mkdir -p ../units/x86_64-openbsd/gtk2
../tools/svn2revisioninc .. revision.inc
Created /tmp/lazarus/ide/revision.inc for revision: Unversioned directory
An unhandled exception occurred at $000003CFF7F87EF2:
EAccessViolation:
  $000003CFF7F87EF2 line 177 of /usr/src/lib/libc/stdlib/atexit.c
  $000003CFF7FD0EC5 line 54 of /usr/src/lib/libc/stdlib/exit.c
  $000003CDDEEAAF53  _FPC_PROC_HALTPROC,  line 72 of x86_64/si_c.inc

An unhandled exception occurred at $000003CDDEEC6140:
EAccessViolation:
  $000003CDDEEC6140  SYSFREEMEM,  line 1222 of ../inc/heap.inc
  $000003CFF7FD0EC5 line 54 of /usr/src/lib/libc/stdlib/exit.c
  $000003CDDEEAAF53  _FPC_PROC_HALTPROC,  line 72 of x86_64/si_c.incWhoops, you've posted in the meantime:

The last one might also depend on other revisions. You could try to download the first revision as a patch and have a look and try to apply it against fixes.

Thanks for the tip, i'll try to reverse those changes.

[marcov]

 Note also that svn2revision is not crucial. Probably it only tries to execute git to get some revision to put in about. Probably the makefiles allow it to ski pit.

Making random symlinks and mixing sources is _NEVER_ good.   Find out why it doesn't work.  In thise case, you modified FPC, not lazarus, so you need to recompile fpc, and then lazarus.