using musl-libc instead of glibc

[robert rozee]

hi,
    has anyone tried running FPC/Lazarus on a machine that uses musl instead of glibc? musl being a glibc alternative:
https://musl.libc.org/

i've just spent 6 hours or so trying to get a couple of distros that use musl running in a VM: first trying Gentoo Linux (https://www.gentoo.org/) and then Void Linux (https://voidlinux.org/). Gentoo proved hopeless as the installing into a VM was incredibly complicated (for a non-expert), while Void was far more civilized but once installed (using the guide here: https://linuxiac.com/void-linux-installation/) refused to boot - just going to a blank screen a few seconds into the boot process.

i'm quite interested in musl as (1) it offers the opportunity to build a FPC/Lazarus application against a 'libc lookalike' without the burden of versioned symbols, and (2) there is the possibility of static linking against musl to produce a binary that, although larger, is not libc-dependent. i'm curious to see how much larger the resultant binary really is, musl apparently being far more modular that glibc and so affording the chance of having minimum extra code pulled in.

any pointers, particularly getting Void Linux running in a VM, would be much appreciated. also on persuading FPC/Lazarus to link against musl for those libc routines that are called directly from pascal sources.


cheers,
rob   :-)

[marcov]

Free Pascal default doesn't link to any libc on Linux.

Just do a ldd ./ppc386 or ppcx64

[robert rozee]

... unless your application has a GUI (ie, uses the LCL) or you use threading. or any other stuff that requires libc.

for example, dumped from the binary of a simple large-display digital clock:

Code: [Select]

user@DH61BE:~/pascal/tests/test 206 (3 simple demos)$ objdump -T project1 | grep 'GLIBC'
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) wcrtomb
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) wcscoll
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.34) dlerror
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) setenv
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) setlocale
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) strcoll
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.34) dlopen
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) iconv_close
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) mbrlen
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.34) dlclose
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) mbrtowc
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.34) __libc_start_main
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.34) dlsym
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) iconv
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) iconv_open
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) nl_langinfo
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) towlower
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.34) dladdr
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) __errno_location
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) towupper
0000000000000000      DF *UND* 0000000000000000 (GLIBC_2.2.5) sched_yield
0000000000000000  w   DF *UND* 0000000000000000 (GLIBC_2.2.5) __cxa_finalize
user@DH61BE:~/pascal/tests/test 206 (3 simple demos)$

[AmatCoder]

Some time ago I used FPC with musl in Alpine Linux.

FPC is packaged there: https://pkgs.alpinelinux.org/package/edge/testing/x86_64/fpc

Lazarus is not packaged but I was able to compile it with a few patchs.

But I should warn you:
AFAICR, you can not compile a static build of a program that requires libc with FreePascal and musl.
Because FPC uses dlopen/dlsym under the hood and musl does not support dlopen and static linking at the same time.

[Thaddy]

FPC does not need any libc as Marcov explained. Lazarus does...

Anyway, what I would have tried in my youth is copy over the old libc to a safe location and rename musl-libc with the old name. That only works if the API is the exact same.
Maybe a worthwile attempt? Make sure the original can be placed back.

[robert rozee]

FPC does not need any libc as Marcov explained. Lazarus does...

if i talk about Lazarus without mention of FPC, people will start complaining that Lazarus is just 'a library and IDE' that are add-ons to FPC. hence why i write FPC/Lazarus, or if you prefer Lazarus/FPC.

i do hope we all know what i am talking about - a GUI application compiled with the FPC compiler into a (linux) binary that then depends upon glibc's versioned symbols being 'just so right' that the program can run.

[AmatCoder]

One moment...
Do you want to compile (and dynamically link) a program against musl and then use it on glibc systems?

Because that will never work...They are incompatible.

[DonAlfredo]

I use Alpine Linux running in a VM on windows11.
Fpcupdeluxe for musl runs fine on this OS.

[robert rozee]

One moment...
Do you want to compile (and dynamically link) a program against musl and then use it on glibc systems?

Because that will never work...They are incompatible.

why are they incompatible? i thought one of the main ideas of musl was to provide a work-alike replacement for glibc - and that if you statically link to musl then you'd end up with a binary image that requires no libc on your target computer? after all, libc is just an abstraction layer that sits between the kernel and applications (and libraries that use it). as far as i've been able to determine there is nothing special in libc that can not be accomplished by talking directly to the linux kernel.

so far: i've got Void Linux running, with a Gnome desktop. Gnome is pretty awful, i feel it may be a creation of Fisher-Price! i've managed to get FPC and Lazarus .rpm packages downloaded on the vm, and 'sort of' installed, but not working.

DonAlfredo: i think i'll have a look at Alpine Linux + fpcupdeluxe next.


cheers,
rob   :-)

[AmatCoder]

Do you read my posts? I write them again:

If you statically link to musl:

AFAICR, you can not compile a static build of a program that requires libc with FreePascal and musl.
Because FPC uses dlopen/dlsym under the hood and musl does not support dlopen and static linking at the same time.

If you dynamically link to musl:

One moment...
Do you want to compile (and dynamically link) a program against musl and then use it on glibc systems?

Because that will never work...They are incompatible.

[robert rozee]

Do you read my posts?

i did read them, including the very interesting linked post from 10 or so years ago (https://www.openwall.com/lists/musl/2012/12/08/4) that states "Presently, it does not work at all" (my underlining). the post highlights some of the issues that i'm still thinking through, and i'm interested in finding out if the 'does not work at all' still holds a decade later.

in your earlier posts you qualified your statements with "AFAICR", which i take it means "as far as i can recall". i interpreted this as in the context of "as far as i can recall... the offender was driving a red car" (with the possibility of the 'fact' being false), whereas you seem to have intended the context of "as far as i can recall... 2+2=4" (a more-or-less absolute fact). my apologies for misinterpreting your meaning of "AFAICR".

i can see some of the issues surrounding the use of dlopen() et al, but am not convinced they are insurmountable. certainly difficult, of that there is no doubt, but then most interesting problems are difficult. in an ideal world i'd like to see the functionality of dlopen() et al implemented in pascal - after all, these libraries that dlopen() et al opens are just ELF files.

but there is an interesting question of: can/do libraries themselves make use of dlopen() internally, or can dlopen() only be called by the user application? and if a library uses dlopen() internally to access another library, does this percolate up to an explicit entry in the application's Dynamic Symbol Table? does the library in fact instruct the application to make use of dlopen() instead of doing it itself?


Thaddy: i'd not be anywhere near brave enough to do what you suggest! what i am thinking of (ultimately) is ways to hand-pick pieces of musl-libc and include them directly into the Lazarus/FPC libraries where required to replace the current calls to (the small subset of problematic) libc routines. static linking of musl is just one step on the path to this end. the 'EULA' of glibc precludes doing this with the GNU code, however the musl license seems to be more permissive.


cheers,
rob   :-)

[robert rozee]

a little further digging leads to this project, that points to a solution:

https://github.com/pfalcon/foreign-dlopen

foreign-dlopen

"There's another model, which we'll call "FFI (Foreign Function Interface) model". It goes along the lines of:

1.    Suppose you have a perfect, closed world application. Statically linked of course.
2.    But you want to go out to dirty bustling outside world (in other words, let your application, or users of your appplication, to dlopen() outside shared libraries).
3.    There're absolutely no expectations or stipulations about which libc is used by those shared libraries. In particular, there's no expectations that libc of your application and external shared lib are the same. Or that you know which libc is used by external lib at all. For example, your static binary may be linked against musl libc, but you may want to load (allow to load) glibc bloat lying in abundance on a typical Linux system.

Again, the only thing you want is to maintain your static perfect world, independent from outside hustle. But, at user discretion, you want to allow this hustling outside world into your address space, by means of dlopen().

This cute project is a proof-of-concept solution for this usecase."

interesting?


cheers,
rob   :-)

[PascalDragon]

One moment...
Do you want to compile (and dynamically link) a program against musl and then use it on glibc systems?

Because that will never work...They are incompatible.

why are they incompatible?

You can't have two dynamic C libraries on your at once (or at least it's very hard to set up). The only thing you could do is to use something like AppImage to bundle your binary and all dependencies (because you'd replace the C library, thus any and every library you use needs to be recompiled against the musl libc as well), because it would then use different library search paths.

i can see some of the issues surrounding the use of dlopen() et al, but am not convinced they are insurmountable. certainly difficult, of that there is no doubt, but then most interesting problems are difficult. in an ideal world i'd like to see the functionality of dlopen() et al implemented in pascal - after all, these libraries that dlopen() et al opens are just ELF files.

but there is an interesting question of: can/do libraries themselves make use of dlopen() internally, or can dlopen() only be called by the user application? and if a library uses dlopen() internally to access another library, does this percolate up to an explicit entry in the application's Dynamic Symbol Table? does the library in fact instruct the application to make use of dlopen() instead of doing it itself?

If an application is statically linked it's essentially missing all named symbols that would be required for dlopen/dlsym to work. Also if you open another library it would need to dynamically link in the C library (because the opened library needs it) and this would then mean that you'd have the C library state twice in your application which is a recipe for desaster.

a little further digging leads to this project, that points to a solution:

https://github.com/pfalcon/foreign-dlopen

foreign-dlopen

[snip]

interesting?

This sounds needlessly complicated and prone to failure.

Anyway, what I would have tried in my youth is copy over the old libc to a safe location and rename musl-libc with the old name. That only works if the API is the exact same.
Maybe a worthwile attempt? Make sure the original can be placed back.

Due to the versioned symbols that current glibc uses this is not possible.

[robert rozee]

a little further digging leads to this project, that points to a solution:
https://github.com/pfalcon/foreign-dlopen
foreign-dlopen
[snip]
interesting?

This sounds needlessly complicated and prone to failure.

many things are prone to failure, but sometimes we are surprised when they don't. i was thinking of the part where pfalcon et al use a 'helper application' to retrieve the needed entry points to glibc and pass them back to the user program to be used to gain access libc.so functions without the user program needing to be dynamically linked to libc itself. the nice thing is that the helper application could potentially be very small, a few dozen or so lines of C code that the user program: (1) writes to a .c file in /tmp, (2) compiles by calling gcc, then (3) runs to retrieve the needed entry points. likely these steps could be accomplished (including running gcc) in a fraction of a second every time the user program is run:
https://github.com/pfalcon/foreign-dlopen/blob/master/src/fdlhelper.c


but then i've stumbled across something perhaps far easier here:
https://developers.redhat.com/blog/2019/08/01/how-the-gnu-c-library-handles-backward-compatibility
(my search inspired by https://news.ycombinator.com/item?id=23745297)

what if FPC scanned through libc.so.6 each time before doing a compile, using:

Code: Text  [Select][+][-]

1. readelf --dyn-syms -W /lib/x86_64-linux-gnu/libc.so.6

to collect a list of all the symbols defined within - including symbols with multiple versions available? FPC could then just use a set of 'earliest defined' symbols to defeat using the default (latest), or select symbols that fall within a 'known good range' of glibc versions.

for instance, take the symbol __libc_start_main:

Code: Text  [Select][+][-]

1. readelf --dyn-syms -W /lib/x86_64-linux-gnu/libc.so.6 | grep "start_main"
2.    674: 0000000000029dc0   328 FUNC    GLOBAL DEFAULT   15 __libc_start_main@@GLIBC_2.34
3.    678: 0000000000029dc0   328 FUNC    GLOBAL DEFAULT   15 __libc_start_main@GLIBC_2.2.5

(from my system's 'Ubuntu GLIBC 2.35-0ubuntu3.1')

now substitute in /usr/share/fpcsrc/3.2.2/rtl/linux/x86_64/si_c.inc and si_g.inc:

Code: Pascal  [Select][+][-]

1. procedure libc_start_main; external name '__libc_start_main@GLIBC_2.2.5';

for

Code: Pascal  [Select][+][-]

1. procedure libc_start_main; external name '__libc_start_main';

(well, not substitute, just change si_c.inc et al to pick up the string from an include file created from the readelf data)

as long as the version number used exists (which we have just checked using readelf) the linker should (as far as i can see, although i am far from an expert) be happy. we end up with a binary file that will run against an early version of glibc without needing to build on an 'old' VM. would this work?


cheers,
rob   :-)

[PascalDragon]

what if FPC scanned through libc.so.6 each time before doing a compile, using:

Code: Text  [Select][+][-]

1. readelf --dyn-syms -W /lib/x86_64-linux-gnu/libc.so.6

to collect a list of all the symbols defined within - including symbols with multiple versions available? FPC could then just use a set of 'earliest defined' symbols to defeat using the default (latest), or select symbols that fall within a 'known good range' of glibc versions.

for instance, take the symbol __libc_start_main:

Code: Text  [Select][+][-]

1. readelf --dyn-syms -W /lib/x86_64-linux-gnu/libc.so.6 | grep "start_main"
2.    674: 0000000000029dc0   328 FUNC    GLOBAL DEFAULT   15 __libc_start_main@@GLIBC_2.34
3.    678: 0000000000029dc0   328 FUNC    GLOBAL DEFAULT   15 __libc_start_main@GLIBC_2.2.5

(from my system's 'Ubuntu GLIBC 2.35-0ubuntu3.1')

now substitute in /usr/share/fpcsrc/3.2.2/rtl/linux/x86_64/si_c.inc and si_g.inc:

Code: Pascal  [Select][+][-]

1. procedure libc_start_main; external name '__libc_start_main@GLIBC_2.2.5';

for

Code: Pascal  [Select][+][-]

1. procedure libc_start_main; external name '__libc_start_main';

(well, not substitute, just change si_c.inc et al to pick up the string from an include file created from the readelf data)

No, we don't want something like that. The issue can be easily fixed by compiling on a system with the suitable version of the libary instead of implemting some frickle scheme.

as long as the version number used exists (which we have just checked using readelf) the linker should (as far as i can see, although i am far from an expert) be happy. we end up with a binary file that will run against an early version of glibc without needing to build on an 'old' VM. would this work?

And what if the user does not want the old behavior? What if the new function fixed some critical functionality and by using the old one you'll run into problems because code you expect to work against the new behavior will run into problems? The __libc_start_main is exactly such an example.