/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found

[marcov]

I am willing to help, if versioned linking might/will be accepted by FPC devs.

First prove that something works, then talk about accepting.

I have already made a FPC source scanner to find all libc-calls based on this header file.
Based on headers from: https://github.com/wheybags/glibc_version_header/blob/master/version_headers/x64/force_link_glibc_2.27.h

That has been attempted before too, but is arch specific, and doesn't actually fix the real problem of C interfacing for libc purposes, namely the structures and macros (like FD_SET etc).

Proposal.
All libc-calls (the externals) might be defined from an include file that has a versioned and an unversioned definition of every call. Would also cleanup the FPC sources a bit. As stated previously, care must be taken to include all libc CPU-OS combo's.

Too many CPU-OS combos and too many places where libc is linked to be practical (we had unit libc once and it was a nightmare), and as said this autogeneration only fixes the calls, not the rest including non Linux *nix that doesn't benefit from this.

I think a saner way would be to have a small patch to the linker backend of the compiler to do substitutions there based on a file ( symbol x in lib y should become  y@4234 in lib x). That can be kept out of tree easily to prove the concept.

[robert rozee]

Too many CPU-OS combos and too many places where libc is linked to be practical

all we need to care about is x86-64 Linux, everything else can be left to fend for itself. if that works, then expand out to cover other variants later on.

use a simple include file, containing something like:

{$IF not DECLARED(symbol_libc_start_main)}
  {$IFDEF LINUX and x86_64}
    const symbol_libc_start_main='__libc_start_main@GLIBC_2.2.5';
          symbol_dlopen         ='dlopen@GLIBC_2.2.5';
          symbol_dlsym          ='dlsym@GLIBC_2.2.5';
          // etc...
  {$ELSE}
    const symbol_libc_start_main='__libc_start_main';
          symbol_dlopen         ='dlopen';
          symbol_dlsym          ='dlsym';
          // etc...
  {$ENDIF}
{$ENDIF}
(yes, i know i have the $IF syntax all messed up!)

then for every external statement substitute:

... external 'libc' name symbol_xxxx
or
... external 'libdl' name symbol_xxxx
as required.

the include file can be included at the top of every file that contains externals.

the necessary version number for each symbol can be extracted relatively easily from libc.so.6 using readelf.


cheers,
rob   :-)

[abouchez]

Proposal.
All libc-calls (the externals) might be defined from an include file that has a versioned and an unversioned definition of every call. Would also cleanup the FPC sources a bit. As stated previously, care must be taken to include all libc CPU-OS combo's.

This include file could define such a constant:
https://github.com/synopse/mORMot2/commit/053b098c398f8f265ce148796aae6da4bc09dc69

[WayneSherman]

But can someone please *exactly* explain in a private mail to michael@freepascal.org why dlopen() would need to be versioned ?

Each exported symbol in glibc has defined behavior, ABI, and API for a specific version only.  Some exported functions (and other symbols) have changed in ways that break existing source code and binaries and which could not be predicted in advance.  There is no guarantee that they won't have breaking changes in the future (any of them including dlopen).

What is symbol versioning good for? Do I need it?
https://sourceware.org/glibc/wiki/FAQ#What_is_symbol_versioning_good_for.3F__Do_I_need_it.3F

Symbol versioning solves problems that are related to interface changes. One version of an interface might have been introduced in a previous version of the GNU C library but the interface or the semantics of the function has been changed in the meantime...
We don't advise building without symbol versioning, since you lose binary compatibility - forever! The binary compatibility you lose is not only against the previous version of the GNU libc but also against all future versions. This means that you will not be able to execute programs that others have compiled.

[WayneSherman]

the necessary version number for each symbol can be extracted relatively easily from libc.so.6 using readelf.

Each FPC rtl interface declaration for glibc (i.e. FPC glibc header translations) must be used with a compatible symbol version in the glibc library.  Is there confidence that all the FPC glibc interface declarations are API compatible with the "base" versions of the glibc symbols?

A related, but separate question:  Is there confidence that the behavior of the "base" version of the symbols is compatible with the way FPC expects it to behave?

[robert rozee]

see here:
https://forum.lazarus.freepascal.org/index.php/topic,58888.msg485518.html#msg485518

there are only about 30 sets of base names in glibc where different versions have different entry points, and most of these are not used in the RTL.

the assumption must be that the current FPC RTL has been written to be compatible with the BASE version of each symbol. if not, we would have seen widespread reports of failures from the many thousands of FPC/Lazarus users out there   


cheers,
rob   :-)

[marcov]

the necessary version number for each symbol can be extracted relatively easily from libc.so.6 using readelf.

Each FPC rtl interface declaration for glibc (i.e. FPC glibc header translations) must be used with a compatible symbol version in the glibc library.  Is there confidence that all the FPC glibc interface declarations are API compatible with the "base" versions of the glibc symbols?

First: there is no specific glibc header. Multiple parts import symbols from libc in isolation in a cross *nix matter. That ranges from Linux,*BSD,OSX to AIX, Solaris to half *nixes like BeOS and in the past Netware-libc.  This roughly follows header divisions in *libc too.

So the base RTL imports base calls like mmap,seek and read/write. (see rtl/unix/bunxh.inc) _IF_ it doesn't use syscalls, which are default. The socket units import socket related calls, the optional memory manager substitute imports malloc/free/realloc, one of several DNS resolves import libc, the Users package does the user and password stuff etc etc, unit DL manages dynamic library loading etc. There is no single unified glibc header, and those local declarations usually have a OS specific part for header parts with variation among OSes, and use *nix wide declarations for the rest, possibly even ifdefed for arch etc.

This is all to minimize maintenance and keep a bit of overview.

I expect no problems with the calls themselves. Symbol versioning would be more about fixating a certain version for a while (and cross distro) because FPC doesn't adapt automatically to different headers (with different structs, constants and macros). So the same build on an old Debian STABLE and the newest rolling ARCH can be seen as using the same headers etc, but linking  to the same default symbol (_read, _write etc), whose version might differ between those distros.

For symbol versioning to improve anything, a debian old stable and a new a flashy distro would both have to implement exactly the same base subset. i.e. using the exact same definition of the STAT record/struct etc.   Or adding some extra constant to an ORed set of values, or change the implementation of some macro.

But as said, nothing of this is proven. It is all theory. Something else might become the next bottleneck preventing cross distro binary generation like changes to startup code (libc initialization!), or linker changes. The problems are not even properly investigated even, it is all anecdotal. We don't even have proper knowledge about omitting the syscalls, even though that option exists for 19 years.

At this stage, interested users should do a long term test to see if any proposed changes or strategy really improve cross-distro compatibility for a longer term (e.g. release cycles). To ease that it would be wise to do a setup with minimal source changes to make it easier to rebase to trunk regularly. See my previous post.

Things like reordering all sources to specific OS-target combinations is out of the question. It is way to invasive for what it solves and hurts all  *nixes for a Linux problem. Same with decorating all symbols in sources. As soon as multiple distros with differing base versions happen during some transition (and the 5+ year window that old LTSes die out), that is an mess.

A mapping file that works on the .o writing backend would be more logical, and could be switched in such cases.

A related, but separate question:  Is there confidence that the behavior of the "base" version of the symbols is compatible with the way FPC expects it to behave?

See above. I don't really think it is detail behaviour of the call itself, but more pinning a certain version that match the translated headers definition, and less tension about what to do, update the header for newer distros, or keep it working with LTSes.
 
The base call prototypes themselves work across *nix mostly OSX, *BSD and Linux. But BSD usually saves compatibility breaks for major version transitions so this problem is easier to manage, and also has the seemless COMPAT mechanisms to keep binaries built on older binaries working on newer.  (but the FPC BSD ports recently suffer from the LLVM transitions).

One could argue though that during a base version transition there should be simply two FPC releases to pick from.

Also who is going to manage all this long term, monitor the glibc lists to provide updates or alert on changes etc etc?

[robert rozee]

I think a saner way would be to have a small patch to the linker backend of the compiler to do substitutions there based on a file ( symbol x in lib y should become  y@4234 in lib x). That can be kept out of tree easily to prove the concept.

marcov: that is an excellent idea. are you able to do this, or able to offer guidance on how to do this?


can we achieve any useful result using:
   --allow-shlib-undefined
or
   --no-allow-shlib-undefined
to prevent the compile-time linker from appending @GLIBC... to symbols in the Dynamic Symbol Table? while this method is not recommended by the GLIBC developers, i think it is still working pursuing as a parallel project that may provide some utility and knowledge.

addendum 1: i may have the linker switch --allow-shlib-undefined wrong here, i'm wanting to talk about the switch that allows unversioned symbols to remain unversioned

[...]
For symbol versioning to improve anything, a debian old stable and a new a flashy distro would both have to implement exactly the same base subset. i.e. using the exact same definition of the STAT record/struct etc.   Or adding some extra constant to an ORed set of values, or change the implementation of some macro.
[...]
Also who is going to manage all this long term, monitor the glibc lists to provide updates or alert on changes etc etc?

"a debian old stable and a new a flashy distro would both have to implement exactly the same base subset"

this is exactly what Florian Weimer has promised us - for a given architecture, the original BASE set (@GLIBC_2.2.5) will remain fixed in stone for life. as new functions are added, they will be brought into the BASE set with later @GLIBC_2.xx version numbers, but there is no compulsion for anyone to use those new functions.

"who is going to manage all this long term"

for a given architecture there is no need for long term management. the original BASE set is fixed in stone. it will never change. it needs to be arrived at once, through a relatively simple process. after that, we can forget it and go back to using FPC/Lazarus to write useful programs for sharing with the wider community.


cheers,
rob   :-)

addendum 2: what if external would accepted a further (optional) parameter, being one of: linkbase, linkauto, linknone? these would yield for the example dlopen, respectively: 'dlopen@GLIBC_2_2_5', 'dlopen@GLIBC_2.34', 'dlopen'.

addendum 3: linkbase could trigger using readelf to extract the base version number for a given symbol at compile-time, hence no need to construct and have stored any translation table.

[robert rozee]

a new approach, based on information provided in the thread:
https://forum.lazarus.freepascal.org/index.php/topic,64158.0.html

this involves modifying the function pd_external() in the file pdecsub.pas to automatically intercept and add versioning to selected glibc symbols:

Code: Pascal  [Select][+][-]

1. // modify the file /usr/share/fpcsrc/3.2.2/compiler/pdecsub.pas
2. // adding the code below to the end of pd_external() at approximately line 2340.
3.  
4. procedure pd_external(pd:tabstractprocdef);
5. {
6.   If import_dll=nil the procedure is assumed to be in another
7.   object file. In that object file it should have the name to
8.   which import_name is pointing to. Otherwise, the procedure is
9.   assumed to be in the DLL to which import_dll is pointing to. In
10.   that case either import_nr<>0 or import_name<>nil is true, so
11.   the procedure is either imported by number or by name. (DM)
12. }
13. var
14.   hs, sym_name, ver_name : string;                     // **** added sym_name and ver_name ****
15.   v:Tconstexprint;
16.   is_java_external: boolean;
17. begin
18. ...
19. ...
20. ...
21.              import_name:=stringdup(get_stringconst);
22.              include(procoptions,po_has_importname);
23.              if import_name^='' then
24.                message(parser_e_empty_import_name);
25.            end;
26.         end;
27.  
28. //      ******** added code follows ********
29. //      writeln('!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!');
30.  
31.         if assigned(import_name) then sym_name:=import_name^
32.                                  else sym_name:=proc_get_importname(tprocdef(pd));
33.                                              
34.         ver_name:='' ;
35.         if sym_name='__libc_start_main' then ver_name:='__libc_start_main@GLIBC_2.2.5';
36.         if sym_name='dlopen'            then ver_name:='dlopen@GLIBC_2.2.5';
37.         if sym_name='dlclose'           then ver_name:='dlclose@GLIBC_2.2.5';
38.         if sym_name='dlsym'             then ver_name:='dlsym@GLIBC_2.2.5';
39.         if sym_name='dlvsym'            then ver_name:='dlvsym@GLIBC_2.2.5';
40.         if sym_name='dladdr'            then ver_name:='dladdr@GLIBC_2.2.5';    
41.         if sym_name='dlerror'           then ver_name:='dlerror@GLIBC_2.2.5';
42.         if sym_name='dlinfo'            then ver_name:='dlinfo@GLIBC_2.2.5';
43.        
44.         if ver_name<>'' then begin
45.                                import_name:=stringdup(ver_name);
46.                                writeln('%%%% versioning ', sym_name, ' >> ', ver_name)
47.                              end
48. //
49. // remember to also add the symlink libdl.so -> libdl.so.2 using:
50. //
51. // sudo ln -s /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so
52. //
53.  
54. //      writeln('------------------------------------------------------------------------')
55. //      ******** end of added code ********
56.     end;
57. end;

after modifying, create a symlink from libdl.so to libdl.so.2 using:
sudo ln -s /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so

then rebuilding the RTL and compiler using rebuild.sh (from Fred vS) placed in the /usr/share/fpcsrc/3.2.2 directory:

Code: Bash  [Select][+][-]

1. #!/bin/sh
2. COMPILER=fpc
3. make clean
4. make all FPC=$COMPILER OPT="-Fl/usr/local/lib"
5. make FPC=$COMPILER install INSTALL_PREFIX=/usr/

you will likely need to add other symbols that also need versioning as 2.2.5, and of course the version number is specific to x86-64 Linux. conditionals should be added around the code to ensure this.

while currently hard-coded, in theory the correct version information could be extracted on-the-fly using calls to readelf, but i have not tried this yet.

ideally, the added functionality within pd_external() should be turned on/off using a compiler switch, as it is normally only going to be needed when compiling the compiler, RTL and LCL. but i don't know how to do this (yet). alternatively, symbols that you want to BASE version could have a unique version added, such as dlopen@BASE, where @BASE would be substituted with the correct version number. but this would then require the work of going through and modifying every external statement in the RTL and LCL.

enjoy!

cheers,
rob   :-)

[WayneSherman]

for a given architecture there is no need for long term management. the original BASE set is fixed in stone. it will never change. it needs to be arrived at once, through a relatively simple process. after that, we can forget it and go back to using FPC/Lazarus to write useful programs for sharing with the wider community.

a new approach...involves modifying the function pd_external() in the file pdecsub.pas to automatically intercept and add versioning to selected glibc symbols...

Since this only needs to be done once, isn't the most direct and simple solution to manually add the symbol version to the external function declarations?:

Code: Pascal  [Select][+][-]

1. function dlopen(__file:Pchar; __mode:longint):pointer;cdecl;external dllib name 'dlopen@GLIBC_2.2.5';
2. //or like fredvs did
3. function dlopen(__file:Pchar; __mode:longint):pointer;cdecl;external dllib name 'dlopen' + LIBC_SUFFIX;

That way if changes are made to the declarations in the future, you see immediately which version is being called (or at least that there might be a specific version if LIBC_SUFFIX is set).  It seems to me that modifying "pdecsub.pas" to hide versioning "magic" in the compiler will produce effects that require special knowledge (and if you don't happen to know about the side effects it is easier to introduce bugs and harder to debug problems).

What if someone declares a specific version in their own code.  Does the "pdecsub.pas" change cause a problem for this?:

Code: Pascal  [Select][+][-]

1. function dlopen_v238(__file:Pchar; __mode:longint):pointer;cdecl;external dllib name 'dlopen@GLIBC_2.3.8';
2. // granted instead of static loading, it would be better to dynamically load any
3. // modern libc functions using gnu_get_libc_version(), dlopen(), and dlsym()

[robert rozee]

Since this only needs to be done once, isn't the most direct and simple solution to manually add the symbol version to the external function declarations?

this is just an experimental alternative. but there are a few unique advantages:

1. you don't need to track down external statements from several different places in the RTL, all symbol versioning is in a single place,
2. it is immune to future changes or additions to the RTL that may slip through unnoticed,
3. the versioning information can potentially be loaded from a file by the compiler, hence can be made editable without rebuilding the compiler + RTL.
addendum: while you can edit the versioning information without rebuilding the RTL, as the RTL is pre-built the already versioned symbols in it (from when it was last built) would not change when a user program was built. editing the versioning list file would just affect symbols in the user-code. hence the binary you created could (quite validly) have two versions of the same symbol present in its Dynamic Symbol Table. (9-august-2023)

a hybrid of the two approaches would be for the code in pd_external() to be triggered by any symbol ending in '@BASE', and then specify that every external in the RTL must end this way - there is no good reason why the RTL should need to use anything other than the BASE versions of glibc functions.

What if someone declares a specific version in their own code.  Does the "pdecsub.pas" change cause a problem for this?:

// granted instead of static loading, it would be better to dynamically load any
// modern libc functions using gnu_get_libc_version(), dlopen(), and dlsym()

symbols that are already versioned are left untouched. i agree 100% that the RTL etc should be rewritten so that it uses dlopen() and dlym() instead of external, it would be a little bit more work but produce a much cleaner result.

---

below is an updated version of the pd_external() code that cleans up the way symbols are checked for translation. you can see how STT could instead be a variable, with content loaded in the initialization section at the end of the unit's source from a text file:

Code: Pascal  [Select][+][-]

1. procedure pd_external(pd:tabstractprocdef);
2. {
3.   If import_dll=nil the procedure is assumed to be in another
4.   object file. In that object file it should have the name to
5.   which import_name is pointing to. Otherwise, the procedure is
6.   assumed to be in the DLL to which import_dll is pointing to. In
7.   that case either import_nr<>0 or import_name<>nil is true, so
8.   the procedure is either imported by number or by name. (DM)
9. }
10. const STT='$__libc_start_main@GLIBC_2.2.5'+    // '$' before each versioned symbol name
11.           '$dlopen@GLIBC_2.2.5'+
12.           '$dlclose@GLIBC_2.2.5'+
13.           '$dlsym@GLIBC_2.2.5'+
14.           '$dlvsym@GLIBC_2.2.5'+
15.           '$dladdr@GLIBC_2.2.5'+  
16.           '$dlerror@GLIBC_2.2.5'+
17.           '$dlinfo@GLIBC_2.2.5'+'$';           // MUST also be a trailing '$' at very end
18.  
19. var
20.   hs, sym_name, ver_name : string;
21.   I:integer;
22.   v:Tconstexprint;
23.   is_java_external: boolean;
24. begin
25.     ...
26.     ...
27.     ...
28.              import_name:=stringdup(get_stringconst);
29.              include(procoptions,po_has_importname);
30.              if import_name^='' then
31.                message(parser_e_empty_import_name);
32.            end;
33.         end;  
34.  
35. //      ******** added code follows ********    
36. //      writeln('!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!');
37.  
38.         if assigned(import_name) then sym_name:=import_name^
39.                                  else sym_name:=proc_get_importname(tprocdef(pd));
40. //      writeln('symbol = ', sym_name);        
41.  
42.         ver_name:='';
43.         if pos('@', sym_name)=0 then        // no versioning present -> attempt translation
44.         begin
45.           I:=pos('$'+sym_name+'@', STT);    // try to find symbol in Symbol Translation Table
46.           if I<>0 then                      // match found, note that it starts with '$'
47.           repeat
48.             inc(I);                         // index of each character in versioned symbol name
49.             ver_name:=ver_name+STT[I]       // append character to ver_name
50.           until STT[I+1]='$'                // repeat until next character to retrieve is a '$'
51.         end;
52.        
53.         if ver_name<>'' then begin
54.                                import_name:=stringdup(ver_name);
55.                                writeln('%%%% versioning ', sym_name, ' >> ', ver_name)
56.                              end
57. //
58. // remember to also add the symlink libdl.so -> libdl.so.2 using:
59. // sudo ln -s /lib/x86_64-linux-gnu/libdl.so.2 /lib/x86_64-linux-gnu/libdl.so
60. //
61. //      writeln('------------------------------------------------------------------------')
62. //      ******** end of added code ********
63.     end;
64. end;

cheers,
rob   :-)

[marcov]

Since this only needs to be done once, isn't the most direct and simple solution to manually add the symbol version to the external function declarations?:

The declarations are shattered over many files and packages and are not Linux specific, but mostly for all *nix and even some half nixes like BeOS

In any case, it would preferable to solve this without source or language modifications, e.g. in the ELF object writer or so.

[robert rozee]

The declarations are shattered over many files and packages and are not Linux specific, but mostly for all *nix and even some half nixes like BeOS

In any case, it would preferable to solve this without source or language modifications, e.g. in the ELF object writer or so.

does BeOS suffer from symbol versioning as well?

one problem with attacking the problem at the ELF end is that then you will trample on any symbols that the programmer wants to be a specific version. for instance, the RTL may need to use an older version of memcpy, while someone writing a program using FPC may depend on a newer version of memcpy. the only ways around this are at the level of either the RTL source, or at the compiler.

there is also a problem with a couple of symbols, setenv and __cxa_finalize. both seem to be referenced from unknown places, with only my fake libdl.so catching them.


cheers,
rob   :-)

[Seenkao]

Без "танцев с бубном" данная проблема так и не была решена?

Почему, в новых системах происходит указание на новую ссылку? Почему компилируя программу в "старой" системе, я не получаю подобных ошибок?

Google translate:
Without "dancing with a tambourine" this problem was never solved?

Why does a new link point to a new link in new systems? Why do I not get similar errors when compiling a program on the “old” system?

[marcov]

does BeOS suffer from symbol versioning as well?

Maybe. It had glibc at one point, but has gone through several iterations.

FreeBSD and *BSD has a more generational model, with very hard compatibility within a major version cycle (2-4 years), and not much guaranteed outside that.

one problem with attacking the problem at the ELF end is that then you will trample on any symbols that the programmer wants to be a specific version. for instance, the RTL may need to use an older version of memcpy, while someone writing a program using FPC may depend on a newer version of memcpy. the only ways around this are at the level of either the RTL source, or at the compiler.

FPC doesn't use memcpy. And such far fetched scenarios can be tackled by the programmer that needs them.

I'm more interested in keeping the standard distribution working out of the box for most users, and rather avoid complex and possible fragile workarounds to theoretical increase, but practically decrease that group.

there is also a problem with a couple of symbols, setenv and __cxa_finalize. both seem to be referenced from unknown places, with only my fake libdl.so catching them.

Might be ld-linux or glibc startup code. Not necessarily FPC.