Recent

Author Topic: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found  (Read 64259 times)

WayneSherman

  • Sr. Member
  • ****
  • Posts: 254
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #195 on: July 15, 2023, 06:11:13 pm »
as long as the version number used exists (which we have just checked using readelf) the linker should (as far as i can see, although i am far from an expert) be happy. we end up with a binary file that will run against an early version of glibc without needing to build on an 'old' VM. would this work?

And what if the user does not want the old behavior? What if the new function fixed some critical functionality and by using the old one you'll run into problems because code you expect to work against the new behavior will run into problems? The __libc_start_main is exactly such an example.

Taking the example of the glob64 function:

Quote
...if we look at the 32-bit libc-2.29.so's dynamic symbol table, we see three versions of the glob64 function (in 2017, the glob function was changed to handle dangling symlinks differently, which would cause older programs to crash, but that's a different story):

Code: [Select]
$ readelf --dyn-syms -W /lib/libc-2.29.so | grep glob64
   411: 0012d0e0  7183 FUNC    GLOBAL DEFAULT   14 glob64@GLIBC_2.1
   412: 0012edb0  7183 FUNC    GLOBAL DEFAULT   14 glob64@GLIBC_2.2
   413: 000b69a0  7183 FUNC    GLOBAL DEFAULT   14 glob64@@GLIBC_2.27

If a security issue is found in glob64, I assert a future release of libc will fix ALL of the implementations.  For other types of bugs fixed in the same function I assert that ALL implementations should get those fixes, unless fixing that bug changes the behavior in a way that breaks too much existing code (and it is not a security issue).  If this is true, then linking to a specific (older) version of a function (which was used to design and test the FPC code) is the correct thing to do.



« Last Edit: July 15, 2023, 08:43:37 pm by WayneSherman »

robert rozee

  • Full Member
  • ***
  • Posts: 210
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #196 on: July 15, 2023, 07:37:06 pm »
checking for matching base name but different entry points reveals:
Code: Bash  [Select][+][-]
  1. user@DH61BE:~$ readelf --dyn-syms -W /lib/x86_64-linux-gnu/libc.so.6 | (sed -u 1q; LC_COLLATE=C sort -k8) | ./project3
  2. _sys_nerr                     GLIBC_2.12     GLIBC_2.2.5    GLIBC_2.3      GLIBC_2.4      
  3. fmemopen                      GLIBC_2.22     GLIBC_2.2.5    
  4. glob64                        GLIBC_2.27     GLIBC_2.2.5    
  5. glob                          GLIBC_2.27     GLIBC_2.2.5    
  6. lio_listio64                  GLIBC_2.34     GLIBC_2.2.5    GLIBC_2.4      
  7. lio_listio                    GLIBC_2.34     GLIBC_2.2.5    GLIBC_2.4      
  8. memcpy                        GLIBC_2.14     GLIBC_2.2.5    
  9. nftw64                        GLIBC_2.3.3    GLIBC_2.2.5    
  10. nftw                          GLIBC_2.3.3    GLIBC_2.2.5    
  11. posix_spawn                   GLIBC_2.15     GLIBC_2.2.5    
  12. posix_spawnp                  GLIBC_2.15     GLIBC_2.2.5    
  13. pthread_attr_getaffinity_np   GLIBC_2.34     GLIBC_2.3.3    GLIBC_2.3.4    
  14. pthread_attr_setaffinity_np   GLIBC_2.32     GLIBC_2.3.3    GLIBC_2.3.4    
  15. pthread_cond_broadcast        GLIBC_2.3.2    GLIBC_2.2.5    
  16. pthread_cond_destroy          GLIBC_2.3.2    GLIBC_2.2.5    
  17. pthread_cond_init             GLIBC_2.3.2    GLIBC_2.2.5    
  18. pthread_cond_signal           GLIBC_2.3.2    GLIBC_2.2.5    
  19. pthread_cond_timedwait        GLIBC_2.3.2    GLIBC_2.2.5    
  20. pthread_cond_wait             GLIBC_2.3.2    GLIBC_2.2.5    
  21. pthread_getaffinity_np        GLIBC_2.32     GLIBC_2.3.3    GLIBC_2.3.4    
  22. pthread_kill                  GLIBC_2.34     GLIBC_2.2.5    
  23. pthread_setaffinity_np        GLIBC_2.34     GLIBC_2.3.3    GLIBC_2.3.4    
  24. quick_exit                    GLIBC_2.24     GLIBC_2.10    
  25. realpath                      GLIBC_2.3      GLIBC_2.2.5    
  26. regexec                       GLIBC_2.3.4    GLIBC_2.2.5    
  27. sched_getaffinity             GLIBC_2.3.4    GLIBC_2.3.3    
  28. sched_setaffinity             GLIBC_2.3.4    GLIBC_2.3.3    
  29. sys_nerr                      GLIBC_2.12     GLIBC_2.2.5    GLIBC_2.3      GLIBC_2.4      
  30. timer_create                  GLIBC_2.34     GLIBC_2.2.5    GLIBC_2.3.3    
  31. timer_delete                  GLIBC_2.34     GLIBC_2.2.5    GLIBC_2.3.3    
  32. timer_getoverrun              GLIBC_2.34     GLIBC_2.2.5    GLIBC_2.3.3    
  33. timer_gettime                 GLIBC_2.34     GLIBC_2.2.5    GLIBC_2.3.3    
  34. timer_settime                 GLIBC_2.34     GLIBC_2.2.5    GLIBC_2.3.3    
  35. user@DH61BE:~$
  36. user@DH61BE:~$

using the following pascal program to do the filtering:
Code: Pascal  [Select][+][-]
  1. program project3;
  2.  
  3. Uses sysutils, strutils;
  4.  
  5. var A0, A1, N0, N1, V1, Si, So:string;
  6.                             go:boolean;
  7.                           I, J:integer;
  8.  
  9. begin
  10.   A0:='';                                              // address, previous iteration
  11.   N0:='';                                              // name, previous iteration
  12.   So:='';                                              // output line buffer
  13.   go:=false;
  14.   while not eof do
  15.   begin
  16.     readln(Si);
  17.  
  18.     I:=pos(': 0', Si);                                 // locate start of 'value' field (address)
  19.     if I<>0 then
  20.     begin
  21.       delete(Si, 1, I+1);
  22.       A1:=leftstr(Si, 16);                             // extract address, 16 hex digits (64-bit)
  23.  
  24.       Si:=trim(Si);                                   // trim off any leading or trailing spaces
  25.       if (rightstr(Si, 1)=')') and (pos(' (', Si)<>0)  then            // check for any trailing ' (nn)'
  26.       begin
  27.         while rightstr(Si, 1)<>' ' do delete(Si, length(Si), 1);       // trim back to a trailing space
  28.         Si:=trim(Si)                                                   // trim off trailing space(s)
  29.       end;
  30.  
  31.       while pos(' ', Si)<>0 do delete(Si, 1, 1);       // delete everything up to 'name' field
  32.  
  33.       I:=pos('@', Si);                                 // locate beginning of versioning (at end of name)
  34.       if I<>0 then
  35.       begin
  36.         N1:=leftstr(Si, I-1);                          // extract unversioned name
  37.         while pos('@', Si)<>0 do delete(Si, 1, 1);     // delete everything up to version (including @ or @@)
  38.         V1:=Si;                                        // and extract version
  39.  
  40.         if N1<>N0 then begin                           // name mismatch -> finish off previous line(s)
  41.                         if go then writeln(So);        // only write out if there is an address mismatch
  42.                         go:=false;
  43.                         So:=''                         // clear output buffer
  44.                       end
  45.                  else if A1<>A0 then go:=true;         // name match + address mismatch -> allow output
  46.  
  47.         if So='' then So:=padright(N1, 30) + padright(V1, 15)          // <name> <1st ver>
  48.                  else So:=So + padright(V1, 15);                       // append <nth ver>
  49.  
  50.         A0:=A1;
  51.         N0:=N1
  52.       end
  53.     end
  54.   end;
  55.   if go then writeln(So)                               // if address mismatch output the pending last line
  56. end.

the question is, how much do we care about these particular libc functions? are any of them used anywhere by the Lazarus LCL? or by programmers in general?


BTW, one of my search results found an appropriate description which struck me as hilarious:
  "Versioned Symbols - A New Level of Hell" :-)

in the windows community it is called "DLL Hell":
https://en.wikipedia.org/wiki/DLL_Hell


cheers,
rob   :-)


addendum: needed LC_COLLATE=C so that sort didn't do stupid things with underscore characters. this brings in sys_nerr and _sys_nerr. also updated the pascal filter program (project2 -> project3) to ensure no version numbers were missing from an output line.
addendum 11-aug-2023: added lines to trim any trailing '(nn)' after the version number. in practice, didn't change output.
« Last Edit: August 10, 2023, 03:56:36 pm by robert rozee »

WayneSherman

  • Sr. Member
  • ****
  • Posts: 254
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #197 on: July 16, 2023, 01:13:36 am »
checking for matching base name but different entry points reveals:
Code: Bash  [Select][+][-]
  1. user@DH61BE:~$ readelf --dyn-syms -W /lib/x86_64-linux-gnu/libc.so.6 | (sed -u 1q; sort -k8) | ./project2
  2.  
  3. fmemopen                      GLIBC_2.22     GLIBC_2.2.5    
  4. glob64                        GLIBC_2.2.5    GLIBC_2.27    
  5. glob                          GLIBC_2.2.5    GLIBC_2.27    
  6. ...

In addition, this shows glibc API and ABI changes and backward compatibility across versions:
https://abi-laboratory.pro/?view=timeline&l=glibc
(the open source tools to generate these reports are linked at the bottom of the page)

See also:  https://sourceware.org/glibc/wiki/Testing/ABI_checker

Breaking changes are shown in the Binary Compatibility reports:
https://abi-laboratory.pro/index.php?view=compat_report&l=glibc&v1=2.35&v2=2.37&obj=7dee8&kind=abi

One example from that page for poll.h, libc.so.6
[−] ppoll ( struct pollfd* fds, nfds_t nfds, struct timespec const* timeout, sigset_t const* sigmask ) @@ GLIBC_2.4
Change:
The parameter sigmask became passed in rbp register instead of r10.
Effect:
Applications will read the wrong memory block instead of the parameter value.
« Last Edit: July 16, 2023, 01:25:52 am by WayneSherman »

TRon

  • Hero Member
  • *****
  • Posts: 3946
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #198 on: July 16, 2023, 02:20:06 am »
In addition, this shows glibc API and ABI changes and backward compatibility across versions:
https://abi-laboratory.pro/?view=timeline&l=glibc
(the open source tools to generate these reports are linked at the bottom of the page)
Thank you for that very informative link (I was not aware such thing even existed).
I do not have to remember anything anymore thanks to total-recall.


WayneSherman

  • Sr. Member
  • ****
  • Posts: 254
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #200 on: July 20, 2023, 11:05:16 pm »
I think glibc or parts of it should be depended on as little as possible which will result in fewer issues and be easier to maintain.

If most agree that the final executable (elf) should have *versioned* symbols, then how they get there is the main question.

Should the symbol versions be specified the the fpc source code or should the linker create the versioned symbols based on the libraries present on the system at compile time?

Related questions:
Are the fpc glibc header translations kept up-to-date with glibc API changes?
Is the fpc runtime itself a user the glibc api?
Does fpc expose any parts of the glibc api for general use by other developers?

robert rozee

  • Full Member
  • ***
  • Posts: 210
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #201 on: July 21, 2023, 02:38:02 pm »
If most agree that the final executable (elf) should have *versioned* symbols, then how they get there is the main question. Should the symbol versions be specified the the fpc source code or should the linker create the versioned symbols based on the libraries present on the system at compile time?
Are the fpc glibc header translations kept up-to-date with glibc API changes?
Is the fpc runtime itself a user the glibc api?
Does fpc expose any parts of the glibc api for general use by other developers?

for programs that use threading or the LCL, glibc functions are used. for a simple 'console application' (non-GUI) without threading, the glibc functions are NOT used. the glibc functions may also be freely used by the programmer, but that is the programmer's problem to deal with. currently the latest version of each glibc symbol is linked to, but the code within the FPC RTL and Lazarus LCL is old - in the case of the RTL dating back to 2005. this is a major problem and will eventually result in FPC/Lazarus no longer being able to generate working linux x86-64 binaries.

i have been in touch with the glibc developers, and the feedback i received is that:
the FPC RTL should be explicitly specifying glibc symbol versions, and the symbol versions used should be the "base" ones which are relatively easy to extract from libc.so.6 in an automated fashion. the "base" symbols do not change over time, and will be maintained essentially for the life of the x86-64 instruction set. these 'base symbols' will pair with the current RTL pascal source without needing to change any code (beyond actually specifying the versioned symbols). these versioned symbols will needed to be extracted once for each architecture from a libc.so.6, and then incorporated in the RTL source. for x86-64 most will likely be @GLIBC_2.2.5

for example dlopen should be implemented thus:
function dlopen(Name : PChar; Flags : longint) : Pointer; cdecl; external libdl name 'dlopen@GLIBC_2.2.5';


the full discussion with the libc developers is available here:
https://sourceware.org/pipermail/libc-alpha/2023-July/thread.html#150163

attached below is a .pdf of the (at the moment) 13 email messages with replies highlighted.


cheers,
rob   :-)


Handoko

  • Hero Member
  • *****
  • Posts: 5387
  • My goal: build my own game engine using Lazarus
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #202 on: July 21, 2023, 04:10:30 pm »
Hello all. I did not follow the discussion but hope it can be useful for others.

I ever had libc issue with my Lazarus but that only happened when I installed latest version of Lazarus on a several years old Ubuntu. I solved the problem by using manual build Lazarus.
« Last Edit: July 21, 2023, 04:12:11 pm by Handoko »

MvC

  • New Member
  • *
  • Posts: 25
    • Free Pascal Core team member
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #203 on: July 22, 2023, 11:15:52 am »
Hello,

I've been notified of this discussion by private mail.

That the libc unit would need to use versioning on linux to cater for the correct struct  layouts, is clear.

But can someone please *exactly* explain in a private mail to michael@freepascal.org why dlopen() would need to be versioned ?

From all messages here (and a lot of private messages) I still do not have a clear picture why dlopen would need to be versioned.
Sure, it could be done easily enough, but even the mails to Florian Weimer do not properly explain the why.
Exactly what problem would that solve ?

Please include a simple console application that can be used to demonstrate the problem.

In order to solve a problem, you need first to exactly identify the problem.
From all the messages here, I still get no clear picture of what exactly you're trying to solve.

Michael.

robert rozee

  • Full Member
  • ***
  • Posts: 210
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #204 on: July 23, 2023, 08:22:02 am »
[...] can someone please *exactly* explain [...] why [a symbol] would need to be versioned ?
[...] In order to solve a problem, you need first to exactly identify the problem.

email sent. in summary, consider memcpy()
https://man7.org/linux/man-pages/man3/memcpy.3.html

       Failure to observe the requirement that the memory areas do not
       overlap has been the source of significant bugs.  (POSIX and the
       C standards are explicit that employing memcpy() with overlapping
       areas produces undefined behavior.)  Most notably, in glibc 2.13
       a performance optimization of memcpy() on some platforms
       (including x86-64) included changing the order in which bytes
       were copied from src to dest.

       This change revealed breakages in a number of applications that
       performed copying with overlapping areas.  Under the previous
       implementation, the order in which the bytes were copied had
       fortuitously hidden the bug, which was revealed when the copying
       order was reversed.  In glibc 2.14, a versioned symbol was added
       so that old binaries (i.e., those linked against glibc versions
       earlier than 2.14) employed a memcpy() implementation that safely
       handles the overlapping buffers case (by providing an "older"
       memcpy() implementation that was aliased to memmove(3)).


with FPC, if memcpy() is not explicitly versioned in the Pascal source, then the version of the memcpy() symbol present in the resulting ELF binary will vary depending on the installed glibc at compile-time. this version decision is made by the compile-time linker: either memcpy@GLIBC_2.2.5 or memcpy@GLIBC_2.14. the linker always selects for the 'newest' version found at compile-time if not directed otherwise.

the resulting binary will behave differently depending on the version of memcpy() selected by the compile-time linker.


the same principal applies to __libc_start_main. changes are 'in the air', and there is suggestion that at some not-too-distant future time the parameters to __libc_start_main may be changed. this will result in a new versioned __libc_start_main being introduced, which FPC will link against unless directed otherwise. this will result in an ELF binary that crashes if created on a machine that has this new version of __libc_start_main available. simple as that.

the same principal as applies to __libc_start_main then applies to dlopen() or any other glibc symbol.

to avoid this Florian Weimer has confirmed that FPC should link against the BASE version of the glibc symbols. essentially, this is the @GLIBC_2.2.5 versions - getting the right version numbers involves some one-off processing of the output from readelf -a libc.so.6.


cheers,
rob   :-)
« Last Edit: July 23, 2023, 08:52:24 am by robert rozee »

abouchez

  • Full Member
  • ***
  • Posts: 124
    • Synopse
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #205 on: July 25, 2023, 09:41:21 am »
Some related discussion and code at https://github.com/synopse/mORMot2/issues/211

Even if my program uses the proper symbol names, we are still forced to compile with an old libc.
We really need some RTL changes at low level, e.g. to resollve the `__libc_start_main@@GLIBC_2.34` hardcoded symbol issue.

In the meanwhile, the only solution seems to compile on an old Linux distribution.
What I sometimes do, if I don't want to use another distribution, is to run the Windows-to-Linux fpcupdeluxe cross-compilers using Wine.  8-)

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 12032
  • FPC developer.
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #206 on: July 25, 2023, 09:50:28 am »
Keep in mind that in general we only use syscall wrapping functions from libc, not the language utilities.   There are usually simple wrappers.

So no memcpy or glob but own move() and opendir/readdir/closedir implementations on top of getdirectoryentries.


abouchez

  • Full Member
  • ***
  • Posts: 124
    • Synopse
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #207 on: July 25, 2023, 11:27:19 am »
Keep in mind that in general we only use syscall wrapping functions from libc, not the language utilities.
You are right:  most of the OS APIs are using direct syscall in the FPC RTL.
But this is not true for the `__libc_start_main` link in system.pp nor libdl nor cthreads units, IIRC.
And a single broken name link is enough to avoid proper execution.

Also note that calling syscall is not a good idea, e.g. when there are some VDSO wrappers available.
clockgettime syscall is 20 times slower with a syscal than the one from libc... and this function could be very often call, e.g. if you measure execution time with accuracy, which is a very common task.
See https://synopse.info/forum/viewtopic.php?pid=31342#p31342
« Last Edit: July 25, 2023, 11:34:47 am by abouchez »

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 12032
  • FPC developer.
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #208 on: July 25, 2023, 12:15:37 pm »
Quote
Also note that calling syscall is not a good idea, e.g. when there are some VDSO wrappers available.
clockgettime syscall is 20 times slower with a syscal than the one from libc... and this function could be very often call, e.g. if you measure execution time with accuracy, which is a very common task.
See https://synopse.info/forum/viewtopic.php?pid=31342#p31342

That a libc call could do more than just syscall  was already known 19 years ago, which is why I crafted the -dFPC_USE_LIBC option back then. (actually originally for FreeBSD, but quickly ported to Linux) for people that encounter such issues. The question is if it should be the default or not, and what that means for the longevity of a release.  IMHO distribution specific packages (.deb .rpm) should have changed long ago, since they are not multi distro packages to begin with.

The value of syscall for bootstrapping using a single binary release was a known factor however, and we were all reluctant to desert it.  Partially because there was so much going on with libc all the time. In this era of LTS releases that has quieted down somewhat, but keep in mind that every breakage of a release tar (still the official Linux distribution of FPC) is one too many.

Personally, I gave up on Linux for anything but local work (so no distributing binaries as per Windows model)

All the things in this thread have come up multiple times over the years, but the point is that nobody really tried to create a multi distribution binary based on it, maintain it for a while and prove that the concept really works rather than just echoing factoids and well meant intentions rather than real experiences.

Specially in this GIT era, it would be so easy to have a branch shadowing trunk or fixes for that.

Keep in mind that in general we only use syscall wrapping functions from libc, not the language utilities.
You are right:  most of the OS APIs are using direct syscall in the FPC RTL.
But this is not true for the `__libc_start_main` link in system.pp nor libdl nor cthreads units, IIRC.
And a single broken name link is enough to avoid proper execution.

Yeah, but they are not used in the compiler. I would at least retain syscall builds for the bootstrap compilers of every release.

DonAlfredo

  • Hero Member
  • *****
  • Posts: 1792
Re: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found
« Reply #209 on: July 25, 2023, 01:10:30 pm »
I am willing to help, if versioned linking might/will be accepted by FPC devs.
I have already made a FPC source scanner to find all libc-calls based on this header file.
Based on headers from: https://github.com/wheybags/glibc_version_header/blob/master/version_headers/x64/force_link_glibc_2.27.h

Proposal.
All libc-calls (the externals) might be defined from an include file that has a versioned and an unversioned definition of every call. Would also cleanup the FPC sources a bit. As stated previously, care must be taken to include all libc CPU-OS combo's.

 

TinyPortal © 2005-2018