FPC 3.2.2 / Lazarus 2.2.6 patch for GLIBC_2.34 versioning issue

[marcov]

To my best knowledge, FPC doesn't export FNMatch with a unmangled symbol?

There are many simple words that are duplicates, like read and write etc, but Pascal namespace mangling keeps that from C.

/usr/share/fpcsrc/3.2.2/rtl/netwlibc/libc.pp:function fnmatch(pattern, _string:Pchar; flags:longint):longint;cdecl;external libc_nlm name 'fnmatch';
/usr/share/fpcsrc/3.2.2/packages/gtk2/src/gtk+/gtk/fnmatch.inc:function fnmatch(__pattern: char; __string: char;
/usr/share/fpcsrc/3.2.2/packages/libc/src/fnmatchh.inc:function fnmatch(__pattern:Pchar; __name:Pchar; __flags:longint):longint;cdecl;external clib name 'fnmatch';

Ok so, gtk2, legacy Kylix libc package and netware (which is also not actively supported).

But while those are mostly dead, gtk2 is still alive, so it might be useful to find out what this is for 

fnmatch is passed as a parameter to pd_external() (that resides in the file pdecsub.pas), and is accompanied by the library name libgtk-x11-2.0.so. if the symbol is versioned to fnmatch@GLIBC_2.2.5 then any attempt to compile a GUI application fails, while non-GUI applications compile fine. suppressing versioning of fnmatch, by excluding any substitution when a library name containing the substring "gtk" is encountered, resolves this problem.

So it is not a problem that exists now, but a problem of your versioning patch, that must distinguish between version symbol in libc, and an unversioned in gtk-X11 lib ?

But even then it should only a problem if something actually _calls_ fnmatch from GTK headers and the symbol is  bound?

this is as far as i drilled down into the source code, at that point deciding i had stumbled upon something ancient and evil.

Yes could be that this is some GTK compatibility wrapper for non *nix platforms like Windows.

[robert rozee]

So it is not a problem that exists now, but a problem of your versioning patch, that must distinguish between version symbol in libc, and an unversioned in gtk-X11 lib ?

nope - the problem very much exists now.

Code: Bash  [Select][+][-]

1. user@DH61BE:/$
2. user@DH61BE:/$ find -name "libgtk-x11-2.0.so*" 2>/dev/null
3. ./usr/lib/i386-linux-gnu/libgtk-x11-2.0.so.0.2400.33
4. ./usr/lib/i386-linux-gnu/libgtk-x11-2.0.so.0
5. ./usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so
6. ./usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0.2400.33
7. ./usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so.0
8. [ ... ]
9. user@DH61BE:/$

gives us the location of the GTK library. then searching it with readelf gives us...

Code: Bash  [Select][+][-]

1. user@DH61BE:/$
2. user@DH61BE:/$ readelf --dyn-syms -W /usr/lib/x86_64-linux-gnu/libgtk-x11-2.0.so | grep "fnmatch"
3. user@DH61BE:/$

... absolutely nothing! so where does the function fnmatch reside?! we know it isn't in glibc - if it were then versioning it to fnmatch@GLIBC_2.2.5 wouldn't have been a problem.

where is the code for fnmatch coming from???


cheers,
rob   :-)

[marcov]

Those headers are from the early GTK2 days, patched several times for later (2.8, 2.10)  versions. Also possibly it depends on the target if the symbol is there, and then it probably wouldn't be on glibc systems, as it is in libc there.

[robert rozee]

Those headers are from the early GTK2 days, patched several times for later (2.8, 2.10)  versions. Also possibly it depends on the target if the symbol is there, and then it probably wouldn't be on glibc systems, as it is in libc there.

that isn't answering the question - where is the code for fnmatch coming from??? 64-bit linux, FPC 3.2.2, Lazarus 2.2.6.

it clearly lands somewhere, and is being used for something. is it the case that nobody knows what code is being executed?


cheers,
rob   :-)

[marcov]

that isn't answering the question - where is the code for fnmatch coming from??? 64-bit linux, FPC 3.2.2, Lazarus 2.2.6.

it clearly lands somewhere, and is being used for something. is it the case that nobody knows what code is being executed?

Well, you can simply comment the declaration and see where the build fails yourself?

[robert rozee]

Well, you can simply comment the declaration and see where the build fails yourself?

the downside of making such modifications to the compiler myself (and here i include the build process as well) is that the compiler is used to compile itself. to date i've been proceeding with extreme care, wary that if i break it the subsequent 'unbreaking' might be less than trivial! FPC needs a working FPC to compile it.


consider: an innocent looking symbol (fnmatch) being associated with a library that does not contain it. yet nothing crashes, no errors are reported, but tinker with it and the compiler no longer functions. and nobody know what the symbol is there for, where exactly it is used...

... sounds rather like a subtle backdoor put in place years ago, waiting for someone to quietly drop in a library that handles fnmatch elsewhere in the operating system to exploit said backdoor.


i am still waiting for the FPC developers to come up with some sort of explanation, as to be honest i have very little understanding of the FPC source code and how it all works myself. most of what i have done thus far has been based upon educated guesses. whereas, the FPC developers are the experts whom i and others depend upon.


cheers,
rob   :-)

[marcov]

Never attribute to malice which can be explained by other reasons and all that. I gave your reasons (legacy, not correctly translating the original header ifdef structure), you chose to ignore it.

First, the compiler does not depend on GTK (or any other packages/ except the build tool), stronger even, it doesn't link to any dynamic library at all, which is easily verifiable, so there is no possible reason for paranoia there.

The only thing that uses gtk2/ is Lazarus when compiled from GTK. Even if you wanted to backdoor lazarus you'd do it in place that backdoors all lazarus, not just the GTK port.

So let's start with any form of proof that it is actually used. If you don't want to do a test compile, can you see in a compiled Lazarus binary that it uses the symbol with nm? 

[robert rozee]

First, the compiler does not depend on GTK (or any other packages/ except the build tool), stronger even, it doesn't link to any dynamic library at all, which is easily verifiable, so there is no possible reason for paranoia there.

if fnmatch is versioned to fnmatch@GLIBC_2.2.5 and FPC/Lazarus is rebuilt using the script provided by Fred vS, then lazbuild is no longer able to compile any GUI project. lazbuild is a command-line tool that does not invoke the lazarus IDE or any other GUI things, yet seems dependant upon the non-GLIBC version of fnmatch that is currently being associated in the source code with libgtk-x11-2.0.so .


can someone please explain (1) where this fnmatch is being used from, and (2) where it is landing? i really don't have the expertise to investigate much further. from my perspective, the source is opaque and unauditable.


cheers,
rob   :-)

[robert rozee]

Never attribute to malice which can be explained by other reasons and all that.

i think you are trying to quote Hanlon's razor, an adage or rule of thumb that states:

"Never attribute to malice that which is adequately explained by stupidity."

(https://en.wikipedia.org/wiki/Hanlon%27s_razor)

personally i would rather refrain from describing the authors of FPC as stupid, it is just not polite talk.


cheers,
rob   :-)

[DonAlfredo]

To my best of knowledge, GTK-fnmatch was used/introduced by Lazarus during the time when GLIBC-fnmatch did not handle the utf8 well.

[marcov]

First, the compiler does not depend on GTK (or any other packages/ except the build tool), stronger even, it doesn't link to any dynamic library at all, which is easily verifiable, so there is no possible reason for paranoia there.

if fnmatch is versioned to fnmatch@GLIBC_2.2.5 and FPC/Lazarus is rebuilt using the script provided by Fred vS, then lazbuild is no longer able to compile any GUI project.

And that is not what I asked. I asked that if you do a plain build, if you can then see a link to fnmatch in the generated binary?

I deliberately didn't want proofs using modified code to avoid discussion about if it is a problem with modification or abnormal instrumentation.

E.g. it could be that the versioning of symbols and/or its scripts somehow frustrates removing unused imports from the binary leading to an error on a symbol that is usually smartlinked out

But yes, the best solution would be now that FPC has been unicode safe for 8 years to clean this out in general. But I did quick greps in the ide/ components/ and lcl/ dirs and can't find any usage.  As it is widgetset dependent one would expect it to be in lcl/

[robert rozee]

And that is not what I asked. I asked that if you do a plain build, if you can then see a link to fnmatch in the generated binary

i'm not quite sure what you are asking. the function fnmatch is used by the compiler or one of the associated utilities that are regenerated by running Fred vS's build script. if i use Fred's script to build the toolchain using fnmatch@GLIBC_2.2.5, the resulting toolchain behaves thus:
1. if i build an arbitrary non-GUI project using lazbuild, it builds successfully.
2. if i try to build an arbitrary GUI project using lazbuild, it will fail to build.

put another way, the non-GLIBC fnmatch is used somewhere in the toolchain. strangely, it is used only in places associated with creating/compiling GUI components; non-GUI projects can still be compiled.


with a clean unadulterated install of lazarus/FPC, when an arbitrary GUI project is built using lazbuild the resulting ELF binary contains no references to fnmatch in the Dynamic Symbol Table.

however, it remains that even when building an unadulterated toolchain, pd_external has the symbol fnmatch passing through it, masked in such a way that it never resolves to libc.so.


cheers,
rob   :-)

[TRon]

@robert:
Being thick today, sorry for that.

Could you help me out and tell me exactly what readelf command (and its parameters) I need to run for every elf file located in my FPC and Lazarus directories ?

Right now, in my results, I see none that seem to match your criteria so am afraid that I am doing something wrong. All I got so far from the thread is symbol fnmatch.

[robert rozee]

hi Ron   :-)

Code: Bash  [Select][+][-]

1. readelf --dyn-syms -W filename | grep -i "fnmatch"

where filename is the ELF binary you want to check, and fnmatch is the symbol you are looking for. the -i switch tells grep to ignore case. if no matching lines are found there will be no output.

to test, try:

Code: Bash  [Select][+][-]

1. readelf --dyn-syms -W /usr/bin/bwbasic | grep -i "GLIBC"

to see all symbols containing GLIBC in the binary /use/bin/bwbasic.

btw, you may find some of the FPC binary files are stashed in /usr/bin and other places.


cheers,
rob   :-)

[TRon]

Code: Bash  [Select][+][-]

1. readelf --dyn-syms -W filename | grep -i "fnmatch"

Ah ok. perfect. In that case I indeed did something wrong earlier. Thank you for the help.

Ended up writing my own findelfsymbols utility, as that proofs to be a lot easier than all that bashing 

Thank you for the example (though I had to use bwrap because I do not seem to have bwbasic installed. Seem to provide similar results though (At least some output that I was able to test against)

btw, you may find some of the FPC binary files are stashed in /usr/bin and other places.

If that would to be the case then I did something horribly wrong 

For me it is (always something like) /home/apps/fpc/x.x.x and /home/apps/lazarus/x.x.x where x.x.x stands for the version number of the corresponding software.

Alas... I checked all files that have an elf header against having a symbol fnmatch (ignoring case in the readelf output) but unfortunately none of them was a match for me.

fwiw: Linux 64 gtk2, FPC 3.2.2 and Lazarus 3.0RC1.

Code: Text  [Select][+][-]

1. $ ./findelfsymbols -r "/home/apps/fpc/3.2.2/*" fnmatch
2. Processed 34456 files in 2366 directories, wasting 8.615 seconds. 7493 of those files were ELF files containing a total of 142 (dynamic) symbols for which 0 matched search criteria "fnmatch"
3.  
4. $ ./findelfsymbols -r "/home/apps/fpc/3.3.1/*" fnmatch
5. Processed 18699 files in 1831 directories, wasting 3.286 seconds. 2781 of those files were ELF files containing a total of 187 (dynamic) symbols for which 0 matched search criteria "fnmatch"
6.  
7. $ ./findelfsymbols -r "/home/apps/lazarus/2.2.4/*" fnmatch
8. Processed 18014 files in 1080 directories, wasting 8.148 seconds. 2055 of those files were ELF files containing a total of 5461 (dynamic) symbols for which 0 matched search criteria "fnmatch"
9.  
10. $ ./findelfsymbols -r "/home/apps/lazarus/2.2.6/*" fnmatch
11. Processed 17256 files in 1053 directories, wasting 6.975 seconds. 1540 of those files were ELF files containing a total of 5461 (dynamic) symbols for which 0 matched search criteria "fnmatch"
12.  
13. $ ./findelfsymbols -r "/home/apps/lazarus/2.3.0/*" fnmatch
14. Processed 20332 files in 1094 directories, wasting 1.993 seconds. 1600 of those files were ELF files containing a total of 5385 (dynamic) symbols for which 0 matched search criteria "fnmatch"
15.  
16. $ ./findelfsymbols -r  "/usr/lib/x86_64-linux-gnu/libgtk-x11*" fnmatch
17. Processed 7385 files in 835 directories, wasting 0.098 seconds. 1 of those files were ELF files containing a total of 5635 (dynamic) symbols for which 0 matched search criteria "fnmatch"
18.  
19.  
20. $ ./findelfsymbols -r "/home/apps/fpc/3.2.2/*" glibc
21. <list of matching symbols omitted>
22. Processed 34456 files in 2366 directories, wasting 8.595 seconds. 7493 of those files were ELF files containing a total of 142 (dynamic) symbols for which 111 matched search criteria "glibc"
23.  
24. $ ./findelfsymbols -r "/home/apps/fpc/3.3.1/*" glibc
25. <list of matching symbols omitted>
26. Processed 18699 files in 1831 directories, wasting 3.284 seconds. 2781 of those files were ELF files containing a total of 187 (dynamic) symbols for which 147 matched search criteria "glibc"
27.  
28. $ ./findelfsymbols -r "/home/apps/lazarus/2.2.4/*" glibc
29. <list of matching symbols omitted>
30. Processed 18014 files in 1080 directories, wasting 2.532 seconds. 2055 of those files were ELF files containing a total of 5461 (dynamic) symbols for which 234 matched search criteria "glibc"
31.  
32. $ ./findelfsymbols -r "/home/apps/lazarus/2.2.6/*" glibc
33. <list of matching symbols omitted>
34. Processed 17256 files in 1053 directories, wasting 1.936 seconds. 1540 of those files were ELF files containing a total of 5461 (dynamic) symbols for which 234 matched search criteria "glibc"
35.  
36. $ ./findelfsymbols -r "/home/apps/lazarus/2.3.0/*" glibc
37. <list of matching symbols omitted>
38. Processed 20332 files in 1094 directories, wasting 2.017 seconds. 1600 of those files were ELF files containing a total of 5385 (dynamic) symbols for which 218 matched search criteria "glibc"
39.  
40. $ ./findelfsymbols -r  "/usr/lib/x86_64-linux-gnu/libgtk-x11*" glibc
41. <list of matching symbols omitted>
42. Processed 7385 files in 835 directories, wasting 0.096 seconds. 1 of those files were ELF files containing a total of 5635 (dynamic) symbols for which 75 matched search criteria "glibc"
43.