[SOLVED] Statically linking multiple packages that use the C runtime

[Khrys]

System information:
OS (Architecture): Windows 10 (AMD64/x86-64)
Lazarus Version: 2.0.12
FPC Version: 3.2.0
C Compiler: GCC 10.3.0 (TDM-GCC-64)

Hello everyone,

this is what I'm trying to achieve:

• Create an IDE-installable Lazarus package that statically links PCRE2; usable on Windows
• Create an IDE-installable Lazarus package that statically links SQLite3; usable on Windows
• Have the ability to independently use either one or both of these packages at once (without relying on glue/dummy packages)

In isolation I've managed to reach the first two goals: I've created two packages that bundle C source code, which (given a working installation of GCC) compile a static version of the respective library via a custom makefile. The Windows targets require some extra hand-holding; notably, instead of a simple  {$linklib c}  they require the GNU-style static libraries  libgcc.a,  libmsvcrt.a  and  libkernel32.a. These can be located automatically via GCC by using eg.  "gcc --print-file-name=libgcc.a", so besides conditionally  {$linklib}ing these, the makefile is able to handle this. The end result is a source-only package that compiles and links just fine on both Linux and Windows, plus it can be installed in (that is, linked into) Lazarus.

Now, this is where the complications begin (all on Windows, because of course it had to be Windows):
On x86-64, while linking PCRE2 I was initially hit with internal compiler error 200603061 ("Only debug sections are allowed to have relocs pointing to unused sections") which stumped me for a while. The first thing I tried was switching to the external linker (-Xe), which solved this issue (and would solve most other issues I'm going to bring up), but then I found out that Lazarus itself won't link with -Xe, making that option a dealbreaker since non-installable packages propagate that limitation to any downstream components. In other words, it seems that for package interoperability only FPC's internal linker may be used.

(I eventually worked around the internal error by stumbling upon GCC's  mcmodel=small  option, which gets rid of the relocations that the internal linker can't handle. So for anyone reading this in the future: If you have problems trying to use FPC's internal linker to link object files stemming from multiple co-dependent translation units, try using  -mcmodel=small)

Ok, so with two individual working packages and knowing that  -Xe  is a no-no, I tried using both packages in the same project, and hit the roadblock I'm still stuck at today: No more than one such package will successfully link at a time. By "such package" I mean a package that bundles a C library like so (also setting  -Fl[...] etc.):

Code: Pascal  [Select][+][-]

1. unit Something_Interface;
2.  
3. implementation
4.  
5. procedure something_foo(); cdecl; external;
6.  
7. implementation
8.  
9. {$linklib something} // Contains definition of something_foo
10. {$if defined(WINDOWS)}
11.   {$linklib gcc}
12.   {$linklib msvcrt}
13.   {$linklib kernel32}
14. {$else}
15.   {$linklib c}
16. {$endif}
17.  
18. end.

The first issue I stumbled upon when I tried using both packages at once was that certain symbols from libgcc, libmsvcrt and libkernel32 remained undefined (_strncmp,  __divdi3,  _imp_CloseHandle@4 etc.) remained undefined. Looking at the extended debug output (-va), I found out why:

Code: [Select]

Debug: [2.001] Opening library [...]/libpcre2.a
Debug: [2.002] Opening library [...]/libgcc.a
Debug: [2.003] Opening library [...]/libmsvcrt.a
Debug: [2.004] Opening library [...]/libkernel32.a
Debug: [2.005] Opening library [...]/libsqlite3.a
A link order issue! After libpcre2 resolves all the symbols it needs, the remaining symbols are discarded. Unfortunately, this happends before libsqlite3 can resolve any additional symbols it needs from those libraries. (Please correct me if I'm wrong, but I think that is what's happening)
To me it seemed like FPC didn't bother opening the same library more than once, so I tried to trick the compiler by adding no-op modifications to the  {$linklib}  file names like  {$linklib .//./libgcc.a}  instead of  {$linklib libgcc.a}  - which actually worked, but only when doing a clean rebuild (to be precise, when no  .ppu  was present for the header translation units). So I tried using the advanced linker options (-XLO,  -XLA,  -XLD; the first two seemed promising), but didn't notice any effects, so I dug into the FPC source (release_3_2_0) and found that  -XLO  and  -XLA  are only handled on Linux, Android and BSD (no other target calls  ExpandAndApplyOrder).
I made a final attempt with the  -k  option, but as I soon found out, it unfortunately only has an effect on Windows targets (uses  ParaLinkOptions) when using the external linker... which I cannot use (as detailed above).

So now my only hope left is help from a true expert - let's hope one stumbles upon this post... 

[Thaddy]

You should be able to combine {$if defined xxx}with {$if declared(xxx.feature)} to exclude duplicate linking. Just a though.

[DonAlfredo]

The internal compiler error 200603061 can (also) be resolved by using FPC trunk/main.

Static linking is not easy, but doable. Few examples.

https://github.com/LongDirtyAnimAlf/km003c/blob/main/PC/Common/libusb/src/libusb.pas
https://github.com/synopse/mORMot2/blob/master/src/db/mormot.db.raw.sqlite3.static.pas
https://github.com/BeRo1985/pasvulkan/blob/master/src/PasVulkan.StaticLinking.pas

[Khrys]

Hi everyone,

thanks for the suggestions. After dissecting the compiler sources some more, I finally found a workaround: mark the libraries that are supposed to come last as  shared  while marking all other libraries as  static:

Code: Pascal  [Select][+][-]

1. unit Something_Interface;
2.  
3. implementation
4.  
5. procedure something_foo(); cdecl; external;
6.  
7. implementation
8.  
9. {$linklib something, static}  // <------| Link first -> static
10. {$if defined(WINDOWS)}
11.   {$linklib gcc, shared}      // <------|
12.   {$linklib msvcrt, shared}   // <------| Link last -> shared
13.   {$linklib kernel32, shared} // <------|
14. {$else}
15.   {$linklib c}
16. {$endif}
17.  
18. end.

Turns out that the internal linker (on Windows) makes no distinction between static and shared libraries apart from deferring file name resolution for the latter, but - and this is crucial - it first goes through all static libraries before moving on to shared libraries (release_3_2_0: link.pas[974..1000]; AddSharedAsStatic = True  on Windows):

Code: Pascal  [Select][+][-]

1. procedure TInternalLinker.ScriptAddSourceStatements(AddSharedAsStatic:boolean);
2.   var
3.     s,s2: TCmdStr;
4.   begin
5.     while not ObjectFiles.Empty do
6.       begin
7.         s:=ObjectFiles.GetFirst;
8.         if s<>'' then
9.           LinkScript.Concat('READOBJECT '+MaybeQuoted(s));
10.       end;
11.     while not StaticLibFiles.Empty do
12.       begin
13.         s:=StaticLibFiles.GetFirst;
14.         if s<>'' then
15.           LinkScript.Concat('READSTATICLIBRARY '+MaybeQuoted(s));
16.       end;
17.     if not AddSharedAsStatic then
18.       exit;
19.     while not SharedLibFiles.Empty do
20.       begin
21.         S:=SharedLibFiles.GetFirst;
22.         if FindLibraryFile(s,target_info.staticClibprefix,target_info.staticClibext,s2) then
23.           LinkScript.Concat('READSTATICLIBRARY '+MaybeQuoted(s2))
24.         else
25.           Comment(V_Error,'Import library not found for '+S);
26.       end;
27.   end;

This effectively allows manually controlling the link order of 2 library groups, which is enough for my intents and purposes. It's not an elegant solution since it relies on an implementation detail of the compiler that may or may not change in the future, but oh well...

[marcov]

When building a mingw based IDE I had similar problems back then in 2010 too.

I solved the problems then, by including the same static libraries multiple times with $linklib, once before and once after the other library with dependencies.

[Thaddy]

The order in which you link is very strict.
I didn't know about Marco's work-around, though.

A good example is the static SQLite linking in mORMot.

[Khrys]

From what I've seen elsewhere online, Marco's solution of specifying libraries multiple times actually seems to be the recommended way to do it. GCC even has  -(  and  -)  options that allow groups of archives with circular dependencies to be searched repeatedly until all undefined symbols are resolved (i.e.  libalpha  depends on  libbeta, but  libbeta  is weirdly intertwined with  libgamma  and  libdelta):

Code: Text  [Select][+][-]

1. gcc -o prog ... -lalpha -( -lbeta -lgamma -ldelta -)

I like to think of references to dependencies/external symbols as shopping lists and the dependencies/libraries themselves as stores: first get a shopping list and only then go to the store, because otherwise you'll forget a bunch of stuff...

But in any case, my problem here was really caused by the way Lazarus packages work in combination with my stubbornness on how I want my packages to interact; I could've just put all  {$linklib}  directives in a single file to sort out the link order myself, but then the packages wouldn't be as self-contained and plug-and-play as I'd want them to be.

(None of my problems here popped up in the Linux target though; everything just works there... I guess it's time to blame Windows again )

[Khrys]

Reviving this thread to document a better solution I accidentally stumbled upon: linker scripts!

Turns out that FPC's internal linker supports (a subset of)  ld  linker script commands:

Code: Text  [Select][+][-]

1. GROUP
2. (
3.   libalpha.a
4.   libbeta.a
5.   libgamma.a
6.   libdelta.a
7. )

Putting the above script in a file called  libfoo.a,  one can simply use  {$linklib foo}  without having to worry about the ordering!
From the GNU docs:

GROUP(file, file, ...)
GROUP(file file ...)

    The GROUP command is like INPUT, except that the named files should all be archives, and they are searched repeatedly until no new undefined references are created. See the description of '-(' in Command-line Options.

[marcov]

The whole idea of the -XLA,XLO commands was to avoid manual intervention in the linking process (including per application linker scripts). I primarily wanted to link new builds of libraries (usually libgdb) with existing release binaries and/or sources. (So no FPC source modifications).   Since various builds might have different border parameters (with and without python, as cygwin or mingw build or nearly entirely standalone), the XLA and XLO stuff allowed me to manipulate the compiler to issue the correct linkorder to the linker (which was then still external, also on Windows). There is some documentation about those in the buildfaq.

But if you have control of your sources, the smartest is to simply remove all linklibs from relevant units, and list them in your main program, and play a bit to get the duplication right. That would be easier than to mess with custom scripts.

Another idea that I had, but never conclusively tested was to have the dependent .a's  (msvcrt, kernel32 etc)  once before the first uses in the main program, and once after, and let the units only list the main dependencies (pcre, sqlite). The basic idea is that the bit above the USES in the main program is always parsed first, and the bit after last.

But of course the correct way would be to work on the internal linker and make it support those relocations and the XLA/XLO commands.

[Khrys]

But of course the correct way would be to work on the internal linker and make it support those relocations and the XLA/XLO commands.

I absolutely agree, with one small addition: support for grouping via the command line, like  ld's  -( -lalpha -lbeta -).

The reason I had to revisit this topic was because while trying to link in yet another C library, I found out that there are circular dependencies even among  mingw-w64's  C libraries (!).
I did a very crude worst-case analysis (assuming that absolutely every function is required) and came up with the following table listing the number of dependencies between archives:

Code: Text  [Select][+][-]

1.             SUBJECT | libmingwex  libmingw32  libgcc  libmsvcrt   libkernel32
2. DEPENDENCY          |
3. --------------------+--------------------------------------------------------
4. libmingwex          | -           -           1       3           -
5. libmingw32          | 2           -           -       -           -
6. libgcc              | 8           3           -       1           -
7. libmsvcrt           | 105         14          9       -           -
8. libkernel32         | 29          8           21      9           -

There wouldn't be any issues if all dependencies were concentrated on one side of the diagonal, but the following ones prevent the dependency graph from being truly acyclic:

libgcc → libmingwex:     __mingw_vfprintf
libmsvcrt → libgcc:      ___chkstk_ms
libmsvcrt → libmingwex:  __ms__vsnprintf, strnlen, wcsnlen

In practice this won't necessarily cause problems, since the affected functions are common enough to most likely be required by downstream libraries anyway, but throwing  libpthread  into the mix broke the camel's back for me (due to its circular dependency with  libgcc).

Knowing that a project could be rendered uncompilable just by calling a function that relies on heretofore unneeded circularly dependent code is not just a spooky thought, but has in fact happened to me before... hence why there absolutely has to be a way to resolve circular dependencies.