Lazarus

Programming => General => Topic started by: TurboRascal on January 06, 2012, 10:41:15 pm

Title: Static linking
Post by: TurboRascal on January 06, 2012, 10:41:15 pm
Somewhere on the forum, I've seen a good answer to a wrong question :) So, not to see this nice informations get forgotten, I'd like to ask more about this - how does static linking work in FPC/Lazarus and where could some more information be found about it. Some of the questions would for example be, what are those other libs that get loaded here, where are those found and why are they needed...

Here goes the quote I'm talking about:

{$IFDEF FPC}
{$MODE Delphi}
{$ENDIF}

interface

uses
{$IFDEF MSWINDOWS}
  Windows,
  Messages,
{$ENDIF}
{$IFDEF LINUX} // for Kylix
  Libc,
  Types,
{$ENDIF}
  Classes,
{$IFDEF USETYPEINFO}
  // this pure pascal version must handle the 64-bits ordinal values and
  // the special layout of the underlying compiler (mostly FPC)
  typinfo,
{$ENDIF}
{$IFNDEF LVCL}
  Contnrs, // for TObjectList
{$ENDIF}
  SysUtils;
{$IFNDEF FPC}
{$L 'OBJ\sqlite3se.obj'}  //for Delphi
{$ELSE}
{$IFDEF MSWINDOWS}
{$linklib libsqlite3se.a}
{$linklib libmsvcrt.a}
{$linklib libkernel32.a}
{$linklib libgcc.a}
{$ELSE}
{$linklib libsqlite3selinux.a}
{$Linklib c}
{$linklib libgcc.a}
{$ENDIF}
{$ENDIF}

and add -Xt in fpc.cfg.
Title: Re: Static linking
Post by: ludob on January 07, 2012, 11:42:22 am
Quote
how does static linking work in FPC/Lazarus and where could some more information be found about it. Some of the questions would for example be, what are those other libs that get loaded here, where are those found and why are they needed...
http://en.wikipedia.org/wiki/Static_library explains what static linking is. FPC is not an exception. Think about a static library as a collection of .o and .ppu files. There are different library formats but the fpc linker only accepts the gnu format. On windows this means that static libraries produced by MS tools can not be used. Mingw or cygwin ports of the gnu binutils need to be used.
To comment on the snippet, I'll do this for windows because I went through the exercise of linking statically sqlite. I'll actually go through the complete steps, for posterity  ;)
- Get the sqlite source from http://www.sqlite.org/download.html (amalgamation: all sources in one file).
The source is written in C and needs to be compiled to a static library using a gnu tool chain. Personally I use wxDev-C++ (http://wxdsgn.sourceforge.net/). This a complete IDE for developing with wxWidgets and probably an overkill for this purpose. The advantage is that the installation is straightforward and that everything needed is installed and, when compiling from the IDE, there is no messing with include/library paths. It is based on the mingw port of the binutils which doesn't require additional run time dlls as opposed to Cygwin.
- in Dev-C++, create a new project, select static library, remove all the files added automatically from the project and add sqlite3.c, save project as sqlite3.dev in dir "whatever", compile project. In whatever\objects\MingW you'll find sqlite3.o. Easy enough.
- sqlite3 being written in C, and C having its own run-time (equivalent to fpc RTL) we'll probably need to pull in a few other libraries. If the program uses windows api's then we'll need to add these libraries also. You could go through the 128.000 lines of code in sqlite3.c to figure out the external dependencies but it is easier to be lazy and have the fpc linker do this for us. Create a new pascal program that uses the newly created library:
Code: [Select]
program TestSqlite3;
{$L sqlite3.o}
function sqlite3_libversion(): pchar; cdecl;external;
begin
  writeln(sqlite3_libversion);
end.
and compile. Make sure the path to sqlite3.o is added or simply copy it to the program directory. Linker stops at 50 undefined symbols. No panic. Let's start with what we recognise.
- InterlockedCompareExchange looks windows api. According msdn defined in kernel32. In the Dev-Cpp\lib directory you find a libkernel32.a. Add the path to it or copy it to the program directory. Add {$linklib libkernel32.a} to your program. Compile. Undefined reduced to 14 !!
- memcpy sounds like the c run time. That is msvcrt on windows. libmsvcrt.a is in Dev-Cpp\lib. Add {$linklib libmsvcrt.a}. Compile. Only 4 undefined symbols remain.
- a quick google on "Undefined symbol: ___udivdi3" shows that this belongs to the gcc run time library. In Dev-Cpp/lib/gcc/mingw32/3.4.2/ there is  libgcc.a. Add {$linklib libgcc.a} to the program. Compile. Bingo! Run program and it'll print the sqlite3 version :D If you have lost track, the complete program is now
Code: [Select]
program project1;
{$L sqlite3.o}
{$linklib libkernel32.a}
{$linklib libmsvcrt.a}
{$linklib libgcc.a}
function sqlite3_libversion(): pchar; cdecl;external;
begin
  writeln(sqlite3_libversion);
end.
- Now to use the statically linked version with sqldb, there is still some work to do. In the test program we declared function sqlite3_libversion() manually and Sqlite3conn is "hard-wired" to use the dynamic library. The way how i solved this is to make a copy of sqlite3conn.pp in my project directory, change uses sqlite3dyn to sqlite3static, create a new sqlite3static from sqlite3.inc (add "unit sqlite3static;" remove uses dynlibs, add the {$linklib xxx},{$L sqlite3.o} lines, change all "external Sqlite3Lib;" to "external;" and remove the contents of the implementation section except for the function sqlite3_version. If interested I can attach this file knowing that it isn't based on the latest svn version of sqlite3.inc.

Hope this answers you questions.
Title: Re: Static linking
Post by: BigChimp on January 07, 2012, 11:49:00 am
Very clear explanation, thanks a lot.

One fairly off topic question (I suspect the answer but just wanted to make sure I get it): why would there be a dependency on GCC - shouldn't MinGW compile against the Microsoft C runtime? Or is that because GCC provides some functions not provided by MSVCRT that are called by sqlite?
Title: Re: Static linking
Post by: ludob on January 07, 2012, 12:12:29 pm
One fairly off topic question (I suspect the answer but just wanted to make sure I get it): why would there be a dependency on GCC - shouldn't MinGW compile against the Microsoft C runtime? Or is that because GCC provides some functions not provided by MSVCRT that are called by sqlite?
The GCC compiler uses the divdi3 functions for 64-bit multiplication and division which are not in MSVCRT. Sqlite isn't calling these functions directly but uses some 64-arithmetic.
Title: Re: Static linking
Post by: marcov on January 07, 2012, 02:10:54 pm
Pass   -k-static or -k--static and see what happens?
Title: Re: Static linking
Post by: marcov on January 07, 2012, 02:12:01 pm
The GCC compiler uses the divdi3 functions for 64-bit multiplication and division which are not in MSVCRT. Sqlite isn't calling these functions directly but uses some 64-arithmetic.

Such helpers are in "libgcc"  The reason is that these functions are relatively small and much used, and avoiding DLL call overhead (how little it is) can be worthwhile.
Title: Re: Static linking
Post by: ludob on January 07, 2012, 02:49:44 pm
Quote
The reason is that these functions are relatively small and much used, and avoiding DLL call overhead (how little it is) can be worthwhile.
The goal of the exercise was to get rid of the sqlite3.dll and have an "all-in" executable. Substituting a dependency on sqlite3.dll with a dependency on libgcc_s_dw_2-1.dll (or whatever the used gcc version requires) wouldn't be too smart neither.
Title: Re: Static linking
Post by: TurboRascal on January 07, 2012, 07:08:09 pm
@ludob: thank you for your answer, you made the effort to write more than was required just by my question, a mini-guide from basics up ;)

I believe that post would made a worthy wiki article, so if it's not much trouble, you might create a page there. If you wish, I can do it for you...
Title: Re: Static linking
Post by: ludob on January 07, 2012, 08:26:28 pm
Quote
I believe that post would made a worthy wiki article, so if it's not much trouble, you might create a page there. If you wish, I can do it for you...
I'll take up your offer. If you think you can turn this into a wiki article, please go ahead. The way I wrote the reply was more a explain by example than a wiki page on static linking in general or one on the static linking of sqlite.
Title: Re: Static linking
Post by: chrnobel on January 07, 2012, 09:29:44 pm
ludob

Thank you very much for the explanation, for me it clarified at lot.

Just at thought, would it be possible to automate this in some way, so that the FPC installed could automate this process, and thereby giving a possibility (also for the not so experienced user) to build programs where the static linking is done upfront?

It also leads me to an other question, maybe off topic, why is there a requirement for having a special library/client for MySQL - I mean all the "clever" work is done one the server, and (as I understand it) all the client (simplified) has to do is to send SQL commands over port 3306, and then receive the result and pass it to the program.
I know that the communication is encrypted, but anyway?

If it was possible to write the client purely in Pascal, and having it to be a part of FPC would be an huge advantage, not only for making programs simpler, but also for having an easy way to say goodbye to Oracle, and using MariaDB instead.
Title: Re: Static linking
Post by: marcov on January 07, 2012, 10:22:28 pm
Quote
The reason is that these functions are relatively small and much used, and avoiding DLL call overhead (how little it is) can be worthwhile.
The goal of the exercise was to get rid of the sqlite3.dll and have an "all-in" executable. Substituting a dependency on sqlite3.dll with a dependency on libgcc_s_dw_2-1.dll (or whatever the used gcc version requires) wouldn't be too smart neither.

(1) You need .a's for all involved libs. Avoid the corresponding dlls in the path.
(2) if you want a certain library (e.g. libgcc) for all cases where you link a certain other lib, use -XLA.   So in the example  -XLAsqlite=sqlite,gcc   
     Don't worry, duplicates are removed.
(3) Some libraries like libgcc need a special place in the link.res file.  Using -XLO you can manipulate this.
(4) I did this kind of stuff before the internal linker existed, and forced the external linker (-Xe) using -k-static

Title: Re: Static linking
Post by: ludob on January 08, 2012, 12:06:49 pm
Just at thought, would it be possible to automate this in some way, so that the FPC installed could automate this process, and thereby giving a possibility (also for the not so experienced user) to build programs where the static linking is done upfront?
The purpose of the write-up was also to demonstrate the process of finding the dependent libraries. Other than parsing all libraries on your system and make a kind of database with exported functions, I don't see how to facilitate this. To automate is even more difficult. What happens if the same function is exported by different libraries. It could be just a name collision or 2 different, incompatible, versions of the library. Both require some knowledge of the libraries to solve the ambiguity.
Static linking upfront: you mean creating a big library containing everything needed. What would be the advantage? It still requires to figure out all dependencies and there would be as many versions as there versions of the contained libraries (think also 32-64bit).  I don't see that advantage over creating a unit that contains a series of {$linklib xxx} lines.
It also leads me to an other question, maybe off topic, why is there a requirement for having a special library/client for MySQL - I mean all the "clever" work is done one the server, and (as I understand it) all the client (simplified) has to do is to send SQL commands over port 3306, and then receive the result and pass it to the program.
I know that the communication is encrypted, but anyway?

If it was possible to write the client purely in Pascal, and having it to be a part of FPC would be an huge advantage, not only for making programs simpler, but also for having an easy way to say goodbye to Oracle, and using MariaDB instead.
The db client is doing a lot more.:
- communication: databases use a proprietary network protocol.
- session management
- result set and metadata handling
- data binding, both for parameters and result sets.   
- ...
I invite you to take a look at the mysql client library source just to get an idea of the amount of code involved. Then look at the mysql bug tracker and ask yourself if you want to duplicate all this in pascal or just use the C library and leave the low level stuff to the experts.
To be honest, I don't see the advantage of rewriting a db client library in pascal. Apart from the big effort, you would be running constantly after the facts because of the proprietary protocol. We run even after the facts in creating the bindings for the existing mysql client libraries (latest in fpc trunk is 5.1)

OOP is all about re-using software. That includes the use of existing libraries.

Title: Re: Static linking
Post by: marcov on January 08, 2012, 06:16:45 pm
- communication: databases use a proprietary network protocol.

(And even when not, the wire protocol is often separately versioned, and the client library might transform requests to fit them to the wire protocol)

Quote
To be honest, I don't see the advantage of rewriting a db client library in pascal. Apart from the big effort, you would be running constantly after the facts because of the proprietary protocol. We run even after the facts in creating the bindings for the existing mysql client libraries (latest in fpc trunk is 5.1)

Mysql is a special problem, due to its licensing and (deliberate!) versioning problems, and because it is used a lot about newbies that haven't ran into that.

This causes
1. more problems than for the other databases
2. the bulk of the people that are capable of maintenance to avoid it in the first place.

That leaves a very small developer base that is knowledgable enough but is forced to use it to handle the bulk of the problems.

In short: if you have _any_ kind of choice, don't use mysql.

Quote
OOP is all about re-using software. That includes the use of existing libraries.

Every software paradigm strives to do so  on paper. OOP is no exception :-)
Title: Re: Static linking
Post by: Shebuka on January 17, 2012, 05:23:23 pm
Hi, what happens if i have a unitA that has {$linklib mylibrary} and then i have a unitB that is created at some point by unitA and that also has {$linklib mylibrary}? Is mylibrary initialized 2 times?

Because i having, at random, some strange behavior and try to exclude as much cases as possible.
Title: Re: Static linking
Post by: marcov on January 17, 2012, 08:57:49 pm
Hi, what happens if i have a unitA that has {$linklib mylibrary} and then i have a unitB that is created at some point by unitA and that also has {$linklib mylibrary}? Is mylibrary initialized 2 times?

No.

Quote
Because i having, at random, some strange behavior and try to exclude as much cases as possible.

Start searching for mistakes in your code. It is statistically the most likely option
Title: Re: Static linking
Post by: herux on January 18, 2012, 03:06:08 am
Hi all

I really like this thread. contains a lot of expert knowledge
 :)
Title: Re: Static linking
Post by: eMko on October 13, 2012, 10:48:49 pm
Good one :) . Shame I could not find it by google just a few days ago  :'(

I had to do an extension to Firefox (also to Chrome and IE - but that's different story) - toolbar which displays some data from university information system (unread mails, exam dates, project deadlines...). Since our information system is proprietary (we actively develop it and sell it to universities), we didn't like the possibility to write the communication code in javascript - XPI file is just renamed zip (all metadata are in files in the archive), so anyone can unzip the firefox extensions and read the sources. FF extensions are able to call external shared libraries packed into the xpi, but there are problems with dependencies - the dll/so should be self-contained or you can get random funny errors.

If I had known how to statically link openssl library into pascal dll, I wouldn't have written that library in C++, the language I totally hate. (Ok, situation is not so bad with boost library, but still... :-/ ).
Title: Re: Static linking
Post by: TurboRascal on October 15, 2012, 02:42:51 pm
To take this opportunity of being reminded of this thread, just to check if you, Ludo, added that info to the wiki? If not, I would do it...
Title: Re: Static linking
Post by: ludob on October 15, 2012, 05:08:03 pm
No, I didn't  ;)
Title: Re: Static linking
Post by: TurboRascal on October 15, 2012, 09:26:35 pm
Ok then I'll do it... unless you do it before ;)
Title: Re: Static linking
Post by: Igor Kokarev on November 20, 2018, 07:41:38 pm
Hi,

I'm trying to link statically mpg123 lib in 64-bit, Windows.

I already linked libkernel32.a, libm.a, libmsvcrt.a, libgcc.a

But I still see several errors while linking:

Undefined symbol: sqrt (cos, sin, pow, log, lseek64)

What .a files from MinGW64 are responsible for these functions?
Title: Re: Static linking
Post by: DonAlfredo on November 20, 2018, 08:21:34 pm
You can add this file into your project:

https://github.com/BeRo1985/pasvulkan/blob/master/src/PasVulkan.StaticLinking.pas

It will provide the necessary functions for static linking.
Title: Re: Static linking
Post by: Igor Kokarev on November 21, 2018, 11:56:07 am
Hi,

Thanks for your advice!

Regrettably this .pas code doesn't work.

I get several linking errors. For example:

undefined symbol: fseek

I found fseek(), but it's linked from some lib?

function fseek(s:pointer;offset:int64;whence:longint):longint; cdecl; external name 'fseek';
Title: Re: Static linking
Post by: DonAlfredo on November 21, 2018, 01:23:07 pm
Well, I have never used the file from Bero. So I do not have any real world experience with it. But I do a lot of static linking with sqlite3, used by the mORMot.

Perhaps you can have a look at it:
https://github.com/synopse/mORMot/blob/master/SynSQLite3Static.pas

This surely works on many systems.
Title: Re: Static linking
Post by: Thaddy on November 21, 2018, 01:34:27 pm
It reads if he wants to link the sourcecode (pas).......
Title: Re: Static linking
Post by: Igor Kokarev on November 21, 2018, 04:08:26 pm
I successfully linked statically mpg123.

I compiled mpg123 lib in MSYS2/MinGW64

then I copied a series of .a and .o files from mpg123 and from MingGW64.

Decoding of MP3 works fine.

Currently I'm trying to optimize a number of used .a and o. files - probably I can implement Pascal version of some C functions.

And also I'm looking for a way to disable debug messages in console from mpg123.

I didn't know how to implement Pascal version of mingw_vfprintf() and lseek64() so I added a dummy functions. I know it's not correct.

Code: Pascal  [Select][+][-]
  1. {$linklib Libs64\libmsvcrt.a}
  2. {$linklib Libs64\libkernel32.a}
  3. {$linklib Libs64\libgcc.a}
  4. {$linklib Libs64\libshlwapi.a}
  5.  
  6. {$L Libs64\compat.o}
  7. {$L Libs64\compat_str.o}
  8.  
  9. {$L Libs64\dct36_avx.o}
  10. {$L Libs64\dct36_x86_64.o}
  11. {$L Libs64\dct64.o}
  12. {$L Libs64\dct64_avx.o}
  13. {$L Libs64\dct64_avx_float.o}
  14. {$L Libs64\dct64_x86_64.o}
  15. {$L Libs64\dct64_x86_64_float.o}
  16. {$L Libs64\dither.o}
  17. {$L Libs64\equalizer.o}
  18. {$L Libs64\feature.o}
  19. {$L Libs64\format.o}
  20. {$L Libs64\frame.o}
  21. {$L Libs64\getcpuflags_x86_64.o}
  22. {$L Libs64\icy.o}
  23. {$L Libs64\icy2utf8.o}
  24. {$L Libs64\id3.o}
  25. {$L Libs64\index.o}
  26. {$L Libs64\layer1.o}
  27. {$L Libs64\layer2.o}
  28. {$L Libs64\layer3.o}
  29. {$L Libs64\lfs_alias.o}
  30. {$L Libs64\lfs_wrap.o}
  31. {$L Libs64\libmpg123.o}
  32. {$L Libs64\ntom.o}
  33. {$L Libs64\optimize.o}
  34. {$L Libs64\parse.o}
  35. {$L Libs64\readers.o}
  36. {$L Libs64\stringbuf.o}
  37. {$L Libs64\synth.o}
  38. {$L Libs64\synth_8bit.o}
  39. {$L Libs64\synth_real.o}
  40. {$L Libs64\synth_s32.o}
  41. {$L Libs64\synth_stereo_avx.o}
  42. {$L Libs64\synth_stereo_avx_float.o}
  43. {$L Libs64\synth_stereo_avx_s32.o}
  44. {$L Libs64\synth_stereo_x86_64.o}
  45. {$L Libs64\synth_stereo_x86_64_float.o}
  46. {$L Libs64\synth_stereo_x86_64_s32.o}
  47. {$L Libs64\synth_x86_64.o}
  48. {$L Libs64\synth_x86_64_float.o}
  49. {$L Libs64\synth_x86_64_s32.o}
  50. {$L Libs64\tabinit.o}
  51.  

Code: Pascal  [Select][+][-]
  1. unit StaticLinkingUtils;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7. uses
  8.   Math;
  9.  
  10. function _log(x: double): double; cdecl;
  11. function _sqrt(x: double): double; cdecl;
  12. function _sin(x: double): double; cdecl;
  13. function _cos(x: double): double; cdecl;
  14. function _pow(x,y: double): double; cdecl;
  15. function _lseek64(ahandle: Pointer; offset: Integer; whence: Integer): Integer; cdecl;
  16. function int_mingw_vfprintf(): Integer; cdecl;
  17.  
  18.  
  19. implementation
  20.  
  21.  
  22. function _log(x: double): double; cdecl; [alias: 'log'];
  23. begin
  24.   result := 0; // ln(x);
  25. end;
  26.  
  27. function _sqrt(x: double): double; cdecl; [alias: 'sqrt'];
  28. begin
  29.   result := sqrt(x);
  30. end;
  31.  
  32. function _sin(x: double): double; cdecl; [alias: 'sin'];
  33. begin
  34.   result := sin(x);
  35. end;
  36.  
  37. function _cos(x: double): double; cdecl; [alias: 'cos'];
  38. begin
  39.   result := cos(x);
  40. end;
  41.  
  42. function _pow(x,y: double): double; cdecl; [alias: 'pow'];
  43. begin
  44.   result := power(x,y);
  45. end;
  46.  
  47. function _lseek64(ahandle: Pointer; offset: Integer; whence: Integer): Integer; cdecl; [alias: 'lseek64'];
  48. begin
  49.   Result:=0;
  50. end;
  51.  
  52. function int_mingw_vfprintf(): Integer; cdecl; [alias: '__mingw_vfprintf'];
  53. begin
  54.   Result:=0;
  55. end;
  56.  
  57. end.
  58.  
Title: Re: Static linking
Post by: Igor Kokarev on November 24, 2018, 05:09:45 pm
mpg123 with static linking freezes frequently on half of MP3 files.

Same code with compiled (by me) DLL works fine.

Any advices?
Title: Re: Static linking
Post by: Igor Kokarev on November 26, 2018, 06:47:37 pm
The problem was in a conflict of functions for mpg123 and zlib. Some global cdecl functions were linked from .a libs and same functions were implemented in Pascal code. So it didn't worked stable.

I have to be careful when use global cdecl functions in different statically linked C libraries.
Title: Re: Static linking
Post by: Igor Kokarev on December 04, 2018, 08:02:19 am
Can you help me implement in Pascal __ms_vsnprintf() for static linking:

From stdio.h in MinGW64:

Code: Pascal  [Select][+][-]
  1. # undef vsnprintf
  2.   __attribute__((__format__ (ms_printf, 3, 0))) __MINGW_ATTRIB_NONNULL(3)
  3.   int __cdecl __ms_vsnprintf(char * __restrict__ d,size_t n,const char * __restrict__ format,va_list arg)
  4.     __MINGW_ATTRIB_DEPRECATED_MSVC2005 __MINGW_ATTRIB_DEPRECATED_SEC_WARN;
  5.  
  6.   __mingw_ovr
  7.   __attribute__((__format__ (ms_printf, 3, 0))) __MINGW_ATTRIB_NONNULL(3)
  8.   int vsnprintf (char * __restrict__ __stream, size_t __n, const char * __restrict__ __format, va_list __local_argv)
  9.   {
  10.     return __ms_vsnprintf (__stream, __n, __format, __local_argv);
  11.   }
  12.  
  13.   __attribute__((__format__ (ms_printf, 3, 4))) __MINGW_ATTRIB_NONNULL(3)
  14.   int __cdecl __ms_snprintf(char * __restrict__ s, size_t n, const char * __restrict__  format, ...);
  15.  

Title: Re: Static linking
Post by: DonAlfredo on December 04, 2018, 08:53:41 am
Again:
https://github.com/BeRo1985/pasvulkan/blob/master/src/PasVulkan.StaticLinking.pas
Title: Re: Static linking
Post by: Igor Kokarev on December 04, 2018, 02:06:23 pm
Thank you very much!
TinyPortal © 2005-2018