Recent

Author Topic: Static linking  (Read 22842 times)

TurboRascal

  • Hero Member
  • *****
  • Posts: 672
  • "Good sysadmin. Bad programmer."™
Static linking
« on: January 06, 2012, 10:41:15 pm »
Somewhere on the forum, I've seen a good answer to a wrong question :) So, not to see this nice informations get forgotten, I'd like to ask more about this - how does static linking work in FPC/Lazarus and where could some more information be found about it. Some of the questions would for example be, what are those other libs that get loaded here, where are those found and why are they needed...

Here goes the quote I'm talking about:

{$IFDEF FPC}
{$MODE Delphi}
{$ENDIF}

interface

uses
{$IFDEF MSWINDOWS}
  Windows,
  Messages,
{$ENDIF}
{$IFDEF LINUX} // for Kylix
  Libc,
  Types,
{$ENDIF}
  Classes,
{$IFDEF USETYPEINFO}
  // this pure pascal version must handle the 64-bits ordinal values and
  // the special layout of the underlying compiler (mostly FPC)
  typinfo,
{$ENDIF}
{$IFNDEF LVCL}
  Contnrs, // for TObjectList
{$ENDIF}
  SysUtils;
{$IFNDEF FPC}
{$L 'OBJ\sqlite3se.obj'}  //for Delphi
{$ELSE}
{$IFDEF MSWINDOWS}
{$linklib libsqlite3se.a}
{$linklib libmsvcrt.a}
{$linklib libkernel32.a}
{$linklib libgcc.a}
{$ELSE}
{$linklib libsqlite3selinux.a}
{$Linklib c}
{$linklib libgcc.a}
{$ENDIF}
{$ENDIF}

and add -Xt in fpc.cfg.
Regards, ArNy the Turbo Rascal
-
"The secret is to give them what they need, not what they want." - Scotty, STTNG:Relics

ludob

  • Hero Member
  • *****
  • Posts: 1173
Re: Static linking
« Reply #1 on: January 07, 2012, 11:42:22 am »
Quote
how does static linking work in FPC/Lazarus and where could some more information be found about it. Some of the questions would for example be, what are those other libs that get loaded here, where are those found and why are they needed...
http://en.wikipedia.org/wiki/Static_library explains what static linking is. FPC is not an exception. Think about a static library as a collection of .o and .ppu files. There are different library formats but the fpc linker only accepts the gnu format. On windows this means that static libraries produced by MS tools can not be used. Mingw or cygwin ports of the gnu binutils need to be used.
To comment on the snippet, I'll do this for windows because I went through the exercise of linking statically sqlite. I'll actually go through the complete steps, for posterity  ;)
- Get the sqlite source from http://www.sqlite.org/download.html (amalgamation: all sources in one file).
The source is written in C and needs to be compiled to a static library using a gnu tool chain. Personally I use wxDev-C++ (http://wxdsgn.sourceforge.net/). This a complete IDE for developing with wxWidgets and probably an overkill for this purpose. The advantage is that the installation is straightforward and that everything needed is installed and, when compiling from the IDE, there is no messing with include/library paths. It is based on the mingw port of the binutils which doesn't require additional run time dlls as opposed to Cygwin.
- in Dev-C++, create a new project, select static library, remove all the files added automatically from the project and add sqlite3.c, save project as sqlite3.dev in dir "whatever", compile project. In whatever\objects\MingW you'll find sqlite3.o. Easy enough.
- sqlite3 being written in C, and C having its own run-time (equivalent to fpc RTL) we'll probably need to pull in a few other libraries. If the program uses windows api's then we'll need to add these libraries also. You could go through the 128.000 lines of code in sqlite3.c to figure out the external dependencies but it is easier to be lazy and have the fpc linker do this for us. Create a new pascal program that uses the newly created library:
Code: [Select]
program TestSqlite3;
{$L sqlite3.o}
function sqlite3_libversion(): pchar; cdecl;external;
begin
  writeln(sqlite3_libversion);
end.
and compile. Make sure the path to sqlite3.o is added or simply copy it to the program directory. Linker stops at 50 undefined symbols. No panic. Let's start with what we recognise.
- InterlockedCompareExchange looks windows api. According msdn defined in kernel32. In the Dev-Cpp\lib directory you find a libkernel32.a. Add the path to it or copy it to the program directory. Add {$linklib libkernel32.a} to your program. Compile. Undefined reduced to 14 !!
- memcpy sounds like the c run time. That is msvcrt on windows. libmsvcrt.a is in Dev-Cpp\lib. Add {$linklib libmsvcrt.a}. Compile. Only 4 undefined symbols remain.
- a quick google on "Undefined symbol: ___udivdi3" shows that this belongs to the gcc run time library. In Dev-Cpp/lib/gcc/mingw32/3.4.2/ there is  libgcc.a. Add {$linklib libgcc.a} to the program. Compile. Bingo! Run program and it'll print the sqlite3 version :D If you have lost track, the complete program is now
Code: [Select]
program project1;
{$L sqlite3.o}
{$linklib libkernel32.a}
{$linklib libmsvcrt.a}
{$linklib libgcc.a}
function sqlite3_libversion(): pchar; cdecl;external;
begin
  writeln(sqlite3_libversion);
end.
- Now to use the statically linked version with sqldb, there is still some work to do. In the test program we declared function sqlite3_libversion() manually and Sqlite3conn is "hard-wired" to use the dynamic library. The way how i solved this is to make a copy of sqlite3conn.pp in my project directory, change uses sqlite3dyn to sqlite3static, create a new sqlite3static from sqlite3.inc (add "unit sqlite3static;" remove uses dynlibs, add the {$linklib xxx},{$L sqlite3.o} lines, change all "external Sqlite3Lib;" to "external;" and remove the contents of the implementation section except for the function sqlite3_version. If interested I can attach this file knowing that it isn't based on the latest svn version of sqlite3.inc.

Hope this answers you questions.

BigChimp

  • Hero Member
  • *****
  • Posts: 5740
  • Add to the wiki - it's free ;)
    • FPCUp, PaperTiger scanning and other open source projects
Re: Static linking
« Reply #2 on: January 07, 2012, 11:49:00 am »
Very clear explanation, thanks a lot.

One fairly off topic question (I suspect the answer but just wanted to make sure I get it): why would there be a dependency on GCC - shouldn't MinGW compile against the Microsoft C runtime? Or is that because GCC provides some functions not provided by MSVCRT that are called by sqlite?
Want quicker answers to your questions? Read http://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

Open source including papertiger OCR/PDF scanning:
https://bitbucket.org/reiniero

Lazarus trunk+FPC trunk x86, Windows x64 unless otherwise specified

ludob

  • Hero Member
  • *****
  • Posts: 1173
Re: Static linking
« Reply #3 on: January 07, 2012, 12:12:29 pm »
One fairly off topic question (I suspect the answer but just wanted to make sure I get it): why would there be a dependency on GCC - shouldn't MinGW compile against the Microsoft C runtime? Or is that because GCC provides some functions not provided by MSVCRT that are called by sqlite?
The GCC compiler uses the divdi3 functions for 64-bit multiplication and division which are not in MSVCRT. Sqlite isn't calling these functions directly but uses some 64-arithmetic.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6608
Re: Static linking
« Reply #4 on: January 07, 2012, 02:10:54 pm »
Pass   -k-static or -k--static and see what happens?

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6608
Re: Static linking
« Reply #5 on: January 07, 2012, 02:12:01 pm »
The GCC compiler uses the divdi3 functions for 64-bit multiplication and division which are not in MSVCRT. Sqlite isn't calling these functions directly but uses some 64-arithmetic.

Such helpers are in "libgcc"  The reason is that these functions are relatively small and much used, and avoiding DLL call overhead (how little it is) can be worthwhile.

ludob

  • Hero Member
  • *****
  • Posts: 1173
Re: Static linking
« Reply #6 on: January 07, 2012, 02:49:44 pm »
Quote
The reason is that these functions are relatively small and much used, and avoiding DLL call overhead (how little it is) can be worthwhile.
The goal of the exercise was to get rid of the sqlite3.dll and have an "all-in" executable. Substituting a dependency on sqlite3.dll with a dependency on libgcc_s_dw_2-1.dll (or whatever the used gcc version requires) wouldn't be too smart neither.

TurboRascal

  • Hero Member
  • *****
  • Posts: 672
  • "Good sysadmin. Bad programmer."™
Re: Static linking
« Reply #7 on: January 07, 2012, 07:08:09 pm »
@ludob: thank you for your answer, you made the effort to write more than was required just by my question, a mini-guide from basics up ;)

I believe that post would made a worthy wiki article, so if it's not much trouble, you might create a page there. If you wish, I can do it for you...
Regards, ArNy the Turbo Rascal
-
"The secret is to give them what they need, not what they want." - Scotty, STTNG:Relics

ludob

  • Hero Member
  • *****
  • Posts: 1173
Re: Static linking
« Reply #8 on: January 07, 2012, 08:26:28 pm »
Quote
I believe that post would made a worthy wiki article, so if it's not much trouble, you might create a page there. If you wish, I can do it for you...
I'll take up your offer. If you think you can turn this into a wiki article, please go ahead. The way I wrote the reply was more a explain by example than a wiki page on static linking in general or one on the static linking of sqlite.

chrnobel

  • Full Member
  • ***
  • Posts: 237
Re: Static linking
« Reply #9 on: January 07, 2012, 09:29:44 pm »
ludob

Thank you very much for the explanation, for me it clarified at lot.

Just at thought, would it be possible to automate this in some way, so that the FPC installed could automate this process, and thereby giving a possibility (also for the not so experienced user) to build programs where the static linking is done upfront?

It also leads me to an other question, maybe off topic, why is there a requirement for having a special library/client for MySQL - I mean all the "clever" work is done one the server, and (as I understand it) all the client (simplified) has to do is to send SQL commands over port 3306, and then receive the result and pass it to the program.
I know that the communication is encrypted, but anyway?

If it was possible to write the client purely in Pascal, and having it to be a part of FPC would be an huge advantage, not only for making programs simpler, but also for having an easy way to say goodbye to Oracle, and using MariaDB instead.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6608
Re: Static linking
« Reply #10 on: January 07, 2012, 10:22:28 pm »
Quote
The reason is that these functions are relatively small and much used, and avoiding DLL call overhead (how little it is) can be worthwhile.
The goal of the exercise was to get rid of the sqlite3.dll and have an "all-in" executable. Substituting a dependency on sqlite3.dll with a dependency on libgcc_s_dw_2-1.dll (or whatever the used gcc version requires) wouldn't be too smart neither.

(1) You need .a's for all involved libs. Avoid the corresponding dlls in the path.
(2) if you want a certain library (e.g. libgcc) for all cases where you link a certain other lib, use -XLA.   So in the example  -XLAsqlite=sqlite,gcc   
     Don't worry, duplicates are removed.
(3) Some libraries like libgcc need a special place in the link.res file.  Using -XLO you can manipulate this.
(4) I did this kind of stuff before the internal linker existed, and forced the external linker (-Xe) using -k-static


ludob

  • Hero Member
  • *****
  • Posts: 1173
Re: Static linking
« Reply #11 on: January 08, 2012, 12:06:49 pm »
Just at thought, would it be possible to automate this in some way, so that the FPC installed could automate this process, and thereby giving a possibility (also for the not so experienced user) to build programs where the static linking is done upfront?
The purpose of the write-up was also to demonstrate the process of finding the dependent libraries. Other than parsing all libraries on your system and make a kind of database with exported functions, I don't see how to facilitate this. To automate is even more difficult. What happens if the same function is exported by different libraries. It could be just a name collision or 2 different, incompatible, versions of the library. Both require some knowledge of the libraries to solve the ambiguity.
Static linking upfront: you mean creating a big library containing everything needed. What would be the advantage? It still requires to figure out all dependencies and there would be as many versions as there versions of the contained libraries (think also 32-64bit).  I don't see that advantage over creating a unit that contains a series of {$linklib xxx} lines.
It also leads me to an other question, maybe off topic, why is there a requirement for having a special library/client for MySQL - I mean all the "clever" work is done one the server, and (as I understand it) all the client (simplified) has to do is to send SQL commands over port 3306, and then receive the result and pass it to the program.
I know that the communication is encrypted, but anyway?

If it was possible to write the client purely in Pascal, and having it to be a part of FPC would be an huge advantage, not only for making programs simpler, but also for having an easy way to say goodbye to Oracle, and using MariaDB instead.
The db client is doing a lot more.:
- communication: databases use a proprietary network protocol.
- session management
- result set and metadata handling
- data binding, both for parameters and result sets.   
- ...
I invite you to take a look at the mysql client library source just to get an idea of the amount of code involved. Then look at the mysql bug tracker and ask yourself if you want to duplicate all this in pascal or just use the C library and leave the low level stuff to the experts.
To be honest, I don't see the advantage of rewriting a db client library in pascal. Apart from the big effort, you would be running constantly after the facts because of the proprietary protocol. We run even after the facts in creating the bindings for the existing mysql client libraries (latest in fpc trunk is 5.1)

OOP is all about re-using software. That includes the use of existing libraries.


marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6608
Re: Static linking
« Reply #12 on: January 08, 2012, 06:16:45 pm »
- communication: databases use a proprietary network protocol.

(And even when not, the wire protocol is often separately versioned, and the client library might transform requests to fit them to the wire protocol)

Quote
To be honest, I don't see the advantage of rewriting a db client library in pascal. Apart from the big effort, you would be running constantly after the facts because of the proprietary protocol. We run even after the facts in creating the bindings for the existing mysql client libraries (latest in fpc trunk is 5.1)

Mysql is a special problem, due to its licensing and (deliberate!) versioning problems, and because it is used a lot about newbies that haven't ran into that.

This causes
1. more problems than for the other databases
2. the bulk of the people that are capable of maintenance to avoid it in the first place.

That leaves a very small developer base that is knowledgable enough but is forced to use it to handle the bulk of the problems.

In short: if you have _any_ kind of choice, don't use mysql.

Quote
OOP is all about re-using software. That includes the use of existing libraries.

Every software paradigm strives to do so  on paper. OOP is no exception :-)

Shebuka

  • Sr. Member
  • ****
  • Posts: 404
Re: Static linking
« Reply #13 on: January 17, 2012, 05:23:23 pm »
Hi, what happens if i have a unitA that has {$linklib mylibrary} and then i have a unitB that is created at some point by unitA and that also has {$linklib mylibrary}? Is mylibrary initialized 2 times?

Because i having, at random, some strange behavior and try to exclude as much cases as possible.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6608
Re: Static linking
« Reply #14 on: January 17, 2012, 08:57:49 pm »
Hi, what happens if i have a unitA that has {$linklib mylibrary} and then i have a unitB that is created at some point by unitA and that also has {$linklib mylibrary}? Is mylibrary initialized 2 times?

No.

Quote
Because i having, at random, some strange behavior and try to exclude as much cases as possible.

Start searching for mistakes in your code. It is statistically the most likely option