zlib. Problem with static linking in 64-bit

[Igor Kokarev]

I'm trying to link statically zlib 1.2.11.

It works fine in 32-bit app on Windows. But in 64-bit app I get linking error:

Miltiple defined symbol .refptr.zcfree
Miltiple defined symbol .refptr.zcalloc
Miltiple defined symbol .refptr.zcfree
Miltiple defined symbol .refptr.zcalloc

I compiled .o files of zlib in MSYS2/MINGW64. Example:

Code: Pascal  [Select][+][-]

1. gcc  -O2 -Wall -c -o inflate.o inflate.c

I found a message on same problem on another website. This guy replied me that he didn't find the solution. The problem is related with refptr in new versions of gcc compiler. And he patched manually .o files in IDA и OllyDb to make it work with fpc.

fpc 3.0.4, Lazarus 1.8.4 64 bit.

[Igor Kokarev]

It seems that fpc is not fully compatible with 64-bit .o files  from gcc which use refptr construction.

[Igor Kokarev]

If I remove the following lines from inflate.c and deflate.c

strm->zalloc = zcalloc;
strm->zfree = zcfree;

And compile .o files in gcc, linking in 64-bit fpc works correctly. Of course, zlib will not work correctly without this code.

I have no idea, why fpc linker doesn't like this code in 64-bit mode.

[DonAlfredo]

Perhaps these files will work for you ?

https://github.com/synopse/mORMot/tree/master/static/x86_64-win64

[Igor Kokarev]

Thanks for your reply.

There is no infback.o file and linking fails.

I found a possible workaround for 64 bit zlib:

1. In zlib files inflate.c and deflate.c remove these lines

strm->zalloc = zcalloc;
strm->zfree = zcfree;

I replaced that code with my code:

strm->zalloc = (voidpf)0;
strm->zfree = (voidpf)0;

Now linking in 64 bit fpc works fine.

2. I wrote Pascal implementation of these functions.

Code: Pascal  [Select][+][-]

1. function zcalloc2(opaque: Pointer; items, size: Integer): Pointer;
2. begin
3.   GetMem(Result, items * size);
4. end;
5.  
6. procedure zcfree2(opaque, block: Pointer);
7. begin
8.   FreeMem(block);
9. end;
10.  

And set callbacks:

zStream.zalloc:=zcalloc2;
zFZStream.zfree:=zcfree2;

BEFORE calls to zlib functions:
deflateInit_
InflateInit_
InflateInit2_

3. My z_stream record is packed and I also had a problem with different SizeOf(z_steam) in my code and in zlib internal functions. So I also corrected zutils.h and I added 4 byte alignment:

#pragma pack(4)
before
typedef struct z_stream_s {

zlib works fine now in 64-bit fpc.

But this code now doesn't work in 32-bit fpc and zlib fails. So I compile 32 bit zlib from original sources (without my additions) and I don't set any callbacks.

I'm not sure that I did everything correctly, so any suggestions are welcome.

[skalogryz]

But this code now doesn't work in 32-bit fpc and zlib fails. So I compile 32 bit zlib from original sources (without my additions) and I don't set any callbacks.

Short:
it's likely because the functions needs to be declared as "cdecl" convention.
(depends on how zlib object file was compiled, but I'd think it was compiled using cdecl)

Long:
Both strm.zalloc and strm.zfree (as well as linking for zcfree and zalloc)
are in zlib library in order to use C memory management with malloc() and free().
Thus any language (in this case freepascal) can provide its own memory management functions.

Providing custom implementation of zcalloc and zcfree is fine and desirable.

The problem comes with C-calls differences for 64-bit and 32-bit.
By default, 32-bit c-language convention is "cdecl". (it's possible to build zlib using a different convention, i.e. stdcall. but it's unlikely)
For 64-bit, there's no special "cdecl". If "cdecl" is specified in pascal source code - it would be ignored for 64-bit.

Thus the following code should resolve the problem for both 32 and 64 bit

Code: Pascal  [Select][+][-]

1. function zcalloc2(opaque: Pointer; items, size: Integer): Pointer; cdecl;
2. begin
3.   GetMem(Result, items * size);
4. end;
5.  
6. procedure zcfree2(opaque, block: Pointer); cdecl;
7. begin
8.   FreeMem(block);
9. end;
10.  
11.  

[Igor Kokarev]

Hi,

Thanks for your reply! Glad to see you.

You're right, I added cdecl to callback functions and now zlib with callbacks also works in 32-bit.

P.S. Do you have any idea why original zlib 1.2.11 .o files cannot be linked in 64-bit fpc?

[skalogryz]

P.S. Do you have any idea why original zlib 1.2.11 .o files cannot be linked in 64-bit fpc?

I presume the issue is with how the object files being built.
I can think you're trying to compile each object file one-by-one, which caused zutil.o file to have zcalloc and zcfree functions to be compiled in.
(since your project, as well as zlib object files are both providing "zcalloc" and "zcfree" object names, it's throwing an error, since it cannot chose which one to use)

The better approach is use build utilities supplied with mingw and the zlib. (i.e. configure, make)
Yet it will require you to learn the configuration options for zlib project.

[Igor Kokarev]

The problem with linking in fpc occurs even if I just link zlib .o files into a new Lazarus project. Without any Pascal code for zlib. So it might be pure fpc problem.

I found at least 2 messages on this problem on other forums (on Russian):

http://www.freepascal.ru/forum/viewtopic.php?f=13&t=6536&start=15#p85904

http://www.elite-games.ru/conference/viewtopic.php?p=3504029&sid=5539e2a9eca9ca8aff8873f28e8560e3#3504029

[skalogryz]

So it might be pure fpc problem.

try to enable external linker then:

Code: [Select]

-Xe        Use external linker 


[Igor Kokarev]

in gcc?

[lucamar]

A suggestion: If using zlib is too onerous, you could try paszlib instead. It's a pure Pascal conversion of the library itself, not just the headers, so it doesn't depend on the C lib. And it includes a handy TZipper class to (de-)compress .zip files.

[skalogryz]

in gcc?

in FPC.
It should exclude FPC linker internal from the process, switching to the external gnu linker.

you could try paszlib instead.

what version of zlib, paszlib replicates?

[fcu]

A suggestion: If using zlib is too onerous, you could try paszlib instead. It's a pure Pascal conversion of the library itself, not just the headers, so it doesn't depend on the C lib. And it includes a handy TZipper class to (de-)compress .zip files.

paszlib is lacking of speed like many other libs e.g fpimage , but there is an alternative which i belive is faster than paszlib
https://github.com/synopse/mORMot/blob/master/PasZip.pas

[lucamar]

what version of zlib, paszlib replicates?

According to paszlib.pas:

Code: Pascal  [Select][+][-]

1. const
2.   ZLIB_VERSION = '1.2';
3.  

although zbase.pas and readme.txt refer respectively to:

Code: Pascal  [Select][+][-]

1. { in zbase.pas }
2. const
3.   ZLIB_VERSION : string[10] = '1.1.2';
4.  

Code: [Select]

(in readme.txt)
Based on the zlib 1.1.2, a general purpose data compression library.
Not modern versions in any case--not that zlib has changed so much...