Call to external library 64bit vs 32bit

[Чебурашка]

Hello,
I have created a very simple library hereafter:

Code: Pascal  [Select][+][-]

1.  
2. library project1;
3.  
4. {$mode objfpc}{$H+}
5.  
6. uses Classes, SysUtils;
7.  
8. function Exp_F(v1: string; v2: Integer): string;
9. begin
10.   Writeln(Format('Received v1=%s', [v1]));
11.   Writeln(Format('Received v2=%d', [v2]));
12.   Writeln(Format('Returning %s%d', [v1, v2]));
13.   Result := Format('%s%d', [v1, v2]);
14. end;
15.  
16. exports Exp_F;
17.  
18. begin
19. end.
20.  
21.  

Then I created a very simple program aimed to load the lib and call the Exp_F function.

Code: Pascal  [Select][+][-]

1.  
2. program project2;
3.  
4. uses
5.   dynlibs;
6.  
7. Type
8.   TEXTFUNC__Ext_F = function(v1: string; v2: Integer): string; cdecl;
9.  
10. var
11.   lib: TLibHandle;
12.   Ext_F: TEXTFUNC__Ext_F;
13.  
14. begin
15.   lib := LoadLibrary('libproject1.so');
16.  
17.   Ext_F := TEXTFUNC__Ext_F(GetProcAddress(lib, 'Exp_F'));
18.  
19.   Ext_F('some nice string', 56);
20.  
21.   UnloadLibrary(lib);
22.   lib := 0;
23.  
24. end.
25.  
26.  

If both are compiled 64 bit then:

Code: Text  [Select][+][-]

1. tt@debian:/mnt/virtual-box-shared/library-test$ ./project2
2. Received v1=some nice string
3. Received v2=56
4. Returning some nice string56
5.  

If both are compiled 32 bit then:

Code: Text  [Select][+][-]

1. tt@debian:~/Documents/test/library-test$ ./project2
2. Runtime error 217 at $0805AF87
3.   $0805AF87
4.   $0805B1E4
5.   $0806902D
6.  

INFO1: I compiled project1 and project2 so that the library libproject1.so and the executable project2 are in same directory. This in linux requires that you add the current directory in the LD_LIBRARY_PATH for project2 to find libproject1.so.

INFO2: I did compile:

• 64 bit under debian 10.11 with uname -a="Linux debian 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux"
• 32 bit under debian 10.11 with uname -a="Linux debian 4.19.0-19-686 #1 SMP Debian 4.19.232-1 (2022-03-07) i686 GNU/Linux"

Any ideas to explain the failure on 32bit?

[marcov]

Your project1 declaration has no cdecl, but your project2 does.

That at least matters more on 32-bit than on 64-bit where there generally is only one calling convention.

[Чебурашка]

Your project1 declaration has no cdecl, but your project2 does.

That at least matters more on 32-bit than on 64-bit where there generally is only one calling convention.

I tried this but is does not work.

Code: Pascal  [Select][+][-]

1.  
2. // ...
3.  
4. function Exp_F(v1: string; v2: Integer): string; cdecl;
5. begin
6.  
7. // ...
8.  
9.  

[af0815]

For interfacing string with a library, should it not be a PChar or PWideChar type ?

[Чебурашка]

For interfacing string with a library, should it not be a PChar or PWideChar type ?

If your hypothesis was correct, then I should get a failure also in 64 bit, but I did not get any failure.

[440bx]

I tried this but is does not work.

When something doesn't work, the first step is to check the values used in the method.  In your case that means, check that LoadLibrary and GetProcAddress executed successfully and, returned the expected values.

It's also quite telling that the code works in 64 bit but not in 32 bit.  That's almost always an indication that there is a calling convention mismatch somewhere (as @marcov correctly pointed out.)

Einstein "recommended" to make things as simple as possible but no simpler.  It looks like your code is a bit "simpler" than it should be.

[Чебурашка]

Added cdecl in library function, switched to PChar as suggested, added logging.
It works correctly.
Also adding Heaptrc extra information, I see that all memory is released correclty.

In the 32bit I made a mistake in the LD_LIBRARY_PATH setting.

Code: Pascal  [Select][+][-]

1. library project1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses Classes, SysUtils;
6.  
7. function Exp_F(v1: PChar; v2: Integer): PChar; cdecl;
8. var
9.   r: string;
10. begin
11.   Writeln(Format('Received v1=%s', [v1]));
12.   Writeln(Format('Received v2=%d', [v2]));
13.   Writeln(Format('Returning %s%d', [v1, v2]));
14.   r := Format('%s%d', [v1, v2]);
15.   Result := GetMem(Length(r) + 1);
16.   FillByte(Result^, Length(r) + 1, 0);
17.   Move(r[1], Result^, Length(r));
18. end;
19.  
20. exports Exp_F;
21.  
22. begin
23. end.
24.  

Code: Pascal  [Select][+][-]

1. program project2;
2.  
3. uses
4.   dynlibs, SysUtils;
5.  
6. Type
7.   TEXTFUNC__Ext_F = function(v1: PChar; v2: Integer): PChar; cdecl;
8.  
9. var
10.   lib: TLibHandle;
11.   Ext_F: TEXTFUNC__Ext_F;
12.   result: PChar;
13.  
14. begin
15.   Writeln('Loading library');
16.   lib := LoadLibrary('libproject1.so');
17.   if (lib <> 0) then
18.   begin
19.        Writeln(' ...success');
20.   end else
21.   begin
22.        Writeln(' ...failed');
23.        Exit();
24.   end;
25.  
26.   Writeln('Assigning funcptr');
27.   Ext_F := TEXTFUNC__Ext_F(GetProcAddress(lib, 'Exp_F'));
28.   if (Assigned(Ext_F)) then
29.   begin
30.        Writeln(' ...success');
31.   end else
32.   begin
33.        Writeln(' ...failed');
34.        Exit();
35.   end;
36.  
37.   Writeln('Calling ext func');
38.   result := Ext_F('some nice string', 56);
39.   Writeln(Format('Got: %s', [result]));
40.   FreeMem(result);
41.   Writeln(' ...success');
42.  
43.   Writeln('Unloading library');
44.   UnloadLibrary(lib);
45.   lib := 0;
46.   Writeln(' ...success');
47.  
48.   Writeln('');
49. end.
50.  

Code: Text  [Select][+][-]

1. tt@debian:/mnt/virtual-box-shared/library-test$ ./project2
2. Loading library
3.  ...success
4. Assigning funcptr
5.  ...success
6. Calling ext func
7. Received v1=some nice string
8. Received v2=56
9. Returning some nice string56
10. Got: some nice string56
11.  ...success
12. Unloading library
13.  ...success
14.  

Code: Text  [Select][+][-]

1. tt@debian:~/Documents/test/ext-lib-call$ ./project2
2. Loading library
3.  ...success
4. Assigning funcptr
5.  ...success
6. Calling ext func
7. Received v1=some nice string
8. Received v2=56
9. Returning some nice string56
10. Got: some nice string56
11.  ...success
12. Unloading library
13.  ...success
14.  

[440bx]

switched to PChar as suggested

No problem with that but, it should work with "string" too.

In the 32bit I made a mistake in the LD_LIBRARY_PATH setting.

that explains why it didn't work in 32bit and, not to beat a dead horse but, if you had checked the result from LoadLibrary, you would have found the problem earlier/faster.

[Fred vS]

lib := LoadLibrary('libproject1.so');

You may also use the full path:

Code: Pascal  [Select][+][-]

1. lib := LoadLibrary('/path/oflibrary/libproject1.so');

[PascalDragon]

switched to PChar as suggested

No problem with that but, it should work with "string" too.

No, it won't. Or at least it's not guaranteed that it will work, because the memory managers of the RTL in the library and the one in the application won't be shared. This might in the best case lead to a memory leak in the worst to an exception if the memory manager tries to free memory that it hadn't allocated itself. This is also the case with the usage of GetMem in the variant with PChar: the memory needs to be freed my the library as well, thus tt needs to also export some FreeMem or whatever function to release memory allocated by the library.

[440bx]

No, it won't. Or at least it's not guaranteed that it will work, because the memory managers of the RTL in the library and the one in the application won't be shared.

I was and, still am, under the impression that if _both_ the library and the program are compiled using the same language (as seems to be the case in this particular post) then there should be no problem.  Isn't that the case with FPC ?

[PascalDragon]

No, it won't. Or at least it's not guaranteed that it will work, because the memory managers of the RTL in the library and the one in the application won't be shared.

I was and, still am, under the impression that if _both_ the library and the program are compiled using the same language (as seems to be the case in this particular post) then there should be no problem.  Isn't that the case with FPC ?

It isn't the case with FPC, it isn't the case with Delphi or with any language that provides its own memory manager. If you manually allocate memory using some third library (for example the operating system itself using VirtualAlloc or something like that and the other module uses the corresponding function to free it) then this will work of course, but not if the runtime of the two modules keep track of memory blocks in their own way which is what FPC's memory manager does (and this involves any operation on managed string types, dynamic arrays, classes or even New and GetMem). If different memory managers are involved it would be as if you'd use VirtualAlloc to allocate the memory, but HeapFree to release it.

[440bx]

It isn't the case with FPC, it isn't the case with Delphi or with any language that provides its own memory manager.

It seems to me that the problem is related to the fact that a string is a managed type because, at least in this case, the memory managers are the same in the main program and the dll but, since a string is a managed type, if the dll alters the allocation of the string that will cause the string's state information kept by the main program to no longer be valid.  Is this the root cause of the problem ?

If you manually allocate memory using some third library (for example the operating system itself using VirtualAlloc or something like that and the other module uses the corresponding function to free it) then this will work of course

Agreed. Because the allocation state of the memory blocks is unique (kept in only one place.)

but not if the runtime of the two modules keep track of memory blocks in their own way which is what FPC's memory manager does (and this involves any operation on managed string types, dynamic arrays, classes or even New and GetMem). If different memory managers are involved it would be as if you'd use VirtualAlloc to allocate the memory, but HeapFree to release it.

I see it as, the memory managers are the same but, the state information of a managed variable they both use may get out of synch if either one alters its allocation.  Correct ?

[PascalDragon]

It isn't the case with FPC, it isn't the case with Delphi or with any language that provides its own memory manager.

It seems to me that the problem is related to the fact that a string is a managed type because, at least in this case, the memory managers are the same in the main program and the dll but, since a string is a managed type, if the dll alters the allocation of the string that will cause the string's state information kept by the main program to no longer be valid.  Is this the root cause of the problem ?

No. The problem is the memory manager alone. With managed types like AnsiString it's simply more apparent cause you can't directly influence their allocations. The problems with strings go away as soon as you make sure that the memory managers are shared between the library and the main binary.

but not if the runtime of the two modules keep track of memory blocks in their own way which is what FPC's memory manager does (and this involves any operation on managed string types, dynamic arrays, classes or even New and GetMem). If different memory managers are involved it would be as if you'd use VirtualAlloc to allocate the memory, but HeapFree to release it.

I see it as, the memory managers are the same but, the state information of a managed variable they both use may get out of synch if either one alters its allocation.  Correct ?

Again, nothing to do with managed types. The memory managers have the same code (albeit located in both the library and the main program), but their state is different between library and main program and that is the issue. Essentially you need to consider them as if being written in a completely different language.

[HeavyUser]

It isn't the case with FPC, it isn't the case with Delphi or with any language that provides its own memory manager.

It seems to me that the problem is related to the fact that a string is a managed type because, at least in this case, the memory managers are the same in the main program and the dll but, since a string is a managed type, if the dll alters the allocation of the string that will cause the string's state information kept by the main program to no longer be valid.  Is this the root cause of the problem ?

No. The problem is the memory manager alone. With managed types like AnsiString it's simply more apparent cause you can't directly influence their allocations. The problems with strings go away as soon as you make sure that the memory managers are shared between the library and the main binary.

but not if the runtime of the two modules keep track of memory blocks in their own way which is what FPC's memory manager does (and this involves any operation on managed string types, dynamic arrays, classes or even New and GetMem). If different memory managers are involved it would be as if you'd use VirtualAlloc to allocate the memory, but HeapFree to release it.

I see it as, the memory managers are the same but, the state information of a managed variable they both use may get out of synch if either one alters its allocation.  Correct ?

Again, nothing to do with managed types. The memory managers have the same code (albeit located in both the library and the main program), but their state is different between library and main program and that is the issue. Essentially you need to consider them as if being written in a completely different language.

OK clear and understandable statement. I have just one question how would you go about to share the memory manager? Lets start when both are written in FPC although a c/c++ dll might interest me too in the future.