Recent

Author Topic: Not able to use Python .so library function  (Read 1313 times)

rnfpc

  • Jr. Member
  • **
  • Posts: 81
Not able to use Python .so library function
« on: June 24, 2019, 05:38:08 pm »
I have this simple Python program which returns a string:

Code: Pascal  [Select]
  1. def hellofn():
  2.    return 'Hello from python fn.';

I have converted this to an .so file with cython and gcc. Now I am trying to call this function from a freepascal program. My code is:

Code: Pascal  [Select]
  1. {$mode objfpc}
  2.  
  3. function hellofn:string;stdcall;external 'libpyfnhello.so';
  4. begin
  5.    writeln(hellofn);
  6. end.

However, on trying to compile with fpc, I get following error output:

Code: Pascal  [Select]
  1. $ fpc rnloadLibrary_static.pas
  2. Free Pascal Compiler version 3.0.0+dfsg-11+deb9u1 [2017/06/10] for x86_64
  3. Copyright (c) 1993-2015 by Florian Klaempfl and others
  4. Target OS: Linux for x86-64
  5. Compiling rnloadLibrary_static.pas
  6. Linking rnloadLibrary_static
  7. /usr/bin/ld.bfd: warning: link.res contains output sections; did you forget -T?
  8. rnloadLibrary_static.o: In function `main':
  9. rnloadLibrary_static.pas:(.text.n_main+0x28): undefined reference to `hellofn'
  10. ./libpyfnhello.so: undefined reference to `PyUnicode_FromFormat'
  11. ./libpyfnhello.so: undefined reference to `PyExc_SystemError'
  12. ./libpyfnhello.so: undefined reference to `PyDict_SetItemString'
  13. ./libpyfnhello.so: undefined reference to `PyDict_Size'
  14. ./libpyfnhello.so: undefined reference to `PyObject_ClearWeakRefs'
  15. ./libpyfnhello.so: undefined reference to `PyFrame_New'
  16. ./libpyfnhello.so: undefined reference to `PyObject_GetAttrString'
  17. ./libpyfnhello.so: undefined reference to `PyImport_AddModule'
  18. ./libpyfnhello.so: undefined reference to `PyBytes_FromStringAndSize'
  19. ./libpyfnhello.so: undefined reference to `PyObject_SetAttrString'
  20. ./libpyfnhello.so: undefined reference to `PyErr_WarnEx'
  21. ./libpyfnhello.so: undefined reference to `PyObject_GC_Del'
  22. ./libpyfnhello.so: undefined reference to `PyCode_New'
  23. ./libpyfnhello.so: undefined reference to `PyImport_GetModuleDict'
  24. ./libpyfnhello.so: undefined reference to `PyObject_GC_Track'
  25. ./libpyfnhello.so: undefined reference to `PyErr_SetString'
  26. ./libpyfnhello.so: undefined reference to `PyMethod_New'
  27. ./libpyfnhello.so: undefined reference to `_PyObject_GC_New'
  28. ./libpyfnhello.so: undefined reference to `PyExc_TypeError'
  29. ./libpyfnhello.so: undefined reference to `PyTuple_GetItem'
  30. ./libpyfnhello.so: undefined reference to `PyMem_Realloc'
  31. ./libpyfnhello.so: undefined reference to `PyErr_ExceptionMatches'
  32. ./libpyfnhello.so: undefined reference to `PyOS_snprintf'
  33. ./libpyfnhello.so: undefined reference to `PyTraceBack_Here'
  34. ./libpyfnhello.so: undefined reference to `PyObject_Free'
  35. ./libpyfnhello.so: undefined reference to `PyInstanceMethod_New'
  36. ./libpyfnhello.so: undefined reference to `PyType_Ready'
  37. ./libpyfnhello.so: undefined reference to `PyErr_Clear'
  38. ./libpyfnhello.so: undefined reference to `PyTuple_New'
  39. ./libpyfnhello.so: undefined reference to `PyThreadState_Get'
  40. ./libpyfnhello.so: undefined reference to `PyErr_Occurred'
  41. ./libpyfnhello.so: undefined reference to `PyModule_Create2'
  42. ./libpyfnhello.so: undefined reference to `PyTuple_GetSlice'
  43. ./libpyfnhello.so: undefined reference to `PyDict_GetItemString'
  44. ./libpyfnhello.so: undefined reference to `_Py_NoneStruct'
  45. ./libpyfnhello.so: undefined reference to `PyDict_New'
  46. ./libpyfnhello.so: undefined reference to `PyUnicode_FromString'
  47. ./libpyfnhello.so: undefined reference to `PyUnicode_InternFromString'
  48. ./libpyfnhello.so: undefined reference to `PyExc_ImportError'
  49. ./libpyfnhello.so: undefined reference to `PyDict_SetItem'
  50. ./libpyfnhello.so: undefined reference to `PyExc_AttributeError'
  51. ./libpyfnhello.so: undefined reference to `PyUnicode_Decode'
  52. ./libpyfnhello.so: undefined reference to `PyErr_Format'
  53. ./libpyfnhello.so: undefined reference to `PyUnicode_FromStringAndSize'
  54. ./libpyfnhello.so: undefined reference to `PyModule_GetDict'
  55. ./libpyfnhello.so: undefined reference to `PyMem_Malloc'
  56. ./libpyfnhello.so: undefined reference to `Py_GetVersion'
  57. ./libpyfnhello.so: undefined reference to `PyObject_GC_UnTrack'
  58. rnloadLibrary_static.pas(11,1) Error: Error while linking
  59. rnloadLibrary_static.pas(11,1) Fatal: There were 1 errors compiling module, stopping
  60. Fatal: Compilation aborted
  61. Error: /usr/bin/ppcx64 returned an error exitcode
  62.  

How can I call this function in .so file?
« Last Edit: June 24, 2019, 05:41:40 pm by rnfpc »

Zoran

  • Hero Member
  • *****
  • Posts: 1424
    • http://wiki.lazarus.freepascal.org/User:Zoran
Re: Not able to use Python .so library function
« Reply #1 on: June 24, 2019, 06:42:06 pm »
Perhaps the problem is in using String (this is Pascal own type, unknown to other languages) in your function.

First, try to make similar function which returns Integer instead of String, just to see if it works.

Then, to exchange a string with the library, try to use void function and pass preallocated PChar buffer to it by reference.

Something like this (untested):
Code: Pascal  [Select]
  1. procedure hellofn(P: PChar); stdcall; external 'libpyfnhello.so';
  2.  
  3. procedure TryHelloFn;
  4. var
  5.   S: String;
  6.   P: PChar;
  7. begin
  8.   P := GetMem(P, 100);
  9.   try
  10.     hellofn(P);
  11.     S := P;
  12.     writeln(S);
  13.   finally
  14.     FreeMem(P);
  15.   end;
  16. end;
  17.  
or, simpler:

Code: Pascal  [Select]
  1. procedure TryHelloFn;
  2. var
  3.   S: String;
  4. begin
  5.   SetLength(S, 100);
  6.   hellofn(@S[1]);
  7.   writeln(S);
  8. end;
  9.  

I don't know how this function should be written in Python. In C:

Code: C  [Select]
  1. void hellofn(char *p) {
  2.   strcpy(p, 'Hello, World!');
  3. }
  4.  

Another thing -- are you sure that stdcall is the right calling convention? Maybe cdecl should be used?
« Last Edit: June 24, 2019, 06:44:40 pm by Zoran »

rnfpc

  • Jr. Member
  • **
  • Posts: 81
Re: Not able to use Python .so library function
« Reply #2 on: June 24, 2019, 07:41:07 pm »
I am getting same error with integer in place of string.

In C, my function would be something like:

Code: Pascal  [Select]
  1. char* hellofn() {
  2.     return 'Hello, World!';
  3. }

And we have to receive this string in Pascal program.

SymbolicFrank

  • Hero Member
  • *****
  • Posts: 635
Re: Not able to use Python .so library function
« Reply #3 on: June 24, 2019, 07:55:04 pm »
The error is that Python cannot find its own memory manager. But even if you linked all that in, you wouldn't be able to use it, as Pascal has its own memory manager. They cannot use the locations of the other.

And Python strings are different from Pascal strings.

You would have to make a buffer in Pascal and give a pointer of that to Python and have it strcpy() the chars into it. In the right format.

PascalDragon

  • Sr. Member
  • ****
  • Posts: 443
  • Compiler Developer
Re: Not able to use Python .so library function
« Reply #4 on: June 25, 2019, 09:36:26 am »
How can I call this function in .so file?
I've never used a precompiled Python file, but there are nevertheless a couple of problems I can spot:
Code: Pascal  [Select]
  1. {$mode objfpc}
  2.  
  3. function hellofn:string;stdcall;external 'libpyfnhello.so';
  4. begin
  5.    writeln(hellofn);
  6. end.
Don't use the Pascal String type to interact with code written in another language. I don't know what kind of string cython/gcc generate, but it will either be some custom type or something akin to PChar or PAnsiChar. Check the documentation of cython for that!

Next is the calling convention. I'd bet that the calling convention will be cdecl instead of stdcall as the latter is only really used on Windows.

Then we have this error:
Code: [Select]
rnloadLibrary_static.o: In function `main':
rnloadLibrary_static.pas:(.text.n_main+0x28): undefined reference to `hellofn'

You'll have to find out how the function exported by the library is really called (you can use the objdump tool for example) and then declare your function as
Code: Pascal  [Select]
  1. external 'libpyfnhello.so' name 'whatever';

Then we have all those other linker errors:
Code: [Select]
./libpyfnhello.so: undefined reference to `PyUnicode_FromFormat'
./libpyfnhello.so: undefined reference to `PyExc_SystemError'
./libpyfnhello.so: undefined reference to `PyDict_SetItemString'
./libpyfnhello.so: undefined reference to `PyDict_Size'
./libpyfnhello.so: undefined reference to `PyObject_ClearWeakRefs'
./libpyfnhello.so: undefined reference to `PyFrame_New'
./libpyfnhello.so: undefined reference to `PyObject_GetAttrString'
./libpyfnhello.so: undefined reference to `PyImport_AddModule'
./libpyfnhello.so: undefined reference to `PyBytes_FromStringAndSize'
./libpyfnhello.so: undefined reference to `PyObject_SetAttrString'
./libpyfnhello.so: undefined reference to `PyErr_WarnEx'
./libpyfnhello.so: undefined reference to `PyObject_GC_Del'
./libpyfnhello.so: undefined reference to `PyCode_New'
./libpyfnhello.so: undefined reference to `PyImport_GetModuleDict'
./libpyfnhello.so: undefined reference to `PyObject_GC_Track'
./libpyfnhello.so: undefined reference to `PyErr_SetString'
./libpyfnhello.so: undefined reference to `PyMethod_New'
./libpyfnhello.so: undefined reference to `_PyObject_GC_New'
./libpyfnhello.so: undefined reference to `PyExc_TypeError'
./libpyfnhello.so: undefined reference to `PyTuple_GetItem'
./libpyfnhello.so: undefined reference to `PyMem_Realloc'
./libpyfnhello.so: undefined reference to `PyErr_ExceptionMatches'
./libpyfnhello.so: undefined reference to `PyOS_snprintf'
./libpyfnhello.so: undefined reference to `PyTraceBack_Here'
./libpyfnhello.so: undefined reference to `PyObject_Free'
./libpyfnhello.so: undefined reference to `PyInstanceMethod_New'
./libpyfnhello.so: undefined reference to `PyType_Ready'
./libpyfnhello.so: undefined reference to `PyErr_Clear'
./libpyfnhello.so: undefined reference to `PyTuple_New'
./libpyfnhello.so: undefined reference to `PyThreadState_Get'
./libpyfnhello.so: undefined reference to `PyErr_Occurred'
./libpyfnhello.so: undefined reference to `PyModule_Create2'
./libpyfnhello.so: undefined reference to `PyTuple_GetSlice'
./libpyfnhello.so: undefined reference to `PyDict_GetItemString'
./libpyfnhello.so: undefined reference to `_Py_NoneStruct'
./libpyfnhello.so: undefined reference to `PyDict_New'
./libpyfnhello.so: undefined reference to `PyUnicode_FromString'
./libpyfnhello.so: undefined reference to `PyUnicode_InternFromString'
./libpyfnhello.so: undefined reference to `PyExc_ImportError'
./libpyfnhello.so: undefined reference to `PyDict_SetItem'
./libpyfnhello.so: undefined reference to `PyExc_AttributeError'
./libpyfnhello.so: undefined reference to `PyUnicode_Decode'
./libpyfnhello.so: undefined reference to `PyErr_Format'
./libpyfnhello.so: undefined reference to `PyUnicode_FromStringAndSize'
./libpyfnhello.so: undefined reference to `PyModule_GetDict'
./libpyfnhello.so: undefined reference to `PyMem_Malloc'
./libpyfnhello.so: undefined reference to `Py_GetVersion'
./libpyfnhello.so: undefined reference to `PyObject_GC_UnTrack'

That means that you're missing some basic Python library. I don't know how that library is called (check the cython documentation again!) you need to add it using
Code: Pascal  [Select]
  1. {$LinkLib thelibraryname}
.

rnfpc

  • Jr. Member
  • **
  • Posts: 81
Re: Not able to use Python .so library function
« Reply #5 on: June 25, 2019, 02:38:24 pm »
Thanks for explanations on all these points. It seems importing .so files from Python needs many adjustments. Are .so files created from other languages like Rust, Go or Nim easier to use in Pascal?

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7234
Re: Not able to use Python .so library function
« Reply #6 on: June 25, 2019, 02:40:54 pm »
Thanks for explanations on all these points. It seems importing .so files from Python needs many adjustments. Are .so files created from other languages like Rust, Go or Nim easier to use in Pascal?

I'd expect roughly the same. You need to know what you are doing on both side. That goes probably for any communication between higher level languages (the above + C++).

The only exceptions are plain C level (which PascalDragon explained how to import on the FPC side, but you will need to research how to export them at the plain C level in the other language too), and, on Windows, COM.