Equivalent to stacking varargs

[MarkMLl]

The attached project is a small fragment illustrating static and dynamic linkage to the Python 2.7 runtimes, allowing a Python script to be embedded in a Pascal program. The python2/python.pas and python2/python_dynamic.pas files are machine-generated from ./python2/PythonDefs.inc, the full version of this file is currently about 325Kb since as well as active declarations it includes documentation in a form which works with Lazarus hover-over hints.

Static linkage works fine. Dynamic linkage works fine /provided/ that functions marked as varargs are commented out, see python2/python_dynamic.pas but in particular this:

Code: Pascal  [Select][+][-]

1. function TPython.PyArg_ParseTuple(args: PPyObject; format: PAnsiChar {;...}): Integer; cdecl varargs;
2.  
3. begin
4.   LoadRoutine(pointer(FPyArg_ParseTuple), 'PyArg_ParseTuple');
5.   result := FPyArg_ParseTuple(args, format)
6. end { TPython.PyArg_ParseTuple } ;
7.  

which if compiled using the makefile results in


lazbuild-trunk PythonDemo.lpi
/usr/local/share/lazarus-trunk/lazbuild --pcp=/home/markMLl/.lazarus-trunk PythonDemo.lpi
SetPrimaryConfigPath NewValue="/home/markMLl/.lazarus-trunk" -> "/home/markMLl/.lazarus-trunk"
Hint: (11030) Start of reading config file /etc/fpc.cfg
Hint: (11031) End of reading config file /etc/fpc.cfg
Free Pascal Compiler version 3.2.0 [2020/11/20] for x86_64
Copyright (c) 1993-2020 by Florian Klaempfl and others
(1002) Target OS: Linux for x86-64
(3104) Compiling PythonDemo.lpr
/home/markMLl/varargs/trunk/PythonDemo.lpr(8,2) Note: (2025) User defined: ===== Intending to link to Python v2 =====
(3104) Compiling ./python2/python_dynamic.pas
/home/markMLl/varargs/trunk/./python2/python_dynamic.pas(103,1) Error: (3178) VarArgs directive (or '...' in MacPas) without CDecl/CPPDecl/MWPascal/StdCall and External
/home/markMLl/varargs/trunk/./python2/python_dynamic.pas(104,3) Error: (5000) Identifier not found "LoadRoutine"
/home/markMLl/varargs/trunk/./python2/python_dynamic.pas(104,23) Error: (5000) Identifier not found "FPyArg_ParseTuple"
/home/markMLl/varargs/trunk/./python2/python_dynamic.pas(105,3) Error: (5000) Identifier not found "result"
/home/markMLl/varargs/trunk/./python2/python_dynamic.pas(105,13) Error: (5000) Identifier not found "FPyArg_ParseTuple"
/home/markMLl/varargs/trunk/./python2/python_dynamic.pas(105,31) Error: (5000) Identifier not found "args"
/home/markMLl/varargs/trunk/./python2/python_dynamic.pas(105,37) Error: (3026) Wrong number of parameters specified for call to "Format"
/usr/local/lib/fpc/3.2.0/units/x86_64-linux/rtl/sysutils.ppu:sysstr.inc(1095,10) Error: (5088) Found declaration: Format(const AnsiString;const Array Of Const):AnsiString;
/usr/local/lib/fpc/3.2.0/units/x86_64-linux/rtl/sysutils.ppu:sysstr.inc(1087,10) Error: (5088) Found declaration: Format(const AnsiString;const Array Of Const;const TFormatSettings):AnsiString;
/home/markMLl/varargs/trunk/./python2/python_dynamic.pas(106,34) Fatal: (2003) Syntax error, "." expected but ";" found
Fatal: (1018) Compilation aborted
Error: (lazarus) Compile Project, Target: PythonDemo-x86_64-linux: stopped with exit code 1
Error: (lazbuild) failed compiling of project /home/markMLl/varargs/trunk/PythonDemo.lpi
make: *** [Makefile:2: all] Error 2


The makefile invokes lazbuild-trunk hence FPC 3.2.0 on Debian Linux x86_64, but broadly speaking I get the same if using FPC directly.

I've obviously read the description of varags in the Reference, but would appreciate any suggestions as to how I should be working round this. I'd obviously prefer it if code that calls the relevant function could be unchanged between static linkage (where the varargs syntax appears OK) and dynamic linkage (where the above problem occurs), but so far I don't have an example of how that function is actually used to test against.

MarkMLl

[MarkMLl]

Later: I've tried converting the varargs modifier to an extra array-of-const parameter, but I'm still getting an error


(3104) Compiling ./python2/python_dynamic.pas
/home/markMLl/varargs/trunk/./python2/PythonDefs.inc(64,63) Error: (3178) VarArgs directive (or '...' in MacPas) without CDecl/CPPDecl/MWPascal/StdCall and External
/home/markMLl/varargs/trunk/./python2/python_dynamic.pas(80,1) Fatal: (10026) There were 1 errors compiling module, stopping
Fatal: (1018) Compilation aborted


where the offending line reads like

Code: Pascal  [Select][+][-]

1.     {$macro on }
2.     {$define LIBPYTHON__:= }
3. function PyArg_ParseTuple(args: PPyObject; format: PAnsiChar; aoc: array of const): Integer; cdecl; LIBPYTHON__
4.  

I presume that what's happening here is that the compiler is finding the combination of an array-of-const and cdecl and saying that it can't be used for an internal (Pascal-language) function.

I think that what I should be doing is defining the interface to the external library as cdecl but the internal shim function as using the Pascal calling convention, or (better) the external one as array-of-const+cdecl and the shim as array-of-const which would allow the caller to use the Object Pascal [] parameter list.

It's obviously a bit of extra work in the program that generates the interface units, but nothing insuperable since it's reasonable to expect the writer of the input file to be fairly careful with his definitions.

MarkMLl

[PascalDragon]

I think that what I should be doing is defining the interface to the external library as cdecl but the internal shim function as using the Pascal calling convention, or (better) the external one as array-of-const+cdecl and the shim as array-of-const which would allow the caller to use the Object Pascal [] parameter list.

It's obviously a bit of extra work in the program that generates the interface units, but nothing insuperable since it's reasonable to expect the writer of the input file to be fairly careful with his definitions.

That is the most useful solution. FPC does not support this, because every C compiler handles the implementation side differently, so we didn't want to bother with this. (And you can be sure if we'd provide support for implementing varargs routines that users would want their code to be useable from C and in Pascal we have array of const already anyway)

[MarkMLl]

I think that what I should be doing is defining the interface to the external library as cdecl but the internal shim function as using the Pascal calling convention, or (better) the external one as array-of-const+cdecl and the shim as array-of-const which would allow the caller to use the Object Pascal [] parameter list.

It's obviously a bit of extra work in the program that generates the interface units, but nothing insuperable since it's reasonable to expect the writer of the input file to be fairly careful with his definitions.

That is the most useful solution. FPC does not support this, because every C compiler handles the implementation side differently, so we didn't want to bother with this. (And you can be sure if we'd provide support for implementing varargs routines that users would want their code to be useable from C and in Pascal we have array of const already anyway)

Thanks for that. So if you could double-check my philosophy:

* Leave the static interface unchanged, i.e. the declarations have cdecl and possibly varargs.

* The field (entry point pointer) in the dynamic interface keep cdecl/varargs.

* The class declaration and implementation in the dynamic interface lose cdecl/varargs.

* Where necessary, the class declaration and implementation have varargs rewritten as array of const.

The simplest way of doing that would be to declare new macros CDECL__ and CDECL_VARARGS___ or similar.

Can you confirm that these examples from the Reference really are equivalent in all situations:

Code: Pascal  [Select][+][-]

1. Function PrintF1(fmt : pchar                       ); cdecl; varargs; external ’c’ name ’printf’;  
2. Function PrintF2(fmt : pchar; Args : Array of const); cdecl;          external ’c’ name ’printf’;
3.  

In MacPas mode, is "..." a precise equivalent of "array of const"?

The preprocessor is handling stuff fairly naively, but it works and saves a vast amount of manual editing. In the past I've used pretty much the same approach to also define the exports from a dynamically-linkable unit.

MarkMLl

[PascalDragon]

I think that what I should be doing is defining the interface to the external library as cdecl but the internal shim function as using the Pascal calling convention, or (better) the external one as array-of-const+cdecl and the shim as array-of-const which would allow the caller to use the Object Pascal [] parameter list.

It's obviously a bit of extra work in the program that generates the interface units, but nothing insuperable since it's reasonable to expect the writer of the input file to be fairly careful with his definitions.

That is the most useful solution. FPC does not support this, because every C compiler handles the implementation side differently, so we didn't want to bother with this. (And you can be sure if we'd provide support for implementing varargs routines that users would want their code to be useable from C and in Pascal we have array of const already anyway)

Thanks for that. So if you could double-check my philosophy:

* Leave the static interface unchanged, i.e. the declarations have cdecl and possibly varargs.

* The field (entry point pointer) in the dynamic interface keep cdecl/varargs.

* The class declaration and implementation in the dynamic interface lose cdecl/varargs.

* Where necessary, the class declaration and implementation have varargs rewritten as array of const.

The simplest way of doing that would be to declare new macros CDECL__ and CDECL_VARARGS___ or similar.

Can you confirm that these examples from the Reference really are equivalent in all situations:

Code: Pascal  [Select][+][-]

1. Function PrintF1(fmt : pchar                       ); cdecl; varargs; external ’c’ name ’printf’;  
2. Function PrintF2(fmt : pchar; Args : Array of const); cdecl;          external ’c’ name ’printf’;
3.  

In MacPas mode, is "..." a precise equivalent of "array of const"?

The preprocessor is handling stuff fairly naively, but it works and saves a vast amount of manual editing. In the past I've used pretty much the same approach to also define the exports from a dynamically-linkable unit.

In MacPas mode ... is an equivalent for declaring the function as varargs, not array of const. Also the two PrintFx functions behave the same on the ABI level, but they need to be called differently as the example in the Language Reference Guide showed further. All other points are correct assumptions however (at least as far as I understood them).

[nanobit]

Can anyone explain the found issue in shorter words?
I only know that in Delphi and FPC this should work without "external" (static):
var pyArg_ParseTuple: function( args: PPyObject; format: PAnsiChar {;...}): Integer; cdecl varargs;
result := pyArg_ParseTuple( args, pFmtPoint, @x, @y);

[MarkMLl]

In MacPas mode ... is an equivalent for declaring the function as varargs, not array of const. Also the two PrintFx functions behave the same on the ABI level,

Thanks for that. Allowing that it's strictly the array of const ... cdecl case that I'm interested in, I think that's adequate.

In the end I've not attempted to convert cdecl varargs; to array of const ... cdecl but have simply inserted a $warning that the usage is deprecated in that context... I think that covers all possibilities.

but they need to be called differently as the example in the Language Reference Guide showed further. All other points are correct assumptions however (at least as far as I understood them).

The calling issue doesn't bother me at all since it's consistent behaviour, much as I argue that the community should be receptive to ideas from other languages on occasion.

(Somewhat later...)

Well, I'm fairly sure that the preprocessor output is as I want it but I'm left with


python_dynamic.pas(108,1) Error: Wrong type "Array Of Const" in array constructor
python_dynamic.pas(154) Fatal: There were 1 errors compiling module, stopping


with the indicated line being

Code: Pascal  [Select][+][-]

1.     103 function TPython.PyArg_ParseTuple(args: PPyObject; format: PAnsiChar; vaoc: array of const): Integer;
2.     104
3.     105 begin
4.     106   LoadRoutine(pointer(FPyArg_ParseTuple), 'PyArg_ParseTuple');
5.     107   result := FPyArg_ParseTuple(args, format, vaoc)
6.     108 end { TPython.PyArg_ParseTuple } ;
7.  

I've found http://free-pascal-general.1045716.n5.nabble.com/passing-quot-array-of-const-quot-into-cdecl-procvar-td5725304.html and a linked StackOverflow question which suggests that while it's an issue with Delphi it shouldn't be with FPC... I tack on my current files in case anybody wants to tell me I'm being silly and will come back to it slightly later.

MarkMLl

[MarkMLl]

Can anyone explain the found issue in shorter words?
I only know that in Delphi and FPC this should work without "external" (static):
var pyArg_ParseTuple: function( args: PPyObject; format: PAnsiChar {;...}): Integer; cdecl varargs;
result := pyArg_ParseTuple( args, pFmtPoint, @x, @y);

It's not so much an "issue" but a question of coding style. What you show in your example assumes that the variable pyArg_ParseTuple has been preinitialised, which can be done either by doing them all when the program starts or by loading each routine on demand. In the latter case that can best be done by wrapping the library call in a method i.e. something like Python.pyArg_ParseTuple(), but that means passing the method's array of const parameter to the library routine i.e. "stacking vararg" or "stacking array of const" parameters.

The bigger picture is that I'm attempting to automate generating interface units, which can get to be a helluva job for even a small external library. I worked out a layout which could be used for both static and dynamic linkage (and also for generating a library) years ago, but it still meant that I had to edit the entry point definition, type definition, variable, variable assignment etc. manually... most of that's just hack work and there's absolutely no reason why it shouldn't be automated. It works in almost all cases...

MarkMLl

[PascalDragon]

Well, I'm fairly sure that the preprocessor output is as I want it but I'm left with


python_dynamic.pas(108,1) Error: Wrong type "Array Of Const" in array constructor
python_dynamic.pas(154) Fatal: There were 1 errors compiling module, stopping


with the indicated line being

Code: Pascal  [Select][+][-]

1.     103 function TPython.PyArg_ParseTuple(args: PPyObject; format: PAnsiChar; vaoc: array of const): Integer;
2.     104
3.     105 begin
4.     106   LoadRoutine(pointer(FPyArg_ParseTuple), 'PyArg_ParseTuple');
5.     107   result := FPyArg_ParseTuple(args, format, vaoc)
6.     108 end { TPython.PyArg_ParseTuple } ;
7.  

I've found http://free-pascal-general.1045716.n5.nabble.com/passing-quot-array-of-const-quot-into-cdecl-procvar-td5725304.html and a linked StackOverflow question which suggests that while it's an issue with Delphi it shouldn't be with FPC... I tack on my current files in case anybody wants to tell me I'm being silly and will come back to it slightly later.

I should have mentioned this: it's not possible (for the compiler) to convert a Pascal-style array of const for a call to a varargs function. You can use code like the one in the mentioned StackOverflow question, but you'll have to implement it for each platform yourself.

[MarkMLl]

I should have mentioned this: it's not possible (for the compiler) to convert a Pascal-style array of const for a call to a varargs function. You can use code like the one in the mentioned StackOverflow question, but you'll have to implement it for each platform yourself.

Ah yes... you have an enviable record of hitting the nail on the head :-)

OK, I'll mark that bit of generated code with $error for the moment. It appears to only affect a half dozen of the (so-far translated) Python entry points although there might obviously be more in the debugging etc. stuff which I hope to be looking at presently, and is not something that I'd be likely to put into a locally-written plugin or backend.

I'll comment here when I have a better fix, in case anybody wants a copy of the preprocessor.

MarkMLl

[nanobit]

What you show in your example assumes that the variable pyArg_ParseTuple has been preinitialised, which can be done either by doing them all when the program starts or by loading each routine on demand. In the latter case that can best be done by wrapping the library call in a method i.e. something like Python.pyArg_ParseTuple(), but that means passing the method's array of const parameter to the library routine i.e. "stacking vararg" or "stacking array of const" parameters.

Ah, I see now, the thread is about a compiler limitation: varargs call cannot be wrapped nicely.
(Pascal-ish would be only "case length(vaoc) of", thus only for a limited choice of calls )

Then I would not write wrappers for them, but initialize the function pointers earlier,
actually I would initialize all pointers in the common first call (probably in a wrapper of Py_Initialize or earlier)

[MarkMLl]

Ah, I see now, the thread is about a compiler limitation: varargs call cannot be wrapped nicely.

Maybe. I definitely don't want to appear to be criticising the compiler developers here.

In practical terms I can almost certainly work round it using a locally-defined Pascal function that takes a fixed number of parameters and then calls the C library entry point via the initialised variable, but that might end up making the static- and dynamically-linked programs different. I'll take more of a look at this presently.

Of course, the joke here is that the sort of language preprocessing I'm doing is easy enough with Pascal or C/C++, but /far/ more hairy with Python since it doesn't have begin/end or any reasonable equivalent.

Then I would not write wrappers for them, but initialize the function pointers earlier,
actually I would initialize all pointers in the common first call (probably in a wrapper of Py_Initialize or earlier)

See the example I posted. The pointers (strictly, procedure variables since they're typed) can be initialised during wrapper object creation, or it can be deferred.

Once the variable is initialised it can obviously be called by local helper functions.

MarkMLl

[nanobit]

Ok, but I considered to provide the variable/property directly (without wrapper method),
thus clients can call it without predefined number of arguments.
property PyArg_ParseTuple: TPyArg_ParseTuple read FPyArg_ParseTuple;

[MarkMLl]

Ok, but I considered to provide the variable/property directly (without wrapper method),
thus clients can call it without predefined number of arguments.
property PyArg_ParseTuple: TPyArg_ParseTuple read FPyArg_ParseTuple;

Thus losing all type checking :-/

OK, that sort of thing has to be done on occasion, but it's a strong incentive for having something like Modula-3's scheme where unsafe operations could be isolated.

MarkMLl

[MarkMLl]

I should have mentioned this: it's not possible (for the compiler) to convert a Pascal-style array of const for a call to a varargs function. You can use code like the one in the mentioned StackOverflow question, but you'll have to implement it for each platform yourself.

I've modified my preprocessor to treat procedures/functions with varargs specially: if dynamic linkage is being used they're converted to properties but one has to call LoadVarargsRoutine() explicitly.

Without intending to be in the least critical, I note for completeness that declarations like

Code: Pascal  [Select][+][-]

1. (* Write the output string described by format to sys.stdout. No exceptions are
2. raised, even if truncation occurs (see below).
3.  
4. format should limit the total size of the formatted output string to 1000 bytes
5. or less – after 1000 bytes, the output string is truncated. In particular, this
6. means that no unrestricted “%s” formats should occur; these should be limited
7. using “%.<N>s” where <N> is a decimal number calculated so that <N> plus the
8. maximum size of other formatted text does not exceed 1000 bytes. Also watch out
9. for “%f”, which can print hundreds of digits for very large numbers.
10.  
11. If a problem occurs, or sys.stdout is unset, the formatted message is written
12. to the real (C level) stdout.
13. *)
14. procedure PySys_WriteStdout(format: pchar); CDECL_VARARGS__ LIBPYTHON__
15. // procedure PySys_WriteStdout(format: pchar; vaoc: array of const); CDECL__ LIBPYTHON__
16. // void PySys_WriteStdout(const char *format, ...)
17.  
18. (* As above, but write to sys.stderr or stderr instead.
19. *)
20. // procedure PySys_WriteStderr(format: pchar); CDECL_VARARGS__ LIBPYTHON__
21. procedure PySys_WriteStderr(format: pchar; vaoc: array of const); CDECL__ LIBPYTHON__
22. // void PySys_WriteStderr(const char *format, ...)
23.  

result in Pascal-style entry points which need to be called differently:

Code: Pascal  [Select][+][-]

1.   Python.LoadVarargsRoutine('PySys_WriteStdout');
2.   if Assigned(Python.PySys_WriteStdout) then
3.     Python.PySys_WriteStdout('%s %s\n', 'Test message to', 'StdOut');
4.   Python.LoadVarargsRoutine('PySys_WriteStderr');
5.   if Assigned(Python.PySys_WriteStderr) then
6.     Python.PySys_WriteStderr('%s %s\n', ['Test message to', 'StdErr']);
7.  

This results in output like


Python is dynamically linked, and has been successfully loaded.
2.7.16 (default, Oct 10 2019, 22:02:15)
[GCC 8.3.0]
Test message to StdErr\nTest message to StdOut\n['/usr/local/src/pythonForFpc/trunk/PythonDemo-x86_64-linux']
Hello, World!


Since I'm not going via a shim procedure I can't convert the \n to an EOL: I don't see that as a significant issue since the parameter is being parsed by Pascal (count the quotes) rather than Python or C/C++.

Hadjab MarkMLl