Recent

Author Topic: Procedure as a record field  (Read 729 times)

peardox

  • New Member
  • *
  • Posts: 33
Procedure as a record field
« on: September 11, 2019, 03:44:58 pm »
https://macpgmr.github.io/MacXPlatform/PascalForTensorFlow.html includes this definition which is required to load an external TensorFlow model file

Code: Pascal  [Select]
  1. type
  2.   TF_Buffer = record
  3.     data: pointer;
  4.     length: csize_t;
  5.     data_deallocator: procedure (data: pointer; length: csize_t); cdecl;
  6.   end;
  7. type
  8.   TF_BufferPtr = ^TF_Buffer;
  9.  

I've got my data_deallocator function as follows

Code: Pascal  [Select]
  1. procedure DeallocateBuffer(data: Pointer; fsize: csize_t);
  2. begin
  3.   Free(data);
  4. end;


Can't work out how to assign it to the record, surely it should be something along the lines of

Code: Pascal  [Select]
  1. var
  2.   rec: TF_BufferPtr;
  3. begin
  4.  
  5.   <Code to read data into data buffer>
  6.  
  7.   rec^.length := fsize;
  8.   rec^.data :=data;
  9.   rec^.data_deallocator := @DeallocateBuffer;
  10. end;
  11.  

It's essentially a callback stored in a record methinks

Is the library wrong or is there some new FPC feature that allows this?

jamie

  • Hero Member
  • *****
  • Posts: 1987
Re: Procedure as a record field
« Reply #1 on: September 11, 2019, 04:32:21 pm »
You have created a pointer type record or you have declared one but have not created any memory to hold it so it's going to bomb anyways.

 In this case I don't see why you actually need that? Use the non pointer version in a location where it will always live, like in the main variable section for example.

Var
 rec:TF_Buffer;


 In code somewhere.

 Rec.Length := ?
 
 and the rest....

 If the call back reference needs an address to it you can provide that via the "@Rec" etc.

 need more info?

peardox

  • New Member
  • *
  • Posts: 33
Re: Procedure as a record field
« Reply #2 on: September 11, 2019, 07:45:38 pm »
@jamie

Can't do that, it's a callback - the callback is in tensorflow.dll / .so/ .dylib (which is written in C)

Not in the extract is a Malloc for xMB of memory - that's why the de-allocator has a Free in it

When Tensorflow has finished with the buffer it calls the address in data_deallocator passing the params supplied in the record.

A valid way suggested in 2010 (old forum posts) was to make TF_Buffer an object (should work)

Pascal2Tensorflow, however, defines it as stated above and there have been new features added to FPC since 2010 - one may be procedures as record fields (it's possible). Well, either that or Pascal2Tensorflow is wrong (H2Pas possibly?)

Simply trying to find the correct way to do this...

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7432
Re: Procedure as a record field
« Reply #3 on: September 11, 2019, 08:29:44 pm »
1. you don't mention the exact errormessage
2. the declaration of the record is cdecl, but the procedure is standard calling convention.

jamie

  • Hero Member
  • *****
  • Posts: 1987
Re: Procedure as a record field
« Reply #4 on: September 11, 2019, 08:43:35 pm »
@peardox
 
  So you telling us this record lives in a remote Module ?
   
  and @marcov has a valid point  :D


peardox

  • New Member
  • *
  • Posts: 33
Re: Procedure as a record field
« Reply #5 on: September 11, 2019, 09:47:45 pm »
No - the record is local, Tensorflow is dynamically loaded

Tensorflow needs an address to call using cdecl passing a pointer and int64 so caller can free allocated memory when no longer required.

The (mostly) C sample I'm working from is at https://github.com/Neargye/hello_tf_c_api/blob/master/src/load_graph.cpp

The somewhat convoluted way it works is you have to read in a Tensorflow model, create and assign it to a TF_Buffer, get TF to do something with it and finally call TF_DeleteBuffer - this is the vital bit... Tensorflow calls the caller to dispose of the buffer then presumably does some internal housekeeping.

Once this works Tensorflow becomes a big black box I can throw images at and get different images out. Stuff like image background removal, style transfer etc - there are loads of Python examples that do this (I'm not releasing a program with Python as a dependency - nightmare!)

jamie

  • Hero Member
  • *****
  • Posts: 1987
Re: Procedure as a record field
« Reply #6 on: September 11, 2019, 10:12:26 pm »
I just looked at that C++ file, good luck, I think you'll need it.

I see already that maybe off on the wrong foot.

jamie

  • Hero Member
  • *****
  • Posts: 1987
Re: Procedure as a record field
« Reply #7 on: September 11, 2019, 10:34:39 pm »
I just looked closer, it looks like all you need is a generic pointer and set the address to the procedure at hand.

 You need to ensure the procedure is correct of course.
data_deallocator:Pointer;

Data_Deallocator := @TheProcedure;

I think that is it..
you need to supply the procedure in your code which must match the C version.

PascalDragon

  • Hero Member
  • *****
  • Posts: 616
  • Compiler Developer
Re: Procedure as a record field
« Reply #8 on: September 12, 2019, 09:16:28 am »
I've got my data_deallocator function as follows

Code: Pascal  [Select]
  1. procedure DeallocateBuffer(data: Pointer; fsize: csize_t);
  2. begin
  3.   Free(data);
  4. end;

As marcov wrote, declare it as cdecl. Otherwise show a full, self contained example that should compile as well as the error message.

peardox

  • New Member
  • *
  • Posts: 33
Re: Procedure as a record field
« Reply #9 on: September 12, 2019, 03:55:07 pm »
Did all that and a few extra things

It now SIGSEGVs on closing form :(

I then moved the form oncreate code to a button onclick and it dies a completely different way!

Time to re-install Laz methinks (not many other plans come to mind)

jamie

  • Hero Member
  • *****
  • Posts: 1987
Re: Procedure as a record field
« Reply #10 on: September 12, 2019, 05:04:06 pm »
The procedure you create must be StdCall or  Cdecl , what ever the Library expects and you can't use a Class method unless it is labeled as a class type and calling convention specified.

 Otherwise, all calls in Fpc are Registered based so this will crash if you don't take these steps
I've got my data_deallocator function as follows

Code: Pascal  [Select]
  1. procedure DeallocateBuffer(data: Pointer; fsize: csize_t); StdCall or Cdecl here ?
  2. begin
  3.   Free(data);
  4. end;

« Last Edit: September 12, 2019, 05:06:54 pm by jamie »

peardox

  • New Member
  • *
  • Posts: 33
Re: Procedure as a record field
« Reply #11 on: September 12, 2019, 06:57:48 pm »
Well, that (hopefully) explains a lot

My C is a bit rusty

The entire Pascal library defines everything as cdecl but the official C headers are all stdcall

Earlier I moved the testing code from oncreate to onclick and things went BADLY wrong

This has been driving me crazy

Time to cross everything...

peardox

  • New Member
  • *
  • Posts: 33
Re: Procedure as a record field
« Reply #12 on: September 12, 2019, 08:39:32 pm »
And now it can't find the library

Giving it a go on the Mac I get more detailed error messages and this looks like a name mangling issue

When everything was cdecl it found them all and overwrote a load of stuff

Now it's stdcall it appears to be using mangled names (but at least it shouldn't mess up memory)

Well, that's how it looks to me...

e.g.

Code: Pascal  [Select]
  1. Undefined symbols for architecture x86_64:
  2.   "_TENSORFLOW_$$_TFE_DELETECONTEXT$TFE_CONTEXTPTR$TF_STATUSPTR", referenced from:
  3.       _TF_$$_UNINITCONTEXT$TFE_CONTEXTPTR in TF.o
  4.   "_TENSORFLOW_$$_TFE_DELETECONTEXTOPTIONS$TFE_CONTEXTOPTIONSPTR", referenced from:
  5.       _TF_$$_INITCONTEXT$$TFE_CONTEXTPTR in TF.o
  6.   "_TENSORFLOW_$$_TFE_DELETEOP$TFE_OPPTR", referenced from:
  7.       _TF$_$TTENSOR_$__$$_EXECOP$ANSISTRING$TTENSOR$TTENSOR$TTENSOR$$TTENSOR in TF.o
  8.   "_TENSORFLOW_$$_TFE_DELETETENSORHANDLE$TFE_TENSORHANDLEPTR", referenced from:
  9.       _TF$_$TTENSOR_$__$$_DESTROY in TF.o
  10.   "_TENSORFLOW_$$_TFE_EXECUTE$TFE_OPPTR$TFE_TENSORHANDLEPTR$CINTPTR$TF_STATUSPTR", referenced from:
  11.       _TF$_$TTENSOR_$__$$_EXECOP$ANSISTRING$TTENSOR$TTENSOR$TTENSOR$$TTENSOR in TF.o
  12.   "_TENSORFLOW_$$_TFE_NEWCONTEXT$TFE_CONTEXTOPTIONSPTR$TF_STATUSPTR$$TFE_CONTEXTPTR", referenced from:
  13.       _TF_$$_INITCONTEXT$$TFE_CONTEXTPTR in TF.o
  14.   "_TENSORFLOW_$$_TFE_NEWCONTEXTOPTIONS$$TFE_CONTEXTOPTIONSPTR", referenced from:
  15.       _TF_$$_INITCONTEXT$$TFE_CONTEXTPTR in TF.o
  16.   "_TENSORFLOW_$$_TFE_NEWOP$TFE_CONTEXTPTR$PCHAR$TF_STATUSPTR$$TFE_OPPTR", referenced from:
  17.       _TF$_$TTENSOR_$__$$_EXECOP$ANSISTRING$TTENSOR$TTENSOR$TTENSOR$$TTENSOR in TF.o
  18.   "_TENSORFLOW_$$_TFE_NEWTENSORHANDLE$TF_TENSORPTR$TF_STATUSPTR$$TFE_TENSORHANDLEPTR", referenced from:
  19.       _TF$_$TTENSOR_$__$$_CREATE$INT64$array_of_INT64$formal$LONGINT$$TTENSOR in TF.o
  20.   "_TENSORFLOW_$$_TFE_OPADDINPUT$TFE_OPPTR$TFE_TENSORHANDLEPTR$TF_STATUSPTR", referenced from:
  21.       _TF$_$TTENSOR_$__$$_EXECOP$ANSISTRING$TTENSOR$TTENSOR$TTENSOR$$TTENSOR in TF.o
  22.   "_TENSORFLOW_$$_TFE_OPSETATTRBOOL$TFE_OPPTR$PCHAR$CHAR", referenced from:
  23.       _TF$_$TTENSOR_$__$$_EXECOP$ANSISTRING$TTENSOR$TTENSOR$TTENSOR$$TTENSOR in TF.o
  24.   "_TENSORFLOW_$$_TFE_OPSETATTRINT$TFE_OPPTR$PCHAR$INT64", referenced from:
  25.       _TF$_$TTENSOR_$__$$_EXECOP$ANSISTRING$TTENSOR$TTENSOR$TTENSOR$$TTENSOR in TF.o
  26.   "_TENSORFLOW_$$_TFE_OPSETATTRTYPE$TFE_OPPTR$PCHAR$INT64", referenced from:
  27.       _TF$_$TTENSOR_$__$$_EXECOP$ANSISTRING$TTENSOR$TTENSOR$TTENSOR$$TTENSOR in TF.o
  28.   "_TENSORFLOW_$$_TFE_TENSORHANDLEDATATYPE$TFE_TENSORHANDLEPTR$$INT64", referenced from:
  29.       _TF$_$TTENSOR_$__$$_GETDATATYPE$$INT64 in TF.o
  30.   "_TENSORFLOW_$$_TFE_TENSORHANDLEDIM$TFE_TENSORHANDLEPTR$LONGINT$TF_STATUSPTR$$INT64", referenced from:
  31.       _TF$_$TTENSOR_$__$$_GETDIM$LONGINT$$LONGINT in TF.o
  32.   "_TENSORFLOW_$$_TFE_TENSORHANDLENUMDIMS$TFE_TENSORHANDLEPTR$TF_STATUSPTR$$LONGINT", referenced from:
  33.       _TF$_$TTENSOR_$__$$_GETRANK$$LONGINT in TF.o
  34.   "_TENSORFLOW_$$_TFE_TENSORHANDLERESOLVE$TFE_TENSORHANDLEPTR$TF_STATUSPTR$$TF_TENSORPTR", referenced from:
  35.       _TF$_$TTENSOR_$__$$_GETSCALARCOUNT$$LONGINT in TF.o
  36.       _TF$_$TTENSOR_$__$$_GETDATA$formal$LONGINT$$BOOLEAN in TF.o
  37.   "_TENSORFLOW_$$_TF_ALLOCATETENSOR$INT64$CINT64PTR$LONGINT$QWORD$$TF_TENSORPTR", referenced from:
  38.       _TF$_$TTENSOR_$__$$_CREATE$INT64$array_of_INT64$formal$LONGINT$$TTENSOR in TF.o
  39.   "_TENSORFLOW_$$_TF_DATATYPESIZE$INT64$$QWORD", referenced from:
  40.       _TF$_$TTENSOR_$__$$_CREATE$INT64$array_of_INT64$formal$LONGINT$$TTENSOR in TF.o
  41.       _TF$_$TTENSOR_$__$$_GETSCALARCOUNT$$LONGINT in TF.o
  42.   "_TENSORFLOW_$$_TF_DELETEBUFFER$TF_BUFFERPTR", referenced from:
  43.       _LOADMODELUNIT_$$_TF_LOADMODEL$ANSISTRING$$LONGINT in loadmodelunit.o
  44.   "_TENSORFLOW_$$_TF_DELETEGRAPH$TF_GRAPHPTR", referenced from:
  45.       _LOADMODELUNIT_$$_TF_LOADMODEL$ANSISTRING$$LONGINT in loadmodelunit.o
  46.   "_TENSORFLOW_$$_TF_DELETEIMPORTGRAPHDEFOPTIONS$TF_IMPORTGRAPHDEFOPTIONSPTR", referenced from:
  47.       _LOADMODELUNIT_$$_TF_LOADMODEL$ANSISTRING$$LONGINT in loadmodelunit.o
  48.   "_TENSORFLOW_$$_TF_DELETESTATUS$TF_STATUSPTR", referenced from:
  49.       _TF_$$_INITCONTEXT$$TFE_CONTEXTPTR in TF.o
  50.       _TF_$$_UNINITCONTEXT$TFE_CONTEXTPTR in TF.o
  51.       _TF$_$TTENSOR_$__$$_CREATE$INT64$array_of_INT64$formal$LONGINT$$TTENSOR in TF.o
  52.       _TF$_$TTENSOR_$__$$_GETRANK$$LONGINT in TF.o
  53.       _TF$_$TTENSOR_$__$$_GETSCALARCOUNT$$LONGINT in TF.o
  54.       _TF$_$TTENSOR_$__$$_GETDIM$LONGINT$$LONGINT in TF.o
  55.       _TF$_$TTENSOR_$__$$_GETDATA$formal$LONGINT$$BOOLEAN in TF.o
  56.       ...
  57.   "_TENSORFLOW_$$_TF_DELETETENSOR$TF_TENSORPTR", referenced from:
  58.       _TF$_$TTENSOR_$__$$_CREATE$INT64$array_of_INT64$formal$LONGINT$$TTENSOR in TF.o
  59.       _TF$_$TTENSOR_$__$$_GETSCALARCOUNT$$LONGINT in TF.o
  60.       _TF$_$TTENSOR_$__$$_GETDATA$formal$LONGINT$$BOOLEAN in TF.o
  61.   "_TENSORFLOW_$$_TF_GETCODE$TF_STATUSPTR$$INT64", referenced from:
  62.       _LOADMODELUNIT_$$_TF_CHECKSTATUS$TF_STATUSPTR$$LONGINT in loadmodelunit.o
  63.       _TF$_$TTENSOR_$__$$_CHECKSTATUS$TF_STATUSPTR$$BOOLEAN in TF.o
  64.   "_TENSORFLOW_$$_TF_GRAPHIMPORTGRAPHDEF$crc2C6A85CB", referenced from:
  65.       _LOADMODELUNIT_$$_TF_LOADMODEL$ANSISTRING$$LONGINT in loadmodelunit.o
  66.   "_TENSORFLOW_$$_TF_MESSAGE$TF_STATUSPTR$$PCHAR", referenced from:
  67.       _LOADMODELUNIT_$$_TF_CHECKSTATUS$TF_STATUSPTR$$LONGINT in loadmodelunit.o
  68.       _TF$_$TTENSOR_$__$$_CHECKSTATUS$TF_STATUSPTR$$BOOLEAN in TF.o
  69.   "_TENSORFLOW_$$_TF_NEWBUFFER$$TF_BUFFERPTR", referenced from:
  70.       _LOADMODELUNIT_$$_TF_READFILE$ANSISTRING$$TF_BUFFERPTR in loadmodelunit.o
  71.   "_TENSORFLOW_$$_TF_NEWGRAPH$$TF_GRAPHPTR", referenced from:
  72.       _LOADMODELUNIT_$$_TF_LOADMODEL$ANSISTRING$$LONGINT in loadmodelunit.o
  73.   "_TENSORFLOW_$$_TF_NEWIMPORTGRAPHDEFOPTIONS$$TF_IMPORTGRAPHDEFOPTIONSPTR", referenced from:
  74.       _LOADMODELUNIT_$$_TF_LOADMODEL$ANSISTRING$$LONGINT in loadmodelunit.o
  75.   "_TENSORFLOW_$$_TF_NEWSTATUS$$TF_STATUSPTR", referenced from:
  76.       _LOADMODELUNIT_$$_TF_LOADMODEL$ANSISTRING$$LONGINT in loadmodelunit.o
  77.       _TF_$$_INITCONTEXT$$TFE_CONTEXTPTR in TF.o
  78.       _TF_$$_UNINITCONTEXT$TFE_CONTEXTPTR in TF.o
  79.       _TF$_$TTENSOR_$__$$_CREATE$INT64$array_of_INT64$formal$LONGINT$$TTENSOR in TF.o
  80.       _TF$_$TTENSOR_$__$$_GETRANK$$LONGINT in TF.o
  81.       _TF$_$TTENSOR_$__$$_GETSCALARCOUNT$$LONGINT in TF.o
  82.       _TF$_$TTENSOR_$__$$_GETDIM$LONGINT$$LONGINT in TF.o
  83.       ...
  84.   "_TENSORFLOW_$$_TF_TENSORBYTESIZE$TF_TENSORPTR$$QWORD", referenced from:
  85.       _TF$_$TTENSOR_$__$$_CREATE$INT64$array_of_INT64$formal$LONGINT$$TTENSOR in TF.o
  86.       _TF$_$TTENSOR_$__$$_GETSCALARCOUNT$$LONGINT in TF.o
  87.       _TF$_$TTENSOR_$__$$_GETDATA$formal$LONGINT$$BOOLEAN in TF.o
  88.   "_TENSORFLOW_$$_TF_TENSORDATA$TF_TENSORPTR$$POINTER", referenced from:
  89.       _TF$_$TTENSOR_$__$$_CREATE$INT64$array_of_INT64$formal$LONGINT$$TTENSOR in TF.o
  90.       _TF$_$TTENSOR_$__$$_GETDATA$formal$LONGINT$$BOOLEAN in TF.o
  91.   "_TENSORFLOW_$$_TF_TENSORTYPE$TF_TENSORPTR$$INT64", referenced from:
  92.       _TF$_$TTENSOR_$__$$_GETSCALARCOUNT$$LONGINT in TF.o
  93.   "_TENSORFLOW_$$_TF_VERSION$$PCHAR", referenced from:
  94.       _LOADMODELUNIT$_$TFORM1_$__$$_FORMCREATE$TOBJECT in loadmodelunit.o
  95. ld: symbol(s) not found for architecture x86_64
  96. An error occurred while linking
  97. LoadModel.lpr(23) Error: (9013) Error while linking
  98. LoadModel.lpr(23) Fatal: (10026) There were 1 errors compiling module, stopping
  99. Fatal: (1018) Compilation aborted
  100. Error: /Users/simon/Applications/Lazarus/fpc/bin/x86_64-darwin/ppcx64 returned an error exitcode
  101.  

peardox

  • New Member
  • *
  • Posts: 33
Re: Procedure as a record field
« Reply #13 on: September 12, 2019, 10:11:13 pm »
Having just changed 216 function / procedure definitions to include external name '<name>';

It still barfs at me

Giving up for the evening...

garlar27

  • Hero Member
  • *****
  • Posts: 615
Re: Procedure as a record field
« Reply #14 on: September 12, 2019, 10:50:49 pm »
AFAIK stdcall is for windows only and cdecl for the *nixes

you can do something like this:
Code: Pascal  [Select]
  1.     procedure DeallocateBuffer(data: Pointer; fsize: csize_t); {$IFDEF WINDOWS}StdCall{$ENDIF} {$IFDEF LINUX}Cdecl{$ENDIF};
  2.     begin
  3.       Free(data);
  4.     end;
  5. // or
  6.     procedure DeallocateBuffer(data: Pointer; fsize: csize_t); {$IFDEF WINDOWS}StdCall{$ELSE}Cdecl{$ENDIF};
  7.     begin
  8.       Free(data);
  9.     end;
  10.