Recent

Author Topic: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript  (Read 5465 times)

DevilDevil

  • New Member
  • *
  • Posts: 49
CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« on: October 22, 2019, 12:30:34 pm »
Hi guys!

I am a developer of CrystalLUA: https://github.com/d-mozulyov/CrystalLUA
This is a library that allows you to bind your native code with scripts in the Lua language.

Now there is a desire to support not only Lua, but also Python, JavaScript. Therefore, a separation is now being made into binding with a virtual machine of the scripting language and an analysis of the RTTI, types, functions. To use RTTI and interact with native data, a separate library has been created: https://github.com/d-mozulyov/Tiny.Rtti

Honestly, I'm a Delphi developer, and I only make sure that the code compiles in Lazarus. I know that the FreePascal community is large, and it would be interesting for them to use native binding with scripts. Therefore, firstly, I urge all interested parties to contribute to the development of these libraries. And secondly, I would like to ask for help in the development of functionality that applies not only to scripts, but also to programming in general.

In Delphi there is a native function call functionality that is not in FreePascal. There is also the ability to create a virtual interface. I want to create the same functionality in Tiny.Invoke, but faster, available for FreePascal and with support for a wider list of calling conventions. I need a person who is well versed in assembler for different platforms and could take on the development of this functionality. I think it’s not necessary to write everything in pure assembler, you can take Clang or GCC and use the built-in assembly capabilities.

I will be glad to feedback and any help  :)

MvC

  • New Member
  • *
  • Posts: 25
    • Free Pascal Core team member
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #1 on: October 22, 2019, 02:41:32 pm »
FPC Trunk already has Invoke and TVirtualInterface

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #2 on: October 22, 2019, 02:56:25 pm »
In Delphi there is a native function call functionality that is not in FreePascal. There is also the ability to create a virtual interface. I want to create the same functionality in Tiny.Invoke, but faster, available for FreePascal and with support for a wider list of calling conventions. I need a person who is well versed in assembler for different platforms and could take on the development of this functionality. I think it’s not necessary to write everything in pure assembler, you can take Clang or GCC and use the built-in assembly capabilities.
As MvC wrote FPC trunk already supports both Invoke and TVirtualInterface though on most platforms the use of the libffi library is required as a function call manager (simply include unit ffi.manager in your project file), exceptions are Win64 as well as the register calling convention on i386 (patches are welcome to implement more platforms directly in FPC). I plan to have most of the work for that merged into the upcoming 3.2 before it's released.

Oh and just because I noticed it in your Tiny.Rtti: FPC 3.2 and newer use PPTypeInfo instead of PTypeInfo (like Delphi does) in preparation for dynamic packages (same also goes for some other such references). So you should recheck your code with FPC 3.2. And maybe also recheck with Trunk, cause some structures changed due to the addition of attributes (which we added in non-Delphi-compatible way). And you should also check your code on non-x86 platforms as these have different alignment requirements and at least FPC (don't know about Delphi) does not use packed record for the Rtti there.

DevilDevil

  • New Member
  • *
  • Posts: 49
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #3 on: October 22, 2019, 05:54:00 pm »
FPC Trunk already has Invoke and TVirtualInterface

I know
But "faster, available for FreePascal and with support for a wider list of calling conventions".

Tiny.Rtti key feature is the rejection of the use of standard RTTI inits. This approach will allow you to work uniformly with different compilers, increase productivity and reduce the size of binary files. In addition, I plan to use the Invoke implementation in some C projects.

DevilDevil

  • New Member
  • *
  • Posts: 49
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #4 on: October 22, 2019, 05:58:50 pm »
In Delphi there is a native function call functionality that is not in FreePascal. There is also the ability to create a virtual interface. I want to create the same functionality in Tiny.Invoke, but faster, available for FreePascal and with support for a wider list of calling conventions. I need a person who is well versed in assembler for different platforms and could take on the development of this functionality. I think it’s not necessary to write everything in pure assembler, you can take Clang or GCC and use the built-in assembly capabilities.
As MvC wrote FPC trunk already supports both Invoke and TVirtualInterface though on most platforms the use of the libffi library is required as a function call manager (simply include unit ffi.manager in your project file), exceptions are Win64 as well as the register calling convention on i386 (patches are welcome to implement more platforms directly in FPC). I plan to have most of the work for that merged into the upcoming 3.2 before it's released.

Oh and just because I noticed it in your Tiny.Rtti: FPC 3.2 and newer use PPTypeInfo instead of PTypeInfo (like Delphi does) in preparation for dynamic packages (same also goes for some other such references). So you should recheck your code with FPC 3.2. And maybe also recheck with Trunk, cause some structures changed due to the addition of attributes (which we added in non-Delphi-compatible way). And you should also check your code on non-x86 platforms as these have different alignment requirements and at least FPC (don't know about Delphi) does not use packed record for the Rtti there.

I publish information in order to help me, and not be overloaded with new work :)
But I do not mind if the operating time will be used in FPC or libffi.

P.S. It is PTypeInfoRef type using - universal RTTI type for both Delphi and FPC.

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #5 on: October 23, 2019, 09:27:18 am »
In Delphi there is a native function call functionality that is not in FreePascal. There is also the ability to create a virtual interface. I want to create the same functionality in Tiny.Invoke, but faster, available for FreePascal and with support for a wider list of calling conventions. I need a person who is well versed in assembler for different platforms and could take on the development of this functionality. I think it’s not necessary to write everything in pure assembler, you can take Clang or GCC and use the built-in assembly capabilities.
As MvC wrote FPC trunk already supports both Invoke and TVirtualInterface though on most platforms the use of the libffi library is required as a function call manager (simply include unit ffi.manager in your project file), exceptions are Win64 as well as the register calling convention on i386 (patches are welcome to implement more platforms directly in FPC). I plan to have most of the work for that merged into the upcoming 3.2 before it's released.

Oh and just because I noticed it in your Tiny.Rtti: FPC 3.2 and newer use PPTypeInfo instead of PTypeInfo (like Delphi does) in preparation for dynamic packages (same also goes for some other such references). So you should recheck your code with FPC 3.2. And maybe also recheck with Trunk, cause some structures changed due to the addition of attributes (which we added in non-Delphi-compatible way). And you should also check your code on non-x86 platforms as these have different alignment requirements and at least FPC (don't know about Delphi) does not use packed record for the Rtti there.

I publish information in order to help me, and not be overloaded with new work :)
But I do not mind if the operating time will be used in FPC or libffi.
And I provided information about the current state of affairs in reply to your request for help. ;)

P.S. It is PTypeInfoRef type using - universal RTTI type for both Delphi and FPC.
But your PTypeInfoRef is wrong for FPC 3.2 and newer. There you need to use PPTypeInfo for the value as well.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 11383
  • FPC developer.
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #6 on: October 23, 2019, 09:37:30 am »
I know
But "faster, available for FreePascal and with support for a wider list of calling conventions".

Any numbers there? How much faster is it then using (FPC trunk's) rtti unit  ?
 

Thaddy

  • Hero Member
  • *****
  • Posts: 14205
  • Probably until I exterminate Putin.
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #7 on: October 23, 2019, 09:50:21 am »
May I suggest that the speed of software is equivalent to the driver (programmer). Some drivers can not handle the speed. They crash.
Specialize a type, not a var.

abouchez

  • Full Member
  • ***
  • Posts: 110
    • Synopse
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #8 on: October 23, 2019, 12:41:29 pm »
If you are interested, there is already some Invoke mechanism in our Open Source https://github.com/synopse/mORMot/blob/master/SQLite3/mORMot.pas unit.

Some distinctive info:
- It allows to execute interface methods - not class methods (it is used internally for SOA or MVC calls) - and with the default calling convention
- It doesn't use TValue (or anything like a temporary storage), but directly some JSON as input/output
- It doesn't use the rtti.pas unit at all
- It works since Delphi 6 and up to latest 10.3.2 on Win32 and Win64
- It works with FPC trunk on Win32, Win64, Linux-386, Linux-x64, Darwin-386, Darwin-x64, Linux-Arm, Linux-Arm64 (currently some problems with ARM, but we will fix the regressions)
- It is thread-safe and fast
- There is interface stubbing/mocking, and some advanced features like call interception (e.g. for logging into a text file or a db)

May be useful in your work.

DevilDevil

  • New Member
  • *
  • Posts: 49
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #9 on: October 23, 2019, 03:38:48 pm »
And I provided information about the current state of affairs in reply to your request for help.

I will be glad if the extended RTTI appears in the FPC. But I do not use FPC. Perhaps you would be interested in investing in the development of this project. Perhaps work on this project will help in correcting FPC RTTI bugs.

But your PTypeInfoRef is wrong for FPC 3.2 and newer. There you need to use PPTypeInfo for the value as well.
I understood. Thanks. But I do not support compatibility with the FPC. Perhaps someone will be interested in this compatibility and send a merge request.

DevilDevil

  • New Member
  • *
  • Posts: 49
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #10 on: October 23, 2019, 03:53:01 pm »
I know
But "faster, available for FreePascal and with support for a wider list of calling conventions".

Any numbers there? How much faster is it then using (FPC trunk's) rtti unit  ?

May I suggest that the speed of software is equivalent to the driver (programmer). Some drivers can not handle the speed. They crash.

I am professionally engaged in high performance, you can see this by looking at my projects: https://github.com/d-mozulyov?tab=repositories
Not everyone needs high performance. Therefore, there is no problem in using standard RTTI units. But, for example, in tasks of active interaction between native code and scripts, the speed of function calls can be important. In addition to the slow TValue, the standard Invoke approach involves copying a lot of data into registers and not always justified return. In my library, the Invloke function will be pre-selected in such a way as to execute Invoke as quickly as possible. Regarding interceptions, I profess the same logic.

DevilDevil

  • New Member
  • *
  • Posts: 49
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #11 on: October 23, 2019, 03:54:22 pm »
If you are interested, there is already some Invoke mechanism in our Open Source https://github.com/synopse/mORMot/blob/master/SQLite3/mORMot.pas unit.

Some distinctive info:
- It allows to execute interface methods - not class methods (it is used internally for SOA or MVC calls) - and with the default calling convention
- It doesn't use TValue (or anything like a temporary storage), but directly some JSON as input/output
- It doesn't use the rtti.pas unit at all
- It works since Delphi 6 and up to latest 10.3.2 on Win32 and Win64
- It works with FPC trunk on Win32, Win64, Linux-386, Linux-x64, Darwin-386, Darwin-x64, Linux-Arm, Linux-Arm64 (currently some problems with ARM, but we will fix the regressions)
- It is thread-safe and fast
- There is interface stubbing/mocking, and some advanced features like call interception (e.g. for logging into a text file or a db)

May be useful in your work.

So heavy unit
But thanks a lot :)

PascalDragon

  • Hero Member
  • *****
  • Posts: 5446
  • Compiler Developer
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #12 on: October 24, 2019, 02:16:51 pm »
And I provided information about the current state of affairs in reply to your request for help.

I will be glad if the extended RTTI appears in the FPC. But I do not use FPC. Perhaps you would be interested in investing in the development of this project. Perhaps work on this project will help in correcting FPC RTTI bugs.
Sorry, but no, I'm busy enough working on FPC.

But your PTypeInfoRef is wrong for FPC 3.2 and newer. There you need to use PPTypeInfo for the value as well.
I understood. Thanks. But I do not support compatibility with the FPC. Perhaps someone will be interested in this compatibility and send a merge request.

I'm confused.  %) Your code already contains quite some {$ifdef fpc} and you also said this in your opening post:
Quote
I want to create the same functionality in Tiny.Invoke, but faster, available for FreePascal and with support for a wider list of calling conventions.
So why do you say now that you don't support compatibility with FPC?

DevilDevil

  • New Member
  • *
  • Posts: 49
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #13 on: October 24, 2019, 03:26:57 pm »
PascalDragon,

Quote
Honestly, I'm a Delphi developer, and I only make sure that the code compiles in Lazarus. I know that the FreePascal community is large, and it would be interesting for them to use native binding with scripts. Therefore, firstly, I urge all interested parties to contribute to the development of these libraries. And secondly, I would like to ask for help in the development of functionality that applies not only to scripts, but also to programming in general.

I have no time and motivation to support the FPC direction. But I will be glad if someone takes on the support of this direction. I will help with all I can.

DevilDevil

  • New Member
  • *
  • Posts: 49
Re: CrystalScripts: Delphi/FPC + Lua, Python and JavaScript
« Reply #14 on: March 24, 2020, 09:31:38 pm »
Hi there!

The other day, the next release of the Tiny.Rtti library took place.
This is a really interesting library, which also shows excellent performance!

(https://github.com/d-mozulyov/Tiny.Rtti/blob/master/data/InvokeBenchmark.png)
(https://github.com/d-mozulyov/Tiny.Rtti/blob/master/data/VirtualInterfaceBenchmark.png)

Low-level code (including assembler) is written in C and compiled by the Clang cross-compiler into object files.
The Delphi compiler easily links these files, but the FPC produces strange errors!
Let's take a simple example (attached to the post).

Code: C++  [Select][+][-]
  1. // test.c
  2. int add(int x, int y, int z)
  3. {
  4.     return (x + y + z);
  5. }

Code: [Select]
:: compile.bat
clang -target x86_64-windows-gnu -c -O3 test.c -o"test.o"
pause

Code: Pascal  [Select][+][-]
  1. program Project1;
  2.  
  3. {$APPTYPE CONSOLE}
  4. {$L test.o}
  5.  
  6. function add(X, Y, Z: Integer): Integer; external;
  7.  
  8. var
  9.   R: Integer;
  10. begin
  11.   R := add(1, 2, 3);
  12.   Writeln('1 + 2 + 3 = ', R);
  13.   Readln;
  14. end.  
 

This design works great in Delphi, but it doesn’t link in FreePascal:
Quote
Project1.dpr(14,1) Error: Undefined symbol: P$PROJECT1_$$_ADD$LONGINT$LONGINT$LONGINT$$LONGINT

Tell me, please, how to fix this?

 

TinyPortal © 2005-2018