* * *

Author Topic: PPL = fpc's BPL ?  (Read 1568 times)

x2nie

  • Sr. Member
  • ****
  • Posts: 478
  • Impossible=I don't know the way
    • impossible is nothing - www.x2nie.com
PPL = fpc's BPL ?
« on: May 16, 2018, 08:27:44 am »
Hi all ! :)


I am finding a way to bring my project LiteZarus (Lazarus too of course, if possible) to be more closely to Delphi7,
in the way like of fast compiling designtime package.


I mean, there shall an easy way to load/unload Lazarus packages (design time package too) IMHO,
just as similar as Delphi ability to quickly install/upgrade/uninstall *.BPL packages.
I think it worth to have this feature, especially for package developer.
Because "Time is competitive weapon" --Andrew S Grove.


So, I start learning ppdump and ppmove.
Luckily I found that FPC already has a tool to generate a PPL file which is less-more like the BPL.
here ftp://www.stud.tue.nl/mirrors/.www/fpc/tools/ppumove.html
Quote
ppumove collects one or several Free Pascal unit files and archives them in a static or shared library.


Also here: http://wiki.freepascal.org/Lazarus/FPC_Libraries#ppumove.2C_.ppu.2C_.ppl
Quote
The ppumove tool included with every FPC installation, converts one or several .ppu and .o files into a dynamic library. It does this by calling the linker to gather all .o files into a .so (windows: .dll) file and removes the .o filename references from the .ppu file. These new .ppu files are normally called .ppl files.


So finally we already have had BPL (PPL) for freepascal+Lazarus, didn't we?
-------------------------


Well, now, by the assumption of PPL = BPL = dynamic library  , Lazarus can easily load/unload/recompile a package without recompile the lazarus.exe. That is my big dream.

My Questions now:
1. Is there any sample / demo of PPL usage? I didn't found any
2. Can we load/unload PPL in runtime ? I assumed PPL is "dynamic library"


I am crazily happy reading below article, but do not sure if it still relevant:
Quote
For example:
You have the output directory of a package (where the .ppu files are):
ppumove -o packagename -e ppl *.ppu
This will convert all .ppu files into .ppl files and creates a libpackagename.so (windows: packagename.dll). Note that under Linux the prefix 'lib' is always prepended.This new library can already be used by other programming languages like C. Or by FPC programs by using the external modifiers. But the initialization/finalization sections must be called automatically. This includes the initialization/finalization of the heap manager. This means no strings or GetMem. Of course FPC programmers are spoiled and they can get more.
(in second link above)


It's maybe a long road to walk by, or almost impossible, but I think it is not impossible. 8-)
« Last Edit: May 16, 2018, 09:09:05 am by x2nie »
When you were logged in, you can see attachments.
Lazarus Trunk @ Windows7 64bit, XP 32bit, Debian under VirtualMachine

Leledumbo

  • Hero Member
  • *****
  • Posts: 7927
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: PPL = fpc's BPL ?
« Reply #1 on: May 16, 2018, 06:49:00 pm »
Your post remind me of the dynamic packages plan. AFAIR last time it was demonstrated to be working, but no idea what's next. PPUMove seems to be working, though. It didn't last time. I tested it on brook framework and I get a single .so containing all symbols in the corresponding .o. I just don't know how to link this .so with my app through the .ppl.

x2nie

  • Sr. Member
  • ****
  • Posts: 478
  • Impossible=I don't know the way
    • impossible is nothing - www.x2nie.com
Re: PPL = fpc's BPL ?
« Reply #2 on: May 16, 2018, 09:07:03 pm »
Your post remind me of the dynamic packages plan. AFAIR last time it was demonstrated to be working,
How? when? where? Is there any URL to learn/try with?
Any info (even at few/tiny amount)  about dynamic package would be very valuable for me 8-)


Quote
PPUMove seems to be working, though. I tested it on brook framework and I get a single .so containing all symbols in the corresponding .o.
Unfortunatelly it didn't work in my side (WinXP + FPC 3.0.0)
following previous command in console, the generated ppl is not a collection of PPU, instead the are generated as much as amount of PPU count. Let say from 10 ppu ~-> generated 10 ppl.
Perhaps ppumove is for linux only? >:D  There is no such .dll created in windows.
Well, ppumove could generate a sinble pmove.bat batch-file, but didn't work either :'(  (raised unknown error)
Quote
I just don't know how to link this .so with my app through the .ppl.
I guessed that the *.so is just common dynamic lib, so that might need some "export" things


Anyway, when I were a student, there are TPUMove (turbo-pascal-compiled/binary-unit-management) that can be used to remove/append a tpu into package. Perhaps it has similar purpose.?
« Last Edit: May 16, 2018, 09:12:31 pm by x2nie »
When you were logged in, you can see attachments.
Lazarus Trunk @ Windows7 64bit, XP 32bit, Debian under VirtualMachine

Leledumbo

  • Hero Member
  • *****
  • Posts: 7927
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: PPL = fpc's BPL ?
« Reply #3 on: May 17, 2018, 12:40:25 am »
How? when? where? Is there any URL to learn/try with?
Any info (even at few/tiny amount)  about dynamic package would be very valuable for me 8-)
If I'm not mistaken it was presented by Almindor and Sven in a Pascal event.
Unfortunatelly it didn't work in my side (WinXP + FPC 3.0.0)
following previous command in console, the generated ppl is not a collection of PPU, instead the are generated as much as amount of PPU count. Let say from 10 ppu ~-> generated 10 ppl.
That's how it's designed to work (see its documention below). So the idea is to distribute all the .ppl and the single .so, as opposed to all .ppu and .o. It could be that the compiler will be made possible to see .ppl as alternative to .ppu and it knows the connection to .so. So, for each unit in uses clause, if the compiler sees .ppl, that unit will be linked dynamically. Otherwise if it sees .ppu, it will be linked statically. It's just... not implemented yet.
Perhaps ppumove is for linux only? >:D  There is no such .dll created in windows.
Well, ppumove could generate a sinble pmove.bat batch-file, but didn't work either :'(  (raised unknown error)
No, it was simply an unfinished product. It should work on all platforms eventually.
I guessed that the *.so is just common dynamic lib, so that might need some "export" things
Not that simple. We have syntax for importing procedural stuffs, this .so contains class structures and their methods, which we don't have syntax support for. How does Delphi import .bpl things containing classes?
Anyway, when I were a student, there are TPUMove (turbo-pascal-compiled/binary-unit-management) that can be used to remove/append a tpu into package. Perhaps it has similar purpose.?
Yes, per its documentation.

PascalDragon

  • Full Member
  • ***
  • Posts: 130
  • Compiler Developer
Re: PPL = fpc's BPL ?
« Reply #4 on: May 19, 2018, 09:36:40 am »
First of, yes, PPL is FPC's equivalent of Delphi's BPL.

However I'd definitely not advice to use ppumove as that as some serious restrictions:
  • works only on targets that don't need indirect imports (e.g. Windows is out with that)
  • does not support dynamic loading
  • does not check whether the packages are still acceptable after a recompile

This is where true support for dynamic packages comes in which is still a WIP though definitely further than a few years ago. The groundwork of changing some references to indirect ones (e.g. RTTI and VMT data) to support the indirect imports required on Windows platforms is already done and the startup code of i386-win32, x86_64-win64, x86_64-linux and all Mac OS X targets is already converted to support dynamic packages.
Having the compiler compile an application that uses packages at compile time is already possible (if dynamic packages are enabled for that target and package files are available as well (e.g. "rtl", "rtl.objpas", "fcl.base", etc.)).
The big chunk missing is dynamically loading a package which requires some changes to the way units are managed inside an application to address the issue of units and their types to be dynamically added and removed during the application's lifetime.

x2nie

  • Sr. Member
  • ****
  • Posts: 478
  • Impossible=I don't know the way
    • impossible is nothing - www.x2nie.com
Re: PPL = fpc's BPL ?
« Reply #5 on: May 20, 2018, 05:00:57 am »
The big chunk missing is dynamically loading a package which requires some changes to the way units are managed inside an application to address the issue of units and their types to be dynamically added and removed during the application's lifetime.
I didn't really undestand what it is mean of? Your last paragraph is a bit complicated words for me.
By the way, I smell that is a challenge to Lazarus IDE to recognize any invalid package because of mismatch version of unit in contained in  that package.
Am I right?
When you were logged in, you can see attachments.
Lazarus Trunk @ Windows7 64bit, XP 32bit, Debian under VirtualMachine

PascalDragon

  • Full Member
  • ***
  • Posts: 130
  • Compiler Developer
Re: PPL = fpc's BPL ?
« Reply #6 on: May 20, 2018, 03:03:20 pm »
When a dynamic package is loaded using LoadPackages() or unloaded with UnloadPackage() there are various house keeping tasks that the RTL needs to do:
- initialize or uninitialize units as needed
- insert new types into the public type list for correct support in the Rtti unit
- insert new resource strings into the internal resource string list for correct support (plus notification for translation handling units so that they know that something new needs to be translated)
- some other things that I don't have in my sight right now

Additionally the compiler needs to generate meta data for each package so that the RTL can ensure that a loaded package (and its dependencies) is compatible with the running application.

Lazarus itself won't have to deal with that. It simply needs to handle LoadPackage raising an exception if a package is not compatible as the main work is done by the RTL.

x2nie

  • Sr. Member
  • ****
  • Posts: 478
  • Impossible=I don't know the way
    • impossible is nothing - www.x2nie.com
Re: PPL = fpc's BPL ?
« Reply #7 on: May 22, 2018, 06:01:42 pm »
Hi PascalDragon, I did read you two posts above again and again within days...
Very interesting.


While I read back the Sven Barth's work[1], that will make me think:  :-X somehow you are him: the guy behind dynamic package. If so, Welcome sir. ! 8-)


Anyway, my original idea (about BPL /dynamic package) was to make Lazarus having multiple form designer. This way, I might able to register two TButton; one in LCL package, and another one in TUI (text user interface/Console) package[2]. Two same class name contained in different unit. Just identical feature of the two TForm class that are both registered within Delphi that are contained in two unit: Forms.pas + QForms.pas (vcldesigner70.bpl + clxdesigner70.bpl).


You know, that idea may only applicable via dynamic package. I mean that may requires switch between 2 form designer; because each form designer will only distinct one class name. That was my idea, not the fpc/lazarus's ability nowadays.


What I said for sure is: there will much more useful idea for Lazarus when dynamic package were come to reality.


---------------------
Now, since we can't use ppumove+*.ppl any longer, and because your explanation is very promising to proof the talk, :
would you like to open a bit sample of application or tools or url that run dynamic package
using fpc/lazarus ? :-*


Further, because it said that Windows okay (20+ isn't yet[3]) perhaps I or someone else can improve of what fpc has done? because FYI, I IMHO BPL highly depends on dll file format (see "An In-Depth Look into the Win32 Portable Executable File Format" article [4]).
I mean, perhaps, is that what your obstacle to implement dynamic package for all OS/CPUs supported by fpc? (I wish I am wrong) :-[


>> If BPL is a mysterious magic for many people, then FPC's Dynamic Package will be a bigger magic to be revealed  8) [size=78%] [/size]




[1] https://svn.freepascal.org/cgi-bin/viewvc.cgi/branches/svenbarth/packages/?sortby=date&view=log
[2] https://forum.lazarus.freepascal.org/index.php/topic,18998.msg244432.html#msg244432
[3] https://forum.lazarus.freepascal.org/index.php?topic=30145.msg191420#msg191420
[4] https://msdn.microsoft.com/en-us/magazine/bb985992.aspx
When you were logged in, you can see attachments.
Lazarus Trunk @ Windows7 64bit, XP 32bit, Debian under VirtualMachine

x2nie

  • Sr. Member
  • ****
  • Posts: 478
  • Impossible=I don't know the way
    • impossible is nothing - www.x2nie.com
When you were logged in, you can see attachments.
Lazarus Trunk @ Windows7 64bit, XP 32bit, Debian under VirtualMachine

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 6323
Re: PPL = fpc's BPL ?
« Reply #9 on: May 23, 2018, 06:02:31 pm »
Anyway, my original idea (about BPL /dynamic package) was to make Lazarus having multiple form designer. This way, I might able to register two TButton; one in LCL package, and another one in TUI (text user interface/Console) package[2].

Not really.  You can't unload LCL in Lazarus, for hopefully obvious reasons.

x2nie

  • Sr. Member
  • ****
  • Posts: 478
  • Impossible=I don't know the way
    • impossible is nothing - www.x2nie.com
Re: PPL = fpc's BPL ?
« Reply #10 on: May 23, 2018, 08:47:30 pm »
Not really.  You can't unload LCL in Lazarus, for hopefully obvious reasons.
I think I know what you mean, marcov. : If Lazarus is a GUI application which that build upon widgetset building block, then it shall deadly depends on LCL. Without LCL = no GUI.
Sure It is still true, but my idea could be true too.


Let say, my earlier idea of mimicking Delphi is based the fact that by using Delphi we could have 2 different TForm that are declared in 2 unit : Forms.pas + QForms.pas. It is enabled because basically Delphi := Delphi.exe + DesignerXX.bpl;
Okay I know that the QForms.pas could been eliminated by current Lazarus version.
But TUI wouldn't became one of "widgetset", because Text-User-Interface stuffs contains some properties (like left/top,width/height) that are incompatible coordinate system to be polymorph-ed with any widgetset.
From this reason, I can't use merge TUI visual component into LCL.
Another reason is it is good to have TLabel,TButton,TScrollbox for TUI rather than TTuiLabel, TTuiButton, TTuiScrollbox in a situation that LCL is actually not exist (or not relevant by unit uses as per-form).


Well, I am using TUI here as IMHO unavoided sample case. But there will be another possibility such as TLabel,TButton,TEdit to be drag/drop inside PHP/HTML designer[1].
Hi, LAMW (android non LCL visual component) is also good to have its own TLabel,TButton,Tedit etc.


Now, back to the fact of  Delphi := Delphi.exe + DesignerXX.bpl;
we can borrow that concept to split Lazarus into 2 (two) big part, one is LazarusIDE.exe and the other is LazarusDesigner.bpl (*.dll packages).
Of course the Lazarus.exe (IDE) it self will be built using LCL (widgetset / windowing stuffs), but the form-designer is freely switchable anytime among LCL and non-LCL packages.
The switch of form designer package (and classes registered contained in it) is driven by activating of a form (form designer).
This way, any FindClass function call will be only limited in designer-package, so no ambiguous class name will be happened.


So, marcov, yes Lazarus.exe will always built upon LCL (including PropertyInspector + Configuration Dialogs etc.),
but form/component designer shall not hardly LCL based only; there is must be another choice freely available when is needed.
Even current Lazarus already has mediator for non LCL visual designer ( /lazarus/examples/designnonlcl/notlcldesigner.lpk ), but it not applicable for real complex non LCL visual component implementation: something like TTabPage couldn't be implemented yet.
This way, I need a real form designer for non LCL visual component rather than simulation/painting those component's face on the canvas of LCL's form (form designer).


All of my talk above is assumed/depending on the lazarus has ability to load package from external library dynamically.


Note:
1) I am not forcing the future of "Lazarus" to have such feature above...
It can also be implemented into another new Lazarus IDE prototype.
2) I believe LCL is great, but having non LCL designer is good too.
3) loading a class from one package to be inherited inside another package is amazing, but separating packages into many library not a must for me.
So, perhaps by having one LCLDesigner.dll (which will be recompiled when synedit.pas has been modified) is enough; and then reloaded into IDE.
This situation, Lazarus.exe then is not required for recompiling.
4) I am most probably not accurate yet, since I didn't try that any further.




[1] https://youtu.be/RIpxPvctE4E?t=31
When you were logged in, you can see attachments.
Lazarus Trunk @ Windows7 64bit, XP 32bit, Debian under VirtualMachine

x2nie

  • Sr. Member
  • ****
  • Posts: 478
  • Impossible=I don't know the way
    • impossible is nothing - www.x2nie.com
Re: PPL = fpc's BPL ?
« Reply #11 on: May 24, 2018, 08:46:32 am »
Your post remind me of the dynamic packages plan. AFAIR last time it was demonstrated to be working,
How? when? where? Is there any URL to learn/try with?
Any info (even at few/tiny amount)  about dynamic package would be very valuable for me 8-)
I did think that the experiment is still in incubator. I was wrong ! :'(
Today,  :o after several hours of tracking/looking for partial $IFDEF words from Sven Packages branches,
I found that that Sven Barth's  work has been merged back to trunk on 2 years ago:
https://github.com/graemeg/freepascal/commit/afbc33abcaf2f8d2d24c641adc0ae64c1db78fcb


How stupid I was, failed several time of compiling that fpc for two days using fpc.2.6.4 +fpc.3.0.0.rc2.
Now, I will download fpc.3.0.4 (the latest version) soon.


I still have no idea how to proof the dynamic package mechanism.
Any help, any demo *.lpr?
When you were logged in, you can see attachments.
Lazarus Trunk @ Windows7 64bit, XP 32bit, Debian under VirtualMachine

x2nie

  • Sr. Member
  • ****
  • Posts: 478
  • Impossible=I don't know the way
    • impossible is nothing - www.x2nie.com
Re: PPL = fpc's BPL ?
« Reply #12 on: May 24, 2018, 07:16:46 pm »
Well, it is my last chance and now is clear for me about something that previously seem as mysterious.


* 2015 the Dynamic Package plan were merged into fpc trunk (3.0.4)[1]
* 2016 only compile time packages work (meaning you can't load additional packages at runtime) [2]
* 2018 The big chunk missing is dynamically loading a package which requires some changes to the way units are managed inside an application to address the issue of units (and their types) to be dynamically added (and removed) during the application's lifetime. [3]


It mean that I can continue trying to realize my dream by just using fpc ver3.0.4 ! not the previous version.
And, I can start from any application, because Windows is one of supported target by Dynamic  Package feature.


Because it was a bit ouf of topic, I close the case.


Thanks you all ! 8-)


[1] https://github.com/graemeg/freepascal/commit/afbc33abcaf2f8d2d24c641adc0ae64c1db78fcb
[2] http://lists.freepascal.org/fpc-devel/2016-November/037438.html
[3] http://forum.lazarus.freepascal.org/index.php/topic,41248.msg286331.html#msg286331
When you were logged in, you can see attachments.
Lazarus Trunk @ Windows7 64bit, XP 32bit, Debian under VirtualMachine

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus