Recent

Author Topic: Lazarus Package Manager - CLI package manager for buildsystems  (Read 6320 times)

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Hi,

as part of a project to build a fully automated building toolchain for lazarus projects, I needed a package manager for lazarus which is capable of loading OPM packages. So I've wrote one. And as some of you might also need something similar, here it is: https://github.com/Warfley/LazarusPackageManager.

What it does:
This little script (or set of scripts to be more accurate) allows for downloading and installing packages to any lazarus installation. It can search the OPM for packages, install packages via HTTP download, GIT or SVN.
It was build for being usable within docker, so there is also an example with a docker image for a project using DCP-Crypt

How to use:
Simple, assuming the lpm script is in the path:
Code: Bash  [Select][+][-]
  1. $> lpm update # synchronize OPM list
  2. $> lpm lazarus add 2.0.6 /developer/lazarus/2.0.6 # add lazarus installation so it knows where to install
  3. $> lpm install 2.0.6 "Synapse 40.1" # install synapse from OPM to lazarus
  4. $> lpm direct-download indy https://packages.lazarus-ide.org/Indy10.zip # manual download of a package (even though this is also in OPM)
  5. $> lpm install 2.0.6 indy # install just downloaded package
  6. $> lazbuild ... # now after installing the packages we can use lazbuild to simply build our projects
For simplicity there is also the build command, which will automatically try to download any missing packages from OPM:
Code: Bash  [Select][+][-]
  1. $> lpm update # synchronize OPM list
  2. $> lpm lazarus add 2.0.6 /developer/lazarus/2.0.6 # add lazarus installation so it knows where to install
  3. $> lpm build 2.0.6 project.lpi # add --yes or -y to not get asked to install new packages

What it does not:
  • Resolve dependencies (todo)
  • Rebuild Lazarus (this is deploy only, not intended for development IDE's, use OPM instead)
  • Remove packages (the main target are throwaway docker builds, so no need to remove these afterwards)
  • Install local packages (i.e. packages located in lazarus/components) (is this required?)

Requirements:
  • python3
  • unzip (if you want to use the OPM or download zip archives)
  • tar (if you want to download tar archives)
  • gzip (if you want to download tar.gz archives)
  • bzip2 (if you want to download tar.bz archives)
  • git (if you want to download git repositories)
  • svn (if you want to download svn repositories)
  • lazarus (obviously)
  • linux (I'm not going to test this on windows, I hate python under windows)

You might wonder why I wrote this in Python and not in Lazarus, thats because python has a really nice argument parser for CLI programs, and also this script is so small that I don't really need the structure of pascal :D
« Last Edit: April 04, 2020, 03:38:29 pm by Warfley »

soerensen3

  • Full Member
  • ***
  • Posts: 213
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #1 on: April 03, 2020, 11:09:58 am »
That's good news! I will try that later!

for installing local packages there is already lazbuild. However it lacks the feature of removing packages as well. This would be handy sometimes if you installed a package and broke your lazarus this way, because the package makes it crash.
Lazarus 1.9 with FPC 3.0.4
Target: Manjaro Linux 64 Bit (4.9.68-1-MANJARO)

valdir.marcos

  • Hero Member
  • *****
  • Posts: 1106
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #2 on: April 03, 2020, 02:41:04 pm »
Hi,

as part of a project to build a fully automated building toolchain for lazarus projects, I needed a package manager for lazarus which is capable of loading OPM packages. So I've wrote one. And as some of you might also need something similar, here it is: https://github.com/Warfley/LazarusPackageManager.

What it does:
This little script (or set of scripts to be more accurate) allows for downloading and installing packages to any lazarus installation. It can search the OPM for packages, install packages via HTTP download, GIT or SVN.
It was build for being usable within docker, so there is also an example with a docker image for a project using DCP-Crypt

How to use:
Simple, assuming the lpm script is in the path:
Code: Bash  [Select][+][-]
  1. $> lpm update # synchronize OPM list
  2. $> lpm lazarus add 2.0.6 /developer/lazarus/2.0.6 # add lazarus installation so it knows where to install
  3. $> lpm install 2.0.6 "Synapse 40.1" # install synapse from OPM to lazarus
  4. $> lpm direct-download indy https://packages.lazarus-ide.org/Indy10.zip # manual download of a package (even though this is also in OPM)
  5. $> lpm install 2.0.6 indy # install just downloaded package
  6. $> lazbuild ... # now after installing the packages we can use lazbuild to simply build our projects

What it does not:
  • Resolve dependencies (todo)
  • Rebuild Lazarus (this is deploy only, not intended for development IDE's, use OPM instead)
  • Remove packages (the main target are throwaway docker builds, so no need to remove these afterwards)
  • Install local packages (i.e. packages located in lazarus/components) (is this required?)

I currently tested this with Synapse and DCP-Crypt, two packages which are runtime only. While I think it should also work with design time packages, I don't know yet (I don't have any test projects), so if you want to try it, feedback would be appriceated


Requirements:
  • python3
  • unzip (if you want to use the OPM or download zip archives)
  • tar (if you want to download tar archives)
  • gzip (if you want to download tar.gz archives)
  • bzip2 (if you want to download tar.bz archives)
  • git (if you want to download git repositories)
  • svn (if you want to download svn repositories)
  • lazarus (obviously)
  • linux (I'm not going to test this on windows, I hate python under windows)

You might wonder why I wrote this in Python and not in Lazarus, thats because python has a really nice argument parser for CLI programs, and also this script is so small that I don't really need the structure of pascal :D
@Warfley
That's good news!
First, I appreciate and thank for you effort.

Second, just out of curiosity.
Would it be harder to use traditional Bash shell script instead of Python since you were already on Linux?
https://en.wikipedia.org/wiki/Shell_script
https://en.wikipedia.org/wiki/Bash_(Unix_shell)


That's good news! I will try that later!

for installing local packages there is already lazbuild. However it lacks the feature of removing packages as well. This would be handy sometimes if you installed a package and broke your lazarus this way, because the package makes it crash.
Very well observed. Thanks.

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #3 on: April 03, 2020, 06:25:00 pm »
Second, just out of curiosity.
Would it be harder to use traditional Bash shell script instead of Python since you were already on Linux?
https://en.wikipedia.org/wiki/Shell_script
https://en.wikipedia.org/wiki/Bash_(Unix_shell)

Yeah, it would be much harder. I'm actually quite a fan of bash scripts, but json parsing alone can give you a real headache, and these scripts actually only do 3 things: 1. parse json, 2. download files, 3. call lazbuild. While bash has a slight edge in ease for calling programs (and therefore downloading thanks to wget, curl or lynx), it is not much easier to weigh out the pain it would be to handle complex datastructures with it.

Python on the other hand supports OOP, which makes handling datastructures represented by json rather easy. Example to downlad a tar.gz and unzip it:
Code: Bash  [Select][+][-]
  1. # bash
  2. wget -O- url | tar -xz
  3. # python
  4. with request.open(url) as req:
  5.   p = Popen(["tar", "-xz"], stdin=req)
  6.   p.communicate()
Sure python is a little more complicated about this, but after all it's rather easy. Then when it comes to parsing json:
Code: Python  [Select][+][-]
  1. serialized = json.load(req)
  2. for key, value in serialized.items():
  3.   ...
While in bash I would need to either write some really dreadful sed queries or use specified tools and first convert the json to a string array which i can than iterate, and here python wins hands down

That's good news! I will try that later!

for installing local packages there is already lazbuild. However it lacks the feature of removing packages as well. This would be handy sometimes if you installed a package and broke your lazarus this way, because the package makes it crash.
Yeah I use internally simply lazbuilds -add-package-link to register the packages (I currently don't know if this is enough for design time packages, but from my understanding lazarus should build them with the project as long as they are registered), and the fact that my tool can't remove packages also relies on the fact that lazbuild can't do this.

When I break my lazarus with new components I usually use make to rebuild it withouth the components (simple make all) and then can start it and uninstall these components before rebuilding it from the IDE. This is especially funny when I'm using it on my raspi, because I don't use a DE on it and only connect via SSH, so to install packages I can easiely use lazbuild but for uninstalling I need to use X11 forwarding.

I really like lazbuild and think it's great to have a CLI interface for building lazarus projects, but it's missing some functionality.

I am also toying with the Idea to make this thing fully OPM compatible (i.e. using OPM configurations and locations) and integrate it directly into Lazbuild, I was looking into it, and decided that for now this is to complicated for what I want to accomplish, but is on my todo list for the future. The goal is to simply do something like calling "lazbuild opm install Indy10" or so.
But as I said, my current goal is to create a toolchain for building lazarus projects within docker, and a python script is more than enough for now.

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4467
  • I like bugs.
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #4 on: April 03, 2020, 07:30:12 pm »
I am also toying with the Idea to make this thing fully OPM compatible (i.e. using OPM configurations and locations) and integrate it directly into Lazbuild, I was looking into it, and decided that for now this is to complicated for what I want to accomplish, but is on my todo list for the future. The goal is to simply do something like calling "lazbuild opm install Indy10" or so.
Actually it should integrate directly into the Lazarus package system. The goal is that the different package sources could integrate seamlessly together. Meaning that a missing dependency of a local user installed package could be resolved by an online package, or vice versa.
Please see unit PackageLinkIntf in package BuildIntf (used to be in IdeIntf).
The package system knows 3 types of packages: Global, Online and User.
Global means a package in Lazarus sources. They are found based on .lpl files.
Online comes through OPM.
User installed packages can be anywhere in file system and their location is remembered by the IDE and LazBuild.

Unfortunately the online packages don't yet integrate fully. The internal data structure does not have enough data for solving dependencies. OPM uses its own logic from outside but it is not part the internal package system.
If somebody wants to improve the situation, please do.

Another thing is that the package system should not depend on LCL. My goal is to remove LCL dependency from LazBuild but that goal is far far away ...
Splitting BuildIntf from IdeIntf was one step but it does not help by itself. It provides interface for units which still have the LCL dependency.
The problem is now solved with a NoGUI widgetset hack which prevents pulling in the actual GUI libs.
« Last Edit: April 03, 2020, 11:47:22 pm by JuhaManninen »
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #5 on: April 03, 2020, 08:20:26 pm »
Another thing is that the package system should not depend on LCL. My goal is to remove LCL dependency from LazBuild but that goal is far far away ...
Splitting BuildIntf from IdeIntf was one step but it does not help by itself. It provides interface for units which still have the LCL dependency.
The problem is now solved with a NoGUI widgetset hack which prevents pulling in the actual GUI libs.
I was actually wondering why lazbuild depended on the LCL when looking into it. It was my first idea to use the IDE internals for this and started looking into how lazbuild handles dependencies, but decided that for what I've wanted to accomplish this was way to much effort.

But thank you for the info about the buildsystem units, I might look into it at some point.

balazsszekely

  • Guest
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #6 on: April 04, 2020, 09:53:28 am »
@Juha
Quote
Unfortunately the online packages don't yet integrate fully.
It does in my opinion. We did it togheter.  :) If you have an unresolved package dependency, the project inspector will show you that is avaible online, more over it offers you to install it. Unfortunately there is a small problem, many users did not liked the idea of internet connection on IDE startup, so it's diseabled by default.

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4467
  • I like bugs.
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #7 on: April 04, 2020, 01:54:56 pm »
@Juha
It does in my opinion. We did it togheter.  :) If you have an unresolved package dependency, the project inspector will show you that is avaible online, more over it offers you to install it.
Ok, my info was maybe outdated. I must study this integration thing again to bring it back.
Recently I have studied the package system for potentially reducing dependencies. It is a complex system, damn!

Quote
Unfortunately there is a small problem, many users did not liked the idea of internet connection on IDE startup, so it's diseabled by default.
Yes, that discussion I remember. :)
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #8 on: April 04, 2020, 03:36:51 pm »
Just added a new command: build

Builds projects using lazbuild, detects missing dependencies and tries to retrieve them via opm. Also added an example using indy for this.

Now compiling a project that only uses OPM dependencies is as simple as:
Code: Bash  [Select][+][-]
  1. $> lpm update
  2. $> lpm lazarus add 2.0.6 /developer/lazarus/2.0.6
  3. $> lpm build 2.0.6 project.lpi # use --yes or -y to avoid having to say yes to installing dependencies (e.g. in docker where no user input is possible)

skrat

  • New member
  • *
  • Posts: 8
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #9 on: December 17, 2021, 06:34:09 pm »
Hi,

First of all, thanks for this!

However, I have some trouble using the lpm. I have created a YML script:

Code: Pascal  [Select][+][-]
  1. # This is an example Starter pipeline configuration
  2. # Use a skeleton to build, test and deploy using manual and parallel steps
  3. # -----
  4. # You can specify a custom docker image from Docker Hub as your build environment.
  5.  
  6. image: debian:sid
  7.  
  8. pipelines:
  9.   default:
  10.     - step:
  11.         name: 'Environment'
  12.         script:
  13.           - echo "Setting environment..."
  14.           - apt-get update && apt-get install --yes lazarus-ide-qt5 unzip
  15.           - apt-get install git -y
  16.           - git clone https://github.com/Warfley/LazarusPackageManager.git
  17.           # - export PATH="$HOME/LazarusPackageManager:$PATH"
  18.           - cd LazarusPackageManager && ./lpm update && ./lpm lazarus add 2.0.12 /usr/lib/lazarus/2.0.12 && ./lpm install dcpcrypt-2.0.4.1 && lazbuild example/example.lpi
  19.     # - step:
  20.     #     name: 'Build and Test'
  21.     #     script:
  22.           - echo "Building project..."
  23.           - echo "$PATH"
  24.           - ./lpm update
  25.           - ./lpm lazarus add 2.0.12 /developer/lazarus/2.0.12 # add lazarus installation so it knows where to install
  26.           # - ./lpm direct-download fpspreadsheet http://packages.lazarus-ide.org/FPSpreadsheet.zip # manual download of a package (even though this is also in OPM)
  27.           # - ./lpm install 2.0.12 fpspreadsheet # install just downloaded package
  28.           # - ./lazbuild WingDesigner.lpi # now after installing the packages we can use lazbuild to simply build our projects
  29.           - ./lpm build -y ../WingDesigner.lpi
  30.  

Sadly the last line fails, saying:

Code: Pascal  [Select][+][-]
  1. [INFO] Downloading FPSpreadsheet.zip from https://packages.lazarus-ide.org/FPSpreadsheet.zip
  2. [INFO] Inflating FPSpreadsheet.zip
  3. [INFO] installing fpspreadsheet/laz_fpspreadsheet.lpk
  4. Traceback (most recent call last):
  5.   File "/opt/atlassian/pipelines/agent/build/LazarusPackageManager/./lpm", line 244, in <module>
  6.     main()
  7.   File "/opt/atlassian/pipelines/agent/build/LazarusPackageManager/./lpm", line 234, in main
  8.     success = performBuild(lpm, opm, args.project, args.modes, args.yes, logger)
  9.   File "/opt/atlassian/pipelines/agent/build/LazarusPackageManager/./lpm", line 149, in performBuild
  10.     performInstall(lpm, opm, [toInstall], logger)
  11.   File "/opt/atlassian/pipelines/agent/build/LazarusPackageManager/./lpm", line 33, in performInstall
  12.     result = result and lpm.installPackage(pkgName)
  13.   File "/opt/atlassian/pipelines/agent/build/LazarusPackageManager/packagemanager.py", line 113, in installPackage
  14.     result = result and call([lazbuild.resolve(), "--add-package-link", pkgFile.resolve()]) == 0
  15.   File "/usr/lib/python3.9/subprocess.py", line 349, in call
  16.     with Popen(*popenargs, **kwargs) as p:
  17.   File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
  18.     self._execute_child(args, executable, preexec_fn, close_fds,
  19.   File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
  20.     raise child_exception_type(errno_num, err_msg, err_filename)
  21. FileNotFoundError: [Errno 2] No such file or directory: PosixPath('/developer/lazarus/2.0.12/lazbuild')

Does anybody happen to know what the issue is or even better, how to fix it?

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: Lazarus Package Manager - CLI package manager for buildsystems
« Reply #10 on: January 06, 2022, 11:24:34 am »
Sorry for answering so late, haven't seen this.
The problem you have is that the Lazarus version you are installing is not located in /developer/lazarus/...

I recommend you using the docker images I provide via the docker hub, something like this:
Code: Pascal  [Select][+][-]
  1. # This is an example Starter pipeline configuration
  2. # Use a skeleton to build, test and deploy using manual and parallel steps
  3. # -----
  4. # You can specify a custom docker image from Docker Hub as your build environment.
  5.  
  6. image: warfley/lpm:stable # use lpm image
  7.  
  8. pipelines:
  9.   default:
  10.     - step:
  11.         name: 'Build and Test'
  12.         script:
  13.           - echo "Updating lpm..."
  14.           - lpm self-update && lpm update
  15.           - echo "Building Project..."
  16.           - lpm build -y WingDesigner.lpi
  17.  

 

TinyPortal © 2005-2018