Recent

Author Topic: [Tutorial] CI/CD with Lazarus and GitLab  (Read 2271 times)

Warfley

  • Hero Member
  • *****
  • Posts: 2040
[Tutorial] CI/CD with Lazarus and GitLab
« on: April 13, 2020, 12:11:18 am »
---
I have no idea if this is the correct subforum for this kind of stuff, if not please let me know!
---

Hey, over the past few weeks I've started to develop a CI/CD workflow for my Lazarus projects with the goal to automatically build, test and deploy them using GitLab.

I'm finally at a stage where most of my Programs successfully now use CI/CD to be built and released, so I've wanted to share my results with others, because I think that many people could benefit from this.

The tutorial can be found here: https://gitlab.com/Warfley/lazarusgitlabci/-/blob/master/tutorial.pdf

What's it about:
It shows how to create a CI/CD workflow for your program which automatically builds a program for Linux and Windows (with 3 different widgetsets), runs unittests and deploys the build results (using GitLabs Artifact feature)

One of the main targets of this workflow is to also get cross compiling easiely running, without having to setup each and every development PC for this (as someone who uses 2 different Computers with 2 different OS' each, I want to streamline the build process and release process as much as possible).

Hopefully this helps people. Have fun with it!

dbannon

  • Hero Member
  • *****
  • Posts: 3742
    • tomboy-ng, a rewrite of the classic Tomboy
Re: [Tutorial] CI/CD with Lazarus and GitLab
« Reply #1 on: April 13, 2020, 01:41:37 am »
Hmm, nice. And nicely documented too.  A couple of comments ?

Firstly, "build, text, deploy" ??  I see only "build" - I consider unit testing with apps heavy with GUI hard, hoped you had a solution.  :-)

Secondly, your build model may be a a touch more complicated that what I use.  I have a bash script on a local machine, nothing so fancy as your gitlab model.  It does not use OPM, all you need to do is download the packages, compile and mention where they are at final build stage.  I only use one additional package, KControls, I get that from its official repository rather than from the OPM repository but either works. So my model to build is -

Pull down, with wget, Lazarus, KControls and app source.
Build lazbuild and LCL with make (don't need eg Lazarus IDE)
fpc option.... option

Lazbuild is only needed to build KControls, the app itself is built with FPC.

I might mention this is not my production build system, just something I include in the src for people who want to build without seeing Lazarus.  For each release, I use another bash script that makes much more use of Lazbuild and build modes to make Linux and windows, 32 and 64 bit binaries and package them up into debs, rpms and a kit suited to pass to Inno Setup to make my Windows release package.

But unit testing, that would be nice.

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

Warfley

  • Hero Member
  • *****
  • Posts: 2040
Re: [Tutorial] CI/CD with Lazarus and GitLab
« Reply #2 on: April 13, 2020, 03:23:58 am »
Hmm, nice. And nicely documented too.  A couple of comments ?
Comments where? The program? this is not of any interest, it's only that there is some example, the whole Idea is that this works for any program. So the thing of interest is the gitlab-ci file. And this is documented by the tutorial. The application was just something I could hack together in 10 minutes for the purpose of this tutorial (And I don't want people to look into that source, it's pretty terrible tbh.).

Firstly, "build, text, deploy" ??  I see only "build" - I consider unit testing with apps heavy with GUI hard, hoped you had a solution.  :-)
If I had a solution for testig GUI's I'd probably have a Turing Award by now  :D. This is an open research question where companies like microsoft pour millions of dollars into to solve.
I wish I'd had the answer to this. But this is not the purpose of this tutorial, I'm assuming you have a project at hand, which already has tests (as we all add a unit test for each new functionality we develop  ;) ). This is about streamlining the process.

Sure if you don't have any tests, this doesnt do any testing. But if you have, you can simply push your code, and get an email if it fails the tests. Especially if you have a lot of tests that take some time you might not want to do them on your PC.

But you are right, the deployment aspect falls a little bit flat. I mostly build server sided programs where Docker-In-Docker can be used to build a docker image which is then used for updating by docker-compose. But explaining this would be way to complicated for this tutorial, so I simply sticked with uploading them to GitLab as artifacts.
I think if I have some time over the next few days I'll add a section how to use the GitLab artifacts to create a GitLab release which can be added to the Tag. This is rather simple and I think fits to the tone of the tutorial.

Secondly, your build model may be a a touch more complicated that what I use.  I have a bash script on a local machine, nothing so fancy as your gitlab model.  It does not use OPM, all you need to do is download the packages, compile and mention where they are at final build stage.  I only use one additional package, KControls, I get that from its official repository rather than from the OPM repository but either works. So my model to build is -

Pull down, with wget, Lazarus, KControls and app source.
Build lazbuild and LCL with make (don't need eg Lazarus IDE)
fpc option.... option

Lazbuild is only needed to build KControls, the app itself is built with FPC.
Sure this will work, but my goal was not to make "a buildsystem" but to create a process which results in a buildsystem. The idea is, you can take any Lazarus project, simply create the build modes (if they are not already created, I usually work with build modes even before this), copy that gitlab-ci.yml script, change the file names, and it works.

It seems complex, but now with this workflow I basically create an automated build system for any of my projects in 5-10 minutes.

This is more or less inspired by node-js applications, where to build and run their tests you only need to use the node docker container with "npm install && npm run compile && npm run tests" and my goal was to basically make compiling of lazarus projects as easy.

I might mention this is not my production build system, just something I include in the src for people who want to build without seeing Lazarus.  For each release, I use another bash script that makes much more use of Lazbuild and build modes to make Linux and windows, 32 and 64 bit binaries and package them up into debs, rpms and a kit suited to pass to Inno Setup to make my Windows release package.

But unit testing, that would be nice.

Davo
I do something similar with my Docker-In-Docker approach. There basically the publishing step starts a bash script which creates a new Docker images and pushes it to the registry. Meaning if a feature is completed I simply tag it, and then after around 10 minutes the container is online, and is picked up by my servers next update cycle.

Deploying this software is literally as simple as writing "git push" into my command line.

Another software I've written, a gui software, which requires a lot of configuration files, takes the exact same gitlab-ci file, but simply does in the publishing step the following:
1. Copy executables to a Release folder
2. Copy config templates/create configs
3. Zip it
4. Upload the zip as artifacts
I then only need to take the download link of the Artifacts and publish it as the new download link on a web page.

This is the beauty of GitLab CI, once you have this basic structure, with only a few bash lines, you can make anything you want with it.

This is not for building the "perfect" buildsystem, but to build a reproducible buildsystem that can be set up for any project within a few minutes.

dbannon

  • Hero Member
  • *****
  • Posts: 3742
    • tomboy-ng, a rewrite of the classic Tomboy
Re: [Tutorial] CI/CD with Lazarus and GitLab
« Reply #3 on: April 13, 2020, 03:59:46 am »
....
Comments where? The program? ....

No, no, sorry, by 'comments' I meant do you mind if I comment ?   And I hope not because I did comment !

Quote

If I had a solution for testig GUI's I'd probably have a Turing Award by now  :D....
Agreed ! Sadly, my only 'unit test' is did it compile ?  And did it link ?  With the GUI so attached to the data, thats how it is.  And if Lazarus has a weakness, its that it encourages that attached-ness.  I have, in the distant past worked on projects with enforced unit test policies. It helped, thats for sure.

Anyway, I like your model.

Davo
Lazarus 3, Linux (and reluctantly Win10/11, OSX Monterey)
My Project - https://github.com/tomboy-notes/tomboy-ng and my github - https://github.com/davidbannon

Warfley

  • Hero Member
  • *****
  • Posts: 2040
Re: [Tutorial] CI/CD with Lazarus and GitLab
« Reply #4 on: April 13, 2020, 04:21:28 am »
Yeah but i would say that the attached-ness is also one of its strengrhs. It allows to build simple programs really fast, but as these programs grow it really becomes a problem. Ive worked with objective C multiple years ago, where your GUI is completely detached from the data, and there it was the complete opposite, writing the simpelest programs was a real pain, but as the programs grow you dont as easily run into corners where you are limited by your gui choices.
But sometimes you just want to wrote a very simple program, and there is lazarus absolutely great

Developing with testability in mind or even as a goal (test driven development) can help, but more often then not i also start programming first and then find out that teating would require rewriting half of the program

But even with great tests, some things are simply not testable, i have a program that uses guis and a lot of threads, and i know of a bug that triggers a seg fault, but i cant reproduce it because it requires some special combination of ef events which might happen once every hour. Frequent enough to be really annoying, but to unpredictable to trace down
« Last Edit: April 13, 2020, 04:26:27 am by Warfley »

440bx

  • Hero Member
  • *****
  • Posts: 6155
Re: [Tutorial] CI/CD with Lazarus and GitLab
« Reply #5 on: April 13, 2020, 05:04:04 am »
i have a program that uses guis and a lot of threads, and i know of a bug that triggers a seg fault, but i cant reproduce it because it requires some special combination of ef events which might happen once every hour. Frequent enough to be really annoying, but to unpredictable to trace down
That sounds like an interesting problem to solve.  Is it a desktop application and does it run on Windows ? :)
FPC v3.2.2 and Lazarus v4.0rc3 on Windows 7 SP1 64bit.

 

TinyPortal © 2005-2018