* * *

Author Topic: Public daily build server  (Read 2431 times)

Chronos

  • Full Member
  • ***
  • Posts: 173
    • PascalClassLibrary
Public daily build server
« on: March 19, 2016, 11:54:30 am »
Is there any daily/on demand/automatic on new commit build server with public web interface so we can easily download new Lazarus or FPC binary for multiple platforms to for example check if fixed FPC bugreport works ok? Sure we can download code and run make manually but at least for combination Lazarus+FPC it is rather problematic on Linux as FPC and Lazarus are installed as special RPM/DEB packages and you can't just run make & make install for new FPC.

Also public build server would allow to watch and evaluate various builds metrics over the time. Or easily provide simple URL to download newer or older binaries or installation files.

I remember there were snapshots builds for Lazarus years ago but they stopped operating for some reason. I don't know if FPC team used daily builds at all or just used local builds and have some manual build script for release binaries preparation.

There is also wiki page about Jenkins http://wiki.freepascal.org/Jenkins which could be used as build server but there is no link for existed running server and preferably such server would be better to be written in pascal.

FPCUP http://wiki.freepascal.org/fpcup itself may be tool to easy already complicated build process of FPC and Lazarus but as application itself is not build server.

I had no success with FPC and Lazarus cross compilation http://wiki.freepascal.org/Cross_compiling to build binaries/installation for various different platforms from single Ubuntu 64-bit server. Also setup for cross complication Lazarus applications is so complicated in comparison to Delphi there you can just install support for Windows,iOS and Android and just change target and build FMX application without downloading additional files, recompiling Delphi libraries or changing you computer system files.

So question is if is there any such daily build server and if not then if there is any such software or additional description how to setup automated daily build server.

Leledumbo

  • Hero Member
  • *****
  • Posts: 7549
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Public daily build server
« Reply #1 on: March 19, 2016, 04:23:41 pm »
So question is if is there any such daily build server and if not then if there is any such software or additional description how to setup automated daily build server.
None that I know of. I think you should first get http://wiki.freepascal.org/Cross_compiling to work on your system, if you want the server to build for as many platforms as possible, otherwise you won't get far from single system build. Once you've made it, follow http://wiki.freepascal.org/Jenkins and make it work. I have a distro neutral script that works exclusively with svn checked out repo to build for the following targets:
  • i386-linux
  • x86_64-linux
  • i386-win32
  • x86_64-win64
  • jvm-java
  • jvm-android
  • arm-android
  • i8086-msdos
  • arm-linux
from x86_64-linux system. I use Manjaro Linux but any distro should be fine as long as you can get the required binutils for all targets (fortunately, AUR contains everything I need, some are even in the official distro package repo).

If you can get the system, I can help setting up the automatic build.

Chronos

  • Full Member
  • ***
  • Posts: 173
    • PascalClassLibrary
Re: Public daily build server
« Reply #2 on: March 19, 2016, 10:22:38 pm »
I have three physical servers available all installed with http://pve.proxmox.com/wiki/Main_Page where multiple virtual servers are running. I prefer Ubuntu as OS for desktops and virtual servers simply because it is most used and well user oriented. So I have Ubuntu 64-bit server up and running. Also tested Jenkins but it is not really suitable for FPC & Lazarus fully featured build server. If you want to build for many platforms and versions you can set up rules which will cause jenkins to do full svn checkout unnecessary many times which takes much space and time. Here distributed versioning system like git would help because it would be enough to full clone repo just once and do local small clones for single branch with depth 1 or optimize checkout is various other ways.
Also I would like to have build system where you could specify target architectures as well as all stable branches and tags and also allow on demand build for specific revisions. So Jenkins is not really proper solution here. Jenkins doesn't provide any significant feature which couldn't be easily implemented lets say in PHP or directly in FPC as CGI application. I will need more fine tuning which Jenkins doesn't allow.

So basic system is x86_64-linux. I tried do to cross complication with make which created some files but this is useless from user point of view. Such build server should not only do "make all" to report possible build problems but also create installation files. I am not aware if this is documented anywhere.

So script of yours which can build several architectures also produce these three installation files which are available here? https://sourceforge.net/projects/lazarus/files/Lazarus%20Linux%20amd64%20DEB/Lazarus%201.6/

What are steps to produce these files? I mean for all architectures/platforms.

Leledumbo

  • Hero Member
  • *****
  • Posts: 7549
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Public daily build server
« Reply #3 on: March 20, 2016, 08:14:00 am »
So basic system is x86_64-linux. I tried do to cross complication with make which created some files but this is useless from user point of view. Such build server should not only do "make all" to report possible build problems but also create installation files. I am not aware if this is documented anywhere.
The generic zipinstall target should work.
So script of yours which can build several architectures also produce these three installation files which are available here? https://sourceforge.net/projects/lazarus/files/Lazarus%20Linux%20amd64%20DEB/Lazarus%201.6/
Nope. It doesn't create any installable files, only build cross compilation tools for those targets in the host system.
What are steps to produce these files? I mean for all architectures/platforms.
Lazarus uses scripts available in $(LazarusDir)/tools/install, while FPC uses the one from fpcbuild repo (under install folder).

Chronos

  • Full Member
  • ***
  • Posts: 173
    • PascalClassLibrary
Re: Public daily build server
« Reply #4 on: March 20, 2016, 10:03:24 am »
FPC make zipinstall is basically useless from OS packaging system perspective. This feature is here probably just because creating archive is much simpler then OS dependent installation packages. Even as name suggest it should create zip archive but create tar.gz by default on Linux so even make target name is misleading.

Lazarus tools/install is only usable and also primary source for building install binaries. But it's shame that Lazarus need to create own FPC install packages because FPC itself can't. There are FPC packages in Ubuntu repo but they are simply mess. FPC itself is separated to many small packages which may be usable for apt-get install but not for manual upgrade from web. Nobody want to download 32(16) packages just because somebody had great idea to cut them to small packages. This not how real world works. And even if new FPC version comes then these all packages are upgraded so possible advantage to download only parts which were changed/upgraded is not real use case.

So at least Lazarus have just two FPC packages fpc and fpc-src. These are 3 packages in comparison to just one package for Windows which is still not perfect but after all FPC and Lazarus are different projects so if makes small sense.

This reminds me that there is still missing basic feature in Lazarus called something like "Make install package...". All Lazarus developers deal with same problem. They create application and want provide install packages for various platforms for their users. It may be Windows .msi package, Linux DEB/RPM/other, Apple/iOS package, Android apk or install file for other platforms. This should be part of FPC or Lazarus. As both FPC and Lazarus needs to be distributed in same way. Yet nobody implemented such feature.

So I will check if Lazarus/tools/install can be used for automated processing across multiple Lazarus/FPC versions and try to make web page with downloadable build.

Such build server would be useful tool not just for Lazarus/FPC itself but any project developed with them.

Leledumbo

  • Hero Member
  • *****
  • Posts: 7549
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Public daily build server
« Reply #5 on: March 20, 2016, 11:07:41 am »
FPC make zipinstall is basically useless from OS packaging system perspective. This feature is here probably just because creating archive is much simpler then OS dependent installation packages.
Nope, that one is very useful for those who don't want FPC managed by package manager (at least I'm in the list). It's also the one provided in sourceforge.
Even as name suggest it should create zip archive but create tar.gz by default on Linux so even make target name is misleading.
Not really:
Quote
$ make help

Targets
all Build a new compiler and all packages
install Install newly build files
zipinstall Create zip/tar of installed files
singlezipinstall Alias for zipinstall
Lazarus tools/install is only usable and also primary source for building install binaries. But it's shame that Lazarus need to create own FPC install packages because FPC itself can't.
Both are independent projects anyway.
There are FPC packages in Ubuntu repo but they are simply mess. FPC itself is separated to many small packages which may be usable for apt-get install but not for manual upgrade from web. Nobody want to download 32(16) packages just because somebody had great idea to cut them to small packages. This not how real world works. And even if new FPC version comes then these all packages are upgraded so possible advantage to download only parts which were changed/upgraded is not real use case.
Perhaps that's a requirement from Debian sponsor who uploads the package, who knows? The RPM package is not splitted that way, though, nor is the Arch Linux package.
This reminds me that there is still missing basic feature in Lazarus called something like "Make install package...". All Lazarus developers deal with same problem. They create application and want provide install packages for various platforms for their users. It may be Windows .msi package, Linux DEB/RPM/other, Apple/iOS package, Android apk or install file for other platforms. This should be part of FPC or Lazarus. As both FPC and Lazarus needs to be distributed in same way. Yet nobody implemented such feature.
No other IDE I know provides such a broad range of installer creation option. It's even impossible to build certain format under different system other than it's intended for (unless someone is crazy enough to reimplement all of them without their official tools which often only runs on a specific system). If you really want to implement this, I suggest to extend LazPackager instead of starting from scratch.

Chronos

  • Full Member
  • ***
  • Posts: 173
    • PascalClassLibrary
Re: Public daily build server
« Reply #6 on: March 20, 2016, 01:07:09 pm »
Nope, that one is very useful for those who don't want FPC managed by package manager (at least I'm in the list). It's also the one provided in sourceforge.
Thats ever lasting problem with both FPC and Lazarus that they are forcing users often to recompile tools itself. This can be useful for its developers but not for users who need to use Lazarus as working tool and focus on their project. I spend significant more time on fixing and debugging Lazarus while no such activity is necessary to use commercial Delphi IDE. So to have simply installable platform native packages are basic requirement of any project which wanted to be successful.

Not really:
Quote
$ make help

Targets
all Build a new compiler and all packages
install Install newly build files
zipinstall Create zip/tar of installed files
singlezipinstall Alias for zipinstall
Just try to run make zipinstall on Linux and it will produce .tar.gz. Thats all story. So feature should be named archive_install or whatever more general name.

Both are independent projects anyway.
That have many bad consequences. Lets imagine that Delphi compiler and Delphi IDE would be created by two different companies. Fortunately this is not the case. But for example Lazarus packages and FPC packages are clear example that current state is far from perfect.

Perhaps that's a requirement from Debian sponsor who uploads the package, who knows? The RPM package is not splitted that way, though, nor is the Arch Linux package.
Sure, I am aware of that, but this wont change that this is bad. At least FPC team could provide correct deb packages on their web site or provide apt repo list as for example Jenkins does.

No other IDE I know provides such a broad range of installer creation option. It's even impossible to build certain format under different system other than it's intended for (unless someone is crazy enough to reimplement all of them without their official tools which often only runs on a specific system). If you really want to implement this, I suggest to extend LazPackager instead of starting from scratch.
Sure, it could be implemented as runtime package. I don't know how complicated it would be to implement such feature into Lazarus. If is there any API documentation for developing runtime packages or just code is documentation.
Also there is a problem that what developer of an application would need is to be able to generate all install binaries for all selected platforms at once. Which would require to support cross-compilation and automated rebuild of project multiple-times for several targets. This is not possible now or at least too hard to setup for ordinal developer.


To go back to original topic. There are several problems with automated build system. For example pairing multiple combinations of Lazarus and FPC versions. Each Lazarus version is expected to be build on some FPC version and packed with different newer FPC version. I don't know if this is recorded somewhere. Also one FPC version need to be build with previous version. And all these working FPC instances need to be available on build server.
I tested create_lazarus_deb.sh script and it generated deb package as expected. But for optimization point of view it rebuilds entire Lazarus on each call. So for daily build to save CPU load it would need to be optimized to do build just once to generate for example RPM and DEB.
Entire process of building need to be optimized to take as low space and time as possible. For example problem with SVN is that it stores second copy of all files in .svn subdirectory a pristine copy. Which gets significant problem if build server as Jenkins try to download code for each job and subjob. Not mentioning wasteful checkouting from main SVN server. This needs to be improved definitely. Just "svn export" itself will not allow Jenkins to detect if new commit was done and build need to be rerun.

Chronos

  • Full Member
  • ***
  • Posts: 173
    • PascalClassLibrary
Re: Public daily build server
« Reply #7 on: March 20, 2016, 01:42:17 pm »
Here you can find Jenkins instance: http://devel.zdechov.net:8080/
Question is how to setup build to be able at least do "make all" on all officially supported architectures.

For example to be able to build Lazarus for Win32 you need working Win32 cross FPC. And this should be created by:
make all OS_TARGET=win32 CPU_TARGET=i386
make crossinstall OS_TARGET=win32 CPU_TARGET=i386
according http://wiki.freepascal.org/Cross_compiling_for_Win32_under_Linux
But unfortunately it would mess up Ubuntu FPC installation. Also build server needs to have several FPC versions installed so they should be better placed to specific directory to not affect Ubuntu installed FPC. But this would require nonstandard configuration to have all FPC directories like /bin,/usr/lib/fpc moved elsewhere.

BeniBela

  • Hero Member
  • *****
  • Posts: 547
    • homepage
Re: Public daily build server
« Reply #8 on: March 20, 2016, 01:48:18 pm »

Nobody want to download 32(16) packages just because somebody had great idea to cut them to small packages. This not how real world works.

On the other hand, when I submitted my app to someone's build server they refused it, because:

> lazarus: We can't and won't install a complete IDE on our buildserver. Can we cherry-pick just the needed libraries?


 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus