Could Free Pascal benefit of something like Apache Maven?

[mjustin]

This is a stackoverflow.com question, which can be found at

http://stackoverflow.com/questions/1893469/could-free-pascal-benefit-of-something-like-apache-maven

Apache Maven is a very popular build and dependency management tool in the Java open source ecosphere. I did some tests to find out if it can handle compiled Free Pascal / Delphi units and found it easy to implement. So it would be possible to

    * release open source libraries precompiled for Free Pascal (or Delphi) in a public Maven repository
    * include metadata in this repository which contains dependency information
    * use Maven on the command line to download the open source library from the public repository, and automatically resolve all dependencies
    * local repositories, working as proxies, could be used to cache frequently used binaries
    * automatic checksum generation and verification (provided by Maven) would reduce the risk of downloading corrupted binaries
    * source code and even documentation files could be provided with the binaries
    * binaries can be provided with or without debug information
    * continuous integration servers like Hudson can be used for building all projects whenever changes have been submitted to the source control system and notify developers about build errors

This way of dependency management could be very beneficial for open source projects which use many third party libraries with complex dependencies. It would avoid typical conflicts caused by using wrong versions.

For the developer, the workflow for editing and building a project would be reduced to a minimum:

    * checkout the project
    * edit source file(s)
    * run mvn package which would automatically download all required third party libraries (precompiled units) if they are not yet on the workstation
    * compile and run

The only additional file for Apache Maven which is required in the project folder is the POM.XML file containing the project information.

One point which I have not investigated is binary file compatibility. Are compiled units incompatible between Free Pascal versions? And do Free Pascal developers usually feel more comfortable with recompiling all source codes (including third party libraries) than to use precompiled binaries?

[Mike J]

Unlike JAVA, you would probably need to do Source, any compiled modules are not compatible with differing processors and os's.

[mjustin]

Unlike JAVA, you would probably need to do Source, any compiled modules are not compatible with differing processors and os's.

The repository would offer the artifacts for all possible target platform versions. This has to be done only once, and it can be an automated process since Maven includes repository management commands. A simple 'mvn deploy' will update the files in the repository. The pom.xml file needs only one entry:

    <distributionManagement>
    <repository>
      <id>example-ftp</id>
      <url>ftp://example.com/usr/home/maven2</url>
    </repository>
  </distributionManagement>


Mike

[Phil]

The repository would offer the artifacts for all possible target platform versions. This has to be done only once, and it can be an automated process since Maven includes repository management commands. A simple 'mvn deploy' will update the files in the repository. The pom.xml file needs only one entry:

Are you talking about things like this?

http://wiki.lazarus.freepascal.org/Components_and_Code_examples

These are the kind of 3rd party components that are used in Lazarus apps. In almost all cases, they are source code units that are compiled by and into the Lazarus IDE. They are not "compiled libraries" if I understand what you mean by that (.dll, .dylib, .so).

Nevertheless, there may be some value here if Maven can be adapted to the realities of Lazarus use. For example, Lazarus itself is a good example of "open source projects which use many third party libraries with complex dependencies". Typically a developer may have a number of packages installed (compiled into) the Lazarus IDE. Can you work with that idea and see if you can devise a way to, say, run a script or maven command that updates the source for all packages currently used by the IDE (from SVN), builds Lazarus with them, etc? Maybe that would be useful in some cases.

List of packages compiled into IDE is in staticpackages.inc under Lazarus config folder (e.g., ~/.lazarus). This file is included ($I) in lazarus.pp.

Thanks.

-Phil

[mjustin]

Can you work with that idea and see if you can devise a way to, say, run a script or maven command that updates the source for all packages currently used by the IDE (from SVN), builds Lazarus with them, etc? Maybe that would be useful in some cases.

Yes, this would e a perfect use case: checkout the Lazarus source code (without third party libraries), execute 'mvn clean' which will check if all required third party libraries are available, and download the missing libraries automatically.

The next step would be 'mvn package'. In my vision, Maven should be able to create a Lazarus project file which already is configured with all the source paths (or library paths) to third party libraries, or to run the compiler with all paths passed as arguments.

[Phil]

Yes, this would e a perfect use case: checkout the Lazarus source code (without third party libraries), execute 'mvn clean' which will check if all required third party libraries are available, and download the missing libraries automatically.

But which packages? All 100 of them? The user has to install each package initially in order for it to be added to staticpackages.inc.

The next step would be 'mvn package'. In my vision, Maven should be able to create a Lazarus project file which already is configured with all the source paths (or library paths) to third party libraries, or to run the compiler with all paths passed as arguments.

Paths are not necessary with packages. Lazarus (and lazbuild) knows the path once a package has been installed.

Thanks.

-Phil

[marcov]

fppkg does this already without the JRE dependancy.

It has been tested in 2.2.4+ with a few packages, but so far nobody stepped up to perfect and manage it.

[mjustin]

But which packages? All 100 of them? The user has to install each package initially in order for it to be added to staticpackages.inc.

This Maven-like tool would do this so the user doesn't have to do it manually. So if the packages are not present, or a new version is required, the missing dependencies will be downloaded. Downloaded packages then will be installed locally, the tool would not download them again next time. All steps required to make the packages known for the build of Lazarus (creation of configuration files) will be done by the tool.

Paths are not necessary with packages. Lazarus (and lazbuild) knows the path once a package has been installed.

'mvn package' is not about creating Lazarus packages. In Maven terminologie it is the name of a build goal which includes compiling and running tests sources and then 8if no test fails) compile the artifact to the target directory. In Maven, every build will create one target artifact (like a DLL, SO or EXE file). Multiproject builds use many single artifacts and nested Maven project directories.

[mjustin]

fppkg does this already without the JRE dependancy.

It has been tested in 2.2.4+ with a few packages, but so far nobody stepped up to perfect and manage it.

Ok, no need to reinvent the wheel then - it just needs somebody who maintains it. I am sure it will find more supporters if it makes development easier.

Cheers,

[Phil]

Paths are not necessary with packages. Lazarus (and lazbuild) knows the path once a package has been installed.

'mvn package' is not about creating Lazarus packages. In Maven terminologie it is the name of a build goal which includes compiling and running tests sources and then 8if no test fails) compile the artifact to the target directory. In Maven, every build will create one target artifact (like a DLL, SO or EXE file). Multiproject builds use many single artifacts and nested Maven project directories.

I think we're misunderstanding each other.

In Lazarus, there are two kinds of packages. FPC has its packages, which apparently can be maintained by fppkg, although I've never used it since the FPC "packages" (fpc-base, gtk2, httpdd20, etc.) are installed or built by default with FPC.

Most Lazarus and Delphi developers, when they refer to packages, are referring to .lpk/.dpk package files that compile and add design and/or runtime components and/or units to the IDE. With Lazarus, these components have to be statically linked into the IDE.

When you use one of these packages in a Lazarus project, it adds a reference to it in the project's .lpi file. No path is saved since the package could easily be installed somewhere else on a different computer. Lazarus supplies the path automatically when compiling the project, either from within the IDE or using the command line lazbuild console app.

So a typical "problem", I suppose, for building a project that has external dependencies, is not so much the FPC "packages" but the Lazarus packages.

So, for example, say somebody gives you an ExtPascal project developed with Lazarus that you want to work on. That is, you're supplied with an .lpi file, .lpr file, one or more .pas/.lfm files, etc.

Now this is probably an extreme case, but it is a real case. With a Lazarus ExtPascal project, before you can open the forms you first need to install up to 4 design time packages. See here for instructions:

http://web.fastermac.net/~MacPgmr/ExtPascal/ExtPToolkitStatus.html#Installation

Note that these are design-time only packages and are not compiled against. To build the ExtPascal app, you also need to download the ExtPascal source (which is not currently a Lazarus package), build the wrapper source units, and compile them. To build the wrapper source units requires downloading and installing the Ext JS documentation.

Furthermore, to run an ExtPascal app, you need a working Web server and you have to copy the Ext JS folder containing the runtime .js files to the Web server.

One thing I suspect about ExtPascal is that more people would try it out if the setup of all this stuff were more automated. So, for example, there are 3 example apps included with my ExtP Toolkit add-on for ExtPascal. They are all Lazarus projects (.lpi). But to try one out requires quite a bit of downloading and installation.

Would this be a place for Maven or some other build system? Let us hear your thoughts on this.

Thanks.

-Phil

[mjustin]

One thing I suspect about ExtPascal is that more people would try it out if the setup of all this stuff were more automated. So, for example, there are 3 example apps included with my ExtP Toolkit add-on for ExtPascal. They are all Lazarus projects (.lpi). But to try one out requires quite a bit of downloading and installation.

Would this be a place for Maven or some other build system? Let us hear your thoughts on this.

Hi Phil,

this is exactly my point: give developers some tools which are so easy to use that all new libraries and components can be installed and used in no time. Give them a standard directory layout and some standard tools to create these directory layouts and project files, and a way to 'zip' a library including all metadata so that it can be reused easily ... and other nice helpers, all bundled in an extensible command-line driven toolkit.

Of course I am not saying Maven is the 'silver bullet' - I am sure that a native Free Pascal build tool would find more followers.

Accidentally I stumbled over the description of fppkg which mentions a tool called FPMake at http://wiki.freepascal.org/FPMake

FPMake looks almost like a Maven clone, if you see the manifest XML file:

Code: [Select]

 <?xml version="1.0"?>
  <packages>
  <package name="my-package">
   <version major="0" minor="7" micro="6" build="1"/>
   <filename>my-package-0.7.6-1.zip</filename>
   <author>my name</author>
   <license>GPL</license>
   <homepageurl>http://www.freepascal.org/</homepageurl>
   <email>myname@freepascal.org</email>
   <description>this is the package description</description>
   <dependencies>
    <dependency>
     <package packagename="rtl"/>
    </dependency>
   </dependencies>
  </package>
  </packages>
The header with package description and version information, and the list of dependencies, this is almost an exact copy of a typical Maven build file (pom.xml). Is FPMake a new tool? I think it deserves a lot of interest because it can be the foundation of a very powerful build system for Free Pascal.

Many thanks for you input,

Kind regards
Michael Justin

[Phil]

I don't know much about FPMake.

Two phrases, apparently contradictory, popped out from your first paragraph: "tools which are so easy to use" and "command-line driven toolkit". Isn't that kind of like saying SVN is easy to use? If it were, we wouldn't have Tortoise, Cornerstone, Versions, etc.

Have you done much programming yet with Lazarus? Maybe you could do a little GUI frontend to Maven as a proof of concept or something. If it's easy to use, maybe people would give it a chance, but offhand I don't see very many people investigating a command line tool from the Java world.

Thanks.

-Phil

[marcov]

I don't know much about FPMake.

Two phrases, apparently contradictory, popped out from your first paragraph: "tools which are so easy to use" and "command-line driven toolkit". Isn't that kind of like saying SVN is easy to use? If it were, we wouldn't have Tortoise, Cornerstone, Versions, etc.

I don't think commandlines are that much harder than a modern GUI. People are only less used to it.

[marcov]

I think we're misunderstanding each other.

In Lazarus, there are two kinds of packages. FPC has its packages, which apparently can be maintained by fppkg, although I've never used it since the FPC "packages" (fpc-base, gtk2, httpdd20, etc.) are installed or built by default with FPC.

http://wiki.freepascal.org/packages%28disambiguation%29

Most Lazarus and Delphi developers, when they refer to packages, are referring to .lpk/.dpk package files that compile and add design and/or runtime components and/or units to the IDE. With Lazarus, these components have to be statically linked into the IDE.

Delphi packages are also used to bundle units, even if not compile to .bpl. This makes "Delphi package" a bit an ambiguous concept.