Lazbuild concurrency issue

[Wallaby]

I use lazbuild to compile my app for multiple platforms. To speed up the process, I want to invoke several instances of lazbuild in parallel. On any modern multi-core machine, this dramatically decreases build time. For example, if building for one platform takes 30 seconds, building for four platforms sequentially takes 2 minutes. However, running four instances of lazbuild on multiple cores compiles everything in 35 seconds.

The issue is concurrency. In my LPI file, I have something like this:

Code: XML  [Select][+][-]

1. <UnitOutputDirectory Value="$(TestDir)\lib\$(TargetCPU)-$(TargetOS)" />

This ensures all PPUs and related files go to a separate directory for each platform and don't conflict with each other during parallel builds. However, the problem is that lazbuild recompiles the project resources and updates the RES file in the project folder. This leads to concurrency issues as multiple processes try to update the same file, resulting in failed builds or broken project resources.

Is there a way to rebuild project resources to UnitOutputDirectory or a similar location, so multiple instances of lazbuild do not conflict with each other? Perhaps this could be a lazbuild improvement.

[MarkMLl]

However, the problem is that lazbuild recompiles the project resources and updates the RES file in the project folder.

That's definitely worth raising a bug report for, particularly if you can demonstrate that it's the only issue in that area.

MarkMLl

[Thaddy]

I thought that lazbuild itself is already multi-threaded or multi-thread capable?

[MarkMLl]

I thought that lazbuild itself is already multi-threaded or multi-thread capable?

But this is about potentially-clashing temporary files, so appears to have been overlooked.

MarkMLl

[JuhaManninen]

I thought that lazbuild itself is already multi-threaded or multi-thread capable?

Lazarus IDE and AFAIK also LazBuild can compile required packages of a project in parallel.
The IDE can compile many buildmodes sequentially at one go, and they can be used for different platforms.
If temporary files and other issues can be solved, those buildmodes could be compiled in parallel.
Then this same system could be used by LazBuild.

[Wallaby]

I have checked the lazbuild's code. Around these lines:

Code: Pascal  [Select][+][-]

1.       // update all lrs files
2.       MainBuildBoss.UpdateProjectAutomaticFiles('');
3.  
4.       // regenerate resources
5.       if not Project1.ProjResources.Regenerate(SrcFileName, False, True, '') then
6.       begin
7.         if ConsoleVerbosity>=-1 then
8.           DebugLn('Error: (lazarus) Project1.Resources.Regenerate failed of ',SrcFilename);
9.       end;  
10.  

This is where multiple lazbuild instances may have concurrency issues by trying to write to the same RES file in the project folder (recompiling resources) from multiple instances. I guess I will create a bug report to see what can be done here.

[MarkMLl]

This is where multiple lazbuild instances may have concurrency issues by trying to write to the same RES file in the project folder (recompiling resources) from multiple instances. I guess I will create a bug report to see what can be done here.

We heard you already. Raise a bug report.

MarkMLl

[Wallaby]

This is where multiple lazbuild instances may have concurrency issues by trying to write to the same RES file in the project folder (recompiling resources) from multiple instances. I guess I will create a bug report to see what can be done here.
We heard you already. Raise a bug report.
MarkMLl

Done: #40986 Lazbuild concurrency issue with multiple instances

[JuhaManninen]

Yes. Just remember the same code is used also by Lazarus IDE. The solution will affect both.

[MarkMLl]

Yes. Just remember the same code is used also by Lazarus IDE. The solution will affect both.

Actually, moving the .res into the lib/ tree by default would be worth doing, since "do I need to check the .res into my VCS or will it be rebuilt?" is rather an FAQ.

MarkMLl