Recent

Author Topic: Lazbuild concurrency issue  (Read 1197 times)

Wallaby

  • Full Member
  • ***
  • Posts: 104
Lazbuild concurrency issue
« on: June 14, 2024, 04:07:23 am »
I use lazbuild to compile my app for multiple platforms. To speed up the process, I want to invoke several instances of lazbuild in parallel. On any modern multi-core machine, this dramatically decreases build time. For example, if building for one platform takes 30 seconds, building for four platforms sequentially takes 2 minutes. However, running four instances of lazbuild on multiple cores compiles everything in 35 seconds.

The issue is concurrency. In my LPI file, I have something like this:

Code: XML  [Select][+][-]
  1. <UnitOutputDirectory Value="$(TestDir)\lib\$(TargetCPU)-$(TargetOS)" />

This ensures all PPUs and related files go to a separate directory for each platform and don't conflict with each other during parallel builds. However, the problem is that lazbuild recompiles the project resources and updates the RES file in the project folder. This leads to concurrency issues as multiple processes try to update the same file, resulting in failed builds or broken project resources.

Is there a way to rebuild project resources to UnitOutputDirectory or a similar location, so multiple instances of lazbuild do not conflict with each other? Perhaps this could be a lazbuild improvement.

MarkMLl

  • Hero Member
  • *****
  • Posts: 8044
Re: Lazbuild concurrency issue
« Reply #1 on: June 14, 2024, 08:39:24 am »
However, the problem is that lazbuild recompiles the project resources and updates the RES file in the project folder.

That's definitely worth raising a bug report for, particularly if you can demonstrate that it's the only issue in that area.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 16201
  • Censorship about opinions does not belong here.
Re: Lazbuild concurrency issue
« Reply #2 on: June 14, 2024, 09:13:57 am »
I thought that lazbuild itself is already multi-threaded or multi-thread capable?
If I smell bad code it usually is bad code and that includes my own code.

MarkMLl

  • Hero Member
  • *****
  • Posts: 8044
Re: Lazbuild concurrency issue
« Reply #3 on: June 14, 2024, 09:25:31 am »
I thought that lazbuild itself is already multi-threaded or multi-thread capable?

But this is about potentially-clashing temporary files, so appears to have been overlooked.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4541
  • I like bugs.
Re: Lazbuild concurrency issue
« Reply #4 on: June 14, 2024, 10:23:58 am »
I thought that lazbuild itself is already multi-threaded or multi-thread capable?
Lazarus IDE and AFAIK also LazBuild can compile required packages of a project in parallel.
The IDE can compile many buildmodes sequentially at one go, and they can be used for different platforms.
If temporary files and other issues can be solved, those buildmodes could be compiled in parallel.
Then this same system could be used by LazBuild.
« Last Edit: June 17, 2024, 08:42:40 pm by JuhaManninen »
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

Wallaby

  • Full Member
  • ***
  • Posts: 104
Re: Lazbuild concurrency issue
« Reply #5 on: June 14, 2024, 10:35:42 am »
I have checked the lazbuild's code. Around these lines:

Code: Pascal  [Select][+][-]
  1.       // update all lrs files
  2.       MainBuildBoss.UpdateProjectAutomaticFiles('');
  3.  
  4.       // regenerate resources
  5.       if not Project1.ProjResources.Regenerate(SrcFileName, False, True, '') then
  6.       begin
  7.         if ConsoleVerbosity>=-1 then
  8.           DebugLn('Error: (lazarus) Project1.Resources.Regenerate failed of ',SrcFilename);
  9.       end;  
  10.  

This is where multiple lazbuild instances may have concurrency issues by trying to write to the same RES file in the project folder (recompiling resources) from multiple instances. I guess I will create a bug report to see what can be done here.

MarkMLl

  • Hero Member
  • *****
  • Posts: 8044
Re: Lazbuild concurrency issue
« Reply #6 on: June 14, 2024, 10:40:32 am »
This is where multiple lazbuild instances may have concurrency issues by trying to write to the same RES file in the project folder (recompiling resources) from multiple instances. I guess I will create a bug report to see what can be done here.

We heard you already. Raise a bug report.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Wallaby

  • Full Member
  • ***
  • Posts: 104
Re: Lazbuild concurrency issue
« Reply #7 on: June 14, 2024, 10:53:57 am »
This is where multiple lazbuild instances may have concurrency issues by trying to write to the same RES file in the project folder (recompiling resources) from multiple instances. I guess I will create a bug report to see what can be done here.
We heard you already. Raise a bug report.
MarkMLl

Done: #40986 Lazbuild concurrency issue with multiple instances

JuhaManninen

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 4541
  • I like bugs.
Re: Lazbuild concurrency issue
« Reply #8 on: June 14, 2024, 10:54:18 am »
Yes. Just remember the same code is used also by Lazarus IDE. The solution will affect both.
Mostly Lazarus trunk and FPC 3.2 on Manjaro Linux 64-bit.

MarkMLl

  • Hero Member
  • *****
  • Posts: 8044
Re: Lazbuild concurrency issue
« Reply #9 on: June 14, 2024, 12:30:25 pm »
Yes. Just remember the same code is used also by Lazarus IDE. The solution will affect both.

Actually, moving the .res into the lib/ tree by default would be worth doing, since "do I need to check the .res into my VCS or will it be rebuilt?" is rather an FAQ.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

 

TinyPortal © 2005-2018