Recent

Author Topic: Build lazarus and fpc in linux is not support multi thread.  (Read 741 times)

gasensor

  • New Member
  • *
  • Posts: 15
In linux build lazarus and fpc form source  is not support multi thread.

Code: Pascal  [Select][+][-]
  1. make all -j2

OS:   Fedora 35
fpc:   3.2.0 src
lazarus : 2.30(test edition)
make:  4.3


PascalDragon

  • Hero Member
  • *****
  • Posts: 4008
  • Compiler Developer
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #1 on: May 13, 2022, 10:27:50 am »
At least in FPC you additionally need to use FPMAKEOPT="-T 2". The reason is that one can not query the value of -j from within the Makefile and the majority of the package compilation is not handled by make, but instead by fpmake.
In Lazarus this might be the case as well, I don't know.

gasensor

  • New Member
  • *
  • Posts: 15
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #2 on: May 14, 2022, 03:54:46 pm »
I compiled freepascal 3.2.0 use command above:

Code: Pascal  [Select][+][-]
  1. make all FPMAKEOPT="-T 2"

And find it works  when compiling packages. Time costing is lower. In Lazarus(src 2.3.0) it do not work.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 9986
  • FPC developer.
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #3 on: May 14, 2022, 05:16:56 pm »
That's on par with expectations. FPC compiles the packages part, and every core added beyond 2 has diminishing returns (though a six core is still faster with -T 6 than with -T 4, just not much).

Free Pascal compiling is not massively parallel like C, but there are reasons for that. (and some of those tradeoffs mean FPC is always quite fast)

gasensor

  • New Member
  • *
  • Posts: 15
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #4 on: May 15, 2022, 05:32:37 am »
But, as a Linux user, use"-j" is a habitual operation to build a large software.


That's on par with expectations. FPC compiles the packages part, and every core added beyond 2 has diminishing returns (though a six core is still faster with -T 6 than with -T 4, just not much).

Free Pascal compiling is not massively parallel like C, but there are reasons for that. (and some of those tradeoffs mean FPC is always quite fast)

PascalDragon

  • Hero Member
  • *****
  • Posts: 4008
  • Compiler Developer
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #5 on: May 16, 2022, 02:52:32 pm »
But, as a Linux user, use"-j" is a habitual operation to build a large software.

And the way FPC builds programs is simply not compatible with -j. Where it works for example is while the RTL is compiled on Linux cause FPC is invoked for each unit separately. If you cross compile from Linux to Windows it wouldn't provide any benefit, because a build unit is used there (and those fewer compiler invocations are done). Similar for building the packages: fpmake is what does the real work, the make related part is essentially only a wrapper around fpmake. And Lazarus in contrast uses lazbuild which also deals with parallelism (though I don't know if it needs to be enabled explicitly using some parameter or if it does parallelism by itself).

And again: one can't query the -j parameter from within a Makefile, so we can't hook up to that parameter.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 9986
  • FPC developer.
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #6 on: May 16, 2022, 03:17:30 pm »
But, as a Linux user, use"-j" is a habitual operation to build a large software.

It is a habitual system for C that doesn't have any form of automatically finding files, requiring every file to be listed on the compiler commandline, or make to handle its dependencies.

FPC, if you ever used it on the commandline, recursively finds files itself. This means that there is only one FPC invocation per package. IOW make -j  (and -T) is only used to parallel compile whole directories rather than on an individual file level.

The makefiles also only contain dependency information on a whole package/directory level.


gasensor

  • New Member
  • *
  • Posts: 15
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #7 on: May 16, 2022, 04:32:37 pm »
OK, I see.

Maybe, using ".NOTPARALLEL" is a good idea. I find it in make's manual 5.4 line4.
https://www.gnu.org/software/make/manual/html_node/Parallel.html#Parallel

And... PascalDragon,

makefile can query the -j parameter just read var "MAKEFLAGS". like this:
Code: Text  [Select][+][-]
  1. all:
  2.         @echo $(MAKEFLAGS)

It retruns:
-j2 --jobserver-auth=3,4

Hope that helps.

MarkMLl

  • Hero Member
  • *****
  • Posts: 4163
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #8 on: May 16, 2022, 04:32:59 pm »
But, as a Linux user, use"-j" is a habitual operation to build a large software.

As a Linux user, you should know better than to try to build a large software system other than by hewing closely to the maintainers' documentation.

This is something that I raised on the ML ten years or more ago, and was content then with much the same answers as you are being given now: it might work on a package-level basis, but not at the level of individual object files.

In practice, the real bottleneck is at the linkage stage, and I'm not aware of a linker that can benefit heavily from multiple CPUs/cores.

Of course, if you contributed any patches which improved build efficiency the core developers would probably look favourably on them...

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

PascalDragon

  • Hero Member
  • *****
  • Posts: 4008
  • Compiler Developer
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #9 on: May 17, 2022, 09:09:36 am »
In practice, the real bottleneck is at the linkage stage, and I'm not aware of a linker that can benefit heavily from multiple CPUs/cores.

Microsoft's MSVC linker makes use of multithreading (one can see that in TaskManager when compiling a large C++ program) and the improvement with that is definitely noticeable.

In general a linker could parallelize the reading of the object files. Most everything else is probably too serialized...

And... PascalDragon,

makefile can query the -j parameter just read var "MAKEFLAGS". like this:
Code: Text  [Select][+][-]
  1. all:
  2.         @echo $(MAKEFLAGS)

It retruns:
-j2 --jobserver-auth=3,4

The problem is not only reading it, but also handling it, because make handles that parallelism by itself. Hooking that up to the parallelism provided by fpmake is the problem.

TL;DR: the current way works. It's documented. We don't intend to change it.

MarkMLl

  • Hero Member
  • *****
  • Posts: 4163
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #10 on: May 17, 2022, 09:35:00 am »
In general a linker could parallelize the reading of the object files. Most everything else is probably too serialized...

The async callbacks provided by the Windows (originally OS/2) API might arguably favour heavy multithreading in a way that the unix API doesn't. Or it might be that the MSVC linker actually suffers from being written as a multithreaded program and /demands/ lots of CPUs/cores.

As a potential point of debate, I'd suggest that an OS should or at least could throw spare CPUs/cores at the job of keeping the various buffers and caches needed by the linker full. In any event, sooner or later one hits the limit of how much traffic can pass through the kernel down to a small number of physical storage devices accessed via a small number of busses.

Of course, if one looks at one of the most frequently examined cases that can benefit from make -j, i.e. a Linux kernel rebuild, one notices that most of the code- i.e. almost all the drivers etc.- is saved in the form of .ko files. Multiple .ko files are definitely built in parallel, but I don't know to what extent the fairly small number of intermediate files which are linked to build a .ko are built in parallel.

If OP wants to make a case for change, then I think he needs to provide convincing evidence that an individual Linux .ko benefits from parallelisation, and then demonstrate that FPC's building an individual package is significantly less efficient... which together would be a fairly hefty analysis job, allowing for the number of .ko files and packages being built simultaneously.

And in any event, he'd have to demonstrate that any change was both compatible with fpcmake (which is central to the wide variety of targets FPC supports) and with things like generics.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

PascalDragon

  • Hero Member
  • *****
  • Posts: 4008
  • Compiler Developer
Re: Build lazarus and fpc in linux is not support multi thread.
« Reply #11 on: May 17, 2022, 02:17:23 pm »
Of course, if one looks at one of the most frequently examined cases that can benefit from make -j, i.e. a Linux kernel rebuild, one notices that most of the code- i.e. almost all the drivers etc.- is saved in the form of .ko files. Multiple .ko files are definitely built in parallel, but I don't know to what extent the fairly small number of intermediate files which are linked to build a .ko are built in parallel.

The object files are also compiled in parallel. Otherwise there wouldn't be any benefit when building a full static kernel (one that has all drivers compiled in).

If OP wants to make a case for change, then I think he needs to provide convincing evidence that an individual Linux .ko benefits from parallelisation, and then demonstrate that FPC's building an individual package is significantly less efficient... which together would be a fairly hefty analysis job, allowing for the number of .ko files and packages being built simultaneously.

This is not comparable and also not applicable. In the C world C files can be build in parallel, because they don't depend on each other (declarations and such are in separate header files that are simply included by each). This is not the case for FPC, because units have dependencies between each other and the compiler simply can't continue compiling a unit until all dependencies of that unit are solved.

 

TinyPortal © 2005-2018