Recent

Author Topic: Does unit recompilation roles have changed in fpc 3.3.1?  (Read 1371 times)

mika

  • Full Member
  • ***
  • Posts: 118
Does unit recompilation roles have changed in fpc 3.3.1?
« on: January 21, 2025, 10:12:20 am »
First things first. I am not talking of recent changes of fpc 3.3.1 nor of any specific changes.
And I am aware that fpc 3.2.2 require time to time full project rebuild in order to get all units in line.

Fpc 3.3.1 seems to have more strict recompilation roles as it is for fpc 3.2.2. I tested current fpc 3.3.1  with large project (~500 units - full compilation +40 seconds). Changing some random unit could lead from 50% to 100% of all units recompilation. Some times no changes lead to recompilation of some addition units beside main project file.
fpc 3.2.2.  100k lines compile  4.7s
fpc 3.3.1.  100k lines compile  6.4s
Fpc 3.3.1 recompile more units and does it slower. As result it gives feeling that fpc 3.3.1 is 5 times slower than fpc 3.2.2.

Changing optimization level to zero does not give compilation speed increase as one could hope for. Maybe there are some hidden or experimental compiler switch to go full lighting speed? Hiting F9 key is no fun at all if it becomes a waiting game.

Is it mine perception that deceiving me or is this a new reality of happy fpc 3.3.1 users and future for fpc 4.0.0?

Please, don't ask for small up to 100 lines example program that demonstrates my frustration.

MarkMLl

  • Hero Member
  • *****
  • Posts: 8193
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #1 on: January 21, 2025, 10:23:13 am »
You refer to "spiting F9"... are you talking about the FPC compiler, the FP text-mode wrapper, or Lazarus?

There are known- but poorly-understood- problems in FPC up to at least 3.2.2 where it sometimes fails to recompile files for no obvious reason. I don't believe that anybody has made real progress in pinning this down, but it appears to be exacerbated by directives/conditionals including $include, and by having units in non-standard places particularly if given as an explicit filename in the uses clause.

There is a possibility that the slowdown you say you're seeing is the result of a countermeasure which attempts to tackle that.

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Thaddy

  • Hero Member
  • *****
  • Posts: 16520
  • Kallstadt seems a good place to evict Trump to.
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #2 on: January 21, 2025, 10:33:02 am »
Trunk/main compilation is indeed currently slower, but not 5 times, but ~20% (1/5) and this may change when it becomes a release(positive or negative).
It also has a lot more features that affect parsing, which affects speed.
The generated code is faster by a considerable margin on most main platforms.
Don't confuse compile speed with executable speed.

The GNU compiler suite is still about 5 times slower than FPC, btw, so what are you complaining about? (yes, that is a fact)
The 640k statement all over again?
« Last Edit: January 21, 2025, 10:40:49 am by Thaddy »
But I am sure they don't want the Trumps back...

cdbc

  • Hero Member
  • *****
  • Posts: 1814
    • http://www.cdbc.dk
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #3 on: January 21, 2025, 10:48:32 am »
Hi
I'm with Thaddy on this one, my experience says roughly 1/5 slower too, I shoved it under more features and optimization, because as mentioned - it runs faster.
Regards Benny
If it ain't broke, don't fix it ;)
PCLinuxOS(rolling release) 64bit -> KDE5 -> FPC 3.2.2 -> Lazarus 2.2.6 up until Jan 2024 from then on it's: KDE5/QT5 -> FPC 3.3.1 -> Lazarus 3.0

mika

  • Full Member
  • ***
  • Posts: 118
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #4 on: January 21, 2025, 11:13:32 am »
I'm with Thaddy on this one, my experience says roughly 1/5 slower too, I shoved it under more features and optimization, because as mentioned - it runs faster.
I have no problem with being 20% slower. I can eat that.

Lets imagine situation. You work on your custom memory manager. Included in main project file as first unit. Do some changes in it. How many units of have to be recompile? fpc 3.3.1 does recompile whole project. Size of project is whats increase time dramatically. Maybe it  is just bad project design. I have what i have.

Fun fact for nonbelievers: comments take time to compile as well.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 10794
  • Debugger - SynEdit - and more
    • wiki
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #5 on: January 21, 2025, 11:14:46 am »
Changing some random unit could lead from 50% to 100% of all units recompilation. Some times no changes lead to recompilation of some addition units beside main project file.
I haven't tested what exactly changed. And if any of the changes affects the below. But...

That can be worded either way.

FPC 3.2.2 (incl. 3.2.3 and maybe also older FPC) sometimes do skip recompiling a unit, that required compilation (or do not completely/correctly recompile it).

Under Lazarus this leads to the occasional "Can not find foo.pas" even though this exists. This happens because the outdate ppu is detected to late, and at that point (due to the grouping into packages) the path to that source is no longer given to FPC (as it should have already done it).

And, I do recall (from longer ago / maybe before 3.2.2) that occasionally a compilation resulted in a randomly broken exe, that would work fine after a clean rebuild. But that I have not observed in some years now.

So, maybe, if we are so lucky, the changes you observed are actually fixes. But as I said I did not test. And there also is no reliable test for this, not that I know of anyway.

As far as my observation goes, likely factors in all of this are/were:
- circular unit references, especially (but not limited to) with inline declarations
- use of generics

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 12031
  • FPC developer.
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #6 on: January 21, 2025, 11:21:37 am »
(and the timeframe of those changes is circa may 2024. Try a main from early 2024 to test if those were the changes that cause your observations)

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 10794
  • Debugger - SynEdit - and more
    • wiki
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #7 on: January 21, 2025, 11:34:11 am »
Lets imagine situation. You work on your custom memory manager. Included in main project file as first unit. Do some changes in it. How many units of have to be recompile? fpc 3.3.1 does recompile whole project. Size of project is whats increase time dramatically. Maybe it  is just bad project design. I have what i have.

Well, then I imagine you have a reproducible example that you could provide?

But also, if I imagine: A mem mgr, included ("used via uses"?) as first unit in the main file. That means that the system unit will still use the normal mem mgr for initialization.

There are some cmd line args that can be passed to fpc and get the unit used even before that.
In case you relay on only your mem mgr being used, otherwise never mind.


Quote
Fun fact for nonbelievers: comments take time to compile as well.
Of course, after all they...
- need to be loaded from disk
- parsed for where they end


Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 10794
  • Debugger - SynEdit - and more
    • wiki
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #8 on: January 21, 2025, 11:37:16 am »
Fun question: How did you take the times and observations?

I hope you did call fpc from command line for all test runs? Just to make sure no other program interfered? E.g. if calling from any editor, ide or other tool, any of those may have randomly injected other changes of which you may not have been aware (and even, which they maybe should not have, so that those changes would not even be expected)

Thaddy

  • Hero Member
  • *****
  • Posts: 16520
  • Kallstadt seems a good place to evict Trump to.
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #9 on: January 21, 2025, 11:38:06 am »
Lets imagine situation. You work on your custom memory manager. Included in main project file as first unit. Do some changes in it. How many units of have to be recompile? fpc 3.3.1 does recompile whole project. Size of project is whats increase time dramatically. Maybe it  is just bad project design. I have what i have.
AH, I see, you are not talking about fpc, but about an updated Lazarus and that indeed needs - currently - a full recompile.
But that is only ONCE.
As long as you stay on the same version you will hardly notice the difference.

About memory managers: show some code or post in private: I have written some and contributed to some, so I know since 1999.

Memory managers are usually short enough to post here.
« Last Edit: January 21, 2025, 11:43:25 am by Thaddy »
But I am sure they don't want the Trumps back...

cdbc

  • Hero Member
  • *****
  • Posts: 1814
    • http://www.cdbc.dk
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #10 on: January 21, 2025, 11:48:20 am »
Hi
Hmmmm, I'd like to think that I'd want the compiler to recompile influenced units, if I make changes in/to my custom memory-manager...  %)
Wouldn't you?!?
Regards Benny
If it ain't broke, don't fix it ;)
PCLinuxOS(rolling release) 64bit -> KDE5 -> FPC 3.2.2 -> Lazarus 2.2.6 up until Jan 2024 from then on it's: KDE5/QT5 -> FPC 3.3.1 -> Lazarus 3.0

mika

  • Full Member
  • ***
  • Posts: 118
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #11 on: January 22, 2025, 11:37:30 pm »
Thank you everyone.
Your answers with little bit more research and my frustration growing into desperation because there is nothing that can be done. I am new in this particular project. Turned out that every unit is somehow interlinked with each other. Most of units do pull in whole thing for recompilation.

Well, then I imagine you have a reproducible example that you could provide?

But also, if I imagine: A mem mgr, included ("used via uses"?) as first unit in the main file. That means that the system unit will still use the normal mem mgr for initialization.

The custom memory manager is just for debugging reasons and sitting on top of original. I have one procedure in interface that I did use in one of root unit. That is all the mystery.

Waiting minute to just find out you have column instead of semi-column. And then wait yet another minute to test small piece of code. This might be reason to stick to fpc 3.2.x little bit longer (maybe forever).

Fun question: How did you take the times and observations?
Eyeballing is my observation method. If you have comment lines 10k and 1k lines of code then remove comments and you will feel delighted with compilation speed.

Proper measurements could be done, but that does not make it less fun.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 10794
  • Debugger - SynEdit - and more
    • wiki
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #12 on: January 23, 2025, 02:01:39 am »
Waiting minute to just find out you have column instead of semi-column. And then wait yet another minute to test small piece of code. This might be reason to stick to fpc 3.2.x little bit longer (maybe forever).

Off topic, but the Lazarus editors highlight can help you here.

1) https://wiki.freepascal.org/New_IDE_features_since#Multiple_user_defined_word_highlight/markup
Setup your own typical mistakes to be highlighted with noticeable colors (e.g. red background). Such as ";="

2) In Lazarus 4.0
The pascal highlighter will have custom tokens, where you can highlight a single ":", if it is a symbol. This will color it in correct places too. (like "var a: byte"). But it will not color it in strings, or if part of ":=".

Fred vS

  • Hero Member
  • *****
  • Posts: 3478
    • StrumPract is the musicians best friend
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #13 on: January 23, 2025, 02:34:42 am »
Same kind of problem here with last fpc main/trunk.

Compiling big apps without the -B parameter makes fpc recompile all the units at first compilation and it is ok.
But recompiling without any change and without -B, all the units are once again recompiled.

I did try the same with fpc 3.2.2 and 3.2.4, the first time, all the units are compiled of course it is ok.
But recompiling without -B, fpc 3.2.2/3.3.4 only recompiles very few units.

To complicate the thing, for little project, it is ok and this make difficult to give example that target the problem.

To resume, for example recompiling mseide with fpc trunk without -B ---> recompilation every time all units.

But compiling this code:

Code: Pascal  [Select][+][-]
  1. program test1;
  2. uses
  3. BGRABitmapTypes;
  4. begin
  5. end.

With this:
Code: Pascal  [Select][+][-]
  1. $ cd /dir/of/bgrabitmap
  2. $ fpc test1.pas

compiles all the units at first time, but without -B at next compilation fpc trunk only does a link of the already compiled units.
« Last Edit: January 23, 2025, 03:01:12 am by Fred vS »
I use Lazarus 2.2.0 32/64 and FPC 3.2.2 32/64 on Debian 11 64 bit, Windows 10, Windows 7 32/64, Windows XP 32,  FreeBSD 64.
Widgetset: fpGUI, MSEgui, Win32, GTK2, Qt.

https://github.com/fredvs
https://gitlab.com/fredvs
https://codeberg.org/fredvs

PascalDragon

  • Hero Member
  • *****
  • Posts: 5855
  • Compiler Developer
Re: Does unit recompilation roles have changed in fpc 3.3.1?
« Reply #14 on: January 23, 2025, 09:15:31 pm »
Is it mine perception that deceiving me or is this a new reality of happy fpc 3.3.1 users and future for fpc 4.0.0?

No, that is not your perception deceiving you. Since the rewrite of the unit compilation code last spring/summer there is an issue with projects with larger dependencies like for example the compiler itself. This is one of the reasons why I don't consider main as even remotely release ready, cause this consistent recompilation is a no-go.

Same kind of problem here with last fpc main/trunk.

Compiling big apps without the -B parameter makes fpc recompile all the units at first compilation and it is ok.
But recompiling without any change and without -B, all the units are once again recompiled.

We are aware of this. The problem is that we haven't yet managed to find a small example that allows to easily reproduce and thus debug this (e.g. compiling the compiler triggers this issue as well). My suspicion is that it's depending on the dependencies between the units, so it should essentially be reproducible with empty units that only contain uses-clauses.

 

TinyPortal © 2005-2018