Recent

Author Topic: How to analyze / debug compiler crashes?  (Read 435 times)

SirTwist

  • New Member
  • *
  • Posts: 28
How to analyze / debug compiler crashes?
« on: June 21, 2026, 10:24:27 am »
Good morning fellow developers,

I am using FP/Laz for roughly 10 years and mostly writing programs for my very own usage or f or our dance sport comunity.

Whenever my projects are growing larger (15.000 lines of code, approx. 20 forms and frames), I encounter "randomly" compiler crashes. In many cases they only ocure when I do a normal compile (Ctrl-F9), and a "build" (Shift-F9) will then run through.
But now I reached again the state where even "Build" will crash. After the crash, the IDE points either to the last line of a random unit (with a column 50,000 or above), or at some 'interface' definition of an generic object definition.

I am heavily using generics. And packages, both written by myself or third-party-packages.

I theoretically know how to raise a bug report, but I would need to upload the whole source code and all the packages etc. into public.

Does anyone have some tips how to debug such a compiler crash by myself?

I am using Lazarus 3.4 with FPC 3.2.2 on Windows 10, installed by pfcupdeluxe. I also tried to update the compiler, but than some of the third-party packages would not compile.

Any help appreciated,

Kind regards,
Sir Twist

Code: Pascal  [Select][+][-]
  1. UlcDmxUniverses.pas(359,57867) Error: Compilation raised exception internally
  2. Verbose: Compilation aborted
  3. Debug: An unhandled exception occurred at $000000010003C5A4:
  4. Debug: EAccessViolation: Access violation
  5. Debug:   $000000010003C5A4
  6. Debug:   $000000010019C48C
  7. Debug:   $000000010019D46E
  8. Debug:   $000000010003DCE5
  9. Debug:   $00000001000D4805
  10. Debug:   $000000010019C469
  11. Debug:   $000000010019D46E
  12. Debug:   $000000010003DCE5
  13. Debug:   $00000001000D4805
  14. Debug:   $000000010019C469
  15. Debug:   $000000010019D7CE
  16. Debug:   $000000010003DCE5
  17. Debug:   $00000001000D4805
  18. Debug:   $000000010019C469
  19. Debug:   $000000010019D46E
  20. Debug:   $000000010003DCE5
  21. Debug:   $00000001000D4805

Fibonacci

  • Hero Member
  • *****
  • Posts: 1029
  • Behold, I bring salvation - FPC Unleashed
Re: How to analyze / debug compiler crashes?
« Reply #1 on: June 21, 2026, 11:05:15 am »
installed by pfcupdeluxe. I also tried to update the compiler, but than some of the third-party packages would not compile.

(...)

Code: Pascal  [Select][+][-]
  1. UlcDmxUniverses.pas(359,57867) Error: Compilation raised exception internally
  2. Verbose: Compilation aborted
  3. Debug: An unhandled exception occurred at $000000010003C5A4:
  4. Debug: EAccessViolation: Access violation
  5. Debug:   $000000010003C5A4
  6. Debug:   $000000010019C48C
  7. Debug:   $000000010019D46E
  8. Debug:   $000000010003DCE5
  9. Debug:   $00000001000D4805
  10. Debug:   $000000010019C469
  11. Debug:   $000000010019D46E
  12. Debug:   $000000010003DCE5
  13. Debug:   $00000001000D4805
  14. Debug:   $000000010019C469
  15. Debug:   $000000010019D7CE
  16. Debug:   $000000010003DCE5
  17. Debug:   $00000001000D4805
  18. Debug:   $000000010019C469
  19. Debug:   $000000010019D46E
  20. Debug:   $000000010003DCE5
  21. Debug:   $00000001000D4805

Setup+ -> Options override -> FPC Options -> tick "[ ] Debug" -> Rebuild, and wait for another crash.
FPC Unleashed: async/await, parallel for, match, tuples, string interpolation, inline vars, autofree, no-RTTI & tons more. Star on GitHub

jamie

  • Hero Member
  • *****
  • Posts: 7832
Re: How to analyze / debug compiler crashes?
« Reply #2 on: June 21, 2026, 11:15:50 am »
Did i read that correctly? 57000+ chars on a single line?
The only true wisdom is knowing you know nothing

jamie

  • Hero Member
  • *****
  • Posts: 7832
Re: How to analyze / debug compiler crashes?
« Reply #3 on: June 21, 2026, 11:36:10 am »
I wanted to add that i stopped using the generics collection unit because it crashes the compiler, ends with a error pointing at the main project file looking for a asm lable . I guess it chews up memory fast.
 I questioned the use of a ref being made to a zero sized record on the heap.
Jamie
The only true wisdom is knowing you know nothing

Thaddy

  • Hero Member
  • *****
  • Posts: 19432
  • Glad to be alive.
Re: How to analyze / debug compiler crashes?
« Reply #4 on: June 21, 2026, 11:47:40 am »
@Jamie
He mentioned the bogus line size in the question:
Quote
After the crash, the IDE points either to the last line of a random unit (with a column 50,000 or above), or at some 'interface' definition of an generic object definition.
I agree the cause can be the use of generics, but many of these issues are fixed in trunk, some back-ported to 3.2.3 /3.4 Rc
« Last Edit: June 21, 2026, 11:59:16 am by Thaddy »
Any "programmer" that knows only one programming language is not a programmer

jcmontherock

  • Sr. Member
  • ****
  • Posts: 360
Re: How to analyze / debug compiler crashes?
« Reply #5 on: June 21, 2026, 11:48:46 am »
... And an application containing 15'000 lines of code !...
Windows 11 UTF8-64 - Lazarus 4.8-64 - FPC 3.2.2

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12537
  • Debugger - SynEdit - and more
    • wiki
Re: How to analyze / debug compiler crashes?
« Reply #6 on: June 21, 2026, 11:51:04 am »
First of all: Debugging the compiler will be the lesser worry you have. Fixing the issue will most likely need some understanding of the structures used in the compiler.

If you want to take that on, https://wiki.freepascal.org/Compiler_development_articles look at stuff like the "FPC internals".

Since you are using FpcUpDeluxe, that should allow you to set he correct arguments for building FPC with debug info:
* for gdb -gw2
* for FpDebug: -gw3
and in any case you likely also want: -gl -Criot

There is a file compiler/ppcx86.lpi which you can open in lazarus and then run in the debugger. Menu Run > Run parameters: for command line
(you may need any of the ppc*.lpi) depending on your target architecture / you should have the ppc already build, and not let the IDE attempt to build it)

Alternatively debug from command line using gdb or lldb.

Whenever my projects are growing larger (15.000 lines of code, approx. 20 forms and frames), I encounter "randomly" compiler crashes. In many cases they only ocure when I do a normal compile (Ctrl-F9), and a "build" (Shift-F9) will then run through.
Which points to ppu loading, or rather to using data loaded from ppu, instead of data generated during the current run of the compiler. One possibility is that some data is not loaded correctly and something is left uninitialized.

If it turns out something that may have been freed => try valgrind.

Quote
But now I reached again the state where even "Build" will crash. After the crash, the IDE points either to the last line of a random unit (with a column 50,000 or above), or at some 'interface' definition of an generic object definition.
FPC 3.3.1 (maybe even 3.2.3) had some fixes around ppu loading. But you then need to adapt any code that doesn't compile.

3.2.3 would be preferable as it likely is more stable.

Have you tried to really remove **all** ppu from all folders/packages before the build?

In general such errors have in part been known to be caused by circular unit references (uses in implementation). You may want to try to get rid of them if you can. (I know, easier said ...)

Quote
I theoretically know how to raise a bug report, but I would need to upload the whole source code and all the packages etc. into public.

Does anyone have some tips how to debug such a compiler crash by myself?

See above... That is going to be a real deep dive.

Maybe if you capture the crash you get an idea how to extract a small sample for reproduction => I wouldn't bet on it.

Also, submitting an issue may not help at all, if it is already fixed in 3.3.1. Because then whats needed instead is the commit that fixed it, so it can be reviewed if it can be merged to 3.2.3 (if not yet done).

Well, if it is just a commit, afaik ppu loading had major rework done in 3.3.1. (but don't know for sure).



Since you may have to rebuild your FPC anyway, try building with -O-
There is a tiny (extremely slim) chance that it may help.

IIRC (and again, sorry not sure), you need than in "OPT" and "OPTNEW"... Something like
make all LINKSMART=0  CREATESMART=0  OPTIMIZE=0  OPT="-O- -gw3 -gl "  OPTNEW="-O- -gw3 -gl  "   

Adding -g- -gtttt  (before the -gl and before the -gw3) may also help. I think its 4 "t" (may be 3). If you have the correct number, it will set uninitialized values to 0/nil => and that may just *hide* the issue (with a real big amount of luck).
« Last Edit: June 21, 2026, 11:53:25 am by Martin_fr »

Thaddy

  • Hero Member
  • *****
  • Posts: 19432
  • Glad to be alive.
Re: How to analyze / debug compiler crashes?
« Reply #7 on: June 21, 2026, 12:00:09 pm »
... And an application containing 15'000 lines of code !...
Frankly that is nothing.. I have apps with 800.000+ lines of code.
( Another example: Kol.pas alone uses the (Delphi) maximum code lines (65535) in a single unit and needs to rely on includes)
« Last Edit: June 21, 2026, 12:08:53 pm by Thaddy »
Any "programmer" that knows only one programming language is not a programmer

SirTwist

  • New Member
  • *
  • Posts: 28
Re: How to analyze / debug compiler crashes?
« Reply #8 on: June 21, 2026, 12:07:22 pm »

I agree the cause can be the use of generics, but many of these issues are fixed in trunk, some back-ported to 3.2.3 /3.4 Rc
Thank you all for taking care. In the meantime, I installed FPC 3.2.3 from fpcupdeluxe, and still got the same crash. Now trying the debugging options which were mentioned in other answers.

Is there any other "generics" replacement available?

Kind regards,
Sir Twist

Thaddy

  • Hero Member
  • *****
  • Posts: 19432
  • Glad to be alive.
Re: How to analyze / debug compiler crashes?
« Reply #9 on: June 21, 2026, 12:10:09 pm »
Depending on how you use generics? Yes, fgl, lgenerics to name two general purpose ones.
Better show us a small example in how you use generics: many people use them wrong.
« Last Edit: June 21, 2026, 12:16:33 pm by Thaddy »
Any "programmer" that knows only one programming language is not a programmer

jamie

  • Hero Member
  • *****
  • Posts: 7832
Re: How to analyze / debug compiler crashes?
« Reply #10 on: June 21, 2026, 12:16:01 pm »
U may wanr to try using a external debug file, i do that to make the finale fule much smaller.

May not fix your issue but it makes me happier.
The only true wisdom is knowing you know nothing

SirTwist

  • New Member
  • *
  • Posts: 28
Re: How to analyze / debug compiler crashes?
« Reply #11 on: June 21, 2026, 12:28:40 pm »
Setup+ -> Options override -> FPC Options -> tick "[ ] Debug" -> Rebuild, and wait for another crash.

Code: Pascal  [Select][+][-]
  1. UlcDmxUniverses.pas(359,57867) Error: Compilation raised exception internally
  2. Verbose: Compilation aborted
  3. Debug: An unhandled exception occurred at $0000000100040514:
  4. Debug: EAccessViolation: Access violation
  5. Debug:   $0000000100040514  ADDDEPENDENCY,  line 915 of fmodule.pas
  6. Debug:   $00000001001A116C  LOADUNITS,  line 519 of pmodules.pas
  7. Debug:   $00000001001A214E  PROC_UNIT,  line 967 of pmodules.pas
  8. Debug:   $0000000100041C55  COMPILE,  line 399 of parser.pas
  9. Debug:   $00000001000D9155  LOADPPU,  line 2133 of fppu.pas
  10. Debug:   $00000001001A1149  LOADUNITS,  line 514 of pmodules.pas
  11. Debug:   $00000001001A214E  PROC_UNIT,  line 967 of pmodules.pas
  12. Debug:   $0000000100041C55  COMPILE,  line 399 of parser.pas
  13. Debug:   $00000001000D9155  LOADPPU,  line 2133 of fppu.pas
  14. Debug:   $00000001001A1149  LOADUNITS,  line 514 of pmodules.pas
  15. Debug:   $00000001001A24AE  PROC_UNIT,  line 1058 of pmodules.pas
  16. Debug:   $0000000100041C55  COMPILE,  line 399 of parser.pas
  17. Debug:   $00000001000D9155  LOADPPU,  line 2133 of fppu.pas
  18. Debug:   $00000001001A1149  LOADUNITS,  line 514 of pmodules.pas
  19. Debug:   $00000001001A214E  PROC_UNIT,  line 967 of pmodules.pas
  20. Debug:   $0000000100041C55  COMPILE,  line 399 of parser.pas
  21. Debug:   $00000001000D9155  LOADPPU,  line 2133 of fppu.pas
  22. Debug:

Hm.... "Add Dependecy" does that mean I have to many units?

SirTwist

  • New Member
  • *
  • Posts: 28
Re: How to analyze / debug compiler crashes?
« Reply #12 on: June 21, 2026, 12:44:08 pm »
Depending on how you use generics? Yes, fgl, lgenerics to name two general purpose ones.
Better show us a small example in how you use generics: many people use them wrong.
Thanks again for diving into that!

I have defined myself a base object TmsObj, and a (non-generics) base list TmsObjList, which implements basically everything from the plain old TList. From TmsObjList I've derived a TmsList<T: TmsObj> generic list.
In my applications, I derive many classes from TmsObj and many lists from TmsList<T>. E.g. in my actual application, I have config objects decleared as TmsCfgObj and a TmsCfgList<T: TmsCfgObj>. Then each type of config object I derived from TmsCfgObj resp. TmsCfgList, e.g. TmsFixture = class(TmsCfgObj) and TmsFixtures = specialize TmsCfgList<TmsFixture> etc etc...
("ms" are my initials, not realted to that one software company...)

Code: Pascal  [Select][+][-]
  1.   TmsObj = class
  2.   private
  3.    ...
  4.   public
  5.     constructor    Create; virtual;
  6.     destructor     Destroy; override;
  7.     property       ID: TGUID read GetID write SetID;
  8.   published
  9.     property       ID_Str: String read GetID_Str write SetID_Str;
  10.     property       Name: String read InternalGetName write InternalSetName;
  11.   end;
  12.  
  13.   TmsObjList = class(TmsObj)
  14.   private
  15.   protected
  16.   public
  17.     constructor    Create(aOwnsObjects: Boolean = False); virtual; reintroduce;
  18.     constructor    Create(aOwner: TmsObj); overload;
  19.     destructor     Destroy; override;
  20.     property       Count: Integer read fCount write SetCount;
  21.     property       Capacity: Integer read fCapacity;
  22.     procedure      Clear; virtual;
  23.     function       New: TmsObj;
  24.     function       Add(aItem: TmsObj): TmsObj; virtual;
  25.     function       Add: TmsObj; overload;
  26.     function       Insert(aIndex: Integer; aItem: TmsObj): TmsObj;
  27.     procedure      Remove(aItem: TmsObj); virtual;
  28.     procedure      Delete(aIndex: Integer); virtual;
  29.     function       ExtractByIndex(aIndex: Integer): TmsObj;
  30.     function       Extract(aItem: TmsObj): TmsObj;
  31.     function       GetEnumerator: TmsObjListEnumerator; {$ifdef FGLINLINE} inline; {$endif}
  32.     property       Items[aIndex: Integer]: TmsObj read GetItem write PutItem; default;
  33.         ...
  34.   end;
  35.   TmsObjListClass = class of TmsObjList;
  36.  
  37.   generic TmsList<T: TmsObj> = class(TmsObjList)
  38.   type
  39.     TmsListEnumeratorSpec = specialize TmsListEnumerator<T>;
  40.   private
  41.     function       GetItem(aIndex: Integer): T;
  42.     procedure      PutItem(aIndex: Integer; aValue: T);
  43.   public
  44.     constructor    Create(aOwnsObjects: Boolean = False); override; overload;
  45.     constructor    Create(aOwner: TmsObj); overload;
  46.     function       New: T;
  47.     function       Add(aItem: T): T; virtual;
  48.     function       Add: T; virtual; overload;
  49.     function       Insert(aIndex: Integer; aItem: T): T;
  50.     procedure      Remove(aItem: T); virtual;
  51.     function       ExtractByIndex(aIndex: Integer): T;
  52.     function       Extract(aItem: T): T;
  53.     function       GetEnumerator: TmsListEnumeratorSpec; {$ifdef FGLINLINE} inline; {$endif}
  54.     //
  55.     property       Items[aIndex: Integer]: T read GetItem write PutItem; default;
  56.   end;
  57.  

jamie

  • Hero Member
  • *****
  • Posts: 7832
Re: How to analyze / debug compiler crashes?
« Reply #13 on: June 21, 2026, 12:56:39 pm »
Maybe u have to many lines in a single unit.
 I to get an internal compiler error with no details. A build takes care of it.
The only true wisdom is knowing you know nothing

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12537
  • Debugger - SynEdit - and more
    • wiki
Re: How to analyze / debug compiler crashes?
« Reply #14 on: June 21, 2026, 02:39:23 pm »
looking at the extract of the callstack.... Based on my (couple of weeks old) copy of 3.2.3 sources.

It first looks like callermodule might be trashed, but that can't be, because its accessed before the call (and didn't crash there).

But it might be (not sure...) an exhausted stack (compile your fpc with stack checking).
At least its nested compile > load > compile > load .... Don't know how deep.

Though this isn't a call, just a field access => if it was deep recursion, I would have expected the error on a diff line.

Just a guess, can't do much more from that bit of stack.

 

TinyPortal © 2005-2018