"Compilation raised exception internally" on recent 3.3.1 compiler builds

[Paradice]

All of the recent development builds of fpc 3.3.1 (up to and including the current build from May 20th 2024) have started throwing an unhandled exception error when compiling my largest project. Target and platform are both Win32/i386. Compiler builds compiled with FpcUpDeluxe. Not using Lazarus at all.

I have a compiler from 3.3.1 dev branch version from Feb 27th 2024 which compiles everything successfully, but newer versions bug out. (not sure exactly what date it failed, I only rebuild FPC every month or so - it's been failing for the past ~3 months).

With recent versions, using -B to build all sources will work as expected, but even immediately after that, attempting a simple recompile of any source file within the project without -B will trigger this error. Unfortunately, reducing it down to a "minimum source to reproduce the error" is tricky, I'm only experiencing this on my main project which is probably ~150 units of various interconnections. Which unit is the last unit "Compiling" before the exception varies, but it seems to always be 3 or 4 units after whatever I first attempt to compile.

Any advice on what to look into? I'm not very experienced with git, hence using FPCUpDeluxe to compile FPC for me.

---
Command line:

Code: Pascal  [Select][+][-]

1. E:\FPC\3.3.1\bin\i386-win32\fpc.exe @E:\Projects\.Meta\FPCsettings331.cfg -dDEBUG E:\Projects\Paradice500\Paradice500.pp

----
Output:

Code: Pascal  [Select][+][-]

1. Free Pascal Compiler version 3.3.1-15711-geff10ee9b7 [2024/05/20] for i386
2. Copyright (c) 1993-2024 by Florian Klaempfl and others
3. Target OS: Win32 for i386
4. Compiling E:\Projects\Paradice500\Paradice500.pp
5. Compiling E:\projects\lx7\lx7core.pp
6. Compiling E:\projects\lx7\lx7debug.pp
7. Compiling coreStrings.pp
8. Error: Compilation raised exception internally
9. Fatal: Compilation aborted
10. An unhandled exception occurred at $00437B8A:
11. EAccessViolation: Access violation
12.   $00437B8A
13.   $0056A0FD
14.   $0056A74D
15.   $0056AA9C
16.   $0056A284
17.   $0056A74D
18.   $0056AA9C
19.   $0056A0E4
20.   $0056A74D
21.   $0056AA9C
22.   $0056A0E4
23.   $0056A74D
24.   $0056AA9C
25.   $0056A0E4
26.   $0056A74D
27.   $0056AA9C
28.   $005B4C37

[jamie]

I don't fully know the switches of the compiler, nor should there be an error like that, which of course is a reportable error.

But, with Laz, I elect to use an external debug file so that it does not attach itself to the EXE., makes it much smaller.

Maybe you are running out of space and the compiler is not catching it at the correct moment?

[Eugene Loza]

Unfortunately things like that happen, and there isn't much you can do about that. If clean-up and build fixes the issue, then you might as well just create a script to delete all o and ppu files in the temporary output folder - this is slightly faster than recompiling the project and all dependencies.

Cutting your project down to minimal reproducible example for a bugreport is almost the only way you can try dealing with the issue. Yes, it takes time (last bugreport took me >3 hours to cut from 40k lines down to 70). Making a bisection of FPC history to find a specific commit that started the bug also may help. Yes, eventually the result may be very delayed - as someone smart enough will need to investigate your bugreport and spend a ton of time understanding what is going wrong and how to fix it.

There is also an option to rewrite/simplify the code, it seems like convoluted generic operations are very prone to confuse FPC. But for that you need to try to figure out what exactly goes wrong. It may help if you get a debug build of FPC, see attached image on how to do that in FPCUpDeluxe.

[Paradice]

Thanks for your help, Eugene! - I've recompiled the compiler in debug mode to see if it gave me any clues... now I'm getting this output:

Code: Pascal  [Select][+][-]

1. E:\FPC\3.3.1\bin\i386-win32\fpc.exe @E:\Projects\.Meta\FPCsettings331.cfg -dRELEASE E:\Projects\Paradice500\Paradice500.pp
2. Compiling Release Version
3. Compiling Release Version
4. Hint: End of reading config file E:\Projects\.Meta\FPCsettings331.cfg
5. Free Pascal Compiler version 3.3.1-15713-gcab3b8c06f [2024/05/20] for i386
6. Copyright (c) 1993-2024 by Florian Klaempfl and others
7. Target OS: Win32 for i386
8. Compiling E:\Projects\Paradice500\Paradice500.pp
9. Compiling E:\projects\lx7\lx7core.pp
10. Compiling E:\projects\lx7\lx7events.pp
11. Compiling coreStrings.pp
12. Error: Compilation raised exception internally
13. Fatal: Compilation aborted
14. An unhandled exception occurred at $00438D2A:
15. EAccessViolation: Access violation
16.   $00438D2A  TMODULE__ADDDEPENDENCY,  line 1005 of fmodule.pas
17.   $0056B2CD  TPPUMODULE__LOAD_USEDUNITS,  line 1958 of fppu.pas
18.   $0056B91D  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2221 of fppu.pas
19.   $0056BC6C  TPPUMODULE__LOADPPU,  line 2362 of fppu.pas
20.   $0056B454  TPPUMODULE__LOAD_USEDUNITS,  line 2014 of fppu.pas
21.   $0056B91D  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2221 of fppu.pas
22.   $0056BC6C  TPPUMODULE__LOADPPU,  line 2362 of fppu.pas
23.   $0056B2B4  TPPUMODULE__LOAD_USEDUNITS,  line 1953 of fppu.pas
24.   $0056B91D  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2221 of fppu.pas
25.   $0056BC6C  TPPUMODULE__LOADPPU,  line 2362 of fppu.pas
26.   $0056B2B4  TPPUMODULE__LOAD_USEDUNITS,  line 1953 of fppu.pas
27.   $0056B91D  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2221 of fppu.pas
28.   $0056BC6C  TPPUMODULE__LOADPPU,  line 2362 of fppu.pas
29.   $0056B2B4  TPPUMODULE__LOAD_USEDUNITS,  line 1953 of fppu.pas
30.   $0056B91D  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2221 of fppu.pas
31.   $0056BC6C  TPPUMODULE__LOADPPU,  line 2362 of fppu.pas
32.   $005B5E07  LOADUNITS,  line 681 of pmodules.pas

So, progress.. it's definitely something to do with units I'm including, somehow (maybe order?). I am guilty of several cases of a unit using another unit in it's interface, and the secondary unit uses the original one in it's implementation. It never feels great doing this but it hasn't caused any issues up until now.

This weekend I will try patiently carving chunks of units & functionality out of my project and see if I can narrow down what's causing the error. I suspect that all of the complexity is what's actually causing the error though so making it easily reproducible is not going to be easy at all.


@Jamie - pretty sure it's not related to my system at all. No problems with space, and have tried the same source files on a couple of different systems (one Intel, one AMD) and different directory structures - same issue on both.

[Eugene Loza]

Hmmm... it doesn't look any more helpful to me other than what you've already seen without debug build, that it fails somewhere around coreStrings.pp (hopefully inside of it) as it tries to LOADPPU (i.e. reuse a prebuilt unit, seems like of one of the dependencies of coreStrings.pp).

I wonder if Lazarus can give you a more verbose error stack trace? I see you are using complicated compilation configuration, I wonder if you can try setting it up through Lazarus or making a "less configured" build? Because in my cases it usually points to some line of code inside of my code which may be directly or indirectly related to the error you experience.

[TRon]

Pardon the intrusion...

I wonder if Lazarus can give you a more verbose error stack trace? I see you are using complicated compilation configuration, I wonder if you can try setting it up through Lazarus or making a "less configured" build? Because in my cases it usually points to some line of code inside of my code which may be directly or indirectly related to the error you experience.

The stacktrace already does that ?

Is raises the exception at line 1005 of unit fmodule.pas

Lazarus does not help provide you a better trace. The best you could do is adding -va tot the compiler, perhaps followed by using some linker specific options for debugging to be able to spit out even more information but in this case I don't believe it will help.

Knowing the revision when/where this happens and try to work your way back to a known working version might be able to pinpoint the change in the source tree that might have caused this error (if any).

Also good would be if the problem is reproduceable with a simple(r) example. That makes it a better candidate to add it as an issue (though the current information present would be enough to issue a report).

[Paradice]

I'd be willing to try compiling different revisions of the compiler in order to identify the exact version change where the error first appears..... if only I understood git well enough to be able to download specific historical revisions.

I'm guessing this is not a capability of FPCUpDeluxe (at least, I don't see it after some clicking around).


@Eugene: it's not specific to coreStrings... If I start compiling from coreStrings, it compiles fine and the error occurs in a different unit instead... it always gets 3/4 units in before crashing.

[Eugene Loza]

Is raises the exception at line 1005 of unit fmodule.pas

Indeed, this is how it looks. But in my experience Lazarus always points to some line of my code, not of FPC (e.g. https://forum.lazarus.freepascal.org/index.php/topic,66084.msg516536.html#msg516536 ), even if the internal error happens inside of FPC units with a similar stack trace. Of course, I haven't gotten this specific error myself (yet ), and not such an expert in FPC internals so can't tell for sure.

if only I understood git well enough to be able to download specific historical revisions.

Again FPCUpDeluxe can help with that:

[TRon]

Indeed, this is how it looks. But in my experience Lazarus always points to some line of my code, not of FPC (e.g. https://forum.lazarus.freepascal.org/index.php/topic,66084.msg516536.html#msg516536 ), even if the internal error happens inside of FPC units with a similar stack trace.

In my experience (but I have ditched the platform for a while now) this should not be a problem on Windows (I am aware your posts/experience states otherwise so can only assume it is a problem in/for some (or all) cases).

Things are worse on Linux and Lazarus ads even more confusing traces, unless ofc you know every nitty gritty details of the (LCL) internals by heart.

Again FPCUpDeluxe can help with that:

Indeed that is a (easy) way to do it, particularly if you are not familiar with git.

It is similar as (manually) downloading a specific revision from gitlab.

Using git though makes things a bit easier because it only requires you to download the source-code once and allows to checkout the different revisions (which is actually the whole purpose of using a vcs) see also here.

The process which can be used in this particular case is named bisecting, see also here.

edit:

BTW @Paradice
I have noticed that there in last 2 weeks there is an attempt in the source tree to make some changes/fixes that are able to severely impact compiler behaviour. So you might perhaps want to sit this one through.

[Paradice]

@Eugene thank you so much for your patience in providing screenshots - I've got a workflow down now and am (manually) bisecting builds between my known good compiler and today.... so far I'm back to Mar 29th's build, which interestingly, still exceptioned but in a different place.

I will provide an update about what build first broke and then attempt to turn it into an error report

For completion, this is the crash with Mar 29th revision of compiler (the date still shows today so I'm guessing that's just compilation date):

Code: Pascal  [Select][+][-]

1. Free Pascal Compiler version 3.3.1-15491-ge42209457e [2024/05/20] for i386
2. Copyright (c) 1993-2024 by Florian Klaempfl and others
3. Target OS: Win32 for i386
4. Compiling E:\Projects\Paradice500\Paradice500.pp
5. Compiling E:\projects\lx7\lx7core.pp
6. Compiling E:\projects\lx7\lx7debug.pp
7. Compiling E:\projects\lx7\lx7strings.pp
8. Compiling E:\projects\lx7\lx7core.pp
9. Compiling E:\projects\lx7\lx7events.pp
10. Compiling E:\projects\lx7\lx7debug.pp
11. Compiling E:\projects\lx7\lx7core.pp
12. Compiling E:\projects\lx7\lx7events.pp
13. Error: Compilation raised exception internally
14. An unhandled exception occurred at $005B62EC:
15. EAccessViolation: Access violation
16.   $005B62EC  CONNECT_LOADED_UNITS,  line 777 of pmodules.pas
17.   $005B6AB0  PROC_UNIT_IMPLEMENTATION,  line 1079 of pmodules.pas
18.   $00454D67  TTASK_HANDLER__CONTINUE,  line 273 of ctask.pas
19.   $004175E4  COMPILE,  line 310 of compiler.pas
20.  

[Eugene Loza]

so far I'm back to Mar 29th's build, which interestingly, still exceptioned but in a different place.

My project raises internal errors (or things related to those) in over a dozen different places - some present only in newer compiler versions, some were fixed in newer compiler versions I guess that's the destiny of every project as it grows big enough, especially while using generics. I've managed to isolate and bugreport the most annoying one though

[Paradice]

OK, have narrowed it down

The Mar 06 2024 build (f41025f5dd7d1b7fb4367f0aceac021a7a1f8478) breaks:

Code: Pascal  [Select][+][-]

1. Free Pascal Compiler version 3.3.1-15377-gf41025f5dd [2024/05/20] for i386
2. Copyright (c) 1993-2024 by Florian Klaempfl and others
3. Target OS: Win32 for i386
4. Compiling E:\Projects\Paradice500\Paradice500.pp
5. Compiling E:\projects\lx7\lx7core.pp
6. Compiling E:\projects\lx7\lx7debug.pp
7. Compiling E:\projects\lx7\lx7strings.pp
8. Compiling E:\projects\lx7\lx7core.pp
9. Compiling E:\projects\lx7\lx7array.pp
10. Compiling E:\projects\lx7\lx7hash.pp
11. Compiling E:\projects\lx7\lx7debug.pp
12. Compiling E:\projects\lx7\lx7files.pp
13. Compiling E:\projects\lx7\lx7sysinfo.pp
14. Compiling E:\projects\lx7\lx7events.pp
15. Compiling coreStrings.pp
16. Error: Compilation raised exception internally
17. Fatal: Compilation aborted
18. An unhandled exception occurred at $0043908A:
19. EAccessViolation: Access violation
20.   $0043908A  TMODULE__ADDDEPENDENCY,  line 996 of fmodule.pas
21.   $00569F2D  TPPUMODULE__LOAD_USEDUNITS,  line 1951 of fppu.pas
22.   $0056A56C  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2206 of fppu.pas
23.   $0056A8A4  TPPUMODULE__LOADPPU,  line 2342 of fppu.pas
24.   $0056A0B4  TPPUMODULE__LOAD_USEDUNITS,  line 2007 of fppu.pas
25.   $0056A56C  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2206 of fppu.pas
26.   $0056A8A4  TPPUMODULE__LOADPPU,  line 2342 of fppu.pas
27.   $00569F14  TPPUMODULE__LOAD_USEDUNITS,  line 1946 of fppu.pas
28.   $0056A56C  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2206 of fppu.pas
29.   $0056A8A4  TPPUMODULE__LOADPPU,  line 2342 of fppu.pas
30.   $00569F14  TPPUMODULE__LOAD_USEDUNITS,  line 1946 of fppu.pas
31.   $0056A56C  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2206 of fppu.pas
32.   $0056A8A4  TPPUMODULE__LOADPPU,  line 2342 of fppu.pas
33.   $00569F14  TPPUMODULE__LOAD_USEDUNITS,  line 1946 of fppu.pas
34.   $0056A56C  TPPUMODULE__TRY_LOAD_PPUFILE,  line 2206 of fppu.pas
35.   $0056A8A4  TPPUMODULE__LOADPPU,  line 2342 of fppu.pas
36.   $005B3E6B  LOADUNITS,  line 675 of pmodules.pas

While the Mar 04 build (bf2a3566fa7e4c9ff400c9ded3cd83a61e4b6e3a0) works as expected:

Code: Pascal  [Select][+][-]

1. Free Pascal Compiler version 3.3.1-15331-gbf2a3566fa [2024/05/20] for i386
2. Copyright (c) 1993-2024 by Florian Klaempfl and others
3. Target OS: Win32 for i386
4. Compiling E:\Projects\Paradice500\Paradice500.pp
5. User defined: /*-------------------------*\
6. User defined: | COMPILING: STEAM Edition  |
7. User defined: |     Debugging Information |
8. User defined: \*-------------------------*/
9. Compiling resource E:\Projects\Paradice500\resource500.rc
10. Compiling resource E:\Projects\Paradice500\Paradice500.or
11. Linking E:\Projects\Paradice500\Paradice500.exe
12. 85 lines compiled, 0.3 sec, 946656 bytes code, 22372 bytes data
13. 1 hint(s) issued
14.  

With all versions of the compiler, a full rebuild of the sources works without errors. Then executing a normal compilation immediately afterward shows the differences.  You can see in the Mar 04 version the compiler clearly realizes that none of the units have changed so it *just* builds the main paradice500.pp file.  While the newer compiler versions start recompiling the units again (so already a bug?) and start doing so, then crash shortly afterwards.

I can see that there were some commits on 4/5 Mar specifically altering things like load_ppu... so that will be what broke the compilation.

Is there anything else I should be collecting to go with a bug report? The sources themselves are still going to be very problematic...

[Eugene Loza]

It's also a better idea to include a direct link to the commit: https://gitlab.com/freepascal.org/fpc/source/-/commit/a5caf91f

And the commit message indeed seems like a very good candidate for your specific error. Potentially you may not need to cut down your project to reproducible minimum to file a bug report this way then. But it'd be very good if you could provide the project to reproduce the bug (if you are able to, e.g. if the project is opensource or you can send it in private to someone who will take care of the ticket - otherwise it will still need some not-minimal reproduction you can attach to the bugreport in the open).

[TRon]

OK, have narrowed it down

Thank you very much for having taken the time to do so.

I can see that there were some commits on 4/5 Mar specifically altering things like load_ppu... so that will be what broke the compilation.

Yes, that is indeed still a problem. There is an existing report though not as specific as yours so your mileage may vary (I would suggest creating a new report, it can always be marked as duplicate)

Is there anything else I should be collecting to go with a bug report? The sources themselves are still going to be very problematic...

No way to re-produce with a minimalistic example ? It can be tiresome but sometimes stripping down an existing project might be able to do that for you. If it is possible to share the sources then please mention that in your report. As mentioned by Eugene that might save you the trouble of cutting down your existing project.

[marcov]

It is a known issue that is still not resolved, try to use -Ur as much as possible.