Lazarus/FPC semi-randomly gets stuck while compiling

[Nel_Fie]

As the title says, I've been running into a bizarre issue: Lazarus somewhat randomly gets stuck while compiling a specific project I'm working on, for no apparent reason.

But Lazarus doesn't hang or crash or freeze. I can interact with it just fine and abort the compilation process (with no apparent error message), it's just that the compilation process itself somehow gets stuck at a certain step and refuses to progress, sort of as if the FPC was independently caught in a loop. But while the step seems to be the exact same for a particular instance of the issue, every separate instance of the issue seems to get stuck at a different step.

Yet it doesn't seem related to what is being compiled at that step: it doesn't display any kind of error message, and when I try to do something about the offending line or file, it simply gets stuck at the second-to-last step instead. While this could suggest that the actual offender is not the last step, but the next step after that which is not yet displayed in Messages; I've tried tracking what would happen afterwards and modify that, but that doesn't seem to have an impact either. The issue only disappears sometimes after I've spent enough time working on my project.

But what makes the whole thing stranger is that the project will sometimes compile just fine again after restarting either Lazarus or my computer, and I've found a "quick-fix" which makes little sense to me: simply switching the Optimization level (in "Options for Project>Compiler Options>Compilation and linking") to a different level solves the issue - but only once.
Since it happens pretty frequently, I have to constantly switch back and forth between different optimization levels in order to work, which isn't a major roadblock, but still a bit of a hassle. So it seems more like Lazarus is caching some kind of data between compilations for this specific project, and this cache messes with subsequent runs, until doing something such as switching a parameter like optimization voids that cache, but I don't know enough about the process to tell if that's a correct interpretation.

Does anyone else had this happen to them before? Or any idea what might cause it? I can't really report it since I have no idea what the root of the issue might be, and all my other projects run fine.

[dbannon]

Nel, I'd suspect a software issue but as no one else has (yet) reported such an issue, I'd doubt it.  And you sound sure its not your code ?

What are you running (FPC ver and Lazarus ver) and on what hardware ?

Compiling often uses a lot of memory. Sometimes, memory that is rarely used. So, next I'd run a aggressive memory checker. Not one of the short, few minute things, a multiple combination, sloooow one. Expect as much as an hour to be sure.

Most linux ISOs have a memory checker available in their boot up menu.

I hope that is not the answer ....

Davo

[440bx]

I'm going to echo @dbannon's comments.

Any kind of problem report should be accompanied by the Lazarus and FPC versions. 

The capricious behavior of the problem you described does raise a good possibility that there might be a hardware problem, likely memory as pointed out by @dbannon.  To be on the safe side, I'd check the hard drive and file system too.

Start by posting the FPC and Lazarus version. 

HTH.

[Martin_fr]

When looking at the output, use the context menu "copy all/original messages". That may give additional info.

I am not sure though if you will get much useful info.

Once you had the problem, does Lazarus still show "compiling" or does it show the compilation ended? => If I read you correctly it will say it's still compiling?
Does the fpc process still run?
Someone reported a similar issue recently (IIRC here on the forum). But if the compiler goes into an endless loop, then that will be hard to narrow down.
https://forum.lazarus.freepascal.org/index.php/topic,69067.0.html

I guess the difference you get when changing compiler settings (or doing a clean build) is how other units are handled. I.e. if the code of other units was loaded from ppu, or gotten compiled from source. One fpc process will compile all units (in the project, not packages), but only those that it can't load from ppu.
The result *should* be the same, but at the very least mem layout will differ, maybe something else in your case...

[wp]

Which Lazarus/FPC versions are you using? I do remember that there was a time when compilation did not terminate under some unclear circumstances; I've never seen this for a long time, though.

[TRon]

Is TS perhaps compiling from/on a remote/external drive (dropbox, VM share) ?

[Nel_Fie]

Thank you all for taking the time to reply.

As a heads-up, I might be a bit sporadic with providing feedback because the issue itself is somewhat sporadic. I had it happen all day yesterday and the day before that, but it hasn't popped up again today.

Nel, I'd suspect a software issue but as no one else has (yet) reported such an issue, I'd doubt it.  And you sound sure its not your code ?

What are you running (FPC ver and Lazarus ver) and on what hardware ?

Compiling often uses a lot of memory. Sometimes, memory that is rarely used. So, next I'd run a aggressive memory checker. Not one of the short, few minute things, a multiple combination, sloooow one. Expect as much as an hour to be sure.

Most linux ISOs have a memory checker available in their boot up menu.

I hope that is not the answer ....

Davo

I mean, it could be my code causing the issue for all I know, but if it is, it's not in a way that Lazarus and its error-spotting mechanisms seem to catch. The closest to a hint for a potential cause that I've come across once, a few weeks back, involved a particular unit being in the "uses" list of another unit's implementation section: moving it up to interface somehow seemed to fix the problem. At the time, I thought that this was the cause of the issue. But then it started happening in other unrelated situations as well, so it doesn't seem like that was it.
But if it really is my code, then why would changing something like the optimization level fix it for a single run?

I'm going to echo @dbannon's comments.

Any kind of problem report should be accompanied by the Lazarus and FPC versions. 

The capricious behavior of the problem you described does raise a good possibility that there might be a hardware problem, likely memory as pointed out by @dbannon.  To be on the safe side, I'd check the hard drive and file system too.

Start by posting the FPC and Lazarus version. 

HTH.

Which Lazarus/FPC versions are you using? I do remember that there was a time when compilation did not terminate under some unclear circumstances; I've never seen this for a long time, though.

To start with, here's everything on my version of Lazarus, FPC, and system:

Lazarus 3.99 (rev main_3_99-2836-gdfce0d15a4)
FPC 3.3.1 x86_64-linux-gtk2

## Hardware Information:
- **Hardware Model:**                              Gigabyte Technology Co., Ltd. F2A88X-D3H
- **Memory:**                                      16,0 GiB
- **Processor:**                                   AMD Athlon™ X4 860K Quad Core × 4
- **Graphics:**                                    NVIDIA GeForce RTX™ 3050
- **Disk Capacity:**                               3,1 TB

## Software Information:
- **Firmware Version:**                            F6
- **OS Name:**                                     Ubuntu 24.04.1 LTS
- **OS Build:**                                    (null)
- **OS Type:**                                     64-bit
- **GNOME Version:**                               46
- **Windowing System:**                            X11
- **Kernel Version:**                              Linux 6.8.0-48-generic

--------

I did two passes on Memtest yesterday and found no errors. How many passes would you recommend for it to be conclusive?

Also, what tools should I use to check on the hard drive and file system?

When looking at the output, use the context menu "copy all/original messages". That may give additional info.

I am not sure though if you will get much useful info.

Once you had the problem, does Lazarus still show "compiling" or does it show the compilation ended? => If I read you correctly it will say it's still compiling?
Does the fpc process still run?
Someone reported a similar issue recently (IIRC here on the forum). But if the compiler goes into an endless loop, then that will be hard to narrow down.
https://forum.lazarus.freepascal.org/index.php/topic,69067.0.html

I guess the difference you get when changing compiler settings (or doing a clean build) is how other units are handled. I.e. if the code of other units was loaded from ppu, or gotten compiled from source. One fpc process will compile all units (in the project, not packages), but only those that it can't load from ppu.
The result *should* be the same, but at the very least mem layout will differ, maybe something else in your case...

Okay, I'll give the context menu of Messages a try once it happens again.

Yes, I'll double-check next time, but everything about the process seemed normal and labelled as still "Compiling", just that it was stuck at a specific step and not making any progress.
I didn't think about checking on the compiler's process in the system monitor, I'll do that the next time it happens.

Is TS perhaps compiling from/on a remote/external drive (dropbox, VM share) ?

No, everything is local to my system: Lazarus, FPC and all files used by the project.

[440bx]

Thank you for that information.

I did two passes on Memtest yesterday and found no errors. How many passes would you recommend for it to be conclusive?

In most cases a few passes is sufficient but sometimes memory failures can be very intermittent.  When I want to be reasonably sure that memory is good, I set the test to run overnight, i.e, about 8 hours, continuously.

if I want to be certain that the memory is working as it should, I compress about 200GB worth of files of various types and sizes then test the archive for errors at least 3 times.  if memory passes that test then, I'm confident it is good.  Note: I have seen memory consistently and repeatedly pass memtest without fail but fail the compress/test mentioned above (and yes, the memory was found to be faulty by the manufacturer which used specialized hardware to test it again.)

As far as the hard drive, check the S.M.A.R.T status.  As far as the file system, I cannot offer any suggestion because I don't know what Linux offers to check that.

HTH.

[Nel_Fie]

The issue started happening again today, so I took the opportunity to check everything brought up so far.

TL;DR : While I can't confirm anything more specific, I can at least confirm that the separate issue brought up by Martin_fr (https://forum.lazarus.freepascal.org/index.php/topic,69067.0.html) is occurring for me as well. It might well be related, although I don't have the expertise to tell. You be the judge.


To run down the list:

1) Using the context menu for Messages and making a copy of everything didn't provide any additional info. It just listed exactly the same things already provided by Messages.
2) I can confirm that Lazarus still displays "compiling".
3) I checked on fpc and ppcx64 in the system monitor. This is where things got a bit more interesting.

I had indeed several copies of ppcx64 running. However, while Lazarus was open, I only managed to get up to 3 processes. One of them was always at 0% CPU, the others stood at around 25% each. But it became stranger when I closed Lazarus, because a bunch more ppcx64 instances popped up, for a total of 6. An instance of fpc was also still running.
The first ppcx64 still remained at 0%, but the other ones oscillated between 15% and 20%, essentially summing up to 100% CPU. And as the author of other thread said, I had to kill them manually to be rid of them. The first one (which always remained at 0%) was not killable in itself though, I had to kill the fpc instance in order to remove it.

For comparison, I re-launched Lazarus afterwards and opened a different, non-problematic project, and compiled it several times. ppcx64 ended every time when the compilation did, as it should. There also wasn't an orphan fpc instance left after closing Lazarus again.

Next time I'll try to confirm the process described by the author of the other thread here: https://forum.lazarus.freepascal.org/index.php/topic,69067.msg535504.html#msg535504

The potential cause described by jamie two messages below the above might also be applicable in my case: my project has a lot of units interlinked in sometimes slightly circuitous ways, and as I described previously, the only time I thought I had found the cause was when the issue seemed to clear up after moving a "uses" for a unit. So that probably points to that.

if I want to be certain that the memory is working as it should, I compress about 200GB worth of files of various types and sizes then test the archive for errors at least 3 times.  if memory passes that test then, I'm confident it is good.  Note: I have seen memory consistently and repeatedly pass memtest without fail but fail the compress/test mentioned above (and yes, the memory was found to be faulty by the manufacturer which used specialized hardware to test it again.)

As far as the hard drive, check the S.M.A.R.T status.  As far as the file system, I cannot offer any suggestion because I don't know what Linux offers to check that.

HTH.

I'm not entirely sure if it's still relevant, considering the rest of my post, but just in case: What tool are you using to test the archive? Or do you still mean using memtest?

[440bx]

I'm not entirely sure if it's still relevant, considering the rest of my post, but just in case: What tool are you using to test the archive? Or do you still mean using memtest?

For the record, only once did I run into the case that memory would consistently pass memtest but _occasionally_ fail the compressed archive test and this was about 15 years ago.  I used Winrar at the time.

With Winrar, sometimes the test would succeed, other times it would fail.  IOW, the result was inconsistent.

I returned the memory to the manufacturer who confirmed there was a bad SIMM (sent 4 back, one was intermittently bad.)   It really is very rare to run into a situation like that.

Summary: if you want to be absolutely certain your memory is working properly, create a very large, 200+ megabyte archive and test it for errors multiple times (at least 4 times), if it passes the test 4 or more times, odds are very good the memory is working as it should but, if there is a weakness somewhere, the archive test will very likely trigger it.

If I had to re-do the test today, I'd use 7zip since I no longer use Winrar.

HTH.

[jamie]

That to me sounds like a thread deadlock, a fight for resources.

I also want to note that since 3.4 Lazarus, I've noticed at least with the 32-bit version at times, after I run an app from the IDE under the debug a few times, it appears like I get a delayed start as if its waiting for something blocking the EXE from starting.

 It never does this when I first enter the IDE and do some short and quick runs but after several times of running the EXE from the IDE debugger is when I'll start to notice this, and I'll see no activity on my computer that I would suspect holding it up.

 But to recap a comment about compiling lots of little, short units causing no error numbers to be reported, I've really have not seen that since I stopped using the "Generics.Collections" unit, or I am just lucky. But I will also note when using that UNIT, I would at times get a report of a  ".Lxxx" not found, but all of this was related to generics.

Have a good day.

[Nel_Fie]

For informational purposes: I had the issue occur again today, and had nine instances of both FPC and ppcx64 left over after closing Lazarus, this time.

For the record, only once did I run into the case that memory would consistently pass memtest but _occasionally_ fail the compressed archive test and this was about 15 years ago.  I used Winrar at the time.

With Winrar, sometimes the test would succeed, other times it would fail.  IOW, the result was inconsistent.

I returned the memory to the manufacturer who confirmed there was a bad SIMM (sent 4 back, one was intermittently bad.)   It really is very rare to run into a situation like that.

Summary: if you want to be absolutely certain your memory is working properly, create a very large, 200+ megabyte archive and test it for errors multiple times (at least 4 times), if it passes the test 4 or more times, odds are very good the memory is working as it should but, if there is a weakness somewhere, the archive test will very likely trigger it.

If I had to re-do the test today, I'd use 7zip since I no longer use Winrar.

HTH.

Thank you. We'll see if it comes to me trying that out.

That to me sounds like a thread deadlock, a fight for resources.

I also want to note that since 3.4 Lazarus, I've noticed at least with the 32-bit version at times, after I run an app from the IDE under the debug a few times, it appears like I get a delayed start as if its waiting for something blocking the EXE from starting.

 It never does this when I first enter the IDE and do some short and quick runs but after several times of running the EXE from the IDE debugger is when I'll start to notice this, and I'll see no activity on my computer that I would suspect holding it up.

 But to recap a comment about compiling lots of little, short units causing no error numbers to be reported, I've really have not seen that since I stopped using the "Generics.Collections" unit, or I am just lucky. But I will also note when using that UNIT, I would at times get a report of a  ".Lxxx" not found, but all of this was related to generics.

Have a good day.

Thank you for the feedback. A question I'd ask then: If it is indeed a thread deadlock, what would be the cause? Is it likely a memory or hardware-related issue on my end, as has been opined so far?

I'm not sure if the run delay issue is related, but I can try to keep an eye out for whether it seems to slow down - although that shouldn't be a surprise, if there are orphaned processes using 100% of my CPU. But I'm working on a 64-bit system, so if it's specific to 32, this might not apply.

[Martin_fr]

From the fact that you have multiple ppcx kept running, I assume that the build involves recompiling some packages?

If so, does any of them has custom path settings? Could there be any shared directory between any 2 of those packages?

If the IDE has 2 build 2 packages, and those 2 packages have no dependency on each other (including recursive through other packages that they depend on), then the IDE will start 2 instance of fpc, running the 2 compilations in parallel.

It's 2 independent processes, it shouldn't cause problems.

They do share the same files from the RTL/FCL, but that is read only for the ppu of those (none of those should be recompiled, neither compiler should even have access to the rtl/fcl sources / unless you have a really wrong setup).

Of course, if the packages have custom path settings, and one includes the ppu output of the other by such a path, then the IDE might not be aware of that. But then the one compiler reading ppu from the output of the other, might find a half finished ppu. And who knows what havoc that may cause.

On the other hand, I regularly encounter an issues with 3.2.3 on Win (but yet not able to make a small example for a bugreport) where I get an error within a package, unless I compile it from scratch => loading of ppu files seems to go wrong...

So there could be a similar issue, but instead of it leading to an error, it leads to an endless loop....

In my case it seems generic related...

thread deadlock

Are there indeed multiple threads in a single process?
Or are there several independent processes?
(afaik, but needs confirmation) each ppc process runs a single thread)

[Nel_Fie]

From the fact that you have multiple ppcx kept running, I assume that the build involves recompiling some packages?

If so, does any of them has custom path settings? Could there be any shared directory between any 2 of those packages?
[...]

I'm honestly not entirely sure, but I think not?
My understanding of Lazarus might be insufficient here (but it also seems that the term "packages" is ambiguous).

If the Project Inspector is anything to go by, I'm only using the LCL package, everything else is units. I've written all of them in the context of the project, except for a json parser called JsonTools, which is also provided as a unit rather than a proper package or somesuch.

thread deadlock

Are there indeed multiple threads in a single process?
Or are there several independent processes?
(afaik, but needs confirmation) each ppc process runs a single thread)

I assume this question is meant for Jamie or other people rather than myself, but just in case: I have no idea.

[Martin_fr]

My question was a long shot. But at this point, I thought better ask to much than skip any potential chance.

From the fact that you have multiple ppcx kept running, I assume that the build involves recompiling some packages?

If so, does any of them has custom path settings? Could there be any shared directory between any 2 of those packages?
[...]

I'm honestly not entirely sure, but I think not?
My understanding of Lazarus might be insufficient here (but it also seems that the term "packages" is ambiguous).

If the Project Inspector is anything to go by, I'm only using the LCL package, everything else is units. I've written all of them in the context of the project, except for a json parser called JsonTools, which is also provided as a unit rather than a proper package or somesuch.

That means, likely not a path issue.

"Package" meant *.lpk Lazarus package => a collection of units managed by Lazarus.

If you only see "LCL" then that includes nested LCLBase, Chm, FreeTypeLaz, FCL, RTL (the last isn't organized as lpk though).
And it sounds like you did not edit any of them.

If you also did not add a path in your "Project options" settings on the "Path" page, then that is fine.
If you did put any path there that points into the Lazarus or FPC sources, or unit or lib folders then that may be an issue.
If you did put a path there that points some other place, that is fine.


I assume you just placed the json parser unit into your project folder? That would be fine too.


However, now it gets interesting (maybe, if I am not mistaken).

LCL (and dependencies) would not likely be rebuild.
Unless you change the "configure rebuild Lazarus" settings, or you have "additions and overrides" affecting packages in your project options (see that tab in project opts).

You can also watch the messages windows: Does it say anything about "compiling package ..."? Or does it only output stuff about compiling your project?

Because, if LCL and dependencies is not recompiled, and if you have no other packages.... And if I am right, then there should be only one instance of fpc and one of ppc. (Someone of the fpc team might be able to confirm that).

Of course if you don't notice that that ppc___ didn't stop, and you compile again, then you would have two.

But I understand that a single compilation (one time pressing F9, or one time choosing "build") leads to several ppc___.