lazarus with debug info

[Molochnik]

When trying to locate an error i found that i need an option like the "debug DCUs" in Delphi.
As far as I understand I need to recompile Lazarus sources (or fpc?) for that.
Is there some concise algorithm how to do it?

[Martin_fr]

It depends on which units you need to debug.

In project option go to "Additions and Overrides" => new target "*" and add a custom option -gw

That affects the LCL, WS (LCL-Win,LCL-gtk) and any component in the components folders, and packages installed via OPM.

It does not affect FPC. That is the RTL, and FCL (anything that is in the fpc folder).

For those you need to rebuild fpc, using the make file.
(I am not sure if you may have do download fpc again, or if it can be done with the included files).

You can always use fpcupdeluxe to build an additional fpc, and in fpcupdeluxe specify -gw

[Molochnik]

Martin_fr
The top line of code in the stack windows i'd like to debug refers to CThreads, so it's FPC, right?

[Martin_fr]

Yes, cthreads is part of the RTL (of fpc).

So you need to rebuild fpc. Essentially something like
  make OPT="-gw"
but there are a few details ... You need a starting compiler, and maybe point to it install to some location in the end (and not replace your current), create a config, ....

So my suggestion is use fpcupdeluxe. Somewhere it should allow you to set the "OPT" for build => put the -gw there (or -gw3 if you use fpdebug in the IDE).

Because each fpc needs to find its correct config, fpcupdeluxe afaik create as script: fpc.sh => that makes sure your new fpc does not get tangled up with the existing.
So once you have that fpc, in the IDE set the compiler (tools > options) to that script.
That compiler will then use the units with debug info (so to toggle between the debug/no-debug units you switch the  compiler)

[Molochnik]

As I expected nothing is going easy on Lazarus.
1) I tried using fpcupdeluxe, it downloaded and built fpc but i could not make the existing Lazarus to connect to the newly build compiler(sources were found properly but system.ppu for unknown reasons is not). I checked and made some corrections in the fpc.cfg for it to properly refer to the units directory, failure.
2) All right I downloaded fpc trunc manually and built it. Another try to connect Lazarus to the new compiler. Now system.ppu is found (no error) but after selecting sources from which it was build i got an error:
"Warning: Found version 3.3.1, expected 3.2.2 You can download FPC and the FPC sources from http://sourceforge.net/projects/lazarus/?source=directory"
I have no idea why Lazarus expects 3.2.2 version.

[Martin_fr]

First of all: about units not find, and config files.

If you compile with (IIRC)   -vtu   then the compiler will tell you where it looked, and where it got its config from (it will be hidden in the messages window, but you can do "copy > copy all and original messages").


You may run into issues because of files in your PATH environment.

i.e. when you put
   /home/m/fpc/bin/fpc
down as fpc, then this fpc will call ppcx86 (or some ppc, depending what cpu you compile for).

And it may find that in $PATH ... I don't know for sure. (I always have all my fpc hand installed outside the PATH).

you can try in the IDE and directly put
   /home/m/fpc/bin/ppcx86
as compiler.
Should work, but you can't change the target cpu in the project options anymore.


Also as you have seen config, especially   fpc.cfg is important.
check what you have in global spaces, like /etc/fpc.cfg
Or diretly in your home folder 
(again  IIRC -vtu  should tell you what is used).

If need is, you can temporarily rename any config that is in the way.

[Molochnik]

1) the PATH is almost clean:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
2) How does lazarus know about 3.2.2 version? Where does it get the info?
3) The "setup" window in fpcupdeluxe has "FPC options", "Laz options" and "debug" checkboxes. So it's here where I should place additional options for compiler, correct? Particularly "-vtu"?
4) After the newly built fpc (or ppcx64) is set as a compiler the lazarus initial config window shows "Error: system.ppu not found. Check your fpc.cfg" What fpc.cfg is used? The one that is next to the compiler binary?
5) In the fpc.cfg file that is located right next to compiler binaries there are following lines:

Code: Pascal  [Select][+][-]

1. .....
2. # searchpath for units and other system dependent things
3. -Fu/home/ravil/fpctrunk/fpc/units/$fpctarget
4. -Fu/home/ravil/fpctrunk/fpc/units/$fpctarget/*
5. -Fu/home/ravil/fpctrunk/fpc/units/$fpctarget/rtl

As for me they look good, if $fpctarget has "x86_64-linux" value then it is correct, but i have no idea what the actual value it contains. Setting "x86_64-linux" directly instead of $fpctarget doesn't change a thing.

[Martin_fr]

2) How does lazarus know about 3.2.2 version? Where does it get the info?

Most likely it did run your configured "fpc" and parsed the version from its output.
And then it compares to the version that it finds in one of the source files in the fpc source folder (not sure which, but there is one that contains it)

So that is why I thought it may have run the correct fpc => but that fpc may have run
/usr/local/bin/ppcx86   or wherever that is... (assuming 64bit otherwise the name differs)

Your fpcdeluxe version would be probably in your home folder (wherever you placed it).
And you self-build version, would be where you did build it, or where you did "make install PREFIX=..." (I think it is PREFIX , double check)

Whichever of them you configured in the IDE, it would have its own fpc and its own ppcx86

3) The "setup" window in fpcupdeluxe has "FPC options", "Laz options" and "debug" checkboxes. So it's here where I should place additional options for compiler, correct? Particularly "-vtu"?

No -vtu is not for building the compiler.

-vtu you would put in Lazarus in the "Project Options" => "custom options" before you try to compile your project (any project where the compiler will then give an error).

Then when you copy "all and original messages" it should say for each unit where it tried to find it. So you can see if and which path has gone wrong.
And afaik it will also tell you which fpc.cfg it loaded. So you can also check that this is correct.

If any path or any file comes from the default fpc (rather from your self build fpc) then you can trace it back. E.g. if it came from a wrong fpc.cfg. Or you can search in the IDE if the wrong path is in any xml, or lpi file.

4) After the newly built fpc (or ppcx64) is set as a compiler the lazarus initial config window shows "Error: system.ppu not found. Check your fpc.cfg" What fpc.cfg is used? The one that is next to the compiler binary?

Yes, the one next to the compiler bin.

Actually good point, I overlooked a detail in one of my earlier posts (because on Windows it all just works ...)

On Linux it wont necessarily  find the fpc.cfg in the same path.
That is why fpcdeluxe would create a fpc.sh file that passes the location as config. If you used fpcdeluxe you would use that fpc.sh file.
And then you could not directly use the ppcx86 => but you could edit the fpc.sh file to use the ppcx86.

Anyway if you have your own build you must make sure the fpc.cfg is found.
I am not sure what the syntax is to pass it in. I thing    fpc @path/fpc.cfg
But needs to be double checked.

I myself have some scripts that put the fpc.cfg in a rather obscure location were it is found. But there were quite some steps needed for that.  I would have to go and re-analyse that ....

One thing you can do is to temporarily remove (backup) and replace the global /etc/fpc.cfg with an fpc.cfg in your own build.

Well, if you have one. fpcupdeluxe would create it.
make ... I am not sure. You may need to call "fpcmkcfg" to create it (but I have to lookup the correct params again...)

You can see why I recommended fpcupdeluxe. Doing it without is quite a challenge.

5) In the fpc.cfg file that is located right next to compiler binaries there are following lines:

Code: Pascal  [Select][+][-]

1. .....
2. # searchpath for units and other system dependent things
3. -Fu/home/ravil/fpctrunk/fpc/units/$fpctarget
4. -Fu/home/ravil/fpctrunk/fpc/units/$fpctarget/*
5. -Fu/home/ravil/fpctrunk/fpc/units/$fpctarget/rtl

As for me they look good, if $fpctarget has "x86_64-linux" value then it is correct, but i have no idea what the actual value it contains. Setting "x86_64-linux" directly instead of $fpctarget doesn't change a thing.

Yes that looks good, just need to ensure that is the fpc.cfg that gets used..

Compiling any project with -vtu probably will. If not use -va that will (but that will generate a ton of other messages // a = all)

[Molochnik]

At last I understood the Lazarus logic when selecting fpc compiler at the start config window:
1) It allows you to select the fpc binary but not fpc.cfg (it knows better where to find it)
2) Then it totally ignores the fpc.cfg located next to the fpc binary (likely it contains some errors)
3) Then it searches throughout known Linux paths. locates the first existed fpc.cfg (obviously it is the one I actually need)
4) Checks the units paths there and accurately informs you that they are improperly set, offers you to correct them
and clarifies that you should do it in the fpc.cfg file (of course without specifying its actual path cause it makes the error look ugly)

After coping and replacing fpc.cfg I made the previously built lazarus totally unusable and after some efforts just removed all of them, downloaded fpcupdeluxe binary, made it to download and install both laz+fpc and the miracle happened. Everything works like charm now! And with debug info.

Though unfortunately it didn't help me to find the error in my program - without notifications the main thread receives sigsegv the program dies (Delphi windows version works fine).

Thanks!

[TRon]

At last I understood the Lazarus logic when selecting fpc compiler at the start config window:

fpc.cfg places that the compiler will look for (in particular order) can be found here.

After coping and replacing fpc.cfg I made the previously built lazarus totally unusable and after some efforts just removed all of them,

Yups, messing with the config file will do that for you unfortunately.

downloaded fpcupdeluxe binary, made it to download and install both laz+fpc and the miracle happened. Everything works like charm now! And with debug info.

That is because FPCUpDeluxe uses a little trick to bypass the places the compiler will look for the fpc.cfg file, see the options -n and @ at the listing of commandline options manual page.

It ensures that the FPC commandline compiler will not try and load a fpc.cfg file other than the one you explicitly tell it to use. One fpc.cfg file to rule them all for me (and nicely tucked away exactly where I want it to be) 

Martin_fr was almost there with his (excellent) explanations.

Though unfortunately it didn't help me to find the error in my program - without notifications the main thread receives sigsegv the program dies (Delphi windows version works fine).

Yeah, now you have gained the knowledge on setting up and configuring the compiler (important to know) you're still stuck with coding related issues.... that is still a bit of a bummer. Does that mean that compiling the RTL with debug info did not shine a (brighter) light for your use case  ?

[Martin_fr]

Unrelated to the topic... But maybe helpful for your bug... At least worth a shot.

Try running your app under valgrind. Note, this will be slow, real dead slow.
(no special fpc needed / just compile your app with -gv  and WITHOUT -gh / no heaptrc)

   valgrind --tool=memcheck  yourapp


If that doesn't get you anything, valgrind has 2 thread checkers. You would need to google them. And note that they will return big amounts of false positives. So it takes a lot of work to sort through them.

[Molochnik]

Yeah, now you have gained the knowledge on setting up and configuring the compiler (important to know) you're still stuck with coding related issues.... that is still a bit of a bummer. Does that mean that compiling the RTL with debug info did not shine a (brighter) light for your use case  ?

yes, true
As per the error I found a workaround but failed to understand the cause.
The code is old and convoluted, but works fine on Delphi/Windows.
The oversimplified version (it is compiled but actually wont work cause it needs some amendments) but the idea is right:

Code: Pascal  [Select][+][-]

1. type
2.   TThreadA = class(TThread)
3.   protected
4.     procedure Execute; override;
5.   public
6.     OnData: TNotifyEvent;
7.   end;
8.  
9.   TThreadB = class(TThread)
10.   protected
11.     IndyClient: TIdTCPClient;
12.     procedure Execute; override;
13.     procedure SendData(AObject: TObject);
14.   public
15.     procedure StartSending;
16.     procedure StopSending;
17.     constructor Create;
18.     destructor Destroy; override;
19.   end;
20.  
21. var
22.   ThreadA: TThreadA;
23.   ThreadB: TThreadB;
24.  
25. implementation
26.  
27. procedure TThreadA.Execute;
28. begin
29.   while not Terminated do
30.   begin
31.     if Assigned(OnData) then
32.       OnData(Self);
33.     Sleep(100);
34.   end;
35. end;
36.  
37. constructor TThreadB.Create;
38. begin
39.   inherited;
40.   IndyClient := TIdTCPClient.Create;
41. end;
42.  
43. destructor TThreadB.Destroy;
44. begin
45.   IndyClient.Free;
46.   inherited;
47. end;
48.  
49. procedure TThreadB.SendData(AObject: TObject);
50. begin
51.   IndyClient.Socket.WriteLn('Hello');
52. end;
53.  
54. procedure TThreadB.StartSending;
55. begin
56.   ThreadA.OnData := SendData;
57. end;
58.  
59. procedure TThreadB.StopSending;
60. begin
61.   ThreadA.OnData := nil;
62.   Sleep(1000);
63. end;
64.  
65. procedure TThreadB.Execute;
66. begin
67.   while not Terminated do
68.   begin
69.     Sleep(1000);
70.   end;
71. end;
72.  
73. initialization
74.  
75. ThreadA := TThreadA.Create;
76. ThreadB := TThreadB.Create;
77.  
78. ThreadB.StartSending;
79. Sleep(1000); // 1 sec, so around 10 "Hellos" will be sent by ThreadA
80. ThreadB.StopSending;
81.  
82. // If threadB is terminated before threadA, later I will get SIGSEGV
83. // If threadA is terminated before threadB, everything will be fine
84. ThreadB.Terminate;
85. ThreadB.WaitFor;
86. ThreadB.Free;
87.  
88. ThreadA.Terminate;
89. ThreadA.WaitFor;
90. ThreadA.Free;
91.  
92. end.

If ThreadB is terminated before ThreadA the code works fine but only once, no matter of sleeps, timeouts, critical section locks etc. After the execution in several seconds SIGSEGV is triggered. No runtime errors before that.
If ThreadA is terminated before ThreadB the code works without causing any problem no matter how many times. Its actually a workaround.

[Martin_fr]

Yeah you could a race condition there. Actual, several...

Lets say you would do

Code: Pascal  [Select][+][-]

1. ThreadB.StopSending;
2.  
3. ThreadA.Terminate;
4. ThreadA.WaitFor;
5. ThreadA.Free;

Which you noted crashes.

"ThreadB.StopSending;" => this removes the OnData from ThreadA.

Except, you can't assume it happens simultaneously.

Code: Pascal  [Select][+][-]

1.     if Assigned(OnData) then
2.       OnData(Self);

For the call "OnData(Self) the thread loads the event-address into a register.

Now this can hapen
- Thread A: copy OnData into register
* Scheduler switches thread
- Main thread, calls ThreadB.StopSending
- Main thread destroys ThreadB
* Scheduler switches thread
- Thread A: call to ThreadB.SendData (from what it has in its register)

And that gives an error as the object for ThreadB is gone.

Now, this can be a hard to catch race condition, because the time slot in which the error happens is tiny.
I am not sure, even if you compile with maximum optimization, the field should be reloaded from memory quite often.

You put in a lot of sleeps, so this should be  really really unlikely....

I wonder... 
You said the debugger was somewhere in CThreads. But that may be misleading.
Because if you call a method on a freed object, then you could end up following virtual methods to random places.

Also, your real code will have more workload than the example. So it is possible that instead of calling an event on the destroyed object, you are still inside an event within the object "ThreadB" when THreadB gets destroyed. Leads to similar errors.


More so, even

Code: Pascal  [Select][+][-]

1.  ThreadA.OnData := nil;

Is not thread save.

This is an event, a method pointer. OnData has 2 fields, the function-address and the object instance.

So the assignment is not an atomic operation. While highly unlikely (less than a lottery jackpot chance), the other thread could be reading OnData, when only half of it was written.

And

Code: Pascal  [Select][+][-]

1.   if Assigned(OnData) then
2.       OnData(Self);

Is also bad (and it is not that unlikely to go wrong)>

You could have
Thread A:  if Assigned(OnData) then
Thread B:   ThreadA.OnData := nil;
   // even if it was atomic, even if it does both fields before the other thread goes on
Thread A      OnData(Self);
 
Only that by the time the call is made, OnData is nil.



In other words quite a few race conditions.

The simplest way is to use critical sections.
If that is not wanted, you can use unprotected writes/reads but  (very roughly described)
- you must the fields by hand in the right order
- have the check do the matching order
- potentially add Read/Write barriers.

In other words, without critical sections => you need a lot of knowledge about thread safety.