Moving away from GDB

[MarkMLl]

For the last 15 years or so, I have habitually compiled both FPC and Lazarus from source on various platforms rather than using prebuilt packages.

On Debian Linux "Bookworm" x86_64, Lazarus 2.2.4 is clearly struggling with GDB: the debugger regularly crashes particularly if hovering over any variable other than a simple scalar.

With reference to https://wiki.freepascal.org/Debugger_Setup#Setting_up_the_debugger_backend_for_the_IDE , Tools -> Options -> Debugger Backend only shows "GNU Debugger (gdb)" in the drop-down at the top of the window. I can see that the "LazDebuggerFP 1.0" package is installed.

What do I do to enable the new-style debugger?

MarkMLl

p.s. I would far prefer, this time round, to not be told that I'm the one at fault for running an experimental OS or that a mere user should not be attempting to build the IDE from source.

[Martin_fr]

GDB works best with "DWARF 2 with sets" ... but even then.  DWARF 3 is a gdb killer. (And that includes mixed compiles, e.g. fpc/rtl has dwarf 3 and your project has dwarf 2.

As for setting up:
https://wiki.lazarus.freepascal.org/Debugger_Setup

A new install should already have FpDebug in the list. If you upgrade to 3.0 then you should be asked at startup.
Also running
   lazarus --setup
brings up the start dialog, that has options to change the settings.


Otherwise:
- Button "Add"
- Select "FpDebug" as type
- press ok

---
You can also set a debugger per project in the project options....

[MarkMLl]

Thanks Martin, I'll give that some thought. However initially:

GDB works best with "DWARF 2 with sets" ... but even then.  DWARF 3 is a gdb killer. (And that includes mixed compiles, e.g. fpc/rtl has dwarf 3 and your project has dwarf 2.

The IDE appears to be using automatic selection of debug type, which I suspect is inherited from when the configuration file was first set up (which is quite possibly several versions ago). FPC (i.e. RTL etc.) was compiled with -O- -gl -Xs- -vt The file command confirms that binaries are unstripped, but doesn't say much else.

Since this could possibly help others, how can I inspect a binary and determine what DWARF variant is in use?

Project options only give GDB as an option. I've not explored your other suggestions yet since I think it's important to first determine DWARF properties.

MarkMLl

[Martin_fr]

"automatic" is decided by FPC. Afaik it's preset for each target. (so it does not depend what was used to build the compiler/rtl).
Afaik, on all major platforms it is some version of DWARF (i.e. stabs, is no longer used on major platforms).

objdump should probably give you insight (I would have to check what the quickest way is). "objdump --dwarf=info" => very very long output. Mixed within a header for each unit, that should have a version.


But to know if you use DWARF 3 or higher: Are your local vars shown all uppercase?
"All upper" = DWARF 2
"mixed case" = DWARF 3 or higher.

So of course you can still have some units that and other units this... But then you need to run the long objdump.


fpc also introduced
 -godwarfcpp
https://www.freepascal.org/docs-html/user/userap1.html

But I have not tested it much.

[rvk]

Since this could possibly help others, how can I inspect a binary and determine what DWARF variant is in use?

Would this work?
readelf --debug-dump=info binary_name | head | grep -A 2 'Compilation Unit @'

And this for all included units?
readelf --debug-dump=info mytest | grep -A 2 'Compilation Unit @'

[MarkMLl]

Since this could possibly help others, how can I inspect a binary and determine what DWARF variant is in use?

Would this work?
readelf --debug-dump=info binary_name | head | grep -A 2 'Compilation Unit @'

And this for all included units?
readelf --debug-dump=info mytest | grep -A 2 'Compilation Unit @'

Since that's an easy one to check,

Code: Text  [Select][+][-]

1. bin.fpc/3.2.2$ readelf --debug-dump=info fp | head | grep -A 2 'Compilation Unit @'
2.   Compilation Unit @ offset 0:
3.    Length:        0x234a (32-bit)
4.    Version:       2
5. bin.fpc/3.2.2$ readelf --debug-dump=info fpc | head | grep -A 2 'Compilation Unit @'
6.   Compilation Unit @ offset 0:
7.    Length:        0x234a (32-bit)
8.    Version:       2
9. bin.fpc/3.2.2$ file fp
10. fp: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.4.0, with debug_info, not stripped
11. bin.fpc/3.2.2$ file fpc
12. fpc: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, for GNU/Linux 2.4.0, with debug_info, not stripped
13.  
14. share.lazarus/lazarus-stable$ readelf --debug-dump=info lazarus | head | grep -A 2 'Compilation Unit @'
15.   Compilation Unit @ offset 0:
16.    Length:        0x2362 (32-bit)
17.    Version:       2
18. share.lazarus/lazarus-stable$ file lazarus
19. lazarus: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.4.0, with debug_info, not stripped
20.  

I'll try to look at some of the other suggestions later.

MarkMLl

[Martin_fr]

I'm not familiar with "readelf", but it looks like it gets the same data. So that would be "Dwarf 2" (with/without sets can't be told, but probably makes no difference).

The grep would also show you each and every unit ("compilation unit")

[MarkMLl]

But to know if you use DWARF 3 or higher: Are your local vars shown all uppercase?
"All upper" = DWARF 2
"mixed case" = DWARF 3 or higher.

The IDE's "Local Variables" debugging window shows locally-declared variables in upper-case, plus a symbol representing what I presume is the starting value in mixed case i.e.

Code: Pascal  [Select][+][-]

1. var
2.   tempExts: TExtractions= nil;
3.  

results in TEMPEXTS and defaulttempExts both with value zero. Hence I presume we're talking about DWARF 2.

The "Add" button at the top of the window has allowed me to add FpDebug, and also automatically selected it. I've not yet been able to put any time into exercising it: I'll report back when I can.

Losing gdb presumably means that it's no longer to use gdbserver.

MarkMLl

[Martin_fr]

The IDE's "Local Variables" debugging window shows locally-declared variables in upper-case, plus a symbol representing what I presume is the starting value in mixed case i.e.

Code: Pascal  [Select][+][-]

1. var
2.   tempExts: TExtractions= nil;
3.  

results in TEMPEXTS and defaulttempExts both with value zero. Hence I presume we're talking about DWARF 2.

Yes, also shown by the readelf.

There are some symbols that FPC may add (in mixed case, like the current function name as alias for "RESULT". Or some symbols with  "$" in it (like "$parentfp". (the exact name varies by fpc version).

With fpdebug, I suggest to switch to Dwarf 3.

The "Add" button at the top of the window has allowed me to add FpDebug, and also automatically selected it. I've not yet been able to put any time into exercising it: I'll report back when I can.

Losing gdb presumably means that it's no longer to use gdbserver.

Unfortunately. But if it is in the list of configured debuggers, you can select it in project settings. You then just have to be careful of certain values. (You can test the -godwarfcpp / but I don't know any details on it / don't use it with fpdebug)

For AVR, there is a fpdebug package that afaik uses gdbserver. (package comes with IDE, but must be installed).

Not gdb server, but if you have issuse, there is a mix of gdb + fpdebug (LazDebuggerFpGdbmi) => runs in gdb, but evals watches via fpdebug (needs local exe for symbols).
This is useful if you debug a lot in the kernel / 3rd party libs => gdb is still better at unrolling the stack in those cases.

In theory that could be extended to gdbserver. But no plans for that at the moment.
Also: It's considerable slower.

[MarkMLl]

Ooh, this is interesting :-)

I had a couple of minutes to knock together so started the IDE and went to set a breakpoint. The IDE responded with a dialogue

"Enable Dwarf 2 (-gw)?"

"The project does not write debug info in Dwarf format. The "FpDebug internal Dwarf-debugger" supports only Dwarf."

With buttons for Dwarf 2, Dwarf 3 and Dwarf 2 with sets with the last of those default.

I've not changed anything yet since I think that reviewing what I've actually got in the RTL and app being compiled (which the IDE clearly doesn't think is Dwarf 2) might be useful.

My notes from earlier in the thread show -O- -gl -Xs- -vt being used for the compiler/RTL, and inspecting the command line that the IDE intends to use suggest that it's propagating the compiler's -gl option.

Reviewing fpc -h output, it looks as though gdb is happy if a binary is compiled with -gl but FpDebug requires -gl -gw or even something more specific. I think it would be extremely useful if the compiler could say "you've got this" as well as "you need that".

For info, Bookworm's gdb --version says 13.1

MarkMLl

[Martin_fr]

-gl  => the l is line info, i.e. includes the unit for dumping traces with resolved addresses as unit/line.

-g  is debug info in either stabs or dwarf. Nowadays this is 99.9% dwarf. But only fpc knows what it is going to use (depending on your target). Since the IDE does not know, it can not relay on it.

GDB accepts stabs or dwarf.
FpDebug accepts only dwarf.

Both gdb and fpdebug "support" dwarf 2 and 3.
In fpdebug dwarf 3 gives the better results.
In gdb there are issues (either issues of gdb, or how fpc writes dwarf 3...)

Hence, if using gdb, stick to dwarf 2 (though in your case, even that did not help / feel free to test dwarf 3, maybe you have a good version of gdb).

[MarkMLl]

For the purpose of my desktop system running Debian "Bookworm":

I'm obliged to use a very recent FPC on account of linker changes, which I think at the moment means 3.2.2. It looks as though a new version of gdb is causing Lazarus problems /possibly/ because I'm still using DWARF 2: if this is the case it might be reasonable to assume that the binutils developers are now oriented towards DWARF 3 (and might even be testing it properly).

I suspect I started using -gl /very/ early on when I needed backtraces to help the developers. Investigating various combinations using a "cogito ergo sum" program (i.e. begin-end only) and looking at the checksum and size of the resulting binary:

Code: Text  [Select][+][-]

1. 1867881469 231872  dummy_
2. 3331781544 1078504 dummy_gl
3. 3331781544 1078504 dummy_gl_gw
4. 3331781544 1078504 dummy_gl_gw2
5. 0375397870 1078504 dummy_gl_gw3
6. 4291773809 1021680 dummy_gw2
7. 3297076883 1021680 dummy_gw3
8.  

i.e. -gl implies -gw2 (but not vice-versa), and switching to -gw3 changes the checksum (as expected) but not the size of the binary (which is suspicious).

I'm going to recompile FPC 3.2.2 (i.e. including the RTL etc.) for DWARF 3 and use that to rebuild Lazarus 2.2.4. I'll report back on the stability of gdb and FpDebug.

Booting a 32-bit laptop and looking at old versions of the compiler, it looks as though -gw3 was supported as far back as 2.2.2. Allowing that I've got FPC 2.2.4 and Lazarus 0.9.24 paired together and assuming that FPC was actually generating debugging records properly, the limiting factor would appear to be the distro's binutils version.

MarkMLl

[MarkMLl]

Looking at this in a bit more detail for a few minutes. The first thing I'd say is that the dwarfdump program is supposed to be able to extract the format and version from a binary, but I've had no success with it, hence am sticking with rvk's suggested

Code: Text  [Select][+][-]

1. $ readelf --debug-dump=info dummy_ | head | grep -A 2 'Compilation Unit @'
2.  

which, now that I've got a better idea what I'm looking for, I see in various other discussions.

Using this test script:

Code: Text  [Select][+][-]

1. #!/bin/sh
2.  
3. echo 'FPC compiler/runtimes built with -gl only.'
4.  
5. echo 'fpc -odummy_ dummy.pas'
6. fpc -odummy_ dummy.pas > /dev/null
7. readelf --debug-dump=info dummy_ | head | grep -A 2 'Compilation Unit @'
8.  
9. echo 'fpc -gl -odummy_gl dummy.pas'
10. fpc -gl -odummy_gl dummy.pas > /dev/null
11. readelf --debug-dump=info dummy_gl | head | grep -A 2 'Compilation Unit @'
12.  
13. echo 'fpc -gl -gw2 -odummy_gl_gw2 dummy.pas'
14. fpc -gl -gw2 -odummy_gl_gw2 dummy.pas > /dev/null
15. readelf --debug-dump=info dummy_gl_gw2 | head | grep -A 2 'Compilation Unit @'
16.  
17. echo 'fpc -gl -gw3 -odummy_gl_gw3 dummy.pas'
18. fpc -gl -gw3 -odummy_gl_gw3 dummy.pas > /dev/null
19. readelf --debug-dump=info dummy_gl_gw3 | head | grep -A 2 'Compilation Unit @'
20.  
21. echo 'fpc -gw2 -odummy_gw2 dummy.pas'
22. fpc -gw2 -odummy_gw2 dummy.pas > /dev/null
23. readelf --debug-dump=info dummy_gw2 | head | grep -A 2 'Compilation Unit @'
24.  
25. echo 'fpc -gw3 -odummy_gw3 dummy.pas'
26. fpc -gw3 -odummy_gw3 dummy.pas > /dev/null
27. readelf --debug-dump=info dummy_gw3 | head | grep -A 2 'Compilation Unit @'
28.  
29. cksum dummy_* | sort
30. rm dummy_*
31.  

and starting off with a version of the compiler etc. built with -gl only, the script produces this output:

Code: Text  [Select][+][-]

1. FPC compiler/runtimes built with -gl only.
2. fpc -odummy_ dummy.pas
3. fpc -gl -odummy_gl dummy.pas
4.   Compilation Unit @ offset 0:
5.    Length:        0x234a (32-bit)
6.    Version:       2
7. fpc -gl -gw2 -odummy_gl_gw2 dummy.pas
8.   Compilation Unit @ offset 0:
9.    Length:        0x234a (32-bit)
10.    Version:       2
11. fpc -gl -gw3 -odummy_gl_gw3 dummy.pas
12.   Compilation Unit @ offset 0:
13.    Length:        0x234a (32-bit)
14.    Version:       2
15. fpc -gw2 -odummy_gw2 dummy.pas
16.   Compilation Unit @ offset 0:
17.    Length:        0x234a (32-bit)
18.    Version:       2
19. fpc -gw3 -odummy_gw3 dummy.pas
20.   Compilation Unit @ offset 0:
21.    Length:        0x234a (32-bit)
22.    Version:       2
23. 1798613560 1078504 dummy_gl
24. 1798613560 1078504 dummy_gl_gw2
25. 2745444480 1078504 dummy_gl_gw3
26. 2927890045  231872 dummy_
27. 3549871481 1021680 dummy_gw2
28. 3765698736 1021680 dummy_gw3
29.  

If instead I use a compiler built with -gl -gw2:

Code: [Select]

FPC compiler/runtimes built with -gl -gw2.
fpc -odummy_ dummy.pas
fpc -gl -odummy_gl dummy.pas
  Compilation Unit @ offset 0:
   Length:        0x234a (32-bit)
   Version:       2
fpc -gl -gw2 -odummy_gl_gw2 dummy.pas
  Compilation Unit @ offset 0:
   Length:        0x234a (32-bit)
   Version:       2
fpc -gl -gw3 -odummy_gl_gw3 dummy.pas
  Compilation Unit @ offset 0:
   Length:        0x234a (32-bit)
   Version:       2
fpc -gw2 -odummy_gw2 dummy.pas
  Compilation Unit @ offset 0:
   Length:        0x234a (32-bit)
   Version:       2
fpc -gw3 -odummy_gw3 dummy.pas
  Compilation Unit @ offset 0:
   Length:        0x234a (32-bit)
   Version:       2
1798613560 1078504 dummy_gl
1798613560 1078504 dummy_gl_gw2
2745444480 1078504 dummy_gl_gw3
2927890045  231872 dummy_
3549871481 1021680 dummy_gw2
3765698736 1021680 dummy_gw3

The good news here is that the checksums suggest that the build is reproducible. The bad news is that compiling the test program with -gw3 still produces a binary with DWARF 2 debug info, if the readelf output is to be believed.

If instead I use a compiler built with -gl -gw3:

Code: Text  [Select][+][-]

1. FPC compiler/runtimes built with -gl -gw3.
2. fpc -odummy_ dummy.pas
3. fpc -gl -odummy_gl dummy.pas
4.   Compilation Unit @ offset 0:
5.    Length:        0x1eda (32-bit)
6.    Version:       3
7. fpc -gl -gw2 -odummy_gl_gw2 dummy.pas
8.   Compilation Unit @ offset 0:
9.    Length:        0x1eda (32-bit)
10.    Version:       3
11. fpc -gl -gw3 -odummy_gl_gw3 dummy.pas
12.   Compilation Unit @ offset 0:
13.    Length:        0x1eda (32-bit)
14.    Version:       3
15. fpc -gw2 -odummy_gw2 dummy.pas
16.   Compilation Unit @ offset 0:
17.    Length:        0x1eda (32-bit)
18.    Version:       3
19. fpc -gw3 -odummy_gw3 dummy.pas
20.   Compilation Unit @ offset 0:
21.    Length:        0x1eda (32-bit)
22.    Version:       3
23. 1358981819 1029376 dummy_gl
24. 1358981819 1029376 dummy_gl_gw2
25. 1605665511  975368 dummy_gw3
26. 1860338280 1029376 dummy_gl_gw3
27. 1874535790  975368 dummy_gw2
28. 2927890045  231872 dummy_
29.  

So we can say definitely (from the checksums) that -gl still implies -gw2, however from the readelf output I'm not sure what is /really/ being put into the binary file.

I've put minimal time into working with the IDE yet, since I think that this takes us back to the issue when FpDebug unexpectedly prompted me for a DWARF version: we need some reliable way of working out that's actually in there.

MarkMLl

[rvk]

Weird. You script outputs this for me:

fpc -odummy_ dummy.pas
fpc -gl -odummy_gl dummy.pas
  Compilation Unit @ offset 0x0:
   Length:        0x77 (32-bit)
   Version:       2
fpc -gl -gw2 -odummy_gl_gw2 dummy.pas
  Compilation Unit @ offset 0x0:
   Length:        0x77 (32-bit)
   Version:       2
fpc -gl -gw3 -odummy_gl_gw3 dummy.pas
  Compilation Unit @ offset 0x0:
   Length:        0x78 (32-bit)
   Version:       3
fpc -gw2 -odummy_gw2 dummy.pas
  Compilation Unit @ offset 0x0:
   Length:        0x77 (32-bit)
   Version:       2
fpc -gw3 -odummy_gw3 dummy.pas
  Compilation Unit @ offset 0x0:
   Length:        0x78 (32-bit)
   Version:       3
116035507 226696 dummy_
2093630322 1076400 dummy_gw2
2856255440 1139768 dummy_gl
2856255440 1139768 dummy_gl_gw2
3552731905 1139768 dummy_gl_gw3
3883557931 1076400 dummy_gw3

Checking my compiled fpc I get this:

# readelf --debug-dump=info ~/dev/fpc/bin/fpc | head | grep -A 2 'Compilation Unit @'
  Compilation Unit @ offset 0x0:
   Length:        0xfad (32-bit)
   Version:       2

It looks like the gwx switch doesn't work for you and it always takes the same as the one compiled with fpc 

[marcov]

Probably the order is important with -gl triggering dwarf2, overriding an earlier -gw3

Note that combinations are also possible, it would be interesting to see what -gw3l  and -glw3 resp. do