[SOLVED] Debugging dynamic (shared object) library on linux

[linvon]

Hello! I am trying debug dynamic library and having some troubles with this on linux. I have Lazarus 3.4 and FPC 3.2.2.
I have 2 separated project: for library and for application
Compiling all this things in debug mode.
Using standard debugger (FpDebug).
Loading libaray in app with LoadLibrary and getting correct lib handle.

But when i put break-point in application project debugger ingores it from app. I also tried set 'host application' in library project - no effect.
Can you help me please?

[Martin_fr]

I don't currently have 3.4 installed on Linux. Only 3.99.  (testing with Fedora 38)

There were some fixes, but I don't know if they cover your case. E.g.:
https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues/40973

In any case in 3.99 it worked both ways
- Using the library project and setting host application
- Using the host project and just loading the library (if that is compiled with debug info)

My test code:

Host project:

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. {$mode delphi}{$H+}
4.  
5. uses
6.   {$IFDEF UNIX}
7.   cthreads,
8.   {$ENDIF}
9.   Classes, dynlibs;
10.  
11. var
12.   hl:TLibHandle;
13.   hello_lib:function(const pc:PAnsiChar):integer;
14. begin
15.  
16.   writeln('Start test');
17.   hl:=LoadLibrary({$ifdef LINUX}'./libproja.so'{$endif}
18.      {$ifdef MSWINDOWS}'./proja.dll'{$endif}
19.      );
20.   if hl>0 then
21.   begin
22.     @hello_lib:=GetProcAddress(hl,'hello_lib');
23.     if Assigned(hello_lib) then
24.     hello_lib('Out form lib OK');
25.   end;
26. end.

lib:

Code: Pascal  [Select][+][-]

1. library proja;
2.  
3. {$mode delphi}{$H+}
4.  
5.  
6. function hello_lib(const pc:PAnsiChar):integer;
7. begin
8.   Result:=0;
9.   writeln(pc);
10. end;
11.  
12. exports
13.   hello_lib;
14. end.

When debugging the host project, and the library has not yet been loaded, then the breakpoint shows as "paused" (a red dot, whit 2 green/yellow bars (like pause on a music player). And then becomes active when the lib is loaded.

I haven't tested what happens if there are units of the same name in both the host, and the lib... IIRC it should work.

I need more info on your case.
- Do you at least get the "paused" breakpoint?
- If you open the debugger "event log" does it say something like "Loaded: libfoo.so (/path/to/libfoo.so)"

[linvon]

Yes, I have "paused" breakpoints, but nothing in "event log".
However, the library works, I just can't debug it.

[Martin_fr]

What is your exact Linux environment?

Maybe something changed in one of the involved libraries.

You can always try to debug your app using the gdb backend. But that has of course other limits.

[linvon]

I will look for the environment.
But with gdb same behaviour.

[korba812]

Which project do you have loaded in Lazarus IDE when debugging? Application or the library? You should open library project, set host application in startup parameters and then run library project with debugging (F9).
Remember to enable debugger information generation in the library project.

[linvon]

I have "project group" with library and application.
Debug information for library always generates

[Martin_fr]

You should open library project, set host application in startup parameters and then run library project with debugging (F9).

It also works (or should work) when you open the main project in the IDE, and it loads the library.


A few more questions:

Both main project and library are fpc projects?
Both have dwarf debug info enabled?

You did not use any special settings for the linker? (static vs dynamic / stripping / ...)

You did not strip any of the exe/so ?
You did not strip any of the dll in /usr/lib... ? (really unlikely)

The reason I am asking...

In order for breakpoint in libraries to work, the debugger must be aware when a lib is loaded. For that there is a "trigger" in one of the system libs.

If FpDebug fails to get that trigger, then breakpoints in libs wont work. (I don't know what the code in gdb is nowadays doing, but google suggests that it at least at some point used the same method, and maybe still does)

Lets do some checks please.

1) I don't know the exact name of your systems shared libs. But they should be something like the examples.

They may just be in /usr/lib or some other lib folder
find the ld lib:

Code: Bash  [Select][+][-]

1. find /usr/ -name 'ld*.so*'

get some info about it (you need to change the name according to what you found above):

Code: Bash  [Select][+][-]

1. objdump -t /usr/lib64/ld-linux-x86-64.so.2 | grep -i _debug; objdump -T /usr/lib64/ld-linux-x86-64.so.2 | grep -i _debug

copy and paste the result here, please.


2)
Can you apply the patch below please.

Then recompile the IDE (Tools > Build Lazarus).
Open a terminal/console/shell, and run lazarus from that terminal (so you can see what it prints to stdout)

Run your project
Copy and paste anything that Lazarus printed.

Mind the below has a tiny functional change too. With some extreme amount of luck it may even fix the issue. (Though I really doubt it, because then gdb should not have the issue too)

Code: Diff  [Select][+][-]

1. diff --git a/components/fpdebug/fpdbglinuxclasses.pas b/components/fpdebug/fpdbglinuxclasses.pas
2. index e72c919e7d..5601c9a9cc 100644
3. --- a/components/fpdebug/fpdbglinuxclasses.pas
4. +++ b/components/fpdebug/fpdbglinuxclasses.pas
5. @@ -1564,7 +1564,7 @@ var
6.  
7.  begin
8.    inherited LoadInfo;
9. -
10. +writeln('####################################################################');
11.    // This would be strange, but you never know.
12.    if Assigned(FSOLibEventBreakpoint) then
13.      Raise Exception.Create('SOLib event-breakpoint already exists.');
14. @@ -1576,6 +1576,7 @@ begin
15.      InterpSection := LoaderList.Items[i].Section['.interp'];
16.      if assigned(InterpSection) then
17.        begin
18. +writeln( LoaderList.Items[i].FileName );
19.        // Try to retrieve the inode of the file (library) in the .interp section.
20.        // This is the filename of the library that handles the dynamic loading.
21.        if FpStat(PChar(InterpSection^.RawData), AStat) = 0 then
22. @@ -1585,10 +1586,16 @@ begin
23.          SynchronizeProcMapsWithLibraryList();
24.          if FLibMap.GetLib(Astat.st_ino, ALib) then
25.            begin
26. + writeln('in library : ', ALib.Name );
27. +
28.            // Set a breakpoint at _dl_debug_state. This procedure is called after
29.            // one or more libraries have been loaded. This breakpoint is used to
30.            // detect the (un)loading of libraries.
31.            FSOLibEventBreakpoint := AddBreak('_dl_debug_state', False, ALib);
32. +writeln(FSOLibEventBreakpoint<> nil);
33. +if FSOLibEventBreakpoint<>nil then writeln(' state ', ord(FSOLibEventBreakpoint.State));
34. +if FSOLibEventBreakpoint.State <> bksOk then FSOLibEventBreakpoint.Free
35. +else
36.            TFpDbgBreakpoint(FSOLibEventBreakpoint).FreeByDbgProcess := True;
37.            end
38.          end;
39.  

[Martin_fr]

Also, can you test/confirm if/that the issue also exist with the sample app that I posted earlier?

[linvon]

Guys you gave me a clue. I doubted that the project compiles with debug information.
1. Then i tried compile library with debug info in separated debug file .dbg

Code: Bash  [Select][+][-]

1. fpc -g -gl -Xg -O- -CX -XX -fPIC -vewnhi -l -olib123.so lib123.lpr

2. Then check the file size:

Code: Bash  [Select][+][-]

1. du -h lib123.so

And got 1.2M and no lib123.dbg file.

I removed flag -Xg and recompile (without lib123.dbg)

Code: Bash  [Select][+][-]

1. fpc -g -gl -O- -CX -XX -fPIC -vewnhi -l -olib123.so lib123.lpr

Check the size again like in paragraph 2 and got 2.1M.
After recompile library with debug info inner I was able to debug and go through to lib code.

For some reason in linux with flag -Xg external debug file not generated ONLY for dynamic libs, with executable no problem.
With flag -Xg under windows .dbg file generated for both cases(lib and executable) without problems

Thank you, Martin_fr for hint!