dwarf trouble with DLL and host application

[MarkBravington]

Hi

I'm writing a DLL to be run by host application R (ver 2.11.1), using this Lazarus version:

Lazarus 0.9.28.2 r22279 FPC 2.2.4 i386-win32-win32/win64

under Window XP x64 Service Pack 2. Everything is 32bit except my computer (and the Windows version). I can do rudimentary debugging *without* dwarf. But when I tried to use dwarf (ticking "Generate dwarf debug information (-gw)" in Project/Compiler_options), debugging just didn't work. Well, I can't set a breakpoint, anyway. Even with dwarf not working, the DLL runs OK inside R though (except for the bugs I am trying to debug...).

I have "Run/Run_parameters/Host_application" and "Run/Run_parameters/Command_line_parameters" set, but not "R/R_p/Launching application" etc.

Below is (most) of the debug output from the with/without dwarf runs. Even without dwarf, there are lots of errors messages, but it still works (sort-of; at any rate, I can set breakpoints and inspect simple variables).

The messages suggest to me that the dwarf version is not finding the source. The DLL and its source are in different directories, the former set by "Project/Project_options/Application/Target_file_name". I've tried setting "extra" paths to my source code and DLL directory (which are different), to no effect. EG in "Project/Compiler_options/Debugger_path_addition" (and when I add the source path there, it resets it to "."), and in "Environment/Options/Debugger/General/Additional_search_path".

There are countless other places in Lazarus where various paths are set; I have little idea what most of them do, and have not included them here (but will happily do so, if anyone thinks they might matter).

Any help much appreciated!

Mark Bravington
mark.bravington@csiro.au


BTW #1: though I know quite a lot about R, Pascal, and DLLs under R, I know nothing about gdb and scary linuxy stuff like that; for better or (probably) worse, I live in the candyfloss castle that is Windows.

BTW #2: while trying dwarf, I also tried ticking the "Use external gdb symbols file (-Xg)" box in Project/Compiler_options, and got this error at the end of building:

srw_raneff3.lpr(307,1) Fatal: Internal error 200602251


library srw_raneff3;
{$mode delphi}{$H+}
{$define REML}

uses
  Classes, Useful, Maths32,
  {$IFNDEF AD} D_Tuseful_obj_unit, {$ENDIF}{AD}
  tuseful_obj_unit, domin
  {$IFDEF REML}
  , Tsrw_raneff3_obj_unit
  , d_tsrw_raneff3_obj_unit
  , Tsrw_raneff3_reml_obj_unit,  D_TSRW_RANEFF3_REML_OBJ_UNIT
  , tw_d_tsrw_raneff3_reml_obj_unit
  {$ENDIF}
  ;
 
**************** DEBUG OUTPUT WITH DWARF ****************** 

(gdb)

<-gdb-set confirm off>
^done

(gdb)

<-gdb-set new-console off>
^done

(gdb)

<-gdb-version>
~"GNU gdb 6.8\n"
~"Copyright (C) 2008 Free Software Foundation, Inc.\n"
~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\n"
~"This is free software: you are free to change and redistribute it.\n"
~"There is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\n"
~"and \"show warranty\" for details.\n"
~"This GDB was configured as \"i686-pc-mingw32\".\n"
^done
(gdb)
<-gdb-set env AD=d:\r2.0\ad\work>
^done
<<SNIPPED-- LOTS OF ENV VARS GET SET HERE>>
(gdb)
<-file-exec-and-symbols "c:/r/r-2.11.1pat/bin/rgui.exe">
~"(no debugging symbols found)\n"
^done
(gdb)
<-environment-cd .>
^done
(gdb)
<-environment-cd "C:/r/r-2.11.1pat/bin/">
^done
(gdb)
<-data-evaluate-expression FPC_THREADVAR_RELOCATE_PROC>
&"No symbol table is loaded.  Use the \"file\" command.\n"
^error,msg="No symbol table is loaded.  Use the \"file\" command."
(gdb)
<info functions FPC_CPUINIT>
&"info functions FPC_CPUINIT\n"
~"All functions matching regular expression \"FPC_CPUINIT\":\n"
^done
(gdb)
<info functions $$_RUNERROR$>
&"info functions $$_RUNERROR$\n"
~"All functions matching regular expression \"$$_RUNERROR$\":\n"
^done
(gdb)
<-exec-arguments R_PROFILE=D:\R2.0\.R_PROFILE R_USER=D:\R2.0>
^done
(gdb)
<-gdb-set language pascal>
^done
(gdb)
<info address main>
&"info address main\n"
&"No symbol \"main\" in current context.\n"
^error,msg="No symbol \"main\" in current context."
(gdb)
<-break-insert -t main>
&"Function \"main\" not defined in loaded symbols.\n"
^error,msg="Function \"main\" not defined in loaded symbols."
(gdb)
<-break-insert FPC_RAISEEXCEPTION>
&"Function \"FPC_RAISEEXCEPTION\" not defined in loaded symbols.\n"
^error,msg="Function \"FPC_RAISEEXCEPTION\" not defined in loaded symbols."
(gdb)
<-break-insert FPC_BREAK_ERROR>
&"Function \"FPC_BREAK_ERROR\" not defined in loaded symbols.\n"
^error,msg="Function \"FPC_BREAK_ERROR\" not defined in loaded symbols."
(gdb)
<-break-insert FPC_RUNERROR>
&"Function \"FPC_RUNERROR\" not defined in loaded symbols.\n"
^error,msg="Function \"FPC_RUNERROR\" not defined in loaded symbols."
(gdb)
<info file>
&"info file\n"
~"Symbols from \"c:/r/r-2.11.1pat/bin/rgui.exe\".\n"
~"Local exec file:\n"
~"\t`c:/r/r-2.11.1pat/bin/rgui.exe', file type pei-i386.\n"
~"\tEntry point: 0x4012a0\t0x00401000 - 0x00405948 is .text\n"
~"\t0x00406000 - 0x00406084 is .data\n"
~"\t0x00407000 - 0x004073f4 is .rdata\n"
~"\t0x00408000 - 0x00408a10 is .bss\n"
~"\t0x00409000 - 0x004094e8 is .idata\n"
~"\t0x0040a000 - 0x0040b078 is .rsrc\n"
^done
(gdb)
<-break-insert -t *4199072>
^done,bkpt={number="1",type="breakpoint",disp="del",enabled="y",addr="0x004012a0",at="",times="0"}
(gdb)
<-exec-run>
^running
(gdb)
~"[New thread 293960.0x47c4c]\n"
&"Error: dll starting at 0x77d40000 not found.\n"
&"Error while mapping shared library sections:\n"
&"NOT_AN_IMAGE: No such file or directory.\n"
&"Error while mapping shared library sections:\n"
&"C:\\WINDOWS\\SysWOW64\\ntdll32.dll: No such file or directory.\n"
&"Error: dll starting at 0x77d40000 not found.\n"
&"Error: dll starting at 0x77c20000 not found.\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
*stopped,thread-id="1",frame={addr="0x004012a0",func="??",args=[]}
(gdb)
<info program>
&"info program\n"
~"\tUsing the running image of child thread 293960.0x47c4c.\n"
~"Program stopped at 0x4012a0.\n"
~"It stopped at a breakpoint that has since been deleted.\n"
~"Type \"info stack\" or \"info registers\" for more information.\n"
^done
(gdb)
<-break-insert srw_raneff3.lpr:243>
&"No source file named srw_raneff3.lpr in loaded symbols.\n"
^error,msg="No source file named srw_raneff3.lpr in loaded symbols."
(gdb)
<-break-insert tsrw_raneff3_reml_obj_unit.pas:358>
&"No source file named tsrw_raneff3_reml_obj_unit.pas in loaded symbols.\n"
^error,msg="No source file named tsrw_raneff3_reml_obj_unit.pas in loaded symbols."
(gdb)
<-break-insert tsrw_raneff3_obj_unit.pas:678>
&"No source file named tsrw_raneff3_obj_unit.pas in loaded symbols.\n"
^error,msg="No source file named tsrw_raneff3_obj_unit.pas in loaded symbols."
(gdb)
<-break-insert srw_raneff3.lpr:33>
&"No source file named srw_raneff3.lpr in loaded symbols.\n"
^error,msg="No source file named srw_raneff3.lpr in loaded symbols."
(gdb)
<-exec-continue>
^running
(gdb)
~"[New thread 293960.0x47c50]\n"
~"[New thread 293960.0x47c54]\n"
~"[Switching to thread 293960.0x47c54]\n"
*stopped,reason="signal-received",signal-name="SIGTRAP",signal-meaning="Trace/breakpoint trap",thread-id="3",frame={addr="0x7d61002e",func="??",args=[]}
(gdb)
<info program>
&"info program\n"
~"\tUsing the running image of child thread 293960.0x47c54.\n"
~"Program stopped at 0x7d61002e.\n"
~"It stopped with signal SIGTRAP, Trace/breakpoint trap.\n"
~"Type \"info stack\" or \"info registers\" for more information.\n"
^done
(gdb)
<-symbol-list-lines D:\r2.0\srw\inst\pascal3\srw_raneff3.lpr>
&"mi_cmd_symbol_list_lines: Unknown source file name.\n"
^error,msg="mi_cmd_symbol_list_lines: Unknown source file name."
(gdb)
<-symbol-list-lines srw_raneff3.lpr>
&"mi_cmd_symbol_list_lines: Unknown source file name.\n"
^error,msg="mi_cmd_symbol_list_lines: Unknown source file name."
(gdb)
<-exec-continue>
^running
(gdb)
~"[New thread 293960.0x47c58]\n"
~"[New thread 293960.0x47c60]\n"
~"[New thread 293960.0x47c5c]\n"
&"Error while reading shared library symbols:\n"
&"Dwarf Error: Could not find abbrev number 12975 [in module d:\\rpackages\\r2.10.0\\srw\\libs\\srw_raneff3.dll]\n"
<<SNIPPED-- SAME ERROR MESSAGE MANY TIMES>>

*************************** DEBUG OUTPUT WITHOUT DWARF ***************************************

<-file-exec-and-symbols "c:/r/r-2.11.1pat/bin/rgui.exe">
~"(no debugging symbols found)\n"
^done
(gdb)
<-environment-cd .>
^done
(gdb)
<-environment-cd "C:/r/r-2.11.1pat/bin/">
^done
(gdb)
<-data-evaluate-expression FPC_THREADVAR_RELOCATE_PROC>
&"No symbol table is loaded.  Use the \"file\" command.\n"
^error,msg="No symbol table is loaded.  Use the \"file\" command."
(gdb)
<info functions FPC_CPUINIT>
&"info functions FPC_CPUINIT\n"
~"All functions matching regular expression \"FPC_CPUINIT\":\n"
^done
(gdb)
<info functions $$_RUNERROR$>
&"info functions $$_RUNERROR$\n"
~"All functions matching regular expression \"$$_RUNERROR$\":\n"
^done
(gdb)
<-exec-arguments R_PROFILE=D:\R2.0\.R_PROFILE R_USER=D:\R2.0>
^done
(gdb)
<-gdb-set language pascal>
^done
(gdb)
<info address main>
&"info address main\n"
&"No symbol \"main\" in current context.\n"
^error,msg="No symbol \"main\" in current context."
(gdb)
<-break-insert -t main>
&"Function \"main\" not defined in loaded symbols.\n"
^error,msg="Function \"main\" not defined in loaded symbols."
(gdb)
<-break-insert FPC_RAISEEXCEPTION>
&"Function \"FPC_RAISEEXCEPTION\" not defined in loaded symbols.\n"
^error,msg="Function \"FPC_RAISEEXCEPTION\" not defined in loaded symbols."
(gdb)
<-break-insert FPC_BREAK_ERROR>
&"Function \"FPC_BREAK_ERROR\" not defined in loaded symbols.\n"
^error,msg="Function \"FPC_BREAK_ERROR\" not defined in loaded symbols."
(gdb)
<-break-insert FPC_RUNERROR>
&"Function \"FPC_RUNERROR\" not defined in loaded symbols.\n"
^error,msg="Function \"FPC_RUNERROR\" not defined in loaded symbols."
(gdb)
<info file>
&"info file\n"
~"Symbols from \"c:/r/r-2.11.1pat/bin/rgui.exe\".\n"
~"Local exec file:\n"
~"\t`c:/r/r-2.11.1pat/bin/rgui.exe', file type pei-i386.\n"
~"\tEntry point: 0x4012a0\t0x00401000 - 0x00405948 is .text\n"
~"\t0x00406000 - 0x00406084 is .data\n"
~"\t0x00407000 - 0x004073f4 is .rdata\n"
~"\t0x00408000 - 0x00408a10 is .bss\n"
~"\t0x00409000 - 0x004094e8 is .idata\n"
~"\t0x0040a000 - 0x0040b078 is .rsrc\n"
^done
(gdb)
<-break-insert -t *4199072>
^done,bkpt={number="2",type="breakpoint",disp="del",enabled="y",addr="0x004012a0",at="",times="0"}
(gdb)
<-exec-run>
^running
(gdb)
~"[New thread 294592.0x47e9c]\n"
&"Error: dll starting at 0x77d40000 not found.\n"
&"Error while mapping shared library sections:\n"
&"NOT_AN_IMAGE: No such file or directory.\n"
&"Error while mapping shared library sections:\n"
&"C:\\WINDOWS\\SysWOW64\\ntdll32.dll: No such file or directory.\n"
&"Error: dll starting at 0x77d40000 not found.\n"
&"Error: dll starting at 0x77c20000 not found.\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
~"(no debugging symbols found)\n"
*stopped,thread-id="1",frame={addr="0x004012a0",func="??",args=[]}
(gdb)
<info program>
&"info program\n"
~"\tUsing the running image of child thread 294592.0x47e9c.\n"
~"Program stopped at 0x4012a0.\n"
~"It stopped at a breakpoint that has since been deleted.\n"
~"Type \"info stack\" or \"info registers\" for more information.\n"
^done
(gdb)
<-break-insert srw_raneff3.lpr:243>
&"No source file named srw_raneff3.lpr in loaded symbols.\n"
^error,msg="No source file named srw_raneff3.lpr in loaded symbols."
(gdb)
<-break-insert tsrw_raneff3_reml_obj_unit.pas:358>
&"No source file named tsrw_raneff3_reml_obj_unit.pas in loaded symbols.\n"
^error,msg="No source file named tsrw_raneff3_reml_obj_unit.pas in loaded symbols."
(gdb)
<-break-insert tsrw_raneff3_obj_unit.pas:678>
&"No source file named tsrw_raneff3_obj_unit.pas in loaded symbols.\n"
^error,msg="No source file named tsrw_raneff3_obj_unit.pas in loaded symbols."
(gdb)
<-break-insert srw_raneff3.lpr:33>
&"No source file named srw_raneff3.lpr in loaded symbols.\n"
^error,msg="No source file named srw_raneff3.lpr in loaded symbols."
(gdb)
<-exec-continue>
^running
(gdb)
~"[New thread 294592.0x47e8c]\n"
~"[New thread 294592.0x47eac]\n"
~"[Switching to thread 294592.0x47eac]\n"
*stopped,reason="signal-received",signal-name="SIGTRAP",signal-meaning="Trace/breakpoint trap",thread-id="3",frame={addr="0x7d61002e",func="??",args=[]}
(gdb)
<info program>
&"info program\n"
~"\tUsing the running image of child thread 294592.0x47eac.\n"
~"Program stopped at 0x7d61002e.\n"
~"It stopped with signal SIGTRAP, Trace/breakpoint trap.\n"
~"Type \"info stack\" or \"info registers\" for more information.\n"
^done
(gdb)
<-symbol-list-lines D:\r2.0\srw\inst\pascal3\srw_raneff3.lpr>
&"mi_cmd_symbol_list_lines: Unknown source file name.\n"
^error,msg="mi_cmd_symbol_list_lines: Unknown source file name."
(gdb)
<-symbol-list-lines srw_raneff3.lpr>
&"mi_cmd_symbol_list_lines: Unknown source file name.\n"
^error,msg="mi_cmd_symbol_list_lines: Unknown source file name."
(gdb)
<-exec-continue>
^running
(gdb)
~"[New thread 294592.0x47e94]\n"
~"[New thread 294592.0x47eb4]\n"
~"[New thread 294592.0x47ea4]\n"
~"[New thread 294592.0x47f28]\n"
~"[Switching to thread 294592.0x47f28]\n"
*stopped,reason="signal-received",signal-name="SIGTRAP",signal-meaning="Trace/breakpoint trap",thread-id="7",frame={addr="0x7d61002e",func="??",args=[]}
(gdb)
<info program>
&"info program\n"
~"\tUsing the running image of child thread 294592.0x47f28.\n"
~"Program stopped at 0x7d61002e.\n"
~"It stopped with signal SIGTRAP, Trace/breakpoint trap.\n"
~"Type \"info stack\" or \"info registers\" for more information.\n"
^done
(gdb)
<-break-insert srw_raneff3.lpr:243>
^done,bkpt={number="3",type="breakpoint",disp="keep",enabled="y",addr="0x06c41b15",func="R_SRW3_GET_DPROBS",file="srw_raneff3.lpr",line="243",times="0"}
(gdb)
<-break-enable 3>
^done
(gdb)
<-exec-continue>
^running
(gdb)

[Martin_fr]

First at all, update to the most recent 0.9.29 fixes /0.9.20 snapshot.
(or wait  a week (or 2, or 3) until 0.9.30 is released)

It contains various debug related fixes.
Make sure after the update that you use the gdb version from the update (7.2)

With this it should be able to debug dll.

[MarkBravington]

I've just tried the most recent 0.9.29, but the problem persists unfortunately-- more details after "BTW" below. Now my version is:

Lazarus 0.9.29 r29679 FPC 2.4.2 i386-win32-win32/win64

and this is with gdb 7.2.

BTW I had great difficulty in finding the precompiled version (otherwise I would have tried this previously). The Lazarus wiki "download" button takes you straight to SourceForge page (scary Linuxy stuff that I don't understand) and I couldn't see anything after 0.9.28 there, let alone precompiled. And there's no mention of "snapshots". The "Navigation/Downloads" link only talks about packages etc.-- no mention of Lazarus itself. The FAQ doesn't seem to discuss getting precompiled newer versions. I only found the snapshot website by searching in the forums. Could just be me, but it might be useful to add some links-to-snapshots in the above places.

Debugging without dwarf info still works (maybe this will be OK-- it now lets me change values, which it didn't with my 0.9.28 version-- thanks!).

With dwarf, the new version now lets me "set" breakpoints when the DLL is loaded by the host, but they don't actually work. Actually, I get a green tick in a *pink* circle, not a red circle, so maybe they aren't really set. And the Debug Output window still shows stuff like this:

&"No source file named tsrw_raneff3_reml_obj_unit.pas.\n"
^done,bkpt={number="18",type="breakpoint",disp="keep",enabled="y",addr="<PENDING>",pending="tsrw_raneff3_reml_obj_unit.pas:358",times="0",original-location="tsrw_raneff3_reml_obj_unit.pas:
358"}
(gdb)
<-break-enable 18>
^done
(gdb)
<-break-insert -f tsrw_raneff3_obj_unit.pas:678>
&"No source file named tsrw_raneff3_obj_unit.pas.\n"
^done,bkpt=

<<etc>>

Unlike before, gdb now reports finding lots of DLLs, including the DLL for the project-- but  that doesn't seem to help it find the sources (which are not in the target DLL's directory.

So still looks like there's a path-finding problem of some sort.

Mark

[Martin_fr]

There is a difference though:

<-break-insert -f tsrw_raneff3_obj_unit.pas:678>

the "-f" wasn't there before. It means force, and should tell gdb to keep the breakpoint in case it becomes available later....

And at least on my 32 bit system, I have once made it to work.

But let's see what else.

1st: you always mention dwarf. Have you tried stabs? (just "-g" instead of "-gw"). I do not expect it to differ, but to be sure....

Also when you speak of debug-info, I assume you refer to the dll.
How is "R" compiled? It seems to bbe without debug info

~"(no debugging symbols found)\n"

Have you tried to compile it with dwarf too?

2nd:

DLL to be run by host application R

I assume "tsrw_raneff3_obj_unit.pas" is part of the DLL?

How to you debug this? Is your open project "R" or the dll (with "R" as startup-app in run-param)?
I remember when I tested, I had the dll as loaded project in the dll, and had set my "R" as starter-app. So gdb would have the libraries director as working dir.
You may want to try and/or play around.

[MarkBravington]

The basic story is:

Lazarus 0.9.28: stabs works (always has for me, but with limited functionality), dwarf doesn't

Lazarus 0.9.29 from the Snapshots page, version as per my previous email (on my work desktop): stabs works, dwarf doesn't
 
azarus 0.9.29 r28390 FPC 2.5.1 i386-win32-win32/win64n64 (on my laptop, can't remember where I got this version): neither stabs nor dwarf worked, aiiiieee

R is a statistical programming language (www.r-project.org) which allows people to load DLLs. R is written in C, and I can't recompile it. Anyway, I've never had to: stabs debugger has always worked, with the open project being my DLL and the Run/Host_application being R (ie the starter app, I presume). That sounds the same as your setup, Martin.

One thing is that my source and my compiled DLL are in different places. I've tried playing around with some path options in Lazarus, but to no effect (but admittedly I don't know what I'm doing, and am just trying random stuff). Nevertheless, it's odd that stabs generally works, but dwarf doesn't...

Mark
mark.bravington@csiro.au

[Martin_fr]

Hm, not sure if I can add much to it....

Anyway the debug output as far as I have seen it seems ok.

Not sure how much the paths, or the place where you keep your source does matter (for the first steps).

The ide gives the following command to gdb:
your first example:
   <-break-insert srw_raneff3.lpr:243>
your 2nd example:
   <-break-insert -f tsrw_raneff3_obj_unit.pas:678>

As you can see, only the filename. So as long as the filename is unique, the path should not matter.
The path only matters, when the IDE tries to open the file. If the IDE can't find it, it should open the disassembler view.

Out of curiosity: You compiled the dll debug info, into the dll file, not into an external .dbg file?

&"No source file named tsrw_raneff3_reml_obj_unit.pas.\n"
^done,bkpt={number="18",type="breakpoint",disp="keep",enabled="y",addr="<PENDING>",pending="tsrw_raneff3_reml_obj_unit.pas:358",times="0",original-location="tsrw_raneff3_reml_obj_unit.pas:

"pending" basically means the breakpoint is not yet set, but gdb keeps it in mind, as soon as it detects any dll that has this file.

But I have no idea why gdb would not detect this in your case.

If you still have your working setup:
- try the newer gdb, on the working files => just to be sure.

or maybe try the 64 bit gdb version (with your 32 bit exe)
http://svn.freepascal.org/svn/lazarus/binaries/x86_64-win64/gdb/bin/gdb.exe
http://svn.freepascal.org/svn/lazarus/binaries/x86_64-win64/gdb/bin/libiconv-2.dll
(dll goes in the gdb.exe dir)

you can find gdb's from older laz versions: http://svn.freepascal.org/svn/lazarus/binaries
or download from mingw.

[MarkBravington]

Martin-- thanks, I have it (ie dwarf under Lazarus 0.9.29 build 29679) working now. The new features, eg ability to change variable values, are very welcome!

It was my fault because of mangled config files while installing parallel versions of Lazarus-- quite a common situation, judging by the forum lists.

Thanks for the advice-- knowing that "it should work" gave me the incentive to persist.

Mark Bravington
CSIRO, Hobart, Australia