IDE Leaks and Traces "Resolve" fails on Mac OSX (

[kcandrews]

Having spent many hours reading forum posts on this topic, it appears that the Lazrus IDE "Leaks and Traces" Resolve when applied to an executable on OSX still fails to work.   Leaks are shown but without any line number or source file references.  As dbannon discovered years ago through much time and effort, it appears some complation/link step or post-link step on OSX strips the symbolic data from the executable.

As a test I ran dwarfdump (Molly suggestion long ago to dbannon) on a simple Lazarus example executable, with no code except for one deliberate unfreed memory allocation, no symbolic information appears in dwarfdump's output.    The same results on two different OSX machines one containing the latest Mac OSX and the other an earlier OSX version, but both machines with the latest FPC and Lazarus version releases from 2023-12-18 - Version 3.0 FPC Version: 3.2.2 Revision: x86_64-darwin-cocoa.

For history, see:
https://forum.lazarus.freepascal.org/index.php/topic,39255.msg270590.html#msg270590
(a bug report had been filed but was closed)

As suggested by dbannon, it appears leak testing can only work on a NON-Mac OS.  But in the hopes someone has come upon a solution, I thought I would raise the issue once again FWIW.

[Martin_fr]

There are several possible causes...

1) Depending on the IDE version, "Leaks and Traces" use either the code from the RTL or from FpDebug to resolve addresses.
If it is the former, then I don't know which fpc version has support for which Mac CPU/OS-version....

2) If it is FpDebug, then the debug info must be on the exe.  (If FpDebug + lldb can debug the exe, then the info should be there)
  I am not sure, if "resolve" needs to select the bundle, or the exe inside the bundle.

3) "Resolve" can only work if the exe is compiled with a "fixed address" (not sure what the term is). That is, it is always loaded to the same address in memory.
If the OS loads it to different addresses on each run, then it can not be resolved => because the trace has absolute addresses, and any part of the exe could have been at any of the addresses.

I don't know how Mac executables behave in regards to the last point.

I do recall that the leaks and traces for Mac had been under investigation "recently" (last year?). I don't recall the outcome.

[kcandrews]

Thanks Martin for your input and expertise as always. Comments interspersed:

You may recall a couple years ago, you and I had a long thread in the course of getting debugging working consistently on the Mac.  The FpDebug & LLDB debug setup were still stablilizing at the time -- but in the end and even now, I was and am still able to debug my Lazarus projects on OSX, so symbolic info clearly is available to the debugger..

As to "resolve," that requires a filename found in the file dialog (not a directory), so resolve only proceeds if given a filename (NOT an .app bundle).  And "Resolve" does not see a Mac .app "bundle" as a directory, nor a "file", so it is impossible to navigate into .app/Contents/MacOS/executable_file using the dialog.  But one can get around that by setting up a link to the executable_file at the same level as the .app bundle.  That executable though seems to be stripped of symbol info, so nothing is resolved symbol-wise.

Dwarfdump of a simple project executable complied with debug info and line numbers shows:

     project1:   file format Mach-O 64-bit x86-64
    .debug_info contents:  (nothing listed!)

So maybe the "fixed address" is the problem or Lazarus debugging gets its symbolic information from somewhere other than the executable stored in the app bundle (object files?).

It may just be that Lazarus IDE Memory Leak tracing will never be possible due to OS compatiblity issues.  My understanding is that in the investigation (dbannon, Molly, and others), no resolution was ever achieved, so dbannon is only able to do leak tracing in NON Mac OS implementations.  My fate as well, I suspect :-)

[Martin_fr]

You might be able to test the address stuff. Set a breakpoint, and open the assemble window. Now run the app a few times in the debugger. Does the address change or not?

If the address changes, then it would require
- the base address to be logged and added to the trace in such a way that it can be found
- leaks and traces to be extended
=> not anytime soon on my todo.


It's a while that I looked at how debug info on Mac is loaded...

IIRC (distant memory), the debug info is loaded from lots of files (all the .o files or something like this). You may try "external debug info", then you get a 2nd folder next to the app, with a different extension, and (afaik) all the files in it.

You can always check in fpdebug where it opens the file, some fpimagereader*
Probably (just glancing over the files, got other project open and in the works): TDbgMachoDataSource.ReadFile;
Then but writeln in there to indicate filenames, and if they where able to be opened.

If it finds, and opens all files ... -> good.


Btw, if you try to get traces with heaptrc... IIRC there was some mention that fpc trunk may be able to print them with already resolved names... But that is all hearsay.

[kcandrews]

So, the addresses shown in the assembler window did NOT vary :-)

Checking the Compiler Options/Debugging/Use external debug symbols file produced (as noted before) a single dSYM file: in my case "project1.dSYM"

I did a dwarfdump on project1.SYM and redirected the huge amount of output to a file and then reviewed it in TextEdit.  I looked specifically for the method Button1Click and here is what that small section looks like:

0x0003790d:     DW_TAG_member
                  DW_AT_name   ("BUTTON1")
                  DW_AT_data_member_location   (DW_OP_plus_uconst 0x7c0)
                  DW_AT_type   (0x00037a8c "TBUTTON")

0x0003791a:     DW_TAG_subprogram
                  DW_AT_name   ("BUTTON1CLICK")
                  DW_AT_prototyped   (0x01)
                  DW_AT_calling_convention   (DW_CC_GNU_borland_fastcall_i386)
                  DW_AT_external   (0x01)
                  DW_AT_low_pc   (0x0000000100056fe0)
                  DW_AT_high_pc   (0x000000010005700a)

0x00037932:       DW_TAG_formal_parameter
                    DW_AT_name   ("this")
                    DW_AT_location   (DW_OP_breg6 RBP-16)
                    DW_AT_artificial   (0x01)
                    DW_AT_type   (0x000378ef "TFORM1")

So this all looks good in that apparently Lazarus did indeed write the symbolic output to the standard Apple OSX dSYM file sized at 19.1 MB. Project1's executable is 15.5 MB under this scenario.  (The location of the dSYM file seems to vary in my couple test cases -- sometimes it exists at the same level as the .app bundle, and otherwise it is actually in the .app bundle.)

Building again but without checkbox set for "Use external debug symbols" of course produces no DSYM file and interestingly a Project1 executable sized at 15.8 MB (vs 15.5 before).  Obviously the increase of .3 MB is not enough to have the executable actually contain the additional 19.1 MB of data within it. 

A little google search on Apple dSYM found this:
[ "A dSYM file is a "debug symbols file". It is generated when the "Strip Debug Symbols" setting is enabled in the build settings of your project.
When this setting is enabled, symbol names of your objects are removed from the resulting compiled binary (one of the many countermeasures to try and prevent would be hackers/crackers from reverse engineering your code, amongst other optimisations for binary size, etc.).
]

To sum up, it appears that "Resolving" using the executable fails since it seems stripped of symbol information.  But it is accessible to Lazarus since it is discoverable when writing the debug info externally! 

So as dbannon discovered long ago, it appears process is stripping the executable of the symbolic info.

I have tinkered with trying to use the Leaks and Traces "resolve button dialog" to access the dSYM file but so far the Lazarus GUI says the file does not exist.

[Martin_fr]

About the "no extra size without dsym".

IIRC (and again: trip down memory lane, way down memory lane), the debug info is in the *.o files in the directories where the ppu files are too.
That means, if you build "Project1" that uses LCL, and then rebuild LCL with different settings ,then the debug info for project1 is no longer good. There are some checksums, so the changed info is ignored.... (Again, caution, that info is from distant memory...)

[kcandrews]

FWIW Here is an interesting section/comment in FpImgReaderMacho.pas.
Not at all my territory but sounds like a confirmation that the executable in Mac OSX will never contain DWARf-debug info, getting that info is a matter of going through all the .o files or using dSym.

To use the dSym file would require a Lazarus Mac user to know he/she must check the external debug checkbox to even have hope.  But then, in my experience, loading the dSym doesn't seem to be automatic proces in that case either.  And I wonder if there was any coordination between the "Leaks and Traces" developers and FPDebug?
...

procedure TDbgMachoDataSource.AddSubFilesToLoaderList(ALoaderList: TObject;
  PrimaryLoader: TObject);
var
  PLoader: TDbgImageLoader absolute PrimaryLoader;
  LList: TDbgImageLoaderList absolute ALoaderList;
  ALoader: TDbgImageLoader;
  fname, dSYMFilename: String;
  i: SizeInt;
begin
  // JvdS: Mach-O binaries do not contain DWARF-debug info. Instead this info
  // is stored inside the .o files, and the executable contains a map (in stabs-
  // format) of all these .o files. An alternative to parsing this map and reading
  // those .o files a dSYM-bundle could be used, which could be generated
  // with dsymutil.

  // PLoader.FileName in Contents/MacOS       

Is JvdS a developer's name who could be queried on this topic?  Are the developers involved in FPDebug and Leaks and Traces still active?

[TRon]

Is JvdS a developer's name who could be queried on this topic?

He is an active developer, yes. See f.e. his gitlab. Usual in this kind of situation is to post a question on the developers ML. I have no idea if he is active on the forums (most developers aren't).

[Martin_fr]

Code: Text  [Select][+][-]

1.   // JvdS: Mach-O binaries do not contain DWARF-debug info. Instead this info
2.   // is stored inside the .o files, and the executable contains a map (in stabs-
3.   // format) of all these .o files. An alternative to parsing this map and reading
4.   // those .o files a dSYM-bundle could be used, which could be generated
5.   // with dsymutil.

Ah, there it is. My memory wasn't that wrong.

If there is NO dsym:
* In your project dir is a lib folder, and it has ppu and o files => and they are queried for the debug info. If you clean out the lib folder, then that debug info is gone.
* Worse, as indicated, for packages. If another project recompiles the package => debug info is gone.

I don't know if the "references" to those o files are absolute or not. So I don't know if those o files will be found, if the app / app-bundle is moved.

[kcandrews]

Important to remind myself that Lazarus is a lot of excellent free work done by many generous people!

Thanks to all!

(I'll retreat back to the idea of setting up another OS to find my leaks.)

[Martin_fr]

Another way to check what happens

Look for "procedure TDwarfCompilationUnit.BuildAddressMap;" in components/fpdebug.

If the files are loaded, this code will build a map of the addresses of all subroutines found.

Code: Pascal  [Select][+][-]

1.                 if FAddressMap.HasId(Info.StartPC)
2.                 then DebugLn(FPDBG_DWARF_WARNINGS, ['WARNING duplicate start address: ', IntToHex(Info.StartPC, FAddressSize * 2)])
3.                 else FAddressMap.Add(Info.StartPC, Info);
4.  

You want to add

writeln(Info.StartPc, ' # ', Info.Name);
or
writeln(DbgHex(Info.StartPc),'-',dbghex(info.EndPC), ' # ', Info.Name);

Then when you try to resolve (and when lazarus has a console to print to) you would see if it loaded any functions. You would also see, if any of them has an address that matches your unresolved addresses.

If none are listed, then the problem is in loading debug info.

If all the addresses are a complete mismatch, the issue is with relocation info (i.e. addresses are written relative to something, and need an offset when loaded)

[kcandrews]

Martin,
" (and when lazarus has a console to print to) "
How do I accomplish providing a console for lazarus?
Thanks,
Keith

[Martin_fr]

In short, open a terminal and run lazarus from there.

I don't recall the name, but a terminal is included on Mac. Then change to the Lazarus dir (where ever that is for you).

Code: Text  [Select][+][-]

1. cd /home/user/lazarus
2. ./lazarus.app/MacOS/Content/lazarus
3.  

Something like that.


To write info to a file

Code: Text  [Select][+][-]

1. ./lazarus.app/MacOS/Content/lazarus  --debug-log=file

To get all the build in debug info (use with or without the debug-log=)

Code: Text  [Select][+][-]

1. ./lazarus.app/MacOS/Content/lazarus  --debug-log=file   --debug-enable=DBG_VERBOSE,FPDBG_DWARF_ERRORS

and any amount of others.

Code: Text  [Select][+][-]

1. grep -nri FindOrRegisterLogGroup  components/fpdebug components/lazdebuggers

[dbannon]

Code: Text  [Select][+][-]

1. ./lazarus.app/MacOS/Content/lazarus  --debug-log=file   --debug-enable=DBG_VERBOSE,FPDBG_DWARF_ERRORS

Things are never as easy as you expect on a Mac !

Code: [Select]

open ./lazarus.app/MacOS/Content/lazarus --args     "--debug-log=file"  "--debug-enable=DBG_VERBOSE,FPDBG_DWARF_ERRORS"
Might work better.

[kcandrews]

Thanks Martin & Davo!
I will follow up on your suggestions...but it will be a few days due to some urgent priorities :-)
Keith