Recent

Author Topic: New debugger for Mac based on lldb (Call for testers)  (Read 68679 times)

kcandrews

  • Jr. Member
  • **
  • Posts: 50
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #90 on: January 30, 2019, 12:53:35 am »
Martin,

Well done!!

I put the "exit" at the top of the LazLogger.CreateIndent function and the backtest is chugging away...without any issues so far (this will be a several hour test).  I did notice that setting a breakpoint "on the fly" while program was running in the lldb debugger caused an unknown class error -- but this is something I never "need" to do.

Also, KUDOS to those involved in making Cocoa work so well under Lazarus.  With the the your lldb debugger setup work and Cocoa (vs Carbon) allowing for easy 64-bit OSX applications, Lazarus will be well set to remain a great environment for all platforms :-)

Will report more later...

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #91 on: January 30, 2019, 01:43:45 am »
I put the "exit" at the top of the LazLogger.CreateIndent function and the backtest is chugging away...without any issues so far (this will be a several hour test). 
I still need to get an official fix in. But the trace you delivered was gold. And errors like this are most often hell to find.

Quote
I did notice that setting a breakpoint "on the fly" while program was running in the lldb debugger caused an unknown class error -- but this is something I never "need" to do.
More details, please?

Error in the IDE,  or in your app?
Setting a breakpoint, normally involves pausing your app, setting the breakpoint and starting it again. But all that is done by lldb.

Quote
Also, KUDOS to those involved in making Cocoa work so well under Lazarus.
You can find them here. If you go further back there are others.
https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/lcl/interfaces/cocoa/?root=lazarus&view=log

Quote
With the the your lldb debugger setup work and Cocoa (vs Carbon) allowing for easy 64-bit OSX applications, Lazarus will be well set to remain a great environment for all platforms :-)

The lldb debugger is making big use of fpDebug. Which also (over the years) is the work of many. But yes, I did the final touch stringing it together with lldb for the Mac. ;)
« Last Edit: January 30, 2019, 01:57:30 am by Martin_fr »

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #92 on: January 30, 2019, 01:57:05 am »
Try with the changes from https://svn.freepascal.org/cgi-bin/viewvc.cgi/trunk/components/lazutils/lazlogger.pas?root=lazarus&r1=60246&r2=60245&pathrev=60246

You can probably download the entire file (even it is from trunk), and replace it in your fixes or RC installation.


Lazlogger will need more fixes. But the way it is used in the IDE, this should be enough.
« Last Edit: January 30, 2019, 01:59:36 am by Martin_fr »

kcandrews

  • Jr. Member
  • **
  • Posts: 50
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #93 on: January 31, 2019, 05:56:15 am »
Martin,

Thanks again.  I did get the your latest file revision from the trunk and rebuilt the IDE.  (So I'm using RC3 but with the several debug related folders updated from the trunk.)  In a number of tests on my big project, due to your fixes, the debugger did not crash and works well with breakpoints I set.  As BigDan and possibly others have said (as you suggested probably needs an FPC fix), on OSX, stepping in/out after a breakpoint doesn't work and just takes me into the assembly code window (appearing to lose the source code "position"). 

But for now, it is great that the breakpoints work to successfully stop and allow me to view variable values.

Glad my backtrace was helpful -- from decades of consulting work often searching for those "hellish" bugs, I know how great it is when they reveal themselves.

All for now!  Thanks for all your work and help to so many :-)

kcandrews

  • Jr. Member
  • **
  • Posts: 50
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #94 on: January 31, 2019, 08:36:04 pm »
Hi Martin,

I do have a question re: "type of debug info" output when compiling a project.

For OSX 10.13.6 (High Sierra), Lazarus 2.0.0RC3, X86_64-darwin-cocoa (alpha) (with your current trunk debugger fixes for the LazDebuggerFPLldb 0.0 package)...

I do know that using gdb I was to set my "Project Options/Compiler Options/Debugging/Type of debug info" to "Dwarf with sets (-gw -godwarfsets).

BUT I am unclear on what "Type of debug info" is required for the new LazDebuggerFPLldb setup since I really don't know the various layers involved making the lldb debugging work (for OSX).  Earlier in this thread there is mention of  "-godwarfcpp" so I added that to my Complier Options/Custom Options -- but this feels like an overlap?

Just quite ignorant on all this...please advise when you have time :-)

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #95 on: January 31, 2019, 09:03:09 pm »
The lldb based debugger(s) need dwarf.

So basically, any dwarf info will do. So far I have not found any differences. (well a few below)

I do not know which versions of dwarf are supported by lldb. Or to which extend. On windows lldb warns, that it does not know all of the dwarf tags. But it still works. So no worry. Using "lldb + fpdebug", it does not matter which tags lldb knows....


Looking at it systematically:

Dwarf info is used for 2 purposes:
A) line info / and the ability to read the stack, and map it to functions and unit/lines)
B) symbol info, displaying watches, locals, ....

If you use lldb WITHOUT fpdebug, then LLDB must understand all of it. So the choice of which dwarf info may have effects on lldb.

The recommended "lldb with fpdebug", lldb mainly needs line-info (IIRC, lldb displays the stack, so the symbols shown as parms in the stackview require symbol info. If lldb has issues there then the various settings (below) may affect this.


fpdebug can use any of the dwarf settings.

And here are the dwarf settings, and any known effects they have on FpDebug. (I have no data, how they affect lldb (for the stacklist))

* Dwarf -gw  (this is actually dwarf2)
- Works fine, but you can not inspect "set of (...)"
- Symbols are shown all uppercased
- issues with strings (strings == pchar)

* Dwarf with sets
Same as above, but can do sets (sets are part of dwarf3, but included here)

* Dwarf3
- Works fine. I recommend this one (unless your lldb bails out, then check out if it works with any of the above)
- Symbols are uppper/lower case as in source
- strings are fine
- also may improve len detection for dynamic array
- widestrings are still pwidechar

*Fpc trung also has dwarf4 -gw4
I am not aware of where it actually adds different info. Not known if it does anything yet

* -godwarfcpp
This is an option that can be added to any of the above.
If or What it does may depend on the version of fpc, and the dwarf version chosen from the above.
- It may impact fpdebug (eg ability to differ between string and pchar)
- From the description I found (IIRC fpc mail list) it attempts to write more c-ish info, so lldb can better function.
  I am unaware if this impacts line info in anyway.
=> IF you do not have any problems without it, then do not use it.
=> Probably it will only do good, if you use lldb without fpdebug, fpc trunk, and dwarf3 or 4



For any of those debug infos there are currently hard coded upper limits for the size of string/array. See the initial post of this thread. This has other reasons.

All else is trial and error.

kcandrews

  • Jr. Member
  • **
  • Posts: 50
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #96 on: January 31, 2019, 09:11:05 pm »
Martin,

Great and thorough explanation on the compiler settings re: dwarf.  Great to have all that in one place.  Thank you.

And since I have your attention again, just curious that in my OSX setup, I can successfully use "Step Over" and "Step Into" IF I use gdb.  But stepping does not work with the new fp/lldb setup (for me).  I'm trying to wrap my head around why this could be an "FPC" problem.  Again, probably just ignorance and thank you for tolerating it :-)

FWIW, I attached a screen shot of the assembler window after a breakpoint successfully stopped.  But when I hit F7 (step) the program goes into some other code entirely (not showing any related source code nearby).  [see the assembler window after F7 in my next message]
« Last Edit: January 31, 2019, 10:20:57 pm by kcandrews »

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #97 on: January 31, 2019, 09:34:31 pm »
About the stepping...
It is not certain where the problem is.

But it is either in fpc, or in gdb/lldb.

It has been seen on different platforms. On some with gdb and on others with lldb. Or maybe on some with both. 
Most of the info relies on reports from others. I have seen it myself with gdb on windows 64 bit (but all ok with 32 bit).

Pure fpdebug (Win/Linux) does not have the problem (as far as I can tell).

The IDE only send a command to gdb/lldb. And gdb/lldb then reports back where it ended the step.

It would require to go through the implementation of gdb or lldb, and see how they determine where the step ends.
This is currently not on by todo list.


Pure fpdebug for Mac is an option for the future.
Current state is: (That is if it works at all, there is code there, but it hadn't been tested in a long time)
1) stepping / breakpoint / pausing / control flow: Only support one thread. Behaviour in multi thread targets is unpredictable.
2) the entire IDE needs to be codesigned

As for 2: There is fpdebugserver, which comes with a small app, that acts as external debugger. Then only that app needs to be codesigned. But that has not been tested in at least as long.


The decision was for lldb now, because lldb is code signed by apple. So no steps to create your own cert and all.

kcandrews

  • Jr. Member
  • **
  • Posts: 50
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #98 on: January 31, 2019, 10:18:12 pm »
Martin,

Understand..thank you.  Since OSX has been and is becoming gdb unfriendly and lldb works there, the decision makes sense.

I'll try my same tests after shifting all my stuff & IDE back to 32 bit (Carbon) to see what stepping does there.

FWIW, I've attached a screen shot follow-up -- where my code goes (assembler-window-wise) after hitting F7 after the previous screen shot.

Keith
« Last Edit: January 31, 2019, 10:45:53 pm by kcandrews »

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #99 on: January 31, 2019, 10:24:27 pm »
I'll try my same tests after shifting all my stuff & IDE back to 32 bit (Carbon) to see what stepping does there.

You can build and debug 32 bit apps from a 64 bit IDE (and vice versa).

Though for debugging cross-bitness you do need svn fixes (or trunk). RC3 has still bugs preventing this.

kcandrews

  • Jr. Member
  • **
  • Posts: 50
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #100 on: February 01, 2019, 04:49:11 pm »
Martin,

If I wanted to take a look at the sending and receiving directives (for stepping) to lldb & gdb, what units are involved?

Thanks!
Keith
« Last Edit: February 01, 2019, 05:24:37 pm by kcandrews »

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #101 on: February 01, 2019, 05:26:47 pm »
The easiest way to see the communication is to open: menu View> Ide Internals > Debugger Output
On mac, you may need to resize the window, if the memo scrambles the text....

Compile the IDE with -dDBG_WITH_DEBUGGER_DEBUG and the window allows sending commands to lldb. However sending step commands can hang the debugger, as internal state is not kept for those commands.
This function is only tested with gdb, and was designed for data inspection. But you can try....


The 2nd way to watch, is to start the IDE from console and use the --debug-enable (from the start of this thread) => the output goes to console.
Or use --debug-log=file and tail the file
Actually you only need  --debug-enable=DBG_CMD_ECHO  the other flags show other info. (like state changes of the internal debugger object).


The code for dealing with stepping is in the package LazDebuggerLldb (it is the same with/without fpdebug)

unit LldbInstructions, contains mini-objects for each instruction and receiving the response.

unit LldbDebugger is in control of triggering them, and dealing with the result.

The IDE calls  TLldbDebugger.RequestCommand
which then calls TLldbDebugger.LldbStep
which creates a TLldbDebuggerCommandRunStep.Create object
This is queued and when the queue get to it, it calls TLldbDebuggerCommandRun.DoExecute
DoExecute will be called twice: First goes to DoInitialExecute, and then goes to TLldbDebuggerCommandRun.ResumeWithNextStepCommand
This creates TLldbInstructionProcessStep.Create and sets the finished handler to RunInstructionSucceeded
However the actual code flow resumes in TLldbDebuggerCommandRun.DoLineDataReceived


A lot of the code deals with what happens if you step over an exception.
If you do (eg step over a procedure that raises an exception) then the breakpoint at fpc_raise_exception is hit, the debugger will stop at that breakpoint, and the step is interrupted.
If the particular exception is ignored that is handled, and the IDE calculates where the step should have ended.


If there is no exception then it should read (in several calls to that method) the stop/stopped messages from lldb, and exit the command after setting the debuggerstate, and calling Finished.

------------------------
It may be possible to detect, if stepping ends inside a sub-function, and maybe get back to the main method in the same way this is done for exception (except you would not need to check for pop_exception, until you actually encounter an exception...



https://bugs.freepascal.org/view.php?id=34159

I don't know what tools there are on Mac for this. But if there is a possibility to strip away, or prevent certain sections of the debug info....

On windows it appears that this helps: objcopy.exe -R .debug_frame project1.exe project1n.exe

Removing a section called ".debug_frame"
« Last Edit: February 01, 2019, 05:37:49 pm by Martin_fr »

kcandrews

  • Jr. Member
  • **
  • Posts: 50
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #102 on: February 01, 2019, 05:47:42 pm »
Martin,
More than enough to go on :-)
Semi-retired, off to try out some new ski boots at Steamboat (Colorado), but then will slowly see what I can learn.
Thank you!

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9791
  • Debugger - SynEdit - and more
    • wiki
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #103 on: February 01, 2019, 10:23:09 pm »
Try the following:

FILE components\lazdebuggers\lazdebuggerlldb\lldbdebugger.pas
LINE 1065 (existing code)
Code: Pascal  [Select][+][-]
  1.     else
  2.     if FUnknownStopReason <> '' then begin
  3.       DoUnknownStopReason(FUnknownStopReason);
  4.     end
  5. // >>>>>>>>>>>>>>>>>>>> BEGIN NEW CODE
  6.     else
  7.     if StrContains(found[1], ' over') and (FMode = cmRun) and
  8.        (FStepAction = saOver) and (FFramePtrAtStart <> 0) and (frame < FFramePtrAtStart)
  9.     then begin
  10.       // SetDebuggerState(dsInternalPause);
  11.       ContinueRunning;
  12.     end
  13. // <<<<<<<<<<<<<<<<<<<<< END NEW CODE
  14.     else
  15.       SetDebuggerState(dsPause);
  16.  
  17.     if (FState = crRunning) then
  18.       exit;
  19.  

It may be that instead of "(frame < FFramePtrAtStart)" it needs to be "(frame > FFramePtrAtStart)"  ("<" vs ">")

I have not tested myself. Need to get out the test equipment....

--
I think/hope  "SetDebuggerState(dsInternalPause);" is not needed. But if it does not work, try with this line.




EDIT:

Just started some tests on gdb. It may not work that way, because the stepping stops before the frame(base)pointer (variable "frame") is changed.

That means the code probably must do a lot more.
It could detect if the stackpointer (rsp/esp) changed (got smaller). But that can happen without entering a procedure (if the code "push"es something onto the stack / this is rare in fpc, especially in none optimized code, but it has to be assumed possible).

Interestingly gdb actually shows a change in the framepointer..... Need to see what lldb does.
« Last Edit: February 01, 2019, 11:29:08 pm by Martin_fr »

kcandrews

  • Jr. Member
  • **
  • Posts: 50
Re: New debugger for Mac based on lldb (Call for testers)
« Reply #104 on: February 02, 2019, 09:50:08 pm »
Hi Martin,

While you've been working, me 2 a bit.  As I had suspected, based on the assembly window looking like maybe some other thread, I've been theorizing that the wrong thread (not the thread where my code is) is being selected on the step-in.  So I used your suggestion of compiling so I could intervene by sending commands to lldb directly.  It turns out if after a breakpoint, I give the lldb command "thread select 10" (my code thread), THEN when I do a step-in using the Lazarus IDE, I wind up at the next line of my source code as expected.

So I haven't looked at your latest post, but I'll do that too.  In the meantime, you may know where the "thread select" directive is either missing, or choosing the wrong thread?  From a little more step work, it appears that this "wrong" thread is regularly being selected -- since the wrong one is usually chosen when I just run to the next breakpoint, and try a step again (at which point the thread is no longer my #10).  But at that point if I give the "thread select 10" again, all works as expected. 

So all this leads me to believe that lldb itself is working correctly and that the directives to it from the IDE need to specify the correct thread. 

As info, other than to learn that gdb works for me both in generating 32 bit and 64 bit OSX applications, I've just been working with lldb since that's what I need to have working.
« Last Edit: February 02, 2019, 10:01:33 pm by kcandrews »

 

TinyPortal © 2005-2018