Lazarus

Using the Lazarus IDE => Debugger => Topic started by: Luc on June 24, 2022, 06:49:51 pm

Title: Debugging problem on Windows 11 ARM
Post by: Luc on June 24, 2022, 06:49:51 pm
Hi,

I have some trouble using fpDebug on Windows 11 ARM + Parallel VM: The debugger stops at the first breakpoint found in the program flow, no problem here. But quite often (not every time) when I press F7 or F8 or F9 the IDE doesn't step and doesn't give back the execution flow to the program being debugged.

Everything is fine under Win8 or Win10 x64 + Parallel VM.

Thanks
Luc
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 24, 2022, 08:35:11 pm
Not quite clear what you do:

- Debugging an arm/avr target, with the avr-embedded FpDebug?
- Debugging an x84-64 program in the VM? (And presumingly the IDE also running in the VM?)

If the latter, then try running:
Code: Text  [Select][+][-]
  1.     lazarus.exe --debug-log=c:\laz-log.txt --debug-enable=DBG_VERBOSE,DBG_STATE,DBG_DATA_MONITORS,DBGMI_QUEUE_DEBUG,FPDBG_DWARF_ERRORS,DBG_VERBOSE,DBG_WARNINGS,DBG_STATE,DBG_EVENTS,DBG_THREAD_AND_FRAME,DBG_ERRORS,DBG_FPDEBUG_VERBOSE,FPDBG_BREAKPOINT_ERRORS,FPDBG_BREAKPOINTS,FPDBG_COMMANDS,FPDBG_THREADS,FPDBG_QUEUE,FPDEBUG_THREAD_CHECK,FPDBG_WINDOWS    

And get the output attached. It may contain names/symbols from your app. If that is an issue, then remove that part first.


Also if you can: try running Lazarus 2.3.


Very important
If you did rebuild your IDE, make sure you used max -O-1  (or FPC 3.2.3 or 3.3.1)

If you used -O2 or above to build the IDE, there is a bug in fpc 3.2.2, and fpdebug will misbehave.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on June 25, 2022, 04:06:06 pm
Thanks Martin,

Sorry to not being clear. I am

Quote
Debugging an x84-64 program in the VM? (And presumingly the IDE also running in the VM?)

I enclosed my current settings (removing the used libs as it should not be relevant here). The IDE was rebuilt with -O1. I did some more tests with a simple project and I don't have any problem with it.

The one I have problems with is far more complex and uses many Forms or Frames.

The laz-log - NQ.txt doesn't seem to bring some light but I also enclose a screenshot of the assembler windows when the debugger hangs (The IDE is still responding but I cannot go back to the program being debugged nor stepping the code).

As a side note, I also found that you still can select "use external debug symbols file" when fpDebug is selected as the default debugger. AFAIK, this option should be disabled in the IDE with fpDebug?

Please let me know if there is something else I can do to solde this breakpoint problem.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 25, 2022, 04:59:59 pm
Quote
As a side note, I also found that you still can select "use external debug symbols file" when fpDebug is selected as the default debugger. AFAIK, this option should be disabled in the IDE with fpDebug?
FpDebug does support this. (at least in 2.3 / not sure when it was added). But it has no benefit. (Unless someone ships their releases with -O-1 -Criot -gh -Sa  => which is what a debug build should normally have... IMHO)


Ok, the log framework appears to have some bug. The debug-enable does not seem to work.

Can you recompile and add the following options in the "configure build lazarus"
Code: Text  [Select][+][-]
  1. -dDBG_VERBOSE
  2. -dDBG_STATE
  3. -dDBG_DATA_MONITORS
  4. -dDBGMI_QUEUE_DEBUG
  5. -dFPDBG_DWARF_ERRORS
  6. -dDBG_VERBOSE
  7. -dDBG_WARNINGS
  8. -dDBG_STATE
  9. -dDBG_EVENTS
  10. -dDBG_THREAD_AND_FRAME
  11. -dDBG_ERRORS
  12. -dDBG_FPDEBUG_VERBOSE
  13. -dFPDBG_BREAKPOINT_ERRORS
  14. -dFPDBG_BREAKPOINTS
  15. -dFPDBG_COMMANDS
  16. -dFPDBG_THREADS
  17. -dFPDBG_QUEUE
  18. -dFPDEBUG_THREAD_CHECK
  19. -dFPDBG_WINDOWS    
  20.  

And then just
Code: Text  [Select][+][-]
  1.  lazarus.exe --debug-log=c:\laz-log.txt
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on June 25, 2022, 06:53:24 pm
Much better now I think  :)

Here is the log file. I was able to step over F8 several times then I tried F9 and it hanged again. It appears to have "access is denied" messages in the log file but I don't know if it is relevant here.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 25, 2022, 09:44:05 pm
Could you please add
Code: Text  [Select][+][-]
  1. -dDebuglnWinDebugEvents

Also could you apply the following patch
Code: Diff  [Select][+][-]
  1. diff --git a/components/fpdebug/fpdbgwinclasses.pas b/components/fpdebug/fpdbgwinclasses.pas
  2. index 86e855ed0f..37b9ba35db 100644
  3. --- a/components/fpdebug/fpdbgwinclasses.pas
  4. +++ b/components/fpdebug/fpdbgwinclasses.pas
  5. @@ -784,7 +784,7 @@ function TDbgWinProcess.Continue(AProcess: TDbgProcess; AThread: TDbgThread;
  6.  var
  7.    EventThread, t: TDbgThread;
  8.  begin
  9. -debugln(FPDBG_WINDOWS, ['TDbgWinProcess.Continue ',SingleStep]);
  10. +debugln(FPDBG_WINDOWS, ['TDbgWinProcess.Continue ',SingleStep, ' ', DbgSTime ]);
  11.    if assigned(AThread) and not FThreadMap.HasId(AThread.ID) then begin
  12.      AThread := nil;
  13.    end;
This will give me some time-stamps.

If you can't get the patch in, then when you get the issue, wait at least half a minute, before pressing stop.
And also, if you can, copy the log-file before pressing stop, though the full file will also be needed, since when the copy is taken not all content may have been written (not flushed to disk).


Further, if possible
- close the locals dialog
- clear (or disable) the watches dialog
- important: open the "debug history" and disable it (power button off)
Though, its not likely to make a difference. The log does not show anything that would point towards watch/locals.





What I can see...

There are several threads. But they may be from the OS, or other dll.

- You did hit a breakpoint, and then several F8 (around line 800)
- Then an F8 did step out, to line 415
- Next a step to line 416
- Then run with F9  (at least that is what it looks like)
  -> which eventually stopped at the same breakpoint again
- Again F9 -> this time stopping at a different breakpoint
- Again F9
  -> and then something goes wrong
... It looks that at some point, during this you did press "stop" ?

From what it looks like, some thread in your app did exit.
I can't tell exactly if that was the thread you were debugging, or one of the other threads (i.e. some thread inserted by the OS, which would mean that thread may well have been ok to exit).

It does look like it may be one of those other threads. (The above "-dDebuglnWinDebugEvents" hopefully will tell).
If so, that is probably a rare event to happen (i.e. the thread-exit happening exactly at a very specific moment in what fpdebug does).

Then again, my guess is that all the "Access is denied." are threads that did finish => and if that many threads go down, something else may have happened.






Note: Several stops at a breakpoint
- well, either you did actually run all the way, and just came back
- Or...
  If a single command wraps several lines
Code: Pascal  [Select][+][-]
  1.   CallFoo(GetBar(),
  2.     GetOther(),
  3.     GetMore()
  4.   );
  Then the first line, is reached several times while the others are executed.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on June 26, 2022, 11:09:25 pm
Thanks Martin for taking the time to investigate this issue.

I followed up all the instructions but the problem is still here.

To simplify the debug process I added a breakpoint in the main form which is triggered by a simple click in the main thread context.

I enclosed 3 files:
v1 is the log saved when the "freezing" happened after 2 x F9 I think (sometime the first F9 freezes the debugger)
v2 is the log saved after stopping the execution. I waited at least 1 min but nothing was added to the log
v3 is the log saved after closing the IDE

There are 5 or 6 threads running at program start (depending on the context). I can try to disable them if you think the problem could be related.

I don't have any problem debugging on OSX + i7 + VM + Win 8.1 + Lazarus x64->x32
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 27, 2022, 11:12:43 am
Still very strange.

According to the messages that Windows sends to the debugger, the debugged app exited without any error (errorcode = 0).
This happens within 1 second from when you tried to continue with F9.


v1 is the log saved when the "freezing" happened

About "freezing", from the log it looks like the debugger stopped "normally" (well, ignoring that it shouldn't stop...).
That is, it should behave as if the app just exited.

I take it, that you probably disabled "show message on stop" from "tools > options > debugger > general".
So you don't get the message box "Execution stopped".
==> You should however see that the red stop-button gets disabled (greyed), and the green-start button enabled again?

So if you say "freeze" what exactly does that describe?
- What are the states of the debugger tool buttons?
- What does the caption of the IDE title-bar say (does it still say "debugging")?
- If, it is stopped, can it be started again with F9 or Run?
  (Note, if you press F8 while the debugger is stopped, then the debugger will hang)


Depending on whether the debugger "stops" or "freezes", there are one or two issues.
- Issue 1: your app terminates, when it certainly should not
- Issue 2: the debugger hangs




About the "your app terminates".

This could be some issue with Windows11 or your VM.

It could be Windows Defender too.
Under Windows-10 there is "Windows security" > "Virus and thread protection" on that page "manage settings" => "virus and thread protection settings":
A lot of on/off switches, and at the bottom of the page "Exclusions" / "Add or remove exclusions"
Add both, the IDE, and the debugged app.


There is a bit of a "chicken and egg" issue with all the "access denied" in the logfile.
I can't tell, if they (all/some of them) are a consequence of your app terminating or if some of them are causing (indirectly, via fpdebug) your app to terminate.
Though in the latter case, I would not expect "error code = 0"

Code: Text  [Select][+][-]
  1.   EVENT for Process 26848 Thread 26740: << EXIT_THREAD_DEBUG_EVENT exitcode:0 True
  2.   EVENT for Process 26848 Thread 13200: << EXIT_PROCESS_DEBUG_EVENT exitcode:0 True

There is one test you could do.
   components\fpdebug\fpdbgwinclasses.pas
   line 1600
Code: Pascal  [Select][+][-]
  1. procedure TDbgWinThread.Suspend;
  2.  
  3. ...
  4.  
  5. procedure TDbgWinThread.Resume;
  6.  
In both procedures insert an "exit;" as very first statement. So both functions will exit without doing anything at all.

Then FpDebug will no longer try to disable a thread that has exited. In case this somehow causes more other threads to exit.
It is possible that some of the threads are valid to exit at those time. This could be threads the OS added, and they are expected to terminate at some time.








F8 while the debugger is stopped
FpDebug can only be started with F9/Run.
However the IDE does not obey that, and will send an F8 (in order to start debugging / as gdb does accept this).
In that case, FpDebug does not start, but the IDE believe it is running => only "reset debugger" will work.



Btw, while not ideal, but you can still use gdb and at least be able to debug (unless gdb has the same issue).
Tools > Options > Debugger Backend



Also can you please confirm: Your IDE is 64bit and your are cross debugging to 32bit? (Which should work)
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 27, 2022, 12:36:28 pm
One more thing, I can see there are libraries unloaded shortly before the breakpoint. However, I can't see their names.

So to make sure they are not related, can you open the debugger event log (and ensure in the Tools > Options ...) that library events are shown (should be on by default).

Then just check the last few entries before the issue arises.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on June 27, 2022, 02:44:46 pm
Indeed, this is strange.
 
Quote
I take it, that you probably disabled "show message on stop" from "tools > options > debugger > general".
So you don't get the message box "Execution stopped".
==> You should however see that the red stop-button gets disabled (greyed), and the green-start button enabled again?

Yes, the message box "Execution stopped" is disabled but the program being debugged never exits by itself and the IDE is always showing "debugging". You can see it in the screenshoot. The Task Manager shows the debugged exe as "Not responding" (PID 5912), the IDE "Debugging" and nothing happen when I hit F9 or any other debugging key. The assembler window shows the last breakpoint hit but all the assembler lines disappear if I resize the assembler windows like if it were not available anymore.

Quote
So if you say "freeze" what exactly does that describe?
- A - What are the states of the debugger tool buttons?
- B - What does the caption of the IDE title-bar say (does it still say "debugging")?
- C - If, it is stopped, can it be started again with F9 or Run?
  (Note, if you press F8 while the debugger is stopped, then the debugger will hang)

A and B: Please have a look at the screenshot
C: No, unable to run the program again

Quote
Depending on whether the debugger "stops" or "freezes", there are one or two issues.
- Issue 1: your app terminates, when it certainly should not
- Issue 2: the debugger hangs

Issue 1: No the debugged app doesn't exit. It is still running but kind of "captured" by the debugger
Issue 2: It seems so but the IDE is still fully responding.

Quote
Also can you please confirm: Your IDE is 64bit and your are cross debugging to 32bit? (Which should work)
Yes this is the case.

I tried to follow your instructions to the best, disabling all the Win 11 system security for lazarus.exe, changing the IDE debugger options, activating all the options in debug_log, applying the "exit" statements (Suspend, Resume) and so on but still no luck  :(

You will find another log file which hopefully could give you some more information.

At the end, I switched to gdb and it seems to work fine so far.
Let me know if I can help solving this problem with fpDebug, may be a simple one but hard to catch...


Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 27, 2022, 02:57:44 pm
Ouch, when I compared the timestamps
814140
814401

I somehow -in my mind  - swapped some digits around, and saw 1 sec difference.... My bad.
So then, all the "access denied" is fine. That is when you stop the app. And expected.

Let's look again.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 27, 2022, 06:17:33 pm
OK, so indeed the app is just kept suspended....

I have 2 ideas. I added a lot of debug output.

Here the patch (or attached the complete file, based on 2.2)
Code: Diff  [Select][+][-]
  1. diff --git a/components/fpdebug/fpdbgwinclasses.pas b/components/fpdebug/fpdbgwinclasses.pas
  2. index 9d5094a167..9b3b757500 100644
  3. --- a/components/fpdebug/fpdbgwinclasses.pas
  4. +++ b/components/fpdebug/fpdbgwinclasses.pas
  5. @@ -722,6 +722,11 @@ class function TDbgWinProcess.isSupported(ATargetInfo: TTargetDescriptor
  6.              (ATargetInfo.machineType in [mt386, mtX86_64]);
  7.  end;
  8.  
  9. +function dbgsThread(AThread: TDbgThread): string;
  10. +begin
  11. +  if AThread = nil then result := 'nil' else Result := IntToStr(AThread.ID);
  12. +end;
  13. +
  14.  function TDbgWinProcess.Continue(AProcess: TDbgProcess; AThread: TDbgThread;
  15.    SingleStep: boolean): boolean;
  16.  
  17. @@ -740,7 +745,7 @@ function TDbgWinProcess.Continue(AProcess: TDbgProcess; AThread: TDbgThread;
  18.  var
  19.    EventThread, t: TDbgThread;
  20.  begin
  21. -debugln(FPDBG_WINDOWS, ['TDbgWinProcess.Continue ',SingleStep]);
  22. +debugln(FPDBG_WINDOWS, ['TDbgWinProcess.Continue ',SingleStep, ' ',DbgSTime, ' thread=',dbgsThread(AThread), ' event-tid=', MDebugEvent.dwThreadId]);
  23.    if assigned(AThread) and not FThreadMap.HasId(AThread.ID) then begin
  24.      AThread := nil;
  25.    end;
  26. @@ -804,6 +809,7 @@ function TDbgWinProcess.Continue(AProcess: TDbgProcess; AThread: TDbgThread;
  27.      if SingleStep then
  28.        TDbgWinThread(AThread).SetSingleStep;
  29.    end;
  30. +debugln('go before continue');
  31.    AProcess.ThreadsBeforeContinue;
  32.  if AThread<>nil then debugln(FPDBG_WINDOWS, ['## ath.iss ',AThread.NextIsSingleStep]);
  33.  
  34. @@ -811,13 +817,18 @@ function TDbgWinProcess.Continue(AProcess: TDbgProcess; AThread: TDbgThread;
  35.      case MDebugEvent.Exception.ExceptionRecord.ExceptionCode of
  36.       EXCEPTION_BREAKPOINT, STATUS_WX86_BREAKPOINT,
  37.       EXCEPTION_SINGLE_STEP, STATUS_WX86_SINGLE_STEP: begin
  38. +debugln(['GOING TO RUN: DBG_CONTINUE ',MDebugEvent.dwThreadId]);
  39.         result := Windows.ContinueDebugEvent(MDebugEvent.dwProcessId, MDebugEvent.dwThreadId, DBG_CONTINUE);
  40.       end
  41. -    else
  42. -      result := Windows.ContinueDebugEvent(MDebugEvent.dwProcessId, MDebugEvent.dwThreadId, DBG_EXCEPTION_NOT_HANDLED);
  43. +     else begin
  44. +debugln(['GOING TO RUN: DBG_EXCEPTION_NOT_HANDLED ',MDebugEvent.dwThreadId]);
  45. +        result := Windows.ContinueDebugEvent(MDebugEvent.dwProcessId, MDebugEvent.dwThreadId, DBG_EXCEPTION_NOT_HANDLED);
  46. +    end;
  47.      end
  48. -  else
  49. +  else begin
  50. +debugln(['GOING TO RUN: DBG_CONTINUE ',MDebugEvent.dwThreadId]);
  51.      result := Windows.ContinueDebugEvent(MDebugEvent.dwProcessId, MDebugEvent.dwThreadId, DBG_CONTINUE);
  52. +  end;
  53.    DebugLn((FPDBG_WINDOWS or DBG_WARNINGS) and (not Result), 'ContinueDebugEvent failed: %d', [Windows.GetLastError]);
  54.    result := true;
  55.    MDebugEvent.dwProcessId := 0; // Flag as running // for assert in ReadThreadState
  56. @@ -1562,6 +1573,7 @@ procedure TDbgWinThread.Suspend;
  57.  var
  58.    r: DWORD;
  59.  begin
  60. +debugln(['About to suspend ', id, ' already=', FIsSuspended, ' skip break=',FIsSkippingBreakPoint]);
  61.    if FIsSuspended then
  62.      exit;
  63.    r := SuspendThread(Handle);
  64. @@ -1586,6 +1598,7 @@ procedure TDbgWinThread.Resume;
  65.  begin
  66.    if not FIsSuspended then
  67.      exit;
  68. +debugln(['About to Resume', id]);
  69.    r := ResumeThread(Handle);
  70.    FIsSuspended := not(r <> DWORD(-1));
  71.    debugln(DBG_WARNINGS and (r = DWORD(-1)), 'Failed to resume Thread %d (handle: %d). Error: %s', [Id, Handle, GetLastErrorText]);
  72. @@ -1609,11 +1622,17 @@ procedure TDbgWinThread.SetSingleStep;
  73.      if not ReadThreadState then
  74.        exit;
  75.    {$ifdef cpux86_64}
  76. -  if (TDbgWinProcess(Process).FBitness = b32) then
  77. +  if (TDbgWinProcess(Process).FBitness = b32) then begin
  78. +debugln(['Settig single step flag ', id, '  current = ', FCurrentContext^.WOW.EFlags or FLAG_TRACE_BIT]);
  79.      FCurrentContext^.WOW.EFlags := FCurrentContext^.WOW.EFlags or FLAG_TRACE_BIT // TODO WOW_FLAG....
  80. -  else
  81. +  end
  82. +  else begin
  83.    {$endif}
  84. +debugln(['Settig single step flag ', id, '  current = ', FCurrentContext^.def.EFlags or FLAG_TRACE_BIT]);
  85.      FCurrentContext^.def.EFlags := FCurrentContext^.def.EFlags or FLAG_TRACE_BIT;
  86. +  {$ifdef cpux86_64}
  87. +  end;
  88. +  {$endif}
  89.    FThreadContextChanged:=true;
  90.  end;
  91.  
  92. @@ -1681,8 +1700,9 @@ function TDbgWinThread.DetectHardwareWatchpoint: Pointer;
  93.  
  94.  procedure TDbgWinThread.BeforeContinue;
  95.  begin
  96. +debugln(['TDbgWinThread.BeforeContinue ', id, ' / ',FThreadContextChanged, '  susp=',FIsSuspended]);
  97.    if ID = MDebugEvent.dwThreadId then begin
  98. -    inherited;
  99. +    inherited;  ///////////////////////////// MOVE this up, outside the "IF"
  100.  
  101.      {$ifdef cpux86_64}
  102.      if (TDbgWinProcess(Process).FBitness = b32) then begin
  103.  




Once you reproduced the error, and have a log for it, find the line
Code: Text  [Select][+][-]
  1.    inherited;  ///////////////////////////// MOVE this up, outside the "IF"
And move it to the begin of the procedure, so it is unconditionally executed (that is actually already done in git main, but apparently wasn't merged.

And depending, on if that helps or not, you could also follow my earlier suggestion:
(if either of my 2 ideas is correct, this should make the issue go away...)
Quote
   components\fpdebug\fpdbgwinclasses.pas
   line 1600
Code: Text  [Select][+][-]
  1.     procedure TDbgWinThread.Suspend;
  2.      
  3.     ...
  4.      
  5.     procedure TDbgWinThread.Resume;
  6.  
     
In both procedures insert an "exit;" as very first statement. So both functions will exit without doing anything at all.




A bit of background.
So the process is not started again. Well it is "started/continued", but it then does not execute a single asm instruction. So it might be started suspended.
And that is where I can think of 2 ways this could happen.

FpDebug suspends some of the threads, while it has to temporarily remove the breakpoint. If it wouldn't suspend the other threads, they could run that code, and would never hit the (temp removed) breakpoint.

1) Maybe FpDebug somehow suspends all threads - the new log will hopefully tell me.

2) When a breakpoint is hit, it's possible the OS creates a temporary thread. Then maybe if that is suspended (depending on some timing), it will hold the main thread too....
Actually, I don't see that on my Windows 10....
But I did see threads starting and stopping in your log - they may of course be something else entirely.

On that note, could you open the "thread window" and when it hangs provide a snapshot?
The log doesn't have the names of the procedures (will be somewhere in the kernel), and the thread window may show those names.




Thanks for helping.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on June 27, 2022, 06:37:18 pm
Thanks Martin,
I will have a look tomorrow
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on June 28, 2022, 09:27:04 am
Hi Martin,

Problem has gone away :)

Not really as this is done by bypassing some procedures but I applied all the patches an it doesn't hang anymore.

A: I started by replacing fpdbgwinclasses.pas by the one you provided, recompiling the fpDebug package and rebuilding the IDE
Result : no change, debugger hangs

B: I moved the "inherited" on top of IF statement and rebuilt fpDebug + IDE
Result: no change, debugger hangs

C: I added Exit; on Supend and Resume procedures and rebuilt fpDebug + IDE
Result: Everything is fine, no more "hang" 

I guess you found the code causing the issue.
I can revert back to step B or A to help find the problematic code if you need so.   
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 28, 2022, 01:31:37 pm
A: I started by replacing fpdbgwinclasses.pas by the one you provided, recompiling the fpDebug package and rebuilding the IDE
Result : no change, debugger hangs

Expected. But could I have the logs for that run please?

And could I have a screenshot of the "threads window" while the debugger hangs?




Quote
C: I added Exit; on Supend and Resume procedures and rebuilt fpDebug + IDE
Result: Everything is fine, no more "hang" 
Ok, at least something to go on.


If you want to use FpDebug, you can use it with those 2 "exit" in place. It should be stable.

The only difference will be:
- If you have more than one thread.
- If at least 2 of them run the same code (i.e. Both of them run through the procedure "DoSomeStuff"
- You have a breakpoint in "DoSomeStuff"
- One thread stopped at that breakpoint

Now you continue (step or run) => the one thread must go over the breakpoint. For this the debugger needs to temporarily remove that breakpoint.

If the other thread reaches the place of that breakpoint, just in that tiny moment, then the other thread will pass the breakpoint without entering pause.

Example, if you have code like
Code: Pascal  [Select][+][-]
  1. for i := 0 to 9999999999 do begin
  2.   x := data[i];
  3.   if x = val then
  4.     CallFoo;  // breakpoint here
  5. end;

And you have several threads running that loop at the same time (each with its own "data"). Then one (or more) of the threads could enter "CallFoo" without stopping at the breakpoint.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on June 28, 2022, 02:50:41 pm
Ok I understand and it should be ok since most of the time I implement the functions or procedures inside the specific thread object I need.

Here are two more logs. I commented the .Suspend and .Resume "Exit;" and you should have the debugln info in the log.
I was not able to screenshot the thread windows at the moment it hangs because then all threads info disappear but you will find the previous screenshots when F9 was still working.

Thanks Martin for your great work with fpDebug. Not an easy piece of code. I hope it will help you put some light on this problem.
 
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 29, 2022, 02:38:24 pm
One question.

Can it be that your app (the project being debugged) is 32bit?

And if so, is the IDE you use 32 or 64 bit?


Also, if you are using a 64bit IDE to debug a 32bit app (which should work), can you ....
(NO log required)

1)
try this patch
Code: Diff  [Select][+][-]
  1. diff --git a/components/fpdebug/fpdbgwinclasses.pas b/components/fpdebug/fpdbgwinclasses.pas
  2. index 505a6bc9b3..891487cff3 100644
  3. --- a/components/fpdebug/fpdbgwinclasses.pas
  4. +++ b/components/fpdebug/fpdbgwinclasses.pas
  5. @@ -316,6 +316,7 @@ function dbgs(AnDbgEvent: DEBUG_EVENT): String; overload;
  6.    _IsWow64Process: function (hProcess:HANDLE; WoW64Process: PBOOL):BOOL; stdcall = nil;
  7.    _Wow64GetThreadContext: function (hThread: THandle; var   lpContext: WOW64_CONTEXT): BOOL; stdcall = nil;
  8.    _Wow64SetThreadContext: function (hThread: THandle; const lpContext: WOW64_CONTEXT): BOOL; stdcall = nil;
  9. +  _Wow64SuspendThread: function (hThread:HANDLE):DWORD; stdcall = nil;
  10.    _DebugBreakProcess: function(Process:HANDLE): WINBOOL; stdcall = nil;
  11.    _GetThreadDescription: function(hThread: THandle; ppszThreadDescription: PPWSTR): HResult; stdcall = nil;
  12.  
  13. @@ -340,6 +341,7 @@ procedure LoadKernelEntryPoints;
  14.    Pointer(_IsWow64Process) := GetProcAddress(hMod, 'IsWow64Process');
  15.    Pointer(_Wow64GetThreadContext) := GetProcAddress(hMod, 'Wow64GetThreadContext');
  16.    Pointer(_Wow64SetThreadContext) := GetProcAddress(hMod, 'Wow64SetThreadContext');
  17. +  Pointer(_Wow64SuspendThread) := GetProcAddress(hMod, 'Wow64SuspendThread');
  18.    {$endif}
  19.  
  20.    DebugLn(DBG_WARNINGS and (DebugBreakAddr = nil), ['WARNING: Failed to get DebugBreakAddr']);
  21. @@ -354,6 +356,7 @@ procedure LoadKernelEntryPoints;
  22.    DebugLn(DBG_WARNINGS and (_IsWow64Process = nil), ['WARNING: Failed to get IsWow64Process']);
  23.    DebugLn(DBG_WARNINGS and (_Wow64GetThreadContext = nil), ['WARNING: Failed to get Wow64GetThreadContext']);
  24.    DebugLn(DBG_WARNINGS and (_Wow64SetThreadContext = nil), ['WARNING: Failed to get Wow64SetThreadContext']);
  25. +  DebugLn(DBG_WARNINGS and (_Wow64SuspendThread = nil), ['WARNING: Failed to get _Wow64SuspendThread']);
  26.    {$endif}
  27.  end;
  28.  
  29. @@ -1641,6 +1644,11 @@ procedure TDbgWinThread.Suspend;
  30.  begin
  31.    if FIsSuspended then
  32.      exit;
  33. +  {$ifdef cpux86_64}
  34. +  if (Process.Mode = dm32) and (_Wow64SuspendThread <> nil) then
  35. +    r := _Wow64SuspendThread(Handle)
  36. +  else
  37. +  {$endif}
  38.    r := SuspendThread(Handle);
  39.    FIsSuspended := r <> DWORD(-1);
  40.    debugln(DBG_WARNINGS and (r = DWORD(-1)), 'Failed to suspend Thread %d (handle: %d). Error: %s', [Id, Handle, GetLastErrorText]);
  41.  
  42.  



2)
... see if the issue also exists if using a 32bit IDE?
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on June 30, 2022, 08:53:30 am
Sorry for my late answer but I had to do some more testing.

Quote
if you are using a 64bit IDE to debug a 32bit app

Yes, this is the case and I think you can see in the logs:

Info: (lazarus) Param[1]="-Twin32"
Info: (lazarus) Param[2]="-Pi386"

I applied your patch, removed the "Exit;" in .Suspend .Resume proc and it works. No more trouble with F9 or F8 so far. I also removed all the Win11 security settings specific to lazarus.exe and MyProject.exe without any problem.

Quote
... see if the issue also exists if using a 32bit IDE?

I didn't try to install the 32bit version yet as I didn't want to mess up with my current settings and reinstall all the packages I use. I will duplicate the VM and give it a try.

I still have some trouble while setting breakpoints while the debugged app is running (not always but quite often). I enclose the lazarus error message and the assembler window when this happen.
I noticed it especially happens when setting a breakpoint on the last "end;" of a procedure (I know not a useful breakpoint but it was for test purpose).

Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 30, 2022, 01:14:56 pm
I applied your patch, removed the "Exit;" in .Suspend .Resume proc and it works. No more trouble with F9 or F8 so far. I also removed all the Win11 security settings specific to lazarus.exe and MyProject.exe without any problem.

Quote
... see if the issue also exists if using a 32bit IDE?

I didn't try to install the 32bit version yet as I didn't want to mess up with my current settings and reinstall all the packages I use. I will duplicate the VM and give it a try.

That's ok, no need to test the 32 bit IDE, if the patch worked. This means it was a problem only in cross debugging.

According to msdn, the Wow64SuspendThread has been around a long time. Though I have never seen it on previous researches, when I did use the normal SuspendThread.
In either case, now (and I am sure that was not the case when I did this in FpDebug) the msdn docs for SuspendThread actually have a link to the Wow64SuspendThread.
Given that, and knowing that SuspendThread used to work for cross debugging, I guess that MS changed something in Win-11.



I committed the fix (about to merge it to fixes 2.2 too).

Many thanks for your help in tracking this.

Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on June 30, 2022, 02:24:33 pm
You are welcome. Thanks again Martin for your assistance, your hard work on fpDebug and all the other parts of Lazarus IDE.
Let me know if I can help testing the IDE crash that occurs while adding breakpoints at runtime.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on June 30, 2022, 06:31:34 pm
You are welcome. Thanks again Martin for your assistance, your hard work on fpDebug and all the other parts of Lazarus IDE.
Let me know if I can help testing the IDE crash that occurs while adding breakpoints at runtime.

You can always help ;)

If you get that crash => same logfile please.
(also, please rebuild your IDE with  -gw -gl  => then there is a chance that an exception includes a full trace in the log)

If you can reproduce it, steps to do so.


I missed the bit on your previous post...

Seems also 32 bit cross debug related..

I haven't see a WOW version of the DebugBreakProcess  though....
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 01, 2022, 11:42:19 am

Here is the log.

The problem occurs when adding or removing a breakpoint while debugging. It doesn't always happen but quite often (every 5 to 10 attempts).

Using same Cross compiling x64 to x32 as before.

Another issue I had is the breakpoints (sometime) not being activated even if the build mode is set to "debug". I think it is related to the project options not being correctly updated when you change from release to debug or the compiler not receiving the right settings. The settings were changed yesterday from "debug" to "release" from the toolbar button and changed today to "debug" from the same button. The solution I found is to uncheck "generate debug info" in project options, rebuild and then check it again then rebuild. Not a big deal but could help if someone is having the same trouble.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 01, 2022, 11:46:42 am
Here the project settings when breakpoints were not available
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 01, 2022, 12:01:57 pm
The problem occurs when adding or removing a breakpoint while debugging. It doesn't always happen but quite often (every 5 to 10 attempts).

While debugging => that means while your target up is running (as opposed to paused)?

If you change breakpoints while already paused (after ending a single step, or after hitting another breakpoint), then that problem does not happen?

(If you use the pause button, I would expect it to also happen)
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 01, 2022, 12:14:56 pm
The pop up "project raised exception illegal instruction" => is that shown by the app or by the IDE (debugger)?

That is, while that pop up is open, is the IDE "blocked" => because the IDE shows a modal dialog?

I could not find it in the log. But if it is shown by the app then that would explain it. Then that would mean that this error is not even reported to the debugger.

Also (even though it does not look like it): you do not have "access violation" or "illegal inst" in the list of "ignored exceptions"? (Project options "language exceptions"
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 01, 2022, 12:16:43 pm
Quote
While debugging => that means while your target up is running (as opposed to paused)?

It happens when the target is paused because a breakpoint was hit, then I remove or add one or more breakpoints.


Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 01, 2022, 01:39:53 pm
Sorry I was wrong in my previous post. The problem happens whether the target exe is paused, not paused or even if I disabled all breakpoints and start activate some breakpoints while the target app is not paused by debugger.

The message dialogs shows the target app name in the error but both the IDE and the target app are not responding while the error dialog is not closed. Once it is closed, any attempt to "continu" will bring back the error dialog.

But I can tell now that the problem is related to the treads created in the target app.
I deactivated all the threads (.created(True) -> suspended). Then no problem at all.
A soon as I activate the threads the problem is here again.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 01, 2022, 02:05:29 pm
Actually after a closer look, it seems that you were doing this while the app is running.
You were removing a breakpoint, and then setting it in the same location again.

Could you try and get an asm snapshot of the code...
That is, before you trigger the issue, have the debugger pause in the function where you will change the breakpoint, and get the asm for the lines. Starting at least one line before where you will set the breakpoint.

The try to trigger it on that line.




At the time when you are setting the breakpoint, the thread that then crashes is at
Thr.Id:22564 10002

I don't know that address 10002 => but I guess it means it is waiting in the OS (i.e., idle / waiting for more events).
Or maybe the kernel just hides the location....

The location where you set the breakpoint => is that code executed regularly?
Would you expect the breakpoint to be hit, (very) soon after setting it?


Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 01, 2022, 02:31:04 pm
Also please add one more patch
Code: Diff  [Select][+][-]
  1. diff --git a/components/fpdebug/fpdbgwinclasses.pas b/components/fpdebug/fpdbgwinclasses.pas
  2. index 2c2c58b72b..6701b07854 100644
  3. --- a/components/fpdebug/fpdbgwinclasses.pas
  4. +++ b/components/fpdebug/fpdbgwinclasses.pas
  5. @@ -995,6 +996,9 @@ function TDbgWinProcess.WaitForDebugEvent(out ProcessIdentifier, ThreadIdentifie
  6.    for TDbgThread(t) in FThreadMap do begin
  7.    if t.ReadThreadState then
  8.      DebugLn(FPDBG_WINDOWS, 'Thr.Id:%d %x  SSTep %s EF %s     DR6:%x  DR7:%x  WP:%x  RegAcc: %d,  SStep: %d  Task: %d, ExcBrk: %d', [t.ID, t.GetInstructionPointerRegisterValue, dbgs(t.FCurrentContext^.def.EFlags and FLAG_TRACE_BIT), dbghex(t.FCurrentContext^.def.EFlags), t.FCurrentContext^.def.Dr6, t.FCurrentContext^.def.Dr7, t.FCurrentContext^.def.Dr6 and 15, t.FCurrentContext^.def.Dr6 and (1<< 13), t.FCurrentContext^.def.Dr6 and (1<< 14), t.FCurrentContext^.def.Dr6 and (1<< 15), t.FCurrentContext^.def.Dr6 and (1<< 16)]);
  9. +  end
  10. +  else begin
  11. +    debugln(['failed threadstate for ', t.id ])
  12.    end;
  13.    {$ENDIF}
  14.  
  15.  
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 01, 2022, 08:47:30 pm
Good news....
The issue has just started to happen on my PC too. (At least an issue that looks really a lot alike).

Strange though.
It's one of my automated testcases for FpDebug. And the big question now is, how has it for years avoided to run into the error.

The change is triggered by an unrelated commit (even before the WOWSuspendThread).
Well by the look of it, it is some sort of race condition.  I still can't see how the change affects it. But that much is reproducible: going one commit forward or backward, and the error comes and goes.

Anyway, I tracked it down, and found the condition that leads to fpdebug getting it wrong.

I still need to figure how to fix it. (And if it then can be merged...)



For the error to happen at least 2 thread must run the code block in which you set the breakpoint. So that both can hit it, and hit it within a very narrow window of time (even if one of them won't report hitting it and rather crash)
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 01, 2022, 09:03:09 pm
Can't tell exactly if it is the same.

All your threads, almost all the time report
   IP=$00010002

If there really is code there, or if this is the OS idle location => then this would be entirely different.

Are you expecting your threads to be idle, or to be in some sort of "I am waiting" call?




EDIT: your log looks different :(

Also let me know on what you expect your threads to do.
Because if this 00010002 is some sort of placeholder => that will make it a lot harder.

By any chance, might that be in any external DLL?

If, so I have seen previously issues with 3rd party dll, that would react unkindly if the app in which they were loaded was showing signs of being debugged. Not saying this is the case, but must be considered.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 02, 2022, 10:29:15 am
I tried to reproduce the problem with a simple project, adding a simple thread: No problem...
I then added another thread + TEvent in the TThread.Execute (which I use quite often):  Again no problem...

I will try to narrow down the problem by disabling all specific DLL needed by the project causing errors and to disable the threads one by one to find what's going on.

Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 02, 2022, 02:09:47 pm
Just some background on what I am currently looking on. And why it does not match your case.


In my testcase I have 9 threads, that all go round and round and round on about 12 asm instructions.
So quite often more than one of them is at the same address, and at the same breakpoint.

If 2 threads hit a breakpoint, windows will inform the debugger twice, and each time it informs the debugger is stops all threads (at least it is supposed to, and does for me).
But in order to get the 2nd notification, the debugger must acknowledge the first. And at this time all threads will run again (well the debugger suspends some of them. But given enough threads and the right pattern some will run).

That also means that the following can happen.
(When there are more than 2 threads, then between the 2 breakpoint notifications, there can be other events. And the breakpoint may be removed in those other events. The below is massively simplified)

-- 2 threads hit breakpoint "A"
-> Notification for Thread-1 at "A"
    The debugger will handle the notification.
    But coincidently at this time, while handling the notification, the breakpoint "A" gets permanently removed.
<- Debugger acknowledges the notification, and app runs again (the other threads will be suspended, but that actually does not matter)
-> Notification for Thread-2 at "A"
    The breakpoint no longer exists, but the thread had hit it before.
    Here the debugger has a bug. => As it does not see a breakpoint, it does not handle it correctly.
<- Debugger acknowledges the notification, and app runs again     

And now the thread with the unhandled breakpoint will crash.



1) Why is it different from your case?
-> The breakpoint must get removed (well you did toggle it, so that includes removal)
-> A notification of type "breakpoint hit" must reach the debugger for that removed breakpoint
    That I have not seen in your log.
-> Also the thread would fail at the address of the breakpoint (actually + 1 byte).
    And your thread fails at this $100002 address. Which was never touched.
    That is, if that is indeed correctly reported....
    I can only guess it must be in a dll... Because your normal code resides at $4....... addresses. And the Kernel usually is at $7f.......



2) What exactly means "does not handle the breakpoint correctly"

Well here is how a breakpoint works (though of course WOW64 for arch/arm will (should) emulate that).

Say the testcase has an instruction like  (this is just dummy data)

Code: Text  [Select][+][-]
  1. Address     mem-dump       asm
  2. 0x410010  71 72 73 74     mov rax, eax

The debugger will replace the first byte with $CC (an int3 instruction, that is handled as breakpoint).

The test process will execute the int3, and its instruction pointer will then be at 0x410011. The byte after the breakpoint.
The OS catches the int3 and tells the debugger.

Now the debugger must do 2 things:
- tell the OS, to clear the signal. So the test app does not get a signal for the int3. (that seems still to happen in the buggy case)
- restore the original code, and restore the instruction pointer to 0x410010 => so the original instruction can be run
- (after that, put the breakpoint back in place)

But due to the bug, the debugger does not do the 2nd step.

That means the test app continues at 0x410011. And it takes the "72 ..." as it's next instruction. Only that was never an instruction, and that is why it crashes.

Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 04, 2022, 06:05:46 pm
Ok, I made some changes to git main. Fixing the issue I mentioned.

I have no idea if it affects your issue. You can test it https://gitlab.com/martin_frb/lazarus/-/commits/f-fpdebug-new-win-brkpoint

You will need to build a 2.3 Lazarus for this. The link is based on the current main branch.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 05, 2022, 03:44:03 pm
Thanks Martin,
I didn't have time yet to find the thread or dll problem. I will let you know ASAP.
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 06, 2022, 11:56:31 am
For information,

I got an error while building FPC 3.2.2 with FPCUPdeluxe in file gdbint.pp

Code: Pascal  [Select][+][-]
  1. Start compiling package gdbint for target x86_64-win64.
  2. File libgdb.a not found
  3.        Compiling gdbint\src\gdbint.pp
  4. The installer encountered the following error:
  5. External command "C:/fpcupdeluxe/fpcsrc/compiler/ppcx64.exe -Twin64 -FUgdbint\units\x86_64-win64\ -FuC:\fpcupdeluxe\fpcsrc\rtl\units\x86_64-win64\ -Figdbint\src -Fl -Ur -Xs -O2 -n -vw-n-h-l-d-u-t-p-c- -dFPC_SOFT_FPUX80 -dx86_64 -dRELEASE -Sc -viq gdbint\src\gdbint.pp" failed with exit code 1. Console output:
  6. Target OS: Win64 for x64
  7. Compiling gdbint\src\gdbint.pp
  8. User defined: using gdb 6.6.x
  9. gdbint.pp(3726) Error: Invalid DLL C:\WINDOWS\system32\kernel32.dll, invalid header size
  10. gdbint.pp(3726) Error: Invalid DLL C:\WINDOWS\system32\user32.dll, invalid header size
  11. Fatal: There were 2 errors compiling module, stopping
  12. Fatal: Compilation aborted
  13. make.exe[1]: *** [packages_all] Error 2
  14. C:\fpcupdeluxe\fpcbootstrap\make.exe: *** [build-stamp.x86_64-win64] Error 2
  15.  
  16. fpcupdeluxe: ERROR: FPCNativeInstaller (BuildModuleCustom: FPC): Error running C:\fpcupdeluxe\fpcbootstrap\make.exe for FPC failed with exit code 2
  17. . Details:
  18.  
  19.  
  20. ERROR: Fpcupdeluxe fatal error !
  21. Sequencer (FPC): Failure running fpcupdeluxe: error executing sequence FPC
  22. Sequencer (Default): Failure running fpcupdeluxe: error executing sequence Default
  23.  


I had to comment out the 2 lines in gdbint.pp (701 and 702) and check "FCP/Laz Build Only in "Setup +", and build only FPC.

Then the Lazarus Update went well.

Is there a way to copy/apply the settings from one Lazarus install to another? I mean having all user settings and needed packages installation done quickly?


Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 06, 2022, 02:50:59 pm
Is there a way to copy/apply the settings from one Lazarus install to another? I mean having all user settings and needed packages installation done quickly?


Not 100%

Your settings are in a collection of xml files in the primary config path (menu: view > ide internals > about ide => then search for it)
Normally  C:\Users\USERNAME\AppData\Local\lazarus

editoroptions.xml can afaik be copiend. (has no paths)
environmentoptions.xml contains paths. And those must be adapted.

Best to check all file for paths.
If it is a regular think, make sure your paths all are using macros like ($lazarusdir) / not sure if that is correct....

Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 06, 2022, 02:52:31 pm
I got an error while building FPC 3.2.2 with FPCUPdeluxe in file gdbint.pp
Maybe report this in the fpcupdeluxe thread ?
https://forum.lazarus.freepascal.org/index.php/topic,34645.0.html
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 07, 2022, 02:20:31 pm
Quote
Maybe report this in the fpcupdeluxe thread ?
Done

Finally, I was able to reproduce the problem with a simple project (Lazarus 2.3.0 and FPC 3.2.2 + Cross Compile winx64->winx32 ). It is related to TThread and TEvent.
The reported address is still the same on my computer:

Here is the sample project and log.

To reproduce the problem you must have at least 2 "TThreadEvent" running:


You should have the error message box (repeating when you click the IDE run green arrow).


Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 12, 2022, 08:12:47 pm
Hi,

Am I the only one having this problem?
Title: Re: Debugging problem on Windows 11 ARM
Post by: Martin_fr on July 12, 2022, 09:20:52 pm
I haven't found time yet to look at your example, and see if I could reproduce....

It may be a while...
Title: Re: Debugging problem on Windows 11 ARM
Post by: Luc on July 12, 2022, 10:12:39 pm
No problem :)
TinyPortal © 2005-2018