Recent

Author Topic: Debugging problem on Windows 11 ARM  (Read 6085 times)

Luc

  • Jr. Member
  • **
  • Posts: 59
Re: Debugging problem on Windows 11 ARM
« Reply #15 on: June 28, 2022, 02:50:41 pm »
Ok I understand and it should be ok since most of the time I implement the functions or procedures inside the specific thread object I need.

Here are two more logs. I commented the .Suspend and .Resume "Exit;" and you should have the debugln info in the log.
I was not able to screenshot the thread windows at the moment it hangs because then all threads info disappear but you will find the previous screenshots when F9 was still working.

Thanks Martin for your great work with fpDebug. Not an easy piece of code. I hope it will help you put some light on this problem.
 
Win 11 ARM (Running in Parallel VM - Mac OSX) - Lazarus 2.2.6 - i386-win32/win64

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9793
  • Debugger - SynEdit - and more
    • wiki
Re: Debugging problem on Windows 11 ARM
« Reply #16 on: June 29, 2022, 02:38:24 pm »
One question.

Can it be that your app (the project being debugged) is 32bit?

And if so, is the IDE you use 32 or 64 bit?


Also, if you are using a 64bit IDE to debug a 32bit app (which should work), can you ....
(NO log required)

1)
try this patch
Code: Diff  [Select][+][-]
  1. diff --git a/components/fpdebug/fpdbgwinclasses.pas b/components/fpdebug/fpdbgwinclasses.pas
  2. index 505a6bc9b3..891487cff3 100644
  3. --- a/components/fpdebug/fpdbgwinclasses.pas
  4. +++ b/components/fpdebug/fpdbgwinclasses.pas
  5. @@ -316,6 +316,7 @@ function dbgs(AnDbgEvent: DEBUG_EVENT): String; overload;
  6.    _IsWow64Process: function (hProcess:HANDLE; WoW64Process: PBOOL):BOOL; stdcall = nil;
  7.    _Wow64GetThreadContext: function (hThread: THandle; var   lpContext: WOW64_CONTEXT): BOOL; stdcall = nil;
  8.    _Wow64SetThreadContext: function (hThread: THandle; const lpContext: WOW64_CONTEXT): BOOL; stdcall = nil;
  9. +  _Wow64SuspendThread: function (hThread:HANDLE):DWORD; stdcall = nil;
  10.    _DebugBreakProcess: function(Process:HANDLE): WINBOOL; stdcall = nil;
  11.    _GetThreadDescription: function(hThread: THandle; ppszThreadDescription: PPWSTR): HResult; stdcall = nil;
  12.  
  13. @@ -340,6 +341,7 @@ procedure LoadKernelEntryPoints;
  14.    Pointer(_IsWow64Process) := GetProcAddress(hMod, 'IsWow64Process');
  15.    Pointer(_Wow64GetThreadContext) := GetProcAddress(hMod, 'Wow64GetThreadContext');
  16.    Pointer(_Wow64SetThreadContext) := GetProcAddress(hMod, 'Wow64SetThreadContext');
  17. +  Pointer(_Wow64SuspendThread) := GetProcAddress(hMod, 'Wow64SuspendThread');
  18.    {$endif}
  19.  
  20.    DebugLn(DBG_WARNINGS and (DebugBreakAddr = nil), ['WARNING: Failed to get DebugBreakAddr']);
  21. @@ -354,6 +356,7 @@ procedure LoadKernelEntryPoints;
  22.    DebugLn(DBG_WARNINGS and (_IsWow64Process = nil), ['WARNING: Failed to get IsWow64Process']);
  23.    DebugLn(DBG_WARNINGS and (_Wow64GetThreadContext = nil), ['WARNING: Failed to get Wow64GetThreadContext']);
  24.    DebugLn(DBG_WARNINGS and (_Wow64SetThreadContext = nil), ['WARNING: Failed to get Wow64SetThreadContext']);
  25. +  DebugLn(DBG_WARNINGS and (_Wow64SuspendThread = nil), ['WARNING: Failed to get _Wow64SuspendThread']);
  26.    {$endif}
  27.  end;
  28.  
  29. @@ -1641,6 +1644,11 @@ procedure TDbgWinThread.Suspend;
  30.  begin
  31.    if FIsSuspended then
  32.      exit;
  33. +  {$ifdef cpux86_64}
  34. +  if (Process.Mode = dm32) and (_Wow64SuspendThread <> nil) then
  35. +    r := _Wow64SuspendThread(Handle)
  36. +  else
  37. +  {$endif}
  38.    r := SuspendThread(Handle);
  39.    FIsSuspended := r <> DWORD(-1);
  40.    debugln(DBG_WARNINGS and (r = DWORD(-1)), 'Failed to suspend Thread %d (handle: %d). Error: %s', [Id, Handle, GetLastErrorText]);
  41.  
  42.  



2)
... see if the issue also exists if using a 32bit IDE?
« Last Edit: June 29, 2022, 02:43:26 pm by Martin_fr »

Luc

  • Jr. Member
  • **
  • Posts: 59
Re: Debugging problem on Windows 11 ARM
« Reply #17 on: June 30, 2022, 08:53:30 am »
Sorry for my late answer but I had to do some more testing.

Quote
if you are using a 64bit IDE to debug a 32bit app

Yes, this is the case and I think you can see in the logs:

Info: (lazarus) Param[1]="-Twin32"
Info: (lazarus) Param[2]="-Pi386"

I applied your patch, removed the "Exit;" in .Suspend .Resume proc and it works. No more trouble with F9 or F8 so far. I also removed all the Win11 security settings specific to lazarus.exe and MyProject.exe without any problem.

Quote
... see if the issue also exists if using a 32bit IDE?

I didn't try to install the 32bit version yet as I didn't want to mess up with my current settings and reinstall all the packages I use. I will duplicate the VM and give it a try.

I still have some trouble while setting breakpoints while the debugged app is running (not always but quite often). I enclose the lazarus error message and the assembler window when this happen.
I noticed it especially happens when setting a breakpoint on the last "end;" of a procedure (I know not a useful breakpoint but it was for test purpose).

Win 11 ARM (Running in Parallel VM - Mac OSX) - Lazarus 2.2.6 - i386-win32/win64

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9793
  • Debugger - SynEdit - and more
    • wiki
Re: Debugging problem on Windows 11 ARM
« Reply #18 on: June 30, 2022, 01:14:56 pm »
I applied your patch, removed the "Exit;" in .Suspend .Resume proc and it works. No more trouble with F9 or F8 so far. I also removed all the Win11 security settings specific to lazarus.exe and MyProject.exe without any problem.

Quote
... see if the issue also exists if using a 32bit IDE?

I didn't try to install the 32bit version yet as I didn't want to mess up with my current settings and reinstall all the packages I use. I will duplicate the VM and give it a try.

That's ok, no need to test the 32 bit IDE, if the patch worked. This means it was a problem only in cross debugging.

According to msdn, the Wow64SuspendThread has been around a long time. Though I have never seen it on previous researches, when I did use the normal SuspendThread.
In either case, now (and I am sure that was not the case when I did this in FpDebug) the msdn docs for SuspendThread actually have a link to the Wow64SuspendThread.
Given that, and knowing that SuspendThread used to work for cross debugging, I guess that MS changed something in Win-11.



I committed the fix (about to merge it to fixes 2.2 too).

Many thanks for your help in tracking this.


Luc

  • Jr. Member
  • **
  • Posts: 59
Re: Debugging problem on Windows 11 ARM
« Reply #19 on: June 30, 2022, 02:24:33 pm »
You are welcome. Thanks again Martin for your assistance, your hard work on fpDebug and all the other parts of Lazarus IDE.
Let me know if I can help testing the IDE crash that occurs while adding breakpoints at runtime.
Win 11 ARM (Running in Parallel VM - Mac OSX) - Lazarus 2.2.6 - i386-win32/win64

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9793
  • Debugger - SynEdit - and more
    • wiki
Re: Debugging problem on Windows 11 ARM
« Reply #20 on: June 30, 2022, 06:31:34 pm »
You are welcome. Thanks again Martin for your assistance, your hard work on fpDebug and all the other parts of Lazarus IDE.
Let me know if I can help testing the IDE crash that occurs while adding breakpoints at runtime.

You can always help ;)

If you get that crash => same logfile please.
(also, please rebuild your IDE with  -gw -gl  => then there is a chance that an exception includes a full trace in the log)

If you can reproduce it, steps to do so.


I missed the bit on your previous post...

Seems also 32 bit cross debug related..

I haven't see a WOW version of the DebugBreakProcess  though....
« Last Edit: June 30, 2022, 06:54:09 pm by Martin_fr »

Luc

  • Jr. Member
  • **
  • Posts: 59
Re: Debugging problem on Windows 11 ARM
« Reply #21 on: July 01, 2022, 11:42:19 am »

Here is the log.

The problem occurs when adding or removing a breakpoint while debugging. It doesn't always happen but quite often (every 5 to 10 attempts).

Using same Cross compiling x64 to x32 as before.

Another issue I had is the breakpoints (sometime) not being activated even if the build mode is set to "debug". I think it is related to the project options not being correctly updated when you change from release to debug or the compiler not receiving the right settings. The settings were changed yesterday from "debug" to "release" from the toolbar button and changed today to "debug" from the same button. The solution I found is to uncheck "generate debug info" in project options, rebuild and then check it again then rebuild. Not a big deal but could help if someone is having the same trouble.
Win 11 ARM (Running in Parallel VM - Mac OSX) - Lazarus 2.2.6 - i386-win32/win64

Luc

  • Jr. Member
  • **
  • Posts: 59
Re: Debugging problem on Windows 11 ARM
« Reply #22 on: July 01, 2022, 11:46:42 am »
Here the project settings when breakpoints were not available
Win 11 ARM (Running in Parallel VM - Mac OSX) - Lazarus 2.2.6 - i386-win32/win64

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9793
  • Debugger - SynEdit - and more
    • wiki
Re: Debugging problem on Windows 11 ARM
« Reply #23 on: July 01, 2022, 12:01:57 pm »
The problem occurs when adding or removing a breakpoint while debugging. It doesn't always happen but quite often (every 5 to 10 attempts).

While debugging => that means while your target up is running (as opposed to paused)?

If you change breakpoints while already paused (after ending a single step, or after hitting another breakpoint), then that problem does not happen?

(If you use the pause button, I would expect it to also happen)

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9793
  • Debugger - SynEdit - and more
    • wiki
Re: Debugging problem on Windows 11 ARM
« Reply #24 on: July 01, 2022, 12:14:56 pm »
The pop up "project raised exception illegal instruction" => is that shown by the app or by the IDE (debugger)?

That is, while that pop up is open, is the IDE "blocked" => because the IDE shows a modal dialog?

I could not find it in the log. But if it is shown by the app then that would explain it. Then that would mean that this error is not even reported to the debugger.

Also (even though it does not look like it): you do not have "access violation" or "illegal inst" in the list of "ignored exceptions"? (Project options "language exceptions"

Luc

  • Jr. Member
  • **
  • Posts: 59
Re: Debugging problem on Windows 11 ARM
« Reply #25 on: July 01, 2022, 12:16:43 pm »
Quote
While debugging => that means while your target up is running (as opposed to paused)?

It happens when the target is paused because a breakpoint was hit, then I remove or add one or more breakpoints.


Win 11 ARM (Running in Parallel VM - Mac OSX) - Lazarus 2.2.6 - i386-win32/win64

Luc

  • Jr. Member
  • **
  • Posts: 59
Re: Debugging problem on Windows 11 ARM
« Reply #26 on: July 01, 2022, 01:39:53 pm »
Sorry I was wrong in my previous post. The problem happens whether the target exe is paused, not paused or even if I disabled all breakpoints and start activate some breakpoints while the target app is not paused by debugger.

The message dialogs shows the target app name in the error but both the IDE and the target app are not responding while the error dialog is not closed. Once it is closed, any attempt to "continu" will bring back the error dialog.

But I can tell now that the problem is related to the treads created in the target app.
I deactivated all the threads (.created(True) -> suspended). Then no problem at all.
A soon as I activate the threads the problem is here again.
Win 11 ARM (Running in Parallel VM - Mac OSX) - Lazarus 2.2.6 - i386-win32/win64

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9793
  • Debugger - SynEdit - and more
    • wiki
Re: Debugging problem on Windows 11 ARM
« Reply #27 on: July 01, 2022, 02:05:29 pm »
Actually after a closer look, it seems that you were doing this while the app is running.
You were removing a breakpoint, and then setting it in the same location again.

Could you try and get an asm snapshot of the code...
That is, before you trigger the issue, have the debugger pause in the function where you will change the breakpoint, and get the asm for the lines. Starting at least one line before where you will set the breakpoint.

The try to trigger it on that line.




At the time when you are setting the breakpoint, the thread that then crashes is at
Thr.Id:22564 10002

I don't know that address 10002 => but I guess it means it is waiting in the OS (i.e., idle / waiting for more events).
Or maybe the kernel just hides the location....

The location where you set the breakpoint => is that code executed regularly?
Would you expect the breakpoint to be hit, (very) soon after setting it?



Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9793
  • Debugger - SynEdit - and more
    • wiki
Re: Debugging problem on Windows 11 ARM
« Reply #28 on: July 01, 2022, 02:31:04 pm »
Also please add one more patch
Code: Diff  [Select][+][-]
  1. diff --git a/components/fpdebug/fpdbgwinclasses.pas b/components/fpdebug/fpdbgwinclasses.pas
  2. index 2c2c58b72b..6701b07854 100644
  3. --- a/components/fpdebug/fpdbgwinclasses.pas
  4. +++ b/components/fpdebug/fpdbgwinclasses.pas
  5. @@ -995,6 +996,9 @@ function TDbgWinProcess.WaitForDebugEvent(out ProcessIdentifier, ThreadIdentifie
  6.    for TDbgThread(t) in FThreadMap do begin
  7.    if t.ReadThreadState then
  8.      DebugLn(FPDBG_WINDOWS, 'Thr.Id:%d %x  SSTep %s EF %s     DR6:%x  DR7:%x  WP:%x  RegAcc: %d,  SStep: %d  Task: %d, ExcBrk: %d', [t.ID, t.GetInstructionPointerRegisterValue, dbgs(t.FCurrentContext^.def.EFlags and FLAG_TRACE_BIT), dbghex(t.FCurrentContext^.def.EFlags), t.FCurrentContext^.def.Dr6, t.FCurrentContext^.def.Dr7, t.FCurrentContext^.def.Dr6 and 15, t.FCurrentContext^.def.Dr6 and (1<< 13), t.FCurrentContext^.def.Dr6 and (1<< 14), t.FCurrentContext^.def.Dr6 and (1<< 15), t.FCurrentContext^.def.Dr6 and (1<< 16)]);
  9. +  end
  10. +  else begin
  11. +    debugln(['failed threadstate for ', t.id ])
  12.    end;
  13.    {$ENDIF}
  14.  
  15.  

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9793
  • Debugger - SynEdit - and more
    • wiki
Re: Debugging problem on Windows 11 ARM
« Reply #29 on: July 01, 2022, 08:47:30 pm »
Good news....
The issue has just started to happen on my PC too. (At least an issue that looks really a lot alike).

Strange though.
It's one of my automated testcases for FpDebug. And the big question now is, how has it for years avoided to run into the error.

The change is triggered by an unrelated commit (even before the WOWSuspendThread).
Well by the look of it, it is some sort of race condition.  I still can't see how the change affects it. But that much is reproducible: going one commit forward or backward, and the error comes and goes.

Anyway, I tracked it down, and found the condition that leads to fpdebug getting it wrong.

I still need to figure how to fix it. (And if it then can be merged...)



For the error to happen at least 2 thread must run the code block in which you set the breakpoint. So that both can hit it, and hit it within a very narrow window of time (even if one of them won't report hitting it and rather crash)

 

TinyPortal © 2005-2018