Recent

Author Topic: Debugger regression in Lazarus 4.0  (Read 7820 times)

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12398
  • Debugger - SynEdit - and more
    • wiki
Re: Debugger regression in Lazarus 4.0
« Reply #15 on: July 17, 2025, 01:35:24 pm »
This is a long shot...

But you could run your lazarus with (may need startlazarus / or you need to find your "rebuild" lazarus in ~/.lazarus/ )
Code: Text  [Select][+][-]
  1.     lazarus   --debug-enable=DBG_VERBOSE,DBG_WARNINGS,DBG_ERRORS,DBG_STATE,DBG_EVENTS,DBG_FPDEBUG_VERBOSE,FPDBG_LINUX,FPDBG_COMMANDS,
  2. FPDBG_THREADS,FPDBG_QUEUE,DBG_DATA_MONITORS,DBG_THREAD_AND_FRAME,DBG_BREAKPOINTS,FPDBG_BREAKPOINT_ERRORS,DBG_VERBOSE_BRKPOINT,FPDBG_COMMANDS,FPDBG_QUEUE

That needs to be from a console, and output would go to the console.

Output can be redirected to a logfile, but if lazarus crashes hard, then if lazarus itself does the redirect, it likely will not be able to write everything when it crashes. Output redirection " > /txtfile.txt" might work better.

This simply would give some feedback on the events the debugger was processing when the cash happens. Just fishing for clues.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12398
  • Debugger - SynEdit - and more
    • wiki
Re: Debugger regression in Lazarus 4.0
« Reply #16 on: July 18, 2025, 02:59:48 pm »
I did a few tests. Compiled with different fpc, loaded your lib (not running anything in it, but making sure it can be loaded into the debugger). Stressed and reviewed the code around the trace...
Got nothing sofar.



Do you have any packages installed (3rd party? Doesn't have to be debugger related)?

TheMouseAUS

  • Full Member
  • ***
  • Posts: 108
Re: Debugger regression in Lazarus 4.0
« Reply #17 on: July 18, 2025, 08:30:21 pm »
I think its going to be a hard one to find. Even the developer of the library found it hard to replicate, and now I cant either (but their definitely is an issue in the library libplctag version 2.66). I have a second less powerful machine, I will try on that one ass see what I get.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12398
  • Debugger - SynEdit - and more
    • wiki
Re: Debugger regression in Lazarus 4.0
« Reply #18 on: July 19, 2025, 09:29:04 am »
As it happens... I just run into a dangling pointer. Though 99.9% likely that it is not related to your issue.

I haven't checked if it is present in the 3.n releases, but it might be...

First of all, in order for it to happen at all, you must have any of the following windows open (and have a value specified in them): Debug-Inspect, Evaluate-Modify, or Mem-viewer.

In my case (when I got the error) it manifested with a stacktrace directly in one of those windows (actually the watches window, but that is related).
However, its a dangling pointer, it could manifest anywhere. It could have damaged the data that was involved in your crash. But, when your target stopped for the access violation, it wouldn't be called. It would have had to have been called earlier (possible), but then your crash would happen even if you didn't get the the access violation. Hence, it is really unlikely to be your case.

On top of that, I couldn't get it with FpDebug (but it may still have been possible). I only was able to get it using the gdb backend (that is why it wasn't found earlier).



So all in all, one bug down. But not the one you have...

TheMouseAUS

  • Full Member
  • ***
  • Posts: 108
Re: Debugger regression in Lazarus 4.0
« Reply #19 on: July 21, 2025, 01:18:23 am »
I can no longer replicate this now. It was crashing every time the external lib had a GPF but now it falls back to the IDE like it should. I only managed to capture it that one time. I tried your long shot suggestion also. I am getting several IDE crashes when I right click on items etc(once again did not suffer from this in v3.6), but thats related to QT6, is it possible that was connected to this issue somehow?

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12398
  • Debugger - SynEdit - and more
    • wiki
Re: Debugger regression in Lazarus 4.0
« Reply #20 on: July 21, 2025, 09:21:21 am »
but thats related to QT6, is it possible that was connected to this issue somehow?

I would say extremely unlikely.

Starting off with the only bit we have
Quote
Code: Text  [Select][+][-]
  1. #0  0x0000000000562704 in AVL_TREE$_$TAVLTREE_$__$$_ROTATERIGHT$TAVLTREENODE ()
  2. #1  0x0000000000561b04 in AVL_TREE$_$TAVLTREE_$__$$_BALANCEAFTERINSERT$TAVLTREENODE ()

This is very well matured code. The chances that this code itself contains a crashing bug... Extremely unlikely, to the point that I say its more likely you win the lottery jackpot twice in a row.... (Let me know if you do ;) )

Then that leaves 2 possibilities that I can think off.

1) Concurrent unprotected access from 2 threads (race condition)
2) Prior damage from a dangling pointer that overwrote those structures with its own data


(1) Can be further limited to "concurrent write access". Since this thread, wrote, and a read access by another thread would have only crashed the reading thread. (Well technically, it could have turned the reading thread into a dangling pointer, but then it would be (2)).
And, from my review, I am pretty certain that this does not happen. Well under normal operations, if the debugger had entered some invalid state already then that may be different, but then the trace is meaningless, and we don't know anything about how it entered the invalid state in first)

(2) A dangling pointer. That is a wide field... Technically that pointer could be in any code, even non debugger code.
But then, It always only struck, when the target hard crashed. If it was in other code, then it would most likely have done the damage to that structure anyway (independent of what the debugged process was doing). And at the latest when the debug session ended, the damage to the structure would always have crashed, even if the access violation never happened.
So then, the dangling pointer likely either
- happened in reaction to the debugger getting the signal that the target crashed.
- happened on some previous debugger event (pause at breakpoint, etc) that only happened when the crash was next (i.e. you always hit that breakpoint or did a single step) before the crash, and when the crash would be next something would be different, e.g. a watch would have a value that let to the dangling pointer...



Of course, calling a menu, that ended in an exception, could have created a dangling pointer...
But the likelihood that this always only happened when the target would crash next...?

Well, there is one possibility, that we haven't explored yet. (But its unlikely too)

You wrote:
Even the developer of the library found it hard to replicate
Well, he didn't replicate the Lazarus crash, he replicated the crash in the debugged library?

Actually, did he yet?

Because, what if the FpDebugger had failed before the crash? Then the debugger could have injected e.g. a breakpoint at the wrong location, and that could have caused the debug target to crash (so the targed would not have a bug, but the bug was injected by the debugger).
Then of course any event anywhere could have tripped the debugger, which would then have tripped the target.

But that would mean that when you get the target library crashing now, it would still be because of that. Only with different damage in the debugger, as it doesn't go down anymore.

TheMouseAUS

  • Full Member
  • ***
  • Posts: 108
Re: Debugger regression in Lazarus 4.0
« Reply #21 on: July 21, 2025, 06:54:28 pm »
Quote

Actually, did he yet?

Because, what if the FpDebugger had failed before the crash? Then the debugger could have injected e.g. a breakpoint at the wrong location, and that could have caused the debug target to crash (so the targed would not have a bug, but the bug was injected by the debugger).
Then of course any event anywhere could have tripped the debugger, which would then have tripped the target.

But that would mean that when you get the target library crashing now, it would still be because of that. Only with different damage in the debugger, as it doesn't go down anymore.


Yes the developer did manage to recreate the race condition in the external library that caused all this in the first place but he had to create a test specifically for it. He has since released a fix.

I dont know what I can do to help find this now as this issue wont break for me. The QT crashes though are a nightmare but thats a different story.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12398
  • Debugger - SynEdit - and more
    • wiki
Re: Debugger regression in Lazarus 4.0
« Reply #22 on: July 21, 2025, 07:31:33 pm »
The QT stuff you best report on the bug tracker.

This issue has to wait till it resurfaces.... Or maybe I stumble on something when I do future work around this code...

zeljko

  • Hero Member
  • *****
  • Posts: 1950
    • http://wiki.lazarus.freepascal.org/User:Zeljan
Re: Debugger regression in Lazarus 4.0
« Reply #23 on: July 22, 2025, 10:02:05 am »
Yes I am using FPC 3.2.2. I did check but valgrind is not available on the repo of the distro I use.

I haven't checked yet what it would take to compile/install it. I do want to do it at some point as I wanted to run my app through to make sure I was handling everything correctly anyway (threads, memory etc).  Heaptrc is showing its pretty good so havent got round to it yet.

I will look into FPC upgrade. I do find it unusual I dont see these bugs on Lazarus 3.6. I have found 4.0 to be a lot more 'crashy', even today just right clicking on a variable i wanted to create a 'watch' for crashed the whole IDE out. Even closing Lazarus I managed to catch this :-

Thread 1 "lazarus" received signal SIGSEGV, Segmentation fault.
0x00007ffff6142f73 in QObject::disconnect(QObject const*, char const*, QObject const*, char const*) () from /usr/lib64/libQt6Core.so.6
(gdb) bt
#0  0x00007ffff6142f73 in QObject::disconnect(QObject const*, char const*, QObject const*, char const*) () from /usr/lib64/libQt6Core.so.6
#1  0x00007ffff7e3658a in QObject_hook::~QObject_hook() () from /usr/lib/libQt6Pas.so.6
#2  0x00000000007831ca in DETACHEVENTS (this=0x7fffc471cc60) at qt6/qtwidgets.pas:16530
#3  0x0000000000763a93 in DEINITIALIZEWIDGET (this=0x7fffc471cc60) at qt6/qtwidgets.pas:2279
#4  0x0000000000763cbb in DESTROY (this=0x7fffc471cc60, vmt=0x0) at qt6/qtwidgets.pas:2342
#5  0x000000000078304f in DESTROY (this=0x7fffc471cc60, vmt=0x1) at qt6/qtwidgets.pas:16504
#6  0x00000000004371eb in SYSTEM$_$TOBJECT_$__$$_FREE ()
#7  0x00007fffffffd7c0 in ?? ()
#8  0x0000000000752831 in RELEASE (this=0x7fffc471cc60) at qt6/qtobjects.pas:1126
#9  0x0000000000763deb in RELEASE (this=0x7fffc471cc60) at qt6/qtwidgets.pas:2382
#10 0x00000000007f68c0 in DESTROYHANDLE (self=0x7fffc5029180, AMENUITEM=0x7fffa8e9b020) at qt6/qtwsmenus.pp:257
#11 0x00000000006d2a26 in DESTROYHANDLE (this=0x7fffa8e9b020) at include/menuitem.inc:905
#12 0x00000000006d29a6 in DESTROYHANDLE (this=0x7fffa8e9ac00) at include/menuitem.inc:900
#13 0x00000000006d29a6 in DESTROYHANDLE (this=0x7fffb9341b80) at include/menuitem.inc:900
#14 0x00000000006d29a6 in DESTROYHANDLE (this=0x7fffc42f3040) at include/menuitem.inc:900
#15 0x00000000006d015b in DESTROYHANDLE (this=0x7fffc50341c0) at include/menu.inc:173
#16 0x00000000006d505b in DESTROY (this=0x7fffc50341c0, vmt=0x1) at include/popupmenu.inc:56
#17 0x00000000005157a6 in CLASSES$_$TCOMPONENT_$__$$_DESTROYCOMPONENTS ()
#18 0x0000000000000001 in ?? ()
#19 0x00007fffe3f87410 in ?? ()
#20 0x00000000ffffffff in ?? ()
#21 0x000000000051570e in CLASSES$_$TCOMPONENT_$__$$_DESTROY ()
#22 0x0000000000000001 in ?? ()
#23 0x00007ffff7ffd000 in ?? () from /lib64/ld-linux-x86-64.so.2
#24 0x00007fffffffe220 in ?? ()
#25 0x0000000000000001 in ?? ()
#26 0x00007fffffffdca8 in ?? ()
#27 0x00000000006cc3fd in DESTROY (this=0x7fffe3f87410, vmt=0x0) at lclclasses.pp:157
#28 0x000000000068a9b1 in DESTROY (this=0x7fffe3f87410, vmt=0x0) at include/control.inc:5227
#29 0x0000000000678b0f in DESTROY (this=0x7fffe3f87410, vmt=0x0) at include/wincontrol.inc:6687
#30 0x000000000068d225 in DESTROY (this=0x7fffe3f87410, vmt=0x0) at include/customcontrol.inc:40
#31 0x0000000000484a25 in DESTROY (this=0x7fffe3f87410, vmt=0x0) at include/scrollingwincontrol.inc:360
#32 0x0000000000486139 in DESTROY (this=0x7fffe3f87410, vmt=0x0) at include/customform.inc:138
#33 0x0000000000c5bc25 in DESTROY (this=0x7fffe3f87410, vmt=0x1) at sourceeditor.pp:7032
#34 0x00000000004371eb in SYSTEM$_$TOBJECT_$__$$_FREE ()
#35 0x00007fffe029d390 in ?? ()
#36 0x0000000000c68631 in FREESOURCEWINDOWS (this=0x7fffd405dc10) at sourceeditor.pp:10063
#37 0x0000000000c69eff in DESTROY (this=0x7fffd405dc10, vmt=0x0) at sourceeditor.pp:10572
#38 0x0000000000c6f682 in DESTROY (this=0x7fffd405dc10, vmt=0x1) at sourceeditor.pp:12007
#39 0x00000000004371eb in SYSTEM$_$TOBJECT_$__$$_FREE ()
#40 0x00007fffffffe000 in ?? ()
#41 0x000000000049e462 in FREETHENNIL (OBJ=0) at lazutilities.pas:76
#42 0x00000000004ae694 in DESTROY (this=0x7fffe029d390, vmt=0x1) at main.pp:1764
#43 0x00000000004371eb in SYSTEM$_$TOBJECT_$__$$_FREE ()
#44 0x00007fffffffe0f0 in ?? ()
#45 0x0000000000425891 in main () at lazarus.pp:169

This is fixed in trunk and in lazarus 4.2 afaik.

 

TinyPortal © 2005-2018