Recent

Author Topic: Random faults  (Read 3693 times)

big_M

  • Full Member
  • ***
  • Posts: 104
Random faults
« on: January 15, 2024, 10:15:34 pm »
Okay guys, I need help here. This is going to drive me crazy...

Since at some point in october my program just randomly throws faults of any kind, and I have absolutely no idea how I can debug this and find out what the problem is.

Following is the behaviour: I code a bit, compile, and run the program. Now it may run just fine, or it fails at pretty much ANY random point!... Be it at the creation of the forms somewhere in the depths of the lcl, somewhere on the startup when loading some files from the drive, or even when using the program normally.

First I thought it was due to a package I installed, because it was around that time that those faults showed up, but that seems to be unrelated.

* I can add or remove completely unrelated lines of code to make the faults appear or disappear!
* The program runs fine outside of Lazarus or without debugger
* Gdb or Fpdebug makes no difference
* Clean and compile sometimes helps, but most often not
* (Linux, qt5, various trunc builds, fpc 3.2.2, multiple pc's...)

What do I do with this?? I'm simply not of a pro enough to tackle this, and it's really starting to annoy me.

440bx

  • Hero Member
  • *****
  • Posts: 6123
Re: Random faults
« Reply #1 on: January 15, 2024, 10:22:58 pm »
* The program runs fine outside of Lazarus or without debugger
Does this mean that you only experience problems when the program is started from the IDE ?  IOW, when you start the program from the command line, it executes perfectly without any problem, is this the case ?
FPC v3.2.2 and Lazarus v4.0rc3 on Windows 7 SP1 64bit.

big_M

  • Full Member
  • ***
  • Posts: 104
Re: Random faults
« Reply #2 on: January 15, 2024, 10:29:17 pm »
Does this mean that you only experience problems when the program is started from the IDE ?  IOW, when you start the program from the command line, it executes perfectly without any problem, is this the case ?
Yes, exactly

alpine

  • Hero Member
  • *****
  • Posts: 1412
Re: Random faults
« Reply #3 on: January 15, 2024, 10:41:30 pm »
What if started in IDE but without debugging, Shift-Ctrl-F9 ?
"I'm sorry Dave, I'm afraid I can't do that."
—HAL 9000

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12173
  • Debugger - SynEdit - and more
    • wiki
Re: Random faults
« Reply #4 on: January 15, 2024, 10:42:14 pm »
What you describe sounds a lot like "write access to a dangling pointer" or "write access to out of bounds array/list entries". The latter would maybe be caught by range checks. The former is rather hard to find.

If you have a dangling pointer (e.g. an object that was freed, but not nil'ed) then the memory where that object was will be re-used later. If then you write to the dangling pointer, then you overwrite the new data at that location. Effectively trashing that data. So when the code tries to use the trashed data, it will fail.

You don't always get the error, because sometimes the dangling pointer points to unused memory => and then no harm done.
You change any code at all, the order of things in memory changes, and the error moves... or hides.



There are a couple of means.

First heaptrc.
- Compile with -gh (project options / debugger / heaptrc - mem leaks)
- In menu Run > Run Parameters: Set the environment for your app to have    HEAPTRC=keepreleased

That is probably going to get you some details, but it's more to confirm that you have incorrect mem access. It may not help much finding where.


You are on Linux. Lucky you. On Linux you can use valgrind (3rd party tool, can be installed to any Linux).

Compile your app with -gv (and without heaptrc).

Then outside the IDE run:
Code: Text  [Select][+][-]
  1. valgrind  --tool=memcheck  --num-callers=35  --log-file=mylog.trc   ./yourapp

This will take some time. Your app will be crawling slow.
Do a few interactions with the app. You don't need to get an error, but you need to make sure the invalid code is run (if the error goes to unused mem, it still will be logged).

When done, in the IDE, menu View > Leaks and traces : load the logfile.

It will tell you when and where "freed" memory was accessed.

Mind there are entries like "4 bytes were accessed (READ) and one of it was outside an allocated block" => those are often false positives.

You are looking for WRITE to FREED mem.


big_M

  • Full Member
  • ***
  • Posts: 104
Re: Random faults
« Reply #5 on: January 16, 2024, 12:34:17 am »
Thanks @Martin_fr for that detailed post, that sounds like a possibility. I will check out valgrind and report back.

What if started in IDE but without debugging, Shift-Ctrl-F9 ?
In that case it also runs without any issue.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12173
  • Debugger - SynEdit - and more
    • wiki
Re: Random faults
« Reply #6 on: January 16, 2024, 12:44:37 am »
About "no issues outside the debugger".

Could be coincidence. Dangling pointers are after all somewhat random.

Could also be timing. Under a debugger, the exe gets occasionally "micro-paused". So timing changes ever so slightly. If you use threads, then that can have an effect. Either if they aren't using proper locking, or - again - if there are dangling pointers, because then the time at which a dangling pointer points to data that will cause an error can shift, as the threads have different timing...

Mind that threads could be hidden from you, in gtk or qt code. And dangling pointers can be passed to them too.

Anyway lets see what valgrind has to say.

marcov

  • Administrator
  • Hero Member
  • *
  • Posts: 12706
  • FPC developer.
Re: Random faults
« Reply #7 on: January 16, 2024, 11:54:44 am »
I'd add uninitialized variables and functions that don't (always) set the return value to the list of causes of vague bugs that point to non deterministic behaviour.

Compiling with -CR and -gttt can also help.

-CR changes every cast of classes to an AS like operator, throwing an exception if they don't match.

-gttt intiializes local variables to random patterns (more t's change the pattern) which can help finding them.

big_M

  • Full Member
  • ***
  • Posts: 104
Re: Random faults
« Reply #8 on: January 16, 2024, 01:42:24 pm »
Okay, so here is the valgrind output. Looks like it found something, there are a bunch of "invalid read/write" entries.

This in one of them, but the rest looks similar:
Code: [Select]
==6738== Memcheck, a memory error detector
==6738== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==6738== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==6738== Command: ./carmaintenance
==6738== Parent PID: 6532
==6738==
==6738== Invalid read of size 8
==6738==    at 0x72E088: QTOBJECTS$_$TQTWIDGETPALETTE_$__$$_SETTEXTCOLOR$PQCOLOR (qtobjects.pas:4968)
==6738==    by 0x73651A: QTWIDGETS$_$TQTWIDGET_$__$$_EVENTFILTER$QOBJECTH$QEVENTH$$BOOLEAN (qtwidgets.pas:2667)
==6738==    by 0x67B198F: QCoreApplicationPrivate::sendThroughObjectEventFilters(QObject*, QEvent*) (in /usr/lib64/libQt5Core.so.5.15.2)
==6738==    by 0x54BE391: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x54C519F: QApplication::notify(QObject*, QEvent*) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x67B1C02: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib64/libQt5Core.so.5.15.2)
==6738==    by 0x54F96BB: QWidgetPrivate::setEnabled_helper(bool) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x73B19A: QTWIDGETS$_$TQTWIDGET_$__$$_SETENABLED$BOOLEAN (qtwidgets.pas:5070)
==6738==    by 0x627AFA: QTINT$_$TQTWIDGETSET_$__$$_ENABLEWINDOW$HWND$BOOLEAN$$BOOLEAN (qtwinapi.inc:1511)
==6738==    by 0x5CEFD6: LCLINTF_$$_ENABLEWINDOW$HWND$BOOLEAN$$BOOLEAN (winapi.inc:235)
==6738==    by 0x64EEA7: CONTROLS$_$TWINCONTROL_$__$$_INITIALIZEWND (wincontrol.inc:7642)
==6738==    by 0x64EABF: CONTROLS$_$TWINCONTROL_$__$$_CREATEWND (wincontrol.inc:7551)
==6738==    by 0x699671: STDCTRLS$_$TCUSTOMBUTTON_$__$$_CREATEWND (buttons.inc:46)
==6738==    by 0x64E136: CONTROLS$_$TWINCONTROL_$__$$_CREATEHANDLE (wincontrol.inc:7435)
==6738==    by 0x64F75F: CONTROLS$_$TWINCONTROL_$__$$_HANDLENEEDED (wincontrol.inc:7882)
==6738==    by 0x64EBC5: CONTROLS$_$TWINCONTROL_$__$$_CREATEWND (wincontrol.inc:7570)
==6738==    by 0x44F2BF: FORMS$_$TSCROLLINGWINCONTROL_$__$$_CREATEWND (scrollingwincontrol.inc:31)
==6738==    by 0x45891B: FORMS$_$TCUSTOMFORM_$__$$_CREATEWND (customform.inc:2729)
==6738==    by 0x459F45: FORMS$_$TFORM_$__$$_CREATEWND (customform.inc:3200)
==6738==    by 0x64E136: CONTROLS$_$TWINCONTROL_$__$$_CREATEHANDLE (wincontrol.inc:7435)
==6738==    by 0x64F75F: CONTROLS$_$TWINCONTROL_$__$$_HANDLENEEDED (wincontrol.inc:7882)
==6738==    by 0x464553: FORMS$_$TAPPLICATION_$__$$_CREATEFORM$TCOMPONENTCLASS$formal (application.inc:2256)
==6738==    by 0x424F27: main (carmaintenance.lpr:28)
==6738==  Address 0x12b62e48 is 120 bytes inside a block of size 144 free'd
==6738==    at 0x4E080EB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6738==    by 0x447822: CMEM_$$_CFREEMEM$POINTER$$QWORD (in /home/Ben/Dokumente/lazarus/CarMaintenance/carmaintenance)
==6738==    by 0x43693A: SYSTEM$_$TOBJECT_$__$$_FREE (in /home/Ben/Dokumente/lazarus/CarMaintenance/carmaintenance)
==6738==    by 0x67B198F: QCoreApplicationPrivate::sendThroughObjectEventFilters(QObject*, QEvent*) (in /usr/lib64/libQt5Core.so.5.15.2)
==6738==    by 0x54BE391: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x54C519F: QApplication::notify(QObject*, QEvent*) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x67B1C02: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib64/libQt5Core.so.5.15.2)
==6738==    by 0x54EE923: QWidgetPrivate::propagatePaletteChange() (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x54EEB24: QWidgetPrivate::setPalette_helper(QPalette const&) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x54F19DB: QWidget::setPalette(QPalette const&) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x72E07F: QTOBJECTS$_$TQTWIDGETPALETTE_$__$$_SETTEXTCOLOR$PQCOLOR (qtobjects.pas:4967)
==6738==    by 0x73651A: QTWIDGETS$_$TQTWIDGET_$__$$_EVENTFILTER$QOBJECTH$QEVENTH$$BOOLEAN (qtwidgets.pas:2667)
==6738==    by 0x67B198F: QCoreApplicationPrivate::sendThroughObjectEventFilters(QObject*, QEvent*) (in /usr/lib64/libQt5Core.so.5.15.2)
==6738==    by 0x54BE391: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x54C519F: QApplication::notify(QObject*, QEvent*) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x67B1C02: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib64/libQt5Core.so.5.15.2)
==6738==    by 0x54F96BB: QWidgetPrivate::setEnabled_helper(bool) (in /usr/lib64/libQt5Widgets.so.5.15.2)
==6738==    by 0x73B19A: QTWIDGETS$_$TQTWIDGET_$__$$_SETENABLED$BOOLEAN (qtwidgets.pas:5070)
==6738==    by 0x627AFA: QTINT$_$TQTWIDGETSET_$__$$_ENABLEWINDOW$HWND$BOOLEAN$$BOOLEAN (qtwinapi.inc:1511)
==6738==    by 0x5CEFD6: LCLINTF_$$_ENABLEWINDOW$HWND$BOOLEAN$$BOOLEAN (winapi.inc:235)
==6738==    by 0x64EEA7: CONTROLS$_$TWINCONTROL_$__$$_INITIALIZEWND (wincontrol.inc:7642)
==6738==    by 0x64EABF: CONTROLS$_$TWINCONTROL_$__$$_CREATEWND (wincontrol.inc:7551)
==6738==    by 0x699671: STDCTRLS$_$TCUSTOMBUTTON_$__$$_CREATEWND (buttons.inc:46)
==6738==    by 0x64E136: CONTROLS$_$TWINCONTROL_$__$$_CREATEHANDLE (wincontrol.inc:7435)
==6738==    by 0x64F75F: CONTROLS$_$TWINCONTROL_$__$$_HANDLENEEDED (wincontrol.inc:7882)
==6738==    by 0x64EBC5: CONTROLS$_$TWINCONTROL_$__$$_CREATEWND (wincontrol.inc:7570)
==6738==    by 0x44F2BF: FORMS$_$TSCROLLINGWINCONTROL_$__$$_CREATEWND (scrollingwincontrol.inc:31)
==6738==    by 0x45891B: FORMS$_$TCUSTOMFORM_$__$$_CREATEWND (customform.inc:2729)
==6738==    by 0x459F45: FORMS$_$TFORM_$__$$_CREATEWND (customform.inc:3200)
==6738==    by 0x64E136: CONTROLS$_$TWINCONTROL_$__$$_CREATEHANDLE (wincontrol.inc:7435)
==6738==    by 0x64F75F: CONTROLS$_$TWINCONTROL_$__$$_HANDLENEEDED (wincontrol.inc:7882)
==6738==    by 0x464553: FORMS$_$TAPPLICATION_$__$$_CREATEFORM$TCOMPONENTCLASS$formal (application.inc:2256)
==6738==    by 0x424F27: main (carmaintenance.lpr:28)
==6738==  Block was alloc'd at
==6738==    at 0x4E056A4: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==6738==    by 0x4477FC: CMEM_$$_CGETMEM$QWORD$$POINTER (in /home/Ben/Dokumente/lazarus/CarMaintenance/carmaintenance)
==6738==    by 0x73C5A2: QTWIDGETS$_$TQTWIDGET_$__$$_GETPALETTE$$TQTWIDGETPALETTE (qtwidgets.pas:5771)
==6738==    by 0x73A22D: QTWIDGETS$_$TQTWIDGET_$__$$_SETCOLOR$PQCOLOR (qtwidgets.pas:4592)
==6738==    by 0x7C1941: QTWSCONTROLS$_$TQTWSWINCONTROL_$__$$_SETCOLOR$TWINCONTROL (qtwscontrols.pp:674)
==6738==    by 0x64EE28: CONTROLS$_$TWINCONTROL_$__$$_INITIALIZEWND (wincontrol.inc:7631)
==6738==    by 0x64EABF: CONTROLS$_$TWINCONTROL_$__$$_CREATEWND (wincontrol.inc:7551)
==6738==    by 0x699671: STDCTRLS$_$TCUSTOMBUTTON_$__$$_CREATEWND (buttons.inc:46)
==6738==    by 0x64E136: CONTROLS$_$TWINCONTROL_$__$$_CREATEHANDLE (wincontrol.inc:7435)
==6738==    by 0x64F75F: CONTROLS$_$TWINCONTROL_$__$$_HANDLENEEDED (wincontrol.inc:7882)
==6738==    by 0x64EBC5: CONTROLS$_$TWINCONTROL_$__$$_CREATEWND (wincontrol.inc:7570)
==6738==    by 0x44F2BF: FORMS$_$TSCROLLINGWINCONTROL_$__$$_CREATEWND (scrollingwincontrol.inc:31)
==6738==    by 0x45891B: FORMS$_$TCUSTOMFORM_$__$$_CREATEWND (customform.inc:2729)
==6738==    by 0x459F45: FORMS$_$TFORM_$__$$_CREATEWND (customform.inc:3200)
==6738==    by 0x64E136: CONTROLS$_$TWINCONTROL_$__$$_CREATEHANDLE (wincontrol.inc:7435)
==6738==    by 0x64F75F: CONTROLS$_$TWINCONTROL_$__$$_HANDLENEEDED (wincontrol.inc:7882)
==6738==    by 0x464553: FORMS$_$TAPPLICATION_$__$$_CREATEFORM$TCOMPONENTCLASS$formal (application.inc:2256)
==6738==    by 0x424F27: main (carmaintenance.lpr:28)

I assume you read them from the bottom up. Do I read this correctly: A section of the memory was allocated by qtwidgets.pas, line 5771, free'd by libQt5Core.so.5.15.2 and then read again by qtwidgets.pas, line 2667? So it's all related to qt5 it seems.

Allocation:
Code: Pascal  [Select][+][-]
  1. function TQtWidget.GetPalette: TQtWidgetPalette;
  2. begin
  3.   if not Assigned(FPalette) then
  4.   begin
  5.     if (ClassType = TQtCustomControl) then
  6.       FPalette := TQtWidgetPalette.Create(WidgetColorRole, TextColorRole,
  7.         TQtCustomControl(Self).viewport.Widget)
  8.     else
  9.       FPalette := TQtWidgetPalette.Create(WidgetColorRole, TextColorRole, Widget); //<---
  10.   end;
  11.   Result := FPalette;
  12. end;

Reading:
Code: Pascal  [Select][+][-]
  1. ...
  2.             if Assigned(FPalette) and Assigned(LCLObject) then
  3.             begin
  4.               // DebugLn('QEventEnabledChange: ',dbgsName(LCLObject),' enabled ',dbgs(getEnabled));
  5.               if not getEnabled then
  6.                 Palette.setTextColor(@Palette.DisabledTextColor) //<--
  7.               else
  8.               begin
  9.                 setInitialColor;
  10.                 setInitialFontColor;
  11.               end;
  12.             end;
  13. ...
  14.  
« Last Edit: January 16, 2024, 02:02:17 pm by big_M »

zeljko

  • Hero Member
  • *****
  • Posts: 1870
    • http://wiki.lazarus.freepascal.org/User:Zeljan
Re: Random faults
« Reply #9 on: January 17, 2024, 09:55:10 pm »
Can you make some simple example project so I can test ?
What controls are disabled in design time ?
EDIT: Just comment that line you mentioned and see if you have same problem. I'm not sure that it makes problem.
« Last Edit: January 17, 2024, 10:00:37 pm by zeljko »

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12173
  • Debugger - SynEdit - and more
    • wiki
Re: Random faults
« Reply #10 on: January 18, 2024, 12:12:18 am »
From the 2nd block (invalid write of 8)


Code: Pascal  [Select][+][-]
  1. procedure TWinControl.InitializeWnd;
  2. ...
  3.       // replace by update style call
  4.       TWSWinControlClass(WidgetSetClass).SetColor(Self);

Calls
Code: Pascal  [Select][+][-]
  1. function TQtWidget.GetPalette: TQtWidgetPalette;
  2. ....
  3.     else
  4.       FPalette := TQtWidgetPalette.Create(WidgetColorRole, TextColorRole, Widget);

Which is the object that then gets destroyed, and written to after that.




Once created it continues in InitializeWnd
Code: Pascal  [Select][+][-]
  1. procedure TWinControl.InitializeWnd;
  2. ...
  3.     if not (csDesigning in ComponentState) then
  4.       EnableWindow(Handle, Enabled);

Code: Pascal  [Select][+][-]
  1. procedure TQtWidget.setEnabled(p1: Boolean);
  2. begin
  3.   if not p1 and (HWND(Self) = GetCapture) then
  4.     ReleaseCapture;
  5.   QWidget_setEnabled(Widget, p1);
  6. end;

Code: Pascal  [Select][+][-]
  1. function TQtWidget.EventFilter(Sender: QObjectH; Event: QEventH): Boolean; cdecl;
  2. ....
  3.         QEventEnabledChange:
  4.           begin
  5. ...
  6.               if not getEnabled then
  7.                 Palette.setTextColor(@Palette.DisabledTextColor)

Code: Pascal  [Select][+][-]
  1. procedure TQtWidgetPalette.setTextColor(const AColor: PQColor);
  2. begin
  3. ...
  4.   QWidget_setPalette(FWidget, FHandle);
  5.  

through the dll, to freeing the above object instance.

The incorrect access is then on the next line
Code: Pascal  [Select][+][-]
  1. procedure TQtWidgetPalette.setTextColor(const AColor: PQColor);
  2. begin
  3. ....
  4.   QWidget_setPalette(FWidget, FHandle);  
  5.   FCurrentTextColor := AColor^;
  6.  

In other word "QWidget_setPalette(FWidget, FHandle);"   must be freeing the very TQtWidgetPalette object, that is making the call. Aka, the current self.


zeljko

  • Hero Member
  • *****
  • Posts: 1870
    • http://wiki.lazarus.freepascal.org/User:Zeljan
Re: Random faults
« Reply #11 on: January 18, 2024, 08:37:03 am »

big_M

  • Full Member
  • ***
  • Posts: 104
Re: Random faults
« Reply #12 on: January 18, 2024, 12:52:39 pm »
Okay, I was able to reproduce it thanks to Martins dissection. The invalid writes get triggered when a control gets disabled. Click at least two times on the button in the example program.

Hm, then it can be related to https://gitlab.com/freepascal.org/lazarus/lazarus/-/commit/7fa32a1e8e8eccc2042312d4671a5bc4580f4c38
What would speak for it is that it only appeared after I started using trunc / 3.0. Before, in 2.2.6 I didn't have this problem.

zeljko

  • Hero Member
  • *****
  • Posts: 1870
    • http://wiki.lazarus.freepascal.org/User:Zeljan
Re: Random faults
« Reply #13 on: January 18, 2024, 08:39:56 pm »
I've clicked 100 times and nothing happens, everything works as expected. Qt5-5.12.8 linux 64 bit is here.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 12173
  • Debugger - SynEdit - and more
    • wiki
Re: Random faults
« Reply #14 on: January 18, 2024, 09:05:49 pm »
According to the valgrind log, in the original app lots of errors already happened during FormCreate (and a button on the form being created).

But, if unlucky you can probably run that app a 1000 times without those (many of those recorded in the log) causing any error. Running under valgrind will record them every single time.

The one I dissected wrote to the freed memory almost immediately. 99.9% chance that that memory isn't yet re-used, and that nothing happens. (for that particular case).

 

TinyPortal © 2005-2018