Recent

Author Topic: Needed advice to debug leaks shown only after switching to new Laz&FPC  (Read 9919 times)

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
I have a medium sized Win32 application running 24/7 in heavy industry for more then a year. It has GUI and 2 communication threads, writing data to Oracle. Threads are updating GUI in thread safe way, and I have also implemented thread safe logging for main thread and communication threads. Application is usually running for months without stopping, and I do not have any memory leaks reported on application exit, as you can see in this heap trace output:
Code: Pascal  [Select][+][-]
  1. D:\Data\MyUserName\My Documents\Lazarus Projects\L1toOracle\L1toOracle.exe
  2. Heap dump by heaptrc unit
  3. 277512 memory blocks allocated : 97836608/98786600
  4. 277512 memory blocks freed     : 97836608/98786600
  5. 0 unfreed memory blocks : 0
  6. True heap size : 327680
  7. True free heap : 327536
  8. Should be : 327680

The problem is that when I want to upgrade to any later FPC/Lazarus I get leaks as reported here (even if I just start application and exit it immediately):
Code: Pascal  [Select][+][-]
  1. d:\-\-\-\-\-\L1toOracle.exe
  2. Heap dump by heaptrc unit
  3. 18330 memory blocks allocated : 2193384/2248376
  4. 18327 memory blocks freed     : 2193316/2248296
  5. 3 unfreed memory blocks : 68
  6. True heap size : 327680 (80 used in System startup)
  7. True free heap : 327280
  8. Should be : 327328
  9. Call trace for block $00081170 size 28
  10.   $0070F4B6
  11.   $0070E2C5
  12.   $004107E6
  13.   $BAADF00D
  14.   $BAADF00D
  15.   $BAADF00D
  16.   $BAADF00D
  17.   $BAADF00D
  18. Call trace for block $0007B370 size 12
  19.   $0070E2C5
  20.   $004107E6
  21.   $BAADF00D
  22.   $BAADF00D
  23.   $BAADF00D
  24.   $BAADF00D
  25.   $BAADF00D
  26.   $BAADF00D
  27. Call trace for block $00081100 size 28
  28.   $006E1D1C
  29.   $004107E6
  30.   $004107E6
  31.   $BAADF00D
  32.   $BAADF00D
  33.   $BAADF00D
  34.   $BAADF00D
  35.   $BAADF00D

Compiling Lazarus with full debug info and running backtrace does not show anything useful, as you can see here:
Code: Pascal  [Select][+][-]
  1. GNU gdb (GDB) 7.9
  2. Copyright (C) 2015 Free Software Foundation, Inc.
  3. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
  4. This is free software: you are free to change and redistribute it.
  5. There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
  6. and "show warranty" for details.
  7. This GDB was configured as "i686-pc-mingw32".
  8. Type "show configuration" for configuration details.
  9. For bug reporting instructions, please see:
  10. <http://www.gnu.org/software/gdb/bugs/>.
  11. Find the GDB manual and other documentation resources online at:
  12. <http://www.gnu.org/software/gdb/documentation/>.
  13. For help, type "help".
  14. Type "apropos word" to search for commands related to "word".
  15. This binary was built by Equation Solution <http://www.Equation.com>...
  16. Reading symbols from L1toOracle.exe...done.
  17. (gdb) run
  18. Starting program: d:\-\-\-\-\-\L1toOracle.exe
  19. [New Thread 7780.0x1778]
  20. [New Thread 7780.0x10d4]
  21. [Thread 7780.0x10d4 exited with code 0]
  22. [New Thread 7780.0x136c]
  23. [New Thread 7780.0x1aac]
  24. [New Thread 7780.0x1670]
  25. [New Thread 7780.0x210]
  26. [Thread 7780.0x136c exited with code 0]
  27. [Thread 7780.0x1aac exited with code 0]
  28. [Thread 7780.0x1670 exited with code 0]
  29. [Thread 7780.0x210 exited with code 0]
  30. [Inferior 1 (process 7780) exited normally]
  31. (gdb) bt
  32. No stack.
  33. (gdb)

In earlier cases when leaks were detected it was mostly my fault that could be located because line numbers and name hints were giving me enough info to get an idea where things went wrong. But this time I have no hints and I do not know where to look. I expected to get more info when I rebuilt Lazarus with "Debug IDE" profile ("-gw -gl -godwarfsets -gh -gt -Co -Cr -Ci -Sa"), but that was not the case. Did I make some error in this step?

I would really appreciate any idea or shared experience of the best strategy to start hunting this kind of leaks.
%) ::) %)

Btw, project was initially done in CodeTyphon 5.60 and I faced the problem when trying to make it a Lazarus project. I have tried latest Lazarus stable, Lazarus trunk, and both Lazarus and FPC trunk, and leaks were always present (trunk was from about a month ago). I have also tried to update CodeTyphon and that was only partially successful. The latest CT version where I do not have any leaks is 5.80 (with all updates installed it reports FPC 3.1.1 SVN 51943 and Lazarus from 2016-04-02). Any later CT version also introduces leaks in my application.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 9792
  • Debugger - SynEdit - and more
    • wiki
Re: Needed advice to debug leaks shown only after switching to new Laz&FPC
« Reply #1 on: December 11, 2017, 02:51:29 pm »
I never saw BaadFood on a stacktrace. That might mean there is a deeper issue.

Maybe you are writing to already freed memory.
Try -gh and set environment
HEAPTRC="keepreleased"


As for the trace. Load in gdb, and try
  info line *0x004107E6

I dont know why the top of your trace has addresses in the 00070.... range. Do you use libraries written in pascal?

Then the leak is in one of those libs, try the info line on the address 007...

The 004... should be in your main app.

Thaddy

  • Hero Member
  • *****
  • Posts: 14204
  • Probably until I exterminate Putin.
Re: Needed advice to debug leaks shown only after switching to new Laz&FPC
« Reply #2 on: December 11, 2017, 03:06:53 pm »
I don't know if it is the case here, but usually $BADF00D indicates heap memory that is allocated in a local routine and not freed: IOW: possibly a locally created class that is not freed.
Specialize a type, not a var.

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Needed advice to debug leaks shown only after switching to new Laz&FPC
« Reply #3 on: December 11, 2017, 03:17:13 pm »
Update: Problem solved. No more leaks!   :D ;) :D
After playing with project options, I have finally managed to get a meaningfull heaptrace:
Code: Pascal  [Select][+][-]
  1. d:\-\-\-\-\-\L1toOracle.exe
  2. Heap dump by heaptrc unit
  3. 17635 memory blocks allocated : 2052749/2105816
  4. 17632 memory blocks freed     : 2052681/2105736
  5. 3 unfreed memory blocks : 68
  6. True heap size : 393216 (80 used in System startup)
  7. True free heap : 392816
  8. Should be : 392864
  9. Call trace for block $00081248 size 28
  10.   $0070D496  TIDTHREADSAFE__CREATE,  line 253 of ./source/IdThreadSafe.pas
  11.   $0070C2A5  IDTHREAD_$$_init$,  line 730 of ./source/IdThread.pas
  12.   $004107E6
  13.   $BAADF00D
  14.   $BAADF00D
  15.   $BAADF00D
  16.   $BAADF00D
  17.   $BAADF00D
  18. Call trace for block $0007B448 size 12
  19.   $0070C2A5  IDTHREAD_$$_init$,  line 730 of ./source/IdThread.pas
  20.   $004107E6
  21.   $BAADF00D
  22.   $BAADF00D
  23.   $BAADF00D
  24.   $BAADF00D
  25.   $BAADF00D
  26.   $BAADF00D
  27. Call trace for block $000811D8 size 28
  28.   $006DFCFC  IDSTACK_$$_init$,  line 1225 of ./source/IdStack.pas
  29.   $004107E6
  30.   $004107E6
  31.   $BAADF00D
  32.   $BAADF00D
  33.   $BAADF00D
  34.   $BAADF00D
  35.   $BAADF00D

It was Indy unit that is not used at all, but somehow it was forgotten and Indy was still in project's packages list. It was enough to delete Indy and leaks have gone. I will test more, but it seams that I can finally use newer Lazarus and FPC.
 :D

@martin: Thank you so much for looking into this problem. Although I solved the problem before applying your suggestions, I find them very valuable. This has been an uncharted terithory for me, and I can learn a lot from them. Thank you very much!
 O:-)

@thaddy: I did not have leaks with old Laz/FPC, so it was not a local class not being freed. As you can see from the latest update, leaks have gone once I deleted not really used Indy package reference from the project. Thank you for looking into this!
 O:-)
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Thaddy

  • Hero Member
  • *****
  • Posts: 14204
  • Probably until I exterminate Putin.
Re: Needed advice to debug leaks shown only after switching to new Laz&FPC
« Reply #4 on: December 11, 2017, 03:23:30 pm »
I recall Indy has some expected memory leaks registered (at least under Delphi)
Specialize a type, not a var.

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Needed advice to debug leaks shown only after switching to new Laz&FPC
« Reply #5 on: December 11, 2017, 09:24:16 pm »
I recall Indy has some expected memory leaks registered (at least under Delphi)

That is correct.  The 3 leaks in the above report are from Indy.

The IdThread unit has a global TIdThreadSafeInteger object, which has an internal TCriticalSection object.

The IdStack unit has a global TCriticalSection object.

These two global objects are intentionally leaked by default, but the leaks can be disabled by re-compiling Indy with FREE_ON_FINAL enabled (but do read the warnings documented at the very bottom of those units' source code).

In Delphi, these 3 leaks are registered with the memory manager so they do not appear in leak reports.  FreePascal does not have that option in the HeapTrc unit (see this discussion).
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Needed advice to debug leaks shown only after switching to new Laz&FPC
« Reply #6 on: December 11, 2017, 11:12:28 pm »
The 3 leaks in the above report are from Indy.
...
These two global objects are intentionally leaked by default, but the leaks can be disabled by re-compiling Indy with FREE_ON_FINAL enabled (but do read the warnings documented at the very bottom of those units' source code).
It would really be nice if this knowledge ends up in a wiki: http://wiki.freepascal.org/Indy_with_Lazarus. If no one else with more knowledge of the issue wants to do it, I will.
 ::)
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Needed advice to debug leaks shown only after switching to new Laz&FPC
« Reply #7 on: December 26, 2017, 03:48:05 pm »
Quote
It would really be nice if this knowledge ends up in a wiki
Here it is. Please review and correct if needed.
http://wiki.lazarus.freepascal.org/Indy_with_Lazarus#Memory_leaking_issue

btw1. In last 2 weeks my application was running 24/7 and I was lucky to have one network down event and several oracle server down events. Exceptions are logged and properly handled so on application exit there were no leaks. Problem definitely solved!

btw2. If anyone is interested, I have contributed thread safe TMemoChannel to MultiLog and assisted author with making MultiLog fully thread safe. All communication threads from my mentioned application log data to the same log file and to the same Memo component without problems. I am not aware of any other logging library that can do this at the moment.
 :D ;) :D
« Last Edit: December 26, 2017, 03:51:24 pm by avra »
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

Thaddy

  • Hero Member
  • *****
  • Posts: 14204
  • Probably until I exterminate Putin.
Re: Needed advice to debug leaks shown only after switching to new Laz&FPC
« Reply #8 on: December 26, 2017, 05:46:47 pm »
Remy, Avra,

The leaks may disappear if you wrap the leaking classes in something like my smartpointer code I borrowed from an idea by Marco Cantu:
http://forum.lazarus.freepascal.org/index.php/topic,33095.msg221076.html#msg221076
http://forum.lazarus.freepascal.org/index.php/topic,37524.msg252382.html#msg252382

Until smartpointers are implemented as standard, that is. Threadsafe.

Note my code is cross-platform and tested on nixes (arm too...) and win32/64.
E.g.:
Code: Pascal  [Select][+][-]
  1. program smartpointersCS;
  2. {$ifdef fpc}{$mode delphi}{$endif}
  3. uses
  4.   classes,smartptrs,SyncObjs;
  5. var
  6.   L:TCriticalSection;
  7.   PL:Auto<TCriticalSection>;
  8. begin
  9.   // Auto create
  10.   L:= PL.Value;
  11.   L.Acquire;
  12.  try
  13.  ... worker.
  14.   finally
  15.    L.Release;
  16.  end;
  17.   // Auto cleanup
  18. end.

This uses the smartptrs unit from the second link.
« Last Edit: December 26, 2017, 06:19:52 pm by Thaddy »
Specialize a type, not a var.

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1312
    • Lebeau Software
Re: Needed advice to debug leaks shown only after switching to new Laz&FPC
« Reply #9 on: December 15, 2021, 06:03:27 pm »
The leaks may disappear if you wrap the leaking classes in something like my smartpointer code I borrowed from an idea by Marco Cantu:
http://forum.lazarus.freepascal.org/index.php/topic,33095.msg221076.html#msg221076
http://forum.lazarus.freepascal.org/index.php/topic,37524.msg252382.html#msg252382

The leaks are not a matter of not being able to free the objects (obviously, they can be, as seen when FREE_ON_FINAL is defined), but more a matter of unit dependency.  The objects in question are shared with other units, and it has been noticed that sometimes units may not always be finalized in the order you are expecting, and that has been observed in the wild.  So rather than freeing these objects by default, they are leaked instead, in case they are still accessed later.  The leak is minor (only a few bytes), and only during unit finalization, which 9x% of the time will only be during app shutdown (unless you are using Indy in a DLL that gets unloaded dynamically during runtime).
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

 

TinyPortal © 2005-2018