Lazarus

Free Pascal => FPC development => Topic started by: BrunoK on July 17, 2019, 01:18:45 pm

Title: [Done] Must Order of loading cmem/heaptrc
Post by: BrunoK on July 17, 2019, 01:18:45 pm
FPC 3.0.6 (derivative of //svn.freepascal.org/svn/fpc/tags/release_3_0_4a_macos_10_14).

Would someone have some idea where I should look/search in the compiler where to patch and so to be able to have cmem/heaptrc to be initialized BEFORE system.pp code :
  { Arguments }
  setup_arguments;

I want that because setup_arguments does use the default heap.inc memory manager and conflicts afterward with cmem's reallocmem / getmem / freemem.

Maybe someone knowledgable could PM me a rough idea how to do that.

Hopefully thanks ...
Title: Re: Order of loading cmem/heaptrc
Post by: Thaddy on July 17, 2019, 01:23:46 pm
See the wiki. Don't mix the two!! Ever.
https://wiki.freepascal.org/heaptrc and https://www.freepascal.org/docs-html/rtl/heaptrc/usage.html

There is a way mentioned as a note on how to select either cmem or heaptrc:
Code: Pascal  [Select]
  1. program test;
  2. {$mode objfpc}
  3. uses
  4. // This will conditionally switch in a memory manager
  5. // based on the presence of heaptrc
  6.  {$if not declared(UseHeapTrace)}cmem;{$ifend}
  7. begin
  8. {$if declared(UseHeapTrace)}
  9.    writeln('Heaptrc is used.',' Heaptrc is active? ', UseHeaptrace);
  10. {$else}
  11.    writeln('No trace of heaptrc');
  12. {$ifend}
  13. end.

The ratio is that heaptrc is also a memory manager and you can't have two without creating problems.

( A sick option is to rename cmem to heaptrc and rebuild, that would insert cmem instead of heaptrc when compiled with -gh   :o 8-) )
Title: Re: Order of loading cmem/heaptrc
Post by: BrunoK on July 17, 2019, 01:37:51 pm
See the wiki. Don't mix the two!! Ever.
Yes, I know that bloody well, my new heaptrc WITH cmem nearly works but bangs because HEAP.INC is USED BEFORE (in ' { Arguments } setup_arguments;') having had the opportunity to install cmem.

Why use cmem ?

because it seems much faster in multihtreading applications.
Title: Re: Order of loading cmem/heaptrc
Post by: Thaddy on July 17, 2019, 01:42:40 pm
See the wiki. Don't mix the two!! Ever.
Yes, I know that bloody well, my new heaptrc WITH cmem nearly works but bangs because HEAP.INC is USED BEFORE (in ' { Arguments } setup_arguments;') having had the opportunity to install cmem.

Why use cmem ?

because it seems much faster in multihtreading applications.

See my last line above. The sick option part..
You are wrong in assuming it is faster. cmem does simply larger allocations, which can easily cause problems in long running applications. It looks faster, but not always and you should test.
Technically cmem is a bare and very basic OS heap based manager. The default memory manager is more advanced than that.
With threads, you need scaling. cmem does not scale at all.
Title: Re: Order of loading cmem/heaptrc
Post by: Martin_fr on July 17, 2019, 01:43:45 pm
use -Fa command line option. From docs:
  -Fa<x>[,y] (for a program) load units <x> and [y] before uses is parsed
Title: Re: Order of loading cmem/heaptrc
Post by: Thaddy on July 17, 2019, 01:49:34 pm
use -Fa command line option. From docs:
  -Fa<x>[,y] (for a program) load units <x> and [y] before uses is parsed
That does not work if heaptrc is selected, though. cmem will still be loaded after heaptrc. Still the same problem about using two memory managers.
Title: Re: Order of loading cmem/heaptrc
Post by: Martin_fr on July 17, 2019, 02:38:47 pm
That does not work if heaptrc is selected, though. cmem will still be loaded after heaptrc. Still the same problem about using two memory managers.
He does not say he wants both at the same time. From what I read he either wants:
- one of them (whichever he chooses) to be loaded
- a mem-mgr of his own (which is a merge of the 2) to be loaded.

So in any case just one mem-mgr.

At least that his how I read his posts
Title: Re: Order of loading cmem/heaptrc
Post by: BrunoK on July 17, 2019, 02:45:42 pm

So in any case just one mem-mgr.

At least that his how I read his posts

YES, exactly, full on target as usual !
Title: Re: Order of loading cmem/heaptrc
Post by: BrunoK on July 17, 2019, 03:01:22 pm
use -Fa command line option. From docs:
  -Fa<x>[,y] (for a program) load units <x> and [y] before uses is parsed
Thanks for the tip.

Sadly, the compiler insists on having  system beeing initialized BEFORE cmem than with heap.inc ...

Will dig further.


See my last line above. The sick option part..
You are wrong in assuming it is faster. cmem does simply larger allocations, which can easily cause problems in long running applications. It looks faster, but not always and you should test.
see https://forum.lazarus.freepascal.org/index.php/topic,44590.0.html.
What took it to the top was using cmem.

I can  see it also in wp's mapviewer that leaks. A thing is sure is that it leaks MUCH faster with cmem ...

Quote
Technically cmem is a bare and very basic OS heap based manager. The default memory manager is more advanced than that.
With threads, you need scaling. cmem does not scale at all.
cmem uses msvcrt in win10 and seems to be a tcmalloc allocator. tcmalloc looks like a subtle piece of software. That heap allocator has much evolved in the last 6 years and seems to be the mainstream heap allocator whereas HEAP.INC doesn't seem to have evolved in the last 10 years.
There are quite a few inconsistencies in heap.inc (those I took care of).

As for memory utilization, the W10 task manager shows both fpc and lazarus using less memory when build with cmem.
Title: Re: Order of loading cmem/heaptrc
Post by: Thaddy on July 17, 2019, 04:17:36 pm
Why not link in tcmalloc directly then?
Note msvcrt can have multiple versions depending on system and - although I have a development build of Windows for 15+ years, I am not aware recent  msvcrt versions have changed their malloc implementation..
The original Google version of tcmalloc can be linked statically too. https://github.com/gperftools/gperftools
It is also cross-platform, which msvcrt definitely is not!

Note that if Google themselves say it is the fastest they have seen, that is most likely correct.
Title: Re: Order of loading cmem/heaptrc
Post by: Martin_fr on July 17, 2019, 04:55:26 pm
Sadly, the compiler insists on having  system beeing initialized BEFORE cmem than with heap.inc ...

You may want to try asking on the fpc-devel mail list.

But I am not sure it is possible. AFAIK system is always used. So even your unit is implicitly using system. Which means never mind how early you load your unit, it will load system before itself.
Title: Re: Order of loading cmem/heaptrc
Post by: marcov on July 17, 2019, 05:26:22 pm
cmem is mostly meant for *nix, e.g. to use valgrind.

I don't think this is really tested functionality on Windows, specially in combination with heaptrc.



Title: Re: Order of loading cmem/heaptrc
Post by: Thaddy on July 17, 2019, 05:32:31 pm
cmem is mostly meant for *nix, e.g. to use valgrind.

I don't think this is really tested functionality on Windows, specially in combination with heaptrc.

It has been used for many years It has the same interface on msvcrt. It is basically a set of stdc library calls, so support by all - most - C compilers. Another example is heapmm from KOL which uses the same calls.
The combination, though, is out of the question. It is a matter of either/or
Title: Re: Order of loading cmem/heaptrc
Post by: BrunoK on July 17, 2019, 05:34:25 pm
You may want to try asking on the fpc-devel mail list.

But I am not sure it is possible. AFAIK system is always used. So even your unit is implicitly using system. Which means never mind how early you load your unit, it will load system before itself.
Thanks for your help.

I think I fingered where I can try to do that.  Will try tomorrow.

Cheers, Bruno

I don't think this is really tested functionality on Windows, specially in combination with heaptrc.
I know it works because I run both FPC and Lazarus with cmem for the last 3 months, not just 5 minutes sometimes, but for hours many days in a week.

Of course it doesn't  work with the current version of heaptrc but it nearly works with mine.
Title: Re: Order of loading cmem/heaptrc
Post by: Akira1364 on July 17, 2019, 05:48:14 pm
I don't think this is expected / supposed to work, as currently implemented. heaptrc.pp is itself a
"memory manager unit" in the same way that cmem.pp is.

Both always replace the default memory manager with their own implementations ("CMemoryManager" in the case of cmem, and "TraceManager" in the case of heaptrc) and as such it is not possible for them to co-exist.

If you want something that is still able to report memory usage the way heaptrc does, but calls into some external malloc implementation, the best way to go would be with a modified version of the heaptrc unit that explicitly uses the external API for the actual allocation / deallocation routines, which will generally return the same kind of values allowing it to track them the same way.

As far as whether CMem specifically is generally useful on Windows though, it definitely does massively improve the performance of multi-threaded applications (on at least 64-bit, not sure about 32-bit.)
Title: Re: Order of loading cmem/heaptrc
Post by: PascalDragon on July 18, 2019, 09:42:05 am
Sadly, the compiler insists on having  system beeing initialized BEFORE cmem than with heap.inc ...

You may want to try asking on the fpc-devel mail list.

But I am not sure it is possible. AFAIK system is always used. So even your unit is implicitly using system. Which means never mind how early you load your unit, it will load system before itself.

Correct.

If BrunoK wants to use the Windows heap manager directly for all allocations, including those in early startup than he needs to implement and set his heap manager directly in the System unit. This involves at least rtl\win32\system.pp, rtl\win64\system.pp, rtl\win\sysheap.inc and rtl\inc\heap.inc.
Title: Re: Order of loading cmem/heaptrc
Post by: BrunoK on July 18, 2019, 10:52:03 am
If BrunoK wants to use the Windows heap manager directly for all allocations, including those in early startup than he needs to implement and set his heap manager directly in the System unit. This involves at least rtl\win32\system.pp, rtl\win64\system.pp, rtl\win\sysheap.inc and rtl\inc\heap.inc.
The idea is to stay near current operation.
I have already added a -dCMEM compiler option in TOption.interpret_option +                    include(init_settings.globalswitches,cs_use_cmem); { new cs_use_cmem in enum) +
in pmodules.pas loaddefaultunits
            { The correct heap manager should be setup BEFORE 'system' initializes
             but it is not the case yet ~bk }
           { Units only required for main module }
           if (cs_use_cmem in current_settings.globalswitches) then
              AddUnit('cmem');

that works without any trouble, but there is still this question of getting them in the correct order in fpc_InitializeUnits.

I'll find ...

My original post concerned only getting hints about that stuff, thats all.

Anyway thank you all, even the usual grumpy knows all.
Title: Re: Order of loading cmem/heaptrc
Post by: marcov on July 18, 2019, 11:27:46 am
But cmem is an unit, and as such implicitly requires system, which means that system initializes before cmem?
Title: Re: Order of loading cmem/heaptrc
Post by: PascalDragon on July 19, 2019, 09:37:15 am
If BrunoK wants to use the Windows heap manager directly for all allocations, including those in early startup than he needs to implement and set his heap manager directly in the System unit. This involves at least rtl\win32\system.pp, rtl\win64\system.pp, rtl\win\sysheap.inc and rtl\inc\heap.inc.
The idea is to stay near current operation.
I have already added a -dCMEM compiler option in TOption.interpret_option +                    include(init_settings.globalswitches,cs_use_cmem); { new cs_use_cmem in enum) +
in pmodules.pas loaddefaultunits
            { The correct heap manager should be setup BEFORE 'system' initializes
             but it is not the case yet ~bk }
           { Units only required for main module }
           if (cs_use_cmem in current_settings.globalswitches) then
              AddUnit('cmem');

that works without any trouble, but there is still this question of getting them in the correct order in fpc_InitializeUnits.
Don't mess with these things! The Windows RTL has a secondary entry point in Exec_Tls_Callback in rtl\win\systlsdir.inc which is called before the normal entry point and which might lead to allocations. It might not do so now, but it's not guaranteed to be this way in the future as well.

The only clean way is to integrate it directly into the System unit.
Title: Re: Order of loading cmem/heaptrc
Post by: BrunoK on July 19, 2019, 11:36:37 am

Don't mess with these things! The Windows RTL has a secondary entry point in Exec_Tls_Callback in rtl\win\systlsdir.inc which is called before the normal entry point and which might lead to allocations. It might not do so now, but it's not guaranteed to be this way in the future as well.

The only clean way is to integrate it directly into the System unit.
You are alright about maybe messing with these things. I  won't  come complaining if it breaks and anyway it is my custom 3.0.6 version.

As an indication of what it has now, unretouched for the last 6 months, is for example :
TList = class(TFPList) refactoring. This and its descendant allocate only one INSTANCE for each TFPList descendent. That approach works without trouble (touch wood) both in cclasses and many List classes (at least those I understand what they do). Apart having only one object instance, as a by effect of that, one can inspect the list instance during debugging and see FCount an FCapacity which are always of interest for TFPList based instances.

That means that my TList is declared in lists.inc as
Code: Pascal  [Select]
  1.   TListEnumerator = TFPListEnumerator;
  2.  
  3.   TList = class(TFPList{ ,IFPObserved})
  4.   private
  5.     procedure CopyMove (aList : TFPList);
  6.     procedure MergeMove (aList : TFPList);
  7.     procedure DoCopy(ListA, ListB : TFPList);
  8.     procedure DoSrcUnique(ListA, ListB : TFPList);
  9.     procedure DoAnd(ListA, ListB : TFPList);
  10.     procedure DoDestUnique(ListA, ListB : TFPList);
  11.     procedure DoOr(ListA, ListB : TFPList);
  12.     procedure DoXOr(ListA, ListB : TFPList);
  13.   protected
  14.     procedure Grow; virtual;
  15.     procedure Put(Index: Integer; Item: Pointer);
  16.     procedure Notify(Ptr: Pointer; Action: TListNotification); virtual;
  17.     procedure SetCount(NewCount: Integer);
  18.   public
  19.     destructor Destroy; override;
  20.     Procedure AddList(AList : TFPList);
  21.     function Add(Item: Pointer): Integer;
  22.     procedure Clear; virtual;
  23.     procedure Delete(Index: Integer);
  24.     function Extract(item: Pointer): Pointer;
  25.     function GetEnumerator: TListEnumerator;
  26.     procedure Insert(Index: Integer; Item: Pointer);
  27.     procedure Assign (ListA: TFPList; AOperator: TListAssignOp=laCopy; ListB: TFPList=nil);
  28.     function Remove(Item: Pointer): Integer;
  29.     property Count: Integer read FCount { GetCount } write SetCount;
  30.     property Items[Index: Integer]: Pointer read Get write Put; default;
  31.   end;      
  32.  
or in  CCLasses
Code: Pascal  [Select]
  1.   { TFPObjectList }
  2.  
  3.   TFPObjectList = class(TFPList)
  4.   private
  5.     FFreeObjects : Boolean;
  6.   protected
  7.     function GetItem(Index: Integer): TObject; {$ifdef CCLASSESINLINE}inline;{$endif}
  8.     procedure SetItem(Index: Integer; AObject: TObject);
  9.   public
  10.     constructor Create;
  11.     constructor Create(FreeObjects : Boolean);
  12.     destructor Destroy; override;
  13.     procedure Clear;
  14.     procedure Delete(Index: Integer);
  15.     function Extract(Item: TObject): TObject; {$ifdef CCLASSESINLINE}inline;{$endif}
  16.     function Remove(AObject: TObject): Integer;
  17.  
  18.     function FindInstanceOf(AClass: TClass; AExact: Boolean; AStartAt: Integer): Integer;
  19.     function First: TObject; {$ifdef CCLASSESINLINE}inline;{$endif}
  20.     function Last: TObject; {$ifdef CCLASSESINLINE}inline;{$endif}
  21.     procedure Assign(Obj:TFPObjectList);
  22.     procedure ConcatListCopy(Obj:TFPObjectList);
  23.     procedure ForEachCall(proc2call:TObjectListCallback;arg:pointer); {$ifdef CCLASSESINLINE}inline;{$endif}
  24.     procedure ForEachCall(proc2call:TObjectListStaticCallback;arg:pointer); {$ifdef CCLASSESINLINE}inline;{$endif}
  25.     property OwnsObjects: Boolean read FFreeObjects write FFreeObjects;
  26.     property Items[Index: Integer]: TObject read GetItem write SetItem; default;
  27.   end;
  28.  
No parasitic FList co-class. This is object pascal !

I'm not much scared any more of trying things, I have fall back backups.

Thanks for your help.
Title: Re: Must Order of loading cmem/heaptrc
Post by: BrunoK on August 13, 2019, 04:42:36 pm
Don't mess with these things! The Windows RTL has a secondary entry point in Exec_Tls_Callback in rtl\win\systlsdir.inc which is called before the normal entry point and which might lead to allocations. It might not do so now, but it's not guaranteed to be this way in the future as well.

The only clean way is to integrate it directly into the System unit.
But cmem is an unit, and as such implicitly requires system, which means that system initializes before cmem?

Win32/64 FPC 3.0.4, the mac one with some other changes.

Done : got cmem + heaptrc loaded before system. As they are static units they do not require any other any other units resources before terminating heaptrc.
One bug found in heaptrc. Changed a bit Heap.inc (ReallocMem inconsistent).

Improvements :
Added a global serial to the heaptrc records.
Hooked (Win only) TObject.NewInstance so one can get the classname of the leaked structure.
Possibility to trail memory allocation via a personalized setable/resetable callback.
Corrected an error related to AllocMem.

Here after :

Output of attached test program

Code: Pascal  [Select]
  1. 0;01609C64;ansistring;getmem;size;100;serial;65
  2. 1;01609D34;ansistring;getmem;size;100;serial;66
  3. 2;01609E04;ansistring;getmem;size;100;serial;67
  4. 3;01609D34;ansistring;freemem;size;100;serial;66
  5. 4;01611C7C;unicodestring;getmem;size;180;serial;68
  6. 5;01611C7C;unicodestring;freemem;size;180;serial;68
  7. 6;01611C7C;unicodestring;getmem;size;180;serial;69
  8. 7;01611C7C;unicodestring;freemem;size;180;serial;69
  9. 8;01609D34;ansistring;getmem;size;100;serial;70
  10. 9;01609D34;ansistring;freemem;size;100;serial;70
  11. 10;01619C94;memory;getmem;size;164;serial;72
  12. 11;01619C94;memory;freemem;size;164;serial;72
  13. 12;01609D34;ansistring;getmem;size;100;serial;73
  14. 13;01609D34;ansistring;freemem;size;100;serial;73
  15. 14;01609C64;ansistring;freemem;size;100;serial;65
  16. 15;01609C64;ansistring;getmem;size;100;serial;74
  17. 16;01592AB4;ansistring;getmem;size;36;serial;75
  18. 17;01609C64;ansistring;freemem;size;100;serial;74
  19. 18;015A1DB4;ansistring;getmem;size;52;serial;76
  20. 19;0159941C;ansistring;getmem;size;84;serial;77
  21. 20;015B1C14;memory;getmem;size;100012;serial;78
  22. 21;015B1C14;memory;reallocmem;size;50012;serial;78
  23. 22;015B1C14;memory;freemem;size;50012;serial;78
  24. 23;015B1C04;TObject;getmem;size;4;serial;79
  25. 24;015B1C04;TObject;freemem;size;4;serial;79
  26. 25;015B1C04;TObject;getmem;size;4;serial;80
  27. 26;01621CAC;memory;getmem;size;132;serial;81
  28. 27;01629CC4;memory;reallocmem;size;388;serial;81
  29. 28;015F209C;TList;getmem;size;20;serial;82
  30. 29;015F209C;TList;freemem;size;20;serial;82
  31. 30;015F209C;TList;getmem;size;20;serial;83
  32. 31;015A1E54;TComponent;getmem;size;52;serial;84
  33. 32;015F211C;TFPList;getmem;size;20;serial;85
  34. 33;01592B44;ansistring;getmem;size;36;serial;86
  35. 34;015A1EF4;unicodestring;getmem;size;52;serial;87
  36. 35;015A1DB4;ansistring;freemem;size;52;serial;76
  37. 36;015F9DC4;ansistring;reallocmem;size;292;serial;77
  38. 37;01592BD4;ansistring;getmem;size;36;serial;88
  39. 38;01592AB4;ansistring;freemem;size;36;serial;75
  40. 39;015F9DC4;ansistring;freemem;size;292;serial;77
  41. 40;015A1DB4;ansistring;getmem;size;52;serial;89
  42. 41;0159941C;ansistring;getmem;size;84;serial;90
  43. Press ENTER to Quit
  44.  
Output of heaptrc with heap.inc
Code: Pascal  [Select]
  1. D:\fpc-laz-asus\Lazarus\bk_test\HeapTrcTest\HookHeap\HookHeap\prjHookHeap.exe
  2. prjHookHeap.exe HeapTrc START session : 13.08.2019 16:29:56
  3. Heap dump by heaptrc unit. Memory manager type : [mmtHeap]
  4. Max trace depth : 16
  5. 90 memory blocks allocated : 157152 / 177464
  6. 81 memory blocks freed     : 162420 / 175836
  7. 2 memory blocks used in System startup : 86
  8. 7 unfreed memory blocks : -5354
  9. Call trace for block $01581CB8 size 48  serial=87 unicodestring
  10.   $004087FC
  11.   $0040856D
  12.   $00401BC1 line 89 of prjHookHeap.lpr
  13. Call trace for block $01572908 size 32  serial=86 ansistring
  14.   $004075E0
  15.   $00406B55
  16.   $00401B8F line 85 of prjHookHeap.lpr
  17. Call trace for block $015D1EE0 size 16  serial=85 TFPList > TObject
  18.   $00429C76
  19.   $00401B67 line 83 of prjHookHeap.lpr
  20. Call trace for block $01581C18 size 52  serial=84 TComponent > TPersistent > TObject
  21.   $0042100F
  22.   $00401B56 line 82 of prjHookHeap.lpr
  23. Call trace for block $015D1E60 size 16  serial=83 TList > TFPList > TObject
  24.   $00429C76
  25.   $00401B3A line 81 of prjHookHeap.lpr
  26. Call trace for block $01609A88 size 213  serial=81 <getmem>
  27. Call trace for block $015919C8 size 4  serial=80 TObject
  28.   $00429C76
  29.   $00401AF0 line 76 of prjHookHeap.lpr
  30. prjHookHeap.exe HeapTrc END THeaptrcObject. : 13.08.2019 16:29:56
  31.  
Output of heaptrc with cmem
Code: Pascal  [Select]
  1. D:\fpc-laz-asus\Lazarus\bk_test\HeapTrcTest\HookHeap\HookHeap\prjHookHeap.exe
  2. prjHookHeap.exe HeapTrc START session : 13.08.2019 16:29:00
  3. Heap dump by heaptrc unit. Memory manager type : [mmtCmem]
  4. Max trace depth : 16
  5. 90 memory blocks allocated : 156624 / 169000
  6. 81 memory blocks freed     : 156157 / 167597
  7. 2 memory blocks used in System startup : 86
  8. 7 unfreed memory blocks : 381
  9. Call trace for block $05E602B4 size 48  serial=87 unicodestring
  10.   $0040883C
  11.   $004085AD
  12.   $00401C01 line 89 of prjHookHeap.lpr
  13. Call trace for block $05E6020C size 32  serial=86 ansistring
  14.   $00407620
  15.   $00406B95
  16.   $00401BCF line 85 of prjHookHeap.lpr
  17. Call trace for block $05E60174 size 16  serial=85 TFPList > TObject
  18.   $00429E56
  19.   $00401BA7 line 83 of prjHookHeap.lpr
  20. Call trace for block $05E600BC size 52  serial=84 TComponent > TPersistent > TObject
  21.   $004211EF
  22.   $00401B96 line 82 of prjHookHeap.lpr
  23. Call trace for block $05F84F5C size 16  serial=83 TList > TFPList > TObject
  24.   $00429E56
  25.   $00401B7A line 81 of prjHookHeap.lpr
  26. Call trace for block $05F84DFC size 213  serial=81 <getmem>
  27. Call trace for block $05F84D74 size 4  serial=80 TObject
  28.   $00429E56
  29.   $00401B30 line 76 of prjHookHeap.lpr
  30. prjHookHeap.exe HeapTrc END THeaptrcObject. : 13.08.2019 16:29:00
  31.  

Title: Re: [Done] Must Order of loading cmem/heaptrc
Post by: Thaddy on August 13, 2019, 06:29:08 pm
Does not compile on arm and crashes on win64 when -glh is used.

Can you add tests? I do not want to discourage you.

[edit]
on linux too.

It seems you have created something that only works for your special case?
https://blog.codinghorror.com/the-works-on-my-machine-certification-program/