Recent

Author Topic: [Done] Must Order of loading cmem/heaptrc  (Read 1183 times)

PascalDragon

  • Hero Member
  • *****
  • Posts: 712
  • Compiler Developer
Re: Order of loading cmem/heaptrc
« Reply #15 on: July 18, 2019, 09:42:05 am »
Sadly, the compiler insists on having  system beeing initialized BEFORE cmem than with heap.inc ...

You may want to try asking on the fpc-devel mail list.

But I am not sure it is possible. AFAIK system is always used. So even your unit is implicitly using system. Which means never mind how early you load your unit, it will load system before itself.

Correct.

If BrunoK wants to use the Windows heap manager directly for all allocations, including those in early startup than he needs to implement and set his heap manager directly in the System unit. This involves at least rtl\win32\system.pp, rtl\win64\system.pp, rtl\win\sysheap.inc and rtl\inc\heap.inc.

BrunoK

  • Full Member
  • ***
  • Posts: 191
  • Retired programmer
Re: Order of loading cmem/heaptrc
« Reply #16 on: July 18, 2019, 10:52:03 am »
If BrunoK wants to use the Windows heap manager directly for all allocations, including those in early startup than he needs to implement and set his heap manager directly in the System unit. This involves at least rtl\win32\system.pp, rtl\win64\system.pp, rtl\win\sysheap.inc and rtl\inc\heap.inc.
The idea is to stay near current operation.
I have already added a -dCMEM compiler option in TOption.interpret_option +                    include(init_settings.globalswitches,cs_use_cmem); { new cs_use_cmem in enum) +
in pmodules.pas loaddefaultunits
            { The correct heap manager should be setup BEFORE 'system' initializes
             but it is not the case yet ~bk }
           { Units only required for main module }
           if (cs_use_cmem in current_settings.globalswitches) then
              AddUnit('cmem');

that works without any trouble, but there is still this question of getting them in the correct order in fpc_InitializeUnits.

I'll find ...

My original post concerned only getting hints about that stuff, thats all.

Anyway thank you all, even the usual grumpy knows all.
Lazarus trunk r. 62137/27.10.2019 (+/- patches regarding TScrollBar, IntitalSetupDialog, Options.Environment options, SearchResults).  Lazarus 3.0.6 raw from svn.
FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903 / 18362.418)

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7603
Re: Order of loading cmem/heaptrc
« Reply #17 on: July 18, 2019, 11:27:46 am »
But cmem is an unit, and as such implicitly requires system, which means that system initializes before cmem?

PascalDragon

  • Hero Member
  • *****
  • Posts: 712
  • Compiler Developer
Re: Order of loading cmem/heaptrc
« Reply #18 on: July 19, 2019, 09:37:15 am »
If BrunoK wants to use the Windows heap manager directly for all allocations, including those in early startup than he needs to implement and set his heap manager directly in the System unit. This involves at least rtl\win32\system.pp, rtl\win64\system.pp, rtl\win\sysheap.inc and rtl\inc\heap.inc.
The idea is to stay near current operation.
I have already added a -dCMEM compiler option in TOption.interpret_option +                    include(init_settings.globalswitches,cs_use_cmem); { new cs_use_cmem in enum) +
in pmodules.pas loaddefaultunits
            { The correct heap manager should be setup BEFORE 'system' initializes
             but it is not the case yet ~bk }
           { Units only required for main module }
           if (cs_use_cmem in current_settings.globalswitches) then
              AddUnit('cmem');

that works without any trouble, but there is still this question of getting them in the correct order in fpc_InitializeUnits.
Don't mess with these things! The Windows RTL has a secondary entry point in Exec_Tls_Callback in rtl\win\systlsdir.inc which is called before the normal entry point and which might lead to allocations. It might not do so now, but it's not guaranteed to be this way in the future as well.

The only clean way is to integrate it directly into the System unit.

BrunoK

  • Full Member
  • ***
  • Posts: 191
  • Retired programmer
Re: Order of loading cmem/heaptrc
« Reply #19 on: July 19, 2019, 11:36:37 am »

Don't mess with these things! The Windows RTL has a secondary entry point in Exec_Tls_Callback in rtl\win\systlsdir.inc which is called before the normal entry point and which might lead to allocations. It might not do so now, but it's not guaranteed to be this way in the future as well.

The only clean way is to integrate it directly into the System unit.
You are alright about maybe messing with these things. I  won't  come complaining if it breaks and anyway it is my custom 3.0.6 version.

As an indication of what it has now, unretouched for the last 6 months, is for example :
TList = class(TFPList) refactoring. This and its descendant allocate only one INSTANCE for each TFPList descendent. That approach works without trouble (touch wood) both in cclasses and many List classes (at least those I understand what they do). Apart having only one object instance, as a by effect of that, one can inspect the list instance during debugging and see FCount an FCapacity which are always of interest for TFPList based instances.

That means that my TList is declared in lists.inc as
Code: Pascal  [Select]
  1.   TListEnumerator = TFPListEnumerator;
  2.  
  3.   TList = class(TFPList{ ,IFPObserved})
  4.   private
  5.     procedure CopyMove (aList : TFPList);
  6.     procedure MergeMove (aList : TFPList);
  7.     procedure DoCopy(ListA, ListB : TFPList);
  8.     procedure DoSrcUnique(ListA, ListB : TFPList);
  9.     procedure DoAnd(ListA, ListB : TFPList);
  10.     procedure DoDestUnique(ListA, ListB : TFPList);
  11.     procedure DoOr(ListA, ListB : TFPList);
  12.     procedure DoXOr(ListA, ListB : TFPList);
  13.   protected
  14.     procedure Grow; virtual;
  15.     procedure Put(Index: Integer; Item: Pointer);
  16.     procedure Notify(Ptr: Pointer; Action: TListNotification); virtual;
  17.     procedure SetCount(NewCount: Integer);
  18.   public
  19.     destructor Destroy; override;
  20.     Procedure AddList(AList : TFPList);
  21.     function Add(Item: Pointer): Integer;
  22.     procedure Clear; virtual;
  23.     procedure Delete(Index: Integer);
  24.     function Extract(item: Pointer): Pointer;
  25.     function GetEnumerator: TListEnumerator;
  26.     procedure Insert(Index: Integer; Item: Pointer);
  27.     procedure Assign (ListA: TFPList; AOperator: TListAssignOp=laCopy; ListB: TFPList=nil);
  28.     function Remove(Item: Pointer): Integer;
  29.     property Count: Integer read FCount { GetCount } write SetCount;
  30.     property Items[Index: Integer]: Pointer read Get write Put; default;
  31.   end;      
  32.  
or in  CCLasses
Code: Pascal  [Select]
  1.   { TFPObjectList }
  2.  
  3.   TFPObjectList = class(TFPList)
  4.   private
  5.     FFreeObjects : Boolean;
  6.   protected
  7.     function GetItem(Index: Integer): TObject; {$ifdef CCLASSESINLINE}inline;{$endif}
  8.     procedure SetItem(Index: Integer; AObject: TObject);
  9.   public
  10.     constructor Create;
  11.     constructor Create(FreeObjects : Boolean);
  12.     destructor Destroy; override;
  13.     procedure Clear;
  14.     procedure Delete(Index: Integer);
  15.     function Extract(Item: TObject): TObject; {$ifdef CCLASSESINLINE}inline;{$endif}
  16.     function Remove(AObject: TObject): Integer;
  17.  
  18.     function FindInstanceOf(AClass: TClass; AExact: Boolean; AStartAt: Integer): Integer;
  19.     function First: TObject; {$ifdef CCLASSESINLINE}inline;{$endif}
  20.     function Last: TObject; {$ifdef CCLASSESINLINE}inline;{$endif}
  21.     procedure Assign(Obj:TFPObjectList);
  22.     procedure ConcatListCopy(Obj:TFPObjectList);
  23.     procedure ForEachCall(proc2call:TObjectListCallback;arg:pointer); {$ifdef CCLASSESINLINE}inline;{$endif}
  24.     procedure ForEachCall(proc2call:TObjectListStaticCallback;arg:pointer); {$ifdef CCLASSESINLINE}inline;{$endif}
  25.     property OwnsObjects: Boolean read FFreeObjects write FFreeObjects;
  26.     property Items[Index: Integer]: TObject read GetItem write SetItem; default;
  27.   end;
  28.  
No parasitic FList co-class. This is object pascal !

I'm not much scared any more of trying things, I have fall back backups.

Thanks for your help.
Lazarus trunk r. 62137/27.10.2019 (+/- patches regarding TScrollBar, IntitalSetupDialog, Options.Environment options, SearchResults).  Lazarus 3.0.6 raw from svn.
FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903 / 18362.418)

BrunoK

  • Full Member
  • ***
  • Posts: 191
  • Retired programmer
Re: Must Order of loading cmem/heaptrc
« Reply #20 on: August 13, 2019, 04:42:36 pm »
Don't mess with these things! The Windows RTL has a secondary entry point in Exec_Tls_Callback in rtl\win\systlsdir.inc which is called before the normal entry point and which might lead to allocations. It might not do so now, but it's not guaranteed to be this way in the future as well.

The only clean way is to integrate it directly into the System unit.
But cmem is an unit, and as such implicitly requires system, which means that system initializes before cmem?

Win32/64 FPC 3.0.4, the mac one with some other changes.

Done : got cmem + heaptrc loaded before system. As they are static units they do not require any other any other units resources before terminating heaptrc.
One bug found in heaptrc. Changed a bit Heap.inc (ReallocMem inconsistent).

Improvements :
Added a global serial to the heaptrc records.
Hooked (Win only) TObject.NewInstance so one can get the classname of the leaked structure.
Possibility to trail memory allocation via a personalized setable/resetable callback.
Corrected an error related to AllocMem.

Here after :

Output of attached test program

Code: Pascal  [Select]
  1. 0;01609C64;ansistring;getmem;size;100;serial;65
  2. 1;01609D34;ansistring;getmem;size;100;serial;66
  3. 2;01609E04;ansistring;getmem;size;100;serial;67
  4. 3;01609D34;ansistring;freemem;size;100;serial;66
  5. 4;01611C7C;unicodestring;getmem;size;180;serial;68
  6. 5;01611C7C;unicodestring;freemem;size;180;serial;68
  7. 6;01611C7C;unicodestring;getmem;size;180;serial;69
  8. 7;01611C7C;unicodestring;freemem;size;180;serial;69
  9. 8;01609D34;ansistring;getmem;size;100;serial;70
  10. 9;01609D34;ansistring;freemem;size;100;serial;70
  11. 10;01619C94;memory;getmem;size;164;serial;72
  12. 11;01619C94;memory;freemem;size;164;serial;72
  13. 12;01609D34;ansistring;getmem;size;100;serial;73
  14. 13;01609D34;ansistring;freemem;size;100;serial;73
  15. 14;01609C64;ansistring;freemem;size;100;serial;65
  16. 15;01609C64;ansistring;getmem;size;100;serial;74
  17. 16;01592AB4;ansistring;getmem;size;36;serial;75
  18. 17;01609C64;ansistring;freemem;size;100;serial;74
  19. 18;015A1DB4;ansistring;getmem;size;52;serial;76
  20. 19;0159941C;ansistring;getmem;size;84;serial;77
  21. 20;015B1C14;memory;getmem;size;100012;serial;78
  22. 21;015B1C14;memory;reallocmem;size;50012;serial;78
  23. 22;015B1C14;memory;freemem;size;50012;serial;78
  24. 23;015B1C04;TObject;getmem;size;4;serial;79
  25. 24;015B1C04;TObject;freemem;size;4;serial;79
  26. 25;015B1C04;TObject;getmem;size;4;serial;80
  27. 26;01621CAC;memory;getmem;size;132;serial;81
  28. 27;01629CC4;memory;reallocmem;size;388;serial;81
  29. 28;015F209C;TList;getmem;size;20;serial;82
  30. 29;015F209C;TList;freemem;size;20;serial;82
  31. 30;015F209C;TList;getmem;size;20;serial;83
  32. 31;015A1E54;TComponent;getmem;size;52;serial;84
  33. 32;015F211C;TFPList;getmem;size;20;serial;85
  34. 33;01592B44;ansistring;getmem;size;36;serial;86
  35. 34;015A1EF4;unicodestring;getmem;size;52;serial;87
  36. 35;015A1DB4;ansistring;freemem;size;52;serial;76
  37. 36;015F9DC4;ansistring;reallocmem;size;292;serial;77
  38. 37;01592BD4;ansistring;getmem;size;36;serial;88
  39. 38;01592AB4;ansistring;freemem;size;36;serial;75
  40. 39;015F9DC4;ansistring;freemem;size;292;serial;77
  41. 40;015A1DB4;ansistring;getmem;size;52;serial;89
  42. 41;0159941C;ansistring;getmem;size;84;serial;90
  43. Press ENTER to Quit
  44.  
Output of heaptrc with heap.inc
Code: Pascal  [Select]
  1. D:\fpc-laz-asus\Lazarus\bk_test\HeapTrcTest\HookHeap\HookHeap\prjHookHeap.exe
  2. prjHookHeap.exe HeapTrc START session : 13.08.2019 16:29:56
  3. Heap dump by heaptrc unit. Memory manager type : [mmtHeap]
  4. Max trace depth : 16
  5. 90 memory blocks allocated : 157152 / 177464
  6. 81 memory blocks freed     : 162420 / 175836
  7. 2 memory blocks used in System startup : 86
  8. 7 unfreed memory blocks : -5354
  9. Call trace for block $01581CB8 size 48  serial=87 unicodestring
  10.   $004087FC
  11.   $0040856D
  12.   $00401BC1 line 89 of prjHookHeap.lpr
  13. Call trace for block $01572908 size 32  serial=86 ansistring
  14.   $004075E0
  15.   $00406B55
  16.   $00401B8F line 85 of prjHookHeap.lpr
  17. Call trace for block $015D1EE0 size 16  serial=85 TFPList > TObject
  18.   $00429C76
  19.   $00401B67 line 83 of prjHookHeap.lpr
  20. Call trace for block $01581C18 size 52  serial=84 TComponent > TPersistent > TObject
  21.   $0042100F
  22.   $00401B56 line 82 of prjHookHeap.lpr
  23. Call trace for block $015D1E60 size 16  serial=83 TList > TFPList > TObject
  24.   $00429C76
  25.   $00401B3A line 81 of prjHookHeap.lpr
  26. Call trace for block $01609A88 size 213  serial=81 <getmem>
  27. Call trace for block $015919C8 size 4  serial=80 TObject
  28.   $00429C76
  29.   $00401AF0 line 76 of prjHookHeap.lpr
  30. prjHookHeap.exe HeapTrc END THeaptrcObject. : 13.08.2019 16:29:56
  31.  
Output of heaptrc with cmem
Code: Pascal  [Select]
  1. D:\fpc-laz-asus\Lazarus\bk_test\HeapTrcTest\HookHeap\HookHeap\prjHookHeap.exe
  2. prjHookHeap.exe HeapTrc START session : 13.08.2019 16:29:00
  3. Heap dump by heaptrc unit. Memory manager type : [mmtCmem]
  4. Max trace depth : 16
  5. 90 memory blocks allocated : 156624 / 169000
  6. 81 memory blocks freed     : 156157 / 167597
  7. 2 memory blocks used in System startup : 86
  8. 7 unfreed memory blocks : 381
  9. Call trace for block $05E602B4 size 48  serial=87 unicodestring
  10.   $0040883C
  11.   $004085AD
  12.   $00401C01 line 89 of prjHookHeap.lpr
  13. Call trace for block $05E6020C size 32  serial=86 ansistring
  14.   $00407620
  15.   $00406B95
  16.   $00401BCF line 85 of prjHookHeap.lpr
  17. Call trace for block $05E60174 size 16  serial=85 TFPList > TObject
  18.   $00429E56
  19.   $00401BA7 line 83 of prjHookHeap.lpr
  20. Call trace for block $05E600BC size 52  serial=84 TComponent > TPersistent > TObject
  21.   $004211EF
  22.   $00401B96 line 82 of prjHookHeap.lpr
  23. Call trace for block $05F84F5C size 16  serial=83 TList > TFPList > TObject
  24.   $00429E56
  25.   $00401B7A line 81 of prjHookHeap.lpr
  26. Call trace for block $05F84DFC size 213  serial=81 <getmem>
  27. Call trace for block $05F84D74 size 4  serial=80 TObject
  28.   $00429E56
  29.   $00401B30 line 76 of prjHookHeap.lpr
  30. prjHookHeap.exe HeapTrc END THeaptrcObject. : 13.08.2019 16:29:00
  31.  

Lazarus trunk r. 62137/27.10.2019 (+/- patches regarding TScrollBar, IntitalSetupDialog, Options.Environment options, SearchResults).  Lazarus 3.0.6 raw from svn.
FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903 / 18362.418)

Thaddy

  • Hero Member
  • *****
  • Posts: 9284
Re: [Done] Must Order of loading cmem/heaptrc
« Reply #21 on: August 13, 2019, 06:29:08 pm »
Does not compile on arm and crashes on win64 when -glh is used.

Can you add tests? I do not want to discourage you.

[edit]
on linux too.

It seems you have created something that only works for your special case?
https://blog.codinghorror.com/the-works-on-my-machine-certification-program/
« Last Edit: August 13, 2019, 06:34:26 pm by Thaddy »
also related to equus asinus.