Recent

Author Topic: [Done] Must Order of loading cmem/heaptrc  (Read 1130 times)

BrunoK

  • Full Member
  • ***
  • Posts: 190
  • Retired programmer
[Done] Must Order of loading cmem/heaptrc
« on: July 17, 2019, 01:18:45 pm »
FPC 3.0.6 (derivative of //svn.freepascal.org/svn/fpc/tags/release_3_0_4a_macos_10_14).

Would someone have some idea where I should look/search in the compiler where to patch and so to be able to have cmem/heaptrc to be initialized BEFORE system.pp code :
  { Arguments }
  setup_arguments;

I want that because setup_arguments does use the default heap.inc memory manager and conflicts afterward with cmem's reallocmem / getmem / freemem.

Maybe someone knowledgable could PM me a rough idea how to do that.

Hopefully thanks ...
« Last Edit: August 13, 2019, 04:42:58 pm by BrunoK »
Lazarus trunk r. 62137/27.10.2019 (+/- patches regarding TScrollBar, IntitalSetupDialog, Options.Environment options, SearchResults).  Lazarus 3.0.6 raw from svn.
FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903 / 18362.418)

Thaddy

  • Hero Member
  • *****
  • Posts: 9191
Re: Order of loading cmem/heaptrc
« Reply #1 on: July 17, 2019, 01:23:46 pm »
See the wiki. Don't mix the two!! Ever.
https://wiki.freepascal.org/heaptrc and https://www.freepascal.org/docs-html/rtl/heaptrc/usage.html

There is a way mentioned as a note on how to select either cmem or heaptrc:
Code: Pascal  [Select]
  1. program test;
  2. {$mode objfpc}
  3. uses
  4. // This will conditionally switch in a memory manager
  5. // based on the presence of heaptrc
  6.  {$if not declared(UseHeapTrace)}cmem;{$ifend}
  7. begin
  8. {$if declared(UseHeapTrace)}
  9.    writeln('Heaptrc is used.',' Heaptrc is active? ', UseHeaptrace);
  10. {$else}
  11.    writeln('No trace of heaptrc');
  12. {$ifend}
  13. end.

The ratio is that heaptrc is also a memory manager and you can't have two without creating problems.

( A sick option is to rename cmem to heaptrc and rebuild, that would insert cmem instead of heaptrc when compiled with -gh   :o 8-) )
« Last Edit: July 17, 2019, 01:38:01 pm by Thaddy »
also related to equus asinus.

BrunoK

  • Full Member
  • ***
  • Posts: 190
  • Retired programmer
Re: Order of loading cmem/heaptrc
« Reply #2 on: July 17, 2019, 01:37:51 pm »
See the wiki. Don't mix the two!! Ever.
Yes, I know that bloody well, my new heaptrc WITH cmem nearly works but bangs because HEAP.INC is USED BEFORE (in ' { Arguments } setup_arguments;') having had the opportunity to install cmem.

Why use cmem ?

because it seems much faster in multihtreading applications.
Lazarus trunk r. 62137/27.10.2019 (+/- patches regarding TScrollBar, IntitalSetupDialog, Options.Environment options, SearchResults).  Lazarus 3.0.6 raw from svn.
FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903 / 18362.418)

Thaddy

  • Hero Member
  • *****
  • Posts: 9191
Re: Order of loading cmem/heaptrc
« Reply #3 on: July 17, 2019, 01:42:40 pm »
See the wiki. Don't mix the two!! Ever.
Yes, I know that bloody well, my new heaptrc WITH cmem nearly works but bangs because HEAP.INC is USED BEFORE (in ' { Arguments } setup_arguments;') having had the opportunity to install cmem.

Why use cmem ?

because it seems much faster in multihtreading applications.

See my last line above. The sick option part..
You are wrong in assuming it is faster. cmem does simply larger allocations, which can easily cause problems in long running applications. It looks faster, but not always and you should test.
Technically cmem is a bare and very basic OS heap based manager. The default memory manager is more advanced than that.
With threads, you need scaling. cmem does not scale at all.
also related to equus asinus.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5714
    • wiki
Re: Order of loading cmem/heaptrc
« Reply #4 on: July 17, 2019, 01:43:45 pm »
use -Fa command line option. From docs:
  -Fa<x>[,y] (for a program) load units <x> and [y] before uses is parsed

Thaddy

  • Hero Member
  • *****
  • Posts: 9191
Re: Order of loading cmem/heaptrc
« Reply #5 on: July 17, 2019, 01:49:34 pm »
use -Fa command line option. From docs:
  -Fa<x>[,y] (for a program) load units <x> and [y] before uses is parsed
That does not work if heaptrc is selected, though. cmem will still be loaded after heaptrc. Still the same problem about using two memory managers.
also related to equus asinus.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5714
    • wiki
Re: Order of loading cmem/heaptrc
« Reply #6 on: July 17, 2019, 02:38:47 pm »
That does not work if heaptrc is selected, though. cmem will still be loaded after heaptrc. Still the same problem about using two memory managers.
He does not say he wants both at the same time. From what I read he either wants:
- one of them (whichever he chooses) to be loaded
- a mem-mgr of his own (which is a merge of the 2) to be loaded.

So in any case just one mem-mgr.

At least that his how I read his posts

BrunoK

  • Full Member
  • ***
  • Posts: 190
  • Retired programmer
Re: Order of loading cmem/heaptrc
« Reply #7 on: July 17, 2019, 02:45:42 pm »

So in any case just one mem-mgr.

At least that his how I read his posts

YES, exactly, full on target as usual !
Lazarus trunk r. 62137/27.10.2019 (+/- patches regarding TScrollBar, IntitalSetupDialog, Options.Environment options, SearchResults).  Lazarus 3.0.6 raw from svn.
FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903 / 18362.418)

BrunoK

  • Full Member
  • ***
  • Posts: 190
  • Retired programmer
Re: Order of loading cmem/heaptrc
« Reply #8 on: July 17, 2019, 03:01:22 pm »
use -Fa command line option. From docs:
  -Fa<x>[,y] (for a program) load units <x> and [y] before uses is parsed
Thanks for the tip.

Sadly, the compiler insists on having  system beeing initialized BEFORE cmem than with heap.inc ...

Will dig further.


See my last line above. The sick option part..
You are wrong in assuming it is faster. cmem does simply larger allocations, which can easily cause problems in long running applications. It looks faster, but not always and you should test.
see https://forum.lazarus.freepascal.org/index.php/topic,44590.0.html.
What took it to the top was using cmem.

I can  see it also in wp's mapviewer that leaks. A thing is sure is that it leaks MUCH faster with cmem ...

Quote
Technically cmem is a bare and very basic OS heap based manager. The default memory manager is more advanced than that.
With threads, you need scaling. cmem does not scale at all.
cmem uses msvcrt in win10 and seems to be a tcmalloc allocator. tcmalloc looks like a subtle piece of software. That heap allocator has much evolved in the last 6 years and seems to be the mainstream heap allocator whereas HEAP.INC doesn't seem to have evolved in the last 10 years.
There are quite a few inconsistencies in heap.inc (those I took care of).

As for memory utilization, the W10 task manager shows both fpc and lazarus using less memory when build with cmem.
Lazarus trunk r. 62137/27.10.2019 (+/- patches regarding TScrollBar, IntitalSetupDialog, Options.Environment options, SearchResults).  Lazarus 3.0.6 raw from svn.
FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903 / 18362.418)

Thaddy

  • Hero Member
  • *****
  • Posts: 9191
Re: Order of loading cmem/heaptrc
« Reply #9 on: July 17, 2019, 04:17:36 pm »
Why not link in tcmalloc directly then?
Note msvcrt can have multiple versions depending on system and - although I have a development build of Windows for 15+ years, I am not aware recent  msvcrt versions have changed their malloc implementation..
The original Google version of tcmalloc can be linked statically too. https://github.com/gperftools/gperftools
It is also cross-platform, which msvcrt definitely is not!

Note that if Google themselves say it is the fastest they have seen, that is most likely correct.
« Last Edit: July 17, 2019, 04:41:29 pm by Thaddy »
also related to equus asinus.

Martin_fr

  • Administrator
  • Hero Member
  • *
  • Posts: 5714
    • wiki
Re: Order of loading cmem/heaptrc
« Reply #10 on: July 17, 2019, 04:55:26 pm »
Sadly, the compiler insists on having  system beeing initialized BEFORE cmem than with heap.inc ...

You may want to try asking on the fpc-devel mail list.

But I am not sure it is possible. AFAIK system is always used. So even your unit is implicitly using system. Which means never mind how early you load your unit, it will load system before itself.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 7508
Re: Order of loading cmem/heaptrc
« Reply #11 on: July 17, 2019, 05:26:22 pm »
cmem is mostly meant for *nix, e.g. to use valgrind.

I don't think this is really tested functionality on Windows, specially in combination with heaptrc.




Thaddy

  • Hero Member
  • *****
  • Posts: 9191
Re: Order of loading cmem/heaptrc
« Reply #12 on: July 17, 2019, 05:32:31 pm »
cmem is mostly meant for *nix, e.g. to use valgrind.

I don't think this is really tested functionality on Windows, specially in combination with heaptrc.

It has been used for many years It has the same interface on msvcrt. It is basically a set of stdc library calls, so support by all - most - C compilers. Another example is heapmm from KOL which uses the same calls.
The combination, though, is out of the question. It is a matter of either/or
also related to equus asinus.

BrunoK

  • Full Member
  • ***
  • Posts: 190
  • Retired programmer
Re: Order of loading cmem/heaptrc
« Reply #13 on: July 17, 2019, 05:34:25 pm »
You may want to try asking on the fpc-devel mail list.

But I am not sure it is possible. AFAIK system is always used. So even your unit is implicitly using system. Which means never mind how early you load your unit, it will load system before itself.
Thanks for your help.

I think I fingered where I can try to do that.  Will try tomorrow.

Cheers, Bruno

I don't think this is really tested functionality on Windows, specially in combination with heaptrc.
I know it works because I run both FPC and Lazarus with cmem for the last 3 months, not just 5 minutes sometimes, but for hours many days in a week.

Of course it doesn't  work with the current version of heaptrc but it nearly works with mine.
« Last Edit: July 17, 2019, 05:36:28 pm by BrunoK »
Lazarus trunk r. 62137/27.10.2019 (+/- patches regarding TScrollBar, IntitalSetupDialog, Options.Environment options, SearchResults).  Lazarus 3.0.6 raw from svn.
FPC 3.0.4 32 bits. (+heaptrc with leaked ClassName+Revisited TList) , Windows 10 Pro x64 (v. 1903 / 18362.418)

Akira1364

  • Hero Member
  • *****
  • Posts: 539
Re: Order of loading cmem/heaptrc
« Reply #14 on: July 17, 2019, 05:48:14 pm »
I don't think this is expected / supposed to work, as currently implemented. heaptrc.pp is itself a
"memory manager unit" in the same way that cmem.pp is.

Both always replace the default memory manager with their own implementations ("CMemoryManager" in the case of cmem, and "TraceManager" in the case of heaptrc) and as such it is not possible for them to co-exist.

If you want something that is still able to report memory usage the way heaptrc does, but calls into some external malloc implementation, the best way to go would be with a modified version of the heaptrc unit that explicitly uses the external API for the actual allocation / deallocation routines, which will generally return the same kind of values allowing it to track them the same way.

As far as whether CMem specifically is generally useful on Windows though, it definitely does massively improve the performance of multi-threaded applications (on at least 64-bit, not sure about 32-bit.)
« Last Edit: July 17, 2019, 06:31:31 pm by Akira1364 »