Heaptrace report missing details

[kaapte]

Hi all,

I have a multithreaded FPC app running on linux. It's permanently running and over the course of the days and weeks, memory usage will suddenly jump up some 100MB or 200MB and never go back down. So I decided to try Heaptrace to find the cause of this.
Usually I use mORMot's fpcx64mm memory manager. The Heaptrace wiki says one should not use any other memory manager when using Heaptrace, so I removed it. However I noticed, memory usage is going up rapidly then, I was at 700MB after 30 minutes and it kept going up. Is that normal? Is Heaptrace collecting data and keep it in memory?

So to my actual issue: I set

Code: Pascal  [Select][+][-]

1. export HEAPTRC="log=heap.trc"

to write the Heaptrace report into a file. That works, but it does not print any details about where in my source code the problems might be. See this report which shows a lot of unfreed memory blocks, but there is no more information. What might be the issue here?
I used these options for compiling: -dDEBUG -MDelphi -gl -gp -gw3 -gh

Thanks!

Heap dump by heaptrc unit of app
1268968142 memory blocks allocated : 154865380079/160289538184
1268845654 memory blocks freed     : 154839922875/160263834208
122488 unfreed memory blocks : 254

[Thaddy]

-dDEBUG -MDelphi -gl -gp -gw3 -gh
Let's examine this.
1. -dDEBUG
How is your debug mode defined in fpc.cfg?
2. -MDelphi
should be -Mdelphi
3. -gl
That is ok.
4. -gp
you are not using stabs but dwarf. Makes no sense.
5. -gw3
This is already the default (in 3.2.2), so you can omit it.
6. -gh
the -g switch can be combined, so make it just one: -glh

But indeed, heaptrc gathers the information in memory and reports on program finished.
Since I did not see a whole of the heaptrc log:
It should report line numbers etc.
But what you have got is a big gaping hole that should be spotted by just reading the code.
(you have read the wiki, and understood that heaptrc should not be added by hand? just -glh will do )
First hang back and look what can cause such massive leak, if you can not find it report back. It should be visible by eye and heaptrc log should identify line numbers, except if you have many inc files. In that case it will be off by some lines.

[kaapte]

Thanks for your response.

1. -dDEBUG
How is your debug mode defined in fpc.cfg?

I don't have a fpc.cfg. So it must use some default mode, I'm not sure which options there are? I also added the -n parameter to make sure, there are no weird things inside a fpc.cfg somewhere.

2. -MDelphi
should be -Mdelphi

Alright, i changed that. However the delphi mode has been working already.

4. -gp
you are not using stabs but dwarf. Makes no sense.

Ok, I will remove that one and try again.

Since I did not see a whole of the heaptrc log:
It should report line numbers etc.

Yes you did see the whole heaptrace log. This is the main issue I'm facing now, that it only just writes these 4 lines into the report and no details like line numbers. Maybe it's just too much information (122488 unfreed memory blocks) and that causes an error. Or some other error preventing it to write the details. I will try a really short session to see if it will output details then.

But what you have got is a big gaping hole that should be spotted by just reading the code.
(you have read the wiki, and understood that heaptrc should not be added by hand? just -glh will do )
First hang back and look what can cause such massive leak, if you can not find it report back. It should be visible by eye and heaptrc log should identify line numbers, except if you have many inc files. In that case it will be off by some lines.

Yes, I noticed memory usage is continuously going up. That is only the case when using heaptrace and disabling mORMot's fpcx64mm memory manager (which is required according to the wiki).
This is a rather complex program and the memory leak usually happens within a single incident in varying time spans (like 1 week it's running at 350MB RAM and then suddenly jump to 450MB, then remain at 450MB for 2 days and then jump to 600MB for example). I'm not able to tell where the leak is, just by looking at the code, I'm afraid
Yes I read the wiki and did not add the heaptrace unit manually.

[DavidL]

I've had to deal with a couple of large C++ programs that exhibited similar behavior.  In both cases it turned out that the OO kiddies didn't realize their "collection classes" chewed up increasingly larger chunks of memory when they hit their growth boundaries over extended run times.  Just a thought...

[Thaddy]

Thanks for your response.

1. -dDEBUG
How is your debug mode defined in fpc.cfg?

I don't have a fpc.cfg. So it must use some default mode, I'm not sure which options there are? I also added the -n parameter to make sure, there are no weird things inside a fpc.cfg somewhere.

There are no defaults for the debug conditional. The debug settings always come from an fpc.cfg but it may have a different cfg name (although few people use that feature) Or the debug settings may be defined on the commandline of course, but not as -dDEBUG. Which will do nothing if there is no config....

Since you are on Linux, try to set up compilation for valgrind with -gv and use valgrind with a lot of verbosity. And/or use -vv which writes an fpcdebug.txt file with lots of additional info.
There should be an fpc.cfg in e.g. /etc/fpc.cfg if it is not under /usrl/bin/fpc/<version> or /usr/local/bin/fpc/<version>. The one in /etc/ is sticky and often overlooked.
I am almost sure that if you do cat /etc/fpc.cfg you will see a config file.....

[kaapte]

Oh I completely missed the one in /etc/fpc.cfg.
There it says:

Code: [Select]

#IFDEF DEBUG
  -gl
  -Crtoi
  #WRITE Compiling Debug Version
#ELSE
  # Strip debuginfo from the executable if not in debug mode
  -Xs
#ENDIF
I was trying to avoid using Valgrind because it's limiting the program to only using 1 thread/core. The program is highly multithreaded and I was afraid this would probably mean the issue might not even come up when it's only running 1 thread at a time.
However I will also give Valgrind a try.

Thanks for your help, guys. I will post any news here.

[Thaddy]

I was trying to avoid using Valgrind because it's limiting the program to only using 1 thread/core.

That is not true, Valgrind will happily target multiple threads.
With the provision that Valgrind will analyze them sequentially, that is true.

[PascalDragon]

2. -MDelphi
should be -Mdelphi

This is irrelevant, because the mode(switch) name is case insensitive.