Finishing thread does not release Virtual memory

[jollytall]

I have an application that makes new threads all the time. They finish normally, no memory leak, etc..
Still the Virtual memory usage of the process is constantly growing. I read many articles and it seems to be the normal process, as they say, because calling the virtual memory deallocation is apparently an expensive process. So what happens, every time when a new thread is launched it reserves memory in the virtual space and when it terminates it releases all physical space, but not the virtual. The next process could maybe use the "abandoned" virtual memory (it is the same structure thread again), but instead it reserves a new virtual address space.
Since on x64 the space is large enough, it is not a frequent problem to deal with, but it is alarming to see that the virtual memory usage goes up fast and I also see some small increase in physical memory usage (maybe for the allocation table).
I would like to somehow release the virtual memory, even if it is a bit expensive.

Somewhere I read a topic about detaching the thread from the process and let it die alone and apparently in such a case the virtual memory reservation "dies" with the left alone thread. I am not sure it is possible, exists, but surely could not find a "DetachThread" procedure (C++ has it).

For information: I use Linux/Debian10 with Lazarus 2.0.12, FPC 3.2.0.

I know I could use a pool of threads to avoid this problem, but prefer not to rewrite the whole application if I can solve it in the current set-up.

[marcov]

The classic cause is memory fragmentation.  A new block is allocated by the application, but it is larger than the empty spaces the heapmanager has, so it allocates it from the OS.

The solution is usually to pool large allocations manually, or break them up in smaller pieces.

[Jonas Maebe]

If you set FreeOnTerminate of your TThread to true, it will run as a detached thread. Warning: in that case you cannot/must not use TThread.WaitFor on it (partly because in theory the thread could even end before you started waiting on it). You will need to implement your own synchronisation if you need similar functionality (e.g. by setting an event from the thread when it is about to finish).

[jollytall]

@marcov,
I think it is unlikely. I call the same object type with the same create parameters many times. They should allocate the same memory, so it should fit in to an earlier one. Even if they are slightly different size (for whatever reason), it is unlikely that when I do it for the 100th time, none of the previous 99 is the same size or larger to be reused. I think that the system does not look for virtual memory blocks allocated but not used by the same process, once a new thread is created.

@Jonas
Thanks, it might be leading to the solution. Two problems:
I do not use TThread, but the one level lower BeginThread, so I would need to call whatever has to be called manually. It would not be a problem if I knew what to call. I also have my own Wait implementation, so that is not a problem either.
I check the TThread implementation and FFreeOnTermiante I found only once in ThreadProc, and it calls Free (i.e. Destroy), but in Destroy I did not find what to do. Can you help?

[MarkMLl]

I would like to somehow release the virtual memory, even if it is a bit expensive.

On Linux,  swapoff -a  should do it.

Yes, it's extreme. But it should have the effect of (a) forcing everything briefly into real memory and (b) in order to do this minimising the amount of demand-loaded executable in memory. You probably want to run  swapon -a  shortly afterwards.

Apart from that, listen to Jonas et al.

MarkMLl

[jollytall]

It does not seem to work, and if I understand it correctly, it not even should. Isn't swapoff/swapon control the allocation of physical memory between RAM and Swap, while it has nothing to do with Virtual memory?

[MarkMLl]

It does not seem to work, and if I understand it correctly, it not even should. Isn't swapoff/swapon control the allocation of physical memory between RAM and Swap, while it has nothing to do with Virtual memory?

Swap is normally referred to as virtual memory, or are you talking about something different?

MarkMLl

[BrunoK]

Maybe try using cmem and see how memory usage vary ?

Of course, no heaptrc possible.

[jollytall]

Swap is normally referred to as virtual memory, or are you talking about something different?
MarkMLl

Swap (or to be more precise Swap + RAM) is often referred as virtual memory. But this is not what Linux (e.g. in Top) calls Virtual memory. Nowadays VIRT can be much larger than the available physical memory including RAM and swap. I found a good description https://serverfault.com/questions/138427/what-does-virtual-memory-size-in-top-mean explaining how it can happen. Also https://man7.org/linux/man-pages/man1/top.1.html has a paragraph on Linux memory types.

The problem to reserve a virtual memory space still reserves (I think) an entry in the allocation table and so the table slowly grows, so over a longer period can cause OOM. (And also the peace of my mind not seeing a number growing permanently beyond any limit.)

[jollytall]

Maybe try using cmem and see how memory usage vary ?

Of course, no heaptrc possible.

I tried adding cmem, but the VIRT usage grows just the same.

[Martin_fr]

They finish normally, no memory leak, etc..

Or none that you know of....

Run your app under valgrind.
Compile with debug info enabled: dwarf  (ideally enabled for packages too, use "Additions and Overrides" to add -gw )

Code: Text  [Select][+][-]

1. valgrind --tool=memcheck  ./project1

[Martin_fr]

Also, its possible that memory is allocated, and only freed when your app shuts down => no memleak, but growing mem usage.

E.g. if you add objects to a global list, and you (including any unit used by you) free all of it in a "finalization" block.

[jollytall]

Thanks Martin,

Honestly, I never used vargrind before and it does not seem to be an easy tool. I get many errors and possibly lost and still reachable bytes. E.g. a summary:
==27332== LEAK SUMMARY:
==27332==    definitely lost: 0 bytes in 0 blocks
==27332==    indirectly lost: 0 bytes in 0 blocks
==27332==      possibly lost: 608 bytes in 2 blocks
==27332==    still reachable: 15,497 bytes in 149 blocks
==27332==         suppressed: 0 bytes in 0 blocks
I need more time to understand it, but most of them are in libcrypto (openssl) and libc. I don't know what I can do with them, and whether it can keep the reserved large virtual memory locked for few Kbyte of still reachable memory.

It will be a long task to figure it all out... Maybe it is easier to cron a daily restart and the problem is gone :-), but I do not like a solution like that.

[marcov]

@marcov,
I think it is unlikely. I call the same object type with the same create parameters many times. They should allocate the same memory, so it should fit in to an earlier one

Unless a smaller came inbetween and partially filled the  large hole, so that the now smaller hole doesn't fit the next major allocation anymore . Anyway, try to recycle some larger memory allocations e.g. using a tthreadlist), and see if that changes behaviour.

[Thaddy]

Wait..wait..waitfor! Use waitfor as already mentioned.