Strange memory allocation problem

[marcov]

Cmem has more workarounds for bad allocation schemes (mass allocating in a thread and releasing in another thread). However the default distro linux one uses a bit more memory overhead iirc.

Relying on cmem on a particular system might cause problems on the next (e.g. OS X or embedded Linux etc). It only masks problems.

Better is to have

• some guard against reallocating for every element (e.g. once every n elements growth).  Pooling repeated large operations yourself (easy with a generic pool class)
• If you employ an exponential growth scheme, make sure it is capped at some top value (typically hundreds of MBs worth, and then stop growing exponentially, but 10% or so.
• Don't rely too much on linear container types (like tstringlist) with very high counts

[MarkMLl]

If it was up to me I would never use dynamic arrays, especially big and growing ones. I would declare a static array which is big enough and an additional length variable. Every time you grow a dynamic array you'll leave a gap which cannot satisfy the further growth without merging, you can have other small allocations that prevents free blocks from merging, etc.

And if the heap manager fragments it's reasonable to assume that it's not deeply integrated with the OS, and that it's forced to copy data around whenever it resizes. That has a significant speed penalty, plus can also leave unwiped data in RAM which is a potential security hazard (which obviously also applies to strings).

MarkMLl

[marcov]

And if the heap manager fragments it's reasonable to assume that it's not deeply integrated with the OS, and that it's forced to copy data around whenever it resizes. That has a significant speed penalty, plus can also leave unwiped data in RAM which is a potential security hazard (which obviously also applies to strings).

Most do copy around afaik, since blocks won't be page aligned etc.  I think it is more a tunable, how much extra space do you allocate/leave for future expansion of a block?  Space/speed tradeoff......

In this case cmem wins, but it might use more memory overall.

[MarkMLl]

Most do copy around afaik, since blocks won't be page aligned etc.  I think it is more a tunable, how much extra space do you allocate/leave for future expansion of a block?  Space/speed tradeoff......

If the allocation were being done at the level of a paging OS and was recognised as being e.g. more than a certain size, then in principle at least it should be possible to base it on a page boundary and to only have to copy the content of the highest page when necessary. I don't know whether that can be done by an application program without recourse to kernel facilities... which of course will vary enormously depending on the OS.

Systems with multiple heaps (for small and large allocations) go back to Knuth: he either wrote one for Burroughs or borrowed their implementation for TAoCP.

MarkMLl

[LV]

“For any possible allocation algorithm, there exists a stream of allocation and deallocation requests that defeat the allocator and force it into severe fragmentation.” — Robson.

https://github.com/pbrinkmeier/kit-lecture-notes/blob/master/betriebssysteme/14-memory-allocation.md

[MathMan]

“For any possible allocation algorithm, there exists a stream of allocation and deallocation requests that defeat the allocator and force it into severe fragmentation.” — Robson.

https://github.com/pbrinkmeier/kit-lecture-notes/blob/master/betriebssysteme/14-memory-allocation.md

Indeed  And that's why for memory intensive programs/applications usually a thorough memory requirements analysis should be done ...

You mentioned earlier that thread starters program seems to fall into a specific category and I fully support that TS should look for secondary implementations/literature to see how his memory management may be improved.