A bug or a feature? SysAllocMem calls MemoryManager.GetMem instead of SysGetMem

[chebmaster]

I am confused.
I tried creating a wrapper memory manager for debugging purposes (i.e. a memory manager that expands on the system one by increasing allocated block sizes, thus adding some extra data to the allocated blocks)
It failed craptacularly because, for example, SysAllocMem() calls not on SysGetMem as one would expect but on MemoryManager.GetMem  - which I already redefined and which adds that record and performs pointer arithmetic before calling SysGetMem - i.e. the block gets added twice, then FreeMem comes trying to untangle tham mess and the everything goes to hell. 
Sadly that wasn't t the only place the MemoryManager.xxx methods get called.

So, my question is: is this done for some compatibility reason, or is this an oversight?

Because as it is now, the only way to get a modded memory manager is to write one from scratch

[marcov]

The Sys variants are afaik the default  implementations for the memorymanager records. They are afaik meant to be replaced en-bloc by another memory manager implementation, and their internals are not documented?

Did you maybe replace getmem in the memory manager, but not allocmem in the memory manager?

[chebmaster]

I surely replaced them *all*.
The problem is that I replaced them with wrappers that called upon them (OldMM.AllocMem or SysAllocMem - it makes no difference). But they, in turn, were calling upon my wrappers instead of calling on each other. Which is fundamentally wrong.

So I am trying to understand: why?
Is there an obscure but legitimate reason, or is this a genuine bug?

I tried hacking around that (see SysGetMem below), but SysReallocMem is too complex for it to work

Code: Pascal  [Select][+][-]

1.   function mml_AllocMem(Size : ptruint) : Pointer;
2.   begin
3.     EnterCriticalSection(CS);
4.     Result:= MyAllocMem(Size);
5.     LeaveCriticalSection(CS);
6.   end;
7.  
8.   function MyAllocMem  (Size : ptruint) : Pointer;
9.   var
10.     b: PMemBlockRec;
11.     newallocsize: ptruint;
12.   begin
13.       if Size = 0 then begin
14.         Exit(nil);
15.       end;
16.  
17.       newallocsize:= Size + sizeof(TMemBlockRec);
18.       b:= SysGetMem(newallocsize);
19.       FillChar(b^, newallocsize, 0);
20.       b^.sig:= defsig;
21.       b^.size:= Size;
22.       b^.alloc_size:= newallocsize;
23.       b^.ThreadId:= GetCurrentThreadID;
24.       if not deactivated then begin
25.         b^.prev:= @StubBlock;
26.         b^.next:= StubBlock.next;
27.         StubBlock.next:= b;
28.         if Assigned(b^.next) then b^.next^.prev:= b;
29.         GetBacktrace(b);
30.       end;
31.       Result:= pointer(ptruint(b) + sizeof(TMemBlockRec));
32.   end;

[chebmaster]

P.S.

Code: Pascal  [Select][+][-]

1.   Function mml_FreeMem (P : pointer) : ptruint;
2.   begin
3.     EnterCriticalSection(CS);
4.     Result:= MyFreeMem(p);
5.     LeaveCriticalSection(CS);
6.   end;
7.  
8.   Function mml_FreeMemSize(p:pointer;Size:ptruint):ptruint;
9.   begin
10.     Result:= mml_FreeMem(p);
11.   end;
12.  
13.   Function myFreeMem (P : pointer) : ptruint;
14.   var
15.     b: PMemBlockRec;
16.     alien: boolean = true;
17.     newallocsize: ptruint;
18.   begin
19.       if not Assigned(p) then begin
20.         Exit(0);
21.       end;
22.       b:= PMemBlockRec(pointer(ptruint(p) - sizeof(TMemBlockRec)));
23.       alien:= b^.sig <> defsig;
24.       if alien then begin
25.         //alien block allocated prior to installation of my memory manager
26.         myFreeMem:= SysFreeMem (p);
27.         Exit
28.       end;
29.       if Assigned(b^.next) then b^.next^.prev:= b^.prev;
30.       if Assigned(b^.prev) then b^.prev^.next:= b^.next;
31.       b^.prev:= nil;
32.       b^.next:= nil;
33.       myFreeMem:= b^.size;
34.       SysFreeMemSize(b, b^.alloc_size);
35.   end;

[Cyrax]

I would not recommend to use directly Sys* variants of memory allocation functions. Rather make a backup of previous memory manager (GetMemoryManager) and use it extensively in your own manager by redirecting calls to old manager with necessary modifications.

[chebmaster]

make a backup of previous memory manager (GetMemoryManager) and use it extensively in your own manager by redirecting calls to old manager with necessary modifications

Have you read my posts at all?
I DID EXACTLY THAT.
IT DOESN'T WORK.
Because the Sys*** functions call MemoryManager.*** functions, not their fellow Sys*** functions.

Please stop giving me noob-targeted answers. I have more than 20 years of experience using Pascal, since Turbo 5.0 / Delphi 2.0 / FPC 1.X

So. To anyone who can understand the topic: Is that a bug or is that an obscure feature?

P.S. Sister topic in Russian: http://freepascal.ru/forum/viewtopic.php?f=1&t=18231

[Cyrax]

Take chill pill, dude. With that kind of attitude, you will not find help here in the future.

I take that this code example won't work because of reasons?

Code: Pascal  [Select][+][-]

1. program Project1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. uses
6.   {$IFDEF UNIX}{$IFDEF UseCThreads}
7.   cthreads,
8.   {$ENDIF}{$ENDIF}
9.   Classes
10.   { you can add units after this };
11.  
12. Var
13.   AOldMemoryManager : TMemoryManager;
14.  
15. Function MyGetmem(Size:ptruint):Pointer;
16. Var
17.   AMemory : PByte;
18. begin
19.   AMemory := AOldMemoryManager.Getmem(Size + 100);
20.   Inc(AMemory, 100);
21.   Result := AMemory;
22. end;
23.  
24. Function MyFreeMem(p:pointer):ptruint;
25. Var
26.   AMemory : PByte;
27. Begin
28.   AMemory := P;
29.   Dec(AMemory, 100);
30.   Result := AOldMemoryManager.FreeMem(AMemory);
31. end;
32.  
33. Var
34.   ANewMemoryManager : TMemoryManager;
35.   AMemory : Pointer;
36. begin
37.   System.GetMemoryManager(AOldMemoryManager);
38.   ANewMemoryManager := AOldMemoryManager;
39.  
40.   ANewMemoryManager.Getmem := @MyGetmem;
41.   ANewMemoryManager.FreeMem := @MyFreeMem;
42.  
43.   System.SetMemoryManager(ANewMemoryManager);
44.   GetMem(AMemory, 10);
45.   FreeMem(AMemory);
46.  
47.   System.SetMemoryManager(AOldMemoryManager);
48. end.
49.  

[chebmaster]

I take that this code example won't work because of reasons?

Yeah 
It would have worked IF SysReallocMem only used MemoryManager.GetMem/FreeMem/MemSize. Alas, it also uses  SysTryResizeMem.
In short, it mixes calls to my wrapper functions that work in the transformed space with calls to routines that should work with original space. Nothing but mess can result.

A wrapper would only work if all it did was non-intrusive stuff like counting number of deallocations during certain phase of the Moon

Take chill pill, dude.

I try, but "intellectual" replies like "Try adding 2 and 2! It's 4, see?" make ANGRY

[molly]

In that case, something seems a bit 'off' in the documentation:

To implement your own memory manager, it is sufficient to construct such a record and to issue a call to SetMemoryManager.

To avoid conflicts with the system memory manager, setting the memory manager should happen as soon as possible in the initialization of your program, i.e. before any call to getmem is processed.

This means in practice that the unit implementing the memory manager should be the first in the uses clause of your program or library, since it will then be initialized before all other units - except the system unit itself, of course.

[marcov]

I assume you did look at heaptrc implementation ?

[Thaddy]

If somebody doesn't want to understand the difference between RAW memory allocation (getmem) and CLEAN memory allocation (allocmem), why bother to answer.
Sometimes it pays to be rude....< grumpy mode >

[Thaddy]

In that case, something seems a bit 'off' in the documentation:

To implement your own memory manager, it is sufficient to construct such a record and to issue a call to SetMemoryManager.

To avoid conflicts with the system memory manager, setting the memory manager should happen as soon as possible in the initialization of your program, i.e. before any call to getmem is processed.

This means in practice that the unit implementing the memory manager should be the first in the uses clause of your program or library, since it will then be initialized before all other units - except the system unit itself, of course.

That's not off, only for heaptrc. The compiler makes that decision. You can..however... write your own heaptrc unit as long as it's called heaptrc.... and with -gh....

Hmm, maybe I should edit the wiki some more...
If you want a memorymanager to be able to do what is asked.... What logically follows is you can't debug properly unless your own memory manager provides for  that.
You aren't silly idiots, are you???

[molly]

@Thaddy:
Remind me to put up a smiley next time i put up documentation with the intention to let TS decide whether the documentation is at fault or his code.

That part of the documentation has been there ... how many years ? And nobody noticed there was something wrong with it.... hmz... maybe ... perhaps .... possibly .... there is something wrong with my code or assumptions ?

There is a perfect example accompanied with that link, using c memory manager.

Note that i'm not excluding any errors that do exists in the documentation.

[marcov]

That's not off, only for heaptrc. The compiler makes that decision. You can..however... write your own heaptrc unit as long as it's called heaptrc.... and with -gh....
..
You aren't silly idiots, are you???

some can't even find -Fa !

[chebmaster]

If somebody doesn't want to understand the difference between RAW memory allocation (getmem) and CLEAN memory allocation (allocmem),

Wut?
I do call FillChar(b^, newallocsize, 0); there.
I was hoping to create a custom MM quickly, in a couple of hours, for a specific debugging purpose. So writing optimal code wasn't my goal. Writing working code was.

This means in practice that the unit implementing the memory manager should be the first in the uses clause of your program or library, since it will then be initialized before all other units - except the system unit itself, of course.

I though I had a workaround for that: my MM marked each block with a signature (at negative offset relative to the memory block returned by it) in and passed the call to OldMM if there was no such signature.
It would have worked, too (with negligible chance of crash), if not the topic of this thread.

heaptrc

It includes lineinfo but lineinfo either causes my custom lineinfo to fail (-gl, no -gw2) or it fails itself (-gw2, no -gl) Either way my game engine reports all exception addresses in useless hexadecimals.

Anyway, since the simple solution failed I'm going to create an extensible memory manager using mmap, mremap, VirtualAlloc, MapViewOfFile and the like. There is no need to hold back 

Should be way easier than taming heaptrc, right?