* * *

Author Topic: A bug or a feature? SysAllocMem calls MemoryManager.GetMem instead of SysGetMem  (Read 2238 times)

chebmaster

  • New member
  • *
  • Posts: 7
I am confused.
I tried creating a wrapper memory manager for debugging purposes (i.e. a memory manager that expands on the system one by increasing allocated block sizes, thus adding some extra data to the allocated blocks)
It failed craptacularly because, for example, SysAllocMem() calls not on SysGetMem as one would expect but on MemoryManager.GetMem  - which I already redefined and which adds that record and performs pointer arithmetic before calling SysGetMem - i.e. the block gets added twice, then FreeMem comes trying to untangle tham mess and the everything goes to hell.  >:D
Sadly that wasn't t the only place the MemoryManager.xxx methods get called.

So, my question is: is this done for some compatibility reason, or is this an oversight?

Because as it is now, the only way to get a modded memory manager is to write one from scratch :(

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 5743
The Sys variants are afaik the default  implementations for the memorymanager records. They are afaik meant to be replaced en-bloc by another memory manager implementation, and their internals are not documented?

Did you maybe replace getmem in the memory manager, but not allocmem in the memory manager?




chebmaster

  • New member
  • *
  • Posts: 7
I surely replaced them *all*.
The problem is that I replaced them with wrappers that called upon them (OldMM.AllocMem or SysAllocMem - it makes no difference). But they, in turn, were calling upon my wrappers instead of calling on each other. Which is fundamentally wrong. >:(

So I am trying to understand: why?
Is there an obscure but legitimate reason, or is this a genuine bug?

I tried hacking around that (see SysGetMem below), but SysReallocMem is too complex for it to work :(

Code: Pascal  [Select]
  1.   function mml_AllocMem(Size : ptruint) : Pointer;
  2.   begin
  3.     EnterCriticalSection(CS);
  4.     Result:= MyAllocMem(Size);
  5.     LeaveCriticalSection(CS);
  6.   end;
  7.  
  8.   function MyAllocMem  (Size : ptruint) : Pointer;
  9.   var
  10.     b: PMemBlockRec;
  11.     newallocsize: ptruint;
  12.   begin
  13.       if Size = 0 then begin
  14.         Exit(nil);
  15.       end;
  16.  
  17.       newallocsize:= Size + sizeof(TMemBlockRec);
  18.       b:= SysGetMem(newallocsize);
  19.       FillChar(b^, newallocsize, 0);
  20.       b^.sig:= defsig;
  21.       b^.size:= Size;
  22.       b^.alloc_size:= newallocsize;
  23.       b^.ThreadId:= GetCurrentThreadID;
  24.       if not deactivated then begin
  25.         b^.prev:= @StubBlock;
  26.         b^.next:= StubBlock.next;
  27.         StubBlock.next:= b;
  28.         if Assigned(b^.next) then b^.next^.prev:= b;
  29.         GetBacktrace(b);
  30.       end;
  31.       Result:= pointer(ptruint(b) + sizeof(TMemBlockRec));
  32.   end;

« Last Edit: February 28, 2017, 07:03:09 pm by chebmaster »

chebmaster

  • New member
  • *
  • Posts: 7
P.S.
Code: Pascal  [Select]
  1.   Function mml_FreeMem (P : pointer) : ptruint;
  2.   begin
  3.     EnterCriticalSection(CS);
  4.     Result:= MyFreeMem(p);
  5.     LeaveCriticalSection(CS);
  6.   end;
  7.  
  8.   Function mml_FreeMemSize(p:pointer;Size:ptruint):ptruint;
  9.   begin
  10.     Result:= mml_FreeMem(p);
  11.   end;
  12.  
  13.   Function myFreeMem (P : pointer) : ptruint;
  14.   var
  15.     b: PMemBlockRec;
  16.     alien: boolean = true;
  17.     newallocsize: ptruint;
  18.   begin
  19.       if not Assigned(p) then begin
  20.         Exit(0);
  21.       end;
  22.       b:= PMemBlockRec(pointer(ptruint(p) - sizeof(TMemBlockRec)));
  23.       alien:= b^.sig <> defsig;
  24.       if alien then begin
  25.         //alien block allocated prior to installation of my memory manager
  26.         myFreeMem:= SysFreeMem (p);
  27.         Exit
  28.       end;
  29.       if Assigned(b^.next) then b^.next^.prev:= b^.prev;
  30.       if Assigned(b^.prev) then b^.prev^.next:= b^.next;
  31.       b^.prev:= nil;
  32.       b^.next:= nil;
  33.       myFreeMem:= b^.size;
  34.       SysFreeMemSize(b, b^.alloc_size);
  35.   end;
« Last Edit: February 28, 2017, 07:15:08 pm by chebmaster »

Cyrax

  • Hero Member
  • *****
  • Posts: 540
I would not recommend to use directly Sys* variants of memory allocation functions. Rather make a backup of previous memory manager (GetMemoryManager) and use it extensively in your own manager by redirecting calls to old manager with necessary modifications.

chebmaster

  • New member
  • *
  • Posts: 7
Quote
make a backup of previous memory manager (GetMemoryManager) and use it extensively in your own manager by redirecting calls to old manager with necessary modifications
Have you read my posts at all? >:(
I DID EXACTLY THAT.
IT DOESN'T WORK.
Because the Sys*** functions call MemoryManager.*** functions, not their fellow Sys*** functions.

Please stop giving me noob-targeted answers. I have more than 20 years of experience using Pascal, since Turbo 5.0 / Delphi 2.0 / FPC 1.X

So. To anyone who can understand the topic: Is that a bug or is that an obscure feature?

P.S. Sister topic in Russian: http://freepascal.ru/forum/viewtopic.php?f=1&t=18231
« Last Edit: March 01, 2017, 08:28:39 am by chebmaster »

Cyrax

  • Hero Member
  • *****
  • Posts: 540
Take chill pill, dude. With that kind of attitude, you will not find help here in the future.

I take that this code example won't work because of reasons?

Code: Pascal  [Select]
  1. program Project1;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. uses
  6.   {$IFDEF UNIX}{$IFDEF UseCThreads}
  7.   cthreads,
  8.   {$ENDIF}{$ENDIF}
  9.   Classes
  10.   { you can add units after this };
  11.  
  12. Var
  13.   AOldMemoryManager : TMemoryManager;
  14.  
  15. Function MyGetmem(Size:ptruint):Pointer;
  16. Var
  17.   AMemory : PByte;
  18. begin
  19.   AMemory := AOldMemoryManager.Getmem(Size + 100);
  20.   Inc(AMemory, 100);
  21.   Result := AMemory;
  22. end;
  23.  
  24. Function MyFreeMem(p:pointer):ptruint;
  25. Var
  26.   AMemory : PByte;
  27. Begin
  28.   AMemory := P;
  29.   Dec(AMemory, 100);
  30.   Result := AOldMemoryManager.FreeMem(AMemory);
  31. end;
  32.  
  33. Var
  34.   ANewMemoryManager : TMemoryManager;
  35.   AMemory : Pointer;
  36. begin
  37.   System.GetMemoryManager(AOldMemoryManager);
  38.   ANewMemoryManager := AOldMemoryManager;
  39.  
  40.   ANewMemoryManager.Getmem := @MyGetmem;
  41.   ANewMemoryManager.FreeMem := @MyFreeMem;
  42.  
  43.   System.SetMemoryManager(ANewMemoryManager);
  44.   GetMem(AMemory, 10);
  45.   FreeMem(AMemory);
  46.  
  47.   System.SetMemoryManager(AOldMemoryManager);
  48. end.
  49.  

chebmaster

  • New member
  • *
  • Posts: 7
Quote
I take that this code example won't work because of reasons?
Yeah  :(
It would have worked IF SysReallocMem only used MemoryManager.GetMem/FreeMem/MemSize. Alas, it also uses  SysTryResizeMem.
In short, it mixes calls to my wrapper functions that work in the transformed space with calls to routines that should work with original space. Nothing but mess can result.

A wrapper would only work if all it did was non-intrusive stuff like counting number of deallocations during certain phase of the Moon :(

Quote
Take chill pill, dude.
I try, but "intellectual" replies like "Try adding 2 and 2! It's 4, see?" make ANGRY >:(


molly

  • Hero Member
  • *****
  • Posts: 1922
In that case, something seems a bit 'off' in the documentation:
Quote
To implement your own memory manager, it is sufficient to construct such a record and to issue a call to SetMemoryManager.

To avoid conflicts with the system memory manager, setting the memory manager should happen as soon as possible in the initialization of your program, i.e. before any call to getmem is processed.

This means in practice that the unit implementing the memory manager should be the first in the uses clause of your program or library, since it will then be initialized before all other units - except the system unit itself, of course.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 5743
I assume you did look at heaptrc implementation ?

Thaddy

  • Hero Member
  • *****
  • Posts: 4525
If somebody doesn't want to understand the difference between RAW memory allocation (getmem) and CLEAN memory allocation (allocmem), why bother to answer.
Sometimes it pays to be rude....< grumpy mode >:D >
"Logically, no number of positive outcomes at the level of experimental testing can confirm a scientific theory, but a single counterexample is logically decisive."

Thaddy

  • Hero Member
  • *****
  • Posts: 4525
In that case, something seems a bit 'off' in the documentation:
Quote
To implement your own memory manager, it is sufficient to construct such a record and to issue a call to SetMemoryManager.

To avoid conflicts with the system memory manager, setting the memory manager should happen as soon as possible in the initialization of your program, i.e. before any call to getmem is processed.

This means in practice that the unit implementing the memory manager should be the first in the uses clause of your program or library, since it will then be initialized before all other units - except the system unit itself, of course.

That's not off, only for heaptrc. The compiler makes that decision. You can..however... write your own heaptrc unit as long as it's called heaptrc.... and with -gh....

Hmm, maybe I should edit the wiki some more...
If you want a memorymanager to be able to do what is asked.... What logically follows is you can't debug properly unless your own memory manager provides for  that. >:(
You aren't silly idiots, are you??? :o ::) :-[
« Last Edit: March 01, 2017, 01:22:17 pm by Thaddy »
"Logically, no number of positive outcomes at the level of experimental testing can confirm a scientific theory, but a single counterexample is logically decisive."

molly

  • Hero Member
  • *****
  • Posts: 1922
@Thaddy:
Remind me to put up a smiley next time i put up documentation with the intention to let TS decide whether the documentation is at fault or his code.

That part of the documentation has been there ... how many years ? And nobody noticed there was something wrong with it.... hmz... maybe ... perhaps .... possibly .... there is something wrong with my code or assumptions ?

There is a perfect example accompanied with that link, using c memory manager.

Note that i'm not excluding any errors that do exists in the documentation.

marcov

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 5743

That's not off, only for heaptrc. The compiler makes that decision. You can..however... write your own heaptrc unit as long as it's called heaptrc.... and with -gh....
..
You aren't silly idiots, are you??? :o ::) :-[

some can't even find -Fa !

chebmaster

  • New member
  • *
  • Posts: 7
Quote
If somebody doesn't want to understand the difference between RAW memory allocation (getmem) and CLEAN memory allocation (allocmem),
Wut?
I do call FillChar(b^, newallocsize, 0); there.
I was hoping to create a custom MM quickly, in a couple of hours, for a specific debugging purpose. So writing optimal code wasn't my goal. Writing working code was.

Quote
This means in practice that the unit implementing the memory manager should be the first in the uses clause of your program or library, since it will then be initialized before all other units - except the system unit itself, of course.
I though I had a workaround for that: my MM marked each block with a signature (at negative offset relative to the memory block returned by it) in and passed the call to OldMM if there was no such signature.
It would have worked, too (with negligible chance of crash), if not the topic of this thread.

Quote
heaptrc
It includes lineinfo but lineinfo either causes my custom lineinfo to fail (-gl, no -gw2) or it fails itself (-gw2, no -gl) Either way my game engine reports all exception addresses in useless hexadecimals.

Anyway, since the simple solution failed I'm going to create an extensible memory manager using mmap, mremap, VirtualAlloc, MapViewOfFile and the like. There is no need to hold back  >:D

Should be way easier than taming heaptrc, right?

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus