Memory Manager

[damieiro]

Hi!
I'm trying to make a memory manager in which the getmen function will have the sintax like in tp7

Code: [Select]

procedure Getmem(  out p: pointer;  Size: PtrUInt);

like in heaph.inc

and not the way is in the TMemoryManager defined (https://www.freepascal.org/docs-html/prog/progsu175.html):

Code: Pascal  [Select][+][-]

1. TMemoryManager = record  
2.     NeedLock    : Boolean;  
3.   Getmem      : Function(Size:PtrInt):Pointer;  /// i prefer procedural way
4.     Freemem     : Function(var p:pointer):PtrInt;  
5.     FreememSize : Function(var p:pointer;Size:PtrInt):PtrInt;  
6.     AllocMem    : Function(Size:PtrInt):Pointer;  
7.     ReAllocMem  : Function(var p:pointer;Size:PtrInt):Pointer;  
8.     MemSize     : function(p:pointer):PtrInt;  
9.     InitThread          : procedure;  
10.     DoneThread          : procedure;  
11.     RelocateHeap        : procedure;  
12.     GetHeapStatus       : function :THeapStatus;  
13.     GetFPCHeapStatus    : function :TFPCHeapStatus;  
14.   end;
15.  

Why like a procedure and not like a function?. Well, i want to check the value of p (for example, is nil or isn't nil, but is unused) , and with the function interface i cannot check it.

Is there an easy way to do it without "wrapping" the function..?

On the other way, i do not know why the procedural way is not the default and the function is.
One can do a easy function getmen with the procedural if you don't mind about p, or you prefer the function sintax, but not the reverse.

[marcov]

The normal getmem is a wrapper that calls the manager. Those two definitions don't have to compatible.

And no, this is not going to change, the function form in the record is Delphi compatible.

[Thaddy]

I think you can extend the MM record, add your procedure and have the function call the procedure. Has to be the last entry of course.

[damieiro]

@Markov:  Why don't change it or at least overload it? It doesn't seem complicated

I know that getmen wraps sysgetmen and getmen by itself is direct call to this procedure. I point to the interface that i cannot change and seems possible to change safely without trouble.

I suppose that when new (p) is in compiler time and is traslated in some call to the direction pointer by getmen in compiler time, because it knows size of (p). So this is made by compiler and not delphi or tp or dialect, so the entry point can be changed without issues. This is a construct like write made by compiler. So new (p) should call the procedural way internally to get the programmer the tool of know (p) as in dispose way.

So let's go for the other ways. That some existent memory manager (not default) exists and we know to maintain their way. Then the procedural way should be filled by default with some kind code calling the old function. But being the procedural way the entry point for the Compiler...

Code: [Select]

procedure getmen (var localpointer:pointer; size:ptrint);
begin
localpointer:=getmen (size)  // the old sysmanager is called correctly
end
This should maintain the old function way, but at least give the programmer a way to see (p). In class constructors, it should be compiled or transformed to point the procedural way.
I think it's logical that the variable in which should be stored the pointer be visible. I do not know why this is not to change. It can be implemented as extension and trasparently by compiler.
 Example of extended TManager respecting old ways. (@Taddy i used your idea in different way )

Code: Pascal  [Select][+][-]

1. TMemoryManager = record  
2.     NeedLock    : Boolean;  
3.   Getmem      : Function(Size:PtrInt):Pointer;  /// i prefer procedural way
4.     Freemem     : Function(var p:pointer):PtrInt;  
5.     FreememSize : Function(var p:pointer;Size:PtrInt):PtrInt;  
6.     AllocMem    : Function(Size:PtrInt):Pointer;  
7.     ReAllocMem  : Function(var p:pointer;Size:PtrInt):Pointer;  
8.     MemSize     : function(p:pointer):PtrInt;  
9.     InitThread          : procedure;  
10.     DoneThread          : procedure;  
11.     RelocateHeap        : procedure;  
12.     GetHeapStatus       : function :THeapStatus;  
13.     GetFPCHeapStatus    : function :TFPCHeapStatus;  
14.     Getmem: procedure(out p:pointer;Size:PtrInt); /// ADD this record field
15.     Getmem : Function(const p: pointer;Size:PtrInt):Pointer;  /// other way
16.  end;
17.  
18.  

Even it could be even more ways for example extending in the interface and passing allways the p as an internal parameter.  Or this p being in some accesible place.
Or if we go creative, even tricky ways and compiler extensions: if a function has Result, a @callerResult  could be a point to the adress of  variable storing the value of the function -> this languaje construct will make any function call know as extraparameter the storing variable, but i only will allow this with some kind of extended compiler flag like const var changing. But i go for the simple way: Changing the entry point and having pointer as parameter.

And as you know, p is pointing to data, but p-8 and p-16 is pointing to pointermetadata (like size), so knowing p, one can know if it is a valid pointer and What kind/class of smartpointer could be).

So Why not is going to change? Where are the compatibility issues? I think the memory management is in a layer previous (how is compiled new and the constructors solver like write (p1,p2,p3...) ), so it can be solved in compile time and not being a language issue.

I think the only "big" think is to change the entry point from the function to the procedural way in compiler time (which i think is were it's done). For compatibility filling it with a call to the function way for old code, and new power for the memory managers with the new code.

Even it shouldn't be a performance issue. If compiler is smart enought this code could be removed by compiler optimization call a that calls b that calls c compiled by call  to c from a...

@Taddy: Not need to extend the record as you say. Actually i can call with the function my own getmem. The issue is the "entry point". It's the reverse. Calling the function with the entry point in procedural getmen (or any two parameters getmen with the p and the size) saving the knowledge of the pointer content. But it's good idea to extend the record to use the new capabilities saving the old ways

PD: Sorry for my bad english. I'm not english native

[lucamar]

It's basically a question of backwards compatibility ... and personal preferences, of which each person has their own. And writing a simple wrapper over the MM is so easy that I don't think it warrants changing the basic interface.

Besides, the semantics of function GetMem() looks correct as is: It Gets (some size of) Mem from wherever and retunrs a pointer to that memory. But that is, of course, my opinion. 

[damieiro]

I agree that function is well defined. I propose a way that is backward compatible, but allows to do more things.
I disagree in matter of tastes. It's not a taste, it's having the information or not, then interface it with your favorite taste

For example.
If you have a code that does

Code: Pascal  [Select][+][-]

1. {this code seems to work allways well, but it doesn't if you manually don't see it}
2. p:=getmem(somesize);  //compiler calls getmem func
3. {do cool things, but yout forgot free p}
4. p:=getmem (somesize);  //compiler calls getmen func
5.  

and if you do not free (p). you have a memory leak. If the memory manager sees that p is used, and no referenced it could free it in running time transparently for you and log it correctly.
If you do not know p, the only thing you can do is when the code finalices take the list of non freed pointers by the freemem procedure and free these when the code exits (if already using your own memory manager, default one doesn't do it).. DEVS, why not default memory manager doesn't free the pointers when code exit/halt as default-> protect the system?. At least with an own memory manager you can see memory leaks (you can log these pointers for example to code correction) but you cannot solve these issues on run time and log then correctly: when it happens, not at the end of program or some fixed point.

it's even worse if you use new (p). In this construct like write you really pass the pointer, but it's called to the memory manager as a function, so you don't see p which is passed. And compiler use size of p for data so new p does something like this

Code: Pascal  [Select][+][-]

1. procedure new (var p:pointer); /// really not a var, because it knows sizeof (p) and a var pointer is untyped pointer (size 0)
2. begin
3. p:=getmem (sizeof (p)) ; /// well, this is pseudocode, size of (p) is 0 it you pass a untyped pointer, but like write, compiler constructs something like this
4. end;
5.  
6.  

Well, the issue, in their most general way is that a memory manager should know what already the compiler knows, not matter of languaje tastes. Then the interface could be the old way, or new way backward compatible...

What i think it should be. Like this. Compiler is the magic word

Code: Pascal  [Select][+][-]

1. p:=getmem(somesize);  //COMPILER calls getmem proc with p as parameter, not losing/fogetting it.
2. {do cool things, but again}
3. p:=getmem (somesize);  // compiler calls getmen proc again. Like it does in new (p)
4. getmem (p,somesize); // Compiler calls the same getmen proc..
5.  

Or at least that Compiler give the programmer a way to know p in a not p parameter call.

Well, i should say that i firmly believe in automatically avoiding manual errors when possible .

[Thaddy]

// well, this is pseudocode, size of (p) is 0 it you pass a untyped pointer,

Bogus. SizeOf(p) is SizeOf(NativePointer), typed or not. Such things don't belong on the forum. People may believe it.

[damieiro]

@Thaddy, try this.. t's correct what you say, but i mean things like this with the (p^) not p (my fault )

Code: Pascal  [Select][+][-]

1.  
2. program pointers;
3.  
4. procedure sizep (var p:pointer);
5. begin
6.   writeln (sizeof (p)); {not possible sizeof (p^), how i could deference it or know at least the size...? sizeof (p^) is 0}
7.   writeln (sizeof (p^)); {ZERO, type size is lost by definition}
8. end;
9.  
10. type
11.   Tmyarray = array [1..100] of byte;
12.   pmyarray = ^TMyarray;
13.  
14. var
15.   test:pmyarray;
16.  
17. begin
18.   writeln (sizeof(test^)); {should be 100, i can deference it}
19.   sizep (test); {shoul be 8 and 0}
20. end.
21.  

Making a new (p) implies that new should receive a anytiped pointer and you should know their size (but it can be deferenced with generic pointer) . So compiler is making new (p) like new (p, sizeof (p^)) and calling it like p:=getmem (sizeof(p^). P cannot be passed maintaining its size unless you use a typed pointed parameter. If you use an untiped one, type information is lost.

It's not possible to make a sizep function (without overloading it with every type) that gives size of different types of pointers, because you cannot pass that. But new (p) allows it because is constructed like write with compiler information of the pointer declaration. So same strategy for proc getmem: pass the pointer, not only the calculated size

If you pass var p (not var p:pointer) the size is zero too

Code: Pascal  [Select][+][-]

1. program pointers;
2.  
3. procedure sizep (var p);
4. begin
5.   writeln (sizeof (p));   {ZERO}      {sizeof (p^) is not valid}
6. end;
7.  
8. type
9.   Tmyarray = array [1..100] of byte;
10.   pmyarray = ^TMyarray;
11.  
12. var
13.   test:pmyarray;
14.  
15. begin
16.   writeln (sizeof(test^)); {should be 100}
17.   sizep (test); {should be 0}
18. end.  
19.  

[ASerge]

I don't understand what the problem is? When you write GetMem(P, Size); or P := GetMem(Size); compiler inserts a call to P := MemoryManager.GetMem(Size);
In case you want to reuse P variable, then do call ReAllocMem(P, Size);

[damieiro]

@aserge

I don't understand what the problem is? When you write GetMem(P, Size); or P := GetMem(Size); compiler inserts a call to P := MemoryManager.GetMem(Size);
In case you want to reuse P variable, then do call ReAllocMem(P, Size

This is the problem, is an example. I want a smarter memory manager to detect leaks automatically, not manually. And know p is better that not knowing it

Code: Pascal  [Select][+][-]

1. {pseudocode}
2.  
3. function memorymanager.getmem (size:ptrint):pointer;
4. var
5. localpointer:pointer;
6. begin
7.    {HOW can know P?}
8.    if p = nil then assignmemory (localpointer, size)  // ALL OK
9.    else { p <> nil }
10.      begin {this is not ok}
11.        log error, free old p, give new p.... do some cool things (smart pointers)
12.      end;  
13.  
14. result:=localpointer;
15.  
16. end;
17.  

Code: Pascal  [Select][+][-]

1. procedure memorymanager.getmem (var p: pointer; size:ptrint)
2. begin
3.     { P is know, because it's passed}
4.   if p = nil then assignmemory (p, size) // ALL OK
5.    else { p <> nil }
6.      begin {this is not ok}
7.        log error, free old p, give new p... {you can manage with the p info}
8.      end;  
9. end;
10.  

What i think is (using your words) is When I write GetMem(P, Size); or P := GetMem(Size);
compiler inserts a call to MemoryManager.GetMem(P,Size) and not do a P:=GetMem(Size). Simply as hell as this (thanks, your example helped me to explain the issue)

[lucamar]

It seems to me that you want the MM to cope with soddy programming, or act like a semi-GC. The point is that in Pascal, as it is today, that's NOT the job of the MM, but of the programmer. The MM "simply" manages the heap, gives you memory when you want it and releases it when you want it to. Thats why there exists heaptrc, etc.: to help with *debugging* problematic code. Once it is debuged? No need for more "magic".

BTW, a pointer doesn't *need* to be nil to be valid for GetMem. Consider:

Code: Pascal  [Select][+][-]

1. APointer:= GetMem(1024);
2. try
3.   {Do work with APointer}
4. finally
5.   FreeMem(APointer, 1024);
6. end;
7. {Here APointer is *NOT* nil but you can safely do:
8.   APointer := GetMem(NeededSize);
9. }
10.  

[damieiro]

@lucamar.

I only say that when I write GetMem(P, Size); or P := GetMem(Size);
compiler inserts a call to MemoryManager.GetMem(P,Size) and not do a P:=MemoryManager.GetMem(Size). Simply as this. And it should do the same for new (p) and every call to memory manager. (constructor... etc).
And if not defined, MemoryManager.getmem (p,size) call the memorymanager function as default to backward compatibility.

Then i would like to manage p. Others not. All ways are perfect. Ones prefer allways nil for unused pointers, others not.
I don't  say one way is better than other

I say that if one wants to manage the content of p, the actual way gives you no information and the passing p gives information. And having that information allows to do some other things. I want the tool. Then you can use the tool in a way or another. But i think we could have a more complete tool

I know that p could have any value before calling getmen.
You even can do things like this

Code: Pascal  [Select][+][-]

1. Var
2. unusedvar:longint;
3. p:pointer;
4.  
5. begin
6. p:=@unusedvar
7. p:=getmem (1024);
8. end;

But even in this case, the memory manager if p is passed can determine if p is a valid pointer or not and do things with it. A memory manager has ways to know that p is a pointer made for the memory manager (having a copy in his own table, for example). As i said above, i would like what value p has as a parameter for memory manager, in the worst case it could simply be ignored like now. And their visibility should not have performance issues, but i think it's useful for a memory manager having it from start.

And even for debugging, a debugging tool could be better than the current debugger heaptpr with this pointer information.. (actively detecting it when happens and not as final log).

[ASerge]

My opinion is not to change RTL, but to change your approach to memory allocation.
If you prefer a procedural approach and the reuse of variables, always use ReAllocMem:

Code: Pascal  [Select][+][-]

1. var P: Pointer = nil;
2. begin
3.   //...
4.   ReAlloc(P, Size);
5.   try
6.     //...
7.     ReAlloc(P, NewSize);
8.     //...
9.     ReAlloc(P, NewSize);
10.     //...
11.   finally
12.     ReAlloc(P, 0);
13.   end;
14.  

If you want to test your code, you can either use the debug properties (heaptrace), or write your own GetMem, which does everything you want.

[Thaddy]

A word of caution: although writing a memory manager is easy, writing a memory manager with debug features is not!
Since you can't easily use language features that require memory (like writeln etc) unless you pre-allocate such memory.
Also note that the system unit already allocates memory (160 bytes), the reason heaptrc can not be in the uses clause, but needs to be included by the compiler option.
All those little details can make it a quite complicated task if you want to do it right.

[taazz]

My opinion is not to change RTL, but to change your approach to memory allocation.
If you prefer a procedural approach and the reuse of variables, always use ReAllocMem:

Code: Pascal  [Select][+][-]

1. var P: Pointer = nil;
2. begin
3.   //...
4.   ReAlloc(P, Size);
5.   try
6.     //...
7.     ReAlloc(P, NewSize);
8.     //...
9.     ReAlloc(P, NewSize);
10.     //...
11.   finally
12.     ReAlloc(P, 0);
13.   end;
14.  

If you want to test your code, you can either use the debug properties (heaptrace), or write your own GetMem, which does everything you want.

I think you missed the point. Its not his code is worried about changing the interface to allow for this is the only way out of testing 3rd party code including the rtl.