[SOLVED] more strange Heap usage for PChar variables

[Hartmut]

Hello,

I am a little confused about the heap usage for a PChar variable. Different programs show different results. In the documentation I read, that (on a 32 bit system) the heap is allocated in blocks of 16 bytes. So I expected e.g. 16 bytes heap usage for PChars with the length from 0 to 15 (because of the trailing zero byte) and 32 bytes heap usage for PChars with the length from 16 to 31. But this is not true, as 'program1' shows:

Code: [Select]

{$mode Delphi} {$H+}
program program1;

uses strings;

var s: string[200];
    pc: pchar;
    h1,h2: TFPCHeapStatus;
    i: integer;
begin
for i:=0 to 50 do
   begin
   s:=space(i);
   h1:=GetFPCHeapStatus;
      pc:=StrAlloc(length(s)+1);
      StrPCopy(pc,s);
   h2:=GetFPCHeapStatus;
   writeln(i, ' ', h2.CurrHeapUsed-h1.CurrHeapUsed);
   StrDispose(pc);
   end;
end.
We have 16 bytes heap usage for PChars with the length from 0 to 11 and 32 bytes for PChars with the length from 12 to 27. So there seems always to be a 'hidden consumer' who takes 4 bytes.

But in another example 'program2', which copies PChars from Ansistrings, we have an other picture:

Code: [Select]

{$mode Delphi} {$H+}
program program2;

uses sysutils;

var SR: TSearchRec;
    pc: pchar;
    h1,h2: TFPCHeapStatus;
    e: longint;
begin
e:=FindFirst('*.*', faAnyFile, SR);
while e=0 do
   begin
   h1:=GetFPCHeapStatus;
      pc:=StrNew(pChar(SR.name));
   h2:=GetFPCHeapStatus;
   writeln(length(pc):2, ' ', h2.CurrHeapUsed-h1.CurrHeapUsed, ' ', pc);
   StrDispose(pc);
   e:=FindNext(SR);
   end;
FindClose(SR);
end.
Here we have 16 bytes heap usage for PChars with the length from 0 to 7 and 32 bytes for PChars with the length from 8 to 23. So here seems always to be a hidden consumer who takes 8 bytes!

But the reason for the difference is not the Ansistring source: 'Program3' copies also PChars from Ansistrings with StrNew(), but is more like 'program1' and has the same heap usage with a hidden consumer of only 4 bytes:

Code: [Select]

{$mode Delphi} {$H+}
program program3;

uses strings;

var s: Ansistring;
    pc: pchar;
    h1,h2: TFPCHeapStatus;
    i: integer;
begin
for i:=0 to 50 do
   begin
   s:=space(i);
   h1:=GetFPCHeapStatus;
      pc:=StrNew(pChar(s));
   h2:=GetFPCHeapStatus;
   writeln(i, ' ', h2.CurrHeapUsed-h1.CurrHeapUsed);
   StrDispose(pc);
   end;
end.
Please can samebody explain to me who this 'hidden' consumer is (something of the heap management?) and why/when he needs sometimes 4 and sometimes 8 bytes? If there is any documentation about that please give me a link. Thanks in advance.

[Hartmut]

Things got more strange. Meanwhile I found out, that the difference, whether the 'hidden consumer' needs additional 4 or 8 bytes, does not depend on the kind of PChar variable assignment, but only depends on the units which are 'use'd:
 - if you use unit strings: 4 bytes are wasted for each PChar
 - if you use unit sysutils: 8 bytes are wasted for each PChar
 - if you use both: it depends on the order you write them (if strings is last, 4 bytes are wasted, if sysutils is last, 8 bytes are wasted).

I don't understand this. Is this a bug? Can anybody explain this?

[marcov]

Well, 4 bytes are used to store the size of the allocation. Note that in some cases also temporary variables may be generated that are deallocated at the end of the function.

Maybe you also run into the fact that searchrec, findfirst and friends are different in sysutils than dos (having ansistring/unicodestring allowing longer  filenames).

[engkin]

There are two different functions that have the same name:
Strings.StrNew calls GetMem directly.
SysUtils.SrtNew calls StrAlloc which adds 4 bytes before it calls GetMem.
In both cases GetMem adds some amount (4 bytes?) to account for a header.
These numbers are for 32 bit systems.

Strings:

Code: [Select]

    function strnew(p : pchar) : pchar;
      var
         len : SizeInt;

      begin
         strnew:=nil;
         if (p=nil) or (p^=#0) then
           exit;
         len:=strlen(p)+1;
         getmem(strnew,len);
         if strnew<>nil then
           move(p^,strnew^,len);
      end;

SysUtils:

Code: [Select]

function strnew(p : pchar) : pchar;
var
  len : longint;
begin
  Result:=nil;
  if (p=nil) or (p^=#0) then
   exit;
  len:=strlen(p)+1;
  Result:=StrAlloc(Len);
  if Result<>nil then
   strmove(Result,p,len);
end;

function StrAlloc(Size: cardinal): PChar;
begin
  inc(size,sizeof(cardinal));
  getmem(result,size);
  cardinal(pointer(result)^):=size;
  inc(result,sizeof(cardinal));
end;

By default the memory manager calls SysGetMem:

Code: [Select]

function SysGetMem(size : ptruint):pointer;
begin
{ Something to allocate ? }
  if size=0 then
    { we always need to allocate something, using heapend is not possible,
      because heappend can be changed by growheap (PFV) }
    size := 1;
{ calc to multiple of 16 after adding the needed bytes for memchunk header }
  if size <= (maxblocksize - sizeof(tmemchunk_fixed_hdr)) then
    begin
      size := (size+(sizeof(tmemchunk_fixed_hdr)+(blocksize-1))) and fixedsizemask;
      result := sysgetmem_fixed(size);
    end
  else
    begin
      if size < high(ptruint)-((sizeof(tmemchunk_var_hdr)+(blocksize-1))) then
        size := (size+(sizeof(tmemchunk_var_hdr)+(blocksize-1))) and sizemask;
      result := sysgetmem_var(size);
    end;
...


[Hartmut]

Thank you very much for that excellent explanation.
It's very strange to me that a core function like StrNew() exists twice in different core units and is implemented so different. I wonder if there is a benefit of using sysutils, which pays 4 bytes more for each PChar variable...

[engkin]

I wonder if there is a benefit of using sysutils, which pays 4 bytes more for each PChar variable...

I believe that SysUtils.StrNew is implemented this way to maintain compatibility with Delphi.

Notice that there are also two StrDispose procedures and the corresponding one should be used to free the memory correctly.

[marcov]

It's confirmed by both the FPC and Delphi help (the additional 4 bytes). I think it is an artifact from TPW or D1 times, before the count was added to the memory manager. (Delphi deprecates the functions btw).

[Hartmut]

Notice that there are also two StrDispose procedures and the corresponding one should be used to free the memory correctly.

Thank you for this very important tip! I did not pay attention to that.