Recent

Author Topic: GetMem question  (Read 726 times)

Blaazen

  • Hero Member
  • *****
  • Posts: 3237
  • POKE 54296,15
    • Eye-Candy Controls
GetMem question
« on: January 26, 2023, 02:37:14 pm »
Hi,
this code
Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. const cCount = 50000000;
  3. var aPI: PInteger;
  4. begin
  5.   aPI:=getmem(cCount*sizeOf(Integer));
  6. end;
allocates ~200MB memory on each button click when compiled in Default mode, i.e. -O1 and heaptrace. I can see raising curve in System Monitor.

When compiled with -O3 and no debug info and no heaptrsace, the curve in System Monitor is flat. It looks like it does not allocate anything. Is it just an optimalization? Allocated memory is not used after all. It's the same when aPI is global variable.

Thanks

EDIT: It's FPC trunk and Linux.
« Last Edit: January 26, 2023, 02:47:46 pm by Blaazen »
Lazarus 2.3.0 (rev main-2_3-2863...) FPC 3.3.1 x86_64-linux-qt Chakra, Qt 4.8.7/5.13.2, Plasma 5.17.3
Lazarus 1.8.2 r57369 FPC 3.0.4 i386-win32-win32/win64 Wine 3.21

Try Eye-Candy Controls: https://sourceforge.net/projects/eccontrols/files/

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: GetMem question
« Reply #1 on: January 26, 2023, 02:51:57 pm »
No this is to be expected. Modern Operating Systems use so called virtual memory, which only get's mapped onto physical memory when it is used.

On Unix systems you can easiely see virtual and physical memory usege. For example:
Code: Pascal  [Select][+][-]
  1. program Test;
  2. begin
  3.   GetMem(1024*1024*1024); // Alloc 1 Gig
  4.   ReadLn; // avoid closing
  5. end.
Compiling and executing and looking into top:
Code: Text  [Select][+][-]
  1. Tasks:  10 total,   1 running,   8 sleeping,   1 stopped,   0 zombie
  2. %Cpu(s):  0.0 us,  0.3 sy,  0.0 ni, 99.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  3. MiB Mem : 15941.52+total, 15191.93+free,  694.270 used,  313.387 buff/cache
  4. MiB Swap: 4096.000 total, 4096.000 free,    0.000 used. 15247.25+avail Mem
  5.  
  6.   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  7. ...
  8.   345 frederic  20   0 1048988      4      0 T 0.000 0.000   0:00.00 test
  9. ...
As you can see, the process test takes about 1 gig of virtual memory (VIRT), but only 4 Kbytes of phsyical memory (RES)

If we now touch this memory once:
Code: Pascal  [Select][+][-]
  1. program Test;
  2. var
  3.   p: PInteger;
  4. begin
  5.   p := GetMem(1024*1024*1024);
  6.   FillChar(p^, 1024*1024*1024, #00);
  7.   ReadLn;
  8. end.
We get in top:
Code: Text  [Select][+][-]
  1. top - 14:45:27 up 5 min,  0 users,  load average: 0.02, 0.01, 0.00
  2. Tasks:  11 total,   2 running,   8 sleeping,   1 stopped,   0 zombie
  3. %Cpu(s):  0.6 us,  3.6 sy,  0.0 ni, 95.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  4. MiB Mem : 15941.52+total, 14512.57+free, 1284.270 used,  405.113 buff/cache
  5. MiB Swap: 4096.000 total, 4096.000 free,    0.000 used. 14657.25+avail Mem
  6.  
  7.   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  8.   544 frederic  20   0 1048988 578216    172 R 86.67 3.542   0:00.13 test
Which no has a massive physical memory usage (it's not quite the virutal one because operating systems are complicated).

This is actually quite adventaguous for development, because this means that (virtual) memory is free, and you can allocate as much as you want, and it will only be physically allocated when you use it.
This allows you to do things like when creating a list, pre-allocate all the memory you could possibly need (e.g. a gigabyte of data), and then fill that list.  The memory will only be used as much as you filled the list.
This can be quite advantaguous for example for gemoetricly growing datastructures or things like that.

Blaazen

  • Hero Member
  • *****
  • Posts: 3237
  • POKE 54296,15
    • Eye-Candy Controls
Re: GetMem question
« Reply #2 on: January 26, 2023, 03:22:03 pm »
Thanks, I was wondering why SetLength(DynArray) allocates physical memory immediatelly and GetMem() doesn't.
Lazarus 2.3.0 (rev main-2_3-2863...) FPC 3.3.1 x86_64-linux-qt Chakra, Qt 4.8.7/5.13.2, Plasma 5.17.3
Lazarus 1.8.2 r57369 FPC 3.0.4 i386-win32-win32/win64 Wine 3.21

Try Eye-Candy Controls: https://sourceforge.net/projects/eccontrols/files/

Warfley

  • Hero Member
  • *****
  • Posts: 1499
Re: GetMem question
« Reply #3 on: January 26, 2023, 03:35:04 pm »
If you look at the documentation for SetLength: https://www.freepascal.org/docs-html/rtl/system/setlength.html
Quote
In case the length is set to a larger length than the current one, the new elements are zeroed out for a dynamic array. For a string, the string is zero-terminated at the correct length.
SetLength initializes the elements, which is touching them.

The same is btw also true for new for managed types:
With byte array:
Code: Pascal  [Select][+][-]
  1. program Test;
  2.  
  3. type
  4.   PLargeArray = ^TLargeArray;
  5.   TLargeArray=array[0..1024*1024*1024-1] of Byte;
  6.  
  7. var
  8.   p: PLargeArray;
  9. begin
  10.   new(p);
  11.   ReadLn;
  12. end.
top:
Code: Text  [Select][+][-]
  1. Tasks:  11 total,   1 running,   9 sleeping,   1 stopped,   0 zombie
  2. %Cpu(s):  0.0 us,  0.0 sy,  0.0 ni,100.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  3. MiB Mem : 15941.52+total, 15063.96+free,  732.316 used,  405.848 buff/cache
  4. MiB Swap: 4096.000 total, 4096.000 free,    0.000 used. 15209.20+avail Mem
  5.  
  6.   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  7. ...
  8.   705 frederic  20   0 1048988      4      0 T 0.000 0.000   0:00.00 test
With string (managed type):
Code: Pascal  [Select][+][-]
  1. program Test;
  2. {$mode ObjFPC}{$H+}
  3.  
  4. type
  5.   PLargeArray = ^TLargeArray;
  6.   TLargeArray=array[0..1024*1024*1024 div sizeof(string)-1] of String;
  7.  
  8. var
  9.   p: PLargeArray;
  10. begin
  11.   new(p);
  12.   ReadLn;
  13. end.
top:
Code: Text  [Select][+][-]
  1. Tasks:  12 total,   2 running,   9 sleeping,   1 stopped,   0 zombie
  2. %Cpu(s):  2.5 us,  1.7 sy,  0.0 ni, 95.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  3. MiB Mem : 15941.52+total, 14796.17+free, 1000.070 used,  405.855 buff/cache
  4. MiB Swap: 4096.000 total, 4096.000 free,    0.000 used. 14941.45+avail Mem
  5.  
  6.   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  7.   731 frederic  20   0 1048992 274016    180 R 86.67 1.679   0:00.13 test

This is because for managed types new calls Initialize (https://www.freepascal.org/docs-html/rtl/system/initialize.html) on the data, which will touch the data once.
Similar "Dispose" calls Finalize(https://www.freepascal.org/docs-html/rtl/system/finalize.html) on the data.

Both of this is required for managed types to work properly, so when you allocate memory with GetMem and dispose with FreeMem, you must call initialize and finalize manually:
Code: Pascal  [Select][+][-]
  1. program Test;
  2. {$mode ObjFPC}{$H+}
  3.  
  4. type
  5.   PLargeArray = ^TLargeArray;
  6.   TLargeArray=array[0..1024*1024*1024 div sizeof(string)-1] of String;
  7.  
  8. var
  9.   p: PLargeArray;
  10. begin
  11.   p := GetMem(SizeOf(p^));
  12.   Initialize(p^);
  13.   // Do Something with p
  14.  
  15.   Finalize(p^);
  16.   FreeMem(p^);
  17. end.
otherwise this won't work with any managed types

 

TinyPortal © 2005-2018