Recent

Author Topic: [SOLVED] discarding an array  (Read 1678 times)

robert rozee

  • Full Member
  • ***
  • Posts: 153
[SOLVED] discarding an array
« on: June 01, 2023, 07:53:24 pm »
perhaps a silly question: is there any way to discard a typed constant array at runtime?

the situation is i have several rather large arrays, of the form
Code: Pascal  [Select][+][-]
  1. Const
  2.   bindata : Array[0..1066635] of byte = (
  3.       31,139,  8,  8,242,169,120,100,  0,  3, 71, 70, 88,116,101,114,109,
  4.        0,172,157,  7,120, 85,197,246,197,111, 66,132,  0,210,123, 15,160,
  5.      244,222, 59,161,247, 94, 85, 48,  4,114, 19,  2, 33,  9, 73, 40,210,
  6.     ...
  7. Const
  8.   patch1 : Array[0..315288] of byte = (
  9.       31,139,  8,  8,237,165,120,100,  0,  3,112, 49,100,105,102,102,115,
  10.       46,116,120,116,  0,236,253,187,210,236, 58,175, 45,  8,250,124,138,
  11.      109,118, 25, 43, 66, 41,165,164, 76,183,162,162,204,118,250,  9,242,
  12. ... etc.

there have been generated by bin2obj. i use them at the start of my application, but thereafter they are no longer needed. indeed, within a second or two my application does:
Code: Pascal  [Select][+][-]
  1.   ExecuteProcess(TempName, '');                                // run binary

in order to keep things as tidy as possible, it would be nice to free up the arrays before essentially going dormant. any suggestions on how to achieve this? the application is strictly linux/unix only if that helps.


cheers,
rob   :-)
« Last Edit: June 05, 2023, 05:26:15 pm by robert rozee »

Eugene Loza

  • Hero Member
  • *****
  • Posts: 656
    • My games in Pascal
Re: discarding an array
« Reply #1 on: June 01, 2023, 08:32:26 pm »
I don't know any way to discard a constant in Pascal... I wonder if there is any reason you don't want to have them as dynamic arrays? You literally just replace "const" with "var" and don't even have to make any conversion. Like this:

Code: Pascal  [Select][+][-]
  1. var
  2.   A: array of Byte = (1,2,3,4);
  3. begin
  4.   //Work with A
  5.   A := nil; // A will be free now
  6. end.

Note: this may be a new feature of FPC 3.3.1
My FOSS games in FreePascal&CastleGameEngine: https://decoherence.itch.io/ (Sources: https://gitlab.com/EugeneLoza)

jamie

  • Hero Member
  • *****
  • Posts: 6077
Re: discarding an array
« Reply #2 on: June 01, 2023, 09:25:53 pm »
Link that data in as a resource and load it when needed into dynamic array.

It will still be attached to the EXE but not in memory once you free the dynamic array.
The only true wisdom is knowing you know nothing

PascalDragon

  • Hero Member
  • *****
  • Posts: 5444
  • Compiler Developer
Re: discarding an array
« Reply #3 on: June 01, 2023, 10:29:14 pm »
perhaps a silly question: is there any way to discard a typed constant array at runtime?

No, for static typed constant arrays this is not possible, they are part of the binary's (readonly) data section.

jcmontherock

  • Full Member
  • ***
  • Posts: 234
Re: discarding an array
« Reply #4 on: June 02, 2023, 05:36:47 pm »
@Eugene:

...Or SetLength(A, 0);
Windows 11 UTF8-64 - Lazarus 3.2-64 - FPC 3.2.2

Kays

  • Hero Member
  • *****
  • Posts: 569
  • Whasup!?
    • KaiBurghardt.de
Re: discarding an array
« Reply #5 on: June 02, 2023, 10:26:56 pm »
[…] No, for static typed constant arrays this is not possible, they are part of the binary's (readonly) data section.
Whoops. Typed constants are writable consequently do not reside in .rodata. Setting {$writeableConst off} will yield a compile-time error.

[…] within a second or two my application does:
Code: Pascal  [Select][+][-]
  1.   ExecuteProcess(TempName, '');                                // run binary
[…] the application is strictly linux/unix only if that helps. […]
You are asking to release unused memory. Considering potential motives for doing that: Does your Pascal program do anything thereafter? It sounds like you wanted to use fpExecV (execve(2)).
« Last Edit: June 02, 2023, 10:28:32 pm by Kays »
Yours Sincerely
Kai Burghardt

PascalDragon

  • Hero Member
  • *****
  • Posts: 5444
  • Compiler Developer
Re: discarding an array
« Reply #6 on: June 02, 2023, 10:51:57 pm »
[…] No, for static typed constant arrays this is not possible, they are part of the binary's (readonly) data section.
Whoops. Typed constants are writable consequently do not reside in .rodata. Setting {$writeableConst off} will yield a compile-time error.

That's why I wrote the “readonly” in brackets: It depends on the state of the $J switch as well as on the platform. On Windows the unwritable typed constants will be placed in a readonly section while on ELF-based systems they won't due to issues with relocation. (And the point was that the data is part of the binary's static data section and thus can't be freed no matter what anway)

VisualLab

  • Sr. Member
  • ****
  • Posts: 290
Re: discarding an array
« Reply #7 on: June 03, 2023, 05:07:11 pm »
Maybe the simplest thing would be to load this data from a file into a dynamic array? And when the data is no longer needed, you can free this array.

robert rozee

  • Full Member
  • ***
  • Posts: 153
Re: discarding an array
« Reply #8 on: June 03, 2023, 08:06:10 pm »
You are asking to release unused memory. Considering potential motives for doing that: Does your Pascal program do anything thereafter? It sounds like you wanted to use fpExecV (execve(2)).

yes, here is the (simplified) source. it is an FPC console application, although carries a compressed Lazarus GUI application (1mb compressed, 3.4mb uncompressed) that the console application runs from /tmp. note that after the GUI application exits, the console application needs to delete the BINnnnnn file it created in /tmp before it can exit - so using FpExecv is not an option. interestingly, UPX apparently somehow manages to delete the file from /tmp just before it runs it!

Code: Pascal  [Select][+][-]
  1. program minimal;
  2.  
  3. uses SysUtils;
  4.  
  5. {$I binary.inc}            // original binary (GFXterm)
  6.  
  7. // to create binary.inc use:
  8. // gzip -k <binary>
  9. // bin2obj -o binary.inc -c bindata <binary>.gz
  10. // rm <binary>.gz
  11.  
  12. var TempName:string;
  13.           SR:TSearchRec;
  14.            F:file of byte;
  15.  
  16. begin
  17.   Randomize;
  18.   repeat
  19.     TempName:=GetTempDir+'BIN'+Copy(IntToStr(100000+random(100000)),2,5)
  20.   until FindFirst(TempName+'*', faAnyFile, SR)<>0;             // LINUX: matches both TempName and TempName+'.extension'
  21.   FindClose(SR);
  22.  
  23.   assign(F, TempName+'.gz');
  24.   rewrite(F);
  25.   blockwrite(F, bindata, sizeof(bindata));                     // write base binary image (.gz)
  26.   close(F);
  27.  
  28.   ExecuteProcess('/bin/gzip', ['-d', TempName]);               // uncompress base binary image
  29.   ExecuteProcess('/bin/chmod', ['+x', TempName]);              // mark runable
  30.   ExecuteProcess(TempName, '');                                // run binary
  31.   DeleteFile(TempName)                                         // delete binary
  32. end.

the GUI application consumes around 28mb of RSS when running, while the console application waits consuming about 1.7mb. although this is less than 7% of the GUI app's RSS, i would still like to reduce it if not too difficult. experiments with reducing the payload from 1mb down to around 400k seems to produce an RSS while waiting of some 4k. i've been hitting a brick wall trying to figure out the reason why the sudden jump from 4k to 1.7mb.


cheers,
rob   :-)
« Last Edit: June 05, 2023, 03:39:13 pm by robert rozee »

jamie

  • Hero Member
  • *****
  • Posts: 6077
Re: discarding an array
« Reply #9 on: June 03, 2023, 09:08:33 pm »
I am no expert with the Linux OS/Kernal however, I noticed you are using the FindFirsrt repeatedly without findclose before you do another findfirst? In windows, that will eat up handles and buffers.

 Normally, you use FindFirsrt then FindNext.

 I don't know what FPC does in the background for this but those are the rules that I know of.
 
 If you do multiple FindFirsrt and only one findclose I can see resources hanging around there?

 Just an observation and maybe it doesn't matter.
The only true wisdom is knowing you know nothing

robert rozee

  • Full Member
  • ***
  • Posts: 153
Re: discarding an array
« Reply #10 on: June 03, 2023, 09:54:10 pm »
If you do multiple FindFirsrt and only one FindClose I can see resources hanging around there.

good point. i should have used:

Code: Pascal  [Select][+][-]
  1.   repeat
  2.     TempName:=GetTempDir+'BIN'+Copy(IntToStr(100000+random(100000)),2,5);
  3.     if FindFirst(TempName+'*', faAnyFile, SR)=0 then FindClose(SR)
  4.                                                 else break
  5.   until false;
  6.  

based upon https://www.freepascal.org/docs-html/rtl/sysutils/findfirst.html:
"A successful FindFirst call must always be followed by a FindClose call with the same Rslt record. Failure to do so will result in memory leaks. If the FindFirst call failed (i.e. returned a nonzero handle) there is no need to call FindClose."

although in the present case the loop will almost never be passed through more than once as we are checking that a file name containing a completely random 5-digit number doesn't exist. so the leak is small!


cheers,
rob   :-)
« Last Edit: June 03, 2023, 09:56:03 pm by robert rozee »

robert rozee

  • Full Member
  • ***
  • Posts: 153
[SOLVED] Re: discarding an array
« Reply #11 on: June 05, 2023, 05:25:52 pm »
i found a (simple) solution, making use of FpFork to split the task over two processes:

Code: Pascal  [Select][+][-]
  1. program minimal;
  2.  
  3. uses SysUtils, BaseUnix;
  4.  
  5. {$I binary.inc}            // original binary (GFXterm)
  6.  
  7. // to create binary.inc use:
  8. // gzip -k <binary>
  9. // bin2obj -o binary.inc -c bindata <binary>.gz
  10. // rm <binary>.gz
  11.  
  12. var TempName:string;
  13.           SR:TSearchRec;
  14.            F:file of byte;
  15.            L:longint;
  16.  
  17. begin
  18.   Randomize;
  19.   repeat
  20.     TempName:=GetTempDir+'BIN'+Copy(IntToStr(100000+random(100000)),2,5);
  21.     if FindFirst(TempName+'*', faAnyFile, SR)<>0 then break;   // LINUX: matches both TempName and TempName+'.extension'
  22.     FindClose(SR)
  23.   until false;
  24.  
  25.   if FpFork=0 then
  26.   begin                                                        // *** child process ***
  27.     assign(F, TempName+'.gz');
  28.     rewrite(F);
  29.     blockwrite(F, bindata, sizeof(bindata));                   // write binary image (.gz)
  30.     close(F);
  31.  
  32.     ExecuteProcess('/bin/gzip', ['-d', TempName]);             // uncompress binary image
  33.     ExecuteProcess('/bin/chmod', ['+x', TempName])             // mark as runable
  34.   end
  35.   else
  36.   begin                                                        // *** parent process ***
  37.     FpWait(L);                                                 // wait for child process to exit
  38.     ExecuteProcess(TempName, '');                              // run binary
  39.     DeleteFile(TempName)                                       // delete binary
  40.   end
  41. end.

the child process does all the moving of data and file writing and hence incurs the memory usage penalty by itself - it exits once this is finished, freeing up the memory used. the parent process then carries on and runs the payload (and cleans up /tmp afterwards), consuming just 4k of RSS on my test computer:

Code: Text  [Select][+][-]
  1. user@DH61BE:~$ top
  2.  
  3. top - 03:06:42 up 4 days,  3:41,  1 user,  load average: 0.94, 0.84, 0.66
  4. Tasks: 294 total,   1 running, 286 sleeping,   0 stopped,   7 zombie
  5. %Cpu(s):  1.5 us,  0.8 sy,  0.0 ni, 97.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
  6. MiB Mem :  15832.5 total,   4092.1 free,   2556.2 used,   9184.2 buff/cache
  7. MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.  12518.5 avail Mem
  8.  
  9.     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  10.                                                      
  11.   69435 user      20   0  340620  28684  22816 S   1.0   0.2   0:04.66 BIN74800  <---- payload (GUI)
  12.   69431 user      20   0    1708      4      0 S   0.0   0.0   0:00.00 minimal   <---- container
  13.                                   ^^^^^
  14. physical memory used ---------------+

this solution is far cleaner than messing around with resources or dynamic arrays - i did get a couple of other solutions working, but they still had significant memory overheads.


cheers,
rob   :-)
« Last Edit: June 05, 2023, 05:31:16 pm by robert rozee »

 

TinyPortal © 2005-2018