[SOLVED] discarding an array

[robert rozee]

perhaps a silly question: is there any way to discard a typed constant array at runtime?

the situation is i have several rather large arrays, of the form

Code: Pascal  [Select][+][-]

1. Const
2.   bindata : Array[0..1066635] of byte = (
3.       31,139,  8,  8,242,169,120,100,  0,  3, 71, 70, 88,116,101,114,109,
4.        0,172,157,  7,120, 85,197,246,197,111, 66,132,  0,210,123, 15,160,
5.      244,222, 59,161,247, 94, 85, 48,  4,114, 19,  2, 33,  9, 73, 40,210,
6.     ...
7. Const
8.   patch1 : Array[0..315288] of byte = (
9.       31,139,  8,  8,237,165,120,100,  0,  3,112, 49,100,105,102,102,115,
10.       46,116,120,116,  0,236,253,187,210,236, 58,175, 45,  8,250,124,138,
11.      109,118, 25, 43, 66, 41,165,164, 76,183,162,162,204,118,250,  9,242,
12. ... etc.

there have been generated by bin2obj. i use them at the start of my application, but thereafter they are no longer needed. indeed, within a second or two my application does:

Code: Pascal  [Select][+][-]

1.   ExecuteProcess(TempName, '');                                // run binary

in order to keep things as tidy as possible, it would be nice to free up the arrays before essentially going dormant. any suggestions on how to achieve this? the application is strictly linux/unix only if that helps.


cheers,
rob   :-)

[Eugene Loza]

I don't know any way to discard a constant in Pascal... I wonder if there is any reason you don't want to have them as dynamic arrays? You literally just replace "const" with "var" and don't even have to make any conversion. Like this:

Code: Pascal  [Select][+][-]

1. var
2.   A: array of Byte = (1,2,3,4);
3. begin
4.   //Work with A
5.   A := nil; // A will be free now
6. end.

Note: this may be a new feature of FPC 3.3.1

[jamie]

Link that data in as a resource and load it when needed into dynamic array.

It will still be attached to the EXE but not in memory once you free the dynamic array.

[PascalDragon]

perhaps a silly question: is there any way to discard a typed constant array at runtime?

No, for static typed constant arrays this is not possible, they are part of the binary's (readonly) data section.

[jcmontherock]

@Eugene:

...Or SetLength(A, 0);

[Kays]

[…] No, for static typed constant arrays this is not possible, they are part of the binary's (readonly) data section.

Whoops. Typed constants are writable consequently do not reside in .rodata. Setting {$writeableConst off} will yield a compile-time error.

[…] within a second or two my application does:

Code: Pascal  [Select][+][-]

1.   ExecuteProcess(TempName, '');                                // run binary

[…] the application is strictly linux/unix only if that helps. […]

You are asking to release unused memory. Considering potential motives for doing that: Does your Pascal program do anything thereafter? It sounds like you wanted to use fpExecV (execve(2)).

[PascalDragon]

[…] No, for static typed constant arrays this is not possible, they are part of the binary's (readonly) data section.

Whoops. Typed constants are writable consequently do not reside in .rodata. Setting {$writeableConst off} will yield a compile-time error.

That's why I wrote the “readonly” in brackets: It depends on the state of the $J switch as well as on the platform. On Windows the unwritable typed constants will be placed in a readonly section while on ELF-based systems they won't due to issues with relocation. (And the point was that the data is part of the binary's static data section and thus can't be freed no matter what anway)

[VisualLab]

Maybe the simplest thing would be to load this data from a file into a dynamic array? And when the data is no longer needed, you can free this array.

[robert rozee]

You are asking to release unused memory. Considering potential motives for doing that: Does your Pascal program do anything thereafter? It sounds like you wanted to use fpExecV (execve(2)).

yes, here is the (simplified) source. it is an FPC console application, although carries a compressed Lazarus GUI application (1mb compressed, 3.4mb uncompressed) that the console application runs from /tmp. note that after the GUI application exits, the console application needs to delete the BINnnnnn file it created in /tmp before it can exit - so using FpExecv is not an option. interestingly, UPX apparently somehow manages to delete the file from /tmp just before it runs it!

Code: Pascal  [Select][+][-]

1. program minimal;
2.  
3. uses SysUtils;
4.  
5. {$I binary.inc}            // original binary (GFXterm)
6.  
7. // to create binary.inc use:
8. // gzip -k <binary>
9. // bin2obj -o binary.inc -c bindata <binary>.gz
10. // rm <binary>.gz
11.  
12. var TempName:string;
13.           SR:TSearchRec;
14.            F:file of byte;
15.  
16. begin
17.   Randomize;
18.   repeat
19.     TempName:=GetTempDir+'BIN'+Copy(IntToStr(100000+random(100000)),2,5)
20.   until FindFirst(TempName+'*', faAnyFile, SR)<>0;             // LINUX: matches both TempName and TempName+'.extension'
21.   FindClose(SR);
22.  
23.   assign(F, TempName+'.gz');
24.   rewrite(F);
25.   blockwrite(F, bindata, sizeof(bindata));                     // write base binary image (.gz)
26.   close(F);
27.  
28.   ExecuteProcess('/bin/gzip', ['-d', TempName]);               // uncompress base binary image
29.   ExecuteProcess('/bin/chmod', ['+x', TempName]);              // mark runable
30.   ExecuteProcess(TempName, '');                                // run binary
31.   DeleteFile(TempName)                                         // delete binary
32. end.

the GUI application consumes around 28mb of RSS when running, while the console application waits consuming about 1.7mb. although this is less than 7% of the GUI app's RSS, i would still like to reduce it if not too difficult. experiments with reducing the payload from 1mb down to around 400k seems to produce an RSS while waiting of some 4k. i've been hitting a brick wall trying to figure out the reason why the sudden jump from 4k to 1.7mb.


cheers,
rob   :-)

[jamie]

I am no expert with the Linux OS/Kernal however, I noticed you are using the FindFirsrt repeatedly without findclose before you do another findfirst? In windows, that will eat up handles and buffers.

 Normally, you use FindFirsrt then FindNext.

 I don't know what FPC does in the background for this but those are the rules that I know of.
 
 If you do multiple FindFirsrt and only one findclose I can see resources hanging around there?

 Just an observation and maybe it doesn't matter.

[robert rozee]

If you do multiple FindFirsrt and only one FindClose I can see resources hanging around there.

good point. i should have used:

Code: Pascal  [Select][+][-]

1.   repeat
2.     TempName:=GetTempDir+'BIN'+Copy(IntToStr(100000+random(100000)),2,5);
3.     if FindFirst(TempName+'*', faAnyFile, SR)=0 then FindClose(SR)
4.                                                 else break
5.   until false;
6.  

based upon https://www.freepascal.org/docs-html/rtl/sysutils/findfirst.html:
"A successful FindFirst call must always be followed by a FindClose call with the same Rslt record. Failure to do so will result in memory leaks. If the FindFirst call failed (i.e. returned a nonzero handle) there is no need to call FindClose."

although in the present case the loop will almost never be passed through more than once as we are checking that a file name containing a completely random 5-digit number doesn't exist. so the leak is small!


cheers,
rob   :-)

[robert rozee]

i found a (simple) solution, making use of FpFork to split the task over two processes:

Code: Pascal  [Select][+][-]

1. program minimal;
2.  
3. uses SysUtils, BaseUnix;
4.  
5. {$I binary.inc}            // original binary (GFXterm)
6.  
7. // to create binary.inc use:
8. // gzip -k <binary>
9. // bin2obj -o binary.inc -c bindata <binary>.gz
10. // rm <binary>.gz
11.  
12. var TempName:string;
13.           SR:TSearchRec;
14.            F:file of byte;
15.            L:longint;
16.  
17. begin
18.   Randomize;
19.   repeat
20.     TempName:=GetTempDir+'BIN'+Copy(IntToStr(100000+random(100000)),2,5);
21.     if FindFirst(TempName+'*', faAnyFile, SR)<>0 then break;   // LINUX: matches both TempName and TempName+'.extension'
22.     FindClose(SR)
23.   until false;
24.  
25.   if FpFork=0 then
26.   begin                                                        // *** child process ***
27.     assign(F, TempName+'.gz');
28.     rewrite(F);
29.     blockwrite(F, bindata, sizeof(bindata));                   // write binary image (.gz)
30.     close(F);
31.  
32.     ExecuteProcess('/bin/gzip', ['-d', TempName]);             // uncompress binary image
33.     ExecuteProcess('/bin/chmod', ['+x', TempName])             // mark as runable
34.   end
35.   else
36.   begin                                                        // *** parent process ***
37.     FpWait(L);                                                 // wait for child process to exit
38.     ExecuteProcess(TempName, '');                              // run binary
39.     DeleteFile(TempName)                                       // delete binary
40.   end
41. end.

the child process does all the moving of data and file writing and hence incurs the memory usage penalty by itself - it exits once this is finished, freeing up the memory used. the parent process then carries on and runs the payload (and cleans up /tmp afterwards), consuming just 4k of RSS on my test computer:

Code: Text  [Select][+][-]

1. user@DH61BE:~$ top
2.  
3. top - 03:06:42 up 4 days,  3:41,  1 user,  load average: 0.94, 0.84, 0.66
4. Tasks: 294 total,   1 running, 286 sleeping,   0 stopped,   7 zombie
5. %Cpu(s):  1.5 us,  0.8 sy,  0.0 ni, 97.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
6. MiB Mem :  15832.5 total,   4092.1 free,   2556.2 used,   9184.2 buff/cache
7. MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.  12518.5 avail Mem
8.  
9.     PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
10.                                                      
11.   69435 user      20   0  340620  28684  22816 S   1.0   0.2   0:04.66 BIN74800  <---- payload (GUI)
12.   69431 user      20   0    1708      4      0 S   0.0   0.0   0:00.00 minimal   <---- container
13.                                   ^^^^^
14. physical memory used ---------------+

this solution is far cleaner than messing around with resources or dynamic arrays - i did get a couple of other solutions working, but they still had significant memory overheads.


cheers,
rob   :-)