using bin2obj, how to create a gzip archive?

[robert rozee]

hi,
    i'm using bin2obj (the utility supplied with FPC) with the -z switch to compress then convert a binary file to a pascal array. this uses the construct CompStream:=TCompressionStream.Create(cldefault,Result) to perform the compression, and according to this page https://linux.die.net/man/1/bin2obj "zlib compression is used".

once the conversion to an array is completed, i want my program to be able to:
1. write the array contents out to a file in /tmp,
2. use the unix gzip utility to uncompress the file,
3. use chmod to mark the file as runnable,
4. and then run it.

all works well, except that i can not for the life of me figure out the exact require header and (particularly) footer needs to be added to keep gzip happy! i started out working from info here: https://unix.stackexchange.com/questions/22834/how-to-uncompress-zlib-data-in-unix, but have now hit a brick wall.

has anyone else been down this path and know the final incantations required?

cheers,
rob   :-)


here is what running my code produces:

Code: Text  [Select][+][-]

1. user@DH61BE:~/pascal/tests/test 203 (bin2obj)$ bin2obj -z -o binary.inc -c bindata GFXterm
2. [ *** hit compile button in IDE *** ]
3. user@DH61BE:~/pascal/tests/test 203 (bin2obj)$ ./project1
4.      100%
5. CRC  = 52 91 A9 E5
6. size = 20 42 36 00
7. user@DH61BE:~/pascal/tests/test 203 (bin2obj)$ uncompress -l GFXrun.gz
8.          compressed        uncompressed  ratio uncompressed_name
9.             1062874             3555872  70.1% GFXrun
10. user@DH61BE:~/pascal/tests/test 203 (bin2obj)$ uncompress  GFXrun.gz
11.  
12. gzip: GFXrun.gz: invalid compressed data--crc error
13.  
14. gzip: GFXrun.gz: invalid compressed data--length error
15.  

and here is my source:

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. uses Sysutils;
4.  
5. {$I binary.inc}
6.  
7. // to create binary.inc use:
8. // bin2obj -z -o binary.inc -c bindata GFXterm
9.  
10. const CRC:int64=$e5a99152;         // generated by /bin/crc32
11.       FSz:int64=3555872;           // original file size
12.  
13. function I2C(I:int64; B:byte):char;
14. begin
15.   I:=I shr (B*8);
16.   result:=chr(I and $FF)
17. end;
18.  
19. var I:integer;
20.     T:text;
21.  
22. begin
23. //assign(T, '/tmp/GFXrun');
24.   assign(T, 'GFXrun.gz');              // work in local directory for the moment
25.   rewrite(T);
26.  
27.   write(T, #$1f, #$8b, #$08, #$00,     // gzip preamble from https://datatracker.ietf.org/doc/html/rfc1952
28.            #$00, #$00, #$00, #$00);
29.  
30.   for I:=low(bindata) to high(bindata) do
31.   begin
32.     write(T,chr(bindata[I]));
33.     if (I and $3)=0 then write((I*100) div high(bindata):8, '%', #13)
34.   end;
35.   writeln(100:8, '%');                 // compressed data written out to file
36.  
37.   write('CRC  = ');
38.   for I:=0 to 3 do
39. //for I:=3 downto 0 do
40.   begin                                // crude way of adding CRC as footer
41.     write(T, I2C(CRC, I));
42.     write(IntToHex(ord(I2C(CRC, I)), 2), #32)
43.   end;
44.   writeln;
45.  
46.   write('size = ');
47.   for I:=0 to 3 do                     // crude way of adding original file size to footer
48. //for I:=3 downto 0 do
49.   begin
50.     write(T, I2C(FSz, I));
51.     write(IntToHex(ord(I2C(FSz, I)), 2), #32)
52.   end;
53.   writeln;
54.  
55.   close(T);                            // stop here as we can't uncompress it yet!!
56. {
57.   ExecuteProcess('/bin/gzip', '-d /tmp/GFXrun');               // FAILS!!!
58.   ExecuteProcess('/bin/chmod', '+x /tmp/GFXrun');              // works OK
59.   ExecuteProcess('/tmp/GFXrun', '');                           // works OK
60.   ExecuteProcess('/bin/rm', '/tmp/GFXrun') }                   // works OK
61. end.
62.  

[robert rozee]

and, of course, in the light of day the solution is obvious: just use gzip to do the compression!

the below works remarkably well, the only major catch is that the default directory for the run application is /tmp.


cheers,
rob   :-)

Code: Pascal  [Select][+][-]

1. program project1;
2.  
3. uses SysUtils;
4.  
5. {$I binary.inc}
6.  
7. // to create binary.inc use:
8. // gzip -k <binary>
9. // bin2obj -o binary.inc -c bindata <binary>.gz
10. // rm <binary>.gz
11.  
12. var TempName:string;
13.            I:integer;
14.            T:text;
15.  
16. begin                                                  // I:=bindata[0] end.
17.   Randomize;
18.   repeat
19.     TempName:=GetTempDir+'BIN'+Copy(IntToStr(100000+random(100000)),2,5)
20.   until not (FileExists(TempName) or FileExists(TempName+'.gz'));
21.   assign(T, TempName+'.gz');
22.   rewrite(T);                                          // create temp file
23.  
24.   for I:=low(bindata) to high(bindata) do
25.   begin
26.     write(T,chr(bindata[I]));
27. { if (I and $FF)=0 then } write((I*100) div sizeof(bindata):3, '%', #13)
28.   end;
29.   writeln(100:3, '% : '+TempName);                     // .gz file created
30.   close(T);
31.  
32.   writeln('uncompress image');
33.   ExecuteProcess('/bin/gzip', ['-d', TempName]);       // uncompress
34.   writeln('apply chmod +x');
35.   ExecuteProcess('/bin/chmod', ['+x', TempName]);      // mark runable
36.   writeln('run binary image');
37.   ExecuteProcess(TempName, '');                        // run binary
38.   writeln('cleanup /tmp');
39.   ExecuteProcess('/bin/rm', TempName);                 // delete binary
40.   writeln('finished OK')
41. end.
42.  

addendum1: slightly updated above code to use GetTempFileName
addendum2: changed to random temp file name with clash detection
addendum3: removed -z parameter i'd accidentally left in comment

[marcov]

Afaik gzip has an additional header before the zlib stream, with CRC32 etc. So that is not 100% the same.  In zstream you can clearly see separate TCompressionStream and TGzipCompressionStream classes.

For a FPC only example, see the  "minigzip.pp" example in the packages/paszlib/examples and the other files there

[robert rozee]

Afaik gzip has an additional header before the zlib stream, with CRC32 etc. So that is not 100% the same.  In zstream you can clearly see separate TCompressionStream and TGzipCompressionStream classes.

aha, that makes sense! so by the looks of it, in /usr/share/fpcsrc/3.2.2/utils/bin2obj.pp one merely needs to change line 127 to create a .gz compatible output?

Code: Pascal  [Select][+][-]

1.   If ComPressData then
2.     begin
3. //  CompStream:=TCompressionStream.Create(cldefault,Result);
4.     CompStream:=TGZipCompressionStream.Create(cldefault,Result);
5.     Result:=CompStream;
6.     end;

if it is in fact this easy, it could be a handy addition to the bin2obj utility, controlled by a command line switch to select between 'zlib' and 'gzip' compatibility. in my present user case it doesn't matter so much, as now that i understand what is going on i can just use the gzip provided by linux.

next step is to find a method of binary-level patching, so that i can create a difference file between two binaries (built against glib 2.34 and glib 'old') and create one or the other on-the-fly to be run depending on the version of libc installed..


cheers,
rob   :-)