Error in TdecompressionStream?

[Grahame Grieve]

I'm having trouble reading the attached file (a gzipped json file). If I read it this way:

Code: Pascal  [Select][+][-]

1. uses zStream;
2.  
3. function inflateCheck(gzipfile:string):string;
4. var
5.   gz: TGZFileStream;
6.   chunk:string;
7.   cnt:integer;
8. const
9.   CHUNKSIZE=4096;
10. begin
11. gz:= TGZFileStream.create(gzipfile,gzopenread);
12. result:='';
13. setlength(chunk,CHUNKSIZE);
14. repeat
15.   cnt:=gz.read(chunk[1],CHUNKSIZE);
16.   if cnt<CHUNKSIZE then
17.     setlength(chunk,cnt);
18.   result:=result+chunk;
19. until cnt<CHUNKSIZE;
20. end;    
21.  

Then I can read the file no problems. But if I read it this way

Code: Pascal  [Select][+][-]

1. uses zStream;
2.  
3. function inflateCheck(gzipfile:string):string;
4. var
5.   f1 : TFileStream;
6.   b2 : TBytesStream;
7.   gz: Tdecompressionstream;
8.   cnt:integer;
9. const
10.   CHUNKSIZE=4096;
11. begin
12.   f1 := TFileStream.create(gzipfile, fmOpenRead);
13.   b2 := TBytesStream.create;
14.   gz:= Tdecompressionstream.create(f1, false);
15.   try
16.     b2.CopyFrom(gz, gz.size);
17.     result := TEncoding.UTF8.GetString(b2.Bytes);
18.   finally
19.     gz.free;
20.     f1.free;
21.     b2.free;
22.   end;
23. end;          
24.  

Then I get "seek in deflate compressed stream failed". Why? I can't see that I'm doing anything wrong in the second approach, and I really don't want to read a file - this is a blob in a database.

Version: Lazarus 2.2.7 (rev lazarus_2_2_6-1-gada7a90f86) FPC 3.2.3 aarch64-darwin-cocoa

The gzipped bytes, btw, were produced using this java code:

Code: Java  [Select][+][-]

1.  public static byte[] gzip(byte[] bytes) throws IOException {
2.     ByteArrayOutputStream bOut = new ByteArrayOutputStream();
3.    
4.     GzipParameters gp = new GzipParameters();
5.     gp.setCompressionLevel(Deflater.BEST_COMPRESSION);
6.     GzipCompressorOutputStream gzip = new GzipCompressorOutputStream(bOut, gp);
7.     gzip.write(bytes);
8.     gzip.flush();
9.     gzip.close();
10.     return bOut.toByteArray();
11.   }
12.  

[Fibonacci]

There are 3 gz formats
- gzdeflate (pure data)
- gzcompress (header added, ZLIB format)
- gzencode (header and footer checksum added, GZIP, used in .gz files)

Provide a sample compressed by your Java code

[Grahame Grieve]

I did - that's the attachment

[Fibonacci]

Didnt notice

decompressed 959 bytes to 3103 bytes

Code: Pascal  [Select][+][-]

1. uses ZStream, Classes;
2.  
3. procedure DecompressStream(src, dst: TStream);
4. var
5.   ds: TDecompressionStream;
6.   d: dword;
7.   buff: array[0..1023] of byte;
8. begin
9.   ds := TDecompressionStream.Create(src, true);
10.   try
11.     repeat
12.       d := ds.Read(buff, 1024);
13.       dst.Write(buff, d);
14.     until
15.       d = 0;
16.   finally
17.     ds.Free;
18.   end;
19. end;
20.  
21. var
22.   ss1, ss2: TStringStream;
23. begin
24.   ss1 := TStringStream.Create;
25.   ss1.LoadFromFile('gzip.gz');
26.   ss1.Position := 10; //SKIP GZIP HEADER
27.  
28.   ss2 := TStringStream.Create;
29.  
30.   DecompressStream(ss1, ss2);
31.  
32.   writeln('decompressed ', ss1.Size, ' bytes to ', ss2.Size, ' bytes');
33.  
34.   ss2.Position := 0;
35.   ss2.SaveToFile('out.txt');
36.  
37.   ss1.Free;
38.   ss2.Free;
39.  
40.   readln;
41. end.

[Fibonacci]

Code: Pascal  [Select][+][-]

1. ss1.Position := 10; //SKIP GZIP HEADER

At the end of the file there is:
- 4 bytes CRC checksum
- 4 bytes decompressed data size

You should cut those too, and maybe check if checksum is valid

[TRon]

...and I really don't want to read a file - this is a blob in a database.

A blobfield can be loaded into a stream, for example a memorystream with savetostream.

Fibonacci's code can simply be adjust to accommodate.

[jamie]

Code: Pascal  [Select][+][-]

1.     b2.CopyFrom(gz, gz.size);
2.     result := TEncoding.UTF8.GetString(b2.Bytes);
3.  

Not sure but, that may have worked if the B2 stream was reset back to the start, I think most encoding is done at the start?

[Grahame Grieve]

Thanks for this - I appreciate it. But I don't understand it. Isn't that what skipHeader is about? How would you know whether you need to manually skip some bytes at the start, and how dod you know how many bytes to skip? is it always 10?

[TRon]

Thanks for this - I appreciate it. But I don't understand it. Isn't that what skipHeader is about? How would you know whether you need to manually skip some bytes at the start, and how dod you know how many bytes to skip? is it always 10?

By (manually) analyzing the file that you posted.

Some gzip send data contains a header. html gzip compression does something similar for example (took me ages to figure out that it included a header). So yes, if your files always come from the same source or are obtained in the same manner then you would have to assume that the header is present.

see also: https://en.wikipedia.org/wiki/Gzip#File_format

[Fibonacci]

For GZIP its always 10 bytes header and 8 bytes footer. Actually in some cases it may contain optional header for eg filename, but if you compress RAW data then it will be 10 bytes.

In first example you used "TGZFileStream", its for files and it knows the file has header and footer.

In your second example you used "Tdecompressionstream" which does things at lower level, it doesnt expect header.

I am not 100% sure but ASkipHeader I did set to true was to skip ZLIB header (2 bytes I think), because (not sure) Tdecompressionstream is for ZLIB with 2 byte header, so you would need to cut 10 bytes GZIP header and add 2 bytes ZLIB header.

[Grahame Grieve]

So (a) would it be good if TDecompressionStream could handle this header? (b) Would it be good if this stuff was documented in the unit? Or somewhere?

[Fibonacci]

would it be good if TDecompressionStream could handle this header?

No, because this is a different header. Other than different header, ZLIB has also Adler32 checksum at the end, while GZIP uses CRC32.

TDecompressionStream is for ZLIB stream, but you can skip the header and decompress pure deflated data.

Would it be good if this stuff was documented in the unit? Or somewhere?

I dont know, maybe it could be explained better somewhere. I am not entitled to write FPC documentation.

Documentation states
"If ASkipHeader is true, then the gzip data header is skipped"

That is not true, what is skipped is ZLIB header.

If its feasible I would suggest you using pure deflated data without any headers. Then in FPC use TDecompressionStream with SkipHeaders - its the same as you would just do "inflate".

[Fibonacci]

Just to make sure about ZLIB header, I compressed a string with TCompressionStream, then I decompressed it using SkipHeaders, and

Code: Pascal  [Select][+][-]

1. ss1.Position := 2; //SKIP ZLIB HEADER

So yes, its 2 bytes + adler32 at the end.

GZIP is for files. Use either ZLIB (with Adler32 checksum) or pure deflate.

[Grahame Grieve]

It's the GZip file header per the documentation here. https://docs.fileformat.com/compression/gz/

I think that TdecompressionStream should be able to read it just like the file variant, but if it can't, it should at least document the difference. (IMHO)

[Fibonacci]

I looked at ZStream unit and here you go, improved version for your use case

decompressed 959 bytes to 3103 bytes
part of decompressed data
{"resourceType":"ConceptMap","

Code: Pascal  [Select][+][-]

1. uses ZStream, Classes;
2.  
3. var
4.   sinput, soutput: TStringStream;
5.   gz: TGZipDecompressionStream;      
6.   buff: array[0..1024*32-1] of byte;
7.   d: dword;
8.  
9. begin
10.   //input
11.   sinput := TStringStream.Create;
12.   sinput.LoadFromFile('gzip.gz');
13.  
14.   //output
15.   soutput := TStringStream.Create;
16.  
17.   //decompress gzip blob
18.   gz := TGZipDecompressionStream.Create(sinput);
19.   repeat
20.     d := gz.Read(buff, sizeof(buff));
21.     soutput.Write(buff, d);
22.   until d = 0;    
23.   gz.Free;
24.  
25.   writeln('decompressed ', sinput.Size, ' bytes to ', soutput.Size, ' bytes');
26.   writeln('part of decompressed data');
27.   writeln(copy(soutput.DataString, 1, 30));
28.  
29.   sinput.Free;
30.   soutput.Free;
31.  
32.   readln;
33. end.