Error in TdecompressionStream?

[Grahame Grieve]

I'm not sure where TGZipDecompressionStream comes from? Not in my version of ZStream and perhaps that's my problem. Anyway, for someone else blocked where I am, this code removes the header from all the gz sources I've run into and means that the bytes can be read by TdecompressionStream:

Code: Pascal  [Select][+][-]

1. function readZLibHeader(stream : TStream) : TBytes;
2. var
3.   b : TBytes;
4.   p : int64;
5.   i : integer;
6. begin
7.   b := StreamToBytes(stream);
8.   if (length(b) < 10) or (b[0] <> $1F) or (b[1] <> $8B) then
9.     result := b
10.   else
11.   begin
12.     i := 10;
13.     if ((b[3] and $08) > 0) then
14.     begin
15.       repeat
16.         inc(i);
17.       until (i = length(b)) or (b[i] = 0);
18.       inc(i);
19.     end;
20.     if i >= length(b) then
21.       result := b
22.     else
23.        result := copy(b, i, length(b)-i-8);
24.   end;
25. end;              
26.  

[Fibonacci]

I'm not sure where TGZipDecompressionStream comes from? Not in my version of ZStream and perhaps that's my problem.

https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/paszlib/src/zstream.pp#L124

[Grahame Grieve]

Thx. might be time to upgrade soon

[Grahame Grieve]

I've found another deflated stream that I can't read with FPC's TDecompression stream. This is part of my unit tests, and it used to work (I can't figure out why, but it sure did).

The stream is attached (z.gz), and I try to decompress it this way:

Code: Pascal  [Select][+][-]

1. function InflateRfc1951(b : TBytes) : TBytes;
2. var
3.   b1, b2 : TBytesStream;
4.   z : TDecompressionStream;
5. begin
6.   b1 := TBytesStream.create(b);
7.   try
8.     z := TDecompressionStream.create(b1, false); // same outcome with true
9.     try
10.       b2  := TBytesStream.Create;
11.       try
12.         b2.CopyFrom(z, z.Size);
13.         result := b2.Bytes;
14.         setLength(result, b2.size);
15.       finally
16.         b2.free;
17.       end;
18.     finally
19.       z.free;
20.     end;
21.   finally
22.     b1.free;
23.   end;
24. end;    
25.  

I get "Project X raised exception class 'Edecompressionerror' with message: Seek in deflate compressed stream failed."

Versions = Lazarus 3.1 (rev lazarus_3_0-15-g9bef988478) FPC 3.3.1 (trunk) aarch64-darwin-cocoa

Btw, I can read it in Java:

Code: Java  [Select][+][-]

1. public static final byte[] inflate(byte[] data) throws IOException, DataFormatException {
2.     final Inflater inflater = new Inflater(true);
3.     inflater.setInput(data);
4.  
5.     try (final ByteArrayOutputStream outputStream = new ByteArrayOutputStream(data.length))
6.     {
7.       byte[] buffer = new byte[BUFFER_SIZE];
8.       while (!inflater.finished())
9.       {
10.         final int count = inflater.inflate(buffer);
11.         outputStream.write(buffer, 0, count);
12.       }
13.  
14.       return outputStream.toByteArray();
15.     }
16.   }
17.  

[jamie]

Your method of setting the RESULT looks wrong.

You are using an inner object to directly set the output, that inner object will be invalid when you free B2.

Also, you are setting the length of results after instead of before.

Maybe you should be setting the length of the RESULT and then copy the contents not assigned, therefor, when you free everything, the output will have a copy of it not an image of it which is just a pointer that is now undefined.

[Grahame Grieve]

Maybe, though the assignment works as a byte copy, but the exception is from the copyFrom routine before that happens

[jamie]

Hmm,. don't know, maybe you need to move the stream pointer back to the start before copy....

[Grahame Grieve]

z.position := 0; before the creation of b2? I tried that, no difference (whew!)

[Grahame Grieve]

I get that error on OSX/M1. Reading the same stream on Win64 with "Lazarus 2.2.6 (rev 0df75f4) FPC 3.2.2 x86_64-win64-win32/win64" results in a two byte return - which is not valid.

Then there's there the tgz file here: https://packages.simplifier.net/hl7.fhir.r4b.core/4.3.0. On OSX, this reads fine, but on win64 / linux, reading the stream terminates about 1% of the way in with no error

I'm pretty frustrated at this point - this stuff used to work, and I'm really not familiar with the inner details or how to debug...?

[Fibonacci]

Try this:
https://github.com/fibodevy/zflate

Code: Pascal  [Select][+][-]

1. if gzdecode_file('hl7.fhir.r4b.core-4.3.0.tgz', 'hl7.fhir.r4b.core-4.3.0.tar',
2.   (function(position, totalsize, outputsize: dword): boolean
3.   var
4.     progress: double;
5.   begin
6.     progress := position/totalsize*100;
7.  
8.     //show progress somewhere?
9.     //Form1.ProgressBar1.Position := trunc(progress);
10.     //Form1.Label3.Caption := (FloatToStrF(progress, ffFixed, 2, 2)+'%').Replace(',', '.');
11.     //Application.ProcessMessages;
12.  
13.     result := true; //continue
14.   end), 10000
15. ) then
16.   ShowMessage('Done!')
17. else begin
18.   if not (zlasterror = ZFLATE_EABORTED) then
19.     ShowMessage('Failed');
20. end;

You can also read whole file to a string and then call gzdecode(your_string) or zdecompress(your_string) and do everything in memory

[domasz]

This works with your z.gz file just fine:

Code: Pascal  [Select][+][-]

1. function inflateCheck(gzipfile:string):string;
2. var
3.   f1 : TFileStream;
4.   gz: Tdecompressionstream;
5.   cnt:integer;
6.   len: Integer;
7.   Chunk: String;
8. const
9.   CHUNKSIZE=4096;
10. begin
11.   f1 := TFileStream.create(gzipfile, fmOpenRead);
12.  
13.   SetLength(Chunk, ChunkSize);
14.   gz:= Tdecompressionstream.create(f1, true);
15.  
16.   try
17.     repeat
18.       len := gz.Read(Chunk[1], ChunkSize);
19.     until len < ChunkSize-1;
20.  
21.     result := result + Copy(chunk, 1, len);
22.   finally
23.     gz.free;
24.     f1.free;
25.   end;
26. end;  

[Grahame Grieve]

@Fibonacci, zflate is very nice code indeed that is exactly what I dreamed of. Thank you very much, it has solved my problem very nicely.