Recent

Author Topic: Error in TdecompressionStream?  (Read 2437 times)

Grahame Grieve

  • Sr. Member
  • ****
  • Posts: 358
Re: Error in TdecompressionStream?
« Reply #15 on: September 25, 2023, 12:13:49 pm »
I'm not sure where TGZipDecompressionStream comes from? Not in my version of ZStream and perhaps that's my problem. Anyway, for someone else blocked where I am, this code removes the header from all the gz sources I've run into and means that the bytes can be read by TdecompressionStream:

Code: Pascal  [Select][+][-]
  1. function readZLibHeader(stream : TStream) : TBytes;
  2. var
  3.   b : TBytes;
  4.   p : int64;
  5.   i : integer;
  6. begin
  7.   b := StreamToBytes(stream);
  8.   if (length(b) < 10) or (b[0] <> $1F) or (b[1] <> $8B) then
  9.     result := b
  10.   else
  11.   begin
  12.     i := 10;
  13.     if ((b[3] and $08) > 0) then
  14.     begin
  15.       repeat
  16.         inc(i);
  17.       until (i = length(b)) or (b[i] = 0);
  18.       inc(i);
  19.     end;
  20.     if i >= length(b) then
  21.       result := b
  22.     else
  23.        result := copy(b, i, length(b)-i-8);
  24.   end;
  25. end;              
  26.  

Fibonacci

  • Sr. Member
  • ****
  • Posts: 340
  • Certified Internal Error Hunter
Re: Error in TdecompressionStream?
« Reply #16 on: September 25, 2023, 12:21:45 pm »
I'm not sure where TGZipDecompressionStream comes from? Not in my version of ZStream and perhaps that's my problem.

https://gitlab.com/freepascal.org/fpc/source/-/blob/main/packages/paszlib/src/zstream.pp#L124

Grahame Grieve

  • Sr. Member
  • ****
  • Posts: 358
Re: Error in TdecompressionStream?
« Reply #17 on: September 25, 2023, 01:25:24 pm »
Thx. might be time to upgrade soon

Grahame Grieve

  • Sr. Member
  • ****
  • Posts: 358
Re: Error in TdecompressionStream?
« Reply #18 on: December 31, 2023, 09:53:59 pm »
I've found another deflated stream that I can't read with FPC's TDecompression stream. This is part of my unit tests, and it used to work (I can't figure out why, but it sure did).

The stream is attached (z.gz), and I try to decompress it this way:

Code: Pascal  [Select][+][-]
  1. function InflateRfc1951(b : TBytes) : TBytes;
  2. var
  3.   b1, b2 : TBytesStream;
  4.   z : TDecompressionStream;
  5. begin
  6.   b1 := TBytesStream.create(b);
  7.   try
  8.     z := TDecompressionStream.create(b1, false); // same outcome with true
  9.     try
  10.       b2  := TBytesStream.Create;
  11.       try
  12.         b2.CopyFrom(z, z.Size);
  13.         result := b2.Bytes;
  14.         setLength(result, b2.size);
  15.       finally
  16.         b2.free;
  17.       end;
  18.     finally
  19.       z.free;
  20.     end;
  21.   finally
  22.     b1.free;
  23.   end;
  24. end;    
  25.  

I get "Project X raised exception class 'Edecompressionerror' with message: Seek in deflate compressed stream failed."

Versions = Lazarus 3.1 (rev lazarus_3_0-15-g9bef988478) FPC 3.3.1 (trunk) aarch64-darwin-cocoa

Btw, I can read it in Java:

Code: Java  [Select][+][-]
  1. public static final byte[] inflate(byte[] data) throws IOException, DataFormatException {
  2.     final Inflater inflater = new Inflater(true);
  3.     inflater.setInput(data);
  4.  
  5.     try (final ByteArrayOutputStream outputStream = new ByteArrayOutputStream(data.length))
  6.     {
  7.       byte[] buffer = new byte[BUFFER_SIZE];
  8.       while (!inflater.finished())
  9.       {
  10.         final int count = inflater.inflate(buffer);
  11.         outputStream.write(buffer, 0, count);
  12.       }
  13.  
  14.       return outputStream.toByteArray();
  15.     }
  16.   }
  17.  

jamie

  • Hero Member
  • *****
  • Posts: 6030
Re: Error in TdecompressionStream?
« Reply #19 on: December 31, 2023, 11:31:43 pm »
Your method of setting the RESULT looks wrong.

You are using an inner object to directly set the output, that inner object will be invalid when you free B2.

Also, you are setting the length of results after instead of before.

Maybe you should be setting the length of the RESULT and then copy the contents not assigned, therefor, when you free everything, the output will have a copy of it not an image of it which is just a pointer that is now undefined.

The only true wisdom is knowing you know nothing

Grahame Grieve

  • Sr. Member
  • ****
  • Posts: 358
Re: Error in TdecompressionStream?
« Reply #20 on: January 01, 2024, 12:29:59 am »
Maybe, though the assignment works as a byte copy, but the exception is from the copyFrom routine before that happens

jamie

  • Hero Member
  • *****
  • Posts: 6030
Re: Error in TdecompressionStream?
« Reply #21 on: January 01, 2024, 01:26:23 am »
Hmm,. don't know, maybe you need to move the stream pointer back to the start before copy....
The only true wisdom is knowing you know nothing

Grahame Grieve

  • Sr. Member
  • ****
  • Posts: 358
Re: Error in TdecompressionStream?
« Reply #22 on: January 01, 2024, 02:44:47 am »
z.position := 0; before the creation of b2? I tried that, no difference (whew!)

Grahame Grieve

  • Sr. Member
  • ****
  • Posts: 358
Re: Error in TdecompressionStream?
« Reply #23 on: January 04, 2024, 09:01:36 pm »
I get that error on OSX/M1. Reading the same stream on Win64 with "Lazarus 2.2.6 (rev 0df75f4) FPC 3.2.2 x86_64-win64-win32/win64" results in a two byte return - which is not valid.

Then there's there the tgz file here: https://packages.simplifier.net/hl7.fhir.r4b.core/4.3.0. On OSX, this reads fine, but on win64 / linux, reading the stream terminates about 1% of the way in with no error

I'm pretty frustrated at this point - this stuff used to work, and I'm really not familiar with the inner details or how to debug...?

Fibonacci

  • Sr. Member
  • ****
  • Posts: 340
  • Certified Internal Error Hunter
Re: Error in TdecompressionStream?
« Reply #24 on: January 04, 2024, 09:18:18 pm »
Try this:
https://github.com/fibodevy/zflate

Code: Pascal  [Select][+][-]
  1. if gzdecode_file('hl7.fhir.r4b.core-4.3.0.tgz', 'hl7.fhir.r4b.core-4.3.0.tar',
  2.   (function(position, totalsize, outputsize: dword): boolean
  3.   var
  4.     progress: double;
  5.   begin
  6.     progress := position/totalsize*100;
  7.  
  8.     //show progress somewhere?
  9.     //Form1.ProgressBar1.Position := trunc(progress);
  10.     //Form1.Label3.Caption := (FloatToStrF(progress, ffFixed, 2, 2)+'%').Replace(',', '.');
  11.     //Application.ProcessMessages;
  12.  
  13.     result := true; //continue
  14.   end), 10000
  15. ) then
  16.   ShowMessage('Done!')
  17. else begin
  18.   if not (zlasterror = ZFLATE_EABORTED) then
  19.     ShowMessage('Failed');
  20. end;

You can also read whole file to a string and then call gzdecode(your_string) or zdecompress(your_string) and do everything in memory

domasz

  • Sr. Member
  • ****
  • Posts: 388
Re: Error in TdecompressionStream?
« Reply #25 on: January 04, 2024, 09:29:08 pm »
This works with your z.gz file just fine:

Code: Pascal  [Select][+][-]
  1. function inflateCheck(gzipfile:string):string;
  2. var
  3.   f1 : TFileStream;
  4.   gz: Tdecompressionstream;
  5.   cnt:integer;
  6.   len: Integer;
  7.   Chunk: String;
  8. const
  9.   CHUNKSIZE=4096;
  10. begin
  11.   f1 := TFileStream.create(gzipfile, fmOpenRead);
  12.  
  13.   SetLength(Chunk, ChunkSize);
  14.   gz:= Tdecompressionstream.create(f1, true);
  15.  
  16.   try
  17.     repeat
  18.       len := gz.Read(Chunk[1], ChunkSize);
  19.     until len < ChunkSize-1;
  20.  
  21.     result := result + Copy(chunk, 1, len);
  22.   finally
  23.     gz.free;
  24.     f1.free;
  25.   end;
  26. end;  

Grahame Grieve

  • Sr. Member
  • ****
  • Posts: 358
Re: Error in TdecompressionStream?
« Reply #26 on: January 05, 2024, 12:38:17 pm »
@Fibonacci, zflate is very nice code indeed that is exactly what I dreamed of. Thank you very much, it has solved my problem very nicely.

 

TinyPortal © 2005-2018