Indy Gzip library compression problem

[dinmil]

Hello

I wrote some http server application for windows 64 /32 which use gzip indy compression to compress output.
Problem is that from time to time I get zlib error -5 when I try to compress some string.
In attachment you can find my isolated test case which proves that some string can not be compressed using Indy routines.
I thing this is bug inside Indy Library, and I need confirmation or fix. Any help is appreciated.

(*
The purpose of test is to show that Indy
library can not GZIP compress some string
String is saved in file uncompresable.xml
String is loaded into memory and
I cut peace of files and compresse it
After some time I found on peace which
cannot be compressed and program returns
zlib error -5 which means that buffer is
not good. If you change MemLevel from 9 to
lower values or compression level from 9 to
no lower values you can compress this string
but eventually if you try some other strings
you get the same result (-5).
Indy use zlib1.dll (ver 1.2. for 32 bit windows
and zlibwapi.dll (ver 1.2.5) for 64 bit windows
I think that something is wrong with Indy
implementation of compression and the way
how -5 error should be interpreted but I'm
not sure. I tried some other pascal zlib libraries
and everyhing is good.
Used version is Windows CodeTyphon 5.1.0, Fpc = 2.7.1,
pl_Indy Source (24-10-2014 SVN Rev 5201)
I appreciate any help from Indy team or CodeTyphon team or
Lazarus team
*)

dll._zip contain zlib1 for 32 windows and zlibwapi from 64 bit windows.
Rename and uncompress it if you do not have your own version.

Regards, Dinko

[ChrisF]

Apparently, you've got a problem with an incomplete or empty buffer case.

And it seems it's not really a new problem, though not exactly yours. See: http://forums2.atozed.com/viewtopic.php?f=7&t=26206

After just a few tests, the following fix was OK for me (but I'm not an expert): Indy 10.6.1 rev 5217 and zlib version 1.2.8.0

In Lib/Protocols/IdZlib.pas, in procedure IndyCompressStream:

Code: [Select]

...
      repeat
//&&&& Fix (add the following lines)
        if (strm.avail_in = 0) and (strm.avail_out = 0) then begin
          WriteOut;
        end;
//&&&& Fix end
        Finished := CCheck(deflate(strm, Z_FINISH)) = Z_STREAM_END;
        WriteOut;
      until Finished;
...

Anyway, I suggest you to report your problem to Remy Lebeau in the Indy forum: http://forums2.atozed.com/index.php,

in order to get a full a more complete answer, and a more proper fix for your problem.

[dinmil]

Thanks, for quick response. I will check soon your fix and report results.

[dinmil]

After some test I done with your fix implemented, I can confirm that this fix solve the problem for me.
I'm still waiting Indy guys for answers.

I wrote test program which gzip and ungzip many strings and checking that Input and Output strings are the same.
So far they are, program is still running :-)

Thank you. You saved my life :-)

[Remy Lebeau]

And it seems it's not really a new problem, though not exactly yours. See: http://forums2.atozed.com/viewtopic.php?f=7&t=26206

The problem is, that discussion was never finalized, so no changes were made to Indy.  I'm not a ZLib expert, and I didn't (still don't) have to debug and test ZLib issues.

After just a few tests, the following fix was OK for me (but I'm not an expert): Indy 10.6.1 rev 5217 and zlib version 1.2.8.0

Which do you think is better?

Code: [Select]

...
      repeat
//&&&& Fix (add the following lines)
        if (strm.avail_in = 0) and (strm.avail_out = 0) then begin
          WriteOut;
        end;
//&&&& Fix end
        Finished := CCheck(deflate(strm, Z_FINISH)) = Z_STREAM_END;
        WriteOut;
      until Finished;
...

Or the fix proposed in the earlier discussion:

Code: [Select]

var
  CResult: Integer;
{... snip ...}
      repeat
        CResult := deflate(strm, Z_FINISH);
        if CResult <> Z_BUF_ERROR then
        begin
          CCheck(CResult);
        end;

        WriteOut;
      until ((CResult = Z_STREAM_END) and (strm.avail_out > 0));


[ChrisF]

I'm certainly not a Zlib expert neither.

But I find the first approach (i.e. calling WriteOut before the final deflate(s)) more logical. Though I think it could be eventually "amended".

The first loop (repeat ... until False) is responsible of providing all the input data to the compression function (deflate Z_NO_FLUSH). The only way to quit this loop without any error is to have no more input data: i.e.  strm.avail_in = 0.

Then, the second loop is finalizing the compression (deflate Z_FINISH). No more input data, but eventually several turns may be necessary to write the finalize data.

The problem is that after the first loop, the output buffer may eventually be full (i.e. strm.avail_out = 0); because WriteOut is -if necessary- called before deflate, not after, in this loop.

And in this case, the first next call to the "finalize compression" step (i.e. deflate Z_FINISH) returns an error (Z_BUF_ERROR).

Though, here is the way I see the whole thing:

Code: [Select]


      // Loop to compress all the input data
      repeat
        ...
      until False;

      // After this loop, if the output buffer is full, write it (to make it no more full)
      //    Note: (strm.avail_in = 0) test may eventually be optional, as it's supposed to be always true ?
      if (strm.avail_in = 0) and (strm.avail_out = 0) then begin
        WriteOut;
      end;

      // Loop for finalizing the compression
      repeat
        Finished := CCheck(deflate(strm, Z_FINISH)) = Z_STREAM_END;
        WriteOut;
      until Finished;


Note: it concerns both the "IndyCompressStream" and the "DoCompressStreamEx" procedures.

[dinmil]

I think last one from ChrisF is correct approach

      repeat
        ...
      until False;

      // After this loop, if the output buffer is full, write it (to make it no more full)
      //    Note: (strm.avail_in = 0) test may eventually be optional, as it's supposed to be always true ?
      if (strm.avail_in = 0) and (strm.avail_out = 0) then begin
        WriteOut;
      end;

      // Loop for finalizing the compression
      repeat
        Finished := CCheck(deflate(strm, Z_FINISH)) = Z_STREAM_END;
        WriteOut;
      until Finished;