Unzip from a stream

[wpflum]

I need to be able to unzip a memory stream to another memory stream but so far I haven't had much luck.  I have the code working at the moment with an actual zip file but what I need to do is take a stream of that file and use it like I used the actual file itself.  Everything I've seen either works from a file to a stream or cannot handle a pkzip compressed file. 

Any suggestions?

[User137]

I need to be able to unzip a memory stream to another memory stream but so far I haven't had much luck.

So where does the luck end? 

You could show what code you have so far, so we could see how to continue from there.

[wpflum]

At the moment I don't have any code since after having errors I did some more research and discovered that Tunzipper doesn't seem to support compression and none of the other unzip packages seem to be able to use a stream as an input so I have nothing to really post.  What I was hoping for was either a verification that nothing available can do a pkzip decompress from a stream or some direction to a package that will or, even more likely, that I'm an idiot and Tunzipper works and I'm using it wrong.  If the last one is the case then I'll rebuild the code back to the way I had it when I was trying to use TUnzipper and post the code.

[BigChimp]

did some more research and discovered that Tunzipper doesn't seem to support compression

Where did you find that gem? TUnzipper obviously decompresses things. It just does not support all kinds of compression.

http://wiki.lazarus.freepascal.org/paszlib#TZipper

PS: IIRC, the zip format as normally used prohibits streaming because it has header data both before the file data and at the end of the total zip file (in the central directory header etc); there's a special format that could be used for streaming suport (again IIRC).

[wpflum]

Sorry, it might have been another built in library for compression that said it didn't support compression just collection, or something like that, it was late yesterday when I gave up and I hadn't gotten back to it this morning, other than replying to your email,  since we had a minor emergency in the form of mostly dead connections between our three stores. 

I went back and started from scratch to see what happens and I now have a simple bit of code that loads a stream with a zip file and then tries to unzip it.

When I run this code I get a "Project unzip raised exception class 'Edecompressionerror' with message: data error".

Does this relate to the problem with a zip file having the twin headers?

Code: [Select]

procedure TForm1.Button1Click(Sender: TObject);
var
  UnZip : TInflater;
  buff1,buff2:tmemorystream;
begin
buff1:=TMemoryStream.Create ;
buff2:=TMemoryStream.Create ;
buff1.LoadFromFile('data.zip') ;
buff1.Position := 0;
buff2.Position :=0;

UnZip := TInflater.Create(buff1,buff2,buff1.Size  );
Unzip.DeCompress ;
end; 

[BigChimp]

Seems like you're a fan of the "throw everything together and see how it goes" approach. So am I, but when that doesn't work it's best to find out why something doesn't work.

Asking on the forum all the time wouldn't be the most efficient way of doing that and you already have the link to the tunzipper documentation as well as example code, and finally you can look at the TUnzipper source code, so:
1. What kind of format does TInflater expect?
2. What does it get?
I suspect 1 and 2 are different.

PS: What happens if the file(s) in the zip ever get shrunk instead of deflated or get compressed using another compression method?

However, more importantly:
3. Why are you writing this code? What is the underlying goal? In other words: why were the examples on that wiki page not good enough? What extra requirements do you have?
And no, don't answer: "I want to unzip to a stream". WHY do you want to unzip to a stream? What do you want to do with that? Why is the example code on the wiki page (including the one that unzips to a stream) not good enough?

The answer to this question will probably give an idea of the right way to solve whatever it is you are trying to solve.

[wpflum]

This was just a small bit of bigger program that I broke out to figure out why it wasn't working.  What I'm trying to do is take a zipped group of database files and attach it to the end of the exe itself so that the data and program to read the data is one item. I've already gotten to the point where I can unzip a file to a stream and use it which was the next step from having to unzip the group of data files to a directory to use them.  Now I want to further idiot proof the entire thing by creating a procedure that will allow an external system, the one that creates the zipped data file, to just attach the zip file to an existing exe file and email it out so that when the data is updated, nightly, the end user just has to run a single file to access the data and not worry about moving the file to a specific directory to use it.

I was digging into the inflator code yesterday but it's a bit overwhelming and above my level of usage of pascal so I was having a problem with figuring out what was what in the code itself. 

Is there some way to set break points in a package?  Since I'm getting a generic 'data error' I wanted to see what exactly was crashing so I tried to add break points in the zipper.pas but Lazarus seems to ignore it.

One final thing, how is a couple of questions, 'all the time' as you put it?  I usually bang my head against the problem for a while, researching on the web, trying to dig into the code itself, but when I hit a wall I try to ask generic enough questions so at least I know if I'm heading up the wrong path, I don't just beg for someone else to code it for me like I've seen before on forums. 

[BigChimp]

Don't use the inflater code. Inflater is IIRC used to decompress a file stream that has been compressed by using the deflate method.
A zip consists of headers, 1 or more file streams compressed using deflate, shrink, or other methods then more records.

Inflate will therefore not decompress a zip file - use TUnzipper.
You don't need to unzip to a stream to be able to unzip it to a file in the end - see the examples on the wiki page.

I already linked to ready-made classes in my other post that allow you to zip up files and add them as a resource to an executable.
The only thing you would then need to do is unzip the files you want to attach and feed them to that code.

Regarding break points: yes, I suppose it is possible, but you will need to compile the package with debug info (optimization off, line info on, generate gdb etc - probably can be set in package options). However if you intend to debug the tunzipper code: it's not a Lazarus package but part of FPC. Therefore you need a debug build of FPC.
(I use fpcup with something like --fpcopt="-g -gl" --lazopt="-g -gl" to get a debug fpc/lazarus build)

The questions all the time is related to your posts, not especially this one. And yes, you do not beg - which does speak volumes
The reason I asked you for your goals etc is simple: often there's a n easier way of getting to a solution than one might think. So please give details as well as the underlying goal you're trying to reach.

[wpflum]

I guess I wasn't clear.  I don't want to unzip to a file at all.  I was trying to get completely away from using a file.  Right now the procedure is, get zip file and unzip it to a stream and use it,  I want to get to the point of, read zip file data from end of program to a stream, unzip to a stream and use it.  I guess it's a lost cause so I'm going to at least get the separate file part of the equation out by extracting the zip data from the exe file and saving it to the temp directory and then using that file with the existing code.

I'm back to the 'all the time' issue.  I asked three unrelated questions, if that is 'all the time' in your book I apologize for monopolizing your time and inundating the forum. 

[hinst]

if you want to store compressed data in ur executable file and then use it at runtime, then I suggest you just compress it using TCompressionStream from zstream unit, then decompress it using TDecompressionStream. It is a lot easier this way in my opinion. This approach is appropriate if you don't absolutely need your data to be in a zip archive file, but just need it to be compressed instead

[hinst]

Some code I wrote:

Compress & decompress streams:

Code: [Select]

unit PasZUtilU;

{$Include PasZUtil.inc}

interface

uses
  Classes, sysutils, zstream;

procedure CompressTransf(const aSource, aDest: TStream; aLevel: Tcompressionlevel);
procedure CompressTransf(const aSource, aDest: TStream);
procedure ReplaceWithCompressed(var a: TStream);
procedure DecompressTransf(const aSource, aDest: TStream);
procedure ReplaceWithDecompressed(var a: TStream);

implementation

type
  TypeOfSize = Int64;

procedure CompressTransf(const aSource, aDest: TStream; aLevel: Tcompressionlevel);
var
  c: Tcompressionstream;
  n: TypeOfSize;
begin
  c := Tcompressionstream.create(aLevel, aDest);
  n := aSource.Size;
  aDest.Write(n, SizeOf(n));
  c.CopyFrom(aSource, aSource.Size);
  c.Free;
end;

procedure CompressTransf(const aSource, aDest: TStream);
begin
  CompressTransf(aSource, aDest, cldefault);
end;

procedure ReplaceWithCompressed(var a: TStream);
var
  b: TMemoryStream;
begin
  a.Position := 0;
  b := TMemoryStream.Create;
  CompressTransf(a, b);
  a.Free;
  a := b;
end;

procedure DecompressTransf(const aSource, aDest: TStream);
var
  d: Tdecompressionstream;
  n: TypeOfSize;
begin
  n := 0;
  aSource.Read(n, SizeOf(n));
  d := Tdecompressionstream.create(aSource);
  aDest.CopyFrom(d, n);
  d.Free;
end;

procedure ReplaceWithDecompressed(var a: TStream);
var
  b: TMemoryStream;
begin
  a.Position := 0;
  b := TMemoryStream.Create;
  DecompressTransf(a, b);
  a.Free;
  a := b;
end;

end.


This code is not for zip streams, just for "some compressed streams". I don't know which compression method zstream unit implements, but it works

[wpflum]

I wish I could but the data is coming from a legacy system and I don't want to mess with it.  The most I can do is a script to append the zip file with a header to the exe.

[hinst]

Actually take a look at: OnOpenInputStream property of of TUnZipper class in zipper unit; it's just what you need. Just assign your handler to this property

[wpflum]

Thanks, I'll take a look at it.  I saw it earlier but the 'On' in front kept throwing me off. Tunzipper is the first code I've used that did it's work using the on events so everytime I saw the stream part with the 'ON' in front I associated with something like a form event.  Now that I think of it considering how the example I'm currently using is doing the heavy lifting in the OnDoneStream I guess I should have seen that.

Thanks again.

[hinst]

If you look at the source:

Code: [Select]

Procedure TUnZipper.OpenInput;
Begin
  if Assigned(FOnOpenInputStream) then
    FOnOpenInputStream(Self, FZipStream);
  if FZipStream = nil then
    FZipStream:=TFileStream.Create(FFileName,fmOpenRead or fmShareDenyWrite);
End;
See what it does: if OnOpenInputSteam is assigned, then it opens stream using this proc, otherwise it attempts to open file.

assign OnOpenInputStream to read from memory stream, then
assign OnCreateStream and OnDoneStream to write resulting file to memory stream