paszlib. Feature request on work with buffers

[LemonParty]

Here's something that also might interest you:
https://github.com/Xelitan/Various-Free-Pascal-TStreams/

Code: Pascal  [Select][+][-]

1. var Bytes: TBytes;
2. begin
3. Stream := TPointerStream.Create(@Bytes[0], Length(Bytes));

This lets you use TBytes, arrays, strings and PBytes without copying as TStreams.

Cool. You generate code really fast.
Just tested. I only changed line

Code: Pascal  [Select][+][-]

1.     property Memory: Pointer read FData;

to

Code: Pascal  [Select][+][-]

1.     property Memory: PByte read FData;

so compiler not complain.

I modified your Zip code to use directly buffers. Attached project creates archive with compressed string. But there is a problem. In explorer I can open this archive and see 'f' as a file and there visible size of it. But when I trying to open this archive with 7zip I getting error

Cannot open the file as [zip] archive. Errors: is not archive

and from Explorer I also can't extract the file. Maybe you can spot an error in my code or have an idea why it failing.

[Tomxe]

Code: Pascal  [Select][+][-]

1. {ZIP DATA 1}
2.   WriteLE(aOutBuf[0], DWord($04034B50));

Fix this

[Tomxe]

Cool. You generate code really fast.

I have lots of compression-related code on the disk, written over the years. Here's something that also might interest you:

Code: Pascal  [Select][+][-]

1. function UnZlib(Compressed: String): String; //gzdecompress from PHP
2. const BuffSize = 4096;
3. var Z: TDecompressionStream;
4.     Read: Integer;
5.     Temp: String;
6.     Str: TStringStream;
7. begin
8.   Str := TStringStream.Create(Compressed);
9.   Z := TDecompressionStream.Create(Str, False);
10.   SetLength(Temp, BuffSize);
11.   Result := '';
12.  
13.   try
14.     repeat
15.       Read := Z.Read(Temp[1], BuffSize);
16.       Result := Result + Copy(Temp, 1, Read);
17.     until Read<BuffSize-1;
18.   finally
19.     Z.Free;
20.     Str.Free;
21.   end;
22. end;
23.  
24.  
25. function Zlib(UnCompressed: String): String; //gzcompress from PHP
26. const BuffSize = 4096;
27. var Z: TCompressionStream;
28.     Read: Integer;
29.     Temp: AnsiString;
30.     Str: TStringStream;
31.     Str2: TStringStream;
32. begin
33.   Str := TStringStream.Create('');
34.   Str2 := TStringStream.Create(UnCompressed);
35.  
36.   Z := TCompressionStream.Create(clMax, Str, False);
37.  
38.   SetLength(Temp, BuffSize);
39.   Result := '';
40.  
41.   try
42.     repeat
43.       Read := Str2.Read(Temp[1], BuffSize);
44.       Z.Write(Temp[1], Read);
45.  
46.     until Read = 0;
47.   finally
48.     Z.Free;
49.     Str2.Free;
50.   end;
51.  
52.   Result := Str.DataString;
53.   Str.Free;
54. end;
55.  
56.  
57. function GZip(const Uncompressed: AnsiString): AnsiString;
58.  
59.   procedure AppendByte(var S: AnsiString; B: Byte);
60.   var
61.     L: Integer;
62.   begin
63.     L := Length(S);
64.     SetLength(S, L + 1);
65.     S[L + 1] := AnsiChar(B);
66.   end;
67.  
68.   procedure AppendUInt32LE(var S: AnsiString; Value: Cardinal);
69.   begin
70.     AppendByte(S, Byte(Value and $FF));
71.     AppendByte(S, Byte((Value shr 8) and $FF));
72.     AppendByte(S, Byte((Value shr 16) and $FF));
73.     AppendByte(S, Byte((Value shr 24) and $FF));
74.   end;
75.  
76.   function CRC32Of(const S: AnsiString): Cardinal;
77.   const
78.     Polynomial = $EDB88320;
79.   var
80.     CRC: Cardinal;
81.     I, J: Integer;
82.     B: Byte;
83.   begin
84.     CRC := $FFFFFFFF;
85.  
86.     for I := 1 to Length(S) do
87.     begin
88.       B := Byte(S[I]);
89.       CRC := CRC xor B;
90.       for J := 0 to 7 do
91.       begin
92.         if (CRC and 1) <> 0 then
93.           CRC := (CRC shr 1) xor Polynomial
94.         else
95.           CRC := CRC shr 1;
96.       end;
97.     end;
98.  
99.     Result := not CRC;
100.   end;
101.  
102. var
103.   Deflated: TStringStream;
104.   Input: TStringStream;
105.   Z: TCompressionStream;
106.   CRC: Cardinal;
107.   UnixTime: Cardinal;
108. begin
109.   Deflated := TStringStream.Create('');
110.   try
111.     Input := TStringStream.Create(String(Uncompressed));
112.     try
113.       Z := TCompressionStream.Create(clMax, Deflated, True);
114.       try
115.         Z.CopyFrom(Input, 0);
116.       finally
117.         Z.Free;
118.       end;
119.  
120.       CRC := CRC32Of(Uncompressed);
121.       UnixTime := DateTimeToUnix(Now);
122.  
123.       Result := '';
124.  
125.       // GZIP header (10 bytes)
126.       AppendByte(Result, $1F);          // ID1
127.       AppendByte(Result, $8B);          // ID2
128.       AppendByte(Result, $08);          //CM = deflate
129.       AppendByte(Result, $00);          // FLG = no extra fields
130.       AppendUInt32LE(Result, UnixTime); // ModTime
131.       AppendByte(Result, $02);          // XFL = max compression
132.       AppendByte(Result, $FF);          // OS = unknown
133.  
134.       Result := Result + AnsiString(Deflated.DataString);
135.  
136.       // Gzip foot
137.       AppendUInt32LE(Result, CRC);                      // CRC32 of unpacked data
138.       AppendUInt32LE(Result, Cardinal(Length(Uncompressed))); // ISIZE mod 2^32
139.     finally
140.       Input.Free;
141.     end;
142.   finally
143.     Deflated.Free;
144.   end;
145. end;
146.  
147. function UnGzip(Compressed: String): String; //gzdecode from PHP
148. const BuffSize = 4096;
149. var Z: TDecompressionStream;
150.     Read: Integer;
151.     Temp: String;
152.     Str: TStringStream;
153.  
154.     ID: Word;
155.     Method: Byte;
156.     Head: Word;
157.     Date: Cardinal;
158.     Flag, OS: Byte;
159.     Skip: Word;
160.     Tmp: String;
161.     Null: Integer;
162. begin
163.   Str := TStringStream.Create(Compressed);
164.  
165.   ID := Str.ReadWord;
166.   Method := Str.ReadByte;
167.   Head := Str.ReadByte;
168.   Date := Str.ReadDWord;
169.   Flag := Str.ReadByte;
170.   OS := Str.ReadByte;
171.  
172.   //FEXTRA
173.   if ((Head shr 2) and 1 = 1) then begin
174.     Skip := Str.ReadByte;
175.     Str.Position := Str.Position + Skip;
176.   end;
177.  
178.   //FNAME
179.   if ((Head shr 3) and 1 = 1) then begin
180.     Tmp := Str.ReadString(256);
181.     Null := Pos(chr(0), Tmp);
182.     Str.Position := Str.Position - Length(Tmp) + Null;
183.   end;
184.  
185.   //FCOMMENT
186.   if ((Head shr 4) and 1 = 1) then begin
187.     Tmp := Str.ReadString(256);
188.     Null := Pos(chr(0), Tmp);
189.     Str.Position := Str.Position - Length(Tmp) + Null;
190.   end;
191.  
192.   //FHCRC
193.   if ((Head shr 1) and 1 = 1) then begin
194.     Str.Position := Str.Position + Skip;
195.   end;
196.  
197.   //https://datatracker.ietf.org/doc/html/rfc1952#page-5
198.   Z := TDecompressionStream.Create(Str, True);
199.   SetLength(Temp, BuffSize);
200.   Result := '';
201.  
202.   try
203.     repeat
204.       Read := Z.Read(Temp[1], BuffSize);
205.       Result := Result + Copy(Temp, 1, Read);
206.     until Read<BuffSize-1;
207.   finally
208.     Z.Free;
209.     Str.Free;
210.   end;
211. end;

[LemonParty]

Yes. Interesting.
I found useful table with offsets for Zip https://en.wikipedia.org/wiki/ZIP_(file_format). And compared with what I calculated by your code – all matches. But don't work. Disappointing.

[Tomxe]

Code: Pascal  [Select][+][-]

1. {ZIP DATA 1}
2.   WriteLE(aOutBuf[0], DWord($04034B50));

With this fix iyour code does work.

[LemonParty]

Oh, that is my fault. I could notice this mistake. Thank you.

[LemonParty]

Could you help me a little?

This is a code of deflate that should work with buffers:

Code: Pascal  [Select][+][-]

1. function Deflate(const Buffer; BufferSize: SizeUInt; Level: TCompressionLevel;
2.   var OutBuffer): SizeUInt;
3. var
4.   aBuffer: array [Byte] of Byte absolute Buffer;
5.   PS: TPointerStream;
6.   CS: TCompressionStream;
7.   TotalSize: SizeUInt;
8.   Written: LongInt;
9. begin
10.   Result:= High(Result);
11.   TotalSize:= 0;
12.   PS:= TPointerStream.Create(@OutBuffer, High(Int64));
13.   CS:= TCompressionStream.Create(Level, PS, True);
14.  
15.   try
16.     CS.WriteBuffer(Buffer, BufferSize);
17.     Result:= PS.Position;
18.   finally
19.     CS.Free;
20.     PS.Free;
21.   end;
22. end;

The same code works fine in Zip function I wrote before. But this code return 0 as the result. To be more precise PS.Position not changes after CS.WriteBuffer(Buffer, BufferSize). What I missed?

[Tomxe]

Is OutBuffer huge enough?

[LemonParty]

Ha. I found the problem.
The right code is:

Code: Pascal  [Select][+][-]

1. function Deflate(const Buffer; BufferSize: SizeUInt; Level: TCompressionLevel;
2.   var OutBuffer): SizeUInt;
3. var
4.   PS: TPointerStream;
5.   CS: TCompressionStream;
6. begin
7.   Result:= High(Result);
8.   PS:= TPointerStream.Create(@OutBuffer, High(Int64));
9.   CS:= TCompressionStream.Create(Level, PS, True);
10.  
11.   try
12.     CS.WriteBuffer(Buffer, BufferSize);
13.   finally
14.     CS.Free;
15.     if Assigned(PS) then
16.       Result:= PS.Position;
17.     PS.Free;
18.   end;
19. end;

PS.Position should call after compression stream was free.

[LemonParty]

I done modifications.

Things I have done:
1. Added overloads for any case. Present: buffers, strings, streams, files.
2. Optimized varsion for buffers and strings. Used direct write to memory.
3. Added also functions to determine worst case size for buffers.
4. Added ability to pick compression level.
5. Added inline to small functions.

Editing reason: setted result for files functions.

[Tomxe]

I done modifications.

Very nice!