Pure Pascal LZ4, LZ5 and LIZ

[Tomxe]

https://github.com/Xelitan/Free-Pascal-port-of-LZ4-compressor-decompressor-/
https://github.com/Xelitan/Free-Pascal-port-of-LZ5-LIZARD-compressor-decompressor-

License: BSD-2

Usage:

Code: Pascal  [Select][+][-]

1. function LZ5CompressStreams (Infile, Outfile: TStream): Integer;
2. function LZ5DecompressStreams(Infile, Outfile: TStream): Integer;
3. function LZ5CompressFile(const Infilename, Outfilename: String): Integer;
4. function LZ5DecompressFile(const Infilename, Outfilename: String): Integer;
5. function LZ5(Uncompressed: AnsiString): AnsiString;
6. function UnLZ5(Compressed: AnsiString): AnsiString;
7.  
8. function LIZCompressStreams (Infile, Outfile: TStream): Integer;
9. function LIZDecompressStreams(Infile, Outfile: TStream): Integer;
10. function LIZCompressFile(const Infilename, Outfilename: String): Integer;
11. function LIZDecompressFile(const Infilename, Outfilename: String): Integer;
12. function LIZ(Uncompressed: AnsiString): AnsiString;
13. function UnLIZ(Compressed: AnsiString): AnsiString;
14.  
15. function LZ4(Uncompressed: AnsiString): AnsiString;
16. function UnLZ4(Compressed: AnsiString): AnsiString;
17. function LZ4CompressFile(const Infilename, Outfilename: String): Integer;
18. function LZ4DecompressFile(const Infilename, Outfilename: String): Integer;
19. function LZ4CompressStreams(Infile, Outfile: TStream): Integer;
20. function LZ4DecompressStreams(Infile, Outfile: TStream): Integer;
21.  

[mika]

There is this error:

Code: Pascal  [Select][+][-]

1. function LIZDecompressFileconst (Infilename, Outfilename: String): Integer;

[Tomxe]

Thanks, fixed.

[Thaddy]

No, all the string and stream parameters should be const
You did a few, but not all and it does make a difference.
This remark goes for ALL the code you presented in several threads: some AI is not very good at applying the const parameter optimization for managed types.

Then again, they are not always aware which string types they are using.. They often miss {$H+/-} and {$mode delphi/unicode} and more...

Which model(s) are you using?

Even claude.ai makes that mistake unless you make it explicitly aware it has to take managed types into account. From the models I use, Only DeepSeek will usually optimize correctly with const.

[Tomxe]

No, all the string and stream parameters should be const
You did a few, but not all and it does make a difference.
This remark goes for ALL the code you presented in several threads: some AI is not very good at applying the const parameter optimization for managed types.

This is not AI's fault. This is mine. I rarely think about such optimizations.

Which model(s) are you using?

zzzcode.ai, Grok, ChatGPT, a bit Claude. Whatever helps speed up translation. They all make weird mistakes. Then I write wrappers for easier usage.

[Thaddy]

zzzcode.ai, Grok, ChatGPT, a bit Claude. Whatever helps speed up translation. They all make weird mistakes. Then I write wrappers for easier usage.

About the same as me: everything free. Don't forget Deepseek, though: it is remarkably good with Pascal code and it shows you its reasoning too. Which I find great.

A candidate for future work may be UPX/UCL

[LemonParty]

Hello.

I have an error on line 420 of xxHash.pas:

Code: Pascal  [Select][+][-]

1.     if (len>=32) then
2.     begin
3.         v1 := seed + PRIME64_1 + PRIME64_2; {this line}
4.         v2 := seed + PRIME64_2;

Also I have an error on line 540 of xxHash.pas:

Code: Pascal  [Select][+][-]

1.     fillbyte(state, sizeof(XXH64_state_t)-8, 0);   { do not write into reserved, for future removal }
2.     state.v1 := seed + PRIME64_1 + PRIME64_2; {this line}
3.     state.v2 := seed + PRIME64_2;

In both cases compiler says

Error: Overflow in arithmetic operation

I use trunk FPC.

[Tomxe]

Maybe something in trunk FPC is broken because it works fine under both 32 and 64 and a few different 3.X Lazarus versions.

[LemonParty]

I checked, those two numbers

Code: Pascal  [Select][+][-]

1.   PRIME64_1 =  Uint64(11400714785074694791);
2.   PRIME64_2 =  Uint64(14029467366897019727);

indeed do overflow if you add them.

I tried to compile this unit with {$Q-}, but – same error. Is this an error in compiler?

[LemonParty]

I also checked with this version of compiler: 3.0.4+dfsg-22 [2019/01/24]. Gives the same error.

[mika]

I have an error on line 420 of xxHash.pas:

Code: Pascal  [Select][+][-]

1.     if (len>=32) then
2.     begin
3.         v1 := seed + PRIME64_1 + PRIME64_2; {this line}
4.         v2 := seed + PRIME64_2;

Also I have an error on line 540 of xxHash.pas:

Code: Pascal  [Select][+][-]

1.     fillbyte(state, sizeof(XXH64_state_t)-8, 0);   { do not write into reserved, for future removal }
2.     state.v1 := seed + PRIME64_1 + PRIME64_2; {this line}
3.     state.v2 := seed + PRIME64_2;

In both cases compiler says

Error: Overflow in arithmetic operation

I use trunk FPC.

It is expected to have overflow there. {$R-} would help.

[LemonParty]

{$R-} not help. In both compilers the error still present.

[Tomxe]

Lazarus 4.4, no errors, no warnings, just some hints.

[LemonParty]

There is indeed an error in your code, look: https://forum.lazarus.freepascal.org/index.php/topic,73975.0.html

The fix is next (starting at line 19 of xxHash):

Code: Pascal  [Select][+][-]

1.   PRIME64_1: UInt64 = 11400714785074694791;
2.   PRIME64_2: UInt64 = 14029467366897019727;
3.   PRIME64_3: UInt64 =  1609587929392839161;
4.   PRIME64_4: UInt64 =  9650029242287828579;
5.   PRIME64_5: UInt64 =  2870177450012600261;

This way it compiles (and I hope works as needed).

After that I wrote a simple program to check how compression works. The program is next:

Code: Pascal  [Select][+][-]

1. {$Mode ObjFPC}{$H+}
2. {$R *.res}
3.  
4. uses
5.   LizSimple;
6.  
7. var
8.   S: String = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.';
9.   Compressed: String;
10. begin
11.   Compressed:= LIZ(S);
12.   Writeln(Length(S), ' ', Length(Compressed));
13.   Readln;
14. end.
15.  

The output is

445 0

Do I use your library right? Why compressed string is empty?

[Tomxe]

uses
  LizSimple;

var
  S: String = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.';
  Compressed: String;
begin
  Compressed:= LIZ(S);
  Writeln(Length(S), ' ', Length(Compressed));
  Readln;
end.
[/code]

The output is

445 0

Sorry about that. Here's my mistake:

these 2 lines:

Code: Pascal  [Select][+][-]

1. if LIZDecompressStreams(InStream, OutStream) <> 0 then
2. ...
3. if LIZCompressStreams(InStream, OutStream) <> 0 then

should be:

Code: Pascal  [Select][+][-]

1. if LIZDecompressStreams(InStream, OutStream) < 0 then
2. ...
3. if LIZCompressStreams(InStream, OutStream) < 0 then

I'll update the Github repo later.