Recent

Author Topic: Zstandard for Lazarus, "without" DLL, Windows only  (Read 3094 times)

domasz

  • Hero Member
  • *****
  • Posts: 557
Zstandard for Lazarus, "without" DLL, Windows only
« on: February 26, 2024, 01:02:51 am »
I prefer single EXE programs but there is no ZStandard implementation in Pascal. So here's my solution for Windows 64 bit, a bit ugly but:

1) no DLL file to distribute
2) antivirus programs don't complain- you can check on virustotal
3) easy to update Zstandard to newest release
4) easy to adapt code for Win32, Linux, MacOS.

It just extracts the DLL to the temporary folder and uses it from there. The DLL is UPX packed to be smaller.

Code: Pascal  [Select][+][-]
  1. uses ZSTDLib, ZSTD
  2.  
Code: Pascal  [Select][+][-]
  1. const BufSize = 40960000;
  2. var InF, OutF: TFileStream;
  3.     Z: TZSTDCompressStream;
  4.     Len: Integer;
  5.     Buf: array of Byte;
  6.     Options: TZSTDCompressOptions;
  7.  
  8.     c1, c2, f: Int64;
  9. begin
  10.   ZSTDDllName := GetTempDir + 'libzstd-117788.dll';
  11.  
  12.   if not FileExists(ZSTDDllName) then begin
  13.     OutF := TFileStream.Create(ZSTDDllName, fmCreate);
  14.     OutF.Write(ZStdBody, Length(ZStdBody));
  15.     OutF.Free;
  16.   end;
  17.  
  18.   QueryPerformanceFrequency(f);
  19.   QueryPerformanceCounter(c1);
  20.  
  21.   InF := TFileStream.Create('whatever.input', fmOpenRead);
  22.  
  23.   OutF := TFileStream.Create('pack.zstd', fmCreate);
  24.  
  25.   Options.Init;
  26.   //Options.CompressionLevel := 100; //max
  27.   Options.CompressionLevel := 2; //fast
  28.  
  29.   Z := TZSTDCompressStream.Create(OutF, Options);
  30.  
  31.   SetLength(Buf, BufSize);
  32.  
  33.   while InF.Position < InF.size do begin
  34.     Len := InF.Read(Buf[0], BufSize);
  35.  
  36.     Z.Write(Buf[0], Len);
  37.     //OutF.Write(Buf[0], Len);
  38.   end;
  39.  
  40.   Z.Free;
  41.   InF.Free;
  42.   OutF.Free;
  43.   //DeleteFile(ZSTDDllName);
  44.  
  45.   QueryPerformanceCounter(c2);
  46.   ShowMessage(Format('%f sec', [ (c2-c1)/f ]));
  47.  

Fibonacci

  • Hero Member
  • *****
  • Posts: 651
  • Internal Error Hunter
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #1 on: February 26, 2024, 01:12:47 am »
You know you can load a DLL directly from memory without saving it to disk? It aint easy though. You need to manually do the reclocations and patch the IAT.

But no need to do that, you can static link this lib with {$linklib xxx.a}.

domasz

  • Hero Member
  • *****
  • Posts: 557
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #2 on: February 26, 2024, 01:28:54 am »
You know you can load a DLL directly from memory without saving it to disk? It aint easy though. You need to manually do the reclocations and patch the IAT.
Yes, but antivirus programs don't like it and I read its problematic in case of 64 bit Windows.

But no need to do that, you can static link this lib with {$linklib xxx.a}.
I can static link this DLL into EXE? I think I need a special .OBJ file or something to static link.
« Last Edit: February 26, 2024, 01:40:20 am by domasz »

Fibonacci

  • Hero Member
  • *****
  • Posts: 651
  • Internal Error Hunter
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #3 on: February 26, 2024, 01:55:48 am »
You need a static library libzstd.a

I found 4 such files on my PC, from MinGW installation or Ruby installation, I dont know where it is available for download, I looked at msys and at github repo and there was no static library for win64.

domasz

  • Hero Member
  • *****
  • Posts: 557
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #4 on: February 26, 2024, 01:59:54 am »
Yes, I would need a binary in very specific format which nobody provides. And not just an .A file because this extension is used by different things.
That's the same problem with most libraries. There is ZStandard with static binaries for Delphi but they don't work in Lazarus. So my method is the only one working.

Fibonacci

  • Hero Member
  • *****
  • Posts: 651
  • Internal Error Hunter
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #5 on: February 26, 2024, 02:13:30 am »
EDIT: My bad, something went wrong, BRB!
« Last Edit: February 26, 2024, 02:17:17 am by Fibonacci »

domasz

  • Hero Member
  • *****
  • Posts: 557
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #6 on: February 26, 2024, 02:17:42 am »
Here you go, ready to compile :) Archive includes libstd.a
That's great, thank you!

Only would be nice to know where the libstd.a comes from (to get newer ones in the future)

Fibonacci

  • Hero Member
  • *****
  • Posts: 651
  • Internal Error Hunter
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #7 on: February 26, 2024, 02:18:41 am »
Wait, I have NO IDEA how that was possible but the file loaded libzstd.dll and it worked :D Investigating.

Fibonacci

  • Hero Member
  • *****
  • Posts: 651
  • Internal Error Hunter
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #8 on: February 26, 2024, 02:43:35 am »
Ok, dunno how that happened, that Pascal source, although I uncommented ifdef to static link the lib, it still used a DLL, and it happens I have a libzstd.dll available in my PATH, so it worked. I posted that project here, but then checked the binary file and there were imports from libzstd.dll.

Here is the fixed version, I removed everything that wanted to load a DLL. More static libs were needed.

{$linklib libzstd.a}
{$linklib libgcc.a}
{$linklib libkernel32.a}
{$linklib libucrt.a}

Archive has ~3.5 MB so I cant upload it here on the forum, so the link is here:
https://github.com/fibodevy/zflate/files/14398693/fpc-zstd-static.zip
(I dropped the file on a new issue on github and closed the tab, so I dont know how long the link will stay valid :D)
« Last Edit: February 26, 2024, 02:51:16 am by Fibonacci »

440bx

  • Hero Member
  • *****
  • Posts: 5009
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #9 on: February 26, 2024, 02:58:37 am »
What I'm going to say  isn't about libzstd but someone may be interested in knowing that Windows NTDLL provides compression and decompression functions.   While they aren't as "effective" as e.g, 7Zip or rar, they are quite decent and no need to distribute a dll (if ntdll is missing, the user has more important problems than decompressing a file... <chuckle>)

I thought I'd mention that in case it might be of interest to someone.
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

KodeZwerg

  • Hero Member
  • *****
  • Posts: 2269
  • Fifty shades of code.
    • Delphi & FreePascal
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #10 on: February 26, 2024, 03:00:10 am »
Do by accident you maybe also got a working version for static lz4 / lz4hc with fpc header flying around?
« Last Edit: Tomorrow at 31:76:97 xm by KodeZwerg »

Fibonacci

  • Hero Member
  • *****
  • Posts: 651
  • Internal Error Hunter
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #11 on: February 26, 2024, 03:08:48 am »
What I'm going to say  isn't about libzstd but someone may be interested in knowing that Windows NTDLL provides compression and decompression functions.

These APIs are:
- compression: RtlGetCompressionWorkSpaceSize() -> RtlCompressBuffer()
- decompression: RtlGetCompressionWorkSpaceSize() -> RtlDecompressBuffer()

Compression ratio is really low IIRC.

Do by accident you maybe also got a working version for static lz4 / lz4hc with fpc header flying around?

Just static libs and C headers :D

440bx

  • Hero Member
  • *****
  • Posts: 5009
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #12 on: February 26, 2024, 03:17:26 am »
Compression ratio is really low IIRC.
yes, it is relatively low but, I'd say it's decent. That said, definitely not nearly as good as what is obtained from more sophisticated algorithms used in 7zip and Winrar.

The really nice feature is: no external dll needed because the routines are part of the O/S.  That's a nice plus.
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

KodeZwerg

  • Hero Member
  • *****
  • Posts: 2269
  • Fifty shades of code.
    • Delphi & FreePascal
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #13 on: February 26, 2024, 05:07:51 am »
Just static libs and C headers :D
Many thanks, will try to do same like you've done with Zstandard and upload when done or cry for help when it fail :D
« Last Edit: Tomorrow at 31:76:97 xm by KodeZwerg »

domasz

  • Hero Member
  • *****
  • Posts: 557
Re: Zstandard for Lazarus, "without" DLL, Windows only
« Reply #14 on: February 26, 2024, 08:39:10 am »

{$linklib libzstd.a}
{$linklib libgcc.a}
{$linklib libkernel32.a}
{$linklib libucrt.a}

Where are these files from? You compiled them by yourself?

I found good files (except libzstd.a) in MinGW release (winlibs-x86_64-posix-seh-gcc-13.2.0-llvm-17.0.6-mingw-w64ucrt-11.0.1-r5.7z).
« Last Edit: February 26, 2024, 08:44:54 am by domasz »

 

TinyPortal © 2005-2018