* * *

Author Topic: A powerful Parallel Compression Library  (Read 680 times)

Sky68

  • New member
  • *
  • Posts: 15
A powerful Parallel Compression Library
« on: May 21, 2018, 11:04:34 pm »
Hello,

My powerful Parallel Compression Library version 4.32 for Windows and Linux and for 32 bit and 64 bit is here..

Author: Amine Moulay Ramdane

Description:

Parallel Compression Library implements Parallel LZ4 , Parallel LZMA , and Parallel Zstd algorithms using my Thread Pool Engine.

- It supports memory streams, file streams and files

- 64 bit supports - lets you create archive files over 4 GB ,  supports archives up  to 2^63 bytes, compresses and  decompresses files up to 2^63 bytes.

- Parallel compression and parallel decompression are extremely fast

- Now it supports processor groups on windows, so that it can use more than 64 logical processors and it scales well.

- It's NUMA-aware and NUMA efficient on windows

- It minimizes efficiently the contention so that it scales well.

- It supports both compression and decompression rate indicator

- You can test the integrity of your compressed file or stream

- Easy programming interface

- Full source codes available.

Now my Parallel compression library is optimized for NUMA(it parallelizes the reads and writes on NUMA nodes)  and it supports processor groups on windows and it uses only two threads that do the IO (and they are not contending) so that it reduces at best the contention, so that it scales well.  Also now the process of calculating the CRC is much more optimized and is fast, and the process of testing the integrity is fast.

I have done a quick calculation of the scalability prediction for my Parallel Compression Library,  and i think it's good: it can scale beyond 100X on NUMA systems.

Here are the parameters of the constructor:

First parameter is: The number of cores you have specify to run the compression algorithm in parallel.

Second parameter is: A boolean parameter that is processorgroups to support processor groups on windows , if it is set to true it will enable you to scale beyond 64 logical processors and it will be NUMA efficient.

Just look at the Easy compression library for example,

if you have noticed it's not a parallel compression library.

http://www.componentace.com/ecl_features.htm

And look at its pricing:

http://www.componentace.com/order/order_product.php?id=4

My parallel compression library costs you 0$ and it's a parallel compression library..

My Parallel compression library was updated,  i have ported the Parallel LZ4 compression algorithm(one of the fastest in the world)  to the Windows 64 bit system, now Parallel LZ4 compression algorithm is working perfectly with Windows 32 bit and 64 bit, if you want to use Windows 64 bit Parallel LZ4 just copy the lz4_2.dll inside the LZ4_64 directory (that you find inside the zip file) to your current directory or to the c:\windows\SysWow64 directory, and if you want to use the Windows 32bit Parallel LZ4  use the lz4_2.dll inside the LZ4_32 directory.

If you want to use Windows 64 bit Parallel LZMA with Windows 64 bit just copy the LZMAStream1.dll inside the LZMA_fpc64  directory and LZMAStream2.dll inside LZMA_dcc64 directory to your
current directory or to the c:\windows\SysWow64 directory, and if you want to use Windows 32bit Parallel LZMA copy the LZMAStream1.dll inside the LZMA_fpc32 directory and LZMAStream2.dll inside LZMA_dcc32 directory to your current directory or to the c:\windows\system32 directory.


You can download it from:

https://sites.google.com/site/scalable68/parallel-compression-library

Language: FPC Pascal v2.2.0+ / Delphi 7+: http://www.freepascal.org/

Operating systems: Windows , Linux

Required FPC switches: -O3 -Sd -dFPC

-Sd for delphi mode....

Required Delphi switches:  -$H+ -DDelphi32

Required Delphi XE-XE5 switches: -DXE       

{$DEFINE CPU32} and {$DEFINE Windows32} for 32 bit systems

{$DEFINE CPU64} and {$DEFINE Windows64} for 64 bit systems


Thank you,
Amine Moulay Ramdane.
« Last Edit: June 03, 2018, 07:15:05 pm by Sky68 »

Sky68

  • New member
  • *
  • Posts: 15
Parallel archiver version 4.76 is here...
« Reply #1 on: May 25, 2018, 06:12:44 pm »
Hello,

My powerful Parallel archiver version 4.76 is here, it works with Delphi and FreePascal and it is much more stable and it is very fast..

Now my Parallel archiver library is optimized for NUMA(it parallelizes the reads and writes on NUMA nodes)  and it supports processor groups on windows and it uses only two threads that do the IO (and they are not contending) so that it reduces at best the contention, so that it scales well.  Also now the process of calculating the CRC is much more optimized and is fast, and the process of testing the integrity is fast.

I have done a quick calculation of the scalability prediction for my Parallel archiver,  and i think it's good: it can scale beyond 100X on NUMA systems.

You can read about it and download it from:

https://sites.google.com/site/scalable68/parallel-archiver


Thank you,
Amine Moulay Ramdane.

« Last Edit: June 03, 2018, 07:16:46 pm by Sky68 »

Thaddy

  • Hero Member
  • *****
  • Posts: 6101
Re: Parallel archiver version 4.76 is here...
« Reply #2 on: May 25, 2018, 08:19:41 pm »
It is always a pleasure to play with your code.. but what does this mean: -dFreePascal ? A duplicate of -dFPC?
Also, I can't find all sources (probably my mistake). Usually you make all your sources available? And the archive contains binaries (dll's) so I can't really test on other platforms.
That said: I know your code is excellent regarding parallel programming and I use some of it (the lock-free stuff).
« Last Edit: May 25, 2018, 08:25:59 pm by Thaddy »
I might not give the answer that you want me to.. Peter Green 1969

Sky68

  • New member
  • *
  • Posts: 15
Re: Parallel archiver version 4.76 is here...
« Reply #3 on: May 25, 2018, 08:50:03 pm »
Hello,

>It is always a pleasure to play with your code.. but what does this >mean: -dFreePascal ? A duplicate of -dFPC?

This part was my old writing, you don't have to use -dFPC or -dFreePascal.

>Also, I can't find all sources (probably my mistake).
>Usually you make all your sources available? And the archive contains binaries (dll's) so I can't really test on other platforms.

I haven't yet given the source code of the Dynamic Link Libraries
for the compress and uncompress functions for Parallel Compression Library and for Parallel archiver, but i am always taking care of updating those Dynamic Link Libraries to the newer versions.

>That said: I know your code is excellent regarding parallel >programming and I use some of it (the lock-free stuff).

Yes i am more specialized in parallel programming and scalable synchronization algorithms, and the lock-free stuff too.


Thank you,
Amine Moulay Ramdane.



« Last Edit: May 25, 2018, 09:06:11 pm by Sky68 »

Thaddy

  • Hero Member
  • *****
  • Posts: 6101
Re: Parallel archiver version 4.76 is here...
« Reply #4 on: May 25, 2018, 09:25:32 pm »
Yes i am more specialized in parallel programming and scalable synchronization algorithms, and the lock-free stuff too.

Thank you,
Amine Moulay Ramdane.
For now can you explain what your mistake was in your lock-free linked list (the first CAS)? Because I am not entirely sure it is you....
And you made a - probably unintended - mistake in this answer that does not correspond to what I know from you .
« Last Edit: May 25, 2018, 09:28:48 pm by Thaddy »
I might not give the answer that you want me to.. Peter Green 1969

Sky68

  • New member
  • *
  • Posts: 15
Re: Parallel archiver version 4.76 is here...
« Reply #5 on: May 25, 2018, 10:42:17 pm »
Hello,


Wich lockfree source code is it ? can you give the name of the file
and wich line of the source code is it ?



Thank you,
Amine Moulay Ramdane.

 

Recent

Get Lazarus at SourceForge.net. Fast, secure and Free Open Source software downloads Open Hub project report for Lazarus