Notes:What you may like to know:
- It is Free and remains as is. I made it to enable people to have a safer, easier and faster life.
- It is made using the latest versions of FreePascal and Lazarus (but no FCL or LCL) and internally uses SQLite and ZSTD statically; there is no external dependency beside OS.
- Source code is available with a permissive licence.
- It is at the beta stage; keep the input files. It is designed to be safe, have crash resistance, and prevent reading problematic data and exploited by many vulnerabilities, but for now it is intended only for evaluation purposes.
- It is fast—really fast.
- It is smart. It configures itself as needed; there are not many dials to play with.
- It is very resource-friendly.
- It will get more updates including Encryption.
If you like to know more, continue.
Safety: Everything is made with safety as the first priority.
If you want to verify the packing process, use these parameters: `--activate-other-options --verify-pack`
It will pack, unpack, and compare the input and output to make sure everything went correctly.
Privacy: No private metadata will be written to the file.
Benchmarking: Numbers are from my personal machine with their corresponding official program in an out-of-the-box configuration. And all are considered to be in a warm state with no antivirus interference. Please test for yourself and report back if possible. On the first try of reading many files, Windows Defender (or any other antivirus) makes everything slow as it is scanning the files. Run any test at least two times.
File Format: This version is Draft 0. At the end of this stage, it will get to Final 1. Any file made now will be supported by future versions, so rest easy.
Internally, it is based on SQLite3 and tries to inherit all the goods like reliability, safety, and speed, but the file header is changed to prevent any mistakes and future compatibility.
You can use the `--activate-other-options --transform-to-sqlite3` parameter to convert a Pack file to SQLite3, and use the `--activate-other-options --transform-to-pack` to go back.
Platforms: For now, it is only available for Windows and Linux. It is designed to run on any architecture supported by FreePascal. Soon, support for macOS, and more will come.
Program Version: They will use incremental versioning. So the first public version is 1, and the next with fixes will be 2, and so on.
Pascal: Other than static builds of SQLite and Zstandard all the code is in Pascal, in Delphi mode (as I prefer the generic style), and it only uses system and objfpc units, so no FCL or LCL. I do not have a plan to support the Delphi compiler, as it has been years since I tried it.
Design: Format and Programs are separate from each other.
Data structure:
The standard format in the Draft 0 version is about having three tables in SQLite:
- One for Items to store files or data information and structure. For example, if it is a folder or a file, what is the name.
- One for Contents. They are packages of raw data from a chunk, a whole, or many of the items. They may be compressed if needed.
- One for the relationship between Items and Contents. Showing what part of the content is used for what part of what item.
This approach, while simple at first glance, enables grouping similar data to fit together to enable better compression, while maintaining fast random access.
All the columns were chosen carefully (to match the SQLite3 file format) to keep the overhead at a minimum and let the reader extract information as needed. So getting a list of items (or files) is a matter of milliseconds.
Using such a
popular, safe, and
well-tested database, ensures safety, crash resistance, and transactional updates,
future compatibility and even being
faster than file systems in some cases, although Pack takes this to another level and, in more cases, is faster than file systems (because of the handling of small files, and working with fast compressed data).
For the curious: Look at the previous `File Format` note to convert Pack files to SQLite3 and back.
Using Zstandard as a well-used,
standard, popular and new proper alternative to DEFLATE, enables Pack to compress fast, and decompress even faster. Although it comes with multi-threading, Pack uses its own, configuring as needed to get even more speed out of this great library, enabling it to even be a little faster than Zstandard's official client while being a container format and supporting multiple files and folders and random access. All the compressed data has
Zstandard signature using the extremely fast
xxHashPrograms:
Implementing Pack and Unpack should be done in maybe a couple of hundreds of lines (and I may do it as a demo). But the officially provided programs use a multithreaded (depending on the CPU specification and input data), queue approach. It scans all the input files or data, estimates how it can split them into contents, and determines if they are worth the compression. Then read them asynchronously, compress if needed using Zstandard, and store them synchronously into SQLite. All items' information is stored at the start to keep accessing them faster, and with fewer disk jumps.
It is designed for new generations of hardware, for multicore CPUs, fast SSDs, and NVMes. Albeit it can show capability with even one core and an old HDD anyway, you see the difference on newer hardware much better.
What made Pack considerable, is that it configures SQLite and Zstandard to their best ability, and the code is written using a custom optimized standard library, so almost no memory allocation is needed, and hopefully no extra CPU cycle is wasted.
Considering the resource usage, user experience, and speed, it can be a proper choice for everyday use as it goes easy on the end-user hardware, and because of its high throughput compared to needed resources, it can be a proper choice to be used on servers and for heavy-duty tasks.
Final Note:SQLite and Zstandard are fantastic; I felt tying them together using a fresh Pack approach produced an interesting outcome, and for now, results show that. Beside these, without dear FreePascal and Lazarus, my life would be harder. My respect for the teams.
Recent
