From e873697625fa2a9f9a31c1851ebd15107b62c207 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Fri, 8 May 2020 10:51:37 -0700 Subject: [PATCH 1/5] updated documentation regarding ZSTD_CLEVEL make it clearer that it's useful for `tar --zstd`. --- programs/README.md | 35 +++++++++++++++++++---------------- 1 file changed, 19 insertions(+), 16 deletions(-) diff --git a/programs/README.md b/programs/README.md index 7668d49a2..a097cdac7 100644 --- a/programs/README.md +++ b/programs/README.md @@ -10,7 +10,7 @@ There are however other Makefile targets that create different variations of CLI - `zstd-decompress` : version of CLI which can only decompress zstd format -#### Compilation variables +### Compilation variables `zstd` scope can be altered by modifying the following `make` variables : - __HAVE_THREAD__ : multithreading is automatically enabled when `pthread` is detected. @@ -69,11 +69,11 @@ There are however other Makefile targets that create different variations of CLI Example : `make zstd BACKTRACE=1` -#### Aggregation of parameters +### Aggregation of parameters CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`. -#### Symlink shortcuts +### Symlink shortcuts It's possible to invoke `zstd` through a symlink. When the name of the symlink has a specific value, it triggers an associated behavior. - `zstdmt` : compress using all cores available on local system. @@ -86,7 +86,7 @@ When the name of the symlink has a specific value, it triggers an associated beh - `ungz`, `unxz` and `unlzma` will do the same, and will also remove source file by default (use `--keep` to preserve). -#### Dictionary builder in Command Line Interface +### Dictionary builder in Command Line Interface Zstd offers a training mode, which can be used to tune the algorithm for a selected type of data, by providing it with a few samples. The result of the training is stored in a file selected with the `-o` option (default name is `dictionary`), @@ -106,7 +106,7 @@ Usage of the dictionary builder and created dictionaries with CLI: 3. Decompress with the dictionary: `zstd --decompress FILE.zst -D dictionaryName` -#### Benchmark in Command Line Interface +### Benchmark in Command Line Interface CLI includes in-memory compression benchmark module for zstd. The benchmark is conducted using given filenames. The files are read into memory and joined together. It makes benchmark more precise as it eliminates I/O overhead. @@ -118,7 +118,7 @@ One can select compression levels starting from `-b` and ending with `-e`. The `-i` parameter selects minimal time used for each of tested levels. -#### Usage of Command Line Interface +### Usage of Command Line Interface The full list of options can be obtained with `-h` or `-H` parameter: ``` Usage : @@ -172,16 +172,19 @@ Benchmark arguments : --priority=rt : set process priority to real-time ``` -#### Restricted usage of Environment Variables -Using environment variables to set parameters has security implications. -Therefore, this avenue is intentionally restricted. -Only `ZSTD_CLEVEL` is supported currently, for setting compression level. -`ZSTD_CLEVEL` can be used to set the level between 1 and 19 (the "normal" range). -If the value of `ZSTD_CLEVEL` is not a valid integer, it will be ignored with a warning message. -`ZSTD_CLEVEL` just replaces the default compression level (`3`). -It can be overridden by corresponding command line arguments. +### Passing parameters through Environment Variables +`ZSTD_CLEVEL` can be used to modify the default compression level of `zstd` +(usually set to `3`) to another value between 1 and 19 (the "normal" range). +This can be useful when `zstd` CLI is invoked in a way that doesn't allow passing arguments. +One such scenario is `tar --zstd`. +As `ZSTD_CLEVEL` only replaces the default compression level, +it can then be overridden by corresponding command line arguments. -#### Long distance matching mode +There is no "generic" way to pass "any kind of parameter" to `zstd` in a pass-through manner. +Using environment variables for this purpose has security implications. +Therefore, this avenue is intentionally restricted and only supports `ZSTD_CLEVEL`. + +### Long distance matching mode The long distance matching mode, enabled with `--long`, is designed to improve the compression ratio for files with long matches at a large distance (up to the maximum window size, `128 MiB`) while still maintaining compression speed. @@ -233,7 +236,7 @@ The below table illustrates this on the [Silesia compression corpus]. | `zstd -10 --long`| `3.566` | `16.2 MB/s` | `415.7 MB/s` | -#### zstdgrep +### zstdgrep `zstdgrep` is a utility which makes it possible to `grep` directly a `.zst` compressed file. It's used the same way as normal `grep`, for example : From d564d564407b341b8c7343508e17251487731642 Mon Sep 17 00:00:00 2001 From: Yann Collet Date: Fri, 8 May 2020 13:35:59 -0700 Subject: [PATCH 2/5] document additional compilation macros ZSTD_NOBENCH, ZSTD_NODICT, ZSTD_NOCOMPRESS, ZSTD_NODECOMPRESS --- programs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/programs/README.md b/programs/README.md index 2f212f68a..53706de72 100644 --- a/programs/README.md +++ b/programs/README.md @@ -61,6 +61,24 @@ There are however other Makefile targets that create different variations of CLI In which case, linking stage will fail if `lz4` library cannot be found. This is useful to prevent silent feature disabling. +- __ZSTD_NOBENCH__ : `zstd` cli will be compiled without its integrated benchmark module. + This can be useful to produce smaller binaries. + In this case, the corresponding unit can also be excluded from compilation target. + +- __ZSTD_NODICT__ : `zstd` cli will be compiled without support for the integrated dictionary builder. + This can be useful to produce smaller binaries. + In this case, the corresponding unit can also be excluded from compilation target. + +- __ZSTD_NOCOMPRESS__ : `zstd` cli will be compiled without support for compression. + The resulting binary will only be able to decompress files. + This can be useful to produce smaller binaries. + A corresponding `Makefile` target using this ability is `zstd-decompress`. + +- __ZSTD_NODECOMPRESS__ : `zstd` cli will be compiled without support for decompression. + The resulting binary will only be able to compress files. + This can be useful to produce smaller binaries. + A corresponding `Makefile` target using this ability is `zstd-compress`. + - __BACKTRACE__ : `zstd` can display a stack backtrace when execution generates a runtime exception. By default, this feature may be degraded/disabled on some platforms unless additional compiler directives are From ef467fd428a26f93d498ef6c3cfb6aa57f9f49c9 Mon Sep 17 00:00:00 2001 From: Bimba Shrestha Date: Mon, 11 May 2020 12:22:45 -0500 Subject: [PATCH 3/5] documenting non-supported grep options for zstdgrep --- programs/README.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/programs/README.md b/programs/README.md index 53706de72..63ce79fc1 100644 --- a/programs/README.md +++ b/programs/README.md @@ -272,6 +272,19 @@ It's used the same way as normal `grep`, for example : `zstdgrep pattern file.zst` `zstdgrep` is _not_ compatible with dictionary compression. +`zstdgrep` does not support the following grep options + +``` +--dereference-recursive (-R) + --directories (-d) + --exclude + --exclude-from + --exclude-dir + --include + --null (-Z), + --null-data (-z) + --recursive (-r) +``` To search into a file compressed with a dictionary, it's necessary to decompress it using `zstd` or `zstdcat`, From f2f86b50ae345ac29c860d98d73e8454ec517a4a Mon Sep 17 00:00:00 2001 From: Nick Terrell Date: Mon, 11 May 2020 12:29:03 -0700 Subject: [PATCH 4/5] Update benchmarks and add fast mode --- README.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index cb956690d..5c300fdc4 100644 --- a/README.md +++ b/README.md @@ -31,10 +31,10 @@ a list of known ports and bindings is provided on [Zstandard homepage](http://ww ## Benchmarks For reference, several fast compression algorithms were tested and compared -on a server running Arch Linux (`Linux version 5.0.5-arch1-1`), +on a server running Arch Linux (`Linux version 5.5.11-arch1-1`), with a Core i9-9900K CPU @ 5.0GHz, using [lzbench], an open-source in-memory benchmark by @inikep -compiled with [gcc] 8.2.1, +compiled with [gcc] 9.3.0, on the [Silesia compression corpus]. [lzbench]: https://github.com/inikep/lzbench @@ -43,18 +43,26 @@ on the [Silesia compression corpus]. | Compressor name | Ratio | Compression| Decompress.| | --------------- | ------| -----------| ---------- | -| **zstd 1.4.4 -1** | 2.884 | 520 MB/s | 1600 MB/s | -| zlib 1.2.11 -1 | 2.743 | 110 MB/s | 440 MB/s | -| brotli 1.0.7 -0 | 2.701 | 430 MB/s | 470 MB/s | -| quicklz 1.5.0 -1 | 2.238 | 600 MB/s | 800 MB/s | -| lzo1x 2.09 -1 | 2.106 | 680 MB/s | 950 MB/s | -| lz4 1.8.3 | 2.101 | 800 MB/s | 4220 MB/s | -| snappy 1.1.4 | 2.073 | 580 MB/s | 2020 MB/s | -| lzf 3.6 -1 | 2.077 | 440 MB/s | 930 MB/s | +| **zstd 1.4.5 -1** | 2.884 | 500 MB/s | 1660 MB/s | +| zlib 1.2.11 -1 | 2.743 | 90 MB/s | 400 MB/s | +| brotli 1.0.7 -0 | 2.703 | 400 MB/s | 450 MB/s | +| **zstd 1.4.5 --fast=1** | 2.434 | 570 MB/s | 2200 MB/s | +| **zstd 1.4.5 --fast=3** | 2.312 | 640 MB/s | 2300 MB/s | +| quicklz 1.5.0 -1 | 2.238 | 560 MB/s | 710 MB/s | +| **zstd 1.4.5 --fast=5** | 2.178 | 700 MB/s | 2420 MB/s | +| lzo1x 2.10 -1 | 2.106 | 690 MB/s | 820 MB/s | +| lz4 1.9.2 | 2.101 | 740 MB/s | 4530 MB/s | +| **zstd 1.4.5 --fast=7** | 2.096 | 750 MB/s | 2480 MB/s | +| lzf 3.6 -1 | 2.077 | 410 MB/s | 860 MB/s | +| snappy 1.1.8 | 2.073 | 560 MB/s | 1790 MB/s | [zlib]: http://www.zlib.net/ [LZ4]: http://www.lz4.org/ +The negative compression levels, specified with `--fast=#`, +offer faster compression and decompression speed in exchange for some loss in +compression ratio compared to level 1, as seen in the table above. + Zstd can also offer stronger compression ratios at the cost of compression speed. Speed vs Compression trade-off is configurable by small increments. Decompression speed is preserved and remains roughly the same at all settings, From c6636afbbb0d545347b17565b007195efaa3764e Mon Sep 17 00:00:00 2001 From: "W. Felix Handte" Date: Mon, 11 May 2020 18:53:32 -0400 Subject: [PATCH 5/5] Fix ZSTD_estimateCCtxSize() Under ASAN `ZSTD_estimateCCtxSize()` provides estimates for one-shot compression, which is guaranteed not to buffer inputs or outputs. So it ignores the sizes of the buffers, assuming they'll be zero. However, the actual workspace allocation logic always allocates those buffers, and when running under ASAN, the workspace surrounds every allocation with 256 bytes of redzone. So the 0-sized buffers end up consuming 512 bytes of space, which is accounted for in the actual allocation path through the use of `ZSTD_cwksp_alloc_size()` but isn't in the estimation path, since it ignores the buffers entirely. This commit fixes this. --- lib/compress/zstd_compress.c | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/lib/compress/zstd_compress.c b/lib/compress/zstd_compress.c index ff8a00e6b..538f8e9e2 100644 --- a/lib/compress/zstd_compress.c +++ b/lib/compress/zstd_compress.c @@ -1144,13 +1144,26 @@ size_t ZSTD_estimateCCtxSize_usingCCtxParams(const ZSTD_CCtx_params* params) size_t const ldmSpace = ZSTD_ldm_getTableSize(params->ldmParams); size_t const ldmSeqSpace = ZSTD_cwksp_alloc_size(ZSTD_ldm_getMaxNbSeq(params->ldmParams, blockSize) * sizeof(rawSeq)); - size_t const neededSpace = entropySpace + blockStateSpace + tokenSpace + - matchStateSize + ldmSpace + ldmSeqSpace; + /* estimateCCtxSize is for one-shot compression. So no buffers should + * be needed. However, we still allocate two 0-sized buffers, which can + * take space under ASAN. */ + size_t const bufferSpace = ZSTD_cwksp_alloc_size(0) + + ZSTD_cwksp_alloc_size(0); + size_t const cctxSpace = ZSTD_cwksp_alloc_size(sizeof(ZSTD_CCtx)); - DEBUGLOG(5, "sizeof(ZSTD_CCtx) : %u", (U32)cctxSpace); + size_t const neededSpace = + cctxSpace + + entropySpace + + blockStateSpace + + ldmSpace + + ldmSeqSpace + + matchStateSize + + tokenSpace + + bufferSpace; + DEBUGLOG(5, "estimate workspace : %u", (U32)neededSpace); - return cctxSpace + neededSpace; + return neededSpace; } }