ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #1225 from jennifermliu/dev

Browse Source 
Split samples when building dictionary for COVER
This commit is contained in:
Nick Terrell
2018-07-13 13:26:15 -07:00
committed by GitHub
parent aa2415402d 612b346ed5
commit 45821fac0c
8 changed files with 76 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
programs/zstd.1.md
+6 -1
View File
@@ -223,11 +223,12 @@ Compression of small files similar to the sample set will be greatly improved.
This compares favorably to 4 bytes default.
However, it's up to the dictionary manager to not assign twice the same ID to
2 different dictionaries.
* `--train-cover[=k#,d=#,steps=#]`:
* `--train-cover[=k#,d=#,steps=#,split=#]`:
Select parameters for the default dictionary builder algorithm named cover.
If _d_ is not specified, then it tries _d_ = 6 and _d_ = 8.
If _k_ is not specified, then it tries _steps_ values in the range [50, 2000].
If _steps_ is not specified, then the default value of 40 is used.
If _split_ is not specified or split <= 0, then the default value of 100 is used.
Requires that _d_ <= _k_.
Selects segments of size _k_ with highest score to put in the dictionary.
@@ -237,6 +238,8 @@ Compression of small files similar to the sample set will be greatly improved.
algorithm will run faster with d <= _8_.
Good values for _k_ vary widely based on the input data, but a safe range is
[2 * _d_, 2000].
If _split_ is 100, all input samples are used for both training and testing
to find optimal _d_ and _k_ to build dictionary.
Supports multithreading if `zstd` is compiled with threading support.
Examples:
@@ -249,6 +252,8 @@ Compression of small files similar to the sample set will be greatly improved.
`zstd --train-cover=k=50 FILEs`
`zstd --train-cover=k=50,split=60 FILEs`
* `--train-legacy[=selectivity=#]`:
Use legacy dictionary builder algorithm with the given dictionary
_selectivity_ (default: 9).