From 02da4497f05ef5724c50689639bcac096d9c29ef Mon Sep 17 00:00:00 2001 From: Tyler Tran Date: Mon, 22 Jul 2019 16:33:22 -0700 Subject: [PATCH 1/4] Adding documentation for shrink flag PR #1656 --- programs/README.md | 4 ++-- programs/zstd.1 | 6 +++--- programs/zstd.1.md | 11 ++++++++++- 3 files changed, 15 insertions(+), 6 deletions(-) diff --git a/programs/README.md b/programs/README.md index d9ef5dd2a..c3a5590d6 100644 --- a/programs/README.md +++ b/programs/README.md @@ -157,8 +157,8 @@ Advanced arguments : Dictionary builder : --train ## : create a dictionary from a training set of files ---train-cover[=k=#,d=#,steps=#,split=#] : use the cover algorithm with optional args ---train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#] : use the fastcover algorithm with optional args +--train-cover[=k=#,d=#,steps=#,split=#,shrink[=#]] : use the cover algorithm with optional args +--train-fastcover[=k=#,d=#,f=#,steps=#,split=#,shrink[=#],accel=#] : use the fastcover algorithm with optional args --train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9) -o file : `file` is dictionary name (default: dictionary) --maxdict=# : limit dictionary to specified size (default: 112640) diff --git a/programs/zstd.1 b/programs/zstd.1 index beca9da42..8ef3edd22 100644 --- a/programs/zstd.1 +++ b/programs/zstd.1 @@ -229,8 +229,8 @@ Split input files in blocks of size # (default: no split) A dictionary ID is a locally unique ID that a decoder can use to verify it is using the right dictionary\. By default, zstd will create a 4\-bytes random number ID\. It\'s possible to give a precise number instead\. Short numbers have an advantage : an ID < 256 will only need 1 byte in the compressed frame header, and an ID < 65536 will only need 2 bytes\. This compares favorably to 4 bytes default\. However, it\'s up to the dictionary manager to not assign twice the same ID to 2 different dictionaries\. . .TP -\fB\-\-train\-cover[=k#,d=#,steps=#,split=#]\fR -Select parameters for the default dictionary builder algorithm named cover\. If \fId\fR is not specified, then it tries \fId\fR = 6 and \fId\fR = 8\. If \fIk\fR is not specified, then it tries \fIsteps\fR values in the range [50, 2000]\. If \fIsteps\fR is not specified, then the default value of 40 is used\. If \fIsplit\fR is not specified or split <= 0, then the default value of 100 is used\. Requires that \fId\fR <= \fIk\fR\. +\fB\-\-train\-cover[=k#,d=#,steps=#,split=#,shrink[=#]]\fR +Select parameters for the default dictionary builder algorithm named cover\. If \fId\fR is not specified, then it tries \fId\fR = 6 and \fId\fR = 8\. If \fIk\fR is not specified, then it tries \fIsteps\fR values in the range [50, 2000]\. If \fIsteps\fR is not specified, then the default value of 40 is used\. If \fIsplit\fR is not specified or split <= 0, then the default value of 100 is used\. Requires that \fId\fR <= \fIk\fR\. If \fIshrink\fR flag is not used, then the default value for \fIshrinkDict\fR of 0 is used\. If \fIshrink\fR is not specified, then the default value for \fIshrinkDictMaxRegression\fR of 1 is used\. . .IP Selects segments of size \fIk\fR with highest score to put in the dictionary\. The score of a segment is computed by the sum of the frequencies of all the subsegments of size \fId\fR\. Generally \fId\fR should be in the range [6, 8], occasionally up to 16, but the algorithm will run faster with d <= \fI8\fR\. Good values for \fIk\fR vary widely based on the input data, but a safe range is [2 * \fId\fR, 2000]\. If \fIsplit\fR is 100, all input samples are used for both training and testing to find optimal \fId\fR and \fIk\fR to build dictionary\. Supports multithreading if \fBzstd\fR is compiled with threading support\. @@ -254,7 +254,7 @@ Examples: \fBzstd \-\-train\-cover=k=50,split=60 FILEs\fR . .TP -\fB\-\-train\-fastcover[=k#,d=#,f=#,steps=#,split=#,accel=#]\fR +\fB\-\-train\-fastcover[=k#,d=#,f=#,steps=#,split=#,shrink[=#],accel=#]\fR Same as cover but with extra parameters \fIf\fR and \fIaccel\fR and different default value of split If \fIsplit\fR is not specified, then it tries \fIsplit\fR = 75\. If \fIf\fR is not specified, then it tries \fIf\fR = 20\. Requires that 0 < \fIf\fR < 32\. If \fIaccel\fR is not specified, then it tries \fIaccel\fR = 1\. Requires that 0 < \fIaccel\fR <= 10\. Requires that \fId\fR = 6 or \fId\fR = 8\. . .IP diff --git a/programs/zstd.1.md b/programs/zstd.1.md index 93c6fa400..ca4d64301 100644 --- a/programs/zstd.1.md +++ b/programs/zstd.1.md @@ -244,13 +244,15 @@ Compression of small files similar to the sample set will be greatly improved. This compares favorably to 4 bytes default. However, it's up to the dictionary manager to not assign twice the same ID to 2 different dictionaries. -* `--train-cover[=k#,d=#,steps=#,split=#]`: +* `--train-cover[=k#,d=#,steps=#,split=#,shrink[=#]]`: Select parameters for the default dictionary builder algorithm named cover. If _d_ is not specified, then it tries _d_ = 6 and _d_ = 8. If _k_ is not specified, then it tries _steps_ values in the range [50, 2000]. If _steps_ is not specified, then the default value of 40 is used. If _split_ is not specified or split <= 0, then the default value of 100 is used. Requires that _d_ <= _k_. + If _shrink_ flag is not used, then the default value for _shrinkDict_ of 0 is used. + If _shrink_ is not specified, then the default value for _shrinkDictMaxRegression_ of 1 is used. Selects segments of size _k_ with highest score to put in the dictionary. The score of a segment is computed by the sum of the frequencies of all the @@ -262,6 +264,9 @@ Compression of small files similar to the sample set will be greatly improved. If _split_ is 100, all input samples are used for both training and testing to find optimal _d_ and _k_ to build dictionary. Supports multithreading if `zstd` is compiled with threading support. + Having _shrink_ enabled takes a truncated dictionary of minimum size and doubles + in size until compression ratio of the truncated dictionary is at most + _shrinkDictMaxRegression_% worse than the compression ratio of the largest dictionary. Examples: @@ -275,6 +280,10 @@ Compression of small files similar to the sample set will be greatly improved. `zstd --train-cover=k=50,split=60 FILEs` + `zstd --train-cover=shrink FILEs` + + `zstd --train-cover=shrink=2 FILEs` + * `--train-fastcover[=k#,d=#,f=#,steps=#,split=#,accel=#]`: Same as cover but with extra parameters _f_ and _accel_ and different default value of split If _split_ is not specified, then it tries _split_ = 75. From f8c1d7979c884ea1ba4f3439225adfcf72c23d6c Mon Sep 17 00:00:00 2001 From: Tyler Tran Date: Mon, 22 Jul 2019 16:36:44 -0700 Subject: [PATCH 2/4] modifying minor nit --- programs/zstd.1.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/programs/zstd.1.md b/programs/zstd.1.md index ca4d64301..3ab2667a0 100644 --- a/programs/zstd.1.md +++ b/programs/zstd.1.md @@ -266,7 +266,7 @@ Compression of small files similar to the sample set will be greatly improved. Supports multithreading if `zstd` is compiled with threading support. Having _shrink_ enabled takes a truncated dictionary of minimum size and doubles in size until compression ratio of the truncated dictionary is at most - _shrinkDictMaxRegression_% worse than the compression ratio of the largest dictionary. + _shrinkDictMaxRegression%_ worse than the compression ratio of the largest dictionary. Examples: From 12d60a9bd902269390159717cb61ac0126f037a9 Mon Sep 17 00:00:00 2001 From: Tyler Tran Date: Wed, 24 Jul 2019 13:52:34 -0700 Subject: [PATCH 3/4] removing changes to zstd.1 --- programs/zstd.1 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/programs/zstd.1 b/programs/zstd.1 index 8ef3edd22..87aa80dd3 100644 --- a/programs/zstd.1 +++ b/programs/zstd.1 @@ -229,8 +229,8 @@ Split input files in blocks of size # (default: no split) A dictionary ID is a locally unique ID that a decoder can use to verify it is using the right dictionary\. By default, zstd will create a 4\-bytes random number ID\. It\'s possible to give a precise number instead\. Short numbers have an advantage : an ID < 256 will only need 1 byte in the compressed frame header, and an ID < 65536 will only need 2 bytes\. This compares favorably to 4 bytes default\. However, it\'s up to the dictionary manager to not assign twice the same ID to 2 different dictionaries\. . .TP -\fB\-\-train\-cover[=k#,d=#,steps=#,split=#,shrink[=#]]\fR -Select parameters for the default dictionary builder algorithm named cover\. If \fId\fR is not specified, then it tries \fId\fR = 6 and \fId\fR = 8\. If \fIk\fR is not specified, then it tries \fIsteps\fR values in the range [50, 2000]\. If \fIsteps\fR is not specified, then the default value of 40 is used\. If \fIsplit\fR is not specified or split <= 0, then the default value of 100 is used\. Requires that \fId\fR <= \fIk\fR\. If \fIshrink\fR flag is not used, then the default value for \fIshrinkDict\fR of 0 is used\. If \fIshrink\fR is not specified, then the default value for \fIshrinkDictMaxRegression\fR of 1 is used\. +\fB\-\-train\-cover[=k#,d=#,steps=#,split=#]\fR \fB\-\-train\-cover[=k#,d=#,steps=#,split=#,shrink[=#]]\fR +Select parameters for the default dictionary builder algorithm named cover\. If \fId\fR is not specified, then it tries \fId\fR = 6 and \fId\fR = 8\. If \fIk\fR is not specified, then it tries \fIsteps\fR values in the range [50, 2000]\. If \fIsteps\fR is not specified, then the default value of 40 is used\. If \fIsplit\fR is not specified or split <= 0, then the default value of 100 is used\. Requires that \fId\fR <= \fIk\fR\. . .IP Selects segments of size \fIk\fR with highest score to put in the dictionary\. The score of a segment is computed by the sum of the frequencies of all the subsegments of size \fId\fR\. Generally \fId\fR should be in the range [6, 8], occasionally up to 16, but the algorithm will run faster with d <= \fI8\fR\. Good values for \fIk\fR vary widely based on the input data, but a safe range is [2 * \fId\fR, 2000]\. If \fIsplit\fR is 100, all input samples are used for both training and testing to find optimal \fId\fR and \fIk\fR to build dictionary\. Supports multithreading if \fBzstd\fR is compiled with threading support\. From 5a61e66f7bb381cc4e2ef9ac0216addcd62f6ec0 Mon Sep 17 00:00:00 2001 From: Tyler Tran Date: Wed, 24 Jul 2019 13:53:50 -0700 Subject: [PATCH 4/4] previous commit did not undo all changes --- programs/zstd.1 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/programs/zstd.1 b/programs/zstd.1 index 87aa80dd3..beca9da42 100644 --- a/programs/zstd.1 +++ b/programs/zstd.1 @@ -229,7 +229,7 @@ Split input files in blocks of size # (default: no split) A dictionary ID is a locally unique ID that a decoder can use to verify it is using the right dictionary\. By default, zstd will create a 4\-bytes random number ID\. It\'s possible to give a precise number instead\. Short numbers have an advantage : an ID < 256 will only need 1 byte in the compressed frame header, and an ID < 65536 will only need 2 bytes\. This compares favorably to 4 bytes default\. However, it\'s up to the dictionary manager to not assign twice the same ID to 2 different dictionaries\. . .TP -\fB\-\-train\-cover[=k#,d=#,steps=#,split=#]\fR \fB\-\-train\-cover[=k#,d=#,steps=#,split=#,shrink[=#]]\fR +\fB\-\-train\-cover[=k#,d=#,steps=#,split=#]\fR Select parameters for the default dictionary builder algorithm named cover\. If \fId\fR is not specified, then it tries \fId\fR = 6 and \fId\fR = 8\. If \fIk\fR is not specified, then it tries \fIsteps\fR values in the range [50, 2000]\. If \fIsteps\fR is not specified, then the default value of 40 is used\. If \fIsplit\fR is not specified or split <= 0, then the default value of 100 is used\. Requires that \fId\fR <= \fIk\fR\. . .IP @@ -254,7 +254,7 @@ Examples: \fBzstd \-\-train\-cover=k=50,split=60 FILEs\fR . .TP -\fB\-\-train\-fastcover[=k#,d=#,f=#,steps=#,split=#,shrink[=#],accel=#]\fR +\fB\-\-train\-fastcover[=k#,d=#,f=#,steps=#,split=#,accel=#]\fR Same as cover but with extra parameters \fIf\fR and \fIaccel\fR and different default value of split If \fIsplit\fR is not specified, then it tries \fIsplit\fR = 75\. If \fIf\fR is not specified, then it tries \fIf\fR = 20\. Requires that 0 < \fIf\fR < 32\. If \fIaccel\fR is not specified, then it tries \fIaccel\fR = 1\. Requires that 0 < \fIaccel\fR <= 10\. Requires that \fId\fR = 6 or \fId\fR = 8\. . .IP