[regression] Add dictionary support

Dictionaries are prebuilt and saved as part of the data object.
The config decides whether or not to use the dictionary if it is
available. Configs that require dictionaries are only run with
data that have dictionaries. The method will skip configs that are
irrelevant, so for example ZSTD_compress() will skip configs with
dictionaries.

I've also trimmed the silesia source to 1MB per file (12 MB total),
and added 500 samples from the github data set with a dictionary.

I've intentionally added an extra line to the `results.csv` to make
the nightly build fail, so that we can see how CircleCI reports it.

Full list of changes:

* Add pre-built dictionaries to the data.
* Add `use_dictionary` and `no_pledged_src_size` flags to the config.
* Add a config using a dictionary for every level.
* Add a config that specifies no pledged source size.
* Support dictionaries and streaming in the `zstdcli` method.
* Add a context-reuse method using `ZSTD_compressCCtx()`.
* Clean up the formatting of the `results.csv` file to align columns.
* Add `--data`, `--config`, and `--method` flags to constrain each
  to a particular value. This is useful for debugging a failure
  or debugging a particular config/method/data.
This commit is contained in:
Nick Terrell
2018-11-30 18:23:01 -08:00
parent 386c9ab58a
commit e859862341
7 changed files with 657 additions and 226 deletions
+19
View File
@@ -16,6 +16,8 @@
#define ZSTD_STATIC_LINKING_ONLY
#include <zstd.h>
#include "data.h"
typedef struct {
ZSTD_cParameter param;
unsigned value;
@@ -41,8 +43,25 @@ typedef struct {
* the parameters will be derived from these.
*/
param_values_t param_values;
/**
* Boolean parameter that says if we should use a dictionary. If the data
* doesn't have a dictionary, this config is skipped. Defaults to no.
*/
int use_dictionary;
/**
* Boolean parameter that says if we should pass the pledged source size
* when the method allows it. Defaults to yes.
*/
int no_pledged_src_size;
} config_t;
/**
* Returns true if the config should skip this data.
* For instance, if the config requires a dictionary but the data doesn't have
* one.
*/
int config_skip_data(config_t const* config, data_t const* data);
#define CONFIG_NO_LEVEL (-ZSTD_TARGETLENGTH_MAX - 1)
/**
* Returns the compression level specified by the config, or CONFIG_NO_LEVEL if