[regression] Add dictionary support
Dictionaries are prebuilt and saved as part of the data object. The config decides whether or not to use the dictionary if it is available. Configs that require dictionaries are only run with data that have dictionaries. The method will skip configs that are irrelevant, so for example ZSTD_compress() will skip configs with dictionaries. I've also trimmed the silesia source to 1MB per file (12 MB total), and added 500 samples from the github data set with a dictionary. I've intentionally added an extra line to the `results.csv` to make the nightly build fail, so that we can see how CircleCI reports it. Full list of changes: * Add pre-built dictionaries to the data. * Add `use_dictionary` and `no_pledged_src_size` flags to the config. * Add a config using a dictionary for every level. * Add a config that specifies no pledged source size. * Support dictionaries and streaming in the `zstdcli` method. * Add a context-reuse method using `ZSTD_compressCCtx()`. * Clean up the formatting of the `results.csv` file to align columns. * Add `--data`, `--config`, and `--method` flags to constrain each to a particular value. This is useful for debugging a failure or debugging a particular config/method/data.
This commit is contained in:
@@ -16,6 +16,8 @@
|
||||
#define ZSTD_STATIC_LINKING_ONLY
|
||||
#include <zstd.h>
|
||||
|
||||
#include "data.h"
|
||||
|
||||
typedef struct {
|
||||
ZSTD_cParameter param;
|
||||
unsigned value;
|
||||
@@ -41,8 +43,25 @@ typedef struct {
|
||||
* the parameters will be derived from these.
|
||||
*/
|
||||
param_values_t param_values;
|
||||
/**
|
||||
* Boolean parameter that says if we should use a dictionary. If the data
|
||||
* doesn't have a dictionary, this config is skipped. Defaults to no.
|
||||
*/
|
||||
int use_dictionary;
|
||||
/**
|
||||
* Boolean parameter that says if we should pass the pledged source size
|
||||
* when the method allows it. Defaults to yes.
|
||||
*/
|
||||
int no_pledged_src_size;
|
||||
} config_t;
|
||||
|
||||
/**
|
||||
* Returns true if the config should skip this data.
|
||||
* For instance, if the config requires a dictionary but the data doesn't have
|
||||
* one.
|
||||
*/
|
||||
int config_skip_data(config_t const* config, data_t const* data);
|
||||
|
||||
#define CONFIG_NO_LEVEL (-ZSTD_TARGETLENGTH_MAX - 1)
|
||||
/**
|
||||
* Returns the compression level specified by the config, or CONFIG_NO_LEVEL if
|
||||
|
||||
Reference in New Issue
Block a user