[zdict] Remove ZDICT_CONTENTSIZE_MIN restriction for ZDICT_finalizeDictionary
Allow the `dictContentSize` to be any size. The finalized dictionary content size must be at least as large as the maximum repcode (8). So we add zero bytes to the dictionary to ensure that we meet that requirement. I've removed this restriction because its been causing us headaches when people complain that dictionary training failed. It fails because there isn't enough useful content to put in the dictionary. Either because every sample is exactly the same and less than ZDICT_CONTENTSIZE_MIN bytes, or there isn't enough content. Instead, we should succeed in creating the dictionary, and it is up to the user to decide if it is worthwhile. It is possible that the tables alone provide enough value. NOTE: This allows us to produce dictionaries with finalized `dictContentSize < ZDICT_CONTENTSIZE_MIN`. But, they are still valid zstd dictionaries. We could remove the `ZDICT_CONTENTSIZE_MIN` macro, but I've decided to leave that for now, so we don't break users.
This commit is contained in:
+2
-2
@@ -237,7 +237,6 @@ typedef struct {
|
||||
* is presumed that the most profitable content is at the end of the dictionary,
|
||||
* since that is the cheapest to reference.
|
||||
*
|
||||
* `dictContentSize` must be >= ZDICT_CONTENTSIZE_MIN bytes.
|
||||
* `maxDictSize` must be >= max(dictContentSize, ZSTD_DICTSIZE_MIN).
|
||||
*
|
||||
* @return: size of dictionary stored into `dstDictBuffer` (<= `maxDictSize`),
|
||||
@@ -272,8 +271,9 @@ ZDICTLIB_API const char* ZDICT_getErrorName(size_t errorCode);
|
||||
* Use them only in association with static linking.
|
||||
* ==================================================================================== */
|
||||
|
||||
#define ZDICT_CONTENTSIZE_MIN 128
|
||||
#define ZDICT_DICTSIZE_MIN 256
|
||||
/* Deprecated: Remove in v1.6.0 */
|
||||
#define ZDICT_CONTENTSIZE_MIN 128
|
||||
|
||||
/*! ZDICT_cover_params_t:
|
||||
* k and d are the only required parameters.
|
||||
|
||||
Reference in New Issue
Block a user