[zdict] Remove ZDICT_CONTENTSIZE_MIN restriction for ZDICT_finalizeDictionary

Allow the `dictContentSize` to be any size. The finalized dictionary
content size must be at least as large as the maximum repcode (8). So we
add zero bytes to the dictionary to ensure that we meet that
requirement.

I've removed this restriction because its been causing us headaches when
people complain that dictionary training failed. It fails because there
isn't enough useful content to put in the dictionary. Either because
every sample is exactly the same and less than ZDICT_CONTENTSIZE_MIN bytes,
or there isn't enough content. Instead, we should succeed in creating
the dictionary, and it is up to the user to decide if it is worthwhile.
It is possible that the tables alone provide enough value.

NOTE: This allows us to produce dictionaries with finalized
`dictContentSize < ZDICT_CONTENTSIZE_MIN`. But, they are still valid
zstd dictionaries. We could remove the `ZDICT_CONTENTSIZE_MIN` macro,
but I've decided to leave that for now, so we don't break users.
This commit is contained in:
Nick Terrell
2021-11-30 18:02:26 -08:00
parent 7847c2fd68
commit 0356e05f07
3 changed files with 53 additions and 13 deletions
-4
View File
@@ -1018,12 +1018,8 @@ zstd -o tmpDict --train "$TESTDIR"/*.c "$PRGDIR"/*.c
test -f tmpDict
zstd --train "$TESTDIR"/*.c "$PRGDIR"/*.c
test -f dictionary
println "- Test dictionary training fails"
echo "000000000000000000000000000000000" > tmpz
zstd --train tmpz tmpz tmpz tmpz tmpz tmpz tmpz tmpz tmpz && die "Dictionary training should fail : source is all zeros"
if [ -n "$hasMT" ]
then
zstd --train -T0 tmpz tmpz tmpz tmpz tmpz tmpz tmpz tmpz tmpz && die "Dictionary training should fail : source is all zeros"
println "- Create dictionary with multithreading enabled"
zstd --train -T0 "$TESTDIR"/*.c "$PRGDIR"/*.c -o tmpDict
fi