Fix superblock mode (#2100)

Fixes:

Enable RLE blocks for superblock mode
Fix the limitation that the literals block must shrink. Instead, when we're within 200 bytes of the next header byte size, we will just use the next one up. That way we should (almost?) always have space for the table.
Remove the limitation that the first sub-block MUST have compressed literals and be compressed. Now one sub-block MUST be compressed (otherwise we fall back to raw block which is okay, since that is streamable). If no block has compressed literals that is okay, we will fix up the next Huffman table.
Handle the case where the last sub-block is uncompressed (maybe it is very small). Before it would skip superblock in this case, now we allow the last sub-block to be uncompressed. To do this we need to regenerate the correct repcodes.
Respect disableLiteralsCompression in superblock mode
Fix superblock mode to handle a block consisting of only compressed literals
Fix a off by 1 error in superblock mode that disabled it whenever there were last literals
Fix superblock mode with long literals/matches (> 0xFFFF)
Allow superblock mode to repeat Huffman tables
Respect ZSTD_minGain().
Tests:

Simple check for the condition in #2096.
When the simple_round_trip fuzzer enables superblock mode, it checks that the compressed size isn't expanded too much.
Remaining limitations:

O(targetCBlockSize^2) because we recompute statistics every sequence
Unable to split literals of length > targetCBlockSize into multiple sequences
Refuses to generate sub-blocks that don't shrink the compressed data, so we could end up with large sub-blocks. We should emit those sections as uncompressed blocks instead.
...
Fixes #2096
This commit is contained in:
Nick Terrell
2020-05-01 16:11:47 -07:00
committed by GitHub
parent 0eaaa4d3a7
commit e103d7b4a6
11 changed files with 378 additions and 134 deletions
+24 -2
View File
@@ -32,9 +32,12 @@ static size_t roundTripTest(void *result, size_t resultCapacity,
FUZZ_dataProducer_t *producer)
{
size_t cSize;
size_t dSize;
int targetCBlockSize = 0;
if (FUZZ_dataProducer_uint32Range(producer, 0, 1)) {
FUZZ_setRandomParameters(cctx, srcSize, producer);
cSize = ZSTD_compress2(cctx, compressed, compressedCapacity, src, srcSize);
FUZZ_ZASSERT(ZSTD_CCtx_getParameter(cctx, ZSTD_c_targetCBlockSize, &targetCBlockSize));
} else {
int const cLevel = FUZZ_dataProducer_int32Range(producer, kMinClevel, kMaxClevel);
@@ -42,14 +45,33 @@ static size_t roundTripTest(void *result, size_t resultCapacity,
cctx, compressed, compressedCapacity, src, srcSize, cLevel);
}
FUZZ_ZASSERT(cSize);
return ZSTD_decompressDCtx(dctx, result, resultCapacity, compressed, cSize);
dSize = ZSTD_decompressDCtx(dctx, result, resultCapacity, compressed, cSize);
FUZZ_ZASSERT(dSize);
/* When superblock is enabled make sure we don't expand the block more than expected. */
if (targetCBlockSize != 0) {
size_t normalCSize;
FUZZ_ZASSERT(ZSTD_CCtx_setParameter(cctx, ZSTD_c_targetCBlockSize, 0));
normalCSize = ZSTD_compress2(cctx, compressed, compressedCapacity, src, srcSize);
FUZZ_ZASSERT(normalCSize);
{
size_t const bytesPerBlock = 3 /* block header */
+ 5 /* Literal header */
+ 6 /* Huffman jump table */
+ 3 /* number of sequences */
+ 1 /* symbol compression modes */;
size_t const expectedExpansion = bytesPerBlock * (1 + (normalCSize / MAX(1, targetCBlockSize)));
size_t const allowedExpansion = (srcSize >> 4) + 3 * expectedExpansion + 10;
FUZZ_ASSERT(cSize <= normalCSize + allowedExpansion);
}
}
return dSize;
}
int LLVMFuzzerTestOneInput(const uint8_t *src, size_t size)
{
size_t const rBufSize = size;
void* rBuf = malloc(rBufSize);
size_t cBufSize = ZSTD_compressBound(size) * 2;
size_t cBufSize = ZSTD_compressBound(size);
void* cBuf;
/* Give a random portion of src data to the producer, to use for