fix : huge (>4GB) stream of blocks
experimental function ZSTD_compressBlock() is designed for very small data in mind, for situation where saving the ~12 bytes of frame header can actually make a difference. Some systems though may have to deal with small and large data entangled. If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round. That's why it's possible to compress in multiple rounds. This is a chain of compressed blocks. Some users push this capability to the limit, encoding gigantic chain of blocks. On crossing the 4GB limit, some internal overflow occurs. This fix moves the overflow correction mechanism higher in the call chain, so that it's applied also to gigantic chains of blocks. Added a test case in fuzzer.c, which crashes before the fix, and pass now.
This commit is contained in:
@@ -1291,6 +1291,20 @@ static int basicUnitTests(U32 seed, double compressibility)
|
||||
if (r != blockSize) goto _output_error; }
|
||||
DISPLAYLEVEL(3, "OK \n");
|
||||
|
||||
/* very long stream of block compression */
|
||||
DISPLAYLEVEL(3, "test%3i : Huge block streaming compression test : ", testNb++);
|
||||
CHECK( ZSTD_compressBegin(cctx, -99) ); /* we just want to quickly overflow internal U32 index */
|
||||
CHECK( ZSTD_getBlockSize(cctx) >= blockSize);
|
||||
{ U64 const toCompress = 5000000000ULL; /* > 4 GB */
|
||||
U64 compressed = 0;
|
||||
while (compressed < toCompress) {
|
||||
size_t const blockCSize = ZSTD_compressBlock(cctx, compressedBuffer, ZSTD_compressBound(blockSize), CNBuffer, blockSize);
|
||||
if (ZSTD_isError(cSize)) goto _output_error;
|
||||
compressed += blockCSize;
|
||||
}
|
||||
}
|
||||
DISPLAYLEVEL(3, "OK \n");
|
||||
|
||||
/* dictionary block compression */
|
||||
DISPLAYLEVEL(3, "test%3i : Dictionary Block compression test : ", testNb++);
|
||||
CHECK( ZSTD_compressBegin_usingDict(cctx, CNBuffer, dictSize, 5) );
|
||||
|
||||
Reference in New Issue
Block a user