Add support for in-place decompression
* Add a function and macro ZSTD_decompressionMargin() that computes the decompression margin for in-place decompression. The function computes a tight margin that works in all cases, and the macro computes an upper bound that will only work if flush isn't used. * When doing in-place decompression, make sure that our output buffer doesn't overlap with the input buffer. This ensures that we don't decide to use the portion of the output buffer that overlaps the input buffer for temporary memory, like for literals. * Add a simple unit test. * Add in-place decompression to the simple_round_trip and stream_round_trip fuzzers. This should help verify that our margin stays correct.
This commit is contained in:
committed by
Nick Terrell
parent
423500d1ae
commit
5b266196a4
+45
@@ -1427,6 +1427,51 @@ ZSTDLIB_STATIC_API unsigned long long ZSTD_decompressBound(const void* src, size
|
||||
* or an error code (if srcSize is too small) */
|
||||
ZSTDLIB_STATIC_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);
|
||||
|
||||
/*! ZSTD_decompressionMargin() :
|
||||
* Zstd supports in-place decompression, where the input and output buffers overlap.
|
||||
* In this case, the output buffer must be at least (Margin + Output_Size) bytes large,
|
||||
* and the input buffer must be at the end of the output buffer.
|
||||
*
|
||||
* _______________________ Output Buffer ________________________
|
||||
* | |
|
||||
* | ____ Input Buffer ____|
|
||||
* | | |
|
||||
* v v v
|
||||
* |---------------------------------------|-----------|----------|
|
||||
* ^ ^ ^
|
||||
* |___________________ Output_Size ___________________|_ Margin _|
|
||||
*
|
||||
* NOTE: See also ZSTD_DECOMPRESSION_MARGIN().
|
||||
* NOTE: This applies only to single-pass decompression through ZSTD_decompress() or
|
||||
* ZSTD_decompressDCtx().
|
||||
* NOTE: This function supports multi-frame input.
|
||||
*
|
||||
* @param src The compressed frame(s)
|
||||
* @param srcSize The size of the compressed frame(s)
|
||||
* @returns The decompression margin or an error that can be checked with ZSTD_isError().
|
||||
*/
|
||||
ZSTDLIB_STATIC_API size_t ZSTD_decompressionMargin(const void* src, size_t srcSize);
|
||||
|
||||
/*! ZSTD_DECOMPRESS_MARGIN() :
|
||||
* Similar to ZSTD_decompressionMargin(), but instead of computing the margin from
|
||||
* the compressed frame, compute it from the original size and the blockSizeLog.
|
||||
* See ZSTD_decompressionMargin() for details.
|
||||
*
|
||||
* WARNING: This macro does not support multi-frame input, the input must be a single
|
||||
* zstd frame. If you need that support use the function, or implement it yourself.
|
||||
*
|
||||
* @param originalSize The original uncompressed size of the data.
|
||||
* @param blockSize The block size == MIN(windowSize, ZSTD_BLOCKSIZE_MAX).
|
||||
* Unless you explicitly set the windowLog smaller than
|
||||
* ZSTD_BLOCKSIZELOG_MAX you can just use ZSTD_BLOCKSIZE_MAX.
|
||||
*/
|
||||
#define ZSTD_DECOMPRESSION_MARGIN(originalSize, blockSize) ((size_t)( \
|
||||
ZSTD_FRAMEHEADERSIZE_MAX /* Frame header */ + \
|
||||
4 /* checksum */ + \
|
||||
((originalSize) == 0 ? 0 : 3 * (((originalSize) + (blockSize) - 1) / blockSize)) /* 3 bytes per block */ + \
|
||||
(blockSize) /* One block of margin */ \
|
||||
))
|
||||
|
||||
typedef enum {
|
||||
ZSTD_sf_noBlockDelimiters = 0, /* Representation of ZSTD_Sequence has no block delimiters, sequences only */
|
||||
ZSTD_sf_explicitBlockDelimiters = 1 /* Representation of ZSTD_Sequence contains explicit block delimiters */
|
||||
|
||||
Reference in New Issue
Block a user