ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix required decompression memory usage reported by -vv + --long

Browse Source 
The use of --long alters the window size internally in the underlying
library (lib/compress/zstd_compress.c:ZSTD_getCParamsFromCCtxParams),
which changes the memory required for decompression. This means that the
reported requirement from the zstd binary when -vv is specified is
incorrect.

A full fix for this would be to add an API call to be able to retrieve
the required decompression memory from the library, but as a
lighterweight fix we can just take account of the fact we've enabled
long mode and update our verbose output appropriately.

Fixes #2968
This commit is contained in:
Jonathan McDowell
2022-02-02 12:39:27 +00:00
parent 4b24ebdcf3
commit 470eb8330a
1 changed files with 7 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
programs/fileio.c
+7 -2
View File
@@ -1301,8 +1301,13 @@ FIO_compressZstdFrame(FIO_ctx_t* const fCtx,
UTIL_HumanReadableSize_t windowSize; UTIL_HumanReadableSize_t windowSize;
CHECK(ZSTD_CCtx_getParameter(ress.cctx, ZSTD_c_windowLog, &windowLog)); CHECK(ZSTD_CCtx_getParameter(ress.cctx, ZSTD_c_windowLog, &windowLog));
if (windowLog == 0) { if (windowLog == 0) {
const ZSTD_compressionParameters cParams = ZSTD_getCParams(compressionLevel, fileSize, 0); if (prefs->ldmFlag) {
windowLog = cParams.windowLog; /* If long mode is set without a window size libzstd will set this size internally */
windowLog = ZSTD_WINDOWLOG_LIMIT_DEFAULT;
} else {
const ZSTD_compressionParameters cParams = ZSTD_getCParams(compressionLevel, fileSize, 0);
windowLog = cParams.windowLog;
}
} }
windowSize = UTIL_makeHumanReadableSize(MAX(1ULL, MIN(1ULL << windowLog, pledgedSrcSize))); windowSize = UTIL_makeHumanReadableSize(MAX(1ULL, MIN(1ULL << windowLog, pledgedSrcSize)));
DISPLAYLEVEL(4, "Decompression will require %.*f%s of memory\n", windowSize.precision, windowSize.value, windowSize.suffix); DISPLAYLEVEL(4, "Decompression will require %.*f%s of memory\n", windowSize.precision, windowSize.value, windowSize.suffix);