[lib] Fix dictionary invalidation logic
Call `ZSTD_enforceMaxDist()` before each block with the beginning of the block. This ensures that `lowLimit` is updated to `dictLimit` whenever the ext-dict is out of range, so we can use prefix mode for speed. This can cause non-determinism because prefix mode and ext-dict mode match finders can return different results. It can also hurt speed because ext-dict match finders are slower. The scenario is: 1. Compress large data with a dictionary. 2. The dictionary goes out of bounds, so we invalidate it. 3. However, we still have `lowLimit < dictLimit`, since it is never updated. 4. We will call the ext-dict match finder instead of the prefix one.
This commit is contained in:
@@ -3915,6 +3915,7 @@ static size_t ZSTD_compress_frameChunk(ZSTD_CCtx* cctx,
|
||||
ZSTD_overflowCorrectIfNeeded(
|
||||
ms, &cctx->workspace, &cctx->appliedParams, ip, ip + blockSize);
|
||||
ZSTD_checkDictValidity(&ms->window, ip + blockSize, maxDist, &ms->loadedDictEnd, &ms->dictMatchState);
|
||||
ZSTD_window_enforceMaxDist(&ms->window, ip, maxDist, &ms->loadedDictEnd, &ms->dictMatchState);
|
||||
|
||||
/* Ensure hash/chain table insertion resumes no sooner than lowlimit */
|
||||
if (ms->nextToUpdate < ms->window.lowLimit) ms->nextToUpdate = ms->window.lowLimit;
|
||||
|
||||
Reference in New Issue
Block a user