[zstdmt] Fix determinism issue with rsyncable mode
The problem occurs in this scenario: 1. We find a synchronization point. 2. We attmept to create the job. 3. We fail because the job table is full: `mtctx->nextJobID > mtctx->doneJobID + mtctx->jobIDMask`. 4. We call `ZSTDMT_compressStream_generic` again. 5. We forget that we're at a sync point already, and we continue looking for the next sync point. This fix is to detect if we're currently paused at a sync point, and if we are then don't load any more input. Caught by zstreamtest. I modified it to make the bug occur more often (~1/100K -> ~1/200) and verified that it is fixed after. I then ran a few hundred thousand unmodified zstreamtest iterations to verify.
This commit is contained in:
@@ -1687,6 +1687,16 @@ findSynchronizationPoint(ZSTDMT_CCtx const* mtctx, ZSTD_inBuffer const input)
|
|||||||
pos = 0;
|
pos = 0;
|
||||||
prev = (BYTE const*)mtctx->inBuff.buffer.start + mtctx->inBuff.filled - RSYNC_LENGTH;
|
prev = (BYTE const*)mtctx->inBuff.buffer.start + mtctx->inBuff.filled - RSYNC_LENGTH;
|
||||||
hash = ZSTD_rollingHash_compute(prev, RSYNC_LENGTH);
|
hash = ZSTD_rollingHash_compute(prev, RSYNC_LENGTH);
|
||||||
|
if ((hash & hitMask) == hitMask) {
|
||||||
|
/* We're already at a sync point so don't load any more until
|
||||||
|
* we're able to flush this sync point.
|
||||||
|
* This likely happened because the job table was full so we
|
||||||
|
* couldn't add our job.
|
||||||
|
*/
|
||||||
|
syncPoint.toLoad = 0;
|
||||||
|
syncPoint.flush = 1;
|
||||||
|
return syncPoint;
|
||||||
|
}
|
||||||
} else {
|
} else {
|
||||||
/* We don't have enough bytes buffered to initialize the hash, but
|
/* We don't have enough bytes buffered to initialize the hash, but
|
||||||
* we know we have at least RSYNC_LENGTH bytes total.
|
* we know we have at least RSYNC_LENGTH bytes total.
|
||||||
|
|||||||
Reference in New Issue
Block a user