Keep the private ZSTD_window_t layout and pointer-difference calculation in C,
then route the size_t-to-U32 conversion and paired limit writes through a small
Rust ABI leaf. This removes the inline C window-clear implementation while
preserving its overflow behavior at both single-threaded and multithreaded
call sites.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib zstd_compress::tests (77 passed)
- make -C programs -j2 zstd
- rustfmt +nightly --edition 2021 rust/src/zstd_compress.rs --check
- make -C tests -j2 test-cli-tests (41 passed)
- make -C tests -j2 test-legacy test-invalidDictionaries test-decodecorpus test-rust-lib-smoke (passed)
- git diff --check
ZSTDMT_expandJobsTable previously kept worker-capacity comparison, old-table
synchronization teardown, replacement allocation, initialization failure cleanup,
and mask updates in C. That left the remaining job-table lifecycle orchestration
outside the Rust storage leaves. Add a narrow Rust ABI that exchanges only
opaque storage and scalar mask state and accepts C callbacks for the private
synchronization lifecycle.
The adapter frees the old table before replacement, reuses the existing Rust
create/free leaves with the caller's custom memory, destroys a partially
initialized replacement before freeing it, and updates jobIDMask only after
successful initialization. C still owns ZSTDMT_CCtx and job descriptors, so
worker-facing fields and platform synchronization stay outside the Rust ABI.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features
compression zstdmt_compress --lib` -- 38 passed.
- `cargo clippy`, `cargo clippy --benches`, and `cargo clippy --tests` -- passed.
- `cargo +nightly fmt --all -- --check` -- passed.
- `make -B -C lib lib-mt` -- passed.
- `make -B -C tests -j2 fullbench poolTests` -- passed.
- `./poolTests` and `./fullbench -i1 -B1000 ../README.md` -- passed.
Move the arithmetic aggregation in ZSTDMT_getFrameProgression across the
existing C/Rust boundary. C still owns MT state traversal, mutex locking,
jobIDMask ordering, nextJobID plus jobReady handling, error normalization,
and the flushed <= produced assertion. Rust constructs the base result and
folds each compact, C-normalized job snapshot into the complete repr(C)
ZSTD_frameProgression, preserving size_t-to-U64 wrapping and active-worker
counting without exposing private job or mutex layouts.
Focused Rust tests cover zero jobs, a ready job, completed jobs, normalized
error output, and mixed active/completed aggregation.
Test Plan:
- All three compression clippy commands passed before formatting.
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- passed.
- All three compression clippy commands passed after formatting.
- cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression -- 332 passed.
- make -B -C lib -j2 lib-mt -- passed.
- make -B -C lib -j2 lib-nomt -- passed.
- make -B -C tests -j2 test-zstream -- passed: 84 named, 6,004, and 8,392
randomized cases.
- git diff --check and git diff --cached --check -- passed.
The zstream build retained the pre-existing warning at
tests/zstreamtest.c:1899 about an unterminated initializer string.
The MT rsync initialization previously computed the rolling-hash prime
power through the C-only ZSTD_rollingHash_primePower helper. Move only that
scalar exponentiation behind the existing Rust MT ABI boundary. The C caller
still owns the MT context assignment, RSYNC_LENGTH constant, and surrounding
initialization.
The Rust helper mirrors ZSTD_ipow with wrapping u64 multiplication and applies
u32 wrapping subtraction before widening the exponent, preserving the C
length == 0 behavior. Rolling-hash scanning, rotation, and all other MT state
remain unchanged.
Test Plan:
- Focused rolling_hash_prime_power Rust tests -- passed (2 tests).
- Required clippy/fmt sequence in normal, bench, and test modes -- passed.
- make -B -C lib -j2 lib -- passed.
- make -C tests test-rust-lib-smoke -- passed.
- tests/fuzzer -s4560 -t56 -i57 -v -- passed (57 tests).
- make -C tests -j2 test-zstream -- passed (84 named, 5,305 + 7,351 randomized).
- git diff --check and git diff --cached --check -- passed.
Keep buffer-pool ownership, allocation, synchronization, and the Rust
pool's saturating accounting in their existing implementations. The C
wrapper still handles NULL, obtains sizeof(*bufPool), and queries the Rust
pool size; Rust now performs only the final size_t addition with wrapping
semantics. The sequence-pool sizing alias continues to call the same C
helper and therefore retains its behavior.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features
--features compression` (before and after formatting) -- passed
- The same clippy command with `--benches` and `--tests` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml` -- passed
- `cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression zstdmt_compress` -- 29 passed
- `make -B -C lib -j2 lib` -- passed
- `make -C tests test-rust-lib-smoke` -- passed
- `tests/fuzzer -s4560 -t56 -i57 -v` -- 57 passed
- `make -C tests -j2 test-zstream` -- 84 named, 7,661, and 8,090
randomized tests passed
- `git diff --check` and `git diff --cached --check` -- passed
The zstream build still emits the pre-existing unterminated-string warning
at tests/zstreamtest.c:1899.
ZSTDMT_sizeof_CCtx previously performed the complete size_t sum in C,
although every component depends on C-owned MT state and sizing helpers.
Keep the NULL guard, pool and dictionary queries, job-table multiplication,
and round-buffer extraction in C. Pass those eight computed components to a
Rust ABI helper that performs the same ordered wrapping additions.
This keeps MT context layout and allocator ownership on the C side while
moving only the scalar arithmetic across the existing C/Rust boundary.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression`
- Same clippy command with `--benches` and `--tests`, nightly fmt, then all three clippy commands again -- passed.
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression` -- 289 passed.
- `make -B -C lib -j2 lib` -- passed.
- `make -C tests test-rust-lib-smoke` -- passed.
- `make -B -C tests -j2 fuzzer` and `tests/fuzzer -s4560 -t56 -i57 -v` -- passed.
- `make -C tests -j2 test-zstream` -- 84 named, 6,268, and 8,862 randomized cases passed.
- `git diff --check` and `git diff --cached --check` -- passed.
The zstream build retains its pre-existing unterminated-string warning at
`tests/zstreamtest.c:1899`.
ZSTDMT_nextInputSizeHint only derives a size_t hint from targetSectionSize
and inBuff.filled. Keep the MT context, public/internal entry point, and
stream orchestration in C, but delegate those two scalars to Rust. The Rust
helper uses wrapping subtraction and preserves the zero-result fallback,
including C's wrapped behavior for invalid overfill.
Test Plan:
- The three compression clippy modes passed before and after
`cargo +nightly fmt --manifest-path rust/Cargo.toml`.
- Focused Rust tests passed: 5 `mt_next_input_size_hint` tests.
- `make -B -C lib -j2 lib` passed.
- `make -C tests test-rust-lib-smoke` passed.
- `tests/fuzzer -s4560 -t56 -i57 -v` passed: 57 tests completed.
- `make -C tests -j2 test-zstream` passed: 84 named, 6,659, and 8,055
randomized tests completed.
- `git diff --check` and `git diff --cached --check` passed.
Move construction of the external-dictionary and active-prefix ranges into a narrow Rust ABI while retaining the C window and logging surface. Preserve byte-range half-open overlap semantics and leave the input-range overlap wrapper available to its other C caller.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (180 passed); root and CLI clippy; make -B -C lib -j2 lib; make -C tests -j2 test-cli-tests (41 passed); make -B -C tests -j2 test-zstream (84 named tests plus 6,845 and 9,628 fuzz cases passed).
Move sequence-store byte accounting, chunk derivation, and repcode resolution into the Rust compression module. Move MT raw-sequence buffer conversions into Rust while preserving the existing C adapters and ABI layouts.
Test Plan:\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- make -B -C lib -j2 lib\n- make -B -C tests -j2 test-cli-tests
The multithreaded compressor already uses Rust-owned buffer and CCtx pools,
but zstdmt_compress.c still allocated and freed its job table directly. Move
the raw table storage and power-of-two sizing behind Rust's custom allocator
ABI. Keep descriptor field access and platform mutex/condition initialization
in C because those layouts remain private and platform-specific. The C
wrapper initializes and destroys synchronization primitives around the Rust
storage calls, preserving failure cleanup while leaving worker job setup,
scheduling, and stream entry points in C as the fallback implementation.
Keep the Rust storage-only ABI free of MT-only C references so single-threaded
archives can omit zstdmt_compress.c without acquiring new unresolved symbols.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression zstdmt_compress --lib` -- passed (5 tests).
- `make -B -C lib lib-mt` -- passed.
- `make -B -C tests -j2 fullbench zstreamtest poolTests` -- passed.
- `./poolTests` -- passed.
- `./fullbench -i1 -B1000 README.md` -- passed, including -T2 scenarios.
- Rebuilt `programs/zstd` and ran a `-T2` compress/decompress `cmp` round-trip
-- passed.
- `rustfmt` and `cargo +nightly fmt --manifest-path rust/Cargo.toml --all
-- --check` -- passed.
- Full `cargo clippy -D warnings` remains blocked by unrelated warnings in
the concurrent `rust/src/fileio_asyncio.rs` worktree changes.
- `./zstreamtest -T5s` reaches an unrelated single-thread maxBlockSize
assertion at `tests/zstreamtest.c:2157`; MT fullbench and CLI smoke pass.
Move the reusable buffer and compression-context pools behind a Rust
implementation with a narrow C ABI adapter. The scheduler, job table, serial
LDM state, and stream orchestration remain in C until their private layouts
are ported.
Keep context types opaque across Rust modules so the pool bridge does not
depend on private C layout declarations. This also keeps the existing custom
allocator and pool replacement contracts intact.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
rust/Cargo.toml --all-targets -- -D warnings
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
rust/Cargo.toml zstdmt_compress
- make -B -C lib lib-mt
- git diff --cached --check
This function was seriously flawed:
* It didn't do output bounds checks
* It produced invalid sequences when an uncompressed or RLE block was emitted
* It produced invalid sequences when the block splitter was enabled
* It produced invalid sequences when ZSTD_c_targetCBlockSize was enabled
I've attempted to fix these issues, but this function is just a bad idea,
so I've marked it as deprecated and unsafe. We should replace it with
`ZSTD_extractSequences()` which operates on a compressed frame.
This PR introduces no functional changes. It attempts to change all
macros currently using `{ }` or some variant of that to to
`do { } while (0)`, and introduces trailing `;` where necessary.
There were no bugs found during this migration.
The bug in Visual Studios warning on this has been fixed since VS2015.
Additionally, we have several instances of `do { } while (0)` which have
been present for several releases, so we don't have to worry about
breaking peoples builds.
Fixes Issue #3830.
within ZSTDMT_.
This pattern is flagged by less forgiving variants of ubsan
notably used during compilation of the Linux Kernel.
There are 2 other places in the code where this pattern is used.
This fixes just one of them.
* Mark all bufferless and block level functions as deprecated
* Update documentation to suggest not using these functions
* Add `_deprecated()` wrappers for functions that we use internally and
call those instead
* Fixes zstd-dll build (https://github.com/facebook/zstd/issues/3492):
- Adds pool.o and threading.o dependency to the zstd-dll target
- Moves custom allocation functions into header to avoid needing to add dependency on common.o
- Adds test target for zstd-dll
- Adds github workflow that buildis zstd-dll
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
This assert slows the loop down by 10x. We can get similar
coverage by asserting at the beginning & end of the loop.
We need this fix because Debian compiles zstd with asserts
enabled. Separately, we should ask them why, and if they would
consider disabling asserts in their builds. Since we don't
optimize for assert enabled builds.
Fixes Issue #3150.
Test failures showed up on the daily cron job. They didn't show up
in CI because the condition is somewhat rare, and didn't trigger
during the CI tests.
This PR fixes up the logic in `findSynchronizationPoint()` to correctly
handle the edge case. It also un-comments an assert that helps catch the
issue, and verify that rsyncable mode is calculating the correct hash.
After the fix, the test that failed passes:
```
./zstreamtest --newapi -t1 --no-big-tests -s9680
```
In degenerate cases `--rsyncable` could create very small blocks (1
byte). This causes the compressed output to be larger than
`ZSTD_compressBound()`. Fix the issue by ensuring that rsyncable mode
never outputs blocks smaller than 128 KB.
The minimum job size is 512 KB, so we shouldn't lose many
synchronization points from skipping any that cause blocks smaller than
128 KB. And even if we do, that is fine, because we'll find the next
one.
This fixes the `raw_dictionary_round_trip` oss-fuzz assert.
Credit to OSS-Fuzz
This flag forces zstd to always load the prefix in ext-dict mode, even
if it happens to be contiguous, to force determinism. It also applies to
dictionaries that are re-processed.
A determinism test case is also added, which fails without
`ZSTD_c_deterministicRefPrefix` and passes with it set.
Question: Should this be the default behavior? It isn't in this PR.
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
The most common information that you want to track between begin() and
end() is the timestamp of the begin function, so you can measure the
duration of the (de)compression call. Allow the tracing library to put
this information inside the `ZSTD_TraceCtx`, so it doesn't need to keep
a global map in this case. If a single uint64_t is not enough, the
tracing library can return a unique identifier (like the context
pointer) instead, and use it as a key in a map.
This keeps the simple case simple.
ddidderr
Yann Collet