`make all` exercises substantially more than the default zstd binary. It also
builds compression-only and decompression-only archives, older contrib tools
that compile program C shims directly, and the single-file amalgamations. The
Rust migration had changed symbol ownership without updating every one of
those feature and link boundaries.
The first failure came from `fileio_asyncio`: it is always compiled, but its
codec bindings unconditionally referenced both `zstd_compress` and
`zstd_decompress`. A compression-only archive therefore required disabled
decoder modules, and the decompression-only configuration had the symmetric
problem. Gate the concrete codec imports, callback types, helpers, and exports
with their Cargo features. Keep the format probe compilable without the
legacy decoder predicate when decompression is disabled.
Once that boundary compiled, the remaining `make all` paths exposed related
integration gaps. Move the C decompression projection declarations out of the
compression preprocessor block, while leaving destination callback types
shared. Link the Rust CLI helpers archive into zlibWrapper, pzstd, and
largeNbDicts, whose util/time/data-generator C files are now declaration shims
rather than implementations.
The generated single-file C sources also call Rust-owned symbols now. Build
and link an appropriately featured Rust archive in their native smoke tests,
and include the pool configuration shim in the decoder case. The full
amalgamation combines `zstd_lazy.c` and `zstd_opt.c` into one translation unit,
so guard their otherwise translation-unit-local dictionary mode enum against
duplicate definition.
A Rust-backed amalgamation is no longer a standalone Emscripten input. Remove
the obsolete emcc/Docker path and report that limitation explicitly; restoring
the WebAssembly smoke test requires a Rust WebAssembly archive and a defined
cross-language amalgamation contract.
Test Plan:
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features compression` -- passed
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features decompression` -- passed
- `sh -n build/single_file_libs/build_decoder_test.sh build/single_file_libs/build_library_test.sh` -- passed
- `make all` -- passed, including native single-file and seekable-format tests
- `git diff --cached --check` -- passed
Parameter-bound lookup already lived in Rust, but the internal C helper still
performed the error and interval comparison locally. Call the existing Rust
bounds predicate directly so the compression parameter policy has one owner
while preserving the C helper and its callers.
Test Plan:
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- Previous capped cargo check and clippy passed for the unchanged Rust API.
The block-compressor dispatch already selected its strategy and row-mode index
in Rust, but C still classified the match state into no-dictionary,
ext-dictionary, attached-dictionary, or dedicated-dictionary mode. Project the
window and match-state scalars into Rust so that four-way policy is centralized
with compressor dispatch while C retains private match-state access.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
The window preparation callbacks already delegated dictionary-distance policy to
Rust, but C still copied projected limits and cleared loaded-dictionary and
match-state pointers. Add a checked pointer projection so Rust applies both
window-limit and dictionary-validity side effects; C retains private window
layout, pointer-index arithmetic, assertions, and diagnostics.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
The MT flush path already delegated checksum serialization to Rust but still
computed XXH64_digest in C. Pass the live C-owned XXH64 state through the
checked projection and let Rust finalize the digest before the existing
little-endian write. C retains serial state storage and job/flush mutation.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- Focused cargo test compiles but remains link-blocked by existing C-owned decompression-view symbols.
The MT serial-turn policy already ran in Rust but still crossed back into C
for the XXH64 update. Pass the live C-owned checksum state through the explicit
ABI and let Rust perform the update after the existing LDM and turn gate. C
retains the private serial state, synchronization, and raw sequence storage.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- Added a Rust checksum-vector test; standalone cargo test remains link-blocked by existing C-owned decompression-view symbols.
Keep the C-owned trace slot behind an explicit pointer projection so the
Rust trace boundary owns the complete enabled-call lifecycle. Rust now reads
the context, publishes the record, and clears the slot after the callback;
C only supplies private CCtx fields and the weak callback adapter.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- Focused trace lifecycle test added; standalone cargo test remains link-blocked by the existing C-owned decompression view symbols.
Move the first-job versus continuation-job trace policy out of the C worker
callback. Rust now decides when the external-dictionary assertion is required
and orders it before the trace, while C retains the private window assertion
and trace-recording leaves.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
Move LDM loadedDictEnd and final nextToUpdate publication out of the C
dictionary-content callbacks. Rust now writes through compact projections with
live window bases, while C retains the configuration-sensitive window,
matchfinder, and LDM operations.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
Replace the dictionary loader's C match-state publication callback with a live
field projection. Rust now publishes nextToUpdate, loadedDictEnd, and
forceNonContiguous at the existing orchestration point while C retains the
private match-state layout. Keep the window base indirect so a preceding
window update cannot leave Rust with a stale offset origin.
Test Plan:
- git diff --cached --check
- capped cargo check --tests
- capped cargo clippy --tests -- -A clippy::manual-bits -D warnings
- capped make -j1
- capped make -j1 -C tests test
Replace the C worker finish callbacks that publish errors, final block sizes,
consumed bytes, and completion signals with a Rust-owned publication path. The
nested projection keeps the job counters and pthread operations private to C,
while Rust preserves the error/success ordering and size_t wrapping behavior.
The consumed-size assertion remains a C validation callback at the original
point before publication.
Test Plan:
- git diff --cached --check
- capped cargo check --tests
- capped cargo clippy --tests -- -A clippy::manual-bits -D warnings
- capped make -j1
- capped make -j1 -C tests test
- standalone hybrid cargo test remains unavailable because its link lacks the
pre-existing ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and
ZSTD_rust_block_context_init symbols
Move the intermediate worker progress state update out of the C callback and
into a Rust-owned publication leaf. The projection keeps the private job
descriptor and pthread objects in C while preserving the lock, counter update,
diagnostic, signal, and unlock order, including size_t wrapping behavior.
Test Plan:
- git diff --cached --check
- capped cargo check --tests
- capped cargo clippy --tests -- -A clippy::manual-bits -D warnings
- capped make -j1
- capped make -j1 -C tests test
- standalone capped cargo test was attempted but its hybrid link lacks the
pre-existing ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and
ZSTD_rust_block_context_init symbols
Replace the dictionary-content Double-Fast callback with a live-field
projection and call the existing Rust table-fill leaf directly. The
projection keeps hash tables, window base, match-state index, and parameters
live through the window/publication callbacks, preserving the timing behavior
of the former C adapter. Preserve the excluded-DFast assertion path and the
34-word dictionary bridge layout.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -A clippy::manual-bits -D warnings
- ulimit -v 41943040; make -j1
- capped strategy-2 compress/decompress round trip with cmp
- Full original suite deferred to the next serialized verification step
The local-dictionary callback is now a C adapter around Rust-owned assignment
policy. Add a small allocator probe and focused tests for by-reference loading,
by-copy data publication, static-CCtx rejection, and allocation failure. The
probe observes the output slots during allocation so the test also protects
the original allocation-before-publication ordering. Strengthen the C and Rust
repr(C) checks with an explicit custom-allocator function-pointer size check.
The capped C-integrated build and original test suite remain deferred to the
main worker, which must run them serially under the repository's 40 GiB virtual
memory limit.
Test Plan:
- `git diff --cached --check` -- passed.
- `rustfmt --check rust/src/zstd_compress_dictionary.rs` -- passed.
- Capped cargo, make, native, fuzzer, and full-suite verification -- not run
by design; deferred to the main worker.
The Fast dictionary-table projection was populated before the Rust
orchestrator ran its window-update and match-state publication callbacks.
That left base and nextToUpdate pointing at the initial sentinel for MT
prefix jobs, which caused ZSTD_rust_fillHashTable to dereference an invalid
range. Project pointers to the live match-state fields so Rust reads the
same values the former C adapter would have read at the Fast branch.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; make -j1
- capped 4 MiB -T0 compress/decompress round trip with cmp
The MT worker previously kept the external-sequence non-empty gate and LDM
invariant assertion in C, bundled with the private CCtx operation. Project the
CCtx pointer and raw-sequence store through a layout-checked seam so Rust owns
the gate and the apply point while C retains only the opaque
ZSTD_referenceExternalSequences() leaf. Preserve the callback ordering after
worker CCtx initialization and keep empty stores as a no-op.
Test Plan:
- git diff --cached --check -- passed before commit.
- Static rg inspection confirmed all projection initializers include the new
sequence-state pointer and no old serialState_applySequences symbol remains.
- Cargo, make, compiler checks, and full tests were not run per worker scope.
The dictionary-content orchestrator still routed the Fast hash-table fill
through a C callback even though the matching leaf already lived in Rust.
Replace that callback slot with a pointer-sized C/Rust projection containing
only the Fast leaf inputs, including a by-value copy of nextToUpdate and the
normalized CDict/full-load flags. Keep the dtlm/tfp invariant assertion in C
and retain all bridge offsets and the double-Fast callback.
A broader MatchState projection would expose private layout and adding fields
would change the established 34-word bridge. The narrow projection keeps
that ABI stable while making the Fast branch call ZSTD_rust_fillHashTable
directly.
Test Plan:
- `git diff --cached --check`
- `rustfmt --check --edition 2021 rust/src/zstd_compress_dictionary.rs` (reports only pre-existing formatting drift outside this change)
- Cargo, make, and full tests intentionally not run per task instructions.
The MT serial turn already belongs to Rust for turn selection, checksum
placement, and advance. The remaining LDM callback still bundled the private
window update, sequence generation, and ldmWindow publication behind one C
operation, which hid that ordering from Rust.
Expose those three private operations as separate C callbacks and invoke them
from Rust in the original order. The first callback retains the original
RawSeqStore and job-size assertions before updating the LDM window; sequence
generation keeps its error assertion; and publication keeps the
ldmWindowMutex lock, copy, signal, and unlock sequence. The serial mutex
remains held until the existing advance callback, and disabled-LDM, empty
input, and skipped-turn behavior remain unchanged.
Test Plan:
- `git diff --cached --check` -- passed.
- Cargo, make, and full tests were not run per request; run the capped serial
verification suite before relying on runtime coverage.
Keep weak-symbol discovery and private dctx/ddict extraction in the C adapter,
but project the exact trace callback types into Rust and invoke them from the
Rust trace lifecycle. Preserve the nullable-hook behavior, trace context
width, dictionary-cold timing, record field layout, size conversions, and the
no-trace build path. Add focused Rust policy tests for callback, record, and
null-projection behavior.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1 -C tests test
The row-matchfinder reset path only used its C callback to read the private
hashSaltEntropy scalar and invoke Rust. Replace that callback slot with the
existing 32-bit scalar, keep the ABI offsets pointer-sized on 32-bit and
64-bit targets, and update the salt directly in Rust at the row-table reset
point. Preserve the C workspace callbacks and verify the real salt transition
in the reset-policy test.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1 -C tests test
The CCtx reset tail already delegated storage reservation and ordering policy
to Rust, but crossed back into C through a redundant callback trampoline.
Project the storage state directly through the existing pointer-sized ABI slot
and invoke the Rust storage operation without the extra C hop. Keep private
workspace/layout callbacks and match-state reset in C, preserve hash -> match
-> storage ordering, and retain allocation-error propagation in Rust tests.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
Keep the compression context extraction and the weak trace hook adapter in C,
where the private CCtx layout and platform-specific tracing declarations remain
available. Project only the scalar trace inputs and callback through an explicit
ABI-checked boundary so Rust owns the enabled check, stable record layout,
version field, compression-only null fields, and end-of-trace policy. The C
entry point still clears traceCtx after Rust returns, preserving the old
single-use lifecycle and keeping trace-disabled builds free of weak-hook calls.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; make -j1 -C tests test
Move buffered-input presence, offset subtraction, and block-size validation into
Rust while keeping the private MT progression query behind its C callback. The
C entry point now projects only the consumed/produced counters, input-buffer
fields, and worker dispatch inputs. Rust preserves the null-buffer behavior and
aborts on the same impossible offset or oversized-buffer invariants as the
original C assertions.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Move the serial LDM dictionary preload policy into Rust. The C MT adapter now
exposes only the private window and hash-table operations through checked
callbacks, while Rust owns empty/raw-content eligibility, loaded-dictionary
reset ordering, and the force-window publication rule. The explicit projection
keeps the dictionary inputs and callback ABI auditable without exposing the
private SerialState layout.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; make -j1 -C tests test
The CCtx reset orchestration already runs in Rust, but its C workspace
adapter still decided when capacity was too small or wasteful and maintained
the oversized-duration counter through C-only predicates. Move those factor,
threshold, overflow, and duration-update rules into Rust. C now supplies only
private workspace metrics and retains low-level allocation, reservation, and
layout operations.
Remove the obsolete C predicates so the workspace resize decision has one
Rust-owned implementation and remains covered at its boundary conditions.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; make -j1 -C tests test
The local-dictionary callback still contained the policy that selected
by-reference versus by-copy loading, rejected internal copies for static
CCtx instances, allocated and copied dictionary bytes, and published the
resulting local-dictionary state. Move that policy into the Rust compression
rewrite so the C callback is only an adapter around the private CCtx and
localDict layouts.
The adapter retains the C custom allocator and output slots, preserving the
existing ownership and free path while keeping the ABI boundary explicit.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -B -j1 -C tests fuzzer
- (cd tests && ulimit -v 41943040; ./fuzzer -v -i1 -s6572)
- ulimit -v 41943040; make -j1 -C tests test
The overflow-correction policy and scalar correction arithmetic were already
Rust-owned, but the stateful window rebase still lived in the C
ZSTD_window_correctOverflow() inline. That left pointer advancement, limit
clamping, and the correction counter as an untracked production C algorithm
on every compression path that protects matchfinder indices.
Replace that inline with a five-pointer projection. C continues to compute the
private source index and choose the correction context, while Rust now owns
the rebasing transition: it advances the regular and dictionary bases, clamps
both limits with the original wrapping U32 behavior, checks the correction
invariants in debug builds, and increments the overflow-correction counter.
The C/Rust projection has explicit layout assertions, and the Rust fixture
covers the pointer, limit, and counter updates together.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
Move the scalar portion of the private CCtx reset tail behind the existing
Rust reset adapter. The C side now publishes explicit pointers to the fields
that Rust must initialize, while retaining workspace ownership, match-state
reset, storage reset, and the XXH64 reset callback that cannot cross the
boundary safely. This removes the last opaque C initializer from this reset
phase without exposing the private ZSTD_CCtx layout to Rust.
The ABI record has compile-time offset and size assertions on both sides, and
the Rust unit fixture now checks the resolved compression parameters,
prefetch policy, pledged-size sentinel handling, counters, content-size flag,
block-size limit, stage, and dictionary metadata. The callback-order tests
continue to verify that compressed-block state reset precedes match-state and
storage cleanup, and that failures stop the tail at the correct callback.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
Remove the C-only DDict offset globals and route production
ZSTD_copyDDictParameters() through the existing C-produced DCtx view. This
keeps the private decoder layout and optional fuzzing fields owned by the
canonical projection while preserving the DDict entropy, prefix, window, and
checksum state updates. Keep the synthetic offset writer test-only so the
Rust unit tests remain independently linkable.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (207 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
- git diff --check
The MT stream initializer previously exposed one C callback that waited for
all submitted jobs and then released their resources. That left the ordering
policy in C even though Rust already owned the surrounding initialization
sequence and the ring-order wait/release policies.
Expose separate wait-all and release-all callbacks at the Rust/C boundary and
invoke them in Rust in the original wait-before-release order. C retains the
worker synchronization, job descriptors, input buffer, and resource-pool
side effects behind the two callbacks, while the focused Rust initializer test
now makes the ordering explicit.
Test Plan:
- capped cargo +nightly fmt for the Rust workspace -- passed
- capped root cargo clippy --all-targets -- -D warnings -- passed
- capped serial make -j1 for the single-threaded library, MT library, and CLI -- passed
- git diff --cached --check -- passed
Transparent stream initialization can resolve an unknown source size to the
multithreaded path and update appliedParams.nbWorkers. The Rust stream2
fallback previously captured that value before initialization, then entered the
serial generic adapter while the context stage was still created. Unknown-size
stdin compression consequently returned StageWrong instead of producing a
frame.
Keep the worker-mode projection live through initialization so Rust observes
the applied value at the existing MT-versus-serial decision point. The ABI
layout assertions now cover the pointer projection, and a focused regression
test verifies that an initialization callback which selects MT dispatches the
MT step and preserves the expected callback order.
Test Plan:
- capped cargo +nightly fmt for the Rust workspace -- passed
- capped root and CLI cargo clippy with -D warnings -- passed
- capped serial make -j1 for single-threaded library, MT library, and CLI -- passed without the prior pointer-sign warning
- capped stdin compression/decompression round trip for unknown-size input -- passed
- capped original suite, including CLI tests, native tests, fuzzer, and both zstream phases -- passed; the final signedness-only ABI spelling cleanup was followed by a clean capped rebuild
- targeted root cargo test attempted but remains blocked at link time by pre-existing decompression bridge symbols (ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and ZSTD_rust_block_context_init)
The public ZSTD_compressStream2_c entry point still contained the complete
fallback state machine even though its validation, initialization policy,
serial-versus-MT dispatch, and result accounting were already represented by
Rust policy helpers. That left the main streaming entry point as a large C
orchestration island and made the Rust rewrite boundary misleading.
Project the scalar stream state and private CCtx operations through a checked
C ABI, then let Rust own the fallback ordering. Rust now validates buffers and
the end directive, handles transparent one-shot completion and stable-input
deferral, performs the initialization and stability decisions, selects the
serial or MT path, and publishes the result policy. C retains the private
context layout, stream adapter, MT operations, checksum/epilogue side effects,
and trace/reset callbacks. ABI offset assertions and focused callback-order
and stable-input tests protect the projection without requiring a standalone
Rust test binary to link every C bridge symbol.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed before and after formatting.
- `git diff --cached --check` -- passed.
- Native `make -j1` and the original suite remain the next post-commit verification gates.
Project the private requested-parameter and context state into a fixed scalar ABI record, leaving only C's parameter-selection leaf and private layout access in the shim. Rust now owns the complete-input eligibility predicate, including advanced-state exclusions, LDM sentinel handling, and the final fast/double-fast strategy gate, while preserving the requested-level and INT_MIN results.
Add focused policy tests for fast and double-fast acceptance, representative advanced-state rejection, the accepted LDM sentinel, non-fast strategies, and null state.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
Move the multithreaded compressStream2 outer coordinator into the Rust compression module. The C bridge now owns only the private ZSTDMT call and consumed/produced counters, while Rust preserves the progress loop, completion ordering, and error handoff.
Keep the MT context mutations, diagnostics, assertions, and buffer-expectation publication in C because they depend on the private context layout. Add callback-driven tests for continue/break, error completion, end trace/reset ordering, and malformed states.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
The public CCtx and CStream size-estimation APIs still selected row-matchfinder modes and walked compression levels in C, leaving a policy-level implementation behind the Rust workspace-sizing bridge. That also made the C layer responsible for the subtle raw-size_t MAX behavior used when an estimator returns an encoded error.
Move the row-mode selection and shared monotonic memory-budget policy into Rust. Keep the C callbacks responsible for constructing private ZSTD_CCtx_params values and for the distinct CCtx/CStream sizing formulas, so no private parameter layout crosses the ABI. Preserve the signed MIN(compressionLevel, 1) start and raw maxima exactly.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
The MT serial-reset bridge now declares its Rust scalar projection before
executing the diagnostic branch. This keeps the declaration ordering valid for
the C90-oriented compiler flags used by the native test builds while leaving
the Rust-owned policy and callback sequence unchanged.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1` -- passed without the
new mixed-declaration warning.
- `git diff --cached --check` -- passed.
The multithreaded stream loop already delegated scalar result classification to
Rust, but its final error/end conditional still mixed completion ordering with
the private C context. That left trace and reset sequencing embedded beside the
MT counters and made the next Rust seam depend on exposing CCtx layout.
Rust now consumes the projected loop action and sequences the final completion
side effects through opaque C callbacks. Continue and break do nothing, worker
errors reset the session, and completed end operations trace before resetting.
The reset callback result is intentionally discarded. C retains the private
context mutations in callback adapters, and the callback invocation remains
inside the existing ZSTD_MULTITHREAD branch before FORWARD_IF_ERROR(flushMin).
C/Rust repr(C) projections and ABI assertions protect the callback/result
boundary, while focused tests cover all four loop actions and end ordering.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_compress.rs` -- passed
- single-thread and `-DZSTD_MULTITHREAD` C syntax checks -- passed; existing warnings only
- `CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed
- `CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --benches -- -D warnings` -- passed
- focused Rust test filter was attempted serially but is blocked by unrelated in-progress `rust/src/zstdmt_compress.rs` test ABI edits; no full test suites were run
ZSTD_compressBegin_internal already routes attach-versus-reload policy through
Rust, but its C insertion callback still inspected a non-null ZSTD_CDict and
selected dictContent, dictContentSize, and dictContentType. That left source
selection coupled to the private CDict layout and kept the Rust bridge from
receiving the same selected source for direct and prepared dictionaries.
Project the three CDict content fields into the repr(C) begin state. Rust now
selects either those fields or the direct dictionary arguments before invoking
the C callback. The callback keeps the private CCtx insertion path and no
longer branches on CDict. ABI offsets and sizes are asserted on both sides,
with focused by-reference and forced-CDict tests covering pointer, size, type,
reset order, and dictionary-result publication.
Test Plan:
- `cargo check --manifest-path rust/Cargo.toml --lib --tests` -- passed under
`ulimit -v 41943040; CARGO_BUILD_JOBS=1`.
- Serial GCC/Clang syntax-only checks for `lib/compress/zstd_compress.c` --
passed under the same cap; only existing warnings were reported.
- `cargo clippy` for lib, benches, and tests with `-D warnings`, plus nightly
format check -- passed serially under the same cap.
- `cargo test --manifest-path rust/Cargo.toml --lib zstd_compress_dictionary
-- --test-threads=1` -- compiled but standalone linking failed on the three
pre-existing C bridge symbols `ZSTD_rust_dctx_trace_view`,
`ZSTD_rust_dctx_view`, and `ZSTD_rust_block_context_init`.
- Full native, upstream, and fuzzer tests were not run per scope.
Add the existing heap-mode bridge declaration beside the other decompression shim prototypes so the new Rust policy seam remains warning-clean under the native C build.
Test Plan:
- git diff --cached --check
- capped full test suite passed before this warning-only fix; final capped native rebuild follows
Route the configured default maximum window size through the Rust policy layer while keeping the build-time C configuration value and decoder layout in C. The explicit projection preserves overridden builds exactly and gives the scalar policy a focused null-input contract and tests.
Test Plan:
- git diff --check -- lib/decompress/zstd_decompress.c rust/src/zstd_decompress.rs
- capped serial cargo check, clippy, nightly fmt, C syntax checks, and focused default-window tests (passed in the worker)
CDict initialization still hard-coded the match-state reset policies in its
C callback, even though Rust already owned the surrounding content branch,
workspace reservation, and callback ordering. That left the reset target and
cleanup policy split across the language boundary and made the intended CDict
reset contract implicit.
Extend the existing repr(C) reset callback with the three private enum values.
Rust now selects make-clean, index-reset, and the CDict reset target before
calling the opaque C operation. C retains the private ZSTD_CDict, workspace,
and match-state layouts and only casts the projected values for
ZSTD_reset_matchState(). Focused by-reference and by-copy probes record the
values and preserve reserve, reset, and insert ordering.
Test Plan:
- `rustfmt --check --edition 2021 rust/src/zstd_compress_dictionary.rs` -- passed.
- GCC and Clang syntax-only checks for `lib/compress/zstd_compress.c` under
`ulimit -v 41943040` -- passed.
- `cargo check --manifest-path rust/Cargo.toml --lib --tests` under the cap -- passed.
- Serial `cargo clippy` lib, benches, and tests passes with `-D warnings`,
plus `cargo +nightly fmt --all -- --check` -- passed.
- `cargo test --manifest-path rust/Cargo.toml --lib zstd_compress_dictionary
-- --test-threads=1` compiled but could not link the standalone test binary
because three pre-existing C bridge symbols are unavailable outside the
native harness; no native, upstream, or fuzzer tests were run.
Move the Rust teardown projection declaration before the null guard so the MT source remains clean under the repository's C90 declaration rules. The initializer only copies pointers, scalars, the allocator, and the callback; the null guard still precedes the bridge call and preserves the existing no-op behavior for an absent job table.
Test Plan:
- git diff --cached --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test (passed before this warning-only fix)
Normal MT context teardown still let the C wrapper directly sequence
per-job synchronization destruction before Rust released the opaque job-table
storage, while failed create and expansion transactions already used a Rust
helper for that same order. Route the normal teardown through a repr(C)
projection so Rust owns the destroy-before-storage-free policy consistently.
The synchronization callback, descriptor storage, and custom allocator remain
opaque C-owned operations. Add matching C/Rust layout assertions and a focused
callback-order test.
Test Plan:
- `git diff --cached --check` -- passed
- Cargo, make, native tests, and heavy verification were not run per request
ZSTD_compress_advanced_internal() still encoded the one-shot begin-then-end
sequence in C after public advanced parameter validation had moved behind Rust.
That left the public compression boundary split between Rust policy and a C
orchestration body, and made the begin-error ordering implicit in the C path.
Add an explicit C-layout state with opaque begin and end callbacks. Rust now
owns the ordering, forwards the source size as the pledged size, propagates a
begin error before invoking the end callback, and returns the end result. The
callbacks retain the private ZSTD_CCtx and ZSTD_CCtx_params layouts, so the
change moves policy and sequencing without exposing window, workspace,
matchfinder, or context internals to Rust. The existing advanced and
using-dictionary wrappers continue to use the same C private operations.
Test Plan:
- `git diff --cached --check` -- passed.
- `rustfmt --check --edition 2021 rust/src/zstd_compress.rs` -- passed.
- Not run: Cargo, make, native tests, builds, or heavy verification per request.
`ZSTD_HEAPMODE` was still returned directly from C, and the stack entry
projection also copied the raw macro. That left the one-shot stack-versus-heap
classification in the C configuration shim even though Rust owns the public
decompression branch and stack action.
Keep the build-time override in C, but pass it through a one-field repr(C)
projection to Rust. Rust normalizes values below one to the stack mode and
values at or above one to the heap mode, preserving the existing behavior
while making the branch policy explicit. The C stack object, private DCtx
layout, platform initialization, and trace boundary remain C-owned; moving
those would either duplicate build configuration or cross the requested ABI
boundary. The stack projection now uses the same normalized result as the
one-shot branch.
ABI layout assertions cover the scalar projection, and focused Rust tests cover
negative, zero, positive, maximum, and missing-configuration inputs.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_decompress.rs` -- passed
- Cargo, Make, native tests, and heavy commands intentionally not run per the
task restriction; integration verification remains pending.
Keep the MT frame-progression callback under the ZSTD_MULTITHREAD preprocessor guard using an ordinary C declaration. The previous warning-cleanup edit accidentally prefixed the declaration with a preprocessor marker, which broke the multithreaded compilation path.
Test Plan:
- git diff --cached --check
- Capped native rebuild failed before this fix at the invalid directive; rerun pending
The MT frame-progression callback is only referenced when ZSTD_MULTITHREAD is enabled. Keep its definition under the same configuration guard so single-threaded library, test, and decode-corpus builds do not report an unused static function while the MT callback remains available to the Rust dispatch bridge.
Test Plan:
- git diff --cached --check
- Capped full tests passed before this warning-only guard
- Capped native rebuild and smoke rerun pending
The Rust-owned frame-progression dispatch invokes the existing C MT progression API through a mutable opaque callback context. Match that API's established signature in the adapter so the new boundary does not introduce a discarded-const qualifier warning while preserving the private MT context.
Test Plan:
- git diff --cached --check
- Capped native make rerun pending after this warning-only fix
ddidderr