`make all` exercises substantially more than the default zstd binary. It also
builds compression-only and decompression-only archives, older contrib tools
that compile program C shims directly, and the single-file amalgamations. The
Rust migration had changed symbol ownership without updating every one of
those feature and link boundaries.
The first failure came from `fileio_asyncio`: it is always compiled, but its
codec bindings unconditionally referenced both `zstd_compress` and
`zstd_decompress`. A compression-only archive therefore required disabled
decoder modules, and the decompression-only configuration had the symmetric
problem. Gate the concrete codec imports, callback types, helpers, and exports
with their Cargo features. Keep the format probe compilable without the
legacy decoder predicate when decompression is disabled.
Once that boundary compiled, the remaining `make all` paths exposed related
integration gaps. Move the C decompression projection declarations out of the
compression preprocessor block, while leaving destination callback types
shared. Link the Rust CLI helpers archive into zlibWrapper, pzstd, and
largeNbDicts, whose util/time/data-generator C files are now declaration shims
rather than implementations.
The generated single-file C sources also call Rust-owned symbols now. Build
and link an appropriately featured Rust archive in their native smoke tests,
and include the pool configuration shim in the decoder case. The full
amalgamation combines `zstd_lazy.c` and `zstd_opt.c` into one translation unit,
so guard their otherwise translation-unit-local dictionary mode enum against
duplicate definition.
A Rust-backed amalgamation is no longer a standalone Emscripten input. Remove
the obsolete emcc/Docker path and report that limitation explicitly; restoring
the WebAssembly smoke test requires a Rust WebAssembly archive and a defined
cross-language amalgamation contract.
Test Plan:
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features compression` -- passed
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features decompression` -- passed
- `sh -n build/single_file_libs/build_decoder_test.sh build/single_file_libs/build_library_test.sh` -- passed
- `make all` -- passed, including native single-file and seekable-format tests
- `git diff --cached --check` -- passed
Parameter-bound lookup already lived in Rust, but the internal C helper still
performed the error and interval comparison locally. Call the existing Rust
bounds predicate directly so the compression parameter policy has one owner
while preserving the C helper and its callers.
Test Plan:
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- Previous capped cargo check and clippy passed for the unchanged Rust API.
The block-compressor dispatch already selected its strategy and row-mode index
in Rust, but C still classified the match state into no-dictionary,
ext-dictionary, attached-dictionary, or dedicated-dictionary mode. Project the
window and match-state scalars into Rust so that four-way policy is centralized
with compressor dispatch while C retains private match-state access.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
The window preparation callbacks already delegated dictionary-distance policy to
Rust, but C still copied projected limits and cleared loaded-dictionary and
match-state pointers. Add a checked pointer projection so Rust applies both
window-limit and dictionary-validity side effects; C retains private window
layout, pointer-index arithmetic, assertions, and diagnostics.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
The MT flush path already delegated checksum serialization to Rust but still
computed XXH64_digest in C. Pass the live C-owned XXH64 state through the
checked projection and let Rust finalize the digest before the existing
little-endian write. C retains serial state storage and job/flush mutation.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- Focused cargo test compiles but remains link-blocked by existing C-owned decompression-view symbols.
The MT serial-turn policy already ran in Rust but still crossed back into C
for the XXH64 update. Pass the live C-owned checksum state through the explicit
ABI and let Rust perform the update after the existing LDM and turn gate. C
retains the private serial state, synchronization, and raw sequence storage.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- Added a Rust checksum-vector test; standalone cargo test remains link-blocked by existing C-owned decompression-view symbols.
Keep the C-owned trace slot behind an explicit pointer projection so the
Rust trace boundary owns the complete enabled-call lifecycle. Rust now reads
the context, publishes the record, and clears the slot after the callback;
C only supplies private CCtx fields and the weak callback adapter.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- Focused trace lifecycle test added; standalone cargo test remains link-blocked by the existing C-owned decompression view symbols.
Move the first-job versus continuation-job trace policy out of the C worker
callback. Rust now decides when the external-dictionary assertion is required
and orders it before the trace, while C retains the private window assertion
and trace-recording leaves.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
Move LDM loadedDictEnd and final nextToUpdate publication out of the C
dictionary-content callbacks. Rust now writes through compact projections with
live window bases, while C retains the configuration-sensitive window,
matchfinder, and LDM operations.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
Replace the dictionary loader's C match-state publication callback with a live
field projection. Rust now publishes nextToUpdate, loadedDictEnd, and
forceNonContiguous at the existing orchestration point while C retains the
private match-state layout. Keep the window base indirect so a preceding
window update cannot leave Rust with a stale offset origin.
Test Plan:
- git diff --cached --check
- capped cargo check --tests
- capped cargo clippy --tests -- -A clippy::manual-bits -D warnings
- capped make -j1
- capped make -j1 -C tests test
Replace the C worker finish callbacks that publish errors, final block sizes,
consumed bytes, and completion signals with a Rust-owned publication path. The
nested projection keeps the job counters and pthread operations private to C,
while Rust preserves the error/success ordering and size_t wrapping behavior.
The consumed-size assertion remains a C validation callback at the original
point before publication.
Test Plan:
- git diff --cached --check
- capped cargo check --tests
- capped cargo clippy --tests -- -A clippy::manual-bits -D warnings
- capped make -j1
- capped make -j1 -C tests test
- standalone hybrid cargo test remains unavailable because its link lacks the
pre-existing ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and
ZSTD_rust_block_context_init symbols
Move the intermediate worker progress state update out of the C callback and
into a Rust-owned publication leaf. The projection keeps the private job
descriptor and pthread objects in C while preserving the lock, counter update,
diagnostic, signal, and unlock order, including size_t wrapping behavior.
Test Plan:
- git diff --cached --check
- capped cargo check --tests
- capped cargo clippy --tests -- -A clippy::manual-bits -D warnings
- capped make -j1
- capped make -j1 -C tests test
- standalone capped cargo test was attempted but its hybrid link lacks the
pre-existing ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and
ZSTD_rust_block_context_init symbols
Replace the dictionary-content Double-Fast callback with a live-field
projection and call the existing Rust table-fill leaf directly. The
projection keeps hash tables, window base, match-state index, and parameters
live through the window/publication callbacks, preserving the timing behavior
of the former C adapter. Preserve the excluded-DFast assertion path and the
34-word dictionary bridge layout.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -A clippy::manual-bits -D warnings
- ulimit -v 41943040; make -j1
- capped strategy-2 compress/decompress round trip with cmp
- Full original suite deferred to the next serialized verification step
Use vec![0; size] in the local-dictionary allocator probe so the focused
Rust tests satisfy the repository's strict clippy configuration. The probe
still owns the allocation until the test releases it and production code is
unchanged.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -A clippy::manual-bits -D warnings
The local-dictionary callback is now a C adapter around Rust-owned assignment
policy. Add a small allocator probe and focused tests for by-reference loading,
by-copy data publication, static-CCtx rejection, and allocation failure. The
probe observes the output slots during allocation so the test also protects
the original allocation-before-publication ordering. Strengthen the C and Rust
repr(C) checks with an explicit custom-allocator function-pointer size check.
The capped C-integrated build and original test suite remain deferred to the
main worker, which must run them serially under the repository's 40 GiB virtual
memory limit.
Test Plan:
- `git diff --cached --check` -- passed.
- `rustfmt --check rust/src/zstd_compress_dictionary.rs` -- passed.
- Capped cargo, make, native, fuzzer, and full-suite verification -- not run
by design; deferred to the main worker.
The Fast dictionary-table projection was populated before the Rust
orchestrator ran its window-update and match-state publication callbacks.
That left base and nextToUpdate pointing at the initial sentinel for MT
prefix jobs, which caused ZSTD_rust_fillHashTable to dereference an invalid
range. Project pointers to the live match-state fields so Rust reads the
same values the former C adapter would have read at the Fast branch.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; make -j1
- capped 4 MiB -T0 compress/decompress round trip with cmp
Keep the MT sequence-state projection helper available to the C ABI
orchestration while excluding it from the standalone Rust test target,
where the production entry point is intentionally not compiled. This
keeps the capped Rust test check warning-free without changing runtime
behavior or the C layout contract.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- git diff --check
The MT worker previously kept the external-sequence non-empty gate and LDM
invariant assertion in C, bundled with the private CCtx operation. Project the
CCtx pointer and raw-sequence store through a layout-checked seam so Rust owns
the gate and the apply point while C retains only the opaque
ZSTD_referenceExternalSequences() leaf. Preserve the callback ordering after
worker CCtx initialization and keep empty stores as a no-op.
Test Plan:
- git diff --cached --check -- passed before commit.
- Static rg inspection confirmed all projection initializers include the new
sequence-state pointer and no old serialState_applySequences symbol remains.
- Cargo, make, compiler checks, and full tests were not run per worker scope.
The dictionary-content orchestrator still routed the Fast hash-table fill
through a C callback even though the matching leaf already lived in Rust.
Replace that callback slot with a pointer-sized C/Rust projection containing
only the Fast leaf inputs, including a by-value copy of nextToUpdate and the
normalized CDict/full-load flags. Keep the dtlm/tfp invariant assertion in C
and retain all bridge offsets and the double-Fast callback.
A broader MatchState projection would expose private layout and adding fields
would change the established 34-word bridge. The narrow projection keeps
that ABI stable while making the Fast branch call ZSTD_rust_fillHashTable
directly.
Test Plan:
- `git diff --cached --check`
- `rustfmt --check --edition 2021 rust/src/zstd_compress_dictionary.rs` (reports only pre-existing formatting drift outside this change)
- Cargo, make, and full tests intentionally not run per task instructions.
The multi-file compression and decompression entry points already delegate
file iteration and resource-sensitive work to Rust, but each still kept the
final summary decision and progress-to-summary ordering in C. Move that
scalar policy into a Rust ABI function. The C side now supplies only exact
legacy display callbacks and unchanged format strings, while Rust receives
plain counters and a summary kind without exposing FIO, cRess, dRess, or
codec layouts. Focused unit tests cover both summary modes, callback order,
argument forwarding, and sessions that must not display a summary.
Test Plan:
- `git diff --check -- programs/fileio.c rust/src/fileio_asyncio.rs` -- passed
- `git diff --cached --check` -- passed
- `rustfmt --check --edition 2021 rust/src/fileio_asyncio.rs` -- parsed the file;
reports pre-existing formatting differences elsewhere in the file, so no
unrelated formatting was applied
- Cargo, make, and runtime tests were intentionally not run per worker scope
The DDict implementation already owned the decoder-table loader in
zstd_decompress.rs, but zstd_ddict.rs still declared it as an external C
helper and disabled that path in Rust unit-test builds. Call the Rust
implementation directly through the ABI-compatible entropy-table
projection and make the module documentation match the actual ownership.
Keep offset_of imported only for the non-test C-context projection so the
Rust test check remains warning-free.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- Standalone cargo test was attempted; linking remains blocked by the
existing C context-projection symbols used by the hybrid test crate.
The MT serial turn already belongs to Rust for turn selection, checksum
placement, and advance. The remaining LDM callback still bundled the private
window update, sequence generation, and ldmWindow publication behind one C
operation, which hid that ordering from Rust.
Expose those three private operations as separate C callbacks and invoke them
from Rust in the original order. The first callback retains the original
RawSeqStore and job-size assertions before updating the LDM window; sequence
generation keeps its error assertion; and publication keeps the
ldmWindowMutex lock, copy, signal, and unlock sequence. The serial mutex
remains held until the existing advance callback, and disabled-LDM, empty
input, and skipped-turn behavior remain unchanged.
Test Plan:
- `git diff --cached --check` -- passed.
- Cargo, make, and full tests were not run per request; run the capped serial
verification suite before relying on runtime coverage.
Keep weak-symbol discovery and private dctx/ddict extraction in the C adapter,
but project the exact trace callback types into Rust and invoke them from the
Rust trace lifecycle. Preserve the nullable-hook behavior, trace context
width, dictionary-cold timing, record field layout, size conversions, and the
no-trace build path. Add focused Rust policy tests for callback, record, and
null-projection behavior.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1 -C tests test
The row-matchfinder reset path only used its C callback to read the private
hashSaltEntropy scalar and invoke Rust. Replace that callback slot with the
existing 32-bit scalar, keep the ABI offsets pointer-sized on 32-bit and
64-bit targets, and update the salt directly in Rust at the row-table reset
point. Preserve the C workspace callbacks and verify the real salt transition
in the reset-policy test.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1 -C tests test
The CCtx reset tail already delegated storage reservation and ordering policy
to Rust, but crossed back into C through a redundant callback trampoline.
Project the storage state directly through the existing pointer-sized ABI slot
and invoke the Rust storage operation without the extra C hop. Keep private
workspace/layout callbacks and match-state reset in C, preserve hash -> match
-> storage ordering, and retain allocation-error propagation in Rust tests.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
Keep the compression context extraction and the weak trace hook adapter in C,
where the private CCtx layout and platform-specific tracing declarations remain
available. Project only the scalar trace inputs and callback through an explicit
ABI-checked boundary so Rust owns the enabled check, stable record layout,
version field, compression-only null fields, and end-of-trace policy. The C
entry point still clears traceCtx after Rust returns, preserving the old
single-use lifecycle and keeping trace-disabled builds free of weak-hook calls.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; make -j1 -C tests test
Move buffered-input presence, offset subtraction, and block-size validation into
Rust while keeping the private MT progression query behind its C callback. The
C entry point now projects only the consumed/produced counters, input-buffer
fields, and worker dispatch inputs. Rust preserves the null-buffer behavior and
aborts on the same impossible offset or oversized-buffer invariants as the
original C assertions.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Move the serial LDM dictionary preload policy into Rust. The C MT adapter now
exposes only the private window and hash-table operations through checked
callbacks, while Rust owns empty/raw-content eligibility, loaded-dictionary
reset ordering, and the force-window publication rule. The explicit projection
keeps the dictionary inputs and callback ABI auditable without exposing the
private SerialState layout.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; make -j1 -C tests test
Move the fileio source-size precedence policy into the Rust backend while
keeping the private CCtx pledge call, error handling, and diagnostics in the
C adapter. A known statted source size, including zero, remains authoritative;
when it is unavailable, a positive declared stream size is used, otherwise the
zstd unknown-content sentinel is preserved. The Rust policy accepts both
sentinels explicitly so the bridge does not couple their representations.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; make -j1 -C tests test
The CCtx reset orchestration already runs in Rust, but its C workspace
adapter still decided when capacity was too small or wasteful and maintained
the oversized-duration counter through C-only predicates. Move those factor,
threshold, overflow, and duration-update rules into Rust. C now supplies only
private workspace metrics and retains low-level allocation, reservation, and
layout operations.
Remove the obsolete C predicates so the workspace resize decision has one
Rust-owned implementation and remains covered at its boundary conditions.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; make -j1 -C tests test
The local-dictionary callback still contained the policy that selected
by-reference versus by-copy loading, rejected internal copies for static
CCtx instances, allocated and copied dictionary bytes, and published the
resulting local-dictionary state. Move that policy into the Rust compression
rewrite so the C callback is only an adapter around the private CCtx and
localDict layouts.
The adapter retains the C custom allocator and output slots, preserving the
existing ownership and free path while keeping the ABI boundary explicit.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -B -j1 -C tests fuzzer
- (cd tests && ulimit -v 41943040; ./fuzzer -v -i1 -s6572)
- ulimit -v 41943040; make -j1 -C tests test
Keep the migration map aligned with the completed adaptive-refresh and window
overflow seams. The CLI description now identifies only the remaining C
progression and diagnostic callbacks, and the compression description records
that Rust owns the stateful window rebase. The explicit list of private C
layout and callback boundaries remains unchanged so the hybrid build is not
mistaken for a finished rewrite.
Test Plan:
- git diff --cached --check
The overflow-correction policy and scalar correction arithmetic were already
Rust-owned, but the stateful window rebase still lived in the C
ZSTD_window_correctOverflow() inline. That left pointer advancement, limit
clamping, and the correction counter as an untracked production C algorithm
on every compression path that protects matchfinder indices.
Replace that inline with a five-pointer projection. C continues to compute the
private source index and choose the correction context, while Rust now owns
the rebasing transition: it advances the regular and dictionary bases, clamps
both limits with the original wrapping U32 behavior, checks the correction
invariants in debug builds, and increments the overflow-correction counter.
The C/Rust projection has explicit layout assertions, and the Rust fixture
covers the pointer, limit, and counter updates together.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
The adaptive compression loop still delegated its refresh clock gate to a
small C callback even though Rust already owned the iteration state and all
adaptive policy decisions. That left timing policy, the last-refresh scalar,
and one projection callback in the C-side orchestration boundary.
Move the one-sixth-second monotonic refresh gate into ZstdAdaptiveState. The
Rust loop now initializes the first refresh timestamp at frame start and
preserves the C callback's strict-greater-than interval check. C retains only
the private progression, parameter-setting, and diagnostic callbacks needed
by the existing file I/O context.
Shrink both sides of the projection together and keep compile-time offset and
size assertions aligned. The adaptive unit fixture now verifies Rust records
the refresh event while continuing to exercise the existing progression and
policy callbacks.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
The default zstd frame loop was already implemented in Rust, but its
completion and error-result policy still lived in the C callback wrapper.
That left the C translation unit deciding when a decoded size was valid and
when a decoding or premature-end status had to become the historical frame
sentinel. Keep the exact operator-facing diagnostics and the private
asynchronous-resource pointers in C, while giving Rust ownership of the
status-to-result policy and callback ordering.
The new C/Rust policy record has compile-time layout assertions on both sides,
and the Rust fixture covers success, decode error, premature end, sentinel
results, and callback sequencing. This keeps the bridge explicit without
making Rust depend on the private dRess_t layout.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
Move the scalar portion of the private CCtx reset tail behind the existing
Rust reset adapter. The C side now publishes explicit pointers to the fields
that Rust must initialize, while retaining workspace ownership, match-state
reset, storage reset, and the XXH64 reset callback that cannot cross the
boundary safely. This removes the last opaque C initializer from this reset
phase without exposing the private ZSTD_CCtx layout to Rust.
The ABI record has compile-time offset and size assertions on both sides, and
the Rust unit fixture now checks the resolved compression parameters,
prefetch policy, pledged-size sentinel handling, counters, content-size flag,
block-size limit, stage, and dictionary metadata. The callback-order tests
continue to verify that compressed-block state reset precedes match-state and
storage cleanup, and that failures stop the tail at the correct callback.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
Move the per-file and multi-file --list row formatters out of programs/fileio.c. C still owns filesystem access, frame analysis, private fileInfo_t storage, and exact status diagnostics; Rust now formats the rows into byte buffers and sends them through a synchronous output callback. This keeps filenames byte-preserving and prevents the private C record from crossing the ABI while preserving the original display thresholds, human-readable size policy, unavailable-content spacing, checksum order, and total-row behavior.
Replace the old display callbacks with one size-delimited output callback shared by the single-file and multi-file projections. Add explicit Rust ABI layout checks, formatter fixtures for normal/unavailable/verbose/checksum/total output, and callback-order assertions for the policy boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (208 passed)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test (all original tests completed successfully)
- git diff --check
Remove the C-only DDict offset globals and route production
ZSTD_copyDDictParameters() through the existing C-produced DCtx view. This
keeps the private decoder layout and optional fuzzing fields owned by the
canonical projection while preserving the DDict entropy, prefix, window, and
checksum state updates. Keep the synthetic offset writer test-only so the
Rust unit tests remain independently linkable.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (207 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
- git diff --check
Move the decompression-resource state declaration ahead of the first
statement in FIO_createDResources. The initializer only captures the
resource address, so moving memset after the declaration preserves the
runtime order while avoiding the ISO C90 mixed-declaration warning.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 (passed after cleanup)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test (passed before this declaration-only cleanup)
- git diff --check (passed)
Move the decompression resource creation order into the Rust CLI policy
layer: dictionary stat and patch-memory preparation, decoder allocation,
window/checksum configuration, dictionary reset/attachment, and asynchronous
pool creation now run through one Rust-owned sequence. Keep dRess_t,
stat_t, dictionary buffers, decoder context, pools, and exact allocation
errors in C callbacks.
Preserve the original patch-from and dictionary-reference behavior while
adding ABI layout assertions, scalar policy coverage, and error short-circuit
tests for the new boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (207 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Make the Rust policy layer own the ordered compression-resource lifecycle:
create the CCtx, prepare patch/dictionary state, create the write and read
pools, validate the dictionary, apply general and multithreaded parameters,
and finally load the dictionary. Keep cRess_t, FIO_Dict_t, file statistics,
AIO pools, and CLI diagnostics in C callbacks so the existing resource
ownership and error behavior remain local to the C backend.
Preserve adaptive window defaults and patch-from parameter adjustment while
adding ABI layout assertions and a lifecycle-order test for the new boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (205 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
The MT stream initializer previously exposed one C callback that waited for
all submitted jobs and then released their resources. That left the ordering
policy in C even though Rust already owned the surrounding initialization
sequence and the ring-order wait/release policies.
Expose separate wait-all and release-all callbacks at the Rust/C boundary and
invoke them in Rust in the original wait-before-release order. C retains the
worker synchronization, job descriptors, input buffer, and resource-pool
side effects behind the two callbacks, while the focused Rust initializer test
now makes the ordering explicit.
Test Plan:
- capped cargo +nightly fmt for the Rust workspace -- passed
- capped root cargo clippy --all-targets -- -D warnings -- passed
- capped serial make -j1 for the single-threaded library, MT library, and CLI -- passed
- git diff --cached --check -- passed
Transparent stream initialization can resolve an unknown source size to the
multithreaded path and update appliedParams.nbWorkers. The Rust stream2
fallback previously captured that value before initialization, then entered the
serial generic adapter while the context stage was still created. Unknown-size
stdin compression consequently returned StageWrong instead of producing a
frame.
Keep the worker-mode projection live through initialization so Rust observes
the applied value at the existing MT-versus-serial decision point. The ABI
layout assertions now cover the pointer projection, and a focused regression
test verifies that an initialization callback which selects MT dispatches the
MT step and preserves the expected callback order.
Test Plan:
- capped cargo +nightly fmt for the Rust workspace -- passed
- capped root and CLI cargo clippy with -D warnings -- passed
- capped serial make -j1 for single-threaded library, MT library, and CLI -- passed without the prior pointer-sign warning
- capped stdin compression/decompression round trip for unknown-size input -- passed
- capped original suite, including CLI tests, native tests, fuzzer, and both zstream phases -- passed; the final signedness-only ABI spelling cleanup was followed by a clean capped rebuild
- targeted root cargo test attempted but remains blocked at link time by pre-existing decompression bridge symbols (ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and ZSTD_rust_block_context_init)
FIO_openDstFile() still owned the complete destination policy loop in C even
though the filesystem opener and status/action classifier were already Rust
leaves. That left confirmation, sparse-mode adjustment, overwrite removal, and
retry ordering duplicated beside private FILE* and CLI state.
Project the C-owned callbacks for diagnostics, preference mutation, user
confirmation, stdout setup, and existing-file removal. Rust now owns the
retry/order state machine and calls the existing narrow opener for each
attempt; C keeps private preferences and context, exact diagnostics, and FILE*
ownership. The callback layout is asserted on both sides, and focused tests
cover status ordering, confirmation/removal retry, setvbuf preservation, and
invalid policy inputs.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1` -- passed, including the single-thread library, MT library, and CLI binary.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- passed: 206 tests.
- `git diff --cached --check` -- passed.
The public ZSTD_compressStream2_c entry point still contained the complete
fallback state machine even though its validation, initialization policy,
serial-versus-MT dispatch, and result accounting were already represented by
Rust policy helpers. That left the main streaming entry point as a large C
orchestration island and made the Rust rewrite boundary misleading.
Project the scalar stream state and private CCtx operations through a checked
C ABI, then let Rust own the fallback ordering. Rust now validates buffers and
the end directive, handles transparent one-shot completion and stable-input
deferral, performs the initialization and stability decisions, selects the
serial or MT path, and publishes the result policy. C retains the private
context layout, stream adapter, MT operations, checksum/epilogue side effects,
and trace/reset callbacks. ABI offset assertions and focused callback-order
and stable-input tests protect the projection without requiring a standalone
Rust test binary to link every C bridge symbol.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed before and after formatting.
- `git diff --cached --check` -- passed.
- Native `make -j1` and the original suite remain the next post-commit verification gates.
Project the private requested-parameter and context state into a fixed scalar ABI record, leaving only C's parameter-selection leaf and private layout access in the shim. Rust now owns the complete-input eligibility predicate, including advanced-state exclusions, LDM sentinel handling, and the final fast/double-fast strategy gate, while preserving the requested-level and INT_MIN results.
Add focused policy tests for fast and double-fast acceptance, representative advanced-state rejection, the accepted LDM sentinel, non-fast strategies, and null state.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
Document Rust ownership of compression-size estimator policy and the CLI adaptive feedback state machine, and describe the private C callbacks that remain at those boundaries.
Test Plan:
- Documentation-only change; git diff --cached --check
Move the multithreaded compressStream2 outer coordinator into the Rust compression module. The C bridge now owns only the private ZSTDMT call and consumed/produced counters, while Rust preserves the progress loop, completion ordering, and error handoff.
Keep the MT context mutations, diagnostics, assertions, and buffer-expectation publication in C because they depend on the private context layout. Add callback-driven tests for continue/break, error completion, end trace/reset ordering, and malformed states.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
Clippy rejects the hand-written round-up expression used to compute the callback-alignment offset in the Rust file-I/O projection. Use the equivalent div_ceil spelling so the ABI assertion remains clear and the root Rust targets pass with warnings denied.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
The zstd file-I/O loop still kept adaptive compression's state machine in C: progression deltas, refresh gating, job-completion checks, input counters, speed decisions, and level updates were interleaved with private FIO and ZSTD state. That left orchestration policy behind the existing scalar Rust predicates.
Move the adaptive state and callback order into Rust. C now supplies scalar progression snapshots, the clock gate, exact diagnostics, and the ignored CCtx parameter setter through callbacks; private FIO_prefs_t, ZSTD_CCtx, ZSTD_frameProgression, clocks, and progress formatting remain C-owned. Preserve the previous-progression publication before backlog evaluation, input counter reset points, and serial/MT level-clamp behavior.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
The public CCtx and CStream size-estimation APIs still selected row-matchfinder modes and walked compression levels in C, leaving a policy-level implementation behind the Rust workspace-sizing bridge. That also made the C layer responsible for the subtle raw-size_t MAX behavior used when an estimator returns an encoded error.
Move the row-mode selection and shared monotonic memory-budget policy into Rust. Keep the C callbacks responsible for constructing private ZSTD_CCtx_params values and for the distinct CCtx/CStream sizing formulas, so no private parameter layout crosses the ABI. Preserve the signed MIN(compressionLevel, 1) start and raw maxima exactly.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Update the migration boundary document to record the two latest multithreaded
policy moves. Rust now owns serial-reset LDM normalization, table sizing, and
publication plus final completion trace/reset ordering, while C still owns the
private job/context state and callbacks.
Test Plan:
- `git diff --cached --check` -- passed.
- Capped native build and full original test target were run before this
documentation-only commit and passed.
The MT serial-reset bridge now declares its Rust scalar projection before
executing the diagnostic branch. This keeps the declaration ordering valid for
the C90-oriented compiler flags used by the native test builds while leaving
the Rust-owned policy and callback sequence unchanged.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1` -- passed without the
new mixed-declaration warning.
- `git diff --cached --check` -- passed.
The multithreaded stream loop already delegated scalar result classification to
Rust, but its final error/end conditional still mixed completion ordering with
the private C context. That left trace and reset sequencing embedded beside the
MT counters and made the next Rust seam depend on exposing CCtx layout.
Rust now consumes the projected loop action and sequences the final completion
side effects through opaque C callbacks. Continue and break do nothing, worker
errors reset the session, and completed end operations trace before resetting.
The reset callback result is intentionally discarded. C retains the private
context mutations in callback adapters, and the callback invocation remains
inside the existing ZSTD_MULTITHREAD branch before FORWARD_IF_ERROR(flushMin).
C/Rust repr(C) projections and ABI assertions protect the callback/result
boundary, while focused tests cover all four loop actions and end ordering.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_compress.rs` -- passed
- single-thread and `-DZSTD_MULTITHREAD` C syntax checks -- passed; existing warnings only
- `CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed
- `CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --benches -- -D warnings` -- passed
- focused Rust test filter was attempted serially but is blocked by unrelated in-progress `rust/src/zstdmt_compress.rs` test ABI edits; no full test suites were run
Move the multi-input single-output decision sequence out of
FIO_multiFilesConcatWarning while preserving the C-owned CLI boundary. The
previous wrapper classified fatal remove cases, emitted the concatenation
warning, disabled --rm, reclassified the action, and then selected quiet
abort or confirmation in C. Add a repr(C) callback projection so Rust owns
only that scalar ordering while C continues to own exact diagnostics, the
confirmation prompt, FIO_prefs_t mutation, and all private state.
The Rust bridge re-runs the action after the C disable-remove callback with
the same has-output/remove arguments as the original wrapper. C and Rust
assert the callback layout, and focused tests cover callback order,
fatal/quiet paths, and ABI offsets. Existing scalar action tests remain in
place.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -B -C programs -j1 fileio.o`
— passed; only pre-existing suffixList C++-compat warnings appeared.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path
rust/cli/Cargo.toml --tests --no-deps` — passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 RUSTFLAGS='-C
link-arg=/tmp/zstd_rust_test_bridges.o' cargo test --manifest-path
rust/cli/Cargo.toml --lib 'fileio_prefs::tests::multi_files_concat_'`
— 8 passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo build --manifest-path
rust/cli/Cargo.toml --lib` — passed; the archive exports
`FIO_rust_multiFilesConcatWarning`.
- `cargo +nightly fmt --manifest-path rust/cli/Cargo.toml -- --check` — not
clean due pre-existing formatting drift in unchanged Rust code; no bulk
formatting was applied.
- Full native/upstream/fuzzer suites were not run by request.
ZSTD_compressBegin_internal already routes attach-versus-reload policy through
Rust, but its C insertion callback still inspected a non-null ZSTD_CDict and
selected dictContent, dictContentSize, and dictContentType. That left source
selection coupled to the private CDict layout and kept the Rust bridge from
receiving the same selected source for direct and prepared dictionaries.
Project the three CDict content fields into the repr(C) begin state. Rust now
selects either those fields or the direct dictionary arguments before invoking
the C callback. The callback keeps the private CCtx insertion path and no
longer branches on CDict. ABI offsets and sizes are asserted on both sides,
with focused by-reference and forced-CDict tests covering pointer, size, type,
reset order, and dictionary-result publication.
Test Plan:
- `cargo check --manifest-path rust/Cargo.toml --lib --tests` -- passed under
`ulimit -v 41943040; CARGO_BUILD_JOBS=1`.
- Serial GCC/Clang syntax-only checks for `lib/compress/zstd_compress.c` --
passed under the same cap; only existing warnings were reported.
- `cargo clippy` for lib, benches, and tests with `-D warnings`, plus nightly
format check -- passed serially under the same cap.
- `cargo test --manifest-path rust/Cargo.toml --lib zstd_compress_dictionary
-- --test-threads=1` -- compiled but standalone linking failed on the three
pre-existing C bridge symbols `ZSTD_rust_dctx_trace_view`,
`ZSTD_rust_dctx_view`, and `ZSTD_rust_block_context_init`.
- Full native, upstream, and fuzzer tests were not run per scope.
Synchronize the ownership map with the default-window decompression policy, the Rust-owned single-file --list ordering, and CDict match-state reset policy moved in the latest migration cycle.
Test Plan:
- git diff --cached --check
Add the existing heap-mode bridge declaration beside the other decompression shim prototypes so the new Rust policy seam remains warning-clean under the native C build.
Test Plan:
- git diff --cached --check
- capped full test suite passed before this warning-only fix; final capped native rebuild follows
Move the single-file --list status gates, diagnostic-versus-metadata ordering, and aggregate projection into Rust. C retains file opening and frame analysis, the private fileInfo_t layout, exact diagnostics, and row formatting behind explicit callbacks, so the user-visible behavior remains unchanged while the policy boundary is auditable.
Test Plan:
- git diff --cached --check
- worker format, check, clippy, and serial C compilation (passed); focused Rust tests compiled but the standalone link hit the pre-existing C bridge-symbol gap
Route the configured default maximum window size through the Rust policy layer while keeping the build-time C configuration value and decoder layout in C. The explicit projection preserves overridden builds exactly and gives the scalar policy a focused null-input contract and tests.
Test Plan:
- git diff --check -- lib/decompress/zstd_decompress.c rust/src/zstd_decompress.rs
- capped serial cargo check, clippy, nightly fmt, C syntax checks, and focused default-window tests (passed in the worker)
CDict initialization still hard-coded the match-state reset policies in its
C callback, even though Rust already owned the surrounding content branch,
workspace reservation, and callback ordering. That left the reset target and
cleanup policy split across the language boundary and made the intended CDict
reset contract implicit.
Extend the existing repr(C) reset callback with the three private enum values.
Rust now selects make-clean, index-reset, and the CDict reset target before
calling the opaque C operation. C retains the private ZSTD_CDict, workspace,
and match-state layouts and only casts the projected values for
ZSTD_reset_matchState(). Focused by-reference and by-copy probes record the
values and preserve reserve, reset, and insert ordering.
Test Plan:
- `rustfmt --check --edition 2021 rust/src/zstd_compress_dictionary.rs` -- passed.
- GCC and Clang syntax-only checks for `lib/compress/zstd_compress.c` under
`ulimit -v 41943040` -- passed.
- `cargo check --manifest-path rust/Cargo.toml --lib --tests` under the cap -- passed.
- Serial `cargo clippy` lib, benches, and tests passes with `-D warnings`,
plus `cargo +nightly fmt --all -- --check` -- passed.
- `cargo test --manifest-path rust/Cargo.toml --lib zstd_compress_dictionary
-- --test-threads=1` compiled but could not link the standalone test binary
because three pre-existing C bridge symbols are unavailable outside the
native harness; no native, upstream, or fuzzer tests were run.
Document the advanced one-shot begin/end seam, MT job-table teardown order, decompression heap-mode policy, and file-removal status policy so the migration boundary stays synchronized with the implementation.
Test Plan:
- git diff --cached --check
Move the Rust teardown projection declaration before the null guard so the MT source remains clean under the repository's C90 declaration rules. The initializer only copies pointers, scalars, the allocator, and the callback; the null guard still precedes the bridge call and preserves the existing no-op behavior for an absent job table.
Test Plan:
- git diff --cached --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test (passed before this warning-only fix)
Apply the repository formatter to the new MT job-table teardown projection and its focused callback-order test. This keeps the accepted lifecycle seam behavior unchanged while restoring the crate's formatting contract.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all
- git diff --cached --check
Normal MT context teardown still let the C wrapper directly sequence
per-job synchronization destruction before Rust released the opaque job-table
storage, while failed create and expansion transactions already used a Rust
helper for that same order. Route the normal teardown through a repr(C)
projection so Rust owns the destroy-before-storage-free policy consistently.
The synchronization callback, descriptor storage, and custom allocator remain
opaque C-owned operations. Add matching C/Rust layout assertions and a focused
callback-order test.
Test Plan:
- `git diff --cached --check` -- passed
- Cargo, make, native tests, and heavy verification were not run per request
ZSTD_compress_advanced_internal() still encoded the one-shot begin-then-end
sequence in C after public advanced parameter validation had moved behind Rust.
That left the public compression boundary split between Rust policy and a C
orchestration body, and made the begin-error ordering implicit in the C path.
Add an explicit C-layout state with opaque begin and end callbacks. Rust now
owns the ordering, forwards the source size as the pledged size, propagates a
begin error before invoking the end callback, and returns the end result. The
callbacks retain the private ZSTD_CCtx and ZSTD_CCtx_params layouts, so the
change moves policy and sequencing without exposing window, workspace,
matchfinder, or context internals to Rust. The existing advanced and
using-dictionary wrappers continue to use the same C private operations.
Test Plan:
- `git diff --cached --check` -- passed.
- `rustfmt --check --edition 2021 rust/src/zstd_compress.rs` -- passed.
- Not run: Cargo, make, native tests, builds, or heavy verification per request.
`ZSTD_HEAPMODE` was still returned directly from C, and the stack entry
projection also copied the raw macro. That left the one-shot stack-versus-heap
classification in the C configuration shim even though Rust owns the public
decompression branch and stack action.
Keep the build-time override in C, but pass it through a one-field repr(C)
projection to Rust. Rust normalizes values below one to the stack mode and
values at or above one to the heap mode, preserving the existing behavior
while making the branch policy explicit. The C stack object, private DCtx
layout, platform initialization, and trace boundary remain C-owned; moving
those would either duplicate build configuration or cross the requested ABI
boundary. The stack projection now uses the same normalized result as the
one-shot branch.
ABI layout assertions cover the scalar projection, and focused Rust tests cover
negative, zero, positive, maximum, and missing-configuration inputs.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_decompress.rs` -- passed
- Cargo, Make, native tests, and heavy commands intentionally not run per the
task restriction; integration verification remains pending.
Keep the MT frame-progression callback under the ZSTD_MULTITHREAD preprocessor guard using an ordinary C declaration. The previous warning-cleanup edit accidentally prefixed the declaration with a preprocessor marker, which broke the multithreaded compilation path.
Test Plan:
- git diff --cached --check
- Capped native rebuild failed before this fix at the invalid directive; rerun pending
Keep the migration boundary accurate after moving frame-progression dispatch, the decompression no-forward-progress threshold, and compression metadata-transfer selection into Rust. The README now states which scalar inputs Rust owns and which private callbacks, probes, and operations remain in C.
Test Plan:
- git diff --cached --check
- Capped clippy, CLI tests, native build, smoke test, and full upstream suite passed before this documentation-only commit
The MT frame-progression callback is only referenced when ZSTD_MULTITHREAD is enabled. Keep its definition under the same configuration guard so single-threaded library, test, and decode-corpus builds do not report an unused static function while the MT callback remains available to the Rust dispatch bridge.
Test Plan:
- git diff --cached --check
- Capped full tests passed before this warning-only guard
- Capped native rebuild and smoke rerun pending
The Rust-owned frame-progression dispatch invokes the existing C MT progression API through a mutable opaque callback context. Match that API's established signature in the adapter so the new boundary does not introduce a discarded-const qualifier warning while preserving the private MT context.
Test Plan:
- git diff --cached --check
- Capped native make rerun pending after this warning-only fix
ZSTD_NO_FORWARD_PROGRESS_MAX is a compile-time override that was returned
directly by the C adapter. Keep the active build value in C, where
configuration belongs, but pass it through an ABI-checked scalar projection to
Rust. Rust now owns the policy boundary and preserves the exact configured
threshold, while the existing stalled-stream comparison, error mapping, and
private DCtx layout remain unchanged.
Test Plan:
- `git diff --cached --check` -- passed
- Cargo, make, builds, native tests, and heavy verification were not run per
request.
Move the public ZSTD_getFrameProgression MT-versus-single-thread worker-count
branch into a Rust-owned scalar projection. C continues to compute the
single-thread scalar inputs and keeps the private ZSTDMT context behind a
callback; the compile-time ZSTD_MULTITHREAD adapter remains local to C. Add
C/Rust layout assertions and focused Rust tests for both dispatch paths and
null-state fallback.
Test Plan:
- `git diff --cached --check`
- Not run: Cargo, make, builds, native tests, and heavy verification per request.
Keep the private stat_t probe, destination opening, metadata syscalls, and format callbacks in C, but route the regular-source plus stdin/stdout exception policy through the Rust fileio preference layer. The Rust ABI bridge normalizes the scalar consumed by the existing destination lifecycle, with compile-time value assertions and focused tests covering regular, non-regular, stdin, stdout, and nonzero scalar inputs.
Test Plan: Not run by request; cargo, make, native tests, and heavy commands were intentionally avoided. Lightweight git diff --check passed.
Apply the repository formatter to the sequence-decoder policy bridge introduced in the preceding decompression refactor. The change is formatting-only.
Test Plan: cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
Keep the opaque ZSTD_CCtx_params copy operation in C, but make Rust own the public advanced2 null-source validation and copy-before-preparation ordering. The bridge now rejects a missing source before any allocator or workspace callback, with explicit ABI assertions and focused order/null-input tests.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Keep the C build-time mutual-exclusion check and private decoder-context assembly in the C shim, but pass the selected force-short/force-long policy through Rust. Rust now owns normalization to the runtime, short, or long sequence decoder mode, with ABI assertions and focused null/build-policy tests.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Keep the target-specific ZSTD_LEGACY_SUPPORT macro in the C translation unit, but pass its scalar value through a Rust policy bridge before legacy dispatch. Rust now accepts only the historical supported range 1 through 7 and normalizes all other values, including a missing projection, to disabled. ABI assertions and focused tests preserve the compile-time configuration contract.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Keep FIO_removeFile responsible for the filesystem operation, exact diagnostics, and C return wrapper, but route its status classification through a Rust policy bridge. Unknown statuses are explicitly treated as failed removal. ABI value assertions and focused mapping tests preserve the C contract.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Apply the repository formatter to the dictionary-size and MT input-publication bridges introduced in the preceding atomic refactors. The change is formatting-only and keeps the semantic seams independently reviewable.
Test Plan: cargo +nightly fmt --manifest-path rust/Cargo.toml --all
Keep the advanced CDict estimator and its private workspace sizing inputs in C, but route the public estimate through a Rust policy bridge. Rust now selects unknown-source/create-CDict parameters and the public by-copy load mode before invoking the opaque C estimator. Focused tests verify callback ordering, selected parameters, and the missing-policy-input guard.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Keep the C-owned compression job callback responsible for private buffer and job-state mutation, but return the range-active decision alongside its result. Rust now owns the outer scheduler policy that publishes or suppresses the reusable input count after job creation, including the error path. The focused seam test covers the callback-clears-range case without exposing the MT context layout.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Move the scalar InfoError-to-action mapping used by FIO_listFile into Rust and reuse the same classifier while aggregating multi-file list results. C retains file opening/parsing, exact diagnostics, metadata formatting, private fileInfo_t state, and the public result values; add layout and mapping coverage.
Test Plan: git diff --cached --check; focused Rust mapping test added; full capped verification will run after the MT and dictionary workers are integrated.
Apply rustfmt to the stream-result and MT input-range policy projections and their focused tests. This is mechanical only; it keeps the new Rust seams compliant with the repository formatter without changing behavior.
Test Plan: ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check; git diff --cached --check.
Move the ready/wrap branch and synchronization ordering from ZSTDMT_tryGetInputRange into Rust. The Rust policy now waits for the prefix range, invokes the C-owned prefix move, waits for the selected source range, and publishes the buffer; C retains memmove, private buffer fields, round-buffer state, and LDM callbacks behind the projection. Add ABI assertions and focused wrapped/ordinary ordering tests.
Test Plan: git diff --cached --check; focused Rust tests added; full capped Rust/native/original-suite verification follows.
Move the post-call ZSTD_compressStream2_c policy into Rust: adapter errors now short-circuit before buffer publication, successful calls publish buffer expectations before calculating pending output, and the private C context remains behind a callback projection. Add ABI assertions and ordering tests for both paths.
Test Plan: git diff --cached --check; focused Rust tests added; full capped Rust/native/original-suite verification will run after the MT slice is integrated.
Move stdin-sentinel classification and source stat/open result policy into the Rust fileio backend while keeping C responsible for diagnostics, binary-mode setup, and FILE ownership. The bridge leaves stat layout and native file utilities behind the existing C ABI and adds a focused no-stat stdin test.
Test Plan: git diff --cached --check; focused Rust test added but full capped verification will run after the remaining workers are integrated.
The new MT-loop policy projection asserted its total size as eight times
`size_of::<usize>()`. Although that describes the padded C layout, clippy
interprets the multiplication as a manual bit-count expression and rejects it
under the repository's `-D warnings` policy.
Express the same ABI invariant from the final field offset plus its field size.
This keeps the assertion tied to the actual projected layout without changing
any runtime policy or C/Rust representation.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_compress.rs` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
The multithreaded branch of `ZSTD_compressStream2_c` performed the complete
post-call result classification in C: it prioritized errors, recognized a
completed end directive, and applied different progress rules for continue,
flush, and end operations. Those decisions were scalar policy around a C-owned
MT call, but remained embedded beside private counters, reset, and tracing.
Rust now classifies one projected iteration and returns an explicit action for
continue, break, error, or completed end. The C shim still owns the MT call,
input/output accounting, error forwarding, private context mutation, reset,
and trace callback; the existing condition ordering and progress comparisons
are preserved. ABI layout assertions and focused tests cover error precedence,
completed end, progress-based continue termination, and pending/full output.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_compress.rs` -- passed
- Full capped Rust/native verification is the next serial step.
The stack-backed one-shot decompression entry point used to make its own
heap-mode rejection, static DCtx initialization, allocation-error mapping,
and dispatch ordering in C. That kept scalar control flow beside the private
DCtx layout even though Rust already owns the decompression policy and context
leaf.
Rust now consumes an ABI-checked scalar projection and callback set. It makes
the branch and ordering decisions, maps a failed static initializer to the
same memory-allocation error, and invokes the existing C-owned context leaf.
The C shim retains the local stack object, private DCtx layout, and
`ZSTD_initStaticDCtx` implementation. Recording-callback tests cover heap
rejection, initializer failure, and successful initialization-to-dispatch
ordering without fabricating the private context layout.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_decompress.rs` -- passed
- Full capped Rust/native verification remains pending until the parallel
compression seam is integrated.
The decompression callback in fileio used to classify result statuses and
select its display action with a C switch. That left scalar result policy in
the C frontend even though Rust already owns the decompression dispatch and
result classification, and it coupled the C callback to a diagnostic enum.
Rust now exposes an ABI-checked status-action classifier. It keeps the
original action ordering and preserves silent handling for statuses that have
no display diagnostic, while the C callback retains the exact diagnostic
strings, source-name formatting, and display operation. Invalid inputs map to
a silent fallback action so the public callback remains behavior-compatible.
Focused Rust tests cover every status class, invalid values, and the exported
ABI result.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/fileio_asyncio.rs` -- passed
- Full capped Rust/native verification remains pending until the parallel
compression and decompression seams are integrated.
FIO_openDstFile retained the filesystem leaf in Rust but its C wrapper still encoded the status precedence for test mode, stdout, same-file protection, sparse-mode adjustment, overwrite prompting, removal, retry, and final errors. Centralize that pure action classification in Rust without moving FILE* handling, metadata, preference mutation, prompts, diagnostics, or retry callbacks across the ABI.\n\nThe action bridge validates status, confirmation, and sparse-state inputs and has focused tests for every destination outcome, sparse-first ordering, prompt acceptance/abort, quiet mode, overwrite mode, setvbuf/open failures, success, and invalid inputs.\n\nTest Plan:\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt/clippy gates passed before staging\n- git diff --cached --check\n- Full capped native and original-test verification follows after the batch is committed.
ZSTDMT_updateCParams_whileCompressing already delegated parameter derivation to Rust, but C still owned the transition publication order and independently updated compressionLevel. Return a Rust-owned scalar result that carries the requested level with the derived compression parameters, while preserving the active frame window and all private MT context state in C.\n\nThe new projection and result have explicit C/Rust layout assertions. Focused Rust tests exercise unknown-size and explicit-size hints, LDM and override inputs, saved-window restoration, requested-level forwarding, and the exported ABI wrapper.\n\nTest Plan:\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt/clippy gates passed before staging\n- git diff --cached --check\n- Full capped native and original-test verification follows after the batch is committed.
ZSTD_compressStream2_c still owns private context mutation, diagnostics, buffer updates, and codec initialization, but its stable-input transparent-initialization decision was a remaining scalar policy island in C. Move the continuation validation, block-size threshold, format-specific progress hint, and wrapping size arithmetic behind a Rust projection. Preserve the original C error messages and return values while keeping all private context state on the C side.\n\nThe bridge uses explicit ABI layout assertions for the mixed pointer/size projection. Focused Rust tests cover buffered and non-continue calls, block-boundary initialization, empty and short stable input, both frame formats, matching continuation state, invalid source/position, and null bridge input.\n\nTest Plan:\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings\n- Full capped native and original-test verification follows after the batch is committed.
The MT stream initializer and its later dictionary update both encoded the
same three-way choice in C: copy a supplied dictionary, attach a borrowed
CDict, or install a raw prefix. The old code also embedded the required
release/clear-before-attach ordering in each callback wrapper. Project the
presence and raw-content flags into Rust, where the branch and ordering are
now explicit and tested. C retains only the private CDict allocation,
prefix-storage, context publication, and destruction callbacks, so the
configured C layouts and allocator behavior remain unchanged.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- `git diff --check` and `rustfmt --edition 2021 --check` -- passed before commit
- Capped native and original-test verification remains pending for the complete batch.
The file-I/O wrapper already delegated dictionary loading to Rust but kept
all status interpretation in C. That duplicated the malloc and mmap status
families and made the platform-specific error branches part of the C policy
surface. Add a Rust classifier that maps the shared numeric loader statuses
to diagnostic actions, including the distinct mmap failure classes. Keep
metadata lookup, platform handles, ownership, and the exact EXM_THROW text
in C, where the configured platform APIs still belong. The classifier also
rejects out-of-range type/status values; the valid malloc and mmap enums
intentionally share numeric values and therefore cannot be distinguished
beyond their family tag.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- passed (192 tests)
- Broader native and original-test verification remains pending for the complete batch.
ZSTD_freeCCtx previously passed one C callback that bundled dictionary,
multithreaded context, workspace, and heap-object teardown. That left the
lifetime order inside private C control flow and made the public destruction
policy harder to audit. Project the four private operations separately and
let Rust enforce the original dictionary, MT-context, workspace, and
object order while preserving the static-context and workspace-embedded
object guards. The callbacks still own the C layouts and custom allocator
operations.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- Broader native and original-test verification remains pending for the complete batch.
ddidderr