Keep the C-owned trace slot behind an explicit pointer projection so the
Rust trace boundary owns the complete enabled-call lifecycle. Rust now reads
the context, publishes the record, and clears the slot after the callback;
C only supplies private CCtx fields and the weak callback adapter.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests (rust)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings (rust)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- Focused trace lifecycle test added; standalone cargo test remains link-blocked by the existing C-owned decompression view symbols.
Move the first-job versus continuation-job trace policy out of the C worker
callback. Rust now decides when the external-dictionary assertion is required
and orders it before the trace, while C retains the private window assertion
and trace-recording leaves.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
Move LDM loadedDictEnd and final nextToUpdate publication out of the C
dictionary-content callbacks. Rust now writes through compact projections with
live window bases, while C retains the configuration-sensitive window,
matchfinder, and LDM operations.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --tests -- -A clippy::manual-bits -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
Replace the dictionary loader's C match-state publication callback with a live
field projection. Rust now publishes nextToUpdate, loadedDictEnd, and
forceNonContiguous at the existing orchestration point while C retains the
private match-state layout. Keep the window base indirect so a preceding
window update cannot leave Rust with a stale offset origin.
Test Plan:
- git diff --cached --check
- capped cargo check --tests
- capped cargo clippy --tests -- -A clippy::manual-bits -D warnings
- capped make -j1
- capped make -j1 -C tests test
Replace the C worker finish callbacks that publish errors, final block sizes,
consumed bytes, and completion signals with a Rust-owned publication path. The
nested projection keeps the job counters and pthread operations private to C,
while Rust preserves the error/success ordering and size_t wrapping behavior.
The consumed-size assertion remains a C validation callback at the original
point before publication.
Test Plan:
- git diff --cached --check
- capped cargo check --tests
- capped cargo clippy --tests -- -A clippy::manual-bits -D warnings
- capped make -j1
- capped make -j1 -C tests test
- standalone hybrid cargo test remains unavailable because its link lacks the
pre-existing ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and
ZSTD_rust_block_context_init symbols
Move the intermediate worker progress state update out of the C callback and
into a Rust-owned publication leaf. The projection keeps the private job
descriptor and pthread objects in C while preserving the lock, counter update,
diagnostic, signal, and unlock order, including size_t wrapping behavior.
Test Plan:
- git diff --cached --check
- capped cargo check --tests
- capped cargo clippy --tests -- -A clippy::manual-bits -D warnings
- capped make -j1
- capped make -j1 -C tests test
- standalone capped cargo test was attempted but its hybrid link lacks the
pre-existing ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and
ZSTD_rust_block_context_init symbols
Replace the dictionary-content Double-Fast callback with a live-field
projection and call the existing Rust table-fill leaf directly. The
projection keeps hash tables, window base, match-state index, and parameters
live through the window/publication callbacks, preserving the timing behavior
of the former C adapter. Preserve the excluded-DFast assertion path and the
34-word dictionary bridge layout.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -A clippy::manual-bits -D warnings
- ulimit -v 41943040; make -j1
- capped strategy-2 compress/decompress round trip with cmp
- Full original suite deferred to the next serialized verification step
Use vec![0; size] in the local-dictionary allocator probe so the focused
Rust tests satisfy the repository's strict clippy configuration. The probe
still owns the allocation until the test releases it and production code is
unchanged.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -A clippy::manual-bits -D warnings
The local-dictionary callback is now a C adapter around Rust-owned assignment
policy. Add a small allocator probe and focused tests for by-reference loading,
by-copy data publication, static-CCtx rejection, and allocation failure. The
probe observes the output slots during allocation so the test also protects
the original allocation-before-publication ordering. Strengthen the C and Rust
repr(C) checks with an explicit custom-allocator function-pointer size check.
The capped C-integrated build and original test suite remain deferred to the
main worker, which must run them serially under the repository's 40 GiB virtual
memory limit.
Test Plan:
- `git diff --cached --check` -- passed.
- `rustfmt --check rust/src/zstd_compress_dictionary.rs` -- passed.
- Capped cargo, make, native, fuzzer, and full-suite verification -- not run
by design; deferred to the main worker.
The Fast dictionary-table projection was populated before the Rust
orchestrator ran its window-update and match-state publication callbacks.
That left base and nextToUpdate pointing at the initial sentinel for MT
prefix jobs, which caused ZSTD_rust_fillHashTable to dereference an invalid
range. Project pointers to the live match-state fields so Rust reads the
same values the former C adapter would have read at the Fast branch.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; make -j1
- capped 4 MiB -T0 compress/decompress round trip with cmp
Keep the MT sequence-state projection helper available to the C ABI
orchestration while excluding it from the standalone Rust test target,
where the production entry point is intentionally not compiled. This
keeps the capped Rust test check warning-free without changing runtime
behavior or the C layout contract.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- git diff --check
The MT worker previously kept the external-sequence non-empty gate and LDM
invariant assertion in C, bundled with the private CCtx operation. Project the
CCtx pointer and raw-sequence store through a layout-checked seam so Rust owns
the gate and the apply point while C retains only the opaque
ZSTD_referenceExternalSequences() leaf. Preserve the callback ordering after
worker CCtx initialization and keep empty stores as a no-op.
Test Plan:
- git diff --cached --check -- passed before commit.
- Static rg inspection confirmed all projection initializers include the new
sequence-state pointer and no old serialState_applySequences symbol remains.
- Cargo, make, compiler checks, and full tests were not run per worker scope.
The dictionary-content orchestrator still routed the Fast hash-table fill
through a C callback even though the matching leaf already lived in Rust.
Replace that callback slot with a pointer-sized C/Rust projection containing
only the Fast leaf inputs, including a by-value copy of nextToUpdate and the
normalized CDict/full-load flags. Keep the dtlm/tfp invariant assertion in C
and retain all bridge offsets and the double-Fast callback.
A broader MatchState projection would expose private layout and adding fields
would change the established 34-word bridge. The narrow projection keeps
that ABI stable while making the Fast branch call ZSTD_rust_fillHashTable
directly.
Test Plan:
- `git diff --cached --check`
- `rustfmt --check --edition 2021 rust/src/zstd_compress_dictionary.rs` (reports only pre-existing formatting drift outside this change)
- Cargo, make, and full tests intentionally not run per task instructions.
The multi-file compression and decompression entry points already delegate
file iteration and resource-sensitive work to Rust, but each still kept the
final summary decision and progress-to-summary ordering in C. Move that
scalar policy into a Rust ABI function. The C side now supplies only exact
legacy display callbacks and unchanged format strings, while Rust receives
plain counters and a summary kind without exposing FIO, cRess, dRess, or
codec layouts. Focused unit tests cover both summary modes, callback order,
argument forwarding, and sessions that must not display a summary.
Test Plan:
- `git diff --check -- programs/fileio.c rust/src/fileio_asyncio.rs` -- passed
- `git diff --cached --check` -- passed
- `rustfmt --check --edition 2021 rust/src/fileio_asyncio.rs` -- parsed the file;
reports pre-existing formatting differences elsewhere in the file, so no
unrelated formatting was applied
- Cargo, make, and runtime tests were intentionally not run per worker scope
The DDict implementation already owned the decoder-table loader in
zstd_decompress.rs, but zstd_ddict.rs still declared it as an external C
helper and disabled that path in Rust unit-test builds. Call the Rust
implementation directly through the ABI-compatible entropy-table
projection and make the module documentation match the actual ownership.
Keep offset_of imported only for the non-test C-context projection so the
Rust test check remains warning-free.
Test Plan:
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- Standalone cargo test was attempted; linking remains blocked by the
existing C context-projection symbols used by the hybrid test crate.
The MT serial turn already belongs to Rust for turn selection, checksum
placement, and advance. The remaining LDM callback still bundled the private
window update, sequence generation, and ldmWindow publication behind one C
operation, which hid that ordering from Rust.
Expose those three private operations as separate C callbacks and invoke them
from Rust in the original order. The first callback retains the original
RawSeqStore and job-size assertions before updating the LDM window; sequence
generation keeps its error assertion; and publication keeps the
ldmWindowMutex lock, copy, signal, and unlock sequence. The serial mutex
remains held until the existing advance callback, and disabled-LDM, empty
input, and skipped-turn behavior remain unchanged.
Test Plan:
- `git diff --cached --check` -- passed.
- Cargo, make, and full tests were not run per request; run the capped serial
verification suite before relying on runtime coverage.
Keep weak-symbol discovery and private dctx/ddict extraction in the C adapter,
but project the exact trace callback types into Rust and invoke them from the
Rust trace lifecycle. Preserve the nullable-hook behavior, trace context
width, dictionary-cold timing, record field layout, size conversions, and the
no-trace build path. Add focused Rust policy tests for callback, record, and
null-projection behavior.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1 -C tests test
The row-matchfinder reset path only used its C callback to read the private
hashSaltEntropy scalar and invoke Rust. Replace that callback slot with the
existing 32-bit scalar, keep the ABI offsets pointer-sized on 32-bit and
64-bit targets, and update the salt directly in Rust at the row-table reset
point. Preserve the C workspace callbacks and verify the real salt transition
in the reset-policy test.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1 -C tests test
The CCtx reset tail already delegated storage reservation and ordering policy
to Rust, but crossed back into C through a redundant callback trampoline.
Project the storage state directly through the existing pointer-sized ABI slot
and invoke the Rust storage operation without the extra C hop. Keep private
workspace/layout callbacks and match-state reset in C, preserve hash -> match
-> storage ordering, and retain allocation-error propagation in Rust tests.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
Keep the compression context extraction and the weak trace hook adapter in C,
where the private CCtx layout and platform-specific tracing declarations remain
available. Project only the scalar trace inputs and callback through an explicit
ABI-checked boundary so Rust owns the enabled check, stable record layout,
version field, compression-only null fields, and end-of-trace policy. The C
entry point still clears traceCtx after Rust returns, preserving the old
single-use lifecycle and keeping trace-disabled builds free of weak-hook calls.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; make -j1 -C tests test
Move buffered-input presence, offset subtraction, and block-size validation into
Rust while keeping the private MT progression query behind its C callback. The
C entry point now projects only the consumed/produced counters, input-buffer
fields, and worker dispatch inputs. Rust preserves the null-buffer behavior and
aborts on the same impossible offset or oversized-buffer invariants as the
original C assertions.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Move the serial LDM dictionary preload policy into Rust. The C MT adapter now
exposes only the private window and hash-table operations through checked
callbacks, while Rust owns empty/raw-content eligibility, loaded-dictionary
reset ordering, and the force-window publication rule. The explicit projection
keeps the dictionary inputs and callback ABI auditable without exposing the
private SerialState layout.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; make -j1 -C tests test
Move the fileio source-size precedence policy into the Rust backend while
keeping the private CCtx pledge call, error handling, and diagnostics in the
C adapter. A known statted source size, including zero, remains authoritative;
when it is unavailable, a positive declared stream size is used, otherwise the
zstd unknown-content sentinel is preserved. The Rust policy accepts both
sentinels explicitly so the bridge does not couple their representations.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; make -j1 -C tests test
The CCtx reset orchestration already runs in Rust, but its C workspace
adapter still decided when capacity was too small or wasteful and maintained
the oversized-duration counter through C-only predicates. Move those factor,
threshold, overflow, and duration-update rules into Rust. C now supplies only
private workspace metrics and retains low-level allocation, reservation, and
layout operations.
Remove the obsolete C predicates so the workspace resize decision has one
Rust-owned implementation and remains covered at its boundary conditions.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; make -j1 -C tests test
The local-dictionary callback still contained the policy that selected
by-reference versus by-copy loading, rejected internal copies for static
CCtx instances, allocated and copied dictionary bytes, and published the
resulting local-dictionary state. Move that policy into the Rust compression
rewrite so the C callback is only an adapter around the private CCtx and
localDict layouts.
The adapter retains the C custom allocator and output slots, preserving the
existing ownership and free path while keeping the ABI boundary explicit.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -B -j1 -C tests fuzzer
- (cd tests && ulimit -v 41943040; ./fuzzer -v -i1 -s6572)
- ulimit -v 41943040; make -j1 -C tests test
Keep the migration map aligned with the completed adaptive-refresh and window
overflow seams. The CLI description now identifies only the remaining C
progression and diagnostic callbacks, and the compression description records
that Rust owns the stateful window rebase. The explicit list of private C
layout and callback boundaries remains unchanged so the hybrid build is not
mistaken for a finished rewrite.
Test Plan:
- git diff --cached --check
The overflow-correction policy and scalar correction arithmetic were already
Rust-owned, but the stateful window rebase still lived in the C
ZSTD_window_correctOverflow() inline. That left pointer advancement, limit
clamping, and the correction counter as an untracked production C algorithm
on every compression path that protects matchfinder indices.
Replace that inline with a five-pointer projection. C continues to compute the
private source index and choose the correction context, while Rust now owns
the rebasing transition: it advances the regular and dictionary bases, clamps
both limits with the original wrapping U32 behavior, checks the correction
invariants in debug builds, and increments the overflow-correction counter.
The C/Rust projection has explicit layout assertions, and the Rust fixture
covers the pointer, limit, and counter updates together.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
The adaptive compression loop still delegated its refresh clock gate to a
small C callback even though Rust already owned the iteration state and all
adaptive policy decisions. That left timing policy, the last-refresh scalar,
and one projection callback in the C-side orchestration boundary.
Move the one-sixth-second monotonic refresh gate into ZstdAdaptiveState. The
Rust loop now initializes the first refresh timestamp at frame start and
preserves the C callback's strict-greater-than interval check. C retains only
the private progression, parameter-setting, and diagnostic callbacks needed
by the existing file I/O context.
Shrink both sides of the projection together and keep compile-time offset and
size assertions aligned. The adaptive unit fixture now verifies Rust records
the refresh event while continuing to exercise the existing progression and
policy callbacks.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
The default zstd frame loop was already implemented in Rust, but its
completion and error-result policy still lived in the C callback wrapper.
That left the C translation unit deciding when a decoded size was valid and
when a decoding or premature-end status had to become the historical frame
sentinel. Keep the exact operator-facing diagnostics and the private
asynchronous-resource pointers in C, while giving Rust ownership of the
status-to-result policy and callback ordering.
The new C/Rust policy record has compile-time layout assertions on both sides,
and the Rust fixture covers success, decode error, premature end, sentinel
results, and callback sequencing. This keeps the bridge explicit without
making Rust depend on the private dRess_t layout.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
Move the scalar portion of the private CCtx reset tail behind the existing
Rust reset adapter. The C side now publishes explicit pointers to the fields
that Rust must initialize, while retaining workspace ownership, match-state
reset, storage reset, and the XXH64 reset callback that cannot cross the
boundary safely. This removes the last opaque C initializer from this reset
phase without exposing the private ZSTD_CCtx layout to Rust.
The ABI record has compile-time offset and size assertions on both sides, and
the Rust unit fixture now checks the resolved compression parameters,
prefetch policy, pledged-size sentinel handling, counters, content-size flag,
block-size limit, stage, and dictionary metadata. The callback-order tests
continue to verify that compressed-block state reset precedes match-state and
storage cleanup, and that failures stop the tail at the correct callback.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
Move the per-file and multi-file --list row formatters out of programs/fileio.c. C still owns filesystem access, frame analysis, private fileInfo_t storage, and exact status diagnostics; Rust now formats the rows into byte buffers and sends them through a synchronous output callback. This keeps filenames byte-preserving and prevents the private C record from crossing the ABI while preserving the original display thresholds, human-readable size policy, unavailable-content spacing, checksum order, and total-row behavior.
Replace the old display callbacks with one size-delimited output callback shared by the single-file and multi-file projections. Add explicit Rust ABI layout checks, formatter fixtures for normal/unavailable/verbose/checksum/total output, and callback-order assertions for the policy boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (208 passed)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test (all original tests completed successfully)
- git diff --check
Remove the C-only DDict offset globals and route production
ZSTD_copyDDictParameters() through the existing C-produced DCtx view. This
keeps the private decoder layout and optional fuzzing fields owned by the
canonical projection while preserving the DDict entropy, prefix, window, and
checksum state updates. Keep the synthetic offset writer test-only so the
Rust unit tests remain independently linkable.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (207 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
- git diff --check
Move the decompression resource creation order into the Rust CLI policy
layer: dictionary stat and patch-memory preparation, decoder allocation,
window/checksum configuration, dictionary reset/attachment, and asynchronous
pool creation now run through one Rust-owned sequence. Keep dRess_t,
stat_t, dictionary buffers, decoder context, pools, and exact allocation
errors in C callbacks.
Preserve the original patch-from and dictionary-reference behavior while
adding ABI layout assertions, scalar policy coverage, and error short-circuit
tests for the new boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (207 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Make the Rust policy layer own the ordered compression-resource lifecycle:
create the CCtx, prepare patch/dictionary state, create the write and read
pools, validate the dictionary, apply general and multithreaded parameters,
and finally load the dictionary. Keep cRess_t, FIO_Dict_t, file statistics,
AIO pools, and CLI diagnostics in C callbacks so the existing resource
ownership and error behavior remain local to the C backend.
Preserve adaptive window defaults and patch-from parameter adjustment while
adding ABI layout assertions and a lifecycle-order test for the new boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (205 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
The MT stream initializer previously exposed one C callback that waited for
all submitted jobs and then released their resources. That left the ordering
policy in C even though Rust already owned the surrounding initialization
sequence and the ring-order wait/release policies.
Expose separate wait-all and release-all callbacks at the Rust/C boundary and
invoke them in Rust in the original wait-before-release order. C retains the
worker synchronization, job descriptors, input buffer, and resource-pool
side effects behind the two callbacks, while the focused Rust initializer test
now makes the ordering explicit.
Test Plan:
- capped cargo +nightly fmt for the Rust workspace -- passed
- capped root cargo clippy --all-targets -- -D warnings -- passed
- capped serial make -j1 for the single-threaded library, MT library, and CLI -- passed
- git diff --cached --check -- passed
Transparent stream initialization can resolve an unknown source size to the
multithreaded path and update appliedParams.nbWorkers. The Rust stream2
fallback previously captured that value before initialization, then entered the
serial generic adapter while the context stage was still created. Unknown-size
stdin compression consequently returned StageWrong instead of producing a
frame.
Keep the worker-mode projection live through initialization so Rust observes
the applied value at the existing MT-versus-serial decision point. The ABI
layout assertions now cover the pointer projection, and a focused regression
test verifies that an initialization callback which selects MT dispatches the
MT step and preserves the expected callback order.
Test Plan:
- capped cargo +nightly fmt for the Rust workspace -- passed
- capped root and CLI cargo clippy with -D warnings -- passed
- capped serial make -j1 for single-threaded library, MT library, and CLI -- passed without the prior pointer-sign warning
- capped stdin compression/decompression round trip for unknown-size input -- passed
- capped original suite, including CLI tests, native tests, fuzzer, and both zstream phases -- passed; the final signedness-only ABI spelling cleanup was followed by a clean capped rebuild
- targeted root cargo test attempted but remains blocked at link time by pre-existing decompression bridge symbols (ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and ZSTD_rust_block_context_init)
FIO_openDstFile() still owned the complete destination policy loop in C even
though the filesystem opener and status/action classifier were already Rust
leaves. That left confirmation, sparse-mode adjustment, overwrite removal, and
retry ordering duplicated beside private FILE* and CLI state.
Project the C-owned callbacks for diagnostics, preference mutation, user
confirmation, stdout setup, and existing-file removal. Rust now owns the
retry/order state machine and calls the existing narrow opener for each
attempt; C keeps private preferences and context, exact diagnostics, and FILE*
ownership. The callback layout is asserted on both sides, and focused tests
cover status ordering, confirmation/removal retry, setvbuf preservation, and
invalid policy inputs.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1` -- passed, including the single-thread library, MT library, and CLI binary.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- passed: 206 tests.
- `git diff --cached --check` -- passed.
The public ZSTD_compressStream2_c entry point still contained the complete
fallback state machine even though its validation, initialization policy,
serial-versus-MT dispatch, and result accounting were already represented by
Rust policy helpers. That left the main streaming entry point as a large C
orchestration island and made the Rust rewrite boundary misleading.
Project the scalar stream state and private CCtx operations through a checked
C ABI, then let Rust own the fallback ordering. Rust now validates buffers and
the end directive, handles transparent one-shot completion and stable-input
deferral, performs the initialization and stability decisions, selects the
serial or MT path, and publishes the result policy. C retains the private
context layout, stream adapter, MT operations, checksum/epilogue side effects,
and trace/reset callbacks. ABI offset assertions and focused callback-order
and stable-input tests protect the projection without requiring a standalone
Rust test binary to link every C bridge symbol.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed before and after formatting.
- `git diff --cached --check` -- passed.
- Native `make -j1` and the original suite remain the next post-commit verification gates.
Project the private requested-parameter and context state into a fixed scalar ABI record, leaving only C's parameter-selection leaf and private layout access in the shim. Rust now owns the complete-input eligibility predicate, including advanced-state exclusions, LDM sentinel handling, and the final fast/double-fast strategy gate, while preserving the requested-level and INT_MIN results.
Add focused policy tests for fast and double-fast acceptance, representative advanced-state rejection, the accepted LDM sentinel, non-fast strategies, and null state.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
Document Rust ownership of compression-size estimator policy and the CLI adaptive feedback state machine, and describe the private C callbacks that remain at those boundaries.
Test Plan:
- Documentation-only change; git diff --cached --check
Move the multithreaded compressStream2 outer coordinator into the Rust compression module. The C bridge now owns only the private ZSTDMT call and consumed/produced counters, while Rust preserves the progress loop, completion ordering, and error handoff.
Keep the MT context mutations, diagnostics, assertions, and buffer-expectation publication in C because they depend on the private context layout. Add callback-driven tests for continue/break, error completion, end trace/reset ordering, and malformed states.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
Clippy rejects the hand-written round-up expression used to compute the callback-alignment offset in the Rust file-I/O projection. Use the equivalent div_ceil spelling so the ABI assertion remains clear and the root Rust targets pass with warnings denied.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
The zstd file-I/O loop still kept adaptive compression's state machine in C: progression deltas, refresh gating, job-completion checks, input counters, speed decisions, and level updates were interleaved with private FIO and ZSTD state. That left orchestration policy behind the existing scalar Rust predicates.
Move the adaptive state and callback order into Rust. C now supplies scalar progression snapshots, the clock gate, exact diagnostics, and the ignored CCtx parameter setter through callbacks; private FIO_prefs_t, ZSTD_CCtx, ZSTD_frameProgression, clocks, and progress formatting remain C-owned. Preserve the previous-progression publication before backlog evaluation, input counter reset points, and serial/MT level-clamp behavior.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
The public CCtx and CStream size-estimation APIs still selected row-matchfinder modes and walked compression levels in C, leaving a policy-level implementation behind the Rust workspace-sizing bridge. That also made the C layer responsible for the subtle raw-size_t MAX behavior used when an estimator returns an encoded error.
Move the row-mode selection and shared monotonic memory-budget policy into Rust. Keep the C callbacks responsible for constructing private ZSTD_CCtx_params values and for the distinct CCtx/CStream sizing formulas, so no private parameter layout crosses the ABI. Preserve the signed MIN(compressionLevel, 1) start and raw maxima exactly.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Update the migration boundary document to record the two latest multithreaded
policy moves. Rust now owns serial-reset LDM normalization, table sizing, and
publication plus final completion trace/reset ordering, while C still owns the
private job/context state and callbacks.
Test Plan:
- `git diff --cached --check` -- passed.
- Capped native build and full original test target were run before this
documentation-only commit and passed.
The multithreaded stream loop already delegated scalar result classification to
Rust, but its final error/end conditional still mixed completion ordering with
the private C context. That left trace and reset sequencing embedded beside the
MT counters and made the next Rust seam depend on exposing CCtx layout.
Rust now consumes the projected loop action and sequences the final completion
side effects through opaque C callbacks. Continue and break do nothing, worker
errors reset the session, and completed end operations trace before resetting.
The reset callback result is intentionally discarded. C retains the private
context mutations in callback adapters, and the callback invocation remains
inside the existing ZSTD_MULTITHREAD branch before FORWARD_IF_ERROR(flushMin).
C/Rust repr(C) projections and ABI assertions protect the callback/result
boundary, while focused tests cover all four loop actions and end ordering.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_compress.rs` -- passed
- single-thread and `-DZSTD_MULTITHREAD` C syntax checks -- passed; existing warnings only
- `CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed
- `CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --benches -- -D warnings` -- passed
- focused Rust test filter was attempted serially but is blocked by unrelated in-progress `rust/src/zstdmt_compress.rs` test ABI edits; no full test suites were run
Move the multi-input single-output decision sequence out of
FIO_multiFilesConcatWarning while preserving the C-owned CLI boundary. The
previous wrapper classified fatal remove cases, emitted the concatenation
warning, disabled --rm, reclassified the action, and then selected quiet
abort or confirmation in C. Add a repr(C) callback projection so Rust owns
only that scalar ordering while C continues to own exact diagnostics, the
confirmation prompt, FIO_prefs_t mutation, and all private state.
The Rust bridge re-runs the action after the C disable-remove callback with
the same has-output/remove arguments as the original wrapper. C and Rust
assert the callback layout, and focused tests cover callback order,
fatal/quiet paths, and ABI offsets. Existing scalar action tests remain in
place.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -B -C programs -j1 fileio.o`
— passed; only pre-existing suffixList C++-compat warnings appeared.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path
rust/cli/Cargo.toml --tests --no-deps` — passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 RUSTFLAGS='-C
link-arg=/tmp/zstd_rust_test_bridges.o' cargo test --manifest-path
rust/cli/Cargo.toml --lib 'fileio_prefs::tests::multi_files_concat_'`
— 8 passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo build --manifest-path
rust/cli/Cargo.toml --lib` — passed; the archive exports
`FIO_rust_multiFilesConcatWarning`.
- `cargo +nightly fmt --manifest-path rust/cli/Cargo.toml -- --check` — not
clean due pre-existing formatting drift in unchanged Rust code; no bulk
formatting was applied.
- Full native/upstream/fuzzer suites were not run by request.
ddidderr