The MT stream initializer previously exposed one C callback that waited for
all submitted jobs and then released their resources. That left the ordering
policy in C even though Rust already owned the surrounding initialization
sequence and the ring-order wait/release policies.
Expose separate wait-all and release-all callbacks at the Rust/C boundary and
invoke them in Rust in the original wait-before-release order. C retains the
worker synchronization, job descriptors, input buffer, and resource-pool
side effects behind the two callbacks, while the focused Rust initializer test
now makes the ordering explicit.
Test Plan:
- capped cargo +nightly fmt for the Rust workspace -- passed
- capped root cargo clippy --all-targets -- -D warnings -- passed
- capped serial make -j1 for the single-threaded library, MT library, and CLI -- passed
- git diff --cached --check -- passed
Transparent stream initialization can resolve an unknown source size to the
multithreaded path and update appliedParams.nbWorkers. The Rust stream2
fallback previously captured that value before initialization, then entered the
serial generic adapter while the context stage was still created. Unknown-size
stdin compression consequently returned StageWrong instead of producing a
frame.
Keep the worker-mode projection live through initialization so Rust observes
the applied value at the existing MT-versus-serial decision point. The ABI
layout assertions now cover the pointer projection, and a focused regression
test verifies that an initialization callback which selects MT dispatches the
MT step and preserves the expected callback order.
Test Plan:
- capped cargo +nightly fmt for the Rust workspace -- passed
- capped root and CLI cargo clippy with -D warnings -- passed
- capped serial make -j1 for single-threaded library, MT library, and CLI -- passed without the prior pointer-sign warning
- capped stdin compression/decompression round trip for unknown-size input -- passed
- capped original suite, including CLI tests, native tests, fuzzer, and both zstream phases -- passed; the final signedness-only ABI spelling cleanup was followed by a clean capped rebuild
- targeted root cargo test attempted but remains blocked at link time by pre-existing decompression bridge symbols (ZSTD_rust_dctx_trace_view, ZSTD_rust_dctx_view, and ZSTD_rust_block_context_init)
The public ZSTD_compressStream2_c entry point still contained the complete
fallback state machine even though its validation, initialization policy,
serial-versus-MT dispatch, and result accounting were already represented by
Rust policy helpers. That left the main streaming entry point as a large C
orchestration island and made the Rust rewrite boundary misleading.
Project the scalar stream state and private CCtx operations through a checked
C ABI, then let Rust own the fallback ordering. Rust now validates buffers and
the end directive, handles transparent one-shot completion and stable-input
deferral, performs the initialization and stability decisions, selects the
serial or MT path, and publishes the result policy. C retains the private
context layout, stream adapter, MT operations, checksum/epilogue side effects,
and trace/reset callbacks. ABI offset assertions and focused callback-order
and stable-input tests protect the projection without requiring a standalone
Rust test binary to link every C bridge symbol.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed before and after formatting.
- `git diff --cached --check` -- passed.
- Native `make -j1` and the original suite remain the next post-commit verification gates.
Project the private requested-parameter and context state into a fixed scalar ABI record, leaving only C's parameter-selection leaf and private layout access in the shim. Rust now owns the complete-input eligibility predicate, including advanced-state exclusions, LDM sentinel handling, and the final fast/double-fast strategy gate, while preserving the requested-level and INT_MIN results.
Add focused policy tests for fast and double-fast acceptance, representative advanced-state rejection, the accepted LDM sentinel, non-fast strategies, and null state.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
Move the multithreaded compressStream2 outer coordinator into the Rust compression module. The C bridge now owns only the private ZSTDMT call and consumed/produced counters, while Rust preserves the progress loop, completion ordering, and error handoff.
Keep the MT context mutations, diagnostics, assertions, and buffer-expectation publication in C because they depend on the private context layout. Add callback-driven tests for continue/break, error completion, end trace/reset ordering, and malformed states.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
The public CCtx and CStream size-estimation APIs still selected row-matchfinder modes and walked compression levels in C, leaving a policy-level implementation behind the Rust workspace-sizing bridge. That also made the C layer responsible for the subtle raw-size_t MAX behavior used when an estimator returns an encoded error.
Move the row-mode selection and shared monotonic memory-budget policy into Rust. Keep the C callbacks responsible for constructing private ZSTD_CCtx_params values and for the distinct CCtx/CStream sizing formulas, so no private parameter layout crosses the ABI. Preserve the signed MIN(compressionLevel, 1) start and raw maxima exactly.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
The MT serial-reset bridge now declares its Rust scalar projection before
executing the diagnostic branch. This keeps the declaration ordering valid for
the C90-oriented compiler flags used by the native test builds while leaving
the Rust-owned policy and callback sequence unchanged.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1` -- passed without the
new mixed-declaration warning.
- `git diff --cached --check` -- passed.
The multithreaded stream loop already delegated scalar result classification to
Rust, but its final error/end conditional still mixed completion ordering with
the private C context. That left trace and reset sequencing embedded beside the
MT counters and made the next Rust seam depend on exposing CCtx layout.
Rust now consumes the projected loop action and sequences the final completion
side effects through opaque C callbacks. Continue and break do nothing, worker
errors reset the session, and completed end operations trace before resetting.
The reset callback result is intentionally discarded. C retains the private
context mutations in callback adapters, and the callback invocation remains
inside the existing ZSTD_MULTITHREAD branch before FORWARD_IF_ERROR(flushMin).
C/Rust repr(C) projections and ABI assertions protect the callback/result
boundary, while focused tests cover all four loop actions and end ordering.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_compress.rs` -- passed
- single-thread and `-DZSTD_MULTITHREAD` C syntax checks -- passed; existing warnings only
- `CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed
- `CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --benches -- -D warnings` -- passed
- focused Rust test filter was attempted serially but is blocked by unrelated in-progress `rust/src/zstdmt_compress.rs` test ABI edits; no full test suites were run
ZSTD_compressBegin_internal already routes attach-versus-reload policy through
Rust, but its C insertion callback still inspected a non-null ZSTD_CDict and
selected dictContent, dictContentSize, and dictContentType. That left source
selection coupled to the private CDict layout and kept the Rust bridge from
receiving the same selected source for direct and prepared dictionaries.
Project the three CDict content fields into the repr(C) begin state. Rust now
selects either those fields or the direct dictionary arguments before invoking
the C callback. The callback keeps the private CCtx insertion path and no
longer branches on CDict. ABI offsets and sizes are asserted on both sides,
with focused by-reference and forced-CDict tests covering pointer, size, type,
reset order, and dictionary-result publication.
Test Plan:
- `cargo check --manifest-path rust/Cargo.toml --lib --tests` -- passed under
`ulimit -v 41943040; CARGO_BUILD_JOBS=1`.
- Serial GCC/Clang syntax-only checks for `lib/compress/zstd_compress.c` --
passed under the same cap; only existing warnings were reported.
- `cargo clippy` for lib, benches, and tests with `-D warnings`, plus nightly
format check -- passed serially under the same cap.
- `cargo test --manifest-path rust/Cargo.toml --lib zstd_compress_dictionary
-- --test-threads=1` -- compiled but standalone linking failed on the three
pre-existing C bridge symbols `ZSTD_rust_dctx_trace_view`,
`ZSTD_rust_dctx_view`, and `ZSTD_rust_block_context_init`.
- Full native, upstream, and fuzzer tests were not run per scope.
CDict initialization still hard-coded the match-state reset policies in its
C callback, even though Rust already owned the surrounding content branch,
workspace reservation, and callback ordering. That left the reset target and
cleanup policy split across the language boundary and made the intended CDict
reset contract implicit.
Extend the existing repr(C) reset callback with the three private enum values.
Rust now selects make-clean, index-reset, and the CDict reset target before
calling the opaque C operation. C retains the private ZSTD_CDict, workspace,
and match-state layouts and only casts the projected values for
ZSTD_reset_matchState(). Focused by-reference and by-copy probes record the
values and preserve reserve, reset, and insert ordering.
Test Plan:
- `rustfmt --check --edition 2021 rust/src/zstd_compress_dictionary.rs` -- passed.
- GCC and Clang syntax-only checks for `lib/compress/zstd_compress.c` under
`ulimit -v 41943040` -- passed.
- `cargo check --manifest-path rust/Cargo.toml --lib --tests` under the cap -- passed.
- Serial `cargo clippy` lib, benches, and tests passes with `-D warnings`,
plus `cargo +nightly fmt --all -- --check` -- passed.
- `cargo test --manifest-path rust/Cargo.toml --lib zstd_compress_dictionary
-- --test-threads=1` compiled but could not link the standalone test binary
because three pre-existing C bridge symbols are unavailable outside the
native harness; no native, upstream, or fuzzer tests were run.
Move the Rust teardown projection declaration before the null guard so the MT source remains clean under the repository's C90 declaration rules. The initializer only copies pointers, scalars, the allocator, and the callback; the null guard still precedes the bridge call and preserves the existing no-op behavior for an absent job table.
Test Plan:
- git diff --cached --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test (passed before this warning-only fix)
Normal MT context teardown still let the C wrapper directly sequence
per-job synchronization destruction before Rust released the opaque job-table
storage, while failed create and expansion transactions already used a Rust
helper for that same order. Route the normal teardown through a repr(C)
projection so Rust owns the destroy-before-storage-free policy consistently.
The synchronization callback, descriptor storage, and custom allocator remain
opaque C-owned operations. Add matching C/Rust layout assertions and a focused
callback-order test.
Test Plan:
- `git diff --cached --check` -- passed
- Cargo, make, native tests, and heavy verification were not run per request
ZSTD_compress_advanced_internal() still encoded the one-shot begin-then-end
sequence in C after public advanced parameter validation had moved behind Rust.
That left the public compression boundary split between Rust policy and a C
orchestration body, and made the begin-error ordering implicit in the C path.
Add an explicit C-layout state with opaque begin and end callbacks. Rust now
owns the ordering, forwards the source size as the pledged size, propagates a
begin error before invoking the end callback, and returns the end result. The
callbacks retain the private ZSTD_CCtx and ZSTD_CCtx_params layouts, so the
change moves policy and sequencing without exposing window, workspace,
matchfinder, or context internals to Rust. The existing advanced and
using-dictionary wrappers continue to use the same C private operations.
Test Plan:
- `git diff --cached --check` -- passed.
- `rustfmt --check --edition 2021 rust/src/zstd_compress.rs` -- passed.
- Not run: Cargo, make, native tests, builds, or heavy verification per request.
Keep the MT frame-progression callback under the ZSTD_MULTITHREAD preprocessor guard using an ordinary C declaration. The previous warning-cleanup edit accidentally prefixed the declaration with a preprocessor marker, which broke the multithreaded compilation path.
Test Plan:
- git diff --cached --check
- Capped native rebuild failed before this fix at the invalid directive; rerun pending
The MT frame-progression callback is only referenced when ZSTD_MULTITHREAD is enabled. Keep its definition under the same configuration guard so single-threaded library, test, and decode-corpus builds do not report an unused static function while the MT callback remains available to the Rust dispatch bridge.
Test Plan:
- git diff --cached --check
- Capped full tests passed before this warning-only guard
- Capped native rebuild and smoke rerun pending
The Rust-owned frame-progression dispatch invokes the existing C MT progression API through a mutable opaque callback context. Match that API's established signature in the adapter so the new boundary does not introduce a discarded-const qualifier warning while preserving the private MT context.
Test Plan:
- git diff --cached --check
- Capped native make rerun pending after this warning-only fix
Move the public ZSTD_getFrameProgression MT-versus-single-thread worker-count
branch into a Rust-owned scalar projection. C continues to compute the
single-thread scalar inputs and keeps the private ZSTDMT context behind a
callback; the compile-time ZSTD_MULTITHREAD adapter remains local to C. Add
C/Rust layout assertions and focused Rust tests for both dispatch paths and
null-state fallback.
Test Plan:
- `git diff --cached --check`
- Not run: Cargo, make, builds, native tests, and heavy verification per request.
Keep the opaque ZSTD_CCtx_params copy operation in C, but make Rust own the public advanced2 null-source validation and copy-before-preparation ordering. The bridge now rejects a missing source before any allocator or workspace callback, with explicit ABI assertions and focused order/null-input tests.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Keep the advanced CDict estimator and its private workspace sizing inputs in C, but route the public estimate through a Rust policy bridge. Rust now selects unknown-source/create-CDict parameters and the public by-copy load mode before invoking the opaque C estimator. Focused tests verify callback ordering, selected parameters, and the missing-policy-input guard.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Keep the C-owned compression job callback responsible for private buffer and job-state mutation, but return the range-active decision alongside its result. Rust now owns the outer scheduler policy that publishes or suppresses the reusable input count after job creation, including the error path. The focused seam test covers the callback-clears-range case without exposing the MT context layout.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Move the ready/wrap branch and synchronization ordering from ZSTDMT_tryGetInputRange into Rust. The Rust policy now waits for the prefix range, invokes the C-owned prefix move, waits for the selected source range, and publishes the buffer; C retains memmove, private buffer fields, round-buffer state, and LDM callbacks behind the projection. Add ABI assertions and focused wrapped/ordinary ordering tests.
Test Plan: git diff --cached --check; focused Rust tests added; full capped Rust/native/original-suite verification follows.
Move the post-call ZSTD_compressStream2_c policy into Rust: adapter errors now short-circuit before buffer publication, successful calls publish buffer expectations before calculating pending output, and the private C context remains behind a callback projection. Add ABI assertions and ordering tests for both paths.
Test Plan: git diff --cached --check; focused Rust tests added; full capped Rust/native/original-suite verification will run after the MT slice is integrated.
The multithreaded branch of `ZSTD_compressStream2_c` performed the complete
post-call result classification in C: it prioritized errors, recognized a
completed end directive, and applied different progress rules for continue,
flush, and end operations. Those decisions were scalar policy around a C-owned
MT call, but remained embedded beside private counters, reset, and tracing.
Rust now classifies one projected iteration and returns an explicit action for
continue, break, error, or completed end. The C shim still owns the MT call,
input/output accounting, error forwarding, private context mutation, reset,
and trace callback; the existing condition ordering and progress comparisons
are preserved. ABI layout assertions and focused tests cover error precedence,
completed end, progress-based continue termination, and pending/full output.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/zstd_compress.rs` -- passed
- Full capped Rust/native verification is the next serial step.
ZSTDMT_updateCParams_whileCompressing already delegated parameter derivation to Rust, but C still owned the transition publication order and independently updated compressionLevel. Return a Rust-owned scalar result that carries the requested level with the derived compression parameters, while preserving the active frame window and all private MT context state in C.\n\nThe new projection and result have explicit C/Rust layout assertions. Focused Rust tests exercise unknown-size and explicit-size hints, LDM and override inputs, saved-window restoration, requested-level forwarding, and the exported ABI wrapper.\n\nTest Plan:\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt/clippy gates passed before staging\n- git diff --cached --check\n- Full capped native and original-test verification follows after the batch is committed.
ZSTD_compressStream2_c still owns private context mutation, diagnostics, buffer updates, and codec initialization, but its stable-input transparent-initialization decision was a remaining scalar policy island in C. Move the continuation validation, block-size threshold, format-specific progress hint, and wrapping size arithmetic behind a Rust projection. Preserve the original C error messages and return values while keeping all private context state on the C side.\n\nThe bridge uses explicit ABI layout assertions for the mixed pointer/size projection. Focused Rust tests cover buffered and non-continue calls, block-boundary initialization, empty and short stable input, both frame formats, matching continuation state, invalid source/position, and null bridge input.\n\nTest Plan:\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings\n- Full capped native and original-test verification follows after the batch is committed.
The MT stream initializer and its later dictionary update both encoded the
same three-way choice in C: copy a supplied dictionary, attach a borrowed
CDict, or install a raw prefix. The old code also embedded the required
release/clear-before-attach ordering in each callback wrapper. Project the
presence and raw-content flags into Rust, where the branch and ordering are
now explicit and tested. C retains only the private CDict allocation,
prefix-storage, context publication, and destruction callbacks, so the
configured C layouts and allocator behavior remain unchanged.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- `git diff --check` and `rustfmt --edition 2021 --check` -- passed before commit
- Capped native and original-test verification remains pending for the complete batch.
ZSTD_freeCCtx previously passed one C callback that bundled dictionary,
multithreaded context, workspace, and heap-object teardown. That left the
lifetime order inside private C control flow and made the public destruction
policy harder to audit. Project the four private operations separately and
let Rust enforce the original dictionary, MT-context, workspace, and
object order while preserving the static-context and workspace-embedded
object guards. The callbacks still own the C layouts and custom allocator
operations.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- Broader native and original-test verification remains pending for the complete batch.
Project the scalar ZSTD_compressStream2 end directive into Rust and keep C responsible for the private streaming context, existing parameter error, and diagnostic text. The Rust range policy preserves the original unsigned enum validation for continue, flush, and end.
Test Plan: ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check; make -j1; ./tests/rustLibSmoke; make -j1 -C tests test (all shell tests, large streaming tests, native tester, fuzzer phases, and zstream tester passed).
Move LDM block-compressor strategy validation, selection ordering, and callback error propagation into Rust while retaining dictionary-mode lookup, MatchState mutation, and the actual codec callback in C. Invalid preflight inputs now use the existing generic error path, and valid strategies preserve the original optimal-parser boundary.
Test Plan: ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check; make -j1; ./tests/rustLibSmoke; make -j1 -C tests test (all shell tests, large streaming tests, native tester, fuzzer phases, and zstream tester passed).
Make Rust the owner of the compression-job preparation sequence: publish the descriptor, reset the reusable input state, and publish terminal frame state last. Keep all MT job and stream layouts in C behind three focused callbacks, and preserve the nonterminal and terminal branches exactly.
Test Plan: ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check; GCC and Clang syntax-only checks; make -j1 -C tests test (all 41 shell tests, fuzzer, zstd tester, and zstream tester passed).
Keep the private static-CDict initializer in C, but return its native size_t status through the callback boundary. Rust now owns the public success-or-NULL mapping, preserving the original workspace validation, callback order, and private layout ownership while making initializer failures explicit and unit-testable.
Test Plan: ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check; GCC and Clang syntax-only checks; make -j1 -C tests test (all 41 shell tests, fuzzer, zstd tester, and zstream tester passed).
The resource-acquisition projection calls its field `ldmEnabled`, but the
first implementation passed the three-state `ZSTD_ParamSwitch_e` value
unchanged. `ZSTD_ps_disable` is 2, so Rust interpreted disabled LDM as enabled
and rejected ordinary jobs whose raw sequence store was intentionally absent.
That surfaced as an allocation error on the small stdin compression path.
Convert the C enum to the same boolean semantic used by the Rust policy before
crossing the ABI boundary. The projection declaration also now precedes the
callback setup statements, keeping the native C90 warning-free style.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `git diff --cached --check` -- passed
- The first capped full upstream run failed at `test-zstd` on the small stdin path with the enum/boolean regression; the suite will be rerun after this fix.
MT worker execution used to combine CCtx acquisition, raw-sequence-store
acquisition, destination-buffer allocation, frame-header publication, and the
LDM resource check in one C callback. That made the resource order and
short-circuit policy another C-owned implementation boundary even though the
operations themselves must remain private to C.
Project the scalar LDM and destination-presence inputs into Rust and let Rust
own the ordering and error normalization. The C callbacks now only acquire
private resources, report readiness, allocate the destination buffer, and
publish the frame-header destination. The ordering remains compatible with the
worker cleanup path: both pool gets happen first, CCtx failure stops before
destination work, destination failure stops before publication, and the LDM
sequence-store check runs after publication.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- Capped GCC and Clang syntax-only checks for `lib/compress/zstdmt_compress.c` -- passed by the worker
- `git diff --cached --check` -- passed
- Native build and full upstream tests are deferred to the post-batch serial verification.
CDict initialization already lets Rust choose by-reference versus by-copy
content, but the C bridge still selected the pointer-rounded content allocation
and the HUF entropy-workspace size. That kept advanced and static CDict
dictionary loading dependent on C policy even though the allocator itself can
remain private.
Compute the content reservation with the C-equivalent wrapping pointer
alignment in Rust and pass both that size and the HUF workspace size through the
existing callbacks. C now only forwards the sizes to
ZSTD_cwksp_reserve_object; private CDict/workspace layouts and allocator
behavior remain behind the ABI bridge. Focused probes cover alignment
boundaries, by-copy content, and both reservation sizes.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `ulimit -v 41943040; gcc -fsyntax-only -std=c99 -DXXH_NAMESPACE=ZSTD_ -DDEBUGLEVEL=0 -DZSTD_MULTITHREAD -DZSTD_LEGACY_SUPPORT=5 -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/deprecated lib/compress/zstd_compress.c` -- passed
- `ulimit -v 41943040; clang -fsyntax-only -std=c99 -DXXH_NAMESPACE=ZSTD_ -DDEBUGLEVEL=0 -DZSTD_MULTITHREAD -DZSTD_LEGACY_SUPPORT=5 -DZSTD_NO_ASM=1 -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/deprecated lib/compress/zstd_compress.c` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- Cargo builds/tests, make, fuzzers, and large upstream tests were not run per task constraints
- GPG signing was unavailable because the configured pinentry could not start;
this commit was created explicitly unsigned with `--no-gpg-sign`
The worker context previously selected the cdict or raw-prefix initialization
path, applied non-first-job parameter updates, and published the frame-header
projection directly in C. That left the branch and failure order intertwined
with private CCtx and parameter layouts.
Project only the first-job and cdict flags into Rust. Rust now validates cdict
placement, selects the initialization path, stops on force-window or
prefix-policy errors, and publishes the header projection only after successful
initialization. C callbacks retain the private cdict, CCtx, parameter mutation,
and frame-header field operations, including the original pledged-size and
force-window behavior.
Focused tests cover cdict ordering, non-first parameter ordering, parameter and
initialization failures, header-publication suppression, and invalid cdict
placement.
Test Plan:
- `rustfmt +nightly --edition 2021 --check rust/src/zstdmt_compress.rs` -- passed.
- Capped GCC syntax-only check for `zstdmt_compress.c` -- passed.
- Capped Clang syntax-only check for `zstdmt_compress.c` -- passed.
- `git diff --cached --check` -- passed.
- Cargo, native builds, fuzzers, and large tests were not run per assignment.
CDict initialization already routes content copying and dictionary insertion
through the Rust orchestrator, but its private C adapter still hard-coded the
full table-load method and the CDict table-fill purpose. That split left an
important advanced-CDict content-loading policy hidden in the C bridge and
made the callback contract less explicit.
Extend the private insertion callback with the two table-loading policy
values. Rust now selects the full-load and for-CDict modes after its content
branch and before invoking the opaque C operation. C retains only the private
CDict layout projection and forwards those selected values to the existing
content loader, preserving the original insertion order and behavior.
The focused initialization probes record both values in the callback and
assert them for by-reference and by-copy dictionaries alongside the existing
ordering and content-copy checks.
Test Plan:
- `rustfmt +nightly --edition 2021 --check rust/src/zstd_compress_dictionary.rs` -- passed
- `ulimit -v 41943040; gcc -fsyntax-only -std=c99 -DXXH_NAMESPACE=ZSTD_ -DDEBUGLEVEL=0 -DZSTD_MULTITHREAD -DZSTD_LEGACY_SUPPORT=5 -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/deprecated lib/compress/zstd_compress.c` -- passed
- `ulimit -v 41943040; clang -fsyntax-only -std=c99 -DXXH_NAMESPACE=ZSTD_ -DDEBUGLEVEL=0 -DZSTD_MULTITHREAD -DZSTD_LEGACY_SUPPORT=5 -DZSTD_NO_ASM=1 -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/deprecated lib/compress/zstd_compress.c` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- Cargo, make, native builds, fuzzers, and large tests were not run per task constraints
ZSTD_initStaticCDict previously resolved automatic row matching and computed the
minimum workspace in C before handing the result to the Rust construction
orchestrator. That left the public static-CDict sizing branch split from the
Rust validation and callback failure policy.
Make the Rust projection carry dictionary size, load method, compression
parameters, and the C-owned layout sizing inputs. Rust now resolves automatic
row matching, computes the dedicated-search workspace requirement, validates
workspace capacity, and passes the resolved mode into the initializer while
preserving create/reserve/move/init order. C retains opaque private cwksp and
CDict operations, allocator/layout behavior, and dictionary-content loading.
Add focused tests for computed sizing policy, capacity boundaries, callback
order, reservation and initialization failures, and null or unaligned
workspaces.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/zstd_compress_dictionary.rs` -- passed
- GCC `-fsyntax-only` checks with multithreaded, non-multithreaded,
`ZSTD_DISABLE_ASM=1`, and `ZSTD_ADDRESS_SANITIZER=1` configurations -- passed
- Clang `-fsyntax-only` check with `ZSTD_MULTITHREAD` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- Cargo, make, native builds, and large tests were not run per task constraints
The MT flush state machine already recognized worker and checksum errors in
Rust, but its single C error callback still bundled waiting for all workers
with releasing job resources. That kept teardown order in C and made the two
cleanup phases impossible to test independently.
Split the C callback into private wait and release leaves, then pass both to
Rust. Rust now owns the error cleanup sequence and always waits for workers
before releasing their resources on either error path. The MT context, job
resources, synchronization, and cleanup operations remain C-owned.
Test Plan:
- `rustup run nightly rustfmt --check --edition 2021 rust/src/zstdmt_compress.rs`
-- passed under the 40 GiB virtual-memory cap.
- `cc -fsyntax-only -std=c99` with the zstd include paths for
`lib/compress/zstdmt_compress.c` -- passed under the cap.
- `git diff --check` and `git diff --cached --check` -- passed.
- Cargo, Make, native builds, fuzzers, and large tests were not run per the
worker OOM and scope constraints.
Move static CCtx construction sequencing out of the monolithic C leaf. Rust
now validates the existing public workspace contract, sequences workspace
creation, CCtx and auxiliary reservations, publication, and BMI2 setup, and
short-circuits callback-reported failures. C callbacks retain the private
cwksp and CCtx layouts plus the native workspace-size and CPU-feature rules.
Add matching ABI layout assertions and focused callback-order, reservation-
failure, workspace-capacity, and NULL-path unit coverage. Static-CDict and
heap-CCtx paths remain unchanged.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/zstd_compress.rs` -- passed
- GCC and Clang `-fsyntax-only` checks on `zstd_compress.c` -- passed
- GCC syntax checks with `ZSTD_DISABLE_ASM=1` and
`ZSTD_ADDRESS_SANITIZER=1` -- passed
- `git diff --check` -- passed
- Cargo, make, native suites, and fuzzers not run per request
The MT worker previously reported errors through one C callback and left a
combined C finish callback responsible for serial completion, private resource
release, output-size publication, consumed-size publication, and signaling.
That kept the branch and lifecycle policy on the C side of the existing Rust
stage scheduler.
Add a narrow finish projection whose C callbacks expose only those private
leaves and the source-size scalar. Rust now publishes an error before serial
completion on every failed resource or codec stage, releases the sequence and
CCtx resources in the original order, publishes the final block size only on
success, then publishes the consumed size and signals the job condition. The
failed path therefore normalizes any codec-reported last-block size to zero
without changing the C-owned mutex, descriptor, pool, or context layouts.
Focused tests cover successful final-block publication, resource failure,
codec failure, cleanup ordering, consumed-size publication, and error-path
normalization.
Test Plan:
- `rustfmt --check --edition 2021 rust/src/zstdmt_compress.rs` -- passed
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy lib/compress/zstdmt_compress.c` -- passed
- `git diff --check -- lib/compress/zstdmt_compress.c rust/src/zstdmt_compress.rs` and `git diff --cached --check` -- passed
- Rust unit tests were added but not run because the request prohibited Cargo and heavy commands
The MT job callback added state publication before its Rust job-construction
call, which left the result declaration after executable statements and emitted
the repository's C90 mixed-declaration warning on every CLI/test build.
Declare the result alongside the other callback-local projections and assign it
at the existing call site. This is a warning-only cleanup with no runtime or
ABI change.
Test Plan:
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy lib/compress/zstdmt_compress.c` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- The prior capped suite reached `rustLibSmoke`; a fresh root library build is
required because its archive predates the display-hook fix
The first MT state-publication fix made the C count current before job
construction, but the Rust result still unconditionally copied its local
pre-job fill count back into `mtctx` after the callback returned. Preparing a
job clears `mtctx->inBuff.buffer` and `mtctx->inBuff.filled`, so the next stream
pass then observed a consumed range as if it were still active and returned a
generic error in the multi-file CLI test.
Keep the C-cleared state whenever job preparation has detached the active
range. Rust's filled count is published only while the C range remains active,
which covers the blocked/table-full path where no job reset occurred. The
callback still receives the current count before preparation, and the C90
projection declaration is kept before executable statements.
Test Plan:
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy lib/compress/zstdmt_compress.c` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- The capped upstream suite reproduced a generic error in the multi-file CLI
path before this correction; the full rerun is pending
The Rust MT stream scheduler now copies input into the reusable round buffer
through its local projection and publishes the final filled count when the
outer call returns. Job creation, however, is a C callback invoked during that
same scheduler call. It still inspected `mtctx->inBuff.filled` while preparing
the job, so the count lagged behind the bytes Rust had just copied and the
existing preparation assertion fired in the multiple-file CLI path.
Publish the callback's `srcSize` into the C context before constructing the job
projection. This restores the ordering of the former C copy callback: the
context count is current before job preparation, while the existing end-of-call
result publication remains available for paths that do not create a job.
Test Plan:
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy lib/compress/zstdmt_compress.c` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- `make -j1 -C tests test` reproduced the pre-fix assertion in
`ZSTDMT_prepareCompressionJob`; the capped full-suite rerun is pending
The advanced-CDict path previously kept a C callback that rebuilt the
ZSTD_rustCDictSizing projection and delegated the actual arithmetic back to
Rust. Replace that callback with a pointer to the explicit sizing projection,
so Rust owns the workspace-size policy while C continues to own private
zstd_cwksp allocation, object reservation, initialization, and teardown. The
projection replaces the old callback slot, preserving every later callback
offset and the synchronous lifetime of the C sizing values.
Update both sides' layout assertions, remove the redundant C helper, and make
the lifecycle tests assert that Rust's computed size reaches allocation and
workspace creation before the existing C-owned callbacks run. Custom-memory
validation and all other CDict callbacks remain unchanged.
Test Plan:
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy lib/compress/zstd_compress.c` -- passed.
- `rustfmt +nightly --edition 2021 --check rust/src/zstd_compress_dictionary.rs` -- passed.
- `git diff --check` and `git diff --cached --check` -- passed.
- Cargo build/test/clippy, Make, fuzzers, and other heavy verification were intentionally not run per request.
The MT streaming scheduler previously delegated each bounded input-buffer copy
through a C-only callback. That callback mutated the C context as a side
effect, which kept a trivial data movement operation outside the Rust stream
path and made the result projection incomplete.
Copy the checked input range directly in Rust with copy_nonoverlapping, after
the existing available-input, destination-capacity, and non-null-source checks.
The C input-range callback remains responsible for exposing the reusable buffer,
while the Rust result now returns its updated filled count. C publishes that
count back to mtctx after the scheduler call. The result ABI has explicit
cross-language layout assertions, and the scheduler test verifies both copied
bytes and fill accounting. Job scheduling and all other MT callbacks are
unchanged.
Test Plan:
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy lib/compress/zstdmt_compress.c` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- Cargo build/test, make, fuzzers, and other heavy checks were not run per task instructions
ddidderr