File-resource creation already delegates the general compression-parameter
policy to Rust, but the multithreaded parameters were still applied in C in a
separate branch. That left parameter ordering, optional overlap handling, and
error short-circuiting outside the Rust policy boundary. Add a narrow ABI
projection with C callbacks for CCtx mutation and diagnostics, and let Rust
apply worker count, job size, optional overlap, and rsyncable in the original
order. The C90 declaration layout and compile-time ABI assertions keep the
existing native program configurations intact.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo test --manifest-path rust/Cargo.toml --all-targets` -- 771 passed
- `cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- 179 passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `make -j1` under `ulimit -v 41943040` -- passed without the new C90 warning
- `make -j1 -C tests test` under `ulimit -v 41943040` -- passed
The compression context still owns private dictionary storage and CDict
objects, but the teardown order is observable because each step can invoke an
allocator or clear state used by later steps. Keep those storage operations as
C callbacks while moving the sequencing policy into the Rust compression
module. The C bridge now contains only callback adapters and compile-time ABI
layout checks, and the Rust unit test pins the original five-step order.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo test --manifest-path rust/Cargo.toml --all-targets` -- 771 passed
- `cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `make -j1` under `ulimit -v 41943040` -- passed
- `make -j1 -C tests test` under `ulimit -v 41943040` -- passed
Move the pure multithreaded worker-parameter policy into Rust: only the first
job keeps the frame checksum flag, worker jobs disable LDM, and every worker
context runs with zero nested workers. Keep the complete C-owned parameter
structure and codec initialization in the adapter, crossing the ABI with a
small layout-checked scalar projection.
Test Plan:
- rustfmt --edition 2021 --check rust/src/zstdmt_compress.rs
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; make -j1 -C tests test
Move the stable input and output expectation update into the Rust compression
module. Keep the private CCtx layout and debug logging in the C adapter while
passing only the buffer-mode scalars and projected buffer fields across the
ABI. Preserve C size_t wrapping for the output remainder and leave buffered
modes unchanged.
Test Plan:
- rustfmt --edition 2021 --check rust/src/zstd_compress.rs
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; make -j1 -C tests test
Optimized dictionary scoring used the requested dictionary ID while building
its scoring CDict, so --dictID could change candidate quality and output
content. Parallel FastCover could also choose different equal-score candidates
based on completion order. Clear the ID only while constructing the scoring
CDict, restore it immediately afterward, and use deterministic (d, k)
tie-breakers in Cover and FastCover. The requested ID now remains output
metadata instead of influencing dictionary content or repeat-run stability.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml scoring_dictionary_id_is_restored_after_the_score --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; make -j1 -C tests test
Move the non-threaded compression parameter sequence and adaptive window policy
out of FIO_createCResources. C continues to own context and resource
allocation, dictionary and pool setup, and the multithread-specific settings;
Rust drives the ordered parameter callbacks and stops on the first error.
The projection mirrors the C ABI and has focused order and short-circuit tests.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml create_c_resources --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
Move ZSTDMT_freeCCtx ownership branches and teardown ordering into Rust while
keeping the MT context, worker pool, synchronization, dictionary, and custom
allocator layouts private to C callbacks. Preserve the original factory and
round-buffer conditions, release jobs before destroying the pools, and keep
free-on-null compatible with the C API.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml free_cctx --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
Move the worker-count branch of ZSTD_toFlushNow into the Rust policy layer.
The C adapter retains the private CCtx and multithread context, exposing only
the worker count and a callback for the MT flush query. Single-threaded
contexts keep the existing zero result and the ABI projection is checked on
both sides.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml to_flush_now --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
Keep fileio's compression-resource teardown order in the Rust policy layer:
dictionary, write pool, read pool, then compression context. The C adapter
keeps cRess_t and each private resource layout local while exposing only
one callback projection to Rust, preserving the existing cleanup behavior.
The Rust entry point also treats a null state as a no-op and verifies the
projection layout at compile time.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml free_c_resources --lib
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
The Rust CLI clippy gate treats the hand-written alignment rounding in the
multi-file list projection layout test as a manual div-ceil implementation.
Use the standard integer operation so the test remains clear, equivalent, and
warning-free under the repository's deny-warnings policy.
Test Plan:
- `cargo fmt --manifest-path rust/cli/Cargo.toml --all` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
Move advanced and static CDict construction ordering into Rust while keeping
private workspace layout, object initialization, dictionary loading, and
allocator leaves in C callbacks. The Rust policy now validates parameters and
memory, computes the workspace contract, sequences allocation/reservation and
initialization, and preserves failure cleanup for both dynamic and static
workspaces. Add callback-order, allocation-failure, reservation-failure, and
initializer-failure tests around the ABI projections.
Test Plan:
- `cargo fmt --manifest-path rust/Cargo.toml --all -- --check`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo build --manifest-path rust/Cargo.toml --release`
Move stdin and empty-input validation, header selection, per-file iteration,
status aggregation, and multi-file total selection into the Rust CLI policy
layer. The callback projection keeps C responsible for its private file-info
storage, file parsing, diagnostics, human-readable formatting, and per-file
listing. Only successful and frame-error records contribute to the aggregate,
matching the original early-return behavior for invalid or truncated inputs.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/fileio_prefs.rs` -- passed.
- `git diff --cached --check` -- passed.
- Cargo, native, and full test commands were not run per the explicit no-heavy-command constraint.
Keep the compression destination ABI assertion byte-accurate while expressing
the final callback offset in the form required by current Clippy. The total
projection size remains nine function-pointer slots.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- git diff --check
Move the residual ZSTD_resetCCtx_internal sequencing into a Rust policy
entrypoint. The Rust side now owns the plan-to-workspace-to-tail order and
scalar hand-off, while C retains private workspace checks, context
initialization, match-state reset, and pointer publication callbacks. Preserve
error short-circuiting and the existing static, dynamic, LDM, and index-reset
behavior.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- Full capped Rust, native, CLI, and upstream test suites to follow
Move the single-file compression destination lifecycle into the Rust file-I/O
policy layer while keeping C responsible for private resources, file handles,
diagnostics, metadata operations, and codec dispatch callbacks. Preserve the
original shared-destination fast path, temporary permissions, handler timing,
metadata-before-close ordering, close-error propagation, stdout cleanup guard,
and failed-artifact removal.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- Full capped Rust, native, CLI, and upstream test suites to follow
Use the pointer-width bit count for the decompression callback projection
assertion so the Rust archive remains clean under warnings-as-errors Clippy.
The assertion keeps the same byte offset on supported targets.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
`ZSTD_selectBlockCompressor` already delegated scalar strategy/index
calculation to the Rust parameter module, but C still owned table selection
and dictionary-mode offsets. Keep the configuration-dependent C function-
pointer tables and NULL entries for excluded compressors as private leaves,
and pass those tables through a narrow ABI state so Rust owns row versus
ordinary selection, dictionary-mode indexing, and selected callback
publication. The legacy C selector signature remains intact for the LDM
caller.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/zstd_compress.rs` -- passed.
- `git diff --check` and `git diff --cached --check` -- passed.
- Heavy cargo/make/test/fuzzer verification not run per the enforced 40 GiB
serial verification limit.
Keep job table allocation, private synchronization callbacks, and
initialization-failure cleanup in one Rust adapter. The C wrapper now only
supplies pthread/object callbacks, preserving rounded capacity and the
destroy-before-custom-free ordering; table expansion reuses the same path.
Test Plan:
- rustfmt --check --edition 2021 rust/src/zstdmt_compress.rs
- git diff --check and git diff --cached --check
- Static inspection only; cargo, make, tests, fuzzers, and native/heavy
commands were not run under the global 40 GiB cap.
Define the decompression trace projection with the exact unsigned-long-long
representation used by the public trace API. The generic U64 typedef can be
a different C type with the same width, which made the Rust trace view fail to
compile when assigned the DCtx trace field.
Test Plan:
- ulimit -v 41943040; make -j1
Use the type's bit width for the pointer-sized tail in the source-file
projection layout assertion. This is numerically identical to eight
pointer-sized bytes on supported targets while keeping the assertion
compatible with the repository's warnings-as-errors lint policy.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
Move ZSTD_loadDictionaryContent orchestration into a Rust-owned policy leaf while keeping C-private match-state, LDM, workspace, window, overflow, and matchfinder operations behind narrow callbacks. Preserve both dictionary suffix limits, LDM-before-main-table ordering, dictionary validity publication, strategy-specific loading, and final nextToUpdate publication without changing ZSTD_compress_insertDictionary dispatch semantics.
Add focused policy tests for the two suffix limits and matchfinder path selection. Preserve the existing global overflow helper as a C callback boundary.
Test Plan:
- rustfmt --check rust/src/zstd_compress_dictionary.rs
- git diff --check and git diff --cached --check
- Static diff/rg inspection of the Rust/C ABI, callback order, suffix truncation, and private-state ownership
- No cargo, make, native, fuzz, or heavy tests run by request
Keep weak-symbol lookup and the conditional traceCtx/DCtx layout in the C
adapter, but project only the trace slot, callback pointers, context pointer,
and dictionary metadata into Rust. Rust now decides begin/end eligibility and
builds the ABI-compatible trace record, preserving null callbacks, streaming
and size conversion semantics. Existing call sites remain in their
success-only branches, so checksum/error paths still emit no trace end.
Test Plan:
- rustfmt --edition 2021 --check rust/src/zstd_decompress.rs
- git diff --check and git diff --cached --check
- Static rg/diff inspection of weak callbacks, traceCtx lifecycle, dictionary
metadata, and success-only trace-end call sites
- No cargo, make, native, fuzz, or heavy tests run by request
Move the ordered policy from FIO_compressFilename_srcFile into a narrow Rust
projection: named-source stat and collision checks, compressed-file exclusion,
source open and close, size-based async selection, source attachment, and
successful-only --rm handling now run in Rust. The destination callback still
hands compression to the existing Rust compression loop.
Keep FIO_ctx_t, cRess_t, stat_t, FILE/pool handles, signal handling, and
CLI diagnostics behind opaque C callbacks, so the ABI carries only filenames,
scalar policy flags, and callback pointers. Focused Rust policy tests cover
named-file short circuits, stdin behavior, async threshold selection, and the
successful-compression removal gate.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/fileio_asyncio.rs` — passed
- `git diff --check` and path-limited `git diff --cached --check` — passed
- Static ABI/order/diff inspection only; policy tests were not executed
- cargo/make/native/fuzzer/heavy tests were not run per request
The Rust-owned frame-progression ring scan consumes ZSTDMT_jobProjection
through ZSTDMT_projectJob. That callback previously copied counters under
job_mutex but read source and prefix metadata after unlocking. Keep the
complete projection in one mutex-protected snapshot, matching the original
ZSTDMT_getFrameProgression access pattern without moving descriptor or
synchronization ownership out of C.
Test Plan:
- `rustfmt --check --edition 2021 rust/src/zstdmt_compress.rs` -- passed.
- `git diff --check` and `git diff --cached --check` -- passed.
- Static inspection only; no cargo, make, native binaries, tests, fuzzers,
or heavyweight commands were run.
Project the external sequence producer callback directly into the Rust
block-delimited sequence transfer leaf. Remove the redundant C forwarding
wrapper while preserving the C-facing sequence conversion API.
Test Plan:
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests fuzzer
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Remove the unreachable C target-sized and split-block adapters now that frame chunks use the direct Rust state projections. The C sequence-store wrapper was likewise unused after those adapters disappeared.
Keep the target and split callback slots as nullable Rust test seams and return a generic error if a caller selects a callback fallback without providing one. Production frame-chunk state now leaves both slots null because the direct projections are authoritative.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Replace the frame-chunk split-discovery callback with an explicit projection of the sequence stores, block-state pointer slots, entropy metadata, parameters, and workspace required by the Rust estimator. Rust now invokes the existing split estimator directly before the Rust-owned partition loop, while C retains only private block-split storage and projection assembly.
Keep the block-state pointer slots live through estimation instead of capturing entropy-table addresses during frame preparation. Sequence-store work can change which compressed-block state is active, and following the slots preserves the original C behavior for dictionary-backed streaming compression.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Project the target-sized block state into frame-chunk compression and invoke
Rust sequence-store construction followed by the existing target-block
post-build leaf. Refresh the per-block isFirstBlock snapshot to preserve the
former C adapter's behavior while leaving the split-block branch on its
existing callback path for a later seam.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib 'zstd_compress::tests::frame_chunk' -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Route frame-chunk internal blocks through the projected build-sequence-store
and block-internal state, using a Rust-owned frame-chunk leaf while retaining
the target and split callbacks in C. Refresh the per-block isFirstBlock
snapshot for the direct path so its behavior remains identical to the former
C adapter, and remove the obsolete C frame-chunk and block-internal adapters.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Share the existing end-state projection between the public C wrapper and
Rust-owned compressed-dictionary orchestration. The CDict callback now reaches
ZSTD_rust_compressEnd through the private dispatcher without re-entering the
public C end wrapper, while the exported API keeps its compatibility wrapper.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Pass the existing overflow-correction projection directly through the
compressContinue ABI so Rust owns the correction decision and ordering while
C retains only the table/workspace callbacks. Remove the redundant C
forwarding adapter and keep the stack projection valid for synchronous
dispatch.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::compress_continue -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Project the match-state and optional LDM window fields directly through the
compressContinue ABI so Rust can reuse the existing window-update leaf. Keep
forceNonContiguous and nextToUpdate updates visible to the following C
callbacks, preserve the match-before-LDM ordering, and remove the C window
update callback while retaining context-sensitive overflow and block calls.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::compress_continue -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Remove the sequence-compression API's C frame-header callback and carry the
validated frame parameters as scalar state. Rust now serializes headers for
both sequence API variants through the same frame-header leaf, while C keeps
only CCtx initialization and checksum callbacks behind the boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::sequence_api -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Remove the C frame-header callback from the Rust-owned compressContinue
orchestration. Project the applied frame parameters and dictionary ID as
scalars, so Rust can call the existing header serializer directly while
preserving pledged-size subtraction, stage transitions, and output accounting.
Route the sequence API's remaining header call directly to the same Rust leaf
and delete the redundant C wrapper.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::compress_continue -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Project the end-frame stage, format flags, window log, and XXH64 state into
Rust so the Rust end-of-frame orchestrator computes the checksum and invokes
the existing Rust epilogue serializer directly. Remove the C epilogue and
checksum adapter while preserving callback order, output accounting, stage
transitions, and error precedence.
Test Plan:
- CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::compress_end -- --nocapture
- CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- CARGO_BUILD_JOBS=1 cargo test
- CARGO_BUILD_JOBS=1 make -j1
- CARGO_BUILD_JOBS=1 make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- CARGO_BUILD_JOBS=1 make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
All commands were run serially with a 40 GiB virtual-memory cap.
Project the prepared CDict window extent and destination match-state fields
through an explicit ABI state so Rust owns the post-reset attachment
operation. Preserve the original empty-dictionary and already-advanced
window branches, including window-limit clearing and loaded-dictionary
referential updates, while keeping private C pointer arithmetic and layout
ownership in the C adapter.
Test Plan:
- CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress_dictionary::tests::reset_cctx_by_attaching_cdict -- --nocapture
- CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- CARGO_BUILD_JOBS=1 cargo test
- CARGO_BUILD_JOBS=1 make -j1
- CARGO_BUILD_JOBS=1 make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- CARGO_BUILD_JOBS=1 make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
All commands were run serially with a 40 GiB virtual-memory cap.
Move the final CDict-copy hashTable3 clearing operation into the Rust reset
orchestration. The C bridge now projects the post-reset destination table
and hash log while retaining only the private reset, workspace dirty/clean,
and metadata operations on the C side.
Preserve C's zero-log contract: hashLog3 == 0 means no table bytes are
cleared and a null hashTable3 is valid. Nonzero logs are bounds-checked before
Rust zeroes the table, keeping the operation between the dirty and clean
transitions.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Move the CDict-to-CCtx hash, chain, and row-match table copies into the
Rust reset orchestration. The C bridge now exposes table-field addresses
and scalar policy inputs while retaining private reset and hashTable3
operations; Rust loads destination pointers only after reset and strips
short-cache tags through the existing Rust leaf.
The destination chain policy must be read after the reset callback updates
appliedParams. Reading it before reset skipped DFAST chain copies and caused
the dictionary source-size zstream test to segfault in the double-fast
external-dictionary matcher. Focused tests cover tagged hash/chain copies,
row-match tag/salt copies, callback ordering, and reset-error short-circuiting.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Replace the remaining CCtx-copy custom-memory callback with explicit source and
destination pointers. Rust now copies the ABI-checked ZSTD_customMem payload in
its original position before reset, while C retains private context access and
the surrounding workspace and table operations.
Test Plan:
- ulimit -v 41943040 cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040 make -j1
- ulimit -v 41943040 make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040 make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Pass the CCtx compressed-block-state pointer slot and CDict source state
through the attach/copy bridges, then perform the final state copy in Rust
after reset has established the destination. Remove the redundant C memcpy
adapters while preserving dictionary attach/copy ordering and null/error
behavior.
Add focused tests for copied repcodes and entropy repeat state.
Test Plan:
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
- ulimit -v 41943040 && cargo fmt --manifest-path rust/Cargo.toml -- --check
- git diff --check
Route ZSTD_initStaticCDict() workspace-size calculation through the Rust
CDict sizing policy, including the by-reference/by-copy, resolved row
match-finder, and dedicated-search branches. Keep private layout constants
at the C boundary and retain C ownership of static workspace construction
and dictionary initialization.
Remove the obsolete C match-state sizing adapter now that no C path calls
it, avoiding a dead implementation after the sizing migration.
Test Plan:
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
- ulimit -v 41943040 && cargo fmt --manifest-path rust/Cargo.toml -- --check
- git diff --check
Move the advanced-CDict workspace-size formula and its by-reference,
by-copy, row-match-finder, and dedicated-search branches into the Rust
compression-parameter layer. Keep private C layout constants and the
already-resolved mode at the C boundary, where the allocator still owns
private object layout details.
Add a focused test covering the load-method and dedicated-search deltas
and the exported advanced-sizing boundary.
Test Plan:
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
- ulimit -v 41943040 && cargo fmt --manifest-path rust/Cargo.toml -- --check
- git diff --check
The transparent stream initializer previously called a C helper that both
inspected local-dictionary state and chose the no-dictionary, already-created,
and create/publish branches. Project that private state through a checked C/Rust
ABI so Rust owns the branch and callback order while C retains the requested
parameters, custom allocator, and CDict implementation details. Keep creation
and publication split so allocation failure cannot publish a partial dictionary.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compress_stream_init_
- CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -j1
- make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
- CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
Move match-table selection and btlazy2 marker policy out of the C overflow
correction wrapper. Rust now decides whether chain and hash3 tables
participate using the strategy, row-matchfinder, dedicated-dictionary, and
log inputs, then delegates cell updates to the existing Rust reducer. C
retains only the private table pointers and scalar projection.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstd_compress::tests::reduce_index_for_match_state -- --nocapture
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Route match-state invalidation through a Rust leaf. C still computes the
private window pointer difference and passes the opaque dictionary-state
pointer, while Rust preserves the window-limit, next-index, dictionary,
optimal-parser, and dictionary-match-state reset order. Add a focused test
for all invalidated fields and use the concrete C pointer type at the ABI
boundary to avoid strict-aliasing warnings.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstd_compress::tests::invalidate_match_state -- --nocapture
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Move the failed-job serial-state orchestration out of zstdmt_compress.c.
Rust now owns the lock, skip decision, serial-counter publication,
broadcast, LDM cleanup ordering, and final unlock. C retains the pthread
objects, private LDM window, and error/debug leaves behind callbacks. Add
ABI layout checks and focused tests for both skipped predecessors and later
jobs that only need the lock/unlock pair.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstdmt_compress::tests::serial_ensure_finished -- --nocapture
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Move the serial turn increment, broadcast, and unlock ordering into a Rust
entry point. C retains the pthread condition variable and mutex operations
behind callbacks, while Rust preserves wrapping unsigned job-counter
semantics and the required broadcast-before-unlock contract. Add ABI layout
checks and focused tests for ordinary and wrapping advancement.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstdmt_compress::tests::serial_ -- --nocapture
- ulimit -v 41943040 && make -j1
Use an explicit break-based loop for the Rust serial wait projection so
Clippy recognizes that the C wait callback can change the shared job counter.
The loop preserves the same lock ownership and condition-wait behavior.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
Move the serial turn wait loop out of zstdmt_compress.c and into a Rust
entry point. Rust now owns the lock, condition-wait, and ready/skip ordering
policy while C retains pthread synchronization behind lock and wait callbacks.
The projection documents that the mutex remains locked on return for the
serial advance callback, preserving the existing worker synchronization
contract. Add ABI layout checks and focused tests for both waiting and skip
paths.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstdmt_compress::tests::serial_wait_for_turn -- --nocapture
- ulimit -v 41943040 && make -j1
Move the MT reusable-input wait decision and lock/overlap/condition-wait
ordering into Rust through a small callback projection. C retains ownership
of the pthread synchronization objects, private LDM window, and diagnostics.
Add layout assertions and callback-order tests for enabled and disabled LDM.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1; cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040; CARGO_BUILD_JOBS=1; cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
Move the bounded nextToUpdate catch-up arithmetic used after very long
matches into Rust. C continues to calculate the source index against the
private match-state window and retains all surrounding assertions and setup;
Rust owns the exact wrapping U32 threshold and 192-byte catch-up policy.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml limit_next_to_update --lib (1 passed)
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib (692 passed)
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && MAKEFLAGS=-j1 make -B -C programs -j1 zstd
- ulimit -v 41943040 && MAKEFLAGS=-j1 make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s (84 tests and both short fuzz rounds passed)
Move the scalar planning portion of ZSTD_resetCCtx_internal across an
explicit C/Rust boundary. Rust now computes window and block sizes, sequence
capacities, buffered I/O sizes, LDM and external-sequence reservations, the
index-reset policy, and the CCtx workspace estimate. C retains private
workspace resizing, reservations, pointer publication, and layout-dependent
callbacks.
Add ABI layout assertions and focused planner tests for buffered sizing, LDM
and index-reset policy, external sequence capacity, and invalid unadjusted LDM
inputs. Keep the migration boundary documented so the hybrid state remains
explicit while deeper CCtx reset and matchfinder operations stay in C.
Test Plan:
- `cargo fmt --manifest-path rust/Cargo.toml -- --check`
- capped focused planner tests: 3 passed
- capped full Rust library tests: 680 passed, 0 failed
- capped `cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings`: passed
- capped native `make -B -C programs -j1 zstd`: passed; only known fileio const-cast warnings
- capped `make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s`: 84 tests and both short fuzzer rounds passed; only known zstream initializer warning
The CCtx workspace estimator still assembled its private sizing formula in C,
even though the match-state and component-sum leaves already lived in Rust.
That left buffering, LDM, static-context, and external-sequence sizing policy
outside the rewrite boundary.
Move the complete size_t formula into a Rust entry point. C now supplies only
private object sizes, sanitizer redzone policy, and LDM scalar fields through a
small ABI record. Rust retains the C wrapping/alignment behavior and focused
branch coverage while the C adapter preserves the existing public estimator
surface.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --release estimate_cctx_workspace_size
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --release (667 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --release --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s (84 tests and both short fuzzer rounds)
ZSTD_createCDict_advanced2 previously selected compression parameters,
resolved dedicated-dictionary-search fallback, and ordered allocation and
initialization entirely in C. That left a large public dictionary boundary
outside the Rust rewrite and made the failure ordering implicit in the C
wrapper.
The Rust params API now owns advanced-CDict parameter preparation, including
the dedicated-search override/fallback and row-matchfinder resolution. A Rust
dictionary bridge owns the create -> init sequence and frees a created CDict
when initialization fails. The C shim retains only the private workspace and
custom-memory allocation, dictionary initialization, and teardown callbacks;
opaque parameter-field pointers and compile-time layout assertions preserve
the existing ABI. Focused probes cover policy publication and all callback
failure-ordering paths, and the migration README records the remaining C
boundary.
Test Plan:
- `rustfmt --check` and `cargo fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `cargo test --manifest-path rust/Cargo.toml --release` -- 663 passed
- `cargo clippy --manifest-path rust/Cargo.toml --release --all-targets -- -D warnings` -- passed
- `make -B -C programs -j1 zstd` -- passed
- `make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s` -- 84 tests and both fuzz rounds passed
Record that Rust now owns advanced CDict one-shot validation and begin/end
ordering while C retains the private operations behind callbacks.
Test Plan:
- Documentation-only change; code validation is covered by 8b783e8.
ddidderr