Remove the unreachable C target-sized and split-block adapters now that frame chunks use the direct Rust state projections. The C sequence-store wrapper was likewise unused after those adapters disappeared.
Keep the target and split callback slots as nullable Rust test seams and return a generic error if a caller selects a callback fallback without providing one. Production frame-chunk state now leaves both slots null because the direct projections are authoritative.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Replace the frame-chunk split-discovery callback with an explicit projection of the sequence stores, block-state pointer slots, entropy metadata, parameters, and workspace required by the Rust estimator. Rust now invokes the existing split estimator directly before the Rust-owned partition loop, while C retains only private block-split storage and projection assembly.
Keep the block-state pointer slots live through estimation instead of capturing entropy-table addresses during frame preparation. Sequence-store work can change which compressed-block state is active, and following the slots preserves the original C behavior for dictionary-backed streaming compression.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Project the target-sized block state into frame-chunk compression and invoke
Rust sequence-store construction followed by the existing target-block
post-build leaf. Refresh the per-block isFirstBlock snapshot to preserve the
former C adapter's behavior while leaving the split-block branch on its
existing callback path for a later seam.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib 'zstd_compress::tests::frame_chunk' -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Route frame-chunk internal blocks through the projected build-sequence-store
and block-internal state, using a Rust-owned frame-chunk leaf while retaining
the target and split callbacks in C. Refresh the per-block isFirstBlock
snapshot for the direct path so its behavior remains identical to the former
C adapter, and remove the obsolete C frame-chunk and block-internal adapters.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Share the existing end-state projection between the public C wrapper and
Rust-owned compressed-dictionary orchestration. The CDict callback now reaches
ZSTD_rust_compressEnd through the private dispatcher without re-entering the
public C end wrapper, while the exported API keeps its compatibility wrapper.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Pass the existing overflow-correction projection directly through the
compressContinue ABI so Rust owns the correction decision and ordering while
C retains only the table/workspace callbacks. Remove the redundant C
forwarding adapter and keep the stack projection valid for synchronous
dispatch.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::compress_continue -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Project the match-state and optional LDM window fields directly through the
compressContinue ABI so Rust can reuse the existing window-update leaf. Keep
forceNonContiguous and nextToUpdate updates visible to the following C
callbacks, preserve the match-before-LDM ordering, and remove the C window
update callback while retaining context-sensitive overflow and block calls.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt --all -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::compress_continue -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Remove the sequence-compression API's C frame-header callback and carry the
validated frame parameters as scalar state. Rust now serializes headers for
both sequence API variants through the same frame-header leaf, while C keeps
only CCtx initialization and checksum callbacks behind the boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::sequence_api -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Remove the C frame-header callback from the Rust-owned compressContinue
orchestration. Project the applied frame parameters and dictionary ID as
scalars, so Rust can call the existing header serializer directly while
preserving pledged-size subtraction, stage transitions, and output accounting.
Route the sequence API's remaining header call directly to the same Rust leaf
and delete the redundant C wrapper.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::compress_continue -- --nocapture
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Project the end-frame stage, format flags, window log, and XXH64 state into
Rust so the Rust end-of-frame orchestrator computes the checksum and invokes
the existing Rust epilogue serializer directly. Remove the C epilogue and
checksum adapter while preserving callback order, output accounting, stage
transitions, and error precedence.
Test Plan:
- CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress::tests::compress_end -- --nocapture
- CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- CARGO_BUILD_JOBS=1 cargo test
- CARGO_BUILD_JOBS=1 make -j1
- CARGO_BUILD_JOBS=1 make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- CARGO_BUILD_JOBS=1 make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
All commands were run serially with a 40 GiB virtual-memory cap.
Project the prepared CDict window extent and destination match-state fields
through an explicit ABI state so Rust owns the post-reset attachment
operation. Preserve the original empty-dictionary and already-advanced
window branches, including window-limit clearing and loaded-dictionary
referential updates, while keeping private C pointer arithmetic and layout
ownership in the C adapter.
Test Plan:
- CARGO_BUILD_JOBS=1 cargo test --lib zstd_compress_dictionary::tests::reset_cctx_by_attaching_cdict -- --nocapture
- CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
- CARGO_BUILD_JOBS=1 cargo test
- CARGO_BUILD_JOBS=1 make -j1
- CARGO_BUILD_JOBS=1 make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- CARGO_BUILD_JOBS=1 make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
All commands were run serially with a 40 GiB virtual-memory cap.
Move the final CDict-copy hashTable3 clearing operation into the Rust reset
orchestration. The C bridge now projects the post-reset destination table
and hash log while retaining only the private reset, workspace dirty/clean,
and metadata operations on the C side.
Preserve C's zero-log contract: hashLog3 == 0 means no table bytes are
cleared and a null hashTable3 is valid. Nonzero logs are bounds-checked before
Rust zeroes the table, keeping the operation between the dirty and clean
transitions.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Move the CDict-to-CCtx hash, chain, and row-match table copies into the
Rust reset orchestration. The C bridge now exposes table-field addresses
and scalar policy inputs while retaining private reset and hashTable3
operations; Rust loads destination pointers only after reset and strips
short-cache tags through the existing Rust leaf.
The destination chain policy must be read after the reset callback updates
appliedParams. Reading it before reset skipped DFAST chain copies and caused
the dictionary source-size zstream test to segfault in the double-fast
external-dictionary matcher. Focused tests cover tagged hash/chain copies,
row-match tag/salt copies, callback ordering, and reset-error short-circuiting.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040; make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Replace the remaining CCtx-copy custom-memory callback with explicit source and
destination pointers. Rust now copies the ABI-checked ZSTD_customMem payload in
its original position before reset, while C retains private context access and
the surrounding workspace and table operations.
Test Plan:
- ulimit -v 41943040 cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040 make -j1
- ulimit -v 41943040 make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040 make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Pass the CCtx compressed-block-state pointer slot and CDict source state
through the attach/copy bridges, then perform the final state copy in Rust
after reset has established the destination. Remove the redundant C memcpy
adapters while preserving dictionary attach/copy ordering and null/error
behavior.
Add focused tests for copied repcodes and entropy repeat state.
Test Plan:
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
- ulimit -v 41943040 && cargo fmt --manifest-path rust/Cargo.toml -- --check
- git diff --check
Route ZSTD_initStaticCDict() workspace-size calculation through the Rust
CDict sizing policy, including the by-reference/by-copy, resolved row
match-finder, and dedicated-search branches. Keep private layout constants
at the C boundary and retain C ownership of static workspace construction
and dictionary initialization.
Remove the obsolete C match-state sizing adapter now that no C path calls
it, avoiding a dead implementation after the sizing migration.
Test Plan:
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
- ulimit -v 41943040 && cargo fmt --manifest-path rust/Cargo.toml -- --check
- git diff --check
Move the advanced-CDict workspace-size formula and its by-reference,
by-copy, row-match-finder, and dedicated-search branches into the Rust
compression-parameter layer. Keep private C layout constants and the
already-resolved mode at the C boundary, where the allocator still owns
private object layout details.
Add a focused test covering the load-method and dedicated-search deltas
and the exported advanced-sizing boundary.
Test Plan:
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
- ulimit -v 41943040 && cargo fmt --manifest-path rust/Cargo.toml -- --check
- git diff --check
The transparent stream initializer previously called a C helper that both
inspected local-dictionary state and chose the no-dictionary, already-created,
and create/publish branches. Project that private state through a checked C/Rust
ABI so Rust owns the branch and callback order while C retains the requested
parameters, custom allocator, and CDict implementation details. Keep creation
and publication split so allocation failure cannot publish a partial dictionary.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compress_stream_init_
- CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -j1
- make -j1 -C tests test-zstream ZSTREAM_TESTTIME=-T2s
- make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
- CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml
Move match-table selection and btlazy2 marker policy out of the C overflow
correction wrapper. Rust now decides whether chain and hash3 tables
participate using the strategy, row-matchfinder, dedicated-dictionary, and
log inputs, then delegates cell updates to the existing Rust reducer. C
retains only the private table pointers and scalar projection.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstd_compress::tests::reduce_index_for_match_state -- --nocapture
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Route match-state invalidation through a Rust leaf. C still computes the
private window pointer difference and passes the opaque dictionary-state
pointer, while Rust preserves the window-limit, next-index, dictionary,
optimal-parser, and dictionary-match-state reset order. Add a focused test
for all invalidated fields and use the concrete C pointer type at the ABI
boundary to avoid strict-aliasing warnings.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstd_compress::tests::invalidate_match_state -- --nocapture
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Move the failed-job serial-state orchestration out of zstdmt_compress.c.
Rust now owns the lock, skip decision, serial-counter publication,
broadcast, LDM cleanup ordering, and final unlock. C retains the pthread
objects, private LDM window, and error/debug leaves behind callbacks. Add
ABI layout checks and focused tests for both skipped predecessors and later
jobs that only need the lock/unlock pair.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstdmt_compress::tests::serial_ensure_finished -- --nocapture
- ulimit -v 41943040 && make -j1
- ulimit -v 41943040 && make -j1 -C tests test-fuzzer FUZZERTEST=-T3s FUZZER_FLAGS=--no-big-tests
Move the serial turn increment, broadcast, and unlock ordering into a Rust
entry point. C retains the pthread condition variable and mutex operations
behind callbacks, while Rust preserves wrapping unsigned job-counter
semantics and the required broadcast-before-unlock contract. Add ABI layout
checks and focused tests for ordinary and wrapping advancement.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstdmt_compress::tests::serial_ -- --nocapture
- ulimit -v 41943040 && make -j1
Use an explicit break-based loop for the Rust serial wait projection so
Clippy recognizes that the C wait callback can change the shared job counter.
The loop preserves the same lock ownership and condition-wait behavior.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
Move the serial turn wait loop out of zstdmt_compress.c and into a Rust
entry point. Rust now owns the lock, condition-wait, and ready/skip ordering
policy while C retains pthread synchronization behind lock and wait callbacks.
The projection documents that the mutex remains locked on return for the
serial advance callback, preserving the existing worker synchronization
contract. Add ABI layout checks and focused tests for both waiting and skip
paths.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040 && CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml zstdmt_compress::tests::serial_wait_for_turn -- --nocapture
- ulimit -v 41943040 && make -j1
ddidderr