Move the scalar planning portion of ZSTD_resetCCtx_internal across an
explicit C/Rust boundary. Rust now computes window and block sizes, sequence
capacities, buffered I/O sizes, LDM and external-sequence reservations, the
index-reset policy, and the CCtx workspace estimate. C retains private
workspace resizing, reservations, pointer publication, and layout-dependent
callbacks.
Add ABI layout assertions and focused planner tests for buffered sizing, LDM
and index-reset policy, external sequence capacity, and invalid unadjusted LDM
inputs. Keep the migration boundary documented so the hybrid state remains
explicit while deeper CCtx reset and matchfinder operations stay in C.
Test Plan:
- `cargo fmt --manifest-path rust/Cargo.toml -- --check`
- capped focused planner tests: 3 passed
- capped full Rust library tests: 680 passed, 0 failed
- capped `cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings`: passed
- capped native `make -B -C programs -j1 zstd`: passed; only known fileio const-cast warnings
- capped `make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s`: 84 tests and both short fuzzer rounds passed; only known zstream initializer warning
The CCtx workspace estimator still assembled its private sizing formula in C,
even though the match-state and component-sum leaves already lived in Rust.
That left buffering, LDM, static-context, and external-sequence sizing policy
outside the rewrite boundary.
Move the complete size_t formula into a Rust entry point. C now supplies only
private object sizes, sanitizer redzone policy, and LDM scalar fields through a
small ABI record. Rust retains the C wrapping/alignment behavior and focused
branch coverage while the C adapter preserves the existing public estimator
surface.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --release estimate_cctx_workspace_size
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --release (667 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --release --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s (84 tests and both short fuzzer rounds)
ZSTD_createCDict_advanced2 previously selected compression parameters,
resolved dedicated-dictionary-search fallback, and ordered allocation and
initialization entirely in C. That left a large public dictionary boundary
outside the Rust rewrite and made the failure ordering implicit in the C
wrapper.
The Rust params API now owns advanced-CDict parameter preparation, including
the dedicated-search override/fallback and row-matchfinder resolution. A Rust
dictionary bridge owns the create -> init sequence and frees a created CDict
when initialization fails. The C shim retains only the private workspace and
custom-memory allocation, dictionary initialization, and teardown callbacks;
opaque parameter-field pointers and compile-time layout assertions preserve
the existing ABI. Focused probes cover policy publication and all callback
failure-ordering paths, and the migration README records the remaining C
boundary.
Test Plan:
- `rustfmt --check` and `cargo fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `cargo test --manifest-path rust/Cargo.toml --release` -- 663 passed
- `cargo clippy --manifest-path rust/Cargo.toml --release --all-targets -- -D warnings` -- passed
- `make -B -C programs -j1 zstd` -- passed
- `make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s` -- 84 tests and both fuzz rounds passed
Record that Rust now owns advanced CDict one-shot validation and begin/end
ordering while C retains the private operations behind callbacks.
Test Plan:
- Documentation-only change; code validation is covered by 8b783e8.
Record that Rust now owns static CCtx workspace validation and dispatch while
C retains private workspace construction and context initialization.
Test Plan:
- Documentation-only change; code validation is covered by 463ceeb86.
Record that Rust now owns static CDict workspace validation and dispatch while
C retains private workspace construction and dictionary initialization.
Test Plan:
- Documentation-only change; code validation is covered by 12b99469b.
Record that Rust now owns CDict null handling and workspace/object teardown
ordering while C retains private workspace and allocator callbacks.
Test Plan:
- Documentation-only change; code validation is covered by 14157558f.
Record that Rust now owns regular and by-reference CDict constructor parameter
selection and default-level normalization while C retains advanced allocation
and workspace lifecycle.
Test Plan:
- Documentation-only change; code validation is covered by 9f4d92808.
Record that Rust now owns public CCtx allocation validation and
allocate-before-init ordering while C retains private context initialization.
Test Plan:
- Documentation-only change; code validation is covered by aa817a56b.
Record that Rust now owns the legacy CDict-begin fixed frame policy and
unknown-source pledge while C retains the private begin callback.
Test Plan:
- Documentation-only change; code validation is covered by 2fdd548a3.
Record that Rust now owns public advanced-begin parameter validation and
init-then-begin ordering while C retains private context initialization and
begin callbacks.
Test Plan:
- Documentation-only change; code validation is covered by feee05372.
Record that Rust now owns the compressBegin_usingDict family’s unknown-source
parameter selection and default-level normalization while C retains the private
begin callback.
Test Plan:
- Documentation-only change; code validation is covered by 9c0b9bd25.
Record that Rust now owns the public usingCDict fixed-frame policy and
begin/end sequencing while C retains the private CDict compression callbacks.
Test Plan:
- Documentation-only change; code validation is covered by 0fbc9ff37.
Record that Rust now owns the public usingDict parameter-selection,
dictionary-presence, and default-level policy while C retains private context
initialization and compression callbacks.
Test Plan:
- Documentation-only change; code validation is covered by 556cdc53b.
Record that Rust now owns public advanced-compression parameter validation and
initialization ordering while C retains private context setup and execution.
Test Plan:
- git diff --cached --check
Record that Rust now owns public CCtx-copy pledge and frame-parameter policy,
while C retains the private workspace and table-copy operation.
Test Plan:
- git diff --cached --check
Record that Rust now selects the CDict attach-versus-copy reset path while C
retains the workspace- and match-state-heavy reset operations.
Test Plan:
- git diff --cached --check
Record that shared compression-begin dictionary selection now runs in Rust,
while C retains the reset, attach, and private dictionary operation callbacks.
Test Plan:
- git diff --cached --check
Clarify that CDict initialization ordering and scalar publication now run in
Rust, while allocation, private dictionary loading, and match/workspace
callbacks remain C-owned.
Test Plan:
- git diff --cached --check
Record that Rust now owns external-producer invocation and successful-path
validation plus external raw-sequence-store reset, while C retains private
transfer, fallback, and context operations.
Test Plan:
- git diff --cached --check
Record that CDict-begin parameter selection, initialization ordering, and
source-window policy now run in Rust while private CDict construction and the
final C begin operation remain on the C side of the migration boundary.
Test Plan:
- git diff --cached --check
Record the separate-destination decompression scheduler in the Rust migration
boundary and clarify that C still owns destination naming and private I/O,
resource, and format callbacks.
Test Plan:
- git diff --cached --check
Record the shared-destination decompression scheduler as a Rust-owned boundary while documenting that C retains destination, resource, and format callbacks.
Test Plan:
- git diff --cached --check
Document the Rust-owned public end-of-frame compression orchestration and both multi-file compression scheduler boundaries, including the C callbacks that remain responsible for private file and codec operations.
Test Plan:
- git diff --cached --check
Record the Rust-owned stable-buffer fallback orchestration and shared-output
multi-file scheduler in the migration boundary documentation while retaining
the C-owned private context, resource, and callback responsibilities.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
17cfedd56 and 035001b76
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that public sequence API orchestration and scalar CLI adaptive
decisions are Rust-owned while private codec/context state and mutation
callbacks remain in C.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
3b989ab3e and b04893799
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that Rust owns single-threaded and MT stream-initialization policy,
including the ordered setup decisions and scalar projections, while C retains
private contexts and mutation callbacks.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by the
preceding stream-initialization commits
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that Rust now owns CCtx dictionary/prefix attachment dispatch and the
multithreaded compression-job stage sequence, while C retains reset policy,
private context and worker state, dictionary-content loading, and codec
callbacks.
Test Plan:
- Documentation-only change; code validation was completed before this commit.
- Rust library all-target tests: 517 passed; legacy feature matrix: 572 passed.
- Native CLI tests (41), test-zstd, fuzzer (319), zstream (152 + 297), and
decode corpus (1,647) passed under the serial 40 GiB virtual-memory cap.
- Commit is intentionally unsigned because configured GPG pinentry was
unavailable and hung during the signing attempt.
Update the Rust migration map after the latest compression and CLI seams
moved. The fileio summary now distinguishes Rust-owned zstd stream
orchestration from C-owned codec, diagnostics, and adaptive-policy callbacks.
The remaining-boundary summary records that dictionary dispatch and the CLI
zstd compression loop have moved, while CDict lifecycle, private dictionary
content loading, and private compressor operations remain in C.
Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no additional build or test run was needed.
GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
Document that Rust now owns sequence-store construction, block post-build
policy, public sequence conversion, and optional-format decompression loops.
Keep the remaining private C callback and dictionary/CLI boundaries explicit so
the hybrid state is not mistaken for a completed all-Rust rewrite.
Test Plan:
- git diff --check (passed)
Update the Rust component map to describe the newly migrated single-threaded
stream state machine and multithreaded scheduler/flush policy. Correct the CLI
boundary so it no longer claims that the optional-format decompression loops
are Rust-owned, and list the remaining sequence, dictionary, and CLI leaves
that still block an all-Rust build.
Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no code tests required.
Document that the Rust CLI now owns the gzip, xz/lzma, and LZ4 codec leaves as
well as MT job creation, while the generic frame-compression state machine and
remaining MT scheduling/outer-flush orchestration remain C-owned. Keep the
hybrid boundary explicit so compatibility-suite success is not mistaken for
completion of the full rewrite.
Test Plan:
- git diff --cached --check
- reviewed rust/README.md against the committed Rust/C ownership boundary
Document that the frame epilogue, produced-output flush policy, and gzip codec
leaf now execute in Rust. Narrow the remaining C-owned list to the generic
compression state machine, MT scheduling/outer flush orchestration, and the
xz/lzma/lz4 codec leaves so the README remains an accurate progress boundary.
Test Plan:
- git diff --cached --check
The migration map still described the optimal parser and benchmark orchestration
as C-owned, and its remaining-work paragraph did not distinguish the newly
ported high-level leaves from the still-private frame and MT orchestration.
That made the README under-report Rust ownership and overstate some remaining
C work.
Update the component map to identify `zstd_opt` price/parse ownership and Rust
benchmark orchestration, and describe the current C boundary as the remaining
frame setup/flush scheduling and format-specific codec leaves. Keep the
hybrid-build warning explicit until those larger orchestration layers move.
Test Plan:
- `git diff --cached --check` -- passed.
- Rust library and CLI Clippy default, benches, and tests -- passed before staging.
- Native library, program, fuzz, CLI, and upstream test matrices -- passed before staging.
Move the post-sequence-store body of ZSTD_compressBlock_internal behind a
small C/Rust state projection. C continues to build the sequence store and
handle the no-compress and sequence-producer error paths, while Rust now owns
sequence collection, entropy emission, the legacy non-first-frame RLE gate,
compressed-block confirmation, and offcode repeat cleanup. Remove the C
wrappers that became dead after those leaves moved behind the Rust body.
The first integration run exposed that leaving the old C finalization label in
place confirmed compressed block state twice, undoing Rust's pointer swap and
breaking a later sparse-file checksum. The C wrapper now returns directly for
the Rust-owned path and retains only its C-owned no-compress cleanup.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1
- cargo check --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-zstd
- make -B -C tests -j2 test-cli-tests
- ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer
- make -B -C tests/fuzz -j2 all
- make -B -C tests/fuzz -j2 sequence_compression_api
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings (pre-existing manual_repeat_n failure)
- cargo clippy --manifest-path rust/Cargo.toml --benches -- -D warnings (pre-existing manual_repeat_n failure)
Move the post-split partition loop out of zstd_compress.c while keeping the
private CCtx, matchfinder sequence-store construction, and split discovery in
C. Rust now receives a layout-asserted projection containing sequence-store
views, block-state slots, workspace and scalar policy, then mirrors the C
loop's dRep/cRep histories, final-literal accounting, repeated single-block
serialization, and final dRep publication.
Remove the obsolete C sequence-store/count/chunk wrappers and the debug-only
C size-estimation path that was coupled to the old loop. Add a focused Rust
fixture covering partition payloads and final literals, and document the new
ownership boundary for the split search and emission paths.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1 (448 passed)
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings (passed)
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings (passed)
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check (passed)
- cargo clippy --tests/--benches remain blocked only by the pre-existing manual_repeat_n lint in one_shot_promotes_nonfirst_rle_blocks
- make -B -C lib -j2 lib (passed)
- make -B -C tests -j2 test-zstd (passed)
- make -B -C tests -j2 test-cli-tests (41 passed)
- ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream (passed)
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer (252 passed)
- make -B -C tests/fuzz -j2 all and sequence_compression_api (passed)
The component map still described decoder context allocation as a C-owned
responsibility after the allocation and custom-memory bridges moved into Rust.
Update the boundary description to distinguish the C-defined context layout and
configuration-sensitive platform, legacy, and trace leaves from Rust-owned
storage allocation and allocator dispatch.
Test Plan:
- `git diff --cached --check` -- passed
Document that the Rust dictionary-builder modules now own sample analysis,
training, and dictionary assembly, with the original C files retained only as
ABI/build shims. Remove the stale statement that only suffix-array
construction has migrated and keep the remaining C boundary focused on
compression orchestration, benchmark reporting, and format-specific CLI I/O.
Test Plan:
- git diff --cached --check
- make -C tests -j2 test-zstd (passed before this documentation-only commit)
ddidderr