The public ZSTD_compressStream2_c entry point still contained the complete
fallback state machine even though its validation, initialization policy,
serial-versus-MT dispatch, and result accounting were already represented by
Rust policy helpers. That left the main streaming entry point as a large C
orchestration island and made the Rust rewrite boundary misleading.
Project the scalar stream state and private CCtx operations through a checked
C ABI, then let Rust own the fallback ordering. Rust now validates buffers and
the end directive, handles transparent one-shot completion and stable-input
deferral, performs the initialization and stability decisions, selects the
serial or MT path, and publishes the result policy. C retains the private
context layout, stream adapter, MT operations, checksum/epilogue side effects,
and trace/reset callbacks. ABI offset assertions and focused callback-order
and stable-input tests protect the projection without requiring a standalone
Rust test binary to link every C bridge symbol.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed before and after formatting.
- `git diff --cached --check` -- passed.
- Native `make -j1` and the original suite remain the next post-commit verification gates.
Document Rust ownership of compression-size estimator policy and the CLI adaptive feedback state machine, and describe the private C callbacks that remain at those boundaries.
Test Plan:
- Documentation-only change; git diff --cached --check
Update the migration boundary document to record the two latest multithreaded
policy moves. Rust now owns serial-reset LDM normalization, table sizing, and
publication plus final completion trace/reset ordering, while C still owns the
private job/context state and callbacks.
Test Plan:
- `git diff --cached --check` -- passed.
- Capped native build and full original test target were run before this
documentation-only commit and passed.
Synchronize the ownership map with the default-window decompression policy, the Rust-owned single-file --list ordering, and CDict match-state reset policy moved in the latest migration cycle.
Test Plan:
- git diff --cached --check
Document the advanced one-shot begin/end seam, MT job-table teardown order, decompression heap-mode policy, and file-removal status policy so the migration boundary stays synchronized with the implementation.
Test Plan:
- git diff --cached --check
Keep the migration boundary accurate after moving frame-progression dispatch, the decompression no-forward-progress threshold, and compression metadata-transfer selection into Rust. The README now states which scalar inputs Rust owns and which private callbacks, probes, and operations remain in C.
Test Plan:
- git diff --cached --check
- Capped clippy, CLI tests, native build, smoke test, and full upstream suite passed before this documentation-only commit
Move the scalar planning portion of ZSTD_resetCCtx_internal across an
explicit C/Rust boundary. Rust now computes window and block sizes, sequence
capacities, buffered I/O sizes, LDM and external-sequence reservations, the
index-reset policy, and the CCtx workspace estimate. C retains private
workspace resizing, reservations, pointer publication, and layout-dependent
callbacks.
Add ABI layout assertions and focused planner tests for buffered sizing, LDM
and index-reset policy, external sequence capacity, and invalid unadjusted LDM
inputs. Keep the migration boundary documented so the hybrid state remains
explicit while deeper CCtx reset and matchfinder operations stay in C.
Test Plan:
- `cargo fmt --manifest-path rust/Cargo.toml -- --check`
- capped focused planner tests: 3 passed
- capped full Rust library tests: 680 passed, 0 failed
- capped `cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings`: passed
- capped native `make -B -C programs -j1 zstd`: passed; only known fileio const-cast warnings
- capped `make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s`: 84 tests and both short fuzzer rounds passed; only known zstream initializer warning
The CCtx workspace estimator still assembled its private sizing formula in C,
even though the match-state and component-sum leaves already lived in Rust.
That left buffering, LDM, static-context, and external-sequence sizing policy
outside the rewrite boundary.
Move the complete size_t formula into a Rust entry point. C now supplies only
private object sizes, sanitizer redzone policy, and LDM scalar fields through a
small ABI record. Rust retains the C wrapping/alignment behavior and focused
branch coverage while the C adapter preserves the existing public estimator
surface.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml -- --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --release estimate_cctx_workspace_size
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --release (667 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --release --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s (84 tests and both short fuzzer rounds)
ZSTD_createCDict_advanced2 previously selected compression parameters,
resolved dedicated-dictionary-search fallback, and ordered allocation and
initialization entirely in C. That left a large public dictionary boundary
outside the Rust rewrite and made the failure ordering implicit in the C
wrapper.
The Rust params API now owns advanced-CDict parameter preparation, including
the dedicated-search override/fallback and row-matchfinder resolution. A Rust
dictionary bridge owns the create -> init sequence and frees a created CDict
when initialization fails. The C shim retains only the private workspace and
custom-memory allocation, dictionary initialization, and teardown callbacks;
opaque parameter-field pointers and compile-time layout assertions preserve
the existing ABI. Focused probes cover policy publication and all callback
failure-ordering paths, and the migration README records the remaining C
boundary.
Test Plan:
- `rustfmt --check` and `cargo fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `cargo test --manifest-path rust/Cargo.toml --release` -- 663 passed
- `cargo clippy --manifest-path rust/Cargo.toml --release --all-targets -- -D warnings` -- passed
- `make -B -C programs -j1 zstd` -- passed
- `make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s` -- 84 tests and both fuzz rounds passed
Record that Rust now owns advanced CDict one-shot validation and begin/end
ordering while C retains the private operations behind callbacks.
Test Plan:
- Documentation-only change; code validation is covered by 8b783e8.
Record that Rust now owns static CCtx workspace validation and dispatch while
C retains private workspace construction and context initialization.
Test Plan:
- Documentation-only change; code validation is covered by 463ceeb86.
Record that Rust now owns static CDict workspace validation and dispatch while
C retains private workspace construction and dictionary initialization.
Test Plan:
- Documentation-only change; code validation is covered by 12b99469b.
Record that Rust now owns CDict null handling and workspace/object teardown
ordering while C retains private workspace and allocator callbacks.
Test Plan:
- Documentation-only change; code validation is covered by 14157558f.
Record that Rust now owns regular and by-reference CDict constructor parameter
selection and default-level normalization while C retains advanced allocation
and workspace lifecycle.
Test Plan:
- Documentation-only change; code validation is covered by 9f4d92808.
Record that Rust now owns public CCtx allocation validation and
allocate-before-init ordering while C retains private context initialization.
Test Plan:
- Documentation-only change; code validation is covered by aa817a56b.
Record that Rust now owns the legacy CDict-begin fixed frame policy and
unknown-source pledge while C retains the private begin callback.
Test Plan:
- Documentation-only change; code validation is covered by 2fdd548a3.
Record that Rust now owns public advanced-begin parameter validation and
init-then-begin ordering while C retains private context initialization and
begin callbacks.
Test Plan:
- Documentation-only change; code validation is covered by feee05372.
Record that Rust now owns the compressBegin_usingDict family’s unknown-source
parameter selection and default-level normalization while C retains the private
begin callback.
Test Plan:
- Documentation-only change; code validation is covered by 9c0b9bd25.
Record that Rust now owns the public usingCDict fixed-frame policy and
begin/end sequencing while C retains the private CDict compression callbacks.
Test Plan:
- Documentation-only change; code validation is covered by 0fbc9ff37.
Record that Rust now owns the public usingDict parameter-selection,
dictionary-presence, and default-level policy while C retains private context
initialization and compression callbacks.
Test Plan:
- Documentation-only change; code validation is covered by 556cdc53b.
Record that Rust now owns public advanced-compression parameter validation and
initialization ordering while C retains private context setup and execution.
Test Plan:
- git diff --cached --check
Record that Rust now owns public CCtx-copy pledge and frame-parameter policy,
while C retains the private workspace and table-copy operation.
Test Plan:
- git diff --cached --check
Record that Rust now selects the CDict attach-versus-copy reset path while C
retains the workspace- and match-state-heavy reset operations.
Test Plan:
- git diff --cached --check
Record that shared compression-begin dictionary selection now runs in Rust,
while C retains the reset, attach, and private dictionary operation callbacks.
Test Plan:
- git diff --cached --check
Clarify that CDict initialization ordering and scalar publication now run in
Rust, while allocation, private dictionary loading, and match/workspace
callbacks remain C-owned.
Test Plan:
- git diff --cached --check
Record that Rust now owns external-producer invocation and successful-path
validation plus external raw-sequence-store reset, while C retains private
transfer, fallback, and context operations.
Test Plan:
- git diff --cached --check
Record that CDict-begin parameter selection, initialization ordering, and
source-window policy now run in Rust while private CDict construction and the
final C begin operation remain on the C side of the migration boundary.
Test Plan:
- git diff --cached --check
Record the separate-destination decompression scheduler in the Rust migration
boundary and clarify that C still owns destination naming and private I/O,
resource, and format callbacks.
Test Plan:
- git diff --cached --check
Record the shared-destination decompression scheduler as a Rust-owned boundary while documenting that C retains destination, resource, and format callbacks.
Test Plan:
- git diff --cached --check
Document the Rust-owned public end-of-frame compression orchestration and both multi-file compression scheduler boundaries, including the C callbacks that remain responsible for private file and codec operations.
Test Plan:
- git diff --cached --check
Record the Rust-owned stable-buffer fallback orchestration and shared-output
multi-file scheduler in the migration boundary documentation while retaining
the C-owned private context, resource, and callback responsibilities.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
17cfedd56 and 035001b76
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that public sequence API orchestration and scalar CLI adaptive
decisions are Rust-owned while private codec/context state and mutation
callbacks remain in C.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
3b989ab3e and b04893799
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that Rust owns single-threaded and MT stream-initialization policy,
including the ordered setup decisions and scalar projections, while C retains
private contexts and mutation callbacks.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by the
preceding stream-initialization commits
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that Rust now owns CCtx dictionary/prefix attachment dispatch and the
multithreaded compression-job stage sequence, while C retains reset policy,
private context and worker state, dictionary-content loading, and codec
callbacks.
Test Plan:
- Documentation-only change; code validation was completed before this commit.
- Rust library all-target tests: 517 passed; legacy feature matrix: 572 passed.
- Native CLI tests (41), test-zstd, fuzzer (319), zstream (152 + 297), and
decode corpus (1,647) passed under the serial 40 GiB virtual-memory cap.
- Commit is intentionally unsigned because configured GPG pinentry was
unavailable and hung during the signing attempt.
Update the Rust migration map after the latest compression and CLI seams
moved. The fileio summary now distinguishes Rust-owned zstd stream
orchestration from C-owned codec, diagnostics, and adaptive-policy callbacks.
The remaining-boundary summary records that dictionary dispatch and the CLI
zstd compression loop have moved, while CDict lifecycle, private dictionary
content loading, and private compressor operations remain in C.
Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no additional build or test run was needed.
GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
Document that Rust now owns sequence-store construction, block post-build
policy, public sequence conversion, and optional-format decompression loops.
Keep the remaining private C callback and dictionary/CLI boundaries explicit so
the hybrid state is not mistaken for a completed all-Rust rewrite.
Test Plan:
- git diff --check (passed)
Update the Rust component map to describe the newly migrated single-threaded
stream state machine and multithreaded scheduler/flush policy. Correct the CLI
boundary so it no longer claims that the optional-format decompression loops
are Rust-owned, and list the remaining sequence, dictionary, and CLI leaves
that still block an all-Rust build.
Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no code tests required.
Document that the Rust CLI now owns the gzip, xz/lzma, and LZ4 codec leaves as
well as MT job creation, while the generic frame-compression state machine and
remaining MT scheduling/outer-flush orchestration remain C-owned. Keep the
hybrid boundary explicit so compatibility-suite success is not mistaken for
completion of the full rewrite.
Test Plan:
- git diff --cached --check
- reviewed rust/README.md against the committed Rust/C ownership boundary
Document that the frame epilogue, produced-output flush policy, and gzip codec
leaf now execute in Rust. Narrow the remaining C-owned list to the generic
compression state machine, MT scheduling/outer flush orchestration, and the
xz/lzma/lz4 codec leaves so the README remains an accurate progress boundary.
Test Plan:
- git diff --cached --check
ddidderr