Commit Graph
11606 Commits
Author SHA1 Message Date
ddidderr 9a68253710 feat(compress): move external producer success path into Rust
Move block-level external sequence producer invocation, result
post-processing, source-length validation, and transfer ordering into a
Rust-owned ABI leaf. Retain the private CCtx transfer callback and C-side
fallback/block-compressor selection, preserving producer-error fallback
eligibility and direct invalid-sequence/transfer failures.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused external_sequence_producer unit tests
2026-07-19 14:41:09 +02:00
ddidderr 724c7c60fa feat(compress): move CDict query leaves into Rust
Move CDict compression-parameter and dictionary-ID snapshot reads behind a
Rust-owned projection while retaining the private C field access and the
existing non-null assertion for ZSTD_getCParamsFromCDict. Preserve the public
NULL dictionary-ID result and ABI-compatible compression-parameter return.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused cdict_query_projection unit tests
2026-07-19 14:26:34 +02:00
ddidderr 993fd0acbe docs(rust): document CDict begin boundary
Record that CDict-begin parameter selection, initialization ordering, and
source-window policy now run in Rust while private CDict construction and the
final C begin operation remain on the C side of the migration boundary.

Test Plan:
- git diff --cached --check
2026-07-19 14:20:30 +02:00
ddidderr 4952b259c4 feat(compress): move CDict begin policy into Rust
Move ZSTD_compressBegin_usingCDict_internal source-size parameter selection,
parameter initialization ordering, and source-window floor into a Rust-owned
ABI bridge while retaining private CCtx/CDict layouts and the final C begin
callback. Preserve strict cutoff, dictionary-size multiplier, unknown-size,
level-zero, and begin-error semantics.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused cdict_begin unit tests
2026-07-19 14:19:43 +02:00
ddidderr 148af6d884 feat(compress): move CCtx free policy into Rust
Move ZSTD_freeCCtx null handling, static-context rejection, content/object
free ordering, and embedded-workspace policy behind a Rust-owned ABI bridge.
C retains dictionary, multithread, workspace, and custom allocator teardown
through callbacks, and ZSTD_freeCStream continues to use the CCtx path.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused free_cctx unit tests
2026-07-19 14:00:21 +02:00
ddidderr 55f28cdd71 feat(compress): move thread-pool attachment policy into Rust
Move ZSTD_CCtx_refThreadPool init-stage validation and pool-slot assignment
behind a Rust-owned ABI bridge while keeping the opaque pool and CCtx storage
in C. Preserve the stage error and avoid mutating the pool on rejection.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused ref_thread_pool unit tests
2026-07-19 13:55:21 +02:00
ddidderr 245cb36f91 feat(compress): move context parameter policy into Rust
Move ZSTD_CCtx_setParameter stage authorization, supported-parameter
validation, static-context worker guard, and max-block side effect into
a Rust-owned ABI bridge while retaining C-owned context and parameter
storage. Preserve update authorization and error ordering.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused set_parameter unit tests
2026-07-19 13:48:06 +02:00
ddidderr b2e7be480a feat(compress): move stored parameter policy into Rust
Move ZSTD_CCtx_setParametersUsingCCtxParams stage, dictionary, and copy policy into a Rust-owned ABI bridge while retaining C-owned parameter object storage. Preserve stage-before-dictionary validation, error codes, and no-mutation behavior on rejected calls.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused set_parameters_using_cctx_params unit tests
2026-07-19 13:40:12 +02:00
ddidderr 1cf5a92e02 feat(compress): move CCtx reset policy into Rust
Move ZSTD_CCtx_reset directive handling into a Rust-owned state machine while keeping dictionary cleanup and parameter reset behind C callbacks. Preserve the original session ordering, stage error boundary, completion/max-block flag updates, and reset error propagation across the ABI bridge.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- focused reset_cctx unit tests
2026-07-19 13:22:25 +02:00
ddidderr 878e18a966 feat(compress): move parameter bundle orchestration into Rust
Move ZSTD_CCtx_setParams's check, frame-parameter, and compression-parameter
orchestration into the Rust projection. C retains callbacks for the existing
context-bound operations while Rust preserves the check -> frame -> compression
order and first-error propagation.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml set_params -- --test-threads=1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 13:09:08 +02:00
ddidderr b4e37ad09f feat(compress): move frame parameter updates into Rust
Move ZSTD_CCtx_setFParams's ordered frame-flag updates into the Rust
projection. C retains the context-bound ZSTD_CCtx_setParameter callback while
Rust preserves content-size, checksum, and dictionary-ID boolean conversions,
update order, and first-error propagation.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml set_fparams -- --test-threads=1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:59:56 +02:00
ddidderr 631dc4a361 feat(compress): move C parameter update policy into Rust
Move ZSTD_CCtx_setCParams's validation and ordered parameter-update policy into
the Rust projection. C retains the private ZSTD_CCtx_setParameter callback while
Rust preserves the exact check/window/chain/hash/search/min-match/target/strategy
order and error propagation.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml set_cparams -- --test-threads=1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:48:13 +02:00
ddidderr af1fd3e0fc feat(compress): move advanced stream init policy into Rust
Move ZSTD_initCStream_advanced's public initialization policy into the Rust
projection. Rust owns legacy pledged-size normalization and the reset, pledge,
parameter-check, parameter-copy, and dictionary-load ordering while C retains
the private CCtx parameter mutation callbacks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_advanced -- --test-threads=1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:37:52 +02:00
ddidderr ad1ec27212 feat(compress): move dictionary stream init policy into Rust
Move ZSTD_initCStream_usingDict's public initialization policy into the Rust
projection. Rust now preserves reset, compression-level, and dictionary-load
ordering while C retains the private context and dictionary callbacks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_using_dict -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:29:14 +02:00
ddidderr 3798dfd82c feat(compress): move basic stream init policy into Rust
Move ZSTD_initCStream's public initialization policy into the Rust
projection. Rust now preserves the reset, CDict-clear, and compression-level
ordering while C retains the private context callbacks and ABI boundary.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:22:23 +02:00
ddidderr 1aab448943 feat(compress): move src-size stream init policy into Rust
Move ZSTD_initCStream_srcSize's public initialization policy into the Rust
projection. Rust now normalizes the pledged source size and preserves the
reset, dictionary-clear, compression-level, and pledge ordering while C
retains the private context callbacks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_src_size -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:13:17 +02:00
ddidderr 0de56049cf feat(compress): move cdict stream init policy into Rust
Move ZSTD_initCStream_usingCDict's reset and CDict-reference ordering into a
small Rust projection. Keep the opaque C context and dictionary operation
behind callbacks with an explicit layout-checked ABI boundary.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_using_cdict -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:05:29 +02:00
ddidderr b0c03f41f5 feat(compress): move cdict stream init policy into Rust
Move ZSTD_initCStream_usingCDict_advanced's reset, pledge, frame-parameter,
and CDict callback ordering into the Rust projection. Keep private CCtx and
CDict mutations in C callbacks and pass frame parameters as scalars across the
ABI boundary.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_using_cdict_advanced -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 11:58:35 +02:00
ddidderr fe6cf3bccc feat(compress): move resetCStream policy into Rust
Move the public CStream reset wrapper's pledge normalization, callback order,
and error short-circuiting into Rust. Keep the C-owned context reset and
pledged-size mutation behind opaque callbacks with an explicit ABI projection.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml reset_cstream -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 11:46:18 +02:00
ddidderr df6ecd0e2f docs(rust): document separate decompression scheduler
Record the separate-destination decompression scheduler in the Rust migration
boundary and clarify that C still owns destination naming and private I/O,
resource, and format callbacks.

Test Plan:
- git diff --cached --check
2026-07-19 11:26:19 +02:00
ddidderr 845f77924a feat(cli): move separate decompression scheduling into Rust
Move the separate-destination multi-file decompression loop into the Rust
projection so Rust owns file iteration, progress counters, and aggregate
error handling. Keep destination-name construction, mirror setup, source
opening, format dispatch, diagnostics, and source removal in the C callback
boundary.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
2026-07-19 11:25:59 +02:00
ddidderr bc8c005c23 docs(rust): document shared decompression scheduler
Record the shared-destination decompression scheduler as a Rust-owned boundary while documenting that C retains destination, resource, and format callbacks.

Test Plan:
- git diff --cached --check
2026-07-19 11:10:29 +02:00
ddidderr 802cb70184 feat(cli): move shared decompression scheduling into Rust
Move the shared-destination branch of FIO_decompressMultipleFilenames into a Rust-owned scheduler. Preserve source order, destination propagation, cursor advancement, success counting, aggregate errors, and C-owned source/format/resource behavior including --rm and output close handling.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_shared_destination -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
2026-07-19 11:10:09 +02:00
ddidderr 4dc6761e60 docs(rust): refresh migration boundary
Document the Rust-owned public end-of-frame compression orchestration and both multi-file compression scheduler boundaries, including the C callbacks that remain responsible for private file and codec operations.

Test Plan:
- git diff --cached --check
2026-07-19 10:52:42 +02:00
ddidderr fbc80de7dc feat(cli): move separate-output scheduling into Rust
Move the separate-destination branch of FIO_compressMultipleFilenames into a Rust-owned scheduler. Preserve source ordering, currFileIdx advancement, successful-file counting, aggregate error handling, and post-loop collision checks while retaining destination construction and private file/resource work in C callbacks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
2026-07-19 10:52:31 +02:00
ddidderr bd7fbc81fc feat(compress): move end-of-frame orchestration into Rust
Move ZSTD_compressEnd_public's continuation, epilogue placement, pledged-size validation, trace ordering, and output accounting into the Rust compression projection. Keep CCtx-dependent continuation, checksum/epilogue, and trace behavior behind opaque C callbacks with compile-time ABI layout checks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compress_end_ -- --test-threads=1
- ulimit -v 41943040; make -B -C lib -j1 lib
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
2026-07-19 10:52:09 +02:00
ddidderr 3b2e8155cd docs(rust): document compress2 and multi-file boundaries
Record the Rust-owned stable-buffer fallback orchestration and shared-output
multi-file scheduler in the migration boundary documentation while retaining
the C-owned private context, resource, and callback responsibilities.

Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
  17cfedd56 and 035001b76

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:32:02 +02:00
ddidderr 035001b769 feat(cli): move shared-destination scheduling into Rust
Move the shared-output arm of FIO_compressMultipleFilenames into a Rust
scheduler. Rust now owns input-order iteration, processed-file accounting,
index advancement, and non-short-circuiting error aggregation while C retains
file validation, private resources, callbacks, diagnostics, and destination
lifetime.

Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml compression_multiple_shared_destination -- --test-threads=1 (2 passed)
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j1 lib
- make -B -C programs -j1 zstd
- make -B -C tests -j1 test-cli-tests (41 passed)
- all commands used CARGO_BUILD_JOBS=1 where applicable and
  ulimit -v 41943040; no heavyweight jobs ran concurrently

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:31:19 +02:00
ddidderr 17cfedd56e feat(compress): move compress2 fallback orchestration into Rust
Move the stable-buffer fallback behind ZSTD_compress2() into a Rust-owned
projection. Rust now owns reset/mode-switch/stream-end ordering and maps
remaining output to dstSize_tooSmall, while C retains the private CCtx reset,
mode mutation, and simple-arguments stream adapter callbacks.

Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml compress2_fallback -- --test-threads=1 (2 passed)
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j1 lib
- make -B -C programs -j1 zstd
- all commands used CARGO_BUILD_JOBS=1 where applicable and
  ulimit -v 41943040; no heavyweight jobs ran concurrently

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:30:25 +02:00
ddidderr 11120fd70c docs(rust): document sequence and adaptive boundaries
Document that public sequence API orchestration and scalar CLI adaptive
decisions are Rust-owned while private codec/context state and mutation
callbacks remain in C.

Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
  3b989ab3e and b04893799

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:15:01 +02:00
ddidderr b048937992 feat(cli): move adaptive decisions into Rust
Move the scalar runtime --adapt predicates and compression-level normalization
into Rust. Rust now decides output blocking, output backlog, input starvation,
blocked-input speedups, and bounded slower/faster level changes through an
explicit projection.

Keep the CLI iteration order, refresh clock, frame-progression snapshots,
private FIO/ZSTD contexts, diagnostics, and ZSTD_CCtx_setParameter() mutation
in C. The new C/Rust layout assertions make the scalar policy ABI explicit
without exposing those private layouts.

Test Plan:
- focused adaptive-policy tests: 6 passed
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1:
  533 passed
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
  --test-threads=1: 169 passed
- legacy compression/decompression/dictionary-builder feature matrix:
  588 passed
- six library/CLI clippy gates with -D warnings
- capped serial native library/program rebuilds, 41 CLI tests, Rust library
  smoke, and full test-zstd round trips including --adapt cases
- capped serial stress gates: 278 fuzzer cases, 84+129+143 zstream cases,
  and 1,601 decode-corpus cases
- every heavyweight command used CARGO_BUILD_JOBS=1 or make -j1 and
  ulimit -v 41943040; no worker/native process remained afterward

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:12:50 +02:00
ddidderr 3b989ab3ee feat(compress): move sequence API orchestration into Rust
Move the high-level orchestration of ZSTD_compressSequences() and
ZSTD_compressSequencesAndLiterals() into Rust. Rust now owns the validation
precedence, CCtx initialization handoff, frame-header and checksum ordering,
block-loop dispatch, and output accounting for both public sequence APIs.

Keep the private CCtx, sequence-store, block-state, checksum, and conversion
layouts in C. C supplies scalar block projections and callbacks for private
initialization, frame-header emission, checksum operations, and sequence-state
preparation, preserving the ABI boundary and the existing codec leaves.

Test Plan:
- focused sequence API policy tests: 4 passed
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1:
  533 passed
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
  --test-threads=1: 169 passed
- legacy compression/decompression/dictionary-builder feature matrix:
  588 passed
- six library/CLI clippy gates with -D warnings
- capped serial native library/program rebuilds, 41 CLI tests, Rust library
  smoke, and full test-zstd round trips
- capped serial stress gates: 278 fuzzer cases, 84+129+143 zstream cases,
  and 1,601 decode-corpus cases
- every heavyweight command used CARGO_BUILD_JOBS=1 or make -j1 and
  ulimit -v 41943040; no worker/native process remained afterward

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:12:21 +02:00
ddidderr fa37b1d533 docs(rust): document stream initialization boundary
Document that Rust owns single-threaded and MT stream-initialization policy,
including the ordered setup decisions and scalar projections, while C retains
private contexts and mutation callbacks.

Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by the
  preceding stream-initialization commits

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 09:43:49 +02:00
ddidderr 15d42bcc95 feat(compress): move MT stream initialization policy into Rust
Move the high-level ZSTDMT_initCStream_internal setup policy into Rust. Rust
now owns worker-count resizing decisions, job-size normalization, unfinished-
job draining order, overlap and section sizing, rsync setup, buffer sizing,
and stream reset sequencing through a scalar projection and callbacks.

Keep MT contexts, pools, job resources, dictionaries, buffers, synchronization,
and serial state private to C. C callbacks perform those private mutations while
Rust controls the transparent initialization flow and can test its normalization
and ordering independently of the private layouts.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- run the legacy Rust feature matrix and all six library/CLI clippy gates with
  -D warnings
- run lib and program native rebuilds plus test-cli-tests,
  test-rust-lib-smoke, and test-zstd with make -j1
- run fuzzer, zstream, and decode-corpus stress gates serially with
  ulimit -v 41943040

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 09:43:28 +02:00
ddidderr a9589d2d7d feat(compress): move stream initialization policy into Rust
Move the transparent ZSTD_CCtx_init_compressStream2 initialization policy into
Rust. Rust now owns the ordered local-dictionary, prefix, parameter-resolution,
pledged-size, worker-selection, and ordinary-buffering decisions through a
scalar projection and explicit callbacks.

Keep the private CCtx and parameter layouts, allocator and trace state, MT
context lifecycle, codec operations, reset behavior, and mutation details in C.
The C shim therefore remains the ABI and private-state boundary while the
high-level stream setup flow is testable in Rust without duplicating those
layouts.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- run the legacy Rust feature matrix and all six library/CLI clippy gates with
  -D warnings
- run lib and program native rebuilds plus test-cli-tests,
  test-rust-lib-smoke, and test-zstd with make -j1
- run fuzzer, zstream, and decode-corpus stress gates serially with
  ulimit -v 41943040

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 09:43:03 +02:00
ddidderr 492c6292c9 docs(rust): refresh compression migration boundary
Document that Rust now owns CCtx dictionary/prefix attachment dispatch and the
multithreaded compression-job stage sequence, while C retains reset policy,
private context and worker state, dictionary-content loading, and codec
callbacks.

Test Plan:
- Documentation-only change; code validation was completed before this commit.
- Rust library all-target tests: 517 passed; legacy feature matrix: 572 passed.
- Native CLI tests (41), test-zstd, fuzzer (319), zstream (152 + 297), and
  decode corpus (1,647) passed under the serial 40 GiB virtual-memory cap.
- Commit is intentionally unsigned because configured GPG pinentry was
  unavailable and hung during the signing attempt.
2026-07-19 09:15:05 +02:00
ddidderr 1655302473 feat(compress): move MT job orchestration into Rust
Move the high-level ZSTDMT compression-job stage sequence into Rust: resource
acquisition, per-job parameter preparation, serial sequence handling, context
initialization, external-sequence application, non-first frame-header repair,
chunk compression/error routing, tracing, and common finalization. Keep C-owned
job descriptors, pools, synchronization, codec contexts, serial state, and
cleanup behind callbacks so private worker state does not cross the boundary.

Test Plan:
- Rust library all-target tests: 517 passed, including MT job-order tests.
- Rust legacy feature matrix: 572 passed.
- Rust and CLI clippy, nightly fmt, native CLI tests (41), and library smoke.
- Native test-zstd, bounded fuzzer (319), zstream (152 + 297), and decode
  corpus (1,647) all passed, including multi-GiB and MT round trips.
- All heavy checks ran serially with CARGO_BUILD_JOBS=1 or make -j1 and
  ulimit -v 41943040 (40 GiB virtual memory).
- Commit is intentionally unsigned because configured GPG pinentry was
  unavailable and hung during the signing attempt.
2026-07-19 09:14:50 +02:00
ddidderr 47b0da7ad7 feat(compress): move CCtx dictionary attachment policy into Rust
Move stage validation, dictionary clearing, and local/CDict/prefix assignment
selection for the three CCtx dictionary APIs into Rust. Keep the private CCtx
layouts, allocator and by-copy lifetime behavior, CDict ownership, reset
policy, and dictionary-content loader behind explicit C callbacks so the
language boundary carries policy rather than private state.

Test Plan:
- Rust library all-target tests: 517 passed.
- Rust legacy feature matrix: 572 passed.
- Rust and CLI clippy, nightly fmt, native CLI tests (41), and library smoke.
- Native test-zstd, bounded fuzzer (319), zstream (152 + 297), and decode
  corpus (1,647) all passed.
- All heavy checks ran serially with CARGO_BUILD_JOBS=1 or make -j1 and
  ulimit -v 41943040 (40 GiB virtual memory).
- Commit is intentionally unsigned because configured GPG pinentry was
  unavailable and hung during the signing attempt.
2026-07-19 09:14:26 +02:00
ddidderr 0492e7f7ec docs(rust): refresh migration boundary
Update the Rust migration map after the latest compression and CLI seams
moved. The fileio summary now distinguishes Rust-owned zstd stream
orchestration from C-owned codec, diagnostics, and adaptive-policy callbacks.
The remaining-boundary summary records that dictionary dispatch and the CLI
zstd compression loop have moved, while CDict lifecycle, private dictionary
content loading, and private compressor operations remain in C.

Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no additional build or test run was needed.

GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
2026-07-19 08:50:29 +02:00
ddidderr a874ab17fd feat(compress): move dictionary dispatch into Rust
Dictionary insertion still kept its high-level policy in
zstd_compress.c: short and missing dictionary handling, content-type
selection, magic and dictionary-ID processing, entropy-header loading, and
compressed-block-state reset were all interleaved with the private
match-state content loader.

Move that dispatch into ZSTD_rust_compressInsertDictionary. Rust now owns the
mode decisions, reset ordering, dictionary magic and ID semantics, entropy
loading, and error propagation. C retains only a narrow opaque callback for
ZSTD_loadDictionaryContent because that operation depends on private
ZSTD_MatchState_t, ldmState_t, workspace, and parameter layouts. The focused
Rust tests cover short/full errors, raw and auto callback selection, full
dictionary entropy loading, and noDictIDFlag behavior.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml dictionary -- --test-threads=1`
  -- 21 passed under the 40 GiB virtual-memory cap.
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1`
  -- 513 passed under the cap before the test-only clippy sentinel cleanup;
  the focused dictionary suite passed again on the exact staged contents.
- `cargo clippy` lib/benches/tests for `rust` and `rust/cli`, with
  `-D warnings`, and nightly formatting -- passed.
- Native library, CLI, full zstd, fuzzer, zstream, and decode-corpus targets
  -- passed serially under the cap.

GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
2026-07-19 08:49:27 +02:00
ddidderr 423854541b feat(cli): move zstd compression loop into Rust
The CLI's zstd-format compression path previously kept the asynchronous
read, compressStream2, output-job, flush, accounting, adaptive-policy, and
progress loop together in fileio.c. That made the high-level stream
orchestration another large C-owned surface even though the Rust fileio
module already owned the neighboring format loops.

Move the format-independent zstd stream loop and read/output accounting into
FIO_rust_compressZstdFrame. The C adapter now projects read-pool, write-pool,
codec, and policy operations through narrow callbacks. C retains the private
ZSTD_CCtx interaction, adaptive-level policy, memory diagnostics, progress
formatting, and CLI error mapping, so no private C layout crosses into Rust.
Focused seam tests cover input/output ordering, final flush behavior, codec
error propagation, and incomplete known-size input handling.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1`
  -- 513 passed under the 40 GiB virtual-memory cap.
- `cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
  --test-threads=1` -- 169 passed under the cap.
- `cargo clippy` lib/benches/tests for `rust` and `rust/cli`, with
  `-D warnings`, and nightly formatting -- passed.
- `make -B -C lib -j1 lib` and `make -B -C programs -j1 zstd` -- passed.
- Native CLI, full zstd, fuzzer, zstream, and decode-corpus targets -- passed
  serially under the cap.

GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
2026-07-19 08:48:50 +02:00
ddidderr 747559e619 docs(rust): refresh migration boundary
Document that Rust now owns sequence-store construction, block post-build
policy, public sequence conversion, and optional-format decompression loops.
Keep the remaining private C callback and dictionary/CLI boundaries explicit so
the hybrid state is not mistaken for a completed all-Rust rewrite.

Test Plan:
- git diff --check (passed)
2026-07-19 08:21:56 +02:00
ddidderr 62389c002f feat(compress): move sequence-store policy into Rust
Move sequence-store construction, small-block handling, repcode setup,
last-literal storage, block-body no-compress transitions, target/split
post-build policy, and public sequence conversion into Rust projections.
Keep matchfinders, LDM, external sequence producers, workspaces, and private
compression context state behind narrow C callbacks and preserve the existing
C-facing sequence API.

Test Plan:
- ulimit -v 41943040; make -B -C lib -j1 lib (passed)
- ulimit -v 41943040; make -B -C tests -j1 test-zstd (passed)
- ulimit -v 41943040; FUZZERTEST=-T5s make -B -C tests -j1 test-fuzzer (284 passed)
- ulimit -v 41943040; ZSTREAM_TESTTIME=-T2s make -B -C tests -j1 test-zstream (210 passed)
- ulimit -v 41943040; DECODECORPUS_TESTTIME=-T10 make -B -C tests -j1 test-decodecorpus (1608 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07 --all-targets -- --test-threads=1 (562 passed)
2026-07-19 08:21:07 +02:00
ddidderr 7282ec4046 feat(cli): move optional decompression loops into Rust
Move gzip, xz/LZMA, and LZ4 frame I/O loops into Rust projections while
keeping zlib, liblzma, LZ4F, asynchronous pool operations, diagnostics, and
private C handles in C callbacks.  Preserve trailing-input accounting and
format-dispatch behavior, and stop faulty LZ4 callbacks from producing an
unbounded no-progress output loop while retaining the codec error value.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib lz4_decompress_projection_reports_errors_and_unfinished_input -- --test-threads=1 (1 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings (passed)
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests (41 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1 (507 passed before final error-value assertion)
2026-07-19 08:20:46 +02:00
ddidderr eb1aa3b6dc docs(rust): refresh migration boundary
Update the Rust component map to describe the newly migrated single-threaded
stream state machine and multithreaded scheduler/flush policy. Correct the CLI
boundary so it no longer claims that the optional-format decompression loops
are Rust-owned, and list the remaining sequence, dictionary, and CLI leaves
that still block an all-Rust build.

Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no code tests required.
2026-07-19 00:17:03 +02:00
ddidderr dbc6992f3b feat(compress): move stream orchestration into Rust
The high-level single-thread stream path previously combined stable-input
rewind, buffer fill, direct-versus-buffered output, pending flush, end-of-frame
reset, and progress hints in C. The multithreaded path likewise owned input
range selection, rsync and end-directive adjustments, job creation decisions,
and outer flush return policy in C.

Add explicit C/Rust projections and callbacks. Rust now drives the
single-thread state machine and the MT scheduling and flush decision sequence,
while C retains CCtx-sensitive block callbacks, reusable buffers, job
descriptors, synchronization, and worker lifecycle. Focused tests cover stable
and buffered stream behavior, errors, pending output, frame completion, and MT
scheduler branches. Sequence-store and dictionary boundaries remain unchanged.

Test Plan:
- Rust lib and CLI checks, tests, formatting, and clippy passed.
- Rust legacy feature matrix passed: 556 tests.
- Native lib and CLI builds passed; CLI tests passed: 41 tests.
- Native test-zstd, fuzzer, zstream, and decode-corpus gates passed.
2026-07-19 00:14:18 +02:00
ddidderr e3d8f61ec5 docs(rust): refresh migration boundary
Document the newly Rust-owned external-sequence block loop, MT input-range
policy, block-decoder wrappers, and CLI decompression result policy so the
remaining C orchestration boundary stays explicit.

Test Plan:
- git diff --cached --check
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
2026-07-18 23:47:38 +02:00
ddidderr cb438dc4f1 feat(cli): move decompression result policy into Rust
Move mixed-format decompression status mapping and successful-file finalization
into Rust. C keeps the diagnostic and private FIO context callbacks, while
Rust decides pass-through/error behavior and invokes final accounting only for
a successful decoded file.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-cli-tests
2026-07-18 23:46:17 +02:00
ddidderr 440585995b feat(decompress): move block decoder wrappers into Rust
Move the hidden fullbench block-decoder entrypoints and the public
ZSTD_decompressBlock compatibility wrapper into Rust. The C translation unit
now only projects the configuration-dependent private DCtx fields into the
Rust block context, preserving the existing ABI and decoder feature modes.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-rust-lib-smoke
- make -B -C tests -j2 test-cli-tests
2026-07-18 23:46:08 +02:00
ddidderr dc95de5212 feat(mt): move input-range reuse policy into Rust
Move multithreaded input-range selection and overlap decisions into Rust using
an explicit scalar projection. C retains the round-buffer mutation, LDM
synchronization, prefix copying, and private job state, while Rust decides
whether a reusable section is available and where it begins.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-cli-tests
- make -B -C tests -j2 test-rust-lib-smoke
2026-07-18 23:46:02 +02:00