Commit Graph
245 Commits
Author SHA1 Message Date
ddidderr 4952b259c4 feat(compress): move CDict begin policy into Rust
Move ZSTD_compressBegin_usingCDict_internal source-size parameter selection,
parameter initialization ordering, and source-window floor into a Rust-owned
ABI bridge while retaining private CCtx/CDict layouts and the final C begin
callback. Preserve strict cutoff, dictionary-size multiplier, unknown-size,
level-zero, and begin-error semantics.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused cdict_begin unit tests
2026-07-19 14:19:43 +02:00
ddidderr 148af6d884 feat(compress): move CCtx free policy into Rust
Move ZSTD_freeCCtx null handling, static-context rejection, content/object
free ordering, and embedded-workspace policy behind a Rust-owned ABI bridge.
C retains dictionary, multithread, workspace, and custom allocator teardown
through callbacks, and ZSTD_freeCStream continues to use the CCtx path.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused free_cctx unit tests
2026-07-19 14:00:21 +02:00
ddidderr 55f28cdd71 feat(compress): move thread-pool attachment policy into Rust
Move ZSTD_CCtx_refThreadPool init-stage validation and pool-slot assignment
behind a Rust-owned ABI bridge while keeping the opaque pool and CCtx storage
in C. Preserve the stage error and avoid mutating the pool on rejection.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused ref_thread_pool unit tests
2026-07-19 13:55:21 +02:00
ddidderr 245cb36f91 feat(compress): move context parameter policy into Rust
Move ZSTD_CCtx_setParameter stage authorization, supported-parameter
validation, static-context worker guard, and max-block side effect into
a Rust-owned ABI bridge while retaining C-owned context and parameter
storage. Preserve update authorization and error ordering.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused set_parameter unit tests
2026-07-19 13:48:06 +02:00
ddidderr b2e7be480a feat(compress): move stored parameter policy into Rust
Move ZSTD_CCtx_setParametersUsingCCtxParams stage, dictionary, and copy policy into a Rust-owned ABI bridge while retaining C-owned parameter object storage. Preserve stage-before-dictionary validation, error codes, and no-mutation behavior on rejected calls.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- make -C tests -j1 test-zstream ZSTREAM_TESTTIME=-T1s
- focused set_parameters_using_cctx_params unit tests
2026-07-19 13:40:12 +02:00
ddidderr 1cf5a92e02 feat(compress): move CCtx reset policy into Rust
Move ZSTD_CCtx_reset directive handling into a Rust-owned state machine while keeping dictionary cleanup and parameter reset behind C callbacks. Preserve the original session ordering, stage error boundary, completion/max-block flag updates, and reset error propagation across the ABI bridge.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j1 zstd
- focused reset_cctx unit tests
2026-07-19 13:22:25 +02:00
ddidderr 878e18a966 feat(compress): move parameter bundle orchestration into Rust
Move ZSTD_CCtx_setParams's check, frame-parameter, and compression-parameter
orchestration into the Rust projection. C retains callbacks for the existing
context-bound operations while Rust preserves the check -> frame -> compression
order and first-error propagation.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml set_params -- --test-threads=1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 13:09:08 +02:00
ddidderr b4e37ad09f feat(compress): move frame parameter updates into Rust
Move ZSTD_CCtx_setFParams's ordered frame-flag updates into the Rust
projection. C retains the context-bound ZSTD_CCtx_setParameter callback while
Rust preserves content-size, checksum, and dictionary-ID boolean conversions,
update order, and first-error propagation.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml set_fparams -- --test-threads=1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:59:56 +02:00
ddidderr 631dc4a361 feat(compress): move C parameter update policy into Rust
Move ZSTD_CCtx_setCParams's validation and ordered parameter-update policy into
the Rust projection. C retains the private ZSTD_CCtx_setParameter callback while
Rust preserves the exact check/window/chain/hash/search/min-match/target/strategy
order and error propagation.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml set_cparams -- --test-threads=1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:48:13 +02:00
ddidderr af1fd3e0fc feat(compress): move advanced stream init policy into Rust
Move ZSTD_initCStream_advanced's public initialization policy into the Rust
projection. Rust owns legacy pledged-size normalization and the reset, pledge,
parameter-check, parameter-copy, and dictionary-load ordering while C retains
the private CCtx parameter mutation callbacks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_advanced -- --test-threads=1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:37:52 +02:00
ddidderr ad1ec27212 feat(compress): move dictionary stream init policy into Rust
Move ZSTD_initCStream_usingDict's public initialization policy into the Rust
projection. Rust now preserves reset, compression-level, and dictionary-load
ordering while C retains the private context and dictionary callbacks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_using_dict -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:29:14 +02:00
ddidderr 3798dfd82c feat(compress): move basic stream init policy into Rust
Move ZSTD_initCStream's public initialization policy into the Rust
projection. Rust now preserves the reset, CDict-clear, and compression-level
ordering while C retains the private context callbacks and ABI boundary.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:22:23 +02:00
ddidderr 1aab448943 feat(compress): move src-size stream init policy into Rust
Move ZSTD_initCStream_srcSize's public initialization policy into the Rust
projection. Rust now normalizes the pledged source size and preserves the
reset, dictionary-clear, compression-level, and pledge ordering while C
retains the private context callbacks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_src_size -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:13:17 +02:00
ddidderr 0de56049cf feat(compress): move cdict stream init policy into Rust
Move ZSTD_initCStream_usingCDict's reset and CDict-reference ordering into a
small Rust projection. Keep the opaque C context and dictionary operation
behind callbacks with an explicit layout-checked ABI boundary.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_using_cdict -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 12:05:29 +02:00
ddidderr b0c03f41f5 feat(compress): move cdict stream init policy into Rust
Move ZSTD_initCStream_usingCDict_advanced's reset, pledge, frame-parameter,
and CDict callback ordering into the Rust projection. Keep private CCtx and
CDict mutations in C callbacks and pass frame parameters as scalars across the
ABI boundary.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_using_cdict_advanced -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 11:58:35 +02:00
ddidderr fe6cf3bccc feat(compress): move resetCStream policy into Rust
Move the public CStream reset wrapper's pledge normalization, callback order,
and error short-circuiting into Rust. Keep the C-owned context reset and
pledged-size mutation behind opaque callbacks with an explicit ABI projection.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml reset_cstream -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
2026-07-19 11:46:18 +02:00
ddidderr 845f77924a feat(cli): move separate decompression scheduling into Rust
Move the separate-destination multi-file decompression loop into the Rust
projection so Rust owns file iteration, progress counters, and aggregate
error handling. Keep destination-name construction, mirror setup, source
opening, format dispatch, diagnostics, and source removal in the C callback
boundary.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
2026-07-19 11:25:59 +02:00
ddidderr 802cb70184 feat(cli): move shared decompression scheduling into Rust
Move the shared-destination branch of FIO_decompressMultipleFilenames into a Rust-owned scheduler. Preserve source order, destination propagation, cursor advancement, success counting, aggregate errors, and C-owned source/format/resource behavior including --rm and output close handling.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_shared_destination -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
2026-07-19 11:10:09 +02:00
ddidderr fbc80de7dc feat(cli): move separate-output scheduling into Rust
Move the separate-destination branch of FIO_compressMultipleFilenames into a Rust-owned scheduler. Preserve source ordering, currFileIdx advancement, successful-file counting, aggregate error handling, and post-loop collision checks while retaining destination construction and private file/resource work in C callbacks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
2026-07-19 10:52:31 +02:00
ddidderr bd7fbc81fc feat(compress): move end-of-frame orchestration into Rust
Move ZSTD_compressEnd_public's continuation, epilogue placement, pledged-size validation, trace ordering, and output accounting into the Rust compression projection. Keep CCtx-dependent continuation, checksum/epilogue, and trace behavior behind opaque C callbacks with compile-time ABI layout checks.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compress_end_ -- --test-threads=1
- ulimit -v 41943040; make -B -C lib -j1 lib
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
2026-07-19 10:52:09 +02:00
ddidderr 035001b769 feat(cli): move shared-destination scheduling into Rust
Move the shared-output arm of FIO_compressMultipleFilenames into a Rust
scheduler. Rust now owns input-order iteration, processed-file accounting,
index advancement, and non-short-circuiting error aggregation while C retains
file validation, private resources, callbacks, diagnostics, and destination
lifetime.

Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml compression_multiple_shared_destination -- --test-threads=1 (2 passed)
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j1 lib
- make -B -C programs -j1 zstd
- make -B -C tests -j1 test-cli-tests (41 passed)
- all commands used CARGO_BUILD_JOBS=1 where applicable and
  ulimit -v 41943040; no heavyweight jobs ran concurrently

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:31:19 +02:00
ddidderr 17cfedd56e feat(compress): move compress2 fallback orchestration into Rust
Move the stable-buffer fallback behind ZSTD_compress2() into a Rust-owned
projection. Rust now owns reset/mode-switch/stream-end ordering and maps
remaining output to dstSize_tooSmall, while C retains the private CCtx reset,
mode mutation, and simple-arguments stream adapter callbacks.

Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml compress2_fallback -- --test-threads=1 (2 passed)
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j1 lib
- make -B -C programs -j1 zstd
- all commands used CARGO_BUILD_JOBS=1 where applicable and
  ulimit -v 41943040; no heavyweight jobs ran concurrently

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:30:25 +02:00
ddidderr b048937992 feat(cli): move adaptive decisions into Rust
Move the scalar runtime --adapt predicates and compression-level normalization
into Rust. Rust now decides output blocking, output backlog, input starvation,
blocked-input speedups, and bounded slower/faster level changes through an
explicit projection.

Keep the CLI iteration order, refresh clock, frame-progression snapshots,
private FIO/ZSTD contexts, diagnostics, and ZSTD_CCtx_setParameter() mutation
in C. The new C/Rust layout assertions make the scalar policy ABI explicit
without exposing those private layouts.

Test Plan:
- focused adaptive-policy tests: 6 passed
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1:
  533 passed
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
  --test-threads=1: 169 passed
- legacy compression/decompression/dictionary-builder feature matrix:
  588 passed
- six library/CLI clippy gates with -D warnings
- capped serial native library/program rebuilds, 41 CLI tests, Rust library
  smoke, and full test-zstd round trips including --adapt cases
- capped serial stress gates: 278 fuzzer cases, 84+129+143 zstream cases,
  and 1,601 decode-corpus cases
- every heavyweight command used CARGO_BUILD_JOBS=1 or make -j1 and
  ulimit -v 41943040; no worker/native process remained afterward

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:12:50 +02:00
ddidderr 3b989ab3ee feat(compress): move sequence API orchestration into Rust
Move the high-level orchestration of ZSTD_compressSequences() and
ZSTD_compressSequencesAndLiterals() into Rust. Rust now owns the validation
precedence, CCtx initialization handoff, frame-header and checksum ordering,
block-loop dispatch, and output accounting for both public sequence APIs.

Keep the private CCtx, sequence-store, block-state, checksum, and conversion
layouts in C. C supplies scalar block projections and callbacks for private
initialization, frame-header emission, checksum operations, and sequence-state
preparation, preserving the ABI boundary and the existing codec leaves.

Test Plan:
- focused sequence API policy tests: 4 passed
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1:
  533 passed
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
  --test-threads=1: 169 passed
- legacy compression/decompression/dictionary-builder feature matrix:
  588 passed
- six library/CLI clippy gates with -D warnings
- capped serial native library/program rebuilds, 41 CLI tests, Rust library
  smoke, and full test-zstd round trips
- capped serial stress gates: 278 fuzzer cases, 84+129+143 zstream cases,
  and 1,601 decode-corpus cases
- every heavyweight command used CARGO_BUILD_JOBS=1 or make -j1 and
  ulimit -v 41943040; no worker/native process remained afterward

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 10:12:21 +02:00
ddidderr 15d42bcc95 feat(compress): move MT stream initialization policy into Rust
Move the high-level ZSTDMT_initCStream_internal setup policy into Rust. Rust
now owns worker-count resizing decisions, job-size normalization, unfinished-
job draining order, overlap and section sizing, rsync setup, buffer sizing,
and stream reset sequencing through a scalar projection and callbacks.

Keep MT contexts, pools, job resources, dictionaries, buffers, synchronization,
and serial state private to C. C callbacks perform those private mutations while
Rust controls the transparent initialization flow and can test its normalization
and ordering independently of the private layouts.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- run the legacy Rust feature matrix and all six library/CLI clippy gates with
  -D warnings
- run lib and program native rebuilds plus test-cli-tests,
  test-rust-lib-smoke, and test-zstd with make -j1
- run fuzzer, zstream, and decode-corpus stress gates serially with
  ulimit -v 41943040

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 09:43:28 +02:00
ddidderr a9589d2d7d feat(compress): move stream initialization policy into Rust
Move the transparent ZSTD_CCtx_init_compressStream2 initialization policy into
Rust. Rust now owns the ordered local-dictionary, prefix, parameter-resolution,
pledged-size, worker-selection, and ordinary-buffering decisions through a
scalar projection and explicit callbacks.

Keep the private CCtx and parameter layouts, allocator and trace state, MT
context lifecycle, codec operations, reset behavior, and mutation details in C.
The C shim therefore remains the ABI and private-state boundary while the
high-level stream setup flow is testable in Rust without duplicating those
layouts.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- run the legacy Rust feature matrix and all six library/CLI clippy gates with
  -D warnings
- run lib and program native rebuilds plus test-cli-tests,
  test-rust-lib-smoke, and test-zstd with make -j1
- run fuzzer, zstream, and decode-corpus stress gates serially with
  ulimit -v 41943040

Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
2026-07-19 09:43:03 +02:00
ddidderr 1655302473 feat(compress): move MT job orchestration into Rust
Move the high-level ZSTDMT compression-job stage sequence into Rust: resource
acquisition, per-job parameter preparation, serial sequence handling, context
initialization, external-sequence application, non-first frame-header repair,
chunk compression/error routing, tracing, and common finalization. Keep C-owned
job descriptors, pools, synchronization, codec contexts, serial state, and
cleanup behind callbacks so private worker state does not cross the boundary.

Test Plan:
- Rust library all-target tests: 517 passed, including MT job-order tests.
- Rust legacy feature matrix: 572 passed.
- Rust and CLI clippy, nightly fmt, native CLI tests (41), and library smoke.
- Native test-zstd, bounded fuzzer (319), zstream (152 + 297), and decode
  corpus (1,647) all passed, including multi-GiB and MT round trips.
- All heavy checks ran serially with CARGO_BUILD_JOBS=1 or make -j1 and
  ulimit -v 41943040 (40 GiB virtual memory).
- Commit is intentionally unsigned because configured GPG pinentry was
  unavailable and hung during the signing attempt.
2026-07-19 09:14:50 +02:00
ddidderr 47b0da7ad7 feat(compress): move CCtx dictionary attachment policy into Rust
Move stage validation, dictionary clearing, and local/CDict/prefix assignment
selection for the three CCtx dictionary APIs into Rust. Keep the private CCtx
layouts, allocator and by-copy lifetime behavior, CDict ownership, reset
policy, and dictionary-content loader behind explicit C callbacks so the
language boundary carries policy rather than private state.

Test Plan:
- Rust library all-target tests: 517 passed.
- Rust legacy feature matrix: 572 passed.
- Rust and CLI clippy, nightly fmt, native CLI tests (41), and library smoke.
- Native test-zstd, bounded fuzzer (319), zstream (152 + 297), and decode
  corpus (1,647) all passed.
- All heavy checks ran serially with CARGO_BUILD_JOBS=1 or make -j1 and
  ulimit -v 41943040 (40 GiB virtual memory).
- Commit is intentionally unsigned because configured GPG pinentry was
  unavailable and hung during the signing attempt.
2026-07-19 09:14:26 +02:00
ddidderr a874ab17fd feat(compress): move dictionary dispatch into Rust
Dictionary insertion still kept its high-level policy in
zstd_compress.c: short and missing dictionary handling, content-type
selection, magic and dictionary-ID processing, entropy-header loading, and
compressed-block-state reset were all interleaved with the private
match-state content loader.

Move that dispatch into ZSTD_rust_compressInsertDictionary. Rust now owns the
mode decisions, reset ordering, dictionary magic and ID semantics, entropy
loading, and error propagation. C retains only a narrow opaque callback for
ZSTD_loadDictionaryContent because that operation depends on private
ZSTD_MatchState_t, ldmState_t, workspace, and parameter layouts. The focused
Rust tests cover short/full errors, raw and auto callback selection, full
dictionary entropy loading, and noDictIDFlag behavior.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml dictionary -- --test-threads=1`
  -- 21 passed under the 40 GiB virtual-memory cap.
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1`
  -- 513 passed under the cap before the test-only clippy sentinel cleanup;
  the focused dictionary suite passed again on the exact staged contents.
- `cargo clippy` lib/benches/tests for `rust` and `rust/cli`, with
  `-D warnings`, and nightly formatting -- passed.
- Native library, CLI, full zstd, fuzzer, zstream, and decode-corpus targets
  -- passed serially under the cap.

GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
2026-07-19 08:49:27 +02:00
ddidderr 423854541b feat(cli): move zstd compression loop into Rust
The CLI's zstd-format compression path previously kept the asynchronous
read, compressStream2, output-job, flush, accounting, adaptive-policy, and
progress loop together in fileio.c. That made the high-level stream
orchestration another large C-owned surface even though the Rust fileio
module already owned the neighboring format loops.

Move the format-independent zstd stream loop and read/output accounting into
FIO_rust_compressZstdFrame. The C adapter now projects read-pool, write-pool,
codec, and policy operations through narrow callbacks. C retains the private
ZSTD_CCtx interaction, adaptive-level policy, memory diagnostics, progress
formatting, and CLI error mapping, so no private C layout crosses into Rust.
Focused seam tests cover input/output ordering, final flush behavior, codec
error propagation, and incomplete known-size input handling.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1`
  -- 513 passed under the 40 GiB virtual-memory cap.
- `cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
  --test-threads=1` -- 169 passed under the cap.
- `cargo clippy` lib/benches/tests for `rust` and `rust/cli`, with
  `-D warnings`, and nightly formatting -- passed.
- `make -B -C lib -j1 lib` and `make -B -C programs -j1 zstd` -- passed.
- Native CLI, full zstd, fuzzer, zstream, and decode-corpus targets -- passed
  serially under the cap.

GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
2026-07-19 08:48:50 +02:00
ddidderr 62389c002f feat(compress): move sequence-store policy into Rust
Move sequence-store construction, small-block handling, repcode setup,
last-literal storage, block-body no-compress transitions, target/split
post-build policy, and public sequence conversion into Rust projections.
Keep matchfinders, LDM, external sequence producers, workspaces, and private
compression context state behind narrow C callbacks and preserve the existing
C-facing sequence API.

Test Plan:
- ulimit -v 41943040; make -B -C lib -j1 lib (passed)
- ulimit -v 41943040; make -B -C tests -j1 test-zstd (passed)
- ulimit -v 41943040; FUZZERTEST=-T5s make -B -C tests -j1 test-fuzzer (284 passed)
- ulimit -v 41943040; ZSTREAM_TESTTIME=-T2s make -B -C tests -j1 test-zstream (210 passed)
- ulimit -v 41943040; DECODECORPUS_TESTTIME=-T10 make -B -C tests -j1 test-decodecorpus (1608 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07 --all-targets -- --test-threads=1 (562 passed)
2026-07-19 08:21:07 +02:00
ddidderr 7282ec4046 feat(cli): move optional decompression loops into Rust
Move gzip, xz/LZMA, and LZ4 frame I/O loops into Rust projections while
keeping zlib, liblzma, LZ4F, asynchronous pool operations, diagnostics, and
private C handles in C callbacks.  Preserve trailing-input accounting and
format-dispatch behavior, and stop faulty LZ4 callbacks from producing an
unbounded no-progress output loop while retaining the codec error value.

Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --lib lz4_decompress_projection_reports_errors_and_unfinished_input -- --test-threads=1 (1 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings (passed)
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests (41 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1 (507 passed before final error-value assertion)
2026-07-19 08:20:46 +02:00
ddidderr dbc6992f3b feat(compress): move stream orchestration into Rust
The high-level single-thread stream path previously combined stable-input
rewind, buffer fill, direct-versus-buffered output, pending flush, end-of-frame
reset, and progress hints in C. The multithreaded path likewise owned input
range selection, rsync and end-directive adjustments, job creation decisions,
and outer flush return policy in C.

Add explicit C/Rust projections and callbacks. Rust now drives the
single-thread state machine and the MT scheduling and flush decision sequence,
while C retains CCtx-sensitive block callbacks, reusable buffers, job
descriptors, synchronization, and worker lifecycle. Focused tests cover stable
and buffered stream behavior, errors, pending output, frame completion, and MT
scheduler branches. Sequence-store and dictionary boundaries remain unchanged.

Test Plan:
- Rust lib and CLI checks, tests, formatting, and clippy passed.
- Rust legacy feature matrix passed: 556 tests.
- Native lib and CLI builds passed; CLI tests passed: 41 tests.
- Native test-zstd, fuzzer, zstream, and decode-corpus gates passed.
2026-07-19 00:14:18 +02:00
ddidderr cb438dc4f1 feat(cli): move decompression result policy into Rust
Move mixed-format decompression status mapping and successful-file finalization
into Rust. C keeps the diagnostic and private FIO context callbacks, while
Rust decides pass-through/error behavior and invokes final accounting only for
a successful decoded file.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-cli-tests
2026-07-18 23:46:17 +02:00
ddidderr 440585995b feat(decompress): move block decoder wrappers into Rust
Move the hidden fullbench block-decoder entrypoints and the public
ZSTD_decompressBlock compatibility wrapper into Rust. The C translation unit
now only projects the configuration-dependent private DCtx fields into the
Rust block context, preserving the existing ABI and decoder feature modes.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-rust-lib-smoke
- make -B -C tests -j2 test-cli-tests
2026-07-18 23:46:08 +02:00
ddidderr dc95de5212 feat(mt): move input-range reuse policy into Rust
Move multithreaded input-range selection and overlap decisions into Rust using
an explicit scalar projection. C retains the round-buffer mutation, LDM
synchronization, prefix copying, and private job state, while Rust decides
whether a reusable section is available and where it begins.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-cli-tests
- make -B -C tests -j2 test-rust-lib-smoke
2026-07-18 23:46:02 +02:00
ddidderr 693b142837 feat(compress): move external sequence literal loop into Rust
Move the external-sequence-and-literals block loop into the Rust compression
module while preserving the C-owned CCtx initialization and sequence
conversion callback. Rust now owns block summaries, literal cursor movement,
entropy emission, block framing, repcode state updates, and completion
validation; C remains the private-context ABI adapter.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-cli-tests
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer
2026-07-18 23:45:55 +02:00
ddidderr fce8d34ffc feat(cli): move xz/lzma and lz4 codec loops into Rust
Move the xz/lzma and LZ4 file-compression orchestration into Rust projection
functions. C retains liblzma and LZ4F contexts, asynchronous pool ownership,
format-specific callbacks, sparse-output handling, and user-facing diagnostics.
The Rust leaves preserve input accounting, output enqueue order, progress
updates, finish/end handling, cleanup, and the original error-code mapping.
Focused projection tests cover success ordering and codec initialization,
header, update, finish, and end failures.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-cli-tests test-zstd
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer
2026-07-18 23:28:53 +02:00
ddidderr 45d54a71ad feat(mt): move compression job creation policy into Rust
Move the multithreaded compression job-creation decision tree into a Rust
projection while keeping C-owned job descriptors, input buffers, pools,
synchronization, worker callbacks, and terminal empty-block serialization
behind explicit callbacks. The Rust policy now preserves ring-table full
checks, prepared-job retry state, prefix advancement, frame checksum rules,
and the non-first empty terminal job path without crossing private C layouts.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07 --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-zstd test-pool test-legacy test-invalidDictionaries
2026-07-18 23:28:36 +02:00
ddidderr 05561c7494 feat(cli): move gzip codec loop into Rust
Move the gzip member read, zlib deflate, finish, output-enqueue, progress,
and accounting loop into Rust. Keep zlib, asynchronous pool, sparse-output,
and diagnostic ownership in C through an explicit projection so optional gzip
builds preserve the existing CLI behavior and error mapping. Add focused tests
for callback ordering, partial input consumption, finish retries, accounting,
and initialization failures.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-cli-tests test-zstd
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
2026-07-18 23:07:51 +02:00
ddidderr 575360c6b1 feat(mt): move produced-output flush into Rust
Move the multithreaded produced-output state machine into Rust while keeping
C-owned job descriptors, synchronization, checksum state, and buffer-pool
cleanup behind projection callbacks. Rust now decides publication progress,
job completion, pending-work results, and frame-completion reporting without
crossing the private ZSTDMT context layout. Add focused tests for partial
output, worker errors, checksum projection, and frame-end state.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07 --all-targets -- --test-threads=1
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-zstd test-rust-lib-smoke
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
2026-07-18 23:07:10 +02:00
ddidderr be7e656d8e feat(cli): move compression format dispatch into Rust
`FIO_compressFilename_internal()` still selected the zstd, gzip, xz/lzma, and
lz4 paths in a C-owned format switch and updated aggregate byte accounting
around those callbacks.  That left the mixed-format CLI dispatch loop outside
the Rust file-I/O orchestration even though the pools and decompression path
were already projected there.

Add a callback table for the codec leaves and move format selection, optional
codec status handling, post-success accounting, and progress-hook ordering to
Rust.  The existing C codec implementations, diagnostics, elapsed-time
formatting, and resource ownership remain unchanged; unsupported optional
formats return explicit statuses so the C wrapper preserves its original
EXM_THROW messages.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1` -- 473 passed.
- Native `test-cli-tests` -- all 41 passed.
- Native `test-zstd` -- passed, including async zstd/gzip/xz/lzma/lz4 round trips.
- `make -B -C programs -j2 zstd` and `make -B -C tests/fuzz -j2 all` -- passed.
2026-07-18 22:41:39 +02:00
ddidderr 4304c0af40 feat(decompress): move legacy dispatch into Rust
Legacy detection, frame-size queries, one-shot decode, and buffered streaming
were still routed through the C header dispatcher even though every historical
v0.1 through v0.7 decoder is already ported under Rust feature gates.  The C
implementation also duplicated the support-level boundary independently from
the Rust archive selection.

Move the dispatcher and stream lifecycle to Rust using the existing DCtx field
projection.  Rust now selects the version, preserves the null-buffer and
position semantics, initializes and frees buffered legacy contexts, and calls
the frozen per-version modules.  C retains only the configuration leaf that
reports ZSTD_LEGACY_SUPPORT plus the private DCtx layout and trace helpers, so
library, program, and test builds can still choose different support levels.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07 --all-targets -- --test-threads=1` -- 528 passed.
- Native `test-legacy`, `test-invalidDictionaries`, `test-decodecorpus`, and `test-zstd` -- passed.
- Strict C syntax checks with legacy support 0, 4, 5, and 7 -- passed.
- `cargo clippy` default, benches, and tests for library and CLI -- passed.
2026-07-18 22:41:17 +02:00
ddidderr 0e0d87ddf5 feat(mt): move empty-job emission into Rust
The multithreaded compressor still handled the terminal empty-job case in C,
including the decision to acquire a buffer, emit the final empty block, and
clear the consumed source range.  That left a small but stateful branch outside
the Rust job helpers and made allocation-failure behavior difficult to test.

Project the terminal job and result through a stable C ABI, then let Rust own
that branch's assertions, buffer callback, block-header serialization, and
failure-preserving result.  C remains responsible for its private job layout
and applies the returned buffer, source, and compressed-size fields exactly as
before.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1` -- 473 passed.
- Native multithreaded cases in `test-fuzzer`, `test-zstream`, `test-cli-tests`, and `test-zstd` -- passed.
- `make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=1 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=0` -- passed.
- `make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=0 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=1` -- passed.
2026-07-18 22:40:56 +02:00
ddidderr 217cc76e74 feat(compress): move continue orchestration into Rust
The public and deprecated block APIs previously entered a C-owned
ZSTD_compressContinue_internal routine, which mixed frame-header sequencing,
window maintenance, block dispatch, and frame-size accounting with the private
CCtx match state.  That made the high-level compression boundary both harder
to test and easy to diverge from the already-ported frame-chunk body.

Add an explicit C/Rust projection containing only scalar state and callback
slots.  Rust now owns stage validation and transitions, header sequencing,
frame-versus-block dispatch, and consumed/produced progression, while C
callbacks retain access to the private match state and context-sensitive block
operations.  The same path now covers compressContinue, deprecated block
compression, and compressEnd without exposing the unstable CCtx layout.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1` -- 473 passed.
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07 --all-targets -- --test-threads=1` -- 528 passed.
- Native `test-fuzzer`, `test-zstream`, `test-decodecorpus`, `test-cli-tests`, and `test-zstd` -- passed.
- `cargo clippy` default, benches, and tests for library and CLI -- passed.
2026-07-18 22:40:35 +02:00
ddidderr 139b45832e feat(cli): move mixed-format dispatch into Rust
Move file I/O's format probing, frame classification, callback dispatch, and
size accounting into Rust. C retains the codec-specific decompression,
pass-through, metadata, and private resource adapters, while Rust coordinates
mixed-format streams and maps callback outcomes back to the CLI error paths.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-cli-tests
- make -B -C tests -j2 test-zstd
2026-07-18 22:09:05 +02:00
ddidderr 2a588f89f2 feat(decompress): move sequence decoder policy into Rust
Port sequence-decoder selection and offset-history policy into Rust. The
Rust block decoder now handles short/long selection, offset-table analysis,
history thresholds, and prefetch sequencing; C projects only the decoder mode
and private context fields required by the ABI.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=1 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=0
- make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=0 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=1
- make -B -C tests -j2 test-zstd
2026-07-18 22:08:56 +02:00
ddidderr 380d8075c6 feat(mt): move job-ring decisions into Rust
Move multithreaded compression's input-retention and pending-output policy
behind scalar job projections. Rust scans the ring and selects source or
prefix ranges while C keeps job descriptors, mutexes, pointer ownership, and
worker-private layout behind the projection callback.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C lib -j2 lib
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-fuzzer
- make -B -C tests -j2 test-zstd
2026-07-18 22:08:46 +02:00
ddidderr f3bc5e98f1 feat(compress): move frame-chunk loop into Rust
Port ZSTD_compress_frameChunk's per-block orchestration into Rust behind an
explicit C-layout projection and callback table. Rust now owns block sizing,
target/split/internal dispatch, block framing and accounting, checksum
sequencing, and terminal frame-state updates. C retains CCtx and match-state
preparation plus the codec-specific callbacks.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C lib -j2 lib
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-zstd
2026-07-18 22:08:37 +02:00
ddidderr 896397f729 fix(rust): satisfy current clippy lints
The current toolchain promoted two test-only patterns to hard errors under
`-D warnings`: integer-to-pointer casts used as non-null dictionary-handle
sentinels, and a repeat/take construction that Clippy now replaces with
`repeat_n`. The casts never represented addresses and were only placeholders
for ownership fields that the cleanup functions must clear. Use the standard
dangling-pointer constructor for those sentinels and the equivalent concise
iterator for the generated test input. Runtime behavior and the ABI are
unchanged, while the repository's Rust lint gate is warning-free again.

Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/Cargo.toml --benches -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml` -- passed
- `cargo +nightly fmt --manifest-path rust/cli/Cargo.toml` -- passed
2026-07-18 21:46:36 +02:00