Move ZSTD_initCStream_srcSize's public initialization policy into the Rust
projection. Rust now normalizes the pledged source size and preserves the
reset, dictionary-clear, compression-level, and pledge ordering while C
retains the private context callbacks.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_src_size -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
Move ZSTD_initCStream_usingCDict's reset and CDict-reference ordering into a
small Rust projection. Keep the opaque C context and dictionary operation
behind callbacks with an explicit layout-checked ABI boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_using_cdict -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
Move ZSTD_initCStream_usingCDict_advanced's reset, pledge, frame-parameter,
and CDict callback ordering into the Rust projection. Keep private CCtx and
CDict mutations in C callbacks and pass frame parameters as scalars across the
ABI boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml init_cstream_using_cdict_advanced -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
Move the public CStream reset wrapper's pledge normalization, callback order,
and error short-circuiting into Rust. Keep the C-owned context reset and
pledged-size mutation behind opaque callbacks with an explicit ABI projection.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml reset_cstream -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- git diff --cached --check
Record the separate-destination decompression scheduler in the Rust migration
boundary and clarify that C still owns destination naming and private I/O,
resource, and format callbacks.
Test Plan:
- git diff --cached --check
Move the separate-destination multi-file decompression loop into the Rust
projection so Rust owns file iteration, progress counters, and aggregate
error handling. Keep destination-name construction, mirror setup, source
opening, format dispatch, diagnostics, and source removal in the C callback
boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Record the shared-destination decompression scheduler as a Rust-owned boundary while documenting that C retains destination, resource, and format callbacks.
Test Plan:
- git diff --cached --check
Move the shared-destination branch of FIO_decompressMultipleFilenames into a Rust-owned scheduler. Preserve source order, destination propagation, cursor advancement, success counting, aggregate errors, and C-owned source/format/resource behavior including --rm and output close handling.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_shared_destination -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Document the Rust-owned public end-of-frame compression orchestration and both multi-file compression scheduler boundaries, including the C callbacks that remain responsible for private file and codec operations.
Test Plan:
- git diff --cached --check
Move the separate-destination branch of FIO_compressMultipleFilenames into a Rust-owned scheduler. Preserve source ordering, currFileIdx advancement, successful-file counting, aggregate error handling, and post-loop collision checks while retaining destination construction and private file/resource work in C callbacks.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Record the Rust-owned stable-buffer fallback orchestration and shared-output
multi-file scheduler in the migration boundary documentation while retaining
the C-owned private context, resource, and callback responsibilities.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
17cfedd56 and 035001b76
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Move the shared-output arm of FIO_compressMultipleFilenames into a Rust
scheduler. Rust now owns input-order iteration, processed-file accounting,
index advancement, and non-short-circuiting error aggregation while C retains
file validation, private resources, callbacks, diagnostics, and destination
lifetime.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml compression_multiple_shared_destination -- --test-threads=1 (2 passed)
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j1 lib
- make -B -C programs -j1 zstd
- make -B -C tests -j1 test-cli-tests (41 passed)
- all commands used CARGO_BUILD_JOBS=1 where applicable and
ulimit -v 41943040; no heavyweight jobs ran concurrently
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Move the stable-buffer fallback behind ZSTD_compress2() into a Rust-owned
projection. Rust now owns reset/mode-switch/stream-end ordering and maps
remaining output to dstSize_tooSmall, while C retains the private CCtx reset,
mode mutation, and simple-arguments stream adapter callbacks.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml compress2_fallback -- --test-threads=1 (2 passed)
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j1 lib
- make -B -C programs -j1 zstd
- all commands used CARGO_BUILD_JOBS=1 where applicable and
ulimit -v 41943040; no heavyweight jobs ran concurrently
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that public sequence API orchestration and scalar CLI adaptive
decisions are Rust-owned while private codec/context state and mutation
callbacks remain in C.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
3b989ab3e and b04893799
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Move the scalar runtime --adapt predicates and compression-level normalization
into Rust. Rust now decides output blocking, output backlog, input starvation,
blocked-input speedups, and bounded slower/faster level changes through an
explicit projection.
Keep the CLI iteration order, refresh clock, frame-progression snapshots,
private FIO/ZSTD contexts, diagnostics, and ZSTD_CCtx_setParameter() mutation
in C. The new C/Rust layout assertions make the scalar policy ABI explicit
without exposing those private layouts.
Test Plan:
- focused adaptive-policy tests: 6 passed
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1:
533 passed
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
--test-threads=1: 169 passed
- legacy compression/decompression/dictionary-builder feature matrix:
588 passed
- six library/CLI clippy gates with -D warnings
- capped serial native library/program rebuilds, 41 CLI tests, Rust library
smoke, and full test-zstd round trips including --adapt cases
- capped serial stress gates: 278 fuzzer cases, 84+129+143 zstream cases,
and 1,601 decode-corpus cases
- every heavyweight command used CARGO_BUILD_JOBS=1 or make -j1 and
ulimit -v 41943040; no worker/native process remained afterward
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Move the high-level orchestration of ZSTD_compressSequences() and
ZSTD_compressSequencesAndLiterals() into Rust. Rust now owns the validation
precedence, CCtx initialization handoff, frame-header and checksum ordering,
block-loop dispatch, and output accounting for both public sequence APIs.
Keep the private CCtx, sequence-store, block-state, checksum, and conversion
layouts in C. C supplies scalar block projections and callbacks for private
initialization, frame-header emission, checksum operations, and sequence-state
preparation, preserving the ABI boundary and the existing codec leaves.
Test Plan:
- focused sequence API policy tests: 4 passed
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1:
533 passed
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
--test-threads=1: 169 passed
- legacy compression/decompression/dictionary-builder feature matrix:
588 passed
- six library/CLI clippy gates with -D warnings
- capped serial native library/program rebuilds, 41 CLI tests, Rust library
smoke, and full test-zstd round trips
- capped serial stress gates: 278 fuzzer cases, 84+129+143 zstream cases,
and 1,601 decode-corpus cases
- every heavyweight command used CARGO_BUILD_JOBS=1 or make -j1 and
ulimit -v 41943040; no worker/native process remained afterward
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that Rust owns single-threaded and MT stream-initialization policy,
including the ordered setup decisions and scalar projections, while C retains
private contexts and mutation callbacks.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by the
preceding stream-initialization commits
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Move the high-level ZSTDMT_initCStream_internal setup policy into Rust. Rust
now owns worker-count resizing decisions, job-size normalization, unfinished-
job draining order, overlap and section sizing, rsync setup, buffer sizing,
and stream reset sequencing through a scalar projection and callbacks.
Keep MT contexts, pools, job resources, dictionaries, buffers, synchronization,
and serial state private to C. C callbacks perform those private mutations while
Rust controls the transparent initialization flow and can test its normalization
and ordering independently of the private layouts.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- run the legacy Rust feature matrix and all six library/CLI clippy gates with
-D warnings
- run lib and program native rebuilds plus test-cli-tests,
test-rust-lib-smoke, and test-zstd with make -j1
- run fuzzer, zstream, and decode-corpus stress gates serially with
ulimit -v 41943040
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Move the transparent ZSTD_CCtx_init_compressStream2 initialization policy into
Rust. Rust now owns the ordered local-dictionary, prefix, parameter-resolution,
pledged-size, worker-selection, and ordinary-buffering decisions through a
scalar projection and explicit callbacks.
Keep the private CCtx and parameter layouts, allocator and trace state, MT
context lifecycle, codec operations, reset behavior, and mutation details in C.
The C shim therefore remains the ABI and private-state boundary while the
high-level stream setup flow is testable in Rust without duplicating those
layouts.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- run the legacy Rust feature matrix and all six library/CLI clippy gates with
-D warnings
- run lib and program native rebuilds plus test-cli-tests,
test-rust-lib-smoke, and test-zstd with make -j1
- run fuzzer, zstream, and decode-corpus stress gates serially with
ulimit -v 41943040
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that Rust now owns CCtx dictionary/prefix attachment dispatch and the
multithreaded compression-job stage sequence, while C retains reset policy,
private context and worker state, dictionary-content loading, and codec
callbacks.
Test Plan:
- Documentation-only change; code validation was completed before this commit.
- Rust library all-target tests: 517 passed; legacy feature matrix: 572 passed.
- Native CLI tests (41), test-zstd, fuzzer (319), zstream (152 + 297), and
decode corpus (1,647) passed under the serial 40 GiB virtual-memory cap.
- Commit is intentionally unsigned because configured GPG pinentry was
unavailable and hung during the signing attempt.
Move the high-level ZSTDMT compression-job stage sequence into Rust: resource
acquisition, per-job parameter preparation, serial sequence handling, context
initialization, external-sequence application, non-first frame-header repair,
chunk compression/error routing, tracing, and common finalization. Keep C-owned
job descriptors, pools, synchronization, codec contexts, serial state, and
cleanup behind callbacks so private worker state does not cross the boundary.
Test Plan:
- Rust library all-target tests: 517 passed, including MT job-order tests.
- Rust legacy feature matrix: 572 passed.
- Rust and CLI clippy, nightly fmt, native CLI tests (41), and library smoke.
- Native test-zstd, bounded fuzzer (319), zstream (152 + 297), and decode
corpus (1,647) all passed, including multi-GiB and MT round trips.
- All heavy checks ran serially with CARGO_BUILD_JOBS=1 or make -j1 and
ulimit -v 41943040 (40 GiB virtual memory).
- Commit is intentionally unsigned because configured GPG pinentry was
unavailable and hung during the signing attempt.
Move stage validation, dictionary clearing, and local/CDict/prefix assignment
selection for the three CCtx dictionary APIs into Rust. Keep the private CCtx
layouts, allocator and by-copy lifetime behavior, CDict ownership, reset
policy, and dictionary-content loader behind explicit C callbacks so the
language boundary carries policy rather than private state.
Test Plan:
- Rust library all-target tests: 517 passed.
- Rust legacy feature matrix: 572 passed.
- Rust and CLI clippy, nightly fmt, native CLI tests (41), and library smoke.
- Native test-zstd, bounded fuzzer (319), zstream (152 + 297), and decode
corpus (1,647) all passed.
- All heavy checks ran serially with CARGO_BUILD_JOBS=1 or make -j1 and
ulimit -v 41943040 (40 GiB virtual memory).
- Commit is intentionally unsigned because configured GPG pinentry was
unavailable and hung during the signing attempt.
Update the Rust migration map after the latest compression and CLI seams
moved. The fileio summary now distinguishes Rust-owned zstd stream
orchestration from C-owned codec, diagnostics, and adaptive-policy callbacks.
The remaining-boundary summary records that dictionary dispatch and the CLI
zstd compression loop have moved, while CDict lifecycle, private dictionary
content loading, and private compressor operations remain in C.
Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no additional build or test run was needed.
GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
Dictionary insertion still kept its high-level policy in
zstd_compress.c: short and missing dictionary handling, content-type
selection, magic and dictionary-ID processing, entropy-header loading, and
compressed-block-state reset were all interleaved with the private
match-state content loader.
Move that dispatch into ZSTD_rust_compressInsertDictionary. Rust now owns the
mode decisions, reset ordering, dictionary magic and ID semantics, entropy
loading, and error propagation. C retains only a narrow opaque callback for
ZSTD_loadDictionaryContent because that operation depends on private
ZSTD_MatchState_t, ldmState_t, workspace, and parameter layouts. The focused
Rust tests cover short/full errors, raw and auto callback selection, full
dictionary entropy loading, and noDictIDFlag behavior.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml dictionary -- --test-threads=1`
-- 21 passed under the 40 GiB virtual-memory cap.
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1`
-- 513 passed under the cap before the test-only clippy sentinel cleanup;
the focused dictionary suite passed again on the exact staged contents.
- `cargo clippy` lib/benches/tests for `rust` and `rust/cli`, with
`-D warnings`, and nightly formatting -- passed.
- Native library, CLI, full zstd, fuzzer, zstream, and decode-corpus targets
-- passed serially under the cap.
GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
The CLI's zstd-format compression path previously kept the asynchronous
read, compressStream2, output-job, flush, accounting, adaptive-policy, and
progress loop together in fileio.c. That made the high-level stream
orchestration another large C-owned surface even though the Rust fileio
module already owned the neighboring format loops.
Move the format-independent zstd stream loop and read/output accounting into
FIO_rust_compressZstdFrame. The C adapter now projects read-pool, write-pool,
codec, and policy operations through narrow callbacks. C retains the private
ZSTD_CCtx interaction, adaptive-level policy, memory diagnostics, progress
formatting, and CLI error mapping, so no private C layout crosses into Rust.
Focused seam tests cover input/output ordering, final flush behavior, codec
error propagation, and incomplete known-size input handling.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1`
-- 513 passed under the 40 GiB virtual-memory cap.
- `cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
--test-threads=1` -- 169 passed under the cap.
- `cargo clippy` lib/benches/tests for `rust` and `rust/cli`, with
`-D warnings`, and nightly formatting -- passed.
- `make -B -C lib -j1 lib` and `make -B -C programs -j1 zstd` -- passed.
- Native CLI, full zstd, fuzzer, zstream, and decode-corpus targets -- passed
serially under the cap.
GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
Document that Rust now owns sequence-store construction, block post-build
policy, public sequence conversion, and optional-format decompression loops.
Keep the remaining private C callback and dictionary/CLI boundaries explicit so
the hybrid state is not mistaken for a completed all-Rust rewrite.
Test Plan:
- git diff --check (passed)
Update the Rust component map to describe the newly migrated single-threaded
stream state machine and multithreaded scheduler/flush policy. Correct the CLI
boundary so it no longer claims that the optional-format decompression loops
are Rust-owned, and list the remaining sequence, dictionary, and CLI leaves
that still block an all-Rust build.
Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no code tests required.
Move mixed-format decompression status mapping and successful-file finalization
into Rust. C keeps the diagnostic and private FIO context callbacks, while
Rust decides pass-through/error behavior and invokes final accounting only for
a successful decoded file.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-cli-tests
Move the hidden fullbench block-decoder entrypoints and the public
ZSTD_decompressBlock compatibility wrapper into Rust. The C translation unit
now only projects the configuration-dependent private DCtx fields into the
Rust block context, preserving the existing ABI and decoder feature modes.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-rust-lib-smoke
- make -B -C tests -j2 test-cli-tests
Move multithreaded input-range selection and overlap decisions into Rust using
an explicit scalar projection. C retains the round-buffer mutation, LDM
synchronization, prefix copying, and private job state, while Rust decides
whether a reusable section is available and where it begins.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-cli-tests
- make -B -C tests -j2 test-rust-lib-smoke
Move the external-sequence-and-literals block loop into the Rust compression
module while preserving the C-owned CCtx initialization and sequence
conversion callback. Rust now owns block summaries, literal cursor movement,
entropy emission, block framing, repcode state updates, and completion
validation; C remains the private-context ABI adapter.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-cli-tests
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer
Document that the Rust CLI now owns the gzip, xz/lzma, and LZ4 codec leaves as
well as MT job creation, while the generic frame-compression state machine and
remaining MT scheduling/outer-flush orchestration remain C-owned. Keep the
hybrid boundary explicit so compatibility-suite success is not mistaken for
completion of the full rewrite.
Test Plan:
- git diff --cached --check
- reviewed rust/README.md against the committed Rust/C ownership boundary
Document that the frame epilogue, produced-output flush policy, and gzip codec
leaf now execute in Rust. Narrow the remaining C-owned list to the generic
compression state machine, MT scheduling/outer flush orchestration, and the
xz/lzma/lz4 codec leaves so the README remains an accurate progress boundary.
Test Plan:
- git diff --cached --check
The migration map still described the optimal parser and benchmark orchestration
as C-owned, and its remaining-work paragraph did not distinguish the newly
ported high-level leaves from the still-private frame and MT orchestration.
That made the README under-report Rust ownership and overstate some remaining
C work.
Update the component map to identify `zstd_opt` price/parse ownership and Rust
benchmark orchestration, and describe the current C boundary as the remaining
frame setup/flush scheduling and format-specific codec leaves. Keep the
hybrid-build warning explicit until those larger orchestration layers move.
Test Plan:
- `git diff --cached --check` -- passed.
- Rust library and CLI Clippy default, benches, and tests -- passed before staging.
- Native library, program, fuzz, CLI, and upstream test matrices -- passed before staging.
`FIO_compressFilename_internal()` still selected the zstd, gzip, xz/lzma, and
lz4 paths in a C-owned format switch and updated aggregate byte accounting
around those callbacks. That left the mixed-format CLI dispatch loop outside
the Rust file-I/O orchestration even though the pools and decompression path
were already projected there.
Add a callback table for the codec leaves and move format selection, optional
codec status handling, post-success accounting, and progress-hook ordering to
Rust. The existing C codec implementations, diagnostics, elapsed-time
formatting, and resource ownership remain unchanged; unsupported optional
formats return explicit statuses so the C wrapper preserves its original
EXM_THROW messages.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1` -- 473 passed.
- Native `test-cli-tests` -- all 41 passed.
- Native `test-zstd` -- passed, including async zstd/gzip/xz/lzma/lz4 round trips.
- `make -B -C programs -j2 zstd` and `make -B -C tests/fuzz -j2 all` -- passed.
Legacy detection, frame-size queries, one-shot decode, and buffered streaming
were still routed through the C header dispatcher even though every historical
v0.1 through v0.7 decoder is already ported under Rust feature gates. The C
implementation also duplicated the support-level boundary independently from
the Rust archive selection.
Move the dispatcher and stream lifecycle to Rust using the existing DCtx field
projection. Rust now selects the version, preserves the null-buffer and
position semantics, initializes and frees buffered legacy contexts, and calls
the frozen per-version modules. C retains only the configuration leaf that
reports ZSTD_LEGACY_SUPPORT plus the private DCtx layout and trace helpers, so
library, program, and test builds can still choose different support levels.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07 --all-targets -- --test-threads=1` -- 528 passed.
- Native `test-legacy`, `test-invalidDictionaries`, `test-decodecorpus`, and `test-zstd` -- passed.
- Strict C syntax checks with legacy support 0, 4, 5, and 7 -- passed.
- `cargo clippy` default, benches, and tests for library and CLI -- passed.
The multithreaded compressor still handled the terminal empty-job case in C,
including the decision to acquire a buffer, emit the final empty block, and
clear the consumed source range. That left a small but stateful branch outside
the Rust job helpers and made allocation-failure behavior difficult to test.
Project the terminal job and result through a stable C ABI, then let Rust own
that branch's assertions, buffer callback, block-header serialization, and
failure-preserving result. C remains responsible for its private job layout
and applies the returned buffer, source, and compressed-size fields exactly as
before.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1` -- 473 passed.
- Native multithreaded cases in `test-fuzzer`, `test-zstream`, `test-cli-tests`, and `test-zstd` -- passed.
- `make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=1 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=0` -- passed.
- `make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=0 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=1` -- passed.
The public and deprecated block APIs previously entered a C-owned
ZSTD_compressContinue_internal routine, which mixed frame-header sequencing,
window maintenance, block dispatch, and frame-size accounting with the private
CCtx match state. That made the high-level compression boundary both harder
to test and easy to diverge from the already-ported frame-chunk body.
Add an explicit C/Rust projection containing only scalar state and callback
slots. Rust now owns stage validation and transitions, header sequencing,
frame-versus-block dispatch, and consumed/produced progression, while C
callbacks retain access to the private match state and context-sensitive block
operations. The same path now covers compressContinue, deprecated block
compression, and compressEnd without exposing the unstable CCtx layout.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1` -- 473 passed.
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07 --all-targets -- --test-threads=1` -- 528 passed.
- Native `test-fuzzer`, `test-zstream`, `test-decodecorpus`, `test-cli-tests`, and `test-zstd` -- passed.
- `cargo clippy` default, benches, and tests for library and CLI -- passed.
Port sequence-decoder selection and offset-history policy into Rust. The
Rust block decoder now handles short/long selection, offset-table analysis,
history thresholds, and prefetch sequencing; C projects only the decoder mode
and private context fields required by the ABI.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=1 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=0
- make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=0 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=1
- make -B -C tests -j2 test-zstd
The current toolchain promoted two test-only patterns to hard errors under
`-D warnings`: integer-to-pointer casts used as non-null dictionary-handle
sentinels, and a repeat/take construction that Clippy now replaces with
`repeat_n`. The casts never represented addresses and were only placeholders
for ownership fields that the cleanup functions must clear. Use the standard
dangling-pointer constructor for those sentinels and the equivalent concise
iterator for the generated test input. Runtime behavior and the ABI are
unchanged, while the repository's Rust lint gate is warning-free again.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/Cargo.toml --benches -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml` -- passed
- `cargo +nightly fmt --manifest-path rust/cli/Cargo.toml` -- passed
Move the post-sequence-store body of ZSTD_compressBlock_internal behind a
small C/Rust state projection. C continues to build the sequence store and
handle the no-compress and sequence-producer error paths, while Rust now owns
sequence collection, entropy emission, the legacy non-first-frame RLE gate,
compressed-block confirmation, and offcode repeat cleanup. Remove the C
wrappers that became dead after those leaves moved behind the Rust body.
The first integration run exposed that leaving the old C finalization label in
place confirmed compressed block state twice, undoing Rust's pointer swap and
breaking a later sparse-file checksum. The C wrapper now returns directly for
the Rust-owned path and retains only its C-owned no-compress cleanup.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1
- cargo check --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-zstd
- make -B -C tests -j2 test-cli-tests
- ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer
- make -B -C tests/fuzz -j2 all
- make -B -C tests/fuzz -j2 sequence_compression_api
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings (pre-existing manual_repeat_n failure)
- cargo clippy --manifest-path rust/Cargo.toml --benches -- -D warnings (pre-existing manual_repeat_n failure)
Move the post-split partition loop out of zstd_compress.c while keeping the
private CCtx, matchfinder sequence-store construction, and split discovery in
C. Rust now receives a layout-asserted projection containing sequence-store
views, block-state slots, workspace and scalar policy, then mirrors the C
loop's dRep/cRep histories, final-literal accounting, repeated single-block
serialization, and final dRep publication.
Remove the obsolete C sequence-store/count/chunk wrappers and the debug-only
C size-estimation path that was coupled to the old loop. Add a focused Rust
fixture covering partition payloads and final literals, and document the new
ownership boundary for the split search and emission paths.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1 (448 passed)
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings (passed)
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings (passed)
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check (passed)
- cargo clippy --tests/--benches remain blocked only by the pre-existing manual_repeat_n lint in one_shot_promotes_nonfirst_rle_blocks
- make -B -C lib -j2 lib (passed)
- make -B -C tests -j2 test-zstd (passed)
- make -B -C tests -j2 test-cli-tests (41 passed)
- ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream (passed)
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer (252 passed)
- make -B -C tests/fuzz -j2 all and sequence_compression_api (passed)
Clarify that the decoder's private C layout remains configuration-sensitive in
C while Rust owns storage allocation, state-machine behavior, and the public
ABI. This keeps the module-level migration boundary consistent with the
allocator bridge now implemented in Rust.
Test Plan:
- `git diff --cached --check` -- passed
The component map still described decoder context allocation as a C-owned
responsibility after the allocation and custom-memory bridges moved into Rust.
Update the boundary description to distinguish the C-defined context layout and
configuration-sensitive platform, legacy, and trace leaves from Rust-owned
storage allocation and allocator dispatch.
Test Plan:
- `git diff --cached --check` -- passed
Document that the Rust dictionary-builder modules now own sample analysis,
training, and dictionary assembly, with the original C files retained only as
ABI/build shims. Remove the stale statement that only suffix-array
construction has migrated and keep the remaining C boundary focused on
compression orchestration, benchmark reporting, and format-specific CLI I/O.
Test Plan:
- git diff --cached --check
- make -C tests -j2 test-zstd (passed before this documentation-only commit)
Move the target-compressed-block body out of zstd_compress.c while keeping
sequence-store construction, matchfinding, context lifetime, and the outer
offcode repeat-mode cleanup in C. Rust now owns the target policy, repeated
block decision, Rust superblock call, compressed-state confirmation, and raw
fallback. The narrow C projection asserts the cross-language layout instead
of exposing ZSTD_CCtx.
Add focused tests for first-block RLE suppression, RLE output, superblock
errors, raw fallback, and state-slot swapping. Remove the C-only maybeRLE
wrapper that became unreachable after the extraction.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-cli-tests
- ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer
The CLI already delegated one zstd frame at a time to Rust, but the C
FIO_decompressFrames loop still owned repeated-frame dispatch. That left
concatenated zstd streams split across the language boundary and made the
Rust frame helper unable to preserve the next mixed-format header itself.
Add a Rust multi-frame adapter that repeatedly probes four buffered bytes,
uses the existing one-frame decoder, accumulates decoded output and progress,
and leaves a following non-zstd header or short trailing input untouched for
C's format dispatcher. Keep C responsible for probing the first format,
non-zstd dispatch, diagnostics, cleanup, and final file accounting. Decoder
errors still preserve the current input for FIO_zstdErrorHelp().
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib fileio_asyncio -- --test-threads=1` -- 20 passed
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `make -B -C tests -j2 test-cli-tests` -- all 41 passed
- `git diff --cached --check` -- passed
- Full root `cargo clippy --all-targets/--benches/--tests -- -D warnings` remains blocked by the pre-existing `manual_repeat_n` warning in `rust/src/zstd_compress.rs` test code; no unrelated change was made.
The standalone fuzz Makefile compiled all seven legacy translation units as
C declaration-only shims while building the Rust archive without the matching
legacy features. It also linked the Rust CLI utility implementation nowhere:
programs/util.c is now only an ABI shim, and libregression.a introduces its
UTIL_* references after the existing fuzz archives have been scanned.
Mirror the selected legacy level in the fuzz Rust feature list and archive
cache directory. Build the helpers-only CLI archive for the migrated utility
functions, make every fuzz target depend on it, and place it after
libregression.a so the linker can extract the needed members.
Test Plan:
- `make -C tests/fuzz -j2 sequence_compression_api` -- passed
- `./tests/fuzz/sequence_compression_api tests/fuzz/sequence_compression_api.c` -- passed
- `git diff --cached --check` -- passed
Move the per-block body of ZSTD_compressSequences_internal behind an explicit
Rust projection. The C wrapper still owns context initialization, public API
validation, frame headers, checksums, and the private CCtx layout. Rust now
owns block sizing and sequence transfer, sequence-store reset, entropy
compression, raw/RLE/compressed block selection, block headers, repcode/state
swapping, repeat-mode transition, and first-block handling.
The bridge passes only the sequence store, block-state pointer slots, workspace,
policy scalars, dictionary size, and the isFirstBlock slot. It does not pass a
CCtx or C callback across the ABI. The tests cover empty-block headers,
capacity errors, first-block RLE restrictions, and entropy fallback decisions.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib zstd_compress -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-cli-tests
- ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream
- FUZZERTEST=-T5s make -C tests -j2 test-fuzzer (covers ZSTD_compressSequences at fuzzer test 190)
- git diff --cached --check
Keep the MT chunk diagnostic after moving chunk iteration into Rust without
retaining a C local that is unused when debug logging is compiled out.
Test Plan:
- make -C lib -j2 lib
- git diff --cached --check
Move the bounded multithreaded output-publication kernel into Rust. Rust now
calculates the flush amount, copies the selected job bytes, and returns the
updated output and job offsets through an explicit C-layout result. C retains
mutex and condition-variable handling, checksum insertion, job retirement,
progress accounting, and the terminal state machine.
The Rust helper validates the scalar bounds before copying and has focused
coverage for empty output space, offset partial flushes, repeated flushes,
complete flushes, sentinels, and invalid bounds.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib zstdmt_compress -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check (blocked by concurrent fileio worker formatting its uncommitted file)
- ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream (worker)
- git diff --cached --check
Update the Rust rewrite documentation to match the current implementation
boundary. The legacy decoder set is complete, and Rust now owns several
CLI file-I/O policy and resource leaves while C retains format dispatch,
metadata, and remaining orchestration. Keep the stated boundary explicit so
passing hybrid tests do not imply that the full rewrite is finished.
Test Plan:
- git diff --cached --check
ZSTDMT_compressionJob kept the chunking kernel in C while Rust already owned
most of the MT pool and progress helpers. The loop mixed compression calls
with mutex and condition-variable publication, making it difficult to test
its exact final-chunk and partial-tail behavior independently.
Move only the chunk-processing kernel into Rust. It calls the existing C
compression entry points through an explicit ABI, reports the final block size
or error in a repr(C) result, and invokes a C progress callback after each
successful intermediate chunk. C retains worker setup, serial and raw-sequence
state, synchronization, error publication, cleanup, and final completion
reporting. Mock compressors cover empty, exact, partial, progress, and error
boundaries without mirroring private MT job state in Rust.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib zstdmt_compress --
--test-threads=1` -- passed (44 tests).
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed.
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed.
- `ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream` -- passed,
including multi-threaded dictionary/checksum coverage.
FIO_openDstFile still performed destination classification, platform-specific
file creation, stdio buffering, and overwrite preparation in C, even though
source opening and removal already used Rust filesystem leaves. That made the
CLI backend's most important output safety path split across two implementations.
Add a Rust destination leaf with a small status ABI. Rust owns sentinel and
same-file classification, regular-file detection, platform binary open and
fdopen, truncation, and buffering. C keeps the user-facing diagnostics,
sparse-mode preference mutation, overwrite prompt, and existing remove-file
wrapper; it performs a second Rust open after an accepted overwrite decision so
no prompt or policy is duplicated. Existing file contents are left untouched
until that C-owned decision is complete.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1`
-- passed (411 tests, including destination classification/open tests).
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings`
-- passed.
- `cargo clippy --manifest-path rust/cli/Cargo.toml --lib
--no-default-features --features cli,compression,decompression,benchmark,
dict-builder -- -D warnings` -- passed.
- Nightly fmt checks for both Rust manifests -- passed.
- `make -B -C tests -j2 test-cli-tests` -- passed (41 tests).
Compression-parameter table selection and adjustment were already implemented
as Rust leaves, but the internal and public ZSTD_getCParams/ZSTD_getParams
helpers still assembled those results in the C translation unit. That left
source-size and dictionary-mode policy duplicated at the C/Rust boundary.
Add Rust implementations for the internal and public getter policies. The
internal helpers preserve a zero source size as a known empty input, while the
public helpers retain the API rule that zero means unknown. The C side keeps
only the build-specific exclusion-mask construction and thin ABI adapters, so
reduced builds continue to select the same available strategy cascade.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1`
-- passed (411 tests).
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings`
-- passed.
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed.
- `make -B -C lib -j2 lib` -- passed.
- Focused tests cover public zero-to-unknown translation, internal zero
semantics, default frame parameters, and dictionary attachment modes.
The Rust simple-compression entry points already owned the dispatch, but
ZSTD_rust_compressCCtxStrategy still called the C parameter cascade to decide
whether a fast Rust frame path was safe. That left a small policy leaf in the
C implementation and made the dispatch boundary harder to test in isolation.
Implement the helper in Rust by using the existing Rust table-selection and
C-parameter-adjustment functions with the same source-size, dictionary-mode,
and automatic-adjustment inputs as the C code. Leave only the C ABI declaration
and add tests for the table thresholds, default-level behavior, and negative
fast-level behavior. The private context reset and fallback compression paths
remain unchanged.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib zstd_compress --
--test-threads=1` -- passed (234 tests).
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed.
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed.
- `make -B -C lib -j2 lib` -- passed.
- `make -C tests -j2 test-rust-lib-smoke` -- passed.
- Full-target clippy remains blocked by pre-existing test-only
`manual_repeat_n` and `manual_dangling_ptr` warnings outside this seam.
The C benchmark translation unit still contained the complete BMK
implementation, so program builds and paramgrill could silently keep using a
second C implementation even though rust/src/benchzstd.rs already provided the
same ABI. Replace that duplicate with a declaration-only shim and retain
compile-time checks for the by-value result and parameter layouts.
The standalone test tools do not link the full CLI archive. Keep their
helpers archive free of the benchmark and trace-only objects, and add a
separate benchmark archive for paramgrill so the shim remains linkable without
pulling program-only trace dependencies into datagen and similar targets.
Test Plan:
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark,dict-builder --lib -- --test-threads=1` -- 165 passed
- `make -B -C programs -j2 zstd` -- passed
- `./programs/zstd -b1 tests/hello` -- passed
- `make -B -C tests paramgrill` -- passed
- `./tests/paramgrill -S tests/hello` -- passed
- `make -C tests -j2 test-fullbench` -- passed
- targeted Rust clippy and nightly rustfmt checks -- passed
- full all-target clippy remains blocked by pre-existing test-only lints in `zstd_compress.rs` and `fileio_backend.rs`
The library reports an absolute maximum compression level of 22, but the CLI
advertises levels 1 through 19 unless `--ultra` is selected. The Rust frontend
used the absolute maximum for both help lines, diverging from the original C
contract and failing the help golden test. Separate the normal help ceiling
from the ultra ceiling and add a focused regression assertion.
Test Plan:
- `cargo fmt --manifest-path rust/cli/Cargo.toml -- --check` -- passed
- `cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings` -- passed
- CLI Rust tests -- 162 passed
- `make -C tests -j2 test-cli-tests` -- all 41 passed
- `git diff --check` -- passed
The v0.2 decoder implementation and C boundary were already Rust-owned, but
its public error predicate, frame-size sentinel behavior, and opaque streaming
context lifecycle lacked focused Rust coverage. Add tests for malformed input,
content-size error reporting, one-shot decoding, reset behavior, and idempotent
context release so future legacy changes are checked at the ABI boundary.
Test Plan:
- Focused v0.2 Cargo tests under both feature sets -- 6 passed each
- `make -C tests -j2 test-legacy` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- Scoped `git diff --check` -- passed
The CLI trace translation unit still contained the complete CSV writer and
strong callback implementations even though the Rust CLI archive already had
the equivalent trace module. Reduce the C file to its public-header shim and
make the Rust implementation authoritative, preserving the exact CSV header,
version assertion, timing fields, and no-trace build behavior. Focused tests
cover header creation and the version-mismatch assertion.
Test Plan:
- CLI tests: 161 default and 127 reduced-feature -- passed
- Focused trace tests: 3/3 in both configurations -- passed
- `make -B -C programs -j2` for all CLI variants -- passed
- Trace compression/decompression smoke test -- passed
- C shim compile with and without `ZSTD_NOTRACE` -- passed
- Clippy, nightly rustfmt, and `git diff --check` -- passed
The optimal block splitter stores up to 196 split points and then needs one
additional partition entry for the terminal sequence boundary consumed by the
C block-emission loop. The previous array declaration provided only the split
slots, so a maximally partitioned block could write the terminal boundary past
the projected Rust/C state. Reserve the extra entry and document the layout;
retain focused tests for equal-cost and estimation-error cases so the splitter
continues to avoid unnecessary or invalid partitions.
Test Plan:
- Focused block-split Rust tests -- 7 passed
- `make -B -C lib -j2 lib` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- Scoped `git diff --check` -- passed
The verbose compression-preference summary was still formatted by the C
file-I/O orchestration layer even though the preference object and the rest of
its policy helpers already lived in Rust. Move the option selection and exact
summary formatting into the Rust CLI archive, keeping the public C function as
a narrow assertion-and-dispatch shim. The Rust formatter preserves the legacy
option spellings, default memory limit, integer casts, and stderr output, with
a focused string-format regression test.
Test Plan:
- `cargo fmt --manifest-path rust/cli/Cargo.toml -- --check` -- passed
- `cargo test --manifest-path rust/cli/Cargo.toml --lib fileio_prefs -- --test-threads=1` -- 46 passed
- `git diff --check` -- passed
Move the active-stage check and pledged-size update behind a scalar Rust ABI
leaf while keeping the CCtx layout and public C entry point in C. Preserve the
stage_wrong error encoding and the unsigned wrap behavior at U64::MAX.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib zstd_compress::tests (80 passed)
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- rustfmt +nightly --edition 2021 rust/src/zstd_compress.rs --check
- make -C programs -j2 zstd
- git diff --check
Route FIO_freeDict through the Rust filesystem backend for malloc-backed and
mapped dictionaries. Keep enum validation in C, pair Windows views with their
file handles, support the non-POSIX malloc fallback, and clear every ownership
field so repeated cleanup remains harmless.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib fileio_backend -- --test-threads=1 (25 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- rustfmt +nightly --edition 2021 rust/src/fileio_backend.rs --check
- make -C programs -j2 zstd
- git diff --check
Keep dictionary-training code out of compressor-only and decompressor-only CLI
archives, where its ZDICT symbols are intentionally absent. Add a dedicated
Rust CLI archive for zstd-dictBuilder, enable the builder feature only for the
full and dictionary-builder programs, and make reduced-feature help use the
feature-independent max-level helper.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib -- --test-threads=1 (158 passed)
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,dict-builder --lib -- --test-threads=1 (124 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,dict-builder --lib -- -D warnings
- make -C programs -j2 zstd zstd-small zstd-frugal zstd-decompress zstd-compress zstd-dictBuilder
- git diff --check
Move the platform-specific dictionary mapping and release operations behind the
Rust CLI archive while keeping size policy, diagnostics, and FIO_Dict_t
ownership decisions in C. POSIX descriptors and Windows handles are cleaned up
on every failure path, and zero-length dictionaries use an owned sentinel so
the existing non-null dictionary invariant remains valid.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib fileio_backend (24 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- make -C programs -j2 zstd
- --mmap-dict compression/decompression round trip with programs/README.md
- programs/zstd --mmap-dict --test on the generated frame
- rustfmt +nightly --edition 2021 rust/src/fileio_backend.rs --check
- git diff --check
Keep the private ZSTD_window_t layout and pointer-difference calculation in C,
then route the size_t-to-U32 conversion and paired limit writes through a small
Rust ABI leaf. This removes the inline C window-clear implementation while
preserving its overflow behavior at both single-threaded and multithreaded
call sites.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib zstd_compress::tests (77 passed)
- make -C programs -j2 zstd
- rustfmt +nightly --edition 2021 rust/src/zstd_compress.rs --check
- make -C tests -j2 test-cli-tests (41 passed)
- make -C tests -j2 test-legacy test-invalidDictionaries test-decodecorpus test-rust-lib-smoke (passed)
- git diff --check
Attached-dictionary matchers derive dictionary and prefix pointers from
32-bit repcode coordinates. A stale offset in the Rust double-fast
post-match path could wrap the subtraction to a large index, while the
C-compatible overlap predicate's intentional unsigned arithmetic then
classified that index as safe. Dictionary training consequently reached an
invalid read in the multithreaded FastCover workers.
Centralize the attached-dictionary coordinate check before pointer formation:
reject disabled or wrapped offsets, keep repcodes no later than the current
position, require four-byte dictionary matches to remain in the translated
dictionary interval, and preserve the active-prefix coordinate region. Apply
the same checked overlap semantics in the optimal parser, where the current
position is available to reject wrapped repcodes. Focused tests cover the
crashing coordinate, dictionary boundaries, prefix boundaries, and valid
active-prefix repcodes.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib zstd_double_fast::tests` -- passed (4 tests)
- `cargo test --manifest-path rust/Cargo.toml --lib zstd_opt::tests::index_overlap_check` -- passed (2 tests)
- `cargo test --manifest-path rust/Cargo.toml --all-targets` -- passed (400 tests)
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed
- `rustfmt +nightly --edition 2021 rust/src/zstd_double_fast.rs rust/src/zstd_opt.rs --check` -- passed
- `make -C programs -j2 zstd` -- passed
- `./programs/zstd --train -B2K tests/tmpCorpusHighCompress -o /tmp/zstd-rust-dict-high-fixed` -- passed
- `make -C tests -j2 test-zstd` -- passed
- Full `cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` remains blocked by the pre-existing `manual_repeat_n` lint at `rust/src/zstd_compress.rs:2351`.
Keep FIO_openSrcFile's stdin sentinel handling, Windows binary-mode setup,
existing diagnostics, and FILE ownership in the C wrapper while moving only
non-stdin source validation and opening behind the Rust CLI ABI. The Rust leaf
returns distinct stat, non-regular, and fopen failure statuses, delegates
metadata and regular/FIFO/block-device classification to the existing utility
ABI, and passes the original C path directly to fopen("rb"). Opaque stat_t and
FILE* pointers cross the boundary, and the output stream is published only on
success; Rust never closes a returned stream.
Focused backend tests cover regular-file reads, missing and directory paths,
unchanged output pointers on failure, empty and spaced paths, and symlinks.
FIFO acceptance remains in the preserved classification order without a test
that could block while opening a named pipe.
Test Plan:
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression --lib fileio_backend` -- 22 passed
- CLI clippy for library, benches, and tests with `cli,compression,decompression,benchmark`, before and after nightly formatting -- clean after the new-code warnings were fixed
- `cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all -- --check` -- passed
- Root Rust clippy with the reduced `compression` feature split for library, benches, and tests -- passed with one pre-existing warning in forbidden `rust/src/zstd_compress.rs`
- `make -B -C programs -j2 zstd` -- passed
- `make -B -C programs -j2 zstd-small zstd-frugal zstd-dictBuilder` -- passed with existing unused-function warnings in compact CLI builds
- `make -C tests -j2 test-cli-tests` -- all 41 CLI tests passed, including file-stat coverage
- `make -C tests -j2 test-zstd` -- reached dictionary training, then hit a segmentation fault in the unrelated concurrent compressor/dictionary path
- `git diff --check` and `git diff --cached --check` -- passed
Include localDict.dict in the eligibility guard for the simple Rust compress2 route. By-reference dictionary loads populate this field without populating dictBuffer, so omitting it incorrectly allowed a dictionary-bearing context to bypass the C dictionary path.
Test Plan:
- ./tests/fuzzer -s9634 -v
- git diff --check
Keep the one-shot Rust frame path on the same ordinary block-emission contract as the C compressor: preserve the initial window index and matcher history across blocks, use direct sequence-store entropy coding, promote non-first RLE blocks, and emit raw blocks below the C compressibility cutoff. Route unsupported strategies through the existing C stateful fallback so all public compression levels retain their expected behavior.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_compress::tests
- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --lib -- -D warnings
- ./tests/fuzzer -s9634 -v
- git diff --cached --check
Keep the C --list orchestration, file ownership, fileInfo_t storage, and
DISPLAYLEVEL/reporting behavior in programs/fileio.c while replacing only the
frame scanner with a guarded Rust FFI leaf. The Rust implementation preserves
the C status values, fread lookahead, large-file seek/tell behavior, frame and
skippable-frame accounting, content-size and window updates, checksum capture,
RLE and invalid-block handling, dictionary-ID aggregation, and truncation
classification without taking ownership of FILE or the output structure.
A C diagnostic bridge keeps the existing CLI messages and warning formatting
outside the scanner. The Rust tests use temporary C streams and focused frame
buffers, including a test-only header API shim so the unit tests remain safe
and link independently of the full CLI binary. The decompression feature guard
also keeps the scanner absent from no-decompression CLI archives.
Test Plan:
- cargo test --no-default-features --features cli,compression,decompression,benchmark --lib fileio_prefs (45 passed)
- cargo test --no-default-features --features cli,compression,decompression,benchmark (152 passed)
- library and CLI clippy for main, benches, and tests, before and after nightly formatting (clean)
- cargo +nightly fmt -- --check for both library and CLI crates (clean)
- make -B -C programs -j2 zstd, zstd-small, zstd-frugal, and zstd-dictBuilder (passed)
- exact --list/-l playTests.sh blocks (passed)
- make -C tests -j2 test-cli-tests (41 passed)
- nm archive guard check confirmed FIO_rust_analyzeFrames only in the decompression-enabled CLI archive
- zstd-decompress remains blocked by the pre-existing DEFAULT_MAX_CLEVEL feature-gating error in rust/src/zstd_cli.rs
- zstd-compress remains blocked by pre-existing unresolved ZDICT_* references; neither blocker is in the owned files
ZSTDMT_expandJobsTable previously kept worker-capacity comparison, old-table
synchronization teardown, replacement allocation, initialization failure cleanup,
and mask updates in C. That left the remaining job-table lifecycle orchestration
outside the Rust storage leaves. Add a narrow Rust ABI that exchanges only
opaque storage and scalar mask state and accepts C callbacks for the private
synchronization lifecycle.
The adapter frees the old table before replacement, reuses the existing Rust
create/free leaves with the caller's custom memory, destroys a partially
initialized replacement before freeing it, and updates jobIDMask only after
successful initialization. C still owns ZSTDMT_CCtx and job descriptors, so
worker-facing fields and platform synchronization stay outside the Rust ABI.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features
compression zstdmt_compress --lib` -- 38 passed.
- `cargo clippy`, `cargo clippy --benches`, and `cargo clippy --tests` -- passed.
- `cargo +nightly fmt --all -- --check` -- passed.
- `make -B -C lib lib-mt` -- passed.
- `make -B -C tests -j2 fullbench poolTests` -- passed.
- `./poolTests` and `./fullbench -i1 -B1000 ../README.md` -- passed.
Keep the C header's ZSTD_wildcopy signature as the enum-facing wrapper, but
move its over-copying implementation into rust/src/mem.rs. The wrapper passes
the enum's int representation to ZSTD_rust_wildcopy, where only the two valid
values are converted to ZstdOverlap. This leaves C's ZSTD_copy16/COPY16
helpers available to their direct compression callers and removes the now
unused COPY8 helper. ZSTD_copy16 is marked unused-safe for C translation units
that include the shared header without calling that direct helper.
The Rust leaf preserves the original first-copy behavior for zero and short
lengths, the source-before-destination 8-byte do-while path, the no-overlap
distance assertion, and the first-then-two-COPY16 loop. Its ABI contract does
not take ownership of caller buffers. ABI tests use 32-byte padded buffers and
exercise no-overlap plus offsets 8 and 15 across the boundary lengths, checking
both copied bytes and guard regions.
Test Plan:
- `cargo test mem::tests` -- passed (4 tests)
- `make lib-nomt` and `make lib-mt` -- passed
- `make -C tests test-zstream` -- passed
- `make -C tests test-fullbench` -- completed; its `-P0` run printed the
existing Scenario 17 diagnostic, but the target returned normally
- `cargo clippy`, `cargo clippy --benches`, `cargo clippy --tests`,
`cargo +nightly fmt`, then the same three clippy commands -- passed on the
final repeat
- `git diff --cached --check` -- passed
The zdict entropy pass previously allocated and drove an opaque C compression
context, keeping eight private context symbols in the Rust link surface. Replace
that path with a private Rust-owned analyzer that selects and adjusts parameters
through existing Rust policy leaves, builds dictionary match tables through the
existing Rust matcher leaves, and projects only the field-level lazy and optimal
state those leaves consume. The sequence store and superblock leaf remain the
source of entropy statistics, including the existing sample-size cap and
compressible-block filtering. Backing allocations stay owned by the analyzer,
so no public API or C compression-context layout is introduced.
Test Plan:
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features compression,dict-builder` -- passed.
- `cargo test --manifest-path rust/Cargo.toml --all-targets --no-default-features --features compression,dict-builder` -- 370 passed.
- Focused entropy-context test -- passed.
- Relevant-feature clippy for the library, benches, and tests with `-D warnings` -- passed.
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed.
- `make -C tests -B -j2 test-zstream` -- passed, including both fuzz phases.
Move dictionary-path statting behind the existing Rust CLI file-I/O backend while keeping FIO_getDictFileStat as the C policy adapter. The adapter still owns the NULL-path early return, errno-based EXM_THROW(31) diagnostic, EXM_THROW(32) regular-file policy, and all dictionary orchestration.
FIO_rust_getDictFileStat crosses only the existing path and stat_t boundary, delegates metadata and regular-file classification to UTIL_stat and UTIL_isRegularFileStat, and returns 0/1/2 for success, stat failure, or non-regular input. Reusing those ABIs preserves the C stat_t representation, errno behavior, and stat's symlink-following semantics. Focused backend tests cover regular, missing, directory, NULL, and symlink paths.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression --lib fileio_backend (15 passed)
- cargo clippy --no-default-features --features compression, including --benches and --tests (passed)
- CLI cargo clippy with cli,compression,decompression,benchmark, including --benches and --tests (passed)
- cargo +nightly fmt checks for the owned Rust module and CLI crate (passed); root-wide check remains limited by unrelated dict_builder_zdict.rs edits
- Existing CLI dictionary scenarios for missing and directory paths retained status 31/32 and exact diagnostics
- git diff --check and git diff --cached --check (passed)
- zstd build variants were attempted; full links remain blocked by unrelated active dict-builder/zstd-cli/archive edits after the owned C file compiled
The frozen v0.7 decoder's two-state FSE tail must emit state 1, reload,
state 2, reload, and possibly a final state 1 symbol. The Rust port used
current-state and end-of-stream early exits, so historical Huffman headers
could stop one terminal weight short and corrupt a literal byte. The legacy
corpus exposed this as the UTF-8 apostrophe's first byte changing from e2 to
e0. Match the historical reload-driven tail and retain the exact v0.7 frame
as a regression test.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features decompression,legacy-v07 legacy::zstd_v07::tests` -- passed (6 tests)
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features decompression,legacy-v07 --lib -- -D warnings` -- passed
- `rustfmt +nightly --edition 2021 rust/src/legacy/zstd_v07.rs` -- passed
- `make -C tests -j2 legacy` -- blocked by the active dict-builder work's
unrelated full-feature compilation gap
Keep the LDM limit-table update on the Rust compression path while leaving
match-state ownership in C. Previously, the C block wrapper combined pointer
subtraction, match-state access, and the bounded scalar update. The wrapper
now computes `curr` from `anchor - window.base`, passes `curr` and
`nextToUpdate` through the narrow U32 ABI, and stores Rust's result before the
existing fast-table dispatch.
The Rust leaf uses explicit wrapping arithmetic to preserve the C U32 behavior:
the strict `curr > nextToUpdate + 1024` threshold and the `MIN(512, ...)`
clamp. Focused tests cover the threshold, one-step update, clamp, nonzero
starting point, and arithmetic wraparound.
Test Plan:
- `cargo test zstd_ldm` -- default-feature test-binary link failed because
existing dict-builder C symbols are not linked.
- `cargo test --no-default-features --features compression zstd_ldm` -- passed
(7 tests).
- `make lib-nomt` -- passed.
- `make lib-mt` -- passed.
- `make -C tests test-zstream` -- passed; it emitted the existing
`tests/zstreamtest.c` unterminated-string warning.
- `cargo clippy`, `cargo clippy --benches`, `cargo clippy --tests`,
`cargo +nightly fmt`, then the same clippy sequence -- passed.
The Rust dispatch gate for ZSTD_compress2 and complete-input
ZSTD_compressStream2 calls was selecting compression parameters with an
unknown content size. For the 30-byte level-4 CCtx reuse case that chooses the
dfast strategy, while the actual source size selects greedy; the Rust frame
then emitted different output from the C-owned ZSTD_compressCCtx path.
Pass the actual source size through both helper ABIs. The one-shot and complete
stream entry points now select the same strategy as C, while the existing
sentinel keeps every non-fast/non-dfast configuration on the original C
fallback. The existing fuzzer test 56 exercises the regression without
modifying the shared C test harness.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression` -- 348 passed.
- Compression clippy for the library, benches, and tests, followed by
nightly fmt and the repeated three clippy checks -- passed.
- `make -C tests -j2 fuzzer` and `./tests/fuzzer -s4560 -t56 -i57 -v` --
passed; the 30-byte test-56 regression now completes.
- `./tests/fuzzer -s4142 -t63 -i64 -v` -- passed all 64 focused cases.
- `make -C lib -j2 lib-mt` and `make -C lib -j2 lib-nomt` -- passed.
- `make -C tests -j2 test-zstream` -- passed 84 deterministic, 4,385 first
randomized, and 8,137 new-API randomized cases.
- `make -C tests -j2 test-fuzzer` reaches test 113's flat-dictionary
efficiency assertion with concurrent unstaged LDM changes; test 56 passes.
The window overflow gate mixed two scalar decisions with C-owned pointer and
window state. The early correction test uses the block-start index and must
wait for dictionary invalidation, while the normal fallback uses the block-end
index and the platform-selected ZSTD_CURRENT_MAX threshold. Keeping those
subtractions and the ZSTD_window_t wrapper in C preserves the existing call
contract without making Rust depend on pointer width or private state.
Move the U32 cycle/MAX arithmetic, correction-count scaling, dictionary gate,
and frequent-policy decision into Rust. C passes the two pointer-derived
indices only through scalar results, and supplies the current-max threshold
and active build policy explicitly so 32/64-bit and fuzzing configurations
remain authoritative.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression` -- 348 passed.
- Compression clippy for the library, benches, and tests, followed by
nightly fmt and the repeated three clippy checks -- passed.
- `make -C lib -j2 lib-mt` and `make -C lib -j2 lib-nomt` -- passed.
- `make -C tests -j2 fuzzer` and `./tests/fuzzer -s4142 -t63 -i64 -v` --
passed all 64 cases.
- `make -C tests -j2 test-zstream` -- passed 84 deterministic, 5,477 first
randomized, and 6,902 new-API randomized cases; the existing unterminated
string initializer warning remains.
Keep the public ZSTD_getCParams() selection and all DISPLAYLEVEL output in
programs/fileio.c, but pass its by-value compression-parameter snapshot to a
Rust policy leaf. Rust now preserves the original source-size highbit and
window clamping rules, validates and updates the patch memory limit before
success outputs, derives cycleLog, updates comprParams->windowLog, and applies
automatic LDM without overwriting the explicit-LDM diagnostic semantics.
The Rust ABI returns the raw file window and the two diagnostic predicates so
C can retain its existing messages and optimal-parser note ordering. C layout
assertions cover the seven-word compression-parameter snapshot on both
platform-width variants; rejected unknown and oversized inputs leave all
outputs unchanged.
Test Plan:
- Rust clippy pre/post nightly-format matrix for library compression and CLI
cli,compression,decompression,benchmark features -- passed.
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features
--features cli,compression,decompression,benchmark` -- 137 passed.
- `make -C programs -j2 zstd` -- passed.
- `make -C tests -j2 test-cli-tests` -- 41 passed.
- `python3 tests/cli-tests/run.py decompression/pass-through.sh` -- passed.
- Focused patch-from round trips, stream-size failure ordering, and automatic
long-mode diagnostics -- passed.
- `make -C tests -j2 test-rust-lib-smoke` -- passed.
- No i686 Rust target is installed; 32-bit policy bounds are covered by
size_of-based constants and C/Rust ABI assertions.
ZSTD_reduceIndex previously selected and reduced each match table through
separate C helpers, even though the 16-cell reduction leaf already lived in
Rust. Keep C responsible for match-state access, table allocation policy,
nullable table selection, btlazy2 strategy selection, and the surrounding
workspace/window/dictionary state transition. Add one narrow Rust ABI that
receives only the validated hash, chain, and hash3 table slices and delegates
to the existing reducer, preserving chain markers only when requested and
keeping threshold and U32 wrapping arithmetic unchanged. Focused tests cover
zero-sized optional tables, marker routing, and threshold behavior.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_compress::tests::reduce_` -- 5 passed.
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression` -- 344 passed.
- `make -C lib -j2 lib-mt` and `make -C lib -j2 lib-nomt` -- passed.
- `make -C tests -j2 test-zstream` -- 84 named, 6,814 standard
randomized, and 8,955 new-API randomized cases passed.
- Required compression clippy library/benches/tests, nightly fmt, and the repeated three clippy checks -- passed.
Keep patch-from orchestration in C while extracting only the bounded window
selection and automatic-LDM decision into the Rust CLI leaf. The C caller
still computes the raw file window with FIO_highbit64, obtains cParams,
updates the memory limit, emits warnings and optimal-parser notes, and writes
comprParams->windowLog and prefs->ldmFlag. Rust receives only scalar policy
inputs and two output pointers, clamps the selected window to the target's
10..30/31 bounds, and compares the unclamped value strictly against cycleLog.
This preserves equality as no-LDM and keeps values above the maximum eligible
to trigger LDM. Focused unit tests cover both clamps and each comparison edge.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features
--features compression` with library, `--benches`, and `--tests`, before and
after formatting -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --
--config skip_children=true` and nightly rustfmt on the owned file -- passed
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features
--features cli,compression,decompression,benchmark fileio_prefs` -- 36 passed
- `make -C programs -j2 zstd` -- passed
- `make -C tests -j2 test-cli-tests` -- 41 passed
- Focused patch-from round trip and automatic long-mode trigger -- passed
ddidderr