Port ZSTD_seqStore_resolveOffCodes and its repcode-resolution helper to Rust. The C shim keeps Repcodes_t and SeqStore_t ownership in the compressor context while Rust preserves long-literal handling, mismatch materialization, and independent decoder/compressor history updates.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (208 tests)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- make -B -C lib -j2 lib
Replace the shared C raw and one-byte RLE fallback block serializers with Rust ABI leaves. Preserve the zstd block headers, payload copies, capacity errors, and all existing C compressor dispatch and context ownership; share the raw serializer with the Rust one-shot path.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (208 tests)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- make -B -C lib -j2 lib
Move malloc-backed dictionary file reads into the Rust CLI backend while keeping C responsible for stat metadata, patch-mode size policy, diagnostics, buffer-type selection, and eventual free(). The Rust leaf returns explicit open, size, allocation, and read statuses and publishes only fully read libc allocations.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (126 tests)
- cargo clippy --manifest-path rust/cli/Cargo.toml
- make -B -C programs -j2 zstd
- make -C tests -j2 test-cli-tests (41 tests)
Port the scalar frame-finalization policy and byte serialization into the Rust compression leaf while keeping checksum finalization, tracing, and the compression context in C. Preserve the init-to-ongoing transition on partial failure and reset the stage only after a successful epilogue.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-zstream (84 deterministic, 6737 and 9012 fuzzer cases)
Route the CLI backend's safe file-removal leaf through a narrow Rust status ABI while preserving C diagnostics, Windows read-only handling, and the existing return convention.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml; cargo clippy --manifest-path rust/cli/Cargo.toml; cargo clippy --manifest-path rust/cli/Cargo.toml --benches; cargo clippy --manifest-path rust/cli/Cargo.toml --tests; make -B -C programs -j2 zstd; make -C tests -j2 test-cli-tests (41 tests); make -B -C programs -j2 zstd-small zstd-frugal.
Port the external-sequence window, offset, and minimum-match checks through a scalar C ABI leaf while retaining the C caller's control flow and diagnostics.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 deterministic, 7063 and 8665 fuzzer cases).
Move ZSTD_isUpdateAuthorized behind the existing Rust parameter API while retaining the C context-stage checks, diagnostics, and mutation flow.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (187 tests)
- cargo clippy for library, benches, and tests
- make -B -C lib -j2 lib
Move the CLI's strategy-dependent cycle-log calculation behind a scalar Rust ABI shim while retaining the C assertion and call-site behavior.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark (116 tests)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets with the full CLI feature set
- make -B -C programs -j2 zstd zstd-small zstd-frugal
- make -C tests -j2 test-cli-tests (41 scenarios)
Move hash-salt mixing and context-reset overflow predicates behind scalar Rust ABI shims. Keep the C-owned match state and window arithmetic in the C wrapper while preserving exact architecture-dependent thresholds.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (186 tests)
- cargo clippy for library, benches, tests, and CLI all-targets
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-zstream (84 deterministic tests; fuzzers 6451 and 9879 cases)
Move patch-from memory-limit sizing and rejection status selection behind a Rust ABI shim. Keep the C diagnostics and assertion in place, and leave the existing preference unchanged when input sizes are unknown or exceed the window limit.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark (113 tests)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets with the CLI feature set
- make -B -C programs -j2 zstd zstd-small zstd-frugal
- make -C tests -j2 test-cli-tests (41 scenarios) and make -B -C tests -j2 test-zstream
Move target block-size and block-splitter policy predicates behind narrow Rust ABI shims while retaining the C debug logging, finalized-mode assertion, and surrounding control flow.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression
- cargo clippy --manifest-path rust/Cargo.toml and native library build
- make -B -C tests -j2 test-zstream (84 deterministic tests; fuzzers 6778 and 8299 cases)
Port the single-file and multiple-file summary decisions through the existing FIO context ABI, including the display-level lookup and the original multiple-file assertion invariant. Keep the C wrappers and call sites unchanged.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark (108 passed); CLI all-target clippy; make -B -C programs -j2 zstd zstd-small zstd-frugal; make -C tests -j2 test-cli-tests (41 passed).
Move construction of the external-dictionary and active-prefix ranges into a narrow Rust ABI while retaining the C window and logging surface. Preserve byte-range half-open overlap semantics and leave the input-range overlap wrapper available to its other C caller.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (180 passed); root and CLI clippy; make -B -C lib -j2 lib; make -C tests -j2 test-cli-tests (41 passed); make -B -C tests -j2 test-zstream (84 named tests plus 6,845 and 9,628 fuzz cases passed).
Port the 16-cell match-table reduction leaf while preserving the window-index threshold, wrapping subtraction, and btlazy2 unsorted marker. Keep ZSTD_reduceIndex and its strategy-specific table selection in C behind the narrow Rust ABI.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (180 passed); root clippy, bench clippy, and test clippy; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 named tests plus 6,845 and 9,628 fuzz cases passed).
Mirror fileio.c's private fileInfo_t with a checked repr(C) ABI and aggregate totals, flags, and file counts through an output-pointer shim. Preserve zeroed non-aggregate fields and use wrapping arithmetic for the original integer operations.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark (106 passed); cargo clippy --all-targets with the same features; make -B -C programs -j2 zstd zstd-small zstd-frugal; make -C tests -j2 test-cli-tests (41 passed).
Port repcode offBase finalization and explicit-delimiter block sizing through the Rust compression ABI. Preserve wrapped sequence arithmetic, external-sequence error codes, and the no-delimiter target-size policy while keeping the C control flow intact.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (176 passed); cargo clippy root, benches, and tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 named tests plus 6,230 and 8,598 fuzz cases passed).
Move sequence-store byte accounting, chunk derivation, and repcode resolution into the Rust compression module. Move MT raw-sequence buffer conversions into Rust while preserving the existing C adapters and ABI layouts.
Test Plan:\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- make -B -C lib -j2 lib\n- make -B -C tests -j2 test-cli-tests
Route file-I/O input/output buffer construction and the LZ4 block-size mapping through Rust ABI leaves while preserving the existing C helper names and public buffer layouts. Add layout, field, and formula coverage.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark; cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets --no-default-features --features cli,compression,decompression,benchmark; make -B -C programs -j2 zstd zstd-small zstd-frugal; make -B -C tests -j2 test-cli-tests
Route public CCtx parameter bounds through the Rust policy implementation while keeping build-specific MT limits supplied by the C configuration. Preserve clamping and validation behavior for migrated parameters and add focused Rust coverage.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-cli-tests; make -B -C tests -j2 test-zstream
Move ZSTD_CCtxParams_init_advanced and its default-resolution policy into the Rust parameter API while retaining the C public ABI wrapper. Preserve validation, zeroing, and the existing row-match, block-splitter, LDM, and repcode defaults.
Test Plan:\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_compress_params_api\n- make -B -C lib -j2 lib
Move CCtx_params allocation, initialization, and release behind Rust ABI helpers while preserving the public C wrappers and custom allocator contract. Keep parameter bounds and policy in the existing C implementation.
Test Plan:\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- cargo +nightly fmt --manifest-path rust/Cargo.toml --all\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression\n- make -B -C lib -j2 lib\n- make -B -C tests -j2 test-rust-lib-smoke\n- make -B -C tests -j2 test-zstream\n- make -B -C tests -j2 test-cli-tests
Move the basename-preserving output-directory helper behind the existing Rust utility ABI. Keep the libc allocation contract and platform separator behavior intact while leaving the C file-I/O engines and higher-level naming policy unchanged.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (93 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets
- make -B -C tests -j2 test-cli-tests (41 passed)
Implement the CLI trace callbacks and CSV logger in Rust, preserving the ZSTD_Trace ABI, timing fields, parameter reporting, mutex-protected writes, and no-op disabled behavior. Add the module to every Rust CLI archive and exclude the duplicate C trace object from Rust-backed program targets.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (92 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets
- make -B -C programs -j2 zstd
- Rust trace smoke: programs/zstd -q --trace=/tmp/zstd-rust-trace-smoke.csv -c /tmp/levels-input
- make -B -C tests -j2 test-cli-tests (41 passed)
Match the original optimal-parser statistics for zero-possible symbols, complete the shortest-path fallback when the forward pass runs past its final position, and refresh the projected window state for the ultra2 second pass. Keep the simple Rust frame leaf restricted to the match finders it implements so lazy and optimal strategies retain their stateful lifecycle.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (135 passed)
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_opt (3 passed)
- make -B -C tests -j2 test-cli-tests (41 passed)
- make -B -C tests -j2 test-zstream (84 named tests and fuzzer suites passed)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
Port the remaining high-level CLI option, validation, environment, terminal-safety, and file-stat compatibility paths into Rust while keeping the existing C file-I/O ABI boundary. Match upstream quiet-mode warnings and exact error text for size and level handling.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (90 passed)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- make -B -C tests -j2 test-cli-tests (41 passed)
Route ordinary complete-input ZSTD_compressStream2 calls through the Rust frame compressor while retaining the C state machine for partial, advanced, and dictionary-backed streams. Track explicit maxBlockSize requests so the migrated path preserves the original byte-identical default behavior, and extend the C archive smoke test across context reuse and fallback cases.
Test Plan:
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (133 passed)
- make -B -C tests -j2 test-rust-lib-smoke and ./tests/rustLibSmoke
- make -B -C tests -j2 test-zstream (passed, including both APIs and maxBlockSize case 78)
Replace the program archive's C benchmark orchestration with the complete
Rust port while retaining the public benchzstd.h ABI and result/error
contracts. Include the module only in full CLI archives and remove only
benchzstd.c from the program source set, leaving C test targets on their
existing implementation.
Test Plan:
- cargo test --manifest-path /tmp/zstd-benchzstd.IHWpLD/Cargo.toml (6 passed)
- cargo check and cargo clippy --all-targets in the focused harness (passed)
- standalone release archive build and BMK_* symbol inspection (passed)
- make -C programs zstd (passed before concurrent CLI edits)
- make -C tests -B paramgrill (passed)
- native synthetic, multi-file, dictionary, and invalid-frame benchmark smoke tests (passed)
Replace programs/fileio_asyncio.c with a declaration-only shim and register
Rust's async file-I/O implementation in the library archive. Preserve the C
ABI for pool and job contexts, including DEBUGLEVEL pointer-based pthread
wrappers, and select separate Rust build configurations for those layouts.
Keep ordered-read queue state locked while waiting, drain queued work before
shutdown frees job buffers, and use platform large-file seeks for sparse writes.
Test Plan:
- `cargo test --no-default-features --lib fileio_asyncio` (7 passed).
- `cargo test --no-default-features --features debug-pthread --lib fileio_asyncio` (7 passed).
- `cargo check --all-targets`, clippy library/benches/tests, and nightly fmt check (passed).
- Threaded program build and README round-trip (passed).
- `make -C tests poolTests` and `./tests/poolTests` (passed).
- Current C shim object defines no `AIO_*` symbols; the Rust archive exports all 20 declarations.
- Full standalone all-target Rust tests and the default CLI archive rebuild remain affected by unrelated C/Rust integration and another worker's in-progress benchmark changes.
The multithreaded compressor already uses Rust-owned buffer and CCtx pools,
but zstdmt_compress.c still allocated and freed its job table directly. Move
the raw table storage and power-of-two sizing behind Rust's custom allocator
ABI. Keep descriptor field access and platform mutex/condition initialization
in C because those layouts remain private and platform-specific. The C
wrapper initializes and destroys synchronization primitives around the Rust
storage calls, preserving failure cleanup while leaving worker job setup,
scheduling, and stream entry points in C as the fallback implementation.
Keep the Rust storage-only ABI free of MT-only C references so single-threaded
archives can omit zstdmt_compress.c without acquiring new unresolved symbols.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression zstdmt_compress --lib` -- passed (5 tests).
- `make -B -C lib lib-mt` -- passed.
- `make -B -C tests -j2 fullbench zstreamtest poolTests` -- passed.
- `./poolTests` -- passed.
- `./fullbench -i1 -B1000 README.md` -- passed, including -T2 scenarios.
- Rebuilt `programs/zstd` and ran a `-T2` compress/decompress `cmp` round-trip
-- passed.
- `rustfmt` and `cargo +nightly fmt --manifest-path rust/Cargo.toml --all
-- --check` -- passed.
- Full `cargo clippy -D warnings` remains blocked by unrelated warnings in
the concurrent `rust/src/fileio_asyncio.rs` worktree changes.
- `./zstreamtest -T5s` reaches an unrelated single-thread maxBlockSize
assertion at `tests/zstreamtest.c:2157`; MT fullbench and CLI smoke pass.
Dispatch ordinary dictionary-free ZSTD_compress2 calls through the Rust frame
compressor while retaining the original C implementation for advanced
contexts. Prepare the C workspace first so allocation downsizing and
static-context limits remain observable.
Mark successful Rust frames so a following one-shot
ZSTD_compressStream2(..., ZSTD_e_end) call uses the same output path. Other
streaming directives and advanced parameters remain on the C implementation.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
rust/Cargo.toml --no-default-features --features compression
zstd_compress::tests
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
rust/Cargo.toml --no-default-features --features compression -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C tests fuzzer
- ./tests/fuzzer -t114 -i1000 --no-big-tests -s1
The basic fuzzer run now reaches the existing Rust one-shot compression-ratio
limit at test 113; the new stateful reuse tests pass before that point.
Move ZSTD_compressCCtx() onto the Rust frame-compression path while
retaining only the C-owned reset needed for its private context layout.
The simple API ignores advanced context parameters by contract, so the
Rust entry point can share the one-shot implementation after reset.
Preserve valid frames when the Rust superblock leaf declines a compressed
block by emitting a raw block instead. Match the fast-strategy literal
policy for negative levels so compatibility behavior remains intact.
Streaming, stateful ZSTD_compress2(), and multithreaded context
orchestration remain C-owned for a later slice. The full fuzzer currently
reaches an existing compress2() superblock-expansion failure after these
simple-API checks; that stateful path is intentionally out of scope here.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression zstd_compress::tests
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
(library, benches, and tests)
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C programs zstd V=1
- make -B -C examples multiple_simple_compression
- ./multiple_simple_compression README.md ../rust/README.md
Move the frozen v0.7 decoder, dictionary handling, bufferless streaming,
and buffered streaming implementation into the Rust legacy module. Keep
the historical C translation unit as a declaration-only ABI shim so the
existing C callers and build selection remain unchanged.
The port preserves the v0.7 entropy and frame boundaries, while fixing the
Rust-side literal-tail bookkeeping and making buffered header loading revisit
the complete frame header before sizing its rolling buffers.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features decompression,legacy-v07 legacy::zstd_v07::tests
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
- make -B -C programs zstd V=1
- decode a v0.7.5 fixture and compare its 200000-byte output
- make -C tests check V=1
Replace the v0.5 and v0.6 legacy decoder translation units with Rust modules
and keep only narrow C registration shims for the public legacy dispatch ABI.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/Cargo.toml --features compression,decompression,dict-builder,legacy-v05,legacy-v06,legacy-v07
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --features compression,decompression,dict-builder,legacy-v05,legacy-v06,legacy-v07 -- -D warnings
- make -C tests check V=1
Delay the legacy dictionary segment copy until the effective content size is
known, preventing writes beyond the caller-provided dictionary capacity.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- make -C tests check V=1
Move filename tables, file-list expansion, core-count helpers, statistics,
and dictionary sample loading behind Rust implementations while retaining
the narrow C ABI used by the CLI.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- make -B -C programs zstd V=1
- make -C tests check V=1
Add Rust parsing and dispatch for the default, Cover, FastCover, and legacy
dictionary-training modes, including their parameter validation, output
defaults, and help text. Keep file loading behind the existing narrow dibio
bridge while the dictionary algorithms are supplied by the Rust library.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/zstd_cli.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- git diff --cached --check
Move the frozen v0.4 frame, entropy, streaming, and dictionary decoder
implementation to Rust while retaining the C translation unit as an ABI
anchor. Register the decoder behind the matching legacy feature and keep its
historical behavior isolated from newer formats.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/legacy/zstd_v04.rs rust/src/legacy/mod.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/Cargo.toml --no-default-features --features decompression,legacy-v04 legacy::zstd_v04
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --no-default-features --features decompression,legacy-v04 -- -D warnings
- C/Rust v0.4 ABI and decompression build checks
- git diff --cached --check
Move FastCover parameter validation, corpus preparation, optimization, and
dictionary training into Rust. Retain the C source as a static-linking ABI
anchor and register the implementation with the compression dictionary-builder
feature set.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/dict_builder_fastcover.rs rust/src/lib.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --no-default-features --features compression,dict-builder -- -D warnings
- FastCover focused Rust tests and deterministic C-reference harness
- git diff --cached --check
ddidderr