Update the Rust rewrite documentation to match the current implementation
boundary. The legacy decoder set is complete, and Rust now owns several
CLI file-I/O policy and resource leaves while C retains format dispatch,
metadata, and remaining orchestration. Keep the stated boundary explicit so
passing hybrid tests do not imply that the full rewrite is finished.
Test Plan:
- git diff --cached --check
The library reports an absolute maximum compression level of 22, but the CLI
advertises levels 1 through 19 unless `--ultra` is selected. The Rust frontend
used the absolute maximum for both help lines, diverging from the original C
contract and failing the help golden test. Separate the normal help ceiling
from the ultra ceiling and add a focused regression assertion.
Test Plan:
- `cargo fmt --manifest-path rust/cli/Cargo.toml -- --check` -- passed
- `cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings` -- passed
- CLI Rust tests -- 162 passed
- `make -C tests -j2 test-cli-tests` -- all 41 passed
- `git diff --check` -- passed
Keep dictionary-training code out of compressor-only and decompressor-only CLI
archives, where its ZDICT symbols are intentionally absent. Add a dedicated
Rust CLI archive for zstd-dictBuilder, enable the builder feature only for the
full and dictionary-builder programs, and make reduced-feature help use the
feature-independent max-level helper.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib -- --test-threads=1 (158 passed)
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,dict-builder --lib -- --test-threads=1 (124 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,dict-builder --lib -- -D warnings
- make -C programs -j2 zstd zstd-small zstd-frugal zstd-decompress zstd-compress zstd-dictBuilder
- git diff --check
Parse attached and separated --patch-from references, preserve the upstream ultra ceiling and conflict checks, and pass the reference path through the existing file-I/O dictionary ABI with patch mode enabled.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- make -C tests -j2 check
- small attached/separate patch-from round-trip and rejection smoke
Port the remaining high-level CLI option, validation, environment, terminal-safety, and file-stat compatibility paths into Rust while keeping the existing C file-I/O ABI boundary. Match upstream quiet-mode warnings and exact error text for size and level handling.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (90 passed)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- make -B -C tests -j2 test-cli-tests (41 passed)
Add Rust parsing and dispatch for the default, Cover, FastCover, and legacy
dictionary-training modes, including their parameter validation, output
defaults, and help text. Keep file loading behind the existing narrow dibio
bridge while the dictionary algorithms are supplied by the Rust library.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/zstd_cli.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- git diff --cached --check
Implement the Rust frontend's --show-default-cparams path, including source
and dictionary size discovery, strategy labels, verbose --zstd parameter
reporting, and the decompression-mode diagnostic. Keep the C parameter API
as the source of the selected values so the CLI remains aligned with the
active compression build.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- rustfmt +nightly --edition 2021 rust/src/zstd_cli.rs
- git diff --check
Depends-on: Rust CLI parser, file-selection, and C fileio bridge ports
Set the C file-I/O patch-from flag explicitly when the Rust frontend creates
preferences. The parser still rejects the unported option, but this keeps the
backend in ordinary dictionary mode instead of reading an unspecified field.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --all-features
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- git diff --check
Refs: rust/src/zstd_cli.rs, programs/fileio.c
Move recursive expansion, file-list inputs, output-directory dispatch, and
frame listing into the Rust command-line frontend while keeping directory
traversal and file I/O in the existing C helpers. Preserve the unsigned fast
compression-level parsing and add focused parser coverage for the new modes.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --all-features
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- git diff --check
Refs: rust/src/zstd_cli.rs, programs/util.c, programs/fileio.c
The C command-line parser accepts short-form levels through an unsigned
32-bit conversion before storing them in the signed compression-level field.
The Rust parser used signed parsing for both ordinary levels and --fast,
rejecting 4294967295 even though it represents the valid fast level -1.
Mirror the conversion and retain the fast-level clamp for the attached form.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --all-features
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- make -C programs zstd
- ./programs/zstd --fast -4294967295 tests/playTests.sh -o /tmp/zstd-fast-test
- ./programs/zstd --fast=4294967295 tests/playTests.sh -o /tmp/zstd-fast-test
Refs: tests/playTests.sh maximum fast-level cases
Move the implementation of programs/benchfn.c into rust/src/benchfn.rs and
wire benchmark mode (-b/-e/-i) into the Rust CLI frontend, which previously
rejected those options as not yet implemented. `zstd -b1 -i0 FILE` and range
runs like `zstd -b5e6 -i0 FILE` work again, including the synthetic-sample
benchmark when no file is given.
benchfn.rs is a faithful port of the run/timing state machine:
BMK_benchFunction keeps the exact loop accounting (first-loop blockResults
and errorFn checks, dstSize summed on the first loop only, 0xE5 warm-up of
result buffers, nbLoops minimum of 1) and BMK_benchTimedFn keeps the same
convergence behavior (x10 workload growth for short runs, budget-based
nbLoops estimation, runs below half the run budget re-tried rather than
reported, best qualifying run returned). Arithmetic that C leaves to
unsigned wrap-around uses wrapping operations so debug builds cannot panic
where release C would wrap.
ABI notes: BMK_runTime_t and BMK_runOutcome_t are returned by value across
the C boundary and BMK_benchParams_t is passed by value, so all three are
repr(C) mirrors of benchfn.h; their field offsets are pinned by const
asserts in Rust and matching C static asserts in the benchfn.c shim, which
is now declaration-only. BMK_timedFnState_t stays opaque, fits the 64-byte
BMK_timedFnState_shell (compile-time checked), and is malloc/free-managed
so creation and destruction remain interchangeable with C callers.
The CLI parses -b (bench mode), -e (range end, digits attach directly,
defaulting to 0 like readU32FromChar) and -i (duration in seconds), then
dispatches through a new ZSTD_rust_cli_bench bridge in the zstdcli.c shim.
The bridge exists because benchmark availability is a C preprocessor
property (ZSTD_NOBENCH): orchestration and reporting stay in C benchzstd.c,
stripped variants (zstd-small, zstd-compress, zstd-decompress) compile the
stub branch and report "benchmark mode is not available in this build", and
the Rust side never references benchmark symbols directly. Level clamping
against ZSTD_maxCLevel() happens in the bridge, where the symbol is
guaranteed to exist whenever benchmarking is compiled in. -T selects the
worker count, defaulting to single-threaded like the C bench path; -S
(separate files) and --priority=rt remain unimplemented.
Makefile updates only extend the Rust source prerequisite lists with
benchfn.rs; the helpers-archive plumbing from the timefn commit already
links fullbench(-lib/-dll/32) and paramgrill, the benchfn consumers among
the C tests. Original C test sources are untouched.
Known pre-existing issues, unchanged by this commit: tests/fullbench-lib
fails to link at the base commit too (libzstd.a precedes fullbench.c in its
link line), and the cli-tests basic/help.sh, compression/levels.sh,
compression/golden.sh, and decompression/pass-through.sh scripts fail
identically with a base-commit binary because the Rust CLI frontend is
still a partial reimplementation.
Test Plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings
&& cargo test --all-targets && cargo build --release
- cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets; repeat tests with
--no-default-features plus features compression / decompression / (none)
- make -C programs zstd; ./programs/zstd -b1 -i0 lib/common/xxhash.c;
./programs/zstd -b5e6 -i0 programs/fileio.c; ./programs/zstd -b1 -i0
(synthetic); echo roundtrip via zstd | zstd -d
- make -C programs zstd-small zstd-compress zstd-decompress zstd-nolegacy
zstd-dictBuilder; zstd-small -b reports benchmark unavailable; compress/
decompress roundtrip across the split binaries
- make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus
poolTests fullbench32 fuzzer32; ./tests/fullbench -i0 (exercises Rust
BMK_benchTimedFn from C); ./tests/fullbench32 -i0; ./tests/fuzzer -i1
--no-big-tests; ./tests/poolTests; make -C tests test-rust-lib-smoke
- cli-tests subset: basic/version.sh, compression/basic.sh,
compression/multiple-files.sh pass; failing scripts match the base commit
Refs: rust/README.md
The Rust CLI frontend rejected --single-thread, but tests/playTests.sh
uses it 28 times, so the flag is required before the original CLI test
suite can gate the migration.
Mirror the C zstdcli.c semantics: --single-thread pins zero workers and
sets a latch that suppresses the automatic core-count resolution, so
fileio receives nbWorkers == 0 and runs its single-thread streaming
mode (slightly different from -T1, which uses one worker thread). A
bare zero from -T0 or the zstdmt program name still auto-detects the
core count, and a later -T# overrides the pinned worker count while the
latch stays set, exactly as the C variable pair behaved.
Test plan:
- cd rust/cli && cargo clippy --all-targets -- -D warnings && cargo
test --all-targets (plus the compression-only and decompression-only
feature matrices); new parser tests cover the flag, the -T override
order, and rejection of an attached value.
- make -C programs zstd, then: --single-thread -3/-19 round-trips
against COPYING via cmp; --single-thread=1 fails with "does not take
an argument".
Move long-distance matching and high-level decompression from C shims into
Rust. The decoder now owns context, dictionary, parameter, one-shot, and
buffered streaming state while C retains allocation/configuration, legacy,
and trace leaves.
Move CLI parsing, safety policy, and dispatch into a separate Rust static
archive. Keeping it separate prevents library builds from retaining FIO
symbols, while C continues to own file opening, replacement, and I/O.
Program targets now select matching compression/decompression archives.
The remaining C boundary is intentional: high-level compression, optimal
parsing, dictionary building, legacy callbacks, and CLI file I/O still need
migration.
Test Plan:
- cargo test --all-targets (native and i686)
- cargo test --all-targets in rust/cli (native and i686)
- CLI crate compression-only and decompression-only feature tests
- native and i686 fuzzer/zstreamtest runs, plus legacy and dictionary tests
- ZSTD_C_PREDICT and ZSTD_HEAPMODE=0 fuzzer coverage
- library, dynamic-link, and program-target build/round-trip matrix
Refs: rust/README.md
ddidderr