Route file-I/O input/output buffer construction and the LZ4 block-size mapping through Rust ABI leaves while preserving the existing C helper names and public buffer layouts. Add layout, field, and formula coverage.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark; cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets --no-default-features --features cli,compression,decompression,benchmark; make -B -C programs -j2 zstd zstd-small zstd-frugal; make -B -C tests -j2 test-cli-tests
Route public CCtx parameter bounds through the Rust policy implementation while keeping build-specific MT limits supplied by the C configuration. Preserve clamping and validation behavior for migrated parameters and add focused Rust coverage.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-cli-tests; make -B -C tests -j2 test-zstream
Move ZSTD_CCtxParams_init_advanced and its default-resolution policy into the Rust parameter API while retaining the C public ABI wrapper. Preserve validation, zeroing, and the existing row-match, block-splitter, LDM, and repcode defaults.
Test Plan:\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_compress_params_api\n- make -B -C lib -j2 lib
Move CCtx_params allocation, initialization, and release behind Rust ABI helpers while preserving the public C wrappers and custom allocator contract. Keep parameter bounds and policy in the existing C implementation.
Test Plan:\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- cargo +nightly fmt --manifest-path rust/Cargo.toml --all\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression\n- make -B -C lib -j2 lib\n- make -B -C tests -j2 test-rust-lib-smoke\n- make -B -C tests -j2 test-zstream\n- make -B -C tests -j2 test-cli-tests
Move the basename-preserving output-directory helper behind the existing Rust utility ABI. Keep the libc allocation contract and platform separator behavior intact while leaving the C file-I/O engines and higher-level naming policy unchanged.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (93 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets
- make -B -C tests -j2 test-cli-tests (41 passed)
Implement the CLI trace callbacks and CSV logger in Rust, preserving the ZSTD_Trace ABI, timing fields, parameter reporting, mutex-protected writes, and no-op disabled behavior. Add the module to every Rust CLI archive and exclude the duplicate C trace object from Rust-backed program targets.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (92 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets
- make -B -C programs -j2 zstd
- Rust trace smoke: programs/zstd -q --trace=/tmp/zstd-rust-trace-smoke.csv -c /tmp/levels-input
- make -B -C tests -j2 test-cli-tests (41 passed)
Match the original optimal-parser statistics for zero-possible symbols, complete the shortest-path fallback when the forward pass runs past its final position, and refresh the projected window state for the ultra2 second pass. Keep the simple Rust frame leaf restricted to the match finders it implements so lazy and optimal strategies retain their stateful lifecycle.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (135 passed)
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_opt (3 passed)
- make -B -C tests -j2 test-cli-tests (41 passed)
- make -B -C tests -j2 test-zstream (84 named tests and fuzzer suites passed)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
Port the remaining high-level CLI option, validation, environment, terminal-safety, and file-stat compatibility paths into Rust while keeping the existing C file-I/O ABI boundary. Match upstream quiet-mode warnings and exact error text for size and level handling.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (90 passed)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- make -B -C tests -j2 test-cli-tests (41 passed)
Route ordinary complete-input ZSTD_compressStream2 calls through the Rust frame compressor while retaining the C state machine for partial, advanced, and dictionary-backed streams. Track explicit maxBlockSize requests so the migrated path preserves the original byte-identical default behavior, and extend the C archive smoke test across context reuse and fallback cases.
Test Plan:
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (133 passed)
- make -B -C tests -j2 test-rust-lib-smoke and ./tests/rustLibSmoke
- make -B -C tests -j2 test-zstream (passed, including both APIs and maxBlockSize case 78)
Replace the program archive's C benchmark orchestration with the complete
Rust port while retaining the public benchzstd.h ABI and result/error
contracts. Include the module only in full CLI archives and remove only
benchzstd.c from the program source set, leaving C test targets on their
existing implementation.
Test Plan:
- cargo test --manifest-path /tmp/zstd-benchzstd.IHWpLD/Cargo.toml (6 passed)
- cargo check and cargo clippy --all-targets in the focused harness (passed)
- standalone release archive build and BMK_* symbol inspection (passed)
- make -C programs zstd (passed before concurrent CLI edits)
- make -C tests -B paramgrill (passed)
- native synthetic, multi-file, dictionary, and invalid-frame benchmark smoke tests (passed)
Replace programs/fileio_asyncio.c with a declaration-only shim and register
Rust's async file-I/O implementation in the library archive. Preserve the C
ABI for pool and job contexts, including DEBUGLEVEL pointer-based pthread
wrappers, and select separate Rust build configurations for those layouts.
Keep ordered-read queue state locked while waiting, drain queued work before
shutdown frees job buffers, and use platform large-file seeks for sparse writes.
Test Plan:
- `cargo test --no-default-features --lib fileio_asyncio` (7 passed).
- `cargo test --no-default-features --features debug-pthread --lib fileio_asyncio` (7 passed).
- `cargo check --all-targets`, clippy library/benches/tests, and nightly fmt check (passed).
- Threaded program build and README round-trip (passed).
- `make -C tests poolTests` and `./tests/poolTests` (passed).
- Current C shim object defines no `AIO_*` symbols; the Rust archive exports all 20 declarations.
- Full standalone all-target Rust tests and the default CLI archive rebuild remain affected by unrelated C/Rust integration and another worker's in-progress benchmark changes.
The multithreaded compressor already uses Rust-owned buffer and CCtx pools,
but zstdmt_compress.c still allocated and freed its job table directly. Move
the raw table storage and power-of-two sizing behind Rust's custom allocator
ABI. Keep descriptor field access and platform mutex/condition initialization
in C because those layouts remain private and platform-specific. The C
wrapper initializes and destroys synchronization primitives around the Rust
storage calls, preserving failure cleanup while leaving worker job setup,
scheduling, and stream entry points in C as the fallback implementation.
Keep the Rust storage-only ABI free of MT-only C references so single-threaded
archives can omit zstdmt_compress.c without acquiring new unresolved symbols.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression zstdmt_compress --lib` -- passed (5 tests).
- `make -B -C lib lib-mt` -- passed.
- `make -B -C tests -j2 fullbench zstreamtest poolTests` -- passed.
- `./poolTests` -- passed.
- `./fullbench -i1 -B1000 README.md` -- passed, including -T2 scenarios.
- Rebuilt `programs/zstd` and ran a `-T2` compress/decompress `cmp` round-trip
-- passed.
- `rustfmt` and `cargo +nightly fmt --manifest-path rust/Cargo.toml --all
-- --check` -- passed.
- Full `cargo clippy -D warnings` remains blocked by unrelated warnings in
the concurrent `rust/src/fileio_asyncio.rs` worktree changes.
- `./zstreamtest -T5s` reaches an unrelated single-thread maxBlockSize
assertion at `tests/zstreamtest.c:2157`; MT fullbench and CLI smoke pass.
Dispatch ordinary dictionary-free ZSTD_compress2 calls through the Rust frame
compressor while retaining the original C implementation for advanced
contexts. Prepare the C workspace first so allocation downsizing and
static-context limits remain observable.
Mark successful Rust frames so a following one-shot
ZSTD_compressStream2(..., ZSTD_e_end) call uses the same output path. Other
streaming directives and advanced parameters remain on the C implementation.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
rust/Cargo.toml --no-default-features --features compression
zstd_compress::tests
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
rust/Cargo.toml --no-default-features --features compression -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C tests fuzzer
- ./tests/fuzzer -t114 -i1000 --no-big-tests -s1
The basic fuzzer run now reaches the existing Rust one-shot compression-ratio
limit at test 113; the new stateful reuse tests pass before that point.
Move ZSTD_compressCCtx() onto the Rust frame-compression path while
retaining only the C-owned reset needed for its private context layout.
The simple API ignores advanced context parameters by contract, so the
Rust entry point can share the one-shot implementation after reset.
Preserve valid frames when the Rust superblock leaf declines a compressed
block by emitting a raw block instead. Match the fast-strategy literal
policy for negative levels so compatibility behavior remains intact.
Streaming, stateful ZSTD_compress2(), and multithreaded context
orchestration remain C-owned for a later slice. The full fuzzer currently
reaches an existing compress2() superblock-expansion failure after these
simple-API checks; that stateful path is intentionally out of scope here.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression zstd_compress::tests
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
(library, benches, and tests)
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C programs zstd V=1
- make -B -C examples multiple_simple_compression
- ./multiple_simple_compression README.md ../rust/README.md
Move the frozen v0.7 decoder, dictionary handling, bufferless streaming,
and buffered streaming implementation into the Rust legacy module. Keep
the historical C translation unit as a declaration-only ABI shim so the
existing C callers and build selection remain unchanged.
The port preserves the v0.7 entropy and frame boundaries, while fixing the
Rust-side literal-tail bookkeeping and making buffered header loading revisit
the complete frame header before sizing its rolling buffers.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features decompression,legacy-v07 legacy::zstd_v07::tests
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
- make -B -C programs zstd V=1
- decode a v0.7.5 fixture and compare its 200000-byte output
- make -C tests check V=1
Replace the v0.5 and v0.6 legacy decoder translation units with Rust modules
and keep only narrow C registration shims for the public legacy dispatch ABI.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/Cargo.toml --features compression,decompression,dict-builder,legacy-v05,legacy-v06,legacy-v07
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --features compression,decompression,dict-builder,legacy-v05,legacy-v06,legacy-v07 -- -D warnings
- make -C tests check V=1
Delay the legacy dictionary segment copy until the effective content size is
known, preventing writes beyond the caller-provided dictionary capacity.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- make -C tests check V=1
Move filename tables, file-list expansion, core-count helpers, statistics,
and dictionary sample loading behind Rust implementations while retaining
the narrow C ABI used by the CLI.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- make -B -C programs zstd V=1
- make -C tests check V=1
Add Rust parsing and dispatch for the default, Cover, FastCover, and legacy
dictionary-training modes, including their parameter validation, output
defaults, and help text. Keep file loading behind the existing narrow dibio
bridge while the dictionary algorithms are supplied by the Rust library.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/zstd_cli.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- git diff --cached --check
Move the frozen v0.4 frame, entropy, streaming, and dictionary decoder
implementation to Rust while retaining the C translation unit as an ABI
anchor. Register the decoder behind the matching legacy feature and keep its
historical behavior isolated from newer formats.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/legacy/zstd_v04.rs rust/src/legacy/mod.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/Cargo.toml --no-default-features --features decompression,legacy-v04 legacy::zstd_v04
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --no-default-features --features decompression,legacy-v04 -- -D warnings
- C/Rust v0.4 ABI and decompression build checks
- git diff --cached --check
Move FastCover parameter validation, corpus preparation, optimization, and
dictionary training into Rust. Retain the C source as a static-linking ABI
anchor and register the implementation with the compression dictionary-builder
feature set.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/dict_builder_fastcover.rs rust/src/lib.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --no-default-features --features compression,dict-builder -- -D warnings
- FastCover focused Rust tests and deterministic C-reference harness
- git diff --cached --check
Move fileio preference and context allocation, defaults, setters, and small
query helpers into Rust while keeping the actual file operations in C. Keep
the C structs and ABI declarations as anchors, and make the CLI build track
the new Rust source.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/fileio_prefs.rs rust/cli/src/lib.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- git diff --cached --check
Move the public ZDICT helpers, legacy trainer, dictionary finalization,
entropy-table construction, and supporting dictionary logic into Rust. Keep
the C translation unit as an ABI anchor and register the Rust module only
when compression and dictionary-builder features are enabled.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/Cargo.toml dict_builder_zdict
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- rustfmt +nightly --check --edition 2021 rust/src/dict_builder_zdict.rs rust/src/lib.rs
- git diff --cached --check
Move the frozen v0.3 frame decoder, entropy tables, and streaming context
state into a feature-gated Rust module. Keep the historical C translation
unit as an ABI shim so existing C callers retain the opaque context and
error-code contracts while v0.4 and newer decoders remain unchanged.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
rust/Cargo.toml --no-default-features --features legacy-v01,legacy-v02,
legacy-v03
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
rust/Cargo.toml --no-default-features --features compression,decompression,
dict-builder,legacy-v01,legacy-v02,legacy-v03
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
rust/Cargo.toml --all-targets --no-default-features --features
compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03
-- -D warnings
- make -B -C lib libzstd.a ZSTD_LEGACY_SUPPORT=3 V=1
- git diff --cached --check
Move the reusable buffer and compression-context pools behind a Rust
implementation with a narrow C ABI adapter. The scheduler, job table, serial
LDM state, and stream orchestration remain in C until their private layouts
are ported.
Keep context types opaque across Rust modules so the pool bridge does not
depend on private C layout declarations. This also keeps the existing custom
allocator and pool replacement contracts intact.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
rust/Cargo.toml --all-targets -- -D warnings
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
rust/Cargo.toml zstdmt_compress
- make -B -C lib lib-mt
- git diff --cached --check
Implement the Rust frontend's --show-default-cparams path, including source
and dictionary size discovery, strategy labels, verbose --zstd parameter
reporting, and the decompression-mode diagnostic. Keep the C parameter API
as the source of the selected values so the CLI remains aligned with the
active compression build.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- rustfmt +nightly --edition 2021 rust/src/zstd_cli.rs
- git diff --check
Depends-on: Rust CLI parser, file-selection, and C fileio bridge ports
Move COVER training, frequency mapping, segment selection, optimization,
dictionary shrinking, and best-candidate synchronization into Rust. The
remaining C translation unit is an ABI shim, while the public ZDICT and COVER
symbols stay available to the existing C dictionary-builder callers.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml dict_builder_cover
- cargo check --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C lib libzstd.a V=1
- git diff --check
Depends-on: Rust compression, entropy, and dictionary-finalization leaves
Move the public ZBUFF compatibility layer into the Rust static archive while
keeping stream contexts opaque and forwarding to the current ZSTD streaming
APIs. The deprecated C translation units are now declaration-only shims, so
the compatibility symbols work in compression-only and decompression-only
feature builds without duplicating private context layouts.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C lib libzstd.a ZSTD_LIB_DEPRECATED=1 V=1
- C ZBUFF compress/decompress smoke test against the rebuilt static archive
Depends-on: Rust compression and decompression streaming ABI ports
Move the binary-tree optimal and ultra block parsers into Rust with a narrow
C projection of match-state, sequence-store, and entropy-table fields. Keep
the established C entry points as wrappers, and avoid reading uninitialized
entropy costs during dictionary tree updates.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- make -B -C tests test-zstream V=1
- git diff --check
Refs: rust/src/zstd_opt.rs, lib/compress/zstd_opt.c
Translate the frozen v0.2 frame, entropy, block, frame-size, and streaming
decoder paths to Rust while preserving the opaque C context ABI. Replace the
3465-line historical C implementation with a declaration-only shim and enable
the Rust module for the legacy-v02 Cargo feature.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v02
- make -B -C lib libzstd.a ZSTD_LEGACY_SUPPORT=2 V=1
- git diff --check
Refs: rust/src/legacy/zstd_v02.rs, lib/legacy/zstd_v02.c
Implement ZSTD_compress in Rust using the migrated parameter, match-finder,
sequence, frame, and superblock leaves. Keep the private context and streaming
entry points C-backed until their configuration-dependent context projection is
ported, and remove only the duplicate C one-shot definition.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- make -B -C tests test-rust-lib-smoke V=1
- git diff --check
Refs: rust/src/zstd_compress.rs, lib/compress/zstd_compress.c
Set the C file-I/O patch-from flag explicitly when the Rust frontend creates
preferences. The parser still rejects the unported option, but this keeps the
backend in ordinary dictionary mode instead of reading an unspecified field.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --all-features
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- git diff --check
Refs: rust/src/zstd_cli.rs, programs/fileio.c
Move recursive expansion, file-list inputs, output-directory dispatch, and
frame listing into the Rust command-line frontend while keeping directory
traversal and file I/O in the existing C helpers. Preserve the unsigned fast
compression-level parsing and add focused parser coverage for the new modes.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --all-features
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- git diff --check
Refs: rust/src/zstd_cli.rs, programs/util.c, programs/fileio.c
Keep the original C translation units as declaration-only build shims while
the Rust CLI helper archive owns datagen and lorem. Track both Rust sources in
the program and test archive prerequisites, and link the helpers-only archive
into C test binaries that still provide their own file I/O and orchestration.
This preserves the existing C test harnesses while making the migrated helper
implementations the single definitions used by the binary and tests.
Test Plan:
- make -C programs zstd V=1
- make -C tests datagen V=1
- make -C tests test-rust-lib-smoke V=1
- git diff --check
Refs: programs/datagen.c, programs/lorem.c, tests/Makefile
The C command-line parser accepts short-form levels through an unsigned
32-bit conversion before storing them in the signed compression-level field.
The Rust parser used signed parsing for both ordinary levels and --fast,
rejecting 4294967295 even though it represents the valid fast level -1.
Mirror the conversion and retain the fast-level clamp for the attached form.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --all-features
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- make -C programs zstd
- ./programs/zstd --fast -4294967295 tests/playTests.sh -o /tmp/zstd-fast-test
- ./programs/zstd --fast=4294967295 tests/playTests.sh -o /tmp/zstd-fast-test
Refs: tests/playTests.sh maximum fast-level cases
Move LOREM_genBlock and LOREM_genBuffer into the Rust CLI helper archive,
preserving the frozen word distribution, seeded sentence and paragraph layout,
tail filling, and C-compatible buffer ABI. Use per-call state so generation is
reentrant without changing the generated bytes.
Focused tests cover C reference vectors, distribution construction,
determinism, zero-sized null buffers, tiny tail-filled buffers, and continued
nonzero-fill generation.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features lorem::tests
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
Refs: programs/lorem.c compatibility port
Move RDG_genBuffer and RDG_genStdout into the Rust CLI helper archive while
preserving the C generator's random-draw order, fixed-point literal table,
sparse-data path, sliding dictionary, and public ABI. Keep a zero-length
buffer a valid no-op even when its pointer is null.
The focused tests include C reference vectors, sparse and pure-noise inputs,
custom literal probabilities, reproducibility, and the null zero-length case.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features datagen::tests
- cargo test --manifest-path rust/cli/Cargo.toml --all-features
Refs: programs/datagen.c compatibility port
The compression round-trip tests built an X1 decode table unconditionally.
That violated the X2 decoder contract when the HUF force feature was enabled
and triggered the production debug assertion instead of testing the selected
configuration. Build the table with the selected X1/X2 reader and reserve the
corresponding table capacity.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,huf-force-decompress-x1,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,huf-force-decompress-x2,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
Refs: HUF force-mode build matrix
Continue the incremental zstd_compress.c migration with its sequence
statistics and seqStore entropy-compression layer. The following now
live in rust/src/zstd_compress_stats.rs:
- ZSTD_seqToCodes(), exported under its original name because the C
dictionary builder (zdict.c) and decodecorpus link against it,
- ZSTD_buildSequencesStatistics() and its dummy variant, whose result
struct no longer crosses the language boundary,
- ZSTD_entropyCompressSeqStore_internal(), _wExtLitBuffer(), and
ZSTD_entropyCompressSeqStore(),
- ZSTD_buildBlockEntropyStats() with its literals/sequences helpers,
- ZSTD_copyBlockSequences() and the ZSTD_updateRep() rules it shares
with the superblock writer.
Boundary: ZSTD_CCtx and ZSTD_CCtx_params stay private to C. These
paths read exactly two parameter fields, so the C shims keep the
original static/exported function names and forward the strategy and
ZSTD_literalsCompressionIsDisabled() as int scalars alongside the
seqStore and entropy-table leaves. The leaf layouts (SeqDef,
SeqStore_t, ZSTD_hufCTables_t, ZSTD_fseCTables_t, entropy metadata,
SeqCollector, ZSTD_Sequence) are pinned by compile-time asserts in
zstd_compress.c and by both-pointer-width layout tests in Rust.
ZSTD_buildSeqStore, block dispatch/splitting, and the block-size
estimation helpers remain C for a later slice.
The superblock module previously round-tripped through the C export of
ZSTD_buildBlockEntropyStats and mirrored the entropy leaf structs
privately. It now calls the crate-internal builder directly, and the
shared struct definitions moved to zstd_compress_stats; consequently
ZSTD_rust_compressSuperBlock() takes the two parameter scalars instead
of an opaque ZSTD_CCtx_params pointer, extracted by its C shim. Its
layout and repcode tests moved with the definitions.
One C helper family gets no shim: the static
ZSTD_entropyCompressSeqStore_wExtLitBuffer() had a single caller and
was folded into the Rust implementation.
Byte-identity was verified against the pre-change compressor: COPYING,
datagen -g5000000 -s7, and datagen -g300000 -s21 -P90, each at levels
1/3/9/19 and --fast=5, plus a superblock-heavy pass at level 19 with
--target-compressed-block-size=1024; all 18 frames are byte-identical,
covering the repeat-mode state machine, longOffsets, RLE/raw fallback,
and dstSize_tooSmall paths through the block splitter and superblock.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (131 tests: new reference
vectors for seqToCodes, RLE table headers, empty-seqStore repeat
copies, literal-stats type selection, and repcode resolution in
copyBlockSequences)
- cargo build --release --no-default-features --features compression
and --features decompression
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests (covers
ZSTD_generateSequences and ZDICT training over the Rust seqToCodes)
- make -C tests test-rust-lib-smoke
- ./tests/zstreamtest -i1 and ./tests/zstreamtest --newapi -t1 -i1
- ./tests/decodecorpus -t -T1s (decodecorpus links Rust seqToCodes)
- /tmp/ref vs /tmp/got frame diff as described above: byte-identical
Port lib/legacy/zstd_v01.c (the frozen zstd v0.1 decoder) to
rust/src/legacy/zstd_v01.rs as the first legacy-format port on the new
scaffolding, and reduce the C file to a declaration-only shim that
keeps its header includes for configuration and platform preprocessor
behavior.
Frozen-decoder policy: zstd_v01.c embeds its own v0.1-era FSE and
Huff0 snapshot, distinct from every other release. The Rust port is a
line-by-line translation with the same table layouts (FSE_DTable as a
u32 header word plus packed newState/symbol/nbBits entries, the Huff0
u16 DTable with byte/nbBits pairs), the same arithmetic including
wrap-around and pointer-comparison quirks (e.g. the offset-vs-base
address check in ZSTD_execSequence), the same internal FSE error space
(size_t)-1..-7, and the same public ZSTD error codes. It reuses no
modern Rust entropy module; its only crate dependency is `errors`,
matching the C file's error_private.h include. The 32-bit-only reload
points are kept as compile-time conditions on usize::BITS.
Symbol takeover boundary: all nine ZSTDv01_* entry points from
zstd_v01.h now come from Rust as context-free #[no_mangle] extern "C"
functions (isError, decompress, decompressDCtx,
findFrameSizeInfoLegacy, createDCtx, freeDCtx, resetDCtx,
nextSrcSizeToDecompress, decompressContinue). zstd_legacy.h only uses
the first four for v0.1; streaming for v0.1-v0.3 intentionally returns
version_unsupported there, unchanged. The ZSTDv01_Dctx struct
definition moves entirely into Rust: C code only ever holds an opaque
pointer (zstd_v01.h forward-declares the type), and the context is
malloc/free-allocated exactly like the C version so create/free may
pair across the language boundary.
Byte-identity verification against the pristine pre-migration C build
(f8745da6, pure C, ZSTD_LEGACY_SUPPORT=1):
- Real v0.1 frames were generated by building the v0.1.0 git tag and
compressing text, random, and 426 KB multi-block inputs. A one-shot
ZSTD_decompress harness linked once against the pristine C libzstd.a
and once against the Rust-backed libzstd.a produced bit-identical
outputs for all frames.
- A direct ZSTDv01_* probe (one-shot decode, dst-too-small, truncated
input, bad magic, findFrameSizeInfoLegacy, and the streaming
continue loop) printed identical results, including exact error
codes (-70 dstSize_tooSmall, -72 srcSize_wrong, -10 prefix_unknown)
and identical dBound values.
- zstd -l -v on v0.1 files matches the pristine binary; CLI streaming
decode of v0.1 fails with the same "Version not supported" in both,
by design of zstd_legacy.h.
Unit tests embed three v0.1.0-generated fixtures (entropy-coded,
raw-block, and four-block frames) plus the truncation, bad-magic,
small-destination, and streaming-API cases, all asserting the exact C
error codes above. Note that `make -C tests test-legacy` only covers
v0.4+ frames, so the embedded fixtures and the harness comparison are
the actual v0.1 coverage.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
--features legacy-v01 -- -D warnings && cargo test --all-targets
--features legacy-v01 (127 tests, 9 for v0.1)
- cargo clippy/test --no-default-features --features
decompression,legacy-v01 (module builds standalone)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests, also with
ZSTD_LEGACY_SUPPORT=1 (mixed Rust v0.1 + C v0.2-0.7 link)
- make -C tests test-rust-lib-smoke && make -C tests test-legacy
- make -C programs zstd (default and ZSTD_LEGACY_SUPPORT=1); nm shows
the nine ZSTDv01_* symbols provided by Rust at level 1
- make -C lib libzstd.a ZSTD_LEGACY_SUPPORT=0 (no legacy symbols) and
meson -Dlegacy_level=1 shared library exporting all nine
The legacy decoders (lib/legacy/zstd_v01.c .. zstd_v07.c) are next in
the Rust migration. Each of those files is a frozen snapshot of the
FSE/Huff0 entropy coders and frame logic of one historical release, so
their ports must not reuse the modern Rust entropy modules and must not
share code with each other: outputs and error codes have to stay
byte-identical to the frozen C forever. This commit installs the
build-system scaffolding so seven per-version ports can land
independently, each adding only its own module file plus a one-line
registration in rust/src/legacy/mod.rs.
Cargo grows features legacy-v01 .. legacy-v07. They are never default
features: the C build defaults differ per build system, so each build
system passes the list explicitly, derived from its own legacy
configuration:
- lib/Makefile and programs/Makefile map ZSTD_LEGACY_SUPPORT=N to the
features for versions N..7 (0 disables legacy), mirroring the
ZSTD_LEGACY_FILES selection in lib/libzstd.mk.
- tests/Makefile always enables all seven features because its
ZSTDLEGACY_FILES wildcard compiles every lib/legacy/*.c regardless of
the dispatch level.
- build/meson maps legacy_level exactly like the makefiles; build/cmake
enables all seven whenever ZSTD_LEGACY_SUPPORT is ON because it
always compiles all seven C files (ZSTD_LEGACY_LEVEL only selects the
C dispatch).
Every build system also encodes the legacy selection in the Rust target
directory name (e.g. c1-d1-default-legacy5), for the same reason the
HUF mode is encoded there: a cached archive built for one configuration
must never be linked into a build expecting another. In tests/Makefile
the legacy level additionally flows into the existing HUF C-mode stamp,
so the flat C test objects (which bake -DZSTD_LEGACY_SUPPORT into the
dispatch) are rebuilt whenever the level changes. In programs/Makefile
the compress-only, decompress-only, and CLI archives keep
level-independent directories (RUST_HUF_MODE) because they are only
linked into ZSTD_LEGACY_SUPPORT=0 program variants and carry no legacy
features.
A feature whose version has not been ported yet gates nothing: the
module registration in rust/src/legacy/mod.rs is added by each port,
so enabling e.g. legacy-v05 today simply leaves that decoder in C.
This is what makes mixed C/Rust legacy levels link cleanly while the
seven ports land in any order.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
-- -D warnings && cargo test --all-targets
- cargo clippy with --no-default-features --features
decompression,legacy-v01 and with all seven legacy features
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke; make -C tests test-legacy
- make -C lib libzstd.a with ZSTD_LEGACY_SUPPORT=0, 1 and default (5)
- cmake configure and meson setup (including -Dlegacy_level=1) emit the
expected --features lists and legacy-suffixed target directories
Move the implementation of programs/benchfn.c into rust/src/benchfn.rs and
wire benchmark mode (-b/-e/-i) into the Rust CLI frontend, which previously
rejected those options as not yet implemented. `zstd -b1 -i0 FILE` and range
runs like `zstd -b5e6 -i0 FILE` work again, including the synthetic-sample
benchmark when no file is given.
benchfn.rs is a faithful port of the run/timing state machine:
BMK_benchFunction keeps the exact loop accounting (first-loop blockResults
and errorFn checks, dstSize summed on the first loop only, 0xE5 warm-up of
result buffers, nbLoops minimum of 1) and BMK_benchTimedFn keeps the same
convergence behavior (x10 workload growth for short runs, budget-based
nbLoops estimation, runs below half the run budget re-tried rather than
reported, best qualifying run returned). Arithmetic that C leaves to
unsigned wrap-around uses wrapping operations so debug builds cannot panic
where release C would wrap.
ABI notes: BMK_runTime_t and BMK_runOutcome_t are returned by value across
the C boundary and BMK_benchParams_t is passed by value, so all three are
repr(C) mirrors of benchfn.h; their field offsets are pinned by const
asserts in Rust and matching C static asserts in the benchfn.c shim, which
is now declaration-only. BMK_timedFnState_t stays opaque, fits the 64-byte
BMK_timedFnState_shell (compile-time checked), and is malloc/free-managed
so creation and destruction remain interchangeable with C callers.
The CLI parses -b (bench mode), -e (range end, digits attach directly,
defaulting to 0 like readU32FromChar) and -i (duration in seconds), then
dispatches through a new ZSTD_rust_cli_bench bridge in the zstdcli.c shim.
The bridge exists because benchmark availability is a C preprocessor
property (ZSTD_NOBENCH): orchestration and reporting stay in C benchzstd.c,
stripped variants (zstd-small, zstd-compress, zstd-decompress) compile the
stub branch and report "benchmark mode is not available in this build", and
the Rust side never references benchmark symbols directly. Level clamping
against ZSTD_maxCLevel() happens in the bridge, where the symbol is
guaranteed to exist whenever benchmarking is compiled in. -T selects the
worker count, defaulting to single-threaded like the C bench path; -S
(separate files) and --priority=rt remain unimplemented.
Makefile updates only extend the Rust source prerequisite lists with
benchfn.rs; the helpers-archive plumbing from the timefn commit already
links fullbench(-lib/-dll/32) and paramgrill, the benchfn consumers among
the C tests. Original C test sources are untouched.
Known pre-existing issues, unchanged by this commit: tests/fullbench-lib
fails to link at the base commit too (libzstd.a precedes fullbench.c in its
link line), and the cli-tests basic/help.sh, compression/levels.sh,
compression/golden.sh, and decompression/pass-through.sh scripts fail
identically with a base-commit binary because the Rust CLI frontend is
still a partial reimplementation.
Test Plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings
&& cargo test --all-targets && cargo build --release
- cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets; repeat tests with
--no-default-features plus features compression / decompression / (none)
- make -C programs zstd; ./programs/zstd -b1 -i0 lib/common/xxhash.c;
./programs/zstd -b5e6 -i0 programs/fileio.c; ./programs/zstd -b1 -i0
(synthetic); echo roundtrip via zstd | zstd -d
- make -C programs zstd-small zstd-compress zstd-decompress zstd-nolegacy
zstd-dictBuilder; zstd-small -b reports benchmark unavailable; compress/
decompress roundtrip across the split binaries
- make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus
poolTests fullbench32 fuzzer32; ./tests/fullbench -i0 (exercises Rust
BMK_benchTimedFn from C); ./tests/fullbench32 -i0; ./tests/fuzzer -i1
--no-big-tests; ./tests/poolTests; make -C tests test-rust-lib-smoke
- cli-tests subset: basic/version.sh, compression/basic.sh,
compression/multiple-files.sh pass; failing scripts match the base commit
Refs: rust/README.md
ddidderr