Move the separate-destination branch of FIO_compressMultipleFilenames into a Rust-owned scheduler. Preserve source ordering, currFileIdx advancement, successful-file counting, aggregate error handling, and post-loop collision checks while retaining destination construction and private file/resource work in C callbacks.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Move the shared-output arm of FIO_compressMultipleFilenames into a Rust
scheduler. Rust now owns input-order iteration, processed-file accounting,
index advancement, and non-short-circuiting error aggregation while C retains
file validation, private resources, callbacks, diagnostics, and destination
lifetime.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml compression_multiple_shared_destination -- --test-threads=1 (2 passed)
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j1 lib
- make -B -C programs -j1 zstd
- make -B -C tests -j1 test-cli-tests (41 passed)
- all commands used CARGO_BUILD_JOBS=1 where applicable and
ulimit -v 41943040; no heavyweight jobs ran concurrently
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Move the scalar runtime --adapt predicates and compression-level normalization
into Rust. Rust now decides output blocking, output backlog, input starvation,
blocked-input speedups, and bounded slower/faster level changes through an
explicit projection.
Keep the CLI iteration order, refresh clock, frame-progression snapshots,
private FIO/ZSTD contexts, diagnostics, and ZSTD_CCtx_setParameter() mutation
in C. The new C/Rust layout assertions make the scalar policy ABI explicit
without exposing those private layouts.
Test Plan:
- focused adaptive-policy tests: 6 passed
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1:
533 passed
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
--test-threads=1: 169 passed
- legacy compression/decompression/dictionary-builder feature matrix:
588 passed
- six library/CLI clippy gates with -D warnings
- capped serial native library/program rebuilds, 41 CLI tests, Rust library
smoke, and full test-zstd round trips including --adapt cases
- capped serial stress gates: 278 fuzzer cases, 84+129+143 zstream cases,
and 1,601 decode-corpus cases
- every heavyweight command used CARGO_BUILD_JOBS=1 or make -j1 and
ulimit -v 41943040; no worker/native process remained afterward
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
The CLI's zstd-format compression path previously kept the asynchronous
read, compressStream2, output-job, flush, accounting, adaptive-policy, and
progress loop together in fileio.c. That made the high-level stream
orchestration another large C-owned surface even though the Rust fileio
module already owned the neighboring format loops.
Move the format-independent zstd stream loop and read/output accounting into
FIO_rust_compressZstdFrame. The C adapter now projects read-pool, write-pool,
codec, and policy operations through narrow callbacks. C retains the private
ZSTD_CCtx interaction, adaptive-level policy, memory diagnostics, progress
formatting, and CLI error mapping, so no private C layout crosses into Rust.
Focused seam tests cover input/output ordering, final flush behavior, codec
error propagation, and incomplete known-size input handling.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1`
-- 513 passed under the 40 GiB virtual-memory cap.
- `cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
--test-threads=1` -- 169 passed under the cap.
- `cargo clippy` lib/benches/tests for `rust` and `rust/cli`, with
`-D warnings`, and nightly formatting -- passed.
- `make -B -C lib -j1 lib` and `make -B -C programs -j1 zstd` -- passed.
- Native CLI, full zstd, fuzzer, zstream, and decode-corpus targets -- passed
serially under the cap.
GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
Move mixed-format decompression status mapping and successful-file finalization
into Rust. C keeps the diagnostic and private FIO context callbacks, while
Rust decides pass-through/error behavior and invokes final accounting only for
a successful decoded file.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-cli-tests
`FIO_compressFilename_internal()` still selected the zstd, gzip, xz/lzma, and
lz4 paths in a C-owned format switch and updated aggregate byte accounting
around those callbacks. That left the mixed-format CLI dispatch loop outside
the Rust file-I/O orchestration even though the pools and decompression path
were already projected there.
Add a callback table for the codec leaves and move format selection, optional
codec status handling, post-success accounting, and progress-hook ordering to
Rust. The existing C codec implementations, diagnostics, elapsed-time
formatting, and resource ownership remain unchanged; unsupported optional
formats return explicit statuses so the C wrapper preserves its original
EXM_THROW messages.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1` -- 473 passed.
- Native `test-cli-tests` -- all 41 passed.
- Native `test-zstd` -- passed, including async zstd/gzip/xz/lzma/lz4 round trips.
- `make -B -C programs -j2 zstd` and `make -B -C tests/fuzz -j2 all` -- passed.
The CLI already delegated one zstd frame at a time to Rust, but the C
FIO_decompressFrames loop still owned repeated-frame dispatch. That left
concatenated zstd streams split across the language boundary and made the
Rust frame helper unable to preserve the next mixed-format header itself.
Add a Rust multi-frame adapter that repeatedly probes four buffered bytes,
uses the existing one-frame decoder, accumulates decoded output and progress,
and leaves a following non-zstd header or short trailing input untouched for
C's format dispatcher. Keep C responsible for probing the first format,
non-zstd dispatch, diagnostics, cleanup, and final file accounting. Decoder
errors still preserve the current input for FIO_zstdErrorHelp().
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib fileio_asyncio -- --test-threads=1` -- 20 passed
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `make -B -C tests -j2 test-cli-tests` -- all 41 passed
- `git diff --cached --check` -- passed
- Full root `cargo clippy --all-targets/--benches/--tests -- -D warnings` remains blocked by the pre-existing `manual_repeat_n` warning in `rust/src/zstd_compress.rs` test code; no unrelated change was made.
FIO_openDstFile still performed destination classification, platform-specific
file creation, stdio buffering, and overwrite preparation in C, even though
source opening and removal already used Rust filesystem leaves. That made the
CLI backend's most important output safety path split across two implementations.
Add a Rust destination leaf with a small status ABI. Rust owns sentinel and
same-file classification, regular-file detection, platform binary open and
fdopen, truncation, and buffering. C keeps the user-facing diagnostics,
sparse-mode preference mutation, overwrite prompt, and existing remove-file
wrapper; it performs a second Rust open after an accepted overwrite decision so
no prompt or policy is duplicated. Existing file contents are left untouched
until that C-owned decision is complete.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1`
-- passed (411 tests, including destination classification/open tests).
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings`
-- passed.
- `cargo clippy --manifest-path rust/cli/Cargo.toml --lib
--no-default-features --features cli,compression,decompression,benchmark,
dict-builder -- -D warnings` -- passed.
- Nightly fmt checks for both Rust manifests -- passed.
- `make -B -C tests -j2 test-cli-tests` -- passed (41 tests).
The C benchmark translation unit still contained the complete BMK
implementation, so program builds and paramgrill could silently keep using a
second C implementation even though rust/src/benchzstd.rs already provided the
same ABI. Replace that duplicate with a declaration-only shim and retain
compile-time checks for the by-value result and parameter layouts.
The standalone test tools do not link the full CLI archive. Keep their
helpers archive free of the benchmark and trace-only objects, and add a
separate benchmark archive for paramgrill so the shim remains linkable without
pulling program-only trace dependencies into datagen and similar targets.
Test Plan:
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark,dict-builder --lib -- --test-threads=1` -- 165 passed
- `make -B -C programs -j2 zstd` -- passed
- `./programs/zstd -b1 tests/hello` -- passed
- `make -B -C tests paramgrill` -- passed
- `./tests/paramgrill -S tests/hello` -- passed
- `make -C tests -j2 test-fullbench` -- passed
- targeted Rust clippy and nightly rustfmt checks -- passed
- full all-target clippy remains blocked by pre-existing test-only lints in `zstd_compress.rs` and `fileio_backend.rs`
The CLI trace translation unit still contained the complete CSV writer and
strong callback implementations even though the Rust CLI archive already had
the equivalent trace module. Reduce the C file to its public-header shim and
make the Rust implementation authoritative, preserving the exact CSV header,
version assertion, timing fields, and no-trace build behavior. Focused tests
cover header creation and the version-mismatch assertion.
Test Plan:
- CLI tests: 161 default and 127 reduced-feature -- passed
- Focused trace tests: 3/3 in both configurations -- passed
- `make -B -C programs -j2` for all CLI variants -- passed
- Trace compression/decompression smoke test -- passed
- C shim compile with and without `ZSTD_NOTRACE` -- passed
- Clippy, nightly rustfmt, and `git diff --check` -- passed
The verbose compression-preference summary was still formatted by the C
file-I/O orchestration layer even though the preference object and the rest of
its policy helpers already lived in Rust. Move the option selection and exact
summary formatting into the Rust CLI archive, keeping the public C function as
a narrow assertion-and-dispatch shim. The Rust formatter preserves the legacy
option spellings, default memory limit, integer casts, and stderr output, with
a focused string-format regression test.
Test Plan:
- `cargo fmt --manifest-path rust/cli/Cargo.toml -- --check` -- passed
- `cargo test --manifest-path rust/cli/Cargo.toml --lib fileio_prefs -- --test-threads=1` -- 46 passed
- `git diff --check` -- passed
Route FIO_freeDict through the Rust filesystem backend for malloc-backed and
mapped dictionaries. Keep enum validation in C, pair Windows views with their
file handles, support the non-POSIX malloc fallback, and clear every ownership
field so repeated cleanup remains harmless.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib fileio_backend -- --test-threads=1 (25 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- rustfmt +nightly --edition 2021 rust/src/fileio_backend.rs --check
- make -C programs -j2 zstd
- git diff --check
Keep dictionary-training code out of compressor-only and decompressor-only CLI
archives, where its ZDICT symbols are intentionally absent. Add a dedicated
Rust CLI archive for zstd-dictBuilder, enable the builder feature only for the
full and dictionary-builder programs, and make reduced-feature help use the
feature-independent max-level helper.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib -- --test-threads=1 (158 passed)
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,dict-builder --lib -- --test-threads=1 (124 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,dict-builder --lib -- -D warnings
- make -C programs -j2 zstd zstd-small zstd-frugal zstd-decompress zstd-compress zstd-dictBuilder
- git diff --check
Move the platform-specific dictionary mapping and release operations behind the
Rust CLI archive while keeping size policy, diagnostics, and FIO_Dict_t
ownership decisions in C. POSIX descriptors and Windows handles are cleaned up
on every failure path, and zero-length dictionaries use an owned sentinel so
the existing non-null dictionary invariant remains valid.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib fileio_backend (24 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- make -C programs -j2 zstd
- --mmap-dict compression/decompression round trip with programs/README.md
- programs/zstd --mmap-dict --test on the generated frame
- rustfmt +nightly --edition 2021 rust/src/fileio_backend.rs --check
- git diff --check
Keep FIO_openSrcFile's stdin sentinel handling, Windows binary-mode setup,
existing diagnostics, and FILE ownership in the C wrapper while moving only
non-stdin source validation and opening behind the Rust CLI ABI. The Rust leaf
returns distinct stat, non-regular, and fopen failure statuses, delegates
metadata and regular/FIFO/block-device classification to the existing utility
ABI, and passes the original C path directly to fopen("rb"). Opaque stat_t and
FILE* pointers cross the boundary, and the output stream is published only on
success; Rust never closes a returned stream.
Focused backend tests cover regular-file reads, missing and directory paths,
unchanged output pointers on failure, empty and spaced paths, and symlinks.
FIFO acceptance remains in the preserved classification order without a test
that could block while opening a named pipe.
Test Plan:
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression --lib fileio_backend` -- 22 passed
- CLI clippy for library, benches, and tests with `cli,compression,decompression,benchmark`, before and after nightly formatting -- clean after the new-code warnings were fixed
- `cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all -- --check` -- passed
- Root Rust clippy with the reduced `compression` feature split for library, benches, and tests -- passed with one pre-existing warning in forbidden `rust/src/zstd_compress.rs`
- `make -B -C programs -j2 zstd` -- passed
- `make -B -C programs -j2 zstd-small zstd-frugal zstd-dictBuilder` -- passed with existing unused-function warnings in compact CLI builds
- `make -C tests -j2 test-cli-tests` -- all 41 CLI tests passed, including file-stat coverage
- `make -C tests -j2 test-zstd` -- reached dictionary training, then hit a segmentation fault in the unrelated concurrent compressor/dictionary path
- `git diff --check` and `git diff --cached --check` -- passed
Keep the C --list orchestration, file ownership, fileInfo_t storage, and
DISPLAYLEVEL/reporting behavior in programs/fileio.c while replacing only the
frame scanner with a guarded Rust FFI leaf. The Rust implementation preserves
the C status values, fread lookahead, large-file seek/tell behavior, frame and
skippable-frame accounting, content-size and window updates, checksum capture,
RLE and invalid-block handling, dictionary-ID aggregation, and truncation
classification without taking ownership of FILE or the output structure.
A C diagnostic bridge keeps the existing CLI messages and warning formatting
outside the scanner. The Rust tests use temporary C streams and focused frame
buffers, including a test-only header API shim so the unit tests remain safe
and link independently of the full CLI binary. The decompression feature guard
also keeps the scanner absent from no-decompression CLI archives.
Test Plan:
- cargo test --no-default-features --features cli,compression,decompression,benchmark --lib fileio_prefs (45 passed)
- cargo test --no-default-features --features cli,compression,decompression,benchmark (152 passed)
- library and CLI clippy for main, benches, and tests, before and after nightly formatting (clean)
- cargo +nightly fmt -- --check for both library and CLI crates (clean)
- make -B -C programs -j2 zstd, zstd-small, zstd-frugal, and zstd-dictBuilder (passed)
- exact --list/-l playTests.sh blocks (passed)
- make -C tests -j2 test-cli-tests (41 passed)
- nm archive guard check confirmed FIO_rust_analyzeFrames only in the decompression-enabled CLI archive
- zstd-decompress remains blocked by the pre-existing DEFAULT_MAX_CLEVEL feature-gating error in rust/src/zstd_cli.rs
- zstd-compress remains blocked by pre-existing unresolved ZDICT_* references; neither blocker is in the owned files
Move dictionary-path statting behind the existing Rust CLI file-I/O backend while keeping FIO_getDictFileStat as the C policy adapter. The adapter still owns the NULL-path early return, errno-based EXM_THROW(31) diagnostic, EXM_THROW(32) regular-file policy, and all dictionary orchestration.
FIO_rust_getDictFileStat crosses only the existing path and stat_t boundary, delegates metadata and regular-file classification to UTIL_stat and UTIL_isRegularFileStat, and returns 0/1/2 for success, stat failure, or non-regular input. Reusing those ABIs preserves the C stat_t representation, errno behavior, and stat's symlink-following semantics. Focused backend tests cover regular, missing, directory, NULL, and symlink paths.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression --lib fileio_backend (15 passed)
- cargo clippy --no-default-features --features compression, including --benches and --tests (passed)
- CLI cargo clippy with cli,compression,decompression,benchmark, including --benches and --tests (passed)
- cargo +nightly fmt checks for the owned Rust module and CLI crate (passed); root-wide check remains limited by unrelated dict_builder_zdict.rs edits
- Existing CLI dictionary scenarios for missing and directory paths retained status 31/32 and exact diagnostics
- git diff --check and git diff --cached --check (passed)
- zstd build variants were attempted; full links remain blocked by unrelated active dict-builder/zstd-cli/archive edits after the owned C file compiled
Keep the public ZSTD_getCParams() selection and all DISPLAYLEVEL output in
programs/fileio.c, but pass its by-value compression-parameter snapshot to a
Rust policy leaf. Rust now preserves the original source-size highbit and
window clamping rules, validates and updates the patch memory limit before
success outputs, derives cycleLog, updates comprParams->windowLog, and applies
automatic LDM without overwriting the explicit-LDM diagnostic semantics.
The Rust ABI returns the raw file window and the two diagnostic predicates so
C can retain its existing messages and optimal-parser note ordering. C layout
assertions cover the seven-word compression-parameter snapshot on both
platform-width variants; rejected unknown and oversized inputs leave all
outputs unchanged.
Test Plan:
- Rust clippy pre/post nightly-format matrix for library compression and CLI
cli,compression,decompression,benchmark features -- passed.
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features
--features cli,compression,decompression,benchmark` -- 137 passed.
- `make -C programs -j2 zstd` -- passed.
- `make -C tests -j2 test-cli-tests` -- 41 passed.
- `python3 tests/cli-tests/run.py decompression/pass-through.sh` -- passed.
- Focused patch-from round trips, stream-size failure ordering, and automatic
long-mode diagnostics -- passed.
- `make -C tests -j2 test-rust-lib-smoke` -- passed.
- No i686 Rust target is installed; 32-bit policy bounds are covered by
size_of-based constants and C/Rust ABI assertions.
Keep patch-from orchestration in C while extracting only the bounded window
selection and automatic-LDM decision into the Rust CLI leaf. The C caller
still computes the raw file window with FIO_highbit64, obtains cParams,
updates the memory limit, emits warnings and optimal-parser notes, and writes
comprParams->windowLog and prefs->ldmFlag. Rust receives only scalar policy
inputs and two output pointers, clamps the selected window to the target's
10..30/31 bounds, and compares the unclamped value strictly against cycleLog.
This preserves equality as no-LDM and keeps values above the maximum eligible
to trigger LDM. Focused unit tests cover both clamps and each comparison edge.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features
--features compression` with library, `--benches`, and `--tests`, before and
after formatting -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --
--config skip_children=true` and nightly rustfmt on the owned file -- passed
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features
--features cli,compression,decompression,benchmark fileio_prefs` -- 36 passed
- `make -C programs -j2 zstd` -- passed
- `make -C tests -j2 test-cli-tests` -- 41 passed
- Focused patch-from round trip and automatic long-mode trigger -- passed
Keep decompression resource ownership and file orchestration in C while the
Rust AIO module handles the pass-through byte-copy leaf through a two-pool ABI.
The Rust implementation derives the block limit from the actual read and write
job buffers, preventing a write-job overrun when their configured sizes differ.
It preserves the existing enqueue/reacquire, consume/refill, EOF, release, and
sparse-write ordering, with pool-backed coverage for synchronous and threaded
operation.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression --lib fileio_asyncio` -- passed, 9 tests.
- Required clippy/fmt sequence against `rust/Cargo.toml` with compression -- passed before and after nightly fmt.
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression --lib` -- passed, 342 tests.
- `make -C programs -j2 zstd` -- passed.
- Manual 200 KiB and empty-input `programs/zstd -dc --pass-through` comparisons -- passed.
Move malloc-backed dictionary file reads into the Rust CLI backend while keeping C responsible for stat metadata, patch-mode size policy, diagnostics, buffer-type selection, and eventual free(). The Rust leaf returns explicit open, size, allocation, and read statuses and publishes only fully read libc allocations.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (126 tests)
- cargo clippy --manifest-path rust/cli/Cargo.toml
- make -B -C programs -j2 zstd
- make -C tests -j2 test-cli-tests (41 tests)
Route the CLI backend's safe file-removal leaf through a narrow Rust status ABI while preserving C diagnostics, Windows read-only handling, and the existing return convention.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml; cargo clippy --manifest-path rust/cli/Cargo.toml; cargo clippy --manifest-path rust/cli/Cargo.toml --benches; cargo clippy --manifest-path rust/cli/Cargo.toml --tests; make -B -C programs -j2 zstd; make -C tests -j2 test-cli-tests (41 tests); make -B -C programs -j2 zstd-small zstd-frugal.
Move the CLI's strategy-dependent cycle-log calculation behind a scalar Rust ABI shim while retaining the C assertion and call-site behavior.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark (116 tests)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets with the full CLI feature set
- make -B -C programs -j2 zstd zstd-small zstd-frugal
- make -C tests -j2 test-cli-tests (41 scenarios)
Move patch-from memory-limit sizing and rejection status selection behind a Rust ABI shim. Keep the C diagnostics and assertion in place, and leave the existing preference unchanged when input sizes are unknown or exceed the window limit.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark (113 tests)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets with the CLI feature set
- make -B -C programs -j2 zstd zstd-small zstd-frugal
- make -C tests -j2 test-cli-tests (41 scenarios) and make -B -C tests -j2 test-zstream
Port the single-file and multiple-file summary decisions through the existing FIO context ABI, including the display-level lookup and the original multiple-file assertion invariant. Keep the C wrappers and call sites unchanged.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark (108 passed); CLI all-target clippy; make -B -C programs -j2 zstd zstd-small zstd-frugal; make -C tests -j2 test-cli-tests (41 passed).
Mirror fileio.c's private fileInfo_t with a checked repr(C) ABI and aggregate totals, flags, and file counts through an output-pointer shim. Preserve zeroed non-aggregate fields and use wrapping arithmetic for the original integer operations.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark (106 passed); cargo clippy --all-targets with the same features; make -B -C programs -j2 zstd zstd-small zstd-frugal; make -C tests -j2 test-cli-tests (41 passed).
Route file-I/O input/output buffer construction and the LZ4 block-size mapping through Rust ABI leaves while preserving the existing C helper names and public buffer layouts. Add layout, field, and formula coverage.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark; cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets --no-default-features --features cli,compression,decompression,benchmark; make -B -C programs -j2 zstd zstd-small zstd-frugal; make -B -C tests -j2 test-cli-tests
Move the basename-preserving output-directory helper behind the existing Rust utility ABI. Keep the libc allocation contract and platform separator behavior intact while leaving the C file-I/O engines and higher-level naming policy unchanged.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (93 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets
- make -B -C tests -j2 test-cli-tests (41 passed)
Implement the CLI trace callbacks and CSV logger in Rust, preserving the ZSTD_Trace ABI, timing fields, parameter reporting, mutex-protected writes, and no-op disabled behavior. Add the module to every Rust CLI archive and exclude the duplicate C trace object from Rust-backed program targets.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (92 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets
- make -B -C programs -j2 zstd
- Rust trace smoke: programs/zstd -q --trace=/tmp/zstd-rust-trace-smoke.csv -c /tmp/levels-input
- make -B -C tests -j2 test-cli-tests (41 passed)
Replace the program archive's C benchmark orchestration with the complete
Rust port while retaining the public benchzstd.h ABI and result/error
contracts. Include the module only in full CLI archives and remove only
benchzstd.c from the program source set, leaving C test targets on their
existing implementation.
Test Plan:
- cargo test --manifest-path /tmp/zstd-benchzstd.IHWpLD/Cargo.toml (6 passed)
- cargo check and cargo clippy --all-targets in the focused harness (passed)
- standalone release archive build and BMK_* symbol inspection (passed)
- make -C programs zstd (passed before concurrent CLI edits)
- make -C tests -B paramgrill (passed)
- native synthetic, multi-file, dictionary, and invalid-frame benchmark smoke tests (passed)
Replace programs/fileio_asyncio.c with a declaration-only shim and register
Rust's async file-I/O implementation in the library archive. Preserve the C
ABI for pool and job contexts, including DEBUGLEVEL pointer-based pthread
wrappers, and select separate Rust build configurations for those layouts.
Keep ordered-read queue state locked while waiting, drain queued work before
shutdown frees job buffers, and use platform large-file seeks for sparse writes.
Test Plan:
- `cargo test --no-default-features --lib fileio_asyncio` (7 passed).
- `cargo test --no-default-features --features debug-pthread --lib fileio_asyncio` (7 passed).
- `cargo check --all-targets`, clippy library/benches/tests, and nightly fmt check (passed).
- Threaded program build and README round-trip (passed).
- `make -C tests poolTests` and `./tests/poolTests` (passed).
- Current C shim object defines no `AIO_*` symbols; the Rust archive exports all 20 declarations.
- Full standalone all-target Rust tests and the default CLI archive rebuild remain affected by unrelated C/Rust integration and another worker's in-progress benchmark changes.
Move filename tables, file-list expansion, core-count helpers, statistics,
and dictionary sample loading behind Rust implementations while retaining
the narrow C ABI used by the CLI.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- make -B -C programs zstd V=1
- make -C tests check V=1
Move fileio preference and context allocation, defaults, setters, and small
query helpers into Rust while keeping the actual file operations in C. Keep
the C structs and ABI declarations as anchors, and make the CLI build track
the new Rust source.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/fileio_prefs.rs rust/cli/src/lib.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/cli/Cargo.toml
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- git diff --cached --check
Keep the original C translation units as declaration-only build shims while
the Rust CLI helper archive owns datagen and lorem. Track both Rust sources in
the program and test archive prerequisites, and link the helpers-only archive
into C test binaries that still provide their own file I/O and orchestration.
This preserves the existing C test harnesses while making the migrated helper
implementations the single definitions used by the binary and tests.
Test Plan:
- make -C programs zstd V=1
- make -C tests datagen V=1
- make -C tests test-rust-lib-smoke V=1
- git diff --check
Refs: programs/datagen.c, programs/lorem.c, tests/Makefile
The legacy decoders (lib/legacy/zstd_v01.c .. zstd_v07.c) are next in
the Rust migration. Each of those files is a frozen snapshot of the
FSE/Huff0 entropy coders and frame logic of one historical release, so
their ports must not reuse the modern Rust entropy modules and must not
share code with each other: outputs and error codes have to stay
byte-identical to the frozen C forever. This commit installs the
build-system scaffolding so seven per-version ports can land
independently, each adding only its own module file plus a one-line
registration in rust/src/legacy/mod.rs.
Cargo grows features legacy-v01 .. legacy-v07. They are never default
features: the C build defaults differ per build system, so each build
system passes the list explicitly, derived from its own legacy
configuration:
- lib/Makefile and programs/Makefile map ZSTD_LEGACY_SUPPORT=N to the
features for versions N..7 (0 disables legacy), mirroring the
ZSTD_LEGACY_FILES selection in lib/libzstd.mk.
- tests/Makefile always enables all seven features because its
ZSTDLEGACY_FILES wildcard compiles every lib/legacy/*.c regardless of
the dispatch level.
- build/meson maps legacy_level exactly like the makefiles; build/cmake
enables all seven whenever ZSTD_LEGACY_SUPPORT is ON because it
always compiles all seven C files (ZSTD_LEGACY_LEVEL only selects the
C dispatch).
Every build system also encodes the legacy selection in the Rust target
directory name (e.g. c1-d1-default-legacy5), for the same reason the
HUF mode is encoded there: a cached archive built for one configuration
must never be linked into a build expecting another. In tests/Makefile
the legacy level additionally flows into the existing HUF C-mode stamp,
so the flat C test objects (which bake -DZSTD_LEGACY_SUPPORT into the
dispatch) are rebuilt whenever the level changes. In programs/Makefile
the compress-only, decompress-only, and CLI archives keep
level-independent directories (RUST_HUF_MODE) because they are only
linked into ZSTD_LEGACY_SUPPORT=0 program variants and carry no legacy
features.
A feature whose version has not been ported yet gates nothing: the
module registration in rust/src/legacy/mod.rs is added by each port,
so enabling e.g. legacy-v05 today simply leaves that decoder in C.
This is what makes mixed C/Rust legacy levels link cleanly while the
seven ports land in any order.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
-- -D warnings && cargo test --all-targets
- cargo clippy with --no-default-features --features
decompression,legacy-v01 and with all seven legacy features
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke; make -C tests test-legacy
- make -C lib libzstd.a with ZSTD_LEGACY_SUPPORT=0, 1 and default (5)
- cmake configure and meson setup (including -Dlegacy_level=1) emit the
expected --features lists and legacy-suffixed target directories
Move the implementation of programs/benchfn.c into rust/src/benchfn.rs and
wire benchmark mode (-b/-e/-i) into the Rust CLI frontend, which previously
rejected those options as not yet implemented. `zstd -b1 -i0 FILE` and range
runs like `zstd -b5e6 -i0 FILE` work again, including the synthetic-sample
benchmark when no file is given.
benchfn.rs is a faithful port of the run/timing state machine:
BMK_benchFunction keeps the exact loop accounting (first-loop blockResults
and errorFn checks, dstSize summed on the first loop only, 0xE5 warm-up of
result buffers, nbLoops minimum of 1) and BMK_benchTimedFn keeps the same
convergence behavior (x10 workload growth for short runs, budget-based
nbLoops estimation, runs below half the run budget re-tried rather than
reported, best qualifying run returned). Arithmetic that C leaves to
unsigned wrap-around uses wrapping operations so debug builds cannot panic
where release C would wrap.
ABI notes: BMK_runTime_t and BMK_runOutcome_t are returned by value across
the C boundary and BMK_benchParams_t is passed by value, so all three are
repr(C) mirrors of benchfn.h; their field offsets are pinned by const
asserts in Rust and matching C static asserts in the benchfn.c shim, which
is now declaration-only. BMK_timedFnState_t stays opaque, fits the 64-byte
BMK_timedFnState_shell (compile-time checked), and is malloc/free-managed
so creation and destruction remain interchangeable with C callers.
The CLI parses -b (bench mode), -e (range end, digits attach directly,
defaulting to 0 like readU32FromChar) and -i (duration in seconds), then
dispatches through a new ZSTD_rust_cli_bench bridge in the zstdcli.c shim.
The bridge exists because benchmark availability is a C preprocessor
property (ZSTD_NOBENCH): orchestration and reporting stay in C benchzstd.c,
stripped variants (zstd-small, zstd-compress, zstd-decompress) compile the
stub branch and report "benchmark mode is not available in this build", and
the Rust side never references benchmark symbols directly. Level clamping
against ZSTD_maxCLevel() happens in the bridge, where the symbol is
guaranteed to exist whenever benchmarking is compiled in. -T selects the
worker count, defaulting to single-threaded like the C bench path; -S
(separate files) and --priority=rt remain unimplemented.
Makefile updates only extend the Rust source prerequisite lists with
benchfn.rs; the helpers-archive plumbing from the timefn commit already
links fullbench(-lib/-dll/32) and paramgrill, the benchfn consumers among
the C tests. Original C test sources are untouched.
Known pre-existing issues, unchanged by this commit: tests/fullbench-lib
fails to link at the base commit too (libzstd.a precedes fullbench.c in its
link line), and the cli-tests basic/help.sh, compression/levels.sh,
compression/golden.sh, and decompression/pass-through.sh scripts fail
identically with a base-commit binary because the Rust CLI frontend is
still a partial reimplementation.
Test Plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings
&& cargo test --all-targets && cargo build --release
- cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets; repeat tests with
--no-default-features plus features compression / decompression / (none)
- make -C programs zstd; ./programs/zstd -b1 -i0 lib/common/xxhash.c;
./programs/zstd -b5e6 -i0 programs/fileio.c; ./programs/zstd -b1 -i0
(synthetic); echo roundtrip via zstd | zstd -d
- make -C programs zstd-small zstd-compress zstd-decompress zstd-nolegacy
zstd-dictBuilder; zstd-small -b reports benchmark unavailable; compress/
decompress roundtrip across the split binaries
- make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus
poolTests fullbench32 fuzzer32; ./tests/fullbench -i0 (exercises Rust
BMK_benchTimedFn from C); ./tests/fullbench32 -i0; ./tests/fuzzer -i1
--no-big-tests; ./tests/poolTests; make -C tests test-rust-lib-smoke
- cli-tests subset: basic/version.sh, compression/basic.sh,
compression/multiple-files.sh pass; failing scripts match the base commit
Refs: rust/README.md
Move the implementation of programs/timefn.c into rust/src/timefn.rs. The
file provides the monotonic nanosecond clock (UTIL_getTime, span helpers,
UTIL_waitForNextTick, UTIL_support_MT_measurements) used by the CLI and by
several C test tools. timefn.c remains as a declaration-only shim so the
original source lists and header configuration keep working, and it pins the
ABI with static asserts: UTIL_time_t is returned by value and must stay a
plain 64-bit counter, which the Rust #[repr(C)] mirror also asserts.
Platform selection mirrors the C preprocessor structure: Windows uses
QueryPerformanceCounter, Apple targets use mach_absolute_time, and other
POSIX systems use libc clock_gettime(CLOCK_MONOTONIC). Only the unix path is
exercised by this environment; the Windows and Apple paths are written from
the C source and compile-checked logically but are untested here. The C90
clock() fallback is unreachable on Rust-supported targets, so multi-threaded
measurement support is always reported.
The symbols live in the program-only zstd-cli-rs package, keeping them out
of library builds. Linking that archive into C test binaries surfaced a
structural problem: rustc's local ThinLTO promotes internal symbols across
codegen units, so extracting the timefn object could drag in the zstd_cli
parser object, whose FIO_* externs test binaries cannot satisfy. The parser
is therefore gated behind a new additive `cli` cargo feature (default on).
Program archives build with cli,compression,decompression as before, while
tests/Makefile links a helpers-only archive (rust/target/cli-helpers) built
with --no-default-features, which contains no fileio references at all.
tests/Makefile gains build rules for the helpers archive and adds it as a
prerequisite of every binary that compiles the timefn shim: fullbench(32),
fullbench-lib, fullbench-dll, fuzzer(32), zstreamtest(32/asan/tsan/ubsan),
paramgrill, decodecorpus, and poolTests. Prerequisite order places the
archive after all C objects in `$^` link lines; the known-broken -dll
recipes filter to %.c, so they name the archive explicitly. Original C test
sources are untouched; only link inputs changed.
zstd-cli-rs now depends on libc (already used by the core crate) for
clock_gettime and the Mach timebase bindings.
Test Plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings
&& cargo test --all-targets && cargo build --release
- cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets; repeat tests with
--no-default-features plus features compression / decompression / (none)
- make -C programs zstd; roundtrip echo hello | zstd | zstd -d
- make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus
poolTests; ./tests/fullbench -i0; ./tests/fuzzer -i1 --no-big-tests;
./tests/poolTests; make -C tests test-rust-lib-smoke
- verified with nm that the helpers archive member defining UTIL_getTime has
no FIO_*/ZSTD_* undefined references
Refs: rust/README.md
The dictionary-builder sources (lib/dictBuilder) are about to start moving
to Rust, beginning with divsufsort. The Rust crate previously only modeled
the compression/decompression module split plus the forced-HUF decoder
modes, so no build could express "this C configuration includes (or
excludes) dictBuilder" to Cargo. Without that, a Rust archive could carry
dictBuilder modules into a build whose C side disabled them, or worse,
omit a migrated implementation from a build whose C shims require it.
Add a `dict-builder` cargo feature and thread it through every build that
consumes the Rust static archive, mirroring exactly how each build system
already gates the dictBuilder C sources:
- rust/Cargo.toml: new `dict-builder` feature, included in the default
set because the C library builds dictBuilder by default
(ZSTD_LIB_DICTBUILDER ?= 1). The feature is empty until the first
dictBuilder module lands.
- lib/Makefile: RUST_CARGO_FEATURES gains dict-builder when
ZSTD_LIB_DICTBUILDER is enabled, following the existing
ZSTD_LIB_COMPRESSION/ZSTD_LIB_DECOMPRESSION pattern. The archive
directory naming grows a matching `b<0|1>` dimension
(c1-d1-b1-default etc.) so differently configured archives never
collide; the repeated config prefix is factored into
RUST_MODULE_CONFIG.
- programs/Makefile: the full-featured archives now request
compression,decompression,dict-builder (equal to the default set, so
the target directory stays shared with tests). The partial-library
variants gain the `b0` name dimension, and zstd-dictBuilder gets its
own lib-c1-d0-b1 archive because it compiles the dictBuilder C sources
without decompression; it previously shared the compression-only
archive, which will lack the migrated dictBuilder symbols.
- tests/Makefile: no flag change needed since tests use the crate default
feature set; a comment now records that dict-builder arrives that way.
- build/cmake/lib/CMakeLists.txt: ZSTD_BUILD_DICTBUILDER now adds the
dict-builder feature and a `b<0|1>` component in the Rust build-config
directory name, in lockstep with the DictBuilderSources gating.
- build/meson/lib/meson.build: meson compiles the dictBuilder sources
unconditionally, so the feature list and config name gain dict-builder
unconditionally (c1-d1-b1-<huf-mode>).
The `dict-builder` feature deliberately does not imply `compression`.
lib/Makefile forces ZSTD_LIB_DICTBUILDER=0 when compression is disabled,
but CMake does not couple the two options, so encoding the C-side
constraint in Cargo would make the Rust archive diverge from the C source
list in that (already unsupported) CMake configuration.
Test plan:
- cd rust && cargo build --release
- cargo build --release --no-default-features \
--features compression,decompression
- cargo build --release --no-default-features \
--features compression,dict-builder
- Full validation (fuzzer, smoke tests, dictionary byte-identity) runs
with the follow-up commit that ports divsufsort onto this scaffolding.
ddidderr