Call the Rust pass-through and frame-analysis leaves directly from their C
callbacks. The C translation unit still owns private resource projections,
filename suffix configuration, diagnostics, and format-specific operations;
this commit only removes two exact return-forwarding layers.
Test Plan:
- clang -fsyntax-only on programs/fileio.c
- git diff --check
- Full capped native/upstream suite pending after this commit
Keep the local buffer variables declared before the direct Rust buffer-policy
calls in the compression stream adapter, preserving the native C90 warning
profile after removing the old C forwarding helper.
Test Plan: `ulimit -v 41943040; make -j1 -C tests test` passed before this declaration-only cleanup; the next capped native gate will recheck it.
Rely on the existing fileio.h declaration now that the filename collision
checker is exported directly from Rust, avoiding a redundant C redeclaration
in the implementation file.
Test Plan: `ulimit -v 41943040; make -j1 -C tests test` passed; CLI lint and unit gates pending.
Move the pure filename-collision, input/output-buffer construction, and
compressed-destination-name helpers to direct Rust-owned caller symbols.
Leave C responsible for diagnostics, filesystem resources, suffix-list policy,
and the surrounding file-processing orchestration.
Test Plan: Pending capped full verification after this atomic ABI cleanup.
Export the file-summary predicates and largest-file-size scan directly from
Rust under their existing caller symbols. Remove the redundant C forwarding
wrappers while preserving pointer contracts, assertions, return widths, and
filesystem behavior.
Test Plan:
- worker capped format, C syntax, and diff checks
- parent capped root clippy and native build
- parent capped upstream make -j1 -C tests test
- parent capped CLI clippy and tests
Export FIO_LZ4_GetBlockSize_FromBlockId directly from Rust under the existing
caller symbol and remove the redundant C forwarding wrapper. Preserve the
block-ID formula and cover its boundary values in the focused Rust test.
Test Plan:
- worker capped nightly rustfmt check
- worker git diff --check
- full serial capped native and upstream suites to run at the next parent gate
Export FIO_highbit64 directly from Rust under the existing caller symbol and
remove the C forwarding wrapper. Preserve the nonzero-input contract and add
boundary tests for the scalar policy leaf.
Test Plan:
- capped nightly rustfmt check
- capped root clippy with all targets and -D warnings
- capped native make -j1
- capped upstream make -j1 -C tests test
- capped CLI clippy and 185 CLI tests
All integrated checks ran at the combined working-tree tip under a serial
40 GiB virtual-memory cap. Standalone root Rust unit linking remains
unavailable because the crate imports C-owned bridge symbols.
Move the malloc-versus-mmap decision used by compression and decompression
resource construction into Rust. The C adapter continues to own platform
loaders, allocation handles, diagnostics, and dictionary I/O; Rust only
combines the explicit mmap preference, patch-mode size threshold, and explicit
disable override. Add enum-value ABI checks and focused policy coverage.
Also hoist the source-size declaration in the C adapter so the migration does
not introduce a C90 declaration-after-statement warning.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings`
- `ulimit -v 41943040; make -j1`
- `ulimit -v 41943040; make -j1 -C tests test`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets`
The integrated checks ran at the combined working-tree tip under a serial
40 GiB virtual-memory limit. Standalone root Rust unit linking remains
unavailable because the crate imports C-owned bridge symbols without a Cargo
build/link setup.
Project the zstd window-too-large diagnostic inputs into Rust so Rust owns
error filtering and the concrete-versus-unsupported guidance choice. C keeps
frame-header parsing, window-log extraction, and the exact display text behind
a callback, preserving the existing diagnostics and fallback behavior.
All heavy verification was run serially with a 40 GiB virtual-memory cap and
one build job.
Test Plan:
- git diff --cached --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets (790 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (184 passed)
- ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; make -j1 (clean after C90 declaration cleanup)
- ulimit -v 41943040; make -j1 -C tests test (passed; the later cleanup only moved a declaration before statements)
Project the decompression-resource cleanup callbacks into Rust so the stable
teardown order remains dictionary, dstream, write pool, and read pool. C
retains ownership of dRess_t and each private resource implementation, with
its existing dstream CHECK behavior preserved. Focused tests cover order and
null or incomplete callback states.
All heavy verification was run serially with a 40 GiB virtual-memory cap and
one build job.
Test Plan:
- git diff --cached --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets (788 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (181 passed)
- ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
File-resource creation already delegates the general compression-parameter
policy to Rust, but the multithreaded parameters were still applied in C in a
separate branch. That left parameter ordering, optional overlap handling, and
error short-circuiting outside the Rust policy boundary. Add a narrow ABI
projection with C callbacks for CCtx mutation and diagnostics, and let Rust
apply worker count, job size, optional overlap, and rsyncable in the original
order. The C90 declaration layout and compile-time ABI assertions keep the
existing native program configurations intact.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo test --manifest-path rust/Cargo.toml --all-targets` -- 771 passed
- `cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- 179 passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `make -j1` under `ulimit -v 41943040` -- passed without the new C90 warning
- `make -j1 -C tests test` under `ulimit -v 41943040` -- passed
Move the non-threaded compression parameter sequence and adaptive window policy
out of FIO_createCResources. C continues to own context and resource
allocation, dictionary and pool setup, and the multithread-specific settings;
Rust drives the ordered parameter callbacks and stops on the first error.
The projection mirrors the C ABI and has focused order and short-circuit tests.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml create_c_resources --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
Keep fileio's compression-resource teardown order in the Rust policy layer:
dictionary, write pool, read pool, then compression context. The C adapter
keeps cRess_t and each private resource layout local while exposing only
one callback projection to Rust, preserving the existing cleanup behavior.
The Rust entry point also treats a null state as a no-op and verifies the
projection layout at compile time.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml free_c_resources --lib
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
Move stdin and empty-input validation, header selection, per-file iteration,
status aggregation, and multi-file total selection into the Rust CLI policy
layer. The callback projection keeps C responsible for its private file-info
storage, file parsing, diagnostics, human-readable formatting, and per-file
listing. Only successful and frame-error records contribute to the aggregate,
matching the original early-return behavior for invalid or truncated inputs.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/fileio_prefs.rs` -- passed.
- `git diff --cached --check` -- passed.
- Cargo, native, and full test commands were not run per the explicit no-heavy-command constraint.
Move the single-file compression destination lifecycle into the Rust file-I/O
policy layer while keeping C responsible for private resources, file handles,
diagnostics, metadata operations, and codec dispatch callbacks. Preserve the
original shared-destination fast path, temporary permissions, handler timing,
metadata-before-close ordering, close-error propagation, stdout cleanup guard,
and failed-artifact removal.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- Full capped Rust, native, CLI, and upstream test suites to follow
Move the ordered policy from FIO_compressFilename_srcFile into a narrow Rust
projection: named-source stat and collision checks, compressed-file exclusion,
source open and close, size-based async selection, source attachment, and
successful-only --rm handling now run in Rust. The destination callback still
hands compression to the existing Rust compression loop.
Keep FIO_ctx_t, cRess_t, stat_t, FILE/pool handles, signal handling, and
CLI diagnostics behind opaque C callbacks, so the ABI carries only filenames,
scalar policy flags, and callback pointers. Focused Rust policy tests cover
named-file short circuits, stdin behavior, async threshold selection, and the
successful-compression removal gate.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/fileio_asyncio.rs` — passed
- `git diff --check` and path-limited `git diff --cached --check` — passed
- Static ABI/order/diff inspection only; policy tests were not executed
- cargo/make/native/fuzzer/heavy tests were not run per request
Move the separate-destination multi-file decompression loop into the Rust
projection so Rust owns file iteration, progress counters, and aggregate
error handling. Keep destination-name construction, mirror setup, source
opening, format dispatch, diagnostics, and source removal in the C callback
boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Move the shared-destination branch of FIO_decompressMultipleFilenames into a Rust-owned scheduler. Preserve source order, destination propagation, cursor advancement, success counting, aggregate errors, and C-owned source/format/resource behavior including --rm and output close handling.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_shared_destination -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Move the separate-destination branch of FIO_compressMultipleFilenames into a Rust-owned scheduler. Preserve source ordering, currFileIdx advancement, successful-file counting, aggregate error handling, and post-loop collision checks while retaining destination construction and private file/resource work in C callbacks.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Move the shared-output arm of FIO_compressMultipleFilenames into a Rust
scheduler. Rust now owns input-order iteration, processed-file accounting,
index advancement, and non-short-circuiting error aggregation while C retains
file validation, private resources, callbacks, diagnostics, and destination
lifetime.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml compression_multiple_shared_destination -- --test-threads=1 (2 passed)
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j1 lib
- make -B -C programs -j1 zstd
- make -B -C tests -j1 test-cli-tests (41 passed)
- all commands used CARGO_BUILD_JOBS=1 where applicable and
ulimit -v 41943040; no heavyweight jobs ran concurrently
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Move the scalar runtime --adapt predicates and compression-level normalization
into Rust. Rust now decides output blocking, output backlog, input starvation,
blocked-input speedups, and bounded slower/faster level changes through an
explicit projection.
Keep the CLI iteration order, refresh clock, frame-progression snapshots,
private FIO/ZSTD contexts, diagnostics, and ZSTD_CCtx_setParameter() mutation
in C. The new C/Rust layout assertions make the scalar policy ABI explicit
without exposing those private layouts.
Test Plan:
- focused adaptive-policy tests: 6 passed
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1:
533 passed
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
--test-threads=1: 169 passed
- legacy compression/decompression/dictionary-builder feature matrix:
588 passed
- six library/CLI clippy gates with -D warnings
- capped serial native library/program rebuilds, 41 CLI tests, Rust library
smoke, and full test-zstd round trips including --adapt cases
- capped serial stress gates: 278 fuzzer cases, 84+129+143 zstream cases,
and 1,601 decode-corpus cases
- every heavyweight command used CARGO_BUILD_JOBS=1 or make -j1 and
ulimit -v 41943040; no worker/native process remained afterward
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
The CLI's zstd-format compression path previously kept the asynchronous
read, compressStream2, output-job, flush, accounting, adaptive-policy, and
progress loop together in fileio.c. That made the high-level stream
orchestration another large C-owned surface even though the Rust fileio
module already owned the neighboring format loops.
Move the format-independent zstd stream loop and read/output accounting into
FIO_rust_compressZstdFrame. The C adapter now projects read-pool, write-pool,
codec, and policy operations through narrow callbacks. C retains the private
ZSTD_CCtx interaction, adaptive-level policy, memory diagnostics, progress
formatting, and CLI error mapping, so no private C layout crosses into Rust.
Focused seam tests cover input/output ordering, final flush behavior, codec
error propagation, and incomplete known-size input handling.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1`
-- 513 passed under the 40 GiB virtual-memory cap.
- `cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
--test-threads=1` -- 169 passed under the cap.
- `cargo clippy` lib/benches/tests for `rust` and `rust/cli`, with
`-D warnings`, and nightly formatting -- passed.
- `make -B -C lib -j1 lib` and `make -B -C programs -j1 zstd` -- passed.
- Native CLI, full zstd, fuzzer, zstream, and decode-corpus targets -- passed
serially under the cap.
GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
Move mixed-format decompression status mapping and successful-file finalization
into Rust. C keeps the diagnostic and private FIO context callbacks, while
Rust decides pass-through/error behavior and invokes final accounting only for
a successful decoded file.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C programs -j2 zstd
- make -B -C tests -j2 test-cli-tests
`FIO_compressFilename_internal()` still selected the zstd, gzip, xz/lzma, and
lz4 paths in a C-owned format switch and updated aggregate byte accounting
around those callbacks. That left the mixed-format CLI dispatch loop outside
the Rust file-I/O orchestration even though the pools and decompression path
were already projected there.
Add a callback table for the codec leaves and move format selection, optional
codec status handling, post-success accounting, and progress-hook ordering to
Rust. The existing C codec implementations, diagnostics, elapsed-time
formatting, and resource ownership remain unchanged; unsupported optional
formats return explicit statuses so the C wrapper preserves its original
EXM_THROW messages.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1` -- 473 passed.
- Native `test-cli-tests` -- all 41 passed.
- Native `test-zstd` -- passed, including async zstd/gzip/xz/lzma/lz4 round trips.
- `make -B -C programs -j2 zstd` and `make -B -C tests/fuzz -j2 all` -- passed.
The CLI already delegated one zstd frame at a time to Rust, but the C
FIO_decompressFrames loop still owned repeated-frame dispatch. That left
concatenated zstd streams split across the language boundary and made the
Rust frame helper unable to preserve the next mixed-format header itself.
Add a Rust multi-frame adapter that repeatedly probes four buffered bytes,
uses the existing one-frame decoder, accumulates decoded output and progress,
and leaves a following non-zstd header or short trailing input untouched for
C's format dispatcher. Keep C responsible for probing the first format,
non-zstd dispatch, diagnostics, cleanup, and final file accounting. Decoder
errors still preserve the current input for FIO_zstdErrorHelp().
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib fileio_asyncio -- --test-threads=1` -- 20 passed
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `make -B -C tests -j2 test-cli-tests` -- all 41 passed
- `git diff --cached --check` -- passed
- Full root `cargo clippy --all-targets/--benches/--tests -- -D warnings` remains blocked by the pre-existing `manual_repeat_n` warning in `rust/src/zstd_compress.rs` test code; no unrelated change was made.
FIO_openDstFile still performed destination classification, platform-specific
file creation, stdio buffering, and overwrite preparation in C, even though
source opening and removal already used Rust filesystem leaves. That made the
CLI backend's most important output safety path split across two implementations.
Add a Rust destination leaf with a small status ABI. Rust owns sentinel and
same-file classification, regular-file detection, platform binary open and
fdopen, truncation, and buffering. C keeps the user-facing diagnostics,
sparse-mode preference mutation, overwrite prompt, and existing remove-file
wrapper; it performs a second Rust open after an accepted overwrite decision so
no prompt or policy is duplicated. Existing file contents are left untouched
until that C-owned decision is complete.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1`
-- passed (411 tests, including destination classification/open tests).
- `cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings`
-- passed.
- `cargo clippy --manifest-path rust/cli/Cargo.toml --lib
--no-default-features --features cli,compression,decompression,benchmark,
dict-builder -- -D warnings` -- passed.
- Nightly fmt checks for both Rust manifests -- passed.
- `make -B -C tests -j2 test-cli-tests` -- passed (41 tests).
The C benchmark translation unit still contained the complete BMK
implementation, so program builds and paramgrill could silently keep using a
second C implementation even though rust/src/benchzstd.rs already provided the
same ABI. Replace that duplicate with a declaration-only shim and retain
compile-time checks for the by-value result and parameter layouts.
The standalone test tools do not link the full CLI archive. Keep their
helpers archive free of the benchmark and trace-only objects, and add a
separate benchmark archive for paramgrill so the shim remains linkable without
pulling program-only trace dependencies into datagen and similar targets.
Test Plan:
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression,benchmark,dict-builder --lib -- --test-threads=1` -- 165 passed
- `make -B -C programs -j2 zstd` -- passed
- `./programs/zstd -b1 tests/hello` -- passed
- `make -B -C tests paramgrill` -- passed
- `./tests/paramgrill -S tests/hello` -- passed
- `make -C tests -j2 test-fullbench` -- passed
- targeted Rust clippy and nightly rustfmt checks -- passed
- full all-target clippy remains blocked by pre-existing test-only lints in `zstd_compress.rs` and `fileio_backend.rs`
The CLI trace translation unit still contained the complete CSV writer and
strong callback implementations even though the Rust CLI archive already had
the equivalent trace module. Reduce the C file to its public-header shim and
make the Rust implementation authoritative, preserving the exact CSV header,
version assertion, timing fields, and no-trace build behavior. Focused tests
cover header creation and the version-mismatch assertion.
Test Plan:
- CLI tests: 161 default and 127 reduced-feature -- passed
- Focused trace tests: 3/3 in both configurations -- passed
- `make -B -C programs -j2` for all CLI variants -- passed
- Trace compression/decompression smoke test -- passed
- C shim compile with and without `ZSTD_NOTRACE` -- passed
- Clippy, nightly rustfmt, and `git diff --check` -- passed
The verbose compression-preference summary was still formatted by the C
file-I/O orchestration layer even though the preference object and the rest of
its policy helpers already lived in Rust. Move the option selection and exact
summary formatting into the Rust CLI archive, keeping the public C function as
a narrow assertion-and-dispatch shim. The Rust formatter preserves the legacy
option spellings, default memory limit, integer casts, and stderr output, with
a focused string-format regression test.
Test Plan:
- `cargo fmt --manifest-path rust/cli/Cargo.toml -- --check` -- passed
- `cargo test --manifest-path rust/cli/Cargo.toml --lib fileio_prefs -- --test-threads=1` -- 46 passed
- `git diff --check` -- passed
Route FIO_freeDict through the Rust filesystem backend for malloc-backed and
mapped dictionaries. Keep enum validation in C, pair Windows views with their
file handles, support the non-POSIX malloc fallback, and clear every ownership
field so repeated cleanup remains harmless.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib fileio_backend -- --test-threads=1 (25 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- rustfmt +nightly --edition 2021 rust/src/fileio_backend.rs --check
- make -C programs -j2 zstd
- git diff --check
Keep dictionary-training code out of compressor-only and decompressor-only CLI
archives, where its ZDICT symbols are intentionally absent. Add a dedicated
Rust CLI archive for zstd-dictBuilder, enable the builder feature only for the
full and dictionary-builder programs, and make reduced-feature help use the
feature-independent max-level helper.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib -- --test-threads=1 (158 passed)
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,dict-builder --lib -- --test-threads=1 (124 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,dict-builder --lib -- -D warnings
- make -C programs -j2 zstd zstd-small zstd-frugal zstd-decompress zstd-compress zstd-dictBuilder
- git diff --check
Move the platform-specific dictionary mapping and release operations behind the
Rust CLI archive while keeping size policy, diagnostics, and FIO_Dict_t
ownership decisions in C. POSIX descriptors and Windows handles are cleaned up
on every failure path, and zero-length dictionaries use an owned sentinel so
the existing non-null dictionary invariant remains valid.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --lib fileio_backend (24 passed)
- cargo clippy --manifest-path rust/cli/Cargo.toml --lib -- -D warnings
- make -C programs -j2 zstd
- --mmap-dict compression/decompression round trip with programs/README.md
- programs/zstd --mmap-dict --test on the generated frame
- rustfmt +nightly --edition 2021 rust/src/fileio_backend.rs --check
- git diff --check
Keep FIO_openSrcFile's stdin sentinel handling, Windows binary-mode setup,
existing diagnostics, and FILE ownership in the C wrapper while moving only
non-stdin source validation and opening behind the Rust CLI ABI. The Rust leaf
returns distinct stat, non-regular, and fopen failure statuses, delegates
metadata and regular/FIFO/block-device classification to the existing utility
ABI, and passes the original C path directly to fopen("rb"). Opaque stat_t and
FILE* pointers cross the boundary, and the output stream is published only on
success; Rust never closes a returned stream.
Focused backend tests cover regular-file reads, missing and directory paths,
unchanged output pointers on failure, empty and spaced paths, and symlinks.
FIFO acceptance remains in the preserved classification order without a test
that could block while opening a named pipe.
Test Plan:
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression --lib fileio_backend` -- 22 passed
- CLI clippy for library, benches, and tests with `cli,compression,decompression,benchmark`, before and after nightly formatting -- clean after the new-code warnings were fixed
- `cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all -- --check` -- passed
- Root Rust clippy with the reduced `compression` feature split for library, benches, and tests -- passed with one pre-existing warning in forbidden `rust/src/zstd_compress.rs`
- `make -B -C programs -j2 zstd` -- passed
- `make -B -C programs -j2 zstd-small zstd-frugal zstd-dictBuilder` -- passed with existing unused-function warnings in compact CLI builds
- `make -C tests -j2 test-cli-tests` -- all 41 CLI tests passed, including file-stat coverage
- `make -C tests -j2 test-zstd` -- reached dictionary training, then hit a segmentation fault in the unrelated concurrent compressor/dictionary path
- `git diff --check` and `git diff --cached --check` -- passed
Keep the C --list orchestration, file ownership, fileInfo_t storage, and
DISPLAYLEVEL/reporting behavior in programs/fileio.c while replacing only the
frame scanner with a guarded Rust FFI leaf. The Rust implementation preserves
the C status values, fread lookahead, large-file seek/tell behavior, frame and
skippable-frame accounting, content-size and window updates, checksum capture,
RLE and invalid-block handling, dictionary-ID aggregation, and truncation
classification without taking ownership of FILE or the output structure.
A C diagnostic bridge keeps the existing CLI messages and warning formatting
outside the scanner. The Rust tests use temporary C streams and focused frame
buffers, including a test-only header API shim so the unit tests remain safe
and link independently of the full CLI binary. The decompression feature guard
also keeps the scanner absent from no-decompression CLI archives.
Test Plan:
- cargo test --no-default-features --features cli,compression,decompression,benchmark --lib fileio_prefs (45 passed)
- cargo test --no-default-features --features cli,compression,decompression,benchmark (152 passed)
- library and CLI clippy for main, benches, and tests, before and after nightly formatting (clean)
- cargo +nightly fmt -- --check for both library and CLI crates (clean)
- make -B -C programs -j2 zstd, zstd-small, zstd-frugal, and zstd-dictBuilder (passed)
- exact --list/-l playTests.sh blocks (passed)
- make -C tests -j2 test-cli-tests (41 passed)
- nm archive guard check confirmed FIO_rust_analyzeFrames only in the decompression-enabled CLI archive
- zstd-decompress remains blocked by the pre-existing DEFAULT_MAX_CLEVEL feature-gating error in rust/src/zstd_cli.rs
- zstd-compress remains blocked by pre-existing unresolved ZDICT_* references; neither blocker is in the owned files
Move dictionary-path statting behind the existing Rust CLI file-I/O backend while keeping FIO_getDictFileStat as the C policy adapter. The adapter still owns the NULL-path early return, errno-based EXM_THROW(31) diagnostic, EXM_THROW(32) regular-file policy, and all dictionary orchestration.
FIO_rust_getDictFileStat crosses only the existing path and stat_t boundary, delegates metadata and regular-file classification to UTIL_stat and UTIL_isRegularFileStat, and returns 0/1/2 for success, stat failure, or non-regular input. Reusing those ABIs preserves the C stat_t representation, errno behavior, and stat's symlink-following semantics. Focused backend tests cover regular, missing, directory, NULL, and symlink paths.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml --no-default-features --features cli,compression,decompression --lib fileio_backend (15 passed)
- cargo clippy --no-default-features --features compression, including --benches and --tests (passed)
- CLI cargo clippy with cli,compression,decompression,benchmark, including --benches and --tests (passed)
- cargo +nightly fmt checks for the owned Rust module and CLI crate (passed); root-wide check remains limited by unrelated dict_builder_zdict.rs edits
- Existing CLI dictionary scenarios for missing and directory paths retained status 31/32 and exact diagnostics
- git diff --check and git diff --cached --check (passed)
- zstd build variants were attempted; full links remain blocked by unrelated active dict-builder/zstd-cli/archive edits after the owned C file compiled
Keep the public ZSTD_getCParams() selection and all DISPLAYLEVEL output in
programs/fileio.c, but pass its by-value compression-parameter snapshot to a
Rust policy leaf. Rust now preserves the original source-size highbit and
window clamping rules, validates and updates the patch memory limit before
success outputs, derives cycleLog, updates comprParams->windowLog, and applies
automatic LDM without overwriting the explicit-LDM diagnostic semantics.
The Rust ABI returns the raw file window and the two diagnostic predicates so
C can retain its existing messages and optimal-parser note ordering. C layout
assertions cover the seven-word compression-parameter snapshot on both
platform-width variants; rejected unknown and oversized inputs leave all
outputs unchanged.
Test Plan:
- Rust clippy pre/post nightly-format matrix for library compression and CLI
cli,compression,decompression,benchmark features -- passed.
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features
--features cli,compression,decompression,benchmark` -- 137 passed.
- `make -C programs -j2 zstd` -- passed.
- `make -C tests -j2 test-cli-tests` -- 41 passed.
- `python3 tests/cli-tests/run.py decompression/pass-through.sh` -- passed.
- Focused patch-from round trips, stream-size failure ordering, and automatic
long-mode diagnostics -- passed.
- `make -C tests -j2 test-rust-lib-smoke` -- passed.
- No i686 Rust target is installed; 32-bit policy bounds are covered by
size_of-based constants and C/Rust ABI assertions.
Keep patch-from orchestration in C while extracting only the bounded window
selection and automatic-LDM decision into the Rust CLI leaf. The C caller
still computes the raw file window with FIO_highbit64, obtains cParams,
updates the memory limit, emits warnings and optimal-parser notes, and writes
comprParams->windowLog and prefs->ldmFlag. Rust receives only scalar policy
inputs and two output pointers, clamps the selected window to the target's
10..30/31 bounds, and compares the unclamped value strictly against cycleLog.
This preserves equality as no-LDM and keeps values above the maximum eligible
to trigger LDM. Focused unit tests cover both clamps and each comparison edge.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features
--features compression` with library, `--benches`, and `--tests`, before and
after formatting -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --
--config skip_children=true` and nightly rustfmt on the owned file -- passed
- `cargo test --manifest-path rust/cli/Cargo.toml --no-default-features
--features cli,compression,decompression,benchmark fileio_prefs` -- 36 passed
- `make -C programs -j2 zstd` -- passed
- `make -C tests -j2 test-cli-tests` -- 41 passed
- Focused patch-from round trip and automatic long-mode trigger -- passed
Keep decompression resource ownership and file orchestration in C while the
Rust AIO module handles the pass-through byte-copy leaf through a two-pool ABI.
The Rust implementation derives the block limit from the actual read and write
job buffers, preventing a write-job overrun when their configured sizes differ.
It preserves the existing enqueue/reacquire, consume/refill, EOF, release, and
sparse-write ordering, with pool-backed coverage for synchronous and threaded
operation.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression --lib fileio_asyncio` -- passed, 9 tests.
- Required clippy/fmt sequence against `rust/Cargo.toml` with compression -- passed before and after nightly fmt.
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression --lib` -- passed, 342 tests.
- `make -C programs -j2 zstd` -- passed.
- Manual 200 KiB and empty-input `programs/zstd -dc --pass-through` comparisons -- passed.
Move malloc-backed dictionary file reads into the Rust CLI backend while keeping C responsible for stat metadata, patch-mode size policy, diagnostics, buffer-type selection, and eventual free(). The Rust leaf returns explicit open, size, allocation, and read statuses and publishes only fully read libc allocations.
Test Plan:
- cargo test --manifest-path rust/cli/Cargo.toml (126 tests)
- cargo clippy --manifest-path rust/cli/Cargo.toml
- make -B -C programs -j2 zstd
- make -C tests -j2 test-cli-tests (41 tests)
Route the CLI backend's safe file-removal leaf through a narrow Rust status ABI while preserving C diagnostics, Windows read-only handling, and the existing return convention.
Test Plan: cargo test --manifest-path rust/cli/Cargo.toml; cargo clippy --manifest-path rust/cli/Cargo.toml; cargo clippy --manifest-path rust/cli/Cargo.toml --benches; cargo clippy --manifest-path rust/cli/Cargo.toml --tests; make -B -C programs -j2 zstd; make -C tests -j2 test-cli-tests (41 tests); make -B -C programs -j2 zstd-small zstd-frugal.
ddidderr