Move the decompression resource creation order into the Rust CLI policy
layer: dictionary stat and patch-memory preparation, decoder allocation,
window/checksum configuration, dictionary reset/attachment, and asynchronous
pool creation now run through one Rust-owned sequence. Keep dRess_t,
stat_t, dictionary buffers, decoder context, pools, and exact allocation
errors in C callbacks.
Preserve the original patch-from and dictionary-reference behavior while
adding ABI layout assertions, scalar policy coverage, and error short-circuit
tests for the new boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (207 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Make the Rust policy layer own the ordered compression-resource lifecycle:
create the CCtx, prepare patch/dictionary state, create the write and read
pools, validate the dictionary, apply general and multithreaded parameters,
and finally load the dictionary. Keep cRess_t, FIO_Dict_t, file statistics,
AIO pools, and CLI diagnostics in C callbacks so the existing resource
ownership and error behavior remain local to the C backend.
Preserve adaptive window defaults and patch-from parameter adjustment while
adding ABI layout assertions and a lifecycle-order test for the new boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (205 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
FIO_openDstFile() still owned the complete destination policy loop in C even
though the filesystem opener and status/action classifier were already Rust
leaves. That left confirmation, sparse-mode adjustment, overwrite removal, and
retry ordering duplicated beside private FILE* and CLI state.
Project the C-owned callbacks for diagnostics, preference mutation, user
confirmation, stdout setup, and existing-file removal. Rust now owns the
retry/order state machine and calls the existing narrow opener for each
attempt; C keeps private preferences and context, exact diagnostics, and FILE*
ownership. The callback layout is asserted on both sides, and focused tests
cover status ordering, confirmation/removal retry, setvbuf preservation, and
invalid policy inputs.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1` -- passed, including the single-thread library, MT library, and CLI binary.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- passed: 206 tests.
- `git diff --cached --check` -- passed.
The zstd file-I/O loop still kept adaptive compression's state machine in C: progression deltas, refresh gating, job-completion checks, input counters, speed decisions, and level updates were interleaved with private FIO and ZSTD state. That left orchestration policy behind the existing scalar Rust predicates.
Move the adaptive state and callback order into Rust. C now supplies scalar progression snapshots, the clock gate, exact diagnostics, and the ignored CCtx parameter setter through callbacks; private FIO_prefs_t, ZSTD_CCtx, ZSTD_frameProgression, clocks, and progress formatting remain C-owned. Preserve the previous-progression publication before backlog evaluation, input counter reset points, and serial/MT level-clamp behavior.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Move the multi-input single-output decision sequence out of
FIO_multiFilesConcatWarning while preserving the C-owned CLI boundary. The
previous wrapper classified fatal remove cases, emitted the concatenation
warning, disabled --rm, reclassified the action, and then selected quiet
abort or confirmation in C. Add a repr(C) callback projection so Rust owns
only that scalar ordering while C continues to own exact diagnostics, the
confirmation prompt, FIO_prefs_t mutation, and all private state.
The Rust bridge re-runs the action after the C disable-remove callback with
the same has-output/remove arguments as the original wrapper. C and Rust
assert the callback layout, and focused tests cover callback order,
fatal/quiet paths, and ABI offsets. Existing scalar action tests remain in
place.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -B -C programs -j1 fileio.o`
— passed; only pre-existing suffixList C++-compat warnings appeared.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path
rust/cli/Cargo.toml --tests --no-deps` — passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 RUSTFLAGS='-C
link-arg=/tmp/zstd_rust_test_bridges.o' cargo test --manifest-path
rust/cli/Cargo.toml --lib 'fileio_prefs::tests::multi_files_concat_'`
— 8 passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo build --manifest-path
rust/cli/Cargo.toml --lib` — passed; the archive exports
`FIO_rust_multiFilesConcatWarning`.
- `cargo +nightly fmt --manifest-path rust/cli/Cargo.toml -- --check` — not
clean due pre-existing formatting drift in unchanged Rust code; no bulk
formatting was applied.
- Full native/upstream/fuzzer suites were not run by request.
Move the single-file --list status gates, diagnostic-versus-metadata ordering, and aggregate projection into Rust. C retains file opening and frame analysis, the private fileInfo_t layout, exact diagnostics, and row formatting behind explicit callbacks, so the user-visible behavior remains unchanged while the policy boundary is auditable.
Test Plan:
- git diff --cached --check
- worker format, check, clippy, and serial C compilation (passed); focused Rust tests compiled but the standalone link hit the pre-existing C bridge-symbol gap
Keep the private stat_t probe, destination opening, metadata syscalls, and format callbacks in C, but route the regular-source plus stdin/stdout exception policy through the Rust fileio preference layer. The Rust ABI bridge normalizes the scalar consumed by the existing destination lifecycle, with compile-time value assertions and focused tests covering regular, non-regular, stdin, stdout, and nonzero scalar inputs.
Test Plan: Not run by request; cargo, make, native tests, and heavy commands were intentionally avoided. Lightweight git diff --check passed.
Keep FIO_removeFile responsible for the filesystem operation, exact diagnostics, and C return wrapper, but route its status classification through a Rust policy bridge. Unknown statuses are explicitly treated as failed removal. ABI value assertions and focused mapping tests preserve the C contract.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Move the scalar InfoError-to-action mapping used by FIO_listFile into Rust and reuse the same classifier while aggregating multi-file list results. C retains file opening/parsing, exact diagnostics, metadata formatting, private fileInfo_t state, and the public result values; add layout and mapping coverage.
Test Plan: git diff --cached --check; focused Rust mapping test added; full capped verification will run after the MT and dictionary workers are integrated.
Move stdin-sentinel classification and source stat/open result policy into the Rust fileio backend while keeping C responsible for diagnostics, binary-mode setup, and FILE ownership. The bridge leaves stat layout and native file utilities behind the existing C ABI and adds a focused no-stat stdin test.
Test Plan: git diff --cached --check; focused Rust test added but full capped verification will run after the remaining workers are integrated.
The decompression callback in fileio used to classify result statuses and
select its display action with a C switch. That left scalar result policy in
the C frontend even though Rust already owns the decompression dispatch and
result classification, and it coupled the C callback to a diagnostic enum.
Rust now exposes an ABI-checked status-action classifier. It keeps the
original action ordering and preserves silent handling for statuses that have
no display diagnostic, while the C callback retains the exact diagnostic
strings, source-name formatting, and display operation. Invalid inputs map to
a silent fallback action so the public callback remains behavior-compatible.
Focused Rust tests cover every status class, invalid values, and the exported
ABI result.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/fileio_asyncio.rs` -- passed
- Full capped Rust/native verification remains pending until the parallel
compression and decompression seams are integrated.
FIO_openDstFile retained the filesystem leaf in Rust but its C wrapper still encoded the status precedence for test mode, stdout, same-file protection, sparse-mode adjustment, overwrite prompting, removal, retry, and final errors. Centralize that pure action classification in Rust without moving FILE* handling, metadata, preference mutation, prompts, diagnostics, or retry callbacks across the ABI.\n\nThe action bridge validates status, confirmation, and sparse-state inputs and has focused tests for every destination outcome, sparse-first ordering, prompt acceptance/abort, quiet mode, overwrite mode, setvbuf/open failures, success, and invalid inputs.\n\nTest Plan:\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt/clippy gates passed before staging\n- git diff --cached --check\n- Full capped native and original-test verification follows after the batch is committed.
The file-I/O wrapper already delegated dictionary loading to Rust but kept
all status interpretation in C. That duplicated the malloc and mmap status
families and made the platform-specific error branches part of the C policy
surface. Add a Rust classifier that maps the shared numeric loader statuses
to diagnostic actions, including the distinct mmap failure classes. Keep
metadata lookup, platform handles, ownership, and the exact EXM_THROW text
in C, where the configured platform APIs still belong. The classifier also
rejects out-of-range type/status values; the valid malloc and mmap enums
intentionally share numeric values and therefore cannot be distinguished
beyond their family tag.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- passed (192 tests)
- Broader native and original-test verification remains pending for the complete batch.
Move the aggregate compression-result classification out of the C wrapper and into the Rust fileio module. Keep C responsible for the user-facing EXM_THROW diagnostics and the existing fallback assertion, so optional-format build guards and command-line behavior remain unchanged. The Rust classifier also makes unexpected statuses explicit without widening the format callback boundary.
Test Plan: ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check; GCC and Clang syntax-only checks; make -j1 -C tests test (all 41 shell tests, fuzzer, zstd tester, and zstream tester passed).
The decompression source-open callback previously both tested for a named
directory and returned the generic open failure. That left a source-resource
rejection in the C orchestration path even though Rust already owns the
per-file ordering.
Add a C-owned directory probe to the projection. Rust invokes it before source
opening for named inputs and rejects a positive result; the C callback retains
the private filesystem helper and exact directory diagnostic. Stdin bypasses
the probe as before, and optional codec callbacks plus the remaining source
open, asynchronous, and cleanup ordering are unchanged.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/fileio_asyncio.rs (passed)
- capped GCC syntax-only check of programs/fileio.c with all optional-format
macros enabled (passed)
- capped Clang syntax-only check of programs/fileio.c with all optional-format
macros enabled (passed)
- git diff --cached --check (passed)
- workspace cargo +nightly fmt --check was not clean because of an unrelated
rust/src/zstdmt_compress.rs formatting diff; that file was left untouched
- no cargo build/test, make, fuzzers, or upstream tests were run
- commit signing was disabled because the configured GPG prompt hung
The gzip codec loop already returns distinct status values, but the C
wrapper still owned the status branch that selected the CLI failure path.
That kept a format-result policy in the implementation-bearing C file even
though the loop itself is Rust. Add a Rust diagnostic mapping for success,
initialization, deflate, finish, end, invalid projection, and unknown
statuses. C now branches on the Rust classification while retaining zlib
result values, EXM_THROW codes, and exact diagnostic strings. This keeps the
ABI and observable behavior unchanged and leaves unrelated source-exclusion
and zstd classification seams untouched.
Test Plan:
- Nightly rustfmt check on fileio_asyncio.rs -- passed.
- Targeted git diff checks -- passed.
- Capped GCC syntax-only compile of fileio.c with codec defines -- passed.
- Focused Rust mapping tests were added; Cargo/tests were not run per the
task's OOM constraint.
The zstd compression stream already runs its scheduling and accounting loop in
Rust, but the C callback still owned the result-status switch that selected
success, codec failure, incomplete input, or invalid projection handling. Move
that status-to-diagnostic policy into the Rust ABI module, matching the
existing LZMA and LZ4 diagnostic seams. C keeps the zstd-specific error-name
lookup, display text, and exception construction, so private codec details and
CLI diagnostics remain on the C side and the observable error behavior is
unchanged. Unknown statuses retain the projection-error fallback.
Test Plan:
- `rustfmt +nightly --edition 2021 --check rust/src/fileio_asyncio.rs` -- passed.
- `git diff --check` and `git diff --cached --check` -- passed.
- Capped GCC syntax-only check of `programs/fileio.c` with all CLI format
feature defines -- passed.
- Cargo, make, native builds, and large tests were not run per worker OOM rules.
The Rust source-file scheduler already owns the ordering around source
exclusion, but its suffix policy and diagnostic still depended on a C table
and helper. That left the policy leaf on the C side of the boundary and
made the Rust scheduler call back into a C-owned decision.
Move the complete 113-entry, case-sensitive suffix policy into the Rust CLI
file-I/O layer and export the exclusion callback through the existing C ABI.
The callback preserves leading dots, the stdin and NULL non-match behavior,
the 0/1 return policy, and the exact display-level-4 diagnostic while leaving
C resource, compression, and asynchronous callbacks unchanged. Focused
checks cover representative suffixes, case sensitivity, stdin/NULL handling,
and the display gate.
Test Plan:
- `cc -fsyntax-only -Iprograms -Ilib -Ilib/common programs/fileio.c` -- passed
- Exact extracted C/Rust suffix-list comparison -- passed; all 113 entries match
- `git diff --check` and `git diff --cached --check` -- passed
- Rust unit tests and Cargo/Make/native suites were not run per request
- `rustfmt +nightly --check --edition 2021 rust/src/fileio_prefs.rs` -- not clean
because it reports pre-existing formatting drift in unchanged code
The Rust zstd stream callback initially called a display helper defined only in
programs/fileio.c. The CLI linked successfully, but the upstream C tests also
link the shared Rust archive without the CLI translation unit, leaving that
symbol unresolved even though those tests do not use the fileio projection.
Keep the diagnostic implementation in C, but pass it as an optional final field
of the zstd compression projection. Rust invokes it from the stream loop after
a successful codec call, using the same directive, input position, input size,
and produced-output count as the former C callback. Test projections can leave
the hook empty, so the reusable Rust archive has no dependency on CLI-only
symbols while the production CLI preserves its level-6 diagnostic.
Test Plan:
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy programs/fileio.c` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- `make -j1 -C tests test` reached the suite but failed before this fix on the
now-removed unresolved `FIO_rust_zstd_compressStreamDisplay` reference
The fileio zstd projection used to route its stream callback through a C
implementation that constructed the public input and output buffer views,
queried pending output, called the streaming codec, and copied four scalar
results back to the Rust-owned loop. That left the central codec operation in
the CLI C translation unit even though the surrounding stream loop was already
in Rust.
Move that callback into Rust while keeping the C CCtx opaque across the
boundary. Rust now builds the public ZSTD_inBuffer and ZSTD_outBuffer views,
invokes ZSTD_toFlushNow and ZSTD_compressStream2, publishes the original
positions and result, and preserves the existing success diagnostic through a
small C display callback. The C projection passes the real CCtx as codecOpaque
and retains the adaptive iteration context and all private CLI state. The
buffer structs are made public within the Rust crate so this seam can reuse the
existing C-compatible definitions without duplicating them.
Test Plan:
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy programs/fileio.c` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- Full capped native and Rust verification remains to be run after this seam
The separate-file decompression callback only reached the existing Rust
FIO_rust_determineDstName ABI through a two-argument C wrapper that supplied
the file-static suffix table and display string. Call the Rust ABI directly at
both decompression call sites, keeping the same suffix inputs and return-value
handling while removing the redundant wrapper and forward declaration.
Test Plan:
- `cc -fsyntax-only -Iprograms -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy programs/fileio.c` -- passed.
- `git diff --check` and `git diff --cached --check` -- passed.
- Cargo, Make, native tests, fuzzers, and other heavy verification were not run
per task constraints.
The --list stdin callback was a stateless string predicate: it ignored its
opaque context and compared the input name with the fixed stdin marker. Move
that callback under its existing caller symbol into the Rust CLI archive, so
C no longer owns this policy leaf. The Rust implementation preserves the
callback ABI and returns the same 1/0 result for the marker and ordinary
names. File opening, frame parsing, diagnostics, human-readable formatting,
private file-info storage, and list orchestration remain C-owned. The optional
codec-version helpers remain untouched because their build-time library
configuration is not propagated to the Rust archive.
Test Plan:
- `clang -fsyntax-only -Iprograms -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dictBuilder programs/fileio.c` -- passed.
- `rustfmt --edition 2021 --check --config skip_children=true rust/cli/src/lib.rs` -- passed.
- `git diff --check` and `git diff --cached --check` -- passed.
- Cargo, clippy, native builds, and upstream tests were not run per the explicit no-heavy-command constraint.
Call the Rust pass-through and frame-analysis leaves directly from their C
callbacks. The C translation unit still owns private resource projections,
filename suffix configuration, diagnostics, and format-specific operations;
this commit only removes two exact return-forwarding layers.
Test Plan:
- clang -fsyntax-only on programs/fileio.c
- git diff --check
- Full capped native/upstream suite pending after this commit
Keep the local buffer variables declared before the direct Rust buffer-policy
calls in the compression stream adapter, preserving the native C90 warning
profile after removing the old C forwarding helper.
Test Plan: `ulimit -v 41943040; make -j1 -C tests test` passed before this declaration-only cleanup; the next capped native gate will recheck it.
Rely on the existing fileio.h declaration now that the filename collision
checker is exported directly from Rust, avoiding a redundant C redeclaration
in the implementation file.
Test Plan: `ulimit -v 41943040; make -j1 -C tests test` passed; CLI lint and unit gates pending.
Move the pure filename-collision, input/output-buffer construction, and
compressed-destination-name helpers to direct Rust-owned caller symbols.
Leave C responsible for diagnostics, filesystem resources, suffix-list policy,
and the surrounding file-processing orchestration.
Test Plan: Pending capped full verification after this atomic ABI cleanup.
Export the file-summary predicates and largest-file-size scan directly from
Rust under their existing caller symbols. Remove the redundant C forwarding
wrappers while preserving pointer contracts, assertions, return widths, and
filesystem behavior.
Test Plan:
- worker capped format, C syntax, and diff checks
- parent capped root clippy and native build
- parent capped upstream make -j1 -C tests test
- parent capped CLI clippy and tests
Export FIO_LZ4_GetBlockSize_FromBlockId directly from Rust under the existing
caller symbol and remove the redundant C forwarding wrapper. Preserve the
block-ID formula and cover its boundary values in the focused Rust test.
Test Plan:
- worker capped nightly rustfmt check
- worker git diff --check
- full serial capped native and upstream suites to run at the next parent gate
Export FIO_highbit64 directly from Rust under the existing caller symbol and
remove the C forwarding wrapper. Preserve the nonzero-input contract and add
boundary tests for the scalar policy leaf.
Test Plan:
- capped nightly rustfmt check
- capped root clippy with all targets and -D warnings
- capped native make -j1
- capped upstream make -j1 -C tests test
- capped CLI clippy and 185 CLI tests
All integrated checks ran at the combined working-tree tip under a serial
40 GiB virtual-memory cap. Standalone root Rust unit linking remains
unavailable because the crate imports C-owned bridge symbols.
Move the malloc-versus-mmap decision used by compression and decompression
resource construction into Rust. The C adapter continues to own platform
loaders, allocation handles, diagnostics, and dictionary I/O; Rust only
combines the explicit mmap preference, patch-mode size threshold, and explicit
disable override. Add enum-value ABI checks and focused policy coverage.
Also hoist the source-size declaration in the C adapter so the migration does
not introduce a C90 declaration-after-statement warning.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings`
- `ulimit -v 41943040; make -j1`
- `ulimit -v 41943040; make -j1 -C tests test`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets`
The integrated checks ran at the combined working-tree tip under a serial
40 GiB virtual-memory limit. Standalone root Rust unit linking remains
unavailable because the crate imports C-owned bridge symbols without a Cargo
build/link setup.
Project the zstd window-too-large diagnostic inputs into Rust so Rust owns
error filtering and the concrete-versus-unsupported guidance choice. C keeps
frame-header parsing, window-log extraction, and the exact display text behind
a callback, preserving the existing diagnostics and fallback behavior.
All heavy verification was run serially with a 40 GiB virtual-memory cap and
one build job.
Test Plan:
- git diff --cached --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets (790 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (184 passed)
- ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; make -j1 (clean after C90 declaration cleanup)
- ulimit -v 41943040; make -j1 -C tests test (passed; the later cleanup only moved a declaration before statements)
Project the decompression-resource cleanup callbacks into Rust so the stable
teardown order remains dictionary, dstream, write pool, and read pool. C
retains ownership of dRess_t and each private resource implementation, with
its existing dstream CHECK behavior preserved. Focused tests cover order and
null or incomplete callback states.
All heavy verification was run serially with a 40 GiB virtual-memory cap and
one build job.
Test Plan:
- git diff --cached --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets (788 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (181 passed)
- ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
File-resource creation already delegates the general compression-parameter
policy to Rust, but the multithreaded parameters were still applied in C in a
separate branch. That left parameter ordering, optional overlap handling, and
error short-circuiting outside the Rust policy boundary. Add a narrow ABI
projection with C callbacks for CCtx mutation and diagnostics, and let Rust
apply worker count, job size, optional overlap, and rsyncable in the original
order. The C90 declaration layout and compile-time ABI assertions keep the
existing native program configurations intact.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo test --manifest-path rust/Cargo.toml --all-targets` -- 771 passed
- `cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- 179 passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `make -j1` under `ulimit -v 41943040` -- passed without the new C90 warning
- `make -j1 -C tests test` under `ulimit -v 41943040` -- passed
Move the non-threaded compression parameter sequence and adaptive window policy
out of FIO_createCResources. C continues to own context and resource
allocation, dictionary and pool setup, and the multithread-specific settings;
Rust drives the ordered parameter callbacks and stops on the first error.
The projection mirrors the C ABI and has focused order and short-circuit tests.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml create_c_resources --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
Keep fileio's compression-resource teardown order in the Rust policy layer:
dictionary, write pool, read pool, then compression context. The C adapter
keeps cRess_t and each private resource layout local while exposing only
one callback projection to Rust, preserving the existing cleanup behavior.
The Rust entry point also treats a null state as a no-op and verifies the
projection layout at compile time.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml free_c_resources --lib
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
Move stdin and empty-input validation, header selection, per-file iteration,
status aggregation, and multi-file total selection into the Rust CLI policy
layer. The callback projection keeps C responsible for its private file-info
storage, file parsing, diagnostics, human-readable formatting, and per-file
listing. Only successful and frame-error records contribute to the aggregate,
matching the original early-return behavior for invalid or truncated inputs.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/fileio_prefs.rs` -- passed.
- `git diff --cached --check` -- passed.
- Cargo, native, and full test commands were not run per the explicit no-heavy-command constraint.
Move the single-file compression destination lifecycle into the Rust file-I/O
policy layer while keeping C responsible for private resources, file handles,
diagnostics, metadata operations, and codec dispatch callbacks. Preserve the
original shared-destination fast path, temporary permissions, handler timing,
metadata-before-close ordering, close-error propagation, stdout cleanup guard,
and failed-artifact removal.
Test Plan:
- cargo fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- Full capped Rust, native, CLI, and upstream test suites to follow
Move the ordered policy from FIO_compressFilename_srcFile into a narrow Rust
projection: named-source stat and collision checks, compressed-file exclusion,
source open and close, size-based async selection, source attachment, and
successful-only --rm handling now run in Rust. The destination callback still
hands compression to the existing Rust compression loop.
Keep FIO_ctx_t, cRess_t, stat_t, FILE/pool handles, signal handling, and
CLI diagnostics behind opaque C callbacks, so the ABI carries only filenames,
scalar policy flags, and callback pointers. Focused Rust policy tests cover
named-file short circuits, stdin behavior, async threshold selection, and the
successful-compression removal gate.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/fileio_asyncio.rs` — passed
- `git diff --check` and path-limited `git diff --cached --check` — passed
- Static ABI/order/diff inspection only; policy tests were not executed
- cargo/make/native/fuzzer/heavy tests were not run per request
Move the separate-destination multi-file decompression loop into the Rust
projection so Rust owns file iteration, progress counters, and aggregate
error handling. Keep destination-name construction, mirror setup, source
opening, format dispatch, diagnostics, and source removal in the C callback
boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Move the shared-destination branch of FIO_decompressMultipleFilenames into a Rust-owned scheduler. Preserve source order, destination propagation, cursor advancement, success counting, aggregate errors, and C-owned source/format/resource behavior including --rm and output close handling.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml decompression_multiple_shared_destination -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Move the separate-destination branch of FIO_compressMultipleFilenames into a Rust-owned scheduler. Preserve source ordering, currFileIdx advancement, successful-file counting, aggregate error handling, and post-loop collision checks while retaining destination construction and private file/resource work in C callbacks.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml compression_multiple_separate -- --test-threads=1
- ulimit -v 41943040; make -B -C programs -j1 zstd
- ulimit -v 41943040; make -B -C tests -j1 test-cli-tests
Move the shared-output arm of FIO_compressMultipleFilenames into a Rust
scheduler. Rust now owns input-order iteration, processed-file accounting,
index advancement, and non-short-circuiting error aggregation while C retains
file validation, private resources, callbacks, diagnostics, and destination
lifetime.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- cargo test --manifest-path rust/Cargo.toml compression_multiple_shared_destination -- --test-threads=1 (2 passed)
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j1 lib
- make -B -C programs -j1 zstd
- make -B -C tests -j1 test-cli-tests (41 passed)
- all commands used CARGO_BUILD_JOBS=1 where applicable and
ulimit -v 41943040; no heavyweight jobs ran concurrently
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Move the scalar runtime --adapt predicates and compression-level normalization
into Rust. Rust now decides output blocking, output backlog, input starvation,
blocked-input speedups, and bounded slower/faster level changes through an
explicit projection.
Keep the CLI iteration order, refresh clock, frame-progression snapshots,
private FIO/ZSTD contexts, diagnostics, and ZSTD_CCtx_setParameter() mutation
in C. The new C/Rust layout assertions make the scalar policy ABI explicit
without exposing those private layouts.
Test Plan:
- focused adaptive-policy tests: 6 passed
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1:
533 passed
- cargo test --manifest-path rust/cli/Cargo.toml --all-targets --
--test-threads=1: 169 passed
- legacy compression/decompression/dictionary-builder feature matrix:
588 passed
- six library/CLI clippy gates with -D warnings
- capped serial native library/program rebuilds, 41 CLI tests, Rust library
smoke, and full test-zstd round trips including --adapt cases
- capped serial stress gates: 278 fuzzer cases, 84+129+143 zstream cases,
and 1,601 decode-corpus cases
- every heavyweight command used CARGO_BUILD_JOBS=1 or make -j1 and
ulimit -v 41943040; no worker/native process remained afterward
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
ddidderr