`make all` exercises substantially more than the default zstd binary. It also
builds compression-only and decompression-only archives, older contrib tools
that compile program C shims directly, and the single-file amalgamations. The
Rust migration had changed symbol ownership without updating every one of
those feature and link boundaries.
The first failure came from `fileio_asyncio`: it is always compiled, but its
codec bindings unconditionally referenced both `zstd_compress` and
`zstd_decompress`. A compression-only archive therefore required disabled
decoder modules, and the decompression-only configuration had the symmetric
problem. Gate the concrete codec imports, callback types, helpers, and exports
with their Cargo features. Keep the format probe compilable without the
legacy decoder predicate when decompression is disabled.
Once that boundary compiled, the remaining `make all` paths exposed related
integration gaps. Move the C decompression projection declarations out of the
compression preprocessor block, while leaving destination callback types
shared. Link the Rust CLI helpers archive into zlibWrapper, pzstd, and
largeNbDicts, whose util/time/data-generator C files are now declaration shims
rather than implementations.
The generated single-file C sources also call Rust-owned symbols now. Build
and link an appropriately featured Rust archive in their native smoke tests,
and include the pool configuration shim in the decoder case. The full
amalgamation combines `zstd_lazy.c` and `zstd_opt.c` into one translation unit,
so guard their otherwise translation-unit-local dictionary mode enum against
duplicate definition.
A Rust-backed amalgamation is no longer a standalone Emscripten input. Remove
the obsolete emcc/Docker path and report that limitation explicitly; restoring
the WebAssembly smoke test requires a Rust WebAssembly archive and a defined
cross-language amalgamation contract.
Test Plan:
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features compression` -- passed
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features decompression` -- passed
- `sh -n build/single_file_libs/build_decoder_test.sh build/single_file_libs/build_library_test.sh` -- passed
- `make all` -- passed, including native single-file and seekable-format tests
- `git diff --cached --check` -- passed
The multi-file compression and decompression entry points already delegate
file iteration and resource-sensitive work to Rust, but each still kept the
final summary decision and progress-to-summary ordering in C. Move that
scalar policy into a Rust ABI function. The C side now supplies only exact
legacy display callbacks and unchanged format strings, while Rust receives
plain counters and a summary kind without exposing FIO, cRess, dRess, or
codec layouts. Focused unit tests cover both summary modes, callback order,
argument forwarding, and sessions that must not display a summary.
Test Plan:
- `git diff --check -- programs/fileio.c rust/src/fileio_asyncio.rs` -- passed
- `git diff --cached --check` -- passed
- `rustfmt --check --edition 2021 rust/src/fileio_asyncio.rs` -- parsed the file;
reports pre-existing formatting differences elsewhere in the file, so no
unrelated formatting was applied
- Cargo, make, and runtime tests were intentionally not run per worker scope
Move the fileio source-size precedence policy into the Rust backend while
keeping the private CCtx pledge call, error handling, and diagnostics in the
C adapter. A known statted source size, including zero, remains authoritative;
when it is unavailable, a positive declared stream size is used, otherwise the
zstd unknown-content sentinel is preserved. The Rust policy accepts both
sentinels explicitly so the bridge does not couple their representations.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; make -j1 -C tests test
The adaptive compression loop still delegated its refresh clock gate to a
small C callback even though Rust already owned the iteration state and all
adaptive policy decisions. That left timing policy, the last-refresh scalar,
and one projection callback in the C-side orchestration boundary.
Move the one-sixth-second monotonic refresh gate into ZstdAdaptiveState. The
Rust loop now initializes the first refresh timestamp at frame start and
preserves the C callback's strict-greater-than interval check. C retains only
the private progression, parameter-setting, and diagnostic callbacks needed
by the existing file I/O context.
Shrink both sides of the projection together and keep compile-time offset and
size assertions aligned. The adaptive unit fixture now verifies Rust records
the refresh event while continuing to exercise the existing progression and
policy callbacks.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests invalidDictionaries
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
The default zstd frame loop was already implemented in Rust, but its
completion and error-result policy still lived in the C callback wrapper.
That left the C translation unit deciding when a decoded size was valid and
when a decoding or premature-end status had to become the historical frame
sentinel. Keep the exact operator-facing diagnostics and the private
asynchronous-resource pointers in C, while giving Rust ownership of the
status-to-result policy and callback ordering.
The new C/Rust policy record has compile-time layout assertions on both sides,
and the Rust fixture covers success, decode error, premature end, sentinel
results, and callback sequencing. This keeps the bridge explicit without
making Rust depend on the private dRess_t layout.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo check --manifest-path rust/Cargo.toml --tests
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
- git diff --check
Move the per-file and multi-file --list row formatters out of programs/fileio.c. C still owns filesystem access, frame analysis, private fileInfo_t storage, and exact status diagnostics; Rust now formats the rows into byte buffers and sends them through a synchronous output callback. This keeps filenames byte-preserving and prevents the private C record from crossing the ABI while preserving the original display thresholds, human-readable size policy, unavailable-content spacing, checksum order, and total-row behavior.
Replace the old display callbacks with one size-delimited output callback shared by the single-file and multi-file projections. Add explicit Rust ABI layout checks, formatter fixtures for normal/unavailable/verbose/checksum/total output, and callback-order assertions for the policy boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (208 passed)
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test (all original tests completed successfully)
- git diff --check
Move the decompression-resource state declaration ahead of the first
statement in FIO_createDResources. The initializer only captures the
resource address, so moving memset after the declaration preserves the
runtime order while avoiding the ISO C90 mixed-declaration warning.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 (passed after cleanup)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1 -C tests test (passed before this declaration-only cleanup)
- git diff --check (passed)
Move the decompression resource creation order into the Rust CLI policy
layer: dictionary stat and patch-memory preparation, decoder allocation,
window/checksum configuration, dictionary reset/attachment, and asynchronous
pool creation now run through one Rust-owned sequence. Keep dRess_t,
stat_t, dictionary buffers, decoder context, pools, and exact allocation
errors in C callbacks.
Preserve the original patch-from and dictionary-reference behavior while
adding ABI layout assertions, scalar policy coverage, and error short-circuit
tests for the new boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (207 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Make the Rust policy layer own the ordered compression-resource lifecycle:
create the CCtx, prepare patch/dictionary state, create the write and read
pools, validate the dictionary, apply general and multithreaded parameters,
and finally load the dictionary. Keep cRess_t, FIO_Dict_t, file statistics,
AIO pools, and CLI diagnostics in C callbacks so the existing resource
ownership and error behavior remain local to the C backend.
Preserve adaptive window defaults and patch-from parameter adjustment while
adding ABI layout assertions and a lifecycle-order test for the new boundary.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo +nightly fmt --manifest-path rust/cli/Cargo.toml --all
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (205 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
FIO_openDstFile() still owned the complete destination policy loop in C even
though the filesystem opener and status/action classifier were already Rust
leaves. That left confirmation, sparse-mode adjustment, overwrite removal, and
retry ordering duplicated beside private FILE* and CLI state.
Project the C-owned callbacks for diagnostics, preference mutation, user
confirmation, stdout setup, and existing-file removal. Rust now owns the
retry/order state machine and calls the existing narrow opener for each
attempt; C keeps private preferences and context, exact diagnostics, and FILE*
ownership. The callback layout is asserted on both sides, and focused tests
cover status ordering, confirmation/removal retry, setvbuf preservation, and
invalid policy inputs.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1` -- passed, including the single-thread library, MT library, and CLI binary.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- passed: 206 tests.
- `git diff --cached --check` -- passed.
The zstd file-I/O loop still kept adaptive compression's state machine in C: progression deltas, refresh gating, job-completion checks, input counters, speed decisions, and level updates were interleaved with private FIO and ZSTD state. That left orchestration policy behind the existing scalar Rust predicates.
Move the adaptive state and callback order into Rust. C now supplies scalar progression snapshots, the clock gate, exact diagnostics, and the ignored CCtx parameter setter through callbacks; private FIO_prefs_t, ZSTD_CCtx, ZSTD_frameProgression, clocks, and progress formatting remain C-owned. Preserve the previous-progression publication before backlog evaluation, input counter reset points, and serial/MT level-clamp behavior.
Test Plan:
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- git diff --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -j1
Move the multi-input single-output decision sequence out of
FIO_multiFilesConcatWarning while preserving the C-owned CLI boundary. The
previous wrapper classified fatal remove cases, emitted the concatenation
warning, disabled --rm, reclassified the action, and then selected quiet
abort or confirmation in C. Add a repr(C) callback projection so Rust owns
only that scalar ordering while C continues to own exact diagnostics, the
confirmation prompt, FIO_prefs_t mutation, and all private state.
The Rust bridge re-runs the action after the C disable-remove callback with
the same has-output/remove arguments as the original wrapper. C and Rust
assert the callback layout, and focused tests cover callback order,
fatal/quiet paths, and ABI offsets. Existing scalar action tests remain in
place.
Test Plan:
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 make -B -C programs -j1 fileio.o`
— passed; only pre-existing suffixList C++-compat warnings appeared.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path
rust/cli/Cargo.toml --tests --no-deps` — passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 RUSTFLAGS='-C
link-arg=/tmp/zstd_rust_test_bridges.o' cargo test --manifest-path
rust/cli/Cargo.toml --lib 'fileio_prefs::tests::multi_files_concat_'`
— 8 passed.
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo build --manifest-path
rust/cli/Cargo.toml --lib` — passed; the archive exports
`FIO_rust_multiFilesConcatWarning`.
- `cargo +nightly fmt --manifest-path rust/cli/Cargo.toml -- --check` — not
clean due pre-existing formatting drift in unchanged Rust code; no bulk
formatting was applied.
- Full native/upstream/fuzzer suites were not run by request.
Move the single-file --list status gates, diagnostic-versus-metadata ordering, and aggregate projection into Rust. C retains file opening and frame analysis, the private fileInfo_t layout, exact diagnostics, and row formatting behind explicit callbacks, so the user-visible behavior remains unchanged while the policy boundary is auditable.
Test Plan:
- git diff --cached --check
- worker format, check, clippy, and serial C compilation (passed); focused Rust tests compiled but the standalone link hit the pre-existing C bridge-symbol gap
Keep the private stat_t probe, destination opening, metadata syscalls, and format callbacks in C, but route the regular-source plus stdin/stdout exception policy through the Rust fileio preference layer. The Rust ABI bridge normalizes the scalar consumed by the existing destination lifecycle, with compile-time value assertions and focused tests covering regular, non-regular, stdin, stdout, and nonzero scalar inputs.
Test Plan: Not run by request; cargo, make, native tests, and heavy commands were intentionally avoided. Lightweight git diff --check passed.
Keep FIO_removeFile responsible for the filesystem operation, exact diagnostics, and C return wrapper, but route its status classification through a Rust policy bridge. Unknown statuses are explicitly treated as failed removal. ABI value assertions and focused mapping tests preserve the C contract.
Test Plan: Not run in this atomic commit; the capped serial Rust, native, smoke, and original test-suite verification follows.
Move the scalar InfoError-to-action mapping used by FIO_listFile into Rust and reuse the same classifier while aggregating multi-file list results. C retains file opening/parsing, exact diagnostics, metadata formatting, private fileInfo_t state, and the public result values; add layout and mapping coverage.
Test Plan: git diff --cached --check; focused Rust mapping test added; full capped verification will run after the MT and dictionary workers are integrated.
Move stdin-sentinel classification and source stat/open result policy into the Rust fileio backend while keeping C responsible for diagnostics, binary-mode setup, and FILE ownership. The bridge leaves stat layout and native file utilities behind the existing C ABI and adds a focused no-stat stdin test.
Test Plan: git diff --cached --check; focused Rust test added but full capped verification will run after the remaining workers are integrated.
The decompression callback in fileio used to classify result statuses and
select its display action with a C switch. That left scalar result policy in
the C frontend even though Rust already owns the decompression dispatch and
result classification, and it coupled the C callback to a diagnostic enum.
Rust now exposes an ABI-checked status-action classifier. It keeps the
original action ordering and preserves silent handling for statuses that have
no display diagnostic, while the C callback retains the exact diagnostic
strings, source-name formatting, and display operation. Invalid inputs map to
a silent fallback action so the public callback remains behavior-compatible.
Focused Rust tests cover every status class, invalid values, and the exported
ABI result.
Test Plan:
- `git diff --cached --check` -- passed
- `rustfmt --edition 2021 --check rust/src/fileio_asyncio.rs` -- passed
- Full capped Rust/native verification remains pending until the parallel
compression and decompression seams are integrated.
FIO_openDstFile retained the filesystem leaf in Rust but its C wrapper still encoded the status precedence for test mode, stdout, same-file protection, sparse-mode adjustment, overwrite prompting, removal, retry, and final errors. Centralize that pure action classification in Rust without moving FILE* handling, metadata, preference mutation, prompts, diagnostics, or retry callbacks across the ABI.\n\nThe action bridge validates status, confirmation, and sparse-state inputs and has focused tests for every destination outcome, sparse-first ordering, prompt acceptance/abort, quiet mode, overwrite mode, setvbuf/open failures, success, and invalid inputs.\n\nTest Plan:\n- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo fmt/clippy gates passed before staging\n- git diff --cached --check\n- Full capped native and original-test verification follows after the batch is committed.
The file-I/O wrapper already delegated dictionary loading to Rust but kept
all status interpretation in C. That duplicated the malloc and mmap status
families and made the platform-specific error branches part of the C policy
surface. Add a Rust classifier that maps the shared numeric loader statuses
to diagnostic actions, including the distinct mmap failure classes. Keep
metadata lookup, platform handles, ownership, and the exact EXM_THROW text
in C, where the configured platform APIs still belong. The classifier also
rejects out-of-range type/status values; the valid malloc and mmap enums
intentionally share numeric values and therefore cannot be distinguished
beyond their family tag.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- passed (192 tests)
- Broader native and original-test verification remains pending for the complete batch.
Move the aggregate compression-result classification out of the C wrapper and into the Rust fileio module. Keep C responsible for the user-facing EXM_THROW diagnostics and the existing fallback assertion, so optional-format build guards and command-line behavior remain unchanged. The Rust classifier also makes unexpected statuses explicit without widening the format callback boundary.
Test Plan: ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check; GCC and Clang syntax-only checks; make -j1 -C tests test (all 41 shell tests, fuzzer, zstd tester, and zstream tester passed).
The decompression source-open callback previously both tested for a named
directory and returned the generic open failure. That left a source-resource
rejection in the C orchestration path even though Rust already owns the
per-file ordering.
Add a C-owned directory probe to the projection. Rust invokes it before source
opening for named inputs and rejects a positive result; the C callback retains
the private filesystem helper and exact directory diagnostic. Stdin bypasses
the probe as before, and optional codec callbacks plus the remaining source
open, asynchronous, and cleanup ordering are unchanged.
Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/fileio_asyncio.rs (passed)
- capped GCC syntax-only check of programs/fileio.c with all optional-format
macros enabled (passed)
- capped Clang syntax-only check of programs/fileio.c with all optional-format
macros enabled (passed)
- git diff --cached --check (passed)
- workspace cargo +nightly fmt --check was not clean because of an unrelated
rust/src/zstdmt_compress.rs formatting diff; that file was left untouched
- no cargo build/test, make, fuzzers, or upstream tests were run
- commit signing was disabled because the configured GPG prompt hung
The gzip codec loop already returns distinct status values, but the C
wrapper still owned the status branch that selected the CLI failure path.
That kept a format-result policy in the implementation-bearing C file even
though the loop itself is Rust. Add a Rust diagnostic mapping for success,
initialization, deflate, finish, end, invalid projection, and unknown
statuses. C now branches on the Rust classification while retaining zlib
result values, EXM_THROW codes, and exact diagnostic strings. This keeps the
ABI and observable behavior unchanged and leaves unrelated source-exclusion
and zstd classification seams untouched.
Test Plan:
- Nightly rustfmt check on fileio_asyncio.rs -- passed.
- Targeted git diff checks -- passed.
- Capped GCC syntax-only compile of fileio.c with codec defines -- passed.
- Focused Rust mapping tests were added; Cargo/tests were not run per the
task's OOM constraint.
The zstd compression stream already runs its scheduling and accounting loop in
Rust, but the C callback still owned the result-status switch that selected
success, codec failure, incomplete input, or invalid projection handling. Move
that status-to-diagnostic policy into the Rust ABI module, matching the
existing LZMA and LZ4 diagnostic seams. C keeps the zstd-specific error-name
lookup, display text, and exception construction, so private codec details and
CLI diagnostics remain on the C side and the observable error behavior is
unchanged. Unknown statuses retain the projection-error fallback.
Test Plan:
- `rustfmt +nightly --edition 2021 --check rust/src/fileio_asyncio.rs` -- passed.
- `git diff --check` and `git diff --cached --check` -- passed.
- Capped GCC syntax-only check of `programs/fileio.c` with all CLI format
feature defines -- passed.
- Cargo, make, native builds, and large tests were not run per worker OOM rules.
The Rust source-file scheduler already owns the ordering around source
exclusion, but its suffix policy and diagnostic still depended on a C table
and helper. That left the policy leaf on the C side of the boundary and
made the Rust scheduler call back into a C-owned decision.
Move the complete 113-entry, case-sensitive suffix policy into the Rust CLI
file-I/O layer and export the exclusion callback through the existing C ABI.
The callback preserves leading dots, the stdin and NULL non-match behavior,
the 0/1 return policy, and the exact display-level-4 diagnostic while leaving
C resource, compression, and asynchronous callbacks unchanged. Focused
checks cover representative suffixes, case sensitivity, stdin/NULL handling,
and the display gate.
Test Plan:
- `cc -fsyntax-only -Iprograms -Ilib -Ilib/common programs/fileio.c` -- passed
- Exact extracted C/Rust suffix-list comparison -- passed; all 113 entries match
- `git diff --check` and `git diff --cached --check` -- passed
- Rust unit tests and Cargo/Make/native suites were not run per request
- `rustfmt +nightly --check --edition 2021 rust/src/fileio_prefs.rs` -- not clean
because it reports pre-existing formatting drift in unchanged code
The Rust zstd stream callback initially called a display helper defined only in
programs/fileio.c. The CLI linked successfully, but the upstream C tests also
link the shared Rust archive without the CLI translation unit, leaving that
symbol unresolved even though those tests do not use the fileio projection.
Keep the diagnostic implementation in C, but pass it as an optional final field
of the zstd compression projection. Rust invokes it from the stream loop after
a successful codec call, using the same directive, input position, input size,
and produced-output count as the former C callback. Test projections can leave
the hook empty, so the reusable Rust archive has no dependency on CLI-only
symbols while the production CLI preserves its level-6 diagnostic.
Test Plan:
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy programs/fileio.c` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- `make -j1 -C tests test` reached the suite but failed before this fix on the
now-removed unresolved `FIO_rust_zstd_compressStreamDisplay` reference
The fileio zstd projection used to route its stream callback through a C
implementation that constructed the public input and output buffer views,
queried pending output, called the streaming codec, and copied four scalar
results back to the Rust-owned loop. That left the central codec operation in
the CLI C translation unit even though the surrounding stream loop was already
in Rust.
Move that callback into Rust while keeping the C CCtx opaque across the
boundary. Rust now builds the public ZSTD_inBuffer and ZSTD_outBuffer views,
invokes ZSTD_toFlushNow and ZSTD_compressStream2, publishes the original
positions and result, and preserves the existing success diagnostic through a
small C display callback. The C projection passes the real CCtx as codecOpaque
and retains the adaptive iteration context and all private CLI state. The
buffer structs are made public within the Rust crate so this seam can reuse the
existing C-compatible definitions without duplicating them.
Test Plan:
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `cc -fsyntax-only -Werror=incompatible-pointer-types -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy programs/fileio.c` -- passed
- `git diff --check` and `git diff --cached --check` -- passed
- Full capped native and Rust verification remains to be run after this seam
The separate-file decompression callback only reached the existing Rust
FIO_rust_determineDstName ABI through a two-argument C wrapper that supplied
the file-static suffix table and display string. Call the Rust ABI directly at
both decompression call sites, keeping the same suffix inputs and return-value
handling while removing the redundant wrapper and forward declaration.
Test Plan:
- `cc -fsyntax-only -Iprograms -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dict -Ilib/legacy programs/fileio.c` -- passed.
- `git diff --check` and `git diff --cached --check` -- passed.
- Cargo, Make, native tests, fuzzers, and other heavy verification were not run
per task constraints.
The --list stdin callback was a stateless string predicate: it ignored its
opaque context and compared the input name with the fixed stdin marker. Move
that callback under its existing caller symbol into the Rust CLI archive, so
C no longer owns this policy leaf. The Rust implementation preserves the
callback ABI and returns the same 1/0 result for the marker and ordinary
names. File opening, frame parsing, diagnostics, human-readable formatting,
private file-info storage, and list orchestration remain C-owned. The optional
codec-version helpers remain untouched because their build-time library
configuration is not propagated to the Rust archive.
Test Plan:
- `clang -fsyntax-only -Iprograms -Ilib -Ilib/common -Ilib/compress -Ilib/decompress -Ilib/dictBuilder programs/fileio.c` -- passed.
- `rustfmt --edition 2021 --check --config skip_children=true rust/cli/src/lib.rs` -- passed.
- `git diff --check` and `git diff --cached --check` -- passed.
- Cargo, clippy, native builds, and upstream tests were not run per the explicit no-heavy-command constraint.
Call the Rust pass-through and frame-analysis leaves directly from their C
callbacks. The C translation unit still owns private resource projections,
filename suffix configuration, diagnostics, and format-specific operations;
this commit only removes two exact return-forwarding layers.
Test Plan:
- clang -fsyntax-only on programs/fileio.c
- git diff --check
- Full capped native/upstream suite pending after this commit
Keep the local buffer variables declared before the direct Rust buffer-policy
calls in the compression stream adapter, preserving the native C90 warning
profile after removing the old C forwarding helper.
Test Plan: `ulimit -v 41943040; make -j1 -C tests test` passed before this declaration-only cleanup; the next capped native gate will recheck it.
Rely on the existing fileio.h declaration now that the filename collision
checker is exported directly from Rust, avoiding a redundant C redeclaration
in the implementation file.
Test Plan: `ulimit -v 41943040; make -j1 -C tests test` passed; CLI lint and unit gates pending.
Move the pure filename-collision, input/output-buffer construction, and
compressed-destination-name helpers to direct Rust-owned caller symbols.
Leave C responsible for diagnostics, filesystem resources, suffix-list policy,
and the surrounding file-processing orchestration.
Test Plan: Pending capped full verification after this atomic ABI cleanup.
Export the file-summary predicates and largest-file-size scan directly from
Rust under their existing caller symbols. Remove the redundant C forwarding
wrappers while preserving pointer contracts, assertions, return widths, and
filesystem behavior.
Test Plan:
- worker capped format, C syntax, and diff checks
- parent capped root clippy and native build
- parent capped upstream make -j1 -C tests test
- parent capped CLI clippy and tests
Export FIO_LZ4_GetBlockSize_FromBlockId directly from Rust under the existing
caller symbol and remove the redundant C forwarding wrapper. Preserve the
block-ID formula and cover its boundary values in the focused Rust test.
Test Plan:
- worker capped nightly rustfmt check
- worker git diff --check
- full serial capped native and upstream suites to run at the next parent gate
Export FIO_highbit64 directly from Rust under the existing caller symbol and
remove the C forwarding wrapper. Preserve the nonzero-input contract and add
boundary tests for the scalar policy leaf.
Test Plan:
- capped nightly rustfmt check
- capped root clippy with all targets and -D warnings
- capped native make -j1
- capped upstream make -j1 -C tests test
- capped CLI clippy and 185 CLI tests
All integrated checks ran at the combined working-tree tip under a serial
40 GiB virtual-memory cap. Standalone root Rust unit linking remains
unavailable because the crate imports C-owned bridge symbols.
Move the malloc-versus-mmap decision used by compression and decompression
resource construction into Rust. The C adapter continues to own platform
loaders, allocation handles, diagnostics, and dictionary I/O; Rust only
combines the explicit mmap preference, patch-mode size threshold, and explicit
disable override. Add enum-value ABI checks and focused policy coverage.
Also hoist the source-size declaration in the C adapter so the migration does
not introduce a C90 declaration-after-statement warning.
Test Plan:
- `ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings`
- `ulimit -v 41943040; make -j1`
- `ulimit -v 41943040; make -j1 -C tests test`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings`
- `ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets`
The integrated checks ran at the combined working-tree tip under a serial
40 GiB virtual-memory limit. Standalone root Rust unit linking remains
unavailable because the crate imports C-owned bridge symbols without a Cargo
build/link setup.
Project the zstd window-too-large diagnostic inputs into Rust so Rust owns
error filtering and the concrete-versus-unsupported guidance choice. C keeps
frame-header parsing, window-log extraction, and the exact display text behind
a callback, preserving the existing diagnostics and fallback behavior.
All heavy verification was run serially with a 40 GiB virtual-memory cap and
one build job.
Test Plan:
- git diff --cached --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets (790 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (184 passed)
- ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; make -j1 (clean after C90 declaration cleanup)
- ulimit -v 41943040; make -j1 -C tests test (passed; the later cleanup only moved a declaration before statements)
Project the decompression-resource cleanup callbacks into Rust so the stable
teardown order remains dictionary, dstream, write pool, and read pool. C
retains ownership of dRess_t and each private resource implementation, with
its existing dstream CHECK behavior preserved. Focused tests cover order and
null or incomplete callback states.
All heavy verification was run serially with a 40 GiB virtual-memory cap and
one build job.
Test Plan:
- git diff --cached --check
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/Cargo.toml --all-targets (788 passed)
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets (181 passed)
- ulimit -v 41943040; cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; make -j1 -C tests test
File-resource creation already delegates the general compression-parameter
policy to Rust, but the multithreaded parameters were still applied in C in a
separate branch. That left parameter ordering, optional overlap handling, and
error short-circuiting outside the Rust policy boundary. Add a narrow ABI
projection with C callbacks for CCtx mutation and diagnostics, and let Rust
apply worker count, job size, optional overlap, and rsyncable in the original
order. The C90 declaration layout and compile-time ABI assertions keep the
existing native program configurations intact.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings` -- passed
- `cargo test --manifest-path rust/Cargo.toml --all-targets` -- 771 passed
- `cargo test --manifest-path rust/cli/Cargo.toml --all-targets` -- 179 passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check` -- passed
- `make -j1` under `ulimit -v 41943040` -- passed without the new C90 warning
- `make -j1 -C tests test` under `ulimit -v 41943040` -- passed
Move the non-threaded compression parameter sequence and adaptive window policy
out of FIO_createCResources. C continues to own context and resource
allocation, dictionary and pool setup, and the multithread-specific settings;
Rust drives the ordered parameter callbacks and stops on the first error.
The projection mirrors the C ABI and has focused order and short-circuit tests.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml create_c_resources --lib
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
- ulimit -v 41943040; make -j1
Keep fileio's compression-resource teardown order in the Rust policy layer:
dictionary, write pool, read pool, then compression context. The C adapter
keeps cRess_t and each private resource layout local while exposing only
one callback projection to Rust, preserving the existing cleanup behavior.
The Rust entry point also treats a null state as a no-op and verifies the
projection layout at compile time.
Test Plan:
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml free_c_resources --lib
- ulimit -v 41943040; make -j1
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo test --manifest-path rust/cli/Cargo.toml --all-targets
- ulimit -v 41943040; CARGO_BUILD_JOBS=1 cargo clippy --manifest-path rust/cli/Cargo.toml --all-targets -- -D warnings
Move stdin and empty-input validation, header selection, per-file iteration,
status aggregation, and multi-file total selection into the Rust CLI policy
layer. The callback projection keeps C responsible for its private file-info
storage, file parsing, diagnostics, human-readable formatting, and per-file
listing. Only successful and frame-error records contribute to the aggregate,
matching the original early-return behavior for invalid or truncated inputs.
Test Plan:
- `rustfmt --edition 2021 --check rust/src/fileio_prefs.rs` -- passed.
- `git diff --cached --check` -- passed.
- Cargo, native, and full test commands were not run per the explicit no-heavy-command constraint.
ddidderr