The compression-parameter override helper previously lived in C and selectively
copied each nonzero field from a caller-provided override structure. Move that
pure field-wise operation into the Rust compression-parameter module, while
keeping the existing repr(C) structure and pointer-based ABI at both C call
sites. Zero-valued overrides continue to leave the current parameter untouched.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression` -- passed (252 tests)
- `make -B -C lib -j2 lib` -- passed
- `make -C tests -j2 fuzzer` and `tests/fuzzer -s4560 -t47 -i48 -v` -- passed
- required clippy passes, nightly fmt, and `git diff --check` -- passed
Keep validation of the applied compression parameters in C and delegate the context-free minimum calculation through a narrow ABI leaf. Preserve the public getter and deprecated block-compression paths.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression
- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --all-targets
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- make -B -C lib -j2 lib
- make -C tests -j2 fuzzer
- tests/fuzzer -s4560 -t47 -i48 -v
Keep C ownership of the block-state structure while delegating the confirmed-block pointer swap through a narrow two-pointer ABI leaf. Preserve every existing confirmation call site and add direct pointer-swap coverage.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression
- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --all-targets
- cargo +nightly fmt --manifest-path rust/Cargo.toml
- make -B -C lib -j2 lib
- tests/fuzzer -s4560 -t47 -i48 -v
- make -C tests -j2 test-zstream
Port the no-delimiter external-sequence block scanner, match splitting, repcode updates, validation, and sequence/literal storage behind a narrow C shim. Preserve trailing bytes as literals when the sequence buffer ends before the requested block, and remove the obsolete C validation wrapper.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (242 passed)
- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression [--benches|--tests] (clean)
- cargo +nightly fmt --manifest-path rust/Cargo.toml (clean)
- make -B -C lib -j2 lib (passed)
- make -C tests -j2 test-zstream (84 deterministic; 5960 and 8307 randomized passed)
- make -C tests -j2 test-fuzzer (existing test 56 CCtx-reuse mismatch; also fails at 37eb75758)
- make -C tests -j2 test-zstd (existing Rust CLI --patch-from gap)
Port ZSTD_dictNCountRepeat and ZSTD_loadCEntropy to a dedicated Rust compression module. Preserve the existing C-facing ZSTD_loadCEntropy symbol through a narrow shim while keeping dictionary content and compression-context ownership in C.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression
- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --benches --tests
- make -B -C lib -j2 lib
- make -C tests -j2 test-invalidDictionaries
- make -C tests -j2 test-zstream
Move ZSTD_transferSequences_wBlockDelim's delimiter scan, literal copying, SeqDef storage, validation, and repcode bookkeeping into the Rust compression statistics module. Keep ZSTD_CCtx and block-state ownership in C through a narrow scalar/pointer shim, with ABI layout assertions for ZSTD_SequencePosition.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression
- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --benches --tests
- make -B -C lib -j2 lib
- make -C tests -j2 test-zstream
Block splitting still estimated literal and sequence section sizes through
three C helpers. Move the pure estimator into zstd_compress_stats.rs so its
histogram, HUF, and FSE cost calculations use the Rust implementations already
behind the compression ABI. Preserve the original literal headers, multi-stream
Huffman surcharge, sequence additional-bit costs, entropy-header accounting,
and fallback estimates; leave C responsible for split recursion and table
ownership.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression` -- 229 passed
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression` plus benches/tests -- passed before and after formatting
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `make -B -C lib -j2 lib` -- passed
- `make -C tests -j2 test-zstream` -- passed, including 84 deterministic and 15,375 randomized cases
The explicit-delimiter sequence path still calculated each block summary in C,
with a separate AVX2 implementation and scalar fallback. Move that pure scan
to zstd_compress_stats.rs and keep a C wrapper for the existing orchestration.
The Rust leaf preserves delimiter inclusion, literal and match totals, and the
external-sequences error when no delimiter is present, while the C ABI is pinned
with a repr(C) layout check.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression` -- 227 passed
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression` plus benches/tests -- passed before and after formatting
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `make -B -C lib -j2 lib` -- passed
- `make -C tests -j2 test-zstream` -- passed, including 84 deterministic and 15,464 randomized cases
The external-sequence block path still converted public ZSTD_Sequence values
in C, with separate scalar and AVX2 implementations. Move that pure format
conversion into zstd_compress_stats.rs so both native configurations share one
implementation. Preserve offBase encoding, u16 truncation, the rep-ignored
contract, and the long-length side-band marker; leave C responsible for block
state, repcode resolution, and long-length bookkeeping.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression` -- 225 passed
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression` plus benches/tests -- passed before and after formatting
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `make -B -C lib -j2 lib` -- passed
- `make -C tests -j2 test-zstream` -- passed, including 84 deterministic and 14,768 randomized cases
Port ZSTD_seqStore_resolveOffCodes and its repcode-resolution helper to Rust. The C shim keeps Repcodes_t and SeqStore_t ownership in the compressor context while Rust preserves long-literal handling, mismatch materialization, and independent decoder/compressor history updates.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (208 tests)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- make -B -C lib -j2 lib
Replace the shared C raw and one-byte RLE fallback block serializers with Rust ABI leaves. Preserve the zstd block headers, payload copies, capacity errors, and all existing C compressor dispatch and context ownership; share the raw serializer with the Rust one-shot path.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (208 tests)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- make -B -C lib -j2 lib
Port the scalar frame-finalization policy and byte serialization into the Rust compression leaf while keeping checksum finalization, tracing, and the compression context in C. Preserve the init-to-ongoing transition on partial failure and reset the stage only after a successful epilogue.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-zstream (84 deterministic, 6737 and 9012 fuzzer cases)
Port the external-sequence window, offset, and minimum-match checks through a scalar C ABI leaf while retaining the C caller's control flow and diagnostics.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 deterministic, 7063 and 8665 fuzzer cases).
Move ZSTD_isUpdateAuthorized behind the existing Rust parameter API while retaining the C context-stage checks, diagnostics, and mutation flow.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (187 tests)
- cargo clippy for library, benches, and tests
- make -B -C lib -j2 lib
Move hash-salt mixing and context-reset overflow predicates behind scalar Rust ABI shims. Keep the C-owned match state and window arithmetic in the C wrapper while preserving exact architecture-dependent thresholds.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (186 tests)
- cargo clippy for library, benches, tests, and CLI all-targets
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-zstream (84 deterministic tests; fuzzers 6451 and 9879 cases)
Move target block-size and block-splitter policy predicates behind narrow Rust ABI shims while retaining the C debug logging, finalized-mode assertion, and surrounding control flow.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression
- cargo clippy --manifest-path rust/Cargo.toml and native library build
- make -B -C tests -j2 test-zstream (84 deterministic tests; fuzzers 6778 and 8299 cases)
Move construction of the external-dictionary and active-prefix ranges into a narrow Rust ABI while retaining the C window and logging surface. Preserve byte-range half-open overlap semantics and leave the input-range overlap wrapper available to its other C caller.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (180 passed); root and CLI clippy; make -B -C lib -j2 lib; make -C tests -j2 test-cli-tests (41 passed); make -B -C tests -j2 test-zstream (84 named tests plus 6,845 and 9,628 fuzz cases passed).
Port the 16-cell match-table reduction leaf while preserving the window-index threshold, wrapping subtraction, and btlazy2 unsorted marker. Keep ZSTD_reduceIndex and its strategy-specific table selection in C behind the narrow Rust ABI.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (180 passed); root clippy, bench clippy, and test clippy; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 named tests plus 6,845 and 9,628 fuzz cases passed).
Port repcode offBase finalization and explicit-delimiter block sizing through the Rust compression ABI. Preserve wrapped sequence arithmetic, external-sequence error codes, and the no-delimiter target-size policy while keeping the C control flow intact.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (176 passed); cargo clippy root, benches, and tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 named tests plus 6,230 and 8,598 fuzz cases passed).
Move sequence-store byte accounting, chunk derivation, and repcode resolution into the Rust compression module. Move MT raw-sequence buffer conversions into Rust while preserving the existing C adapters and ABI layouts.
Test Plan:\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- make -B -C lib -j2 lib\n- make -B -C tests -j2 test-cli-tests
Route public CCtx parameter bounds through the Rust policy implementation while keeping build-specific MT limits supplied by the C configuration. Preserve clamping and validation behavior for migrated parameters and add focused Rust coverage.
Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-cli-tests; make -B -C tests -j2 test-zstream
Move ZSTD_CCtxParams_init_advanced and its default-resolution policy into the Rust parameter API while retaining the C public ABI wrapper. Preserve validation, zeroing, and the existing row-match, block-splitter, LDM, and repcode defaults.
Test Plan:\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_compress_params_api\n- make -B -C lib -j2 lib
Move CCtx_params allocation, initialization, and release behind Rust ABI helpers while preserving the public C wrappers and custom allocator contract. Keep parameter bounds and policy in the existing C implementation.
Test Plan:\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- cargo +nightly fmt --manifest-path rust/Cargo.toml --all\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression\n- make -B -C lib -j2 lib\n- make -B -C tests -j2 test-rust-lib-smoke\n- make -B -C tests -j2 test-zstream\n- make -B -C tests -j2 test-cli-tests
Match the original optimal-parser statistics for zero-possible symbols, complete the shortest-path fallback when the forward pass runs past its final position, and refresh the projected window state for the ultra2 second pass. Keep the simple Rust frame leaf restricted to the match finders it implements so lazy and optimal strategies retain their stateful lifecycle.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (135 passed)
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_opt (3 passed)
- make -B -C tests -j2 test-cli-tests (41 passed)
- make -B -C tests -j2 test-zstream (84 named tests and fuzzer suites passed)
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
Route ordinary complete-input ZSTD_compressStream2 calls through the Rust frame compressor while retaining the C state machine for partial, advanced, and dictionary-backed streams. Track explicit maxBlockSize requests so the migrated path preserves the original byte-identical default behavior, and extend the C archive smoke test across context reuse and fallback cases.
Test Plan:
- cargo clippy --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --benches
- cargo clippy --manifest-path rust/Cargo.toml --tests
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (133 passed)
- make -B -C tests -j2 test-rust-lib-smoke and ./tests/rustLibSmoke
- make -B -C tests -j2 test-zstream (passed, including both APIs and maxBlockSize case 78)
The multithreaded compressor already uses Rust-owned buffer and CCtx pools,
but zstdmt_compress.c still allocated and freed its job table directly. Move
the raw table storage and power-of-two sizing behind Rust's custom allocator
ABI. Keep descriptor field access and platform mutex/condition initialization
in C because those layouts remain private and platform-specific. The C
wrapper initializes and destroys synchronization primitives around the Rust
storage calls, preserving failure cleanup while leaving worker job setup,
scheduling, and stream entry points in C as the fallback implementation.
Keep the Rust storage-only ABI free of MT-only C references so single-threaded
archives can omit zstdmt_compress.c without acquiring new unresolved symbols.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression zstdmt_compress --lib` -- passed (5 tests).
- `make -B -C lib lib-mt` -- passed.
- `make -B -C tests -j2 fullbench zstreamtest poolTests` -- passed.
- `./poolTests` -- passed.
- `./fullbench -i1 -B1000 README.md` -- passed, including -T2 scenarios.
- Rebuilt `programs/zstd` and ran a `-T2` compress/decompress `cmp` round-trip
-- passed.
- `rustfmt` and `cargo +nightly fmt --manifest-path rust/Cargo.toml --all
-- --check` -- passed.
- Full `cargo clippy -D warnings` remains blocked by unrelated warnings in
the concurrent `rust/src/fileio_asyncio.rs` worktree changes.
- `./zstreamtest -T5s` reaches an unrelated single-thread maxBlockSize
assertion at `tests/zstreamtest.c:2157`; MT fullbench and CLI smoke pass.
Dispatch ordinary dictionary-free ZSTD_compress2 calls through the Rust frame
compressor while retaining the original C implementation for advanced
contexts. Prepare the C workspace first so allocation downsizing and
static-context limits remain observable.
Mark successful Rust frames so a following one-shot
ZSTD_compressStream2(..., ZSTD_e_end) call uses the same output path. Other
streaming directives and advanced parameters remain on the C implementation.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
rust/Cargo.toml --no-default-features --features compression
zstd_compress::tests
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
rust/Cargo.toml --no-default-features --features compression -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C tests fuzzer
- ./tests/fuzzer -t114 -i1000 --no-big-tests -s1
The basic fuzzer run now reaches the existing Rust one-shot compression-ratio
limit at test 113; the new stateful reuse tests pass before that point.
Move ZSTD_compressCCtx() onto the Rust frame-compression path while
retaining only the C-owned reset needed for its private context layout.
The simple API ignores advanced context parameters by contract, so the
Rust entry point can share the one-shot implementation after reset.
Preserve valid frames when the Rust superblock leaf declines a compressed
block by emitting a raw block instead. Match the fast-strategy literal
policy for negative levels so compatibility behavior remains intact.
Streaming, stateful ZSTD_compress2(), and multithreaded context
orchestration remain C-owned for a later slice. The full fuzzer currently
reaches an existing compress2() superblock-expansion failure after these
simple-API checks; that stateful path is intentionally out of scope here.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features
--features compression zstd_compress::tests
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
(library, benches, and tests)
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C programs zstd V=1
- make -B -C examples multiple_simple_compression
- ./multiple_simple_compression README.md ../rust/README.md
Move the reusable buffer and compression-context pools behind a Rust
implementation with a narrow C ABI adapter. The scheduler, job table, serial
LDM state, and stream orchestration remain in C until their private layouts
are ported.
Keep context types opaque across Rust modules so the pool bridge does not
depend on private C layout declarations. This also keeps the existing custom
allocator and pool replacement contracts intact.
Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
rust/Cargo.toml --all-targets -- -D warnings
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
rust/Cargo.toml zstdmt_compress
- make -B -C lib lib-mt
- git diff --cached --check
Move the binary-tree optimal and ultra block parsers into Rust with a narrow
C projection of match-state, sequence-store, and entropy-table fields. Keep
the established C entry points as wrappers, and avoid reading uninitialized
entropy costs during dictionary tree updates.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- make -B -C tests test-zstream V=1
- git diff --check
Refs: rust/src/zstd_opt.rs, lib/compress/zstd_opt.c
Implement ZSTD_compress in Rust using the migrated parameter, match-finder,
sequence, frame, and superblock leaves. Keep the private context and streaming
entry points C-backed until their configuration-dependent context projection is
ported, and remove only the duplicate C one-shot definition.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- make -B -C tests test-rust-lib-smoke V=1
- git diff --check
Refs: rust/src/zstd_compress.rs, lib/compress/zstd_compress.c
Continue the incremental zstd_compress.c migration with its sequence
statistics and seqStore entropy-compression layer. The following now
live in rust/src/zstd_compress_stats.rs:
- ZSTD_seqToCodes(), exported under its original name because the C
dictionary builder (zdict.c) and decodecorpus link against it,
- ZSTD_buildSequencesStatistics() and its dummy variant, whose result
struct no longer crosses the language boundary,
- ZSTD_entropyCompressSeqStore_internal(), _wExtLitBuffer(), and
ZSTD_entropyCompressSeqStore(),
- ZSTD_buildBlockEntropyStats() with its literals/sequences helpers,
- ZSTD_copyBlockSequences() and the ZSTD_updateRep() rules it shares
with the superblock writer.
Boundary: ZSTD_CCtx and ZSTD_CCtx_params stay private to C. These
paths read exactly two parameter fields, so the C shims keep the
original static/exported function names and forward the strategy and
ZSTD_literalsCompressionIsDisabled() as int scalars alongside the
seqStore and entropy-table leaves. The leaf layouts (SeqDef,
SeqStore_t, ZSTD_hufCTables_t, ZSTD_fseCTables_t, entropy metadata,
SeqCollector, ZSTD_Sequence) are pinned by compile-time asserts in
zstd_compress.c and by both-pointer-width layout tests in Rust.
ZSTD_buildSeqStore, block dispatch/splitting, and the block-size
estimation helpers remain C for a later slice.
The superblock module previously round-tripped through the C export of
ZSTD_buildBlockEntropyStats and mirrored the entropy leaf structs
privately. It now calls the crate-internal builder directly, and the
shared struct definitions moved to zstd_compress_stats; consequently
ZSTD_rust_compressSuperBlock() takes the two parameter scalars instead
of an opaque ZSTD_CCtx_params pointer, extracted by its C shim. Its
layout and repcode tests moved with the definitions.
One C helper family gets no shim: the static
ZSTD_entropyCompressSeqStore_wExtLitBuffer() had a single caller and
was folded into the Rust implementation.
Byte-identity was verified against the pre-change compressor: COPYING,
datagen -g5000000 -s7, and datagen -g300000 -s21 -P90, each at levels
1/3/9/19 and --fast=5, plus a superblock-heavy pass at level 19 with
--target-compressed-block-size=1024; all 18 frames are byte-identical,
covering the repeat-mode state machine, longOffsets, RLE/raw fallback,
and dstSize_tooSmall paths through the block splitter and superblock.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (131 tests: new reference
vectors for seqToCodes, RLE table headers, empty-seqStore repeat
copies, literal-stats type selection, and repcode resolution in
copyBlockSequences)
- cargo build --release --no-default-features --features compression
and --features decompression
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests (covers
ZSTD_generateSequences and ZDICT training over the Rust seqToCodes)
- make -C tests test-rust-lib-smoke
- ./tests/zstreamtest -i1 and ./tests/zstreamtest --newapi -t1 -i1
- ./tests/decodecorpus -t -T1s (decodecorpus links Rust seqToCodes)
- /tmp/ref vs /tmp/got frame diff as described above: byte-identical
Move the context-free compression-parameter logic of zstd_compress.c to
rust/src/zstd_compress_params.rs: the compression-level tables (formerly
clevels.h), parameter bounds/checking/clamping, cycle log, level-table
selection, source/dictionary parameter adjustment, default frame
parameters, and the match-state/CDict size estimators.
The boundary follows the module's design notes: Rust owns only leaves
whose behavior is independent of C preprocessor configuration. C keeps
the public ZSTD_* symbols and feeds the leaves everything that is
configuration-owned as explicit scalars:
- The ZSTD_EXCLUDE_*_BLOCK_COMPRESSOR strategy cascade stays in
ZSTD_adjustCParams_internal() ahead of the Rust adjustment leaf, so
reduced builds keep their fallback policy.
- Workspace estimation receives struct sizes (ZSTD_CDict, ZSTD_match_t,
ZSTD_optimal_t), HUF workspace size, and the sanitizer redzone size,
because those depend on private layouts and ASAN configuration.
- ZSTD_cParam_getBounds() forwards only the compression level and the
seven core parameters; all other parameter bounds remain C.
- Frozen private constants the leaves hardcode (short-cache and row-hash
tag widths, MaxML/MaxLL/MaxOff/Litbits, ZSTD_OPT_SIZE, cwksp alignment)
are pinned by ZSTD_STATIC_ASSERTs at the C call sites.
Two latent 32-bit bugs in the previously unwired module were fixed
before integration: dictAndWindowLog's max window size and the
window-resize threshold were hardcoded to the 64-bit constants
(1<<31, 1<<30) instead of deriving from ZSTD_WINDOWLOG_MAX, which is
30 on 32-bit targets.
clevels.h is no longer included anywhere but stays in-tree as the
reference for mechanical comparison against the Rust table.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (125 tests)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke
- Byte-identity: zstd CLI frames for COPYING, a 250 KB C source, and a
5 MB datagen sample at levels 1/3/9/19/--fast=5 are identical between
this change and its parent commit.
Continue the incremental zstd_compress.c migration with its frame
serialization leaves: ZSTD_writeFrameHeader(), the public
ZSTD_writeSkippableFrame() ABI, and ZSTD_writeLastEmptyBlock() now live
in rust/src/zstd_compress_frame.rs.
ZSTD_writeFrameHeader() reads five fields out of ZSTD_CCtx_params, whose
layout is private to zstd_compress.c and sensitive to build
configuration. Rather than mirror that structure in Rust, the C
function keeps its original static signature and forwards the scalar
fields to ZSTD_rust_writeFrameHeader(), so no parameter-structure layout
crosses the language boundary. The other two functions take only
pointer/size arguments and are exported directly, replacing their C
bodies outright.
Behavior differences are limited to hardening in release builds: the
Rust leaf clamps a window-size shift that C would leave undefined for a
malformed windowLog, and ZSTD_writeSkippableFrame() rejects a srcSize +
header overflow instead of comparing against a wrapped sum. Both paths
are unreachable through validated callers, so compressed output is
byte-identical.
Remaining zstd_compress.c work: parameter selection and validation,
context lifecycle, block dispatch, dictionary loading, and the streaming
state machine.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (118 tests, includes new frame
header/skippable/last-block reference vectors)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke
Move long-distance matching and high-level decompression from C shims into
Rust. The decoder now owns context, dictionary, parameter, one-shot, and
buffered streaming state while C retains allocation/configuration, legacy,
and trace leaves.
Move CLI parsing, safety policy, and dispatch into a separate Rust static
archive. Keeping it separate prevents library builds from retaining FIO
symbols, while C continues to own file opening, replacement, and I/O.
Program targets now select matching compression/decompression archives.
The remaining C boundary is intentional: high-level compression, optimal
parsing, dictionary building, legacy callbacks, and CLI file I/O still need
migration.
Test Plan:
- cargo test --all-targets (native and i686)
- cargo test --all-targets in rust/cli (native and i686)
- CLI crate compression-only and decompression-only feature tests
- native and i686 fuzzer/zstreamtest runs, plus legacy and dictionary tests
- ZSTD_C_PREDICT and ZSTD_HEAPMODE=0 fuzzer coverage
- library, dynamic-link, and program-target build/round-trip matrix
Refs: rust/README.md
Move the binary-tree table maintenance shared by optimal compression strategies
into Rust. A small C projection preserves the private match-state boundary;
the dynamic-programming parser and match selection remain C-owned for now.
The Rust implementation covers prefix and external-dictionary tree updates and
preserves the optional C predictor path. The component map now distinguishes
this migrated tree layer from the remaining optimal parser.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- make -B -C tests fuzzer32 MOREFLAGS=-DZSTD_C_PREDICT
- ./tests/fuzzer32 -s5346 -i1 --no-big-tests
- byte-compare C and Rust-control CLI output at levels 13, 16, 19, and 22
for random, patterned, zero, and dictionary inputs with and without
ZSTD_C_PREDICT
Refs: rust/README.md
Move the two-table fast match finder into Rust while retaining a small C
projection of the private match-state. The Rust implementation handles normal,
attached-dictionary, and external-dictionary matching without exposing the
full C context ABI. The C entry points and build-time exclusion guards remain
unchanged for configured consumers.
Document the migrated matcher in the Rust component map and narrow the
remaining-C boundary accordingly.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- timeout 20s stdbuf -oL ./tests/poolTests
Refs: rust/README.md
ddidderr