Continue the incremental zstd_compress.c migration with its frame
serialization leaves: ZSTD_writeFrameHeader(), the public
ZSTD_writeSkippableFrame() ABI, and ZSTD_writeLastEmptyBlock() now live
in rust/src/zstd_compress_frame.rs.
ZSTD_writeFrameHeader() reads five fields out of ZSTD_CCtx_params, whose
layout is private to zstd_compress.c and sensitive to build
configuration. Rather than mirror that structure in Rust, the C
function keeps its original static signature and forwards the scalar
fields to ZSTD_rust_writeFrameHeader(), so no parameter-structure layout
crosses the language boundary. The other two functions take only
pointer/size arguments and are exported directly, replacing their C
bodies outright.
Behavior differences are limited to hardening in release builds: the
Rust leaf clamps a window-size shift that C would leave undefined for a
malformed windowLog, and ZSTD_writeSkippableFrame() rejects a srcSize +
header overflow instead of comparing against a wrapped sum. Both paths
are unreachable through validated callers, so compressed output is
byte-identical.
Remaining zstd_compress.c work: parameter selection and validation,
context lifecycle, block dispatch, dictionary loading, and the streaming
state machine.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (118 tests, includes new frame
header/skippable/last-block reference vectors)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke
Move long-distance matching and high-level decompression from C shims into
Rust. The decoder now owns context, dictionary, parameter, one-shot, and
buffered streaming state while C retains allocation/configuration, legacy,
and trace leaves.
Move CLI parsing, safety policy, and dispatch into a separate Rust static
archive. Keeping it separate prevents library builds from retaining FIO
symbols, while C continues to own file opening, replacement, and I/O.
Program targets now select matching compression/decompression archives.
The remaining C boundary is intentional: high-level compression, optimal
parsing, dictionary building, legacy callbacks, and CLI file I/O still need
migration.
Test Plan:
- cargo test --all-targets (native and i686)
- cargo test --all-targets in rust/cli (native and i686)
- CLI crate compression-only and decompression-only feature tests
- native and i686 fuzzer/zstreamtest runs, plus legacy and dictionary tests
- ZSTD_C_PREDICT and ZSTD_HEAPMODE=0 fuzzer coverage
- library, dynamic-link, and program-target build/round-trip matrix
Refs: rust/README.md
Move the binary-tree table maintenance shared by optimal compression strategies
into Rust. A small C projection preserves the private match-state boundary;
the dynamic-programming parser and match selection remain C-owned for now.
The Rust implementation covers prefix and external-dictionary tree updates and
preserves the optional C predictor path. The component map now distinguishes
this migrated tree layer from the remaining optimal parser.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- make -B -C tests fuzzer32 MOREFLAGS=-DZSTD_C_PREDICT
- ./tests/fuzzer32 -s5346 -i1 --no-big-tests
- byte-compare C and Rust-control CLI output at levels 13, 16, 19, and 22
for random, patterned, zero, and dictionary inputs with and without
ZSTD_C_PREDICT
Refs: rust/README.md
Move the two-table fast match finder into Rust while retaining a small C
projection of the private match-state. The Rust implementation handles normal,
attached-dictionary, and external-dictionary matching without exposing the
full C context ABI. The C entry points and build-time exclusion guards remain
unchanged for configured consumers.
Document the migrated matcher in the Rust component map and narrow the
remaining-C boundary accordingly.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- timeout 20s stdbuf -oL ./tests/poolTests
Refs: rust/README.md
Move the context-free public ZSTD_compressBound ABI into Rust while leaving
the stateful compressor context in C. The Rust implementation preserves the
size-width-specific input limit, macro arithmetic, and srcSize_wrong error
contract used by C callers.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_compress_api::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_compress_api::tests
- make -B -C lib -j2 libzstd.a
- fuzzer -s5346 -i1 --no-big-tests
Refs: public ZSTD_COMPRESSBOUND macro in lib/zstd.h
Move fast hash-table filling and no-dictionary, external-dictionary, and
attached-CDict match loops into Rust. The C shim keeps `ZSTD_MatchState_t`
opaque, extracts only its needed fields, and asserts the mirrored SeqStore
leaf layout so existing compression dispatch remains ABI-compatible.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_fast::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_fast::tests -- --nocapture
- byte-identical C-control frames for dictionary and streaming variants
- fuzzer -s2066 -i5 --no-big-tests
- invalidDictionaries and zstreamtest -i1
Refs: Rust superblock port fde1d70c
Move subblock partitioning, literal and sequence section emission, entropy
fallback, and repcode repair into Rust. Keep a narrow C wrapper to read the
opaque compression context and invoke the existing C entropy-stat builder;
the Rust side owns only verified C-shaped leaf layouts.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_compress_superblock::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_compress_superblock::tests
- byte-exact C control across 16 target-size and input-shape cases
- fuzzer, zstreamtest, invalidDictionaries, and bounded native test matrix
Refs: Rust sequence entropy port 887af204
Move FSE table-costing, table construction, encoding selection, and sequence
bitstream emission into Rust while retaining the existing C internal-header
boundary. The shim preserves all five C ABI entry points, so the surrounding
compressor continues to consume C-shaped sequence and entropy-table storage.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu zstd_compress_sequences::tests
- C-vs-Rust differential across 163 payload cases on x86_64 and i686
- fuzzer, zstreamtest, and invalidDictionaries
Refs: Rust FSE compression port 5f9d607d
Move raw, RLE, and Huffman literal-section encoding into the Rust
compatibility archive. The surrounding C block compressor continues to pass
its existing entropy workspace and Huffman-table state through the unchanged
internal ABI.
The port preserves literal header choices, compression-gain decisions,
repeat-table transitions, and 1X/4X encoder selection. It lets ordinary C
compression paths exercise Rust code without widening this commit into the
remaining frame compressor.
Test Plan:
- cargo clippy, cargo clippy --benches, cargo clippy --tests, and nightly fmt
- cargo test --all-targets (94 passed) and feature-only cargo checks
- native fuzzer, zstreamtest, invalidDictionaries, and CLI round trip
- compare literals and next-table bytes with pristine C on x86_64 and i686
Refs: rust/README.md compression-primitives map
Move Huffman table construction, table serialization, and one- and
four-stream payload encoding from huf_compress.c into the Rust compatibility
archive. The declaration-only C shim preserves the existing internal ABI,
so the remaining C compressor can call the migrated implementation unchanged.
The translation keeps CTable layouts, workspace checks, repeat-table
selection, and bitstream output compatible with the original encoder. This
lets native C consumers exercise the Rust implementation through libzstd.
Test Plan:
- cargo clippy, cargo clippy --benches, and cargo clippy --tests
- cargo +nightly fmt and cargo test --all-targets (88 passed)
- cargo check --no-default-features --features compression
- rebuild and run tests fuzzer, zstreamtest, and invalidDictionaries
- compare tables, headers, and 1X/4X streams with a renamed pristine C build
Refs: rust/README.md component map
Move ZSTD_splitBlock into Rust while retaining its caller-owned workspace
contract. The implementation preserves the C fingerprint sampling heuristic
and calls the migrated histogram primitive without adding a hot-path
allocation.
Reference-vector and randomized differential tests compare every split level
with the original C translation unit. The C source remains as a declaration
shim so existing source lists resolve the Rust ABI during the migration.
Test Plan:
- cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests
- cargo test --all-targets and cargo build --release
- cargo test/build --target i686-unknown-linux-gnu
- 264-case original-C/Rust differential harness
- C fuzzer, zstreamtest, and fuzzer32 smoke runs
Refs: rust/README.md
Move FSE normalization, count-header writing, compression-table construction,
and stream encoding from C to Rust. The original C source remains as a
header-only shim, preserving the public header configuration while native
consumers receive the Rust archive exports.
This completes both codec directions for FSE in the migration crate and lets
the unchanged C compatibility tests exercise the Rust encoder implementation.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets
- cargo build --release
- Compile the C shim with strict warnings enabled
- 1,000-case pristine-C differential for tables, headers, and payloads
- ./tests/fuzzer -v -T10s
- ./tests/decodecorpus -t -T5
Refs: rust/README.md
Move byte histogram counting into Rust for FSE and Huffman compression callers.
The implementation preserves the C workspace contract, checked alphabet path,
fast path, empty-input behavior, and C ABI while the original C source becomes
a header-only compatibility shim.
Focused tests cover count accumulation, invalid alphabets, workspace failures,
and striped large inputs. Higher-level compression remains C for now.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
- make -C tests fuzzer
- ./tests/fuzzer -i1 --no-big-tests
- compile hist.c with -Werror and -Wredundant-decls
Refs: rust/README.md
this is a prototype definition error:
`_mm_storeu_si128()` should accept a `void*` pointer,
since it explicitly states that it accepts unaligned addresses
yet requiring a `__m128i*` tells otherwise, and requires the compiler the enforce this alignment.
seems like a prototype interface error:
input parameter should have been `const void*`,
since the documentation is explicit that input doesn't have to be aligned,
but `const __m256i*` makes the compiler enforce it.
do not solve the equation, even though some members cancel each other,
this is done for clarity,
we'll let the compiler do the resolution at compile time.
ddidderr
Yann Collet