Move the two-table fast match finder into Rust while retaining a small C
projection of the private match-state. The Rust implementation handles normal,
attached-dictionary, and external-dictionary matching without exposing the
full C context ABI. The C entry points and build-time exclusion guards remain
unchanged for configured consumers.
Document the migrated matcher in the Rust component map and narrow the
remaining-C boundary accordingly.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- timeout 20s stdbuf -oL ./tests/poolTests
Refs: rust/README.md
Move the context-free public ZSTD_compressBound ABI into Rust while leaving
the stateful compressor context in C. The Rust implementation preserves the
size-width-specific input limit, macro arithmetic, and srcSize_wrong error
contract used by C callers.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_compress_api::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_compress_api::tests
- make -B -C lib -j2 libzstd.a
- fuzzer -s5346 -i1 --no-big-tests
Refs: public ZSTD_COMPRESSBOUND macro in lib/zstd.h
Move fast hash-table filling and no-dictionary, external-dictionary, and
attached-CDict match loops into Rust. The C shim keeps `ZSTD_MatchState_t`
opaque, extracts only its needed fields, and asserts the mirrored SeqStore
leaf layout so existing compression dispatch remains ABI-compatible.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_fast::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_fast::tests -- --nocapture
- byte-identical C-control frames for dictionary and streaming variants
- fuzzer -s2066 -i5 --no-big-tests
- invalidDictionaries and zstreamtest -i1
Refs: Rust superblock port fde1d70c
Move subblock partitioning, literal and sequence section emission, entropy
fallback, and repcode repair into Rust. Keep a narrow C wrapper to read the
opaque compression context and invoke the existing C entropy-stat builder;
the Rust side owns only verified C-shaped leaf layouts.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_compress_superblock::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_compress_superblock::tests
- byte-exact C control across 16 target-size and input-shape cases
- fuzzer, zstreamtest, invalidDictionaries, and bounded native test matrix
Refs: Rust sequence entropy port 887af204
Move FSE table-costing, table construction, encoding selection, and sequence
bitstream emission into Rust while retaining the existing C internal-header
boundary. The shim preserves all five C ABI entry points, so the surrounding
compressor continues to consume C-shaped sequence and entropy-table storage.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu zstd_compress_sequences::tests
- C-vs-Rust differential across 163 payload cases on x86_64 and i686
- fuzzer, zstreamtest, and invalidDictionaries
Refs: Rust FSE compression port 5f9d607d
Move raw, RLE, and Huffman literal-section encoding into the Rust
compatibility archive. The surrounding C block compressor continues to pass
its existing entropy workspace and Huffman-table state through the unchanged
internal ABI.
The port preserves literal header choices, compression-gain decisions,
repeat-table transitions, and 1X/4X encoder selection. It lets ordinary C
compression paths exercise Rust code without widening this commit into the
remaining frame compressor.
Test Plan:
- cargo clippy, cargo clippy --benches, cargo clippy --tests, and nightly fmt
- cargo test --all-targets (94 passed) and feature-only cargo checks
- native fuzzer, zstreamtest, invalidDictionaries, and CLI round trip
- compare literals and next-table bytes with pristine C on x86_64 and i686
Refs: rust/README.md compression-primitives map
Move Huffman table construction, table serialization, and one- and
four-stream payload encoding from huf_compress.c into the Rust compatibility
archive. The declaration-only C shim preserves the existing internal ABI,
so the remaining C compressor can call the migrated implementation unchanged.
The translation keeps CTable layouts, workspace checks, repeat-table
selection, and bitstream output compatible with the original encoder. This
lets native C consumers exercise the Rust implementation through libzstd.
Test Plan:
- cargo clippy, cargo clippy --benches, and cargo clippy --tests
- cargo +nightly fmt and cargo test --all-targets (88 passed)
- cargo check --no-default-features --features compression
- rebuild and run tests fuzzer, zstreamtest, and invalidDictionaries
- compare tables, headers, and 1X/4X streams with a renamed pristine C build
Refs: rust/README.md component map
Move ZSTD_splitBlock into Rust while retaining its caller-owned workspace
contract. The implementation preserves the C fingerprint sampling heuristic
and calls the migrated histogram primitive without adding a hot-path
allocation.
Reference-vector and randomized differential tests compare every split level
with the original C translation unit. The C source remains as a declaration
shim so existing source lists resolve the Rust ABI during the migration.
Test Plan:
- cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests
- cargo test --all-targets and cargo build --release
- cargo test/build --target i686-unknown-linux-gnu
- 264-case original-C/Rust differential harness
- C fuzzer, zstreamtest, and fuzzer32 smoke runs
Refs: rust/README.md
Move FSE normalization, count-header writing, compression-table construction,
and stream encoding from C to Rust. The original C source remains as a
header-only shim, preserving the public header configuration while native
consumers receive the Rust archive exports.
This completes both codec directions for FSE in the migration crate and lets
the unchanged C compatibility tests exercise the Rust encoder implementation.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets
- cargo build --release
- Compile the C shim with strict warnings enabled
- 1,000-case pristine-C differential for tables, headers, and payloads
- ./tests/fuzzer -v -T10s
- ./tests/decodecorpus -t -T5
Refs: rust/README.md
Move byte histogram counting into Rust for FSE and Huffman compression callers.
The implementation preserves the C workspace contract, checked alphabet path,
fast path, empty-input behavior, and C ABI while the original C source becomes
a header-only compatibility shim.
Focused tests cover count accumulation, invalid alphabets, workspace failures,
and striped large inputs. Higher-level compression remains C for now.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
- make -C tests fuzzer
- ./tests/fuzzer -i1 --no-big-tests
- compile hist.c with -Werror and -Wredundant-decls
Refs: rust/README.md
this is a prototype definition error:
`_mm_storeu_si128()` should accept a `void*` pointer,
since it explicitly states that it accepts unaligned addresses
yet requiring a `__m128i*` tells otherwise, and requires the compiler the enforce this alignment.
seems like a prototype interface error:
input parameter should have been `const void*`,
since the documentation is explicit that input doesn't have to be aligned,
but `const __m256i*` makes the compiler enforce it.
do not solve the equation, even though some members cancel each other,
this is done for clarity,
we'll let the compiler do the resolution at compile time.
ddidderr
Yann Collet