Move Huffman table construction, table serialization, and one- and
four-stream payload encoding from huf_compress.c into the Rust compatibility
archive. The declaration-only C shim preserves the existing internal ABI,
so the remaining C compressor can call the migrated implementation unchanged.
The translation keeps CTable layouts, workspace checks, repeat-table
selection, and bitstream output compatible with the original encoder. This
lets native C consumers exercise the Rust implementation through libzstd.
Test Plan:
- cargo clippy, cargo clippy --benches, and cargo clippy --tests
- cargo +nightly fmt and cargo test --all-targets (88 passed)
- cargo check --no-default-features --features compression
- rebuild and run tests fuzzer, zstreamtest, and invalidDictionaries
- compare tables, headers, and 1X/4X streams with a renamed pristine C build
Refs: rust/README.md component map
Move ZSTD_splitBlock into Rust while retaining its caller-owned workspace
contract. The implementation preserves the C fingerprint sampling heuristic
and calls the migrated histogram primitive without adding a hot-path
allocation.
Reference-vector and randomized differential tests compare every split level
with the original C translation unit. The C source remains as a declaration
shim so existing source lists resolve the Rust ABI during the migration.
Test Plan:
- cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests
- cargo test --all-targets and cargo build --release
- cargo test/build --target i686-unknown-linux-gnu
- 264-case original-C/Rust differential harness
- C fuzzer, zstreamtest, and fuzzer32 smoke runs
Refs: rust/README.md
Move FSE normalization, count-header writing, compression-table construction,
and stream encoding from C to Rust. The original C source remains as a
header-only shim, preserving the public header configuration while native
consumers receive the Rust archive exports.
This completes both codec directions for FSE in the migration crate and lets
the unchanged C compatibility tests exercise the Rust encoder implementation.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets
- cargo build --release
- Compile the C shim with strict warnings enabled
- 1,000-case pristine-C differential for tables, headers, and payloads
- ./tests/fuzzer -v -T10s
- ./tests/decodecorpus -t -T5
Refs: rust/README.md
Move byte histogram counting into Rust for FSE and Huffman compression callers.
The implementation preserves the C workspace contract, checked alphabet path,
fast path, empty-input behavior, and C ABI while the original C source becomes
a header-only compatibility shim.
Focused tests cover count accumulation, invalid alphabets, workspace failures,
and striped large inputs. Higher-level compression remains C for now.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
- make -C tests fuzzer
- ./tests/fuzzer -i1 --no-big-tests
- compile hist.c with -Werror and -Wredundant-decls
Refs: rust/README.md
this is a prototype definition error:
`_mm_storeu_si128()` should accept a `void*` pointer,
since it explicitly states that it accepts unaligned addresses
yet requiring a `__m128i*` tells otherwise, and requires the compiler the enforce this alignment.
seems like a prototype interface error:
input parameter should have been `const void*`,
since the documentation is explicit that input doesn't have to be aligned,
but `const __m256i*` makes the compiler enforce it.
do not solve the equation, even though some members cancel each other,
this is done for clarity,
we'll let the compiler do the resolution at compile time.
Move towards a stronger guarantee of reproducibility by removing this small difference for machines without SSE2/Neon.
The SIMD behavior is now the default for all platforms.
The optimal parser with LDM enabled using minMatch > 3 could generate a match
length of 3 when minMatch >= 4. This is not allowed.
1. Fix the bug
2. Add validation logic to `ZSTD_buildSeqStore()` in debug mode for all block
compressors that checks we never generate too short a match. This way we don't
rely on the `generate_sequences` fuzzer to find this issue.
Credit to OSS-Fuzz
ddidderr
Yann Collet