Move the two-table fast match finder into Rust while retaining a small C
projection of the private match-state. The Rust implementation handles normal,
attached-dictionary, and external-dictionary matching without exposing the
full C context ABI. The C entry points and build-time exclusion guards remain
unchanged for configured consumers.
Document the migrated matcher in the Rust component map and narrow the
remaining-C boundary accordingly.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- timeout 20s stdbuf -oL ./tests/poolTests
Refs: rust/README.md
Move the context-free public ZSTD_compressBound ABI into Rust while leaving
the stateful compressor context in C. The Rust implementation preserves the
size-width-specific input limit, macro arithmetic, and srcSize_wrong error
contract used by C callers.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_compress_api::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_compress_api::tests
- make -B -C lib -j2 libzstd.a
- fuzzer -s5346 -i1 --no-big-tests
Refs: public ZSTD_COMPRESSBOUND macro in lib/zstd.h
Move fast hash-table filling and no-dictionary, external-dictionary, and
attached-CDict match loops into Rust. The C shim keeps `ZSTD_MatchState_t`
opaque, extracts only its needed fields, and asserts the mirrored SeqStore
leaf layout so existing compression dispatch remains ABI-compatible.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_fast::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_fast::tests -- --nocapture
- byte-identical C-control frames for dictionary and streaming variants
- fuzzer -s2066 -i5 --no-big-tests
- invalidDictionaries and zstreamtest -i1
Refs: Rust superblock port fde1d70c
Move subblock partitioning, literal and sequence section emission, entropy
fallback, and repcode repair into Rust. Keep a narrow C wrapper to read the
opaque compression context and invoke the existing C entropy-stat builder;
the Rust side owns only verified C-shaped leaf layouts.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_compress_superblock::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_compress_superblock::tests
- byte-exact C control across 16 target-size and input-shape cases
- fuzzer, zstreamtest, invalidDictionaries, and bounded native test matrix
Refs: Rust sequence entropy port 887af204
Move FSE table-costing, table construction, encoding selection, and sequence
bitstream emission into Rust while retaining the existing C internal-header
boundary. The shim preserves all five C ABI entry points, so the surrounding
compressor continues to consume C-shaped sequence and entropy-table storage.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu zstd_compress_sequences::tests
- C-vs-Rust differential across 163 payload cases on x86_64 and i686
- fuzzer, zstreamtest, and invalidDictionaries
Refs: Rust FSE compression port 5f9d607d
Move raw, RLE, and Huffman literal-section encoding into the Rust
compatibility archive. The surrounding C block compressor continues to pass
its existing entropy workspace and Huffman-table state through the unchanged
internal ABI.
The port preserves literal header choices, compression-gain decisions,
repeat-table transitions, and 1X/4X encoder selection. It lets ordinary C
compression paths exercise Rust code without widening this commit into the
remaining frame compressor.
Test Plan:
- cargo clippy, cargo clippy --benches, cargo clippy --tests, and nightly fmt
- cargo test --all-targets (94 passed) and feature-only cargo checks
- native fuzzer, zstreamtest, invalidDictionaries, and CLI round trip
- compare literals and next-table bytes with pristine C on x86_64 and i686
Refs: rust/README.md compression-primitives map
Move Huffman table construction, table serialization, and one- and
four-stream payload encoding from huf_compress.c into the Rust compatibility
archive. The declaration-only C shim preserves the existing internal ABI,
so the remaining C compressor can call the migrated implementation unchanged.
The translation keeps CTable layouts, workspace checks, repeat-table
selection, and bitstream output compatible with the original encoder. This
lets native C consumers exercise the Rust implementation through libzstd.
Test Plan:
- cargo clippy, cargo clippy --benches, and cargo clippy --tests
- cargo +nightly fmt and cargo test --all-targets (88 passed)
- cargo check --no-default-features --features compression
- rebuild and run tests fuzzer, zstreamtest, and invalidDictionaries
- compare tables, headers, and 1X/4X streams with a renamed pristine C build
Refs: rust/README.md component map
Build feature-matched Rust archives for the native Make library path. Flatten
their members into libzstd.a so static consumers resolve C-to-Rust and
Rust-to-C references in one archive, and whole-archive link shared outputs so
every migrated public ABI export remains available.
Rust compression and decompression features now mirror the C partial-library
switches. This prevents compression-only artifacts from retaining DDict
references to decompression-only C helpers.
Add a C archive smoke test that round-trips data and exercises the DDict
lifecycle through lib/libzstd.a, rather than direct test-object links.
Test Plan:
- cargo fmt, strict Clippy, cargo test --all-targets, release build
- Rust checks with compression-only, decompression-only, and no features
- full, partial, forced-HUF, and 32-bit static/shared Make library probes
- test-rust-lib-smoke, fuzzer, zstreamtest, and invalidDictionaries
Refs: rust/README.md
Move ZSTD_splitBlock into Rust while retaining its caller-owned workspace
contract. The implementation preserves the C fingerprint sampling heuristic
and calls the migrated histogram primitive without adding a hot-path
allocation.
Reference-vector and randomized differential tests compare every split level
with the original C translation unit. The C source remains as a declaration
shim so existing source lists resolve the Rust ABI during the migration.
Test Plan:
- cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests
- cargo test --all-targets and cargo build --release
- cargo test/build --target i686-unknown-linux-gnu
- 264-case original-C/Rust differential harness
- C fuzzer, zstreamtest, and fuzzer32 smoke runs
Refs: rust/README.md
Move ZSTD_DDict allocation, ownership, construction, and public ABI exports
into Rust. The C decoder keeps ZSTD_loadDEntropy until its table loader moves.
The C shim now exposes C-computed DCtx field offsets. This keeps dictionary
state correct across conditional C layouts, including fuzz fields and 32-bit
static-BMI2 builds, without duplicating decoder-context layout in Rust.
Test Plan:
- cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests
- cargo test --all-targets and cargo build --release
- cargo test/build --target i686-unknown-linux-gnu
- C fuzzer, zstreamtest, invalidDictionaries, and fuzzer32 smoke runs
- fuzz-enabled and i686 -mbmi2 C/Rust dictionary roundtrip harnesses
Refs: rust/README.md
Move Huffman decoding-table construction and X1/X2 single- and four-stream
decoding into Rust. The original C translation unit is now a declaration shim,
so unchanged C callers and tests resolve the public decoder ABI from the Rust
archive.
The portable implementation preserves the C workspace and error contracts,
retains the existing assembly-only internal fast loops, and mirrors the C
32-bit decode cadence so X2 writes remain within its required output window.
Forced X1/X2 modes follow the matching archive configuration from the build
integration.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets for default, forced X1, and forced X2 modes
- cargo build and cargo test for i686-unknown-linux-gnu in all three modes
- Strict normal, X1, and X2 C shim compilation
- Original fuzzer and fullbench compatibility targets
Refs: rust/README.md
Depends-on: d89ebb31
Move FSE normalization, count-header writing, compression-table construction,
and stream encoding from C to Rust. The original C source remains as a
header-only shim, preserving the public header configuration while native
consumers receive the Rust archive exports.
This completes both codec directions for FSE in the migration crate and lets
the unchanged C compatibility tests exercise the Rust encoder implementation.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets
- cargo build --release
- Compile the C shim with strict warnings enabled
- 1,000-case pristine-C differential for tables, headers, and payloads
- ./tests/fuzzer -v -T10s
- ./tests/decodecorpus -t -T5
Refs: rust/README.md
Move byte histogram counting into Rust for FSE and Huffman compression callers.
The implementation preserves the C workspace contract, checked alphabet path,
fast path, empty-input behavior, and C ABI while the original C source becomes
a header-only compatibility shim.
Focused tests cover count accumulation, invalid alphabets, workspace failures,
and striped large inputs. Higher-level compression remains C for now.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
- make -C tests fuzzer
- ./tests/fuzzer -i1 --no-big-tests
- compile hist.c with -Werror and -Wredundant-decls
Refs: rust/README.md
Replace the common worker pool and debug pthread wrappers with ABI-compatible
Rust implementations. The pool preserves bounded-queue, resize, drain-on-free,
custom-allocator, and no-thread behavior while C supplies only the build-time
multithreading configuration bit.
This moves runtime support without changing C tests or public headers. Codec
and CLI implementations remain C while their dependencies are migrated.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
- make -C tests poolTests
- ./tests/poolTests
- compile pool and threading shims with -Werror in MT and no-thread modes
Refs: rust/README.md
Move FSE normalized-count parsing, Huffman statistics decoding, FSE table
construction, and FSE stream decompression into Rust. The C source files now
only retain the headers needed by the existing build configuration.
The implementation keeps the public C ABI and verifies C-generated balanced,
skewed, and Huffman-statistics streams. Compression and the higher-level frame
decoder remain C for now.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
- compile both C shims with -Werror and -Wredundant-decls
Refs: rust/README.md
The common C shims include headers that already declare every Rust-exported
symbol. Remove duplicate declarations so strict C builds do not report
redundant declarations or macro-expanded aliases.
The shims remain intentionally declaration-only; Rust still owns the exported
implementation.
Test Plan:
- compile debug.c with -Wall -Wextra -Werror
- compile error_private.c with -Wall -Wextra -Werror
- compile zstd_common.c with -Wall -Wextra -Werror
Refs: rust/README.md
Introduce the Rust static library and move the shared byte, bitstream, CPU,
error, xxHash, debug, and public-common implementations into it. Thin C shims
preserve the existing header-driven C build while original tests link the Rust
archive.
This establishes the ABI-safe foundation for later codec and CLI ports;
entropy coding, runtime support, codecs, dictionaries, and the CLI remain C.
The top-down migration map documents that boundary and its validation path.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
Refs: rust/README.md
this is a prototype definition error:
`_mm_storeu_si128()` should accept a `void*` pointer,
since it explicitly states that it accepts unaligned addresses
yet requiring a `__m128i*` tells otherwise, and requires the compiler the enforce this alignment.
seems like a prototype interface error:
input parameter should have been `const void*`,
since the documentation is explicit that input doesn't have to be aligned,
but `const __m256i*` makes the compiler enforce it.
ddidderr
Yann Collet