ddidderr
f9adcf6aef
Move the two-table fast match finder into Rust while retaining a small C projection of the private match-state. The Rust implementation handles normal, attached-dictionary, and external-dictionary matching without exposing the full C context ABI. The C entry points and build-time exclusion guards remain unchanged for configured consumers. Document the migrated matcher in the Rust component map and narrow the remaining-C boundary accordingly. Test Plan: - cargo test --all-targets - cargo test --target i686-unknown-linux-gnu --all-targets - cargo clippy && cargo clippy --benches && cargo clippy --tests - cargo +nightly fmt - make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests - ./tests/fuzzer -s5346 -i1 --no-big-tests - ./tests/zstreamtest -i3000 -s334462 - ./tests/invalidDictionaries - timeout 20s stdbuf -oL ./tests/poolTests Refs: rust/README.md
86 lines
3.8 KiB
Markdown
86 lines
3.8 KiB
Markdown
# Rust rewrite
|
|
|
|
This directory contains the in-progress Rust replacement for the zstd library
|
|
and command-line program. During the migration, the crate is built as a static
|
|
library and linked into the original C test programs. Production C translation
|
|
units become declaration-only shims as their implementations move to Rust; the
|
|
original C tests remain unchanged and provide compatibility coverage.
|
|
|
|
## Component map
|
|
|
|
The crate is organized from low-level representation helpers toward the public
|
|
zstd ABI:
|
|
|
|
- Common primitives
|
|
- `mem`, `bits`, `bitstream`, and `cpu` implement byte-order, bitstream, and
|
|
target-feature operations used by the codecs.
|
|
- `errors`, `debug`, `xxhash`, and `zstd_common` provide common exported ABI
|
|
functions and state.
|
|
- `common` contains shared frame constants and internal data types.
|
|
- Entropy coding
|
|
- `entropy_common` reads FSE normalized counts and Huffman statistics.
|
|
- `fse_decompress` builds FSE decoding tables and decodes FSE streams.
|
|
- `fse_compress` normalizes counts, writes FSE headers, builds compression
|
|
tables, and encodes FSE streams.
|
|
- `huf_compress` builds Huffman compression tables, writes table headers,
|
|
and encodes one- and four-stream Huffman payloads.
|
|
- `huf_decompress` builds Huffman decoding tables and decodes X1 and X2
|
|
Huffman streams.
|
|
- Compression primitives
|
|
- `hist` counts byte frequencies for FSE and Huffman compression.
|
|
- `zstd_presplit` chooses split points for full compression blocks.
|
|
- `zstd_compress_literals` emits raw, RLE, and Huffman literal sections
|
|
while preserving the compressor's Huffman-table repeat state.
|
|
- `zstd_fast` and `zstd_double_fast` implement the single- and two-table
|
|
fast block match finders, including attached and external dictionary paths.
|
|
- Runtime support
|
|
- `threading` provides platform pthread wrappers required by zstd headers.
|
|
- `pool` implements the bounded worker pool used by multithreaded compression.
|
|
- Dictionary support
|
|
- `zstd_ddict` owns, loads, copies, and references decode dictionaries.
|
|
|
|
The lazy and optimal block matchers, general decompression, dictionary-building,
|
|
legacy, and CLI translation units are still C. They must move before the
|
|
rewrite is complete. Keeping that boundary explicit prevents a passing hybrid
|
|
build from being mistaken for the final all-Rust result.
|
|
|
|
## Compatibility boundary
|
|
|
|
The public ABI continues to come from the existing headers under `lib/`.
|
|
Exported Rust functions therefore use C layout and calling conventions. A C
|
|
source file whose implementation has moved to Rust remains in the original
|
|
makefile source list as a small shim so header configuration and platform
|
|
preprocessor behavior stay available during the transition.
|
|
|
|
The library, test, and program makefiles select an archive directory for the
|
|
active C configuration: enabled compression/decompression modules, default or
|
|
forced HUF X1/X2, and the matching Rust target for 32-bit C binaries. The
|
|
native static archive flattens Rust object members rather than nesting a Rust
|
|
archive, while the native shared library retains all migrated Rust exports.
|
|
When the HUF mode changes, the test and program paths also rebuild cached C
|
|
outputs before linking. This prevents original C tests from using a stale or
|
|
configuration-incompatible implementation.
|
|
|
|
## Validation
|
|
|
|
Run focused Rust checks from this directory:
|
|
|
|
```sh
|
|
cargo fmt --check
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo test --all-targets
|
|
cargo build --release
|
|
```
|
|
|
|
Then run original compatibility tests from the repository root, starting with
|
|
the narrow target for the component being migrated. For example:
|
|
|
|
```sh
|
|
make -C tests fuzzer
|
|
./tests/fuzzer -i1 --no-big-tests
|
|
make -C tests test-rust-lib-smoke
|
|
```
|
|
|
|
Broader `tests/Makefile` targets remain the authoritative integration gates as
|
|
more of the library and CLI are rewritten.
|