ddidderr
0a6a5f5267
Move Huffman table construction, table serialization, and one- and four-stream payload encoding from huf_compress.c into the Rust compatibility archive. The declaration-only C shim preserves the existing internal ABI, so the remaining C compressor can call the migrated implementation unchanged. The translation keeps CTable layouts, workspace checks, repeat-table selection, and bitstream output compatible with the original encoder. This lets native C consumers exercise the Rust implementation through libzstd. Test Plan: - cargo clippy, cargo clippy --benches, and cargo clippy --tests - cargo +nightly fmt and cargo test --all-targets (88 passed) - cargo check --no-default-features --features compression - rebuild and run tests fuzzer, zstreamtest, and invalidDictionaries - compare tables, headers, and 1X/4X streams with a renamed pristine C build Refs: rust/README.md component map
82 lines
3.5 KiB
Markdown
82 lines
3.5 KiB
Markdown
# Rust rewrite
|
|
|
|
This directory contains the in-progress Rust replacement for the zstd library
|
|
and command-line program. During the migration, the crate is built as a static
|
|
library and linked into the original C test programs. Production C translation
|
|
units become declaration-only shims as their implementations move to Rust; the
|
|
original C tests remain unchanged and provide compatibility coverage.
|
|
|
|
## Component map
|
|
|
|
The crate is organized from low-level representation helpers toward the public
|
|
zstd ABI:
|
|
|
|
- Common primitives
|
|
- `mem`, `bits`, `bitstream`, and `cpu` implement byte-order, bitstream, and
|
|
target-feature operations used by the codecs.
|
|
- `errors`, `debug`, `xxhash`, and `zstd_common` provide common exported ABI
|
|
functions and state.
|
|
- `common` contains shared frame constants and internal data types.
|
|
- Entropy coding
|
|
- `entropy_common` reads FSE normalized counts and Huffman statistics.
|
|
- `fse_decompress` builds FSE decoding tables and decodes FSE streams.
|
|
- `fse_compress` normalizes counts, writes FSE headers, builds compression
|
|
tables, and encodes FSE streams.
|
|
- `huf_compress` builds Huffman compression tables, writes table headers,
|
|
and encodes one- and four-stream Huffman payloads.
|
|
- `huf_decompress` builds Huffman decoding tables and decodes X1 and X2
|
|
Huffman streams.
|
|
- Compression primitives
|
|
- `hist` counts byte frequencies for FSE and Huffman compression.
|
|
- `zstd_presplit` chooses split points for full compression blocks.
|
|
- Runtime support
|
|
- `threading` provides platform pthread wrappers required by zstd headers.
|
|
- `pool` implements the bounded worker pool used by multithreaded compression.
|
|
- Dictionary support
|
|
- `zstd_ddict` owns, loads, copies, and references decode dictionaries.
|
|
|
|
The remaining block compression, general decompression, dictionary-building,
|
|
legacy, and CLI translation units are still C. They must move before the
|
|
rewrite is complete. Keeping that boundary explicit prevents a passing hybrid
|
|
build from being mistaken for the final all-Rust result.
|
|
|
|
## Compatibility boundary
|
|
|
|
The public ABI continues to come from the existing headers under `lib/`.
|
|
Exported Rust functions therefore use C layout and calling conventions. A C
|
|
source file whose implementation has moved to Rust remains in the original
|
|
makefile source list as a small shim so header configuration and platform
|
|
preprocessor behavior stay available during the transition.
|
|
|
|
The library, test, and program makefiles select an archive directory for the
|
|
active C configuration: enabled compression/decompression modules, default or
|
|
forced HUF X1/X2, and the matching Rust target for 32-bit C binaries. The
|
|
native static archive flattens Rust object members rather than nesting a Rust
|
|
archive, while the native shared library retains all migrated Rust exports.
|
|
When the HUF mode changes, the test and program paths also rebuild cached C
|
|
outputs before linking. This prevents original C tests from using a stale or
|
|
configuration-incompatible implementation.
|
|
|
|
## Validation
|
|
|
|
Run focused Rust checks from this directory:
|
|
|
|
```sh
|
|
cargo fmt --check
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo test --all-targets
|
|
cargo build --release
|
|
```
|
|
|
|
Then run original compatibility tests from the repository root, starting with
|
|
the narrow target for the component being migrated. For example:
|
|
|
|
```sh
|
|
make -C tests fuzzer
|
|
./tests/fuzzer -i1 --no-big-tests
|
|
make -C tests test-rust-lib-smoke
|
|
```
|
|
|
|
Broader `tests/Makefile` targets remain the authoritative integration gates as
|
|
more of the library and CLI are rewritten.
|