ddidderr
959e485201
Continue the incremental zstd_compress.c migration with its frame serialization leaves: ZSTD_writeFrameHeader(), the public ZSTD_writeSkippableFrame() ABI, and ZSTD_writeLastEmptyBlock() now live in rust/src/zstd_compress_frame.rs. ZSTD_writeFrameHeader() reads five fields out of ZSTD_CCtx_params, whose layout is private to zstd_compress.c and sensitive to build configuration. Rather than mirror that structure in Rust, the C function keeps its original static signature and forwards the scalar fields to ZSTD_rust_writeFrameHeader(), so no parameter-structure layout crosses the language boundary. The other two functions take only pointer/size arguments and are exported directly, replacing their C bodies outright. Behavior differences are limited to hardening in release builds: the Rust leaf clamps a window-size shift that C would leave undefined for a malformed windowLog, and ZSTD_writeSkippableFrame() rejects a srcSize + header overflow instead of comparing against a wrapped sum. Both paths are unreachable through validated callers, so compressed output is byte-identical. Remaining zstd_compress.c work: parameter selection and validation, context lifecycle, block dispatch, dictionary loading, and the streaming state machine. Test plan: - cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets (118 tests, includes new frame header/skippable/last-block reference vectors) - make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests - make -C tests test-rust-lib-smoke
117 lines
5.5 KiB
Markdown
117 lines
5.5 KiB
Markdown
# Rust rewrite
|
|
|
|
This directory contains the in-progress Rust replacement for the zstd library
|
|
and command-line program. During the migration, the crate is built as a static
|
|
library and linked into the original C test programs. Production C translation
|
|
units become declaration-only shims as their implementations move to Rust; the
|
|
original C tests remain unchanged and provide compatibility coverage.
|
|
|
|
## Component map
|
|
|
|
The crate is organized from low-level representation helpers toward the public
|
|
zstd ABI:
|
|
|
|
- Common primitives
|
|
- `mem`, `bits`, `bitstream`, and `cpu` implement byte-order, bitstream, and
|
|
target-feature operations used by the codecs.
|
|
- `errors`, `debug`, `xxhash`, and `zstd_common` provide common exported ABI
|
|
functions and state.
|
|
- `common` contains shared frame constants and internal data types.
|
|
- Entropy coding
|
|
- `entropy_common` reads FSE normalized counts and Huffman statistics.
|
|
- `fse_decompress` builds FSE decoding tables and decodes FSE streams.
|
|
- `fse_compress` normalizes counts, writes FSE headers, builds compression
|
|
tables, and encodes FSE streams.
|
|
- `huf_compress` builds Huffman compression tables, writes table headers,
|
|
and encodes one- and four-stream Huffman payloads.
|
|
- `huf_decompress` builds Huffman decoding tables and decodes X1 and X2
|
|
Huffman streams.
|
|
- Compression primitives
|
|
- `hist` counts byte frequencies for FSE and Huffman compression.
|
|
- `zstd_presplit` chooses split points for full compression blocks.
|
|
- `zstd_compress_literals` emits raw, RLE, and Huffman literal sections
|
|
while preserving the compressor's Huffman-table repeat state.
|
|
- `zstd_compress_frame` serializes frame headers, skippable frames, and the
|
|
last empty block; it takes scalar frame parameters so the C-owned
|
|
`ZSTD_CCtx_params` layout never crosses the language boundary.
|
|
- `zstd_fast` and `zstd_double_fast` implement the single- and two-table
|
|
fast block match finders, including attached and external dictionary paths.
|
|
- `zstd_lazy` implements greedy, lazy, lazy2, and binary-tree matching,
|
|
including row-based and dictionary search variants.
|
|
- `zstd_opt_tree` maintains the binary-tree index used by optimal matching;
|
|
the dynamic-programming optimal parser itself remains in C for now.
|
|
- `zstd_ldm` implements long-distance-match parameter selection, table
|
|
maintenance, sequence generation, and sequence consumption.
|
|
- Runtime support
|
|
- `threading` provides platform pthread wrappers required by zstd headers.
|
|
- `pool` implements the bounded worker pool used by multithreaded compression.
|
|
- Dictionary support
|
|
- `zstd_ddict` owns, loads, copies, and references decode dictionaries.
|
|
- Block decompression
|
|
- `zstd_decompress_block` decodes literal and sequence sections, maintains
|
|
FSE/Huffman repeat state, and executes compressed-block sequences.
|
|
- `zstd_decompress` owns the public decompression context, one-shot,
|
|
dictionary, parameter, and streaming state machines. Its C shim retains
|
|
configuration-dependent context allocation plus legacy and trace leaves.
|
|
- Command-line frontend
|
|
- `zstd_cli` owns the Rust parser, safety policy, and dispatch. It is built
|
|
by the separate `cli/` static-library package only for program archives,
|
|
so library builds do not acquire program-only dependencies. The C
|
|
`fileio` backend still owns file opening, safe replacement, sparse writes,
|
|
metadata, and streaming I/O.
|
|
|
|
The optimal block matcher, high-level frame compression, dictionary-building,
|
|
legacy decoding callbacks, and the CLI file-I/O backend are still C. They must
|
|
move before the rewrite is complete. Keeping that boundary explicit prevents a
|
|
passing hybrid build from being mistaken for the final all-Rust result.
|
|
|
|
## Compatibility boundary
|
|
|
|
The public ABI continues to come from the existing headers under `lib/`.
|
|
Exported Rust functions therefore use C layout and calling conventions. A C
|
|
source file whose implementation has moved to Rust remains in the original
|
|
makefile source list as a small shim so header configuration and platform
|
|
preprocessor behavior stay available during the transition.
|
|
|
|
The library, test, and program makefiles select an archive directory for the
|
|
active C configuration: enabled compression/decompression modules, default or
|
|
forced HUF X1/X2, and the matching Rust target for 32-bit C binaries. The
|
|
native static archive flattens Rust object members rather than nesting a Rust
|
|
archive, while the native shared library retains all migrated Rust exports.
|
|
When the HUF mode changes, the test and program paths also rebuild cached C
|
|
outputs before linking. This prevents original C tests from using a stale or
|
|
configuration-incompatible implementation.
|
|
|
|
## Validation
|
|
|
|
Run focused Rust checks from this directory:
|
|
|
|
```sh
|
|
cargo fmt --check
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo test --all-targets
|
|
cargo build --release
|
|
```
|
|
|
|
The program-only Rust archive has its own feature matrix and should be checked
|
|
from `rust/cli` as well:
|
|
|
|
```sh
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo test --all-targets
|
|
cargo test --no-default-features --features compression --all-targets
|
|
cargo test --no-default-features --features decompression --all-targets
|
|
```
|
|
|
|
Then run original compatibility tests from the repository root, starting with
|
|
the narrow target for the component being migrated. For example:
|
|
|
|
```sh
|
|
make -C tests fuzzer
|
|
./tests/fuzzer -i1 --no-big-tests
|
|
make -C tests test-rust-lib-smoke
|
|
```
|
|
|
|
Broader `tests/Makefile` targets remain the authoritative integration gates as
|
|
more of the library and CLI are rewritten.
|