ddidderr
24488e4eaa
Port lib/legacy/zstd_v01.c (the frozen zstd v0.1 decoder) to
rust/src/legacy/zstd_v01.rs as the first legacy-format port on the new
scaffolding, and reduce the C file to a declaration-only shim that
keeps its header includes for configuration and platform preprocessor
behavior.
Frozen-decoder policy: zstd_v01.c embeds its own v0.1-era FSE and
Huff0 snapshot, distinct from every other release. The Rust port is a
line-by-line translation with the same table layouts (FSE_DTable as a
u32 header word plus packed newState/symbol/nbBits entries, the Huff0
u16 DTable with byte/nbBits pairs), the same arithmetic including
wrap-around and pointer-comparison quirks (e.g. the offset-vs-base
address check in ZSTD_execSequence), the same internal FSE error space
(size_t)-1..-7, and the same public ZSTD error codes. It reuses no
modern Rust entropy module; its only crate dependency is `errors`,
matching the C file's error_private.h include. The 32-bit-only reload
points are kept as compile-time conditions on usize::BITS.
Symbol takeover boundary: all nine ZSTDv01_* entry points from
zstd_v01.h now come from Rust as context-free #[no_mangle] extern "C"
functions (isError, decompress, decompressDCtx,
findFrameSizeInfoLegacy, createDCtx, freeDCtx, resetDCtx,
nextSrcSizeToDecompress, decompressContinue). zstd_legacy.h only uses
the first four for v0.1; streaming for v0.1-v0.3 intentionally returns
version_unsupported there, unchanged. The ZSTDv01_Dctx struct
definition moves entirely into Rust: C code only ever holds an opaque
pointer (zstd_v01.h forward-declares the type), and the context is
malloc/free-allocated exactly like the C version so create/free may
pair across the language boundary.
Byte-identity verification against the pristine pre-migration C build
(f8745da6, pure C, ZSTD_LEGACY_SUPPORT=1):
- Real v0.1 frames were generated by building the v0.1.0 git tag and
compressing text, random, and 426 KB multi-block inputs. A one-shot
ZSTD_decompress harness linked once against the pristine C libzstd.a
and once against the Rust-backed libzstd.a produced bit-identical
outputs for all frames.
- A direct ZSTDv01_* probe (one-shot decode, dst-too-small, truncated
input, bad magic, findFrameSizeInfoLegacy, and the streaming
continue loop) printed identical results, including exact error
codes (-70 dstSize_tooSmall, -72 srcSize_wrong, -10 prefix_unknown)
and identical dBound values.
- zstd -l -v on v0.1 files matches the pristine binary; CLI streaming
decode of v0.1 fails with the same "Version not supported" in both,
by design of zstd_legacy.h.
Unit tests embed three v0.1.0-generated fixtures (entropy-coded,
raw-block, and four-block frames) plus the truncation, bad-magic,
small-destination, and streaming-API cases, all asserting the exact C
error codes above. Note that `make -C tests test-legacy` only covers
v0.4+ frames, so the embedded fixtures and the harness comparison are
the actual v0.1 coverage.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
--features legacy-v01 -- -D warnings && cargo test --all-targets
--features legacy-v01 (127 tests, 9 for v0.1)
- cargo clippy/test --no-default-features --features
decompression,legacy-v01 (module builds standalone)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests, also with
ZSTD_LEGACY_SUPPORT=1 (mixed Rust v0.1 + C v0.2-0.7 link)
- make -C tests test-rust-lib-smoke && make -C tests test-legacy
- make -C programs zstd (default and ZSTD_LEGACY_SUPPORT=1); nm shows
the nine ZSTDv01_* symbols provided by Rust at level 1
- make -C lib libzstd.a ZSTD_LEGACY_SUPPORT=0 (no legacy symbols) and
meson -Dlegacy_level=1 shared library exporting all nine
158 lines
7.6 KiB
Markdown
158 lines
7.6 KiB
Markdown
# Rust rewrite
|
|
|
|
This directory contains the in-progress Rust replacement for the zstd library
|
|
and command-line program. During the migration, the crate is built as a static
|
|
library and linked into the original C test programs. Production C translation
|
|
units become declaration-only shims as their implementations move to Rust; the
|
|
original C tests remain unchanged and provide compatibility coverage.
|
|
|
|
## Component map
|
|
|
|
The crate is organized from low-level representation helpers toward the public
|
|
zstd ABI:
|
|
|
|
- Common primitives
|
|
- `mem`, `bits`, `bitstream`, and `cpu` implement byte-order, bitstream, and
|
|
target-feature operations used by the codecs.
|
|
- `errors`, `debug`, `xxhash`, and `zstd_common` provide common exported ABI
|
|
functions and state.
|
|
- `common` contains shared frame constants and internal data types.
|
|
- Entropy coding
|
|
- `entropy_common` reads FSE normalized counts and Huffman statistics.
|
|
- `fse_decompress` builds FSE decoding tables and decodes FSE streams.
|
|
- `fse_compress` normalizes counts, writes FSE headers, builds compression
|
|
tables, and encodes FSE streams.
|
|
- `huf_compress` builds Huffman compression tables, writes table headers,
|
|
and encodes one- and four-stream Huffman payloads.
|
|
- `huf_decompress` builds Huffman decoding tables and decodes X1 and X2
|
|
Huffman streams.
|
|
- Compression primitives
|
|
- `hist` counts byte frequencies for FSE and Huffman compression.
|
|
- `zstd_presplit` chooses split points for full compression blocks.
|
|
- `zstd_compress_literals` emits raw, RLE, and Huffman literal sections
|
|
while preserving the compressor's Huffman-table repeat state.
|
|
- `zstd_compress_frame` serializes frame headers, skippable frames, and the
|
|
last empty block; it takes scalar frame parameters so the C-owned
|
|
`ZSTD_CCtx_params` layout never crosses the language boundary.
|
|
- `zstd_fast` and `zstd_double_fast` implement the single- and two-table
|
|
fast block match finders, including attached and external dictionary paths.
|
|
- `zstd_lazy` implements greedy, lazy, lazy2, and binary-tree matching,
|
|
including row-based and dictionary search variants.
|
|
- `zstd_opt_tree` maintains the binary-tree index used by optimal matching;
|
|
the dynamic-programming optimal parser itself remains in C for now.
|
|
- `zstd_ldm` implements long-distance-match parameter selection, table
|
|
maintenance, sequence generation, and sequence consumption.
|
|
- Runtime support
|
|
- `threading` provides platform pthread wrappers required by zstd headers.
|
|
- `pool` implements the bounded worker pool used by multithreaded compression.
|
|
- Dictionary support
|
|
- `zstd_ddict` owns, loads, copies, and references decode dictionaries.
|
|
- Legacy decoding
|
|
- `legacy` hosts one frozen module per historical format; `legacy::zstd_v01`
|
|
ports the self-contained v0.1 decoder. Versions v0.2 through v0.7 are
|
|
still C.
|
|
- Block decompression
|
|
- `zstd_decompress_block` decodes literal and sequence sections, maintains
|
|
FSE/Huffman repeat state, and executes compressed-block sequences.
|
|
- `zstd_decompress` owns the public decompression context, one-shot,
|
|
dictionary, parameter, and streaming state machines. Its C shim retains
|
|
configuration-dependent context allocation plus legacy and trace leaves.
|
|
- Command-line frontend
|
|
- `zstd_cli` owns the Rust parser, safety policy, and dispatch. It is built
|
|
by the separate `cli/` static-library package only for program archives,
|
|
so library builds do not acquire program-only dependencies. The C
|
|
`fileio` backend still owns file opening, safe replacement, sparse writes,
|
|
metadata, and streaming I/O.
|
|
|
|
The optimal block matcher, high-level frame compression, dictionary-building,
|
|
the legacy v0.2-v0.7 decoders, and the CLI file-I/O backend are still C. They
|
|
must move before the rewrite is complete. Keeping that boundary explicit
|
|
prevents a passing hybrid build from being mistaken for the final all-Rust
|
|
result.
|
|
|
|
## Legacy decoding
|
|
|
|
Each `lib/legacy/zstd_v0N.c` file is a frozen snapshot of the entropy coders
|
|
and frame logic of one historical release. The Rust ports in `src/legacy/`
|
|
keep that property: every version owns its own frozen FSE/Huff0 and frame
|
|
logic, ported line by line, and must never reuse the modern entropy modules
|
|
or share code with other legacy versions. Outputs and error codes must be
|
|
byte-identical to the original C files. Their only shared dependency is the
|
|
`errors` module, matching the C files' `error_private.h` include.
|
|
|
|
Cargo features `legacy-v01` .. `legacy-v07` gate the per-version modules and
|
|
are never default features. The build systems derive the feature list from
|
|
the C configuration:
|
|
|
|
- `lib/Makefile` and `programs/Makefile` map `ZSTD_LEGACY_SUPPORT=N` to the
|
|
features for versions >= N (0 disables legacy), matching the
|
|
`ZSTD_LEGACY_FILES` selection in `lib/libzstd.mk`.
|
|
- `tests/Makefile` always enables all seven features because the test
|
|
objects compile every `lib/legacy/*.c` file regardless of dispatch level.
|
|
- `build/meson` maps `legacy_level` like the makefiles; `build/cmake`
|
|
enables all seven whenever `ZSTD_LEGACY_SUPPORT` is on because it always
|
|
compiles all seven C files.
|
|
|
|
Every build system also encodes the legacy selection in the Rust target
|
|
directory name (for example `c1-d1-default-legacy5`), for the same reason the
|
|
HUF mode is encoded there: a cached archive built for one configuration must
|
|
never be linked into a build that expects another.
|
|
|
|
A feature whose version has not been ported yet gates nothing; the original
|
|
C file still provides that decoder, so mixed C/Rust legacy levels link
|
|
cleanly. Porting a version means adding `src/legacy/zstd_v0N.rs`, registering
|
|
it in `src/legacy/mod.rs` behind its feature, and reducing
|
|
`lib/legacy/zstd_v0N.c` to a declaration-only shim. For v0.1 the streaming
|
|
`ZSTDv01_Dctx` state lives entirely in Rust: C code only ever holds an opaque
|
|
pointer, so the C-side struct definition is gone.
|
|
|
|
## Compatibility boundary
|
|
|
|
The public ABI continues to come from the existing headers under `lib/`.
|
|
Exported Rust functions therefore use C layout and calling conventions. A C
|
|
source file whose implementation has moved to Rust remains in the original
|
|
makefile source list as a small shim so header configuration and platform
|
|
preprocessor behavior stay available during the transition.
|
|
|
|
The library, test, and program makefiles select an archive directory for the
|
|
active C configuration: enabled compression/decompression modules, default or
|
|
forced HUF X1/X2, and the matching Rust target for 32-bit C binaries. The
|
|
native static archive flattens Rust object members rather than nesting a Rust
|
|
archive, while the native shared library retains all migrated Rust exports.
|
|
When the HUF mode changes, the test and program paths also rebuild cached C
|
|
outputs before linking. This prevents original C tests from using a stale or
|
|
configuration-incompatible implementation.
|
|
|
|
## Validation
|
|
|
|
Run focused Rust checks from this directory:
|
|
|
|
```sh
|
|
cargo fmt --check
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo test --all-targets
|
|
cargo build --release
|
|
```
|
|
|
|
The program-only Rust archive has its own feature matrix and should be checked
|
|
from `rust/cli` as well:
|
|
|
|
```sh
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo test --all-targets
|
|
cargo test --no-default-features --features compression --all-targets
|
|
cargo test --no-default-features --features decompression --all-targets
|
|
```
|
|
|
|
Then run original compatibility tests from the repository root, starting with
|
|
the narrow target for the component being migrated. For example:
|
|
|
|
```sh
|
|
make -C tests fuzzer
|
|
./tests/fuzzer -i1 --no-big-tests
|
|
make -C tests test-rust-lib-smoke
|
|
```
|
|
|
|
Broader `tests/Makefile` targets remain the authoritative integration gates as
|
|
more of the library and CLI are rewritten.
|