ddidderr
fef5f4478a
Move the implementation of programs/benchfn.c into rust/src/benchfn.rs and wire benchmark mode (-b/-e/-i) into the Rust CLI frontend, which previously rejected those options as not yet implemented. `zstd -b1 -i0 FILE` and range runs like `zstd -b5e6 -i0 FILE` work again, including the synthetic-sample benchmark when no file is given. benchfn.rs is a faithful port of the run/timing state machine: BMK_benchFunction keeps the exact loop accounting (first-loop blockResults and errorFn checks, dstSize summed on the first loop only, 0xE5 warm-up of result buffers, nbLoops minimum of 1) and BMK_benchTimedFn keeps the same convergence behavior (x10 workload growth for short runs, budget-based nbLoops estimation, runs below half the run budget re-tried rather than reported, best qualifying run returned). Arithmetic that C leaves to unsigned wrap-around uses wrapping operations so debug builds cannot panic where release C would wrap. ABI notes: BMK_runTime_t and BMK_runOutcome_t are returned by value across the C boundary and BMK_benchParams_t is passed by value, so all three are repr(C) mirrors of benchfn.h; their field offsets are pinned by const asserts in Rust and matching C static asserts in the benchfn.c shim, which is now declaration-only. BMK_timedFnState_t stays opaque, fits the 64-byte BMK_timedFnState_shell (compile-time checked), and is malloc/free-managed so creation and destruction remain interchangeable with C callers. The CLI parses -b (bench mode), -e (range end, digits attach directly, defaulting to 0 like readU32FromChar) and -i (duration in seconds), then dispatches through a new ZSTD_rust_cli_bench bridge in the zstdcli.c shim. The bridge exists because benchmark availability is a C preprocessor property (ZSTD_NOBENCH): orchestration and reporting stay in C benchzstd.c, stripped variants (zstd-small, zstd-compress, zstd-decompress) compile the stub branch and report "benchmark mode is not available in this build", and the Rust side never references benchmark symbols directly. Level clamping against ZSTD_maxCLevel() happens in the bridge, where the symbol is guaranteed to exist whenever benchmarking is compiled in. -T selects the worker count, defaulting to single-threaded like the C bench path; -S (separate files) and --priority=rt remain unimplemented. Makefile updates only extend the Rust source prerequisite lists with benchfn.rs; the helpers-archive plumbing from the timefn commit already links fullbench(-lib/-dll/32) and paramgrill, the benchfn consumers among the C tests. Original C test sources are untouched. Known pre-existing issues, unchanged by this commit: tests/fullbench-lib fails to link at the base commit too (libzstd.a precedes fullbench.c in its link line), and the cli-tests basic/help.sh, compression/levels.sh, compression/golden.sh, and decompression/pass-through.sh scripts fail identically with a base-commit binary because the Rust CLI frontend is still a partial reimplementation. Test Plan: - cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets && cargo build --release - cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets; repeat tests with --no-default-features plus features compression / decompression / (none) - make -C programs zstd; ./programs/zstd -b1 -i0 lib/common/xxhash.c; ./programs/zstd -b5e6 -i0 programs/fileio.c; ./programs/zstd -b1 -i0 (synthetic); echo roundtrip via zstd | zstd -d - make -C programs zstd-small zstd-compress zstd-decompress zstd-nolegacy zstd-dictBuilder; zstd-small -b reports benchmark unavailable; compress/ decompress roundtrip across the split binaries - make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus poolTests fullbench32 fuzzer32; ./tests/fullbench -i0 (exercises Rust BMK_benchTimedFn from C); ./tests/fullbench32 -i0; ./tests/fuzzer -i1 --no-big-tests; ./tests/poolTests; make -C tests test-rust-lib-smoke - cli-tests subset: basic/version.sh, compression/basic.sh, compression/multiple-files.sh pass; failing scripts match the base commit Refs: rust/README.md
140 lines
6.9 KiB
Markdown
140 lines
6.9 KiB
Markdown
# Rust rewrite
|
|
|
|
This directory contains the in-progress Rust replacement for the zstd library
|
|
and command-line program. During the migration, the crate is built as a static
|
|
library and linked into the original C test programs. Production C translation
|
|
units become declaration-only shims as their implementations move to Rust; the
|
|
original C tests remain unchanged and provide compatibility coverage.
|
|
|
|
## Component map
|
|
|
|
The crate is organized from low-level representation helpers toward the public
|
|
zstd ABI:
|
|
|
|
- Common primitives
|
|
- `mem`, `bits`, `bitstream`, and `cpu` implement byte-order, bitstream, and
|
|
target-feature operations used by the codecs.
|
|
- `errors`, `debug`, `xxhash`, and `zstd_common` provide common exported ABI
|
|
functions and state.
|
|
- `common` contains shared frame constants and internal data types.
|
|
- Entropy coding
|
|
- `entropy_common` reads FSE normalized counts and Huffman statistics.
|
|
- `fse_decompress` builds FSE decoding tables and decodes FSE streams.
|
|
- `fse_compress` normalizes counts, writes FSE headers, builds compression
|
|
tables, and encodes FSE streams.
|
|
- `huf_compress` builds Huffman compression tables, writes table headers,
|
|
and encodes one- and four-stream Huffman payloads.
|
|
- `huf_decompress` builds Huffman decoding tables and decodes X1 and X2
|
|
Huffman streams.
|
|
- Compression primitives
|
|
- `hist` counts byte frequencies for FSE and Huffman compression.
|
|
- `zstd_presplit` chooses split points for full compression blocks.
|
|
- `zstd_compress_literals` emits raw, RLE, and Huffman literal sections
|
|
while preserving the compressor's Huffman-table repeat state.
|
|
- `zstd_compress_frame` serializes frame headers, skippable frames, and the
|
|
last empty block; it takes scalar frame parameters so the C-owned
|
|
`ZSTD_CCtx_params` layout never crosses the language boundary.
|
|
- `zstd_compress_params` owns the compression-level tables (formerly
|
|
`clevels.h`), parameter bounds, clamping, validation, table selection,
|
|
source/dictionary adjustment, and match-state/CDict size estimation.
|
|
The C integration layer keeps the public `ZSTD_*` symbols and feeds the
|
|
leaves configuration-owned scalars: the excluded-block-compressor
|
|
strategy cascade, struct sizes, and sanitizer redzone policy.
|
|
- `zstd_fast` and `zstd_double_fast` implement the single- and two-table
|
|
fast block match finders, including attached and external dictionary paths.
|
|
- `zstd_lazy` implements greedy, lazy, lazy2, and binary-tree matching,
|
|
including row-based and dictionary search variants.
|
|
- `zstd_opt_tree` maintains the binary-tree index used by optimal matching;
|
|
the dynamic-programming optimal parser itself remains in C for now.
|
|
- `zstd_ldm` implements long-distance-match parameter selection, table
|
|
maintenance, sequence generation, and sequence consumption.
|
|
- Dictionary building
|
|
- `divsufsort` constructs the suffix array that drives the legacy `ZDICT`
|
|
trainer (`ZDICT_trainFromBuffer_legacy`). The sample analysis and
|
|
dictionary assembly in `zdict.c`, `cover.c`, and `fastcover.c` remain C.
|
|
- Runtime support
|
|
- `threading` provides platform pthread wrappers required by zstd headers.
|
|
- `pool` implements the bounded worker pool used by multithreaded compression.
|
|
- Dictionary support
|
|
- `zstd_ddict` owns, loads, copies, and references decode dictionaries.
|
|
- Block decompression
|
|
- `zstd_decompress_block` decodes literal and sequence sections, maintains
|
|
FSE/Huffman repeat state, and executes compressed-block sequences.
|
|
- `zstd_decompress` owns the public decompression context, one-shot,
|
|
dictionary, parameter, and streaming state machines. Its C shim retains
|
|
configuration-dependent context allocation plus legacy and trace leaves.
|
|
- Command-line frontend
|
|
- `zstd_cli` owns the Rust parser, safety policy, and dispatch. It is built
|
|
by the separate `cli/` static-library package only for program archives,
|
|
so library builds do not acquire program-only dependencies. The C
|
|
`fileio` backend still owns file opening, safe replacement, sparse writes,
|
|
metadata, and streaming I/O.
|
|
- `timefn` provides the monotonic nanosecond clock behind `UTIL_time_t`,
|
|
and `benchfn` owns the benchmark run/timing loop (`BMK_benchFunction`,
|
|
`BMK_benchTimedFn`) used by the CLI benchmark mode and by C test tools.
|
|
Both live in the `cli/` package, but C test binaries (fullbench, fuzzer,
|
|
zstreamtest, paramgrill, ...) link a helpers-only build of that archive,
|
|
produced without the package's `cli` feature, because the parser layer
|
|
requires the C `fileio` backend that tests do not compile. Benchmark
|
|
orchestration and reporting (`benchzstd.c`) remain C, reached from the
|
|
Rust parser through the `ZSTD_NOBENCH`-gated bridge in `zstdcli.c`.
|
|
|
|
The optimal block matcher, high-level frame compression, dictionary-building
|
|
except suffix-array construction, legacy decoding callbacks, benchmark
|
|
orchestration (`benchzstd`), and the CLI
|
|
file-I/O backend are still C. They must move before the rewrite is complete.
|
|
Keeping that boundary explicit prevents a passing hybrid build from being
|
|
mistaken for the final all-Rust result.
|
|
|
|
## Compatibility boundary
|
|
|
|
The public ABI continues to come from the existing headers under `lib/`.
|
|
Exported Rust functions therefore use C layout and calling conventions. A C
|
|
source file whose implementation has moved to Rust remains in the original
|
|
makefile source list as a small shim so header configuration and platform
|
|
preprocessor behavior stay available during the transition.
|
|
|
|
The library, test, and program makefiles select an archive directory for the
|
|
active C configuration: enabled compression/decompression/dictionary-builder
|
|
modules, default or forced HUF X1/X2, and the matching Rust target for 32-bit
|
|
C binaries. The
|
|
native static archive flattens Rust object members rather than nesting a Rust
|
|
archive, while the native shared library retains all migrated Rust exports.
|
|
When the HUF mode changes, the test and program paths also rebuild cached C
|
|
outputs before linking. This prevents original C tests from using a stale or
|
|
configuration-incompatible implementation.
|
|
|
|
## Validation
|
|
|
|
Run focused Rust checks from this directory:
|
|
|
|
```sh
|
|
cargo fmt --check
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo test --all-targets
|
|
cargo build --release
|
|
```
|
|
|
|
The program-only Rust archive has its own feature matrix and should be checked
|
|
from `rust/cli` as well:
|
|
|
|
```sh
|
|
cargo clippy --all-targets -- -D warnings
|
|
cargo test --all-targets
|
|
cargo test --no-default-features --features cli,compression --all-targets
|
|
cargo test --no-default-features --features cli,decompression --all-targets
|
|
cargo test --no-default-features --all-targets
|
|
```
|
|
|
|
Then run original compatibility tests from the repository root, starting with
|
|
the narrow target for the component being migrated. For example:
|
|
|
|
```sh
|
|
make -C tests fuzzer
|
|
./tests/fuzzer -i1 --no-big-tests
|
|
make -C tests test-rust-lib-smoke
|
|
```
|
|
|
|
Broader `tests/Makefile` targets remain the authoritative integration gates as
|
|
more of the library and CLI are rewritten.
|