ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,180 Commits 17 Branches 77 Tags
45cb1f509e30a429ea6d374580037400a002bc86
Commit Graph
4 Commits
Author SHA1 Message Date
ddidderr ecf03ce020 build(rust): link Rust CLI generators into program tests 
Keep the original C translation units as declaration-only build shims while
the Rust CLI helper archive owns datagen and lorem. Track both Rust sources in
the program and test archive prerequisites, and link the helpers-only archive
into C test binaries that still provide their own file I/O and orchestration.
This preserves the existing C test harnesses while making the migrated helper
implementations the single definitions used by the binary and tests.

Test Plan:
- make -C programs zstd V=1
- make -C tests datagen V=1
- make -C tests test-rust-lib-smoke V=1
- git diff --check

Refs: programs/datagen.c, programs/lorem.c, tests/Makefile
 2026-07-12 09:19:15 +02:00
ddidderr fef5f4478a feat(rust): port benchmark loop and CLI bench mode 
Move the implementation of programs/benchfn.c into rust/src/benchfn.rs and
wire benchmark mode (-b/-e/-i) into the Rust CLI frontend, which previously
rejected those options as not yet implemented. `zstd -b1 -i0 FILE` and range
runs like `zstd -b5e6 -i0 FILE` work again, including the synthetic-sample
benchmark when no file is given.

benchfn.rs is a faithful port of the run/timing state machine:
BMK_benchFunction keeps the exact loop accounting (first-loop blockResults
and errorFn checks, dstSize summed on the first loop only, 0xE5 warm-up of
result buffers, nbLoops minimum of 1) and BMK_benchTimedFn keeps the same
convergence behavior (x10 workload growth for short runs, budget-based
nbLoops estimation, runs below half the run budget re-tried rather than
reported, best qualifying run returned). Arithmetic that C leaves to
unsigned wrap-around uses wrapping operations so debug builds cannot panic
where release C would wrap.

ABI notes: BMK_runTime_t and BMK_runOutcome_t are returned by value across
the C boundary and BMK_benchParams_t is passed by value, so all three are
repr(C) mirrors of benchfn.h; their field offsets are pinned by const
asserts in Rust and matching C static asserts in the benchfn.c shim, which
is now declaration-only. BMK_timedFnState_t stays opaque, fits the 64-byte
BMK_timedFnState_shell (compile-time checked), and is malloc/free-managed
so creation and destruction remain interchangeable with C callers.

The CLI parses -b (bench mode), -e (range end, digits attach directly,
defaulting to 0 like readU32FromChar) and -i (duration in seconds), then
dispatches through a new ZSTD_rust_cli_bench bridge in the zstdcli.c shim.
The bridge exists because benchmark availability is a C preprocessor
property (ZSTD_NOBENCH): orchestration and reporting stay in C benchzstd.c,
stripped variants (zstd-small, zstd-compress, zstd-decompress) compile the
stub branch and report "benchmark mode is not available in this build", and
the Rust side never references benchmark symbols directly. Level clamping
against ZSTD_maxCLevel() happens in the bridge, where the symbol is
guaranteed to exist whenever benchmarking is compiled in. -T selects the
worker count, defaulting to single-threaded like the C bench path; -S
(separate files) and --priority=rt remain unimplemented.

Makefile updates only extend the Rust source prerequisite lists with
benchfn.rs; the helpers-archive plumbing from the timefn commit already
links fullbench(-lib/-dll/32) and paramgrill, the benchfn consumers among
the C tests. Original C test sources are untouched.

Known pre-existing issues, unchanged by this commit: tests/fullbench-lib
fails to link at the base commit too (libzstd.a precedes fullbench.c in its
link line), and the cli-tests basic/help.sh, compression/levels.sh,
compression/golden.sh, and decompression/pass-through.sh scripts fail
identically with a base-commit binary because the Rust CLI frontend is
still a partial reimplementation.

Test Plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings
  && cargo test --all-targets && cargo build --release
- cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D
  warnings && cargo test --all-targets; repeat tests with
  --no-default-features plus features compression / decompression / (none)
- make -C programs zstd; ./programs/zstd -b1 -i0 lib/common/xxhash.c;
  ./programs/zstd -b5e6 -i0 programs/fileio.c; ./programs/zstd -b1 -i0
  (synthetic); echo roundtrip via zstd | zstd -d
- make -C programs zstd-small zstd-compress zstd-decompress zstd-nolegacy
  zstd-dictBuilder; zstd-small -b reports benchmark unavailable; compress/
  decompress roundtrip across the split binaries
- make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus
  poolTests fullbench32 fuzzer32; ./tests/fullbench -i0 (exercises Rust
  BMK_benchTimedFn from C); ./tests/fullbench32 -i0; ./tests/fuzzer -i1
  --no-big-tests; ./tests/poolTests; make -C tests test-rust-lib-smoke
- cli-tests subset: basic/version.sh, compression/basic.sh,
  compression/multiple-files.sh pass; failing scripts match the base commit

Refs: rust/README.md
 2026-07-11 14:26:21 +02:00
ddidderr 54f5c29742 feat(rust): port program timing helpers 
Move the implementation of programs/timefn.c into rust/src/timefn.rs. The
file provides the monotonic nanosecond clock (UTIL_getTime, span helpers,
UTIL_waitForNextTick, UTIL_support_MT_measurements) used by the CLI and by
several C test tools. timefn.c remains as a declaration-only shim so the
original source lists and header configuration keep working, and it pins the
ABI with static asserts: UTIL_time_t is returned by value and must stay a
plain 64-bit counter, which the Rust #[repr(C)] mirror also asserts.

Platform selection mirrors the C preprocessor structure: Windows uses
QueryPerformanceCounter, Apple targets use mach_absolute_time, and other
POSIX systems use libc clock_gettime(CLOCK_MONOTONIC). Only the unix path is
exercised by this environment; the Windows and Apple paths are written from
the C source and compile-checked logically but are untested here. The C90
clock() fallback is unreachable on Rust-supported targets, so multi-threaded
measurement support is always reported.

The symbols live in the program-only zstd-cli-rs package, keeping them out
of library builds. Linking that archive into C test binaries surfaced a
structural problem: rustc's local ThinLTO promotes internal symbols across
codegen units, so extracting the timefn object could drag in the zstd_cli
parser object, whose FIO_* externs test binaries cannot satisfy. The parser
is therefore gated behind a new additive `cli` cargo feature (default on).
Program archives build with cli,compression,decompression as before, while
tests/Makefile links a helpers-only archive (rust/target/cli-helpers) built
with --no-default-features, which contains no fileio references at all.

tests/Makefile gains build rules for the helpers archive and adds it as a
prerequisite of every binary that compiles the timefn shim: fullbench(32),
fullbench-lib, fullbench-dll, fuzzer(32), zstreamtest(32/asan/tsan/ubsan),
paramgrill, decodecorpus, and poolTests. Prerequisite order places the
archive after all C objects in `$^` link lines; the known-broken -dll
recipes filter to %.c, so they name the archive explicitly. Original C test
sources are untouched; only link inputs changed.

zstd-cli-rs now depends on libc (already used by the core crate) for
clock_gettime and the Mach timebase bindings.

Test Plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings
  && cargo test --all-targets && cargo build --release
- cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D
  warnings && cargo test --all-targets; repeat tests with
  --no-default-features plus features compression / decompression / (none)
- make -C programs zstd; roundtrip echo hello | zstd | zstd -d
- make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus
  poolTests; ./tests/fullbench -i0; ./tests/fuzzer -i1 --no-big-tests;
  ./tests/poolTests; make -C tests test-rust-lib-smoke
- verified with nm that the helpers archive member defining UTIL_getTime has
  no FIO_*/ZSTD_* undefined references

Refs: rust/README.md
 2026-07-11 14:25:32 +02:00
ddidderr fe7e24c770 feat(rust): migrate high-level runtime paths 
Move long-distance matching and high-level decompression from C shims into
Rust. The decoder now owns context, dictionary, parameter, one-shot, and
buffered streaming state while C retains allocation/configuration, legacy,
and trace leaves.

Move CLI parsing, safety policy, and dispatch into a separate Rust static
archive. Keeping it separate prevents library builds from retaining FIO
symbols, while C continues to own file opening, replacement, and I/O.
Program targets now select matching compression/decompression archives.

The remaining C boundary is intentional: high-level compression, optimal
parsing, dictionary building, legacy callbacks, and CLI file I/O still need
migration.

Test Plan:
- cargo test --all-targets (native and i686)
- cargo test --all-targets in rust/cli (native and i686)
- CLI crate compression-only and decompression-only feature tests
- native and i686 fuzzer/zstreamtest runs, plus legacy and dictionary tests
- ZSTD_C_PREDICT and ZSTD_HEAPMODE=0 fuzzer coverage
- library, dynamic-link, and program-target build/round-trip matrix

Refs: rust/README.md
 2026-07-11 09:03:41 +02:00