Move the implementation of programs/benchfn.c into rust/src/benchfn.rs and
wire benchmark mode (-b/-e/-i) into the Rust CLI frontend, which previously
rejected those options as not yet implemented. `zstd -b1 -i0 FILE` and range
runs like `zstd -b5e6 -i0 FILE` work again, including the synthetic-sample
benchmark when no file is given.
benchfn.rs is a faithful port of the run/timing state machine:
BMK_benchFunction keeps the exact loop accounting (first-loop blockResults
and errorFn checks, dstSize summed on the first loop only, 0xE5 warm-up of
result buffers, nbLoops minimum of 1) and BMK_benchTimedFn keeps the same
convergence behavior (x10 workload growth for short runs, budget-based
nbLoops estimation, runs below half the run budget re-tried rather than
reported, best qualifying run returned). Arithmetic that C leaves to
unsigned wrap-around uses wrapping operations so debug builds cannot panic
where release C would wrap.
ABI notes: BMK_runTime_t and BMK_runOutcome_t are returned by value across
the C boundary and BMK_benchParams_t is passed by value, so all three are
repr(C) mirrors of benchfn.h; their field offsets are pinned by const
asserts in Rust and matching C static asserts in the benchfn.c shim, which
is now declaration-only. BMK_timedFnState_t stays opaque, fits the 64-byte
BMK_timedFnState_shell (compile-time checked), and is malloc/free-managed
so creation and destruction remain interchangeable with C callers.
The CLI parses -b (bench mode), -e (range end, digits attach directly,
defaulting to 0 like readU32FromChar) and -i (duration in seconds), then
dispatches through a new ZSTD_rust_cli_bench bridge in the zstdcli.c shim.
The bridge exists because benchmark availability is a C preprocessor
property (ZSTD_NOBENCH): orchestration and reporting stay in C benchzstd.c,
stripped variants (zstd-small, zstd-compress, zstd-decompress) compile the
stub branch and report "benchmark mode is not available in this build", and
the Rust side never references benchmark symbols directly. Level clamping
against ZSTD_maxCLevel() happens in the bridge, where the symbol is
guaranteed to exist whenever benchmarking is compiled in. -T selects the
worker count, defaulting to single-threaded like the C bench path; -S
(separate files) and --priority=rt remain unimplemented.
Makefile updates only extend the Rust source prerequisite lists with
benchfn.rs; the helpers-archive plumbing from the timefn commit already
links fullbench(-lib/-dll/32) and paramgrill, the benchfn consumers among
the C tests. Original C test sources are untouched.
Known pre-existing issues, unchanged by this commit: tests/fullbench-lib
fails to link at the base commit too (libzstd.a precedes fullbench.c in its
link line), and the cli-tests basic/help.sh, compression/levels.sh,
compression/golden.sh, and decompression/pass-through.sh scripts fail
identically with a base-commit binary because the Rust CLI frontend is
still a partial reimplementation.
Test Plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings
&& cargo test --all-targets && cargo build --release
- cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets; repeat tests with
--no-default-features plus features compression / decompression / (none)
- make -C programs zstd; ./programs/zstd -b1 -i0 lib/common/xxhash.c;
./programs/zstd -b5e6 -i0 programs/fileio.c; ./programs/zstd -b1 -i0
(synthetic); echo roundtrip via zstd | zstd -d
- make -C programs zstd-small zstd-compress zstd-decompress zstd-nolegacy
zstd-dictBuilder; zstd-small -b reports benchmark unavailable; compress/
decompress roundtrip across the split binaries
- make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus
poolTests fullbench32 fuzzer32; ./tests/fullbench -i0 (exercises Rust
BMK_benchTimedFn from C); ./tests/fullbench32 -i0; ./tests/fuzzer -i1
--no-big-tests; ./tests/poolTests; make -C tests test-rust-lib-smoke
- cli-tests subset: basic/version.sh, compression/basic.sh,
compression/multiple-files.sh pass; failing scripts match the base commit
Refs: rust/README.md
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized
The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.
The copyright and license of `divsufsort.{h,c}` is not changed.
benchfn used to rely on mem.h, and util,
which in turn relied on platform.h.
Using benchfn outside of zstd required to bring all these dependencies.
Now, dependency is reduced to timefn only.
This required to create a separate timefn from util,
and rewrite benchfn and timefn to no longer need mem.h.
Separating timefn from util has a wide effect accross the code base,
as usage of time functions is widespread.
A lot of build scripts had to be updated to also include timefn.
as suggested in #1441.
generally U32 and unsigned are the same thing,
except when they are not ...
case : 32-bit compilation for MIPS (uint32_t == unsigned long)
A vast majority of transformation consists in transforming U32 into unsigned.
In rare cases, it's the other way around (typically for internal code, such as seeds).
Among a few issues this patches solves :
- some parameters were declared with type `unsigned` in *.h,
but with type `U32` in their implementation *.c .
- some parameters have type unsigned*,
but the caller user a pointer to U32 instead.
These fixes are useful.
However, the bulk of changes is about %u formating,
which requires unsigned type,
but generally receives U32 values instead,
often just for brevity (U32 is shorter than unsigned).
These changes are generally minor, or even annoying.
As a consequence, the amount of code changed is larger than I would expect for such a patch.
Testing is also a pain :
it requires manually modifying `mem.h`,
in order to lie about `U32`
and force it to be an `unsigned long` typically.
On a 64-bit system, this will break the equivalence unsigned == U32.
Unfortunately, it will also break a few static_assert(), controlling structure sizes.
So it also requires modifying `debug.h` to make `static_assert()` a noop.
And then reverting these changes.
So it's inconvenient, and as a consequence,
this property is currently not checked during CI tests.
Therefore, these problems can emerge again in the future.
I wonder if it is worth ensuring proper distinction of U32 != unsigned in CI tests.
It's another restriction for coding, adding more frustration during merge tests,
since most platforms don't need this distinction (hence contributor will not see it),
and while this can matter in theory, the number of platforms impacted seems minimal.
Thoughts ?
ddidderr
Christoph Grüninger