Record the Rust-owned stable-buffer fallback orchestration and shared-output
multi-file scheduler in the migration boundary documentation while retaining
the C-owned private context, resource, and callback responsibilities.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
17cfedd56 and 035001b76
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that public sequence API orchestration and scalar CLI adaptive
decisions are Rust-owned while private codec/context state and mutation
callbacks remain in C.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by
3b989ab3e and b04893799
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that Rust owns single-threaded and MT stream-initialization policy,
including the ordered setup decisions and scalar projections, while C retains
private contexts and mutation callbacks.
Test Plan:
- git diff --cached --check
- documentation-only boundary update; code validation is covered by the
preceding stream-initialization commits
Commit is intentionally unsigned because GPG pinentry hangs in this
non-interactive environment.
Document that Rust now owns CCtx dictionary/prefix attachment dispatch and the
multithreaded compression-job stage sequence, while C retains reset policy,
private context and worker state, dictionary-content loading, and codec
callbacks.
Test Plan:
- Documentation-only change; code validation was completed before this commit.
- Rust library all-target tests: 517 passed; legacy feature matrix: 572 passed.
- Native CLI tests (41), test-zstd, fuzzer (319), zstream (152 + 297), and
decode corpus (1,647) passed under the serial 40 GiB virtual-memory cap.
- Commit is intentionally unsigned because configured GPG pinentry was
unavailable and hung during the signing attempt.
Update the Rust migration map after the latest compression and CLI seams
moved. The fileio summary now distinguishes Rust-owned zstd stream
orchestration from C-owned codec, diagnostics, and adaptive-policy callbacks.
The remaining-boundary summary records that dictionary dispatch and the CLI
zstd compression loop have moved, while CDict lifecycle, private dictionary
content loading, and private compressor operations remain in C.
Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no additional build or test run was needed.
GPG signing was attempted but unavailable because no pinentry process was
available; this repository's preceding commits are unsigned, so this commit
uses the explicit unsigned fallback.
Document that Rust now owns sequence-store construction, block post-build
policy, public sequence conversion, and optional-format decompression loops.
Keep the remaining private C callback and dictionary/CLI boundaries explicit so
the hybrid state is not mistaken for a completed all-Rust rewrite.
Test Plan:
- git diff --check (passed)
Update the Rust component map to describe the newly migrated single-threaded
stream state machine and multithreaded scheduler/flush policy. Correct the CLI
boundary so it no longer claims that the optional-format decompression loops
are Rust-owned, and list the remaining sequence, dictionary, and CLI leaves
that still block an all-Rust build.
Test Plan:
- `git diff --cached --check` -- passed.
- Documentation-only change; no code tests required.
Document that the Rust CLI now owns the gzip, xz/lzma, and LZ4 codec leaves as
well as MT job creation, while the generic frame-compression state machine and
remaining MT scheduling/outer-flush orchestration remain C-owned. Keep the
hybrid boundary explicit so compatibility-suite success is not mistaken for
completion of the full rewrite.
Test Plan:
- git diff --cached --check
- reviewed rust/README.md against the committed Rust/C ownership boundary
Document that the frame epilogue, produced-output flush policy, and gzip codec
leaf now execute in Rust. Narrow the remaining C-owned list to the generic
compression state machine, MT scheduling/outer flush orchestration, and the
xz/lzma/lz4 codec leaves so the README remains an accurate progress boundary.
Test Plan:
- git diff --cached --check
The migration map still described the optimal parser and benchmark orchestration
as C-owned, and its remaining-work paragraph did not distinguish the newly
ported high-level leaves from the still-private frame and MT orchestration.
That made the README under-report Rust ownership and overstate some remaining
C work.
Update the component map to identify `zstd_opt` price/parse ownership and Rust
benchmark orchestration, and describe the current C boundary as the remaining
frame setup/flush scheduling and format-specific codec leaves. Keep the
hybrid-build warning explicit until those larger orchestration layers move.
Test Plan:
- `git diff --cached --check` -- passed.
- Rust library and CLI Clippy default, benches, and tests -- passed before staging.
- Native library, program, fuzz, CLI, and upstream test matrices -- passed before staging.
Move the post-sequence-store body of ZSTD_compressBlock_internal behind a
small C/Rust state projection. C continues to build the sequence store and
handle the no-compress and sequence-producer error paths, while Rust now owns
sequence collection, entropy emission, the legacy non-first-frame RLE gate,
compressed-block confirmation, and offcode repeat cleanup. Remove the C
wrappers that became dead after those leaves moved behind the Rust body.
The first integration run exposed that leaving the old C finalization label in
place confirmed compressed block state twice, undoing Rust's pointer swap and
breaking a later sparse-file checksum. The C wrapper now returns directly for
the Rust-owned path and retains only its C-owned no-compress cleanup.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1
- cargo check --manifest-path rust/Cargo.toml --lib
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-zstd
- make -B -C tests -j2 test-cli-tests
- ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer
- make -B -C tests/fuzz -j2 all
- make -B -C tests/fuzz -j2 sequence_compression_api
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings (pre-existing manual_repeat_n failure)
- cargo clippy --manifest-path rust/Cargo.toml --benches -- -D warnings (pre-existing manual_repeat_n failure)
Move the post-split partition loop out of zstd_compress.c while keeping the
private CCtx, matchfinder sequence-store construction, and split discovery in
C. Rust now receives a layout-asserted projection containing sequence-store
views, block-state slots, workspace and scalar policy, then mirrors the C
loop's dRep/cRep histories, final-literal accounting, repeated single-block
serialization, and final dRep publication.
Remove the obsolete C sequence-store/count/chunk wrappers and the debug-only
C size-estimation path that was coupled to the old loop. Add a focused Rust
fixture covering partition payloads and final literals, and document the new
ownership boundary for the split search and emission paths.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --lib -- --test-threads=1 (448 passed)
- cargo clippy --manifest-path rust/Cargo.toml --lib -- -D warnings (passed)
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings (passed)
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check (passed)
- cargo clippy --tests/--benches remain blocked only by the pre-existing manual_repeat_n lint in one_shot_promotes_nonfirst_rle_blocks
- make -B -C lib -j2 lib (passed)
- make -B -C tests -j2 test-zstd (passed)
- make -B -C tests -j2 test-cli-tests (41 passed)
- ZSTREAM_TESTTIME=-T2s make -B -C tests -j2 test-zstream (passed)
- FUZZERTEST=-T5s make -B -C tests -j2 test-fuzzer (252 passed)
- make -B -C tests/fuzz -j2 all and sequence_compression_api (passed)
The component map still described decoder context allocation as a C-owned
responsibility after the allocation and custom-memory bridges moved into Rust.
Update the boundary description to distinguish the C-defined context layout and
configuration-sensitive platform, legacy, and trace leaves from Rust-owned
storage allocation and allocator dispatch.
Test Plan:
- `git diff --cached --check` -- passed
Document that the Rust dictionary-builder modules now own sample analysis,
training, and dictionary assembly, with the original C files retained only as
ABI/build shims. Remove the stale statement that only suffix-array
construction has migrated and keep the remaining C boundary focused on
compression orchestration, benchmark reporting, and format-specific CLI I/O.
Test Plan:
- git diff --cached --check
- make -C tests -j2 test-zstd (passed before this documentation-only commit)
Update the Rust rewrite documentation to match the current implementation
boundary. The legacy decoder set is complete, and Rust now owns several
CLI file-I/O policy and resource leaves while C retains format dispatch,
metadata, and remaining orchestration. Keep the stated boundary explicit so
passing hybrid tests do not imply that the full rewrite is finished.
Test Plan:
- git diff --cached --check
Continue the incremental zstd_compress.c migration with its sequence
statistics and seqStore entropy-compression layer. The following now
live in rust/src/zstd_compress_stats.rs:
- ZSTD_seqToCodes(), exported under its original name because the C
dictionary builder (zdict.c) and decodecorpus link against it,
- ZSTD_buildSequencesStatistics() and its dummy variant, whose result
struct no longer crosses the language boundary,
- ZSTD_entropyCompressSeqStore_internal(), _wExtLitBuffer(), and
ZSTD_entropyCompressSeqStore(),
- ZSTD_buildBlockEntropyStats() with its literals/sequences helpers,
- ZSTD_copyBlockSequences() and the ZSTD_updateRep() rules it shares
with the superblock writer.
Boundary: ZSTD_CCtx and ZSTD_CCtx_params stay private to C. These
paths read exactly two parameter fields, so the C shims keep the
original static/exported function names and forward the strategy and
ZSTD_literalsCompressionIsDisabled() as int scalars alongside the
seqStore and entropy-table leaves. The leaf layouts (SeqDef,
SeqStore_t, ZSTD_hufCTables_t, ZSTD_fseCTables_t, entropy metadata,
SeqCollector, ZSTD_Sequence) are pinned by compile-time asserts in
zstd_compress.c and by both-pointer-width layout tests in Rust.
ZSTD_buildSeqStore, block dispatch/splitting, and the block-size
estimation helpers remain C for a later slice.
The superblock module previously round-tripped through the C export of
ZSTD_buildBlockEntropyStats and mirrored the entropy leaf structs
privately. It now calls the crate-internal builder directly, and the
shared struct definitions moved to zstd_compress_stats; consequently
ZSTD_rust_compressSuperBlock() takes the two parameter scalars instead
of an opaque ZSTD_CCtx_params pointer, extracted by its C shim. Its
layout and repcode tests moved with the definitions.
One C helper family gets no shim: the static
ZSTD_entropyCompressSeqStore_wExtLitBuffer() had a single caller and
was folded into the Rust implementation.
Byte-identity was verified against the pre-change compressor: COPYING,
datagen -g5000000 -s7, and datagen -g300000 -s21 -P90, each at levels
1/3/9/19 and --fast=5, plus a superblock-heavy pass at level 19 with
--target-compressed-block-size=1024; all 18 frames are byte-identical,
covering the repeat-mode state machine, longOffsets, RLE/raw fallback,
and dstSize_tooSmall paths through the block splitter and superblock.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (131 tests: new reference
vectors for seqToCodes, RLE table headers, empty-seqStore repeat
copies, literal-stats type selection, and repcode resolution in
copyBlockSequences)
- cargo build --release --no-default-features --features compression
and --features decompression
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests (covers
ZSTD_generateSequences and ZDICT training over the Rust seqToCodes)
- make -C tests test-rust-lib-smoke
- ./tests/zstreamtest -i1 and ./tests/zstreamtest --newapi -t1 -i1
- ./tests/decodecorpus -t -T1s (decodecorpus links Rust seqToCodes)
- /tmp/ref vs /tmp/got frame diff as described above: byte-identical
Port lib/legacy/zstd_v01.c (the frozen zstd v0.1 decoder) to
rust/src/legacy/zstd_v01.rs as the first legacy-format port on the new
scaffolding, and reduce the C file to a declaration-only shim that
keeps its header includes for configuration and platform preprocessor
behavior.
Frozen-decoder policy: zstd_v01.c embeds its own v0.1-era FSE and
Huff0 snapshot, distinct from every other release. The Rust port is a
line-by-line translation with the same table layouts (FSE_DTable as a
u32 header word plus packed newState/symbol/nbBits entries, the Huff0
u16 DTable with byte/nbBits pairs), the same arithmetic including
wrap-around and pointer-comparison quirks (e.g. the offset-vs-base
address check in ZSTD_execSequence), the same internal FSE error space
(size_t)-1..-7, and the same public ZSTD error codes. It reuses no
modern Rust entropy module; its only crate dependency is `errors`,
matching the C file's error_private.h include. The 32-bit-only reload
points are kept as compile-time conditions on usize::BITS.
Symbol takeover boundary: all nine ZSTDv01_* entry points from
zstd_v01.h now come from Rust as context-free #[no_mangle] extern "C"
functions (isError, decompress, decompressDCtx,
findFrameSizeInfoLegacy, createDCtx, freeDCtx, resetDCtx,
nextSrcSizeToDecompress, decompressContinue). zstd_legacy.h only uses
the first four for v0.1; streaming for v0.1-v0.3 intentionally returns
version_unsupported there, unchanged. The ZSTDv01_Dctx struct
definition moves entirely into Rust: C code only ever holds an opaque
pointer (zstd_v01.h forward-declares the type), and the context is
malloc/free-allocated exactly like the C version so create/free may
pair across the language boundary.
Byte-identity verification against the pristine pre-migration C build
(f8745da6, pure C, ZSTD_LEGACY_SUPPORT=1):
- Real v0.1 frames were generated by building the v0.1.0 git tag and
compressing text, random, and 426 KB multi-block inputs. A one-shot
ZSTD_decompress harness linked once against the pristine C libzstd.a
and once against the Rust-backed libzstd.a produced bit-identical
outputs for all frames.
- A direct ZSTDv01_* probe (one-shot decode, dst-too-small, truncated
input, bad magic, findFrameSizeInfoLegacy, and the streaming
continue loop) printed identical results, including exact error
codes (-70 dstSize_tooSmall, -72 srcSize_wrong, -10 prefix_unknown)
and identical dBound values.
- zstd -l -v on v0.1 files matches the pristine binary; CLI streaming
decode of v0.1 fails with the same "Version not supported" in both,
by design of zstd_legacy.h.
Unit tests embed three v0.1.0-generated fixtures (entropy-coded,
raw-block, and four-block frames) plus the truncation, bad-magic,
small-destination, and streaming-API cases, all asserting the exact C
error codes above. Note that `make -C tests test-legacy` only covers
v0.4+ frames, so the embedded fixtures and the harness comparison are
the actual v0.1 coverage.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
--features legacy-v01 -- -D warnings && cargo test --all-targets
--features legacy-v01 (127 tests, 9 for v0.1)
- cargo clippy/test --no-default-features --features
decompression,legacy-v01 (module builds standalone)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests, also with
ZSTD_LEGACY_SUPPORT=1 (mixed Rust v0.1 + C v0.2-0.7 link)
- make -C tests test-rust-lib-smoke && make -C tests test-legacy
- make -C programs zstd (default and ZSTD_LEGACY_SUPPORT=1); nm shows
the nine ZSTDv01_* symbols provided by Rust at level 1
- make -C lib libzstd.a ZSTD_LEGACY_SUPPORT=0 (no legacy symbols) and
meson -Dlegacy_level=1 shared library exporting all nine
The legacy decoders (lib/legacy/zstd_v01.c .. zstd_v07.c) are next in
the Rust migration. Each of those files is a frozen snapshot of the
FSE/Huff0 entropy coders and frame logic of one historical release, so
their ports must not reuse the modern Rust entropy modules and must not
share code with each other: outputs and error codes have to stay
byte-identical to the frozen C forever. This commit installs the
build-system scaffolding so seven per-version ports can land
independently, each adding only its own module file plus a one-line
registration in rust/src/legacy/mod.rs.
Cargo grows features legacy-v01 .. legacy-v07. They are never default
features: the C build defaults differ per build system, so each build
system passes the list explicitly, derived from its own legacy
configuration:
- lib/Makefile and programs/Makefile map ZSTD_LEGACY_SUPPORT=N to the
features for versions N..7 (0 disables legacy), mirroring the
ZSTD_LEGACY_FILES selection in lib/libzstd.mk.
- tests/Makefile always enables all seven features because its
ZSTDLEGACY_FILES wildcard compiles every lib/legacy/*.c regardless of
the dispatch level.
- build/meson maps legacy_level exactly like the makefiles; build/cmake
enables all seven whenever ZSTD_LEGACY_SUPPORT is ON because it
always compiles all seven C files (ZSTD_LEGACY_LEVEL only selects the
C dispatch).
Every build system also encodes the legacy selection in the Rust target
directory name (e.g. c1-d1-default-legacy5), for the same reason the
HUF mode is encoded there: a cached archive built for one configuration
must never be linked into a build expecting another. In tests/Makefile
the legacy level additionally flows into the existing HUF C-mode stamp,
so the flat C test objects (which bake -DZSTD_LEGACY_SUPPORT into the
dispatch) are rebuilt whenever the level changes. In programs/Makefile
the compress-only, decompress-only, and CLI archives keep
level-independent directories (RUST_HUF_MODE) because they are only
linked into ZSTD_LEGACY_SUPPORT=0 program variants and carry no legacy
features.
A feature whose version has not been ported yet gates nothing: the
module registration in rust/src/legacy/mod.rs is added by each port,
so enabling e.g. legacy-v05 today simply leaves that decoder in C.
This is what makes mixed C/Rust legacy levels link cleanly while the
seven ports land in any order.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
-- -D warnings && cargo test --all-targets
- cargo clippy with --no-default-features --features
decompression,legacy-v01 and with all seven legacy features
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke; make -C tests test-legacy
- make -C lib libzstd.a with ZSTD_LEGACY_SUPPORT=0, 1 and default (5)
- cmake configure and meson setup (including -Dlegacy_level=1) emit the
expected --features lists and legacy-suffixed target directories
Move the implementation of programs/benchfn.c into rust/src/benchfn.rs and
wire benchmark mode (-b/-e/-i) into the Rust CLI frontend, which previously
rejected those options as not yet implemented. `zstd -b1 -i0 FILE` and range
runs like `zstd -b5e6 -i0 FILE` work again, including the synthetic-sample
benchmark when no file is given.
benchfn.rs is a faithful port of the run/timing state machine:
BMK_benchFunction keeps the exact loop accounting (first-loop blockResults
and errorFn checks, dstSize summed on the first loop only, 0xE5 warm-up of
result buffers, nbLoops minimum of 1) and BMK_benchTimedFn keeps the same
convergence behavior (x10 workload growth for short runs, budget-based
nbLoops estimation, runs below half the run budget re-tried rather than
reported, best qualifying run returned). Arithmetic that C leaves to
unsigned wrap-around uses wrapping operations so debug builds cannot panic
where release C would wrap.
ABI notes: BMK_runTime_t and BMK_runOutcome_t are returned by value across
the C boundary and BMK_benchParams_t is passed by value, so all three are
repr(C) mirrors of benchfn.h; their field offsets are pinned by const
asserts in Rust and matching C static asserts in the benchfn.c shim, which
is now declaration-only. BMK_timedFnState_t stays opaque, fits the 64-byte
BMK_timedFnState_shell (compile-time checked), and is malloc/free-managed
so creation and destruction remain interchangeable with C callers.
The CLI parses -b (bench mode), -e (range end, digits attach directly,
defaulting to 0 like readU32FromChar) and -i (duration in seconds), then
dispatches through a new ZSTD_rust_cli_bench bridge in the zstdcli.c shim.
The bridge exists because benchmark availability is a C preprocessor
property (ZSTD_NOBENCH): orchestration and reporting stay in C benchzstd.c,
stripped variants (zstd-small, zstd-compress, zstd-decompress) compile the
stub branch and report "benchmark mode is not available in this build", and
the Rust side never references benchmark symbols directly. Level clamping
against ZSTD_maxCLevel() happens in the bridge, where the symbol is
guaranteed to exist whenever benchmarking is compiled in. -T selects the
worker count, defaulting to single-threaded like the C bench path; -S
(separate files) and --priority=rt remain unimplemented.
Makefile updates only extend the Rust source prerequisite lists with
benchfn.rs; the helpers-archive plumbing from the timefn commit already
links fullbench(-lib/-dll/32) and paramgrill, the benchfn consumers among
the C tests. Original C test sources are untouched.
Known pre-existing issues, unchanged by this commit: tests/fullbench-lib
fails to link at the base commit too (libzstd.a precedes fullbench.c in its
link line), and the cli-tests basic/help.sh, compression/levels.sh,
compression/golden.sh, and decompression/pass-through.sh scripts fail
identically with a base-commit binary because the Rust CLI frontend is
still a partial reimplementation.
Test Plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings
&& cargo test --all-targets && cargo build --release
- cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets; repeat tests with
--no-default-features plus features compression / decompression / (none)
- make -C programs zstd; ./programs/zstd -b1 -i0 lib/common/xxhash.c;
./programs/zstd -b5e6 -i0 programs/fileio.c; ./programs/zstd -b1 -i0
(synthetic); echo roundtrip via zstd | zstd -d
- make -C programs zstd-small zstd-compress zstd-decompress zstd-nolegacy
zstd-dictBuilder; zstd-small -b reports benchmark unavailable; compress/
decompress roundtrip across the split binaries
- make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus
poolTests fullbench32 fuzzer32; ./tests/fullbench -i0 (exercises Rust
BMK_benchTimedFn from C); ./tests/fullbench32 -i0; ./tests/fuzzer -i1
--no-big-tests; ./tests/poolTests; make -C tests test-rust-lib-smoke
- cli-tests subset: basic/version.sh, compression/basic.sh,
compression/multiple-files.sh pass; failing scripts match the base commit
Refs: rust/README.md
Move the implementation of programs/timefn.c into rust/src/timefn.rs. The
file provides the monotonic nanosecond clock (UTIL_getTime, span helpers,
UTIL_waitForNextTick, UTIL_support_MT_measurements) used by the CLI and by
several C test tools. timefn.c remains as a declaration-only shim so the
original source lists and header configuration keep working, and it pins the
ABI with static asserts: UTIL_time_t is returned by value and must stay a
plain 64-bit counter, which the Rust #[repr(C)] mirror also asserts.
Platform selection mirrors the C preprocessor structure: Windows uses
QueryPerformanceCounter, Apple targets use mach_absolute_time, and other
POSIX systems use libc clock_gettime(CLOCK_MONOTONIC). Only the unix path is
exercised by this environment; the Windows and Apple paths are written from
the C source and compile-checked logically but are untested here. The C90
clock() fallback is unreachable on Rust-supported targets, so multi-threaded
measurement support is always reported.
The symbols live in the program-only zstd-cli-rs package, keeping them out
of library builds. Linking that archive into C test binaries surfaced a
structural problem: rustc's local ThinLTO promotes internal symbols across
codegen units, so extracting the timefn object could drag in the zstd_cli
parser object, whose FIO_* externs test binaries cannot satisfy. The parser
is therefore gated behind a new additive `cli` cargo feature (default on).
Program archives build with cli,compression,decompression as before, while
tests/Makefile links a helpers-only archive (rust/target/cli-helpers) built
with --no-default-features, which contains no fileio references at all.
tests/Makefile gains build rules for the helpers archive and adds it as a
prerequisite of every binary that compiles the timefn shim: fullbench(32),
fullbench-lib, fullbench-dll, fuzzer(32), zstreamtest(32/asan/tsan/ubsan),
paramgrill, decodecorpus, and poolTests. Prerequisite order places the
archive after all C objects in `$^` link lines; the known-broken -dll
recipes filter to %.c, so they name the archive explicitly. Original C test
sources are untouched; only link inputs changed.
zstd-cli-rs now depends on libc (already used by the core crate) for
clock_gettime and the Mach timebase bindings.
Test Plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings
&& cargo test --all-targets && cargo build --release
- cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets; repeat tests with
--no-default-features plus features compression / decompression / (none)
- make -C programs zstd; roundtrip echo hello | zstd | zstd -d
- make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus
poolTests; ./tests/fullbench -i0; ./tests/fuzzer -i1 --no-big-tests;
./tests/poolTests; make -C tests test-rust-lib-smoke
- verified with nm that the helpers archive member defining UTIL_getTime has
no FIO_*/ZSTD_* undefined references
Refs: rust/README.md
Move the dictionary builder's suffix-array construction from
lib/dictBuilder/divsufsort.c to rust/src/divsufsort.rs, the first
dictBuilder module to migrate. It rides on the dict-builder cargo
feature dimension introduced by the previous commit.
divsufsort() is a self-contained algorithm (two-stage sort of type-B*
substrings via sssort, rank refinement via trsort, then induced sorting
of the full array), so its context-free signature allows a direct symbol
takeover: the Rust #[no_mangle] export provides the existing `divsufsort`
symbol and the C file becomes a declaration-only shim that just keeps the
header's prototypes in the build. Only divsufsort() moved; divbwt() has
no callers anywhere in zstd, so it is now declaration-only, keeping the
Rust export surface minimal. The unused openMP parameter is retained for
signature compatibility (zstd never defines LIBBSC_OPENMP).
The port is a mechanical translation of the exact configuration zstd
compiles: ALPHABET_SIZE=256, SS_INSERTIONSORT_THRESHOLD=8,
SS_BLOCKSIZE=1024, SS_MISORT_STACKSIZE=16, SS_SMERGE_STACKSIZE=32,
TR_STACKSIZE=64. Every C `int*` cursor into the SA buffer becomes an
`isize` index into a single `&mut [i32]` slice, preserving the pointer
arithmetic (including transient one-before-the-range cursors and the
bitwise-complement rank marking) while staying bounds-checked; all value
arithmetic keeps C int semantics. The C -1/-2 error results are
preserved, with Vec::try_reserve_exact standing in for the bucket-array
malloc failure path. Behavior is bit-identical by construction and by
measurement (see test plan); runtime on an 11 MB training buffer is
within ~5% of the C build end-to-end.
Users see no behavioral change: dictionaries trained through
ZDICT_trainFromBuffer_legacy() are byte-identical to the C build. The
only external difference is that the never-called `divbwt` symbol is no
longer defined in the library.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
-- -D warnings && cargo test --all-targets && cargo build --release
(125 tests pass; new unit tests cover empty/one/two-byte inputs,
all-equal bytes, an exact hand-computed "abracadabra" SA, and
fixed-seed LCG buffers at 256-, 4-, and 2-symbol alphabets verified
against a naive reference sort plus permutation/sorted invariants)
- Feature matrix: cargo build --release --no-default-features
--features compression,decompression (and decompression-only,
compression-only, compression,dict-builder); `divsufsort` is exported
only when dict-builder is enabled
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests (includes
ZDICT training tests): pass
- make -C tests test-rust-lib-smoke: pass
- make -C tests test-invalidDictionaries: pass
- make -C programs zstd zstd-dictBuilder zstd-small zstd-compress
zstd-decompress: build; compress/decompress round-trip verified
- Byte-identity vs pristine C build (commit 959e4852): a harness calling
ZDICT_trainFromBuffer_legacy() (the only zstd path reaching
divsufsort) and divsufsort() directly, linked against both libzstd.a
builds, produces byte-identical dictionaries (80,288 B and full
112,640 B capacity) and byte-identical suffix arrays on a 1 MB source
set and an 11 MB binary/repetitive set; a differential driver over 148
random and structured buffers (sizes 3..6000, alphabets 1..256,
Fibonacci word, sawtooth, 6 KB near-constant) shows zero mismatches.
The CLI --train path could not be exercised because the Rust CLI
frontend rejects --train in both the pristine and ported builds (a
pre-existing migration gap unrelated to this change).
The dictionary-builder sources (lib/dictBuilder) are about to start moving
to Rust, beginning with divsufsort. The Rust crate previously only modeled
the compression/decompression module split plus the forced-HUF decoder
modes, so no build could express "this C configuration includes (or
excludes) dictBuilder" to Cargo. Without that, a Rust archive could carry
dictBuilder modules into a build whose C side disabled them, or worse,
omit a migrated implementation from a build whose C shims require it.
Add a `dict-builder` cargo feature and thread it through every build that
consumes the Rust static archive, mirroring exactly how each build system
already gates the dictBuilder C sources:
- rust/Cargo.toml: new `dict-builder` feature, included in the default
set because the C library builds dictBuilder by default
(ZSTD_LIB_DICTBUILDER ?= 1). The feature is empty until the first
dictBuilder module lands.
- lib/Makefile: RUST_CARGO_FEATURES gains dict-builder when
ZSTD_LIB_DICTBUILDER is enabled, following the existing
ZSTD_LIB_COMPRESSION/ZSTD_LIB_DECOMPRESSION pattern. The archive
directory naming grows a matching `b<0|1>` dimension
(c1-d1-b1-default etc.) so differently configured archives never
collide; the repeated config prefix is factored into
RUST_MODULE_CONFIG.
- programs/Makefile: the full-featured archives now request
compression,decompression,dict-builder (equal to the default set, so
the target directory stays shared with tests). The partial-library
variants gain the `b0` name dimension, and zstd-dictBuilder gets its
own lib-c1-d0-b1 archive because it compiles the dictBuilder C sources
without decompression; it previously shared the compression-only
archive, which will lack the migrated dictBuilder symbols.
- tests/Makefile: no flag change needed since tests use the crate default
feature set; a comment now records that dict-builder arrives that way.
- build/cmake/lib/CMakeLists.txt: ZSTD_BUILD_DICTBUILDER now adds the
dict-builder feature and a `b<0|1>` component in the Rust build-config
directory name, in lockstep with the DictBuilderSources gating.
- build/meson/lib/meson.build: meson compiles the dictBuilder sources
unconditionally, so the feature list and config name gain dict-builder
unconditionally (c1-d1-b1-<huf-mode>).
The `dict-builder` feature deliberately does not imply `compression`.
lib/Makefile forces ZSTD_LIB_DICTBUILDER=0 when compression is disabled,
but CMake does not couple the two options, so encoding the C-side
constraint in Cargo would make the Rust archive diverge from the C source
list in that (already unsupported) CMake configuration.
Test plan:
- cd rust && cargo build --release
- cargo build --release --no-default-features \
--features compression,decompression
- cargo build --release --no-default-features \
--features compression,dict-builder
- Full validation (fuzzer, smoke tests, dictionary byte-identity) runs
with the follow-up commit that ports divsufsort onto this scaffolding.
Move the context-free compression-parameter logic of zstd_compress.c to
rust/src/zstd_compress_params.rs: the compression-level tables (formerly
clevels.h), parameter bounds/checking/clamping, cycle log, level-table
selection, source/dictionary parameter adjustment, default frame
parameters, and the match-state/CDict size estimators.
The boundary follows the module's design notes: Rust owns only leaves
whose behavior is independent of C preprocessor configuration. C keeps
the public ZSTD_* symbols and feeds the leaves everything that is
configuration-owned as explicit scalars:
- The ZSTD_EXCLUDE_*_BLOCK_COMPRESSOR strategy cascade stays in
ZSTD_adjustCParams_internal() ahead of the Rust adjustment leaf, so
reduced builds keep their fallback policy.
- Workspace estimation receives struct sizes (ZSTD_CDict, ZSTD_match_t,
ZSTD_optimal_t), HUF workspace size, and the sanitizer redzone size,
because those depend on private layouts and ASAN configuration.
- ZSTD_cParam_getBounds() forwards only the compression level and the
seven core parameters; all other parameter bounds remain C.
- Frozen private constants the leaves hardcode (short-cache and row-hash
tag widths, MaxML/MaxLL/MaxOff/Litbits, ZSTD_OPT_SIZE, cwksp alignment)
are pinned by ZSTD_STATIC_ASSERTs at the C call sites.
Two latent 32-bit bugs in the previously unwired module were fixed
before integration: dictAndWindowLog's max window size and the
window-resize threshold were hardcoded to the 64-bit constants
(1<<31, 1<<30) instead of deriving from ZSTD_WINDOWLOG_MAX, which is
30 on 32-bit targets.
clevels.h is no longer included anywhere but stays in-tree as the
reference for mechanical comparison against the Rust table.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (125 tests)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke
- Byte-identity: zstd CLI frames for COPYING, a 250 KB C source, and a
5 MB datagen sample at levels 1/3/9/19/--fast=5 are identical between
this change and its parent commit.
Continue the incremental zstd_compress.c migration with its frame
serialization leaves: ZSTD_writeFrameHeader(), the public
ZSTD_writeSkippableFrame() ABI, and ZSTD_writeLastEmptyBlock() now live
in rust/src/zstd_compress_frame.rs.
ZSTD_writeFrameHeader() reads five fields out of ZSTD_CCtx_params, whose
layout is private to zstd_compress.c and sensitive to build
configuration. Rather than mirror that structure in Rust, the C
function keeps its original static signature and forwards the scalar
fields to ZSTD_rust_writeFrameHeader(), so no parameter-structure layout
crosses the language boundary. The other two functions take only
pointer/size arguments and are exported directly, replacing their C
bodies outright.
Behavior differences are limited to hardening in release builds: the
Rust leaf clamps a window-size shift that C would leave undefined for a
malformed windowLog, and ZSTD_writeSkippableFrame() rejects a srcSize +
header overflow instead of comparing against a wrapped sum. Both paths
are unreachable through validated callers, so compressed output is
byte-identical.
Remaining zstd_compress.c work: parameter selection and validation,
context lifecycle, block dispatch, dictionary loading, and the streaming
state machine.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (118 tests, includes new frame
header/skippable/last-block reference vectors)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke
Move long-distance matching and high-level decompression from C shims into
Rust. The decoder now owns context, dictionary, parameter, one-shot, and
buffered streaming state while C retains allocation/configuration, legacy,
and trace leaves.
Move CLI parsing, safety policy, and dispatch into a separate Rust static
archive. Keeping it separate prevents library builds from retaining FIO
symbols, while C continues to own file opening, replacement, and I/O.
Program targets now select matching compression/decompression archives.
The remaining C boundary is intentional: high-level compression, optimal
parsing, dictionary building, legacy callbacks, and CLI file I/O still need
migration.
Test Plan:
- cargo test --all-targets (native and i686)
- cargo test --all-targets in rust/cli (native and i686)
- CLI crate compression-only and decompression-only feature tests
- native and i686 fuzzer/zstreamtest runs, plus legacy and dictionary tests
- ZSTD_C_PREDICT and ZSTD_HEAPMODE=0 fuzzer coverage
- library, dynamic-link, and program-target build/round-trip matrix
Refs: rust/README.md
Move the binary-tree table maintenance shared by optimal compression strategies
into Rust. A small C projection preserves the private match-state boundary;
the dynamic-programming parser and match selection remain C-owned for now.
The Rust implementation covers prefix and external-dictionary tree updates and
preserves the optional C predictor path. The component map now distinguishes
this migrated tree layer from the remaining optimal parser.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- make -B -C tests fuzzer32 MOREFLAGS=-DZSTD_C_PREDICT
- ./tests/fuzzer32 -s5346 -i1 --no-big-tests
- byte-compare C and Rust-control CLI output at levels 13, 16, 19, and 22
for random, patterned, zero, and dictionary inputs with and without
ZSTD_C_PREDICT
Refs: rust/README.md
Move literal decoding, sequence-table construction, FSE sequence decoding, and
sequence execution into Rust. The C shim retains ownership of the configured
decoder context and passes only the leaf state needed by the block codec.
Correct the two high offset-code bases while porting the tables. Their prior
values made valid large-window streams decode as corrupted data; the new unit
test fixes the exact values and a native zstream regression exercises them.
High-level frame and streaming context control remains C for now.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- timeout 20s stdbuf -oL ./tests/poolTests
Refs: rust/README.md
Move the two-table fast match finder into Rust while retaining a small C
projection of the private match-state. The Rust implementation handles normal,
attached-dictionary, and external-dictionary matching without exposing the
full C context ABI. The C entry points and build-time exclusion guards remain
unchanged for configured consumers.
Document the migrated matcher in the Rust component map and narrow the
remaining-C boundary accordingly.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- timeout 20s stdbuf -oL ./tests/poolTests
Refs: rust/README.md
Move raw, RLE, and Huffman literal-section encoding into the Rust
compatibility archive. The surrounding C block compressor continues to pass
its existing entropy workspace and Huffman-table state through the unchanged
internal ABI.
The port preserves literal header choices, compression-gain decisions,
repeat-table transitions, and 1X/4X encoder selection. It lets ordinary C
compression paths exercise Rust code without widening this commit into the
remaining frame compressor.
Test Plan:
- cargo clippy, cargo clippy --benches, cargo clippy --tests, and nightly fmt
- cargo test --all-targets (94 passed) and feature-only cargo checks
- native fuzzer, zstreamtest, invalidDictionaries, and CLI round trip
- compare literals and next-table bytes with pristine C on x86_64 and i686
Refs: rust/README.md compression-primitives map
Move Huffman table construction, table serialization, and one- and
four-stream payload encoding from huf_compress.c into the Rust compatibility
archive. The declaration-only C shim preserves the existing internal ABI,
so the remaining C compressor can call the migrated implementation unchanged.
The translation keeps CTable layouts, workspace checks, repeat-table
selection, and bitstream output compatible with the original encoder. This
lets native C consumers exercise the Rust implementation through libzstd.
Test Plan:
- cargo clippy, cargo clippy --benches, and cargo clippy --tests
- cargo +nightly fmt and cargo test --all-targets (88 passed)
- cargo check --no-default-features --features compression
- rebuild and run tests fuzzer, zstreamtest, and invalidDictionaries
- compare tables, headers, and 1X/4X streams with a renamed pristine C build
Refs: rust/README.md component map
Build feature-matched Rust archives for the native Make library path. Flatten
their members into libzstd.a so static consumers resolve C-to-Rust and
Rust-to-C references in one archive, and whole-archive link shared outputs so
every migrated public ABI export remains available.
Rust compression and decompression features now mirror the C partial-library
switches. This prevents compression-only artifacts from retaining DDict
references to decompression-only C helpers.
Add a C archive smoke test that round-trips data and exercises the DDict
lifecycle through lib/libzstd.a, rather than direct test-object links.
Test Plan:
- cargo fmt, strict Clippy, cargo test --all-targets, release build
- Rust checks with compression-only, decompression-only, and no features
- full, partial, forced-HUF, and 32-bit static/shared Make library probes
- test-rust-lib-smoke, fuzzer, zstreamtest, and invalidDictionaries
Refs: rust/README.md
Move ZSTD_splitBlock into Rust while retaining its caller-owned workspace
contract. The implementation preserves the C fingerprint sampling heuristic
and calls the migrated histogram primitive without adding a hot-path
allocation.
Reference-vector and randomized differential tests compare every split level
with the original C translation unit. The C source remains as a declaration
shim so existing source lists resolve the Rust ABI during the migration.
Test Plan:
- cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests
- cargo test --all-targets and cargo build --release
- cargo test/build --target i686-unknown-linux-gnu
- 264-case original-C/Rust differential harness
- C fuzzer, zstreamtest, and fuzzer32 smoke runs
Refs: rust/README.md
Move ZSTD_DDict allocation, ownership, construction, and public ABI exports
into Rust. The C decoder keeps ZSTD_loadDEntropy until its table loader moves.
The C shim now exposes C-computed DCtx field offsets. This keeps dictionary
state correct across conditional C layouts, including fuzz fields and 32-bit
static-BMI2 builds, without duplicating decoder-context layout in Rust.
Test Plan:
- cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests
- cargo test --all-targets and cargo build --release
- cargo test/build --target i686-unknown-linux-gnu
- C fuzzer, zstreamtest, invalidDictionaries, and fuzzer32 smoke runs
- fuzz-enabled and i686 -mbmi2 C/Rust dictionary roundtrip harnesses
Refs: rust/README.md
Mode-specific Rust archives exposed the test makefiles' flat C-object cache:
switching from default to forced HUF mode could relink stale default C objects
with a newly built forced Rust archive. That hybrid has incompatible decoder
table expectations and can corrupt dictionary decompression.
Track each makefile's effective HUF mode with an empty archive stamp. It is a
safe normal linker prerequisite, and every mode transition advances its mtime
so cached C objects and direct-source binaries rebuild while unchanged modes
remain incremental.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets
- Default -> forced-X1 -> default fuzzer builds without -B, each running
./fuzzer -s5346 -i1 --no-big-tests
- Equivalent zstd-small default/forced/default rebuild and --version checks
Refs: rust/README.md
Fixes: d89ebb31
Move Huffman decoding-table construction and X1/X2 single- and four-stream
decoding into Rust. The original C translation unit is now a declaration shim,
so unchanged C callers and tests resolve the public decoder ABI from the Rust
archive.
The portable implementation preserves the C workspace and error contracts,
retains the existing assembly-only internal fast loops, and mirrors the C
32-bit decode cadence so X2 writes remain within its required output window.
Forced X1/X2 modes follow the matching archive configuration from the build
integration.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets for default, forced X1, and forced X2 modes
- cargo build and cargo test for i686-unknown-linux-gnu in all three modes
- Strict normal, X1, and X2 C shim compilation
- Original fuzzer and fullbench compatibility targets
Refs: rust/README.md
Depends-on: d89ebb31
Derive the Rust static-library configuration from the effective C HUF decoder
flags and place each mode in its own Cargo target directory. This prevents a
forced X1/X2 C build from silently reusing the default Rust archive.
The test and program makefiles now also build and link an i686 Rust archive
for their 32-bit targets, including zstd32. zstd-small retains its existing
default configuration while preserving an explicitly requested HUF mode.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets for default, forced X1, and forced X2 modes
- cargo build --release --target i686-unknown-linux-gnu for each mode
- make -n checks for default, forced, direct -D, and 32-bit test/CLI targets
Refs: rust/README.md
Move FSE normalization, count-header writing, compression-table construction,
and stream encoding from C to Rust. The original C source remains as a
header-only shim, preserving the public header configuration while native
consumers receive the Rust archive exports.
This completes both codec directions for FSE in the migration crate and lets
the unchanged C compatibility tests exercise the Rust encoder implementation.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets
- cargo build --release
- Compile the C shim with strict warnings enabled
- 1,000-case pristine-C differential for tables, headers, and payloads
- ./tests/fuzzer -v -T10s
- ./tests/decodecorpus -t -T5
Refs: rust/README.md
Make Rust sources normal prerequisites of native C test executables and CLI
builds. A changed Rust module now rebuilds libzstd_rs.a and relinks the target,
instead of relying on a manually prepared archive that can silently be stale.
The archive is linked after C objects, allowing migrated C shims to resolve
their Rust ABI symbols while remaining compatible with the existing makefile
flows. The migration guide now documents this behavior.
Test Plan:
- make -B -C tests fuzzer
- ./tests/fuzzer -i1 --no-big-tests
- make -B -C programs zstd
- ./programs/zstd --version
Refs: rust/README.md
Move byte histogram counting into Rust for FSE and Huffman compression callers.
The implementation preserves the C workspace contract, checked alphabet path,
fast path, empty-input behavior, and C ABI while the original C source becomes
a header-only compatibility shim.
Focused tests cover count accumulation, invalid alphabets, workspace failures,
and striped large inputs. Higher-level compression remains C for now.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
- make -C tests fuzzer
- ./tests/fuzzer -i1 --no-big-tests
- compile hist.c with -Werror and -Wredundant-decls
Refs: rust/README.md
Replace the common worker pool and debug pthread wrappers with ABI-compatible
Rust implementations. The pool preserves bounded-queue, resize, drain-on-free,
custom-allocator, and no-thread behavior while C supplies only the build-time
multithreading configuration bit.
This moves runtime support without changing C tests or public headers. Codec
and CLI implementations remain C while their dependencies are migrated.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
- make -C tests poolTests
- ./tests/poolTests
- compile pool and threading shims with -Werror in MT and no-thread modes
Refs: rust/README.md
Move FSE normalized-count parsing, Huffman statistics decoding, FSE table
construction, and FSE stream decompression into Rust. The C source files now
only retain the headers needed by the existing build configuration.
The implementation keeps the public C ABI and verifies C-generated balanced,
skewed, and Huffman-statistics streams. Compression and the higher-level frame
decoder remain C for now.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
- compile both C shims with -Werror and -Wredundant-decls
Refs: rust/README.md
Introduce the Rust static library and move the shared byte, bitstream, CPU,
error, xxHash, debug, and public-common implementations into it. Thin C shims
preserve the existing header-driven C build while original tests link the Rust
archive.
This establishes the ABI-safe foundation for later codec and CLI ports;
entropy coding, runtime support, codecs, dictionaries, and the CLI remain C.
The top-down migration map documents that boundary and its validation path.
Test Plan:
- cargo fmt --check
- cargo test --all-targets
- cargo clippy --all-targets -- -D warnings
- cargo build --release
Refs: rust/README.md
ddidderr