63fdbc3fc271d279a33fd85bd4cfb623ceeb96fc
28
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 ddidderr
|
daeac5238c
|
feat(rust): port block entropy statistics
Continue the incremental zstd_compress.c migration with its sequence statistics and seqStore entropy-compression layer. The following now live in rust/src/zstd_compress_stats.rs: - ZSTD_seqToCodes(), exported under its original name because the C dictionary builder (zdict.c) and decodecorpus link against it, - ZSTD_buildSequencesStatistics() and its dummy variant, whose result struct no longer crosses the language boundary, - ZSTD_entropyCompressSeqStore_internal(), _wExtLitBuffer(), and ZSTD_entropyCompressSeqStore(), - ZSTD_buildBlockEntropyStats() with its literals/sequences helpers, - ZSTD_copyBlockSequences() and the ZSTD_updateRep() rules it shares with the superblock writer. Boundary: ZSTD_CCtx and ZSTD_CCtx_params stay private to C. These paths read exactly two parameter fields, so the C shims keep the original static/exported function names and forward the strategy and ZSTD_literalsCompressionIsDisabled() as int scalars alongside the seqStore and entropy-table leaves. The leaf layouts (SeqDef, SeqStore_t, ZSTD_hufCTables_t, ZSTD_fseCTables_t, entropy metadata, SeqCollector, ZSTD_Sequence) are pinned by compile-time asserts in zstd_compress.c and by both-pointer-width layout tests in Rust. ZSTD_buildSeqStore, block dispatch/splitting, and the block-size estimation helpers remain C for a later slice. The superblock module previously round-tripped through the C export of ZSTD_buildBlockEntropyStats and mirrored the entropy leaf structs privately. It now calls the crate-internal builder directly, and the shared struct definitions moved to zstd_compress_stats; consequently ZSTD_rust_compressSuperBlock() takes the two parameter scalars instead of an opaque ZSTD_CCtx_params pointer, extracted by its C shim. Its layout and repcode tests moved with the definitions. One C helper family gets no shim: the static ZSTD_entropyCompressSeqStore_wExtLitBuffer() had a single caller and was folded into the Rust implementation. Byte-identity was verified against the pre-change compressor: COPYING, datagen -g5000000 -s7, and datagen -g300000 -s21 -P90, each at levels 1/3/9/19 and --fast=5, plus a superblock-heavy pass at level 19 with --target-compressed-block-size=1024; all 18 frames are byte-identical, covering the repeat-mode state machine, longOffsets, RLE/raw fallback, and dstSize_tooSmall paths through the block splitter and superblock. Test plan: - cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets (131 tests: new reference vectors for seqToCodes, RLE table headers, empty-seqStore repeat copies, literal-stats type selection, and repcode resolution in copyBlockSequences) - cargo build --release --no-default-features --features compression and --features decompression - make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests (covers ZSTD_generateSequences and ZDICT training over the Rust seqToCodes) - make -C tests test-rust-lib-smoke - ./tests/zstreamtest -i1 and ./tests/zstreamtest --newapi -t1 -i1 - ./tests/decodecorpus -t -T1s (decodecorpus links Rust seqToCodes) - /tmp/ref vs /tmp/got frame diff as described above: byte-identical |
||
|
ddidderr
|
322f99dfd0
|
feat(rust): port legacy v0.1 decoder
Port lib/legacy/zstd_v01.c (the frozen zstd v0.1 decoder) to
rust/src/legacy/zstd_v01.rs as the first legacy-format port on the new
scaffolding, and reduce the C file to a declaration-only shim that
keeps its header includes for configuration and platform preprocessor
behavior.
Frozen-decoder policy: zstd_v01.c embeds its own v0.1-era FSE and
Huff0 snapshot, distinct from every other release. The Rust port is a
line-by-line translation with the same table layouts (FSE_DTable as a
u32 header word plus packed newState/symbol/nbBits entries, the Huff0
u16 DTable with byte/nbBits pairs), the same arithmetic including
wrap-around and pointer-comparison quirks (e.g. the offset-vs-base
address check in ZSTD_execSequence), the same internal FSE error space
(size_t)-1..-7, and the same public ZSTD error codes. It reuses no
modern Rust entropy module; its only crate dependency is `errors`,
matching the C file's error_private.h include. The 32-bit-only reload
points are kept as compile-time conditions on usize::BITS.
Symbol takeover boundary: all nine ZSTDv01_* entry points from
zstd_v01.h now come from Rust as context-free #[no_mangle] extern "C"
functions (isError, decompress, decompressDCtx,
findFrameSizeInfoLegacy, createDCtx, freeDCtx, resetDCtx,
nextSrcSizeToDecompress, decompressContinue). zstd_legacy.h only uses
the first four for v0.1; streaming for v0.1-v0.3 intentionally returns
version_unsupported there, unchanged. The ZSTDv01_Dctx struct
definition moves entirely into Rust: C code only ever holds an opaque
pointer (zstd_v01.h forward-declares the type), and the context is
malloc/free-allocated exactly like the C version so create/free may
pair across the language boundary.
Byte-identity verification against the pristine pre-migration C build
(
|
||
|
ddidderr
|
bdd35c838d
|
build(rust): add legacy feature scaffolding
The legacy decoders (lib/legacy/zstd_v01.c .. zstd_v07.c) are next in the Rust migration. Each of those files is a frozen snapshot of the FSE/Huff0 entropy coders and frame logic of one historical release, so their ports must not reuse the modern Rust entropy modules and must not share code with each other: outputs and error codes have to stay byte-identical to the frozen C forever. This commit installs the build-system scaffolding so seven per-version ports can land independently, each adding only its own module file plus a one-line registration in rust/src/legacy/mod.rs. Cargo grows features legacy-v01 .. legacy-v07. They are never default features: the C build defaults differ per build system, so each build system passes the list explicitly, derived from its own legacy configuration: - lib/Makefile and programs/Makefile map ZSTD_LEGACY_SUPPORT=N to the features for versions N..7 (0 disables legacy), mirroring the ZSTD_LEGACY_FILES selection in lib/libzstd.mk. - tests/Makefile always enables all seven features because its ZSTDLEGACY_FILES wildcard compiles every lib/legacy/*.c regardless of the dispatch level. - build/meson maps legacy_level exactly like the makefiles; build/cmake enables all seven whenever ZSTD_LEGACY_SUPPORT is ON because it always compiles all seven C files (ZSTD_LEGACY_LEVEL only selects the C dispatch). Every build system also encodes the legacy selection in the Rust target directory name (e.g. c1-d1-default-legacy5), for the same reason the HUF mode is encoded there: a cached archive built for one configuration must never be linked into a build expecting another. In tests/Makefile the legacy level additionally flows into the existing HUF C-mode stamp, so the flat C test objects (which bake -DZSTD_LEGACY_SUPPORT into the dispatch) are rebuilt whenever the level changes. In programs/Makefile the compress-only, decompress-only, and CLI archives keep level-independent directories (RUST_HUF_MODE) because they are only linked into ZSTD_LEGACY_SUPPORT=0 program variants and carry no legacy features. A feature whose version has not been ported yet gates nothing: the module registration in rust/src/legacy/mod.rs is added by each port, so enabling e.g. legacy-v05 today simply leaves that decoder in C. This is what makes mixed C/Rust legacy levels link cleanly while the seven ports land in any order. Test plan: - cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets - cargo clippy with --no-default-features --features decompression,legacy-v01 and with all seven legacy features - make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests - make -C tests test-rust-lib-smoke; make -C tests test-legacy - make -C lib libzstd.a with ZSTD_LEGACY_SUPPORT=0, 1 and default (5) - cmake configure and meson setup (including -Dlegacy_level=1) emit the expected --features lists and legacy-suffixed target directories |
||
|
ddidderr
|
fef5f4478a
|
feat(rust): port benchmark loop and CLI bench mode
Move the implementation of programs/benchfn.c into rust/src/benchfn.rs and wire benchmark mode (-b/-e/-i) into the Rust CLI frontend, which previously rejected those options as not yet implemented. `zstd -b1 -i0 FILE` and range runs like `zstd -b5e6 -i0 FILE` work again, including the synthetic-sample benchmark when no file is given. benchfn.rs is a faithful port of the run/timing state machine: BMK_benchFunction keeps the exact loop accounting (first-loop blockResults and errorFn checks, dstSize summed on the first loop only, 0xE5 warm-up of result buffers, nbLoops minimum of 1) and BMK_benchTimedFn keeps the same convergence behavior (x10 workload growth for short runs, budget-based nbLoops estimation, runs below half the run budget re-tried rather than reported, best qualifying run returned). Arithmetic that C leaves to unsigned wrap-around uses wrapping operations so debug builds cannot panic where release C would wrap. ABI notes: BMK_runTime_t and BMK_runOutcome_t are returned by value across the C boundary and BMK_benchParams_t is passed by value, so all three are repr(C) mirrors of benchfn.h; their field offsets are pinned by const asserts in Rust and matching C static asserts in the benchfn.c shim, which is now declaration-only. BMK_timedFnState_t stays opaque, fits the 64-byte BMK_timedFnState_shell (compile-time checked), and is malloc/free-managed so creation and destruction remain interchangeable with C callers. The CLI parses -b (bench mode), -e (range end, digits attach directly, defaulting to 0 like readU32FromChar) and -i (duration in seconds), then dispatches through a new ZSTD_rust_cli_bench bridge in the zstdcli.c shim. The bridge exists because benchmark availability is a C preprocessor property (ZSTD_NOBENCH): orchestration and reporting stay in C benchzstd.c, stripped variants (zstd-small, zstd-compress, zstd-decompress) compile the stub branch and report "benchmark mode is not available in this build", and the Rust side never references benchmark symbols directly. Level clamping against ZSTD_maxCLevel() happens in the bridge, where the symbol is guaranteed to exist whenever benchmarking is compiled in. -T selects the worker count, defaulting to single-threaded like the C bench path; -S (separate files) and --priority=rt remain unimplemented. Makefile updates only extend the Rust source prerequisite lists with benchfn.rs; the helpers-archive plumbing from the timefn commit already links fullbench(-lib/-dll/32) and paramgrill, the benchfn consumers among the C tests. Original C test sources are untouched. Known pre-existing issues, unchanged by this commit: tests/fullbench-lib fails to link at the base commit too (libzstd.a precedes fullbench.c in its link line), and the cli-tests basic/help.sh, compression/levels.sh, compression/golden.sh, and decompression/pass-through.sh scripts fail identically with a base-commit binary because the Rust CLI frontend is still a partial reimplementation. Test Plan: - cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets && cargo build --release - cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets; repeat tests with --no-default-features plus features compression / decompression / (none) - make -C programs zstd; ./programs/zstd -b1 -i0 lib/common/xxhash.c; ./programs/zstd -b5e6 -i0 programs/fileio.c; ./programs/zstd -b1 -i0 (synthetic); echo roundtrip via zstd | zstd -d - make -C programs zstd-small zstd-compress zstd-decompress zstd-nolegacy zstd-dictBuilder; zstd-small -b reports benchmark unavailable; compress/ decompress roundtrip across the split binaries - make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus poolTests fullbench32 fuzzer32; ./tests/fullbench -i0 (exercises Rust BMK_benchTimedFn from C); ./tests/fullbench32 -i0; ./tests/fuzzer -i1 --no-big-tests; ./tests/poolTests; make -C tests test-rust-lib-smoke - cli-tests subset: basic/version.sh, compression/basic.sh, compression/multiple-files.sh pass; failing scripts match the base commit Refs: rust/README.md |
||
|
ddidderr
|
54f5c29742
|
feat(rust): port program timing helpers
Move the implementation of programs/timefn.c into rust/src/timefn.rs. The file provides the monotonic nanosecond clock (UTIL_getTime, span helpers, UTIL_waitForNextTick, UTIL_support_MT_measurements) used by the CLI and by several C test tools. timefn.c remains as a declaration-only shim so the original source lists and header configuration keep working, and it pins the ABI with static asserts: UTIL_time_t is returned by value and must stay a plain 64-bit counter, which the Rust #[repr(C)] mirror also asserts. Platform selection mirrors the C preprocessor structure: Windows uses QueryPerformanceCounter, Apple targets use mach_absolute_time, and other POSIX systems use libc clock_gettime(CLOCK_MONOTONIC). Only the unix path is exercised by this environment; the Windows and Apple paths are written from the C source and compile-checked logically but are untested here. The C90 clock() fallback is unreachable on Rust-supported targets, so multi-threaded measurement support is always reported. The symbols live in the program-only zstd-cli-rs package, keeping them out of library builds. Linking that archive into C test binaries surfaced a structural problem: rustc's local ThinLTO promotes internal symbols across codegen units, so extracting the timefn object could drag in the zstd_cli parser object, whose FIO_* externs test binaries cannot satisfy. The parser is therefore gated behind a new additive `cli` cargo feature (default on). Program archives build with cli,compression,decompression as before, while tests/Makefile links a helpers-only archive (rust/target/cli-helpers) built with --no-default-features, which contains no fileio references at all. tests/Makefile gains build rules for the helpers archive and adds it as a prerequisite of every binary that compiles the timefn shim: fullbench(32), fullbench-lib, fullbench-dll, fuzzer(32), zstreamtest(32/asan/tsan/ubsan), paramgrill, decodecorpus, and poolTests. Prerequisite order places the archive after all C objects in `$^` link lines; the known-broken -dll recipes filter to %.c, so they name the archive explicitly. Original C test sources are untouched; only link inputs changed. zstd-cli-rs now depends on libc (already used by the core crate) for clock_gettime and the Mach timebase bindings. Test Plan: - cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets && cargo build --release - cd rust/cli && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets; repeat tests with --no-default-features plus features compression / decompression / (none) - make -C programs zstd; roundtrip echo hello | zstd | zstd -d - make -C tests fullbench fuzzer zstreamtest paramgrill decodecorpus poolTests; ./tests/fullbench -i0; ./tests/fuzzer -i1 --no-big-tests; ./tests/poolTests; make -C tests test-rust-lib-smoke - verified with nm that the helpers archive member defining UTIL_getTime has no FIO_*/ZSTD_* undefined references Refs: rust/README.md |
||
|
ddidderr
|
caf12dda22
|
feat(rust): port divsufsort
Move the dictionary builder's suffix-array construction from
lib/dictBuilder/divsufsort.c to rust/src/divsufsort.rs, the first
dictBuilder module to migrate. It rides on the dict-builder cargo
feature dimension introduced by the previous commit.
divsufsort() is a self-contained algorithm (two-stage sort of type-B*
substrings via sssort, rank refinement via trsort, then induced sorting
of the full array), so its context-free signature allows a direct symbol
takeover: the Rust #[no_mangle] export provides the existing `divsufsort`
symbol and the C file becomes a declaration-only shim that just keeps the
header's prototypes in the build. Only divsufsort() moved; divbwt() has
no callers anywhere in zstd, so it is now declaration-only, keeping the
Rust export surface minimal. The unused openMP parameter is retained for
signature compatibility (zstd never defines LIBBSC_OPENMP).
The port is a mechanical translation of the exact configuration zstd
compiles: ALPHABET_SIZE=256, SS_INSERTIONSORT_THRESHOLD=8,
SS_BLOCKSIZE=1024, SS_MISORT_STACKSIZE=16, SS_SMERGE_STACKSIZE=32,
TR_STACKSIZE=64. Every C `int*` cursor into the SA buffer becomes an
`isize` index into a single `&mut [i32]` slice, preserving the pointer
arithmetic (including transient one-before-the-range cursors and the
bitwise-complement rank marking) while staying bounds-checked; all value
arithmetic keeps C int semantics. The C -1/-2 error results are
preserved, with Vec::try_reserve_exact standing in for the bucket-array
malloc failure path. Behavior is bit-identical by construction and by
measurement (see test plan); runtime on an 11 MB training buffer is
within ~5% of the C build end-to-end.
Users see no behavioral change: dictionaries trained through
ZDICT_trainFromBuffer_legacy() are byte-identical to the C build. The
only external difference is that the never-called `divbwt` symbol is no
longer defined in the library.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
-- -D warnings && cargo test --all-targets && cargo build --release
(125 tests pass; new unit tests cover empty/one/two-byte inputs,
all-equal bytes, an exact hand-computed "abracadabra" SA, and
fixed-seed LCG buffers at 256-, 4-, and 2-symbol alphabets verified
against a naive reference sort plus permutation/sorted invariants)
- Feature matrix: cargo build --release --no-default-features
--features compression,decompression (and decompression-only,
compression-only, compression,dict-builder); `divsufsort` is exported
only when dict-builder is enabled
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests (includes
ZDICT training tests): pass
- make -C tests test-rust-lib-smoke: pass
- make -C tests test-invalidDictionaries: pass
- make -C programs zstd zstd-dictBuilder zstd-small zstd-compress
zstd-decompress: build; compress/decompress round-trip verified
- Byte-identity vs pristine C build (commit
|
||
|
ddidderr
|
91c80fc8e7
|
build(rust): add dict-builder cargo feature dimension
The dictionary-builder sources (lib/dictBuilder) are about to start moving
to Rust, beginning with divsufsort. The Rust crate previously only modeled
the compression/decompression module split plus the forced-HUF decoder
modes, so no build could express "this C configuration includes (or
excludes) dictBuilder" to Cargo. Without that, a Rust archive could carry
dictBuilder modules into a build whose C side disabled them, or worse,
omit a migrated implementation from a build whose C shims require it.
Add a `dict-builder` cargo feature and thread it through every build that
consumes the Rust static archive, mirroring exactly how each build system
already gates the dictBuilder C sources:
- rust/Cargo.toml: new `dict-builder` feature, included in the default
set because the C library builds dictBuilder by default
(ZSTD_LIB_DICTBUILDER ?= 1). The feature is empty until the first
dictBuilder module lands.
- lib/Makefile: RUST_CARGO_FEATURES gains dict-builder when
ZSTD_LIB_DICTBUILDER is enabled, following the existing
ZSTD_LIB_COMPRESSION/ZSTD_LIB_DECOMPRESSION pattern. The archive
directory naming grows a matching `b<0|1>` dimension
(c1-d1-b1-default etc.) so differently configured archives never
collide; the repeated config prefix is factored into
RUST_MODULE_CONFIG.
- programs/Makefile: the full-featured archives now request
compression,decompression,dict-builder (equal to the default set, so
the target directory stays shared with tests). The partial-library
variants gain the `b0` name dimension, and zstd-dictBuilder gets its
own lib-c1-d0-b1 archive because it compiles the dictBuilder C sources
without decompression; it previously shared the compression-only
archive, which will lack the migrated dictBuilder symbols.
- tests/Makefile: no flag change needed since tests use the crate default
feature set; a comment now records that dict-builder arrives that way.
- build/cmake/lib/CMakeLists.txt: ZSTD_BUILD_DICTBUILDER now adds the
dict-builder feature and a `b<0|1>` component in the Rust build-config
directory name, in lockstep with the DictBuilderSources gating.
- build/meson/lib/meson.build: meson compiles the dictBuilder sources
unconditionally, so the feature list and config name gain dict-builder
unconditionally (c1-d1-b1-<huf-mode>).
The `dict-builder` feature deliberately does not imply `compression`.
lib/Makefile forces ZSTD_LIB_DICTBUILDER=0 when compression is disabled,
but CMake does not couple the two options, so encoding the C-side
constraint in Cargo would make the Rust archive diverge from the C source
list in that (already unsupported) CMake configuration.
Test plan:
- cd rust && cargo build --release
- cargo build --release --no-default-features \
--features compression,decompression
- cargo build --release --no-default-features \
--features compression,dict-builder
- Full validation (fuzzer, smoke tests, dictionary byte-identity) runs
with the follow-up commit that ports divsufsort onto this scaffolding.
|
||
|
ddidderr
|
529cd297e2
|
feat(rust): port compression-parameter selection
Move the context-free compression-parameter logic of zstd_compress.c to rust/src/zstd_compress_params.rs: the compression-level tables (formerly clevels.h), parameter bounds/checking/clamping, cycle log, level-table selection, source/dictionary parameter adjustment, default frame parameters, and the match-state/CDict size estimators. The boundary follows the module's design notes: Rust owns only leaves whose behavior is independent of C preprocessor configuration. C keeps the public ZSTD_* symbols and feeds the leaves everything that is configuration-owned as explicit scalars: - The ZSTD_EXCLUDE_*_BLOCK_COMPRESSOR strategy cascade stays in ZSTD_adjustCParams_internal() ahead of the Rust adjustment leaf, so reduced builds keep their fallback policy. - Workspace estimation receives struct sizes (ZSTD_CDict, ZSTD_match_t, ZSTD_optimal_t), HUF workspace size, and the sanitizer redzone size, because those depend on private layouts and ASAN configuration. - ZSTD_cParam_getBounds() forwards only the compression level and the seven core parameters; all other parameter bounds remain C. - Frozen private constants the leaves hardcode (short-cache and row-hash tag widths, MaxML/MaxLL/MaxOff/Litbits, ZSTD_OPT_SIZE, cwksp alignment) are pinned by ZSTD_STATIC_ASSERTs at the C call sites. Two latent 32-bit bugs in the previously unwired module were fixed before integration: dictAndWindowLog's max window size and the window-resize threshold were hardcoded to the 64-bit constants (1<<31, 1<<30) instead of deriving from ZSTD_WINDOWLOG_MAX, which is 30 on 32-bit targets. clevels.h is no longer included anywhere but stays in-tree as the reference for mechanical comparison against the Rust table. Test plan: - cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets (125 tests) - make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests - make -C tests test-rust-lib-smoke - Byte-identity: zstd CLI frames for COPYING, a 250 KB C source, and a 5 MB datagen sample at levels 1/3/9/19/--fast=5 are identical between this change and its parent commit. |
||
|
ddidderr
|
959e485201
|
feat(rust): port frame serialization
Continue the incremental zstd_compress.c migration with its frame serialization leaves: ZSTD_writeFrameHeader(), the public ZSTD_writeSkippableFrame() ABI, and ZSTD_writeLastEmptyBlock() now live in rust/src/zstd_compress_frame.rs. ZSTD_writeFrameHeader() reads five fields out of ZSTD_CCtx_params, whose layout is private to zstd_compress.c and sensitive to build configuration. Rather than mirror that structure in Rust, the C function keeps its original static signature and forwards the scalar fields to ZSTD_rust_writeFrameHeader(), so no parameter-structure layout crosses the language boundary. The other two functions take only pointer/size arguments and are exported directly, replacing their C bodies outright. Behavior differences are limited to hardening in release builds: the Rust leaf clamps a window-size shift that C would leave undefined for a malformed windowLog, and ZSTD_writeSkippableFrame() rejects a srcSize + header overflow instead of comparing against a wrapped sum. Both paths are unreachable through validated callers, so compressed output is byte-identical. Remaining zstd_compress.c work: parameter selection and validation, context lifecycle, block dispatch, dictionary loading, and the streaming state machine. Test plan: - cd rust && cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test --all-targets (118 tests, includes new frame header/skippable/last-block reference vectors) - make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests - make -C tests test-rust-lib-smoke |
||
|
ddidderr
|
fe7e24c770 |
feat(rust): migrate high-level runtime paths
Move long-distance matching and high-level decompression from C shims into Rust. The decoder now owns context, dictionary, parameter, one-shot, and buffered streaming state while C retains allocation/configuration, legacy, and trace leaves. Move CLI parsing, safety policy, and dispatch into a separate Rust static archive. Keeping it separate prevents library builds from retaining FIO symbols, while C continues to own file opening, replacement, and I/O. Program targets now select matching compression/decompression archives. The remaining C boundary is intentional: high-level compression, optimal parsing, dictionary building, legacy callbacks, and CLI file I/O still need migration. Test Plan: - cargo test --all-targets (native and i686) - cargo test --all-targets in rust/cli (native and i686) - CLI crate compression-only and decompression-only feature tests - native and i686 fuzzer/zstreamtest runs, plus legacy and dictionary tests - ZSTD_C_PREDICT and ZSTD_HEAPMODE=0 fuzzer coverage - library, dynamic-link, and program-target build/round-trip matrix Refs: rust/README.md |
||
|
ddidderr
|
031592da1e |
feat(rust): port optimal binary-tree updates
Move the binary-tree table maintenance shared by optimal compression strategies into Rust. A small C projection preserves the private match-state boundary; the dynamic-programming parser and match selection remain C-owned for now. The Rust implementation covers prefix and external-dictionary tree updates and preserves the optional C predictor path. The component map now distinguishes this migrated tree layer from the remaining optimal parser. Test Plan: - cargo test --all-targets - cargo test --target i686-unknown-linux-gnu --all-targets - cargo clippy && cargo clippy --benches && cargo clippy --tests - cargo +nightly fmt - make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries - ./tests/fuzzer -s5346 -i1 --no-big-tests - ./tests/zstreamtest -i3000 -s334462 - ./tests/invalidDictionaries - make -B -C tests fuzzer32 MOREFLAGS=-DZSTD_C_PREDICT - ./tests/fuzzer32 -s5346 -i1 --no-big-tests - byte-compare C and Rust-control CLI output at levels 13, 16, 19, and 22 for random, patterned, zero, and dictionary inputs with and without ZSTD_C_PREDICT Refs: rust/README.md |
||
|
ddidderr
|
e5eebd4892 |
feat(rust): port compressed block decoding
Move literal decoding, sequence-table construction, FSE sequence decoding, and sequence execution into Rust. The C shim retains ownership of the configured decoder context and passes only the leaf state needed by the block codec. Correct the two high offset-code bases while porting the tables. Their prior values made valid large-window streams decode as corrupted data; the new unit test fixes the exact values and a native zstream regression exercises them. High-level frame and streaming context control remains C for now. Test Plan: - cargo test --all-targets - cargo test --target i686-unknown-linux-gnu --all-targets - cargo clippy && cargo clippy --benches && cargo clippy --tests - cargo +nightly fmt - make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests - ./tests/fuzzer -s5346 -i1 --no-big-tests - ./tests/zstreamtest -i3000 -s334462 - ./tests/invalidDictionaries - timeout 20s stdbuf -oL ./tests/poolTests Refs: rust/README.md |
||
|
ddidderr
|
7f16a07375 |
feat(rust): port lazy block matching
Move greedy, lazy, lazy2, and binary-tree block matchers into Rust. A narrow C state projection preserves configuration-specific context layout while Rust owns hash-chain, row-based, attached-dictionary, and external-dictionary searches and sequence-store updates. The component map now records the lazy family as migrated; the optimized matcher and high-level contexts remain outside this commit. Test Plan: - cargo test --all-targets - cargo test --target i686-unknown-linux-gnu --all-targets - cargo clippy && cargo clippy --benches && cargo clippy --tests - cargo +nightly fmt - make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests - ./tests/fuzzer -s5346 -i1 --no-big-tests - ./tests/zstreamtest -i3000 -s334462 - ./tests/invalidDictionaries - timeout 20s stdbuf -oL ./tests/poolTests Refs: rust/README.md |
||
|
ddidderr
|
f9adcf6aef |
feat(rust): port double-fast block matching
Move the two-table fast match finder into Rust while retaining a small C projection of the private match-state. The Rust implementation handles normal, attached-dictionary, and external-dictionary matching without exposing the full C context ABI. The C entry points and build-time exclusion guards remain unchanged for configured consumers. Document the migrated matcher in the Rust component map and narrow the remaining-C boundary accordingly. Test Plan: - cargo test --all-targets - cargo test --target i686-unknown-linux-gnu --all-targets - cargo clippy && cargo clippy --benches && cargo clippy --tests - cargo +nightly fmt - make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests - ./tests/fuzzer -s5346 -i1 --no-big-tests - ./tests/zstreamtest -i3000 -s334462 - ./tests/invalidDictionaries - timeout 20s stdbuf -oL ./tests/poolTests Refs: rust/README.md |
||
|
ddidderr
|
6a762660b7 |
feat(rust): port compression literal blocks
Move raw, RLE, and Huffman literal-section encoding into the Rust compatibility archive. The surrounding C block compressor continues to pass its existing entropy workspace and Huffman-table state through the unchanged internal ABI. The port preserves literal header choices, compression-gain decisions, repeat-table transitions, and 1X/4X encoder selection. It lets ordinary C compression paths exercise Rust code without widening this commit into the remaining frame compressor. Test Plan: - cargo clippy, cargo clippy --benches, cargo clippy --tests, and nightly fmt - cargo test --all-targets (94 passed) and feature-only cargo checks - native fuzzer, zstreamtest, invalidDictionaries, and CLI round trip - compare literals and next-table bytes with pristine C on x86_64 and i686 Refs: rust/README.md compression-primitives map |
||
|
ddidderr
|
0a6a5f5267
|
feat(rust): port Huffman compression
Move Huffman table construction, table serialization, and one- and four-stream payload encoding from huf_compress.c into the Rust compatibility archive. The declaration-only C shim preserves the existing internal ABI, so the remaining C compressor can call the migrated implementation unchanged. The translation keeps CTable layouts, workspace checks, repeat-table selection, and bitstream output compatible with the original encoder. This lets native C consumers exercise the Rust implementation through libzstd. Test Plan: - cargo clippy, cargo clippy --benches, and cargo clippy --tests - cargo +nightly fmt and cargo test --all-targets (88 passed) - cargo check --no-default-features --features compression - rebuild and run tests fuzzer, zstreamtest, and invalidDictionaries - compare tables, headers, and 1X/4X streams with a renamed pristine C build Refs: rust/README.md component map |
||
|
ddidderr
|
d784406374
|
build(rust): link migrated code into native libraries
Build feature-matched Rust archives for the native Make library path. Flatten their members into libzstd.a so static consumers resolve C-to-Rust and Rust-to-C references in one archive, and whole-archive link shared outputs so every migrated public ABI export remains available. Rust compression and decompression features now mirror the C partial-library switches. This prevents compression-only artifacts from retaining DDict references to decompression-only C helpers. Add a C archive smoke test that round-trips data and exercises the DDict lifecycle through lib/libzstd.a, rather than direct test-object links. Test Plan: - cargo fmt, strict Clippy, cargo test --all-targets, release build - Rust checks with compression-only, decompression-only, and no features - full, partial, forced-HUF, and 32-bit static/shared Make library probes - test-rust-lib-smoke, fuzzer, zstreamtest, and invalidDictionaries Refs: rust/README.md |
||
|
ddidderr
|
24d01b92fb
|
feat(rust): port compression block pre-splitting
Move ZSTD_splitBlock into Rust while retaining its caller-owned workspace contract. The implementation preserves the C fingerprint sampling heuristic and calls the migrated histogram primitive without adding a hot-path allocation. Reference-vector and randomized differential tests compare every split level with the original C translation unit. The C source remains as a declaration shim so existing source lists resolve the Rust ABI during the migration. Test Plan: - cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests - cargo test --all-targets and cargo build --release - cargo test/build --target i686-unknown-linux-gnu - 264-case original-C/Rust differential harness - C fuzzer, zstreamtest, and fuzzer32 smoke runs Refs: rust/README.md |
||
|
ddidderr
|
ca70ca8061
|
feat(rust): port decode dictionaries
Move ZSTD_DDict allocation, ownership, construction, and public ABI exports into Rust. The C decoder keeps ZSTD_loadDEntropy until its table loader moves. The C shim now exposes C-computed DCtx field offsets. This keeps dictionary state correct across conditional C layouts, including fuzz fields and 32-bit static-BMI2 builds, without duplicating decoder-context layout in Rust. Test Plan: - cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests - cargo test --all-targets and cargo build --release - cargo test/build --target i686-unknown-linux-gnu - C fuzzer, zstreamtest, invalidDictionaries, and fuzzer32 smoke runs - fuzz-enabled and i686 -mbmi2 C/Rust dictionary roundtrip harnesses Refs: rust/README.md |
||
|
ddidderr
|
65afd94867
|
fix(build): rebuild C outputs on HUF mode changes
Mode-specific Rust archives exposed the test makefiles' flat C-object cache:
switching from default to forced HUF mode could relink stale default C objects
with a newly built forced Rust archive. That hybrid has incompatible decoder
table expectations and can corrupt dictionary decompression.
Track each makefile's effective HUF mode with an empty archive stamp. It is a
safe normal linker prerequisite, and every mode transition advances its mtime
so cached C objects and direct-source binaries rebuild while unchanged modes
remain incremental.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets
- Default -> forced-X1 -> default fuzzer builds without -B, each running
./fuzzer -s5346 -i1 --no-big-tests
- Equivalent zstd-small default/forced/default rebuild and --version checks
Refs: rust/README.md
Fixes:
|
||
|
ddidderr
|
27932113fd
|
feat(rust): port Huffman decompression
Move Huffman decoding-table construction and X1/X2 single- and four-stream
decoding into Rust. The original C translation unit is now a declaration shim,
so unchanged C callers and tests resolve the public decoder ABI from the Rust
archive.
The portable implementation preserves the C workspace and error contracts,
retains the existing assembly-only internal fast loops, and mirrors the C
32-bit decode cadence so X2 writes remain within its required output window.
Forced X1/X2 modes follow the matching archive configuration from the build
integration.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets for default, forced X1, and forced X2 modes
- cargo build and cargo test for i686-unknown-linux-gnu in all three modes
- Strict normal, X1, and X2 C shim compilation
- Original fuzzer and fullbench compatibility targets
Refs: rust/README.md
Depends-on:
|
||
|
ddidderr
|
d89ebb31e3
|
build(rust): select archives by target and HUF mode
Derive the Rust static-library configuration from the effective C HUF decoder flags and place each mode in its own Cargo target directory. This prevents a forced X1/X2 C build from silently reusing the default Rust archive. The test and program makefiles now also build and link an i686 Rust archive for their 32-bit targets, including zstd32. zstd-small retains its existing default configuration while preserving an explicitly requested HUF mode. Test Plan: - cargo clippy; cargo clippy --benches; cargo clippy --tests - cargo +nightly fmt, then repeat the Clippy checks - cargo test --all-targets for default, forced X1, and forced X2 modes - cargo build --release --target i686-unknown-linux-gnu for each mode - make -n checks for default, forced, direct -D, and 32-bit test/CLI targets Refs: rust/README.md |
||
|
ddidderr
|
5f9d607ddd
|
feat(rust): port FSE compression primitives
Move FSE normalization, count-header writing, compression-table construction, and stream encoding from C to Rust. The original C source remains as a header-only shim, preserving the public header configuration while native consumers receive the Rust archive exports. This completes both codec directions for FSE in the migration crate and lets the unchanged C compatibility tests exercise the Rust encoder implementation. Test Plan: - cargo clippy; cargo clippy --benches; cargo clippy --tests - cargo +nightly fmt, then repeat the Clippy checks - cargo test --all-targets - cargo build --release - Compile the C shim with strict warnings enabled - 1,000-case pristine-C differential for tables, headers, and payloads - ./tests/fuzzer -v -T10s - ./tests/decodecorpus -t -T5 Refs: rust/README.md |
||
|
ddidderr
|
ebc43676b7
|
build(rust): rebuild the archive for C test and CLI links
Make Rust sources normal prerequisites of native C test executables and CLI builds. A changed Rust module now rebuilds libzstd_rs.a and relinks the target, instead of relying on a manually prepared archive that can silently be stale. The archive is linked after C objects, allowing migrated C shims to resolve their Rust ABI symbols while remaining compatible with the existing makefile flows. The migration guide now documents this behavior. Test Plan: - make -B -C tests fuzzer - ./tests/fuzzer -i1 --no-big-tests - make -B -C programs zstd - ./programs/zstd --version Refs: rust/README.md |
||
|
ddidderr
|
158cd680d9
|
feat(rust): port histogram counting primitives
Move byte histogram counting into Rust for FSE and Huffman compression callers. The implementation preserves the C workspace contract, checked alphabet path, fast path, empty-input behavior, and C ABI while the original C source becomes a header-only compatibility shim. Focused tests cover count accumulation, invalid alphabets, workspace failures, and striped large inputs. Higher-level compression remains C for now. Test Plan: - cargo fmt --check - cargo test --all-targets - cargo clippy --all-targets -- -D warnings - cargo build --release - make -C tests fuzzer - ./tests/fuzzer -i1 --no-big-tests - compile hist.c with -Werror and -Wredundant-decls Refs: rust/README.md |
||
|
ddidderr
|
26b5e202ee
|
feat(rust): port worker pool and pthread wrappers
Replace the common worker pool and debug pthread wrappers with ABI-compatible Rust implementations. The pool preserves bounded-queue, resize, drain-on-free, custom-allocator, and no-thread behavior while C supplies only the build-time multithreading configuration bit. This moves runtime support without changing C tests or public headers. Codec and CLI implementations remain C while their dependencies are migrated. Test Plan: - cargo fmt --check - cargo test --all-targets - cargo clippy --all-targets -- -D warnings - cargo build --release - make -C tests poolTests - ./tests/poolTests - compile pool and threading shims with -Werror in MT and no-thread modes Refs: rust/README.md |
||
|
ddidderr
|
a0f2b2a14c
|
feat(rust): port FSE entropy decoding
Move FSE normalized-count parsing, Huffman statistics decoding, FSE table construction, and FSE stream decompression into Rust. The C source files now only retain the headers needed by the existing build configuration. The implementation keeps the public C ABI and verifies C-generated balanced, skewed, and Huffman-statistics streams. Compression and the higher-level frame decoder remain C for now. Test Plan: - cargo fmt --check - cargo test --all-targets - cargo clippy --all-targets -- -D warnings - cargo build --release - compile both C shims with -Werror and -Wredundant-decls Refs: rust/README.md |
||
|
ddidderr
|
089f8e5b4d
|
feat(rust): add common primitive compatibility layer
Introduce the Rust static library and move the shared byte, bitstream, CPU, error, xxHash, debug, and public-common implementations into it. Thin C shims preserve the existing header-driven C build while original tests link the Rust archive. This establishes the ABI-safe foundation for later codec and CLI ports; entropy coding, runtime support, codecs, dictionaries, and the CLI remain C. The top-down migration map documents that boundary and its validation path. Test Plan: - cargo fmt --check - cargo test --all-targets - cargo clippy --all-targets -- -D warnings - cargo build --release Refs: rust/README.md |