feat(rust): port divsufsort
Move the dictionary builder's suffix-array construction from
lib/dictBuilder/divsufsort.c to rust/src/divsufsort.rs, the first
dictBuilder module to migrate. It rides on the dict-builder cargo
feature dimension introduced by the previous commit.
divsufsort() is a self-contained algorithm (two-stage sort of type-B*
substrings via sssort, rank refinement via trsort, then induced sorting
of the full array), so its context-free signature allows a direct symbol
takeover: the Rust #[no_mangle] export provides the existing `divsufsort`
symbol and the C file becomes a declaration-only shim that just keeps the
header's prototypes in the build. Only divsufsort() moved; divbwt() has
no callers anywhere in zstd, so it is now declaration-only, keeping the
Rust export surface minimal. The unused openMP parameter is retained for
signature compatibility (zstd never defines LIBBSC_OPENMP).
The port is a mechanical translation of the exact configuration zstd
compiles: ALPHABET_SIZE=256, SS_INSERTIONSORT_THRESHOLD=8,
SS_BLOCKSIZE=1024, SS_MISORT_STACKSIZE=16, SS_SMERGE_STACKSIZE=32,
TR_STACKSIZE=64. Every C `int*` cursor into the SA buffer becomes an
`isize` index into a single `&mut [i32]` slice, preserving the pointer
arithmetic (including transient one-before-the-range cursors and the
bitwise-complement rank marking) while staying bounds-checked; all value
arithmetic keeps C int semantics. The C -1/-2 error results are
preserved, with Vec::try_reserve_exact standing in for the bucket-array
malloc failure path. Behavior is bit-identical by construction and by
measurement (see test plan); runtime on an 11 MB training buffer is
within ~5% of the C build end-to-end.
Users see no behavioral change: dictionaries trained through
ZDICT_trainFromBuffer_legacy() are byte-identical to the C build. The
only external difference is that the never-called `divbwt` symbol is no
longer defined in the library.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
-- -D warnings && cargo test --all-targets && cargo build --release
(125 tests pass; new unit tests cover empty/one/two-byte inputs,
all-equal bytes, an exact hand-computed "abracadabra" SA, and
fixed-seed LCG buffers at 256-, 4-, and 2-symbol alphabets verified
against a naive reference sort plus permutation/sorted invariants)
- Feature matrix: cargo build --release --no-default-features
--features compression,decompression (and decompression-only,
compression-only, compression,dict-builder); `divsufsort` is exported
only when dict-builder is enabled
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests (includes
ZDICT training tests): pass
- make -C tests test-rust-lib-smoke: pass
- make -C tests test-invalidDictionaries: pass
- make -C programs zstd zstd-dictBuilder zstd-small zstd-compress
zstd-decompress: build; compress/decompress round-trip verified
- Byte-identity vs pristine C build (commit 959e4852): a harness calling
ZDICT_trainFromBuffer_legacy() (the only zstd path reaching
divsufsort) and divsufsort() directly, linked against both libzstd.a
builds, produces byte-identical dictionaries (80,288 B and full
112,640 B capacity) and byte-identical suffix arrays on a 1 MB source
set and an 11 MB binary/repetitive set; a differential driver over 148
random and structured buffers (sizes 3..6000, alphabets 1..256,
Fibonacci word, sawtooth, 6 KB near-constant) shows zero mismatches.
The CLI --train path could not be exercised because the Rust CLI
frontend rejects --train in both the pristine and ported builds (a
pre-existing migration gap unrelated to this change).
This commit is contained in:
+9
-4
@@ -48,6 +48,10 @@ zstd ABI:
|
||||
the dynamic-programming optimal parser itself remains in C for now.
|
||||
- `zstd_ldm` implements long-distance-match parameter selection, table
|
||||
maintenance, sequence generation, and sequence consumption.
|
||||
- Dictionary building
|
||||
- `divsufsort` constructs the suffix array that drives the legacy `ZDICT`
|
||||
trainer (`ZDICT_trainFromBuffer_legacy`). The sample analysis and
|
||||
dictionary assembly in `zdict.c`, `cover.c`, and `fastcover.c` remain C.
|
||||
- Runtime support
|
||||
- `threading` provides platform pthread wrappers required by zstd headers.
|
||||
- `pool` implements the bounded worker pool used by multithreaded compression.
|
||||
@@ -66,10 +70,11 @@ zstd ABI:
|
||||
`fileio` backend still owns file opening, safe replacement, sparse writes,
|
||||
metadata, and streaming I/O.
|
||||
|
||||
The optimal block matcher, high-level frame compression, dictionary-building,
|
||||
legacy decoding callbacks, and the CLI file-I/O backend are still C. They must
|
||||
move before the rewrite is complete. Keeping that boundary explicit prevents a
|
||||
passing hybrid build from being mistaken for the final all-Rust result.
|
||||
The optimal block matcher, high-level frame compression, dictionary-building
|
||||
except suffix-array construction, legacy decoding callbacks, and the CLI
|
||||
file-I/O backend are still C. They must move before the rewrite is complete.
|
||||
Keeping that boundary explicit prevents a passing hybrid build from being
|
||||
mistaken for the final all-Rust result.
|
||||
|
||||
## Compatibility boundary
|
||||
|
||||
|
||||
Reference in New Issue
Block a user