feat(rust): port divsufsort

Move the dictionary builder's suffix-array construction from
lib/dictBuilder/divsufsort.c to rust/src/divsufsort.rs, the first
dictBuilder module to migrate.  It rides on the dict-builder cargo
feature dimension introduced by the previous commit.

divsufsort() is a self-contained algorithm (two-stage sort of type-B*
substrings via sssort, rank refinement via trsort, then induced sorting
of the full array), so its context-free signature allows a direct symbol
takeover: the Rust #[no_mangle] export provides the existing `divsufsort`
symbol and the C file becomes a declaration-only shim that just keeps the
header's prototypes in the build.  Only divsufsort() moved; divbwt() has
no callers anywhere in zstd, so it is now declaration-only, keeping the
Rust export surface minimal.  The unused openMP parameter is retained for
signature compatibility (zstd never defines LIBBSC_OPENMP).

The port is a mechanical translation of the exact configuration zstd
compiles: ALPHABET_SIZE=256, SS_INSERTIONSORT_THRESHOLD=8,
SS_BLOCKSIZE=1024, SS_MISORT_STACKSIZE=16, SS_SMERGE_STACKSIZE=32,
TR_STACKSIZE=64.  Every C `int*` cursor into the SA buffer becomes an
`isize` index into a single `&mut [i32]` slice, preserving the pointer
arithmetic (including transient one-before-the-range cursors and the
bitwise-complement rank marking) while staying bounds-checked; all value
arithmetic keeps C int semantics.  The C -1/-2 error results are
preserved, with Vec::try_reserve_exact standing in for the bucket-array
malloc failure path.  Behavior is bit-identical by construction and by
measurement (see test plan); runtime on an 11 MB training buffer is
within ~5% of the C build end-to-end.

Users see no behavioral change: dictionaries trained through
ZDICT_trainFromBuffer_legacy() are byte-identical to the C build.  The
only external difference is that the never-called `divbwt` symbol is no
longer defined in the library.

Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
  -- -D warnings && cargo test --all-targets && cargo build --release
  (125 tests pass; new unit tests cover empty/one/two-byte inputs,
  all-equal bytes, an exact hand-computed "abracadabra" SA, and
  fixed-seed LCG buffers at 256-, 4-, and 2-symbol alphabets verified
  against a naive reference sort plus permutation/sorted invariants)
- Feature matrix: cargo build --release --no-default-features
  --features compression,decompression (and decompression-only,
  compression-only, compression,dict-builder); `divsufsort` is exported
  only when dict-builder is enabled
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests (includes
  ZDICT training tests): pass
- make -C tests test-rust-lib-smoke: pass
- make -C tests test-invalidDictionaries: pass
- make -C programs zstd zstd-dictBuilder zstd-small zstd-compress
  zstd-decompress: build; compress/decompress round-trip verified
- Byte-identity vs pristine C build (commit 959e4852): a harness calling
  ZDICT_trainFromBuffer_legacy() (the only zstd path reaching
  divsufsort) and divsufsort() directly, linked against both libzstd.a
  builds, produces byte-identical dictionaries (80,288 B and full
  112,640 B capacity) and byte-identical suffix arrays on a 1 MB source
  set and an 11 MB binary/repetitive set; a differential driver over 148
  random and structured buffers (sizes 3..6000, alphabets 1..256,
  Fibonacci word, sawtooth, 6 KB near-constant) shows zero mismatches.
  The CLI --train path could not be exercised because the Rust CLI
  frontend rejects --train in both the pristine and ported builds (a
  pre-existing migration gap unrelated to this change).
This commit is contained in:
2026-07-11 14:23:39 +02:00
parent 91c80fc8e7
commit caf12dda22
4 changed files with 2772 additions and 1890 deletions
+9 -4
View File
@@ -48,6 +48,10 @@ zstd ABI:
the dynamic-programming optimal parser itself remains in C for now.
- `zstd_ldm` implements long-distance-match parameter selection, table
maintenance, sequence generation, and sequence consumption.
- Dictionary building
- `divsufsort` constructs the suffix array that drives the legacy `ZDICT`
trainer (`ZDICT_trainFromBuffer_legacy`). The sample analysis and
dictionary assembly in `zdict.c`, `cover.c`, and `fastcover.c` remain C.
- Runtime support
- `threading` provides platform pthread wrappers required by zstd headers.
- `pool` implements the bounded worker pool used by multithreaded compression.
@@ -66,10 +70,11 @@ zstd ABI:
`fileio` backend still owns file opening, safe replacement, sparse writes,
metadata, and streaming I/O.
The optimal block matcher, high-level frame compression, dictionary-building,
legacy decoding callbacks, and the CLI file-I/O backend are still C. They must
move before the rewrite is complete. Keeping that boundary explicit prevents a
passing hybrid build from being mistaken for the final all-Rust result.
The optimal block matcher, high-level frame compression, dictionary-building
except suffix-array construction, legacy decoding callbacks, and the CLI
file-I/O backend are still C. They must move before the rewrite is complete.
Keeping that boundary explicit prevents a passing hybrid build from being
mistaken for the final all-Rust result.
## Compatibility boundary