Move the context-free compression-parameter logic of zstd_compress.c to
rust/src/zstd_compress_params.rs: the compression-level tables (formerly
clevels.h), parameter bounds/checking/clamping, cycle log, level-table
selection, source/dictionary parameter adjustment, default frame
parameters, and the match-state/CDict size estimators.
The boundary follows the module's design notes: Rust owns only leaves
whose behavior is independent of C preprocessor configuration. C keeps
the public ZSTD_* symbols and feeds the leaves everything that is
configuration-owned as explicit scalars:
- The ZSTD_EXCLUDE_*_BLOCK_COMPRESSOR strategy cascade stays in
ZSTD_adjustCParams_internal() ahead of the Rust adjustment leaf, so
reduced builds keep their fallback policy.
- Workspace estimation receives struct sizes (ZSTD_CDict, ZSTD_match_t,
ZSTD_optimal_t), HUF workspace size, and the sanitizer redzone size,
because those depend on private layouts and ASAN configuration.
- ZSTD_cParam_getBounds() forwards only the compression level and the
seven core parameters; all other parameter bounds remain C.
- Frozen private constants the leaves hardcode (short-cache and row-hash
tag widths, MaxML/MaxLL/MaxOff/Litbits, ZSTD_OPT_SIZE, cwksp alignment)
are pinned by ZSTD_STATIC_ASSERTs at the C call sites.
Two latent 32-bit bugs in the previously unwired module were fixed
before integration: dictAndWindowLog's max window size and the
window-resize threshold were hardcoded to the 64-bit constants
(1<<31, 1<<30) instead of deriving from ZSTD_WINDOWLOG_MAX, which is
30 on 32-bit targets.
clevels.h is no longer included anywhere but stays in-tree as the
reference for mechanical comparison against the Rust table.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (125 tests)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke
- Byte-identity: zstd CLI frames for COPYING, a 250 KB C source, and a
5 MB datagen sample at levels 1/3/9/19/--fast=5 are identical between
this change and its parent commit.
Continue the incremental zstd_compress.c migration with its frame
serialization leaves: ZSTD_writeFrameHeader(), the public
ZSTD_writeSkippableFrame() ABI, and ZSTD_writeLastEmptyBlock() now live
in rust/src/zstd_compress_frame.rs.
ZSTD_writeFrameHeader() reads five fields out of ZSTD_CCtx_params, whose
layout is private to zstd_compress.c and sensitive to build
configuration. Rather than mirror that structure in Rust, the C
function keeps its original static signature and forwards the scalar
fields to ZSTD_rust_writeFrameHeader(), so no parameter-structure layout
crosses the language boundary. The other two functions take only
pointer/size arguments and are exported directly, replacing their C
bodies outright.
Behavior differences are limited to hardening in release builds: the
Rust leaf clamps a window-size shift that C would leave undefined for a
malformed windowLog, and ZSTD_writeSkippableFrame() rejects a srcSize +
header overflow instead of comparing against a wrapped sum. Both paths
are unreachable through validated callers, so compressed output is
byte-identical.
Remaining zstd_compress.c work: parameter selection and validation,
context lifecycle, block dispatch, dictionary loading, and the streaming
state machine.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets -- -D
warnings && cargo test --all-targets (118 tests, includes new frame
header/skippable/last-block reference vectors)
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke
Move the context-free public ZSTD_compressBound ABI into Rust while leaving
the stateful compressor context in C. The Rust implementation preserves the
size-width-specific input limit, macro arithmetic, and srcSize_wrong error
contract used by C callers.
Test Plan:
- cargo clippy
- cargo clippy --benches
- cargo clippy --tests
- cargo +nightly fmt
- cargo test zstd_compress_api::tests -- --nocapture
- cargo test --target i686-unknown-linux-gnu zstd_compress_api::tests
- make -B -C lib -j2 libzstd.a
- fuzzer -s5346 -i1 --no-big-tests
Refs: public ZSTD_COMPRESSBOUND macro in lib/zstd.h
this is a prototype definition error:
`_mm_storeu_si128()` should accept a `void*` pointer,
since it explicitly states that it accepts unaligned addresses
yet requiring a `__m128i*` tells otherwise, and requires the compiler the enforce this alignment.
seems like a prototype interface error:
input parameter should have been `const void*`,
since the documentation is explicit that input doesn't have to be aligned,
but `const __m256i*` makes the compiler enforce it.
do not solve the equation, even though some members cancel each other,
this is done for clarity,
we'll let the compiler do the resolution at compile time.
Move towards a stronger guarantee of reproducibility by removing this small difference for machines without SSE2/Neon.
The SIMD behavior is now the default for all platforms.
The optimal parser with LDM enabled using minMatch > 3 could generate a match
length of 3 when minMatch >= 4. This is not allowed.
1. Fix the bug
2. Add validation logic to `ZSTD_buildSeqStore()` in debug mode for all block
compressors that checks we never generate too short a match. This way we don't
rely on the `generate_sequences` fuzzer to find this issue.
Credit to OSS-Fuzz
ddidderr
Yann Collet