Move the hidden fullbench block-decoder entrypoints and the public
ZSTD_decompressBlock compatibility wrapper into Rust. The C translation unit
now only projects the configuration-dependent private DCtx fields into the
Rust block context, preserving the existing ABI and decoder feature modes.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --tests -- -D warnings
- make -B -C lib -j2 lib
- make -B -C tests -j2 test-rust-lib-smoke
- make -B -C tests -j2 test-cli-tests
Legacy detection, frame-size queries, one-shot decode, and buffered streaming
were still routed through the C header dispatcher even though every historical
v0.1 through v0.7 decoder is already ported under Rust feature gates. The C
implementation also duplicated the support-level boundary independently from
the Rust archive selection.
Move the dispatcher and stream lifecycle to Rust using the existing DCtx field
projection. Rust now selects the version, preserves the null-buffer and
position semantics, initializes and frees buffered legacy contexts, and calls
the frozen per-version modules. C retains only the configuration leaf that
reports ZSTD_LEGACY_SUPPORT plus the private DCtx layout and trace helpers, so
library, program, and test builds can still choose different support levels.
Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03,legacy-v04,legacy-v05,legacy-v06,legacy-v07 --all-targets -- --test-threads=1` -- 528 passed.
- Native `test-legacy`, `test-invalidDictionaries`, `test-decodecorpus`, and `test-zstd` -- passed.
- Strict C syntax checks with legacy support 0, 4, 5, and 7 -- passed.
- `cargo clippy` default, benches, and tests for library and CLI -- passed.
Port sequence-decoder selection and offset-history policy into Rust. The
Rust block decoder now handles short/long selection, offset-table analysis,
history thresholds, and prefetch sequencing; C projects only the decoder mode
and private context fields required by the ABI.
Test Plan:
- cargo test --manifest-path rust/Cargo.toml --all-targets -- --test-threads=1
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=1 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=0
- make -B -C lib -j2 lib ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT=0 ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG=1
- make -B -C tests -j2 test-zstd
Move the legacy buffered-stream dispatcher out of the C decompression adapter
while keeping ZSTD_DCtx_s private and C-owned. Rust now consumes the existing
ZSTD_rustDctxView projection, detects the legacy version at input.pos, preserves
static-context rejection and stream-stage reset behavior, and handles legacy
context replacement through the version-gated v0.4-v0.7 Rust stream APIs.
The Rust boundary retains the C helper's NULL normalization, dictionary
lifetime sentinel, previous-version cleanup semantics, input/output position
updates, and version-unsupported and no-legacy-support fallbacks. The C legacy
free helper remains unchanged; only the dispatcher declaration and body were
removed from the adapter. Focused tests cover unsupported versions, stage
reset, context reuse/switching, and partial v0.4 input/output progress.
Test Plan:
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression` -- passed
- Same clippy command with `--benches` and `--tests` -- passed
- `cargo +nightly fmt --manifest-path rust/Cargo.toml`, followed by all three clippy commands -- passed
- Decompression/legacy clippy with `-D warnings` -- passed
- All 107 Rust tests with v0.4-v0.7 features -- passed
- Focused legacy dispatcher tests: 4 passed
- `make -C lib -j2 lib-mt` and `make -C lib -j2 lib-nomt` -- passed
- `make -C tests -j2 test-legacy` -- built, then failed in the existing simple one-shot path with one v0.8 output byte mismatch (`e0` vs expected `e2`) before the streaming check; the new dispatcher symbol was not called on that failing path
Move long-distance matching and high-level decompression from C shims into
Rust. The decoder now owns context, dictionary, parameter, one-shot, and
buffered streaming state while C retains allocation/configuration, legacy,
and trace leaves.
Move CLI parsing, safety policy, and dispatch into a separate Rust static
archive. Keeping it separate prevents library builds from retaining FIO
symbols, while C continues to own file opening, replacement, and I/O.
Program targets now select matching compression/decompression archives.
The remaining C boundary is intentional: high-level compression, optimal
parsing, dictionary building, legacy callbacks, and CLI file I/O still need
migration.
Test Plan:
- cargo test --all-targets (native and i686)
- cargo test --all-targets in rust/cli (native and i686)
- CLI crate compression-only and decompression-only feature tests
- native and i686 fuzzer/zstreamtest runs, plus legacy and dictionary tests
- ZSTD_C_PREDICT and ZSTD_HEAPMODE=0 fuzzer coverage
- library, dynamic-link, and program-target build/round-trip matrix
Refs: rust/README.md
Move literal decoding, sequence-table construction, FSE sequence decoding, and
sequence execution into Rust. The C shim retains ownership of the configured
decoder context and passes only the leaf state needed by the block codec.
Correct the two high offset-code bases while porting the tables. Their prior
values made valid large-window streams decode as corrupted data; the new unit
test fixes the exact values and a native zstream regression exercises them.
High-level frame and streaming context control remains C for now.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- timeout 20s stdbuf -oL ./tests/poolTests
Refs: rust/README.md
Move ZSTD_DDict allocation, ownership, construction, and public ABI exports
into Rust. The C decoder keeps ZSTD_loadDEntropy until its table loader moves.
The C shim now exposes C-computed DCtx field offsets. This keeps dictionary
state correct across conditional C layouts, including fuzz fields and 32-bit
static-BMI2 builds, without duplicating decoder-context layout in Rust.
Test Plan:
- cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests
- cargo test --all-targets and cargo build --release
- cargo test/build --target i686-unknown-linux-gnu
- C fuzzer, zstreamtest, invalidDictionaries, and fuzzer32 smoke runs
- fuzz-enabled and i686 -mbmi2 C/Rust dictionary roundtrip harnesses
Refs: rust/README.md
Move Huffman decoding-table construction and X1/X2 single- and four-stream
decoding into Rust. The original C translation unit is now a declaration shim,
so unchanged C callers and tests resolve the public decoder ABI from the Rust
archive.
The portable implementation preserves the C workspace and error contracts,
retains the existing assembly-only internal fast loops, and mirrors the C
32-bit decode cadence so X2 writes remain within its required output window.
Forced X1/X2 modes follow the matching archive configuration from the build
integration.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets for default, forced X1, and forced X2 modes
- cargo build and cargo test for i686-unknown-linux-gnu in all three modes
- Strict normal, X1, and X2 C shim compilation
- Original fuzzer and fullbench compatibility targets
Refs: rust/README.md
Depends-on: d89ebb31
Mark that `huf_decompress_amd64.S` supports BTI and PAC, which it trivially does because it is empty for aarch64.
The issue only requested BTI markings, but it also makes sense to mark PAC, which is the only other feature.
Also run add a test for this mode to the ARM64 QEMU test. Before this PR it warns on `huf_decompress_amd64.S`, after it doesn't.
Fixes Issue #3841.
in this new method, when an `offset==0` is detected,
it's converted into (size_t)(-1), instead of 1.
The logic is that (size_t)(-1) is effectively an extremely large positive number,
which will not pass the offset distance test at next stage (`execSequence()`).
Checked the source code, and offset is always checked (as it should),
using a formula which is not vulnerable to arithmetic overflow:
```
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart),
```
The benefit is that such a case (offset==0) is always detected as corrupted data
as opposed to relying on the checksum to detect the error.
The patch adds a validation to ensure that the last field, which is
reserved, must be all-zeroes in ZSTD_decodeSeqHeaders. This prevents
potential corruption from going undetected.
Fixes an issue where corrupted input could lead to undefined behavior
due to improper validation of reserved bits.
Signed-off-by: aimuz <mr.imuz@gmail.com>
This PR introduces no functional changes. It attempts to change all
macros currently using `{ }` or some variant of that to to
`do { } while (0)`, and introduces trailing `;` where necessary.
There were no bugs found during this migration.
The bug in Visual Studios warning on this has been fixed since VS2015.
Additionally, we have several instances of `do { } while (0)` which have
been present for several releases, so we don't have to worry about
breaking peoples builds.
Fixes Issue #3830.
`HUF_DecompressFastArgs_init()` was adding 0 to NULL. Fix it by exiting
early for empty outputs. This is no change in behavior, because the
function was already exiting 0 in this case, just slightly later.
* Rename `ilimit` to `ilowest` and set it equal to `src` instead of
`src + 6 + 8`. This is safe because the fast decoding loops guarantee
to never read below `ilowest` already. This allows the fast decoder to
run for at least two more iterations, because it consumes at most 7
bytes per iteration.
* Continue the fast loop all the way until the number of safe iterations
is 0. Initially, I thought that when it got towards the end, the
computation of how many iterations of safe might become expensive. But
it ends up being slower to have to decode each of the 4 streams
individually, which makes sense.
This drastically speeds up the Huffman decoder on the `github` dataset
for the issue raised in #3762, measured with `zstd -b1e1r github/`.
| Decoder | Speed before | Speed after |
|----------|--------------|-------------|
| Fallback | 477 MB/s | 477 MB/s |
| Fast C | 384 MB/s | 492 MB/s |
| Assembly | 385 MB/s | 501 MB/s |
We can also look at the speed delta for different block sizes of silesia
using `zstd -b1e1r silesia.tar -B#`.
| Decoder | -B1K ∆ | -B2K ∆ | -B4K ∆ | -B8K ∆ | -B16K ∆ | -B32K ∆ | -B64K ∆ | -B128K ∆ |
|----------|--------|--------|--------|--------|---------|---------|---------|----------|
| Fast C | +11.2% | +8.2% | +6.1% | +4.4% | +2.7% | +1.5% | +0.6% | +0.2% |
| Assembly | +12.5% | +9.0% | +6.2% | +3.6% | +1.5% | +0.7% | +0.2% | +0.03% |
gcc in the linux kernel was not unrolling the inner loops of the Huffman
decoder, which was destroying decoding performance. The compiler was
generating crazy code with all sorts of branches. I suspect because of
Spectre mitigations, but I'm not certain. Once the loops were manually
unrolled, performance was restored.
Additionally, when gcc couldn't prove that the variable left shift in
the 4X2 decode loop wasn't greater than 63, it inserted checks to verify
it. To fix this, mask `entry.nbBits & 0x3F`, which allows gcc to eliete
this check. This is a no op, because `entry.nbBits` is guaranteed to be
less than 64.
Lastly, introduce the `HUF_DISABLE_FAST_DECODE` macro to disable the
fast C loops for Issue #3762. So if even after this change, there is a
performance regression, users can opt-out at compile time.
* Remove all pointer-overflow suppressions from our UBSAN builds/tests.
* Add `ZSTD_ALLOW_POINTER_OVERFLOW_ATTR` macro to suppress
pointer-overflow at a per-function level. This is a superior approach
because it also applies to users who build zstd with UBSAN.
* Add `ZSTD_wrappedPtr{Diff,Add,Sub}()` that use these suppressions.
The end goal is to only tag these functions with
`ZSTD_ALLOW_POINTER_OVERFLOW`. But we can start by annoting functions
that rely on pointer overflow, and gradually transition to using
these.
* Add `ZSTD_maybeNullPtrAdd()` to simplify pointer addition when the
pointer may be `NULL`.
* Fix all the fuzzer issues that came up. I'm sure there will be a lot
more, but these are the ones that came up within a few minutes of
running the fuzzers, and while running GitHub CI.
Refine the macro guards to define the functions exactly when they are
needed.
This fixes the chromium build with zstd.
Thanks to @GregTho for reporting!
The sequence section starts with a number, which tells how sequences are present in the section.
If this number if 0, the section automatically ends.
The number 0 can be represented using the 1 byte or the 2 bytes formats.
That's because the 2-bytes formats fully overlaps the 1 byte format.
However, when 0 is represented using the 2-bytes format,
the decoder was expecting the sequence section to continue,
and was looking for FSE tables, which is incorrect.
Fixed this behavior, in both the reference decoder and the educational behavior.
In practice, this behavior never happens,
because the encoder will always select the 1-byte format to represent 0,
since this is more efficient.
Completed the fix with a new golden sample for tests,
a clarification of the specification,
and a decoder errata paragraph.
Reduces memory when blocks are guaranteed to be smaller than allowed by
the format. This is useful for streaming compression in conjunction with
ZSTD_c_maxBlockSize.
This PR saves 2 * (formatMaxBlockSize - paramMaxBlockSize) when streaming.
Once it is rebased on top of PR #3616 it will save
3 * (formatMaxBlockSize - paramMaxBlockSize).
The split literals buffer patch increased streaming decompression memory
by 64KB (shrunk lit buffer from 128KB to 64KB, and added 128KB). This
patch removes the added 128KB buffer, because it isn't necessary.
The buffer was there because the literals compression code didn't know
the true `blockSizeMax` of the frame, and always put split literals so
they ended 128KB - 32 from the beginning of the block. Instead, we can
pass down the true `blockSizeMax` and ensure that the split literals
end up at `blockSizeMax - 32` from the beginning of the block. We
already reserve a full `blockSizeMax` bytes in streaming mode, so we
won't be overwriting the extDict window.
ddidderr
Pavel P