Move long-distance matching and high-level decompression from C shims into
Rust. The decoder now owns context, dictionary, parameter, one-shot, and
buffered streaming state while C retains allocation/configuration, legacy,
and trace leaves.
Move CLI parsing, safety policy, and dispatch into a separate Rust static
archive. Keeping it separate prevents library builds from retaining FIO
symbols, while C continues to own file opening, replacement, and I/O.
Program targets now select matching compression/decompression archives.
The remaining C boundary is intentional: high-level compression, optimal
parsing, dictionary building, legacy callbacks, and CLI file I/O still need
migration.
Test Plan:
- cargo test --all-targets (native and i686)
- cargo test --all-targets in rust/cli (native and i686)
- CLI crate compression-only and decompression-only feature tests
- native and i686 fuzzer/zstreamtest runs, plus legacy and dictionary tests
- ZSTD_C_PREDICT and ZSTD_HEAPMODE=0 fuzzer coverage
- library, dynamic-link, and program-target build/round-trip matrix
Refs: rust/README.md
Move literal decoding, sequence-table construction, FSE sequence decoding, and
sequence execution into Rust. The C shim retains ownership of the configured
decoder context and passes only the leaf state needed by the block codec.
Correct the two high offset-code bases while porting the tables. Their prior
values made valid large-window streams decode as corrupted data; the new unit
test fixes the exact values and a native zstream regression exercises them.
High-level frame and streaming context control remains C for now.
Test Plan:
- cargo test --all-targets
- cargo test --target i686-unknown-linux-gnu --all-targets
- cargo clippy && cargo clippy --benches && cargo clippy --tests
- cargo +nightly fmt
- make -B -C tests -j2 fuzzer zstreamtest invalidDictionaries poolTests
- ./tests/fuzzer -s5346 -i1 --no-big-tests
- ./tests/zstreamtest -i3000 -s334462
- ./tests/invalidDictionaries
- timeout 20s stdbuf -oL ./tests/poolTests
Refs: rust/README.md
Move ZSTD_DDict allocation, ownership, construction, and public ABI exports
into Rust. The C decoder keeps ZSTD_loadDEntropy until its table loader moves.
The C shim now exposes C-computed DCtx field offsets. This keeps dictionary
state correct across conditional C layouts, including fuzz fields and 32-bit
static-BMI2 builds, without duplicating decoder-context layout in Rust.
Test Plan:
- cargo fmt, cargo clippy, cargo clippy --benches, cargo clippy --tests
- cargo test --all-targets and cargo build --release
- cargo test/build --target i686-unknown-linux-gnu
- C fuzzer, zstreamtest, invalidDictionaries, and fuzzer32 smoke runs
- fuzz-enabled and i686 -mbmi2 C/Rust dictionary roundtrip harnesses
Refs: rust/README.md
Move Huffman decoding-table construction and X1/X2 single- and four-stream
decoding into Rust. The original C translation unit is now a declaration shim,
so unchanged C callers and tests resolve the public decoder ABI from the Rust
archive.
The portable implementation preserves the C workspace and error contracts,
retains the existing assembly-only internal fast loops, and mirrors the C
32-bit decode cadence so X2 writes remain within its required output window.
Forced X1/X2 modes follow the matching archive configuration from the build
integration.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets for default, forced X1, and forced X2 modes
- cargo build and cargo test for i686-unknown-linux-gnu in all three modes
- Strict normal, X1, and X2 C shim compilation
- Original fuzzer and fullbench compatibility targets
Refs: rust/README.md
Depends-on: d89ebb31
Mark that `huf_decompress_amd64.S` supports BTI and PAC, which it trivially does because it is empty for aarch64.
The issue only requested BTI markings, but it also makes sense to mark PAC, which is the only other feature.
Also run add a test for this mode to the ARM64 QEMU test. Before this PR it warns on `huf_decompress_amd64.S`, after it doesn't.
Fixes Issue #3841.
in this new method, when an `offset==0` is detected,
it's converted into (size_t)(-1), instead of 1.
The logic is that (size_t)(-1) is effectively an extremely large positive number,
which will not pass the offset distance test at next stage (`execSequence()`).
Checked the source code, and offset is always checked (as it should),
using a formula which is not vulnerable to arithmetic overflow:
```
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart),
```
The benefit is that such a case (offset==0) is always detected as corrupted data
as opposed to relying on the checksum to detect the error.
The patch adds a validation to ensure that the last field, which is
reserved, must be all-zeroes in ZSTD_decodeSeqHeaders. This prevents
potential corruption from going undetected.
Fixes an issue where corrupted input could lead to undefined behavior
due to improper validation of reserved bits.
Signed-off-by: aimuz <mr.imuz@gmail.com>
This PR introduces no functional changes. It attempts to change all
macros currently using `{ }` or some variant of that to to
`do { } while (0)`, and introduces trailing `;` where necessary.
There were no bugs found during this migration.
The bug in Visual Studios warning on this has been fixed since VS2015.
Additionally, we have several instances of `do { } while (0)` which have
been present for several releases, so we don't have to worry about
breaking peoples builds.
Fixes Issue #3830.
`HUF_DecompressFastArgs_init()` was adding 0 to NULL. Fix it by exiting
early for empty outputs. This is no change in behavior, because the
function was already exiting 0 in this case, just slightly later.
* Rename `ilimit` to `ilowest` and set it equal to `src` instead of
`src + 6 + 8`. This is safe because the fast decoding loops guarantee
to never read below `ilowest` already. This allows the fast decoder to
run for at least two more iterations, because it consumes at most 7
bytes per iteration.
* Continue the fast loop all the way until the number of safe iterations
is 0. Initially, I thought that when it got towards the end, the
computation of how many iterations of safe might become expensive. But
it ends up being slower to have to decode each of the 4 streams
individually, which makes sense.
This drastically speeds up the Huffman decoder on the `github` dataset
for the issue raised in #3762, measured with `zstd -b1e1r github/`.
| Decoder | Speed before | Speed after |
|----------|--------------|-------------|
| Fallback | 477 MB/s | 477 MB/s |
| Fast C | 384 MB/s | 492 MB/s |
| Assembly | 385 MB/s | 501 MB/s |
We can also look at the speed delta for different block sizes of silesia
using `zstd -b1e1r silesia.tar -B#`.
| Decoder | -B1K ∆ | -B2K ∆ | -B4K ∆ | -B8K ∆ | -B16K ∆ | -B32K ∆ | -B64K ∆ | -B128K ∆ |
|----------|--------|--------|--------|--------|---------|---------|---------|----------|
| Fast C | +11.2% | +8.2% | +6.1% | +4.4% | +2.7% | +1.5% | +0.6% | +0.2% |
| Assembly | +12.5% | +9.0% | +6.2% | +3.6% | +1.5% | +0.7% | +0.2% | +0.03% |
gcc in the linux kernel was not unrolling the inner loops of the Huffman
decoder, which was destroying decoding performance. The compiler was
generating crazy code with all sorts of branches. I suspect because of
Spectre mitigations, but I'm not certain. Once the loops were manually
unrolled, performance was restored.
Additionally, when gcc couldn't prove that the variable left shift in
the 4X2 decode loop wasn't greater than 63, it inserted checks to verify
it. To fix this, mask `entry.nbBits & 0x3F`, which allows gcc to eliete
this check. This is a no op, because `entry.nbBits` is guaranteed to be
less than 64.
Lastly, introduce the `HUF_DISABLE_FAST_DECODE` macro to disable the
fast C loops for Issue #3762. So if even after this change, there is a
performance regression, users can opt-out at compile time.
* Remove all pointer-overflow suppressions from our UBSAN builds/tests.
* Add `ZSTD_ALLOW_POINTER_OVERFLOW_ATTR` macro to suppress
pointer-overflow at a per-function level. This is a superior approach
because it also applies to users who build zstd with UBSAN.
* Add `ZSTD_wrappedPtr{Diff,Add,Sub}()` that use these suppressions.
The end goal is to only tag these functions with
`ZSTD_ALLOW_POINTER_OVERFLOW`. But we can start by annoting functions
that rely on pointer overflow, and gradually transition to using
these.
* Add `ZSTD_maybeNullPtrAdd()` to simplify pointer addition when the
pointer may be `NULL`.
* Fix all the fuzzer issues that came up. I'm sure there will be a lot
more, but these are the ones that came up within a few minutes of
running the fuzzers, and while running GitHub CI.
Refine the macro guards to define the functions exactly when they are
needed.
This fixes the chromium build with zstd.
Thanks to @GregTho for reporting!
The sequence section starts with a number, which tells how sequences are present in the section.
If this number if 0, the section automatically ends.
The number 0 can be represented using the 1 byte or the 2 bytes formats.
That's because the 2-bytes formats fully overlaps the 1 byte format.
However, when 0 is represented using the 2-bytes format,
the decoder was expecting the sequence section to continue,
and was looking for FSE tables, which is incorrect.
Fixed this behavior, in both the reference decoder and the educational behavior.
In practice, this behavior never happens,
because the encoder will always select the 1-byte format to represent 0,
since this is more efficient.
Completed the fix with a new golden sample for tests,
a clarification of the specification,
and a decoder errata paragraph.
Reduces memory when blocks are guaranteed to be smaller than allowed by
the format. This is useful for streaming compression in conjunction with
ZSTD_c_maxBlockSize.
This PR saves 2 * (formatMaxBlockSize - paramMaxBlockSize) when streaming.
Once it is rebased on top of PR #3616 it will save
3 * (formatMaxBlockSize - paramMaxBlockSize).
The split literals buffer patch increased streaming decompression memory
by 64KB (shrunk lit buffer from 128KB to 64KB, and added 128KB). This
patch removes the added 128KB buffer, because it isn't necessary.
The buffer was there because the literals compression code didn't know
the true `blockSizeMax` of the frame, and always put split literals so
they ended 128KB - 32 from the beginning of the block. Instead, we can
pass down the true `blockSizeMax` and ensure that the split literals
end up at `blockSizeMax - 32` from the beginning of the block. We
already reserve a full `blockSizeMax` bytes in streaming mode, so we
won't be overwriting the extDict window.
detected by @terrelln,
these issue could be triggered in specific scenarios
namely decompression of certain invalid magic-less frames,
or requested properties from certain invalid skippable frames.
* add check for valid dest buffer and fuzz on random dest ptr when malloc 0
* add uptrval to linux-kernel
* remove bin files
* get rid of uptrval
* restrict max pointer value check to platforms where sizeof(size_t) == sizeof(void*)
Looking at the __builtin_expect in ZSTD_decodeSequence:
{ size_t offset;
#if defined(__clang__)
if (LIKELY(ofBits > 1)) {
#else
if (ofBits > 1) {
#endif
ZSTD_STATIC_ASSERT(ZSTD_lo_isLongOffset == 1);
From profile-annotated assembly, the probability of ofBits > 1 is about 75%
(101k counts out of 135k counts). This is much smaller than the recommended
likelihood to use __builtin_expect which is 99%. As a result, clang moved the
else block further away which hurts cache locality. Removing this
__built_expect along with two others in ZSTD_decodeSequence gave better
performance when PGO is enabled. I suggest to remove these branch hints and
rely on PGO which leverages runtime profiles from actual workload to calculate
branch probability instead.
* Mark all bufferless and block level functions as deprecated
* Update documentation to suggest not using these functions
* Add `_deprecated()` wrappers for functions that we use internally and
call those instead
* Fixes zstd-dll build (https://github.com/facebook/zstd/issues/3492):
- Adds pool.o and threading.o dependency to the zstd-dll target
- Moves custom allocation functions into header to avoid needing to add dependency on common.o
- Adds test target for zstd-dll
- Adds github workflow that buildis zstd-dll
ddidderr
Pavel P