The legacy decoders (lib/legacy/zstd_v01.c .. zstd_v07.c) are next in
the Rust migration. Each of those files is a frozen snapshot of the
FSE/Huff0 entropy coders and frame logic of one historical release, so
their ports must not reuse the modern Rust entropy modules and must not
share code with each other: outputs and error codes have to stay
byte-identical to the frozen C forever. This commit installs the
build-system scaffolding so seven per-version ports can land
independently, each adding only its own module file plus a one-line
registration in rust/src/legacy/mod.rs.
Cargo grows features legacy-v01 .. legacy-v07. They are never default
features: the C build defaults differ per build system, so each build
system passes the list explicitly, derived from its own legacy
configuration:
- lib/Makefile and programs/Makefile map ZSTD_LEGACY_SUPPORT=N to the
features for versions N..7 (0 disables legacy), mirroring the
ZSTD_LEGACY_FILES selection in lib/libzstd.mk.
- tests/Makefile always enables all seven features because its
ZSTDLEGACY_FILES wildcard compiles every lib/legacy/*.c regardless of
the dispatch level.
- build/meson maps legacy_level exactly like the makefiles; build/cmake
enables all seven whenever ZSTD_LEGACY_SUPPORT is ON because it
always compiles all seven C files (ZSTD_LEGACY_LEVEL only selects the
C dispatch).
Every build system also encodes the legacy selection in the Rust target
directory name (e.g. c1-d1-default-legacy5), for the same reason the
HUF mode is encoded there: a cached archive built for one configuration
must never be linked into a build expecting another. In tests/Makefile
the legacy level additionally flows into the existing HUF C-mode stamp,
so the flat C test objects (which bake -DZSTD_LEGACY_SUPPORT into the
dispatch) are rebuilt whenever the level changes. In programs/Makefile
the compress-only, decompress-only, and CLI archives keep
level-independent directories (RUST_HUF_MODE) because they are only
linked into ZSTD_LEGACY_SUPPORT=0 program variants and carry no legacy
features.
A feature whose version has not been ported yet gates nothing: the
module registration in rust/src/legacy/mod.rs is added by each port,
so enabling e.g. legacy-v05 today simply leaves that decoder in C.
This is what makes mixed C/Rust legacy levels link cleanly while the
seven ports land in any order.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
-- -D warnings && cargo test --all-targets
- cargo clippy with --no-default-features --features
decompression,legacy-v01 and with all seven legacy features
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke; make -C tests test-legacy
- make -C lib libzstd.a with ZSTD_LEGACY_SUPPORT=0, 1 and default (5)
- cmake configure and meson setup (including -Dlegacy_level=1) emit the
expected --features lists and legacy-suffixed target directories
Move long-distance matching and high-level decompression from C shims into
Rust. The decoder now owns context, dictionary, parameter, one-shot, and
buffered streaming state while C retains allocation/configuration, legacy,
and trace leaves.
Move CLI parsing, safety policy, and dispatch into a separate Rust static
archive. Keeping it separate prevents library builds from retaining FIO
symbols, while C continues to own file opening, replacement, and I/O.
Program targets now select matching compression/decompression archives.
The remaining C boundary is intentional: high-level compression, optimal
parsing, dictionary building, legacy callbacks, and CLI file I/O still need
migration.
Test Plan:
- cargo test --all-targets (native and i686)
- cargo test --all-targets in rust/cli (native and i686)
- CLI crate compression-only and decompression-only feature tests
- native and i686 fuzzer/zstreamtest runs, plus legacy and dictionary tests
- ZSTD_C_PREDICT and ZSTD_HEAPMODE=0 fuzzer coverage
- library, dynamic-link, and program-target build/round-trip matrix
Refs: rust/README.md
Mode-specific Rust archives exposed the test makefiles' flat C-object cache:
switching from default to forced HUF mode could relink stale default C objects
with a newly built forced Rust archive. That hybrid has incompatible decoder
table expectations and can corrupt dictionary decompression.
Track each makefile's effective HUF mode with an empty archive stamp. It is a
safe normal linker prerequisite, and every mode transition advances its mtime
so cached C objects and direct-source binaries rebuild while unchanged modes
remain incremental.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets
- Default -> forced-X1 -> default fuzzer builds without -B, each running
./fuzzer -s5346 -i1 --no-big-tests
- Equivalent zstd-small default/forced/default rebuild and --version checks
Refs: rust/README.md
Fixes: d89ebb31
Derive the Rust static-library configuration from the effective C HUF decoder
flags and place each mode in its own Cargo target directory. This prevents a
forced X1/X2 C build from silently reusing the default Rust archive.
The test and program makefiles now also build and link an i686 Rust archive
for their 32-bit targets, including zstd32. zstd-small retains its existing
default configuration while preserving an explicitly requested HUF mode.
Test Plan:
- cargo clippy; cargo clippy --benches; cargo clippy --tests
- cargo +nightly fmt, then repeat the Clippy checks
- cargo test --all-targets for default, forced X1, and forced X2 modes
- cargo build --release --target i686-unknown-linux-gnu for each mode
- make -n checks for default, forced, direct -D, and 32-bit test/CLI targets
Refs: rust/README.md
Make Rust sources normal prerequisites of native C test executables and CLI
builds. A changed Rust module now rebuilds libzstd_rs.a and relinks the target,
instead of relying on a manually prepared archive that can silently be stale.
The archive is linked after C objects, allowing migrated C shims to resolve
their Rust ABI symbols while remaining compatible with the existing makefile
flows. The migration guide now documents this behavior.
Test Plan:
- make -B -C tests fuzzer
- ./tests/fuzzer -i1 --no-big-tests
- make -B -C programs zstd
- ./programs/zstd --version
Refs: rust/README.md
The -Werror=missing-profile caused thread/zlib/lzma/lz4 detection failure
when build with profile-use, thus caused ZSTD_MULTITHREAD etc. is not
defined for profile-use, then there will be many profile mismatch information
in output and the final binary reports run error sometimes as below:
Error : ZSTD_CCtx_setParameter(ctx, ZSTD_c_nbWorkers, adv->nbWorkers) failed : Unsupported parameter
Signed-off-by: Xionghu Luo <xionghuluo@tencent.com>
- switched the patter and input of $filter into the right places
- added pattern wildcard to MSYS_NT & CYGWIN_NT as they change with windows versions
- correctly identify MSYS2, even in an env like MINGW64
They are Linux-like environments under Windows and have all the tools needed to support staged installation and testing.
Beware: this only affects the make build system.
* Fixes zstd-dll build (https://github.com/facebook/zstd/issues/3492):
- Adds pool.o and threading.o dependency to the zstd-dll target
- Moves custom allocation functions into header to avoid needing to add dependency on common.o
- Adds test target for zstd-dll
- Adds github workflow that buildis zstd-dll
In GCC, we can add a couple more flags to give us confidence that the profile
data is actually being found and used.
Also, my system for example doesn't have a binary installed under the name
`llvm-profdata`, but it does have, e.g., `llvm-profdata-13`, etc. So this
commit adds a variable that can be overridden.
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
Since your Makefile uses obj/$(HASH_DIR) for object files, this code does not work correctly for GCC. Because profiles are saved in one directory, and are expected in another when reading.
`$(RM) zstd *.o` - this line doesn't delete object files.
Clang stores profiles in the current directory, so the problem doesn't appear when compiling with Clang.
Also this code will work if BUILD_DIR is set.
* Fixing compiler warnings
* Replace the old -s flag with the -Wl,-s flag
* Fixing compiler warnings
* Fixing the linker strip flag and tests/code not working as expected on AIX
* Refactored fileio.c:
- Extracted asyncio code to fileio_asyncio.c/.h
- Moved type definitions to fileio_types.h
- Moved common macro definitions needed by both fileio.c and fileio_asyncio.c to fileio_common.h
* Bugfix - rename fileio_asycio to fileio_asyncio
* Added copyrights & license to new files
* CR fixes
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
`make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.
The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.
The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
If build_dir is set the zstd build complains about md5sum not being found.
Fix this by checking if build_dir is set before checking and using the hash tool
just like in lib/Makefile .
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
ddidderr
Po-Chuan Hsieh