The legacy decoders (lib/legacy/zstd_v01.c .. zstd_v07.c) are next in
the Rust migration. Each of those files is a frozen snapshot of the
FSE/Huff0 entropy coders and frame logic of one historical release, so
their ports must not reuse the modern Rust entropy modules and must not
share code with each other: outputs and error codes have to stay
byte-identical to the frozen C forever. This commit installs the
build-system scaffolding so seven per-version ports can land
independently, each adding only its own module file plus a one-line
registration in rust/src/legacy/mod.rs.
Cargo grows features legacy-v01 .. legacy-v07. They are never default
features: the C build defaults differ per build system, so each build
system passes the list explicitly, derived from its own legacy
configuration:
- lib/Makefile and programs/Makefile map ZSTD_LEGACY_SUPPORT=N to the
features for versions N..7 (0 disables legacy), mirroring the
ZSTD_LEGACY_FILES selection in lib/libzstd.mk.
- tests/Makefile always enables all seven features because its
ZSTDLEGACY_FILES wildcard compiles every lib/legacy/*.c regardless of
the dispatch level.
- build/meson maps legacy_level exactly like the makefiles; build/cmake
enables all seven whenever ZSTD_LEGACY_SUPPORT is ON because it
always compiles all seven C files (ZSTD_LEGACY_LEVEL only selects the
C dispatch).
Every build system also encodes the legacy selection in the Rust target
directory name (e.g. c1-d1-default-legacy5), for the same reason the
HUF mode is encoded there: a cached archive built for one configuration
must never be linked into a build expecting another. In tests/Makefile
the legacy level additionally flows into the existing HUF C-mode stamp,
so the flat C test objects (which bake -DZSTD_LEGACY_SUPPORT into the
dispatch) are rebuilt whenever the level changes. In programs/Makefile
the compress-only, decompress-only, and CLI archives keep
level-independent directories (RUST_HUF_MODE) because they are only
linked into ZSTD_LEGACY_SUPPORT=0 program variants and carry no legacy
features.
A feature whose version has not been ported yet gates nothing: the
module registration in rust/src/legacy/mod.rs is added by each port,
so enabling e.g. legacy-v05 today simply leaves that decoder in C.
This is what makes mixed C/Rust legacy levels link cleanly while the
seven ports land in any order.
Test plan:
- cd rust && cargo fmt --check && cargo clippy --all-targets
-- -D warnings && cargo test --all-targets
- cargo clippy with --no-default-features --features
decompression,legacy-v01 and with all seven legacy features
- make -C tests fuzzer && ./tests/fuzzer -i1 --no-big-tests
- make -C tests test-rust-lib-smoke; make -C tests test-legacy
- make -C lib libzstd.a with ZSTD_LEGACY_SUPPORT=0, 1 and default (5)
- cmake configure and meson setup (including -Dlegacy_level=1) emit the
expected --features lists and legacy-suffixed target directories
The dictionary-builder sources (lib/dictBuilder) are about to start moving
to Rust, beginning with divsufsort. The Rust crate previously only modeled
the compression/decompression module split plus the forced-HUF decoder
modes, so no build could express "this C configuration includes (or
excludes) dictBuilder" to Cargo. Without that, a Rust archive could carry
dictBuilder modules into a build whose C side disabled them, or worse,
omit a migrated implementation from a build whose C shims require it.
Add a `dict-builder` cargo feature and thread it through every build that
consumes the Rust static archive, mirroring exactly how each build system
already gates the dictBuilder C sources:
- rust/Cargo.toml: new `dict-builder` feature, included in the default
set because the C library builds dictBuilder by default
(ZSTD_LIB_DICTBUILDER ?= 1). The feature is empty until the first
dictBuilder module lands.
- lib/Makefile: RUST_CARGO_FEATURES gains dict-builder when
ZSTD_LIB_DICTBUILDER is enabled, following the existing
ZSTD_LIB_COMPRESSION/ZSTD_LIB_DECOMPRESSION pattern. The archive
directory naming grows a matching `b<0|1>` dimension
(c1-d1-b1-default etc.) so differently configured archives never
collide; the repeated config prefix is factored into
RUST_MODULE_CONFIG.
- programs/Makefile: the full-featured archives now request
compression,decompression,dict-builder (equal to the default set, so
the target directory stays shared with tests). The partial-library
variants gain the `b0` name dimension, and zstd-dictBuilder gets its
own lib-c1-d0-b1 archive because it compiles the dictBuilder C sources
without decompression; it previously shared the compression-only
archive, which will lack the migrated dictBuilder symbols.
- tests/Makefile: no flag change needed since tests use the crate default
feature set; a comment now records that dict-builder arrives that way.
- build/cmake/lib/CMakeLists.txt: ZSTD_BUILD_DICTBUILDER now adds the
dict-builder feature and a `b<0|1>` component in the Rust build-config
directory name, in lockstep with the DictBuilderSources gating.
- build/meson/lib/meson.build: meson compiles the dictBuilder sources
unconditionally, so the feature list and config name gain dict-builder
unconditionally (c1-d1-b1-<huf-mode>).
The `dict-builder` feature deliberately does not imply `compression`.
lib/Makefile forces ZSTD_LIB_DICTBUILDER=0 when compression is disabled,
but CMake does not couple the two options, so encoding the C-side
constraint in Cargo would make the Rust archive diverge from the C source
list in that (already unsupported) CMake configuration.
Test plan:
- cd rust && cargo build --release
- cargo build --release --no-default-features \
--features compression,decompression
- cargo build --release --no-default-features \
--features compression,dict-builder
- Full validation (fuzzer, smoke tests, dictionary byte-identity) runs
with the follow-up commit that ports divsufsort onto this scaffolding.
Build feature-matched Rust archives for the native Make library path. Flatten
their members into libzstd.a so static consumers resolve C-to-Rust and
Rust-to-C references in one archive, and whole-archive link shared outputs so
every migrated public ABI export remains available.
Rust compression and decompression features now mirror the C partial-library
switches. This prevents compression-only artifacts from retaining DDict
references to decompression-only C helpers.
Add a C archive smoke test that round-trips data and exercises the DDict
lifecycle through lib/libzstd.a, rather than direct test-object links.
Test Plan:
- cargo fmt, strict Clippy, cargo test --all-targets, release build
- Rust checks with compression-only, decompression-only, and no features
- full, partial, forced-HUF, and 32-bit static/shared Make library probes
- test-rust-lib-smoke, fuzzer, zstreamtest, and invalidDictionaries
Refs: rust/README.md
- switched the patter and input of $filter into the right places
- added pattern wildcard to MSYS_NT & CYGWIN_NT as they change with windows versions
- correctly identify MSYS2, even in an env like MINGW64
Multi-threaded static library require -pthread to correctly link and works.
The pkg-config we provide tho only works with dynamic multi-threaded library
and won't provide the correct libs and cflags values if lib-mt is used.
To handle this, introduce an env variable MT to permit advanced user to
install and generate a correct pkg-config file for lib-mt or detect if
lib-mt target is called.
With MT env set on calling make install-pc, libzstd.pc.in is a
pkg-config file for a multi-threaded static library.
On calling make lib-mt, a libzstd.pc is generated for a multi-threaded
static library as it's what asked by the user by forcing it.
libzstd.pc is changed to PHONY to force regeneration of it on calling
lib targets or install-pc to handle case where the same directory is
used for mixed compilation.
This was notice while migrating from meson to make build system where
meson generates a correct .pc file while make doesn't.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
They are Linux-like environments under Windows and have all the tools needed to support staged installation and testing.
Beware: this only affects the make build system.
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
`make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.
The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.
The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
Add the libzstd.pc target to the lib target in lib/Makefile, which makes
it inherit LDFLAGS_DYNLIB from the lib-mt target. This allows us to add
a Libs.private field to libzstd.pc which gets conditionally populated
with '-pthread'.
The 1.5.0 release notes mention that the static library isn't
multi-threaded by default, due to concern for people building static
binaries with libzstd:
Now the dynamic library supports multi-threaded compression by
default. Note that this property is not extended to the static
library because doing so would have impacted the build script of
existing client applications (requiring them to add -pthread to their
recipe), thus potentially breaking their build.
To get closer to being able to enable multi-threading for all library
builds by default, this commit makes it so that any libzstd consumer
using pkg-config gets the correct flags.
We also fix the indentation of the rule for libzstd.pc and move it
outside the if/endif block for install rules (which uses a list of OSs
where the rules were validated), so the rule is available for all users
of the 'lib*' targets.
Even with -fvisibility=hidden added to CFLAGS, any symbol which is
given a default visibility attribute ends up exported in the dynamic
library. This happens through zstd_internal.h which defines
..._STATIC_LINKING_ONLY before including various header files, and is
included for example in lib/common/pool.c.
To avoid this, this patch distinguishes static and non-static APIs, by
using ZSTDLIB_API only for the latter, and introducing
ZSTDLIB_STATIC_API for the former. For now, both are exported, but
non-static APIs can be hidden by overriding the definition
ZSTDLIB_STATIC_API. lib/Makefile is modified to allow this using
make CPPFLAGS_DYNLIB=-DZSTDLIB_STATIC_API=ZSTDLIB_HIDDEN
In addition, API declarations are dropped from zstd_compress.c (they
aren't needed there).
Signed-off-by: Stephen Kitt <steve@sk2.org>
`zstd_errors.h` and `zdict.h` are public headers, so they deserve to be
in the root `lib/` directory with `zstd.h`, not mixed in with our private
headers.
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
ddidderr
