`make all` exercises substantially more than the default zstd binary. It also
builds compression-only and decompression-only archives, older contrib tools
that compile program C shims directly, and the single-file amalgamations. The
Rust migration had changed symbol ownership without updating every one of
those feature and link boundaries.
The first failure came from `fileio_asyncio`: it is always compiled, but its
codec bindings unconditionally referenced both `zstd_compress` and
`zstd_decompress`. A compression-only archive therefore required disabled
decoder modules, and the decompression-only configuration had the symmetric
problem. Gate the concrete codec imports, callback types, helpers, and exports
with their Cargo features. Keep the format probe compilable without the
legacy decoder predicate when decompression is disabled.
Once that boundary compiled, the remaining `make all` paths exposed related
integration gaps. Move the C decompression projection declarations out of the
compression preprocessor block, while leaving destination callback types
shared. Link the Rust CLI helpers archive into zlibWrapper, pzstd, and
largeNbDicts, whose util/time/data-generator C files are now declaration shims
rather than implementations.
The generated single-file C sources also call Rust-owned symbols now. Build
and link an appropriately featured Rust archive in their native smoke tests,
and include the pool configuration shim in the decoder case. The full
amalgamation combines `zstd_lazy.c` and `zstd_opt.c` into one translation unit,
so guard their otherwise translation-unit-local dictionary mode enum against
duplicate definition.
A Rust-backed amalgamation is no longer a standalone Emscripten input. Remove
the obsolete emcc/Docker path and report that limitation explicitly; restoring
the WebAssembly smoke test requires a Rust WebAssembly archive and a defined
cross-language amalgamation contract.
Test Plan:
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features compression` -- passed
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features decompression` -- passed
- `sh -n build/single_file_libs/build_decoder_test.sh build/single_file_libs/build_library_test.sh` -- passed
- `make all` -- passed, including native single-file and seekable-format tests
- `git diff --cached --check` -- passed
When two threads are using a WorkQueue and the reader thread exits due
to an error, it must call WorkQueue::finish() to wake up the writer
thread. Otherwise, if the queue is full and the writer thread is waiting
for a free slot, it could hang forever.
This can happen in pratice when decompressing a large, corrupted file
that does not contain pzstd skippable frames.
Doing this check with a direct c++ snippet is prone to portability problems:
- \043 is not portable between shells: dash expands it to #,
bash does not;
- using # directly works with make 4.3 but does not with make 4.2.
Let's just use the c++ version that covers both the code and the gtest.
Rather than remove the flag entirely, as proposed in #3499, this commit uses
the newest C++ standard the compiler supports. This retains the selection of
using only standardized features (excluding GNU extensions) and keeps the
recency requirements of the codebase explicit.
Tested with various versions of `g++` and `clang++`.
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
* Add `Portability.h` to fix min/max issues.
* Fix conversion warnings
* Assert that windowLog <= 23, which is currently always the case.
This could be loosened, but we aren't looking to add new functionality.
Fixes on top of PR #3375 by @eli-schwartz, which added Windows CI for contrib & programs.
gcc-5 didn't like the l-value overload for defaulted operator=. There is
no reason it needs to be l-value overloaded, so just remove it.
I'm not sure why the build broke for @mckaygerhard in Issue #2811, since
this code hasn't changed since it was added. But, there is no harm in
fixing it.
Fixes issue #2811.
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
`make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.
The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.
The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
These are replaced by the corresponding context resets. When
converting resetCStream, CCtx_setPledgedSrcSize isn't called if the
source size is "unknown".
This helps reduce the reliance on "static only" symbols, as well as
reducing the use of deprecated functions.
Signed-off-by: Stephen Kitt <steve@sk2.org>
pzstd requires C++11, which older C++ standard libraries like libstdc++
as used on OS X 10.8 and earlier don't support. The user might address
this by setting "CXXFLAGS=-stdlib=libc++". This flag must be used both
at compile time and at link time. Asking the user to also put the flag
in LDFLAGS is undesirable because then the flag would also be used when
linking C code, which would be inappropriate.
Whilst working on the Reproducible Builds effort [0], we noticed
that zstd could not be built reproducibly.
This is due to the manual page encoding the number of CPUs from the
build machine and thus varies across builds.
This was originally filed in Debian as #897904 [1].
[0] https://reproducible-builds.org/
[1] https://bugs.debian.org/897904
Signed-off-by: Chris Lamb <lamby@debian.org>
It can have other effects, such as pulling in extra libraries. Without it,
riscv build fails with undefined reference to `__atomic_compare_exchange_1'.
Fedora now enables this option by default, resulting
in the following build failure:
Logging.h: In instantiation of
'void pzstd::Logger::operator()(int, const char*, Args ...)
Pzstd.cpp:413:48: required from here
Logging.h:46:17: error: format not a string literal and no format arguments
[-Werror=format-security]
std::fprintf(out_, fmt, args...);
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~
ddidderr
