ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

fix(build): restore complete mixed Rust build matrix

Browse Source 
`make all` exercises substantially more than the default zstd binary. It also
builds compression-only and decompression-only archives, older contrib tools
that compile program C shims directly, and the single-file amalgamations. The
Rust migration had changed symbol ownership without updating every one of
those feature and link boundaries.

The first failure came from `fileio_asyncio`: it is always compiled, but its
codec bindings unconditionally referenced both `zstd_compress` and
`zstd_decompress`. A compression-only archive therefore required disabled
decoder modules, and the decompression-only configuration had the symmetric
problem. Gate the concrete codec imports, callback types, helpers, and exports
with their Cargo features. Keep the format probe compilable without the
legacy decoder predicate when decompression is disabled.

Once that boundary compiled, the remaining `make all` paths exposed related
integration gaps. Move the C decompression projection declarations out of the
compression preprocessor block, while leaving destination callback types
shared. Link the Rust CLI helpers archive into zlibWrapper, pzstd, and
largeNbDicts, whose util/time/data-generator C files are now declaration shims
rather than implementations.

The generated single-file C sources also call Rust-owned symbols now. Build
and link an appropriately featured Rust archive in their native smoke tests,
and include the pool configuration shim in the decoder case. The full
amalgamation combines `zstd_lazy.c` and `zstd_opt.c` into one translation unit,
so guard their otherwise translation-unit-local dictionary mode enum against
duplicate definition.

A Rust-backed amalgamation is no longer a standalone Emscripten input. Remove
the obsolete emcc/Docker path and report that limitation explicitly; restoring
the WebAssembly smoke test requires a Rust WebAssembly archive and a defined
cross-language amalgamation contract.

Test Plan:
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features compression` -- passed
- `cargo check --manifest-path rust/Cargo.toml --no-default-features --features decompression` -- passed
- `sh -n build/single_file_libs/build_decoder_test.sh build/single_file_libs/build_library_test.sh` -- passed
- `make all` -- passed, including native single-file and seekable-format tests
- `git diff --cached --check` -- passed
This commit is contained in:
ddidderr
2026-07-22 06:56:06 +02:00
parent 701cf40471
commit da0e2c8380
9 changed files with 129 additions and 124 deletions
Download Patch File Download Diff File Expand all files Collapse all files
build/single_file_libs/build_decoder_test.sh
+14 -55
View File
@@ -3,58 +3,8 @@
# Temporary compiled binary
OUT_FILE="tempbin"
# Optional temporary compiled WebAssembly
OUT_WASM="temp.wasm"
# Source files to compile using Emscripten.
IN_FILES="examples/emscripten.c"
# Emscripten build using emcc.
emscripten_emcc_build() {
# Compile the same example as above
CC_FLAGS="-Wall -Wextra -Wshadow -Werror -Os -g0 -flto"
emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
# Did compilation work?
if [ $? -ne 0 ]; then
echo "Compiling ${IN_FILES}: FAILED"
exit 1
fi
echo "Compiling ${IN_FILES}: PASSED"
rm -f $OUT_WASM
}
# Emscripten build using docker.
emscripten_docker_build() {
docker container run --rm \
--volume $PWD:/code \
--workdir /code \
emscripten/emsdk:latest \
emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
# Did compilation work?
if [ $? -ne 0 ]; then
echo "Compiling ${IN_FILES} (using docker): FAILED"
exit 1
fi
echo "Compiling ${IN_FILES} (using docker): PASSED"
rm -f $OUT_WASM
}
# Try Emscripten build using emcc or docker.
try_emscripten_build() {
which emcc > /dev/null
if [ $? -eq 0 ]; then
emscripten_emcc_build
return $?
fi
which docker > /dev/null
if [ $? -eq 0 ]; then
emscripten_docker_build
return $?
fi
echo "(Skipping Emscripten test)"
}
RUST_TARGET_DIR="../../rust/target/single-file-decompress"
RUST_ARCHIVE="$RUST_TARGET_DIR/release/libzstd_rs.a"
# Amalgamate the sources
./create_single_file_decoder.sh
@@ -65,8 +15,17 @@ if [ $? -ne 0 ]; then
fi
echo "Single file decoder creation script: PASSED"
# Build the Rust decoder implementation referenced by the amalgamated C shims.
${CARGO:-cargo} build --manifest-path ../../rust/Cargo.toml --release \
--target-dir "$RUST_TARGET_DIR" --no-default-features --features decompression
if [ $? -ne 0 ]; then
echo "Building Rust decoder archive: FAILED"
exit 1
fi
# Compile the generated output
cc -Wall -Wextra -Wshadow -Werror -Os -g0 -o $OUT_FILE examples/simple.c
cc -Wall -Wextra -Wshadow -Werror -Os -g0 -I../../lib -o $OUT_FILE \
examples/simple.c ../../lib/common/pool.c "$RUST_ARCHIVE"
# Did compilation work?
if [ $? -ne 0 ]; then
echo "Compiling simple.c: FAILED"
@@ -85,7 +44,7 @@ if [ $retVal -ne 0 ]; then
fi
echo "Running simple.c: PASSED"
# Try Emscripten build if emcc or docker command is available.
try_emscripten_build
# The generated C now calls Rust, so it is no longer a standalone Emscripten input.
echo "(Skipping Emscripten test for Rust-backed amalgamation)"
exit 0
build/single_file_libs/build_library_test.sh
+15 -55
View File
@@ -6,58 +6,8 @@ ZSTD_SRC_ROOT="../../lib"
# Temporary compiled binary
OUT_FILE="tempbin"
# Optional temporary compiled WebAssembly
OUT_WASM="temp.wasm"
# Source files to compile using Emscripten.
IN_FILES="zstd.c examples/roundtrip.c"
# Emscripten build using emcc.
emscripten_emcc_build() {
# Compile the same example as above
CC_FLAGS="-Wall -Wextra -Wshadow -Werror -Os -g0 -flto"
emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
# Did compilation work?
if [ $? -ne 0 ]; then
echo "Compiling ${IN_FILES}: FAILED"
exit 1
fi
echo "Compiling ${IN_FILES}: PASSED"
rm -f $OUT_WASM
}
# Emscripten build using docker.
emscripten_docker_build() {
docker container run --rm \
--volume $PWD:/code \
--workdir /code \
emscripten/emsdk:latest \
emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
# Did compilation work?
if [ $? -ne 0 ]; then
echo "Compiling ${IN_FILES} (using docker): FAILED"
exit 1
fi
echo "Compiling ${IN_FILES} (using docker): PASSED"
rm -f $OUT_WASM
}
# Try Emscripten build using emcc or docker.
try_emscripten_build() {
which emcc > /dev/null
if [ $? -eq 0 ]; then
emscripten_emcc_build
return $?
fi
which docker > /dev/null
if [ $? -eq 0 ]; then
emscripten_docker_build
return $?
fi
echo "(Skipping Emscripten test)"
}
RUST_TARGET_DIR="../../rust/target/single-file-library"
RUST_ARCHIVE="$RUST_TARGET_DIR/release/libzstd_rs.a"
# Amalgamate the sources
./create_single_file_library.sh
@@ -72,8 +22,18 @@ echo "Single file library creation script: PASSED"
cp "$ZSTD_SRC_ROOT/zstd.h" examples/zstd.h
cp "$ZSTD_SRC_ROOT/zstd_errors.h" examples/zstd_errors.h
# Build the Rust implementation referenced by the amalgamated C shims.
${CARGO:-cargo} build --manifest-path ../../rust/Cargo.toml --release \
--target-dir "$RUST_TARGET_DIR"
if [ $? -ne 0 ]; then
echo "Building Rust library archive: FAILED"
exit 1
fi
# Compile the generated output
cc -Wall -Wextra -Werror -Wshadow -pthread -I. -Os -g0 -o $OUT_FILE zstd.c examples/roundtrip.c
cc -Wall -Wextra -Werror -Wshadow \
-Wno-unused-variable -Wno-unused-parameter -Wno-unused-but-set-variable \
-pthread -I. -Os -g0 -o $OUT_FILE zstd.c examples/roundtrip.c "$RUST_ARCHIVE"
# Did compilation work?
if [ $? -ne 0 ]; then
echo "Compiling roundtrip.c: FAILED"
@@ -92,7 +52,7 @@ if [ $retVal -ne 0 ]; then
fi
echo "Running roundtrip.c: PASSED"
# Try Emscripten build if emcc or docker command is available.
try_emscripten_build
# The generated C now calls Rust, so it is no longer a standalone Emscripten input.
echo "(Skipping Emscripten test for Rust-backed amalgamation)"
exit 0