ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

build(rust): add dict-builder cargo feature dimension

Browse Source 
The dictionary-builder sources (lib/dictBuilder) are about to start moving
to Rust, beginning with divsufsort. The Rust crate previously only modeled
the compression/decompression module split plus the forced-HUF decoder
modes, so no build could express "this C configuration includes (or
excludes) dictBuilder" to Cargo. Without that, a Rust archive could carry
dictBuilder modules into a build whose C side disabled them, or worse,
omit a migrated implementation from a build whose C shims require it.

Add a `dict-builder` cargo feature and thread it through every build that
consumes the Rust static archive, mirroring exactly how each build system
already gates the dictBuilder C sources:

- rust/Cargo.toml: new `dict-builder` feature, included in the default
  set because the C library builds dictBuilder by default
  (ZSTD_LIB_DICTBUILDER ?= 1). The feature is empty until the first
  dictBuilder module lands.
- lib/Makefile: RUST_CARGO_FEATURES gains dict-builder when
  ZSTD_LIB_DICTBUILDER is enabled, following the existing
  ZSTD_LIB_COMPRESSION/ZSTD_LIB_DECOMPRESSION pattern. The archive
  directory naming grows a matching `b<0|1>` dimension
  (c1-d1-b1-default etc.) so differently configured archives never
  collide; the repeated config prefix is factored into
  RUST_MODULE_CONFIG.
- programs/Makefile: the full-featured archives now request
  compression,decompression,dict-builder (equal to the default set, so
  the target directory stays shared with tests). The partial-library
  variants gain the `b0` name dimension, and zstd-dictBuilder gets its
  own lib-c1-d0-b1 archive because it compiles the dictBuilder C sources
  without decompression; it previously shared the compression-only
  archive, which will lack the migrated dictBuilder symbols.
- tests/Makefile: no flag change needed since tests use the crate default
  feature set; a comment now records that dict-builder arrives that way.
- build/cmake/lib/CMakeLists.txt: ZSTD_BUILD_DICTBUILDER now adds the
  dict-builder feature and a `b<0|1>` component in the Rust build-config
  directory name, in lockstep with the DictBuilderSources gating.
- build/meson/lib/meson.build: meson compiles the dictBuilder sources
  unconditionally, so the feature list and config name gain dict-builder
  unconditionally (c1-d1-b1-<huf-mode>).

The `dict-builder` feature deliberately does not imply `compression`.
lib/Makefile forces ZSTD_LIB_DICTBUILDER=0 when compression is disabled,
but CMake does not couple the two options, so encoding the C-side
constraint in Cargo would make the Rust archive diverge from the C source
list in that (already unsupported) CMake configuration.

Test plan:
- cd rust && cargo build --release
- cargo build --release --no-default-features \
    --features compression,decompression
- cargo build --release --no-default-features \
    --features compression,dict-builder
- Full validation (fuzzer, smoke tests, dictionary byte-identity) runs
  with the follow-up commit that ports divsufsort onto this scaffolding.
This commit is contained in:
ddidderr
2026-07-11 09:38:04 +02:00
parent 959e485201
commit eecd2e168a
7 changed files with 39 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files
build/cmake/lib/CMakeLists.txt
+6 -1
View File
@@ -142,6 +142,7 @@ endif()
set(_zstd_rust_features)
set(_zstd_rust_compression 0)
set(_zstd_rust_decompression 0)
set(_zstd_rust_dictbuilder 0)
if(ZSTD_BUILD_COMPRESSION)
list(APPEND _zstd_rust_features compression)
set(_zstd_rust_compression 1)
@@ -150,6 +151,10 @@ if(ZSTD_BUILD_DECOMPRESSION)
list(APPEND _zstd_rust_features decompression)
set(_zstd_rust_decompression 1)
endif()
if(ZSTD_BUILD_DICTBUILDER)
list(APPEND _zstd_rust_features dict-builder)
set(_zstd_rust_dictbuilder 1)
endif()
set(_zstd_rust_huf_mode default)
if(_zstd_huf_force_x1)
@@ -190,7 +195,7 @@ if(_zstd_rust_features)
endif()
set(_zstd_rust_build_config
"c${_zstd_rust_compression}-d${_zstd_rust_decompression}-${_zstd_rust_huf_mode}")
"c${_zstd_rust_compression}-d${_zstd_rust_decompression}-b${_zstd_rust_dictbuilder}-${_zstd_rust_huf_mode}")
set(ZSTD_RUST_MANIFEST "${ZSTD_SOURCE_DIR}/rust/Cargo.toml")
set(ZSTD_RUST_TARGET_DIR
"${CMAKE_CURRENT_BINARY_DIR}/rust-target/${_zstd_rust_build_config}")
build/meson/lib/meson.build
+4 -2
View File
@@ -73,7 +73,9 @@ if rust_huf_force_x1 and rust_huf_force_x2
error('HUF_FORCE_DECOMPRESS_X1 and HUF_FORCE_DECOMPRESS_X2 are mutually exclusive')
endif
rust_features = ['compression', 'decompression']
# Meson always compiles the dictBuilder sources above, so the Rust archive
# must always carry the matching dict-builder module set.
rust_features = ['compression', 'decompression', 'dict-builder']
rust_huf_mode = 'default'
rust_huf_c_args = []
if rust_huf_force_x1
@@ -96,7 +98,7 @@ if rust_target == ''
endif
endif
rust_build_config = 'c1-d1-' + rust_huf_mode
rust_build_config = 'c1-d1-b1-' + rust_huf_mode
rust_target_dir = join_paths(meson.current_build_dir(), 'rust-target', rust_build_config)
is_msvc = cc_id == compiler_msvc or cc_id == 'clang-cl'
rust_staticlib_name = is_msvc ? 'zstd_rs.lib' : 'libzstd_rs.a'
lib/Makefile
+7 -3
View File
@@ -84,16 +84,20 @@ endif
ifneq ($(ZSTD_LIB_DECOMPRESSION),0)
RUST_CARGO_FEATURES += decompression
endif
ifneq ($(ZSTD_LIB_DICTBUILDER),0)
RUST_CARGO_FEATURES += dict-builder
endif
RUST_MODULE_CONFIG := c$(ZSTD_LIB_COMPRESSION)-d$(ZSTD_LIB_DECOMPRESSION)-b$(ZSTD_LIB_DICTBUILDER)
RUST_HUF_FEATURE :=
RUST_BUILD_CONFIG := c$(ZSTD_LIB_COMPRESSION)-d$(ZSTD_LIB_DECOMPRESSION)-default
RUST_BUILD_CONFIG := $(RUST_MODULE_CONFIG)-default
ifneq ($(RUST_HUF_FORCE_X1),0)
RUST_HUF_FEATURE := huf-force-decompress-x1
RUST_BUILD_CONFIG := c$(ZSTD_LIB_COMPRESSION)-d$(ZSTD_LIB_DECOMPRESSION)-huf-force-decompress-x1
RUST_BUILD_CONFIG := $(RUST_MODULE_CONFIG)-huf-force-decompress-x1
endif
ifneq ($(RUST_HUF_FORCE_X2),0)
RUST_HUF_FEATURE := huf-force-decompress-x2
RUST_BUILD_CONFIG := c$(ZSTD_LIB_COMPRESSION)-d$(ZSTD_LIB_DECOMPRESSION)-huf-force-decompress-x2
RUST_BUILD_CONFIG := $(RUST_MODULE_CONFIG)-huf-force-decompress-x2
endif
ifneq ($(RUST_HUF_FEATURE),)
ifneq ($(ZSTD_LIB_DECOMPRESSION),0)
programs/Makefile
+15 -4
View File
@@ -71,7 +71,7 @@ RUST_TARGET_32 ?= i686-unknown-linux-gnu
RUST_STATICLIB_32 := $(RUST_TARGET_DIR)/$(RUST_TARGET_32)/release/libzstd_rs.a
RUST_CARGO_FLAGS := --manifest-path $(RUST_MANIFEST) --release \
--target-dir $(RUST_TARGET_DIR) --no-default-features
RUST_CARGO_FLAGS += --features compression,decompression
RUST_CARGO_FLAGS += --features compression,decompression,dict-builder
ifneq ($(RUST_HUF_FEATURE),)
RUST_CARGO_FLAGS += --features $(RUST_HUF_FEATURE)
endif
@@ -96,7 +96,7 @@ $(RUST_CLI_STATICLIB): $(RUST_CLI_SOURCES)
$(RUST_CLI_STATICLIB_32): $(RUST_CLI_SOURCES)
$(CARGO) build $(RUST_CLI_CARGO_FLAGS) --target $(RUST_TARGET_32)
RUST_DECOMPRESS_BUILD_CONFIG := lib-c0-d1-$(RUST_BUILD_CONFIG)
RUST_DECOMPRESS_BUILD_CONFIG := lib-c0-d1-b0-$(RUST_BUILD_CONFIG)
RUST_DECOMPRESS_TARGET_DIR := $(RUST_DIR)/target/$(RUST_DECOMPRESS_BUILD_CONFIG)
RUST_DECOMPRESS_STATICLIB := $(RUST_DECOMPRESS_TARGET_DIR)/release/libzstd_rs.a
RUST_DECOMPRESS_CARGO_FLAGS := --manifest-path $(RUST_MANIFEST) --release \
@@ -119,7 +119,7 @@ RUST_DECOMPRESS_CLI_CARGO_FLAGS := --manifest-path $(RUST_CLI_MANIFEST) --releas
$(RUST_DECOMPRESS_CLI_STATICLIB): $(RUST_CLI_SOURCES)
$(CARGO) build $(RUST_DECOMPRESS_CLI_CARGO_FLAGS)
RUST_COMPRESS_BUILD_CONFIG := lib-c1-d0-$(RUST_BUILD_CONFIG)
RUST_COMPRESS_BUILD_CONFIG := lib-c1-d0-b0-$(RUST_BUILD_CONFIG)
RUST_COMPRESS_TARGET_DIR := $(RUST_DIR)/target/$(RUST_COMPRESS_BUILD_CONFIG)
RUST_COMPRESS_STATICLIB := $(RUST_COMPRESS_TARGET_DIR)/release/libzstd_rs.a
RUST_COMPRESS_CARGO_FLAGS := --manifest-path $(RUST_MANIFEST) --release \
@@ -129,6 +129,17 @@ RUST_COMPRESS_CARGO_FLAGS := --manifest-path $(RUST_MANIFEST) --release \
$(RUST_COMPRESS_STATICLIB): $(RUST_SOURCES)
$(CARGO) build $(RUST_COMPRESS_CARGO_FLAGS)
RUST_DICTBUILDER_BUILD_CONFIG := lib-c1-d0-b1-$(RUST_BUILD_CONFIG)
RUST_DICTBUILDER_TARGET_DIR := $(RUST_DIR)/target/$(RUST_DICTBUILDER_BUILD_CONFIG)
RUST_DICTBUILDER_STATICLIB := $(RUST_DICTBUILDER_TARGET_DIR)/release/libzstd_rs.a
RUST_DICTBUILDER_CARGO_FLAGS := --manifest-path $(RUST_MANIFEST) --release \
--target-dir $(RUST_DICTBUILDER_TARGET_DIR) \
--no-default-features \
--features compression,dict-builder
$(RUST_DICTBUILDER_STATICLIB): $(RUST_SOURCES)
$(CARGO) build $(RUST_DICTBUILDER_CARGO_FLAGS)
RUST_COMPRESS_CLI_BUILD_CONFIG := cli-c1-d0-$(RUST_BUILD_CONFIG)
RUST_COMPRESS_CLI_TARGET_DIR := $(RUST_DIR)/target/$(RUST_COMPRESS_CLI_BUILD_CONFIG)
RUST_COMPRESS_CLI_STATICLIB := $(RUST_COMPRESS_CLI_TARGET_DIR)/release/libzstd_cli_rs.a
@@ -412,7 +423,7 @@ zstd-compress: $(ZSTDLIB_COMMON_SRC) $(ZSTDLIB_COMPRESS_SRC) zstdcli.c util.c ti
## zstd-dictBuilder: executable supporting dictionary creation and compression (only)
CLEAN += zstd-dictBuilder
zstd-dictBuilder: $(ZSTDLIB_COMMON_SRC) $(ZSTDLIB_COMPRESS_SRC) $(ZDICT_SRC) zstdcli.c util.c timefn.c fileio.c fileio_asyncio.c dibio.c $(RUST_COMPRESS_STATICLIB) $(RUST_COMPRESS_CLI_STATICLIB)
zstd-dictBuilder: $(ZSTDLIB_COMMON_SRC) $(ZSTDLIB_COMPRESS_SRC) $(ZDICT_SRC) zstdcli.c util.c timefn.c fileio.c fileio_asyncio.c dibio.c $(RUST_DICTBUILDER_STATICLIB) $(RUST_COMPRESS_CLI_STATICLIB)
$(CC) $(FLAGS) -DZSTD_NOBENCH -DZSTD_NODECOMPRESS -DZSTD_NOTRACE $^ -o $@$(EXT)
RUST_DIRECT_LINK_TARGETS := zstd32 zstd-nolegacy zstd-small zstd-frugal \
rust/Cargo.toml
+2 -1
View File
@@ -7,9 +7,10 @@ edition = "2021"
crate-type = ["staticlib"]
[features]
default = ["compression", "decompression"]
default = ["compression", "decompression", "dict-builder"]
compression = []
decompression = []
dict-builder = []
huf-force-decompress-x1 = []
huf-force-decompress-x2 = []
rust/README.md
+3 -2
View File
@@ -74,8 +74,9 @@ makefile source list as a small shim so header configuration and platform
preprocessor behavior stay available during the transition.
The library, test, and program makefiles select an archive directory for the
active C configuration: enabled compression/decompression modules, default or
forced HUF X1/X2, and the matching Rust target for 32-bit C binaries. The
active C configuration: enabled compression/decompression/dictionary-builder
modules, default or forced HUF X1/X2, and the matching Rust target for 32-bit
C binaries. The
native static archive flattens Rust object members rather than nesting a Rust
archive, while the native shared library retains all migrated Rust exports.
When the HUF mode changes, the test and program paths also rebuild cached C
tests/Makefile
+2
View File
@@ -79,6 +79,8 @@ RUST_TARGET_DIR := $(RUST_DIR)/target/$(RUST_BUILD_CONFIG)
RUST_STATICLIB := $(RUST_TARGET_DIR)/release/libzstd_rs.a
RUST_TARGET_32 ?= i686-unknown-linux-gnu
RUST_STATICLIB_32 := $(RUST_TARGET_DIR)/$(RUST_TARGET_32)/release/libzstd_rs.a
# Tests build every library module, so they use the crate's default feature
# set (compression, decompression, and dict-builder) plus any forced HUF mode.
RUST_CARGO_FLAGS := --manifest-path $(RUST_MANIFEST) --release \
--target-dir $(RUST_TARGET_DIR)
ifneq ($(RUST_HUF_FEATURE),)