Commit Graph
4882 Commits
Author SHA1 Message Date
ddidderr 33c937bc87 feat(compress): move deprecated block-size formula to Rust
Keep validation of the applied compression parameters in C and delegate the context-free minimum calculation through a narrow ABI leaf. Preserve the public getter and deprecated block-compression paths.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --all-targets

- cargo +nightly fmt --manifest-path rust/Cargo.toml

- make -B -C lib -j2 lib

- make -C tests -j2 fuzzer

- tests/fuzzer -s4560 -t47 -i48 -v
2026-07-18 08:33:07 +02:00
ddidderr 7f22db844d feat(compress): move sequence-store validation to Rust
Keep the C DEBUGLEVEL gate and delegate the read-only match-length validation through the existing SeqStore ABI. Preserve long-match side-band decoding and the original 3-or-4 byte lower-bound assertion.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --all-targets

- cargo +nightly fmt --manifest-path rust/Cargo.toml

- make -B -C lib -j2 lib

- make -C tests -j2 fuzzer

- tests/fuzzer -s4560 -t47 -i48 -v
2026-07-18 08:28:49 +02:00
ddidderr 56ae331f09 feat(compress): move block-state confirmation to Rust
Keep C ownership of the block-state structure while delegating the confirmed-block pointer swap through a narrow two-pointer ABI leaf. Preserve every existing confirmation call site and add direct pointer-swap coverage.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --all-targets

- cargo +nightly fmt --manifest-path rust/Cargo.toml

- make -B -C lib -j2 lib

- tests/fuzzer -s4560 -t47 -i48 -v

- make -C tests -j2 test-zstream
2026-07-18 08:23:28 +02:00
ddidderr b81109095f feat(compress): move block split search to Rust
Port the recursive post-block-split partition search behind a narrow Rust ABI leaf while keeping CCtx ownership, entropy-state projection, and block emission in C. Preserve left-first midpoint insertion, minimum-sequence gating, the split cap, and error-as-no-split behavior.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --all-targets

- cargo +nightly fmt --manifest-path rust/Cargo.toml

- make -B -C lib -j2 lib

- tests/fuzzer -s4560 -t47 -i48 -v
2026-07-18 08:08:40 +02:00
ddidderr 7ee445c431 feat(compress): move no-delimiter sequence transfer to Rust
Port the no-delimiter external-sequence block scanner, match splitting, repcode updates, validation, and sequence/literal storage behind a narrow C shim. Preserve trailing bytes as literals when the sequence buffer ends before the requested block, and remove the obsolete C validation wrapper.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (242 passed)

- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression [--benches|--tests] (clean)

- cargo +nightly fmt --manifest-path rust/Cargo.toml (clean)

- make -B -C lib -j2 lib (passed)

- make -C tests -j2 test-zstream (84 deterministic; 5960 and 8307 randomized passed)

- make -C tests -j2 test-fuzzer (existing test 56 CCtx-reuse mismatch; also fails at 37eb75758)

- make -C tests -j2 test-zstd (existing Rust CLI --patch-from gap)
2026-07-18 07:49:11 +02:00
ddidderr 37eb757585 feat(compress): move dictionary entropy loading to Rust
Port ZSTD_dictNCountRepeat and ZSTD_loadCEntropy to a dedicated Rust compression module. Preserve the existing C-facing ZSTD_loadCEntropy symbol through a narrow shim while keeping dictionary content and compression-context ownership in C.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --benches --tests

- make -B -C lib -j2 lib

- make -C tests -j2 test-invalidDictionaries

- make -C tests -j2 test-zstream
2026-07-18 07:33:06 +02:00
ddidderr 77841fcfd2 feat(compress): move explicit sequence transfer to Rust
Move ZSTD_transferSequences_wBlockDelim's delimiter scan, literal copying, SeqDef storage, validation, and repcode bookkeeping into the Rust compression statistics module. Keep ZSTD_CCtx and block-state ownership in C through a narrow scalar/pointer shim, with ABI layout assertions for ZSTD_SequencePosition.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression --benches --tests

- make -B -C lib -j2 lib

- make -C tests -j2 test-zstream
2026-07-18 07:24:18 +02:00
ddidderr 511882e213 feat(compress): move block size estimation to Rust
Block splitting still estimated literal and sequence section sizes through
three C helpers. Move the pure estimator into zstd_compress_stats.rs so its
histogram, HUF, and FSE cost calculations use the Rust implementations already
behind the compression ABI. Preserve the original literal headers, multi-stream
Huffman surcharge, sequence additional-bit costs, entropy-header accounting,
and fallback estimates; leave C responsible for split recursion and table
ownership.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression` -- 229 passed
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression` plus benches/tests -- passed before and after formatting
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `make -B -C lib -j2 lib` -- passed
- `make -C tests -j2 test-zstream` -- passed, including 84 deterministic and 15,375 randomized cases
2026-07-18 07:08:27 +02:00
ddidderr d334aaafa3 feat(compress): move external block summary to Rust
The explicit-delimiter sequence path still calculated each block summary in C,
with a separate AVX2 implementation and scalar fallback. Move that pure scan
to zstd_compress_stats.rs and keep a C wrapper for the existing orchestration.
The Rust leaf preserves delimiter inclusion, literal and match totals, and the
external-sequences error when no delimiter is present, while the C ABI is pinned
with a repr(C) layout check.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression` -- 227 passed
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression` plus benches/tests -- passed before and after formatting
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `make -B -C lib -j2 lib` -- passed
- `make -C tests -j2 test-zstream` -- passed, including 84 deterministic and 15,464 randomized cases
2026-07-18 06:56:38 +02:00
ddidderr 18f5419cc6 feat(compress): move external sequence conversion to Rust
The external-sequence block path still converted public ZSTD_Sequence values
in C, with separate scalar and AVX2 implementations. Move that pure format
conversion into zstd_compress_stats.rs so both native configurations share one
implementation. Preserve offBase encoding, u16 truncation, the rep-ignored
contract, and the long-length side-band marker; leave C responsible for block
state, repcode resolution, and long-length bookkeeping.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression` -- 225 passed
- `cargo clippy --manifest-path rust/Cargo.toml --no-default-features --features compression` plus benches/tests -- passed before and after formatting
- `cargo +nightly fmt --manifest-path rust/Cargo.toml --all` -- passed
- `make -B -C lib -j2 lib` -- passed
- `make -C tests -j2 test-zstream` -- passed, including 84 deterministic and 14,768 randomized cases
2026-07-18 06:50:48 +02:00
ddidderr 21f8e8958c feat(compress): move target block policy to Rust
Port the target-sized block action policy behind a scalar Rust ABI while keeping sequence inspection, superblock compression, block-state confirmation, diagnostics, and raw/RLE emission in C. Preserve the strict three-byte header boundary, destination-too-small fallback, and first-block RLE guard with focused tests.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_compress::tests::target_block

- cargo clippy --manifest-path rust/Cargo.toml

- cargo clippy --manifest-path rust/Cargo.toml --benches

- cargo clippy --manifest-path rust/Cargo.toml --tests

- cargo +nightly fmt --manifest-path rust/Cargo.toml --all

- make -B -C lib -j2 lib
2026-07-18 06:37:07 +02:00
ddidderr b19ca569f7 feat(compress): move MT synchronization policy to Rust
Port the rsyncable synchronization-point scan and rolling hash policy behind a scalar ABI. Keep the MT scheduler, input buffers, and context ownership in C while adding boundary-focused Rust coverage for disabled, short, buffered, spanning, and hit/no-hit paths.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml

- cargo clippy --manifest-path rust/Cargo.toml --benches

- cargo clippy --manifest-path rust/Cargo.toml --tests

- cargo +nightly fmt --manifest-path rust/Cargo.toml --all

- make -B -C lib -j2 lib

- make -B -C tests -j2 test-zstream (pending post-commit native gate)
2026-07-18 06:18:30 +02:00
ddidderr 0272ae867b feat(compress): move frame progression accounting to Rust
Port the scalar consumed/produced counter updates and pledged-size overrun predicate behind a narrow Rust ABI while leaving C diagnostics and context ownership intact. Add focused coverage for unknown, exact, and overrun pledges.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml

- cargo clippy --manifest-path rust/Cargo.toml --benches

- cargo clippy --manifest-path rust/Cargo.toml --tests

- cargo +nightly fmt --manifest-path rust/Cargo.toml --all

- make -B -C lib -j2 lib (pending post-commit native gate)
2026-07-18 06:18:02 +02:00
ddidderr 3bbfacdfd0 feat(compress): move repcode reconciliation to Rust
Port ZSTD_seqStore_resolveOffCodes and its repcode-resolution helper to Rust. The C shim keeps Repcodes_t and SeqStore_t ownership in the compressor context while Rust preserves long-literal handling, mismatch materialization, and independent decoder/compressor history updates.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (208 tests)

- cargo clippy --manifest-path rust/Cargo.toml

- cargo clippy --manifest-path rust/Cargo.toml --benches

- cargo clippy --manifest-path rust/Cargo.toml --tests

- make -B -C lib -j2 lib
2026-07-18 06:03:55 +02:00
ddidderr 48eb895af3 feat(compress): move raw and RLE block emitters to Rust
Replace the shared C raw and one-byte RLE fallback block serializers with Rust ABI leaves. Preserve the zstd block headers, payload copies, capacity errors, and all existing C compressor dispatch and context ownership; share the raw serializer with the Rust one-shot path.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (208 tests)

- cargo clippy --manifest-path rust/Cargo.toml

- cargo clippy --manifest-path rust/Cargo.toml --benches

- cargo clippy --manifest-path rust/Cargo.toml --tests

- make -B -C lib -j2 lib
2026-07-18 06:03:39 +02:00
ddidderr 459782ad55 feat(compress): move frame epilogue serialization to Rust
Port the scalar frame-finalization policy and byte serialization into the Rust compression leaf while keeping checksum finalization, tracing, and the compression context in C. Preserve the init-to-ongoing transition on partial failure and reset the stage only after a successful epilogue.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml

- cargo clippy --manifest-path rust/Cargo.toml --benches

- cargo clippy --manifest-path rust/Cargo.toml --tests

- make -B -C lib -j2 lib

- make -B -C tests -j2 test-zstream (84 deterministic, 6737 and 9012 fuzzer cases)
2026-07-18 05:49:37 +02:00
ddidderr 327272f8b3 feat(compress): move optimal block policy to Rust
Move the frame chunk pre-split policy behind a scalar Rust ABI while retaining C ownership of ZSTD_CCtx and workspace extraction.

Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 deterministic, 6697 and 9217 fuzzer cases).
2026-07-18 05:38:54 +02:00
ddidderr 1a05952013 feat(compress): move external sequence validation to Rust
Port the external-sequence window, offset, and minimum-match checks through a scalar C ABI leaf while retaining the C caller's control flow and diagnostics.

Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 deterministic, 7063 and 8665 fuzzer cases).
2026-07-18 05:19:19 +02:00
ddidderr 4c6a00722f feat(compress): move parameter update policy to Rust
Move ZSTD_isUpdateAuthorized behind the existing Rust parameter API while retaining the C context-stage checks, diagnostics, and mutation flow.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (187 tests)

- cargo clippy for library, benches, and tests

- make -B -C lib -j2 lib
2026-07-18 05:11:26 +02:00
ddidderr cba0373a2b feat(compress): move hash and window reset policies to Rust
Move hash-salt mixing and context-reset overflow predicates behind scalar Rust ABI shims. Keep the C-owned match state and window arithmetic in the C wrapper while preserving exact architecture-dependent thresholds.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (186 tests)

- cargo clippy for library, benches, tests, and CLI all-targets

- make -B -C lib -j2 lib

- make -B -C tests -j2 test-zstream (84 deterministic tests; fuzzers 6451 and 9879 cases)
2026-07-18 05:06:32 +02:00
ddidderr 59eda8deac feat(compress): move block policy predicates to Rust
Move target block-size and block-splitter policy predicates behind narrow Rust ABI shims while retaining the C debug logging, finalized-mode assertion, and surrounding control flow.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression

- cargo clippy --manifest-path rust/Cargo.toml and native library build

- make -B -C tests -j2 test-zstream (84 deterministic tests; fuzzers 6778 and 8299 cases)
2026-07-18 04:54:15 +02:00
ddidderr d9fe5d25ab feat(compress): move MT overlap window policy to Rust
Move construction of the external-dictionary and active-prefix ranges into a narrow Rust ABI while retaining the C window and logging surface. Preserve byte-range half-open overlap semantics and leave the input-range overlap wrapper available to its other C caller.

Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (180 passed); root and CLI clippy; make -B -C lib -j2 lib; make -C tests -j2 test-cli-tests (41 passed); make -B -C tests -j2 test-zstream (84 named tests plus 6,845 and 9,628 fuzz cases passed).
2026-07-18 04:43:41 +02:00
ddidderr 2cd0b29790 feat(compress): move match table reduction to Rust
Port the 16-cell match-table reduction leaf while preserving the window-index threshold, wrapping subtraction, and btlazy2 unsorted marker. Keep ZSTD_reduceIndex and its strategy-specific table selection in C behind the narrow Rust ABI.

Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (180 passed); root clippy, bench clippy, and test clippy; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 named tests plus 6,845 and 9,628 fuzz cases passed).
2026-07-18 04:43:26 +02:00
ddidderr d5ae2a450b feat(compress): move external sequence sizing leaves to Rust
Port repcode offBase finalization and explicit-delimiter block sizing through the Rust compression ABI. Preserve wrapped sequence arithmetic, external-sequence error codes, and the no-delimiter target-size policy while keeping the C control flow intact.

Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (176 passed); cargo clippy root, benches, and tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-zstream (84 named tests plus 6,230 and 8,598 fuzz cases passed).
2026-07-18 04:31:22 +02:00
ddidderr e80fbe4b06 feat(compress): move sequence policy leaves to Rust
Move external-sequence postprocessing and MT overlap detection into the Rust compression modules. Keep C stateful dispatch and locking code intact while preserving error encoding, delimiter handling, and half-open range semantics.

Test Plan:\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- make -B -C lib -j2 lib\n- make -C tests -j2 test-cli-tests
2026-07-18 04:14:27 +02:00
ddidderr bdb14b4fbf feat(compress): move sequence leaves to Rust
Move sequence-store byte accounting, chunk derivation, and repcode resolution into the Rust compression module. Move MT raw-sequence buffer conversions into Rust while preserving the existing C adapters and ABI layouts.

Test Plan:\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- make -B -C lib -j2 lib\n- make -B -C tests -j2 test-cli-tests
2026-07-18 04:05:48 +02:00
ddidderr 1edec2cc0e feat(compress): move seq-store leaves to Rust
Move compressed-block state reset, final-literal storage, sequence-store reset, sequence-length summation, RLE detection, and RLE heuristics into the Rust compression statistics module. Preserve the C helper surface and add focused pointer/layout and boundary tests.

Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C programs -j2 zstd zstd-small zstd-frugal; make -B -C tests -j2 test-cli-tests; make -B -C tests -j2 test-zstream
2026-07-18 03:55:57 +02:00
ddidderr 8ac9855309 feat(compress): move matchfinder policy to Rust
Move row matchfinder, block-splitter, LDM, chain-table, external-repcode, and tagged-dictionary policy leaves into Rust while preserving the existing C helper names and private parameter flow. Add boundary coverage for each policy decision.

Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-cli-tests; make -B -C tests -j2 test-zstream
2026-07-18 03:55:30 +02:00
ddidderr 429d5176e9 feat(compress): move MT sizing policy to Rust
Move target job-log, overlap-log, and overlap-size policy into the Rust multithreaded compression module, retaining the C context projection and logging wrapper. Add policy tests for LDM, strategy defaults, bounds, and overlap sizing.

Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C programs -j2 zstd zstd-small zstd-frugal; make -B -C tests -j2 test-cli-tests; make -B -C tests -j2 test-zstream
2026-07-18 03:40:12 +02:00
ddidderr 6aaabce208 feat(compress): move CCtx parameter bounds to Rust
Route public CCtx parameter bounds through the Rust policy implementation while keeping build-specific MT limits supplied by the C configuration. Preserve clamping and validation behavior for migrated parameters and add focused Rust coverage.

Test Plan: cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression; cargo clippy --manifest-path rust/Cargo.toml; cargo clippy --manifest-path rust/Cargo.toml --benches; cargo clippy --manifest-path rust/Cargo.toml --tests; make -B -C lib -j2 lib; make -B -C tests -j2 test-cli-tests; make -B -C tests -j2 test-zstream
2026-07-18 03:38:56 +02:00
ddidderr 3a10d86c76 feat(compress): move advanced CCtx parameter init to Rust
Move ZSTD_CCtxParams_init_advanced and its default-resolution policy into the Rust parameter API while retaining the C public ABI wrapper. Preserve validation, zeroing, and the existing row-match, block-splitter, LDM, and repcode defaults.

Test Plan:\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_compress_params_api\n- make -B -C lib -j2 lib
2026-07-18 03:14:33 +02:00
ddidderr 3feb9370c8 feat(compress): move CCtx parameter allocation to Rust
Move CCtx_params allocation, initialization, and release behind Rust ABI helpers while preserving the public C wrappers and custom allocator contract. Keep parameter bounds and policy in the existing C implementation.

Test Plan:\n- cargo clippy --manifest-path rust/Cargo.toml\n- cargo clippy --manifest-path rust/Cargo.toml --benches\n- cargo clippy --manifest-path rust/Cargo.toml --tests\n- cargo +nightly fmt --manifest-path rust/Cargo.toml --all\n- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression\n- make -B -C lib -j2 lib\n- make -B -C tests -j2 test-rust-lib-smoke\n- make -B -C tests -j2 test-zstream\n- make -B -C tests -j2 test-cli-tests
2026-07-18 03:01:08 +02:00
ddidderr 49b3405d63 fix(compress): restore optimal parser strategy compatibility
Match the original optimal-parser statistics for zero-possible symbols, complete the shortest-path fallback when the forward pass runs past its final position, and refresh the projected window state for the ultra2 second pass. Keep the simple Rust frame leaf restricted to the match finders it implements so lazy and optimal strategies retain their stateful lifecycle.

Test Plan:

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (135 passed)

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_opt (3 passed)

- make -B -C tests -j2 test-cli-tests (41 passed)

- make -B -C tests -j2 test-zstream (84 named tests and fuzzer suites passed)

- cargo clippy --manifest-path rust/Cargo.toml

- cargo clippy --manifest-path rust/Cargo.toml --benches

- cargo clippy --manifest-path rust/Cargo.toml --tests
2026-07-18 02:39:58 +02:00
ddidderr 23d45378bd feat(compress): route complete stream frames through Rust
Route ordinary complete-input ZSTD_compressStream2 calls through the Rust frame compressor while retaining the C state machine for partial, advanced, and dictionary-backed streams. Track explicit maxBlockSize requests so the migrated path preserves the original byte-identical default behavior, and extend the C archive smoke test across context reuse and fallback cases.

Test Plan:

- cargo clippy --manifest-path rust/Cargo.toml

- cargo clippy --manifest-path rust/Cargo.toml --benches

- cargo clippy --manifest-path rust/Cargo.toml --tests

- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression (133 passed)

- make -B -C tests -j2 test-rust-lib-smoke and ./tests/rustLibSmoke

- make -B -C tests -j2 test-zstream (passed, including both APIs and maxBlockSize case 78)
2026-07-18 01:52:28 +02:00
ddidderr a15bf1d375 feat(compress): move MT job-table storage lifecycle to Rust
The multithreaded compressor already uses Rust-owned buffer and CCtx pools,
but zstdmt_compress.c still allocated and freed its job table directly. Move
the raw table storage and power-of-two sizing behind Rust's custom allocator
ABI. Keep descriptor field access and platform mutex/condition initialization
in C because those layouts remain private and platform-specific. The C
wrapper initializes and destroys synchronization primitives around the Rust
storage calls, preserving failure cleanup while leaving worker job setup,
scheduling, and stream entry points in C as the fallback implementation.

Keep the Rust storage-only ABI free of MT-only C references so single-threaded
archives can omit zstdmt_compress.c without acquiring new unresolved symbols.

Test Plan:
- `cargo test --manifest-path rust/Cargo.toml --no-default-features
  --features compression zstdmt_compress --lib` -- passed (5 tests).
- `make -B -C lib lib-mt` -- passed.
- `make -B -C tests -j2 fullbench zstreamtest poolTests` -- passed.
- `./poolTests` -- passed.
- `./fullbench -i1 -B1000 README.md` -- passed, including -T2 scenarios.
- Rebuilt `programs/zstd` and ran a `-T2` compress/decompress `cmp` round-trip
  -- passed.
- `rustfmt` and `cargo +nightly fmt --manifest-path rust/Cargo.toml --all
  -- --check` -- passed.
- Full `cargo clippy -D warnings` remains blocked by unrelated warnings in
  the concurrent `rust/src/fileio_asyncio.rs` worktree changes.
- `./zstreamtest -T5s` reaches an unrelated single-thread maxBlockSize
  assertion at `tests/zstreamtest.c:2157`; MT fullbench and CLI smoke pass.
2026-07-18 01:40:18 +02:00
ddidderr 5c448120cf feat(compress): route simple context frames through Rust
Dispatch ordinary dictionary-free ZSTD_compress2 calls through the Rust frame
compressor while retaining the original C implementation for advanced
contexts. Prepare the C workspace first so allocation downsizing and
static-context limits remain observable.

Mark successful Rust frames so a following one-shot
ZSTD_compressStream2(..., ZSTD_e_end) call uses the same output path. Other
streaming directives and advanced parameters remain on the C implementation.

Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
  rust/Cargo.toml --no-default-features --features compression
  zstd_compress::tests
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
  rust/Cargo.toml --no-default-features --features compression -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C tests fuzzer
- ./tests/fuzzer -t114 -i1000 --no-big-tests -s1

The basic fuzzer run now reaches the existing Rust one-shot compression-ratio
limit at test 113; the new stateful reuse tests pass before that point.
2026-07-12 19:16:14 +02:00
ddidderr e66ad5f563 feat(compress): port simple context compression to Rust
Move ZSTD_compressCCtx() onto the Rust frame-compression path while
retaining only the C-owned reset needed for its private context layout.
The simple API ignores advanced context parameters by contract, so the
Rust entry point can share the one-shot implementation after reset.

Preserve valid frames when the Rust superblock leaf declines a compressed
block by emitting a raw block instead. Match the fast-strategy literal
policy for negative levels so compatibility behavior remains intact.

Streaming, stateful ZSTD_compress2(), and multithreaded context
orchestration remain C-owned for a later slice. The full fuzzer currently
reaches an existing compress2() superblock-expansion failure after these
simple-API checks; that stateful path is intentionally out of scope here.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features
  --features compression zstd_compress::tests
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
  (library, benches, and tests)
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C programs zstd V=1
- make -B -C examples multiple_simple_compression
- ./multiple_simple_compression README.md ../rust/README.md
2026-07-12 18:48:44 +02:00
ddidderr 6711aef1d4 feat(legacy): port the v0.7 decoder to Rust
Move the frozen v0.7 decoder, dictionary handling, bufferless streaming,
and buffered streaming implementation into the Rust legacy module.  Keep
the historical C translation unit as a declaration-only ABI shim so the
existing C callers and build selection remain unchanged.

The port preserves the v0.7 entropy and frame boundaries, while fixing the
Rust-side literal-tail bookkeeping and making buffered header loading revisit
the complete frame header before sizing its rolling buffers.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features decompression,legacy-v07 legacy::zstd_v07::tests
- cargo clippy --manifest-path rust/Cargo.toml -- -D warnings
- make -B -C programs zstd V=1
- decode a v0.7.5 fixture and compare its 200000-byte output
- make -C tests check V=1
2026-07-12 18:34:28 +02:00
ddidderr 66b7858728 feat(legacy): port the v0.5 and v0.6 decoders to Rust
Replace the v0.5 and v0.6 legacy decoder translation units with Rust modules
and keep only narrow C registration shims for the public legacy dispatch ABI.

Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/Cargo.toml --features compression,decompression,dict-builder,legacy-v05,legacy-v06,legacy-v07
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --features compression,decompression,dict-builder,legacy-v05,legacy-v06,legacy-v07 -- -D warnings
- make -C tests check V=1
2026-07-12 18:07:37 +02:00
ddidderr 0d3dae8fd8 feat(compress): port the CCtx parameter object API to Rust
Move CCtx parameter reset/init, parameter set/get, and external sequence
producer registration to Rust with a private C-layout mirror. Keep allocation,
advanced initialization, bounds policy, and context-dependent APIs in C, and
retain C layout assertions and helper bridges for the shared object.

Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/zstd_compress_params_api.rs rust/src/lib.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression zstd_compress_params_api
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --no-default-features --features compression -- -D warnings
- make -B -C lib libzstd.a V=1
- git diff --cached --check
2026-07-12 10:48:40 +02:00
ddidderr 393b2bec45 feat(legacy): port the v0.4 decoder to Rust
Move the frozen v0.4 frame, entropy, streaming, and dictionary decoder
implementation to Rust while retaining the C translation unit as an ABI
anchor. Register the decoder behind the matching legacy feature and keep its
historical behavior isolated from newer formats.

Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/legacy/zstd_v04.rs rust/src/legacy/mod.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/Cargo.toml --no-default-features --features decompression,legacy-v04 legacy::zstd_v04
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --no-default-features --features decompression,legacy-v04 -- -D warnings
- C/Rust v0.4 ABI and decompression build checks
- git diff --cached --check
2026-07-12 10:43:32 +02:00
ddidderr b8ccfd7b87 feat(dict): port the FastCover trainer to Rust
Move FastCover parameter validation, corpus preparation, optimization, and
dictionary training into Rust. Retain the C source as a static-linking ABI
anchor and register the implementation with the compression dictionary-builder
feature set.

Test Plan:
- rustfmt +nightly --check --edition 2021 rust/src/dict_builder_fastcover.rs rust/src/lib.rs
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets --no-default-features --features compression,dict-builder -- -D warnings
- FastCover focused Rust tests and deterministic C-reference harness
- git diff --cached --check
2026-07-12 10:40:51 +02:00
ddidderr 6a9b2d30b5 feat(dict): port the public zdict builder to Rust
Move the public ZDICT helpers, legacy trainer, dictionary finalization,
entropy-table construction, and supporting dictionary logic into Rust. Keep
the C translation unit as an ABI anchor and register the Rust module only
when compression and dictionary-builder features are enabled.

Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path rust/Cargo.toml dict_builder_zdict
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- rustfmt +nightly --check --edition 2021 rust/src/dict_builder_zdict.rs rust/src/lib.rs
- git diff --cached --check
2026-07-12 10:38:41 +02:00
ddidderr 63fdbc3fc2 feat(legacy): port the v0.3 decoder to Rust
Move the frozen v0.3 frame decoder, entropy tables, and streaming context
state into a feature-gated Rust module. Keep the historical C translation
unit as an ABI shim so existing C callers retain the opaque context and
error-code contracts while v0.4 and newer decoders remain unchanged.

Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
  rust/Cargo.toml --no-default-features --features legacy-v01,legacy-v02,
  legacy-v03
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
  rust/Cargo.toml --no-default-features --features compression,decompression,
  dict-builder,legacy-v01,legacy-v02,legacy-v03
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
  rust/Cargo.toml --all-targets --no-default-features --features
  compression,decompression,dict-builder,legacy-v01,legacy-v02,legacy-v03
  -- -D warnings
- make -B -C lib libzstd.a ZSTD_LEGACY_SUPPORT=3 V=1
- git diff --cached --check
2026-07-12 10:14:50 +02:00
ddidderr 45cb1f509e feat(compress): port multithreaded resource pools
Move the reusable buffer and compression-context pools behind a Rust
implementation with a narrow C ABI adapter. The scheduler, job table, serial
LDM state, and stream orchestration remain in C until their private layouts
are ported.

Keep context types opaque across Rust modules so the pool bridge does not
depend on private C layout declarations. This also keeps the existing custom
allocator and pool replacement contracts intact.

Test Plan:
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo clippy --manifest-path
  rust/Cargo.toml --all-targets -- -D warnings
- RUSTC_WRAPPER= CARGO_BUILD_RUSTC_WRAPPER= cargo test --manifest-path
  rust/Cargo.toml zstdmt_compress
- make -B -C lib lib-mt
- git diff --cached --check
2026-07-12 10:13:41 +02:00
ddidderr 468ccfa388 feat(rust): port the COVER dictionary trainer
Move COVER training, frequency mapping, segment selection, optimization,
dictionary shrinking, and best-candidate synchronization into Rust.  The
remaining C translation unit is an ABI shim, while the public ZDICT and COVER
symbols stay available to the existing C dictionary-builder callers.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml dict_builder_cover
- cargo check --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- make -B -C lib libzstd.a V=1
- git diff --check

Depends-on: Rust compression, entropy, and dictionary-finalization leaves
2026-07-12 10:08:03 +02:00
ddidderr 70260b308b feat(rust): port deprecated ZBUFF streaming wrappers
Move the public ZBUFF compatibility layer into the Rust static archive while
keeping stream contexts opaque and forwarding to the current ZSTD streaming
APIs.  The deprecated C translation units are now declaration-only shims, so
the compatibility symbols work in compression-only and decompression-only
feature builds without duplicating private context layouts.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml -- --check
- make -B -C lib libzstd.a ZSTD_LIB_DEPRECATED=1 V=1
- C ZBUFF compress/decompress smoke test against the rebuilt static archive

Depends-on: Rust compression and decompression streaming ABI ports
2026-07-12 09:59:58 +02:00
ddidderr 5d03dca147 feat(compress): port the optimal parser to Rust
Move the binary-tree optimal and ultra block parsers into Rust with a narrow
C projection of match-state, sequence-store, and entropy-table fields. Keep
the established C entry points as wrappers, and avoid reading uninitialized
entropy costs during dictionary tree updates.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- make -B -C tests test-zstream V=1
- git diff --check

Refs: rust/src/zstd_opt.rs, lib/compress/zstd_opt.c
2026-07-12 09:51:45 +02:00
ddidderr 9b909aaef8 feat(legacy): port the v0.2 decoder to Rust
Translate the frozen v0.2 frame, entropy, block, frame-size, and streaming
decoder paths to Rust while preserving the opaque C context ABI. Replace the
3465-line historical C implementation with a declaration-only shim and enable
the Rust module for the legacy-v02 Cargo feature.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml --no-default-features --features compression,decompression,dict-builder,legacy-v02
- make -B -C lib libzstd.a ZSTD_LEGACY_SUPPORT=2 V=1
- git diff --check

Refs: rust/src/legacy/zstd_v02.rs, lib/legacy/zstd_v02.c
2026-07-12 09:40:38 +02:00
ddidderr f7af4cb71f feat(compress): port the public one-shot compressor to Rust
Implement ZSTD_compress in Rust using the migrated parameter, match-finder,
sequence, frame, and superblock leaves. Keep the private context and streaming
entry points C-backed until their configuration-dependent context projection is
ported, and remove only the duplicate C one-shot definition.

Test Plan:
- cargo test --manifest-path rust/Cargo.toml
- cargo clippy --manifest-path rust/Cargo.toml --all-targets -- -D warnings
- cargo +nightly fmt --manifest-path rust/Cargo.toml --all -- --check
- make -B -C tests test-rust-lib-smoke V=1
- git diff --check

Refs: rust/src/zstd_compress.rs, lib/compress/zstd_compress.c
2026-07-12 09:34:08 +02:00