ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,958 Commits 17 Branches 77 Tags
d4ae5c3752c0f4a918cadcfb9489c91e59bb1f47
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet d4ae5c3752 update installed lib64gcc version 2025-01-15 15:52:27 -08:00
Yann Collet 642157cc45 short-test-1: separate tests, giving each a name 
to better observe where the problem might be
 2025-01-15 15:39:20 -08:00
Yann Collet 80ff61de1d fixed cmake-build-and-test-check 
missing the liblzma library
 2025-01-15 15:28:23 -08:00
Yann Collet 196e76efe1 fix make-external-compressors "only-lzma" test 
missing the liblzma-dev package
 2025-01-15 15:24:27 -08:00
Yann Collet 0e819c9f93 fix meson-linux build test 
missing the liblzma package
 2025-01-15 15:20:28 -08:00
Yann Collet 908a95889b fix clang-pgo test 2025-01-15 15:05:44 -08:00
Yann Collet f5dbdac818 fixed macro name typo 2025-01-12 23:08:26 -08:00
Yann Collet 7b294caf46 add one valid test case 
ZSTD_compressSequencesAndLiterals() may return a specific error code
when data to compress is non-compressible.
 2024-12-23 19:43:17 -08:00
Yann Collet f8725e80cc added fuzzer test for compressSequencesAndLiterals() 
piggy-backing onto existing compressSequences() fuzzer test
 2024-12-23 18:42:51 -08:00
Yann Collet 47cbfc87a9 restore invocation of ZSTD_entropyCompressSeqStore() 
in the ZSTD_compressSequences() pipeline
 2024-12-20 10:37:01 -08:00
Yann Collet 522adc34eb minor: use MEM_writeLE24() 
so that an empty frame needs only 3 bytes of dstCapacity.
 2024-12-20 10:37:01 -08:00
Yann Collet b7a9e69d8d added parameter litCapacity 
to ZSTD_compressSequencesAndLiterals()
to enforce the litCapacity >= litSize+8 condition.
 2024-12-20 10:37:01 -08:00
Yann Collet 76445bb379 add a check, to return an error if Sequence validation is enabled 
since ZSTD_compressSequencesAndLiterals() doesn't support it.
 2024-12-20 10:37:01 -08:00
Yann Collet 52a9bc6fca fixed minor error in preparation of one fullbench scenario 2024-12-20 10:37:00 -08:00
Yann Collet ab0f1798e8 ensure that srcSize is controlled 2024-12-20 10:37:00 -08:00
Yann Collet b339efff2b add dedicated error code for special case 
ZSTD_compressSequencesAndLiterals() cannot produce an uncompressed block
 2024-12-20 10:37:00 -08:00
Yann Collet a80f55f47d added a test for ZSTD_compressSequencesAndLiterals 
checks that srcSize is present in the frame header
and bounds the window size.
 2024-12-20 10:37:00 -08:00
Yann Collet 0a54f6f288 ZSTD_compressSequencesAndLiterals requires srcSize as parameter 
this makes it possible to adjust windowSize to its tightest.
 2024-12-20 10:37:00 -08:00
Yann Collet ad023b392f fixed minor error in one benchmark scenario 2024-12-20 10:37:00 -08:00
Yann Collet b7b4e86347 fixed minor conversion warning 2024-12-20 10:37:00 -08:00
Yann Collet 12c47d3262 improved speed of the Sequences converter 2024-12-20 10:37:00 -08:00
Yann Collet 95ad9e47ff added benchmark for ZSTD_convertBlockSequences_wBlockDelim() 2024-12-20 10:37:00 -08:00
Yann Collet d48e330ae1 change name to ZSTD_convertSequences*() 2024-12-20 10:37:00 -08:00
Yann Collet 61ac8311e0 attempt to silence Visual Studio warning about fopen() 2024-12-20 10:37:00 -08:00
Yann Collet f617e86b71 fixed incorrect assert 2024-12-20 10:37:00 -08:00
Yann Collet 47edd0acf4 removed fullbench-dll project from visual solutions 2024-12-20 10:37:00 -08:00
Yann Collet 6f8c1046d0 update Visual Studio solutions 2024-12-20 10:37:00 -08:00
Yann Collet f0d0d95234 added tests 
check that ZSTD_compressAndLiterals() also controls that the `srcSize` field is exact.
 2024-12-20 10:37:00 -08:00
Yann Collet 31b5ef2539 ZSTD_compressSequencesAndLiterals() now supports multi-blocks frames. 2024-12-20 10:36:59 -08:00
Yann Collet 5164d44dab change advanced parameter name: ZSTD_c_repcodeResolution 
and updated its documentation.
Note: older name ZSTD_c_searchForExternalRepcodes remains supported via #define
 2024-12-20 10:36:59 -08:00
Yann Collet ca8bd83373 minor: cleaner function parameter repcodeResolution 2024-12-20 10:36:59 -08:00
Yann Collet d2d0fdac42 updated documentation on validateSequence 2024-12-20 10:36:59 -08:00
Yann Collet 1f6d6815c3 optimization: instantiate specialized version without Sequence checking code 
results in +4% compression speed,
thanks to removal of branches in the hot loop.
 2024-12-20 10:36:59 -08:00
Yann Collet a288751de7 minor optimization: only track seqPos->posInSrc when validateSequences is enabled 
note: very minor saving, no performance impact
 2024-12-20 10:36:59 -08:00
Yann Collet 8ab04097ed add the compressSequences() benchmark scenario 2024-12-20 10:36:59 -08:00
Yann Collet c050ae4fb8 fullbench: preparation functions are now in charge of allocating buffers 2024-12-20 10:36:59 -08:00
Yann Collet 8b7e1b795d minor variable renaming 2024-12-20 10:36:59 -08:00
Yann Collet c540976a4b minor: test reordering 2024-12-20 10:36:59 -08:00
Yann Collet 4c097b4939 refactor fullbench 
to make it easier to add new scenarios
 2024-12-20 10:36:58 -08:00
Yann Collet 09964c6276 prepare fullbench for a refactor 2024-12-20 10:36:58 -08:00
Yann Collet 0b013b2688 added unit tests to ZSTD_compressSequencesAndLiterals() 
seems to work as expected,
correctly control that `litSize` and `srcSize` are exactly correct.
 2024-12-20 10:36:58 -08:00
Yann Collet 14a21e43b3 produced ZSTD_compressSequencesAndLiterals() as a separate pipeline 
only supports explicit delimiter mode, at least for the time being
 2024-12-20 10:36:58 -08:00
Yann Collet bcb15091aa minor: more accurate variable scope 2024-12-20 10:36:58 -08:00
Yann Collet 047db4f1f8 ZSTD_SequenceCopier_f no returns the nb of bytes consumed from input 
which feels much more natural
 2024-12-20 10:36:58 -08:00
Yann Collet 4ef9d7d585 codemod: ZSTD_cParamMode_e -> ZSTD_CParamMode_e 2024-12-20 10:36:58 -08:00
Yann Collet 56cfb7816a codemod: ZSTD_paramSwitch_e -> ZSTD_ParamSwitch_e 2024-12-20 10:36:58 -08:00
Yann Collet 13b9296d79 minor simplification 2024-12-20 10:36:58 -08:00
Yann Collet 2503b64345 fix minor artifact error in single_file_lib 2024-12-20 10:36:58 -08:00
Yann Collet e0f3aaee46 doc: add mention of frame checksum incompatibility 
for new prototype ZSTD_compressSequencesAndLiterals()
 2024-12-20 10:36:57 -08:00
Yann Collet 08edecb78c codemod: ZSTD_blockCompressor -> ZSTD_BlockCompressor_f 2024-12-20 10:36:57 -08:00
Yann Collet 25bef24c5c codemod: rawSeqStore_t -> RawSeqStore_t 2024-12-20 10:36:57 -08:00
Yann Collet 41c667c0fd codemod: repcodes_t -> Repcodes_t 2024-12-20 10:36:57 -08:00
Yann Collet 5df80acedb codemod: ZSTD_matchState_t -> ZSTD_MatchState_t 2024-12-20 10:36:57 -08:00
Yann Collet fa468944f2 codemod: ZSTD_buildSeqStore_e -> ZSTD_BuildSeqStore_e 2024-12-20 10:36:57 -08:00
Yann Collet 30671d77af codemod: ZSTD_sequencePosition -> ZSTD_SequencePosition 2024-12-20 10:36:57 -08:00
Yann Collet 5359d16d8d enable proper type 2024-12-20 10:36:57 -08:00
Yann Collet 03d95f9d13 fix proper type for .forceNonContiguous 2024-12-20 10:36:57 -08:00
Yann Collet 76dd3a98c4 scope: ZSTD_copySequencesToSeqStore*() are private to ZSTD_compress.c 
no need to publish them outside of this unit.
 2024-12-20 10:36:57 -08:00
Yann Collet 1ac79ba1b6 minor: simplify ZSTD_selectSequenceCopier 2024-12-20 10:36:56 -08:00
Yann Collet 894ea31281 codemod: ZSTD_sequenceCopier -> ZSTD_SequenceCopier_f 2024-12-20 10:36:56 -08:00
Yann Collet c97522f7fb codemod: ZSTD_sequenceFormat_e -> ZSTD_SequenceFormat_e 
since it's a type name.

Note: in contrast with previous names, this one is on the Public API side.
So there is a #define, so that existing programs using ZSTD_sequenceFormat_e still work.
 2024-12-20 10:36:56 -08:00
Yann Collet 0165eeb441 created ZSTD_entropyCompressSeqStore_wExtLitBuffer() 
can receive externally defined buffer of literals
 2024-12-20 10:36:56 -08:00
Yann Collet e9f8a119b4 ZSTD_entropyCompressSeqStore_internal() can accept an externally defined literals buffer 2024-12-20 10:36:56 -08:00
Yann Collet 0442e43aca codemod: ZSTD_defaultPolicy_e -> ZSTD_DefaultPolicy_e 2024-12-20 10:36:56 -08:00
Yann Collet 477a01067f codemod: symbolEncodingType_e -> SymbolEncodingType_e 2024-12-20 10:36:56 -08:00
Yann Collet 8d4506bc94 codemod: ZSTD_sequenceLength -> ZSTD_SequenceLength 2024-12-20 10:36:55 -08:00
Yann Collet a2245721ca codemod: seqStore_t -> SeqStore_t 
same idea, SeqStore_t is a type name, it should start with a Capital letter.
 2024-12-20 10:36:55 -08:00
Yann Collet 9671813375 codemod: seqDef -> SeqDef 
SeqDef is a type name, so it should start with a Capital letter.
It's an internal symbol, no impact on public API.
 2024-12-20 10:36:55 -08:00
Yann Collet b4a40a845f move Sequences definition to zstd_compress_internal.h 
they should not be in common/zstd_internal.h,
since these definitions are not shared beyond lib/compress/.
 2024-12-20 10:36:55 -08:00
Yann Collet a00f45a037 created ZSTD_storeSeqOnly() 
makes it possible to register a sequence without copying its literals.
 2024-12-20 10:36:04 -08:00
Yann Collet 125f05282b publish new symbol ZSTD_compressSequencesAndLiterals() 2024-12-20 10:36:04 -08:00
Yann Collet 2e02cd330d inform manual users that it's automatically generated 
suggested by @Eugeny1
 2024-10-31 15:06:48 -07:00
Yann Collet adbb536d00 fixed single file library test 
copy "zstd_errors.h" alongside "zstd.h" to pass compilation test.
 2024-10-31 14:38:36 -07:00
Yann Collet d9553fd218 elevated ZSTD_getErrorCode() to stable status 
answering #4183
 2024-10-31 14:15:50 -07:00
Yann Collet bbaba45589 change experimental parameter name 
from ZSTD_c_useBlockSplitter to ZSTD_c_splitAfterSequences.
 2024-10-31 13:43:40 -07:00
Yann Collet 4f93206d62 changed variable name to ZSTD_c_blockSplitterLevel 
suggested by @terrelln
 2024-10-29 11:12:09 -07:00
Yann Collet f593ccda04 removed trace left over 2024-10-28 16:57:01 -07:00
Yann Collet fcbf6b014a fixed minor conversion warning 2024-10-28 16:47:38 -07:00
Yann Collet 37706a677c added a test 
test both that the new parameter works as intended,
and that the over-split protection works as intended
 2024-10-28 16:31:15 -07:00
Yann Collet 226ae73311 expose new parameter ZSTD_c_blockSplitter_level 2024-10-28 16:31:15 -07:00
Yann Collet 01474bf73b add internal compression parameter preBlockSplitter_level 
not yet exposed to the interface.

Also: renames `useBlockSplitter` to `postBlockSplitter`
to better qualify the difference between the 2 settings.
 2024-10-28 16:31:15 -07:00
Yann Collet 4e1a87976f update regression results 2024-10-25 16:33:00 -07:00
Yann Collet 5b4ce643f0 update ZSTD_splitBlock() documentation 2024-10-25 16:25:02 -07:00
Yann Collet e557abc8a0 new block splitting variant _fromBorders 
less precise but still suitable for `fast` strategy.
 2024-10-25 16:13:55 -07:00
Yann Collet c63b5d2294 update regression results 2024-10-24 14:47:55 -07:00
Yann Collet da2c0dffd8 add faster block splitting heuristic, suitable for dfast strategy 2024-10-24 14:37:00 -07:00
Yann Collet 2366a87ddc fix minor visual conversion warning 2024-10-24 13:38:12 -07:00
Yann Collet ea082692aa added block splitter variant for greedy & lazy (levels 5 to 7) 2024-10-24 13:27:01 -07:00
Yann Collet 326c45bb8e complete sample11 with reduced fingerprint size 2024-10-24 13:17:56 -07:00
Yann Collet ca6e55cbf5 reduce splitBlock arguments 2024-10-24 13:17:56 -07:00
Yann Collet 94d7b07425 organize specialization at recordFingerprint level 2024-10-24 13:17:56 -07:00
Yann Collet 566763fdc9 new variant, sampling by 11 2024-10-24 13:17:56 -07:00
Yann Collet 70c77d20d6 update regression results 
first block is no longer splitted since adding the @savings over-split protection
 2024-10-24 11:43:16 -07:00
Yann Collet 90095f056d apply limit conditions for all splitting strategies 
instead of just for blind split.

This is in anticipation of adversarial input,
that would intentionally target the sampling pattern of the split detector.

Note that, even without this protection, splitting can never expand beyond ZSTD_COMPRESSBOUND(),
because this upper limit uses a 1KB block size worst case scenario,
and splitting never creates blocks thath small.

The protection is more to ensure that data is not expanded by more than 3-bytes per 128 KB full block,
which is a much stricter limit.
 2024-10-24 11:36:56 -07:00
Yann Collet bbda1acf85 update regression results 2024-10-23 15:56:56 -07:00
Yann Collet c80645a055 stricter limits to ensure expansion factor with blind-split strategy 
issue reported by @terrelln
 2024-10-23 14:55:10 -07:00
Yann Collet 7d3e5e3ba1 split all full 128 KB blocks 
this helps make the streaming behavior more consistent,
since it does no longer depend on having more data presented on the input.

suggested by @terrelln
 2024-10-23 14:18:48 -07:00
Yann Collet b68ddce818 rewrite fingerprint storage to no longer need 64-bit members 
so that it can be stored using standard alignment requirement (sizeof(void*)).

Distance function still requires 64-bit signed multiplication though,
so it won't change the issue regarding the bug in ubsan for clang 32-bit on github ci.
 2024-10-23 11:50:57 -07:00
Yann Collet 18b1e67223 fixed extraneous return 
strict C90 compliance test
 2024-10-23 11:50:57 -07:00
Yann Collet d2eeed53dc updated compression results 
due to integration of `sample5` strategy, leading to better compression ratios on a range of levels
 2024-10-23 11:50:57 -07:00