ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,786 Commits 17 Branches 77 Tags
90095f056d4ccb8e0ed0942b2e30a664f0489932
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 90095f056d apply limit conditions for all splitting strategies 
instead of just for blind split.

This is in anticipation of adversarial input,
that would intentionally target the sampling pattern of the split detector.

Note that, even without this protection, splitting can never expand beyond ZSTD_COMPRESSBOUND(),
because this upper limit uses a 1KB block size worst case scenario,
and splitting never creates blocks thath small.

The protection is more to ensure that data is not expanded by more than 3-bytes per 128 KB full block,
which is a much stricter limit.
 2024-10-24 11:36:56 -07:00
Yann Collet bbda1acf85 update regression results 2024-10-23 15:56:56 -07:00
Yann Collet c80645a055 stricter limits to ensure expansion factor with blind-split strategy 
issue reported by @terrelln
 2024-10-23 14:55:10 -07:00
Yann Collet 7d3e5e3ba1 split all full 128 KB blocks 
this helps make the streaming behavior more consistent,
since it does no longer depend on having more data presented on the input.

suggested by @terrelln
 2024-10-23 14:18:48 -07:00
Yann Collet b68ddce818 rewrite fingerprint storage to no longer need 64-bit members 
so that it can be stored using standard alignment requirement (sizeof(void*)).

Distance function still requires 64-bit signed multiplication though,
so it won't change the issue regarding the bug in ubsan for clang 32-bit on github ci.
 2024-10-23 11:50:57 -07:00
Yann Collet 18b1e67223 fixed extraneous return 
strict C90 compliance test
 2024-10-23 11:50:57 -07:00
Yann Collet d2eeed53dc updated compression results 
due to integration of `sample5` strategy, leading to better compression ratios on a range of levels
 2024-10-23 11:50:57 -07:00
Yann Collet 0be334d208 fixes static state allocation check 
detected by @felixhandte
 2024-10-23 11:50:57 -07:00
Yann Collet 06b7cfabf8 rewrote ZSTD_cwksp_initialAllocStart() to be easier to read 
following a discussion with @felixhandte
 2024-10-23 11:50:57 -07:00
Yann Collet 16450d0732 rewrite penalty update 
suggested by @terrelln
 2024-10-23 11:50:57 -07:00
Yann Collet 1ec5f9f1f6 changed loop exit condition so that there is no need to assert() within the loop. 2024-10-23 11:50:57 -07:00
Yann Collet 4662f6e646 renamed: FingerPrint => Fingerprint 
suggested by @terrelln
 2024-10-23 11:50:57 -07:00
Yann Collet ea85dc7af6 conservatively estimate over-splitting in presence of incompressible loss 
ensure data can never be expanded by more than 3 bytes per full block.
 2024-10-23 11:50:57 -07:00
Yann Collet 5ae34e4c96 ensure lastBlock is correctly determined 
reported by @terrelln
 2024-10-23 11:50:57 -07:00
Yann Collet 7bad787d8b made ZSTD_isPower2() an inline function 2024-10-23 11:50:57 -07:00
Yann Collet a167571db5 added a faster block splitter variant 
that samples 1 in 5 positions.

This variant is fast enough for lazy2 and btlazy2,
but it's less good in combination with post-splitter at higher levels (>= btopt).
 2024-10-23 11:50:57 -07:00
Yann Collet 1c62e714ab minor split optimization 
let's fill the initial stats directly into target fingerprint
 2024-10-23 11:50:57 -07:00
Yann Collet dac26eaeac updated regression test results 2024-10-23 11:50:57 -07:00
Yann Collet cae8d13294 splitter workspace is now provided by ZSTD_CCtx* 2024-10-23 11:50:56 -07:00
Yann Collet 4685eafa81 fix alignment test 
for non 64-bit systems
 2024-10-23 11:50:56 -07:00
Yann Collet 433f4598ad fixed minor conversion warnings on Visual 2024-10-23 11:50:56 -07:00
Yann Collet 73a6653653 ZSTD_splitBlock_4k() uses externally provided workspace 
ideally, this workspace would be provided from the ZSTD_CCtx* state
 2024-10-23 11:50:56 -07:00
Yann Collet 7f015c2fd7 replaced uasan32 test by asan32 test 2024-10-23 11:50:56 -07:00
Yann Collet 31d48e9ffa fixing minor formatting issue in 32-bit mode with logs enabled 2024-10-23 11:50:56 -07:00
Yann Collet 76ad1d6903 fixed VS2010 solution 2024-10-23 11:50:56 -07:00
Yann Collet cdddcaaec9 new Makefile target mesonbuild 
for easier local testing
 2024-10-23 11:50:56 -07:00
Yann Collet 6939235f01 fixed meson build 2024-10-23 11:50:56 -07:00
Yann Collet 80a912dec1 fixed zstreamtest 2024-10-23 11:50:56 -07:00
Yann Collet 6dc52122e6 fixed c90 comment style 2024-10-23 11:50:56 -07:00
Yann Collet 20c3d176cd fix assert 2024-10-23 11:50:56 -07:00
Yann Collet 0d4b520657 only split full blocks 
short term simplification
 2024-10-23 11:50:56 -07:00
Yann Collet dd38c677eb fixed single-library build 2024-10-23 11:50:56 -07:00
Yann Collet 8b3887f579 fixed kernel build 2024-10-23 11:50:56 -07:00
Yann Collet f83ed087f6 fixed RLE detection test 2024-10-23 11:50:56 -07:00
Yann Collet 83a3402a92 fix overlap write scenario in presence of incompressible data 2024-10-23 11:50:56 -07:00
Yann Collet fa147cbb4d more ZSTD_memset() to apply 2024-10-23 11:50:56 -07:00
Yann Collet 6021b6663a minor C++-ism 
though I really wonder if this is a property worth maintaining.
 2024-10-23 11:50:56 -07:00
Yann Collet e2d7d08888 use ZSTD_memset() 
for better portability on Linux kernel
 2024-10-23 11:50:56 -07:00
Yann Collet 586ca96fec do not use new as variable name 2024-10-23 11:50:56 -07:00
Yann Collet 9e52789962 fixed strict C90 semantic 2024-10-23 11:50:56 -07:00
Yann Collet a5bce4ae84 XP: add a pre-splitter 
instead of ingesting only full blocks, make an analysis of data, and infer where to split.
 2024-10-23 11:50:56 -07:00
Yann Collet 41d870fbbf updated regression tests results 2024-10-17 11:06:26 -07:00
Yann Collet ff8e98bebe enable regression tests at pull request time 
was transferred from circleci,
but was only triggered on push into dev,
i.e. after pull request is merged.
 2024-10-17 09:45:16 -07:00
Yann Collet 47d4f5662d rewrite code in the manner suggested by @terrelln 2024-10-17 09:37:23 -07:00
Yann Collet 61d08b0e42 fix test 
a margin of 4 is insufficient to guarantee compression success.
 2024-10-17 09:37:23 -07:00
Yann Collet 6326775166 slightly improved compression ratio at levels 3 & 4 
The compression ratio benefits are small but consistent, i.e. always positive.
On `silesia.tar` corpus, this modification saves ~75 KB at level 3.
The measured speed cost is negligible, i.e. below noise level, between 0 and -1%.
 2024-10-17 09:37:23 -07:00
Yann Collet 730d2dce41 fix test 2024-10-15 18:44:40 -07:00
Yann Collet c2abfc5ba4 minor improvement to level 3 dictionary compression ratio 2024-10-15 17:58:33 -07:00
Yann Collet e63896eb58 small dictionary compression speed improvement 
not as good as small-blocks improvement,
but generally positive.
 2024-10-15 17:48:35 -07:00
Yann Collet e6740355e3 attempt parallel test running with -j 2024-10-11 18:01:28 -07:00
Yann Collet 6f2e29a234 measure if -O2 makes the test complete faster 2024-10-11 17:30:55 -07:00
Yann Collet 1024aa9252 attempt to make 32-bit tests faster 
this is the longest CI test, reaching ~40mn on last PR
 2024-10-11 16:24:25 -07:00
Yann Collet 8e5823b65c rename variable name 
findMatch -> matchFound
since it's a test, as opposed to an active search operation.
suggested by @terrelln
 2024-10-11 15:38:12 -07:00
Yann Collet 83de00316c fixed parameter ordering in dfast 
noticed by @terrelln
 2024-10-11 15:36:15 -07:00
Yann Collet 3b343dcfb1 refactor huffman prefix code paragraph 2024-10-07 17:15:07 -07:00
Yann Collet b84653fc83 fix dependency for nightly github actions tests 2024-10-03 15:10:16 -07:00
Yann Collet a8b86d024a refactor documentation of the FSE decoding table build process 2024-10-02 23:09:06 -07:00
Yann Collet de6cc98e07 fix incorrect pointer manipulation 
fix #4155
 2024-10-01 09:25:26 -07:00
Yann Collet 3d5d3f5630 removing nightly tests built on circleci 2024-09-30 21:38:29 -07:00
Yann Collet 039f404faa update documentation to specify that Dictionary can be used for benchmark 
fix #4139
 2024-09-25 16:56:01 -07:00
Yann Collet 7a48dc230c fix doc nit: ZDICT_DICTSIZE_MIN 
fix #4142
 2024-09-19 09:50:30 -07:00
Yann Collet 09cb37cbb1 Limit range of operations on Indexes in 32-bit mode 
and use unsigned type.
This reduce risks that an operation produces a negative number when crossing the 2 GB limit.
 2024-08-21 11:03:43 -07:00
Yann Collet 14b8d398fd fix c90 comment style 2024-08-06 12:47:30 -07:00
Yann Collet 89451cafbd formatString_u() can display numbers > 100 
fixes #4110
 2024-08-06 11:44:37 -07:00
Yann Collet cb784edf5d added android-ndk-build 2024-07-30 11:34:49 -07:00
Yann Collet 80170f6aad fix macos build 
weird: after replacing the UNAME line with an identical one,
it does work properly now(??).
Possibly a case of hidden special character?
 2024-06-18 20:24:00 -07:00
Yann Collet d7a84a683f added cygwin install test 2024-06-12 10:57:15 -07:00
Yann Collet 2acf90431a minor:doc: specify decompression behavior in presence of multiple concatenated frames 
directly at ZSTD_decompress() level.
 2024-06-03 18:30:23 -07:00
Yann Collet a86f5f3f33 update documentation of ZSTD_decompressStream() 
slightly more precise, by recommending to check the return value.
fix #4030
 2024-04-23 09:31:35 -07:00
Yann Collet ff7a151f2e update test name 2024-04-21 10:44:50 -07:00
Yann Collet a2f145f059 decompression errors always display the full origin filename 
instead of the truncated size-limited version.
 2024-04-01 11:49:21 -07:00
Yann Collet e0ee0fccf8 updated benchmarks for v1.5.6 2024-03-27 18:19:31 -07:00
Yann Collet ebf24b7b77 fix nodejs deprecation warning 
by updating msys2 action
 2024-03-27 16:10:15 -07:00
Yann Collet 273d1279ca try to silence some scorecard warnings 2024-03-21 13:25:48 -07:00
Yann Collet 6f1215b874 fix ZSTD_TARGETCBLOCKSIZE_MIN test 
when requested CBlockSize is too low,
bound it to the minimum
instead of returning an error.
 2024-03-18 14:10:08 -07:00
Yann Collet c5da438dc0 fix typo 2024-03-18 12:33:22 -07:00
Yann Collet 3d18d9a9ce updated API manual 2024-03-18 12:30:54 -07:00
Yann Collet 902c7ec1fe add doc on CCtx UB state 2024-03-18 12:30:35 -07:00
Yann Collet 5d82c2b57c add a paragraph on UB DCtx state after error 2024-03-18 12:17:41 -07:00
Yann Collet f5728da365 update targetCBlockSize documentation 2024-03-18 12:04:02 -07:00
Yann Collet 351498b932 update CHANGELOG for v1.5.6 2024-03-14 17:01:34 -07:00
Yann Collet 686e7e4b4b updated version to v1.5.6 2024-03-14 15:38:14 -07:00
Yann Collet 9cc3304614 add line number to debug traces 2024-03-14 12:11:11 -07:00
Yann Collet c8ab027227 reduce the amount of includes in "cover.h" 2024-03-13 11:29:28 -07:00
Yann Collet ff6713fd72 fix duplicated paragraph 
reported by @zougloub
 2024-03-12 13:09:10 -07:00
Yann Collet 5473b72a05 updated documentation 
following recommendations by @zougloub at #3698
 2024-03-12 12:27:33 -07:00
Yann Collet 83ec3d0164 no longer truncate file name in verbose mode 
fix #3702
 2024-03-12 11:27:42 -07:00
Yann Collet 37ff4f91eb removed golden-decompression/.gitignore 
replaced by an exclusion rule in tests/.gitignore
 2024-03-12 10:47:27 -07:00
Yann Collet 0ae98ba215 add same .gitignore rule in golden-decompression/ 
as requested by @embg
 2024-03-12 09:47:54 -07:00
Yann Collet b39c76765b Add the zeroSeq sample 
that should have been part of #3674
 2024-03-12 09:28:25 -07:00
Yann Collet 74e856a195 add tests inspired from #2927 
centered around -T# and --fast=# arguments
 2024-03-11 17:57:57 -07:00
Yann Collet f6039f3d5f cmake refactor: move HP-UX specific logic into its own function 
reduce visual clutter of corner case configuration.
 2024-03-11 16:22:13 -07:00
Yann Collet 1362699e87 minor man page clarification 2024-03-11 12:23:37 -07:00
Yann Collet fbd9e628ae added tests 2024-03-11 12:17:34 -07:00
Yann Collet c610a01d7d fix #3719 
only disable `--rm` at end of command line parsing,
so that `-c` only disables `--rm` if it's effectively selected,
and not if it's overriden by a later `-o FILE` command.
 2024-03-11 11:38:55 -07:00
Yann Collet eb5f7a7fa2 produced golden sample for the offset==0 decoder test 
is correctly detected as corrupted by new version,
and is accepted (changed into offset==1) by older version.

updated documentation accordingly, with an hexadecimal representation.
 2024-03-09 00:33:44 -08:00
Yann Collet d2f56ba442 update documentation 2024-03-08 15:55:30 -08:00
Yann Collet a9fb8d4c41 new method to deal with offset==0 
in this new method, when an `offset==0` is detected,
it's converted into (size_t)(-1), instead of 1.

The logic is that (size_t)(-1) is effectively an extremely large positive number,
which will not pass the offset distance test at next stage (`execSequence()`).
Checked the source code, and offset is always checked (as it should),
using a formula which is not vulnerable to arithmetic overflow:
```
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart),
```

The benefit is that such a case (offset==0) is always detected as corrupted data
as opposed to relying on the checksum to detect the error.
 2024-03-08 15:26:06 -08:00
Yann Collet aed172a8fe minor: fix incorrect debug level 2024-03-08 14:29:44 -08:00
Yann Collet ad590275b4 added RISC-V emulation tests on Github CI 2024-03-07 16:54:44 -08:00