ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,485 Commits 17 Branches 77 Tags
7b94de78ff2c3634cd7ef32f0e16f4722c03578f
Commit Graph
5043 Commits
Author SHA1 Message Date
Nick TerrellandNick Terrell 94c102038b [cpu] Backport fix for rbx clobbering on Windows with Clang 
Backport folly fix for rbx clobbering: https://github.com/facebook/folly/commit/f22f88b8b9d70160388f0f149bc9abaeb82c250b

This supercedes PR #3646.
 2024-03-13 09:45:40 -04:00
Yonatan KomornikandGitHub b20703f273 Updates ZSTD_RowFindBestMatch comment (#3947) 
Updates the comment on the head of `ZSTD_RowFindBestMatch` to make sure it's aligned with recent changes to the hash table.
 2024-03-12 15:10:07 -07:00
Elliot GorokhovskyandGitHub f65b9e27ce Exercise ZSTD_findDecompressedSize() in the simple decompression fuzzer (#3959) 
* Improve decompression fuzzer

* Fix legacy frame header fuzzer crash, add unit test
 2024-03-12 17:07:06 -04:00
Yann ColletandGitHub db996d253e Merge pull request #3933 from facebook/fix3819 
prevent XXH64 from being autovectorized by XXH512 by default
 2024-03-12 09:46:48 -07:00
Yann ColletandGitHub 3b573ba1f0 Merge pull request #3936 from facebook/debug5 
minor: fix incorrect debug level
 2024-03-12 09:45:20 -07:00
Yann ColletandGitHub fe7c645a7e Merge pull request #3937 from facebook/offset0 
new method to deal with offset==0 erroneous edge case
 2024-03-12 09:25:06 -07:00
acceptacrossandGitHub 8ba5bc4729 chore: fix some typos (#3949) 
Signed-off-by: acceptacross <csqcqs@gmail.com>
 2024-03-12 11:44:42 -04:00
Elliot GorokhovskyandGitHub e0872806df Use ZSTD_LEGACY_SUPPORT=5 in make test (#3943) 2024-03-12 10:08:26 -04:00
Yann ColletandGitHub 372fddf4e6 Merge pull request #3860 from likema/fix-xxhash-aix-51 
Fix building xxhash on AIX 5.1
 2024-03-09 15:34:02 -08:00
Yann Collet a9fb8d4c41 new method to deal with offset==0 
in this new method, when an `offset==0` is detected,
it's converted into (size_t)(-1), instead of 1.

The logic is that (size_t)(-1) is effectively an extremely large positive number,
which will not pass the offset distance test at next stage (`execSequence()`).
Checked the source code, and offset is always checked (as it should),
using a formula which is not vulnerable to arithmetic overflow:
```
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart),
```

The benefit is that such a case (offset==0) is always detected as corrupted data
as opposed to relying on the checksum to detect the error.
 2024-03-08 15:26:06 -08:00
Yann Collet aed172a8fe minor: fix incorrect debug level 2024-03-08 14:29:44 -08:00
Yann Collet 007cda88ca prevent XXH64 from being autovectorized by XXH512 by default 
backport fix https://github.com/Cyan4973/xxHash/pull/924 from libxxhash
 2024-03-07 16:43:13 -08:00
Yann ColletandGitHub 8689633fdf Merge pull request #3840 from aimuz/fix-reserved 
lib/decompress: check for reserved bit corruption in zstd
 2024-03-05 13:40:12 -08:00
Yann ColletandGitHub e385c3dd46 Merge pull request #3753 from facebook/make2 
minor Makefile refactoring
 2024-03-03 19:13:00 -08:00
Yann Collet 8d31e8ec42 sizeBlockSequences() also tracks uncompressed size 
and only defines a sub-block boundary when
it believes that it is compressible.

It's effectively an optimization,
avoiding a compression cycle to reach the same conclusion.
 2024-02-26 14:31:12 -08:00
Yann Collet d23b95d21d minor refactor for clarity 
since we can ensure that nbSubBlocks>0
 2024-02-26 14:06:34 -08:00
Yann Collet 86db60752d optimization: bail out faster in presence of incompressible data 2024-02-26 13:27:59 -08:00
Yann Collet ef82b214ad nit: comment indentation 
as reported by @terrelln
 2024-02-26 13:23:59 -08:00
Yann Collet aa8592c532 minor: reformulate nbSubBlocks assignment 2024-02-26 13:21:14 -08:00
Yann Collet e0412c2062 fix extraneous semicolon ';' 
as reported by @terrelln
 2024-02-26 12:26:54 -08:00
Yann Collet 1fafd0c4ae fix minor visual static analyzer warning 
it's a false positive,
but change the code nonetheless to make it more obvious to the static analyzer.
 2024-02-25 19:45:32 -08:00
Yann Collet 038a8a906b targetCBlockSize: modified splitting strategy to generate blocks of more regular size 
notably avoiding to feature a larger first block
 2024-02-25 17:39:29 -08:00
Yann Collet f8372191f5 reduced minimum compressed block size 
with the intention to match the transport layer size,
such as Ethernet and 4G mobile networks.
 2024-02-24 01:59:16 -08:00
Yann Collet f77f634d41 update API documentation 2024-02-24 01:28:17 -08:00
Yann Collet 4b51526412 fix partial block uncompressed 2024-02-24 01:24:58 -08:00
Yann Collet 6719794379 fixed some regressionTests 
but not all
 2024-02-23 18:48:29 -08:00
Yann Collet 0591e7eea1 minor: fix overly cautious conversion warning 2024-02-23 16:05:09 -08:00
Yann Collet 3b40100058 fix long sequences (> 64 KB) 2024-02-23 15:35:12 -08:00
Yann Collet 6b11fc436c fix issue with incompressible sections 2024-02-23 14:53:56 -08:00
Yann Collet cc4530924b speed optimized version of targetCBlockSize 
note that the size of individual compressed blocks will vary more wildly with this modification.
But it seems good enough for a first test, and fix the speed regression issue.
Further refinements can be attempted later.
 2024-02-23 14:03:26 -08:00
Yann ColletandGitHub 621a263fb2 Merge pull request #3903 from gruenich/feature/reduce-scope-of-variables 
Reduce scope of variables
 2024-02-22 09:42:02 -08:00
Yann Collet e62e15df19 fix clangbuild 
notably -Wconversion and -Wdocumentation
 2024-02-20 22:43:22 -08:00
Christoph Grüninger b921f1aad6 Reduce scope of variables 
This improves readability, keeps variables local, and
prevents the unintended use (e.g. typo) later on.
Found by Cppcheck (variableScope)
 2024-02-11 22:00:03 +01:00
Yann Collet b0e8580dc7 fix fuzz issue 5131069967892480 2024-02-08 16:38:20 -08:00
Yann Collet 22574d848d fix issue 5921623844651008 
ossfuzz managed to create a scenario which triggers an `assert`.
This fixes it, by giving +1 more space for the backward search pass.
 2024-02-06 13:01:14 -08:00
Yann Collet b88c593d8f added or updated code comments 
as suggested by @terrelln,
to make the code of the optimal parser a bit more understandable.
 2024-02-05 18:32:25 -08:00
Yann Collet 6c35fb2e8c fix msan warnings 2024-02-05 01:21:06 -08:00
Yann Collet 641749fc09 fix uasan dictionary_stream_round_trip fuzz test 2024-02-05 00:36:10 -08:00
Yann Collet fe2e2ad36d use ZSTD_memcpy() 
which can be redirected in Linux kernel mode
 2024-02-03 19:57:38 -08:00
Yann Collet 0ae21d8c31 removed trace control 2024-02-03 19:32:59 -08:00
Yann Collet 5474edbe60 fixed wrong assert 
by introducing ZSTD_OPT_SIZE
 2024-02-03 19:31:53 -08:00
Yann Collet e5af24c5fa fixed wrong assert 2024-02-03 17:48:29 -08:00
Yann Collet 8168a451e5 minor optimization, mostly for clarity 2024-02-03 17:26:47 -08:00
Yann Collet d31018e223 finally, a version that generalizes well 
While it's not always strictly a win,
it's a win for files that see a noticeably compression ratio increase,
while it's a very small noise for other files.

Downside is, this patch is less efficient for 32-bit arrays of integer
than the previous patch which was introducing losses for other files,
but it's still a net improvement on this scenario.
 2024-02-03 14:26:18 -08:00
Yann Collet 0166b2ba80 modification: differentiate literal update at pos+1 
helps when litlen==1 is cheaper than litlen==0

works great on pathological arr[u32] examples
but doesn't generalize well on other files.

silesia/x-ray is amoung the most negatively affected ones.
 2024-01-31 11:20:43 -08:00
Yann Collet 4683667785 refactor optimal parser 
store stretches as intermediate solution instead of sequences.
makes it possible to link a solution to a predecessor.
 2024-01-31 02:51:46 -08:00
Yann Collet de10f56be2 improve high compression ratio for file like #3793 
this works great for 32-bit arrays,
notably the synthetic ones, with extreme regularity,
unfortunately, it's not universal,
and in some cases, it's a loss.
Crucially, on average, it's a loss on silesia.
The most negatively impacted file is x-ray.
It deserves an investigation before suggesting it as an evolution.
 2024-01-29 23:25:24 -08:00
Yann Collet e6f4b46493 playTests.sh does no longer needs grep -E 
it makes the test script more portable across posix systems
because `grep -E` is not guaranteed
while `grep` is fairly common.
 2024-01-15 11:16:46 -08:00
Like Ma 66269e74a0 Fix building xxhash on AIX 5.1 2024-01-14 00:09:48 +08:00
Yann ColletandGitHub a07cae3976 Merge pull request #3847 from michoecho/fix_nullptr_deref_in_createCDict 
Fix a nullptr dereference in ZSTD_createCDict_advanced2()
 2023-12-30 13:23:39 -08:00