ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,656 Commits 17 Branches 77 Tags
dev
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 80170f6aad fix macos build 
weird: after replacing the UNAME line with an identical one,
it does work properly now(??).
Possibly a case of hidden special character?
 2024-06-18 20:24:00 -07:00
Yann Collet d7a84a683f added cygwin install test 2024-06-12 10:57:15 -07:00
Yann Collet 2acf90431a minor:doc: specify decompression behavior in presence of multiple concatenated frames 
directly at ZSTD_decompress() level.
 2024-06-03 18:30:23 -07:00
Yann Collet a86f5f3f33 update documentation of ZSTD_decompressStream() 
slightly more precise, by recommending to check the return value.
fix #4030
 2024-04-23 09:31:35 -07:00
Yann Collet ff7a151f2e update test name 2024-04-21 10:44:50 -07:00
Yann Collet a2f145f059 decompression errors always display the full origin filename 
instead of the truncated size-limited version.
 2024-04-01 11:49:21 -07:00
Yann Collet e0ee0fccf8 updated benchmarks for v1.5.6 2024-03-27 18:19:31 -07:00
Yann Collet ebf24b7b77 fix nodejs deprecation warning 
by updating msys2 action
 2024-03-27 16:10:15 -07:00
Yann Collet 273d1279ca try to silence some scorecard warnings 2024-03-21 13:25:48 -07:00
Yann Collet 6f1215b874 fix ZSTD_TARGETCBLOCKSIZE_MIN test 
when requested CBlockSize is too low,
bound it to the minimum
instead of returning an error.
 2024-03-18 14:10:08 -07:00
Yann Collet c5da438dc0 fix typo 2024-03-18 12:33:22 -07:00
Yann Collet 3d18d9a9ce updated API manual 2024-03-18 12:30:54 -07:00
Yann Collet 902c7ec1fe add doc on CCtx UB state 2024-03-18 12:30:35 -07:00
Yann Collet 5d82c2b57c add a paragraph on UB DCtx state after error 2024-03-18 12:17:41 -07:00
Yann Collet f5728da365 update targetCBlockSize documentation 2024-03-18 12:04:02 -07:00
Yann Collet 351498b932 update CHANGELOG for v1.5.6 2024-03-14 17:01:34 -07:00
Yann Collet 686e7e4b4b updated version to v1.5.6 2024-03-14 15:38:14 -07:00
Yann Collet 9cc3304614 add line number to debug traces 2024-03-14 12:11:11 -07:00
Yann Collet c8ab027227 reduce the amount of includes in "cover.h" 2024-03-13 11:29:28 -07:00
Yann Collet ff6713fd72 fix duplicated paragraph 
reported by @zougloub
 2024-03-12 13:09:10 -07:00
Yann Collet 5473b72a05 updated documentation 
following recommendations by @zougloub at #3698
 2024-03-12 12:27:33 -07:00
Yann Collet 83ec3d0164 no longer truncate file name in verbose mode 
fix #3702
 2024-03-12 11:27:42 -07:00
Yann Collet 37ff4f91eb removed golden-decompression/.gitignore 
replaced by an exclusion rule in tests/.gitignore
 2024-03-12 10:47:27 -07:00
Yann Collet 0ae98ba215 add same .gitignore rule in golden-decompression/ 
as requested by @embg
 2024-03-12 09:47:54 -07:00
Yann Collet b39c76765b Add the zeroSeq sample 
that should have been part of #3674
 2024-03-12 09:28:25 -07:00
Yann Collet 74e856a195 add tests inspired from #2927 
centered around -T# and --fast=# arguments
 2024-03-11 17:57:57 -07:00
Yann Collet f6039f3d5f cmake refactor: move HP-UX specific logic into its own function 
reduce visual clutter of corner case configuration.
 2024-03-11 16:22:13 -07:00
Yann Collet 1362699e87 minor man page clarification 2024-03-11 12:23:37 -07:00
Yann Collet fbd9e628ae added tests 2024-03-11 12:17:34 -07:00
Yann Collet c610a01d7d fix #3719 
only disable `--rm` at end of command line parsing,
so that `-c` only disables `--rm` if it's effectively selected,
and not if it's overriden by a later `-o FILE` command.
 2024-03-11 11:38:55 -07:00
Yann Collet eb5f7a7fa2 produced golden sample for the offset==0 decoder test 
is correctly detected as corrupted by new version,
and is accepted (changed into offset==1) by older version.

updated documentation accordingly, with an hexadecimal representation.
 2024-03-09 00:33:44 -08:00
Yann Collet d2f56ba442 update documentation 2024-03-08 15:55:30 -08:00
Yann Collet a9fb8d4c41 new method to deal with offset==0 
in this new method, when an `offset==0` is detected,
it's converted into (size_t)(-1), instead of 1.

The logic is that (size_t)(-1) is effectively an extremely large positive number,
which will not pass the offset distance test at next stage (`execSequence()`).
Checked the source code, and offset is always checked (as it should),
using a formula which is not vulnerable to arithmetic overflow:
```
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart),
```

The benefit is that such a case (offset==0) is always detected as corrupted data
as opposed to relying on the checksum to detect the error.
 2024-03-08 15:26:06 -08:00
Yann Collet aed172a8fe minor: fix incorrect debug level 2024-03-08 14:29:44 -08:00
Yann Collet ad590275b4 added RISC-V emulation tests on Github CI 2024-03-07 16:54:44 -08:00
Yann Collet 007cda88ca prevent XXH64 from being autovectorized by XXH512 by default 
backport fix https://github.com/Cyan4973/xxHash/pull/924 from libxxhash
 2024-03-07 16:43:13 -08:00
Yann Collet 2abe8d63e0 fix LLU->ULL 
LLU is a correct prefix according to C99 & C11 standards (but not C90).
However, older versions of Visual Studio do not work with it.
Replace by ULL, which doesn't have this issue.

Fixes https://github.com/facebook/zstd/issues/3647
 2024-03-04 00:16:01 -08:00
Yann Collet 4fb0a77314 update -V documentation 
to answer #3727 comment
 2024-03-03 23:24:40 -08:00
Yann Collet bb4f85db42 fix version of actions/checkout 2024-03-03 18:47:08 -08:00
Yann Collet 8d31e8ec42 sizeBlockSequences() also tracks uncompressed size 
and only defines a sub-block boundary when
it believes that it is compressible.

It's effectively an optimization,
avoiding a compression cycle to reach the same conclusion.
 2024-02-26 14:31:12 -08:00
Yann Collet d23b95d21d minor refactor for clarity 
since we can ensure that nbSubBlocks>0
 2024-02-26 14:06:34 -08:00
Yann Collet 86db60752d optimization: bail out faster in presence of incompressible data 2024-02-26 13:27:59 -08:00
Yann Collet ef82b214ad nit: comment indentation 
as reported by @terrelln
 2024-02-26 13:23:59 -08:00
Yann Collet aa8592c532 minor: reformulate nbSubBlocks assignment 2024-02-26 13:21:14 -08:00
Yann Collet e0412c2062 fix extraneous semicolon ';' 
as reported by @terrelln
 2024-02-26 12:26:54 -08:00
Yann Collet 1fafd0c4ae fix minor visual static analyzer warning 
it's a false positive,
but change the code nonetheless to make it more obvious to the static analyzer.
 2024-02-25 19:45:32 -08:00
Yann Collet 038a8a906b targetCBlockSize: modified splitting strategy to generate blocks of more regular size 
notably avoiding to feature a larger first block
 2024-02-25 17:39:29 -08:00
Yann Collet 4d2bf7f0f2 removed sprintf usage from zstdcli.c 
some static analyzers flag this standard C90 function as unsafe.
 2024-02-24 23:03:40 -08:00
Yann Collet f8372191f5 reduced minimum compressed block size 
with the intention to match the transport layer size,
such as Ethernet and 4G mobile networks.
 2024-02-24 01:59:16 -08:00
Yann Collet f77f634d41 update API documentation 2024-02-24 01:28:17 -08:00
Yann Collet 4b51526412 fix partial block uncompressed 2024-02-24 01:24:58 -08:00
Yann Collet 6719794379 fixed some regressionTests 
but not all
 2024-02-23 18:48:29 -08:00
Yann Collet 0591e7eea1 minor: fix overly cautious conversion warning 2024-02-23 16:05:09 -08:00
Yann Collet 3b40100058 fix long sequences (> 64 KB) 2024-02-23 15:35:12 -08:00
Yann Collet 6b11fc436c fix issue with incompressible sections 2024-02-23 14:53:56 -08:00
Yann Collet cc4530924b speed optimized version of targetCBlockSize 
note that the size of individual compressed blocks will vary more wildly with this modification.
But it seems good enough for a first test, and fix the speed regression issue.
Further refinements can be attempted later.
 2024-02-23 14:03:26 -08:00
Yann Collet 68a232c591 benchmark more can test targetCBlockSize 2024-02-23 13:13:03 -08:00
Yann Collet b34517a440 fix cmake build 2024-02-20 20:20:40 -08:00
Yann Collet c2d3570338 fix meson datagen build 2024-02-20 18:17:28 -08:00
Yann Collet 1e240af30a fix datagen size control 2024-02-20 18:06:56 -08:00
Yann Collet 7a225c0c46 internal benchmark: can select size of generated synthetic sample 2024-02-20 15:47:09 -08:00
Yann Collet 83598aa106 datagen generates lorem ipsum by default 2024-02-20 15:24:25 -08:00
Yann Collet 7003c9905e increase word dictionary 
for higher variety of messages.
Now, level 5 compresses better than level 4 (by a hair).
 2024-02-20 13:27:36 -08:00
Yann Collet 3dbd861b7d runtime weight distribution table 
and made small words a bit more common.
 2024-02-20 12:26:37 -08:00
Yann Collet 5a1bb4a4e0 add question marks 
and (slightly) longer sentences.
 2024-02-20 00:37:21 -08:00
Yann Collet 40874d4aea enriched vocabulary again 
using real latin sentences from Cicero.

Compression ratio lower again, closer to "real" text,

now level 6 is way better than level 4.

level 5 is still lower than level 4,
but at least it's now higher than level 3.
 2024-02-20 00:30:29 -08:00
Yann Collet 1e046ce7fa increase vocabulary size 
makes compression a bit less good,
hence a bit more comparable with real text (though still too easy to compress).
level 6 is now stronger than level 4, by a hair.
However, there is still a ratio dip at level 5.
 2024-02-20 00:12:32 -08:00
Yann Collet 22574d848d fix issue 5921623844651008 
ossfuzz managed to create a scenario which triggers an `assert`.
This fixes it, by giving +1 more space for the backward search pass.
 2024-02-06 13:01:14 -08:00
Yann Collet b88c593d8f added or updated code comments 
as suggested by @terrelln,
to make the code of the optimal parser a bit more understandable.
 2024-02-05 18:32:25 -08:00
Yann Collet 0166b2ba80 modification: differentiate literal update at pos+1 
helps when litlen==1 is cheaper than litlen==0

works great on pathological arr[u32] examples
but doesn't generalize well on other files.

silesia/x-ray is amoung the most negatively affected ones.
 2024-01-31 11:20:43 -08:00
Yann Collet 4683667785 refactor optimal parser 
store stretches as intermediate solution instead of sequences.
makes it possible to link a solution to a predecessor.
 2024-01-31 02:51:46 -08:00
Yann Collet de10f56be2 improve high compression ratio for file like #3793 
this works great for 32-bit arrays,
notably the synthetic ones, with extreme regularity,
unfortunately, it's not universal,
and in some cases, it's a loss.
Crucially, on average, it's a loss on silesia.
The most negatively impacted file is x-ray.
It deserves an investigation before suggesting it as an evolution.
 2024-01-29 23:25:24 -08:00
Yann Collet fd03971252 blindfix meson recipe 
note: absence of GLOB capability within meson makes its maintenance more painful.
 2024-01-29 15:50:21 -08:00
Yann Collet befcec1788 fix cmake recipe 2024-01-29 15:45:16 -08:00
Yann Collet 3ce4c6e046 fix Visual Studio solutions 
note: we probably don't want to maintain VS2008 solution anymore.
Its successor VS2010 is > 10 years old,
which is more or less the limit after which we can stop supporting old compilers.
 2024-01-29 15:24:42 -08:00
Yann Collet a261375996 fix paramgrill Makefile recipe 2024-01-29 15:18:27 -08:00
Yann Collet d0b7da30e2 add a lorem ipsum generator 
this generator replaces the statistical generator
for the general case when no statistic is requested.

Generated data features a compression level speed / ratio curve
which is more in line with expectation.
 2024-01-29 15:00:32 -08:00
Yann Collet e1ef81a3ae add sparc64 compilation test 2024-01-28 20:25:04 -08:00
Yann Collet 2fc7248412 fix cmakebuild test 
write it in a way which is more compatible with older versions of cmake (<3.13)

Also:
fix pzstd compilation (notably on macos)
 2024-01-27 17:30:06 -08:00
Yann Collet c7611d6964 disable Intel CET Compatibility tests 
The binary blob that must be downloaded from intel.com is no longer available
 2024-01-27 15:37:29 -08:00
Yann Collet 81f444f4f9 made playTests.sh more compatible with older versions of grep 
replaced `\+` by `*`.
`\+` means `[1-N]`,
while `*` means `[0-N]`,
so it's not strictly equivalent
but `\+` happens to be badly supported on some flavors of grep,
and for the purpose of these tests, `*` is good enough.
 2024-01-16 12:14:35 -08:00
Yann Collet e6f4b46493 playTests.sh does no longer needs grep -E 
it makes the test script more portable across posix systems
because `grep -E` is not guaranteed
while `grep` is fairly common.
 2024-01-15 11:16:46 -08:00
Yann Collet 59dcc47579 update license text 2023-11-16 16:19:25 -08:00
Yann Collet 3fd5f9f52d fix the copyright linter 2023-11-13 15:50:42 -08:00
Yann Collet 592b1acb18 update xxhash to v0.8.2 
List of updates : https://github.com/Cyan4973/xxHash/releases/tag/v0.8.2

This is also a preparation task before taking care of #3819
 2023-11-13 15:42:07 -08:00
Yann Collet 24dabde507 revert to manually defining DTable 
thus avoiding the analyzer and ubsan to associate DTable to a size of 1.
 2023-10-18 22:45:57 -07:00
Yann Collet d988e00a7f baby-step towards solving flexArray issue #3785 
the flexArray in structure FSE_DecompressWksp
is just a way to derive a pointer easily,
without risk/complexity of calculating it manually.

Not sure if this change is good enough to avoid ubsan warnings though.
 2023-10-18 16:21:39 -07:00
Yann Collet 3fc14e411b added some documentation on ZSTD_estimate*Size() variants 
as a follow up for #3747
 2023-09-13 11:35:19 -07:00
Yann Collet 607933a2ff minor simplification for dependency generation 
also : fix zstd-nomt exclusion and test
 2023-09-12 13:46:03 -07:00
Yann Collet f4dbfce79c define LIB_SRCDIR and LIB_BINDIR 2023-09-12 13:46:03 -07:00
Yann Collet feaa8ac50d renamed STATLIB into STATICLIB 
for improved clarity
 2023-09-12 13:46:03 -07:00
Yann Collet 4edfaa93b7 default targets of lib/ and programs/ have different names 
avoid risks on overlapping in case of include
 2023-09-12 13:46:03 -07:00
Yann Collet b69d06a810 add include guards 
alleviate risks of double inclusion (typically via transitive includes)
 2023-09-12 13:46:03 -07:00
Yann Collet a07d7c4e29 added ZSTD_decompressDCtx() benchmark option to fullbench 
useful to compare the difference between ZSTD_decompress
and ZSTD_decompressDCtx().
 2023-08-16 10:43:39 -07:00
Yann Collet e4aeaebc20 fixed incorrect test in Win32 pthread wrapper 
reported by @Banzai24-yht in #3683
 2023-06-20 08:34:26 -07:00
Yann Collet c123e69ad0 fixed static analyzer false positive regarding @sequence initialization 
make a mock initialization to please the tool
 2023-06-16 16:24:48 -07:00
Yann Collet c60dcedcc9 adapted long decoder to new decodeSequences 
removed older decodeSequences
 2023-06-16 15:52:00 -07:00
Yann Collet 33fca19dd4 changed ZSTD_decompressSequences_bodySplitLitBuffer() decoding loop 
to behave more like the regular decoding loop.
 2023-06-16 15:32:07 -07:00
Yann Collet 84e898a76c removed _old variant from splitLit 2023-06-16 14:42:28 -07:00
Yann Collet 02134fad12 changed (partially) the decodeSequences flow logic 
this allows detecting overflow events without a checksum.
 2023-06-16 11:57:12 -07:00