ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,069 Commits 17 Branches 77 Tags
73f4c890cd07dbc6e0f24f7399456a060a5a7105
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet ade583948d Merge branch 'tableLevels' of github.com:facebook/zstd into tableLevels 2018-05-18 18:23:40 -07:00
Yann Collet 5381369cb1 Merge branch 'dev' into tableLevels 2018-05-18 18:23:27 -07:00
Yann Collet 38c2c46823 Merge branch 'dev' into tableLevels 2018-05-18 17:17:45 -07:00
Yann Collet b0b3fb517d updated compression levels for blocks of 256KB 2018-05-18 17:17:12 -07:00
Yann Collet 63eeeaa1dd update table levels for blocks <= 16K 
also : allow hlog to be slighly larger than windowlog,
as it's apparently good for both speed and compression ratio.
 2018-05-16 16:13:37 -07:00
Yann Collet d59cf02df0 decompress: changed error code when input is too large 
ZSTD_decompress() can decompress multiple frames sent as a single input.
But the input size must be the exact sum of all compressed frames, no more.

In the case of a mistake on srcSize, being larger than required,
ZSTD_decompress() will try to decompress a new frame after current one, and fail.
As a consequence, it will issue an error code, ERROR(prefix_unknown).

While the error is technically correct
(the decoder could not recognise the header of _next_ frame),
it's confusing, as users will believe that the first header of the first frame is wrong,
which is not the case (it's correct).
It makes it more difficult to understand that the error is in the source size, which is too large.

This patch changes the error code provided in such a scenario.
If (at least) a first frame was successfully decoded,
and then following bytes are garbage values,
the decoder assumes the provided input size is wrong (too large),
and issue the error code ERROR(srcSize_wrong).
 2018-05-14 15:32:28 -07:00
Yann Collet c8c67f7c84 Merge branch 'dev' into tableLevels 2018-05-14 11:55:52 -07:00
Yann Collet 3870db1ba5 Merge branch 'dev' into tableLevels 2018-05-14 11:52:05 -07:00
Yann Collet 2c392952f9 paramgrill: use NB_LEVELS_TRACKED in loop 
make it easier to generate/track more levels
than ZSTD_maxClevel()
 2018-05-13 17:25:53 -07:00
Yann Collet c9227ee16b update table for 128 KB blocks 2018-05-13 17:15:07 -07:00
Yann Collet b4250489cf update compression levels for large inputs 2018-05-13 01:53:38 -07:00
Yann Collet 9cd5c63771 cli: control numeric argument overflow 
exit on overflow
backported from paramgrill
added associated test case
 2018-05-12 14:29:33 -07:00
Yann Collet 3f89cd1081 minor : factor out errorOut() 2018-05-12 14:09:32 -07:00
Yann Collet b824d213cb fix #1115 2018-05-12 10:21:30 -07:00
Yann Collet 50993901b2 paramgrill: subtle change in level spacing 
distance between levels is slightly increased
to compensate for level 1 speed improvements
and the will to have stronger level 19
extending the range of speed to cover.
 2018-05-12 09:40:04 -07:00
Yann Collet a3f2e84a37 added programmable constraints 2018-05-11 19:43:08 -07:00
Yann Collet 17c19fbbb5 generalized use of readU32FromChar() 
and check input overflow
 2018-05-11 17:32:26 -07:00
Yann Collet 761758982e replaced FSE_count by FSE_count_simple 
to reduce usage of stack memory.

Also : tweaked a few comments, as suggested by @terrelln
 2018-05-11 16:03:37 -07:00
Yann Collet 3193d692c2 minor patch, ensuring LIBDIR is created before installation 
follow-up from #1123
 2018-05-11 11:31:48 -07:00
Yann Collet 99ddca43a6 fixed wrong assertion 
base can actually overflow
 2018-05-10 19:48:09 -07:00
Yann Collet 0d7626672d fixed c++ conversion warning 2018-05-10 18:17:21 -07:00
Yann Collet 09d0fa29ee minor adjusting of weights 2018-05-10 18:13:48 -07:00
Yann Collet 1a26ec6e8d opt: init statistics from dictionary 
instead of starting from fake "default" statistics.
 2018-05-10 17:59:12 -07:00
Yann Collet 74b1c75d64 btopt : minor adjustment of update frequencies 2018-05-10 16:32:36 -07:00
Yann Collet ac6105463a opt: minor improvements to log traces 
slight improvement when using fractional-bit evaluation (opt:dictionay)
 2018-05-09 15:46:11 -07:00
Yann Collet c39061cb7b fixed declaration-after-statement warning 2018-05-09 12:07:25 -07:00
Yann Collet 4d5bd32a00 added traces to look at symbol costs 
evaluation looks correct.
 2018-05-09 12:00:12 -07:00
Yann Collet c0da0f5e9e switchable bit-approximation / fractional-bit accuracy modes 
also : makes it possible to select nb of fractional bits.
 2018-05-09 10:48:09 -07:00
Yann Collet ba2ad9b6b9 implemented fractional bit cost evaluation 
for FSE symbols.

While it seems to work, the gains are negligible compared to rough maxNbBits evaluation.
There are even a few losses sometimes, that still need to be explained.
Furthermode, there are still cases where btlazy2 does a better job than btopt,
which seems rather strange too.
 2018-05-08 17:43:13 -07:00
Yann Collet 1aff63b114 opt: shift all costs by 8 bits (* 256) 
making it possible to represent fractional bit costs.
 2018-05-08 16:19:04 -07:00
Yann Collet 6a3c34aa58 opt: estimate cost of both Hufman and FSE symbols 
For FSE symbols : provide an upper bound,
in nb of bits,
since cost function is not able to store fractional bit costs.
 2018-05-08 16:11:21 -07:00
Yann Collet 338f738c24 pass entropy tables to optimal parser 
for proper estimation of symbol's weights
when using dictionary compression.

Note : using only huffman costs is not good enough,
presumably because sequence symbol costs are incorrect.
 2018-05-08 15:37:06 -07:00
Yann Collet a155061328 minor code refactor for readability 
removed some useless operations from optimal parser
(should not change performance, too small a difference)
 2018-05-08 12:32:44 -07:00
Yann Collet ad4524d605 fix ZSTD_compressBlock() associated with CDict 
reported by @let-def.

It's actually a bug in ZSTD_compressBegin_usingCDict()
which would pass a wrong pledgedSrcSize value (0 instead of ZSTD_CONTENTSIZE_UNKNOWN)
resulting in wrong window size, resulting in downsized seqStore,
resulting in segfault when writing into the seqStore later in the process.

Added a test in fuzzer to cover this use case (fails before the patch).
 2018-05-07 12:54:13 -07:00
Yann Collet 82ad249645 Clarifications of Zstandard format specification 
from IETF RFC review
 2018-04-30 12:36:55 -07:00
Yann Collet 12f60b8c98 clarified documentation related to refPrefix() 2018-04-25 10:17:06 -07:00
Yann Collet ace856a835 updated documentation of streaming compression api 2018-04-24 14:44:27 -07:00
Yann Collet 04212178b5 doc : clarified advanced API usage 
sticky parameters only work with `ZSTD_compress_generic()`
 2018-04-10 11:40:36 -07:00
Yann Collet ad5ba6cdcf updated comment on parameters that can be changed during compression 2018-04-09 17:39:07 -07:00
Yann Collet f35b8ba9da updated ZSTD_p_chainLog description 2018-04-05 11:05:11 -07:00
Yann Collet 8be984ec45 fixed comments as suggested by @terrelln 2018-03-30 20:09:27 -07:00
Yann Collet e6e848bfe9 added ZSTD_getFrameHeader_advanced() 
makes it possible to request frame header from a magicless frame
 2018-03-29 17:51:08 -06:00
Yann Collet a6694838e1 added more code documentation for ZSTD_getFrameHeader() 2018-03-29 15:24:17 -06:00
Yann Collet 96b480a96d added ovlog tests 
check that setting ovlog has a direct impact on compression ratio
(hence is not silently ignored)
 2018-03-29 13:08:09 -06:00
Yann Collet 525f3fab33 restored ability to manually set overlapLog 2018-03-28 11:33:41 -06:00
Yann Collet eba13dba2e updated NEWS for v1.3.4 2018-03-26 10:23:46 -07:00
Yann Collet bffb4b46a1 updated VS studio readme 2018-03-26 09:55:10 -07:00
Yann Collet 4d753b0ca8 Merge branch 'dev' into srcPackage 2018-03-24 09:55:13 -07:00
Yann Collet ad15c1b724 added __has_attribute() define for non-clang compilers 2018-03-23 19:04:48 -07:00
Yann Collet 52ca7c6c56 make DYNAMIC_BMI2 support of clang conditional to __has_attribute() 
to support older clang versions such as 3.4
 2018-03-23 18:45:42 -07:00
Yann Collet 6a8d69c427 Merge branch 'dev' of github.com:facebook/zstd into dev 2018-03-23 18:39:53 -07:00
Yann Collet f991897bb1 Revert "create .tar.zst source package" 
This reverts commit 929608146a.
 2018-03-23 18:39:42 -07:00
Yann Collet 6356b007fb make artifacts ande extended tests only on master 2018-03-23 17:57:37 -07:00
Yann Collet d9b2cbf650 restored certutil command line 2018-03-23 17:57:11 -07:00
Yann Collet 956a3b7006 clean certutil output 2018-03-23 17:52:16 -07:00
Yann Collet 2489e71ff1 added sha256 sig 2018-03-23 17:40:53 -07:00
Yann Collet fa294b8590 created zstd archive from git archive 2018-03-23 17:20:05 -07:00
Yann Collet 308f405c73 export zstd/master from github using svn 2018-03-23 17:06:11 -07:00
Yann Collet b42ffb91f7 git clone master branch (hardcoded) 2018-03-23 16:52:32 -07:00
Yann Collet 17e937bcd0 git clone for clean src tar 2018-03-23 16:49:34 -07:00
Yann Collet a59dac279a added tar command 2018-03-23 16:36:18 -07:00
Yann Collet a75104d8f8 added path print 2018-03-23 16:27:28 -07:00
Yann Collet 97bc1aec47 removed src artifact lines 2018-03-23 16:24:54 -07:00
Yann Collet b94d08c566 commented src archive lines 2018-03-23 16:23:23 -07:00
Yann Collet b1f898ec82 added srcPackage to list of branch producing artifacts 2018-03-23 16:15:56 -07:00
Yann Collet 929608146a create .tar.zst source package 2018-03-23 15:00:04 -07:00
Yann Collet d6c6b62958 updated graph for dictionary compression 2018-03-22 18:44:56 -07:00
Yann Collet 184e1c8b8c updated benchmark figures 2018-03-22 18:22:10 -07:00
Yann Collet 01082a39bd restored simple status line during zstd compression 
the more advanced one, featuring amount of data buffered,
is triggered on `-v`.
 2018-03-22 17:49:46 -07:00
Yann Collet f15e2fd55e fixed fuzz test 2018-03-21 16:45:15 -07:00
Yann Collet b7b80e1569 fixed paramgrill 2018-03-21 16:09:15 -07:00
Yann Collet 93b8262c02 added large negative level test case 2018-03-21 15:54:44 -07:00
Yann Collet 153bc1c004 removed limit ZSTD_TARGETLENGTH_MAX 
this makes it possible to specify extremely large negative compression levels,
achieving the side effect as "no compression".

It will also be possible to define larger targetlength for ultra compression mode.

There is no adverse side effect due to removing this limit.
 2018-03-21 15:50:05 -07:00
Yann Collet a99c4a3621 Merge branch 'dev' into advancedDecompress 2018-03-21 06:08:28 -07:00
Yann Collet 453d487807 fixed minor declaration warning 2018-03-20 18:03:56 -07:00
Yann Collet e0cb8d19c6 fixed legacy test case 2018-03-20 17:48:22 -07:00
Yann Collet 878728dc26 fixed several comments by @terrelln 2018-03-20 16:35:14 -07:00
Yann Collet b18cb7e0b7 zstreamtest : slightly decreased time to build dictionary size 2018-03-20 16:18:48 -07:00
Yann Collet 6cda8c932c added test with ZSTD_decompress_generic() + ZSTD_DCtx_refPrefix() 
also :
clarified stage condition to accept new parameters,
fixed initializers correspondingly.
 2018-03-20 16:16:13 -07:00
Yann Collet 2af41592ea added a test with ZSTD_decompress_generic() + ZSTD_DCtx_loadDictionary_byReference() 2018-03-20 15:59:25 -07:00
Yann Collet 0dadb6b70d implemented ZSTD_DCtx_refPrefix*() 2018-03-20 15:45:56 -07:00
Yann Collet 569b8ba4d9 implemented ZSTD_DCtx_refDDict() 2018-03-20 15:43:49 -07:00
Yann Collet 6873fec658 changed dictMore for dictContentType 
which seems clearer to describe what the variable/argument is about.
 2018-03-20 15:13:14 -07:00
Yann Collet 31b54b6eea updated ZSTD_initStaticDDict() prototype 
can also specify dictContentType.
 2018-03-20 14:52:02 -07:00
Yann Collet 353117c5d7 implemented ZSTD_DCtx_loadDictionary*() 
this required updating ZSTD_createDDict_advanced()
to accept a dictContentType parameter (raw, full, auto).
 2018-03-20 13:40:29 -07:00
Yann Collet 2ed5af0766 merge bmi2 implementation of encodeSequence into zstd_compress.c 2018-03-19 19:10:31 -07:00
Yann Collet 9e6ba88363 moved snap to /contrib 2018-03-19 16:15:06 -07:00
Yann Collet a7b532a347 added docker readme 2018-03-19 16:13:12 -07:00
Yann Collet e5ddfaede8 add a test for unordered parameters 2018-03-19 16:02:51 -07:00
Yann Collet c8b3d389fd updated CCtxParams API 
to respect naming convention :
ZSTD_CCtxParams_*()
 2018-03-19 15:07:26 -07:00
Yann Collet 6f4d0778a5 make it possible to express compression parameters in any order 2018-03-19 14:41:23 -07:00
Yann Collet 9618c0c804 make it possible to specify LDM parameters in any order 2018-03-19 11:07:04 -07:00
Yann Collet cbc71e40f6 moving LRM parameters out of experimental section 
into "normal" range,
start pinned at 160.
 2018-03-15 17:22:40 -07:00
Yann Collet 5373e44ba7 fixed contrib/adaptive-compression 2018-03-15 17:10:15 -07:00
Yann Collet 97816400ca added /contrib projects to make all 2018-03-15 16:40:14 -07:00
Yann Collet 355cb645bf fixed seekable format example 2018-03-15 16:29:28 -07:00
Yann Collet 38cbcb5f1a removed LRM exploratory experiment 2018-03-15 16:26:08 -07:00
Yann Collet a909c293c6 Merge branch 'dev' into hufCompress 2018-03-14 16:11:25 -07:00
Yann Collet 50f763ec44 fixed several comments are underlined by @terrelln 2018-03-13 14:23:14 -07:00
Yann Collet a95a88af57 removed huf_compress_impl.h 
re-imported all functions inside huf_compress.c
for easier source editing.

Also updated a bunch of code comments
for clarification.
 2018-03-13 14:14:05 -07:00