ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,025 Commits 17 Branches 77 Tags
0d7626672d19744330c4a2e8fcb022b47a40b618
Commit Graph
2214 Commits
Author SHA1 Message Date
Yann Collet 0d7626672d fixed c++ conversion warning 2018-05-10 18:17:21 -07:00
Yann Collet 09d0fa29ee minor adjusting of weights 2018-05-10 18:13:48 -07:00
Yann Collet 1a26ec6e8d opt: init statistics from dictionary 
instead of starting from fake "default" statistics.
 2018-05-10 17:59:12 -07:00
Yann Collet 74b1c75d64 btopt : minor adjustment of update frequencies 2018-05-10 16:32:36 -07:00
Yann Collet ac6105463a opt: minor improvements to log traces 
slight improvement when using fractional-bit evaluation (opt:dictionay)
 2018-05-09 15:46:11 -07:00
Yann Collet c39061cb7b fixed declaration-after-statement warning 2018-05-09 12:07:25 -07:00
Yann Collet 4d5bd32a00 added traces to look at symbol costs 
evaluation looks correct.
 2018-05-09 12:00:12 -07:00
Yann Collet c0da0f5e9e switchable bit-approximation / fractional-bit accuracy modes 
also : makes it possible to select nb of fractional bits.
 2018-05-09 10:48:09 -07:00
Yann Collet ba2ad9b6b9 implemented fractional bit cost evaluation 
for FSE symbols.

While it seems to work, the gains are negligible compared to rough maxNbBits evaluation.
There are even a few losses sometimes, that still need to be explained.
Furthermode, there are still cases where btlazy2 does a better job than btopt,
which seems rather strange too.
 2018-05-08 17:43:13 -07:00
Yann Collet 1aff63b114 opt: shift all costs by 8 bits (* 256) 
making it possible to represent fractional bit costs.
 2018-05-08 16:19:04 -07:00
Yann Collet 6a3c34aa58 opt: estimate cost of both Hufman and FSE symbols 
For FSE symbols : provide an upper bound,
in nb of bits,
since cost function is not able to store fractional bit costs.
 2018-05-08 16:11:21 -07:00
Yann Collet 338f738c24 pass entropy tables to optimal parser 
for proper estimation of symbol's weights
when using dictionary compression.

Note : using only huffman costs is not good enough,
presumably because sequence symbol costs are incorrect.
 2018-05-08 15:37:06 -07:00
Yann Collet a155061328 minor code refactor for readability 
removed some useless operations from optimal parser
(should not change performance, too small a difference)
 2018-05-08 12:32:44 -07:00
Peter Seiderer 64bfdca5b9 Split library install target into pc, static, shared and include only target 
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
 2018-04-30 20:32:32 +02:00
Nick Terrell ca77822ddf Fix parameter adjustment with dictionary 
The new advanced API basically set `requestedParams = appliedParams` when
using a dictionary. This halted all parameter adjustment, which can hurt
compression ratio if, for example, the window log is small for the first
call, but the rest of the files are large.

This patch fixes the bug, and checks that the `requestedParams` don't change
in the new advanced API when using a dictionary, and generally in the fuzzer.
 2018-04-25 16:32:29 -07:00
Yann Collet 12f60b8c98 clarified documentation related to refPrefix() 2018-04-25 10:17:06 -07:00
Yann Collet ace856a835 updated documentation of streaming compression api 2018-04-24 14:44:27 -07:00
taigaconandNick Terrell 2c3ad05812 Fix the problem that enables DYNAMIC_BMI2 macro by mistake on ARM architecture with Clang (#1110) 2018-04-23 15:41:50 -07:00
Nick Terrell e8c9dc5cea Fix documentation 2018-04-13 12:43:38 -07:00
Nick Terrell c0987986e5 Only reset CDict in ZSTD_CCtx_resetParameters() 2018-04-13 11:26:40 -07:00
Nick Terrell 9f76eebd17 Add ZSTD_CCtx_resetParameters() function 
* Fix docs for `ZSTD_CCtx_reset()`.
* Add `ZSTD_CCtx_resetParameters()`.

Fixes #1094.
 2018-04-12 16:54:07 -07:00
Nick TerrellandGitHub 3c3f59e68f Enforce pledgeSrcSize whenever known (#1106) 
The test fails before the patch and passes after.

Fixes #1095.
 2018-04-12 16:02:03 -07:00
Nick Terrell 280a236e9e Add ZSTD_CCtx(Param)?_getParameter() function 
Closes #1096.
 2018-04-12 11:50:12 -07:00
Yann Collet 04212178b5 doc : clarified advanced API usage 
sticky parameters only work with `ZSTD_compress_generic()`
 2018-04-10 11:40:36 -07:00
Yann Collet ad5ba6cdcf updated comment on parameters that can be changed during compression 2018-04-09 17:39:07 -07:00
Yann ColletandGitHub 1da629f2ad Merge pull request #1104 from terrelln/fast-train 
Allow negative compression levels in training
 2018-04-09 14:16:20 -07:00
Nick Terrell 569e2abccd Allow negative compression levels in training 
* Set `dictCLevel` in `zstdcli.c`.
* Only set to default level if the compression level `== 0`, not `<= 0`.
 2018-04-09 12:12:03 -07:00
Yann ColletandGitHub 4195b36dd7 Merge pull request #1100 from bket/stable_sort 
zstd requires a stable sort.
 2018-04-05 11:39:27 -07:00
Yann Collet f35b8ba9da updated ZSTD_p_chainLog description 2018-04-05 11:05:11 -07:00
Björn Ketelaars 462aed6811 zstd requires a stable sort. 
On OpenBSD qsort() is not guaranteed to be stable, their mergesort() is.
This fixes issue #1088. All the hard work has been done by @terrelln.
 2018-04-05 07:59:16 +02:00
Yann ColletandGitHub 55f67502f4 Merge pull request #1098 from terrelln/nd-mt 
Only load extra table positions for CDicts
 2018-04-02 15:38:20 -07:00
Nick Terrell 295ab0dbfa Only load extra table positions for CDicts 
Zstdmt uses prefixes to load the overlap between segments. Loading extra
positions makes compression non-deterministic, depending on the previous
job the context was used for. Since loading extra position takes extra
time as well, only do it when creating a `ZSTD_CDict`.

Fixes #1077.
 2018-04-02 14:41:30 -07:00
Yann ColletandGitHub 5b616fa269 Merge pull request #1090 from bket/openbsd 
Fix building zstd on OpenBSD.
 2018-04-02 14:15:26 -07:00
Björn Ketelaars 9d3048346d Fix building zstd on OpenBSD. 2018-03-31 10:46:20 +02:00
Yann Collet 8be984ec45 fixed comments as suggested by @terrelln 2018-03-30 20:09:27 -07:00
Yann Collet e6e848bfe9 added ZSTD_getFrameHeader_advanced() 
makes it possible to request frame header from a magicless frame
 2018-03-29 17:51:08 -06:00
Yann Collet a6694838e1 added more code documentation for ZSTD_getFrameHeader() 2018-03-29 15:24:17 -06:00
René Rebe 21eb26d664 fixed legacy/zstd_v* with older gcc version, by guarding builtin_* 
like in other files
 2018-03-25 20:35:15 +02:00
Yann Collet ad15c1b724 added __has_attribute() define for non-clang compilers 2018-03-23 19:04:48 -07:00
Yann Collet 52ca7c6c56 make DYNAMIC_BMI2 support of clang conditional to __has_attribute() 
to support older clang versions such as 3.4
 2018-03-23 18:45:42 -07:00
Yann ColletandGitHub 29b021f9a0 Merge pull request #1067 from facebook/targetLength 
removed limit ZSTD_TARGETLENGTH_MAX
 2018-03-22 10:38:33 -07:00
Nick Terrell ad344033df Fix broken assertion 
The `avgJobSize` must not be lower than 256 KB for single-pass mode.
In `zstd.h` we say the minimum value for `ZSTD_p_jobSize` is 1 MB,
so ensure that we always pick a size >= 1 MB.

Found by libFuzzer fuzzer tests with large input limits.
 2018-03-21 16:20:30 -07:00
Yann Collet 153bc1c004 removed limit ZSTD_TARGETLENGTH_MAX 
this makes it possible to specify extremely large negative compression levels,
achieving the side effect as "no compression".

It will also be possible to define larger targetlength for ultra compression mode.

There is no adverse side effect due to removing this limit.
 2018-03-21 15:50:05 -07:00
Yann Collet a99c4a3621 Merge branch 'dev' into advancedDecompress 2018-03-21 06:08:28 -07:00
Yann ColletandGitHub 87b0cf05bd Merge pull request #1057 from facebook/lrmSettings 
LRM parameters
 2018-03-21 05:59:39 -07:00
Yann ColletandGitHub d1bf609abf Merge pull request #1059 from terrelln/mt-ldm 
Integrate ldm with zstdmt
 2018-03-20 17:50:20 -07:00
Yann Collet e0cb8d19c6 fixed legacy test case 2018-03-20 17:48:22 -07:00
Yann Collet 878728dc26 fixed several comments by @terrelln 2018-03-20 16:35:14 -07:00
Yann ColletandGitHub e1c52faace Merge pull request #1060 from facebook/compressImpl 
merge bmi2 implementation of encodeSequence into zstd_compress.c
 2018-03-20 16:19:42 -07:00
Yann Collet 6cda8c932c added test with ZSTD_decompress_generic() + ZSTD_DCtx_refPrefix() 
also :
clarified stage condition to accept new parameters,
fixed initializers correspondingly.
 2018-03-20 16:16:13 -07:00