ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,104 Commits 17 Branches 77 Tags
ef984e730775da20b47ff8e2837ccac32b95c53f
Commit Graph
2678 Commits
Author SHA1 Message Date
Yann Collet ef984e7307 fix debug levels 
as reported by @terrelln.
2 is reserved for temporary usage only.
 2018-12-18 13:40:07 -08:00
Yann Collet 635783da12 btultra2 and very small srcSize 
When srcSize is small,
the nb of symbols produced is likely too small to warrant dedicated probability tables.
In which case, predefined distribution tables will be used instead.

There is a cheap algorithm in btultra initialization :
it presumes default distribution will be used if srcSize <= 1024.

btultra2 now uses the same threshold to shut down probability estimation,
since measured frequencies won't be used at entropy stage,
and therefore relying on them to determine sequence cost is misleading,
resulting in worse compression ratios.

This fixes btultra2 performance issue on very small input.

Note that, a proper way should be
to determine which symbol is going to use predefined probaility
and which symbol is going to use dynamic ones.
But the current algorithm is unable to make a "per-symbol" decision.
So this will require significant modifications.
 2018-12-18 12:32:58 -08:00
Yann Collet 373ff8b983 play around with rescale weights 2018-12-17 15:48:34 -08:00
Nick TerrellandGitHub 75fa3f2eb7 Merge pull request #1446 from terrelln/overflow 
[libzstd] Fix infinite loop in decompression
 2018-12-13 16:21:15 -08:00
Nick Terrell aaea4ef924 [libzstd] Fix infinite loop in decompression 
When we switched `ZSTD_SKIPPABLEHEADERSIZE` to a macro, the places where we do:

    MEM_readLE32(ptr) + ZSTD_SKIPPABLEHEADERSIZE

can now overflow `(unsigned)-8` to `0` and we infinite loop. We now check
the frame size and reject sizes that overflow a U32.

Note that this bug never made it into a release, and was only in the dev branch
for a few days.

Credit to OSS-Fuzz
 2018-12-13 15:13:19 -08:00
Yann Collet 9a92ed401d updated compression results.csv 
and fixed nit
 2018-12-12 20:30:09 -08:00
Yann Collet 9792acda3b Merge branch 'dev' into btultra2 2018-12-12 20:18:27 -08:00
Yann Collet 52b94f902c add clarification for ZSTD_CCtx_setPledgedSrcSize() 
as requested in #1391
 2018-12-11 12:08:21 -08:00
Yann ColletandGitHub 9c3265a53f Merge pull request #1417 from facebook/advancedAPI 
Advanced API
 2018-12-10 18:48:15 -08:00
Yann Collet 5e6aaa3abb fixed btultra2 usage with prefix 
notably while using multi-threading
 2018-12-10 18:45:03 -08:00
Yann Collet 3619c34399 fix assert position within ZSTD_compress2() 2018-12-10 17:42:35 -08:00
Yann Collet 5a1e01e5f1 clarified experimentalParam 2018-12-10 17:36:20 -08:00
Yann Collet c226a7b9f3 fixed ZSTD_compress2() 
as suggested by @terrelln
 2018-12-10 17:33:49 -08:00
Yann Collet 37e314a68d updated clevel table for large inputs 2018-12-09 22:38:05 -08:00
Yann Collet c9c4c7ec8c update clevel table for 256K 2018-12-08 21:40:08 -08:00
Yann Collet 8075d75f9c update clevel table for 128K 2018-12-08 10:42:55 -08:00
Yann Collet 95b152ab33 updated clevel table for 16K 
to introduce btultra2
 2018-12-07 20:12:43 -08:00
Yann Collet d613fd9afe linked btultra2 as strategy9 
and ensure zstdbench detects out-of-bound parameters
 2018-12-06 19:27:37 -08:00
Yann Collet 34aa401afd updated documentation 
introducing ZSTD_btultra2
 2018-12-06 17:22:19 -08:00
Yann Collet ae370b0e12 minor bound refinements 2018-12-06 16:51:17 -08:00
Yann Collet 39e28982cf introduced constants ZSTD_STRATEGY_MIN and ZSTD_STRATEGY_MAX 2018-12-06 16:16:16 -08:00
Yann Collet c3c3488981 fixed c++ assignment to enum 2018-12-06 15:57:55 -08:00
Yann Collet be9e561da4 changed ZSTD_c_compressionStrategy into ZSTD_c_strategy 
also : fixed paramgrill, and limit conditions
 2018-12-06 15:00:52 -08:00
Yann Collet e9448cdf4c introduced strategy btultra2 
note : not yet applied on any compression level
 2018-12-06 13:38:09 -08:00
Yann Collet 0c404a48f0 moved ZSTD_WINDOWLOG_LIMIT_DEFAULT into static-linking-only area 2018-12-06 10:57:19 -08:00
Yann Collet 96d887429b clarified usage of word "job" 
only applies in MT / async context now.
 2018-12-06 10:14:34 -08:00
Yann Collet 3583d19c4e changed parameter names from ZSTD_p_* to ZSTD_c_* 
for naming consistency
 2018-12-05 17:26:02 -08:00
Yann Collet c2053310e5 updated API documentation 2018-12-05 16:23:00 -08:00
Yann Collet 3e042d5cc0 ZSTD_decompressDCtx() is compatible with sticky parameters 2018-12-04 17:30:58 -08:00
Yann Collet d7da3fc90a merge dedicated dParam setters 2018-12-04 17:06:48 -08:00
Yann Collet 4b5a4f02d7 write the switch()case: differently 
so that it please both
compilers which warn for dead code after the switch
and
compilers which do not detect that all branches terminate.
 2018-12-04 16:59:26 -08:00
Yann Collet 85b02bf142 fixed silent conversion warning 2018-12-04 15:57:16 -08:00
Yann Collet aec945f0dc implemented ZSTD_dParam_getBounds() 
and ZSTD_DCtx_setParameter()
 2018-12-04 15:35:37 -08:00
Yann Collet 34e146f548 advanced decompression function replaces by normal streaming one 
advanced parameters compatible with ZSTD_decompressStream().
 2018-12-04 10:28:36 -08:00
Yann Collet 7ef7dc561a check availability of --color=never command on grep and egrep 
before applying them.
Fixes #1436
 2018-12-03 15:46:55 -08:00
Yann Collet 6ced8f7c7c joined normal streaming API with advanced one 2018-12-03 14:22:38 -08:00
Yann Collet da1f3066a3 preparative for ZSTD_DCtx_setParameter() 2018-11-30 15:59:50 -08:00
Yann Collet d8e215cbee created ZSTD_compress2() and ZSTD_compressStream2() 
ZSTD_compress_generic() is renamed ZSTD_compressStream2().

Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
 2018-11-30 11:25:56 -08:00
Mitchell Grenier a424899637 Fix buck for lib 2018-11-30 13:45:16 +00:00
Yann Collet d3a0c71259 pushed experimental parameters 
into ZSTD_STATIC_LINKING_ONLY section
 2018-11-21 16:18:55 -08:00
Yann Collet d4d4e109e9 getParameter fills an int* 
rather than an unsigned*
for consistency
since type of setParameter() changed to int.
 2018-11-21 15:37:26 -08:00
Yann Collet fea920615c promote ZSTD_findFrameCompressedSize() into staging area 2018-11-21 15:25:50 -08:00
Yann Collet 41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet 5d3592398d fixed fall-through 2018-11-20 16:09:33 -08:00
Yann Collet 5c6d4b18ac completed implementation of ZSTD_cParam_getBounds() 
for all parameters
 2018-11-20 16:06:00 -08:00
Yann Collet 2e7fd6a2cb fixed remaining searchLength invocations 2018-11-20 15:13:27 -08:00
Yann Collet e874dacc08 changed searchLength into minMatch 
refactored all relevant API and calls
for consistency.
 2018-11-20 14:56:07 -08:00
Yann Collet 114bd4346e changed enum type name to ZSTD_ResetDirective 
for naming consistency :
types should start with a capital letter (after prefix)
 2018-11-20 12:00:20 -08:00
Yann Collet 3b838abf97 ZSTD_CCtx_setParameter : value argument is now int 
for compatibility with compression level
 2018-11-20 11:53:01 -08:00
Yann Collet 19e5f2a35b removed some constants and _simpleArgs() from staging 
constants that *may* change in the future
will be accessed through functions instead
(to be created).

_simpleArgs() variant do not have (yet) a clear enough added value
to deserve "stable" status.
 2018-11-19 17:38:15 -08:00