Yann Collet
a7fdceeccd
changed dynamic fse threshold for offset
...
recent experienced showed that
default distribution table for offset
can get it wrong pretty quickly with the nb of symbols,
while it remains a reasonable choice much longer for lengths symbols.
Changed the formula,
so that dynamic threshold is now 32 symbols for offsets.
It remains at 64 symbols for lengths.
Detection based on defaultNormLog
2018-05-25 17:41:16 -07:00
Yann Collet and GitHub
744f36aac4
Merge pull request #1141 from facebook/staticDictCost
...
Random stuff on High Compression mode
2018-05-25 15:43:32 -07:00
Yann Collet
e2c0e3d437
slightly nudge choices towards less sequences
...
also slightly improve some strange detrimental corner cases.
2018-05-25 14:52:21 -07:00
Yann Collet
f6ad59ab5c
Merge branch 'dev' into staticDictCost
2018-05-24 16:21:02 -07:00
Yann Collet and GitHub
a7c75740b3
Merge pull request #1149 from terrelln/fuzz-py
...
Small fixes to fuzz.py
2018-05-24 15:56:09 -07:00
Nick Terrell
fdd4d8510f
Improve compiler detection to work on Mac
2018-05-24 14:21:12 -07:00
Nick Terrell
ac852abb8b
Define BIT_DEBUG for --debug
2018-05-24 14:21:12 -07:00
Nick Terrell
2a9975f77b
Increase the maximum file size
2018-05-24 14:21:12 -07:00
Nick Terrell
e712a3a0a3
Small fixes to fuzz.py
2018-05-24 14:21:12 -07:00
Yann Collet and GitHub
47d2331ed8
Merge pull request #1150 from facebook/fracFse
...
fix corner case when requiring cost of an FSE symbol
2018-05-24 14:19:30 -07:00
Yann Collet
b5ef32fea7
Merge branch 'dev' into fracFse
2018-05-24 14:09:49 -07:00
Yann Collet
776128d16f
fix corner case when requiring cost of an FSE symbol
...
ensure that, when frequency[symbol]==0,
result is (tableLog + 1) bits
with both upper-bit and fractional-bit estimates.
Also : enable BIT_DEBUG in /tests
2018-05-24 13:59:11 -07:00
Yann Collet and GitHub
08c5be5db3
Merge pull request #1117 from felixhandte/zstd-fast-in-place-dict
...
ZSTD_fast: Support Searching the Dictionary Context In-Place
2018-05-23 19:32:25 -07:00
Nick Terrell and GitHub
06b70179da
Work around bug in zstd decoder ( #1147 )
...
Work around bug in zstd decoder
Pull request #1144 exercised a new path in the zstd decoder that proved to
be buggy. Avoid the extremely rare bug by emitting an uncompressed block.
2018-05-23 18:02:30 -07:00
Yann Collet and GitHub
712318a244
Merge pull request #1146 from terrelln/fse-fix
...
[zstd] Fix decompression edge case
2018-05-23 16:41:42 -07:00
Nick Terrell
f2d0924b87
Variable declarations
2018-05-23 14:58:58 -07:00
W. Felix Handte
d9c7e67125
Assert that Dict and Current Window are Adjacent in Index Space
2018-05-23 17:53:03 -04:00
W. Felix Handte
298d24fa57
Make loadedDictEnd an Index, not the Dict Len
2018-05-23 17:53:03 -04:00
W. Felix Handte
7ef85e0618
Fixes in re Comments
2018-05-23 17:53:03 -04:00
W. Felix Handte
582b7f85ed
Don't Attach Empty Dict Contents
...
In weird corner cases, they produce unexpected results...
2018-05-23 17:53:03 -04:00
W. Felix Handte
9c92223468
Avoid Undefined Behavior in Match Ptr Calculation
2018-05-23 17:53:03 -04:00
W. Felix Handte
a44ab3b475
Remove Out-of-Date Comment
2018-05-23 17:53:03 -04:00
W. Felix Handte
95bdf20a87
Moar Renames
2018-05-23 17:53:03 -04:00
W. Felix Handte
7e0402e738
Also Attach Dict When Source Size is Unknown
2018-05-23 17:53:03 -04:00
W. Felix Handte
3ba70cc759
Clear the Dictionary When Sliding the Window
2018-05-23 17:53:03 -04:00
W. Felix Handte
b05ae9b608
Refine ip Initialization to Avoid ARM Weirdness
2018-05-23 17:53:03 -04:00
W. Felix Handte
1a7b34ef28
Use New Index Invariant to Simplify Conditionals
2018-05-23 17:53:03 -04:00
W. Felix Handte
2d598e6fed
Force Working Context Indices Greater than Dict Indices
2018-05-23 17:53:03 -04:00
W. Felix Handte
d005e5daf4
Whitespace Fix
2018-05-23 17:53:03 -04:00
W. Felix Handte
154eb09419
Switch to Original Match Calc for noDict Repcode Check
2018-05-23 17:53:03 -04:00
W. Felix Handte
191fc74a51
Rename 'hasDict' to 'dictMode'
2018-05-23 17:53:03 -04:00
W. Felix Handte
ae4fcf7816
Respond to PR Comments; Formatting/Style/Lint Fixes
2018-05-23 17:53:03 -04:00
W. Felix Handte
ca26cecc7a
Rename and Reformat
2018-05-23 17:53:03 -04:00
W. Felix Handte
66bc1ca641
Change Cut-Off to 8 KB
2018-05-23 17:53:03 -04:00
W. Felix Handte
c31ee3c7f8
Fix Rep Code Initialization
2018-05-23 17:53:03 -04:00
W. Felix Handte
b67196f30d
Coalesce hasDictMatchState and extDict Checks into One Enum and Rename Stuff
2018-05-23 17:53:03 -04:00
W. Felix Handte
265c2869d1
Split Wrapper Functions to Cause Inlining
2018-05-23 17:53:03 -04:00
W. Felix Handte
6929964d65
Add bounds check in repcode tests
2018-05-23 17:53:03 -04:00
W. Felix Handte
70a537d1d7
Initial Repcode Check Support for Ext Dict Ctx
2018-05-23 17:53:03 -04:00
W. Felix Handte
8d24ff0353
Preliminary Support in ZSTD_compressBlock_fast_generic() for Ext Dict Ctx
2018-05-23 17:53:03 -04:00
W. Felix Handte
d18a405779
Refer to the Dictionary Match State In-Place (Sometimes)
2018-05-23 17:53:03 -04:00
Nick Terrell
c92dd11940
Error if reported size is too large in edge case
2018-05-23 14:47:20 -07:00
Nick Terrell
a97e9a627a
[zstd] Fix decompression edge case
...
This edge case is only possible with the new optimal encoding selector,
since before zstd would always choose `set_basic` for small numbers of
sequences.
Fix `FSE_readNCount()` to support buffers < 4 bytes.
Credit to OSS-Fuzz
2018-05-23 12:16:00 -07:00
Yann Collet and GitHub
27dc078aa6
Merge pull request #1144 from terrelln/fse-entropy
...
Approximate FSE encoding costs for selection
2018-05-22 19:25:37 -07:00
Yann Collet and GitHub
4a498f03dc
Merge pull request #1145 from terrelln/spec
...
Clarify what happens when Number_of_Sequences == 0
2018-05-22 16:21:40 -07:00
Nick Terrell
73f4c890cd
Clarify what happens when Number_of_Sequences == 0
2018-05-22 16:12:33 -07:00
Nick Terrell
e3959d5eba
Fixes
2018-05-22 16:06:33 -07:00
Yann Collet
7a8b3496b4
Merge branch 'dev' into staticDictCost
2018-05-22 15:10:05 -07:00
Yann Collet
a8ddf1d370
disable 2-passes strategy
2018-05-22 15:06:36 -07:00
Nick Terrell
49cf880513
Approximate FSE encoding costs for selection
...
Estimate the cost for using FSE modes `set_basic`, `set_compressed`, and
`set_repeat`, and select the one with the lowest cost.
* The cost of `set_basic` is computed using the cross-entropy cost
function `ZSTD_crossEntropyCost()`, using the normalized default count
and the count.
* The cost of `set_repeat` is computed using `FSE_bitCost()`. We check the
previous table to see if it is able to represent the distribution.
* The cost of `set_compressed` is computed with the entropy cost function
`ZSTD_entropyCost()`, together with the cost of writing the normalized
count `ZSTD_NCountCost()`.
2018-05-22 14:33:22 -07:00
Yann Collet