ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,050 Commits 17 Branches 77 Tags
0b31304c8de8904c0628ad2e10c85026dcb9c17d
Commit Graph
5050 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 0b31304c8d Merge branch 'dev' into staticDictCost 2018-05-14 18:09:26 -07:00
Yann Collet 2c26df0e13 opt: removed static prices 
after testing, it's actually always better to use dynamic prices
albeit initialised from dictionary.
 2018-05-14 18:04:08 -07:00
Yann ColletandGitHub f372ffc64d Merge pull request #1127 from facebook/staticDictCost 
Improved optimal parser with dictionary
 2018-05-14 17:45:50 -07:00
Yann ColletandGitHub 174bd3d4a7 Merge pull request #1131 from facebook/zstdcli 
minor: control numeric argument overflow
 2018-05-14 11:53:58 -07:00
Yann ColletandGitHub 5d76201fee Merge pull request #1130 from facebook/man 
fix #1115
 2018-05-14 11:52:53 -07:00
Yann ColletandGitHub 902db38798 Merge pull request #1129 from facebook/paramgrill 
Paramgrill refactoring
 2018-05-14 11:52:41 -07:00
Yann ColletandGitHub 4da0216db0 Merge pull request #1133 from felixhandte/travis-fix 
Make Travis CI Run `apt-get update`
 2018-05-14 09:59:43 -07:00
W. Felix Handte e26be5a7b3 Travis CI Runs apt-get Update 2018-05-14 11:55:21 -04:00
Yann Collet 2c392952f9 paramgrill: use NB_LEVELS_TRACKED in loop 
make it easier to generate/track more levels
than ZSTD_maxClevel()
 2018-05-13 17:25:53 -07:00
Yann Collet 9cd5c63771 cli: control numeric argument overflow 
exit on overflow
backported from paramgrill
added associated test case
 2018-05-12 14:29:33 -07:00
Yann Collet 3f89cd1081 minor : factor out errorOut() 2018-05-12 14:09:32 -07:00
Yann Collet b824d213cb fix #1115 2018-05-12 10:21:30 -07:00
Yann Collet 50993901b2 paramgrill: subtle change in level spacing 
distance between levels is slightly increased
to compensate for level 1 speed improvements
and the will to have stronger level 19
extending the range of speed to cover.
 2018-05-12 09:40:04 -07:00
Yann Collet a3f2e84a37 added programmable constraints 2018-05-11 19:43:08 -07:00
Yann Collet 17c19fbbb5 generalized use of readU32FromChar() 
and check input overflow
 2018-05-11 17:32:26 -07:00
Yann Collet 761758982e replaced FSE_count by FSE_count_simple 
to reduce usage of stack memory.

Also : tweaked a few comments, as suggested by @terrelln
 2018-05-11 16:03:37 -07:00
Yann ColletandGitHub 66b81817b5 Merge pull request #1128 from facebook/libdir 
minor Makefile patch
 2018-05-11 11:47:59 -07:00
Yann Collet 3193d692c2 minor patch, ensuring LIBDIR is created before installation 
follow-up from #1123
 2018-05-11 11:31:48 -07:00
Yann Collet 99ddca43a6 fixed wrong assertion 
base can actually overflow
 2018-05-10 19:48:09 -07:00
Yann Collet 0d7626672d fixed c++ conversion warning 2018-05-10 18:17:21 -07:00
Yann Collet 09d0fa29ee minor adjusting of weights 2018-05-10 18:13:48 -07:00
Yann Collet 1a26ec6e8d opt: init statistics from dictionary 
instead of starting from fake "default" statistics.
 2018-05-10 17:59:12 -07:00
Yann Collet 74b1c75d64 btopt : minor adjustment of update frequencies 2018-05-10 16:32:36 -07:00
Yann ColletandGitHub 498ab7bb12 Merge pull request #1123 from baruchsiach/fix-install-pc 
lib/Makefile: create include directory before headers installation
 2018-05-10 10:38:19 -07:00
Yann Collet ac6105463a opt: minor improvements to log traces 
slight improvement when using fractional-bit evaluation (opt:dictionay)
 2018-05-09 15:46:11 -07:00
Yann Collet c39061cb7b fixed declaration-after-statement warning 2018-05-09 12:07:25 -07:00
Yann Collet 4d5bd32a00 added traces to look at symbol costs 
evaluation looks correct.
 2018-05-09 12:00:12 -07:00
Yann Collet c0da0f5e9e switchable bit-approximation / fractional-bit accuracy modes 
also : makes it possible to select nb of fractional bits.
 2018-05-09 10:48:09 -07:00
Yann ColletandGitHub 33dfc54069 Merge pull request #1124 from terrelln/playtests-gnu 
Write to /dev/random for test
 2018-05-09 09:39:02 -07:00
Yann Collet ba2ad9b6b9 implemented fractional bit cost evaluation 
for FSE symbols.

While it seems to work, the gains are negligible compared to rough maxNbBits evaluation.
There are even a few losses sometimes, that still need to be explained.
Furthermode, there are still cases where btlazy2 does a better job than btopt,
which seems rather strange too.
 2018-05-08 17:43:13 -07:00
Yann Collet 1aff63b114 opt: shift all costs by 8 bits (* 256) 
making it possible to represent fractional bit costs.
 2018-05-08 16:19:04 -07:00
Yann Collet 6a3c34aa58 opt: estimate cost of both Hufman and FSE symbols 
For FSE symbols : provide an upper bound,
in nb of bits,
since cost function is not able to store fractional bit costs.
 2018-05-08 16:11:21 -07:00
Yann Collet 338f738c24 pass entropy tables to optimal parser 
for proper estimation of symbol's weights
when using dictionary compression.

Note : using only huffman costs is not good enough,
presumably because sequence symbol costs are incorrect.
 2018-05-08 15:37:06 -07:00
Yann Collet a155061328 minor code refactor for readability 
removed some useless operations from optimal parser
(should not change performance, too small a difference)
 2018-05-08 12:32:44 -07:00
Nick Terrell 2dde9d5aba Write to /dev/random for test 2018-05-08 11:06:01 -07:00
Baruch Siach 9a0643b633 lib/Makefile: create include directory before headers installation 
Make sure that $(INCLUDEDIR) exists before copying the headers there.
Otherwise, the contest of header files is copied over
$(DESTDIR)$(INCLUDEDIR), making it a regular file.

While at it, remove $(DESTDIR)$(INCLUDEDIR) from the list of directories
to create in the install-pc target. The install-pc target does not need
this directory.
 2018-05-08 20:59:44 +03:00
Yann ColletandGitHub 87125c2c01 Merge pull request #1122 from facebook/fixCDictBlock 
fix ZSTD_compressBlock() associated with ZSTD_CDict
 2018-05-07 17:13:59 -07:00
Yann Collet ad4524d605 fix ZSTD_compressBlock() associated with CDict 
reported by @let-def.

It's actually a bug in ZSTD_compressBegin_usingCDict()
which would pass a wrong pledgedSrcSize value (0 instead of ZSTD_CONTENTSIZE_UNKNOWN)
resulting in wrong window size, resulting in downsized seqStore,
resulting in segfault when writing into the seqStore later in the process.

Added a test in fuzzer to cover this use case (fails before the patch).
 2018-05-07 12:54:13 -07:00
Yann ColletandGitHub ef1abd3c07 Merge pull request #1120 from lamby/897904-libzstd-please-make-the-build-reproducible 
Make the build reproducible
 2018-05-04 12:39:40 -07:00
Chris Lamb 2dbe408a49 Make the build reproducible 
Whilst working on the Reproducible Builds effort [0], we noticed
that zstd could not be built reproducibly.

This is due to the manual page encoding the number of CPUs from the
build machine and thus varies across builds.

This was originally filed in Debian as #897904 [1].

 [0] https://reproducible-builds.org/
 [1] https://bugs.debian.org/897904

Signed-off-by: Chris Lamb <lamby@debian.org>
 2018-05-04 08:39:51 -07:00
Yann ColletandGitHub 5ec92c4d96 Merge pull request #1118 from terrelln/playtests-typo 
Fix playTests.sh typo
 2018-05-03 14:26:26 -07:00
Nick Terrell 263134ce66 Fix playTests.sh typo 
Thanks to @mestia for reporting and finding the bug!
Fixes #1116.
 2018-05-03 13:03:08 -07:00
cyan4973 b095bffac6 ignore Windows build/test artefacts 2018-05-01 05:45:46 -07:00
Yann ColletandGitHub 1fc15dfd9d Merge pull request #1114 from pseiderer/ps-devel-001 
Split library install target into pc, static, shared and include only target
 2018-04-30 14:59:59 -07:00
Yann Collet 82ad249645 Clarifications of Zstandard format specification 
from IETF RFC review
 2018-04-30 12:36:55 -07:00
Peter Seiderer 64bfdca5b9 Split library install target into pc, static, shared and include only target 
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
 2018-04-30 20:32:32 +02:00
Yann ColletandGitHub a11f0c67bb Merge pull request #1113 from terrelln/dict-fix 
Fix parameter adjustment with dictionary
 2018-04-26 10:31:25 -07:00
Nick Terrell ca77822ddf Fix parameter adjustment with dictionary 
The new advanced API basically set `requestedParams = appliedParams` when
using a dictionary. This halted all parameter adjustment, which can hurt
compression ratio if, for example, the window log is small for the first
call, but the rest of the files are large.

This patch fixes the bug, and checks that the `requestedParams` don't change
in the new advanced API when using a dictionary, and generally in the fuzzer.
 2018-04-25 16:32:29 -07:00
Yann Collet 12f60b8c98 clarified documentation related to refPrefix() 2018-04-25 10:17:06 -07:00
Yann ColletandGitHub 449cccf6c9 Merge pull request #1112 from facebook/lz4dec 
fixed decoding bogus lz4 frame
 2018-04-24 16:50:54 -07:00