ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,111 Commits 17 Branches 77 Tags
324c2cd9087d93f511c10d51cf9131843041a164
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 324c2cd908 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-04-11 14:42:46 -07:00
Yann Collet 0e30059ba1 cli : FIO_createDictBuffer() replaces FIO_loadFile() 
makes it more explicit that it allocates a buffer
and that it's meant to be used for dictionary.

Also : simplified function a bit,
now only works for dictionaries up to DICTSIZE_MAX
 2017-04-11 14:41:02 -07:00
Yann Collet 0181fef545 ensure cctx internal buffer is correctly sized in case of memory error 2017-04-06 01:25:26 -07:00
Yann Collet 36c2a03757 updated comments for ZSTD_resetCStream() 2017-04-05 22:06:21 -07:00
Yann Collet ce80098f14 improved zstreamtest --mt to trap bug #644 2017-04-05 16:34:09 -07:00
Yann Collet 003a244324 DStream : ensure correct size of internal buffers in case of error 2017-04-05 15:28:56 -07:00
Yann Collet 02d37aa1c1 ensure correct size of internal buffers in case of error 2017-04-05 14:53:51 -07:00
Yann Collet 9631e603ae paramgrill : results from optimizer use cli format 
Best found configuration is displayed
using documented cli advanced parameter syntax.
 2017-04-04 16:54:33 -07:00
Yann Collet 89b32f3db0 fix paramgrill -O# 
ensure proposed config does not require more memory than necessary
 2017-04-04 16:41:11 -07:00
Yann Collet 2724f25567 fixed paramgrill -O# 
find optimal settings for a minimum speed
 2017-04-04 16:31:17 -07:00
Yann Collet c2007388a5 fixed bench.c : optional advanced parameters applied 
before creating cdict
 2017-04-04 15:35:06 -07:00
Yann Collet 81d6380139 minor bench.c adjustments 
shorter debug messages
no need to check decompressedLength==0 twice
 2017-04-04 15:21:09 -07:00
Yann Collet 7cf78f1be7 Protects ZSTD_compressBegin_usingCDict() vs NULL cdict dereference 
Will issue an error (GENERIC) is cdict==NULL
 2017-04-04 12:38:14 -07:00
Yann Collet a36330a27d minor man update, for overlapSize 2017-04-03 14:57:24 -07:00
Yann Collet 805c5a3efb updated documentation on multithreading modes 2017-04-01 00:36:31 -07:00
Yann Collet 30c7698970 optimize ZSTDMT_compress() memory usage 
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).

Also : fix ZSTD_compressBound() for small srcSize
 2017-03-31 18:27:03 -07:00
Yann Collet 3f75d52527 Changed ZSTD_compressBound() 
required so that if Total = A+B
compressBound(Total) <= compressBound(A) + compressBound(B)
under condition of a minimum size for A and B

Will help for ZSTDMT_compress() memory allocation
 2017-03-31 17:11:38 -07:00
Yann Collet 7b70a1969e Merge branch 'dev' into zstdmt 2017-03-31 16:22:33 -07:00
Yann Collet d11c7bf9bf Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-31 16:19:52 -07:00
Yann Collet 596ee423bb Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-31 16:19:33 -07:00
Yann Collet a935d67bf1 minor typo fixes in specification 2017-03-31 16:19:04 -07:00
Yann Collet 14433ca1ad numerous typos and clarifications in format specification 
fix limit values of Window_Size
bump version to 0.2.5
 2017-03-31 15:45:58 -07:00
Yann Collet eea7858e2b fixed minor warnings in debug code 2017-03-30 16:47:19 -07:00
Yann Collet 34cc487d05 overlap at full windowSize for max compression level 
as it provides max compression ratio
 2017-03-30 16:23:22 -07:00
Yann Collet 458e955c23 improved ZSTDMT_compress() 
Use a bit more threads by default.
Uses overlap segments to boost compression ratio (like the streaming variant)
 2017-03-30 15:51:58 -07:00
Yann Collet 274f59919d Changed memory strategy to __packed for gcc 
Method 1 __packed is always as good or better than memcpy().
But it's not portable, as it depends on compiler extension.

For gcc, __pakced directive works fine.
Furthermore, gcc has serious performance issues with memcpy() on ARM 32 bits.
See #620
 2017-03-30 12:52:14 -07:00
Yann Collet 6851db48e0 created contrib/cleanTabs 2017-03-30 12:42:44 -07:00
Yann Collet 3d58a1f9b9 fixed Visual fuzzer project 2017-03-30 12:28:09 -07:00
Yann Collet 2bcefcc50d fixed Visual Studio projects 2017-03-29 22:03:27 -07:00
Yann Collet ca5a8bbe36 re-added patch ... 2017-03-29 17:15:27 -07:00
Yann Collet 1e1e26f2ac fixed #634 : ZSTDMT_compressCCtx() doesn't provide frame content size in header 2017-03-29 17:09:59 -07:00
Yann Collet 0ef6803707 added ZSTD_getFrameContentSize() to ZSTDMT unit test in fuzzer 
Now tests fail, because ZSTDMT_compress() doesn't fill frame content size correctly.
 2017-03-29 16:58:57 -07:00
Yann Collet 2e2e78de47 removed unnecessary restriction on minmatchLength 
it's now transparently translated to nearest value when unsupported
(7->6) (3->4)
 2017-03-29 16:02:47 -07:00
Yann Collet 26769d88bc Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-29 15:21:30 -07:00
Yann Collet 933ce4a1dd fix : minmatch 7 conversion 
minmatch 7 now converted to minmatch 6 for strategies which do not support 7
Used to folded into "default", which applied minmatch 4
 2017-03-29 14:35:38 -07:00
Yann Collet 5bde4be5a1 fix : bench automatically adapts parameters to srcSize 2017-03-29 12:10:38 -07:00
Yann Collet 57458d4cea Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-27 14:43:53 -07:00
Yann Collet 41fefd573a Improved speed tests 
Better compatibility with Mac OS-X
Force attribution to a selected core # (Linux)
 2017-03-26 23:52:19 -07:00
Yann Collet 4cf0093571 restored bonus rule 2017-03-26 14:51:00 -07:00
Yann Collet 69017bf253 Merge branch 'dev' into LegacyDictBuilder 2017-03-26 14:39:13 -07:00
Yann Collet 582760818f minor refactor 
add const
changed if for easier to add new conditions
 2017-03-26 03:04:56 -07:00
Yann Collet 858f72eeb8 fixed dictBuilder issue 
dictionary loading would fail during entropy analysis
 2017-03-26 02:50:00 -07:00
Yann Collet ecee9f2ef8 fixed conversion warnings 2017-03-26 00:59:14 -07:00
Yann Collet 4c41d37fcc changed test for new syntax 
--dictID= and --maxdict=
 2017-03-24 18:36:56 -07:00
Yann Collet d41f707e88 minor improvement : remove duplicates with 1 char prefix difference 2017-03-24 17:56:45 -07:00
Yann Collet 96aa3019b2 changed advanced commands --maxdict= and --dictID= 
now works with the `=` variant, which is the recommended one.
Old variant `--dictID #` still works, for compatibility with existing scripts.
Long term objective is to remove the old variant..
 2017-03-24 16:04:29 -07:00
Yann Collet 9da3b215ec Ensure all limits derived from same constants 
Now uses ZDICT_DICTSIZE_MIN and ZDICT_CONTENTSIZE_MIN
from zdict.h.

Also : reduced values to 256 and 128 respectively
 2017-03-24 15:02:09 -07:00
Yann Collet 16a0b10781 fixed ZSTD_loadZstdDictionary() 
forgot to add the dictionary content
(tests were not failing, just compressing less).

Also : added size protections when adding dict content
since hc/bt table filling would fail if size < 8
 2017-03-24 12:46:46 -07:00
Yann Collet 23776ce290 fixed ERROR_GENERIC on dstSize_tooSmall 
required by users which depends on this error code to size dest buffer
 2017-03-23 17:59:50 -07:00
Yann Collet f332ece468 dictBuilder fails to create dictionary on certain input 
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
 2017-03-23 16:24:02 -07:00
Yann Collet bea78e8fc2 limited CDict acceptation criteria to be the same as DDict 2017-03-23 15:46:06 -07:00
Yann Collet 35fc66f257 updated .gitignore for new cmake script 2017-03-21 17:52:42 -07:00
Yann Collet f3dfcdccd1 bump version number 2017-03-21 12:18:28 -07:00
Yann Collet 8a7f416871 fixed typo in server type 2017-03-17 15:33:09 -07:00
Yann Collet dcc04944f6 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-17 14:26:17 -07:00
Yann Collet e5c4f04033 updated compression graph 2017-03-17 14:25:57 -07:00
Yann Collet 0a189b63fe fix minor details in man page 2017-03-16 16:33:53 -07:00
Yann Collet 118a8e9ade Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-14 04:20:12 -07:00
Yann Collet 7ae3039f41 updated NEWS for v1.1.4 2017-03-14 04:19:51 -07:00
Yann Collet baa9b114f8 minor text refactor in readme 2017-03-07 16:24:54 -08:00
Yann Collet 38ab1db3cd fixed lzbench link 2017-03-06 17:24:34 -08:00
Yann Collet eeb9758c39 fix : remove mempcpy line in bench 2017-03-06 17:22:47 -08:00
Yann Collet 764c2fdfed updated benchmark table 
zstd v1.1.3, new station i7-6700K
 2017-03-06 17:20:44 -08:00
Yann Collet 3db00373c5 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-05 21:18:25 -08:00
Yann Collet 8b1d004031 added -Wformat-security flag, as recommended by @pixelb 2017-03-05 21:17:32 -08:00
Yann Collet 1f2c95c5f3 minor code refactor in HUF module 2017-03-05 21:07:20 -08:00
Yann Collet e02409fdc3 update NEWS on @iburinoc's 32-bits version improvement 2017-03-02 17:14:57 -08:00
Yann Collet fe5d27062e disable prefetch-decode for 32-bits target 
This decoder variant is detrimental to x86 architecture
likely due to register pressure.

Note that the variant is disabled for all 32-bits targets.
It's unclear if it would help for different architectures,
such as ARM, MIPS or PowerPC.
 2017-03-02 17:09:21 -08:00
Yann Collet 78208bd8be fixed : build zstd cli after libzstd 2017-03-01 21:02:06 -08:00
Yann Collet 27526c7201 make : added target shortest 
shortest only run fast part of playTests.sh .
cc @iburinoc
 2017-03-01 17:02:49 -08:00
Yann Collet c1c040eae1 added gzip tests 
also : made sure zstd --format=gzip -V
would fail if gzip compatibility is not supported
 2017-03-01 16:49:20 -08:00
Yann Collet 76f0494089 xxhash can be included twice in any order 
Previously,

followed by :

would fail to include the static definitions,
because the second include was simply skipped by guard macro.

Now it works as intended :
the missing static part is included during the second include.
 2017-03-01 13:29:29 -08:00
Yann Collet 4bcc69b761 solves warnings when compiling with global XXH_STATIC_LINKING_ONLY 
XXH_STATIC_LINKING_ONLY protection macro is intended to be triggered just before the include.
The main idea is to keep this setting local :
user module shall explicitly understand and accept the static linking restriction
which becomes transparent when triggering the macro at project level.
Global definition also triggers redefinition warnings for user modules which do locally define the macro.

This new version compiles lib and cli without warning when the macro is set globally.
That's not a scenario to be recommended, since it trades a local effect for a global one,
but it was easy enough to provide from zstd side.
 2017-03-01 11:33:25 -08:00
Yann Collet 43764cdb1d updated NEWS for 1.1.4 
cmake, performance
 2017-02-28 17:44:17 -08:00
Yann Collet 22d79762ef fixed multi frames 2017-02-28 02:12:42 -08:00
Yann Collet a33ae64204 fixed decoding skippable frames 2017-02-28 01:15:28 -08:00
Yann Collet c0b1731bce added test for decompression with NULL dict and NULL DDict 
previous version of ZSTD_decompressMultiFrame() would fail that test
 2017-02-28 01:02:46 -08:00
Yann Collet d1760113ec Improved speed of ZSTD_decompressStream() 
When ZSTD_decompressStream() detects
that there is enough space in dst
to complete decompression in a single pass,
delegates to ZSTD_decompress(),
for an extra ~5% speed boost
 2017-02-28 00:14:28 -08:00
Yann Collet 1d7f30f9d4 Merge branch 'decompressStream' into dev 2017-02-27 20:55:22 -08:00
Yann Collet 952d06fa9c fullbench : -i0 displays list of functions to bench 2017-02-27 17:58:02 -08:00
Yann Collet 67d86a74a5 added test case : --rm on stdin 
must remain silent (instead of failing)
 2017-02-27 16:09:20 -08:00
Yann Collet ef569bf75f Merge branch 'dev' of github.com:facebook/zstd into dev 2017-02-27 15:58:38 -08:00
Yann Collet dccd6b6f65 cli : fix : --rm is silent when input is stdin 
previously, app would produce an error message, and stop.
 2017-02-27 15:57:50 -08:00
Yann Collet 0b9b894b2d reduced ZSTD_DDict memory usage 
saved 128 KB
 2017-02-27 00:27:30 -08:00
Yann Collet bd7fa21deb added ZSTD_refDDict() 
Now DDict does no longer depends on DCtx duplication
 2017-02-26 14:43:07 -08:00
Yann Collet d73eebc00f loadEntropy works on new ZSTD_entropy_t type 2017-02-26 10:16:42 -08:00
Yann Collet 8629f0e41f created entropy structure type 2017-02-25 18:33:31 -08:00
Yann Collet 8dff956dbf Added DDict unit test in fuzzer 
also : slightly modified loadEntropy :
know src must points at start of dictionary
 2017-02-25 10:11:15 -08:00
Yann Collet df9f9296e3 attempt to fix pthreat linking error 
replacing -lpthread by -pthread
 2017-02-24 00:16:05 -08:00
Yann Collet 14312d833e zstdmt : fix : loading prefix from previous segments 
There used to be a (very small) chance that
loading prefix from previous segment
would be confused with a real zstd dictionary.
For that to happen, the prefix needs to start
with the same value as dictionary magic.
That's 1 chance in 4 billions if all values have equal probability.
But in fact, since some values are more common (0x00000000 for example)
others are less common, and dictionary magic was selected to be one of them,
so probabilities are likely even lower.

Anyway, this risk is no down to zero
by adding a new CCtx parameter : ZSTD_p_forceRawDict

Current parameter policy : the parameter "stick" to its CCtx,
so any dictionary loading after ZSTD_p_forceRawDict is set
will be loaded in "raw" ("content only") mode,
even if CCtx is re-used multiple times with multiple different dictionary.
It's up to the user to reset this value differently if it needs so.
 2017-02-23 23:42:12 -08:00
Yann Collet 831b4890ce minor tests/Makefile refactoring 
and update of zstd_manual,html
 2017-02-23 23:09:10 -08:00
Yann Collet 1f3d54ddb4 fixed malloc(0) potential issue 
Added test cases to cover #556 patch
 2017-02-22 11:08:00 -08:00
Yann Collet 88ba64702e fixed c90/gnu90/gnu99 tests 2017-02-22 10:52:36 -08:00
Yann Collet 7757577341 added manual target in root Makefile 
`manual` target is added to `all` target
 2017-02-22 01:10:43 -08:00
Yann Collet 508404514c added manual target to contrib/gen_html/Makefile 2017-02-22 00:57:50 -08:00
Yann Collet 3b89eb0c20 updated NEWS with meson build by Dima Krasner 2017-02-20 01:26:05 -08:00
Yann Collet 2252d29a5a Merge branch 'dev' of github.com:facebook/zstd into dev 2017-02-15 12:00:50 -08:00
Yann Collet 4596037042 updated fse version 
feature minor refactoring (removing FSE_abs())
also : fix a few minor issues recently introduced in examples
 2017-02-15 12:00:03 -08:00
Yann Collet c09d16ba8c preset behavior for gzip, gunzip and gzcat 
when zstd is called through a link named gzip, gunzip or gzcat,
provides the same behavior as the related program.
gzip compresses using --format=gz
both gzip and gunzip enable --rm by default
 2017-02-14 10:45:19 -08:00
Yann Collet 58af614ef2 push version and NEWS to v1.1.4 2017-02-13 18:32:44 -08:00