ddidderr/zstd-rs
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,199 Commits 17 Branches 77 Tags
97cbdcabddfc916d838aed06cd332b444f485953
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet e42afbc6fa Comply with suggested comments by @terrelln 
created FSE_CTABLE_SIZE() and FSE_DTABLE_SIZE()
 2017-04-26 11:39:35 -07:00
Yann Collet 230d7acc7d cli : add support for --threads=# command 
updated documentation
add relevant test case
 2017-04-21 11:38:13 -07:00
Yann Collet d0b1846cf4 ignore more cmake build artefacts 2017-04-21 10:59:36 -07:00
Yann Collet 7271203bdb transferred entropy scratch space from CCtx into workSpace 
Saved 6 KB
 2017-04-20 23:21:19 -07:00
Yann Collet a408645f50 made some room for entropy scratch space 2017-04-20 23:09:39 -07:00
Yann Collet 71aaa32c3c transferred FSE tables from CCtx into workspace 
Saved 5 KB from CCtx
 2017-04-20 23:03:38 -07:00
Yann Collet 71ddeb67b1 made room in workspace for FSE tables 
still need to be transferred from CCtx into workspace
 2017-04-20 22:54:54 -07:00
Yann Collet a34a39c183 changed size evaluation of entropy tables 
so that memcpy() does no longer depends on fse pointer being a static table
 2017-04-20 18:26:25 -07:00
Yann Collet 7bb60b17d8 init entropy table pointers only once 
per workSpace resize
 2017-04-20 17:38:56 -07:00
Yann Collet e6fa70a0a1 reorganized ZSTD_resetCCtx_internal() 
clearer separation between variables and buffers
clearer buffers category
kept static buffers at the beginning, favoring cache locality
(it will be easier to add FSE tables there later)

This break a few assumptions that hashTable was always at the beginning.
This is fixed.
And remaining assumptions (namely that tables stand next to each other in memory)
are now tested with assert.
 2017-04-20 17:28:31 -07:00
Yann Collet c17e020c9a disable assert when compiling paramgrill 
paramgrill is a benchmark calibration function.
Speed accuracy is critical, it cannot be altered by assert.
 2017-04-20 12:50:02 -07:00
Yann Collet 16f9c572fc Merge branch 'dev' into compressionFlow 2017-04-20 11:16:40 -07:00
Yann Collet e348dad305 minor long line reformatting 2017-04-20 11:14:13 -07:00
Yann Collet e847730452 slightly refined README comments on lib-mt 2017-04-18 23:15:28 -07:00
Yann Collet 2c5514c759 fixed ZSTDMT_initCStream_advanced() 
Must use the new ZSTD_compressBegin_usingCDict_advanced()
to enforce correct frame parameters
 2017-04-18 22:52:41 -07:00
Yann Collet 0bb381dad8 added test for ZSTD_initCStream_advanced() 
when params.fParams.contentSizeFlags = 1
and pledgedSrcSize = 0
then srcSize should be considered 0 (empty), and not "unknown"
 2017-04-18 15:08:52 -07:00
Yann Collet a4cab80183 added ZSTD_copyCCtx_internal() 
which respects provided fParams.
 2017-04-18 14:54:54 -07:00
Yann Collet b402f1ef37 added make list 2017-04-18 14:34:24 -07:00
Yann Collet 30fb499208 Changed ZSTD_resetCCtx_advanced() into ZSTD_resetCCtx_internal() 
for naming consistency :
_advanced() can be invoked
while _internal() are strictly static
 2017-04-18 14:08:50 -07:00
Yann Collet 715b9aa113 created ZSTD_compressBegin_usingCDict_advanced() 2017-04-18 13:55:53 -07:00
Yann Collet 291f0cb54a Merge branch 'dev' into compressionFlow 2017-04-18 03:18:34 -07:00
Yann Collet af4f45b682 Improved code comments for block functions 2017-04-18 03:17:44 -07:00
Yann Collet 4f818182b8 clarified frame parameters for ZSTD_compress*_usingCDict() 
created ZSTD_compressBegin_usingCDict_internal(),
which gives direct control to frame Parameters.
ZSTD_resetCStream_internal() now points into it.
 2017-04-17 18:29:06 -07:00
Yann Collet 9c4c970399 Merge branch 'dev' into compressionFlow 2017-04-17 16:33:09 -07:00
Yann Collet c47c68f6ca proper evaluation of Huffman CTable size 2017-04-17 16:14:21 -07:00
Yann Collet f0c9437fdf Merge branch 'dev' into compressionFlow 2017-04-17 15:48:13 -07:00
Yann Collet 415cb68e53 Merge branch 'dev' into compressionFlow 2017-04-17 14:51:05 -07:00
Yann Collet 2afb075382 Merge branch 'dev' into compressionFlow 2017-04-14 15:08:31 -07:00
Yann Collet f913cbed33 fixed : memory leak in fuzzer test 2017-04-13 22:46:51 -07:00
Yann Collet 88009a8ba2 removed srcSize control from CStream 
since it's already done from lower bufferless API level
 2017-04-12 00:51:24 -07:00
Yann Collet 20d5e03893 content size is controlled at bufferless level 
so it's active for all entry points

Also : added relevant test (wrong content size) in fuzzer
 2017-04-11 18:34:02 -07:00
Yann Collet 6ab02e16b8 Merge branch 'dev' into compressionFlow 2017-04-11 17:42:15 -07:00
Yann Collet 5c42d0edc8 cli : better status display for zstdmt in 1-thread mode 2017-04-11 16:57:32 -07:00
Yann Collet 324c2cd908 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-04-11 14:42:46 -07:00
Yann Collet 0e30059ba1 cli : FIO_createDictBuffer() replaces FIO_loadFile() 
makes it more explicit that it allocates a buffer
and that it's meant to be used for dictionary.

Also : simplified function a bit,
now only works for dictionaries up to DICTSIZE_MAX
 2017-04-11 14:41:02 -07:00
Yann Collet 4ee6b15dac force contentSizeFlag=0 when using ZSTD_initCStream_usingCDict() 
because by definition srcSize is not known when using this prototype.
added relevant test

Note : this use was already working, because at a later stage
(both ZSTD_compressBegin_usingCDict() and ZSTD_copyCCtx())
pledgedSrcSize=0 is translated into "unknown", no matter the frame parameter.
This is not correct, but of little importance,
as the medium term plan is to no longer set fParams within CDict
 2017-04-11 11:59:44 -07:00
Yann Collet ab9162ebb4 simplified call graph 
by calling ZSTD_compressBegin_internal() instead of ZSTD_compressBegin_advanced()
 2017-04-11 10:46:20 -07:00
Yann Collet e88034fe26 simplified ZSTD_initCStream*() flow 
all variants converge towards ZSTD_initCStream_stage2()
 2017-04-10 22:24:02 -07:00
Yann Collet 4b987ad8ce Introduce ZSTD_initCStream_internal() 
This is now the regroup point for ZSTD_initCStream*() functions

ZSTD_initCStream_advanced() now properly checks for parameters validity.

Also : added <assert.h> usage inside zstd_compress.c
Needs ZSTD_DEBUG=1 macro to be triggered.
Will be triggered by default from `tests` directory
 2017-04-10 17:50:44 -07:00
Yann Collet 0181fef545 ensure cctx internal buffer is correctly sized in case of memory error 2017-04-06 01:25:26 -07:00
Yann Collet 36c2a03757 updated comments for ZSTD_resetCStream() 2017-04-05 22:06:21 -07:00
Yann Collet ce80098f14 improved zstreamtest --mt to trap bug #644 2017-04-05 16:34:09 -07:00
Yann Collet 003a244324 DStream : ensure correct size of internal buffers in case of error 2017-04-05 15:28:56 -07:00
Yann Collet 02d37aa1c1 ensure correct size of internal buffers in case of error 2017-04-05 14:53:51 -07:00
Yann Collet 9631e603ae paramgrill : results from optimizer use cli format 
Best found configuration is displayed
using documented cli advanced parameter syntax.
 2017-04-04 16:54:33 -07:00
Yann Collet 89b32f3db0 fix paramgrill -O# 
ensure proposed config does not require more memory than necessary
 2017-04-04 16:41:11 -07:00
Yann Collet 2724f25567 fixed paramgrill -O# 
find optimal settings for a minimum speed
 2017-04-04 16:31:17 -07:00
Yann Collet c2007388a5 fixed bench.c : optional advanced parameters applied 
before creating cdict
 2017-04-04 15:35:06 -07:00
Yann Collet 81d6380139 minor bench.c adjustments 
shorter debug messages
no need to check decompressedLength==0 twice
 2017-04-04 15:21:09 -07:00
Yann Collet 7cf78f1be7 Protects ZSTD_compressBegin_usingCDict() vs NULL cdict dereference 
Will issue an error (GENERIC) is cdict==NULL
 2017-04-04 12:38:14 -07:00
Yann Collet a36330a27d minor man update, for overlapSize 2017-04-03 14:57:24 -07:00
Yann Collet 805c5a3efb updated documentation on multithreading modes 2017-04-01 00:36:31 -07:00
Yann Collet 30c7698970 optimize ZSTDMT_compress() memory usage 
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).

Also : fix ZSTD_compressBound() for small srcSize
 2017-03-31 18:27:03 -07:00
Yann Collet 3f75d52527 Changed ZSTD_compressBound() 
required so that if Total = A+B
compressBound(Total) <= compressBound(A) + compressBound(B)
under condition of a minimum size for A and B

Will help for ZSTDMT_compress() memory allocation
 2017-03-31 17:11:38 -07:00
Yann Collet 7b70a1969e Merge branch 'dev' into zstdmt 2017-03-31 16:22:33 -07:00
Yann Collet d11c7bf9bf Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-31 16:19:52 -07:00
Yann Collet 596ee423bb Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-31 16:19:33 -07:00
Yann Collet a935d67bf1 minor typo fixes in specification 2017-03-31 16:19:04 -07:00
Yann Collet 14433ca1ad numerous typos and clarifications in format specification 
fix limit values of Window_Size
bump version to 0.2.5
 2017-03-31 15:45:58 -07:00
Yann Collet eea7858e2b fixed minor warnings in debug code 2017-03-30 16:47:19 -07:00
Yann Collet 34cc487d05 overlap at full windowSize for max compression level 
as it provides max compression ratio
 2017-03-30 16:23:22 -07:00
Yann Collet 458e955c23 improved ZSTDMT_compress() 
Use a bit more threads by default.
Uses overlap segments to boost compression ratio (like the streaming variant)
 2017-03-30 15:51:58 -07:00
Yann Collet 274f59919d Changed memory strategy to __packed for gcc 
Method 1 __packed is always as good or better than memcpy().
But it's not portable, as it depends on compiler extension.

For gcc, __pakced directive works fine.
Furthermore, gcc has serious performance issues with memcpy() on ARM 32 bits.
See #620
 2017-03-30 12:52:14 -07:00
Yann Collet 6851db48e0 created contrib/cleanTabs 2017-03-30 12:42:44 -07:00
Yann Collet 3d58a1f9b9 fixed Visual fuzzer project 2017-03-30 12:28:09 -07:00
Yann Collet 2bcefcc50d fixed Visual Studio projects 2017-03-29 22:03:27 -07:00
Yann Collet ca5a8bbe36 re-added patch ... 2017-03-29 17:15:27 -07:00
Yann Collet 1e1e26f2ac fixed #634 : ZSTDMT_compressCCtx() doesn't provide frame content size in header 2017-03-29 17:09:59 -07:00
Yann Collet 0ef6803707 added ZSTD_getFrameContentSize() to ZSTDMT unit test in fuzzer 
Now tests fail, because ZSTDMT_compress() doesn't fill frame content size correctly.
 2017-03-29 16:58:57 -07:00
Yann Collet 2e2e78de47 removed unnecessary restriction on minmatchLength 
it's now transparently translated to nearest value when unsupported
(7->6) (3->4)
 2017-03-29 16:02:47 -07:00
Yann Collet 26769d88bc Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-29 15:21:30 -07:00
Yann Collet 933ce4a1dd fix : minmatch 7 conversion 
minmatch 7 now converted to minmatch 6 for strategies which do not support 7
Used to folded into "default", which applied minmatch 4
 2017-03-29 14:35:38 -07:00
Yann Collet 5bde4be5a1 fix : bench automatically adapts parameters to srcSize 2017-03-29 12:10:38 -07:00
Yann Collet 57458d4cea Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-27 14:43:53 -07:00
Yann Collet 41fefd573a Improved speed tests 
Better compatibility with Mac OS-X
Force attribution to a selected core # (Linux)
 2017-03-26 23:52:19 -07:00
Yann Collet 4cf0093571 restored bonus rule 2017-03-26 14:51:00 -07:00
Yann Collet 69017bf253 Merge branch 'dev' into LegacyDictBuilder 2017-03-26 14:39:13 -07:00
Yann Collet 582760818f minor refactor 
add const
changed if for easier to add new conditions
 2017-03-26 03:04:56 -07:00
Yann Collet 858f72eeb8 fixed dictBuilder issue 
dictionary loading would fail during entropy analysis
 2017-03-26 02:50:00 -07:00
Yann Collet ecee9f2ef8 fixed conversion warnings 2017-03-26 00:59:14 -07:00
Yann Collet 4c41d37fcc changed test for new syntax 
--dictID= and --maxdict=
 2017-03-24 18:36:56 -07:00
Yann Collet d41f707e88 minor improvement : remove duplicates with 1 char prefix difference 2017-03-24 17:56:45 -07:00
Yann Collet 96aa3019b2 changed advanced commands --maxdict= and --dictID= 
now works with the `=` variant, which is the recommended one.
Old variant `--dictID #` still works, for compatibility with existing scripts.
Long term objective is to remove the old variant..
 2017-03-24 16:04:29 -07:00
Yann Collet 9da3b215ec Ensure all limits derived from same constants 
Now uses ZDICT_DICTSIZE_MIN and ZDICT_CONTENTSIZE_MIN
from zdict.h.

Also : reduced values to 256 and 128 respectively
 2017-03-24 15:02:09 -07:00
Yann Collet 16a0b10781 fixed ZSTD_loadZstdDictionary() 
forgot to add the dictionary content
(tests were not failing, just compressing less).

Also : added size protections when adding dict content
since hc/bt table filling would fail if size < 8
 2017-03-24 12:46:46 -07:00
Yann Collet 23776ce290 fixed ERROR_GENERIC on dstSize_tooSmall 
required by users which depends on this error code to size dest buffer
 2017-03-23 17:59:50 -07:00
Yann Collet f332ece468 dictBuilder fails to create dictionary on certain input 
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
 2017-03-23 16:24:02 -07:00
Yann Collet bea78e8fc2 limited CDict acceptation criteria to be the same as DDict 2017-03-23 15:46:06 -07:00
Yann Collet 35fc66f257 updated .gitignore for new cmake script 2017-03-21 17:52:42 -07:00
Yann Collet f3dfcdccd1 bump version number 2017-03-21 12:18:28 -07:00
Yann Collet 8a7f416871 fixed typo in server type 2017-03-17 15:33:09 -07:00
Yann Collet dcc04944f6 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-17 14:26:17 -07:00
Yann Collet e5c4f04033 updated compression graph 2017-03-17 14:25:57 -07:00
Yann Collet 0a189b63fe fix minor details in man page 2017-03-16 16:33:53 -07:00
Yann Collet 118a8e9ade Merge branch 'dev' of github.com:facebook/zstd into dev 2017-03-14 04:20:12 -07:00
Yann Collet 7ae3039f41 updated NEWS for v1.1.4 2017-03-14 04:19:51 -07:00
Yann Collet baa9b114f8 minor text refactor in readme 2017-03-07 16:24:54 -08:00
Yann Collet 38ab1db3cd fixed lzbench link 2017-03-06 17:24:34 -08:00
Yann Collet eeb9758c39 fix : remove mempcpy line in bench 2017-03-06 17:22:47 -08:00
Yann Collet 764c2fdfed updated benchmark table 
zstd v1.1.3, new station i7-6700K
 2017-03-06 17:20:44 -08:00