ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,094 Commits 17 Branches 77 Tags
7f975c0c2881c1f6b2523fdc64489cda280c6de1
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 9a92ed401d updated compression results.csv 
and fixed nit
 2018-12-12 20:30:09 -08:00
Yann Collet 9792acda3b Merge branch 'dev' into btultra2 2018-12-12 20:18:27 -08:00
Yann Collet 52b94f902c add clarification for ZSTD_CCtx_setPledgedSrcSize() 
as requested in #1391
 2018-12-11 12:08:21 -08:00
Yann Collet 5e6aaa3abb fixed btultra2 usage with prefix 
notably while using multi-threading
 2018-12-10 18:45:03 -08:00
Yann Collet 3619c34399 fix assert position within ZSTD_compress2() 2018-12-10 17:42:35 -08:00
Yann Collet 5a1e01e5f1 clarified experimentalParam 2018-12-10 17:36:20 -08:00
Yann Collet c226a7b9f3 fixed ZSTD_compress2() 
as suggested by @terrelln
 2018-12-10 17:33:49 -08:00
Yann Collet ea441ee4ae make test can accept DEBUGLEVEL argument 2018-12-09 23:20:51 -08:00
Yann Collet 37e314a68d updated clevel table for large inputs 2018-12-09 22:38:05 -08:00
Yann Collet c9c4c7ec8c update clevel table for 256K 2018-12-08 21:40:08 -08:00
Yann Collet 8075d75f9c update clevel table for 128K 2018-12-08 10:42:55 -08:00
Yann Collet 95b152ab33 updated clevel table for 16K 
to introduce btultra2
 2018-12-07 20:12:43 -08:00
Yann Collet b71bfb6cf2 paramgrill: add status line 
get information on which config is currently tested
so that console get animated during long tests.
 2018-12-07 16:02:24 -08:00
Yann Collet 27b253fadc added tests for strategy=9 (btultra2) 2018-12-07 14:20:54 -08:00
Yann Collet e68c2d86e7 refactor paramgrill for clarity 
restored ability to copy/paste the resulting compression level table into zstd_compress.c .
 2018-12-07 14:07:54 -08:00
Yann Collet d613fd9afe linked btultra2 as strategy9 
and ensure zstdbench detects out-of-bound parameters
 2018-12-06 19:27:37 -08:00
Yann Collet 34aa401afd updated documentation 
introducing ZSTD_btultra2
 2018-12-06 17:22:19 -08:00
Yann Collet ae370b0e12 minor bound refinements 2018-12-06 16:51:17 -08:00
Yann Collet 39e28982cf introduced constants ZSTD_STRATEGY_MIN and ZSTD_STRATEGY_MAX 2018-12-06 16:16:16 -08:00
Yann Collet c3c3488981 fixed c++ assignment to enum 2018-12-06 15:57:55 -08:00
Yann Collet be9e561da4 changed ZSTD_c_compressionStrategy into ZSTD_c_strategy 
also : fixed paramgrill, and limit conditions
 2018-12-06 15:00:52 -08:00
Yann Collet e9448cdf4c introduced strategy btultra2 
note : not yet applied on any compression level
 2018-12-06 13:38:09 -08:00
Yann Collet 0c404a48f0 moved ZSTD_WINDOWLOG_LIMIT_DEFAULT into static-linking-only area 2018-12-06 10:57:19 -08:00
Yann Collet 96d887429b clarified usage of word "job" 
only applies in MT / async context now.
 2018-12-06 10:14:34 -08:00
Yann Collet 3583d19c4e changed parameter names from ZSTD_p_* to ZSTD_c_* 
for naming consistency
 2018-12-05 17:26:02 -08:00
Yann Collet c2053310e5 updated API documentation 2018-12-05 16:23:00 -08:00
Yann Collet 3e042d5cc0 ZSTD_decompressDCtx() is compatible with sticky parameters 2018-12-04 17:30:58 -08:00
Yann Collet d7da3fc90a merge dedicated dParam setters 2018-12-04 17:06:48 -08:00
Yann Collet 4b5a4f02d7 write the switch()case: differently 
so that it please both
compilers which warn for dead code after the switch
and
compilers which do not detect that all branches terminate.
 2018-12-04 16:59:26 -08:00
Yann Collet 85b02bf142 fixed silent conversion warning 2018-12-04 15:57:16 -08:00
Yann Collet 2fb8d1a392 fixed declaration-after-statement warnings 2018-12-04 15:54:01 -08:00
Yann Collet aec945f0dc implemented ZSTD_dParam_getBounds() 
and ZSTD_DCtx_setParameter()
 2018-12-04 15:35:37 -08:00
Yann Collet 34e146f548 advanced decompression function replaces by normal streaming one 
advanced parameters compatible with ZSTD_decompressStream().
 2018-12-04 10:28:36 -08:00
Yann Collet 7ef7dc561a check availability of --color=never command on grep and egrep 
before applying them.
Fixes #1436
 2018-12-03 15:46:55 -08:00
Yann Collet 6ced8f7c7c joined normal streaming API with advanced one 2018-12-03 14:22:38 -08:00
Yann Collet da1f3066a3 preparative for ZSTD_DCtx_setParameter() 2018-11-30 15:59:50 -08:00
Yann Collet d8e215cbee created ZSTD_compress2() and ZSTD_compressStream2() 
ZSTD_compress_generic() is renamed ZSTD_compressStream2().

Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
 2018-11-30 11:25:56 -08:00
Yann Collet d3a0c71259 pushed experimental parameters 
into ZSTD_STATIC_LINKING_ONLY section
 2018-11-21 16:18:55 -08:00
Yann Collet d4d4e109e9 getParameter fills an int* 
rather than an unsigned*
for consistency
since type of setParameter() changed to int.
 2018-11-21 15:37:26 -08:00
Yann Collet fea920615c promote ZSTD_findFrameCompressedSize() into staging area 2018-11-21 15:25:50 -08:00
Yann Collet 41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet 5d3592398d fixed fall-through 2018-11-20 16:09:33 -08:00
Yann Collet 5c6d4b18ac completed implementation of ZSTD_cParam_getBounds() 
for all parameters
 2018-11-20 16:06:00 -08:00
Yann Collet 2e7fd6a2cb fixed remaining searchLength invocations 2018-11-20 15:13:27 -08:00
Yann Collet e874dacc08 changed searchLength into minMatch 
refactored all relevant API and calls
for consistency.
 2018-11-20 14:56:07 -08:00
Yann Collet 114bd4346e changed enum type name to ZSTD_ResetDirective 
for naming consistency :
types should start with a capital letter (after prefix)
 2018-11-20 12:00:20 -08:00
Yann Collet 3b838abf97 ZSTD_CCtx_setParameter : value argument is now int 
for compatibility with compression level
 2018-11-20 11:53:01 -08:00
Yann Collet 19e5f2a35b removed some constants and _simpleArgs() from staging 
constants that *may* change in the future
will be accessed through functions instead
(to be created).

_simpleArgs() variant do not have (yet) a clear enough added value
to deserve "stable" status.
 2018-11-19 17:38:15 -08:00
Yann Collet f07e13b729 fixed fuzz src 2018-11-15 16:53:45 -08:00
Yann Collet 5c68639186 updated ZSTD_DCtx_reset() 
signature and behavior is now the same as ZSTD_CCtx_reset()
 2018-11-15 16:12:39 -08:00
Yann Collet 4542b9f1cf updated manual 2018-11-15 10:52:24 -08:00
Yann Collet 06c8d5a4f4 Merge branch 'dev' into advancedAPI 
fixed rsyncable
 2018-11-15 10:51:24 -08:00
Yann Collet 21a42bf5f9 added advanced decompression api 2018-11-14 16:54:54 -08:00
Yann Collet cf9f4b63b8 fixed fuzz test src code 2018-11-14 14:46:49 -08:00
Yann Collet 7b0391e37e finalized retrofit of ZSTD_CCtx_reset() 
updated all depending sources
 2018-11-14 13:05:35 -08:00
Yann Collet 5adbad4059 Merge branch 'dev' into advancedAPI 2018-11-14 13:00:37 -08:00
Yann Collet ff8d371708 modified ZSTD_CCtx_reset() 
which now accepts an enum,
to distinguish between resetting the session, or the parameters (or both).

removed ZSTD_CCtx_resetParameters(), which is redundant.

start replacing invocation of ZSTD_CCtx_reset*() functions

Updated advanced API documentation

trimmed down amount of API staged in RC,
in particular, all functions related to ZSTD_CCtxParams()
seem too advanced.
 2018-11-14 12:33:57 -08:00
Yann Collet d7e10a774a added constant ZSTD_WINDOWLOG_LIMIT_DEFAULT 
answering #1407.

Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
 2018-11-13 18:12:34 -08:00
Yann Collet 2c8fde538f added constant ZSTD_MAGIC_SKIPPABLE_MASK 
and updated several API comments
 2018-11-13 17:36:35 -08:00
Yann Collet b83d1e7714 removed some static const variables 
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
 2018-11-13 16:56:32 -08:00
Yann Collet 975abf8f5d circumvent gcc-8 strict aliasing warning 
because gcc considers that
`void* const *`  and `void**` are 2 different types ...
 2018-11-13 16:04:35 -08:00
Yann Collet 768a264200 Merge branch 'dev' of github.com:facebook/zstd into dev 2018-11-13 15:56:36 -08:00
Yann Collet 092c4abd4c bumped version number to v1.3.8 2018-11-13 15:53:38 -08:00
Yann Collet 3ba0d6dd27 fixed decode-only test condition 2018-11-13 14:15:12 -08:00
Yann Collet a29e3d1d01 fix debug trace 2018-11-13 13:27:22 -08:00
Yann Collet e4abd28769 fixed VS2010+ build script 2018-11-13 13:22:16 -08:00
Yann Collet f15312e884 fixed cmake build script 2018-11-13 13:16:41 -08:00
Yann Collet b830ccca5c changed benchfn api 
to use structure for function parameters
as it expresses much clearer than a long list of parameters,
since each parameter can now be named.
 2018-11-13 13:12:50 -08:00
Yann Collet 9867cdb847 benchfn can provided faulty return value 
with BMK_extract_returnValue()
 2018-11-13 12:01:17 -08:00
Yann Collet d38063f8ae separated bench module into benchfn and benchzstd 
it shall be possible to use benchfn
without any dependency on zstd.
 2018-11-13 11:01:59 -08:00
Yann Collet 626040ab53 changed PREFETCH() macro into PREFETCH_L2() 
which is more accurate
 2018-11-12 17:05:32 -08:00
Yann Collet 483759a3de Improves decompression speed when using cold dictionary 
by triggering the prefetching decoder path
(which used to be dedicated to long-range offsets only).

Figures on my laptop :
no content prefetch : ~300 MB/s (for reference)
full content prefetch : ~325 MB/s (before this patch)
new prefetch path : ~375 MB/s (after this patch)

The benchmark speed is already significant,
but another side-effect is that this version
prefetch less data into memory,
since it only prefetches what's needed, instead of the full dictionary.

This is supposed to help highly active environments
such as active databases,
that can't be properly measured in benchmark environment (too clean).

Also :
fixed the largeNbDict test program
which was working improperly when setting nbBlocks > nbFiles.
 2018-11-08 17:00:23 -08:00
Yann Collet 20fb9e7f36 reduced assertion strength 
one limit case can apparently be generated during fuzzer tests
 2018-11-08 12:57:34 -08:00
Yann Collet f40c6b4395 added a simple test case for decode-only benchmark mode 
as it was broken and no one noticed
 2018-11-08 12:51:42 -08:00
Yann Collet 9126da5b5c improve long-range decoder speed 
on enwik9 at level 22 (which is almost a worst case scenario),
speed improves by +7% on my laptop (415 -> 445 MB/s)
 2018-11-08 12:47:46 -08:00
Yann Collet 8bed4012bd fixed decompression-only benchmark 2018-11-08 12:36:39 -08:00
Yann Collet 3e5cdf1b6a fixed T36302429 2018-11-05 17:50:30 -08:00
Yann Collet 2caa995558 just add an assert() in ZSTD_insertBtAndGetAllMatches() 
to express a condition on ll0 .
May help static analyzer as in #1397
 2018-11-05 17:13:32 -08:00
Yann Collet acd75a1448 fixed a second memset() on NULL 
not sure why it only triggers now,
this code has been around for a while.

Introduced a new error code : dstBuffer_null,
I couldn't express anything even remotely similar with existing error codes set.
 2018-10-29 15:03:57 -07:00
Yann Collet 9c58098200 fixed memcpy() on NULL warning 
memcpy(NULL, src, 0) is undefined behavior.
 2018-10-29 13:57:37 -07:00
Yann Collet fc20b3c441 added flag -Wc++-compat 
for library and cli
 2018-10-26 16:38:23 -07:00
Yann Collet 4860311d99 renamed NEWS as CHANGELOG 
to conform fb publication rules.
 2018-10-26 15:57:13 -07:00
Yann Collet 9bf00707c7 minor clarifications of history update rules 2018-10-26 15:51:51 -07:00
Yann Collet 1866bd374a Merge branch 'dev' into huf_Refactor 2018-10-26 15:25:01 -07:00
Yann Collet 8d56f4baee added a few comments for clarifications 2018-10-26 15:21:52 -07:00
Yann Collet 450356b5af Merge branch 'dev' into decompressblock 2018-10-26 15:03:43 -07:00
Yann Collet 7b74405150 refactor HUF_compress_internal for clarity 
changed workspace parameter convention
to always provide workspaceSize,
so that size can be explicitly checked.

Also, use more enum to make the meaning of some parameters more explicit.
 2018-10-26 13:21:37 -07:00
Yann Collet 4ed9233db6 fixed VS2010 project 2018-10-26 11:25:40 -07:00
Yann Collet b832498aee fixed : extraneous function prototype in fullbench.c 
now provided through zstd_internal.h
 2018-10-26 10:39:49 -07:00
Yann Collet 2b4914082e created zstd_decompress_block module 
isolate all logic associated with block decompression
into its own module.

zstd_decompress is still in charge
of context creation/destruction,
frames, headers, streaming, special blocks, etc.

Compressed blocks themselves are now handled within zstd_decompress_block .
 2018-10-25 16:28:41 -07:00
Yann Collet cb320a9fc0 added comment on public ddict functions 2018-10-24 16:50:03 -07:00
Yann Collet 806a5c84e4 support decompressing an empty frame into NULL 
fix #1385
decompressing into NULL was an automatic error.
It is now allowed, as long as the content of the frame is empty.

Seems to simplify things for `arrow`.
Maybe some other projects rely on this behavior ?
 2018-10-24 16:34:35 -07:00
Yann Collet b7421f8e69 fix and refactored libzstd_partial_build.sh 
make the test intention clearer
(easier to maintain)

also resist the existence of zstd_ddict.o within libzstd.a
 2018-10-24 11:32:09 -07:00
Yann Collet debff3929b fixed warnings in testpools 2018-10-24 10:36:06 -07:00
Yann Collet cc3612e1c5 added simple guard macros 
in case of accidental multi-includes
 2018-10-23 17:55:23 -07:00
Yann Collet 2f7e1c8737 fixed VS2010 build script 2018-10-23 17:44:43 -07:00
Yann Collet 1ae38772df restored cmake build tests in Travis 
for some reason, these tests were no longer played on Travis.

As a consequence, I discovered some left-over from the util.c change while testing.
 2018-10-23 17:40:15 -07:00
Yann Collet ccd2d426fc separate DDict logic into its own module 
created zstd_ddict.c within lib/decompress
 2018-10-23 17:25:49 -07:00
Yann Collet fab150ce9f fix : removed zstdmt symlink on uninstall 2018-10-22 12:51:56 -07:00
Yann Collet f181799082 fix decodecorpus incorrect frame generation 
fix #1379
decodecorpus was generating one extraneous byte when `nbSeq==0`.
This is disallowed by the specification.

The reference decoder was just skipping the extraneous byte.
It is now stricter, and flag such situation as an error.
 2018-10-20 18:56:21 -07:00