ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,197 Commits 17 Branches 77 Tags
92a3e3e9cadc4c22ce0c6da79a8f4124fa6f37be
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 0d212dd22d updated CHANGELOG for v1.3.8 2018-12-22 12:10:29 -08:00
Yann Collet c8d1fda982 update aarch64 test to xenial 
in an attempt to circumvent the `ld` bug
 2018-12-21 15:08:48 -08:00
Yann Collet 18434d76b8 added strerror in comment 
as suggested by @felixhandte
 2018-12-20 17:27:08 -08:00
Yann Collet ed2fb6bd57 fixed : better error message when dictionary missing 
during benchmark.
Also : refactored ZSTD_fillHashTable(),
just for readability (it does the same thing)
 2018-12-20 17:20:07 -08:00
Yann Collet 068c9b89be fixed zlibwrapper examples build 2018-12-20 15:28:03 -08:00
Yann Collet 95784c654c fixed shadowing of stat variable 
some standard lib declares a `stat` variable at global scope
shadowing local declarations ....
 2018-12-20 14:56:44 -08:00
Yann Collet e129174d1d fixed shadowing of variable time 
some standard lib do define `time` as a global variable
shadowing local declarations ...
 2018-12-20 14:54:05 -08:00
Yann Collet 0ed8ee4a37 fixed wrong assert condition 2018-12-20 14:46:23 -08:00
Yann Collet ffba142406 fixed file identity detection in 32-bit mode 
also :
some library decided to use `index` as a global variable declared in standard header
shadowing the ones used in fastcover.c  :(
 2018-12-20 14:30:30 -08:00
Yann Collet 65a441a8f0 fixed stdlib implementation of time functions 
generated redefinitions
 2018-12-20 14:02:50 -08:00
Yann Collet 72dbf1bcd0 removed strncpy() from util.c 
as Visual surprisingly complains about their usage.
Replaced by memcpy()
 2018-12-20 12:27:12 -08:00
Yann Collet 105fa953cb use strerror() to generate error message 
as suggested by @terrelln .

also:
- hopefully fixed Windows version
- changed the test, so that it passes on non-english OS stdlib errors.
 2018-12-20 09:16:40 -08:00
Yann Collet de4eb06a77 fixed 1 transitive include 2018-12-19 18:38:37 -08:00
Yann Collet 173ef9dea2 fixed : detection of non-existing file 
better error message
with test
 2018-12-19 18:30:57 -08:00
Yann Collet 0f2d443e10 added system error strings 
to more failure cases in fileio.c
as requested by fbpkg
 2018-12-19 17:25:58 -08:00
Yann Collet 2898afab52 fixed OSSfuzz 11849 
The problem was already masked,
due to no longer accepting tiny blocks for statistics.

But in case it could still happen with not-so-tiny blocks,
there is a stricter control which ensures that
nothing was already loaded prior to statistics collection.
 2018-12-19 16:54:15 -08:00
Yann Collet 78c4ea4930 added tests case 2018-12-19 14:10:27 -08:00
Yann Collet 8e0e495ce8 fixed: compression ratio discrepancy 
depending on initialization,
the first byte of a new frame was invalidated or not.

As a consequence, one match opportunity was available or not,
resulting in slightly different compressed sizes
(on average, 1 or 2 bytes once every 20 frames).

It impacted ratio comparison between one-shot and streaming modes.

This fix makes the first byte of a new frame always a valid match.
Now compressed size is always the same.
It also improves compressed size by a negligible amount.
 2018-12-19 10:11:06 -08:00
Yann Collet ef984e7307 fix debug levels 
as reported by @terrelln.
2 is reserved for temporary usage only.
 2018-12-18 13:40:07 -08:00
Yann Collet 635783da12 btultra2 and very small srcSize 
When srcSize is small,
the nb of symbols produced is likely too small to warrant dedicated probability tables.
In which case, predefined distribution tables will be used instead.

There is a cheap algorithm in btultra initialization :
it presumes default distribution will be used if srcSize <= 1024.

btultra2 now uses the same threshold to shut down probability estimation,
since measured frequencies won't be used at entropy stage,
and therefore relying on them to determine sequence cost is misleading,
resulting in worse compression ratios.

This fixes btultra2 performance issue on very small input.

Note that, a proper way should be
to determine which symbol is going to use predefined probaility
and which symbol is going to use dynamic ones.
But the current algorithm is unable to make a "per-symbol" decision.
So this will require significant modifications.
 2018-12-18 12:32:58 -08:00
Yann Collet 373ff8b983 play around with rescale weights 2018-12-17 15:48:34 -08:00
Yann Collet 96adc846c5 fixed tests 
with correct pointer type
 2018-12-13 16:50:19 -08:00
Yann Collet 8be145a8c1 fixed default job size 2018-12-13 16:38:08 -08:00
Yann Collet 62180b27d5 zstdmt parameter getter/setter use int 2018-12-13 15:47:34 -08:00
Yann Collet 34f01e600f fixed multiple conversions 
from 64-bit to 32-bit
 2018-12-13 14:02:22 -08:00
Yann Collet 1993f5d412 fixed ovlog tests 
and updated man page
 2018-12-12 21:09:14 -08:00
Yann Collet f2f86d369b Merge branch 'btultra2' into ovlog_def 2018-12-12 20:58:14 -08:00
Yann Collet 9a92ed401d updated compression results.csv 
and fixed nit
 2018-12-12 20:30:09 -08:00
Yann Collet 9792acda3b Merge branch 'dev' into btultra2 2018-12-12 20:18:27 -08:00
Yann Collet c313a85ee2 removed exception code for overlapLog level 22 
CLI used to set overlapLog at value 9 when level == 22.
This is no longer necessary (handled internally within library)
 2018-12-11 18:13:06 -08:00
Yann Collet 7bb8dfc62f new overlapLog default values 
varies between 6 and 9, depending on strategy
 2018-12-11 18:10:29 -08:00
Yann Collet eee789b7ea continued: changed to overlapLog 
in deeper code layer.
for consistency.
 2018-12-11 17:41:42 -08:00
Yann Collet 9b784dec7f changed parameter name to ZSTD_c_overlapLog 
from overlapSizeLog.

Reasoning :
`overlapLog` is already used everwhere, in the code, command line and documentation.
`ZSTD_c_overlapSizeLog` feels unnecessarily different.
 2018-12-11 16:55:33 -08:00
Yann Collet 52b94f902c add clarification for ZSTD_CCtx_setPledgedSrcSize() 
as requested in #1391
 2018-12-11 12:08:21 -08:00
Yann Collet 5e6aaa3abb fixed btultra2 usage with prefix 
notably while using multi-threading
 2018-12-10 18:45:03 -08:00
Yann Collet 3619c34399 fix assert position within ZSTD_compress2() 2018-12-10 17:42:35 -08:00
Yann Collet 5a1e01e5f1 clarified experimentalParam 2018-12-10 17:36:20 -08:00
Yann Collet c226a7b9f3 fixed ZSTD_compress2() 
as suggested by @terrelln
 2018-12-10 17:33:49 -08:00
Yann Collet ea441ee4ae make test can accept DEBUGLEVEL argument 2018-12-09 23:20:51 -08:00
Yann Collet 37e314a68d updated clevel table for large inputs 2018-12-09 22:38:05 -08:00
Yann Collet c9c4c7ec8c update clevel table for 256K 2018-12-08 21:40:08 -08:00
Yann Collet 8075d75f9c update clevel table for 128K 2018-12-08 10:42:55 -08:00
Yann Collet 95b152ab33 updated clevel table for 16K 
to introduce btultra2
 2018-12-07 20:12:43 -08:00
Yann Collet b71bfb6cf2 paramgrill: add status line 
get information on which config is currently tested
so that console get animated during long tests.
 2018-12-07 16:02:24 -08:00
Yann Collet 27b253fadc added tests for strategy=9 (btultra2) 2018-12-07 14:20:54 -08:00
Yann Collet e68c2d86e7 refactor paramgrill for clarity 
restored ability to copy/paste the resulting compression level table into zstd_compress.c .
 2018-12-07 14:07:54 -08:00
Yann Collet d613fd9afe linked btultra2 as strategy9 
and ensure zstdbench detects out-of-bound parameters
 2018-12-06 19:27:37 -08:00
Yann Collet 34aa401afd updated documentation 
introducing ZSTD_btultra2
 2018-12-06 17:22:19 -08:00
Yann Collet ae370b0e12 minor bound refinements 2018-12-06 16:51:17 -08:00
Yann Collet 39e28982cf introduced constants ZSTD_STRATEGY_MIN and ZSTD_STRATEGY_MAX 2018-12-06 16:16:16 -08:00
Yann Collet c3c3488981 fixed c++ assignment to enum 2018-12-06 15:57:55 -08:00
Yann Collet be9e561da4 changed ZSTD_c_compressionStrategy into ZSTD_c_strategy 
also : fixed paramgrill, and limit conditions
 2018-12-06 15:00:52 -08:00
Yann Collet e9448cdf4c introduced strategy btultra2 
note : not yet applied on any compression level
 2018-12-06 13:38:09 -08:00
Yann Collet 0c404a48f0 moved ZSTD_WINDOWLOG_LIMIT_DEFAULT into static-linking-only area 2018-12-06 10:57:19 -08:00
Yann Collet 96d887429b clarified usage of word "job" 
only applies in MT / async context now.
 2018-12-06 10:14:34 -08:00
Yann Collet 3583d19c4e changed parameter names from ZSTD_p_* to ZSTD_c_* 
for naming consistency
 2018-12-05 17:26:02 -08:00
Yann Collet c2053310e5 updated API documentation 2018-12-05 16:23:00 -08:00
Yann Collet 3e042d5cc0 ZSTD_decompressDCtx() is compatible with sticky parameters 2018-12-04 17:30:58 -08:00
Yann Collet d7da3fc90a merge dedicated dParam setters 2018-12-04 17:06:48 -08:00
Yann Collet 4b5a4f02d7 write the switch()case: differently 
so that it please both
compilers which warn for dead code after the switch
and
compilers which do not detect that all branches terminate.
 2018-12-04 16:59:26 -08:00
Yann Collet 85b02bf142 fixed silent conversion warning 2018-12-04 15:57:16 -08:00
Yann Collet 2fb8d1a392 fixed declaration-after-statement warnings 2018-12-04 15:54:01 -08:00
Yann Collet aec945f0dc implemented ZSTD_dParam_getBounds() 
and ZSTD_DCtx_setParameter()
 2018-12-04 15:35:37 -08:00
Yann Collet 34e146f548 advanced decompression function replaces by normal streaming one 
advanced parameters compatible with ZSTD_decompressStream().
 2018-12-04 10:28:36 -08:00
Yann Collet 7ef7dc561a check availability of --color=never command on grep and egrep 
before applying them.
Fixes #1436
 2018-12-03 15:46:55 -08:00
Yann Collet 6ced8f7c7c joined normal streaming API with advanced one 2018-12-03 14:22:38 -08:00
Yann Collet da1f3066a3 preparative for ZSTD_DCtx_setParameter() 2018-11-30 15:59:50 -08:00
Yann Collet d8e215cbee created ZSTD_compress2() and ZSTD_compressStream2() 
ZSTD_compress_generic() is renamed ZSTD_compressStream2().

Note that, for the time being,
the "stable" API and advanced one use different parameter planes :
setting parameters using the advanced API does not influence ZSTD_compressStream()
and using ZSTD_initCStream() does not influence parameters for ZSTD_compressStream2().
 2018-11-30 11:25:56 -08:00
Yann Collet d3a0c71259 pushed experimental parameters 
into ZSTD_STATIC_LINKING_ONLY section
 2018-11-21 16:18:55 -08:00
Yann Collet d4d4e109e9 getParameter fills an int* 
rather than an unsigned*
for consistency
since type of setParameter() changed to int.
 2018-11-21 15:37:26 -08:00
Yann Collet fea920615c promote ZSTD_findFrameCompressedSize() into staging area 2018-11-21 15:25:50 -08:00
Yann Collet 41c7d0b1e1 changed hashEveryLog into hashRateLog 2018-11-21 14:36:57 -08:00
Yann Collet 5d3592398d fixed fall-through 2018-11-20 16:09:33 -08:00
Yann Collet 5c6d4b18ac completed implementation of ZSTD_cParam_getBounds() 
for all parameters
 2018-11-20 16:06:00 -08:00
Yann Collet 2e7fd6a2cb fixed remaining searchLength invocations 2018-11-20 15:13:27 -08:00
Yann Collet e874dacc08 changed searchLength into minMatch 
refactored all relevant API and calls
for consistency.
 2018-11-20 14:56:07 -08:00
Yann Collet 114bd4346e changed enum type name to ZSTD_ResetDirective 
for naming consistency :
types should start with a capital letter (after prefix)
 2018-11-20 12:00:20 -08:00
Yann Collet 3b838abf97 ZSTD_CCtx_setParameter : value argument is now int 
for compatibility with compression level
 2018-11-20 11:53:01 -08:00
Yann Collet 19e5f2a35b removed some constants and _simpleArgs() from staging 
constants that *may* change in the future
will be accessed through functions instead
(to be created).

_simpleArgs() variant do not have (yet) a clear enough added value
to deserve "stable" status.
 2018-11-19 17:38:15 -08:00
Yann Collet f07e13b729 fixed fuzz src 2018-11-15 16:53:45 -08:00
Yann Collet 5c68639186 updated ZSTD_DCtx_reset() 
signature and behavior is now the same as ZSTD_CCtx_reset()
 2018-11-15 16:12:39 -08:00
Yann Collet 4542b9f1cf updated manual 2018-11-15 10:52:24 -08:00
Yann Collet 06c8d5a4f4 Merge branch 'dev' into advancedAPI 
fixed rsyncable
 2018-11-15 10:51:24 -08:00
Yann Collet 21a42bf5f9 added advanced decompression api 2018-11-14 16:54:54 -08:00
Yann Collet cf9f4b63b8 fixed fuzz test src code 2018-11-14 14:46:49 -08:00
Yann Collet 7b0391e37e finalized retrofit of ZSTD_CCtx_reset() 
updated all depending sources
 2018-11-14 13:05:35 -08:00
Yann Collet 5adbad4059 Merge branch 'dev' into advancedAPI 2018-11-14 13:00:37 -08:00
Yann Collet ff8d371708 modified ZSTD_CCtx_reset() 
which now accepts an enum,
to distinguish between resetting the session, or the parameters (or both).

removed ZSTD_CCtx_resetParameters(), which is redundant.

start replacing invocation of ZSTD_CCtx_reset*() functions

Updated advanced API documentation

trimmed down amount of API staged in RC,
in particular, all functions related to ZSTD_CCtxParams()
seem too advanced.
 2018-11-14 12:33:57 -08:00
Yann Collet d7e10a774a added constant ZSTD_WINDOWLOG_LIMIT_DEFAULT 
answering #1407.

Also : removed obsolete function ZSTD_setDStreamParameter()
which could only be used with one parameter (DStream_p_maxWindowSize).
Now replaced by ZSTD_DCtx_setWindowSize() (which exists since a few revisions)
 2018-11-13 18:12:34 -08:00
Yann Collet 2c8fde538f added constant ZSTD_MAGIC_SKIPPABLE_MASK 
and updated several API comments
 2018-11-13 17:36:35 -08:00
Yann Collet b83d1e7714 removed some static const variables 
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
 2018-11-13 16:56:32 -08:00
Yann Collet 975abf8f5d circumvent gcc-8 strict aliasing warning 
because gcc considers that
`void* const *`  and `void**` are 2 different types ...
 2018-11-13 16:04:35 -08:00
Yann Collet 768a264200 Merge branch 'dev' of github.com:facebook/zstd into dev 2018-11-13 15:56:36 -08:00
Yann Collet 092c4abd4c bumped version number to v1.3.8 2018-11-13 15:53:38 -08:00
Yann Collet 3ba0d6dd27 fixed decode-only test condition 2018-11-13 14:15:12 -08:00
Yann Collet a29e3d1d01 fix debug trace 2018-11-13 13:27:22 -08:00
Yann Collet e4abd28769 fixed VS2010+ build script 2018-11-13 13:22:16 -08:00
Yann Collet f15312e884 fixed cmake build script 2018-11-13 13:16:41 -08:00
Yann Collet b830ccca5c changed benchfn api 
to use structure for function parameters
as it expresses much clearer than a long list of parameters,
since each parameter can now be named.
 2018-11-13 13:12:50 -08:00
Yann Collet 9867cdb847 benchfn can provided faulty return value 
with BMK_extract_returnValue()
 2018-11-13 12:01:17 -08:00