ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,638 Commits 17 Branches 77 Tags
112c18d6d5ad36e1aa8a912f7b3e73dfb7cb1974
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet deb03d8a26 fixed UTIL_isLink() for Mac OS-X 2018-09-06 17:32:16 -07:00
Yann Collet 161ecddf3c added code of conduct 
as requested by Facebook OSS program.
 2018-09-06 11:20:39 -07:00
Yann Collet c57a856d64 fixed minor static analyzer warning 2018-09-05 14:33:51 -07:00
Yann Collet 1d487d587f updated documentation 2018-09-04 14:57:45 -07:00
Yann Collet 11b8b8c100 silenced false-positive scan-build warning 2018-08-31 10:01:06 -07:00
Yann Collet 0ff67511e6 fixed link order for old compilers 2018-08-30 16:43:28 -07:00
Yann Collet f76253bb70 minor : createDictionaryBuffer() can create dictionaries of different sizes 2018-08-30 16:24:44 -07:00
Yann Collet 39c55a118f fixed minor compatibility issues with older compilers 2018-08-30 16:00:57 -07:00
Yann Collet 39ef91a599 -std=c99 for largeNbDicts 2018-08-30 14:59:23 -07:00
Yann Collet 4086b2871b largeNbDicts compatible with multiple source files 
splitting is disabled by default, but can be re-enabled using usual command -B#
update commands to look like zstd ones
 2018-08-30 14:38:49 -07:00
Yann Collet a5a77965d3 make all includes contrib/largeNbDicts 2018-08-29 16:17:22 -07:00
Yann Collet d89fa814c1 added a README 
for documentation
 2018-08-28 18:19:19 -07:00
Yann Collet 6444c50035 increases randomness of ddict ptrs 2018-08-28 18:13:46 -07:00
Yann Collet 6c398df241 level, block size and nb dicts can be set on command line 2018-08-28 18:05:31 -07:00
Yann Collet 0c66a44d1b first working test program 
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block

current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size

dictionary can be selected on command line, with -D
 2018-08-28 15:47:07 -07:00
Yann Collet 0491037db9 Merge branch 'bench' into largeNbDicts 2018-08-28 11:26:46 -07:00
Yann Collet 55affc09de timedFn : measurement delay is programmable 
instead of hard-coded 1 second per measurement
 2018-08-28 11:26:27 -07:00
Yann Collet 0ff9b67552 paramgrill: removed useless tests 
designed to compensate iter_mode,
but since only time_mode is available now,
all tests are guaranteed to last a minimum amount of time.
 2018-08-27 19:07:17 -07:00
Yann Collet 9e26893e07 paramgrill: fixed a bunch of div-by-zero 
they were pretty easy to trigger by the way,
just start an extended paramgrill session
to find a compression table based on any sample,
it would necessarily happen at some point.
 2018-08-27 18:47:09 -07:00
Yann Collet 0071e8348f restored assert() in paramgrill 
assert() in paramgrill are not in the benchmark path.
They should remain active, as they don't impact measurements, and their runtime is insignificant.
 2018-08-27 17:52:04 -07:00
Yann Collet 274b60e6e6 largeNbDicts can compress and compare dict vs noDict 2018-08-27 17:08:44 -07:00
Yann Collet d97e92dfad Merge branch 'bench' into largeNbDicts 2018-08-27 12:12:51 -07:00
Yann Collet 01dcd0fd17 bench: minor api update, for consistency 
BMK_benchTimedFn()
BMK_isCompleted_TimedFn() uses TimedFnState
 2018-08-26 21:30:18 -07:00
Yann Collet 6782725155 first sketch for largeNbDicts test program 2018-08-26 19:29:12 -07:00
Yann Collet c3a4baaf6e fixed minor warnings 
valgrind: memory leak of a few bytes in fullbench
static analyzer: uninitialized data passed as result
 2018-08-24 23:25:35 -07:00
Yann Collet 2279f3d127 bench: reduce nb of return type 
runOutcome is enough
removed timedFnOutcome
 2018-08-24 17:28:38 -07:00
Yann Collet 6ce7b08f17 fix minor warnings 
gcc : prototype with 0 parameter must be labelled (void)
visual : const property must be identical in both declaration and implementation
 2018-08-24 15:59:57 -07:00
Yann Collet 7b23cc4d1e fixed fullbench behavior 
now same as v1.3.5
 2018-08-24 12:40:10 -07:00
Yann Collet 4da5bdf482 fixed zstd -b speed result 
the benchmark was displaying the speed of last run
instead of the best of all previous runs.
 2018-08-23 18:13:49 -07:00
Yann Collet b0e1f3982d fixed paramgrill 
to work with new bench.c
 2018-08-23 17:21:38 -07:00
Yann Collet 1f9ec13621 introduced MB_UNIT 
so that all benchmarking programs use the same speed scale
 2018-08-23 16:03:30 -07:00
Yann Collet d39a25c5ed update fullbench.c to work with new bench.h 2018-08-23 15:00:09 -07:00
Yann Collet 2e45badff4 refactored bench.c 
for clarity and safety, especially at interface level
 2018-08-23 14:21:18 -07:00
Yann Collet 77e805e3db bench: changed creation/reset function to timedFnState 
for consistency
 2018-08-21 18:19:27 -07:00
Yann Collet 36d6165a2d Makefile: added variable SCANBUILD 
so that a different version of scan-build can be selected
 2018-08-16 16:44:13 -07:00
Yann Collet 1515f0bb0d fixed more issues detected by recent version of scan-build 
test run on Linux
 2018-08-16 15:20:25 -07:00
Yann Collet 31224cc126 added static analyzer tests to travis CI 2018-08-15 17:44:27 -07:00
Yann Collet 5291d9ac31 fix scope of scan-build tests 
exclude zlib code
 2018-08-15 17:41:44 -07:00
Yann Collet da55865e47 ensure dependency for zlib wrapper 2018-08-15 16:43:13 -07:00
Yann Collet 42a02ab745 fixed minor warnings issued by scan-build 2018-08-15 14:36:02 -07:00
Yann Collet 3692c31598 Merge branch 'dev' into scanbuild 2018-08-15 13:50:49 -07:00
Yann Collet 6e66bbf5dd fixed several minor issues detected by scan-build 
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
 2018-08-14 16:55:35 -07:00
Yann Collet 754942cb79 fixed assert() condition 2018-08-09 15:57:19 -07:00
Yann Collet 79a35ac20d minor code comments improvements 2018-08-09 15:16:31 -07:00
Yann Collet 51e71a5ec7 added zstdgrep documentation 
presenting `zstdgrep` limit regarding dictionary compression
with workaround recommended by @tobwen (#1268)
 2018-08-09 12:28:25 -07:00
Yann Collet d1afd48e44 some errors may only happen when optimizations are enabled 2018-08-08 12:37:08 -07:00
Yann Collet 23ab7875e1 try adding gcc-8 compilation tests 2018-08-08 10:26:54 -07:00
Yann Collet 93750a54a9 try to improve some travis test speed 2018-08-08 10:22:19 -07:00
Yann Collet 5808027abf Merge branch 'dev' into fix1241 2018-08-03 16:08:33 -07:00
Yann Collet 2fdab1629b fix unused variable warning 2018-08-03 08:30:01 -07:00
Yann Collet ca785c4b20 fix .travis.yml 2018-08-03 07:59:33 -07:00
Yann Collet 5203f01774 fix : zstd cli can be built with build macro ZSTD_NOBENCH 
which disables bench.c module
 2018-08-03 07:54:29 -07:00
Yann Collet b002eac69a Merge branch 'dev' into travisTest 2018-08-02 11:53:46 -07:00
Yann Collet 064b8e6785 Merge branch 'travisTest' of github.com:facebook/zstd into travisTest 2018-08-02 11:52:12 -07:00
Yann Collet bbd78df59b add build macro NO_PREFETCH 
prevent usage of prefetch intrinsic commands
which are not supported by c2rust
(see https://github.com/immunant/c2rust/issues/13)
 2018-07-06 17:06:04 -07:00
Yann Collet b5207aadfa make build tests more unforgiving 
`-Werror` will ensure they fail if there is the slightest warning.

fix a minor warning specific to `zstd_decompress` variant.
 2018-06-29 17:10:56 -07:00
Yann Collet 328ec7e34c fix test-zstd can be run with parallel compilation 
fix #1221
 2018-06-29 16:52:21 -07:00
Yann Collet ab1ebd6578 removed \ at end of line 2018-06-27 19:29:18 -07:00
Yann Collet 63c5b7b159 Merge branch 'dev' into travisTest 2018-06-27 19:28:24 -07:00
Yann Collet 1fd621ff6d minor man page update 
regarding advanced parameter `tlen`
which was recently changed.
`0` in association with `ZSTD_fast` now means "normal fast mode".
 2018-06-27 18:49:02 -07:00
Yann Collet ed1a429873 test multi-lines travis yaml file 2018-06-27 16:57:28 -07:00
Yann Collet a55ffbb31b removed unused imaged 2018-06-27 16:03:46 -07:00
Yann Collet c0007893b4 added graph for cdict speed v1.3.5 improvements 2018-06-27 15:58:34 -07:00
Yann Collet 9c277f137c attempt to re-enable arm64 tests 2018-06-27 13:19:14 -07:00
Yann Collet 4489daec09 slightly adjusted default-distribution threshold 
depending on strategy.
fast favors faster compression and decompression speeds.
 2018-06-26 20:10:45 -07:00
Yann Collet ff773bfcde zeroise freq table with memset() 
improves decoding speed by ~5% in github_users sample set
 2018-06-26 17:24:41 -07:00
Yann Collet 7b9bbf77c9 switched to a sizeof() version 
avoid -Werror=unused-variable issue
 2018-06-26 14:08:35 -07:00
Yann Collet f98ec46979 updated DEBUG_STATIC_ASSERT() 
following suggestion from #1209
 2018-06-26 12:04:59 -07:00
Yann Collet 31769ce702 error on no forward progress 
streaming decoders, such as ZSTD_decompressStream() or ZSTD_decompress_generic(),
may end up making no forward progress,
(aka no byte read from input __and__ no byte written to output),
due to unusual parameters conditions,
such as providing an output buffer already full.

In such case, the caller may be caught in an infinite loop,
calling the streaming decompression function again and again,
without making any progress.

This version detects such situation, and generates an error instead :
ZSTD_error_dstSize_tooSmall when output buffer is full,
ZSTD_error_srcSize_wrong when input buffer is empty.

The detection tolerates a number of attempts before triggering an error,
controlled by ZSTD_NO_FORWARD_PROGRESS_MAX macro constant,
which is set to 16 by default, and can be re-defined at compilation time.
This behavior tolerates potentially existing implementations
where such cases happen sporadically, like once or twice,
which is not dangerous (only infinite loops are),
without generating an error, hence without breaking these implementations.
 2018-06-22 17:58:21 -07:00
Yann Collet fbd5dfc1b1 changed POOL_resize() return type to int 
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
 2018-06-22 12:14:59 -07:00
Yann Collet 698fd00afb huf: increase threshold detection of poorly compressible data 2018-06-21 18:32:38 -07:00
Yann Collet c1e6347717 fixed minor typos, detected by @terrelln 2018-06-21 18:08:11 -07:00
Yann Collet 243cd9d8bb add a cond_broadcast after resize 
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
 2018-06-21 18:04:58 -07:00
Yann Collet 7639db939f updated Zstandard frame format 
adding clarifications from IETF RFC DISCUSS.
 2018-06-21 17:55:55 -07:00
Yann Collet 818e72b4d5 added extended POOL test 
abrupt end + downsizing with running jobs remaining in queue.

also : POOL_resize() requires numThreads >= 1
 2018-06-21 14:58:59 -07:00
Yann Collet 7d80ada5ca added a test for POOL (multithreading) 
ensuring all jobs in queue are nonetheless completed
when POOL is instructed to end abruptly (POOL_free())
 2018-06-21 12:24:36 -07:00
Yann Collet 6de249c1c6 fixed: bug when counting nb of active threads 
when queueSize > 1

also : added a test in testpool.c
       verifying resizing is effective.
 2018-06-20 18:28:49 -07:00
Yann Collet 6b48eb12c0 change control of threadLimit 
now limits maximum nb of active threads
even when queueSize > 1.
 2018-06-20 14:35:39 -07:00
Yann Collet 62469c9f41 fixed wrong size in pthread struct transfer 2018-06-19 20:14:03 -07:00
Yann Collet 166901dc72 reduced POOL_resize() restriction 
It's not necessary to ensure that no job is ongoing.
The pool is only expanded, existing threads are preserved.
In case of error, the only option is to return NULL and terminate the thread pool anyway.
 2018-06-19 18:07:18 -07:00
Yann Collet 066fbbfe1c make zstdmt resize its context 
when nbThreads change.

Technically, it only expands.
But when instructed to use less threads,
the thread pool will limit nb of concurrent threads.
 2018-06-19 17:28:56 -07:00
Yann Collet 4567c57199 finalized POOL_resize() 
POOL_ctx* POOL_resize(POOL_ctx* ctx, size_t numThreads)

The function may fail, and returns a NULL pointer in this case.
 2018-06-19 16:03:12 -07:00
Yann Collet 1c714fda3f introduced POOL_resize() 
not complete yet :
finalize behavior in case of unfinished expansion
 2018-06-18 20:46:39 -07:00
Yann Collet c9e8ee93a7 removed specific --opaqueapi test 
from zstreamtest.

This test is now integrated within --newapi,
which dynamically switches between the 2 modes randomly.

The main outcome is reduced testing time.
 2018-06-18 19:20:37 -07:00
Yann Collet d8462ecba2 Merge branch 'dev' into huf_rename 2018-06-14 20:42:10 -04:00
Yann Collet b7e5ebef2a grouped X2 function together 2018-06-14 20:41:50 -04:00
Yann Collet 507bef196d added debug.h for cmake 2018-06-14 19:48:40 -04:00
Yann Collet 6901c94cd6 avoid duplicate code comments 
when a function is decribed in hist.h,
do not describe it again in hist.c
to avoid future doc synchronization issues.
 2018-06-14 19:47:05 -04:00
Yann Collet 78e5e2887e Visual Studio project blind fix 2018-06-14 16:30:52 -04:00
Yann Collet 1adf84ccb7 renamed all HUF_decompress*X4*() functions into *X2 
to underline they generate up to 2 symbols per decoding,
in preparation for a future *X3 variant.
 2018-06-14 15:17:03 -04:00
Yann Collet a09af5eb6b renamed all HUF_decompress*X2*() functions into *X1 
to underline they generate one symbol per decoding operation.

The new naming scheme will make it easier to introduce an *X3 variant.
 2018-06-14 15:08:43 -04:00
Yann Collet 7fee966f02 fix dctx initialization within ZSTD_decompress in stack mode 
when ZSTD_HEAPMODE=0 (which is not default).

Also : added an associated test (test-fuzzer-stackmode)
run on travis CI

fix #1186
 2018-06-14 10:22:24 -04:00
Yann Collet fc682263d0 fixed g_debuglevel variable name 
in debug.h
 2018-06-13 20:02:33 -04:00
Yann Collet 2d76defbfe grouped all histogram functions into hist.c 
renamed functions with HIST_* prefix
 2018-06-13 19:49:31 -04:00
Yann Collet fa41bcc2c2 grouped debug functions into debug.h 
There were 2 competing set of debug functions
within zstd_internal.h and bitstream.h.
They were mostly duplicate, and required care to avoid messing with each other.

There is now a single implementation, shared by both.

Significant change :
The macro variable ZSTD_DEBUG does no longer exist,
it has been replaced by DEBUGLEVEL,
which required modifying several source files.
 2018-06-13 15:43:09 -04:00
Yann Collet 56961e4ced fixed minor conversion warning 2018-06-07 16:59:33 -07:00
Yann Collet 0b1833c2cb fixed regressiontest 
ZSTD_TARGETLEN_MIN no longer exists
since is would be tautological to check if an unsigned value is < 0.
 2018-06-07 16:07:46 -07:00
Yann Collet 3050733042 Merge branch 'dev' into negLevels 2018-06-07 15:51:35 -07:00
Yann Collet c2c47e24e0 support targetlen==0 with strategy==ZSTD_fast 
to mean "normal compression",
targetlen >= 1 now means "disable huffman compression of literals"
 2018-06-07 15:49:01 -07:00
Yann Collet a57b4df85f removed literalCompression directive 
in this version, literal compression is always disabled for ZSTD_fast strategy.

Performance parity between ZSTD_compress_advanced() and ZSTD_compress_generic()
 2018-06-07 15:24:12 -07:00