ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,677 Commits 17 Branches 77 Tags
8f27e8cf3de23b87fc3185145316c278f7315488
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 7269fe6cd3 minor code comment update 2018-09-14 16:06:35 -07:00
Yann Collet 5512400677 updated code comments, based on @terrelln review 2018-09-13 16:44:04 -07:00
Yann Collet d195eec97e fixed msan error 
cold dictionary is detected through a comparison with dictEnd,
which was not initialized at the beginning of first DCtx usage.
 2018-09-13 12:29:52 -07:00
Yann Collet 674dd21bd0 final parameter tuning 2018-09-12 17:25:34 -07:00
Yann Collet 419dfd4ea3 clean traces 2018-09-12 16:40:28 -07:00
Yann Collet 2618253da2 fixed PREFETCH() macro 
for corner cases and platforms without this instruction
 2018-09-12 16:15:37 -07:00
Yann Collet 44d3b83bb1 conditional dict content prefetching 
based on nbSeq.
 2018-09-12 15:35:21 -07:00
Yann Collet 5fb5ed3b31 adjust heuristic decisions 2018-09-12 12:32:09 -07:00
Yann Collet c49ccbc8e7 largeNbDicts : can select a nb of blocks 
will automatically truncate or repeat input as needed,
to create the requested nb of blocks.
default: nb of files, eventually increased appropriately if blockSize is set
 2018-09-12 11:31:28 -07:00
Yann Collet 4de344d505 added conditional prefetch 
depending on amount of work to do.
 2018-09-12 10:29:47 -07:00
Yann Collet 63a519dbf6 implemented first prefetch 
based on dictID.
dictContent is prefetched up to 32 KB
(no contentSize adaptation)
 2018-09-11 17:23:44 -07:00
Yann Collet 3675ef4762 added comment about minimum size of FSE tables 
required for DDict creation,
which use this space as workspace during Hufman table building stage.
 2018-09-10 11:24:17 -07:00
Yann Collet f97ca36eab strengthened conditions for using workplace into fse table space 
ensure that the structure layout is as expected.
will trigger an error if it changes in the future.

Another solution would be to use a union,
this would be cleaner and get rid of these static asserts.

However, in order to keep the current code unmodified,
it would be necessary to use an un-named unions.
And apparently, un-named unions are only possible on "recent" compilers (C99+).
 2018-09-06 17:54:13 -07:00
Yann Collet b036b9a177 Merge branch 'islink' into Dworkspace 2018-09-06 17:33:27 -07:00
Yann Collet deb03d8a26 fixed UTIL_isLink() for Mac OS-X 2018-09-06 17:32:16 -07:00
Yann Collet 87406548f0 reduced DDict size, by -2KB 
corresponding to the removal of workspace
which is needed while building huffman table
and is now either present in DCtx,
or temporarily borrowed from available FSE table space.
 2018-09-06 17:07:53 -07:00
Yann Collet 161ecddf3c added code of conduct 
as requested by Facebook OSS program.
 2018-09-06 11:20:39 -07:00
Yann Collet c57a856d64 fixed minor static analyzer warning 2018-09-05 14:33:51 -07:00
Yann Collet 55a8f84a2c spec clarification 
following #1305 comments from @ulikunitz
 2018-09-05 12:31:33 -07:00
Yann Collet 1d487d587f updated documentation 2018-09-04 14:57:45 -07:00
Yann Collet 11b8b8c100 silenced false-positive scan-build warning 2018-08-31 10:01:06 -07:00
Yann Collet 0ff67511e6 fixed link order for old compilers 2018-08-30 16:43:28 -07:00
Yann Collet f76253bb70 minor : createDictionaryBuffer() can create dictionaries of different sizes 2018-08-30 16:24:44 -07:00
Yann Collet 39c55a118f fixed minor compatibility issues with older compilers 2018-08-30 16:00:57 -07:00
Yann Collet 39ef91a599 -std=c99 for largeNbDicts 2018-08-30 14:59:23 -07:00
Yann Collet 4086b2871b largeNbDicts compatible with multiple source files 
splitting is disabled by default, but can be re-enabled using usual command -B#
update commands to look like zstd ones
 2018-08-30 14:38:49 -07:00
Yann Collet a5a77965d3 make all includes contrib/largeNbDicts 2018-08-29 16:17:22 -07:00
Yann Collet d89fa814c1 added a README 
for documentation
 2018-08-28 18:19:19 -07:00
Yann Collet 6444c50035 increases randomness of ddict ptrs 2018-08-28 18:13:46 -07:00
Yann Collet 6c398df241 level, block size and nb dicts can be set on command line 2018-08-28 18:05:31 -07:00
Yann Collet 0c66a44d1b first working test program 
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block

current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size

dictionary can be selected on command line, with -D
 2018-08-28 15:47:07 -07:00
Yann Collet 0491037db9 Merge branch 'bench' into largeNbDicts 2018-08-28 11:26:46 -07:00
Yann Collet 55affc09de timedFn : measurement delay is programmable 
instead of hard-coded 1 second per measurement
 2018-08-28 11:26:27 -07:00
Yann Collet 0ff9b67552 paramgrill: removed useless tests 
designed to compensate iter_mode,
but since only time_mode is available now,
all tests are guaranteed to last a minimum amount of time.
 2018-08-27 19:07:17 -07:00
Yann Collet 9e26893e07 paramgrill: fixed a bunch of div-by-zero 
they were pretty easy to trigger by the way,
just start an extended paramgrill session
to find a compression table based on any sample,
it would necessarily happen at some point.
 2018-08-27 18:47:09 -07:00
Yann Collet 0071e8348f restored assert() in paramgrill 
assert() in paramgrill are not in the benchmark path.
They should remain active, as they don't impact measurements, and their runtime is insignificant.
 2018-08-27 17:52:04 -07:00
Yann Collet 274b60e6e6 largeNbDicts can compress and compare dict vs noDict 2018-08-27 17:08:44 -07:00
Yann Collet d97e92dfad Merge branch 'bench' into largeNbDicts 2018-08-27 12:12:51 -07:00
Yann Collet 01dcd0fd17 bench: minor api update, for consistency 
BMK_benchTimedFn()
BMK_isCompleted_TimedFn() uses TimedFnState
 2018-08-26 21:30:18 -07:00
Yann Collet 6782725155 first sketch for largeNbDicts test program 2018-08-26 19:29:12 -07:00
Yann Collet c3a4baaf6e fixed minor warnings 
valgrind: memory leak of a few bytes in fullbench
static analyzer: uninitialized data passed as result
 2018-08-24 23:25:35 -07:00
Yann Collet 2279f3d127 bench: reduce nb of return type 
runOutcome is enough
removed timedFnOutcome
 2018-08-24 17:28:38 -07:00
Yann Collet 6ce7b08f17 fix minor warnings 
gcc : prototype with 0 parameter must be labelled (void)
visual : const property must be identical in both declaration and implementation
 2018-08-24 15:59:57 -07:00
Yann Collet 7b23cc4d1e fixed fullbench behavior 
now same as v1.3.5
 2018-08-24 12:40:10 -07:00
Yann Collet 4da5bdf482 fixed zstd -b speed result 
the benchmark was displaying the speed of last run
instead of the best of all previous runs.
 2018-08-23 18:13:49 -07:00
Yann Collet b0e1f3982d fixed paramgrill 
to work with new bench.c
 2018-08-23 17:21:38 -07:00
Yann Collet 1f9ec13621 introduced MB_UNIT 
so that all benchmarking programs use the same speed scale
 2018-08-23 16:03:30 -07:00
Yann Collet d39a25c5ed update fullbench.c to work with new bench.h 2018-08-23 15:00:09 -07:00
Yann Collet 2e45badff4 refactored bench.c 
for clarity and safety, especially at interface level
 2018-08-23 14:21:18 -07:00
Yann Collet 77e805e3db bench: changed creation/reset function to timedFnState 
for consistency
 2018-08-21 18:19:27 -07:00
Yann Collet 36d6165a2d Makefile: added variable SCANBUILD 
so that a different version of scan-build can be selected
 2018-08-16 16:44:13 -07:00
Yann Collet 1515f0bb0d fixed more issues detected by recent version of scan-build 
test run on Linux
 2018-08-16 15:20:25 -07:00
Yann Collet 31224cc126 added static analyzer tests to travis CI 2018-08-15 17:44:27 -07:00
Yann Collet 5291d9ac31 fix scope of scan-build tests 
exclude zlib code
 2018-08-15 17:41:44 -07:00
Yann Collet da55865e47 ensure dependency for zlib wrapper 2018-08-15 16:43:13 -07:00
Yann Collet 42a02ab745 fixed minor warnings issued by scan-build 2018-08-15 14:36:02 -07:00
Yann Collet 3692c31598 Merge branch 'dev' into scanbuild 2018-08-15 13:50:49 -07:00
Yann Collet 6e66bbf5dd fixed several minor issues detected by scan-build 
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
 2018-08-14 16:55:35 -07:00
Yann Collet 754942cb79 fixed assert() condition 2018-08-09 15:57:19 -07:00
Yann Collet 79a35ac20d minor code comments improvements 2018-08-09 15:16:31 -07:00
Yann Collet 51e71a5ec7 added zstdgrep documentation 
presenting `zstdgrep` limit regarding dictionary compression
with workaround recommended by @tobwen (#1268)
 2018-08-09 12:28:25 -07:00
Yann Collet d1afd48e44 some errors may only happen when optimizations are enabled 2018-08-08 12:37:08 -07:00
Yann Collet 23ab7875e1 try adding gcc-8 compilation tests 2018-08-08 10:26:54 -07:00
Yann Collet 93750a54a9 try to improve some travis test speed 2018-08-08 10:22:19 -07:00
Yann Collet 5808027abf Merge branch 'dev' into fix1241 2018-08-03 16:08:33 -07:00
Yann Collet 2fdab1629b fix unused variable warning 2018-08-03 08:30:01 -07:00
Yann Collet ca785c4b20 fix .travis.yml 2018-08-03 07:59:33 -07:00
Yann Collet 5203f01774 fix : zstd cli can be built with build macro ZSTD_NOBENCH 
which disables bench.c module
 2018-08-03 07:54:29 -07:00
Yann Collet b002eac69a Merge branch 'dev' into travisTest 2018-08-02 11:53:46 -07:00
Yann Collet 064b8e6785 Merge branch 'travisTest' of github.com:facebook/zstd into travisTest 2018-08-02 11:52:12 -07:00
Yann Collet bbd78df59b add build macro NO_PREFETCH 
prevent usage of prefetch intrinsic commands
which are not supported by c2rust
(see https://github.com/immunant/c2rust/issues/13)
 2018-07-06 17:06:04 -07:00
Yann Collet b5207aadfa make build tests more unforgiving 
`-Werror` will ensure they fail if there is the slightest warning.

fix a minor warning specific to `zstd_decompress` variant.
 2018-06-29 17:10:56 -07:00
Yann Collet 328ec7e34c fix test-zstd can be run with parallel compilation 
fix #1221
 2018-06-29 16:52:21 -07:00
Yann Collet ab1ebd6578 removed \ at end of line 2018-06-27 19:29:18 -07:00
Yann Collet 63c5b7b159 Merge branch 'dev' into travisTest 2018-06-27 19:28:24 -07:00
Yann Collet 1fd621ff6d minor man page update 
regarding advanced parameter `tlen`
which was recently changed.
`0` in association with `ZSTD_fast` now means "normal fast mode".
 2018-06-27 18:49:02 -07:00
Yann Collet ed1a429873 test multi-lines travis yaml file 2018-06-27 16:57:28 -07:00
Yann Collet a55ffbb31b removed unused imaged 2018-06-27 16:03:46 -07:00
Yann Collet c0007893b4 added graph for cdict speed v1.3.5 improvements 2018-06-27 15:58:34 -07:00
Yann Collet 9c277f137c attempt to re-enable arm64 tests 2018-06-27 13:19:14 -07:00
Yann Collet 4489daec09 slightly adjusted default-distribution threshold 
depending on strategy.
fast favors faster compression and decompression speeds.
 2018-06-26 20:10:45 -07:00
Yann Collet ff773bfcde zeroise freq table with memset() 
improves decoding speed by ~5% in github_users sample set
 2018-06-26 17:24:41 -07:00
Yann Collet 7b9bbf77c9 switched to a sizeof() version 
avoid -Werror=unused-variable issue
 2018-06-26 14:08:35 -07:00
Yann Collet f98ec46979 updated DEBUG_STATIC_ASSERT() 
following suggestion from #1209
 2018-06-26 12:04:59 -07:00
Yann Collet 31769ce702 error on no forward progress 
streaming decoders, such as ZSTD_decompressStream() or ZSTD_decompress_generic(),
may end up making no forward progress,
(aka no byte read from input __and__ no byte written to output),
due to unusual parameters conditions,
such as providing an output buffer already full.

In such case, the caller may be caught in an infinite loop,
calling the streaming decompression function again and again,
without making any progress.

This version detects such situation, and generates an error instead :
ZSTD_error_dstSize_tooSmall when output buffer is full,
ZSTD_error_srcSize_wrong when input buffer is empty.

The detection tolerates a number of attempts before triggering an error,
controlled by ZSTD_NO_FORWARD_PROGRESS_MAX macro constant,
which is set to 16 by default, and can be re-defined at compilation time.
This behavior tolerates potentially existing implementations
where such cases happen sporadically, like once or twice,
which is not dangerous (only infinite loops are),
without generating an error, hence without breaking these implementations.
 2018-06-22 17:58:21 -07:00
Yann Collet fbd5dfc1b1 changed POOL_resize() return type to int 
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
 2018-06-22 12:14:59 -07:00
Yann Collet 698fd00afb huf: increase threshold detection of poorly compressible data 2018-06-21 18:32:38 -07:00
Yann Collet c1e6347717 fixed minor typos, detected by @terrelln 2018-06-21 18:08:11 -07:00
Yann Collet 243cd9d8bb add a cond_broadcast after resize 
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
 2018-06-21 18:04:58 -07:00
Yann Collet 7639db939f updated Zstandard frame format 
adding clarifications from IETF RFC DISCUSS.
 2018-06-21 17:55:55 -07:00
Yann Collet 818e72b4d5 added extended POOL test 
abrupt end + downsizing with running jobs remaining in queue.

also : POOL_resize() requires numThreads >= 1
 2018-06-21 14:58:59 -07:00
Yann Collet 7d80ada5ca added a test for POOL (multithreading) 
ensuring all jobs in queue are nonetheless completed
when POOL is instructed to end abruptly (POOL_free())
 2018-06-21 12:24:36 -07:00
Yann Collet 6de249c1c6 fixed: bug when counting nb of active threads 
when queueSize > 1

also : added a test in testpool.c
       verifying resizing is effective.
 2018-06-20 18:28:49 -07:00
Yann Collet 6b48eb12c0 change control of threadLimit 
now limits maximum nb of active threads
even when queueSize > 1.
 2018-06-20 14:35:39 -07:00
Yann Collet 62469c9f41 fixed wrong size in pthread struct transfer 2018-06-19 20:14:03 -07:00
Yann Collet 166901dc72 reduced POOL_resize() restriction 
It's not necessary to ensure that no job is ongoing.
The pool is only expanded, existing threads are preserved.
In case of error, the only option is to return NULL and terminate the thread pool anyway.
 2018-06-19 18:07:18 -07:00
Yann Collet 066fbbfe1c make zstdmt resize its context 
when nbThreads change.

Technically, it only expands.
But when instructed to use less threads,
the thread pool will limit nb of concurrent threads.
 2018-06-19 17:28:56 -07:00
Yann Collet 4567c57199 finalized POOL_resize() 
POOL_ctx* POOL_resize(POOL_ctx* ctx, size_t numThreads)

The function may fail, and returns a NULL pointer in this case.
 2018-06-19 16:03:12 -07:00
Yann Collet 1c714fda3f introduced POOL_resize() 
not complete yet :
finalize behavior in case of unfinished expansion
 2018-06-18 20:46:39 -07:00
Yann Collet c9e8ee93a7 removed specific --opaqueapi test 
from zstreamtest.

This test is now integrated within --newapi,
which dynamically switches between the 2 modes randomly.

The main outcome is reduced testing time.
 2018-06-18 19:20:37 -07:00