ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,751 Commits 17 Branches 77 Tags
a180ea07c4d945bf9400a70494bf458841a44437
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 404a7bfed0 moved again overflow correction 
cannot work from within ZSTD_compressBlock()
 2018-09-26 18:06:53 -07:00
Yann Collet 0e2dbac18a changed overflow correction place 
keep one in compress_frameChunk(),
so that it's tested at every loop
in case some user simply some large mulit-GB input in a single invocation.

Add one in ZSTD_compressBlock(),
since compressBlock() explicitly skips frameChunk().
 2018-09-26 15:35:38 -07:00
Yann Collet f98c69d77c fix : huge (>4GB) stream of blocks 
experimental function ZSTD_compressBlock() is designed for very small data in mind,
for situation where saving the ~12 bytes of frame header can actually make a difference.

Some systems though may have to deal with small and large data entangled.
If it's larger than a block (> 128KB), compressBlock() cannot compress them in one round.

That's why it's possible to compress in multiple rounds.
This is a chain of compressed blocks.

Some users push this capability to the limit, encoding gigantic chain of blocks.
On crossing the 4GB limit, some internal overflow occurs.

This fix moves the overflow correction mechanism higher in the call chain,
so that it's applied also to gigantic chains of blocks.

Added a test case in fuzzer.c, which crashes before the fix, and pass now.
 2018-09-26 14:24:28 -07:00
Yann Collet 08f68d83c5 fixed usage of grep in Makefile 
when terminal uses colors
as suggested by @danielshir (#1294)
 2018-09-25 16:56:53 -07:00
Yann Collet 72a3adf826 updated format documentation 
to match last edits of RFC8478.
 2018-09-25 16:34:26 -07:00
Yann Collet 6c51bf420c bounds for --adapt mode 
can supply min and max compression level through advanced command :
--adapt=min=#,max=#
 2018-09-25 16:03:28 -07:00
Yann Collet 63abaf2171 fixed ! tests 
Sometimes, it's necessary to test that a certain command fail, as expected.
Such failure is actually a success, and must not stop the flow of tests.

Several tests were prefixed with `!` to invert return code.
This does not work : it effectively makes the tests pass no matter what.

Use instead function die(), which is meant to trap successes, and transform them into errors.
 2018-09-25 15:57:28 -07:00
Yann Collet 04f47bbdd2 Merge branch 'dev' into adapt 2018-09-24 16:56:45 -07:00
Yann Collet 0250ac74ce fixed minor scan-build warnings 2018-09-24 00:52:19 -07:00
Yann Collet 292d8e4a83 added some tests based on limits.h 
in order to ensure proper type mapping
when not using stdint.h
 2018-09-23 23:57:30 -07:00
Yann Collet 0fc07eb1fd fixed zstd-decompress 
which cannot support ZSTD_minCLevel()
 2018-09-22 17:21:39 -07:00
Yann Collet 364041c6dd enforce minimum compression level limit 
using ZSTD_minCLevel()
 2018-09-22 16:10:10 -07:00
Yann Collet 0e211bdd18 fixed constant comparison on 32-bits systems 2018-09-21 18:23:32 -07:00
Yann Collet 71a5210617 avoid recompiling dll every time under mingw 2018-09-21 17:40:30 -07:00
Yann Collet 123fac6b6d fix pzstd compatibility with mingw 
some details changed with introduction of gcc7
 2018-09-21 17:36:00 -07:00
Yann Collet 484f40697b fix constant redeclaration in paramgrill 2018-09-21 17:28:37 -07:00
Yann Collet 00c18c0c88 simplified "slows down when compression blocked" 2018-09-21 16:35:43 -07:00
Yann Collet c484345a82 Merge branch 'mingw' into adapt 2018-09-21 16:00:46 -07:00
Yann Collet bfff4f4809 ensure all writes to job->cSize are mutex protected 
even when reporting errors,
using a macro for code brevity, as suggested by @terrelln,
 2018-09-21 16:00:39 -07:00
Yann Collet 32b7cf1bcf fixed tautological tests 
involving ZSTD_TARGETLENGTH_MIN (== 0)
 2018-09-21 15:04:43 -07:00
Yann Collet c044345f8f Merge branch 'mingw' into minclevel 2018-09-21 14:56:57 -07:00
Yann Collet 54001f3dbc fix mingw compatibility 
only enable backtraces for platforms we know support it
aka mac OS-X and Linux.
can be extended later.
 2018-09-21 14:46:09 -07:00
Yann Collet a54c86cfc6 defined a minimum negative level 
which can be probed using new function ZSTD_minCLevel().

Also : redefined ZSTD_TARGETLENGTH_MIN/MAX for consistency

used the opportunity to bump version number to v1.3.6
 2018-09-20 16:52:03 -07:00
Yann Collet db97310ace fixed versions-test to only test v0.5+ 
since zstd_devel is no longer compatible with v0.4+
 2018-09-20 14:59:11 -07:00
Yann Collet b2939163e1 Changed default legacy support to v0.5+ 
thus dropping read support for v0.4.

It's always possible to re-enable it, by changing build macro ZSTD_LEGACY_SUPPORT to 4.
 2018-09-20 14:30:20 -07:00
Yann Collet 7992942d66 fixed complex tsan issue 
when job->consumed == job->src.size , compression job is presumed completed,
so it must be the very last action done in worker thread.
 2018-09-20 13:47:31 -07:00
Yann Collet 15519479ba fixed minor gcc warning on a unused variable 2018-09-20 13:00:11 -07:00
Yann Collet 45010da074 updated man page 
and added `--adapt` test in `playTests.sh`
 2018-09-19 17:37:22 -07:00
Yann Collet 6b07a66aec fixed minor reporting discrepancy in MT mode 2018-09-19 16:30:55 -07:00
Yann Collet ca02ebee07 removed static variables 
so that --adapt can work on multiple input files too
 2018-09-19 15:25:50 -07:00
Yann Collet 89bc309d90 error out when --adapt is associated with --single-thread 
since they are not compatible
 2018-09-19 14:49:13 -07:00
Yann Collet 2f78228f65 Merge branch 'dev' into adapt 2018-09-19 12:43:42 -07:00
Yann Collet 005f000aed updated documentation of *refPrefix() 
indicating the equivalence with `diff` operation.
 2018-09-18 13:07:08 -07:00
Yann Collet 7269fe6cd3 minor code comment update 2018-09-14 16:06:35 -07:00
Yann Collet 5512400677 updated code comments, based on @terrelln review 2018-09-13 16:44:04 -07:00
Yann Collet d195eec97e fixed msan error 
cold dictionary is detected through a comparison with dictEnd,
which was not initialized at the beginning of first DCtx usage.
 2018-09-13 12:29:52 -07:00
Yann Collet 674dd21bd0 final parameter tuning 2018-09-12 17:25:34 -07:00
Yann Collet 419dfd4ea3 clean traces 2018-09-12 16:40:28 -07:00
Yann Collet 2618253da2 fixed PREFETCH() macro 
for corner cases and platforms without this instruction
 2018-09-12 16:15:37 -07:00
Yann Collet 44d3b83bb1 conditional dict content prefetching 
based on nbSeq.
 2018-09-12 15:35:21 -07:00
Yann Collet 5fb5ed3b31 adjust heuristic decisions 2018-09-12 12:32:09 -07:00
Yann Collet c49ccbc8e7 largeNbDicts : can select a nb of blocks 
will automatically truncate or repeat input as needed,
to create the requested nb of blocks.
default: nb of files, eventually increased appropriately if blockSize is set
 2018-09-12 11:31:28 -07:00
Yann Collet 4de344d505 added conditional prefetch 
depending on amount of work to do.
 2018-09-12 10:29:47 -07:00
Yann Collet 63a519dbf6 implemented first prefetch 
based on dictID.
dictContent is prefetched up to 32 KB
(no contentSize adaptation)
 2018-09-11 17:23:44 -07:00
Yann Collet 3675ef4762 added comment about minimum size of FSE tables 
required for DDict creation,
which use this space as workspace during Hufman table building stage.
 2018-09-10 11:24:17 -07:00
Yann Collet f97ca36eab strengthened conditions for using workplace into fse table space 
ensure that the structure layout is as expected.
will trigger an error if it changes in the future.

Another solution would be to use a union,
this would be cleaner and get rid of these static asserts.

However, in order to keep the current code unmodified,
it would be necessary to use an un-named unions.
And apparently, un-named unions are only possible on "recent" compilers (C99+).
 2018-09-06 17:54:13 -07:00
Yann Collet b036b9a177 Merge branch 'islink' into Dworkspace 2018-09-06 17:33:27 -07:00
Yann Collet deb03d8a26 fixed UTIL_isLink() for Mac OS-X 2018-09-06 17:32:16 -07:00
Yann Collet 87406548f0 reduced DDict size, by -2KB 
corresponding to the removal of workspace
which is needed while building huffman table
and is now either present in DCtx,
or temporarily borrowed from available FSE table space.
 2018-09-06 17:07:53 -07:00
Yann Collet 161ecddf3c added code of conduct 
as requested by Facebook OSS program.
 2018-09-06 11:20:39 -07:00
Yann Collet c57a856d64 fixed minor static analyzer warning 2018-09-05 14:33:51 -07:00
Yann Collet 55a8f84a2c spec clarification 
following #1305 comments from @ulikunitz
 2018-09-05 12:31:33 -07:00
Yann Collet 1d487d587f updated documentation 2018-09-04 14:57:45 -07:00
Yann Collet 11b8b8c100 silenced false-positive scan-build warning 2018-08-31 10:01:06 -07:00
Yann Collet 0ff67511e6 fixed link order for old compilers 2018-08-30 16:43:28 -07:00
Yann Collet f76253bb70 minor : createDictionaryBuffer() can create dictionaries of different sizes 2018-08-30 16:24:44 -07:00
Yann Collet 39c55a118f fixed minor compatibility issues with older compilers 2018-08-30 16:00:57 -07:00
Yann Collet 39ef91a599 -std=c99 for largeNbDicts 2018-08-30 14:59:23 -07:00
Yann Collet 4086b2871b largeNbDicts compatible with multiple source files 
splitting is disabled by default, but can be re-enabled using usual command -B#
update commands to look like zstd ones
 2018-08-30 14:38:49 -07:00
Yann Collet a5a77965d3 make all includes contrib/largeNbDicts 2018-08-29 16:17:22 -07:00
Yann Collet d89fa814c1 added a README 
for documentation
 2018-08-28 18:19:19 -07:00
Yann Collet 6444c50035 increases randomness of ddict ptrs 2018-08-28 18:13:46 -07:00
Yann Collet 6c398df241 level, block size and nb dicts can be set on command line 2018-08-28 18:05:31 -07:00
Yann Collet 0c66a44d1b first working test program 
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block

current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size

dictionary can be selected on command line, with -D
 2018-08-28 15:47:07 -07:00
Yann Collet 0491037db9 Merge branch 'bench' into largeNbDicts 2018-08-28 11:26:46 -07:00
Yann Collet 55affc09de timedFn : measurement delay is programmable 
instead of hard-coded 1 second per measurement
 2018-08-28 11:26:27 -07:00
Yann Collet 0ff9b67552 paramgrill: removed useless tests 
designed to compensate iter_mode,
but since only time_mode is available now,
all tests are guaranteed to last a minimum amount of time.
 2018-08-27 19:07:17 -07:00
Yann Collet 9e26893e07 paramgrill: fixed a bunch of div-by-zero 
they were pretty easy to trigger by the way,
just start an extended paramgrill session
to find a compression table based on any sample,
it would necessarily happen at some point.
 2018-08-27 18:47:09 -07:00
Yann Collet 0071e8348f restored assert() in paramgrill 
assert() in paramgrill are not in the benchmark path.
They should remain active, as they don't impact measurements, and their runtime is insignificant.
 2018-08-27 17:52:04 -07:00
Yann Collet 274b60e6e6 largeNbDicts can compress and compare dict vs noDict 2018-08-27 17:08:44 -07:00
Yann Collet d97e92dfad Merge branch 'bench' into largeNbDicts 2018-08-27 12:12:51 -07:00
Yann Collet 01dcd0fd17 bench: minor api update, for consistency 
BMK_benchTimedFn()
BMK_isCompleted_TimedFn() uses TimedFnState
 2018-08-26 21:30:18 -07:00
Yann Collet 6782725155 first sketch for largeNbDicts test program 2018-08-26 19:29:12 -07:00
Yann Collet c3a4baaf6e fixed minor warnings 
valgrind: memory leak of a few bytes in fullbench
static analyzer: uninitialized data passed as result
 2018-08-24 23:25:35 -07:00
Yann Collet 2279f3d127 bench: reduce nb of return type 
runOutcome is enough
removed timedFnOutcome
 2018-08-24 17:28:38 -07:00
Yann Collet 6ce7b08f17 fix minor warnings 
gcc : prototype with 0 parameter must be labelled (void)
visual : const property must be identical in both declaration and implementation
 2018-08-24 15:59:57 -07:00
Yann Collet 7b23cc4d1e fixed fullbench behavior 
now same as v1.3.5
 2018-08-24 12:40:10 -07:00
Yann Collet 4da5bdf482 fixed zstd -b speed result 
the benchmark was displaying the speed of last run
instead of the best of all previous runs.
 2018-08-23 18:13:49 -07:00
Yann Collet b0e1f3982d fixed paramgrill 
to work with new bench.c
 2018-08-23 17:21:38 -07:00
Yann Collet 1f9ec13621 introduced MB_UNIT 
so that all benchmarking programs use the same speed scale
 2018-08-23 16:03:30 -07:00
Yann Collet d39a25c5ed update fullbench.c to work with new bench.h 2018-08-23 15:00:09 -07:00
Yann Collet 2e45badff4 refactored bench.c 
for clarity and safety, especially at interface level
 2018-08-23 14:21:18 -07:00
Yann Collet 77e805e3db bench: changed creation/reset function to timedFnState 
for consistency
 2018-08-21 18:19:27 -07:00
Yann Collet c71c4f23d7 fix "unused parameter" in single-thread mode 
within newly added ZSD_toFlushNow()
 2018-08-20 11:40:10 -07:00
Yann Collet 105677c6db created ZSTDMT_toFlushNow() 
tells in a non-blocking way if there is something ready to flush right now.
only works with multi-threading for the time being.

Useful to know if flush speed will be limited by lack of production.
 2018-08-17 18:11:54 -07:00
Yann Collet 09e63c58ac fix : no longer slow down on input saturation 
only slows down when all buffers are full
 2018-08-17 16:27:43 -07:00
Yann Collet 8b674d7dc7 ensured compression level is maxed at ZSTD_maxCLevel() 2018-08-17 16:01:56 -07:00
Yann Collet b4e7f71055 Merge branch 'dev' into adapt 2018-08-17 15:54:13 -07:00
Yann Collet 36d6165a2d Makefile: added variable SCANBUILD 
so that a different version of scan-build can be selected
 2018-08-16 16:44:13 -07:00
Yann Collet 1515f0bb0d fixed more issues detected by recent version of scan-build 
test run on Linux
 2018-08-16 15:20:25 -07:00
Yann Collet 31224cc126 added static analyzer tests to travis CI 2018-08-15 17:44:27 -07:00
Yann Collet 5291d9ac31 fix scope of scan-build tests 
exclude zlib code
 2018-08-15 17:41:44 -07:00
Yann Collet da55865e47 ensure dependency for zlib wrapper 2018-08-15 16:43:13 -07:00
Yann Collet 42a02ab745 fixed minor warnings issued by scan-build 2018-08-15 14:36:02 -07:00
Yann Collet 3692c31598 Merge branch 'dev' into scanbuild 2018-08-15 13:50:49 -07:00
Yann Collet 6e66bbf5dd fixed several minor issues detected by scan-build 
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
 2018-08-14 16:55:35 -07:00
Yann Collet 3e4617ef54 frameProgression reports nbActiveWorkers and output flushed 2018-08-14 11:49:25 -07:00
Yann Collet 0853f86044 adaptive mode uses default window size of 8 MB 2018-08-13 13:13:22 -07:00
Yann Collet 33f7709c71 fileio: changed parameter type from ptr to plain structure 
safer : this parameter is read-only,
we don't want original structure to be modified
 2018-08-13 13:02:03 -07:00
Yann Collet 09c9cf3f51 simplified rateLimiter 
resists better to changing in/out conditions
limits risks of "catching up"
 2018-08-13 12:13:47 -07:00