ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,930 Commits 17 Branches 77 Tags
b94d08c566fb6f658a2c53d992662cfee1647480
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet b94d08c566 commented src archive lines 2018-03-23 16:23:23 -07:00
Yann Collet b1f898ec82 added srcPackage to list of branch producing artifacts 2018-03-23 16:15:56 -07:00
Yann Collet 929608146a create .tar.zst source package 2018-03-23 15:00:04 -07:00
Yann Collet d6c6b62958 updated graph for dictionary compression 2018-03-22 18:44:56 -07:00
Yann Collet 184e1c8b8c updated benchmark figures 2018-03-22 18:22:10 -07:00
Yann Collet 01082a39bd restored simple status line during zstd compression 
the more advanced one, featuring amount of data buffered,
is triggered on `-v`.
 2018-03-22 17:49:46 -07:00
Yann Collet f15e2fd55e fixed fuzz test 2018-03-21 16:45:15 -07:00
Yann Collet b7b80e1569 fixed paramgrill 2018-03-21 16:09:15 -07:00
Yann Collet 93b8262c02 added large negative level test case 2018-03-21 15:54:44 -07:00
Yann Collet 153bc1c004 removed limit ZSTD_TARGETLENGTH_MAX 
this makes it possible to specify extremely large negative compression levels,
achieving the side effect as "no compression".

It will also be possible to define larger targetlength for ultra compression mode.

There is no adverse side effect due to removing this limit.
 2018-03-21 15:50:05 -07:00
Yann Collet a99c4a3621 Merge branch 'dev' into advancedDecompress 2018-03-21 06:08:28 -07:00
Yann Collet 453d487807 fixed minor declaration warning 2018-03-20 18:03:56 -07:00
Yann Collet e0cb8d19c6 fixed legacy test case 2018-03-20 17:48:22 -07:00
Yann Collet 878728dc26 fixed several comments by @terrelln 2018-03-20 16:35:14 -07:00
Yann Collet b18cb7e0b7 zstreamtest : slightly decreased time to build dictionary size 2018-03-20 16:18:48 -07:00
Yann Collet 6cda8c932c added test with ZSTD_decompress_generic() + ZSTD_DCtx_refPrefix() 
also :
clarified stage condition to accept new parameters,
fixed initializers correspondingly.
 2018-03-20 16:16:13 -07:00
Yann Collet 2af41592ea added a test with ZSTD_decompress_generic() + ZSTD_DCtx_loadDictionary_byReference() 2018-03-20 15:59:25 -07:00
Yann Collet 0dadb6b70d implemented ZSTD_DCtx_refPrefix*() 2018-03-20 15:45:56 -07:00
Yann Collet 569b8ba4d9 implemented ZSTD_DCtx_refDDict() 2018-03-20 15:43:49 -07:00
Yann Collet 6873fec658 changed dictMore for dictContentType 
which seems clearer to describe what the variable/argument is about.
 2018-03-20 15:13:14 -07:00
Yann Collet 31b54b6eea updated ZSTD_initStaticDDict() prototype 
can also specify dictContentType.
 2018-03-20 14:52:02 -07:00
Yann Collet 353117c5d7 implemented ZSTD_DCtx_loadDictionary*() 
this required updating ZSTD_createDDict_advanced()
to accept a dictContentType parameter (raw, full, auto).
 2018-03-20 13:40:29 -07:00
Yann Collet 2ed5af0766 merge bmi2 implementation of encodeSequence into zstd_compress.c 2018-03-19 19:10:31 -07:00
Yann Collet 9e6ba88363 moved snap to /contrib 2018-03-19 16:15:06 -07:00
Yann Collet a7b532a347 added docker readme 2018-03-19 16:13:12 -07:00
Yann Collet e5ddfaede8 add a test for unordered parameters 2018-03-19 16:02:51 -07:00
Yann Collet c8b3d389fd updated CCtxParams API 
to respect naming convention :
ZSTD_CCtxParams_*()
 2018-03-19 15:07:26 -07:00
Yann Collet 6f4d0778a5 make it possible to express compression parameters in any order 2018-03-19 14:41:23 -07:00
Yann Collet 9618c0c804 make it possible to specify LDM parameters in any order 2018-03-19 11:07:04 -07:00
Yann Collet cbc71e40f6 moving LRM parameters out of experimental section 
into "normal" range,
start pinned at 160.
 2018-03-15 17:22:40 -07:00
Yann Collet 5373e44ba7 fixed contrib/adaptive-compression 2018-03-15 17:10:15 -07:00
Yann Collet 97816400ca added /contrib projects to make all 2018-03-15 16:40:14 -07:00
Yann Collet 355cb645bf fixed seekable format example 2018-03-15 16:29:28 -07:00
Yann Collet 38cbcb5f1a removed LRM exploratory experiment 2018-03-15 16:26:08 -07:00
Yann Collet a909c293c6 Merge branch 'dev' into hufCompress 2018-03-14 16:11:25 -07:00
Yann Collet 50f763ec44 fixed several comments are underlined by @terrelln 2018-03-13 14:23:14 -07:00
Yann Collet a95a88af57 removed huf_compress_impl.h 
re-imported all functions inside huf_compress.c
for easier source editing.

Also updated a bunch of code comments
for clarification.
 2018-03-13 14:14:05 -07:00
Yann Collet 2291b85a1e changed ZSTD_p_literalCompression into ZSTD_p_compressLiterals 
prefer verb+object construction
 2018-03-12 11:44:10 -07:00
Yann Collet a57d43d4d4 updated documentation of targetLength 2018-03-12 11:35:01 -07:00
Yann Collet f24566b597 minor bench improvements 
- do not test level 0, as it is converted into level 3,
  which feels strange when compressing multiple levels
- Use direct synchronous mode when a single worker is requested.
 2018-03-12 04:02:57 -07:00
Yann Collet 6a9b41b731 create command --fast[=#] 
access negative compression levels from command line
for both compression and benchmark modes.

also : ensure proper propagation of parameters
through ZSTD_compress_generic() interface.

added relevant cli tests.
 2018-03-11 20:01:23 -07:00
Yann Collet a146ee04ae added negative compression levels 
negative compression level trade compression ratio for more compression speed.
They turn off huffman compression of literals,
and use row 0 as baseline with a stepSize = -cLevel.

added associated test in fuzzer

also added : new advanced parameter ZSTD_p_literalCompression
 2018-03-11 05:21:53 -07:00
Yann Collet facc09aa03 minor compression level adaptation 
level 12 compresses slightly more and faster
due to better btlazy2 mode
 2018-03-11 03:06:52 -07:00
Yann Collet fe321f9e2a re-integrate ZSTD_decompressSequencesLong() into zstd_decompress.c 
removed zstd_decompress_impl.h
 2018-03-09 19:48:06 -08:00
Yann Collet 89a2ebb971 incorporated ZSTD_decompressSequences() into zstd_decompress() 2018-03-09 19:35:57 -08:00
Yann Collet cdb1f1433e incorporated ZSTD_initFseState() inside zstd_decompress.c 2018-03-09 18:16:10 -08:00
Yann Collet a166eae1ba incorporate ZSTD_decodeSequenceLong() within zstd_decompress.c 2018-03-09 18:11:14 -08:00
Yann Collet 17626ba56e restored ZSTD_decodeSequence() into zstd_decompress.c 2018-03-09 18:03:25 -08:00
Yann Collet 8c6dbf490b updated NEWS in preparation for v1.3.4 2018-03-09 16:22:34 -08:00
Yann Collet 5414b8ea01 Merge branch 'dev' of github.com:facebook/zstd into dev 2018-03-09 11:53:24 -08:00
Yann Collet e916b9090e gen_html: changed CFLAGS for CXXFLAGS 
since it's associated with $(CXX)
 2018-03-09 11:52:14 -08:00
Yann Collet db147ea620 improved comments 
following @terrelln suggestions
 2018-03-06 18:15:26 -08:00
Yann Collet 06ca9c7d7c fixed 0-seq blocks in block-decompression mode 2018-03-06 01:50:19 -08:00
Yann Collet 9a91afe6ef long offset mode : new default threshold for 32-bit 2018-03-05 16:41:08 -08:00
Yann Collet 7bd7a3ad43 long offset mode : new default threshold for 64-bits mode 2018-03-05 16:16:49 -08:00
Yann Collet c0393a538f fixed counting long distance weights 2018-03-05 15:12:10 -08:00
Yann Collet a70f7e10fa Merge branch 'benchDecode' into longOffsetMode 2018-03-05 14:09:00 -08:00
Yann Collet 03e7e14192 fix benchmark issue when measuring only decoding speed 
zstd bench module can focus on decompression speed _only_.
This is useful when trying to measure performance
on large input data compressed using a high level
as compression time becomes problematic (too long).

This mode is triggered by command : zstd -b -d

Problem was : in such a mode,
measured decoding speed was > 10% slower
than in nominal mode (compression + decompression),
making decompression benchmark mode much less useful.

This patch fixes the issue.
It's not completely clear why, but
moving the `memcpy()` operation sooner in the pipeline fixed it.

I can still measure some difference, but it is in the < 2% range,
so it's much more tolerable.

also : it doesn't matter anymore in which order are selected
commands `-b` and `-d`.
The combination always triggers bench_decodeOnly mode.
 2018-03-05 13:57:41 -08:00
Yann Collet 41bd10446e Merge branch 'dev' into longOffsetMode 2018-03-05 13:10:10 -08:00
Yann Collet cb789d2df8 re-inserted offset evaluation 2018-03-05 13:08:59 -08:00
Yann Collet b91ddf0ae6 Merge branch 'dev' into longOffsetMode 2018-03-05 11:59:54 -08:00
Yann Collet d02b44cf55 DYNAMIC_BMI2 enabled for clang 
clang only claims compatibility with gcc 4.2.
Consequently, recent patch which reserved DYNAMIC_BMI2 for gcc >= 4.8
also disabled it for clang.

fix : __clang__ is now enough to enable DYNAMIC_BMI2
(associated with other existing conditions : x64/x64, !bmi2)
 2018-03-04 16:05:59 -08:00
Yann Collet 45b09e7625 limit DYNAMIC_BMI2 to gcc >= 4.8 
attribute bmi2 not supported by gcc 4.4
 2018-03-01 15:02:18 -08:00
Yann Collet b01552a07a force inlining of HUF_decodeSymbol*() functions 
which was not done properly by gcc 4.8
resulting in major performance difference.

ex :
zstd -b1 silesia.tar
before : dec 680 MB/s
after  : dec 710 MB/s  (without bmi2)
after  : dec 770 MB/s  (with DYNAMIC_BMI2)
 2018-03-01 11:31:45 -08:00
Yann Collet 564cb1b640 update doc/README.md on provided tools to test 3rd party implementations 2018-02-27 17:37:05 -08:00
Yann Collet 33a3f18848 fixed wrong size test 2018-02-26 18:27:51 -08:00
Yann Collet 89741653ab added error code workSpace_tooSmall 2018-02-26 15:11:50 -08:00
Yann Collet 6cdf690441 minor cleaning of huff0 
Update code documentation, and properly names a few "magic constants".
Also, HUF_compress_internal() gets a cleaner way
to determine size of tables inside workspace.
 2018-02-26 14:52:23 -08:00
Yann Collet 653383f74a minor nit from Mac XCode 2018-02-22 15:44:26 -08:00
Yann Collet 0fd4df6ed3 Implemented BMI2 functions directly within huf_decompress.c 
This makes it easier to edit for maintenance and evolutions
(I plan to experiment modifications in huffman decompression functions).

The methology followed seems broadly applicable to other BMI2 modules.

Performance was tracked rigorously at each step,
there is no noticeable loss (nor win) of performance compared to `#include` version.

Note however that 4X decoder variants tend to be extremely sensitive to code alignment.
This source code resulted in pretty good performance for gcc 7.2 and 7.3,
but future changes (even in other parts of the code) might trigger the issue again.
 2018-02-22 10:51:47 -08:00
Yann Collet 6e481504ee fullbench includes assert.h 
as it is missing for Windows
 2018-02-21 11:42:23 -08:00
Yann Collet 9c5a8040a9 fixed huf_compress workspace size 2018-02-21 11:34:49 -08:00
Yann Collet 364ce19463 update fullbench measurement methodology 
to use less calls to time(), like bench.c.

also upgraded accuracy to nanosecond.
 2018-02-21 09:43:32 -08:00
Yann Collet 25d00d10fc fixed minor conversion warning 2018-02-20 16:52:28 -08:00
Yann Collet 3538a535bf use TIMELOOP_NANOSEC 
as suggested by @terrelln
 2018-02-20 15:33:56 -08:00
Yann Collet d3364aa39e improve benchmark measurement for small inputs 
by invoking time() once per batch, instead of once per compression / decompression.
Batch is dynamically resized so that each round lasts approximately 1 second.

Also : increases time accuracy to nanosecond
 2018-02-20 14:58:40 -08:00
Yann Collet 70163bf0d3 added clarification comments in zstd_errors.h 
answering some points in #1018
 2018-02-20 12:54:49 -08:00
Yann Collet 5cb1144872 fixed --single-thread 
was incorrectly set to -T0 (use as many cores as possible) previously
 2018-02-13 14:56:35 -08:00
Yann Collet 9b184359e2 pretify last unit test output 2018-02-13 10:09:01 -08:00
Yann Collet 2524cbd847 added code comment on how to generate default tables 
as suggested by @terrelln
 2018-02-13 10:02:25 -08:00
Yann Collet 71c07966bb added SEQSYMBOL_TABLE_SIZE() 
as suggested by @terrelln's comment
 2018-02-12 16:52:15 -08:00
Yann Collet 821efa466e fixed logo path 2018-02-10 21:05:48 -08:00
Yann Collet 5f7495371e Merge branch 'dev' into fasterDec 2018-02-10 14:24:44 -08:00
Yann Collet 9945e60ac4 Merge branch 'dev' into flexibleLevel 2018-02-10 11:54:49 -08:00
Yann Collet 04a3f85ce7 fixed gcc warning on a switch code path 2018-02-09 16:16:27 -08:00
Yann Collet 75689838e4 specify new command --single-thread 2018-02-09 15:55:41 -08:00
Yann Collet af48f0b62b fix : offset table pointer when using default table 2018-02-09 15:15:46 -08:00
Yann Collet 426944c3e3 fixed strict aliasing issue 
tuned threshold
 2018-02-09 13:24:11 -08:00
Yann Collet 64ee732694 decide long-offset mode based on offcode statistics 
threshold vaguely estimated
 2018-02-09 12:33:28 -08:00
Yann Collet c72091556b fixed minor nit as per @terrelln's comments 2018-02-09 09:46:08 -08:00
Yann Collet 4beaeaace5 Merge branch 'dev' into flexibleLevel 2018-02-09 09:15:05 -08:00
Yann Collet 6bfe50ad48 re-enabled ZSTD_decompressSequencesLong() 2018-02-09 09:14:25 -08:00
Yann Collet 1850597eaa pre-calculated default decoding tables 2018-02-09 06:01:02 -08:00
Yann Collet ab75df21ed fixed mono-symbol distribution 2018-02-09 05:12:13 -08:00
Yann Collet 421a2716d8 fixed default fse distributions 
but would be better to pre-calculate tables, for speed
 2018-02-09 04:50:58 -08:00
Yann Collet 95424409ea addBits and baseline into FSE decoding table 
note : unfinished
- need new default tables
- need modify long mode
 2018-02-09 04:25:15 -08:00
Yann Collet cc61a3694a Merge branch 'dev' into fasterDec 2018-02-09 02:41:02 -08:00
Yann Collet d6e841d609 fixed streaming_memory_usage example 
also:
ensure zstd.h is read from ../lib (instead of /usr/include)
 2018-02-07 23:42:09 -08:00
Yann Collet de68c2ff10 Merged ZSTD_preserveUnsortedMark() into ZSTD_reduceIndex() 
as it's faster, due to one memory scan instead of two
(confirmed by microbenchmark).

Note : as ZSTD_reduceIndex() is rarely invoked,
it does not translate into a visible gain.
Consider it an exercise in auto-vectorization and micro-benchmarking.
 2018-02-07 14:22:35 -08:00
Yann Collet 0170cf9a7a minor : modified ZSTD_preserveUnsortedMark() to be more vectorization friendly 2018-02-05 11:46:02 -08:00