ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,777 Commits 17 Branches 77 Tags
5cb1144872acbfdf995996624efd5f1ea0cd01aa
Commit Graph
4777 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 5cb1144872 fixed --single-thread 
was incorrectly set to -T0 (use as many cores as possible) previously
 2018-02-13 14:56:35 -08:00
Yann ColletandGitHub 9716250197 Merge pull request #1014 from facebook/fasterDec 
Faster decoding speed
 2018-02-13 12:05:54 -08:00
Yann Collet 9b184359e2 pretify last unit test output 2018-02-13 10:09:01 -08:00
Yann Collet 2524cbd847 added code comment on how to generate default tables 
as suggested by @terrelln
 2018-02-13 10:02:25 -08:00
Yann Collet 71c07966bb added SEQSYMBOL_TABLE_SIZE() 
as suggested by @terrelln's comment
 2018-02-12 16:52:15 -08:00
Yann Collet 821efa466e fixed logo path 2018-02-10 21:05:48 -08:00
Yann Collet 5f7495371e Merge branch 'dev' into fasterDec 2018-02-10 14:24:44 -08:00
Yann ColletandGitHub 992c2370f6 Merge pull request #1010 from facebook/flexibleLevel 
Updatable compression parameters
 2018-02-10 14:19:54 -08:00
Yann Collet 9945e60ac4 Merge branch 'dev' into flexibleLevel 2018-02-10 11:54:49 -08:00
Yann ColletandGitHub 4e3db17cab Merge pull request #1013 from facebook/fasterDec32 
Disable Long Offset mode in 32-bits
 2018-02-09 16:13:55 -08:00
Yann Collet 75689838e4 specify new command --single-thread 2018-02-09 15:55:41 -08:00
Yann Collet c72091556b fixed minor nit as per @terrelln's comments 2018-02-09 09:46:08 -08:00
Yann Collet 4beaeaace5 Merge branch 'dev' into flexibleLevel 2018-02-09 09:15:05 -08:00
Yann Collet 6bfe50ad48 re-enabled ZSTD_decompressSequencesLong() 2018-02-09 09:14:25 -08:00
Yann Collet 1850597eaa pre-calculated default decoding tables 2018-02-09 06:01:02 -08:00
Yann Collet ab75df21ed fixed mono-symbol distribution 2018-02-09 05:12:13 -08:00
Yann Collet 421a2716d8 fixed default fse distributions 
but would be better to pre-calculate tables, for speed
 2018-02-09 04:50:58 -08:00
Yann Collet 95424409ea addBits and baseline into FSE decoding table 
note : unfinished
- need new default tables
- need modify long mode
 2018-02-09 04:25:15 -08:00
Yann Collet cc61a3694a Merge branch 'dev' into fasterDec 2018-02-09 02:41:02 -08:00
Yann Collet d6e841d609 fixed streaming_memory_usage example 
also:
ensure zstd.h is read from ../lib (instead of /usr/include)
 2018-02-07 23:42:09 -08:00
Yann Collet de68c2ff10 Merged ZSTD_preserveUnsortedMark() into ZSTD_reduceIndex() 
as it's faster, due to one memory scan instead of two
(confirmed by microbenchmark).

Note : as ZSTD_reduceIndex() is rarely invoked,
it does not translate into a visible gain.
Consider it an exercise in auto-vectorization and micro-benchmarking.
 2018-02-07 14:22:35 -08:00
Yann Collet 0170cf9a7a minor : modified ZSTD_preserveUnsortedMark() to be more vectorization friendly 2018-02-05 11:46:02 -08:00
Yann Collet 94efb1749d faster decoding in 32-bits mode for long offsets (tentative) 
On my laptop:
Before:
./zstd32 -b --zstd=wlog=27 silesia.tar enwik8 -S
 3#silesia.tar       : 211984896 ->  66683478 (3.179),  97.6 MB/s , 400.7 MB/s
 3#enwik8            : 100000000 ->  35643153 (2.806),  76.5 MB/s , 303.2 MB/s

After:
./zstd32 -b --zstd=wlog=27 silesia.tar enwik8 -S
 3#silesia.tar       : 211984896 ->  66683478 (3.179),  97.4 MB/s , 435.0 MB/s
 3#enwik8            : 100000000 ->  35643153 (2.806),  76.2 MB/s , 338.1 MB/s

Mileage vary, depending on file, and cpu type.
But a generic rule is : x86 benefits less from "long-offset mode" than x64,
maybe due to register pressure.
On "entropy", long-mode is _never_ a win for x86.
On my laptop though, it may, depending on file and compression level
(enwik8 benefits more from "long-mode" than silesia).
 2018-02-04 01:49:31 -08:00
Yann Collet 5188749e1c ensure compression parameters are updated when only compression level is changed 2018-02-02 16:31:20 -08:00
Yann Collet 4b525af53a zstdmt: applies new parameters on the fly 
when invoked from ZSTD_compress_generic()
 2018-02-02 15:58:13 -08:00
Yann Collet 90eca318a7 fileio: create dedicated function to generate zstd frames 
like other formats
 2018-02-02 14:24:56 -08:00
Yann ColletandGitHub 1291d9d7cf Merge pull request #1006 from systemcrash/patch-2 
Update README.md
 2018-02-02 10:04:55 -08:00
Yann ColletandGitHub 549d26ae71 Merge pull request #1005 from systemcrash/dev 
Update zstd.1
 2018-02-02 10:04:40 -08:00
Yann Collet 6c492af284 fixed minor conversion warning 2018-02-01 20:16:00 -08:00
Yann Collet 209df52ba2 Changed nbThreads for nbWorkers 
This makes it easier to explain that nbWorkers=0 --> single-threaded mode,
while nbWorkers=1 --> asynchronous mode (one mode thread on top of the "main" caller thread).
No need for an additional asynchronous mode flag.
nbWorkers>=2 works the same as nbThreads>=2 previously.
 2018-02-01 19:29:30 -08:00
Yann Collet 4b6a94f0cc clarified comments on LDM parameters 2018-02-01 17:07:27 -08:00
Yann Collet 60fa90b6c0 zstdmt: added ability to change compression parameters during compression 2018-02-01 16:13:31 -08:00
Yann Collet 2bfc79ab8d removed bitstream.h dependency 2018-02-01 16:13:04 -08:00
Yann ColletandGitHub f25351e23e Merge pull request #1009 from terrelln/fail-early 
Test for incorrect pledgeSrcSize earlier
 2018-02-01 14:43:32 -08:00
Nick Terrell 48acaddff9 Test for incorrect pledgeSrcSize earlier 2018-02-01 12:04:05 -08:00
Yann ColletandGitHub 727bb7f090 Merge pull request #1008 from terrelln/hlog3 
Fix hashLog3 size when copying cdict tables
 2018-01-31 12:49:07 -08:00
Nick Terrell 9c8c69e41a [fuzzer] Check ZSTD_initStaticCDict() for every level 2018-01-31 11:12:54 -08:00
Nick Terrell ab3346af07 Fix hashLog3 size when copying cdict tables 2018-01-31 11:12:17 -08:00
Yann ColletandGitHub 823a28a1f4 Merge pull request #1000 from facebook/progressiveFlush 
Progressive flush
 2018-01-30 22:49:47 -08:00
Yann Collet b173b8d8da updated zstd api manual 2018-01-30 15:05:12 -08:00
Yann Collet a2ba629971 fixed function declaration ZSTD_getBlockSize() 2018-01-30 15:03:39 -08:00
Yann Collet 2cb0740b6b zstdmt: changed naming convention 
to avoid confusion with blocks.

also:
- jobs are cut into chunks of 512KB now, to reduce nb of mutex calls.
- fix function declaration ZSTD_getBlockSizeMax()
- fix outdated comment
 2018-01-30 14:43:36 -08:00
systemcrashandGitHub 6b57387728 Update README.md 
spelling
 2018-01-29 18:42:20 +01:00
systemcrashandGitHub d13a75c969 Update zstd.1 2018-01-29 18:38:02 +01:00
Yann Collet 9f8ed23b5b bumped version number to v1.3.4 
also added a paragraph on using compression level with training mode
as this is a recurrent question (see for example #1004)
 2018-01-27 22:23:26 -08:00
Yann Collet ba0cd8cf78 fixed minor conversion warning for C++ compilation mode 2018-01-26 18:18:42 -08:00
Yann Collet caf9e96dc3 job mutex creation is checked 2018-01-26 18:09:25 -08:00
Yann Collet 9c40ae7ff1 zstdmt: there is now one mutex/cond per job 2018-01-26 17:55:08 -08:00
Yann Collet 77e36273de zstdmt: minor code refactor for clarity 2018-01-26 17:08:58 -08:00
Yann Collet 27c5853c42 zstdmt: job table correctly cleaned after synchronous ZSTDMT_compress() 2018-01-26 14:35:54 -08:00