ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,447 Commits 17 Branches 77 Tags
d0a1d45582181efd76350922a240f14d7893c985
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet d0a1d45582 ZSTDMT_{flush,end}Stream() now block on next job completion when nothing to flush 
The main issue was to avoid a caller to continually loop on {flush,end}Stream()
when there was nothing ready to be flushed but still some compression work ongoing in a worker thread.
The continuous loop would have resulted in wasted energy.
The new version makes call to {flush,end}Stream blocking when there is nothing ready to be flushed.
Of course, if all worker threads have exhausted job, it will return zero (all flush completed).

Note : There are still some remaining issues to report error codes
and properly collect back resources into pools when an error is triggered.
 2017-01-17 16:15:18 -08:00
Yann Collet a73c412932 completed ZSTDMT streaming compression 
Provides the baseline compression API :
size_t ZSTDMT_initCStream(ZSTDMT_CCtx* zcs, int compressionLevel);
size_t ZSTDMT_compressStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
size_t ZSTDMT_flushStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output);
size_t ZSTDMT_endStream(ZSTDMT_CCtx* zcs, ZSTD_outBuffer* output);

Not tested yet
 2017-01-17 15:31:16 -08:00
Yann Collet 5b726dbe4d fix gcc-arm warning "suggest braces around empty body" 2017-01-12 17:46:46 +01:00
Yann Collet ad9f6bd123 zstdmt : fix : resources properly collected even when early fail 
In previous version, main function would return early when detecting a job error.
Late threads resources were therefore not collected back into pools.
New version just register the error, but continue the collecting process.
All buffers and context should be released back to pool before leaving main function.
 2017-01-12 03:06:35 +01:00
Yann Collet b05c4828ea zstdmt : correctly check for cctx and buffer allocation 
Result from getBuffer and getCCtx could be NULL when allocation fails.
Now correctly checks : job creation stop and last job reports an allocation error.
releaseBuffer and releaseCCtx are now also compatible with NULL input.

Identified a new potential issue :
when early job fails, later jobs are not collected for resource retrieval.
 2017-01-12 02:01:28 +01:00
Yann Collet 107bcbbbc2 zstdmt : changed internal naming from frame to chunk 
Since the result of mt compression is a single frame,
changed naming, which implied the concatenation of multiple frames.

minor : ensures that content size is written in header
 2017-01-12 01:25:46 +01:00
Yann Collet 5eb749e734 ZSTDMT_compress() creates a single frame 
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()

As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().

This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
 2017-01-11 18:21:25 +01:00
Yann Collet 04cbc36499 minor refactor (release CCtx 1st) and comment clarification 2017-01-11 16:08:08 +01:00
Yann Collet 085179bb78 fixed ZSTDMT_createCCtx() : checked inner objects are properly created 2017-01-11 15:58:05 +01:00
Yann Collet 8ce1cc2bec improved ZSTD_createCCtxPool() cancellation 
use ZSTD_freeCCtxPool() to release the partially created pool.
avoids to duplicate logic.

Also : identified a new difficult corner case :
when freeing the Pool, all CCtx should be previously released back to the pool.
Otherwise, it means some CCtx are still in use.
There is currently no clear policy on what to do in such a case.
Note : it's supposed to never happen.
Since pool creation/usage is static, it has no external user,
which limits risks.
 2017-01-11 15:44:26 +01:00
Yann Collet 47557ba2b2 fixed ZSTDMT_createCCtxPool() when inner CCtx creation fails 2017-01-11 15:35:56 +01:00
Yann Collet 8566799781 separated ppc and ppc64 tests, for more regular timing 2017-01-03 00:25:01 +01:00
Yann Collet 6334b04d61 compile object files, for faster recompilation 2017-01-02 03:22:18 +01:00
Yann Collet 747452677d fixed cmake tests 2017-01-02 02:05:45 +01:00
Yann Collet cdb2763f4a new Makefile target zstdmt 2017-01-02 01:43:56 +01:00
Yann Collet f1cb55192c fixed linux warnings 2017-01-02 01:11:55 +01:00
Yann Collet 0ec6a95ba1 minor fixes 2017-01-02 00:49:42 +01:00
Yann Collet 2ec635a162 use pthread_cond to send signals between threads 2017-01-01 17:31:33 +01:00
Yann Collet 3b9d434356 extended ZSTDMT code support for non-MT systems and WIN32 (preliminary) 2016-12-31 16:32:19 +01:00
Yann Collet c8efc1c874 simplified Buffer Pool 2016-12-31 14:45:33 +01:00
Yann Collet 3b29dbd9e8 new zstdmt version using generic treadpool 2016-12-31 06:04:25 +01:00
Yann Collet c6a6417458 bench correctly measures time for multi-threaded compression (posix only) 2016-12-31 03:31:26 +01:00
Yann Collet e70912c72b Changed : input divided into roughly equal parts. 
Debug : can measure time waiting for mutexes to unlock.
 2016-12-29 01:24:01 +01:00
Yann Collet 6c0ed9483a compression threads use ZSTD_compressCCtx() 2016-12-28 17:08:28 +01:00
Yann Collet ab7a579180 added -T command , to set nb of threads 2016-12-28 16:11:09 +01:00
Yann Collet ce9e1452fd protect buffer pool with a mutex 2016-12-28 15:31:19 +01:00
Yann Collet 3d93f2fce7 first zstdmt sketch 2016-12-27 07:19:36 +01:00
Yann Collet 39c105c605 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-23 22:25:31 +01:00
Yann Collet aca113f4f5 fixed ZSTD_sizeof_?Dict() 2016-12-23 22:25:03 +01:00
Yann Collet d76d1a9ef0 added ZDICT_finalizeDictionary() 2016-12-22 20:18:43 +01:00
Yann Collet 9ceb49e097 fixed zlib_wrapper conversion warnings 2016-12-22 15:26:33 +01:00
Yann Collet 7cedbd1936 fixed allmost target 2016-12-22 12:43:00 +01:00
Yann Collet a86a09ea0d removed examples from standard C tests, since they contain some POSIX elements 2016-12-22 11:31:39 +01:00
Yann Collet ba75e9d8c3 fix : zlib wrapper compile in gnu90 mode 2016-12-21 19:57:18 +01:00
Yann Collet 0819abe3c1 added ZSTD_createDDict_byReference() body 2016-12-21 19:25:15 +01:00
Yann Collet 4e5eea61a8 added ZSTD_createDDict_byReference() 2016-12-21 16:44:35 +01:00
Yann Collet 8333106b8a Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-21 16:44:24 +01:00
Yann Collet 9518744025 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-21 16:20:41 +01:00
Yann Collet 1f57c2ed32 added : ZSTD_createCDict_byReference() 2016-12-21 16:20:11 +01:00
Yann Collet d564faa3c6 fix : ZSTD_initCStream_srcSize() correctly set srcSize in frame header 2016-12-18 21:39:15 +01:00
Yann Collet 1496c3dc47 Fix : size estimation when some samples are very large 2016-12-18 11:58:23 +01:00
Yann Collet 36321d2e32 updated NEWS 2016-12-17 19:37:55 +01:00
Yann Collet 31ff2a23be fix Travis long test list; 
added zbufftest-dll
 2016-12-17 19:10:10 +01:00
Yann Collet d46ecb58a5 added dll compilation tests 2016-12-17 16:28:12 +01:00
Yann Collet dc9931205a updated manual 2016-12-14 14:53:47 +01:00
Yann Collet b23420fa48 updated NEWS 2016-12-13 19:47:17 +01:00
Yann Collet 2b36b238d3 changed variable name to estimatedSrcSize, to emphasize it does not need to be exact 2016-12-13 17:59:55 +01:00
Yann Collet e795c8a5f6 Added ZSTD_initCStream_srcSize(). 
Added relevant test cases in zstreamtest
 2016-12-13 17:00:14 +01:00
Yann Collet 5397a66b19 minor BMI version check 2016-12-13 15:21:06 +01:00
Yann Collet 086985316f updated NEWS 2016-12-13 11:45:19 +01:00
Yann Collet fda539f50b minor coding style changes 2016-12-12 01:03:23 +01:00
Yann Collet c3a5c4bef8 introduced cycleLog 2016-12-12 00:47:30 +01:00
Yann Collet c261f71f6a minor variation of rescale fix 2016-12-12 00:25:07 +01:00
Yann Collet 383b8088a3 minor lib build refactoring 2016-12-08 18:42:27 -08:00
Yann Collet 6e754fe76a fixed lib soname. 
example : simple_compression : size overflow check
 2016-12-08 18:26:56 -08:00
Yann Collet 16a5752588 streaming example uses stable API 2016-12-08 17:28:26 -08:00
Yann Collet ef013f0edc Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-08 12:07:06 -08:00
Yann Collet d2908b9215 updated NEWS 2016-12-07 16:41:33 -08:00
Yann Collet 426a9d4b71 changed : dll : only approved ZSTD symbols are now exposed. All other symbols remain internal. 2016-12-07 16:39:34 -08:00
Yann Collet c0a1d6deb0 better cleaning 2016-12-07 15:58:32 -08:00
Yann Collet 2a826e592b compatibility with zstd-frugal (noBench mode) 2016-12-06 17:56:20 -08:00
Yann Collet e63c631aaf decode benchmark, multi-files 2016-12-06 17:46:49 -08:00
Yann Collet d946501d2c decode benchmark - single file (hidden option) 2016-12-06 16:49:23 -08:00
Yann Collet 1628fe08fc cmake : SHARED_LIBRARY_OUTPUT_NAME is user-selectable, by @aparamon (#469) 2016-12-06 13:45:42 -08:00
Yann Collet 94d1a93d28 changed environment variable comparison to sh compatible 2016-12-06 12:02:56 -08:00
Yann Collet 6b9a983261 changed gzstd build messages 2016-12-06 11:23:25 -08:00
Yann Collet 379908be3d fixed zstd.h for manual 2016-12-06 10:36:15 -08:00
Yann Collet 825dffbc43 moved zbuff source files into lib/deprecated 2016-12-05 19:28:19 -08:00
Yann Collet 8f8e2b0b4a fixed initialization warning 2016-12-05 18:00:50 -08:00
Yann Collet e7a41a5955 added : dictID retrieval functions. 
added : unit tests for dictID retrieval functions
 2016-12-05 16:21:06 -08:00
Yann Collet 9ffbeea875 API : changed : streaming decompression : implicit reset on starting new frames 2016-12-02 18:37:38 -08:00
Yann Collet db85a6e09a added zstdgrep 2016-12-02 15:57:07 -08:00
Yann Collet efaf104b2d added zstdless 2016-12-02 15:24:40 -08:00
Yann Collet 743b33f57e fix zstdcat 2016-12-02 15:18:57 -08:00
Yann Collet 5bd4237beb minor refactor 2016-12-02 12:40:57 -08:00
Yann Collet 2238312c2f fix dict loading 2016-12-02 11:36:11 -08:00
Yann Collet b89af20353 reduced table sizes for HUF_readDTableX4 2016-12-01 18:24:59 -08:00
Yann Collet a0d742b1e4 introduced HUF_buildCTable_wksp(), to reduce stack memory usage 2016-12-01 17:47:30 -08:00
Yann Collet 850c76d045 removed test artefact 2016-12-01 16:26:25 -08:00
Yann Collet 643d9a234b replaced usage of FSE_buildCTable by FSE_buildCTable_wksp, using less stack space in the process 2016-12-01 16:24:04 -08:00
Yann Collet e928f7e16d introduced ext_wksp variants of count to reduce stack memory usage 2016-12-01 16:13:35 -08:00
Yann Collet 979cab412b fixed some minor visual silent cast warnings. 
introduced FSE_count_parallel_wksp().
 2016-11-30 18:10:38 -08:00
Yann Collet 5e00b848a8 FSE_compress_wksp() uses less stack space 2016-11-30 16:46:13 -08:00
Yann Collet d79a9a00d9 Introduced FSE_compress_wksp() and FSE_buildCTable_wksp() to reduce stack memory usage 2016-11-30 15:52:20 -08:00
Yann Collet 766431909f introduced FSE_decompress_wksp(), to use less stack space 2016-11-30 12:36:45 -08:00
Yann Collet 95eb43be09 updated pkg config file 2016-11-30 11:06:58 -08:00
Yann Collet 99672d1231 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-11-30 11:03:59 -08:00
Yann Collet 1cc37ad8ee modified NEWS 2016-11-29 18:06:59 -08:00
Yann Collet ff504de391 minor decompression speed improvement 2016-11-29 17:42:46 -08:00
Yann Collet b766c8029c update NEWS 2016-11-29 17:11:01 -08:00
Yann Collet 25f46dcc0f minor const 2016-11-29 16:59:27 -08:00
Yann Collet a56ac2815c restored normal decoder speed 2016-11-29 15:30:23 -08:00
Yann Collet 37870d7a66 fixed minor visual warning 2016-11-29 14:31:57 -08:00
Yann Collet 167c494748 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-11-29 14:05:15 -08:00
Yann Collet 4f5350f610 long matches support overflow 2016-11-29 13:12:24 -08:00
Yann Collet 52e136ed3d long decoder compatible with round and separate buffers 2016-11-28 19:59:11 -08:00
Yann Collet ce3527ca0c combined normal and long decoder 2016-11-28 18:38:52 -08:00
Yann Collet 8993bee997 restored normal mode 2016-11-28 16:11:30 -08:00
Yann Collet 764e70a4f3 added decodeSequencesLong 2016-11-28 15:50:16 -08:00
Yann Collet 73f88a66f1 added prefetch 2016-11-23 15:43:30 -08:00