ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,725 Commits 17 Branches 77 Tags
e3d1f7057dd8cddf28bb3d40fba1ea3dc3f94de4
Commit Graph
5201 Commits
Author SHA1 Message Date
Stella Lau d775519296 Add cctxParam versions of internal functions 2017-08-18 17:37:58 -07:00
Yann Collet 32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Stella Lau 63b8c98531 Pass cctx parameters to MTCtx 2017-08-18 16:17:24 -07:00
Stella Lau 399ae013d4 Add function to apply cctx params 2017-08-18 13:01:55 -07:00
Stella Lau 81d89d82a6 Move nbThreads to cctx params 2017-08-18 12:08:57 -07:00
Stella Lau 2300c58a6f Move dictContentByRef to cctx params 2017-08-18 12:03:16 -07:00
Stella Lau b6cb2ed8cb Move dictMode to cctxParams 2017-08-18 11:43:31 -07:00
Stella Lau 97e27affcb Move compression level to cctx params 2017-08-18 11:20:08 -07:00
Stella Lau c0221124d5 Add function to set opaque parameters 2017-08-17 19:30:22 -07:00
Stella Lau 4169f49171 Add initialization/allocation functions for opaque params 2017-08-17 18:45:04 -07:00
Stella Lau ade95b8bed Add opaque interfaces for static initialization 2017-08-17 18:13:08 -07:00
Stella Lau 699f11b4f7 Create opaque parameter structure 2017-08-17 17:33:46 -07:00
Yann ColletandGitHub f9e6590715 Merge pull request #796 from terrelln/is-error 
[FSE][HUF] Inline error checks
 2017-08-15 12:37:28 -07:00
Yann ColletandGitHub 2dbcfc6994 Merge pull request #794 from terrelln/force-inline 
[libzstd] Fix FORCE_INLINE macro
 2017-08-15 12:03:44 -07:00
Nick Terrell 07c6ff588e [FSE][HUF] Inline error checks 
Caught by Clang's optimization remarks.
 2017-08-15 11:23:28 -07:00
Nick Terrell 565e925eb7 [libzstd] Fix FORCE_INLINE macro 2017-08-14 21:12:05 -07:00
Roman Gershman b9d4f4fb74 Fix ZSTD_estimateDStreamSize function after ZSTD_DStream and ZSTD_DCtx were merged 2017-08-13 13:29:42 +03:00
Nick Terrell 9ba97182d1 [CI] Add gcc7build test 2017-08-08 13:28:56 -07:00
Yann ColletandGitHub d9f2893eb9 Merge pull request #782 from terrelln/dstSizeTooSmall 
Fix compression failure on incompressible data
 2017-08-07 14:52:02 -07:00
Yann ColletandGitHub 8049556928 Merge pull request #778 from terrelln/bad-huff 
[libzstd] Fix bug in Huffman decompresser
 2017-08-07 14:05:58 -07:00
Nick Terrell abe12b3399 [libzstd] Fix bug in Huffman decompresser 
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.

* Make the decompresser accept Huffman compressed literals whose size
  increases.
* Add a test case that exposes the bug. The compressed file has to be
  statically generated, since the compressor won't normally produce files
  that expose the bug.
 2017-08-07 12:37:48 -07:00
Nick Terrell 308047eb5d Fix compression failure on incompressible data 
If the destination buffer is the minimum allowed size in
`ZSTD_compressSequences()` (2^17), then if the block isn't compressible
compression might fail with `dstSize_tooSmall`, when it should instead emit
a raw uncompressed block.

Additionally, `ZSTD_compressLiterals()` implicitly called
`ZSTD_noCompressLiterals()` if Huffman compression failed. Make that
explicit.
 2017-08-07 11:45:24 -07:00
Stella Lau 73ba58955f Signal after finishing job when queueSize=0 2017-08-01 20:12:06 -07:00
Stella Lau 1d76da1d87 Replace marker with queueEmpty variable and update pool.h comment 2017-08-01 12:30:16 -07:00
Stella Lau 5adceeed01 Allow queueSize=0 in pool.c and update poolTests 2017-07-31 10:10:16 -07:00
Yann ColletandGitHub e1222544be Merge pull request #753 from paulcruz74/adapt-approach-3 
adaptive compression v1
 2017-07-27 10:00:10 -07:00
Nick Terrell ae20d413da [libzstd] Fix CHECK_V_F macros 2017-07-25 12:52:01 -07:00
Yann Collet a90b16e150 Visual blind fix 2 2017-07-20 15:57:55 -07:00
Yann Collet b4d460f32c pool.c : blindfix for Visual warnings 2017-07-20 01:13:14 -07:00
Yann Collet 3974d2b38a blind fix for Windows Multithreading module 
adds a fake 0 return value for mutex/cond init
 2017-07-19 13:33:21 -07:00
Paul Cruz 6945b3c43d removed previous version of completion for compression 2017-07-19 11:51:50 -07:00
Yann Collet b71363b967 check pthread_*_init() success condition 2017-07-19 01:05:40 -07:00
Nick Terrell cc1522351f [libzstd] Fix bug in Huffman encoding 
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.

The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.

Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 243.7 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 226.7 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 246.5 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 229.2 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 149.3 MB/s , 963.6 MB/s

Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
 2017-07-18 13:20:40 -07:00
Yann ColletandGitHub 77d67fb167 Merge pull request #766 from terrelln/real-block-split 
[libzstd] Pull optimal parser state out of seqStore_t
 2017-07-18 08:26:24 -07:00
Yann ColletandGitHub 14c83b05c7 Merge pull request #765 from terrelln/real-block-split 
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
 2017-07-17 19:25:55 -07:00
Nick Terrell 7a28b9e4a3 [libzstd] Pull optimal parser state out of seqStore_t 2017-07-17 15:29:11 -07:00
Yann ColletandGitHub 3381bf4b84 Merge pull request #764 from terrelln/real-block-split 
[libzstd] Refactor ZSTD_compressSequences()
 2017-07-17 14:46:01 -07:00
Nick Terrell e198230645 [libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences() 2017-07-17 12:27:24 -07:00
Nick Terrell 634f012420 [libzstd] Refactor ZSTD_compressSequences() 2017-07-17 11:36:11 -07:00
Paul Cruz 50ce4eaeb6 added error detection for pthread initialization, added compression completion measurement, fixed const values 2017-07-17 10:12:44 -07:00
Yann Collet 3b0cff3c33 fixed clang's -Wdocumentation 2017-07-13 18:58:30 -07:00
Yann Collet 2bd6440be0 pinned down error code enum values 
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.

Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
 2017-07-13 17:12:16 -07:00
Nick Terrell 830ef4152a [libzstd] Increase granularity of FSECTable repeat mode 2017-07-13 12:45:39 -07:00
Yann ColletandGitHub d985319337 Merge pull request #759 from terrelln/real-block-split 
[libzstd] Pull CTables into sub-structure
 2017-07-13 10:24:19 -07:00
Yann Collet 3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) 
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
 2017-07-13 10:17:23 -07:00
Yann Collet 132e6efd76 switched ZSTDMT_compress_advanced() last argument to overlapLog 
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
 2017-07-13 02:22:58 -07:00
Yann Collet 4e77f7761d clarified comment on ZSTD_p_contentSizeFlag 2017-07-13 02:09:07 -07:00
Nick Terrell de0414b736 [libzstd] Pull CTables into sub-structure 2017-07-12 19:49:19 -07:00
Yann Collet 8ef666c325 slightly increased buffer pool, to cover normal "full load" scenarios 
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
 2017-07-12 14:23:34 -07:00
Yann Collet 052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum 
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
 2017-07-11 17:18:26 -07:00