ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,124 Commits 17 Branches 77 Tags
d007eb5f9fec9fad15e3d22c71de19339e0c2b9b
Commit Graph
76 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell c8a9ac312b Fix dynamic libzstd symlinks 2016-11-03 12:32:48 -07:00
Nick Terrell 4bafb5aa97 The static library was moved to libzstd.a 2016-11-03 11:32:45 -07:00
Nick Terrell d82efd8a70 ZSTD_compress_usingDict() when dict gets loaded 
Specify that when `dict == NULL || dictSize < 8` no dictionary
gets loaded.
Also add some periods.
 2016-11-02 18:07:16 -07:00
Nick Terrell 25086d9bc6 [pzstd] Move -I flags to PZSTD_CPPFLAGS 2016-10-28 14:24:15 -07:00
Nick Terrell 10bfd0c0d5 Fix ZSTD_execSequence() performance regression 
Commit ae1cb3b3d0 caused the regression.
It is an instruction alignment issue, because if it is `U64 i` instead
of `U32 i`, the regression returns.  This patch fixes the regression
in gcc, but only gets some of the clang performance back.

Benchmarks:
Run on `silesia.tar`.  I only show levels 1-5 because the performance
regression was uniform across all levels.  I did one run on levels
1-19 and it looked good.

| Build | Level | Before | While | After |
|-------|-------|-------:|------:|------:|
| gcc   |     1 |  931.4 | 904.4 | 932.8 |
| gcc   |     2 |  849.1 | 822.6 | 851.2 |
| gcc   |     3 |  815.6 | 790.6 | 818.9 |
| gcc   |     4 |  794.1 | 770.7 | 798.0 |
| gcc   |     5 |  785.7 | 760.7 | 788.8 |
| clang |     1 |  705.5 | 683.2 | 693.8 |
| clang |     2 |  670.0 | 649.2 | 660.7 |
| clang |     3 |  659.6 | 639.8 | 651.4 |
| clang |     4 |  652.5 | 634.7 | 645.9 |
| clang |     5 |  646.9 | 625.5 | 637.7 |
 2016-10-27 16:19:57 -07:00
Nick Terrell ac4310d303 [pzstd] Fix test mode for streaming input 2016-10-26 00:09:39 -07:00
Nick Terrell eb7873a048 ZSTD_execSequence() accepts match in last 7 bytes 
The zstd reference compressor will not emit a match in the last 7
bytes of a block.  The decompressor will also not accept a match
in the last 7 bytes.  This patch makes the decompressor accept a
match in the last 7 bytes.
 2016-10-25 21:24:15 -07:00
Nick Terrell b2c39a22b0 Fix compiler narrowing warning 2016-10-24 14:50:13 -07:00
Nick Terrell f698ad6deb Merge remote-tracking branch 'upstream/dev' into fixes 
* upstream/dev:
  added doc\zstd_manual.html
  added contrib\gen_html
  zstd_compression_format.md moved to doc/
  Fix small bug in ZSTD_execSequence()
  improved ZSTD_compressBlock_opt_extDict_generic
  protect ZSTD_decodeFrameHeader() from invalid usage, as suggested by @spaskob
  zstd_opt.h: small improvement in compression ratio
  improved dicitonary segment merge
  use implicit rules to compile zstd_decompress.c
  detect early impossible decompression scenario in legacy decoder v0.5
  no repeat mode in legacy v0.5
  fixed invalid invocation of dictionary in legacy decoder v0.5
  fix edge case
  fix command line interpretation
  fixed minor corner case
  zstd.h: added the Introduction section
  fixed clang 3.5 warnings
  zstd.h: updated comments
 2016-10-24 13:10:13 -07:00
Nick Terrell f9c9af3c2e Reject dictionaries with incomplete entropy tables 
If a dictionary specifies that a symbol has probability zero in its
`matchLength`, `literalLength`, or `offset` FSE table, but the symbol
appears when compressing input, the compressor fails.

Ensure that dictionaries support all `matchLength`, and `literalLength`
codes.  They must also support all of the `offset` codes required to
represent every possible offset that can appear in the first block.
 2016-10-24 10:42:44 -07:00
Nick Terrell ae1cb3b3d0 Fix small bug in ZSTD_execSequence() 
`memmove(op, match, sequence.matchLength)` is not the desired behavior.
Overlap is allowed, and handled as if we did `*op++ = *match++`, which
is not how `memmove()` handles overlap.

Only triggered if both of the following conditions are met:
* The match spans extDict & currentPrefixSegment
* `oLitEnd <= oend_w < oLitEnd + length1 < oMatchEnd <= oend`.

These two conditions imply that the block is less than 15 bytes long.
This bug isn't triggered by the streaming API, because it allocates
enough space for the window size + the block size, so there cannot be
a match that is within 8 bytes of the end and overlaps with itself.
It cannot be triggered by the block decompression API because all of
the decompressed data is in the currentPrefixSegment.

Introduced by commit 7158584399
 2016-10-21 12:13:44 -07:00
Nick Terrell d760529a05 Fix stack buffer overrun when weightTotal == 0 
If `weightTotal == 0`, then `BIT_highbit32(weightTotal)` is
undefined behavior in the case that it calls `__builtin_clz()`.
If `tableLog == HUF_TABLELOG_ABSOLUTEMAX` then we will access one
byte beyond the end of the buffer.
 2016-10-19 11:39:11 -07:00
Nick Terrell bb68062c59 Unitialized memory read in ZSTD_decodeSeqHeaders() 
Caused by two things:
1. Not checking that `ip` is in range except for the first byte.
2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.
 2016-10-18 16:41:33 -07:00
Nick Terrell 7b06ad7a05 Backport fix from commit 125d817 
This fixes a read of unitialized memory.
Full commit hash: 125d81774f.
 2016-10-18 14:52:34 -07:00
Nick Terrell f45b157d95 Backport fix from commit 9e8b09a 
Fixes uninitialized memory reads.
Full commit hash: 9e8b09a7bd
 2016-10-18 14:22:49 -07:00
Nick Terrell fd98087047 Fix stack buffer overflow in HUF_readCTable() 
If `w ==0` on line 153, then `CTable[n].nbBits == tableLog + 1`.
Then `nbPerRank[CTable[n].nbBits]` and `valPerRank[CTable[n].nbBits]`
are stack buffer overflows.
 2016-10-17 18:16:59 -07:00
Nick Terrell bfd943ace5 Fix buffer overrun in ZSTD_loadDictEntropyStats() 
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadDictEntropyStats()`.  This caused `FSE_buildCTable()`
to stack/heap overflow in a few places.

The benchmarks look good, there is no obvious compression performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e10 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 271.6 MB/s , 716.8 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 204.8 MB/s , 671.1 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.8 MB/s , 658.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 136.4 MB/s , 665.3 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.9 MB/s , 647.0 MB/s
   6#silesia.tar       : 211988480 ->  62979643 (3.366),  65.2 MB/s , 670.4 MB/s
   7#silesia.tar       : 211988480 ->  61974560 (3.421),  44.9 MB/s , 688.2 MB/s
   8#silesia.tar       : 211988480 ->  61028308 (3.474),  32.4 MB/s , 711.9 MB/s
   9#silesia.tar       : 211988480 ->  60416751 (3.509),  21.1 MB/s , 718.1 MB/s
  10#silesia.tar       : 211988480 ->  60174239 (3.523),  22.2 MB/s , 721.8 MB/s

  > ./compress_zstds/zstd.opt.1 -i10 -b1 -e10 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 273.8 MB/s , 722.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 203.2 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 157.4 MB/s , 666.5 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 132.1 MB/s , 661.9 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.8 MB/s , 641.6 MB/s
   6#silesia.tar       : 211988480 ->  62979643 (3.366),  63.1 MB/s , 677.0 MB/s
   7#silesia.tar       : 211988480 ->  61974560 (3.421),  44.3 MB/s , 678.2 MB/s
   8#silesia.tar       : 211988480 ->  61028308 (3.474),  33.1 MB/s , 708.9 MB/s
   9#silesia.tar       : 211988480 ->  60416751 (3.509),  21.5 MB/s , 710.1 MB/s
  10#silesia.tar       : 211988480 ->  60174239 (3.523),  21.9 MB/s , 723.9 MB/s
 2016-10-17 16:55:52 -07:00
Nick Terrell 4db751668f Fix buffer overrun in ZSTD_loadEntropy() 
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadEntropy()`.  This caused `FSE_buildDTable(dctx->MLTable, ...)`
to overwrite the beginning of `dctx->hufTable`.

The benchmarks look good, there is no obvious performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 268.2 MB/s , 701.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.5 MB/s , 666.9 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 154.9 MB/s , 655.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 128.9 MB/s , 648.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.4 MB/s , 633.4 MB/s

  > ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 266.1 MB/s , 703.7 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.0 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.2 MB/s , 656.2 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 133.2 MB/s , 647.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.3 MB/s , 633.3 MB/s
 2016-10-17 15:51:15 -07:00
Nick Terrell ccfcc643da Check if dict is empty before reading first byte 2016-10-17 11:46:03 -07:00
Nick Terrell 8c6c686d0a [pzstd] Fix lantent bug in WorkQueue::push() 2016-10-14 15:26:56 -07:00
Nick Terrell baa152e56e [pzstd] Add Logger class 2016-10-14 15:26:55 -07:00
Nick Terrell e9e151ce31 [pzstd] Reuse ZSTD_{C,D}Stream 2016-10-14 15:26:55 -07:00
Nick Terrell 48294b57c3 [pzstd] Put ErrorHolder into SharedState 2016-10-14 15:26:55 -07:00
Nick Terrell 9b603ee284 [pzstd] Run the reading thread separately 2016-10-14 15:26:55 -07:00
Nick Terrell 4cb5e90a5c [pzstd] Add asan and tsan tests to travis 
gcc-6 tsan is buggy.
It fails to use the correct linker.
It is also broken with `-pie` with linux kernels newer than 4.1, but previous versions require `-pie`...
 2016-10-14 15:26:55 -07:00
Nick Terrell 96e0702c00 [pzstd] Print the correct width ints 2016-10-14 15:26:55 -07:00
Nick Terrell 8b4e84249b [pzstd] Fix Makefile 2016-10-14 15:26:50 -07:00
Nick Terrell 3b9cdf9220 Fix ubsan failures (pass NULL to memcpy) 2016-10-12 20:54:42 -07:00
Nick Terrell 7158584399 Fix ZSTD_execSequence() edge case 2016-10-12 10:05:26 -07:00
Nick Terrell 3e82f2b61a Remove references to GPLv2 license 
License headers added to Makefiles were taken from `zstd/Makefile`.
 2016-10-11 15:28:40 -07:00
Nick Terrell 87629978d3 [pzstd] Fix latent bug in WorkQueue 2016-10-06 15:51:58 -07:00
Nick Terrell b0f6d73002 [pzstd] Remove empty else statement 2016-10-06 12:58:23 -07:00
Nick Terrell 5fa71396e0 [pzstd] Fix typo in readme 2016-10-06 12:45:47 -07:00
Nick Terrell 3980167488 [pzstd] Add status update for MB written 2016-09-23 15:47:26 -07:00
Nick Terrell dac0376908 [pzstd] Add header required for Visual Studios 2016-09-23 14:38:25 -07:00
Nick Terrell d249889b9f [pzstd] Print (de)compression results 2016-09-23 13:42:50 -07:00
Nick Terrell e5b60e859b [pzstd] Update README to reflect new CLI 2016-09-23 13:07:54 -07:00
Nick Terrell cd5c52fe37 [pzstd] Add tests to appveyor MinGW64 2016-09-22 19:00:54 -07:00
Nick Terrell 5ca471990b [pzstd] Spawn less threads in tests 
MinGW thread performance degrades significantly when there are
a lot of threads, so limit the number of threads spawned to ~10.
 2016-09-22 18:59:22 -07:00
Nick Terrell 2b4de225e1 Don't redefine macro in util.h 2016-09-22 18:02:39 -07:00
Nick Terrell 3b4093ca5c [pzstd] Add 32 bit tests to travis-ci 2016-09-22 17:45:24 -07:00
Nick Terrell 5b2c0dbed0 Add include guards to datagen.h 2016-09-22 17:12:50 -07:00
Nick Terrell 5eaf5da723 [pzstd] Turn on warnings + quiet them 2016-09-22 16:12:29 -07:00
Nick Terrell c3b8f2b715 Merge remote-tracking branch 'upstream/dev' into dev 
* upstream/dev:
  Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize.
  cli : better adaptation to small files
  fixed : cli : forgotten mandatory argument
  Implemented "command must be followed by argument" protection suggested by @terrelln (#375)
  minor refactor for clarity
  minor build comment
  gmake necessary on *BSD systems.
  added error check
  added error checking
  fixed cmake test
  completed change from projects to build
  changed projects to build
  bumped version number
  streaming compression example can handle situations where input buffer size is manually set to a small value.
  added comments on buffer sizes guarantees
 2016-09-21 17:47:25 -07:00
Nick Terrell 0a5910b23b [pzstd] Fix and test 32 bit support 2016-09-21 17:47:09 -07:00
Nick Terrell 5c9adff7f8 [pzstd] Check if input is a directory 2016-09-21 16:25:08 -07:00
Nick Terrell f1073c1da7 [pzstd] Fix invalid argument message 2016-09-21 16:04:44 -07:00
Nick Terrell 1c209a4feb [pzstd] Reduce memory usage to 60-75% of previous 2016-09-21 15:12:23 -07:00
Nick Terrell 254c5b1692 [pzstd] Make CLI compatible with zstd 2016-09-21 14:29:47 -07:00
Nick Terrell e48fbb9f4c Specify that dictionary ID is little-endian 2016-09-07 14:39:32 -07:00
Nick Terrell 4d4d1ad3b3 Fix minor potential narrowing bug 2016-09-06 20:27:11 -07:00
Nick Terrell 823bf3d08d Fix invalid narrowing conversion to size_t 2016-09-06 20:11:02 -07:00
Nick Terrell b3ed23e18e [pzstd] Add appveyor build commands 2016-09-06 15:01:23 -07:00
Nick Terrell 4db9fbdec7 [pzstd] Compile with minGW 64 2016-09-06 14:00:20 -07:00
Nick Terrell 378d12bb0c [pzstd] Changes to compile on VS2015 2016-09-06 12:43:07 -07:00
Nick Terrell 4c202815c7 [pzstd] Smart default # of threads (#331) 2016-09-06 12:41:36 -07:00
Nick Terrell 9622fe499d Fix memory usage issues. 2016-09-02 20:11:22 -07:00
Nick Terrell 64c1c065cc Add optional max size to work queue 2016-09-02 13:53:23 -07:00
Nick Terrell 2fcf8a4b99 Update tests to reflect new default options 2016-09-02 12:59:14 -07:00
Nick Terrell ac14348a28 When reading from stdin, write to stdout by default 2016-09-02 12:35:36 -07:00
Nick Terrell 1e2f6a1f5d Clean up compiler warnings + Build pzstd on travis 2016-09-02 12:23:49 -07:00
Nick Terrell 2ebe1cf732 Put libzstd last 2016-09-01 21:12:39 -07:00
Nick Terrell 7bf8c4d7ff Add zstd/lib in includes 2016-09-01 20:53:34 -07:00
Nick Terrell 724e3d534f Put linker -l commands at the end 2016-09-01 20:45:55 -07:00
Nick Terrell 779c489a9c Remove if statement, make language cpp 2016-09-01 20:01:07 -07:00
Nick Terrell 4738e221ca Update travis-ci g++ for pzstds job 2016-09-01 19:46:08 -07:00
Nick Terrell 7aa7e5aaab Merge remote-tracking branch 'upstream/dev' into dev 2016-09-01 18:27:18 -07:00
Nick Terrell 7df55e17e9 Fix up Makefiles, and fix include issues compiling with gcc 2016-09-01 18:26:19 -07:00
Nick Terrell bff1c92c87 Minor tweaks to pzstd graph 2016-09-01 17:53:23 -07:00
Nick Terrell 2741677a8d Update scale for compression speed graph 2016-09-01 17:43:07 -07:00
Nick Terrell 2b0830b067 Randomize tests so travis-ci tests can check for existing failures 2016-09-01 16:39:45 -07:00
Nick Terrell 0c28f62d26 Update travis-ci config to include pzstd 2016-09-01 16:26:27 -07:00
Nick Terrell 040cfd8e7d Get ready to add tests to travis-ci 2016-09-01 16:21:19 -07:00
Nick Terrell 4c3b1881f2 Remove old benchmark code 2016-09-01 15:32:10 -07:00
Nick Terrell f381d2d39c Fix small README things 2016-09-01 15:30:36 -07:00
Nick Terrell c932520960 Add PZstandard to contrib/ 2016-09-01 15:25:31 -07:00