ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,024 Commits 17 Branches 77 Tags
d760529a058e38bb7a24b5be31060d4b2a020f6f
Commit Graph
54 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell d760529a05 Fix stack buffer overrun when weightTotal == 0 
If `weightTotal == 0`, then `BIT_highbit32(weightTotal)` is
undefined behavior in the case that it calls `__builtin_clz()`.
If `tableLog == HUF_TABLELOG_ABSOLUTEMAX` then we will access one
byte beyond the end of the buffer.
 2016-10-19 11:39:11 -07:00
Nick Terrell bb68062c59 Unitialized memory read in ZSTD_decodeSeqHeaders() 
Caused by two things:
1. Not checking that `ip` is in range except for the first byte.
2. `ZSTDv0{5,6}_decodeLiteralsBlock()` could return a value larger than `srcSize`.
 2016-10-18 16:41:33 -07:00
Nick Terrell 7b06ad7a05 Backport fix from commit 125d817 
This fixes a read of unitialized memory.
Full commit hash: 125d81774f.
 2016-10-18 14:52:34 -07:00
Nick Terrell f45b157d95 Backport fix from commit 9e8b09a 
Fixes uninitialized memory reads.
Full commit hash: 9e8b09a7bd
 2016-10-18 14:22:49 -07:00
Nick Terrell fd98087047 Fix stack buffer overflow in HUF_readCTable() 
If `w ==0` on line 153, then `CTable[n].nbBits == tableLog + 1`.
Then `nbPerRank[CTable[n].nbBits]` and `valPerRank[CTable[n].nbBits]`
are stack buffer overflows.
 2016-10-17 18:16:59 -07:00
Nick Terrell bfd943ace5 Fix buffer overrun in ZSTD_loadDictEntropyStats() 
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadDictEntropyStats()`.  This caused `FSE_buildCTable()`
to stack/heap overflow in a few places.

The benchmarks look good, there is no obvious compression performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e10 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 271.6 MB/s , 716.8 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 204.8 MB/s , 671.1 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.8 MB/s , 658.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 136.4 MB/s , 665.3 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.9 MB/s , 647.0 MB/s
   6#silesia.tar       : 211988480 ->  62979643 (3.366),  65.2 MB/s , 670.4 MB/s
   7#silesia.tar       : 211988480 ->  61974560 (3.421),  44.9 MB/s , 688.2 MB/s
   8#silesia.tar       : 211988480 ->  61028308 (3.474),  32.4 MB/s , 711.9 MB/s
   9#silesia.tar       : 211988480 ->  60416751 (3.509),  21.1 MB/s , 718.1 MB/s
  10#silesia.tar       : 211988480 ->  60174239 (3.523),  22.2 MB/s , 721.8 MB/s

  > ./compress_zstds/zstd.opt.1 -i10 -b1 -e10 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 273.8 MB/s , 722.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 203.2 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 157.4 MB/s , 666.5 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 132.1 MB/s , 661.9 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.8 MB/s , 641.6 MB/s
   6#silesia.tar       : 211988480 ->  62979643 (3.366),  63.1 MB/s , 677.0 MB/s
   7#silesia.tar       : 211988480 ->  61974560 (3.421),  44.3 MB/s , 678.2 MB/s
   8#silesia.tar       : 211988480 ->  61028308 (3.474),  33.1 MB/s , 708.9 MB/s
   9#silesia.tar       : 211988480 ->  60416751 (3.509),  21.5 MB/s , 710.1 MB/s
  10#silesia.tar       : 211988480 ->  60174239 (3.523),  21.9 MB/s , 723.9 MB/s
 2016-10-17 16:55:52 -07:00
Nick Terrell 4db751668f Fix buffer overrun in ZSTD_loadEntropy() 
The table log set by `FSE_readNCount()` was not checked in
`ZSTD_loadEntropy()`.  This caused `FSE_buildDTable(dctx->MLTable, ...)`
to overwrite the beginning of `dctx->hufTable`.

The benchmarks look good, there is no obvious performance regression:

  > ./zstds/zstd.opt.0 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 268.2 MB/s , 701.0 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.5 MB/s , 666.9 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 154.9 MB/s , 655.6 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 128.9 MB/s , 648.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  98.4 MB/s , 633.4 MB/s

  > ./zstds/zstd.opt.2 -i10 -b1 -e5 ~/bench/silesia.tar
   1#silesia.tar       : 211988480 ->  73656930 (2.878), 266.1 MB/s , 703.7 MB/s
   2#silesia.tar       : 211988480 ->  70162842 (3.021), 199.0 MB/s , 666.6 MB/s
   3#silesia.tar       : 211988480 ->  66997986 (3.164), 156.2 MB/s , 656.2 MB/s
   4#silesia.tar       : 211988480 ->  66002591 (3.212), 133.2 MB/s , 647.4 MB/s
   5#silesia.tar       : 211988480 ->  65008480 (3.261),  96.3 MB/s , 633.3 MB/s
 2016-10-17 15:51:15 -07:00
Nick Terrell ccfcc643da Check if dict is empty before reading first byte 2016-10-17 11:46:03 -07:00
Nick Terrell 3b9cdf9220 Fix ubsan failures (pass NULL to memcpy) 2016-10-12 20:54:42 -07:00
Nick Terrell 7158584399 Fix ZSTD_execSequence() edge case 2016-10-12 10:05:26 -07:00
Nick Terrell 3e82f2b61a Remove references to GPLv2 license 
License headers added to Makefiles were taken from `zstd/Makefile`.
 2016-10-11 15:28:40 -07:00
Nick Terrell 3980167488 [pzstd] Add status update for MB written 2016-09-23 15:47:26 -07:00
Nick Terrell dac0376908 [pzstd] Add header required for Visual Studios 2016-09-23 14:38:25 -07:00
Nick Terrell d249889b9f [pzstd] Print (de)compression results 2016-09-23 13:42:50 -07:00
Nick Terrell e5b60e859b [pzstd] Update README to reflect new CLI 2016-09-23 13:07:54 -07:00
Nick Terrell cd5c52fe37 [pzstd] Add tests to appveyor MinGW64 2016-09-22 19:00:54 -07:00
Nick Terrell 5ca471990b [pzstd] Spawn less threads in tests 
MinGW thread performance degrades significantly when there are
a lot of threads, so limit the number of threads spawned to ~10.
 2016-09-22 18:59:22 -07:00
Nick Terrell 2b4de225e1 Don't redefine macro in util.h 2016-09-22 18:02:39 -07:00
Nick Terrell 3b4093ca5c [pzstd] Add 32 bit tests to travis-ci 2016-09-22 17:45:24 -07:00
Nick Terrell 5b2c0dbed0 Add include guards to datagen.h 2016-09-22 17:12:50 -07:00
Nick Terrell 5eaf5da723 [pzstd] Turn on warnings + quiet them 2016-09-22 16:12:29 -07:00
Nick Terrell c3b8f2b715 Merge remote-tracking branch 'upstream/dev' into dev 
* upstream/dev:
  Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize.
  cli : better adaptation to small files
  fixed : cli : forgotten mandatory argument
  Implemented "command must be followed by argument" protection suggested by @terrelln (#375)
  minor refactor for clarity
  minor build comment
  gmake necessary on *BSD systems.
  added error check
  added error checking
  fixed cmake test
  completed change from projects to build
  changed projects to build
  bumped version number
  streaming compression example can handle situations where input buffer size is manually set to a small value.
  added comments on buffer sizes guarantees
 2016-09-21 17:47:25 -07:00
Nick Terrell 0a5910b23b [pzstd] Fix and test 32 bit support 2016-09-21 17:47:09 -07:00
Nick Terrell 5c9adff7f8 [pzstd] Check if input is a directory 2016-09-21 16:25:08 -07:00
Nick Terrell f1073c1da7 [pzstd] Fix invalid argument message 2016-09-21 16:04:44 -07:00
Nick Terrell 1c209a4feb [pzstd] Reduce memory usage to 60-75% of previous 2016-09-21 15:12:23 -07:00
Nick Terrell 254c5b1692 [pzstd] Make CLI compatible with zstd 2016-09-21 14:29:47 -07:00
Nick Terrell e48fbb9f4c Specify that dictionary ID is little-endian 2016-09-07 14:39:32 -07:00
Nick Terrell 4d4d1ad3b3 Fix minor potential narrowing bug 2016-09-06 20:27:11 -07:00
Nick Terrell 823bf3d08d Fix invalid narrowing conversion to size_t 2016-09-06 20:11:02 -07:00
Nick Terrell b3ed23e18e [pzstd] Add appveyor build commands 2016-09-06 15:01:23 -07:00
Nick Terrell 4db9fbdec7 [pzstd] Compile with minGW 64 2016-09-06 14:00:20 -07:00
Nick Terrell 378d12bb0c [pzstd] Changes to compile on VS2015 2016-09-06 12:43:07 -07:00
Nick Terrell 4c202815c7 [pzstd] Smart default # of threads (#331) 2016-09-06 12:41:36 -07:00
Nick Terrell 9622fe499d Fix memory usage issues. 2016-09-02 20:11:22 -07:00
Nick Terrell 64c1c065cc Add optional max size to work queue 2016-09-02 13:53:23 -07:00
Nick Terrell 2fcf8a4b99 Update tests to reflect new default options 2016-09-02 12:59:14 -07:00
Nick Terrell ac14348a28 When reading from stdin, write to stdout by default 2016-09-02 12:35:36 -07:00
Nick Terrell 1e2f6a1f5d Clean up compiler warnings + Build pzstd on travis 2016-09-02 12:23:49 -07:00
Nick Terrell 2ebe1cf732 Put libzstd last 2016-09-01 21:12:39 -07:00
Nick Terrell 7bf8c4d7ff Add zstd/lib in includes 2016-09-01 20:53:34 -07:00
Nick Terrell 724e3d534f Put linker -l commands at the end 2016-09-01 20:45:55 -07:00
Nick Terrell 779c489a9c Remove if statement, make language cpp 2016-09-01 20:01:07 -07:00
Nick Terrell 4738e221ca Update travis-ci g++ for pzstds job 2016-09-01 19:46:08 -07:00
Nick Terrell 7aa7e5aaab Merge remote-tracking branch 'upstream/dev' into dev 2016-09-01 18:27:18 -07:00
Nick Terrell 7df55e17e9 Fix up Makefiles, and fix include issues compiling with gcc 2016-09-01 18:26:19 -07:00
Nick Terrell bff1c92c87 Minor tweaks to pzstd graph 2016-09-01 17:53:23 -07:00
Nick Terrell 2741677a8d Update scale for compression speed graph 2016-09-01 17:43:07 -07:00
Nick Terrell 2b0830b067 Randomize tests so travis-ci tests can check for existing failures 2016-09-01 16:39:45 -07:00
Nick Terrell 0c28f62d26 Update travis-ci config to include pzstd 2016-09-01 16:26:27 -07:00
Nick Terrell 040cfd8e7d Get ready to add tests to travis-ci 2016-09-01 16:21:19 -07:00
Nick Terrell 4c3b1881f2 Remove old benchmark code 2016-09-01 15:32:10 -07:00
Nick Terrell f381d2d39c Fix small README things 2016-09-01 15:30:36 -07:00
Nick Terrell c932520960 Add PZstandard to contrib/ 2016-09-01 15:25:31 -07:00