ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,391 Commits 17 Branches 77 Tags
86e83e926f1d8c03447865b340fc6126093eb517
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell 86e83e926f [libzstd] Set CLEVEL_CUSTOM correctly 
In `ZSTD_compressBegin_advanced()`, `ZSTD_parameters` are used to set the
compression parameters, but the level didn't get set to `CLEVEL_CUSTOM`, so
`ZSTD_compressBlock()` used the wrong parameters when checking the source
size.
 2017-10-02 13:43:30 -07:00
Nick Terrell c5d6dde502 Don't size -= 1 in ZSTD_adjustCParams() 
The window size could end up too small if the source size is 2^n + 1.

Credit to OSS-Fuzz
 2017-09-30 14:20:06 -07:00
Nick Terrell d9c1e9125f [fuzz] Small changes for oss-fuzz integration 2017-09-27 18:23:06 -07:00
Nick Terrell b555b7ef41 [libzstd][opt] Simplify repcode logic 2017-09-27 15:30:12 -07:00
Nick Terrell 6c41adfb28 [libzstd] pthread function prefixed with ZSTD_ 
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
 2017-09-27 11:48:48 -07:00
Nick Terrell 471aa385b3 [fuzz] Speed up round trip tests 
* Enforce smaller maximum values for parameters
* Adjust parameters to the source size

The memory usage is reduced by about 5x, which makes the fuzzers run at
least twice as fast, even more so with ASAN/MSAN enabled.
 2017-09-26 14:03:43 -07:00
Nick Terrell c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Nick Terrell 76cb38d085 [zstd] Backport kernel patch from @ColinIanKing 
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com.
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED).
* Benchmarks showed no difference in speed.
 2017-09-25 16:18:23 -07:00
Nick Terrell 917a213254 [fuzz] Determine flags based on compiler version 2017-09-25 15:32:36 -07:00
Nick Terrell 77d5bc2d62 [fuzz][CI] Add regression tests to the CI 2017-09-25 15:32:31 -07:00
Nick Terrell 11e21f23cb [fuzz] Mention the corpora in the README 2017-09-25 15:31:38 -07:00
Nick Terrell 6bb781e0f1 [fuzz] Add regressiontest targets 2017-09-25 15:31:33 -07:00
Nick Terrell bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Nick Terrell bfad5568b5 [fuzz] Make simple_round_trip compile cleanly 2017-09-25 13:28:45 -07:00
Nick Terrell 23199b6daf [fuzz] Fix fuzz.py env flags parsing 2017-09-25 13:28:18 -07:00
Nick Terrell 1c23b64049 [fuzz] fuzz.py can minimize and zip corpora 
* "minimize" minimizes the corpora into an output directory.
* "zip" zips up the minimized corpora, which are ready to deploy.
 2017-09-25 12:04:12 -07:00
Nick Terrell d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Nick Terrell 74718d7e43 [bitstream] Allow adding 31 bits at a time 2017-09-19 13:57:33 -07:00
Nick Terrell 6c9ed76676 [ldm] Fix corner case where minMatch < 8 
There is a potential read buffer overflow when minMatch < 8.

fix-fuzz-failure
 2017-09-19 13:49:37 -07:00
Nick Terrell 18442a31ff [libzstd] Fix bad window size assert 
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.

fix-fuzz-failure
 2017-09-19 13:47:59 -07:00
Nick Terrell 1fe762e236 [zstdcli] Fix LDM advanced options parsing 2017-09-18 14:49:35 -07:00
Nick Terrell cae3e3c652 [fse] Fix FSE_optimalTableLog() for srcSize==1 2017-09-18 14:11:18 -07:00
Nick Terrell 5f22479517 [block] Don't use fParams in ZSTD_decompressBlock() 2017-09-15 17:37:20 -07:00
Nick Terrell 39357c41cb [fuzzer] Fuzz long range matching & new API 2017-09-14 14:48:08 -07:00
Nick Terrell 9712d5ebe6 [fuzzer] Fix bugs in fuzz.py 2017-09-13 19:08:35 -07:00
Nick Terrell a6f08b4783 [fuzzer] Fix FUZZ_seed() 2017-09-13 18:41:32 -07:00
Nick Terrell 6c6412cef9 [fuzzer] Update README.md 2017-09-13 18:23:52 -07:00
Nick Terrell 6b8236cf7e [fuzz] Add fuzzing helper script 2017-09-13 17:45:21 -07:00
Nick Terrell b7e1522330 Add block fuzzers 2017-09-13 17:44:41 -07:00
Nick Terrell def3214d74 [fuzzer] Handle single empty directory 2017-09-13 17:44:30 -07:00
Nick Terrell 8b6c80ada8 Update fuzzer Makefile 2017-09-13 16:16:57 -07:00
Nick Terrell 677c2cbf89 Update fuzzer sources 2017-09-13 16:16:57 -07:00
Nick Terrell 6ab4d5e904 [bench] Use higher resolution timer on POSIX 
The timer used was only accurate up to 0.01 seconds. This timer is accurate up to 1 ns.
It is a monotonic timer that measures the real time difference, not on CPU time.
 2017-09-12 16:46:44 -07:00
Nick Terrell a4eac0db29 Update build scripts 2017-09-05 17:10:29 -07:00
Nick Terrell 721726d688 Split parsers out of zstd_compress.c 2017-09-05 17:10:25 -07:00
Nick Terrell 423b133568 [POOL] Allow free on NULL when multithreading is disabled 2017-09-05 11:18:13 -07:00
Nick Terrell f9252d8347 [linux-kernel] Update license 2017-08-31 12:48:36 -07:00
Nick Terrell 9822f97721 [error] Don't guard undef X with ifdef X 2017-08-29 11:54:38 -07:00
Nick Terrell 02033be08c [pool] Visual Studios disallows empty structs 2017-08-28 17:19:01 -07:00
Nick Terrell 7c365eb02c [threading] Fix ERROR macro after including windows.h 2017-08-28 16:25:02 -07:00
Nick Terrell db3f5372df [zstdmt] Use POOL_create_advanced() 2017-08-24 18:12:28 -07:00
Nick Terrell de6c6bce85 Fix zstd_internal.h for C++ mode 2017-08-24 18:09:50 -07:00
Nick Terrell 26dc040a7b [pool] Accept custom allocators 2017-08-24 17:01:41 -07:00
Nick Terrell 89dc856cae [pool] Fix formatting 2017-08-24 16:48:32 -07:00
Nick Terrell 376f435914 [dictBuilder] Set default compression level to 3 2017-08-24 16:21:05 -07:00
Nick Terrell 29c2d9a4d0 [cover] Turn down notification for ZDICT subroutines 2017-08-21 14:28:31 -07:00
Nick Terrell 98de3f6847 [cover] Add dictionary size to compressed size 2017-08-21 14:23:17 -07:00
Nick Terrell 9a54a315aa [cover] Convert score to U32 and check for zero 2017-08-21 13:30:07 -07:00
Nick Terrell d49eb40c03 [cover] Stop when segmentSize is less than d 2017-08-21 13:10:03 -07:00
Nick Terrell 3587556873 [cover] Test small maxdict 2017-08-21 11:16:47 -07:00
Nick Terrell f306d400c0 [cover] Fix divide by zero 2017-08-21 11:12:11 -07:00
Nick Terrell 07c6ff588e [FSE][HUF] Inline error checks 
Caught by Clang's optimization remarks.
 2017-08-15 11:23:28 -07:00
Nick Terrell 57e2df6651 [kernel] Update squashfs-tools patch 2017-08-14 22:43:36 -07:00
Nick Terrell 565e925eb7 [libzstd] Fix FORCE_INLINE macro 2017-08-14 21:12:05 -07:00
Nick Terrell 8b6702a00d [linux-kernel] Update patches for v5 2017-08-09 13:03:40 -07:00
Nick Terrell 9ba97182d1 [CI] Add gcc7build test 2017-08-08 13:28:56 -07:00
Nick Terrell abe12b3399 [libzstd] Fix bug in Huffman decompresser 
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.

* Make the decompresser accept Huffman compressed literals whose size
  increases.
* Add a test case that exposes the bug. The compressed file has to be
  statically generated, since the compressor won't normally produce files
  that expose the bug.
 2017-08-07 12:37:48 -07:00
Nick Terrell 308047eb5d Fix compression failure on incompressible data 
If the destination buffer is the minimum allowed size in
`ZSTD_compressSequences()` (2^17), then if the block isn't compressible
compression might fail with `dstSize_tooSmall`, when it should instead emit
a raw uncompressed block.

Additionally, `ZSTD_compressLiterals()` implicitly called
`ZSTD_noCompressLiterals()` if Huffman compression failed. Make that
explicit.
 2017-08-07 11:45:24 -07:00
Nick Terrell 7393b49fbd [linux-kernel] Update patches for v4 2017-08-04 16:57:03 -07:00
Nick Terrell ae20d413da [libzstd] Fix CHECK_V_F macros 2017-07-25 12:52:01 -07:00
Nick Terrell 7d3ac0710d [linux] Update patches for v3 2017-07-20 13:33:55 -07:00
Nick Terrell d0b27483ae [zstdcli] Fix -t in streaming mode 2017-07-18 14:45:49 -07:00
Nick Terrell cc1522351f [libzstd] Fix bug in Huffman encoding 
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.

The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.

Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 243.7 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 226.7 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 246.5 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 229.2 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 149.3 MB/s , 963.6 MB/s

Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
 2017-07-18 13:20:40 -07:00
Nick Terrell 7a28b9e4a3 [libzstd] Pull optimal parser state out of seqStore_t 2017-07-17 15:29:11 -07:00
Nick Terrell e198230645 [libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences() 2017-07-17 12:27:24 -07:00
Nick Terrell 634f012420 [libzstd] Refactor ZSTD_compressSequences() 2017-07-17 11:36:11 -07:00
Nick Terrell 830ef4152a [libzstd] Increase granularity of FSECTable repeat mode 2017-07-13 12:45:39 -07:00
Nick Terrell de0414b736 [libzstd] Pull CTables into sub-structure 2017-07-12 19:49:19 -07:00
Nick Terrell bea0f0cfa0 [fuzz] Move from fuzz/ to tests/fuzz/ 2017-07-03 12:40:12 -07:00
Nick Terrell c80fc50a8d [libzstd] Fix memcpy() on potential NULL source 
* `ZSTD_decompressStream_generic()` `ip` may be `NULL` for one of the calls
  to `memcpy()`
* Assert the source is not `NULL` for calls to `memcpy()` where I believe
  the source should not be `NULL`.
 2017-07-03 12:31:55 -07:00
Nick Terrell 4be7f0d45c [fuzz] Add libFuzzer targets 
* The regression driver serves both as a regression test, and as a binary for afl-fuzz.
* Next, we want to check in a seed corpus for each target. Then we can run the regression
  test binary on them on Travis or Circle CI.
 2017-06-30 17:39:56 -07:00
Nick Terrell 88481e4052 [pzstd] Remove appveyor tests 
The appveyor tests sometimes hang, and pzstd is now deprecated in favor of
zstdmt, so delete the tests.
 2017-06-30 16:31:13 -07:00
Nick Terrell 56e3964d85 [man] Specify that strategies start at 1 2017-06-30 16:29:37 -07:00
Nick Terrell ce91b64f00 [linux-kernel] Update patches for v2 
* Reduce stack usage of many zstd functions, none use over 388 B anymore.
* Remove an incorrect `const` in `xxhash`.
 2017-06-28 22:01:46 -07:00
Nick Terrell c2edb78cbe [tests] Fix HAVE_LZMA flag 2017-06-26 22:58:31 -07:00
Nick Terrell 5b7fd7c422 [zdict] Make COVER the default algorithm 2017-06-26 21:09:22 -07:00
Nick Terrell d88a441406 [zstdcli] Document HAVE_LZMA and zstd-noxz 2017-06-26 11:25:00 -07:00
Nick Terrell 6aeb50ea0d [zstdcli] Add unlzma and unxz symlinks 2017-06-26 11:24:36 -07:00
Nick Terrell eb7c2074e8 [zstdcli] Disable lzma support for MSAN testing 2017-06-26 11:23:56 -07:00
Nick Terrell 849ecf3510 [zstdcli] Support xz by default when liblzma is available 2017-06-23 17:11:38 -07:00
Nick Terrell f570609286 [pzstd] Fix deadlock in the case of errors 
Fixes #720.
 2017-06-22 18:09:42 -07:00
Nick Terrell aff86be75d [linux] Write all the patch summaries 2017-06-21 20:23:44 -07:00
Nick Terrell 0a94fead41 [linux] Rename last 2 diffs to patch names 2017-06-21 15:27:07 -07:00
Nick Terrell f3ad4062ba [linux] Add summaries to xxhash and zstd patches 2017-06-21 15:27:07 -07:00
Nick Terrell 34b0a0883a [linux] Rename diffs to patch names 2017-06-21 15:27:07 -07:00
Nick Terrell b789860db0 [linux] Switch BtrFS compression level to 1 2017-06-21 15:27:02 -07:00
Nick Terrell 631d3664b0 [linux] Add xxhash and zstd in-kernel tests 2017-06-19 19:09:41 -07:00
Nick Terrell e7093ca18c [linux] Add libfuzzer targets 2017-06-19 19:09:15 -07:00
Nick Terrell a48e129d48 [linux] Port libzstd UBSAN fix 2017-06-19 15:48:30 -07:00
Nick Terrell 55f9cd4942 [libzstd] Fix UBSAN failure 2017-06-19 15:12:28 -07:00
Nick Terrell 9ad11bea4e [linux] Fix decompression memory allocation 2017-06-19 12:32:04 -07:00
Nick Terrell 53a67ec1a6 [linux-kernel] Fix duplicate symbols when built-in to kernel 2017-05-25 18:34:13 -07:00
Nick Terrell e63fff9b97 [linux-kernel] Fix clang-format edge case with goto labels 2017-05-25 13:56:46 -07:00
Nick Terrell 74b12f4014 [linux-kernel] Fix some up clang-format edge cases 2017-05-24 13:52:36 -07:00
Nick Terrell 55fc1f91fd [zstd] Fix up formatting edge cases for clang-format 2017-05-24 13:50:10 -07:00
Nick Terrell a1280406b0 [libzstd] Allow users to define custom visibility 2017-05-19 18:01:59 -07:00
Nick Terrell ddc68e174b [linux-kernel] Fix a weird clang-format corner case 2017-05-17 15:06:41 -07:00
Nick Terrell a2b21c552f [linux-kernel] Update tests Makefile 2017-05-17 14:56:37 -07:00
Nick Terrell 8d8e65c6c3 [linux-kernel] Update zstd.diff 2017-05-17 14:52:12 -07:00
Nick Terrell 06fa1b0560 [linux-kernel] Move ZSTD_STATIC_ASSERT to zstd_internal.h 2017-05-17 14:51:50 -07:00