ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,061 Commits 17 Branches 77 Tags
4f73b3b55d83b082aad180816a5d336eff33a243
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell 07c6ff588e [FSE][HUF] Inline error checks 
Caught by Clang's optimization remarks.
 2017-08-15 11:23:28 -07:00
Nick Terrell 57e2df6651 [kernel] Update squashfs-tools patch 2017-08-14 22:43:36 -07:00
Nick Terrell 565e925eb7 [libzstd] Fix FORCE_INLINE macro 2017-08-14 21:12:05 -07:00
Nick Terrell 8b6702a00d [linux-kernel] Update patches for v5 2017-08-09 13:03:40 -07:00
Nick Terrell 9ba97182d1 [CI] Add gcc7build test 2017-08-08 13:28:56 -07:00
Nick Terrell abe12b3399 [libzstd] Fix bug in Huffman decompresser 
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.

* Make the decompresser accept Huffman compressed literals whose size
  increases.
* Add a test case that exposes the bug. The compressed file has to be
  statically generated, since the compressor won't normally produce files
  that expose the bug.
 2017-08-07 12:37:48 -07:00
Nick Terrell 308047eb5d Fix compression failure on incompressible data 
If the destination buffer is the minimum allowed size in
`ZSTD_compressSequences()` (2^17), then if the block isn't compressible
compression might fail with `dstSize_tooSmall`, when it should instead emit
a raw uncompressed block.

Additionally, `ZSTD_compressLiterals()` implicitly called
`ZSTD_noCompressLiterals()` if Huffman compression failed. Make that
explicit.
 2017-08-07 11:45:24 -07:00
Nick Terrell 7393b49fbd [linux-kernel] Update patches for v4 2017-08-04 16:57:03 -07:00
Nick Terrell ae20d413da [libzstd] Fix CHECK_V_F macros 2017-07-25 12:52:01 -07:00
Nick Terrell 7d3ac0710d [linux] Update patches for v3 2017-07-20 13:33:55 -07:00
Nick Terrell d0b27483ae [zstdcli] Fix -t in streaming mode 2017-07-18 14:45:49 -07:00
Nick Terrell cc1522351f [libzstd] Fix bug in Huffman encoding 
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.

The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.

Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 243.7 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 226.7 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 246.5 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 229.2 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 149.3 MB/s , 963.6 MB/s

Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
 2017-07-18 13:20:40 -07:00
Nick Terrell 7a28b9e4a3 [libzstd] Pull optimal parser state out of seqStore_t 2017-07-17 15:29:11 -07:00
Nick Terrell e198230645 [libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences() 2017-07-17 12:27:24 -07:00
Nick Terrell 634f012420 [libzstd] Refactor ZSTD_compressSequences() 2017-07-17 11:36:11 -07:00
Nick Terrell 830ef4152a [libzstd] Increase granularity of FSECTable repeat mode 2017-07-13 12:45:39 -07:00
Nick Terrell de0414b736 [libzstd] Pull CTables into sub-structure 2017-07-12 19:49:19 -07:00
Nick Terrell bea0f0cfa0 [fuzz] Move from fuzz/ to tests/fuzz/ 2017-07-03 12:40:12 -07:00
Nick Terrell c80fc50a8d [libzstd] Fix memcpy() on potential NULL source 
* `ZSTD_decompressStream_generic()` `ip` may be `NULL` for one of the calls
  to `memcpy()`
* Assert the source is not `NULL` for calls to `memcpy()` where I believe
  the source should not be `NULL`.
 2017-07-03 12:31:55 -07:00
Nick Terrell 4be7f0d45c [fuzz] Add libFuzzer targets 
* The regression driver serves both as a regression test, and as a binary for afl-fuzz.
* Next, we want to check in a seed corpus for each target. Then we can run the regression
  test binary on them on Travis or Circle CI.
 2017-06-30 17:39:56 -07:00
Nick Terrell 88481e4052 [pzstd] Remove appveyor tests 
The appveyor tests sometimes hang, and pzstd is now deprecated in favor of
zstdmt, so delete the tests.
 2017-06-30 16:31:13 -07:00
Nick Terrell 56e3964d85 [man] Specify that strategies start at 1 2017-06-30 16:29:37 -07:00
Nick Terrell ce91b64f00 [linux-kernel] Update patches for v2 
* Reduce stack usage of many zstd functions, none use over 388 B anymore.
* Remove an incorrect `const` in `xxhash`.
 2017-06-28 22:01:46 -07:00
Nick Terrell c2edb78cbe [tests] Fix HAVE_LZMA flag 2017-06-26 22:58:31 -07:00
Nick Terrell 5b7fd7c422 [zdict] Make COVER the default algorithm 2017-06-26 21:09:22 -07:00
Nick Terrell d88a441406 [zstdcli] Document HAVE_LZMA and zstd-noxz 2017-06-26 11:25:00 -07:00
Nick Terrell 6aeb50ea0d [zstdcli] Add unlzma and unxz symlinks 2017-06-26 11:24:36 -07:00
Nick Terrell eb7c2074e8 [zstdcli] Disable lzma support for MSAN testing 2017-06-26 11:23:56 -07:00
Nick Terrell 849ecf3510 [zstdcli] Support xz by default when liblzma is available 2017-06-23 17:11:38 -07:00
Nick Terrell f570609286 [pzstd] Fix deadlock in the case of errors 
Fixes #720.
 2017-06-22 18:09:42 -07:00
Nick Terrell aff86be75d [linux] Write all the patch summaries 2017-06-21 20:23:44 -07:00
Nick Terrell 0a94fead41 [linux] Rename last 2 diffs to patch names 2017-06-21 15:27:07 -07:00
Nick Terrell f3ad4062ba [linux] Add summaries to xxhash and zstd patches 2017-06-21 15:27:07 -07:00
Nick Terrell 34b0a0883a [linux] Rename diffs to patch names 2017-06-21 15:27:07 -07:00
Nick Terrell b789860db0 [linux] Switch BtrFS compression level to 1 2017-06-21 15:27:02 -07:00
Nick Terrell 631d3664b0 [linux] Add xxhash and zstd in-kernel tests 2017-06-19 19:09:41 -07:00
Nick Terrell e7093ca18c [linux] Add libfuzzer targets 2017-06-19 19:09:15 -07:00
Nick Terrell a48e129d48 [linux] Port libzstd UBSAN fix 2017-06-19 15:48:30 -07:00
Nick Terrell 55f9cd4942 [libzstd] Fix UBSAN failure 2017-06-19 15:12:28 -07:00
Nick Terrell 9ad11bea4e [linux] Fix decompression memory allocation 2017-06-19 12:32:04 -07:00
Nick Terrell 53a67ec1a6 [linux-kernel] Fix duplicate symbols when built-in to kernel 2017-05-25 18:34:13 -07:00
Nick Terrell e63fff9b97 [linux-kernel] Fix clang-format edge case with goto labels 2017-05-25 13:56:46 -07:00
Nick Terrell 74b12f4014 [linux-kernel] Fix some up clang-format edge cases 2017-05-24 13:52:36 -07:00
Nick Terrell 55fc1f91fd [zstd] Fix up formatting edge cases for clang-format 2017-05-24 13:50:10 -07:00
Nick Terrell a1280406b0 [libzstd] Allow users to define custom visibility 2017-05-19 18:01:59 -07:00
Nick Terrell ddc68e174b [linux-kernel] Fix a weird clang-format corner case 2017-05-17 15:06:41 -07:00
Nick Terrell a2b21c552f [linux-kernel] Update tests Makefile 2017-05-17 14:56:37 -07:00
Nick Terrell 8d8e65c6c3 [linux-kernel] Update zstd.diff 2017-05-17 14:52:12 -07:00
Nick Terrell 06fa1b0560 [linux-kernel] Move ZSTD_STATIC_ASSERT to zstd_internal.h 2017-05-17 14:51:50 -07:00
Nick Terrell 67101c6c77 [linux-kernel] clang-format lib/zstd/ 2017-05-17 14:40:46 -07:00
Nick Terrell 588579f3a1 [linux-kernel] Rename MEM_* functions to ZSTD_* 2017-05-17 14:39:57 -07:00
Nick Terrell 581e5fbe83 Add clang-format config 2017-05-17 13:49:19 -07:00
Nick Terrell d1a5790f8e [linux-kernel] Dual license with GPLv2 2017-05-17 12:08:15 -07:00
Nick Terrell fb10c91b4f [linux-kernel] Update zstd.diff 2017-05-16 17:21:22 -07:00
Nick Terrell 358280f107 Remove cross-platform macros and dead code 2017-05-16 17:16:00 -07:00
Nick Terrell 8eb7a32527 [linux-kernel] Fix whitespace edge case with script 2017-05-16 16:55:52 -07:00
Nick Terrell 0ce0782c97 [linux-kernel] Update kernelize.sh script for edge cases 2017-05-16 16:55:03 -07:00
Nick Terrell 0d26da2f25 [linux-kernel] Don't undef current (use curr) 2017-05-16 16:48:58 -07:00
Nick Terrell f3c9bd014c [linux-kernel] Remove unused variables in test 2017-05-16 16:13:51 -07:00
Nick Terrell 90a1c94965 [linux-kernel] Update header comments 2017-05-16 16:10:17 -07:00
Nick Terrell 1d42ca8f6d Add zstd.diff 2017-05-16 16:02:05 -07:00
Nick Terrell b3ebe9b77c [linux-kernel] Separate xxhash into its own module 2017-05-16 15:52:35 -07:00
Nick Terrell 374f868354 Update whitespace 2017-05-10 17:48:42 -07:00
Nick Terrell a99ece9e79 Update zstd.1.md to use btultra 2017-05-10 16:51:18 -07:00
Nick Terrell 5f2c7213c7 Merge remote-tracking branch 'upstream/dev' into btopt 
* upstream/dev: (305 commits)
  added test for ZSTD_estimateCStreamSize()
  changed variable name, for clarity
  fixed ZSTD_estimateCStreamSize()
  shortened ZSTD_createCStream_Advanced()
  fixed symbols test
  added ZSTD_estimateDStreamSize()
  changed name frameParams into frameHeader
  regroup memory usage function declarations
  separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
  bumped version number
  added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
  Updated ZSTD_freeCCtx()
  updated ZSTD_estimateCCtxSize()
  Updated ZSTD_sizeof_CCtx()
  merged CCtx and CStream as a single same object
  cli : -d and -t do not stop after a failed decompression
  added dev branch CircleCI badge
  added dev branch Appveyor badge
  keep dev branch status only
  creates a binary archive without the `programs` directory
  ...
 2017-05-10 16:49:58 -07:00
Nick Terrell cf4f9403fa [kernel] Update README with SquashFS patch 
Take patch from PR #682 by @iburinoc and update benchmarks.
 2017-05-03 11:07:39 -07:00
Nick Terrell 99972fa9a8 [btrfs] Fix typo in pr_warn() message 2017-05-02 13:57:40 -07:00
Nick Terrell 79a85e1371 [btrfs] Update patch to use ZSTD_init{C,D}Stream() 2017-05-02 13:41:57 -07:00
Nick Terrell f376d47c11 [CLI] Switch dictionary builder on CLI to cover 2017-05-02 11:18:27 -07:00
Nick Terrell 020b960e13 [cover] Make optimization faster 2017-05-02 11:02:48 -07:00
Nick Terrell f2d9ef1dc0 [cover] Optimize case where d <= 8 2017-05-02 11:02:43 -07:00
Nick Terrell 865918dd04 Fix typo in zdict.h 2017-05-02 11:02:37 -07:00
Nick Terrell feba969a69 Fix LZ4_MSG in xzstd 2017-05-02 11:02:16 -07:00
Nick Terrell 0bd5d25d02 [pzstd] Add logging statements to tests 2017-04-27 09:55:19 -07:00
Nick Terrell 5087c1761f Rename ZSTD_create*() to ZSTD_init*() 2017-04-26 13:08:20 -07:00
Nick Terrell b633377d0e Add BtrFS benchmarks 2017-04-11 12:40:53 -07:00
Nick Terrell b4dd3378f1 Add BtrFS patch for 4.10 kernel 2017-04-11 11:50:24 -07:00
Nick Terrell adb1974aec Switch comments to kernel style + limit to 80 cols 2017-04-10 16:52:59 -07:00
Nick Terrell bed6c0a4b0 Remove ZSTDLIB_API from decompress.c 2017-04-10 14:49:34 -07:00
Nick Terrell 9c7371b3ee Improve comments and ordering 2017-04-04 19:02:28 -07:00
Nick Terrell 97693aff98 Add a README 2017-04-04 18:39:22 -07:00
Nick Terrell 7f510a7797 Add Kconfig and Makefile diffs 2017-04-04 18:37:55 -07:00
Nick Terrell 60b34b0d1c fix gitignore 2017-04-04 18:15:21 -07:00
Nick Terrell bb213798d9 Rename zstd_(de)compress.c to (de)compress.c 2017-04-04 18:09:02 -07:00
Nick Terrell 1b3856463e Undef current if defined 2017-04-04 17:21:49 -07:00
Nick Terrell c53bacc881 Fix up some old names 2017-04-04 17:03:48 -07:00
Nick Terrell 2e4dbd2105 Really move the makefile 2017-04-04 17:02:28 -07:00
Nick Terrell 5859101396 Move Makefile to lib/zstd 2017-04-04 17:00:12 -07:00
Nick Terrell d0bbceac40 Add fs/btrfs/zstd.c 2017-04-04 16:56:52 -07:00
Nick Terrell 2eb623a6eb Explicitly convert scratchBuffer to unsigned* 2017-04-04 16:38:13 -07:00
Nick Terrell 405d2a1027 Explicitly convert scratchBuffer to unsigned* 2017-04-04 16:35:31 -07:00
Nick Terrell fd95be0ade Move sources to lib/zstd/ 2017-04-04 16:30:11 -07:00
Nick Terrell 1af700ea43 Remove unused FSE/HUF functions 2017-04-04 16:22:58 -07:00
Nick Terrell 16a739cab0 Switch call of FSE_count() to FSE_count_wksp() 2017-04-04 16:17:21 -07:00
Nick Terrell dd62829ccf Remove more restrict 2017-04-04 15:55:48 -07:00
Nick Terrell 9c257dc268 Fix up xxhash 2017-04-04 15:54:39 -07:00
Nick Terrell 0888251fb1 Switch intptr_t to ptrdiff_t 2017-04-04 15:49:38 -07:00
Nick Terrell 675839254d Move zstd.h to linux/zstd.h 2017-04-04 15:44:27 -07:00
Nick Terrell b06507221e Test module macros 2017-04-04 12:56:44 -07:00
Nick Terrell b1b582b9fa Add module macros 2017-04-04 12:56:35 -07:00