ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,753 Commits 17 Branches 77 Tags
421a2716d8e9c6ed796357810c8502aa4002222e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell 48acaddff9 Test for incorrect pledgeSrcSize earlier 2018-02-01 12:04:05 -08:00
Nick Terrell 9c8c69e41a [fuzzer] Check ZSTD_initStaticCDict() for every level 2018-01-31 11:12:54 -08:00
Nick Terrell ab3346af07 Fix hashLog3 size when copying cdict tables 2018-01-31 11:12:17 -08:00
Nick Terrell 9d96761520 Set repcodes for empty ZSTD_CDict 
When the dictionary is <= 8 bytes, no data is loaded from the dictionary.
In this case the repcodes weren't set, because they were inserted after the
size check. Fix this problem in general by first setting the cdict state to
a clean state of an empty dictionary, then filling the state from there.
 2018-01-18 13:28:30 -08:00
Nick Terrell 16bd0fd4df Reduce size of ZSTD_CDict 
Shaves 492,076 B off of the `ZSTD_CDict`.
The size of a `ZSTD_CDict` created from a 112,640 B dictionary is:

| Level | Before (B) | After (B) |
|-------|------------|-----------|
| 1     | 648,448    | 156,412   |
| 3     | 1,140,008  | 647,932   |
 2018-01-17 11:50:49 -08:00
Nick Terrell aae267a2e1 Reorganize block state 2018-01-16 11:17:50 -08:00
Nick Terrell 887cd4e35e Split ZSTD_CCtx into smaller sub-structures 2018-01-16 11:17:50 -08:00
Nick Terrell 9a211d1f05 Load more dictionary positions into table if empty 
If the hash table is empty load positions into the hash table
that we would otherwise skip.

| Level | Data Set     | Improvement |
|-------|--------------|-------------|
| 1     | github       | 0.44%       |
| 1     | hg-changelog | 0.13%       |
| 1     | hg-commands  | 1.28%       |
| 1     | hg-manifest  | 0.70%       |
| 3     | github       | 0.74%       |
| 3     | hg-changelog | 0.87%       |
| 3     | hg-commands  | 1.74%       |
| 3     | hg-manifest  | 0.23%       |
 2018-01-12 16:17:22 -08:00
Nick Terrell 4b7c4e5f41 Add test for cdict window log adjustment 2018-01-11 16:45:16 -08:00
Nick Terrell b610b777d3 Increase windowLog from CDict based on the srcSize when known 2018-01-11 16:23:21 -08:00
Nick Terrell ed9611dc62 [fileio] Don't call FIO_remove() on stdout or /dev/null 2018-01-05 11:50:24 -08:00
Nick Terrell 282ad05e0a [fileio] Use FIO_remove() everywhere for safety 2018-01-05 11:44:45 -08:00
Nick Terrell fd63140e1c [util] Refuse to set file stat on non-regular file 2018-01-05 11:44:25 -08:00
Nick Terrell 8adebbd0f8 Add test case for PR #972 2018-01-03 15:52:18 -08:00
Nick Terrell f48d34edba [fileio] Merge end loop for small optimization 2017-12-14 15:52:24 -08:00
Nick Terrell 4680e85bdf Allow -o with multiple files 2017-12-13 17:44:34 -08:00
Nick Terrell 82bc8fe0cc [fileio] Refuse to remove non-regular file 2017-12-13 13:38:26 -08:00
Nick Terrell 22727a7467 Fix cdict compressor repcodes 2017-12-13 11:31:20 -08:00
Nick Terrell b5e7f6c0f3 [fileio] Fix window size MB calculation 
Test command:
```
head -c 10000 /dev/zero | ./zstd -c --zstd=wlog=12 | ./zstd -M2048 -t
```
 2017-12-13 10:57:01 -08:00
Nick Terrell dab8cfa3c7 Combine definitions of SEC_TO_MICRO 2017-11-30 19:40:53 -08:00
Nick Terrell 9a2f6f477b Use util.h for timing 2017-11-30 14:57:25 -08:00
Nick Terrell a6052af0e8 [zstd] Fix rare bug with signal handler 2017-11-17 16:38:56 -08:00
Nick Terrell e19b0822bc Test large skippable frames 2017-11-01 13:10:03 -07:00
Nick Terrell 1fc4f593da Allow skippable frames of any size 2017-11-01 13:07:26 -07:00
Nick Terrell 86b8134cad [libzstd] Fix parameter selection for empty input 
ZSTD_compress() and friends would treat an empty input as an unknown size
when selecting parameters. Thus, they would drastically overallocate the
context. Tell ZSTD_getParams() that the source size is 1 when it is empty.
 2017-10-25 17:24:15 -07:00
Nick Terrell b495140f67 Update BUCK files 
* Correct XXH namespace (Fixes #901)
* Multithreading always enabled
* GZIP/LZ4/LZMA always enabled
* Legacy support always fully enabled
 2017-10-25 12:47:57 -07:00
Nick Terrell e600b5d0da [test] Exercise all codes in dictionary tables 2017-10-16 18:05:36 -07:00
Nick Terrell 7f961ba6cd Don't allow default tables to repeat 
It isn't useful in any case to repeat default tables.
Saves a few bytes on Silesia, since we don't trigger the dictionary
heuristic.

Before: 211988480 => 73651998 bytes
After:  211988480 => 73651721 bytes
 2017-10-16 11:37:56 -07:00
Nick Terrell ced6e6189c Add DEBUGLOG() that prints FSE encoding types 2017-10-13 14:55:23 -07:00
Nick Terrell 24ac2dbd2a Fix invalid use of dictionary offcode table 
Fixes #888.
 2017-10-13 12:47:03 -07:00
Nick Terrell 6d8778d8e7 [doc] Add images for release notes 2017-10-09 16:44:31 -07:00
Nick Terrell 6dd958eea2 [zstdcli] Add window size to verbose list 
```
> zstd --list -v file1 file2 file3
*** zstd command line interface 64-bits v1.3.2, by Yann Collet ***
Window Size: 512.00 KB (524288 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64

Window Size: 8192.00 KB (8388608 B)
Compressed Size: 0.02 KB (19 B)
Check: XXH64

Window Size: 512.00 KB (524288 B)
Compressed Size: 0.01 KB (15 B)
Check: None

```
 2017-10-04 12:26:28 -07:00
Nick Terrell a86a7097ec Ensure dictionary Huff table can encode any symbol 
* Ensure that the dictionary Huffman CTable has maxSymbolValue 255.
* Fix a stack buffer overflow during compression dictionary loading.
 2017-10-03 13:22:13 -07:00
Nick Terrell 86e83e926f [libzstd] Set CLEVEL_CUSTOM correctly 
In `ZSTD_compressBegin_advanced()`, `ZSTD_parameters` are used to set the
compression parameters, but the level didn't get set to `CLEVEL_CUSTOM`, so
`ZSTD_compressBlock()` used the wrong parameters when checking the source
size.
 2017-10-02 13:43:30 -07:00
Nick Terrell c5d6dde502 Don't size -= 1 in ZSTD_adjustCParams() 
The window size could end up too small if the source size is 2^n + 1.

Credit to OSS-Fuzz
 2017-09-30 14:20:06 -07:00
Nick Terrell d9c1e9125f [fuzz] Small changes for oss-fuzz integration 2017-09-27 18:23:06 -07:00
Nick Terrell b555b7ef41 [libzstd][opt] Simplify repcode logic 2017-09-27 15:30:12 -07:00
Nick Terrell 6c41adfb28 [libzstd] pthread function prefixed with ZSTD_ 
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
 2017-09-27 11:48:48 -07:00
Nick Terrell 471aa385b3 [fuzz] Speed up round trip tests 
* Enforce smaller maximum values for parameters
* Adjust parameters to the source size

The memory usage is reduced by about 5x, which makes the fuzzers run at
least twice as fast, even more so with ASAN/MSAN enabled.
 2017-09-26 14:03:43 -07:00
Nick Terrell c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Nick Terrell 76cb38d085 [zstd] Backport kernel patch from @ColinIanKing 
* Make the U32 table in `FSE_normalizeCount()` static.
* Patch from https://lkml.kernel.org/r/20170922145946.14316-1-colin.king@canonical.com.
* Clang makes non-static tables static anyways. gcc however, does [weird things](https://godbolt.org/g/fvTcED).
* Benchmarks showed no difference in speed.
 2017-09-25 16:18:23 -07:00
Nick Terrell 917a213254 [fuzz] Determine flags based on compiler version 2017-09-25 15:32:36 -07:00
Nick Terrell 77d5bc2d62 [fuzz][CI] Add regression tests to the CI 2017-09-25 15:32:31 -07:00
Nick Terrell 11e21f23cb [fuzz] Mention the corpora in the README 2017-09-25 15:31:38 -07:00
Nick Terrell 6bb781e0f1 [fuzz] Add regressiontest targets 2017-09-25 15:31:33 -07:00
Nick Terrell bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Nick Terrell bfad5568b5 [fuzz] Make simple_round_trip compile cleanly 2017-09-25 13:28:45 -07:00
Nick Terrell 23199b6daf [fuzz] Fix fuzz.py env flags parsing 2017-09-25 13:28:18 -07:00
Nick Terrell 1c23b64049 [fuzz] fuzz.py can minimize and zip corpora 
* "minimize" minimizes the corpora into an output directory.
* "zip" zips up the minimized corpora, which are ready to deploy.
 2017-09-25 12:04:12 -07:00
Nick Terrell d6abb28951 Prepare for ZSTD_WINDOWLOG_MAX == 31 2017-09-21 17:18:41 -07:00
Nick Terrell 74718d7e43 [bitstream] Allow adding 31 bits at a time 2017-09-19 13:57:33 -07:00
Nick Terrell 6c9ed76676 [ldm] Fix corner case where minMatch < 8 
There is a potential read buffer overflow when minMatch < 8.

fix-fuzz-failure
 2017-09-19 13:49:37 -07:00
Nick Terrell 18442a31ff [libzstd] Fix bad window size assert 
The window size is not validated or used in the one-pass API, so there
shouldn't be an assert based on it.

fix-fuzz-failure
 2017-09-19 13:47:59 -07:00
Nick Terrell 1fe762e236 [zstdcli] Fix LDM advanced options parsing 2017-09-18 14:49:35 -07:00
Nick Terrell cae3e3c652 [fse] Fix FSE_optimalTableLog() for srcSize==1 2017-09-18 14:11:18 -07:00
Nick Terrell 5f22479517 [block] Don't use fParams in ZSTD_decompressBlock() 2017-09-15 17:37:20 -07:00
Nick Terrell 39357c41cb [fuzzer] Fuzz long range matching & new API 2017-09-14 14:48:08 -07:00
Nick Terrell 9712d5ebe6 [fuzzer] Fix bugs in fuzz.py 2017-09-13 19:08:35 -07:00
Nick Terrell a6f08b4783 [fuzzer] Fix FUZZ_seed() 2017-09-13 18:41:32 -07:00
Nick Terrell 6c6412cef9 [fuzzer] Update README.md 2017-09-13 18:23:52 -07:00
Nick Terrell 6b8236cf7e [fuzz] Add fuzzing helper script 2017-09-13 17:45:21 -07:00
Nick Terrell b7e1522330 Add block fuzzers 2017-09-13 17:44:41 -07:00
Nick Terrell def3214d74 [fuzzer] Handle single empty directory 2017-09-13 17:44:30 -07:00
Nick Terrell 8b6c80ada8 Update fuzzer Makefile 2017-09-13 16:16:57 -07:00
Nick Terrell 677c2cbf89 Update fuzzer sources 2017-09-13 16:16:57 -07:00
Nick Terrell 6ab4d5e904 [bench] Use higher resolution timer on POSIX 
The timer used was only accurate up to 0.01 seconds. This timer is accurate up to 1 ns.
It is a monotonic timer that measures the real time difference, not on CPU time.
 2017-09-12 16:46:44 -07:00
Nick Terrell a4eac0db29 Update build scripts 2017-09-05 17:10:29 -07:00
Nick Terrell 721726d688 Split parsers out of zstd_compress.c 2017-09-05 17:10:25 -07:00
Nick Terrell 423b133568 [POOL] Allow free on NULL when multithreading is disabled 2017-09-05 11:18:13 -07:00
Nick Terrell f9252d8347 [linux-kernel] Update license 2017-08-31 12:48:36 -07:00
Nick Terrell 9822f97721 [error] Don't guard undef X with ifdef X 2017-08-29 11:54:38 -07:00
Nick Terrell 02033be08c [pool] Visual Studios disallows empty structs 2017-08-28 17:19:01 -07:00
Nick Terrell 7c365eb02c [threading] Fix ERROR macro after including windows.h 2017-08-28 16:25:02 -07:00
Nick Terrell db3f5372df [zstdmt] Use POOL_create_advanced() 2017-08-24 18:12:28 -07:00
Nick Terrell de6c6bce85 Fix zstd_internal.h for C++ mode 2017-08-24 18:09:50 -07:00
Nick Terrell 26dc040a7b [pool] Accept custom allocators 2017-08-24 17:01:41 -07:00
Nick Terrell 89dc856cae [pool] Fix formatting 2017-08-24 16:48:32 -07:00
Nick Terrell 376f435914 [dictBuilder] Set default compression level to 3 2017-08-24 16:21:05 -07:00
Nick Terrell 29c2d9a4d0 [cover] Turn down notification for ZDICT subroutines 2017-08-21 14:28:31 -07:00
Nick Terrell 98de3f6847 [cover] Add dictionary size to compressed size 2017-08-21 14:23:17 -07:00
Nick Terrell 9a54a315aa [cover] Convert score to U32 and check for zero 2017-08-21 13:30:07 -07:00
Nick Terrell d49eb40c03 [cover] Stop when segmentSize is less than d 2017-08-21 13:10:03 -07:00
Nick Terrell 3587556873 [cover] Test small maxdict 2017-08-21 11:16:47 -07:00
Nick Terrell f306d400c0 [cover] Fix divide by zero 2017-08-21 11:12:11 -07:00
Nick Terrell 07c6ff588e [FSE][HUF] Inline error checks 
Caught by Clang's optimization remarks.
 2017-08-15 11:23:28 -07:00
Nick Terrell 57e2df6651 [kernel] Update squashfs-tools patch 2017-08-14 22:43:36 -07:00
Nick Terrell 565e925eb7 [libzstd] Fix FORCE_INLINE macro 2017-08-14 21:12:05 -07:00
Nick Terrell 8b6702a00d [linux-kernel] Update patches for v5 2017-08-09 13:03:40 -07:00
Nick Terrell 9ba97182d1 [CI] Add gcc7build test 2017-08-08 13:28:56 -07:00
Nick Terrell abe12b3399 [libzstd] Fix bug in Huffman decompresser 
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.

* Make the decompresser accept Huffman compressed literals whose size
  increases.
* Add a test case that exposes the bug. The compressed file has to be
  statically generated, since the compressor won't normally produce files
  that expose the bug.
 2017-08-07 12:37:48 -07:00
Nick Terrell 308047eb5d Fix compression failure on incompressible data 
If the destination buffer is the minimum allowed size in
`ZSTD_compressSequences()` (2^17), then if the block isn't compressible
compression might fail with `dstSize_tooSmall`, when it should instead emit
a raw uncompressed block.

Additionally, `ZSTD_compressLiterals()` implicitly called
`ZSTD_noCompressLiterals()` if Huffman compression failed. Make that
explicit.
 2017-08-07 11:45:24 -07:00
Nick Terrell 7393b49fbd [linux-kernel] Update patches for v4 2017-08-04 16:57:03 -07:00
Nick Terrell ae20d413da [libzstd] Fix CHECK_V_F macros 2017-07-25 12:52:01 -07:00
Nick Terrell 7d3ac0710d [linux] Update patches for v3 2017-07-20 13:33:55 -07:00
Nick Terrell d0b27483ae [zstdcli] Fix -t in streaming mode 2017-07-18 14:45:49 -07:00
Nick Terrell cc1522351f [libzstd] Fix bug in Huffman encoding 
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.

The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.

Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 243.7 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 226.7 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 246.5 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 229.2 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 149.3 MB/s , 963.6 MB/s

Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
 2017-07-18 13:20:40 -07:00
Nick Terrell 7a28b9e4a3 [libzstd] Pull optimal parser state out of seqStore_t 2017-07-17 15:29:11 -07:00
Nick Terrell e198230645 [libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences() 2017-07-17 12:27:24 -07:00
Nick Terrell 634f012420 [libzstd] Refactor ZSTD_compressSequences() 2017-07-17 11:36:11 -07:00
Nick Terrell 830ef4152a [libzstd] Increase granularity of FSECTable repeat mode 2017-07-13 12:45:39 -07:00