ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,916 Commits 17 Branches 77 Tags
edd8327097d7e909b52025056933f86980495734
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell edd8327097 Update CHANGELOG for the next release 2020-09-09 17:13:34 -07:00
Nick Terrell b92569a522 [doc] Document new build macros in lib/README.md 2020-09-09 17:13:16 -07:00
Nick TerrellandGitHub 5118e30cfe Merge pull request #2289 from terrelln/zstd-kernel-2 
[linux-kernel] Prepare zstd for the Linux Kernel
 2020-09-09 15:38:02 -07:00
Nick Terrell 3420c202af [linux-kernel] Update README with initial instructions on how to update zstd in the kernel 2020-09-09 14:36:22 -07:00
Nick Terrell aab4bf7b0d [linux-kernel] Add test that checks the ifdef hardwiring 2020-09-09 14:36:19 -07:00
Nick Terrell 29c5de8780 Add linux-kernel freestanding 2020-09-09 14:35:57 -07:00
Nick Terrell 1c3cb2c05c [contrib] Add preprocessor hardwiring to freestanding.py 2020-09-09 14:35:39 -07:00
Nick Terrell 79ded1b4a9 [lib] Add ZSTD_NO_UNUSED_FUNCTIONS macro to hide unused functions 
The unused function definitions are hidden behind a
`#ifndef ZSTD_NO_UNUSED_FUNCTIONS` check.

Initially hiding all functions which are unused and take up more than
2KB of stack space, because these will show up as warnings in the
Linux Kernel build system.
 2020-09-09 14:35:39 -07:00
Nick Terrell ac3a136b0a [lib] Replace 64-bit divisions with ZSTD_div64() 2020-09-09 14:35:39 -07:00
Nick Terrell a90779397a [lib] Reduce zstd stack usage by 1KB 2020-09-09 14:35:39 -07:00
Nick Terrell 046aca190f Fix ZSTD_initCStream_advanced() with no dictionary and static allocation 2020-09-09 14:35:39 -07:00
Nick Terrell e975de289c Add ZSTD_NO_INTRINSICS macro to avoid explicit intrinsics 2020-09-09 14:35:39 -07:00
Nick Terrell f91ed5c766 [lib] s/current/curr because it collides with Linux Kernel macro 2020-09-09 14:35:39 -07:00
Nick TerrellandGitHub 5e4efd22d4 Merge pull request #2291 from i-do-cpp/fix-compression-level-default 
Fix setParameter not falling back to default compression level
 2020-09-08 16:42:34 -07:00
Nick TerrellandGitHub 6da8acd231 Merge pull request #2293 from allanjude/coverity 
Resolve Coverity 1432392 Unintentional integer overflow
 2020-09-03 13:58:45 -07:00
Nick TerrellandGitHub bcedab0731 Merge pull request #2279 from senhuang42/show_warnings_with_multiple_input_files 
Add a warning whenever (de)compressing multiple files
 2020-09-02 11:32:11 -07:00
Nick TerrellandGitHub 4c408866f0 Merge pull request #2264 from terrelln/zstd-kernel 
Prepare for freestanding library and add initial translator script
 2020-08-26 18:15:25 -07:00
Nick Terrell ae455dde08 [contrib] Add freestanding translator prototype 
This is the idea, some of the functionality isn't yet implemented.
 2020-08-26 12:26:05 -07:00
Nick Terrell c465f24457 ZSTD_ prefix mem{cpy,move,set},malloc,calloc,free 2020-08-26 12:26:03 -07:00
Nick Terrell a686d306d2 Rename ZSTD_{malloc,calloc,free} to ZSTD_custom{Malloc,Calloc,Free} 2020-08-26 12:25:08 -07:00
Nick Terrell 80f577baa2 Move standard includes to zstd_deps.h 2020-08-26 12:25:08 -07:00
Nick TerrellandGitHub cf83aceaf3 Merge pull request #2282 from terrelln/ncount-fix 
[bug] Fix FSE_readNCount()
 2020-08-26 10:31:07 -07:00
Nick TerrellandGitHub 3bcdcaa042 Merge pull request #2285 from terrelln/oss-fuzz-fixes 
[fuzz] Fix oss-fuzz test problems
 2020-08-25 18:35:16 -07:00
Nick Terrell ae163015b1 [fuzz] Fix stream_decompress timeouts 2020-08-25 17:13:09 -07:00
Nick Terrell 49eeb2d1fc [fuzz] Disable superblock expansion test 2020-08-25 17:13:06 -07:00
Nick Terrell 51dd7e69f9 [github actions] Disable armbuild and armfuzz tests 2020-08-25 16:28:41 -07:00
Nick Terrell 4193638996 [bug] Fix FSE_readNCount() 
* Fix bug introduced in PR #2271
* Fix long-standing bug that is impossible to trigger inside of zstd
* Add a fuzzer that makes sure the normalized count always round trips
  correctly
 2020-08-25 15:42:41 -07:00
Nick TerrellandGitHub 614e446000 Merge pull request #2271 from terrelln/small-blocks 
Small block optimizations
 2020-08-24 18:54:33 -07:00
Nick Terrell 52f33a1da5 Fix compiler warnings 2020-08-24 16:09:45 -07:00
Nick TerrellandGitHub 6f301a7903 Merge pull request #2272 from terrelln/dstSize_tooSmall 
[fix] Always return dstSize_tooSmall when it is the case
 2020-08-24 15:01:17 -07:00
Nick Terrell 6d2f750b37 Document the BMI2 default() functions 2020-08-24 14:44:33 -07:00
Nick Terrell cebe0b5c0b Improve FSE_normalizeCount() docs 2020-08-24 13:58:34 -07:00
Nick Terrell 1302f8d676 [fix] Always return dstSize_tooSmall when it is the case 2020-08-24 13:38:13 -07:00
Nick Terrell 8def0e5fd3 Fix up code after reading through 2020-08-24 12:24:45 -07:00
Nick Terrell 8f8bd2d1ac [regression] Update results.csv 2020-08-20 12:41:35 -07:00
Nick Terrell 575731b6db Use ncount=1 when < 4096 symbols 2020-08-18 16:47:53 -07:00
Nick Terrell 612e947c5e wire up bmi2 support 2020-08-17 16:35:28 -07:00
Nick Terrell ba1fd17a9f speed up literal header decoding 2020-08-17 12:17:53 -07:00
Nick Terrell 6004c1117f speed up small blocks 2020-08-16 23:03:38 -07:00
Nick Terrell a8006264cf small blocks benchmark 2020-08-14 18:57:20 -07:00
Nick TerrellandGitHub 49069a3f0a Merge pull request #2265 from terrelln/prefetch-fix 
Prefer __builtin_prefetch over inline asm
 2020-08-11 10:07:53 -07:00
Nick Terrell e3bda594ae Prefer __builtin_prefetch over inline asm 
Reorder the ifdefs for the PREFETCH macros so that the compiler builtin is
favored over the inline assembly for aarch64.
 2020-08-10 22:17:18 -07:00
Nick TerrellandGitHub f85a0f8bcf Merge pull request #2256 from helloguo/dev 
Optimize ZSTD_wildcopy
 2020-07-29 11:57:49 -07:00
Nick TerrellandGitHub 01261bc8b6 Merge pull request #2252 from helloguo/dev 
Fix preprocessor in ZSTD_wildcopy
 2020-07-27 12:59:10 -07:00
Nick TerrellandGitHub bd21e4b264 Merge pull request #2242 from evverx/cifuzz-ubsan-msan 
cifuzz: turn on UBsan and MSan
 2020-07-13 14:26:13 -04:00
Nick TerrellandGitHub bdf716bc64 Merge pull request #2239 from evverx/evverx-cifuzz 
cifuzz: don't upload artifacts when CIFuzz fails to build fuzz targets
 2020-07-10 11:26:15 -04:00
Nick TerrellandGitHub a2b6a8fbc6 Merge pull request #2220 from dkcasset/fix-lib-Makefile-older-sed-version 
Replace -E option with equivalent -r for older versions of sed
 2020-06-29 17:19:24 -07:00
Nick TerrellandGitHub 3690fe30f3 Merge pull request #2221 from BioDataAnalysis/bda_install_dll 
Install MSVC runtime shared libraries (DLL) with cmake
 2020-06-29 17:14:50 -07:00
Nick TerrellandGitHub 7afd5d85d3 Merge pull request #2218 from terrelln/assert-seq 
Fix unused variable warnings in fuzzing build mode without asserts
 2020-06-22 17:41:18 -07:00
Nick TerrellandGitHub 081691a3aa Merge pull request #2217 from terrelln/cover-redundant 
[cover] Remove unnecessary mask and dedup hash functions
 2020-06-22 17:41:13 -07:00
Nick TerrellandGitHub 370933fa20 Merge pull request #2209 from Niadb/dev 
Explicitly use __cdecl for qsort, to avoid warning when default calling convention is not __cdecl
 2020-06-22 17:41:03 -07:00
Nick Terrell cce0edfdbe Fix unused variable warnings in fuzzing build mode without asserts 
Fix unused vairable warnings when `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is defined but asserts are disabled.

Fixes #2210.
 2020-06-22 12:56:57 -07:00
Nick Terrell 2312b819af [cover] Remove unnecessary mask and dedup hash functions 
* Remove the unnecessary mask, since `ZSTD_hash*()` already ensures
  the output is mod 2^h.
* Dedup the hash functions and use `ZSTD_hash*()` directly.
 2020-06-22 12:52:13 -07:00
Nick TerrellandGitHub 78601d0806 Merge pull request #2212 from cwoffenden/single-file-dict 
Single-file libs now include dictBuilder
 2020-06-22 12:46:54 -07:00
Nick TerrellandGitHub 34c0d85916 Merge pull request #2214 from terrelln/superblock-cross-entropy 
[superblock] Add defensive assert and bounds check
 2020-06-22 12:27:58 -07:00
Nick Terrell 1047097dad [superblock] Add defensive assert and bounds check 
The bound check condition should always be met because we selected `set_basic` as
our encoding type. But that code is very far away, so assert it is true so if it is
ever false we can catch it, and add a bounds check.

Fixes #2213.
 2020-06-22 10:21:38 -07:00
Nick TerrellandGitHub b6a9ded994 Merge pull request #2202 from terrelln/dict-fi 
[lib] Allow compression dictionaries with missing symbols
 2020-06-15 10:54:33 -07:00
Nick Terrell 08981d2638 [lib] Allow compression dictionaries with missing symbols 
Allow compression to use dictionaries with missing symbols in their
entropy tables. We set the FSE repeat mode to check when there are
missing symbols, and set the FSE repeat mode to valid when all symbols
are present.

Note that when not all symbols are present, the heuristics which favor
dictionary tables for lower compression levels won't activate.

Tested by manually creating a dictionary with missing symbols of every
type, and validing that the compressor rejects it before this change,
and accepts it after this change. Also, I ran the `dictionary_loader`
fuzzer for >1 hour of CPU time without running into cases where
compression succeeds, but decompression fails.

Fixes #2174.
 2020-06-12 17:57:19 -07:00
Nick TerrellandGitHub 9eb2ccc9fb Merge pull request #2167 from terrelln/pzstd-ms-fix 
[pzstd] Fix compilation error in MinGW
 2020-05-22 23:38:27 -07:00
Nick Terrell 5aa5aa4df7 [pzstd] Fix compilation error in MinGW 2020-05-22 22:26:02 -07:00
Nick Terrell 651d3d73e0 [test] Update the ldm loadedDictEnd test to cover zstdmt 2020-05-19 16:14:14 -07:00
Nick Terrell 3cc227e90e [ldm][mt] Fix loadedDictEnd 2020-05-19 15:55:03 -07:00
Nick TerrellandGitHub 0dcd3eec43 Merge pull request #2152 from terrelln/simple-rt-bound 
[fuzz] Expand the allowedExpansion
 2020-05-19 12:56:11 -07:00
Nick Terrell b82bf711fc [fuzz] Expand the allowedExpansion 2020-05-19 11:42:53 -07:00
Nick TerrellandGitHub 9778f46014 Merge pull request #2150 from terrelln/ldm-dict-reset 
[ldm] Reset loadedDictEnd when the context is reset
 2020-05-18 18:33:01 -07:00
Nick Terrell 7b317b4876 [test] Test that the ldm dictionary gets invalidated on reset 2020-05-18 16:00:28 -07:00
Nick Terrell 87dbd6d4bf [test] Improve LDM forceMaxWindow test 2020-05-18 15:11:18 -07:00
Nick Terrell b2092c6dc4 [ldm] Reset loadedDictEnd when the context is reset 2020-05-18 12:35:44 -07:00
Nick Terrell 608075abb2 [test][regression] Update results.csv 2020-05-14 17:06:39 -07:00
Nick TerrellandGitHub 9ff4991d73 Merge pull request #2138 from terrelln/ldm-fix 
[lib] Fix bug in loading LDM dictionary in MT mode
 2020-05-14 13:32:00 -07:00
Nick Terrell bf0591e1e2 [test] Expose the LDM+MT+dict bug in a unit test 2020-05-14 12:06:55 -07:00
Nick Terrell add7ed2d4a [lib] Fix bug in loading LDM dictionary in MT mode 
Exposed when loading a dictionary < LDM minMatch bytes in MT mode.

Test Plan:
```
CC=clang make -j zstreamtest MOREFLAGS="-O0 -fsanitize=address"
./zstreamtest -vv -i100000000 -t1 --newapi -s7065 -t3925297
```

TODO: Add an explicit test that loads a small dictionary in MT mode
 2020-05-14 11:52:28 -07:00
Nick Terrell 70c80e19e6 [greedy] Fix performance instability 2020-05-12 17:51:16 -07:00
Nick TerrellandGitHub c3e921c639 Merge pull request #2131 from terrelln/raw-dict-fuzzer 
Fix rare scenario with lazy parser, dictionary, and repcodes
 2020-05-12 17:44:31 -07:00
Nick Terrell f800e72a3c [lib] Fix assertion when dictionary is prefix 2020-05-12 14:33:59 -07:00
Nick Terrell 3c1eba4d99 [lib] Fix lazy repcode validity checks 2020-05-12 12:25:06 -07:00
Nick Terrell 4e0515916d [lib] Fix repcode validation in no dict mode 2020-05-12 11:57:15 -07:00
Nick Terrell 6d687a8816 [lib] Fix dictionary + repcodes + optimal parser 2020-05-12 10:36:53 -07:00
Nick Terrell 4b88bd3ee0 [lib][fuzz] Assert sequences are valid in round trip tests 2020-05-11 20:38:49 -07:00
Nick Terrell 80d3585e31 [lib] Fix lazy parser with dictionary + repcodes 2020-05-11 19:04:30 -07:00
Nick Terrell 1185dfb8d1 [fuzz] Add raw dictionary content fuzzer 2020-05-11 19:03:33 -07:00
Nick Terrell 301a62fe08 [fuzz] Fix compress bound for dictionary_round_trip 2020-05-11 19:00:52 -07:00
Nick TerrellandGitHub 738f49178b Merge pull request #2128 from terrelln/benches 
Update benchmarks and add fast mode
 2020-05-11 13:53:18 -07:00
Nick Terrell f2f86b50ae Update benchmarks and add fast mode 2020-05-11 12:36:36 -07:00
Nick TerrellandGitHub 32b5a8ec41 Merge pull request #2118 from Artoria2e5/patch-1 
programs/README.md: update help text
 2020-05-08 11:13:44 -07:00
Nick TerrellandNick Terrell 45c66dd298 [zdict] Stabilize ZDICT_finalizeDictionary() 2020-05-07 10:37:01 -07:00
Nick TerrellandNick Terrell 5717bd39ee [lib] Fix NULL pointer dereference 
When the output buffer is `NULL` with size 0, but the frame content size
is non-zero, we will write to the NULL pointer because our bounds check
underflowed.

This was exposed by a recent PR that allowed an empty frame into the
single-pass shortcut in streaming mode.

* Fix the bug.
* Fix another NULL dereference in zstd-v1.
* Overflow checks in 32-bit mode.
* Add a dedicated test.
* Expose the bug in the dedicated simple_decompress fuzzer.
* Switch all mallocs in fuzzers to return NULL for size=0.
* Fix a new timeout in a fuzzer.

Neither clang nor gcc show a decompression speed regression on x86-64.
On x86-32 clang is slightly positive and gcc loses 2.5% of speed.

Credit to OSS-Fuzz.
 2020-05-06 12:09:02 -07:00
Nick TerrellandGitHub e103d7b4a6 Fix superblock mode (#2100) 
Fixes:

Enable RLE blocks for superblock mode
Fix the limitation that the literals block must shrink. Instead, when we're within 200 bytes of the next header byte size, we will just use the next one up. That way we should (almost?) always have space for the table.
Remove the limitation that the first sub-block MUST have compressed literals and be compressed. Now one sub-block MUST be compressed (otherwise we fall back to raw block which is okay, since that is streamable). If no block has compressed literals that is okay, we will fix up the next Huffman table.
Handle the case where the last sub-block is uncompressed (maybe it is very small). Before it would skip superblock in this case, now we allow the last sub-block to be uncompressed. To do this we need to regenerate the correct repcodes.
Respect disableLiteralsCompression in superblock mode
Fix superblock mode to handle a block consisting of only compressed literals
Fix a off by 1 error in superblock mode that disabled it whenever there were last literals
Fix superblock mode with long literals/matches (> 0xFFFF)
Allow superblock mode to repeat Huffman tables
Respect ZSTD_minGain().
Tests:

Simple check for the condition in #2096.
When the simple_round_trip fuzzer enables superblock mode, it checks that the compressed size isn't expanded too much.
Remaining limitations:

O(targetCBlockSize^2) because we recompute statistics every sequence
Unable to split literals of length > targetCBlockSize into multiple sequences
Refuses to generate sub-blocks that don't shrink the compressed data, so we could end up with large sub-blocks. We should emit those sections as uncompressed blocks instead.
...
Fixes #2096
 2020-05-01 16:11:47 -07:00
Nick TerrellandGitHub 0ed07f6dfe Merge pull request #2094 from terrelln/stable-dst 
[lib] Add ZSTD_d_stableOutBuffer + fix single-pass mode for empty frames
 2020-04-28 17:53:24 -07:00
Nick Terrell 55a57d46be Add extra warnings about not modifying the ZSTD_outBuffer 2020-04-28 12:07:42 -07:00
Nick TerrellandGitHub 108a5572a5 Merge pull request #2048 from nocnokneo/ctest-support 
Add CTest support
 2020-04-28 11:01:13 -07:00
Nick Terrell 1343b815f8 [fuzz] Fuzz test ZSTD_d_stableOutBuffer 2020-04-27 20:04:04 -07:00
Nick Terrell 77a2945c43 Add some comments 2020-04-27 20:04:04 -07:00
Nick Terrell f33de06c3e [lib] Fix single-pass mode for empty frames 2020-04-27 20:04:01 -07:00
Nick Terrell a4ff217baf [lib] Add ZSTD_d_stableOutBuffer 2020-04-27 18:09:44 -07:00
Nick Terrell b104f8e3eb [zstd] Fix typo in ZSTD_dParameter 2020-04-27 12:12:28 -07:00
Nick TerrellandGitHub 5fcbc484c8 Merge pull request #2040 from caoyzh/dev-2 
Optimize by prefetching on aarch64
 2020-04-08 13:14:47 -07:00
Nick TerrellandGitHub 1665462573 Merge pull request #2054 from terrelln/license-fix 
Standardize and fix copyright and licenses
 2020-03-27 11:00:01 -07:00
Nick TerrellandGitHub 8fda220d70 Merge pull request #2053 from terrelln/fix-play-test 
[test] Fix playTests.sh with space in binary path
 2020-03-26 21:08:10 -07:00
Nick Terrell c7ae28a67d [circleci] Run test-license.py 
Run it first because it is very fast (~1 second)
 2020-03-26 20:13:16 -07:00