ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,053 Commits 17 Branches 77 Tags
d69d08ed6c83563b57d98132e1e3f2487880781e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell d69d08ed6c Merge pull request #2326 from terrelln/kernel-test-fix 
Fix issues and warnings exposed by Kernel Test Robot
 2020-09-29 13:51:23 -07:00
Nick Terrell f1cbeec039 [superblock] Reduce stack usage by correctly sizing header buffers 2020-09-24 19:42:04 -07:00
Nick Terrell 6a1e526ea7 [lib] Add ZSTD_COMPRESS_HEAPMODE tuning parameter 2020-09-24 19:42:04 -07:00
Nick Terrell b841387218 [freestanding] Improve macro resolution to handle #if X 2020-09-24 19:42:04 -07:00
Nick Terrell caecd8c211 Allow user to override ASAN/MSAN detection 
Rename ADDRESS_SANITIZER -> ZSTD_ADDRESS_SANITIZER and same for
MEMORY_SANITIZER. Also set it to 0/1 instead of checking for defined.
This allows the user to override ASAN/MSAN detection for platforms that
don't support it.
 2020-09-24 19:42:04 -07:00
Nick Terrell 88fac5d514 Remove call to memset 
The previous commit fixes the test so it errors on calls to mem*()
functions from <string.h>.
 2020-09-24 19:42:04 -07:00
Nick Terrell 683150e59f [linux-kernel] Avoid including <string.h> in the tests 2020-09-24 19:42:04 -07:00
Nick Terrell 9ae0483858 Reorganize zstd_deps.h and mem.h + replace mem.h for the kernel 2020-09-24 19:41:59 -07:00
Nick Terrell 260fc75028 Move __has_builtin() fallback define to compiler.h 2020-09-24 15:51:08 -07:00
Nick Terrell 4d63ee57f5 Move ASAN/MSAN support declarations to compiler.h 2020-09-24 15:51:08 -07:00
Nick Terrell b09ec5c2b9 Remove MEM_STATIC_ASSERT and use DEBUG_STATIC_ASSERT instead 2020-09-24 15:51:04 -07:00
Nick Terrell 9009b59f09 [linux-kernel] Add missing semicolon in zstd_deps.h 2020-09-23 23:26:10 -07:00
Nick Terrell 9261476b7d [lib] Wrap customMem xor checks in parens for readability 
This clarifies operator precedence, and quiets cppcheck in
the Kernel Test Robot. I think this is a slight bonus to
readability, so I am accepting the suggestion.
 2020-09-23 23:26:07 -07:00
Nick Terrell 2fc1d50ee7 Merge pull request #2324 from terrelln/kernel-test-fix 
Silence warnings reported by Kernel Test Robot
 2020-09-23 15:01:56 -07:00
Nick Terrell a70fa342f0 [CI][linux-kernel] Enable -Wunused-const-variable and -Wunused-but-set-variable 2020-09-23 12:59:57 -07:00
Nick Terrell dec7fb03ec [lib] Silence -Wunused-const-variable warnings 2020-09-23 12:59:57 -07:00
Nick Terrell 978659d34f [linux-kernel] Fix unused variable warnings with malloc, calloc, and free 2020-09-23 12:59:57 -07:00
Nick Terrell 8170e3ca6d Merge pull request #2320 from terrelln/test-fix 
[tests] Don't write to stdout
 2020-09-22 10:58:48 -07:00
Nick Terrell 973f2adeec [tests] Don't write to stdout 2020-09-22 00:40:27 -07:00
Nick Terrell d96e98cfde Merge pull request #2308 from terrelln/zstd-kernel 
[contrib][linux-kernel] Add decompress_sources.h
 2020-09-14 14:56:30 -07:00
Nick Terrell 5651fe9b49 [contrib][linux-kernel] Add decompress_sources.h 
Add decompress_sources.h, which includes all the decompression .c files.
This is used for kernel decompression.

Also, add a test which checks that including decompress_sources.h works.
 2020-09-14 12:49:23 -07:00
Nick Terrell 868804d322 Merge pull request #2301 from terrelln/prepare-for-release 
Prepare library for a release
 2020-09-10 15:42:16 -07:00
Nick Terrell da30a78c68 [lib] Bump version number to 1.4.6 2020-09-09 17:13:45 -07:00
Nick Terrell edd8327097 Update CHANGELOG for the next release 2020-09-09 17:13:34 -07:00
Nick Terrell b92569a522 [doc] Document new build macros in lib/README.md 2020-09-09 17:13:16 -07:00
Nick Terrell 5118e30cfe Merge pull request #2289 from terrelln/zstd-kernel-2 
[linux-kernel] Prepare zstd for the Linux Kernel
 2020-09-09 15:38:02 -07:00
Nick Terrell 3420c202af [linux-kernel] Update README with initial instructions on how to update zstd in the kernel 2020-09-09 14:36:22 -07:00
Nick Terrell aab4bf7b0d [linux-kernel] Add test that checks the ifdef hardwiring 2020-09-09 14:36:19 -07:00
Nick Terrell 29c5de8780 Add linux-kernel freestanding 2020-09-09 14:35:57 -07:00
Nick Terrell 1c3cb2c05c [contrib] Add preprocessor hardwiring to freestanding.py 2020-09-09 14:35:39 -07:00
Nick Terrell 79ded1b4a9 [lib] Add ZSTD_NO_UNUSED_FUNCTIONS macro to hide unused functions 
The unused function definitions are hidden behind a
`#ifndef ZSTD_NO_UNUSED_FUNCTIONS` check.

Initially hiding all functions which are unused and take up more than
2KB of stack space, because these will show up as warnings in the
Linux Kernel build system.
 2020-09-09 14:35:39 -07:00
Nick Terrell ac3a136b0a [lib] Replace 64-bit divisions with ZSTD_div64() 2020-09-09 14:35:39 -07:00
Nick Terrell a90779397a [lib] Reduce zstd stack usage by 1KB 2020-09-09 14:35:39 -07:00
Nick Terrell 046aca190f Fix ZSTD_initCStream_advanced() with no dictionary and static allocation 2020-09-09 14:35:39 -07:00
Nick Terrell e975de289c Add ZSTD_NO_INTRINSICS macro to avoid explicit intrinsics 2020-09-09 14:35:39 -07:00
Nick Terrell f91ed5c766 [lib] s/current/curr because it collides with Linux Kernel macro 2020-09-09 14:35:39 -07:00
Nick Terrell 5e4efd22d4 Merge pull request #2291 from i-do-cpp/fix-compression-level-default 
Fix setParameter not falling back to default compression level
 2020-09-08 16:42:34 -07:00
Nick Terrell 6da8acd231 Merge pull request #2293 from allanjude/coverity 
Resolve Coverity 1432392 Unintentional integer overflow
 2020-09-03 13:58:45 -07:00
Nick Terrell bcedab0731 Merge pull request #2279 from senhuang42/show_warnings_with_multiple_input_files 
Add a warning whenever (de)compressing multiple files
 2020-09-02 11:32:11 -07:00
Nick Terrell 4c408866f0 Merge pull request #2264 from terrelln/zstd-kernel 
Prepare for freestanding library and add initial translator script
 2020-08-26 18:15:25 -07:00
Nick Terrell ae455dde08 [contrib] Add freestanding translator prototype 
This is the idea, some of the functionality isn't yet implemented.
 2020-08-26 12:26:05 -07:00
Nick Terrell c465f24457 ZSTD_ prefix mem{cpy,move,set},malloc,calloc,free 2020-08-26 12:26:03 -07:00
Nick Terrell a686d306d2 Rename ZSTD_{malloc,calloc,free} to ZSTD_custom{Malloc,Calloc,Free} 2020-08-26 12:25:08 -07:00
Nick Terrell 80f577baa2 Move standard includes to zstd_deps.h 2020-08-26 12:25:08 -07:00
Nick Terrell cf83aceaf3 Merge pull request #2282 from terrelln/ncount-fix 
[bug] Fix FSE_readNCount()
 2020-08-26 10:31:07 -07:00
Nick Terrell 3bcdcaa042 Merge pull request #2285 from terrelln/oss-fuzz-fixes 
[fuzz] Fix oss-fuzz test problems
 2020-08-25 18:35:16 -07:00
Nick Terrell ae163015b1 [fuzz] Fix stream_decompress timeouts 2020-08-25 17:13:09 -07:00
Nick Terrell 49eeb2d1fc [fuzz] Disable superblock expansion test 2020-08-25 17:13:06 -07:00
Nick Terrell 51dd7e69f9 [github actions] Disable armbuild and armfuzz tests 2020-08-25 16:28:41 -07:00
Nick Terrell 4193638996 [bug] Fix FSE_readNCount() 
* Fix bug introduced in PR #2271
* Fix long-standing bug that is impossible to trigger inside of zstd
* Add a fuzzer that makes sure the normalized count always round trips
  correctly
 2020-08-25 15:42:41 -07:00
Nick Terrell 614e446000 Merge pull request #2271 from terrelln/small-blocks 
Small block optimizations
 2020-08-24 18:54:33 -07:00
Nick Terrell 52f33a1da5 Fix compiler warnings 2020-08-24 16:09:45 -07:00
Nick Terrell 6f301a7903 Merge pull request #2272 from terrelln/dstSize_tooSmall 
[fix] Always return dstSize_tooSmall when it is the case
 2020-08-24 15:01:17 -07:00
Nick Terrell 6d2f750b37 Document the BMI2 default() functions 2020-08-24 14:44:33 -07:00
Nick Terrell cebe0b5c0b Improve FSE_normalizeCount() docs 2020-08-24 13:58:34 -07:00
Nick Terrell 1302f8d676 [fix] Always return dstSize_tooSmall when it is the case 2020-08-24 13:38:13 -07:00
Nick Terrell 8def0e5fd3 Fix up code after reading through 2020-08-24 12:24:45 -07:00
Nick Terrell 8f8bd2d1ac [regression] Update results.csv 2020-08-20 12:41:35 -07:00
Nick Terrell 575731b6db Use ncount=1 when < 4096 symbols 2020-08-18 16:47:53 -07:00
Nick Terrell 612e947c5e wire up bmi2 support 2020-08-17 16:35:28 -07:00
Nick Terrell ba1fd17a9f speed up literal header decoding 2020-08-17 12:17:53 -07:00
Nick Terrell 6004c1117f speed up small blocks 2020-08-16 23:03:38 -07:00
Nick Terrell a8006264cf small blocks benchmark 2020-08-14 18:57:20 -07:00
Nick Terrell 49069a3f0a Merge pull request #2265 from terrelln/prefetch-fix 
Prefer __builtin_prefetch over inline asm
 2020-08-11 10:07:53 -07:00
Nick Terrell e3bda594ae Prefer __builtin_prefetch over inline asm 
Reorder the ifdefs for the PREFETCH macros so that the compiler builtin is
favored over the inline assembly for aarch64.
 2020-08-10 22:17:18 -07:00
Nick Terrell f85a0f8bcf Merge pull request #2256 from helloguo/dev 
Optimize ZSTD_wildcopy
 2020-07-29 11:57:49 -07:00
Nick Terrell 01261bc8b6 Merge pull request #2252 from helloguo/dev 
Fix preprocessor in ZSTD_wildcopy
 2020-07-27 12:59:10 -07:00
Nick Terrell bd21e4b264 Merge pull request #2242 from evverx/cifuzz-ubsan-msan 
cifuzz: turn on UBsan and MSan
 2020-07-13 14:26:13 -04:00
Nick Terrell bdf716bc64 Merge pull request #2239 from evverx/evverx-cifuzz 
cifuzz: don't upload artifacts when CIFuzz fails to build fuzz targets
 2020-07-10 11:26:15 -04:00
Nick Terrell a2b6a8fbc6 Merge pull request #2220 from dkcasset/fix-lib-Makefile-older-sed-version 
Replace -E option with equivalent -r for older versions of sed
 2020-06-29 17:19:24 -07:00
Nick Terrell 3690fe30f3 Merge pull request #2221 from BioDataAnalysis/bda_install_dll 
Install MSVC runtime shared libraries (DLL) with cmake
 2020-06-29 17:14:50 -07:00
Nick Terrell 7afd5d85d3 Merge pull request #2218 from terrelln/assert-seq 
Fix unused variable warnings in fuzzing build mode without asserts
 2020-06-22 17:41:18 -07:00
Nick Terrell 081691a3aa Merge pull request #2217 from terrelln/cover-redundant 
[cover] Remove unnecessary mask and dedup hash functions
 2020-06-22 17:41:13 -07:00
Nick Terrell 370933fa20 Merge pull request #2209 from Niadb/dev 
Explicitly use __cdecl for qsort, to avoid warning when default calling convention is not __cdecl
 2020-06-22 17:41:03 -07:00
Nick Terrell cce0edfdbe Fix unused variable warnings in fuzzing build mode without asserts 
Fix unused vairable warnings when `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is defined but asserts are disabled.

Fixes #2210.
 2020-06-22 12:56:57 -07:00
Nick Terrell 2312b819af [cover] Remove unnecessary mask and dedup hash functions 
* Remove the unnecessary mask, since `ZSTD_hash*()` already ensures
  the output is mod 2^h.
* Dedup the hash functions and use `ZSTD_hash*()` directly.
 2020-06-22 12:52:13 -07:00
Nick Terrell 78601d0806 Merge pull request #2212 from cwoffenden/single-file-dict 
Single-file libs now include dictBuilder
 2020-06-22 12:46:54 -07:00
Nick Terrell 34c0d85916 Merge pull request #2214 from terrelln/superblock-cross-entropy 
[superblock] Add defensive assert and bounds check
 2020-06-22 12:27:58 -07:00
Nick Terrell 1047097dad [superblock] Add defensive assert and bounds check 
The bound check condition should always be met because we selected `set_basic` as
our encoding type. But that code is very far away, so assert it is true so if it is
ever false we can catch it, and add a bounds check.

Fixes #2213.
 2020-06-22 10:21:38 -07:00
Nick Terrell b6a9ded994 Merge pull request #2202 from terrelln/dict-fi 
[lib] Allow compression dictionaries with missing symbols
 2020-06-15 10:54:33 -07:00
Nick Terrell 08981d2638 [lib] Allow compression dictionaries with missing symbols 
Allow compression to use dictionaries with missing symbols in their
entropy tables. We set the FSE repeat mode to check when there are
missing symbols, and set the FSE repeat mode to valid when all symbols
are present.

Note that when not all symbols are present, the heuristics which favor
dictionary tables for lower compression levels won't activate.

Tested by manually creating a dictionary with missing symbols of every
type, and validing that the compressor rejects it before this change,
and accepts it after this change. Also, I ran the `dictionary_loader`
fuzzer for >1 hour of CPU time without running into cases where
compression succeeds, but decompression fails.

Fixes #2174.
 2020-06-12 17:57:19 -07:00
Nick Terrell 9eb2ccc9fb Merge pull request #2167 from terrelln/pzstd-ms-fix 
[pzstd] Fix compilation error in MinGW
 2020-05-22 23:38:27 -07:00
Nick Terrell 5aa5aa4df7 [pzstd] Fix compilation error in MinGW 2020-05-22 22:26:02 -07:00
Nick Terrell 651d3d73e0 [test] Update the ldm loadedDictEnd test to cover zstdmt 2020-05-19 16:14:14 -07:00
Nick Terrell 3cc227e90e [ldm][mt] Fix loadedDictEnd 2020-05-19 15:55:03 -07:00
Nick Terrell 0dcd3eec43 Merge pull request #2152 from terrelln/simple-rt-bound 
[fuzz] Expand the allowedExpansion
 2020-05-19 12:56:11 -07:00
Nick Terrell b82bf711fc [fuzz] Expand the allowedExpansion 2020-05-19 11:42:53 -07:00
Nick Terrell 9778f46014 Merge pull request #2150 from terrelln/ldm-dict-reset 
[ldm] Reset loadedDictEnd when the context is reset
 2020-05-18 18:33:01 -07:00
Nick Terrell 7b317b4876 [test] Test that the ldm dictionary gets invalidated on reset 2020-05-18 16:00:28 -07:00
Nick Terrell 87dbd6d4bf [test] Improve LDM forceMaxWindow test 2020-05-18 15:11:18 -07:00
Nick Terrell b2092c6dc4 [ldm] Reset loadedDictEnd when the context is reset 2020-05-18 12:35:44 -07:00
Nick Terrell 608075abb2 [test][regression] Update results.csv 2020-05-14 17:06:39 -07:00
Nick Terrell 9ff4991d73 Merge pull request #2138 from terrelln/ldm-fix 
[lib] Fix bug in loading LDM dictionary in MT mode
 2020-05-14 13:32:00 -07:00
Nick Terrell bf0591e1e2 [test] Expose the LDM+MT+dict bug in a unit test 2020-05-14 12:06:55 -07:00
Nick Terrell add7ed2d4a [lib] Fix bug in loading LDM dictionary in MT mode 
Exposed when loading a dictionary < LDM minMatch bytes in MT mode.

Test Plan:
```
CC=clang make -j zstreamtest MOREFLAGS="-O0 -fsanitize=address"
./zstreamtest -vv -i100000000 -t1 --newapi -s7065 -t3925297
```

TODO: Add an explicit test that loads a small dictionary in MT mode
 2020-05-14 11:52:28 -07:00
Nick Terrell 70c80e19e6 [greedy] Fix performance instability 2020-05-12 17:51:16 -07:00
Nick Terrell c3e921c639 Merge pull request #2131 from terrelln/raw-dict-fuzzer 
Fix rare scenario with lazy parser, dictionary, and repcodes
 2020-05-12 17:44:31 -07:00
Nick Terrell f800e72a3c [lib] Fix assertion when dictionary is prefix 2020-05-12 14:33:59 -07:00
Nick Terrell 3c1eba4d99 [lib] Fix lazy repcode validity checks 2020-05-12 12:25:06 -07:00
Nick Terrell 4e0515916d [lib] Fix repcode validation in no dict mode 2020-05-12 11:57:15 -07:00