ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,160 Commits 17 Branches 77 Tags
de985b1c1ac2625597abdaa214dc7841f9f513f2
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick TerrellandGitHub 8c46c1d851 Merge pull request #2356 from bsdimp/neon 
aarch64: use __ARM_NEON instead of __aarch64__ to control use of neon
 2020-10-13 15:42:46 -07:00
Nick TerrellandGitHub 1e47a75abf Merge pull request #2337 from terrelln/adjust-params 
Fix ZSTD_adjustCParams_internal() to handle dictionary logic
 2020-10-12 18:42:24 -07:00
Nick Terrell 7e6f91ed84 [minor] Improve docs and add an assert in response to review 2020-10-12 16:43:17 -07:00
Nick Terrell d5c688e8ae Fix ZSTD_adjustCParams_internal() to handle dictionary logic 
Pass in the `ZSTD_cParamMode_e` to select how we define our cparams.
Based on the mode we either take the `dictSize` into account or we set
it to `0`. See the documentation for `ZSTD_cParamMode_e`.

Some of the modes currently share the same behavior. But they have
distinct modes because they are drastically different cases. E.g.
compression + reprocessing the dictionary and creating a cdict.

Additionally, when downsizing the hashLog and chainLog take the
(adjusted) dictionary size into account, since the size of the
dictionary gets added onto the window size.

Adds a simple test to ensure that we aren't downsizing too far.
 2020-10-12 12:50:04 -07:00
Nick Terrell fadaab8c7c [minor improvement] Pass 0 as the content size in the DDS 
The DDS structure can't be copied into the working tables like the DMS.
So it doesn't need to account for the source size when sizing its
parameters, just the dictionary size.
 2020-10-12 12:47:21 -07:00
Nick Terrell 48ef15fb47 [minor improvement] Pass dictSize when selecting parameters 
When selecting parameters in streaming compression with a dictionary use
the dictionary size to select the parameters.
 2020-10-12 12:47:19 -07:00
Nick Terrell 012818df99 [refactor] Remove ZSTD_resetCStream_internal() 
This function is only called in one place. It isn't a logical separation
of duties, and it was only obsfucating the code now, so inline it.
 2020-10-12 12:46:10 -07:00
Nick Terrell 7083f79008 [bug] Fix dictContentType when reprocessing cdict 
Conditions to trigger:
* CDict is loaded as raw content.
* CDict starts with the zstd dictionary magic number.
* The CDict is reprocessed (not attached or copied).
* The new API is used (streaming or `ZSTD_compress2()`).

Bug: The dictionary is loaded as a zstd dictionary, not a raw content
dictionary, because the dict content type is set to `ZSTD_dct_auto`.

Fix: Pass in the dictionary content type from cdict creation to the call
to `ZSTD_compress_insertDictionary()`.

Test: Added a test case that exposes the bug, and fixed the raw
content tests to not modify the `dictBuffer`, which makes all future
tests with the `dictBuffer` raw content, which doesn't seem intentional.
 2020-10-12 12:46:10 -07:00
Nick TerrellandGitHub 4b4d8b4dc9 Merge pull request #2338 from terrelln/comments 
Add comments to ZSTD_getLowest{Match,Prefix}Index()
 2020-10-01 18:56:24 -07:00
Nick TerrellandGitHub 0057c4acf7 Merge pull request #2333 from terrelln/stable-dst 
Reset all decompression parameters in ZSTD_DCtx_reset()
 2020-10-01 18:56:11 -07:00
Nick Terrell 2e7d174130 Reset all decompression parameters in ZSTD_DCtx_reset() 
* Reset all decompression parameters in `ZSTD_DCtx_reset()` when
  resetting parameters.
* Add a test case.
 2020-10-01 14:19:21 -07:00
Nick Terrell 27c969ed07 Add comments to ZSTD_getLowest{Match,Prefix}Index() 
Clarify how we handle dictionaries in each case.
 2020-10-01 13:21:46 -07:00
Nick TerrellandGitHub d69d08ed6c Merge pull request #2326 from terrelln/kernel-test-fix 
Fix issues and warnings exposed by Kernel Test Robot
 2020-09-29 13:51:23 -07:00
Nick Terrell f1cbeec039 [superblock] Reduce stack usage by correctly sizing header buffers 2020-09-24 19:42:04 -07:00
Nick Terrell 6a1e526ea7 [lib] Add ZSTD_COMPRESS_HEAPMODE tuning parameter 2020-09-24 19:42:04 -07:00
Nick Terrell b841387218 [freestanding] Improve macro resolution to handle #if X 2020-09-24 19:42:04 -07:00
Nick Terrell caecd8c211 Allow user to override ASAN/MSAN detection 
Rename ADDRESS_SANITIZER -> ZSTD_ADDRESS_SANITIZER and same for
MEMORY_SANITIZER. Also set it to 0/1 instead of checking for defined.
This allows the user to override ASAN/MSAN detection for platforms that
don't support it.
 2020-09-24 19:42:04 -07:00
Nick Terrell 88fac5d514 Remove call to memset 
The previous commit fixes the test so it errors on calls to mem*()
functions from <string.h>.
 2020-09-24 19:42:04 -07:00
Nick Terrell 683150e59f [linux-kernel] Avoid including <string.h> in the tests 2020-09-24 19:42:04 -07:00
Nick Terrell 9ae0483858 Reorganize zstd_deps.h and mem.h + replace mem.h for the kernel 2020-09-24 19:41:59 -07:00
Nick Terrell 260fc75028 Move __has_builtin() fallback define to compiler.h 2020-09-24 15:51:08 -07:00
Nick Terrell 4d63ee57f5 Move ASAN/MSAN support declarations to compiler.h 2020-09-24 15:51:08 -07:00
Nick Terrell b09ec5c2b9 Remove MEM_STATIC_ASSERT and use DEBUG_STATIC_ASSERT instead 2020-09-24 15:51:04 -07:00
Nick Terrell 9009b59f09 [linux-kernel] Add missing semicolon in zstd_deps.h 2020-09-23 23:26:10 -07:00
Nick Terrell 9261476b7d [lib] Wrap customMem xor checks in parens for readability 
This clarifies operator precedence, and quiets cppcheck in
the Kernel Test Robot. I think this is a slight bonus to
readability, so I am accepting the suggestion.
 2020-09-23 23:26:07 -07:00
Nick TerrellandGitHub 2fc1d50ee7 Merge pull request #2324 from terrelln/kernel-test-fix 
Silence warnings reported by Kernel Test Robot
 2020-09-23 15:01:56 -07:00
Nick Terrell a70fa342f0 [CI][linux-kernel] Enable -Wunused-const-variable and -Wunused-but-set-variable 2020-09-23 12:59:57 -07:00
Nick Terrell dec7fb03ec [lib] Silence -Wunused-const-variable warnings 2020-09-23 12:59:57 -07:00
Nick Terrell 978659d34f [linux-kernel] Fix unused variable warnings with malloc, calloc, and free 2020-09-23 12:59:57 -07:00
Nick TerrellandGitHub 8170e3ca6d Merge pull request #2320 from terrelln/test-fix 
[tests] Don't write to stdout
 2020-09-22 10:58:48 -07:00
Nick Terrell 973f2adeec [tests] Don't write to stdout 2020-09-22 00:40:27 -07:00
Nick TerrellandGitHub d96e98cfde Merge pull request #2308 from terrelln/zstd-kernel 
[contrib][linux-kernel] Add decompress_sources.h
 2020-09-14 14:56:30 -07:00
Nick Terrell 5651fe9b49 [contrib][linux-kernel] Add decompress_sources.h 
Add decompress_sources.h, which includes all the decompression .c files.
This is used for kernel decompression.

Also, add a test which checks that including decompress_sources.h works.
 2020-09-14 12:49:23 -07:00
Nick TerrellandGitHub 868804d322 Merge pull request #2301 from terrelln/prepare-for-release 
Prepare library for a release
 2020-09-10 15:42:16 -07:00
Nick Terrell da30a78c68 [lib] Bump version number to 1.4.6 2020-09-09 17:13:45 -07:00
Nick Terrell edd8327097 Update CHANGELOG for the next release 2020-09-09 17:13:34 -07:00
Nick Terrell b92569a522 [doc] Document new build macros in lib/README.md 2020-09-09 17:13:16 -07:00
Nick TerrellandGitHub 5118e30cfe Merge pull request #2289 from terrelln/zstd-kernel-2 
[linux-kernel] Prepare zstd for the Linux Kernel
 2020-09-09 15:38:02 -07:00
Nick Terrell 3420c202af [linux-kernel] Update README with initial instructions on how to update zstd in the kernel 2020-09-09 14:36:22 -07:00
Nick Terrell aab4bf7b0d [linux-kernel] Add test that checks the ifdef hardwiring 2020-09-09 14:36:19 -07:00
Nick Terrell 29c5de8780 Add linux-kernel freestanding 2020-09-09 14:35:57 -07:00
Nick Terrell 1c3cb2c05c [contrib] Add preprocessor hardwiring to freestanding.py 2020-09-09 14:35:39 -07:00
Nick Terrell 79ded1b4a9 [lib] Add ZSTD_NO_UNUSED_FUNCTIONS macro to hide unused functions 
The unused function definitions are hidden behind a
`#ifndef ZSTD_NO_UNUSED_FUNCTIONS` check.

Initially hiding all functions which are unused and take up more than
2KB of stack space, because these will show up as warnings in the
Linux Kernel build system.
 2020-09-09 14:35:39 -07:00
Nick Terrell ac3a136b0a [lib] Replace 64-bit divisions with ZSTD_div64() 2020-09-09 14:35:39 -07:00
Nick Terrell a90779397a [lib] Reduce zstd stack usage by 1KB 2020-09-09 14:35:39 -07:00
Nick Terrell 046aca190f Fix ZSTD_initCStream_advanced() with no dictionary and static allocation 2020-09-09 14:35:39 -07:00
Nick Terrell e975de289c Add ZSTD_NO_INTRINSICS macro to avoid explicit intrinsics 2020-09-09 14:35:39 -07:00
Nick Terrell f91ed5c766 [lib] s/current/curr because it collides with Linux Kernel macro 2020-09-09 14:35:39 -07:00
Nick TerrellandGitHub 5e4efd22d4 Merge pull request #2291 from i-do-cpp/fix-compression-level-default 
Fix setParameter not falling back to default compression level
 2020-09-08 16:42:34 -07:00
Nick TerrellandGitHub 6da8acd231 Merge pull request #2293 from allanjude/coverity 
Resolve Coverity 1432392 Unintentional integer overflow
 2020-09-03 13:58:45 -07:00
Nick TerrellandGitHub bcedab0731 Merge pull request #2279 from senhuang42/show_warnings_with_multiple_input_files 
Add a warning whenever (de)compressing multiple files
 2020-09-02 11:32:11 -07:00
Nick TerrellandGitHub 4c408866f0 Merge pull request #2264 from terrelln/zstd-kernel 
Prepare for freestanding library and add initial translator script
 2020-08-26 18:15:25 -07:00
Nick Terrell ae455dde08 [contrib] Add freestanding translator prototype 
This is the idea, some of the functionality isn't yet implemented.
 2020-08-26 12:26:05 -07:00
Nick Terrell c465f24457 ZSTD_ prefix mem{cpy,move,set},malloc,calloc,free 2020-08-26 12:26:03 -07:00
Nick Terrell a686d306d2 Rename ZSTD_{malloc,calloc,free} to ZSTD_custom{Malloc,Calloc,Free} 2020-08-26 12:25:08 -07:00
Nick Terrell 80f577baa2 Move standard includes to zstd_deps.h 2020-08-26 12:25:08 -07:00
Nick TerrellandGitHub cf83aceaf3 Merge pull request #2282 from terrelln/ncount-fix 
[bug] Fix FSE_readNCount()
 2020-08-26 10:31:07 -07:00
Nick TerrellandGitHub 3bcdcaa042 Merge pull request #2285 from terrelln/oss-fuzz-fixes 
[fuzz] Fix oss-fuzz test problems
 2020-08-25 18:35:16 -07:00
Nick Terrell ae163015b1 [fuzz] Fix stream_decompress timeouts 2020-08-25 17:13:09 -07:00
Nick Terrell 49eeb2d1fc [fuzz] Disable superblock expansion test 2020-08-25 17:13:06 -07:00
Nick Terrell 51dd7e69f9 [github actions] Disable armbuild and armfuzz tests 2020-08-25 16:28:41 -07:00
Nick Terrell 4193638996 [bug] Fix FSE_readNCount() 
* Fix bug introduced in PR #2271
* Fix long-standing bug that is impossible to trigger inside of zstd
* Add a fuzzer that makes sure the normalized count always round trips
  correctly
 2020-08-25 15:42:41 -07:00
Nick TerrellandGitHub 614e446000 Merge pull request #2271 from terrelln/small-blocks 
Small block optimizations
 2020-08-24 18:54:33 -07:00
Nick Terrell 52f33a1da5 Fix compiler warnings 2020-08-24 16:09:45 -07:00
Nick TerrellandGitHub 6f301a7903 Merge pull request #2272 from terrelln/dstSize_tooSmall 
[fix] Always return dstSize_tooSmall when it is the case
 2020-08-24 15:01:17 -07:00
Nick Terrell 6d2f750b37 Document the BMI2 default() functions 2020-08-24 14:44:33 -07:00
Nick Terrell cebe0b5c0b Improve FSE_normalizeCount() docs 2020-08-24 13:58:34 -07:00
Nick Terrell 1302f8d676 [fix] Always return dstSize_tooSmall when it is the case 2020-08-24 13:38:13 -07:00
Nick Terrell 8def0e5fd3 Fix up code after reading through 2020-08-24 12:24:45 -07:00
Nick Terrell 8f8bd2d1ac [regression] Update results.csv 2020-08-20 12:41:35 -07:00
Nick Terrell 575731b6db Use ncount=1 when < 4096 symbols 2020-08-18 16:47:53 -07:00
Nick Terrell 612e947c5e wire up bmi2 support 2020-08-17 16:35:28 -07:00
Nick Terrell ba1fd17a9f speed up literal header decoding 2020-08-17 12:17:53 -07:00
Nick Terrell 6004c1117f speed up small blocks 2020-08-16 23:03:38 -07:00
Nick Terrell a8006264cf small blocks benchmark 2020-08-14 18:57:20 -07:00
Nick TerrellandGitHub 49069a3f0a Merge pull request #2265 from terrelln/prefetch-fix 
Prefer __builtin_prefetch over inline asm
 2020-08-11 10:07:53 -07:00
Nick Terrell e3bda594ae Prefer __builtin_prefetch over inline asm 
Reorder the ifdefs for the PREFETCH macros so that the compiler builtin is
favored over the inline assembly for aarch64.
 2020-08-10 22:17:18 -07:00
Nick TerrellandGitHub f85a0f8bcf Merge pull request #2256 from helloguo/dev 
Optimize ZSTD_wildcopy
 2020-07-29 11:57:49 -07:00
Nick TerrellandGitHub 01261bc8b6 Merge pull request #2252 from helloguo/dev 
Fix preprocessor in ZSTD_wildcopy
 2020-07-27 12:59:10 -07:00
Nick TerrellandGitHub bd21e4b264 Merge pull request #2242 from evverx/cifuzz-ubsan-msan 
cifuzz: turn on UBsan and MSan
 2020-07-13 14:26:13 -04:00
Nick TerrellandGitHub bdf716bc64 Merge pull request #2239 from evverx/evverx-cifuzz 
cifuzz: don't upload artifacts when CIFuzz fails to build fuzz targets
 2020-07-10 11:26:15 -04:00
Nick TerrellandGitHub a2b6a8fbc6 Merge pull request #2220 from dkcasset/fix-lib-Makefile-older-sed-version 
Replace -E option with equivalent -r for older versions of sed
 2020-06-29 17:19:24 -07:00
Nick TerrellandGitHub 3690fe30f3 Merge pull request #2221 from BioDataAnalysis/bda_install_dll 
Install MSVC runtime shared libraries (DLL) with cmake
 2020-06-29 17:14:50 -07:00
Nick TerrellandGitHub 7afd5d85d3 Merge pull request #2218 from terrelln/assert-seq 
Fix unused variable warnings in fuzzing build mode without asserts
 2020-06-22 17:41:18 -07:00
Nick TerrellandGitHub 081691a3aa Merge pull request #2217 from terrelln/cover-redundant 
[cover] Remove unnecessary mask and dedup hash functions
 2020-06-22 17:41:13 -07:00
Nick TerrellandGitHub 370933fa20 Merge pull request #2209 from Niadb/dev 
Explicitly use __cdecl for qsort, to avoid warning when default calling convention is not __cdecl
 2020-06-22 17:41:03 -07:00
Nick Terrell cce0edfdbe Fix unused variable warnings in fuzzing build mode without asserts 
Fix unused vairable warnings when `FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION` is defined but asserts are disabled.

Fixes #2210.
 2020-06-22 12:56:57 -07:00
Nick Terrell 2312b819af [cover] Remove unnecessary mask and dedup hash functions 
* Remove the unnecessary mask, since `ZSTD_hash*()` already ensures
  the output is mod 2^h.
* Dedup the hash functions and use `ZSTD_hash*()` directly.
 2020-06-22 12:52:13 -07:00
Nick TerrellandGitHub 78601d0806 Merge pull request #2212 from cwoffenden/single-file-dict 
Single-file libs now include dictBuilder
 2020-06-22 12:46:54 -07:00
Nick TerrellandGitHub 34c0d85916 Merge pull request #2214 from terrelln/superblock-cross-entropy 
[superblock] Add defensive assert and bounds check
 2020-06-22 12:27:58 -07:00
Nick Terrell 1047097dad [superblock] Add defensive assert and bounds check 
The bound check condition should always be met because we selected `set_basic` as
our encoding type. But that code is very far away, so assert it is true so if it is
ever false we can catch it, and add a bounds check.

Fixes #2213.
 2020-06-22 10:21:38 -07:00
Nick TerrellandGitHub b6a9ded994 Merge pull request #2202 from terrelln/dict-fi 
[lib] Allow compression dictionaries with missing symbols
 2020-06-15 10:54:33 -07:00
Nick Terrell 08981d2638 [lib] Allow compression dictionaries with missing symbols 
Allow compression to use dictionaries with missing symbols in their
entropy tables. We set the FSE repeat mode to check when there are
missing symbols, and set the FSE repeat mode to valid when all symbols
are present.

Note that when not all symbols are present, the heuristics which favor
dictionary tables for lower compression levels won't activate.

Tested by manually creating a dictionary with missing symbols of every
type, and validing that the compressor rejects it before this change,
and accepts it after this change. Also, I ran the `dictionary_loader`
fuzzer for >1 hour of CPU time without running into cases where
compression succeeds, but decompression fails.

Fixes #2174.
 2020-06-12 17:57:19 -07:00
Nick TerrellandGitHub 9eb2ccc9fb Merge pull request #2167 from terrelln/pzstd-ms-fix 
[pzstd] Fix compilation error in MinGW
 2020-05-22 23:38:27 -07:00
Nick Terrell 5aa5aa4df7 [pzstd] Fix compilation error in MinGW 2020-05-22 22:26:02 -07:00
Nick Terrell 651d3d73e0 [test] Update the ldm loadedDictEnd test to cover zstdmt 2020-05-19 16:14:14 -07:00
Nick Terrell 3cc227e90e [ldm][mt] Fix loadedDictEnd 2020-05-19 15:55:03 -07:00
Nick TerrellandGitHub 0dcd3eec43 Merge pull request #2152 from terrelln/simple-rt-bound 
[fuzz] Expand the allowedExpansion
 2020-05-19 12:56:11 -07:00
Nick Terrell b82bf711fc [fuzz] Expand the allowedExpansion 2020-05-19 11:42:53 -07:00
Nick TerrellandGitHub 9778f46014 Merge pull request #2150 from terrelln/ldm-dict-reset 
[ldm] Reset loadedDictEnd when the context is reset
 2020-05-18 18:33:01 -07:00