ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,093 Commits 17 Branches 77 Tags
7e6f91ed84e62513c2d12e7e40254704f593000c
Commit Graph
8093 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell 7e6f91ed84 [minor] Improve docs and add an assert in response to review 2020-10-12 16:43:17 -07:00
Nick Terrell d5c688e8ae Fix ZSTD_adjustCParams_internal() to handle dictionary logic 
Pass in the `ZSTD_cParamMode_e` to select how we define our cparams.
Based on the mode we either take the `dictSize` into account or we set
it to `0`. See the documentation for `ZSTD_cParamMode_e`.

Some of the modes currently share the same behavior. But they have
distinct modes because they are drastically different cases. E.g.
compression + reprocessing the dictionary and creating a cdict.

Additionally, when downsizing the hashLog and chainLog take the
(adjusted) dictionary size into account, since the size of the
dictionary gets added onto the window size.

Adds a simple test to ensure that we aren't downsizing too far.
 2020-10-12 12:50:04 -07:00
Nick Terrell fadaab8c7c [minor improvement] Pass 0 as the content size in the DDS 
The DDS structure can't be copied into the working tables like the DMS.
So it doesn't need to account for the source size when sizing its
parameters, just the dictionary size.
 2020-10-12 12:47:21 -07:00
Nick Terrell 48ef15fb47 [minor improvement] Pass dictSize when selecting parameters 
When selecting parameters in streaming compression with a dictionary use
the dictionary size to select the parameters.
 2020-10-12 12:47:19 -07:00
Nick Terrell 012818df99 [refactor] Remove ZSTD_resetCStream_internal() 
This function is only called in one place. It isn't a logical separation
of duties, and it was only obsfucating the code now, so inline it.
 2020-10-12 12:46:10 -07:00
Nick Terrell 7083f79008 [bug] Fix dictContentType when reprocessing cdict 
Conditions to trigger:
* CDict is loaded as raw content.
* CDict starts with the zstd dictionary magic number.
* The CDict is reprocessed (not attached or copied).
* The new API is used (streaming or `ZSTD_compress2()`).

Bug: The dictionary is loaded as a zstd dictionary, not a raw content
dictionary, because the dict content type is set to `ZSTD_dct_auto`.

Fix: Pass in the dictionary content type from cdict creation to the call
to `ZSTD_compress_insertDictionary()`.

Test: Added a test case that exposes the bug, and fixed the raw
content tests to not modify the `dictBuffer`, which makes all future
tests with the `dictBuffer` raw content, which doesn't seem intentional.
 2020-10-12 12:46:10 -07:00
Yann ColletandGitHub b951ad20a2 Merge pull request #2329 from senhuang42/prevent_summary_updates_when_using_stdout 
Prevent summary updates when using stdout
 2020-10-09 01:01:36 -07:00
Yann ColletandGitHub 12541931fa Merge pull request #2328 from marxin/zstd-pool-api 
Allow external creation of POOLs that can be shared.
 2020-10-09 01:00:50 -07:00
Yann ColletandGitHub 6fdb0cb8d9 Merge pull request #2303 from senhuang42/let_cdict_take_clevel_priority 
For ZSTD_compressStream2(), let cdict take compression level priority
 2020-10-09 00:48:30 -07:00
Yann ColletandGitHub c3ee284ca2 Merge pull request #2319 from facebook/fullbench_stream2 
update fullbench for compressStream2()
 2020-10-09 00:40:59 -07:00
senhuang42 7259b258d1 Add callsites to zstdcli.c and tests to playTests.sh 2020-10-07 13:47:38 -04:00
senhuang42 93cd9d8a6e Add hasStdoutOutput setter to fileio.h 2020-10-07 13:44:25 -04:00
senhuang42 dd3dac9ce0 Fixed logic for stdout output 2020-10-07 13:43:27 -04:00
senhuang42 1ebe360d0f Add new stdoutOutput field 2020-10-07 13:42:34 -04:00
Martin Liska b684900a4a Allow external creation of POOLs that can be shared. 2020-10-07 12:44:33 +02:00
Yann ColletandGitHub f7d4943788 Merge pull request #2330 from senhuang42/fix_stdinout_error_messages 
Improve error messages on console input/output
 2020-10-06 15:58:37 -07:00
Nick TerrellandGitHub 4b4d8b4dc9 Merge pull request #2338 from terrelln/comments 
Add comments to ZSTD_getLowest{Match,Prefix}Index()
 2020-10-01 18:56:24 -07:00
Nick TerrellandGitHub 0057c4acf7 Merge pull request #2333 from terrelln/stable-dst 
Reset all decompression parameters in ZSTD_DCtx_reset()
 2020-10-01 18:56:11 -07:00
Nick Terrell 2e7d174130 Reset all decompression parameters in ZSTD_DCtx_reset() 
* Reset all decompression parameters in `ZSTD_DCtx_reset()` when
  resetting parameters.
* Add a test case.
 2020-10-01 14:19:21 -07:00
Nick Terrell 27c969ed07 Add comments to ZSTD_getLowest{Match,Prefix}Index() 
Clarify how we handle dictionaries in each case.
 2020-10-01 13:21:46 -07:00
Yann ColletandGitHub cc88eb7594 Merge pull request #2317 from animalize/msvc_inline 
Let MSVC force inline ZSTD_hashPtr() function
 2020-09-30 08:27:53 -07:00
Yann ColletandGitHub 6932216e9a Merge pull request #2321 from senhuang42/disallow_repcode_0_in_dict 
Update documentation about repcodes in dictionaries
 2020-09-30 08:27:21 -07:00
Yann ColletandGitHub 83461ce963 Merge pull request #2322 from senhuang42/guard_against_stdin_for_warning_prompts 
Don't let warning messages consume input from stdin
 2020-09-30 08:26:50 -07:00
Nick TerrellandGitHub d69d08ed6c Merge pull request #2326 from terrelln/kernel-test-fix 
Fix issues and warnings exposed by Kernel Test Robot
 2020-09-29 13:51:23 -07:00
senhuang42 ce56810a32 Modify error messages on console input/output 2020-09-28 12:15:18 -04:00
senhuang42 02422db841 Fix Stdin typo 2020-09-25 11:51:35 -04:00
Yann ColletandGitHub 236b98e7de Merge pull request #2310 from senhuang42/fix_multifile_status_bar 
Fix multifile status bar and summary - clear out extraneous characters
 2020-09-25 07:48:33 -07:00
Nick Terrell f1cbeec039 [superblock] Reduce stack usage by correctly sizing header buffers 2020-09-24 19:42:04 -07:00
Nick Terrell 6a1e526ea7 [lib] Add ZSTD_COMPRESS_HEAPMODE tuning parameter 2020-09-24 19:42:04 -07:00
Nick Terrell b841387218 [freestanding] Improve macro resolution to handle #if X 2020-09-24 19:42:04 -07:00
Nick Terrell caecd8c211 Allow user to override ASAN/MSAN detection 
Rename ADDRESS_SANITIZER -> ZSTD_ADDRESS_SANITIZER and same for
MEMORY_SANITIZER. Also set it to 0/1 instead of checking for defined.
This allows the user to override ASAN/MSAN detection for platforms that
don't support it.
 2020-09-24 19:42:04 -07:00
Nick Terrell 88fac5d514 Remove call to memset 
The previous commit fixes the test so it errors on calls to mem*()
functions from <string.h>.
 2020-09-24 19:42:04 -07:00
Nick Terrell 683150e59f [linux-kernel] Avoid including <string.h> in the tests 2020-09-24 19:42:04 -07:00
Nick Terrell 9ae0483858 Reorganize zstd_deps.h and mem.h + replace mem.h for the kernel 2020-09-24 19:41:59 -07:00
Nick Terrell 260fc75028 Move __has_builtin() fallback define to compiler.h 2020-09-24 15:51:08 -07:00
Nick Terrell 4d63ee57f5 Move ASAN/MSAN support declarations to compiler.h 2020-09-24 15:51:08 -07:00
Nick Terrell b09ec5c2b9 Remove MEM_STATIC_ASSERT and use DEBUG_STATIC_ASSERT instead 2020-09-24 15:51:04 -07:00
senhuang42 9f7212a48b Update unit tests 2020-09-24 16:44:33 -04:00
senhuang42 88f4410390 Add more useful failure message when stdin is an input 2020-09-24 16:29:12 -04:00
senhuang42 93d63eaeb8 Expand UTIL_requireUserConfirmation to include stdin input check 2020-09-24 15:58:06 -04:00
senhuang42 432186cbea Add FIO_determineHasStdinInput() function and member to fCtx 2020-09-24 15:55:30 -04:00
senhuang42 0e8ac6b995 Add fCtx to FIO_openDstFile() 2020-09-24 15:49:30 -04:00
Yann ColletandGitHub c6c0a57c53 Merge pull request #2315 from senhuang42/allow_zstd_suffix 
Support .zstd suffix only for decompression
 2020-09-24 09:44:48 -07:00
Nick Terrell 9009b59f09 [linux-kernel] Add missing semicolon in zstd_deps.h 2020-09-23 23:26:10 -07:00
Nick Terrell 9261476b7d [lib] Wrap customMem xor checks in parens for readability 
This clarifies operator precedence, and quiets cppcheck in
the Kernel Test Robot. I think this is a slight bonus to
readability, so I am accepting the suggestion.
 2020-09-23 23:26:07 -07:00
Nick TerrellandGitHub 2fc1d50ee7 Merge pull request #2324 from terrelln/kernel-test-fix 
Silence warnings reported by Kernel Test Robot
 2020-09-23 15:01:56 -07:00
Nick Terrell a70fa342f0 [CI][linux-kernel] Enable -Wunused-const-variable and -Wunused-but-set-variable 2020-09-23 12:59:57 -07:00
Nick Terrell dec7fb03ec [lib] Silence -Wunused-const-variable warnings 2020-09-23 12:59:57 -07:00
Nick Terrell 978659d34f [linux-kernel] Fix unused variable warnings with malloc, calloc, and free 2020-09-23 12:59:57 -07:00
senhuang42 21cd640b93 Add unit tests to guard against bad stdin 2020-09-22 14:55:41 -04:00