ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,093 Commits 17 Branches 77 Tags
efff5d8b2df02e2eed5decea1b321faef6b14931
Commit Graph
1428 Commits
Author SHA1 Message Date
Nick Terrell ede4f97153 [zstdmt] Fix bug where extra empty blocks are emitted 
When zstdmt cannot get a buffer and `ZSTD_e_end` is passed an empty
compression job can be created. Additionally, `mtctx->frameEnded` can be
set to 1, which could potentially cause problems like unterminated blocks.

The fix is to adjust to `ZSTD_e_flush` even when we can't get a buffer.
 2020-10-12 12:55:17 -07:00
Nick Terrell 9ab9229e11 [zstreamtest] Add compression determinism tests 
* Run compression twice and check the compressed data is byte-identical.
  The compression loop had to be rewritten to ensure deteriminism. It is
  guaranteed by always making maximal forward progress.
* When nbWorkers > 0, change the number of workers 1/8 of the time.
* Run in single-pass mode 1/4 of the time.

I've run a few hundred thousand iterations of zstreamtest and have seen
no deteriminism issues so far. Before the zstdmt fix that skips the
single-pass shortcut non-determinism showed up in a few hundred
iterations.
 2020-10-12 12:55:17 -07:00
Nick Terrell c51a9e79b9 [zstdmt] Rip out the zstdmt API 
This commit leaves only the functions used by zstd_compress.c. All other
functions have been removed from the API. The ZSTDMT unit tests in
fuzzer.c and zstreamtest.c have been rewritten to use the ZSTD API. And
the --mt zstreamtest tests have been ripped out.
 2020-10-12 12:55:16 -07:00
Yann ColletandGitHub b951ad20a2 Merge pull request #2329 from senhuang42/prevent_summary_updates_when_using_stdout 
Prevent summary updates when using stdout
 2020-10-09 01:01:36 -07:00
Yann ColletandGitHub c3ee284ca2 Merge pull request #2319 from facebook/fullbench_stream2 
update fullbench for compressStream2()
 2020-10-09 00:40:59 -07:00
senhuang42 7259b258d1 Add callsites to zstdcli.c and tests to playTests.sh 2020-10-07 13:47:38 -04:00
Nick TerrellandGitHub 0057c4acf7 Merge pull request #2333 from terrelln/stable-dst 
Reset all decompression parameters in ZSTD_DCtx_reset()
 2020-10-01 18:56:11 -07:00
Nick Terrell 2e7d174130 Reset all decompression parameters in ZSTD_DCtx_reset() 
* Reset all decompression parameters in `ZSTD_DCtx_reset()` when
  resetting parameters.
* Add a test case.
 2020-10-01 14:19:21 -07:00
Yann ColletandGitHub 83461ce963 Merge pull request #2322 from senhuang42/guard_against_stdin_for_warning_prompts 
Don't let warning messages consume input from stdin
 2020-09-30 08:26:50 -07:00
senhuang42 9f7212a48b Update unit tests 2020-09-24 16:44:33 -04:00
Yann ColletandGitHub c6c0a57c53 Merge pull request #2315 from senhuang42/allow_zstd_suffix 
Support .zstd suffix only for decompression
 2020-09-24 09:44:48 -07:00
senhuang42 21cd640b93 Add unit tests to guard against bad stdin 2020-09-22 14:55:41 -04:00
senhuang42 7aa3da1cd7 Use IS_CONSOLE macro to detect that we're indeed using a console 2020-09-22 14:15:52 -04:00
Nick Terrell 973f2adeec [tests] Don't write to stdout 2020-09-22 00:40:27 -07:00
Yann Collet 5618e000bd update fullbench for compressStream2() 
makes it possible to measure scenarios such as #2314
 2020-09-21 07:19:20 -07:00
Felix HandteandGitHub 200c960f1d Merge pull request #2311 from felixhandte/ddss-fix-cparam-derivation 
Fix Compression Parameter Derivation Bugs Introduced by DDSS Changes
 2020-09-18 14:02:14 -04:00
senhuang42 07034952df Add -f to .zstd decompression CLI test 2020-09-18 13:01:45 -04:00
senhuang42 6b6cc80196 Support .zstd suffix only for decompression 2020-09-18 12:49:51 -04:00
W. Felix Handte 9398acb245 Move Last Two Long Tests in fuzzer.c into Separate --long-tests Section 2020-09-17 13:31:10 -04:00
W. Felix Handte f23a321781 Update Regression Test Results 2020-09-17 12:23:05 -04:00
Yann ColletandGitHub e583e0be8c Merge pull request #2299 from senhuang42/env_var_num_threads 
Allow environment variable to specify number of threads for compression
 2020-09-14 14:04:19 -07:00
Yann Collet dec1a78d3e minor fix casting for Visual 2020-09-14 11:46:23 -07:00
Yann Collet c91a0855f8 check endDirective in ZSTD_compressStream2() 
fix #2297
also :
- `assert()` `endDirective` in `ZSTD_compressStream_internal()`, for debug mode
- add relevant tests
 2020-09-14 10:56:08 -07:00
W. Felix Handte d6246d4a0f Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout 
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
 2020-09-10 23:35:42 -04:00
W. Felix Handte 6d3f816b3e Test Fewer Dictionary Sizes 2020-09-10 22:30:52 -04:00
W. Felix Handte b6df3fd438 Fix Debug Logging in 32-bit Build 2020-09-10 22:10:02 -04:00
W. Felix Handte 2cc2b40a1b Test DDSS A Little More Thoroughly 2020-09-10 22:10:02 -04:00
W. Felix Handte b81f3a37f9 Easy: Fix Test 2020-09-10 18:51:52 -04:00
W. Felix Handte 2cf6cfc55f Add Fuzzer Test for the Various Dict Attachment Strategies 2020-09-10 18:51:52 -04:00
Nick Terrell a90779397a [lib] Reduce zstd stack usage by 1KB 2020-09-09 14:35:39 -07:00
senhuang42 a71963c7b8 nbThreads instead of numThreads 2020-09-09 12:40:00 -04:00
senhuang42 0a170b20a8 Add ZSTD_NUMTHREADS tests to playTests.sh 2020-09-08 10:34:50 -04:00
senhuang42 3aec385a10 Fix merge conflicts 2020-08-26 15:43:38 -04:00
Yann ColletandGitHub a8c66881e5 Merge pull request #2283 from senhuang42/progress_bars_for_multiple_files 
Refreshing progress bar for processing multiple files
 2020-08-26 11:54:50 -07:00
Nick TerrellandGitHub cf83aceaf3 Merge pull request #2282 from terrelln/ncount-fix 
[bug] Fix FSE_readNCount()
 2020-08-26 10:31:07 -07:00
senhuang42 a73e131f10 Adjust playTests.sh refuse overwrite test to include -q 2020-08-26 11:40:05 -04:00
Nick Terrell ae163015b1 [fuzz] Fix stream_decompress timeouts 2020-08-25 17:13:09 -07:00
Nick Terrell 49eeb2d1fc [fuzz] Disable superblock expansion test 2020-08-25 17:13:06 -07:00
Nick Terrell 4193638996 [bug] Fix FSE_readNCount() 
* Fix bug introduced in PR #2271
* Fix long-standing bug that is impossible to trigger inside of zstd
* Add a fuzzer that makes sure the normalized count always round trips
  correctly
 2020-08-25 15:42:41 -07:00
Yann ColletandGitHub f82d9865b9 Merge pull request #2278 from senhuang42/ignore_checksum_advanced_param 
New advanced decompression param to ignore checksums
 2020-08-25 12:08:53 -07:00
Nick TerrellandGitHub 614e446000 Merge pull request #2271 from terrelln/small-blocks 
Small block optimizations
 2020-08-24 18:54:33 -07:00
senhuang42 dde97de6c4 Only ask to proceed if using --rm, otherwise just display warning. -f bypasses it all. More robust tests 2020-08-24 20:20:39 -04:00
senhuang42 1acf243540 Add a warning whenever (de)compressing multiple files into one source, or into stdout 2020-08-24 19:10:03 -04:00
Nick Terrell 52f33a1da5 Fix compiler warnings 2020-08-24 16:09:45 -07:00
senhuang42 a030560d62 Add new DCtx param: validateChecksum and update unit tests 2020-08-24 17:28:00 -04:00
Nick Terrell 1302f8d676 [fix] Always return dstSize_tooSmall when it is the case 2020-08-24 13:38:13 -07:00
senhuang42 44c54a3e31 Addressing comments: more comments, cleanup, remove extra function, checksum logic 2020-08-24 16:14:19 -04:00
Nick Terrell 8def0e5fd3 Fix up code after reading through 2020-08-24 12:24:45 -07:00
senhuang42 ffaa0df76d Document change in CLI for --no-check during decompression in --help menu 2020-08-24 09:49:12 -04:00
senhuang42 e3f5f9658a Added CLI tests for --no-check, fixed ignore checksum logic 2020-08-22 16:05:40 -04:00