ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,490 Commits 17 Branches 77 Tags
9d31c704d569cd5bfed24d1f13c767c5e25d183d
Commit Graph
1176 Commits
Author SHA1 Message Date
Nick Terrell 9d31c704d5 Don't shrink window log when streaming with a dictionary 
Fixes #2442.

1. When creating a dictionary keep the same behavior as before.
   Assume the source size is 513 bytes when adjusting parameters.
2. When calling ZSTD_getCParams() or ZSTD_adjustCParams() keep
   the same behavior as before.
3. When attaching a dictionary keep the same behavior of ignoring
   the dictionary size. When streaming this will select the
   largest parameters and not adjust them down. But, the CDict
   will use the correctly sized parameters, which seems like the
   right tradeoff.
4. When not attaching a dictionary (either forced not to, or
   using a prefix dictionary) we select parameters based on the
   dictionary size + source size, and assume the source size is
   small, which is the same behavior as before. But, now we don't
   adjust the window log (and hash and chain log) down when the
   source size is unknown.

When the source size is unknown all cdicts should attach, except
when the user disables attaching, or `forceWindow` is used. This
means that when streaming with a CDict we end up in the good case
where we get small CDict parameters, and large source parameters.

TODO: Add a streaming + dictionary regression test case.
 2021-01-04 15:54:09 -08:00
Nick Terrell 66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
senhuang42 5c41490bfe Use pre-defined constants 2020-12-21 11:52:05 -05:00
senhuang42 7e11bd012b Implement skippable frame function 2020-12-21 11:13:22 -05:00
Yann Collet 0b39531d75 moving all references to release branch 
was previously `master`
 2020-12-16 23:00:35 -08:00
W. Felix Handte 9dab03db90 Create Enum to Represent Static/Dynamic Allocation Distinction in cwksp 2020-12-09 14:57:37 -05:00
W. Felix Handte db9e73cb07 Don't ASAN-Poison Statically-Allocated Workspaces 
Addresses #2286.
 2020-12-09 13:00:47 -05:00
Nick Terrell c238db046f Merge pull request #2414 from terrelln/mt-progress 
[lib] Ensure that multithreaded compression always makes some progress
 2020-12-04 16:30:08 -08:00
Nick Terrell 4c58cb8383 [lib] Ensure that multithreaded compression always makes some progress 2020-12-03 20:25:14 -08:00
Nick Terrell 6672689e7e Merge pull request #2406 from terrelln/linux-wrapper-api 
[linux] Add the linux wrapper API
 2020-12-02 16:49:03 -08:00
Nick Terrell 894ae36675 Merge pull request #2390 from animalize/clamp_level 
Clamp compression level
 2020-12-02 14:35:58 -08:00
senhuang42 2cbd038528 Move max nb seq check to per-block 2020-12-02 12:11:32 -05:00
Nick Terrell 3cda5fae77 [minor][lib] Remove double semicolon 2020-12-02 01:08:08 -08:00
senhuang42 3efe9c902b Add sequence nb validation to compressSequences(), adjust minMatch comparisons 2020-12-01 10:54:45 -05:00
senhuang42 4c5f337248 Use cctx's minMatch instead of global MINMATCH, make fuzzer use validation 2020-11-30 15:41:20 -05:00
sen c5fbd55dac Merge pull request #2387 from senhuang42/compress_sequence_API 
[RFC] New sequence compression API
 2020-11-20 16:54:20 -05:00
senhuang42 7742f076b4 Add experimental param for sequence validation 2020-11-20 11:57:41 -05:00
senhuang42 0e32928b7d Remove unnecessary repcode backup, apply style choices, use function pointer 2020-11-20 11:02:19 -05:00
senandNick Terrell e924a0fa51 Explicit cast for visual warnings 
Github has automatic commits now! Cool

Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
 2020-11-19 17:32:40 -05:00
senhuang42 dcbbf7c09f Unroll isRLE loop 2020-11-19 12:38:13 -05:00
senhuang42 05c0229668 Clean up visual conversion warnings 2020-11-18 15:36:29 -05:00
senhuang42 d6d7ba2a1f Modification to offset validation to include entire sequence 2020-11-17 10:13:22 -05:00
senhuang42 8f3136a9c7 Fix assert edge case, improve documentation in zstd.h 2020-11-16 18:05:35 -05:00
senhuang42 f6baad87d6 Fix warnings and make validation enabled by default 2020-11-16 12:00:06 -05:00
senhuang42 55b90ef010 Fix unit tests to agree with new changes 2020-11-16 11:36:37 -05:00
senhuang42 7f563b0519 Add new sequence format as an experimental CCtx param 2020-11-16 10:49:17 -05:00
senhuang42 347824ad73 Overhaul logic to simplify, add in proper validations, fix match splitting 2020-11-16 10:49:17 -05:00
senhuang42 46824cb018 Add new sequence compress api params to cctx 2020-11-16 10:49:17 -05:00
senhuang42 48405b4633 Fix srcSize=0 edge case 2020-11-16 10:49:17 -05:00
senhuang42 022e6d81e7 Fix literals length calculation 2020-11-16 10:49:17 -05:00
senhuang42 dad20b5ccb Remove dstCapacity error check 2020-11-16 10:49:17 -05:00
senhuang42 b8e16a2057 Remove extraneous function in this API 2020-11-16 10:49:17 -05:00
senhuang42 f29507c4fc Add check comparing offset to window size 2020-11-16 10:49:17 -05:00
senhuang42 7a6e46a92f Fix MSAN errors 2020-11-16 10:49:17 -05:00
senhuang42 cc2642bd17 Address edge case with endPosInSequence 2020-11-16 10:49:17 -05:00
senhuang42 fd10007174 Change debug levels to appropriate ones 2020-11-16 10:49:17 -05:00
senhuang42 2db8441245 Add RLE support 2020-11-16 10:49:17 -05:00
senhuang42 dfef298336 Fix various build warnings 2020-11-16 10:49:17 -05:00
senhuang42 2bbdddf24e Add test case to roundtrip using ZSTD_getSequences() and ZSTD_compressSequences() 2020-11-16 10:49:16 -05:00
senhuang42 5fd69f8173 Add documentation for new api functions 2020-11-16 10:49:16 -05:00
senhuang42 e8b7fdb64b Refactor for enhanced code clarity 2020-11-16 10:49:16 -05:00
senhuang42 c675fb46f1 Rename internal function compressSequences(), and promote new *_ext() functions to their actual name 2020-11-16 10:49:16 -05:00
senhuang42 013434e1e4 Add another API function to compress with existing CCTX 2020-11-16 10:49:16 -05:00
senhuang42 c44ce29013 More adjustments to improve code clarity 2020-11-16 10:49:16 -05:00
senhuang42 48f67da854 Pull compressStream2() transparent initialization into its own function 2020-11-16 10:49:16 -05:00
senhuang42 c86151f53c Add initial support for new ZSTD_Sequence mode 2020-11-16 10:49:16 -05:00
senhuang42 e0f26afce9 Add sequence compression format param 2020-11-16 10:49:16 -05:00
senhuang42 f51af9a609 Always ensure sequenceRange updates properly, add more error forwarding 2020-11-16 10:49:16 -05:00
senhuang42 1a449688fd Various minor logical refactors to improve clarity 2020-11-16 10:49:16 -05:00
senhuang42 e5fe485dcc Fix cSize calculation for noCompressBlocks 2020-11-16 10:49:16 -05:00