ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,782 Commits 17 Branches 77 Tags
13449d7ce1da1fcf037600e33afe4fa6bc85d12e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
senandGitHub 13449d7ce1 Add PHONY targets to makefiles (#2629) 2021-05-07 14:03:19 -04:00
senandGitHub 9e94b7cac5 Assert no divison by 0, correct superblocks 0 sequences case (#2592) 2021-05-07 13:26:56 -04:00
senandGitHub d8d6e48a0a Add threadPool unit tests to fuzzer.c (#2604) 2021-05-07 11:13:44 -04:00
senandGitHub 91465e23b2 [1.5.0] Enable multithreading in lib build by default (#2584) 
* Update lib Makefile to have new targets

* Update lib/README.md for mt
 2021-05-07 11:13:30 -04:00
senandGitHub 698f261b35 [1.5.0] Deprecate some functions (#2582) 
* Add deprecated macro to zstd.h, mark certain functions as deprecated

* Remove ZSTD_compress.c dependencies on deprecated functions
 2021-05-06 17:59:32 -04:00
senandGitHub 6030cdfede Add --progress flag (#2595) 2021-05-06 14:50:28 -04:00
senandGitHub d6be7659b0 Add seekable roundtrip fuzzer (#2617) 2021-05-06 10:08:21 -04:00
Sen Huang e6c8a5dd40 Fix incorrect usages of repIndex across all strategies 2021-05-04 19:50:55 -04:00
senandGitHub cc31bb8b66 Merge pull request #2598 from senhuang42/reduce_index_rowhash_fix 
Fix chaintable check to include rowhash in ZSTD_reduceIndex()
 2021-05-03 17:34:39 -04:00
senandGitHub 4c5cc345fb Merge pull request #2581 from senhuang42/lcm_stable 
[1.5.0] Promote ZSTD_c_literalCompressionMode to stable params
 2021-05-03 11:59:19 -04:00
senandGitHub cdc979ddb3 Merge pull request #2580 from senhuang42/defaultclevel_to_stable 
[1.5.0] Promote ZSTD_defaultCLevel() into stable API
 2021-05-03 11:59:05 -04:00
senhuang42 61fe571af6 Fix chaintable check to include rowhash in ZSTD_reduceIndex() 2021-04-30 19:52:04 -04:00
senandGitHub f8afc66573 Merge pull request #2588 from senhuang42/update_regressiontest 
[regressiontest] Update results.csv
 2021-04-27 12:40:07 -04:00
senhuang42 33abda4400 Update results.csv 2021-04-26 15:55:23 -04:00
senandGitHub 3e2fbfd056 Merge pull request #2579 from senhuang42/getcdictID_to_stable 
[1.5.0] Promote ZSTD_getDictID_fromCDict() into stable API
 2021-04-26 09:55:43 -04:00
Sen Huang 3c595a4a79 Add literalCompressionMode to stable cParams 2021-04-26 09:55:06 -04:00
senandGitHub 14c11c7459 Merge pull request #2586 from senhuang42/dds_fuzz 
Add DDS to oss fuzzer
 2021-04-24 12:39:16 -04:00
senhuang42 f80dec66b0 Add DDS to oss fuzzer 2021-04-22 18:21:43 -04:00
senandGitHub 3eb3845898 Merge pull request #2583 from senhuang42/remove_zbuff 
[1.5.0] Remove ZBUFF
 2021-04-20 17:05:32 -04:00
senhuang42 a423305e7b Remove ZBUFF tests 2021-04-19 17:27:05 -04:00
senhuang42 3b98987496 Remove building of ZBUFF/deprecated folder by default 2021-04-19 17:12:00 -04:00
Sen Huang c5869677d9 Moved ZSTD_defaultCLevel() into stable API 2021-04-16 10:15:40 -07:00
Sen Huang 9c1ca3c00b Moved ZSTD_getDictID_fromCDict() into stable API 2021-04-16 10:14:29 -07:00
senandGitHub 12c045f74d Merge pull request #2574 from senhuang42/repcode_mismatch_detector_fix 
Correct the block splitter mismatched repcodes detection.
 2021-04-12 23:27:43 -04:00
senandGitHub ebd41ebe56 Merge pull request #2572 from senhuang42/row_hash_hashcache_one_off_error_fix 
Adjust nb elements to prefetch in ZSTD_row_fillHashCache()
 2021-04-12 15:38:53 -04:00
Sen Huang 8844f93957 Adjust nb elements to prefetch in ZSTD_row_fillHashCache() 2021-04-12 14:24:58 -04:00
Sen Huang 550f76f131 Correct the detection of mismatched repcodes 2021-04-09 09:08:51 -07:00
senandGitHub 56421f34e4 Merge pull request #2494 from senhuang42/row_hash2 
SIMD Row Based Matchfinder 🚀
 2021-04-08 10:55:14 -04:00
Sen Huang 4d63d6e8aa Update results.csv, add Row hash to regression test 2021-04-07 10:31:41 -07:00
senandGitHub f71aabb5b5 Move clevel override to after initLocalDict() (#2571) 2021-04-06 21:05:37 -04:00
senandGitHub f1e8b565c2 Maintain two repcode histories for block splitting, replace invalid repcodes (#2569) 2021-04-06 17:25:55 -04:00
senandGitHub e38124555e Fix dictionary force reloading clevel selection (#2570) 
* Move cdict clevel override to before localdict init

* Update results.csv after dict load changes
 2021-04-06 15:35:09 -04:00
senandGitHub 980f3bbf83 [cwksp] Align all allocated "tables" and "aligneds" to 64 bytes (#2546) 
* Perform 64-byte alignment of wksp tables and aligneds internally

* Clean up cwskp_finalize() function to only do two allocs

* Refactor aligned/buffer reservation code, remove ASAN req for alignment reservations

* Change from allocating 128 bytes always to allocating only buffer space as needed for tables/aligned

* Back out aligned/table reservation order restriction

* Add stricter bounds for new/resized wksps, fix comment in zstd_cwksp.h
 2021-04-01 20:07:19 -04:00
senandGitHub 255925c231 Fix repcode-related OSS-fuzz issues in block splitter (#2560) 
* Do not emit last partitions of blocks as RLE/uncompressed

* Fix repcode updates within block splitter

* Add a entropytables confirm function, redo ZSTD_confirmRepcodesAndEntropyTables() for better function signature

* Add a repcode updater to block splitter, no longer need to force emit compressed blocks
 2021-03-31 15:14:59 -04:00
senandGitHub 84ccb81e7c Merge pull request #2561 from senhuang42/longlength_enum 
Add enum for representing long length ID
 2021-03-26 15:55:12 -04:00
Sen Huang b1a43455f8 Add enum for representing long length ID 2021-03-26 10:41:09 -07:00
senandGitHub ab216bc2c5 Merge pull request #2559 from senhuang42/add_dict_regression_tests_backup 
Add different dict modes to compression ratio regression test, update results.csv
 2021-03-25 19:26:06 -04:00
Sen Huang bbbd578f45 Update results.csv 2021-03-25 11:16:37 -07:00
senandGitHub 4fe2e7ae14 Merge pull request #2558 from senhuang42/msan_block_splitter_fix 
Fix block splitter minor MSAN warning.
 2021-03-25 13:51:43 -04:00
Sen Huang f27e326456 Restrict dictmode regression tests only to advanced API, fix some compiler warnings 2021-03-25 10:39:08 -07:00
Sen Huang 1cadf86b39 Add tests to regression tests for dict 2021-03-25 10:39:08 -07:00
senandGitHub b0407b9f0e Merge pull request #2555 from senhuang42/default_clevel_func 
Add ZSTD_defaultCLevel() function to public API
 2021-03-25 13:07:28 -04:00
Sen Huang ef4e26bda5 Add clang msan fuzz test to github actions 2021-03-25 09:11:43 -07:00
Sen Huang 2a907bf4aa Move lastCountSize into a returned struct, fix MSAN error 2021-03-25 09:11:15 -07:00
Sen Huang e398744a35 Add ZSTD_defaultCLevel() function to public API 2021-03-25 08:04:00 -07:00
senandGitHub bf542c8a8d Merge pull request #2447 from senhuang42/block_splitter_v2 
Recursive block splitting
 2021-03-24 12:27:22 -04:00
Sen Huang 5b566ebe08 Rename *compressSequences*() functions for clarity 2021-03-24 08:21:29 -07:00
Sen Huang 0ef1f935b7 Add a fallback in case the total blocksize of split blocks exceeds raw block size 2021-03-24 08:21:29 -07:00
Sen Huang c90e81a692 Enable block splitter by default when applicable 2021-03-24 08:21:29 -07:00
Sen Huang e34332834a Clean up various functions, add debuglogging for estimate vs. actual sizes 2021-03-24 08:21:29 -07:00
Sen Huang 41c3eae6d9 Fix various fuzzer failures: repcode history, superblocks 2021-03-24 08:21:29 -07:00
senhuang42 0633bf17c3 Change 1.3.4 bugfix to be cross-compatible with superblocks and normal compression 2021-03-24 08:21:29 -07:00
senhuang42 eb1ee8686d Refactor buildSequencesStatistics() to avoid pointer increment for superblocks 2021-03-24 08:21:29 -07:00
senhuang42 e2bb215117 Add unit tests and fuzzer param 2021-03-24 08:21:09 -07:00
senhuang42 de52de1347 Add recursive block split algorithm 2021-03-24 08:21:09 -07:00
senhuang42 f06f6626ed Update function names for consistency 2021-03-24 08:20:54 -07:00
senhuang42 c56d6e49e8 Add block splitter to experimental params 2021-03-24 08:20:54 -07:00
senhuang42 2949a95224 Refactor block compression logic into single function 2021-03-24 08:20:54 -07:00
senhuang42 c05c090cc2 Centralize entropy statistics calculations to zstd_compress.c 2021-03-24 08:20:29 -07:00
senandGitHub c48889f097 Merge pull request #2538 from senhuang42/monotonicity_test 
Add memory monotonicity test over srcSize
 2021-03-22 16:54:34 -04:00
Sen Huang dff4a0e867 Make ZSTD_estimateCCtxSize_internal() loop through all srcSize parameter sets as well 2021-03-21 16:15:31 -07:00
senandGitHub eace4abc25 Merge pull request #2540 from senhuang42/fix_dds_supported 
Fix dedicated dict search isSupported() requirements.
 2021-03-17 23:25:53 -04:00
Sen Huang 77ae664ba6 Fix ZSTD_dedicatedDictSearch_isSupported() requirements 2021-03-16 17:36:05 -07:00
senhuang42 386111adec Add a nbSeq argument to compressSequences() 
Refactor ZSTD_compressBlock_internal() to do the block header write within and add nbSeq argument to compressSequences()
 2021-03-16 14:04:22 -07:00
senhuang42 98764493cf Move block header write into compressBlock_internal() 2021-03-16 14:04:22 -07:00
senandGitHub 413b3198b0 Merge pull request #2534 from foxeng/fix-seek-descriptor-check 
Fix seek table descriptor check when loading
 2021-03-16 13:09:00 -04:00
Sen Huang b9dd821441 Add mem monotonicity test over srcSize 2021-03-16 08:24:26 -07:00
senandGitHub a3feed8dcd Merge pull request #2517 from PaulBone/num_cores 
Make the number of physical CPU cores detection more robust
 2021-03-08 11:14:34 -05:00
senandGitHub 0388054aab Merge pull request #2516 from senhuang42/seekable_hang_fix 
Seekable hang fix
 2021-03-03 17:30:46 -05:00
Sen Huang 527a20c3cd Fix seekable decompress hanging 2021-03-02 14:30:03 -08:00
senhuang42 444c4650a0 Add newline to end of cli help message 2021-02-17 12:30:42 -05:00
senandGitHub 7e6c53c7fe Merge pull request #2465 from senhuang42/cformat_2021 
[license] Change year to 2021 compression_format.md
 2021-01-11 12:35:34 -05:00
senhuang42 1d6d64afa3 Change year to 2021 for compression format file 2021-01-11 08:53:29 -05:00
senandGitHub 69085db61c Merge pull request #2446 from senhuang42/multiple_ddicts_v3 
[RFC] Support references to multiple DDicts
 2021-01-08 16:49:45 -05:00
senhuang42 9ae0dd9336 Fix Visual and staticanalyze warnings 2021-01-07 17:58:37 -05:00
senhuang42 4f7584e7a3 Allow freestanding lib script regex to detect XXH64( 2021-01-07 12:29:12 -05:00
senhuang42 17222654bf Add streaming decompression to unit test 2021-01-07 12:29:12 -05:00
senhuang42 c2c9b8a7ec Address comments, clean up interface/internals 2021-01-07 12:29:12 -05:00
senhuang42 22b7bff2bc Add unit test, improve documentation 2021-01-07 12:29:12 -05:00
senhuang42 ea52fc3606 Use XXHash for hash function, create a sensible public interface 2021-01-07 12:29:12 -05:00
senhuang42 7c1a79f232 Add debuglog statements 2021-01-07 12:29:11 -05:00
senhuang42 d1a6a9d285 Reference requested dict ID at decompression time 2021-01-07 12:29:11 -05:00
senhuang42 5a6d3eef2b Allocate memory for DDict hash set when parameter is set 2021-01-07 12:29:11 -05:00
senhuang42 fd5b608f1c Add parameter to control multiple DDicts 2021-01-07 12:29:11 -05:00
senhuang42 f933668d3f Implement hashset for dictIDs 2021-01-07 12:29:11 -05:00
senhuang42 5c41490bfe Use pre-defined constants 2020-12-21 11:52:05 -05:00
senhuang42 339d8ba103 Add unit test 2020-12-21 11:33:27 -05:00
senhuang42 7e11bd012b Implement skippable frame function 2020-12-21 11:13:22 -05:00
senandGitHub f8224ab116 Merge pull request #2418 from senhuang42/fix_paramgrill_warnings 
Fix various paramgrill conversion warnings
 2020-12-08 12:50:23 -05:00
senhuang42 b9ab6bc061 Fix various conversion warnings 2020-12-08 10:07:28 -05:00
senandGitHub f34d2f4192 Merge pull request #2408 from senhuang42/seekable_hang_fix 
Remove possibility of hanging when using seekable decompression
 2020-12-07 08:46:27 -05:00
senhuang42 db249bef4b Allow paramgrill in cmake on windows 2020-12-03 10:39:16 -05:00
senhuang42 260b85acf5 Fix MSVC 2019 warnings 2020-12-03 10:36:45 -05:00
senhuang42 a8693ddef7 Add seekable tests to CI 2020-12-03 09:25:45 -05:00
senhuang42 26f89d47aa Clean up makefile for seekable tests 2020-12-03 09:25:04 -05:00
senhuang42 152b55879c Add unit tests to seekable 2020-12-02 15:33:12 -05:00
senandGitHub a6ee1b3047 Merge pull request #2407 from senhuang42/sequence_compression_nbseqcheck_fix 
[oss-fuzz] Move sequence compression API nbSequences check
 2020-12-02 14:40:02 -05:00
senhuang42 9db49a3989 Add a forward progress requirement bound to seekable streaming decompression 2020-12-02 12:24:16 -05:00
senhuang42 2cbd038528 Move max nb seq check to per-block 2020-12-02 12:11:32 -05:00
senandGitHub 993baba315 Merge pull request #2405 from senhuang42/sequence_compression_minmatch_fix 
Sequence Compression API - make validator use cctx minMatch instead of global MINMATCH
 2020-12-01 17:34:01 -05:00