ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,861 Commits 17 Branches 77 Tags
a21b9036fec3ff2e7fd93fb534cd4e8244c385db
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
senandGitHub a21b9036fe Merge pull request #2678 from senhuang42/big_endian_no_intrinsics_fuzztest 
Fuzzer test with no intrinsics on S390x (big endian)
 2021-06-09 18:46:52 +03:00
senandGitHub dd33ec9db0 Merge pull request #2700 from gauthamkrishna9991/update_readme_travis_link 
Update README for Travis CI Badge
 2021-06-09 10:41:35 +03:00
senandGitHub 5fb3884f33 Merge pull request #2691 from senhuang42/per_pr_regressiontest 
Make regression test run on every PR
 2021-06-07 12:44:31 +03:00
Sen Huang 2ff5c7b59f Add no intrinsics fuzztest, rowhash compression size test, and S390X to travis 2021-06-07 00:54:53 -07:00
senhuang42 88acf0ac65 Make regression test run on every PR 2021-06-07 00:47:56 -07:00
senandGitHub 0a96d00064 Merge pull request #2684 from senhuang42/old_api_ldm_blocksplit 
Add ldm and block splitter auto-enable to old api
 2021-06-03 10:10:27 +03:00
senandGitHub 18d02cbf2e Merge pull request #2686 from senhuang42/arm64fuzztest 
Add arm64 fuzz test to travis
 2021-05-27 13:50:26 -04:00
senhuang42 56b7dd121c Add arm64 fuzz test to travis 2021-05-26 14:56:47 -04:00
senandGitHub 2ee2cf9cdf Merge pull request #2682 from senhuang42/armbuild_fixtest 
Make GH Actions CI tests run apt-get update before apt-get install
 2021-05-26 14:14:43 -04:00
senhuang42 d278bede33 Update apt-get prior to tests that install packages 2021-05-26 13:38:36 -04:00
senandGitHub 414bcf239b Merge pull request #2675 from senhuang42/ci_overhaul 
[CI][1/2] Re-do the github actions workflows, migrate various travis and appveyor tests.
 2021-05-26 13:35:53 -04:00
senhuang42 939276cd0c Add ldm and block splitter auto-enable to old api 2021-05-24 13:09:32 -04:00
senhuang42 bb0cd722b6 Migrate travis CI tests 2021-05-21 09:59:57 -04:00
senandGitHub d92fef0f0a Merge pull request #2667 from senhuang42/arm_tests_ci 
[CI] Add ARM tests back into CI
 2021-05-17 14:29:20 -04:00
senandGitHub 9fe1072229 Merge pull request #2668 from senhuang42/update_ci_platforms 
[CI] Fix zlib-wrapper test
 2021-05-17 14:29:04 -04:00
senhuang42 5a75417d2b [ci] Add ARM tests back into CI 2021-05-17 13:10:57 -04:00
senhuang42 38ffe9658e [ci] Use *-latest for platforms to test on 2021-05-17 13:04:55 -04:00
senandGitHub 06718087f8 Remove deprecate flag for vcx (#2647) 2021-05-13 19:33:48 -04:00
senandGitHub 40def70387 Add source level deprecation warning disabling to certain tests/utils (#2645) 2021-05-13 14:41:21 -04:00
senandGitHub a51e342ed1 Update CHANGELOG to include patch from fix (#2642) 2021-05-12 16:46:05 -04:00
senandGitHub c730b8c5a3 Remove const data members in threadpooltest payload (#2639) (#2640) 2021-05-12 16:09:48 -04:00
senandGitHub b35c250bf3 Remove const data members in threadpooltest payload (#2639) 2021-05-12 12:56:57 -04:00
senandGitHub 01fe4796fb Add mt lib build to CL, shuffle around bugs section (#2638) 2021-05-12 11:31:31 -04:00
senandGitHub 9c23ea9e2b Bump version to 1.5.0, rebuild documentation (#2634) 2021-05-11 16:32:09 -04:00
senandGitHub 13449d7ce1 Add PHONY targets to makefiles (#2629) 2021-05-07 14:03:19 -04:00
senandGitHub 9e94b7cac5 Assert no divison by 0, correct superblocks 0 sequences case (#2592) 2021-05-07 13:26:56 -04:00
senandGitHub d8d6e48a0a Add threadPool unit tests to fuzzer.c (#2604) 2021-05-07 11:13:44 -04:00
senandGitHub 91465e23b2 [1.5.0] Enable multithreading in lib build by default (#2584) 
* Update lib Makefile to have new targets

* Update lib/README.md for mt
 2021-05-07 11:13:30 -04:00
senandGitHub 698f261b35 [1.5.0] Deprecate some functions (#2582) 
* Add deprecated macro to zstd.h, mark certain functions as deprecated

* Remove ZSTD_compress.c dependencies on deprecated functions
 2021-05-06 17:59:32 -04:00
senandGitHub 6030cdfede Add --progress flag (#2595) 2021-05-06 14:50:28 -04:00
senandGitHub d6be7659b0 Add seekable roundtrip fuzzer (#2617) 2021-05-06 10:08:21 -04:00
Sen Huang e6c8a5dd40 Fix incorrect usages of repIndex across all strategies 2021-05-04 19:50:55 -04:00
senandGitHub cc31bb8b66 Merge pull request #2598 from senhuang42/reduce_index_rowhash_fix 
Fix chaintable check to include rowhash in ZSTD_reduceIndex()
 2021-05-03 17:34:39 -04:00
senandGitHub 4c5cc345fb Merge pull request #2581 from senhuang42/lcm_stable 
[1.5.0] Promote ZSTD_c_literalCompressionMode to stable params
 2021-05-03 11:59:19 -04:00
senandGitHub cdc979ddb3 Merge pull request #2580 from senhuang42/defaultclevel_to_stable 
[1.5.0] Promote ZSTD_defaultCLevel() into stable API
 2021-05-03 11:59:05 -04:00
senhuang42 61fe571af6 Fix chaintable check to include rowhash in ZSTD_reduceIndex() 2021-04-30 19:52:04 -04:00
senandGitHub f8afc66573 Merge pull request #2588 from senhuang42/update_regressiontest 
[regressiontest] Update results.csv
 2021-04-27 12:40:07 -04:00
senhuang42 33abda4400 Update results.csv 2021-04-26 15:55:23 -04:00
senandGitHub 3e2fbfd056 Merge pull request #2579 from senhuang42/getcdictID_to_stable 
[1.5.0] Promote ZSTD_getDictID_fromCDict() into stable API
 2021-04-26 09:55:43 -04:00
Sen Huang 3c595a4a79 Add literalCompressionMode to stable cParams 2021-04-26 09:55:06 -04:00
senandGitHub 14c11c7459 Merge pull request #2586 from senhuang42/dds_fuzz 
Add DDS to oss fuzzer
 2021-04-24 12:39:16 -04:00
senhuang42 f80dec66b0 Add DDS to oss fuzzer 2021-04-22 18:21:43 -04:00
senandGitHub 3eb3845898 Merge pull request #2583 from senhuang42/remove_zbuff 
[1.5.0] Remove ZBUFF
 2021-04-20 17:05:32 -04:00
senhuang42 a423305e7b Remove ZBUFF tests 2021-04-19 17:27:05 -04:00
senhuang42 3b98987496 Remove building of ZBUFF/deprecated folder by default 2021-04-19 17:12:00 -04:00
Sen Huang c5869677d9 Moved ZSTD_defaultCLevel() into stable API 2021-04-16 10:15:40 -07:00
Sen Huang 9c1ca3c00b Moved ZSTD_getDictID_fromCDict() into stable API 2021-04-16 10:14:29 -07:00
senandGitHub 12c045f74d Merge pull request #2574 from senhuang42/repcode_mismatch_detector_fix 
Correct the block splitter mismatched repcodes detection.
 2021-04-12 23:27:43 -04:00
senandGitHub ebd41ebe56 Merge pull request #2572 from senhuang42/row_hash_hashcache_one_off_error_fix 
Adjust nb elements to prefetch in ZSTD_row_fillHashCache()
 2021-04-12 15:38:53 -04:00
Sen Huang 8844f93957 Adjust nb elements to prefetch in ZSTD_row_fillHashCache() 2021-04-12 14:24:58 -04:00
Sen Huang 550f76f131 Correct the detection of mismatched repcodes 2021-04-09 09:08:51 -07:00
senandGitHub 56421f34e4 Merge pull request #2494 from senhuang42/row_hash2 
SIMD Row Based Matchfinder 🚀
 2021-04-08 10:55:14 -04:00
Sen Huang 4d63d6e8aa Update results.csv, add Row hash to regression test 2021-04-07 10:31:41 -07:00
senandGitHub f71aabb5b5 Move clevel override to after initLocalDict() (#2571) 2021-04-06 21:05:37 -04:00
senandGitHub f1e8b565c2 Maintain two repcode histories for block splitting, replace invalid repcodes (#2569) 2021-04-06 17:25:55 -04:00
senandGitHub e38124555e Fix dictionary force reloading clevel selection (#2570) 
* Move cdict clevel override to before localdict init

* Update results.csv after dict load changes
 2021-04-06 15:35:09 -04:00
senandGitHub 980f3bbf83 [cwksp] Align all allocated "tables" and "aligneds" to 64 bytes (#2546) 
* Perform 64-byte alignment of wksp tables and aligneds internally

* Clean up cwskp_finalize() function to only do two allocs

* Refactor aligned/buffer reservation code, remove ASAN req for alignment reservations

* Change from allocating 128 bytes always to allocating only buffer space as needed for tables/aligned

* Back out aligned/table reservation order restriction

* Add stricter bounds for new/resized wksps, fix comment in zstd_cwksp.h
 2021-04-01 20:07:19 -04:00
senandGitHub 255925c231 Fix repcode-related OSS-fuzz issues in block splitter (#2560) 
* Do not emit last partitions of blocks as RLE/uncompressed

* Fix repcode updates within block splitter

* Add a entropytables confirm function, redo ZSTD_confirmRepcodesAndEntropyTables() for better function signature

* Add a repcode updater to block splitter, no longer need to force emit compressed blocks
 2021-03-31 15:14:59 -04:00
senandGitHub 84ccb81e7c Merge pull request #2561 from senhuang42/longlength_enum 
Add enum for representing long length ID
 2021-03-26 15:55:12 -04:00
Sen Huang b1a43455f8 Add enum for representing long length ID 2021-03-26 10:41:09 -07:00
senandGitHub ab216bc2c5 Merge pull request #2559 from senhuang42/add_dict_regression_tests_backup 
Add different dict modes to compression ratio regression test, update results.csv
 2021-03-25 19:26:06 -04:00
Sen Huang bbbd578f45 Update results.csv 2021-03-25 11:16:37 -07:00
senandGitHub 4fe2e7ae14 Merge pull request #2558 from senhuang42/msan_block_splitter_fix 
Fix block splitter minor MSAN warning.
 2021-03-25 13:51:43 -04:00
Sen Huang f27e326456 Restrict dictmode regression tests only to advanced API, fix some compiler warnings 2021-03-25 10:39:08 -07:00
Sen Huang 1cadf86b39 Add tests to regression tests for dict 2021-03-25 10:39:08 -07:00
senandGitHub b0407b9f0e Merge pull request #2555 from senhuang42/default_clevel_func 
Add ZSTD_defaultCLevel() function to public API
 2021-03-25 13:07:28 -04:00
Sen Huang ef4e26bda5 Add clang msan fuzz test to github actions 2021-03-25 09:11:43 -07:00
Sen Huang 2a907bf4aa Move lastCountSize into a returned struct, fix MSAN error 2021-03-25 09:11:15 -07:00
Sen Huang e398744a35 Add ZSTD_defaultCLevel() function to public API 2021-03-25 08:04:00 -07:00
senandGitHub bf542c8a8d Merge pull request #2447 from senhuang42/block_splitter_v2 
Recursive block splitting
 2021-03-24 12:27:22 -04:00
Sen Huang 5b566ebe08 Rename *compressSequences*() functions for clarity 2021-03-24 08:21:29 -07:00
Sen Huang 0ef1f935b7 Add a fallback in case the total blocksize of split blocks exceeds raw block size 2021-03-24 08:21:29 -07:00
Sen Huang c90e81a692 Enable block splitter by default when applicable 2021-03-24 08:21:29 -07:00
Sen Huang e34332834a Clean up various functions, add debuglogging for estimate vs. actual sizes 2021-03-24 08:21:29 -07:00
Sen Huang 41c3eae6d9 Fix various fuzzer failures: repcode history, superblocks 2021-03-24 08:21:29 -07:00
senhuang42 0633bf17c3 Change 1.3.4 bugfix to be cross-compatible with superblocks and normal compression 2021-03-24 08:21:29 -07:00
senhuang42 eb1ee8686d Refactor buildSequencesStatistics() to avoid pointer increment for superblocks 2021-03-24 08:21:29 -07:00
senhuang42 e2bb215117 Add unit tests and fuzzer param 2021-03-24 08:21:09 -07:00
senhuang42 de52de1347 Add recursive block split algorithm 2021-03-24 08:21:09 -07:00
senhuang42 f06f6626ed Update function names for consistency 2021-03-24 08:20:54 -07:00
senhuang42 c56d6e49e8 Add block splitter to experimental params 2021-03-24 08:20:54 -07:00
senhuang42 2949a95224 Refactor block compression logic into single function 2021-03-24 08:20:54 -07:00
senhuang42 c05c090cc2 Centralize entropy statistics calculations to zstd_compress.c 2021-03-24 08:20:29 -07:00
senandGitHub c48889f097 Merge pull request #2538 from senhuang42/monotonicity_test 
Add memory monotonicity test over srcSize
 2021-03-22 16:54:34 -04:00
Sen Huang dff4a0e867 Make ZSTD_estimateCCtxSize_internal() loop through all srcSize parameter sets as well 2021-03-21 16:15:31 -07:00
senandGitHub eace4abc25 Merge pull request #2540 from senhuang42/fix_dds_supported 
Fix dedicated dict search isSupported() requirements.
 2021-03-17 23:25:53 -04:00
Sen Huang 77ae664ba6 Fix ZSTD_dedicatedDictSearch_isSupported() requirements 2021-03-16 17:36:05 -07:00
senhuang42 386111adec Add a nbSeq argument to compressSequences() 
Refactor ZSTD_compressBlock_internal() to do the block header write within and add nbSeq argument to compressSequences()
 2021-03-16 14:04:22 -07:00
senhuang42 98764493cf Move block header write into compressBlock_internal() 2021-03-16 14:04:22 -07:00
senandGitHub 413b3198b0 Merge pull request #2534 from foxeng/fix-seek-descriptor-check 
Fix seek table descriptor check when loading
 2021-03-16 13:09:00 -04:00
Sen Huang b9dd821441 Add mem monotonicity test over srcSize 2021-03-16 08:24:26 -07:00
senandGitHub a3feed8dcd Merge pull request #2517 from PaulBone/num_cores 
Make the number of physical CPU cores detection more robust
 2021-03-08 11:14:34 -05:00
senandGitHub 0388054aab Merge pull request #2516 from senhuang42/seekable_hang_fix 
Seekable hang fix
 2021-03-03 17:30:46 -05:00
Sen Huang 527a20c3cd Fix seekable decompress hanging 2021-03-02 14:30:03 -08:00
senhuang42 444c4650a0 Add newline to end of cli help message 2021-02-17 12:30:42 -05:00
senandGitHub 7e6c53c7fe Merge pull request #2465 from senhuang42/cformat_2021 
[license] Change year to 2021 compression_format.md
 2021-01-11 12:35:34 -05:00
senhuang42 1d6d64afa3 Change year to 2021 for compression format file 2021-01-11 08:53:29 -05:00
senandGitHub 69085db61c Merge pull request #2446 from senhuang42/multiple_ddicts_v3 
[RFC] Support references to multiple DDicts
 2021-01-08 16:49:45 -05:00
senhuang42 9ae0dd9336 Fix Visual and staticanalyze warnings 2021-01-07 17:58:37 -05:00
senhuang42 4f7584e7a3 Allow freestanding lib script regex to detect XXH64( 2021-01-07 12:29:12 -05:00