Commit Graph
3798 Commits
Author SHA1 Message Date
Nick Terrell 1ffa80a09e [easy] Rewrite rowHashLog computation
`ZSTD_highbit32(1u << x) == x` when it isn't undefined behavior.
2021-05-04 11:43:20 -07:00
Nick TerrellandGitHub a8ecf4ff88 Merge pull request #2597 from terrelln/public-headers
[1.5.0] Move `zstd_errors.h` and `zdict.h` to `lib/` root
2021-05-04 11:28:41 -07:00
W. Felix Handte ee122baacf Detect Presence of md5 on Darwin
This fixes #2568.
2021-05-04 12:33:19 -04:00
Nick Terrell 34aff7ea06 Bug fix & run overflow correction much more frequently in tests
* Fix overflow correction when `windowLog < cycleLog`. Previously, we
  got the correction wrong in this case, and our chain tables and binary
  trees would be corrupted. Now, we work as long as `maxDist` is a power
  of two, by adding `MAX(maxDist, cycleSize)` to our indices.
* When `ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY` is defined to non-zero
  run overflow correction as frequently as allowed without impacting
  compression ratio.
* Enable `ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY` in `fuzzer` and
  `zstreamtest` as well as all the OSS-Fuzz fuzzers. This has a 5-10%
  speed penalty at most, which seems reasonable.
2021-05-03 15:21:47 -07:00
senandGitHub cc31bb8b66 Merge pull request #2598 from senhuang42/reduce_index_rowhash_fix
Fix chaintable check to include rowhash in ZSTD_reduceIndex()
2021-05-03 17:34:39 -04:00
senandGitHub 4c5cc345fb Merge pull request #2581 from senhuang42/lcm_stable
[1.5.0] Promote ZSTD_c_literalCompressionMode to stable params
2021-05-03 11:59:19 -04:00
senandGitHub cdc979ddb3 Merge pull request #2580 from senhuang42/defaultclevel_to_stable
[1.5.0] Promote ZSTD_defaultCLevel() into stable API
2021-05-03 11:59:05 -04:00
senhuang42 61fe571af6 Fix chaintable check to include rowhash in ZSTD_reduceIndex() 2021-04-30 19:52:04 -04:00
Nick Terrell 09149beaf8 [1.5.0] Move zstd_errors.h and zdict.h to lib/ root
`zstd_errors.h` and `zdict.h` are public headers, so they deserve to be
in the root `lib/` directory with `zstd.h`, not mixed in with our private
headers.
2021-04-30 15:13:54 -07:00
Nick Terrell 6cee3c2c4f [trace] Remove default definitions of weak symbols
Instead of providing a default no-op implementation, check the symbols
for `NULL` before accessing them. Providing a default implementation
doesn't reliably work with dynamic linking. Depending on link order the
default implementations may not be overridden. By skipping the default
implementation, all link order issues are resolved. If the symbols
aren't provided the weak function will be `NULL`.
2021-04-26 16:05:39 -07:00
senandGitHub 3e2fbfd056 Merge pull request #2579 from senhuang42/getcdictID_to_stable
[1.5.0] Promote ZSTD_getDictID_fromCDict() into stable API
2021-04-26 09:55:43 -04:00
Sen Huang 3c595a4a79 Add literalCompressionMode to stable cParams 2021-04-26 09:55:06 -04:00
felixhandteandsenhuang42 efa6dfa729 Apply DDS adjustments to avoid assert failures 2021-04-23 16:41:00 -04:00
senhuang42 3b98987496 Remove building of ZBUFF/deprecated folder by default 2021-04-19 17:12:00 -04:00
Sen Huang c5869677d9 Moved ZSTD_defaultCLevel() into stable API 2021-04-16 10:15:40 -07:00
Sen Huang 9c1ca3c00b Moved ZSTD_getDictID_fromCDict() into stable API 2021-04-16 10:14:29 -07:00
senandGitHub 12c045f74d Merge pull request #2574 from senhuang42/repcode_mismatch_detector_fix
Correct the block splitter mismatched repcodes detection.
2021-04-12 23:27:43 -04:00
Sen Huang 8844f93957 Adjust nb elements to prefetch in ZSTD_row_fillHashCache() 2021-04-12 14:24:58 -04:00
Sen Huang 550f76f131 Correct the detection of mismatched repcodes 2021-04-09 09:08:51 -07:00
Sen Huang 4d63d6e8aa Update results.csv, add Row hash to regression test 2021-04-07 10:31:41 -07:00
Nick TerrellandSen Huang 4694423c4f Add and integrate lazy row hash strategy 2021-04-07 09:53:34 -07:00
senandGitHub f71aabb5b5 Move clevel override to after initLocalDict() (#2571) 2021-04-06 21:05:37 -04:00
senandGitHub f1e8b565c2 Maintain two repcode histories for block splitting, replace invalid repcodes (#2569) 2021-04-06 17:25:55 -04:00
senandGitHub e38124555e Fix dictionary force reloading clevel selection (#2570)
* Move cdict clevel override to before localdict init

* Update results.csv after dict load changes
2021-04-06 15:35:09 -04:00
Nick TerrellandGitHub 8383fc828d Merge pull request #2541 from ihsinme/patch-1
simple fix for using bit operator.
2021-04-02 13:01:09 -07:00
senandGitHub 980f3bbf83 [cwksp] Align all allocated "tables" and "aligneds" to 64 bytes (#2546)
* Perform 64-byte alignment of wksp tables and aligneds internally

* Clean up cwskp_finalize() function to only do two allocs

* Refactor aligned/buffer reservation code, remove ASAN req for alignment reservations

* Change from allocating 128 bytes always to allocating only buffer space as needed for tables/aligned

* Back out aligned/table reservation order restriction

* Add stricter bounds for new/resized wksps, fix comment in zstd_cwksp.h
2021-04-01 20:07:19 -04:00
senandGitHub 255925c231 Fix repcode-related OSS-fuzz issues in block splitter (#2560)
* Do not emit last partitions of blocks as RLE/uncompressed

* Fix repcode updates within block splitter

* Add a entropytables confirm function, redo ZSTD_confirmRepcodesAndEntropyTables() for better function signature

* Add a repcode updater to block splitter, no longer need to force emit compressed blocks
2021-03-31 15:14:59 -04:00
Nick TerrellandNick Terrell a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
senandGitHub 84ccb81e7c Merge pull request #2561 from senhuang42/longlength_enum
Add enum for representing long length ID
2021-03-26 15:55:12 -04:00
Sen Huang b1a43455f8 Add enum for representing long length ID 2021-03-26 10:41:09 -07:00
senandGitHub 4fe2e7ae14 Merge pull request #2558 from senhuang42/msan_block_splitter_fix
Fix block splitter minor MSAN warning.
2021-03-25 13:51:43 -04:00
senandGitHub b0407b9f0e Merge pull request #2555 from senhuang42/default_clevel_func
Add ZSTD_defaultCLevel() function to public API
2021-03-25 13:07:28 -04:00
Sen Huang 2a907bf4aa Move lastCountSize into a returned struct, fix MSAN error 2021-03-25 09:11:15 -07:00
Sen Huang e398744a35 Add ZSTD_defaultCLevel() function to public API 2021-03-25 08:04:00 -07:00
Nick TerrellandGitHub f8ac0ea7ef Merge pull request #2539 from terrelln/linux-kernel-fixes
Fixes for the next linux kernel patch version
2021-03-24 10:34:29 -07:00
senandGitHub bf542c8a8d Merge pull request #2447 from senhuang42/block_splitter_v2
Recursive block splitting
2021-03-24 12:27:22 -04:00
Sen Huang 5b566ebe08 Rename *compressSequences*() functions for clarity 2021-03-24 08:21:29 -07:00
Sen Huang 0ef1f935b7 Add a fallback in case the total blocksize of split blocks exceeds raw block size 2021-03-24 08:21:29 -07:00
Sen Huang c90e81a692 Enable block splitter by default when applicable 2021-03-24 08:21:29 -07:00
Sen Huang e34332834a Clean up various functions, add debuglogging for estimate vs. actual sizes 2021-03-24 08:21:29 -07:00
Sen Huang 41c3eae6d9 Fix various fuzzer failures: repcode history, superblocks 2021-03-24 08:21:29 -07:00
senhuang42 0633bf17c3 Change 1.3.4 bugfix to be cross-compatible with superblocks and normal compression 2021-03-24 08:21:29 -07:00
senhuang42 eb1ee8686d Refactor buildSequencesStatistics() to avoid pointer increment for superblocks 2021-03-24 08:21:29 -07:00
senhuang42 e2bb215117 Add unit tests and fuzzer param 2021-03-24 08:21:09 -07:00
senhuang42 de52de1347 Add recursive block split algorithm 2021-03-24 08:21:09 -07:00
senhuang42 f06f6626ed Update function names for consistency 2021-03-24 08:20:54 -07:00
senhuang42 c56d6e49e8 Add block splitter to experimental params 2021-03-24 08:20:54 -07:00
senhuang42 2949a95224 Refactor block compression logic into single function 2021-03-24 08:20:54 -07:00
senhuang42 c05c090cc2 Centralize entropy statistics calculations to zstd_compress.c 2021-03-24 08:20:29 -07:00
senandGitHub c48889f097 Merge pull request #2538 from senhuang42/monotonicity_test
Add memory monotonicity test over srcSize
2021-03-22 16:54:34 -04:00