ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,056 Commits 17 Branches 77 Tags
b25d57461e9778e9002fdfffc41eefdf6911aac7
Commit Graph
9056 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick TerrellandGitHub b25d57461e Merge pull request #2794 from terrelln/spdx-license 
[contrib][linux] Fix up SPDX license identifiers
 2021-09-22 17:33:31 -07:00
Yann ColletandGitHub 441eb2654b Merge pull request #2793 from facebook/rdoc 
Minor documentation update
 2021-09-22 17:07:49 -07:00
Nick Terrell 4d347a902b [contrib][linux] Fix up SPDX license identifiers 
Correctly identify that we are GPL v2+ or BSD 3 clause, as pointed out
in issue #2663.
 2021-09-22 15:09:42 -07:00
Yann Collet 8150891939 regenerated zstdless.1 2021-09-22 14:48:51 -07:00
Yann Collet 3addf2f277 updated zstdgrep man page 2021-09-22 14:30:59 -07:00
Yann Collet 70b36c2308 update zstdgrep doc 
to mention ripgrep alternative which transparently supports zstd-compressed files.
 2021-09-22 14:30:06 -07:00
Yann Collet 999f8778af updated man pages 
using ronn-ng
 2021-09-22 14:18:24 -07:00
Yann Collet 2832bbbbdc emphasize usage of -r in documentation 
notably as a way to overcome shell expansion limitations,
notably in a scenario of dictionary training.
 2021-09-22 14:00:20 -07:00
senandGitHub 044c8b4722 Merge pull request #2779 from senhuang42/fse_fix 
Fix NCountWriteBound
 2021-09-22 13:51:21 -04:00
senandGitHub 1e99d36361 Merge pull request #2788 from senhuang42/param_switch 
Use new paramSwitch enum for row matchfinder and block splitter
 2021-09-22 13:27:55 -04:00
Nick TerrellandGitHub 5208e14ae0 Merge pull request #2791 from terrelln/huf-asm-fixes 
[huf] Fix compilation when DYNAMIC_BMI2=0 && BMI2 is supported
 2021-09-22 09:59:02 -07:00
senhuang42 99b5e7b8c2 Add test case for FSE over-write 2021-09-22 12:03:46 -04:00
Nick Terrell 9450876a9d [huf] Fix compilation when DYNAMIC_BMI2=0 && BMI2 is supported 
* Fix compilation issues pointed out in PR #2790.
* Add test cases to GitHub actions that test all combinations of
  `DYNAMIC_BMI2` BMI2 support.
 2021-09-21 16:49:13 -07:00
senhuang42 06f42c3bfd Use new paramSwitch enum for LDM 2021-09-21 14:22:09 -04:00
senhuang42 b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
Nick TerrellandGitHub 8385355fac Merge pull request #2722 from terrelln/huf-asm 
Assembly implementation of 4X1 & 4X2 Huffman
 2021-09-20 17:32:30 -07:00
Nick Terrell a5f2c45528 Huffman ASM 2021-09-20 14:46:43 -07:00
Nick TerrellandGitHub 51b123d5f7 Merge pull request #2784 from terrelln/huf-asm-fuzzer 
[fuzzer] Add Huffman decompression fuzzer
 2021-09-20 11:57:27 -07:00
Yann ColletandGitHub 847dcb65bf Merge pull request #2787 from animalize/msvc2019 
add msvc2019 to build.generic.cmd
 2021-09-20 11:14:51 -07:00
Ma Lin 4eef208889 add msvc2019 to build.generic.cmd 2021-09-19 10:28:16 +08:00
Nick Terrell d7542aacd9 [fuzzer] Add huf_decompress fuzzer 
Add a fuzzer for Huffman decompression. Fix several bugs in Huffman
decompression, mostly related to `op == NULL` and pointer underflow.
 2021-09-17 15:00:49 -07:00
Nick TerrellandGitHub 6392c382bf Merge pull request #2783 from terrelln/huf-asm-makefiles 
[build] Add support for ASM files in Make + CMake
 2021-09-17 14:56:47 -07:00
Nick Terrell 8bf699aa59 [build] Add support for ASM files in Make + CMake 
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
  Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
  same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
  `make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.

The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.

The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
 2021-09-17 14:13:53 -07:00
senandGitHub 9d2a45a705 Merge pull request #2778 from senhuang42/opt_inlining_revert 
Revert opt outlining change
 2021-09-15 14:22:10 -04:00
Sen Huang a7aa2c5df6 Fix NCountWriteBound 2021-09-15 09:51:42 -07:00
Sen Huang bd84e4a9d3 Revert opt outlining change 2021-09-15 09:08:41 -07:00
Nick TerrellandGitHub 2fabd370bb Merge pull request #2777 from terrelln/oss-fuzz-fix 
[rsyncable] Fix test failures
 2021-09-14 13:20:22 -07:00
Nick Terrell 9d9e2ed00b [rsyncable] Fix test failures 
Test failures showed up on the daily cron job. They didn't show up
in CI because the condition is somewhat rare, and didn't trigger
during the CI tests.

This PR fixes up the logic in `findSynchronizationPoint()` to correctly
handle the edge case. It also un-comments an assert that helps catch the
issue, and verify that rsyncable mode is calculating the correct hash.

After the fix, the test that failed passes:

```
./zstreamtest --newapi -t1 --no-big-tests -s9680
```
 2021-09-14 12:28:53 -07:00
Yann ColletandGitHub 2e6f5bc0d8 Merge pull request #2771 from facebook/opt_investigation 
Improve optimal parser performance on small data
 2021-09-14 10:36:34 -07:00
Nick TerrellandGitHub d22bbed5db Merge pull request #2776 from terrelln/oss-fuzz-fix 
[rsyncable] Ensure ZSTD_compressBound() is respected
 2021-09-14 09:37:43 -07:00
Yann Collet fd94b9d1c9 Merge branch 'dev' into opt_investigation 2021-09-14 01:15:51 -07:00
senandGitHub 29f595ef76 Merge pull request #2773 from senhuang42/huffman_bugix 
Fix dictionary training huffman segfault and small speed improvement
 2021-09-13 21:02:42 -04:00
Nick Terrell a418b4e478 [rsyncable] Ensure ZSTD_compressBound() is respected 
In degenerate cases `--rsyncable` could create very small blocks (1
byte). This causes the compressed output to be larger than
`ZSTD_compressBound()`. Fix the issue by ensuring that rsyncable mode
never outputs blocks smaller than 128 KB.

The minimum job size is 512 KB, so we shouldn't lose many
synchronization points from skipping any that cause blocks smaller than
128 KB. And even if we do, that is fine, because we'll find the next
one.

This fixes the `raw_dictionary_round_trip` oss-fuzz assert.

Credit to OSS-Fuzz
 2021-09-13 17:14:07 -07:00
Sen Huang d45d0ad9d8 Update regression test 2021-09-13 12:41:02 -04:00
Sen Huang 4a498fb9c3 Add a dictionary training large corpus test 2021-09-13 12:29:17 -04:00
Sen Huang 1daf3c8dbc Use 32 buckets for log2 bucketing in huffman sort 2021-09-13 12:29:16 -04:00
Yann Collet b6b2855b80 updated regression tests 2021-09-12 10:22:35 -07:00
Yann Collet f58e63bee7 Merge branch 'dev' into opt_investigation 2021-09-12 01:42:49 -07:00
Yann Collet 640c5b1f77 fix automated_benchmarking 
make it able to process text output sent into either stdout or stderr
 2021-09-12 01:36:18 -07:00
Yann ColletandGitHub c10067c44e Merge pull request #2775 from eli-schwartz/meson 
meson: fix type error for integer option
 2021-09-10 05:47:52 -07:00
Eli Schwartz 193aa49673 meson: fix type error for integer option 
meson forgave using the wrong type, but this isn't guaranteed. muon
simply failed.
 2021-09-09 23:40:58 -04:00
Felix HandteandGitHub d68aa19a2f Merge pull request #2749 from felixhandte/zstd-fast-pipelined 
Pipelined Implementation of ZSTD_fast (~+5% Speed)
 2021-09-09 17:05:30 -04:00
Yann Collet 5449ede2e6 make automated-benchmarking faster 
by employing parallel compilation of object files.
 2021-09-08 15:12:28 -07:00
Yann Collet b7f46ebc23 use ZSTD_memcpy() for better portability 
notably within kernel space
 2021-09-08 14:45:53 -07:00
Yann Collet 4f0b1b9ee5 update regression tests 2021-09-08 14:37:42 -07:00
Yann Collet 7fce9a41b5 change update rate to 12/11/11/11 
better for large files, and sources with relatively "stable" entropy,
like silesia.tar.
slightly worse for files with rapidly changing entropy,
like Calgary.tar/.

Updated small files tests in fuzzer
 2021-09-08 14:05:57 -07:00
Yann Collet ef78611c26 change update rate to 11/10/10/10 
better for larger blocks,
very small inefficiency on small block.
 2021-09-08 08:58:28 -07:00
Yann Collet 42a3ed752a removed frequency booster for stat initialization of btultra2 
used to be necessary to counter-balance the fixed-weight frequency update
which has been recently changed for an adaptive rate (targeting stable starting frequency stats).
 2021-09-08 07:56:43 -07:00
Yann Collet b096a5c626 updated regression tests 2021-09-07 09:55:14 -07:00
senandGitHub bcc68275f1 Merge pull request #2769 from senhuang42/typo_fix 
[easy] Fix patch-from help msg typo
 2021-09-07 11:09:55 -04:00