ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,902 Commits 17 Branches 77 Tags
0152435ab0abdff45cb25aa6c8f3fc60d81ebcda
Commit Graph
8902 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
binhdvoandGitHub 0152435ab0 Merge pull request #2708 from binhdvo/skippable 
Add API for fetching skippable frame content
 2021-06-14 19:00:31 -04:00
Binh Vo 9d9f7680f8 Add API for fetching skippable frame content 2021-06-14 16:01:28 -04:00
Nick TerrellandGitHub 6917c4ecdf Merge pull request #2709 from terrelln/bounds-check-fix 
[fix] Add missing bounds checks during compression
 2021-06-14 12:33:50 -07:00
Nick Terrell 05b6773fbc [fix] Add missing bounds checks during compression 
* The block splitter missed a bounds check, so when the buffer is too small it
  passes an erroneously large size to `ZSTD_entropyCompressSeqStore()`, which
  can then write the compressed data past the end of the buffer. This is a new
  regression in v1.5.0 when the block splitter is enabled. It is either enabled
  explicitly, or implicitly when using the optimal parser and `ZSTD_compress2()`
  or `ZSTD_compressStream*()`.
* `HUF_writeCTable_wksp()` omits a bounds check when calling
  `HUF_compressWeights()`. If it is called with `dstCapacity == 0` it will pass
  an erroneously large size to `HUF_compressWeights()`, which can then write
  past the end of the buffer. This bug has been present for ages. However, I
  believe that zstd cannot trigger the bug, because it never calls
  `HUF_compress*()` with `dstCapacity == 0` because of [this check][1].

Credit to: Oss-Fuzz

[1]: https://github.com/facebook/zstd/blob/89127e5ee2f3c1e141668fa6d4ee91245f05d132/lib/compress/zstd_compress_literals.c#L100
 2021-06-14 11:35:33 -07:00
binhdvoandGitHub 89127e5ee2 Merge pull request #2705 from binhdvo/bootcamp 
Add support for negative values in advanced flags
 2021-06-11 14:08:23 -04:00
Binh Vo 6fad35c6a1 Add support for negative levels in --adapt=min and --adapt=max" 2021-06-11 12:13:09 -04:00
Felix HandteandGitHub 67a25968f0 Merge pull request #2702 from felixhandte/human_size_output 
Format File Sizes Human-Readable in the CLI
 2021-06-10 16:53:43 -04:00
W. Felix Handte 8c00807bbc Whitespace Fixes to Improve Cross-Line Alignment 2021-06-10 13:28:38 -04:00
W. Felix Handte 94cf57bb13 Update Tests to Reflect New Formatting 2021-06-10 13:14:18 -04:00
W. Felix Handte 87e94e3e39 Convert Other Size Displays to Use Human-Readable Formatting 2021-06-10 12:57:46 -04:00
W. Felix Handte 2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB") 
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
 2021-06-10 12:53:07 -04:00
W. Felix Handte 9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte bc46b6efe4 Apply to Other Print Statement as Well 2021-06-10 12:53:07 -04:00
W. Felix Handte 7e0058848c Fix Whitespace 2021-06-10 12:53:07 -04:00
W. Felix Handte 93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte 464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte 9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting 
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 8e0a9695d7 Attempt to fix a failing test with help from @aqrit 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 1eb852854b Some fixes to address things @felixhandte found 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 376a2730a8 Try enabling the BIG strings now the unsigned long long is in effect 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 20b9b00b41 Try unsigned long long 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 64385ef7cb Update humanSize() to skip the big numbers (it requires 64 bit) 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 1ef6f3d079 Use unsigned long instead to help with some tests 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte e5fc830795 human_size() should use size_t 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 35576e63ce Convert tabs to spaces 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 77001f00fb Use human_size() on the "multiple files compressed" output also 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 894698d3b6 Use human_size() in the benchmark output also 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 4e0d9f1cc8 Move the variable declarations to the top 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte eefdbcd93a Make the variable types match 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte b6b23dfe64 Convert names to CamelCase 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte b70175e5ec Put the human_size() function in util.c 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 26fab1d963 Make the CLI output the file sizes in human readable format 2021-06-10 12:53:07 -04:00
senandGitHub d5f3568c4b Merge pull request #2697 from senhuang42/entropy_repeat_fix 
[bug] Fix entropy repeat mode bug
 2021-06-10 16:39:17 +03:00
Yann ColletandGitHub 2962583492 Merge pull request #2704 from facebook/revert-2703-bootcamp 
Revert "Add support for --long-param flag, fix #2104"
 2021-06-09 13:03:28 -07:00
binhdvoandGitHub 325952f878 Revert "Add support for --long-param flag, fix #2104" 2021-06-09 15:35:43 -04:00
binhdvoandGitHub 78e16b15f1 Merge pull request #2703 from binhdvo/bootcamp 
Add support for --long-param flag, fix #2104
 2021-06-09 15:03:38 -04:00
Binh Vo 6583fa3f0a Add support for --long-param flag 2021-06-09 14:07:52 -04:00
Yann ColletandGitHub 05d70903a6 Merge pull request #2698 from binhdvo/bootcamp 
Fix --progress flag to properly control progress display and default …
 2021-06-09 09:56:01 -07:00
senandGitHub a21b9036fe Merge pull request #2678 from senhuang42/big_endian_no_intrinsics_fuzztest 
Fuzzer test with no intrinsics on S390x (big endian)
 2021-06-09 18:46:52 +03:00
senandGitHub dd33ec9db0 Merge pull request #2700 from gauthamkrishna9991/update_readme_travis_link 
Update README for Travis CI Badge
 2021-06-09 10:41:35 +03:00
Goutham KrishnaandGitHub 912bb9fbf3 Update README for Travis CI Badge 
### Updating Badge link to the newTravis CI link.
- Update badge root to `api.travis-ci.com` (new)
  from `travis-ci.org` (old), which was migrated.
 2021-06-09 12:38:22 +05:30
aqritandGitHub dd4f6aa9e6 Flatten ZSTD_row_getMatchMask (#2681) 
* Flatten ZSTD_row_getMatchMask

* Remove the SIMD abstraction layer.
* Add big endian support.
* Align `hashTags` within `tagRow` to a 16-byte boundary. 
* Switch SSE2 to use aligned reads.
* Optimize scalar path using SWAR.
* Optimize neon path for `n == 32`
* Work around minor clang issue for NEON (https://bugs.llvm.org/show_bug.cgi?id=49577)

* replace memcpy with MEM_readST

* silence alignment warnings

* fix neon casts

* Update zstd_lazy.c

* unify simd preprocessor detection (#3)

* remove duplicate asserts

* tweak rotates

* improve endian detection

* add cast

there is a fun little catch-22 with gcc: result from pmovmskb has to be cast to uint32_t to avoid a zero-extension
but must be uint16_t to get gcc to generate a rotate instruction..

* more casts

* fix casts

better work-around for the (bogus) warning: unary minus on unsigned
 2021-06-09 08:50:25 +03:00
Binh Vo d2f31b6627 Fix --progress flag to properly control progress display and default progress display on when using -v 2021-06-08 17:24:38 -04:00
Felix HandteandGitHub 8a3bdfaa7b Merge pull request #2654 from wolfpld/dev 
Initialize "potentially uninitialized" pointers.
 2021-06-07 13:04:19 -04:00
senandGitHub 5fb3884f33 Merge pull request #2691 from senhuang42/per_pr_regressiontest 
Make regression test run on every PR
 2021-06-07 12:44:31 +03:00
Sen Huang 2ff5c7b59f Add no intrinsics fuzztest, rowhash compression size test, and S390X to travis 2021-06-07 00:54:53 -07:00
senhuang42 88acf0ac65 Make regression test run on every PR 2021-06-07 00:47:56 -07:00
Sen Huang 923e5ad3f5 Fix entropy repeat mode bug 2021-06-07 00:32:03 -07:00
Yann ColletandGitHub cefafc0b6e Merge pull request #2693 from binhdvo/bootcamp 
Add documentation for --patch-from
 2021-06-03 12:41:23 -07:00