ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,485 Commits 17 Branches 77 Tags
7b94de78ff2c3634cd7ef32f0e16f4722c03578f
Commit Graph
11485 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
makise-homura 3cd085cec3 Clarify no-tree-vectorize usage for ICC and LCC 2021-07-14 20:00:44 +03:00
makise-homura a5f518ae27 Change zstdcli's main() declaration due to -Wmain on some compilers 2021-07-14 19:55:47 +03:00
makise-homura d4ad02c721 Add support for MCST LCC compiler 2021-07-10 03:57:06 +03:00
Jonathan Ringer e1f85dbca3 pzstd: fix linking for static builds 2021-07-09 13:21:15 -07:00
binhdvoandGitHub b3e372c171 Merge pull request #2717 from binhdvo/bootcamp 
Proactively skip huffman compression based on sampling where non-comp…
 2021-07-01 10:39:58 -04:00
senandGitHub d90bc0e0b6 Merge pull request #2720 from senhuang42/remove_folder 
Remove folder when done with test
 2021-06-30 14:37:21 -04:00
senhuang42 f5f6cc2e48 Remove folder when done with test 2021-06-30 13:09:00 -04:00
Binh Vo dc5b693f1e Proactively skip huffman compression based on sampling where non-compressibility is suspected 2021-06-30 11:02:47 -04:00
Nick TerrellandGitHub 7b43e69b45 Merge pull request #2655 from heitbaum/patch-1 
[trace] remove zstd_trace.c reference from freestanding
 2021-06-29 18:33:40 -07:00
Nick TerrellandGitHub 609be382ac Merge pull request #2719 from danlark1/danlark_iwyu 
Include what you use in zstd_ldm_geartab
 2021-06-29 16:53:10 -07:00
Nick TerrellandGitHub 094b26081f Merge pull request #2689 from danlark1/dev 
Optimize zstd decompression by another x%
 2021-06-29 11:34:36 -07:00
Danila Kutenin e855b78be6 Include what you use in zstd_ldm_geartab 2021-06-29 17:57:53 +01:00
Danila Kutenin 2c2c9e7dfd Add possible improvements for gcc-11 2021-06-29 09:06:47 +01:00
Yann ColletandGitHub 88f107b5b9 Merge pull request #2716 from facebook/fix2701 
remove invalid test
 2021-06-24 10:57:18 -07:00
senandGitHub 45d707e908 Merge pull request #2715 from senhuang42/sequence_api_3 
[RFC] Add internal API for converting ZSTD_Sequence into seqStore
 2021-06-24 13:02:11 -04:00
Yann Collet 139a2f1e0c remove invalid test 
`--mt` is no longer supported by `zstreamtest`
(relevant API entry point has been removed from `libzstd`).

fix #2701
 2021-06-24 09:07:55 -07:00
senhuang42 76466dfadf Add simple API for converting ZSTD_Sequence into seqStore 2021-06-23 12:10:48 -04:00
Érico Nogueira 4d09952701 [lib] Fix libzstd.pc for lib-mt builds 
Add the libzstd.pc target to the lib target in lib/Makefile, which makes
it inherit LDFLAGS_DYNLIB from the lib-mt target. This allows us to add
a Libs.private field to libzstd.pc which gets conditionally populated
with '-pthread'.

The 1.5.0 release notes mention that the static library isn't
multi-threaded by default, due to concern for people building static
binaries with libzstd:

   Now the dynamic library supports multi-threaded compression by
   default.  Note that this property is not extended to the static
   library because doing so would have impacted the build script of
   existing client applications (requiring them to add -pthread to their
   recipe), thus potentially breaking their build.

To get closer to being able to enable multi-threading for all library
builds by default, this commit makes it so that any libzstd consumer
using pkg-config gets the correct flags.

We also fix the indentation of the rule for libzstd.pc and move it
outside the if/endif block for install rules (which uses a list of OSs
where the rules were validated), so the rule is available for all users
of the 'lib*' targets.
 2021-06-19 17:42:24 -03:00
Felix HandteandGitHub 628f65b79c Merge pull request #2714 from luisdallos/build-macros-typos 
lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros
 2021-06-18 16:16:44 -04:00
Usuario 8bdce1ff97 lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros 2021-06-18 10:07:39 -04:00
binhdvoandGitHub b99d9f5ebd Merge pull request #2710 from binhdvo/bootcamp 
Add option to use logical cores for default threads
 2021-06-17 09:06:18 -04:00
Binh Vo 6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
binhdvoandGitHub 0152435ab0 Merge pull request #2708 from binhdvo/skippable 
Add API for fetching skippable frame content
 2021-06-14 19:00:31 -04:00
Binh Vo 9d9f7680f8 Add API for fetching skippable frame content 2021-06-14 16:01:28 -04:00
Nick TerrellandGitHub 6917c4ecdf Merge pull request #2709 from terrelln/bounds-check-fix 
[fix] Add missing bounds checks during compression
 2021-06-14 12:33:50 -07:00
Nick Terrell 05b6773fbc [fix] Add missing bounds checks during compression 
* The block splitter missed a bounds check, so when the buffer is too small it
  passes an erroneously large size to `ZSTD_entropyCompressSeqStore()`, which
  can then write the compressed data past the end of the buffer. This is a new
  regression in v1.5.0 when the block splitter is enabled. It is either enabled
  explicitly, or implicitly when using the optimal parser and `ZSTD_compress2()`
  or `ZSTD_compressStream*()`.
* `HUF_writeCTable_wksp()` omits a bounds check when calling
  `HUF_compressWeights()`. If it is called with `dstCapacity == 0` it will pass
  an erroneously large size to `HUF_compressWeights()`, which can then write
  past the end of the buffer. This bug has been present for ages. However, I
  believe that zstd cannot trigger the bug, because it never calls
  `HUF_compress*()` with `dstCapacity == 0` because of [this check][1].

Credit to: Oss-Fuzz

[1]: https://github.com/facebook/zstd/blob/89127e5ee2f3c1e141668fa6d4ee91245f05d132/lib/compress/zstd_compress_literals.c#L100
 2021-06-14 11:35:33 -07:00
binhdvoandGitHub 89127e5ee2 Merge pull request #2705 from binhdvo/bootcamp 
Add support for negative values in advanced flags
 2021-06-11 14:08:23 -04:00
Binh Vo 6fad35c6a1 Add support for negative levels in --adapt=min and --adapt=max" 2021-06-11 12:13:09 -04:00
koalaandGitHub e00412f6b5 Z_PREFIX zError function 
When a project use zError function,linker can not find z_zError function
 2021-06-11 19:29:27 +08:00
Felix HandteandGitHub 67a25968f0 Merge pull request #2702 from felixhandte/human_size_output 
Format File Sizes Human-Readable in the CLI
 2021-06-10 16:53:43 -04:00
W. Felix Handte 8c00807bbc Whitespace Fixes to Improve Cross-Line Alignment 2021-06-10 13:28:38 -04:00
W. Felix Handte 94cf57bb13 Update Tests to Reflect New Formatting 2021-06-10 13:14:18 -04:00
W. Felix Handte 87e94e3e39 Convert Other Size Displays to Use Human-Readable Formatting 2021-06-10 12:57:46 -04:00
W. Felix Handte 2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB") 
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
 2021-06-10 12:53:07 -04:00
W. Felix Handte 9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte bc46b6efe4 Apply to Other Print Statement as Well 2021-06-10 12:53:07 -04:00
W. Felix Handte 7e0058848c Fix Whitespace 2021-06-10 12:53:07 -04:00
W. Felix Handte 93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte 464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte 9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting 
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 8e0a9695d7 Attempt to fix a failing test with help from @aqrit 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 1eb852854b Some fixes to address things @felixhandte found 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 376a2730a8 Try enabling the BIG strings now the unsigned long long is in effect 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 20b9b00b41 Try unsigned long long 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 64385ef7cb Update humanSize() to skip the big numbers (it requires 64 bit) 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 1ef6f3d079 Use unsigned long instead to help with some tests 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte e5fc830795 human_size() should use size_t 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 35576e63ce Convert tabs to spaces 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 77001f00fb Use human_size() on the "multiple files compressed" output also 2021-06-10 12:53:07 -04:00