ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,964 Commits 17 Branches 77 Tags
1c97ec73d7efc412ae32bee4f88cbcd39ba3f146
Commit Graph
8964 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 1c97ec73d7 added qemu tests 
running zstd library on emulated targets
 2021-08-29 16:28:41 -07:00
senandGitHub 1f3fc1936c Merge pull request #2753 from senhuang42/better_error_msg 
[easy] Fix zstd bench error message
 2021-08-23 20:37:48 -04:00
senhuang42 dce48f53df Fix benchzstd error message 2021-08-23 19:10:16 -04:00
Yann ColletandGitHub be82a0ab8f Merge pull request #2746 from eli-schwartz/meson-fixup 
meson fixups
 2021-08-23 15:57:47 -07:00
Yann ColletandGitHub 18a20b3ad7 Merge pull request #2752 from facebook/hashLog3max 
make ZSTD_HASHLOG3_MAX private
 2021-08-20 12:51:17 -07:00
Yann Collet 2de42174bb make ZSTD_HASHLOG3_MAX private 
This is an implementation detail,
it doesn't belong to public space (zstd.h).
 2021-08-20 09:52:42 -07:00
senandGitHub ae998544de Merge pull request #2750 from senhuang42/sb_compress 
Improve branch misses on FSE symbol spreading
 2021-08-20 12:47:24 -04:00
senhuang42 da095ed899 Improve branch misses on FSE symbol spreading 2021-08-18 10:22:22 -07:00
Eli Schwartz 9748608aeb meson: set the symbol visibility of the shared library to hidden 
This matches the Makefile build. Due to one private xxhash symbol in use
by the program, it recompiles a private copy of xxhash.

Due to the test binaries making extensive (?) use of private symbols, it
doesn't even attempt to link to shared libzstd, and instead, all of the
original object files are added to libtestcommon itself for private
linkage. This, too, matches the Makefile build.

Ref. #2261
 2021-08-09 22:03:17 -04:00
Eli Schwartz d95a3f5097 meson: fix warnings in build files 
meson prefers that project-level options for Wall/Wextra/pedantic be
used, rather than hardcoding raw flags in add_project_arguments. If you
do the latter anyway, it raises a meson warning.

Set the default options for the project to use all this.

Also move the -Werror comment to the project default options with
appropriate format, but leave it commented out since it does not work.
 2021-08-09 20:17:00 -04:00
Eli Schwartz ee3355df30 meson: link tests with a convenience library of sources used by multiple binaries 
This saves repeatedly compiling them, resulting in 16 fewer compile
steps, in exchange for one `ar` step.
 2021-08-09 20:15:44 -04:00
senandGitHub 6a258043f5 Merge pull request #2692 from senhuang42/rebalance_clevel 
[RFC] Rebalance compression levels
 2021-08-06 12:51:31 -04:00
Sen Huang 539b3aab9b Optimize 32-bit VecMask_next() 2021-08-04 17:14:58 -04:00
Felix HandteandGitHub ae131282f5 Merge pull request #2742 from felixhandte/set-mtime 
Set mtime on Output Files
 2021-08-04 16:44:46 -04:00
senhuang42 e411040ea1 Add 64 row entry support for lazy 2021-08-04 16:19:12 -04:00
senhuang42 31820e032c Rebalance clevels for lazy 2021-08-04 16:18:52 -04:00
senandGitHub f9b034086a Merge pull request #2732 from senhuang42/huf_sort_improvement 
[HUF] Improve Huffman sorting algorithm
 2021-08-04 16:07:56 -04:00
W. Felix Handte a8f4612eab Remove sleep()s in Test; Replace with Artificial mtime 
This behavior of `touch` is standardized in POSIX, so it should be available.
 2021-08-04 15:56:46 -04:00
W. Felix Handte 9cd6c1ff4d Update mtime and atime for Written Files 2021-08-04 14:49:56 -04:00
W. Felix Handte a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
senhuang42 aa1957477b Improve Huffman sorting algorithm 2021-08-04 12:43:34 -04:00
W. Felix Handte ead41bcb4e Add Test to Verify mtime is Copied to Destination 2021-08-03 17:22:58 -04:00
Nick TerrellandGitHub 6ee70bae46 Merge pull request #2733 from terrelln/huf-cspeed 
[HUF] Improve Huffman encoding speed
 2021-08-03 12:59:54 -04:00
Nick Terrell d8a0797268 [fuzz] Add Huffman round trip fuzzer 
* Add a Huffman round trip fuzzer
* Fix two minor bugs in Huffman that aren't exposed in zstd
  - Incorrect weight comparison (weights are allowed to be equal to
    table log).
  - HUF_compress1X_usingCTable_internal() can return compressed
    size >= source size, so the assert that `cSize <= 65535` isn't
    correct, and it needs to be checked instead.
 2021-08-03 08:10:06 -07:00
senandGitHub 5c46f62006 Merge pull request #2677 from senhuang42/ci_overhaul_2 
[CI][2/2] Migrate CI tests which (currently) fail
 2021-08-02 09:55:49 -04:00
Sen Huang 5ec7897a26 Fix static analyzer warnings 2021-07-29 09:11:12 -07:00
Sen Huang 43aad811db Add VS2019 test, migrate minimal decompressor macros to GHActions 2021-07-29 08:49:36 -07:00
Nick TerrellandGitHub 32cfe15edc Merge pull request #2724 from jonringer/fix-pzstd-static-build 
pzstd: fix linking for static builds
 2021-07-28 14:53:01 -07:00
Nick TerrellandGitHub d69e007cca Merge pull request #2707 from koalabearguo/dev 
Z_PREFIX  zError function
 2021-07-28 14:52:15 -07:00
Felix HandteandGitHub 495597699c Merge pull request #2729 from felixhandte/fix-ddss-load 
Fix DDSS Load
 2021-07-28 13:12:50 -04:00
Nick Terrell 46f2710562 [HUF] Improve Huffman encoding speed 
Improve Huffman encoding speed by 20% for gcc and 10% for clang.

| Compiler |     Benchmark     | Config  |   Dataset   | Ratio | Speed MB/s (dev) | Speed MB/s (huf-cspeed) | Speed MB/s (huf-cspeed - dev) |
|----------|-------------------|---------|-------------|-------|------------------|-------------------------|-------------------------------|
| gcc      | compress          | level_1 | enwik7      | 2.43  | 253.70           | 258.72                  | 2.0%                          |
| gcc      | compress          | level_1 | silesia     | 2.88  | 341.90           | 348.15                  | 1.8%                          |
| gcc      | compress_literals | level_1 | enwik7      | 1.49  | 761.83           | 912.76                  | 19.8%                         |
| gcc      | compress_literals | level_1 | silesia     | 1.28  | 754.83           | 902.37                  | 19.5%                         |
| gcc      | compress_literals | level_7 | enwik7      | 1.29  | 502.81           | 552.79                  | 9.9%                          |
| gcc      | compress_literals | level_7 | silesia     | 1.11  | 675.97           | 776.44                  | 14.9%                         |
| clang    | compress          | level_1 | enwik7      | 2.43  | 277.54           | 280.98                  | 1.2%                          |
| clang    | compress          | level_1 | silesia     | 2.88  | 369.98           | 375.46                  | 1.5%                          |
| clang    | compress_literals | level_1 | enwik7      | 1.49  | 828.83           | 918.41                  | 10.8%                         |
| clang    | compress_literals | level_1 | silesia     | 1.28  | 815.81           | 905.41                  | 11.0%                         |
| clang    | compress_literals | level_7 | enwik7      | 1.29  | 533.13           | 553.30                  | 3.8%                          |
| clang    | compress_literals | level_7 | silesia     | 1.11  | 714.52           | 775.38                  | 8.5%                          |
 2021-07-27 15:10:35 -07:00
W. Felix Handte da58821ff2 Fix DDSS Load 
This PR fixes an incorrect comparison in figuring out `minChain` in
`ZSTD_dedicatedDictSearch_lazy_loadDictionary()`. This incorrect comparison
had been masked by the fact that `idx` was always 1, until @terrelln changed
that in #2726.

Credit-to: OSS-Fuzz
 2021-07-27 11:49:44 -04:00
Nick TerrellandGitHub d2b5e5496d Merge pull request #2726 from terrelln/oss-fuzz-fix 
[bug-fix] Fix a determinism bug with the DUBT
 2021-07-21 13:46:21 -07:00
Nick Terrell ba044bd6f1 [bug-fix] Fix a determinism bug with the DUBT 
The DUBT can be non-deterministic if an index is equal to
`ZSTD_DUBT_UNSORTED_MARK`. Ensure that never happens by starting the
indices at 2.

This bug was found by the OSS-Fuzz determinism fuzzer. With this change
the fuzzer test passes. And I've confirmed that this is the root cause,
not just hiding the problem.

Aside: This took me a long time to figure out, because I thought I had
tried this first thing. But, apparantly I messed it up, because when I
was going through it again with @felixhandte, I was pointing out that it
wasn't the case, but it turns out it was.

Credit to: OSS-Fuzz
 2021-07-15 13:02:49 -07:00
Yann ColletandGitHub b18febe52c Merge pull request #2725 from makise-homura/mcst-lcc-support 
Add support for MCST LCC compiler
 2021-07-15 00:09:12 +02:00
makise-homura 3cd085cec3 Clarify no-tree-vectorize usage for ICC and LCC 2021-07-14 20:00:44 +03:00
makise-homura a5f518ae27 Change zstdcli's main() declaration due to -Wmain on some compilers 2021-07-14 19:55:47 +03:00
makise-homura d4ad02c721 Add support for MCST LCC compiler 2021-07-10 03:57:06 +03:00
Jonathan Ringer e1f85dbca3 pzstd: fix linking for static builds 2021-07-09 13:21:15 -07:00
binhdvoandGitHub b3e372c171 Merge pull request #2717 from binhdvo/bootcamp 
Proactively skip huffman compression based on sampling where non-comp…
 2021-07-01 10:39:58 -04:00
senandGitHub d90bc0e0b6 Merge pull request #2720 from senhuang42/remove_folder 
Remove folder when done with test
 2021-06-30 14:37:21 -04:00
senhuang42 f5f6cc2e48 Remove folder when done with test 2021-06-30 13:09:00 -04:00
Binh Vo dc5b693f1e Proactively skip huffman compression based on sampling where non-compressibility is suspected 2021-06-30 11:02:47 -04:00
Nick TerrellandGitHub 7b43e69b45 Merge pull request #2655 from heitbaum/patch-1 
[trace] remove zstd_trace.c reference from freestanding
 2021-06-29 18:33:40 -07:00
Nick TerrellandGitHub 609be382ac Merge pull request #2719 from danlark1/danlark_iwyu 
Include what you use in zstd_ldm_geartab
 2021-06-29 16:53:10 -07:00
Nick TerrellandGitHub 094b26081f Merge pull request #2689 from danlark1/dev 
Optimize zstd decompression by another x%
 2021-06-29 11:34:36 -07:00
Danila Kutenin e855b78be6 Include what you use in zstd_ldm_geartab 2021-06-29 17:57:53 +01:00
Danila Kutenin 2c2c9e7dfd Add possible improvements for gcc-11 2021-06-29 09:06:47 +01:00
Yann ColletandGitHub 88f107b5b9 Merge pull request #2716 from facebook/fix2701 
remove invalid test
 2021-06-24 10:57:18 -07:00
senandGitHub 45d707e908 Merge pull request #2715 from senhuang42/sequence_api_3 
[RFC] Add internal API for converting ZSTD_Sequence into seqStore
 2021-06-24 13:02:11 -04:00