ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,983 Commits 17 Branches 77 Tags
35932ab2f1e129dce0d19bfa82787dc4dc262eed
Commit Graph
8983 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
W. Felix Handte 35932ab2f1 Prefetch Input in Incompressible Sections (+0.25% Speed) 2021-09-01 14:15:03 -04:00
W. Felix Handte b092dd75b7 Shrink Pipeline from 4 Positions to 3 2021-09-01 14:15:03 -04:00
W. Felix Handte 387840af79 Re-Order Operations for Slightly Better Performance 2021-09-01 14:15:03 -04:00
W. Felix Handte bc768bccc0 Track Step Size Statefully, Rather than Recalculating Every Time 2021-09-01 14:15:03 -04:00
W. Felix Handte 80bc12b33a Initial Pipelined Implementation for ZSTD_fast 2021-09-01 14:15:03 -04:00
W. Felix Handte ab8aa49b8d Fix Benchmark Corruption Display 2021-09-01 14:15:03 -04:00
Yann ColletandGitHub 6715096611 Merge pull request #2758 from facebook/qemu 
added qemu tests
 2021-08-31 09:56:50 -07:00
Yann Collet c1de65535f Merge branch 'dev' into qemu 2021-08-31 08:16:46 -07:00
Yann ColletandGitHub 6933ac67d3 Merge pull request #2757 from facebook/transferGA 
Reduce test time on TravisCI
 2021-08-31 07:40:21 -07:00
Yann Collet 333ecf6865 add powerpc qemu emulation 2021-08-30 06:37:50 -07:00
Yann Collet 2b27d07d06 attempt at adding m68k qemu tests 
with optional success (for the time being)
 2021-08-29 21:39:06 -07:00
Yann Collet 1e5c90cb5b remove qemu tests 
that are being transfered to GA in #2758.
This represents a saving of ~25mn of cpu time on TravisCI.
 2021-08-29 20:54:18 -07:00
Yann Collet 74b4171fb8 fix alignment condition in FSE_buildCTable 
2-bytes alignment is enough for 16-bit fields
 2021-08-29 19:05:04 -07:00
Yann Collet f21977c5e6 fix playTests.sh when EXE_PREFIX not null 2021-08-29 17:20:12 -07:00
Yann Collet 18191c85c9 adding optional QEMU_SYS 2021-08-29 16:43:32 -07:00
Yann Collet 1c97ec73d7 added qemu tests 
running zstd library on emulated targets
 2021-08-29 16:28:41 -07:00
Yann Collet b341aa2f95 remove versions-compatibility test from GA 
since it fails on Github Actions specifically.

The test is run on TravisCI for the time being.
Its duration has been reduced to ~6mn anyway.
 2021-08-29 15:47:04 -07:00
Yann Collet 72bd2a83a0 reduce length of scanbuild static analyzer test 
This was ~30mn, by far the longest run on travisCI.
That's because it re-analyzes multiple times the same files (library files notably).
It also performs actions that make no sense for the static analyzer purpose,
such as building the single-file library.

Reduced time spent in this test by reducing its scope :
just build the CLI, and obviously the library along it.
These are the only ones that really deserve to be analyzed.

Unfortunately, it still results in a number of false positives when using newer versions of scanbuild
(each version of scanbuild generates a different list of false positives).
These will have to be fixed before transfering to Github Actions.
 2021-08-29 15:26:31 -07:00
Yann Collet 7f37b8a547 accelerate versionsCompatibilityTest 
by allowing parallel build of units,
and reducing optimization levels.

Parallel build is only effective on "recent" versions of `zstd`,
as previously, the list of units was passed as a list of source files,
which is something neither `make` nor `gcc` can parallelize.
So its impact is mildly effective (-20%).

Reducing optimization level to `-O1` makes compilation much faster.
It also makes runtime slower,
but in this test, compilation time dominates run time.
The savings are very significant (-50%).

On my test system, it reduces the length of this test from 13mn to 5mn.
 2021-08-29 14:48:11 -07:00
Yann Collet ef69539849 transferred inter-versions compatibility tests to GA 2021-08-29 11:53:56 -07:00
senandGitHub 1f3fc1936c Merge pull request #2753 from senhuang42/better_error_msg 
[easy] Fix zstd bench error message
 2021-08-23 20:37:48 -04:00
senhuang42 dce48f53df Fix benchzstd error message 2021-08-23 19:10:16 -04:00
Yann ColletandGitHub be82a0ab8f Merge pull request #2746 from eli-schwartz/meson-fixup 
meson fixups
 2021-08-23 15:57:47 -07:00
Yann ColletandGitHub 18a20b3ad7 Merge pull request #2752 from facebook/hashLog3max 
make ZSTD_HASHLOG3_MAX private
 2021-08-20 12:51:17 -07:00
Yann Collet 2de42174bb make ZSTD_HASHLOG3_MAX private 
This is an implementation detail,
it doesn't belong to public space (zstd.h).
 2021-08-20 09:52:42 -07:00
senandGitHub ae998544de Merge pull request #2750 from senhuang42/sb_compress 
Improve branch misses on FSE symbol spreading
 2021-08-20 12:47:24 -04:00
senhuang42 da095ed899 Improve branch misses on FSE symbol spreading 2021-08-18 10:22:22 -07:00
Eli Schwartz 9748608aeb meson: set the symbol visibility of the shared library to hidden 
This matches the Makefile build. Due to one private xxhash symbol in use
by the program, it recompiles a private copy of xxhash.

Due to the test binaries making extensive (?) use of private symbols, it
doesn't even attempt to link to shared libzstd, and instead, all of the
original object files are added to libtestcommon itself for private
linkage. This, too, matches the Makefile build.

Ref. #2261
 2021-08-09 22:03:17 -04:00
Eli Schwartz d95a3f5097 meson: fix warnings in build files 
meson prefers that project-level options for Wall/Wextra/pedantic be
used, rather than hardcoding raw flags in add_project_arguments. If you
do the latter anyway, it raises a meson warning.

Set the default options for the project to use all this.

Also move the -Werror comment to the project default options with
appropriate format, but leave it commented out since it does not work.
 2021-08-09 20:17:00 -04:00
Eli Schwartz ee3355df30 meson: link tests with a convenience library of sources used by multiple binaries 
This saves repeatedly compiling them, resulting in 16 fewer compile
steps, in exchange for one `ar` step.
 2021-08-09 20:15:44 -04:00
senandGitHub 6a258043f5 Merge pull request #2692 from senhuang42/rebalance_clevel 
[RFC] Rebalance compression levels
 2021-08-06 12:51:31 -04:00
Sen Huang 539b3aab9b Optimize 32-bit VecMask_next() 2021-08-04 17:14:58 -04:00
Felix HandteandGitHub ae131282f5 Merge pull request #2742 from felixhandte/set-mtime 
Set mtime on Output Files
 2021-08-04 16:44:46 -04:00
senhuang42 e411040ea1 Add 64 row entry support for lazy 2021-08-04 16:19:12 -04:00
senhuang42 31820e032c Rebalance clevels for lazy 2021-08-04 16:18:52 -04:00
senandGitHub f9b034086a Merge pull request #2732 from senhuang42/huf_sort_improvement 
[HUF] Improve Huffman sorting algorithm
 2021-08-04 16:07:56 -04:00
W. Felix Handte a8f4612eab Remove sleep()s in Test; Replace with Artificial mtime 
This behavior of `touch` is standardized in POSIX, so it should be available.
 2021-08-04 15:56:46 -04:00
W. Felix Handte 9cd6c1ff4d Update mtime and atime for Written Files 2021-08-04 14:49:56 -04:00
W. Felix Handte a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
senhuang42 aa1957477b Improve Huffman sorting algorithm 2021-08-04 12:43:34 -04:00
W. Felix Handte ead41bcb4e Add Test to Verify mtime is Copied to Destination 2021-08-03 17:22:58 -04:00
Nick TerrellandGitHub 6ee70bae46 Merge pull request #2733 from terrelln/huf-cspeed 
[HUF] Improve Huffman encoding speed
 2021-08-03 12:59:54 -04:00
Nick Terrell d8a0797268 [fuzz] Add Huffman round trip fuzzer 
* Add a Huffman round trip fuzzer
* Fix two minor bugs in Huffman that aren't exposed in zstd
  - Incorrect weight comparison (weights are allowed to be equal to
    table log).
  - HUF_compress1X_usingCTable_internal() can return compressed
    size >= source size, so the assert that `cSize <= 65535` isn't
    correct, and it needs to be checked instead.
 2021-08-03 08:10:06 -07:00
senandGitHub 5c46f62006 Merge pull request #2677 from senhuang42/ci_overhaul_2 
[CI][2/2] Migrate CI tests which (currently) fail
 2021-08-02 09:55:49 -04:00
Sen Huang 5ec7897a26 Fix static analyzer warnings 2021-07-29 09:11:12 -07:00
Sen Huang 43aad811db Add VS2019 test, migrate minimal decompressor macros to GHActions 2021-07-29 08:49:36 -07:00
Nick TerrellandGitHub 32cfe15edc Merge pull request #2724 from jonringer/fix-pzstd-static-build 
pzstd: fix linking for static builds
 2021-07-28 14:53:01 -07:00
Nick TerrellandGitHub d69e007cca Merge pull request #2707 from koalabearguo/dev 
Z_PREFIX  zError function
 2021-07-28 14:52:15 -07:00
Felix HandteandGitHub 495597699c Merge pull request #2729 from felixhandte/fix-ddss-load 
Fix DDSS Load
 2021-07-28 13:12:50 -04:00
Nick Terrell 46f2710562 [HUF] Improve Huffman encoding speed 
Improve Huffman encoding speed by 20% for gcc and 10% for clang.

| Compiler |     Benchmark     | Config  |   Dataset   | Ratio | Speed MB/s (dev) | Speed MB/s (huf-cspeed) | Speed MB/s (huf-cspeed - dev) |
|----------|-------------------|---------|-------------|-------|------------------|-------------------------|-------------------------------|
| gcc      | compress          | level_1 | enwik7      | 2.43  | 253.70           | 258.72                  | 2.0%                          |
| gcc      | compress          | level_1 | silesia     | 2.88  | 341.90           | 348.15                  | 1.8%                          |
| gcc      | compress_literals | level_1 | enwik7      | 1.49  | 761.83           | 912.76                  | 19.8%                         |
| gcc      | compress_literals | level_1 | silesia     | 1.28  | 754.83           | 902.37                  | 19.5%                         |
| gcc      | compress_literals | level_7 | enwik7      | 1.29  | 502.81           | 552.79                  | 9.9%                          |
| gcc      | compress_literals | level_7 | silesia     | 1.11  | 675.97           | 776.44                  | 14.9%                         |
| clang    | compress          | level_1 | enwik7      | 2.43  | 277.54           | 280.98                  | 1.2%                          |
| clang    | compress          | level_1 | silesia     | 2.88  | 369.98           | 375.46                  | 1.5%                          |
| clang    | compress_literals | level_1 | enwik7      | 1.49  | 828.83           | 918.41                  | 10.8%                         |
| clang    | compress_literals | level_1 | silesia     | 1.28  | 815.81           | 905.41                  | 11.0%                         |
| clang    | compress_literals | level_7 | enwik7      | 1.29  | 533.13           | 553.30                  | 3.8%                          |
| clang    | compress_literals | level_7 | silesia     | 1.11  | 714.52           | 775.38                  | 8.5%                          |
 2021-07-27 15:10:35 -07:00