Commit Graph
9267 Commits
Author SHA1 Message Date
W. Felix Handte b8434cb754 Allow Templating ZSTD_fast Matchfinders on Acceleration (Lvl < -1) 2021-12-13 14:46:57 -05:00
W. Felix Handte ace6a7e746 Decompose step into Two Variables
This avoids an additional addition, at the cost of an additional variable.
2021-12-10 16:44:23 -05:00
W. Felix Handte 22501cd283 Stagger Application of stepSize in ZSTD_fast
This replicates the behavior of @terrelln's `ZSTD_fast` implementation. That
is, it always looks at adjacent pairs of positions, and only applies the
acceleration every other position. This produces a more fine-grained
acceleration.
2021-12-10 16:44:23 -05:00
Yann Collet 4cc5e2818a complete changelog with #2885 2021-12-09 09:53:45 -08:00
Yann ColletandGitHub c077b530a0 Merge pull request #2917 from facebook/change151
Update changelog for v1.5.1
2021-12-09 08:45:34 -08:00
Felix HandteandGitHub 0c26d98c0d Merge pull request #2910 from felixhandte/reject-irregular-dicts
Reject Irregular Dictionary Files
2021-12-09 11:44:37 -05:00
Yann Collet 3d738307b4 Update changelog for v1.5.1 2021-12-08 16:55:38 -08:00
Yann ColletandGitHub 57383d2317 Merge pull request #2914 from facebook/xxhash081
updated xxHash to latest v0.8.1
2021-12-08 16:48:46 -08:00
Yann Collet 3ce265fea8 remove offending static assert lines
no idea why visual + clang-cl + appveyor don't like them,
I've not been able to reproduce the issue locally,
but these static assert are very unlikely to deliver a useful signal,
I can't imagine a situation where they will be wrong,
and if they are, then a ton of other things will be broken way before reaching that point.
2021-12-08 15:05:17 -08:00
Nick TerrellandGitHub 8b40095b3f Merge pull request #2916 from terrelln/issue-2906
Remove possible NULL pointer addition
2021-12-08 16:51:10 -05:00
Yann Collet 16241b7d26 altered copyright title 2021-12-08 13:18:41 -08:00
W. Felix Handte 9985e10fda Reject Irregular Dictionary Files
I hadn't seen #2890, so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.

Addresses #2874.
2021-12-08 16:17:04 -05:00
Yann Collet a9cd6164d7 removed declarations of XXH3 symbols when XXH_NO_XXH3 is defined
on top of implementations, which were already scoped out.
2021-12-08 12:56:16 -08:00
Yann Collet 27e706de88 replaces malloc / free / memcpy by Zstandard's version 2021-12-08 12:51:04 -08:00
Nick Terrell b94407b6cf Remove possible NULL pointer addition
Refactor `ZSTDMT_isOverlapped()` to do NULL checks before computing the
end pointer.

Fixes #2906.
2021-12-08 12:40:40 -08:00
Nick TerrellandGitHub 859e0500ab Merge pull request #2915 from terrelln/oss-fuzz-build-fix
Fix oss-fuzz build
2021-12-08 15:32:49 -05:00
Felix HandteandGitHub 33562c55de Merge pull request #2912 from felixhandte/pkg-config-fix
Fix Up #2659; Build libzstd.pc Whenever Building the Lib on Unix
2021-12-08 15:22:56 -05:00
Nick Terrell aa7729c9f3 Fix oss-fuzz build
Disable assembly when dataflow sanitizer is enabled.

This regressed in PR #2893, which accidentally removed the check for
dataflow sanitizer.
2021-12-08 11:01:52 -08:00
Yann Collet fb3522a3fe fixed very minor cast warning under cygwin 2021-12-08 09:48:56 -08:00
W. Felix Handte 9f1dee8fa5 Fix Up #2659; Build libzstd.pc Whenever Building the Lib on Unix 2021-12-08 12:43:34 -05:00
Yann Collet 1c7d2c4dd5 updated xxHash to latest v0.8.1
with minor modifications directly embedded in source :
- does not compile XXH3
- namespace emulation (ZSTD_ prefix)

Incidentally fix #2824
2021-12-07 21:16:15 -08:00
binhdvoandGitHub 38dfc4699e Imply -q when stderr is not a tty (#2884)
* Imply -q when stderr is not a tty
2021-12-07 16:56:19 -05:00
Felix HandteandGitHub 9118ee04c2 Merge pull request #2659 from ericonr/pc
[lib] Fix libzstd.pc for lib-mt builds
2021-12-07 14:18:38 -05:00
Nick TerrellandGitHub b6b4c9a3da Merge pull request #2907 from Hello71/armv6-fix-legacy
Apply FORCE_MEMORY_ACCESS=1 to legacy
2021-12-06 15:41:22 -05:00
Nick TerrellandGitHub e7b0ae385e Merge pull request #2890 from 15596858998/dec_1201
fixbug CLI's -D fails when the argument is not a regular file
2021-12-06 13:18:15 -05:00
Alex Xu (Hello71) 3d773d7013 Apply FORCE_MEMORY_ACCESS=1 to legacy
See #2633, #2881.
2021-12-05 22:51:44 -05:00
Nick TerrellandGitHub 486472c453 Merge pull request #2893 from terrelln/issue-2789
[asm] Share portability macros and restrict ASM further
2021-12-03 14:07:30 -05:00
Felix HandteandGitHub d2c86ec898 Merge pull request #2897 from felixhandte/zstd-deprecated-avoid-deprecated
Avoid Using Deprecated Functions in Deprecated Code
2021-12-03 12:09:58 -05:00
Felix HandteandGitHub fd6c34f641 Merge pull request #2899 from felixhandte/cmake-disable-multithreading-android
Disable Multithreading in CMake Builds for Android
2021-12-03 12:09:39 -05:00
Nick Terrell c284569457 [asm] Share portability macros and restrict ASM further
Move portability macros to `lib/common/portability_macros.h`. This file
only contains platform/feature detection (e.g. 0/1 macros). This file is
shared between C and ASM code, so it cannot include any C code.

Rename `HUF_` ASM macros to be `ZSTD_` prefixed, and move to the new
header.

Restrict `ZSTD_ASM_SUPPORTED` to `__GNUC__`, because we need the GAS
assembler.

Finally, only include the ASM code if we are actually going to use it.
This disables it on all Windows platforms, which should resolve the
problem brought up in Issue #2789.
2021-12-02 16:58:04 -08:00
Nick TerrellandGitHub 647c1b6615 Merge pull request #2900 from terrelln/issue-2893-test
[CI] Add cmake windows build
2021-12-02 19:51:33 -05:00
Nick TerrellandGitHub 014bbb29f8 Merge pull request #2898 from terrelln/issue-2862
Improve zstd_opt build speed and size
2021-12-02 19:49:43 -05:00
Nick Terrell a74a36985a [CI] Add cmake windows build
Build on windows with cmake to ensure everything compiles.
2021-12-02 15:23:33 -08:00
Yann ColletandGitHub 1bf3d8a475 Merge pull request #2896 from facebook/m68k
Zstandard compiles and run on m68k cpus
2021-12-02 14:25:45 -08:00
W. Felix Handte 4a82bc9d00 Disable Multithreading in CMake Builds for Android 2021-12-02 17:23:42 -05:00
Nick Terrell e5bfaeede7 Improve zstd_opt build speed and size
Use the same trick as we did for zstd_lazy in PR #2828:
* Create one search function specialization for each (dictMode, mls).
* Select the search function pointer at the top of the match finder.

Additionally, we no longer inline `ZSTD_compressBlock_opt_generic` into
every function, since `dictMode` is no longer used as a template. Create
two specializations, for opt levels 0 and 2, and call one of the two
specializations.

Lastly, remove the hack that disabled inlining for zstd_opt for the
Linux Kernel, as we've gotten most of the benefit already.

Compilation time sees a ~4x reduction:

| Compiler | Flags                            | Dev Time (s) | PR Time (s) | Delta |
|----------|----------------------------------|--------------|-------------|-------|
| gcc      | -O3                              |         10.1 |         2.3 |  -77% |
| gcc      | -O3 -fsanitize=address,undefined |         61.1 |        10.2 |  -83% |
| clang    | -O3                              |          9.0 |         2.1 |  -76% |
| clang    | -O3 -fsanitize=address,undefined |         33.5 |         5.1 |  -84% |

Build size is reduced by 150KB - 200KB:

| Compiler | Dev libzstd.a Size (B) | PR libzstd.a Size (B) | Delta |
|----------|------------------------|-----------------------|-------|
| gcc      |                1327476 |               1177108 |  -11% |
| clang    |                1378324 |               1167780 |  -15% |

There is a <2% speed loss in all cases:

| Compiler | Level | Dev Speed (MB/s) | PR Speed (MB/s) | Delta  |
|----------|-------|------------------|-----------------|--------|
| gcc      |    16 |             4.78 |            4.72 | -1.25% |
| gcc      |    17 |             3.49 |            3.46 | -0.85% |
| gcc      |    18 |             2.92 |            2.86 | -2.04% |
| gcc      |    19 |             2.61 |            2.61 |  0.00% |
| clang    |    16 |             4.69 |            4.80 |  2.34% |
| clang    |    17 |             3.53 |            3.49 | -1.13% |
| clang    |    18 |             2.86 |            2.85 | -0.34% |
| clang    |    19 |             2.61 |            2.61 |  0.00% |

Fixes Issue #2862.
2021-12-02 14:19:41 -08:00
W. Felix Handte e688317652 Fix Include Path 2021-12-02 16:53:52 -05:00
Nick TerrellandGitHub 01ecd6ffc0 Merge pull request #2892 from terrelln/issue-2785
[CircleCI] Fix short-tests-0
2021-12-02 16:20:56 -05:00
W. Felix Handte d82d67d073 Migrate to FORWARD_IF_ERROR 2021-12-02 16:06:07 -05:00
Yann Collet 30b9db8ae4 changed macro name to ZSTD_ALIGNOF
for better consistency
2021-12-02 12:57:42 -08:00
Yann Collet 1d025d871b bound alignment backup to sizeof(void*) 2021-12-02 11:30:03 -08:00
W. Felix Handte a3ee9815c7 Avoid Using Deprecated Functions in Deprecated Code
`lib/deprecated` is no longer built by zstd's bundled build files. However,
users may try to build these files when they import the source tree into
their own build systems. And if they have `-Wdeprecated-declarations` on,
this can produce warnings.

This PR migrates these files away from using deprecated declarations.

This addresses #2767.
2021-12-02 14:25:33 -05:00
Yann Collet 80a13fd645 move the alignment macro to compiler.h
because mem.h is dropped in the Linux kernel.

Changed macro definition order (gcc/clang/msvc before c11)
due to a limitation in the kernel source builder.

Changed the backup to sizeof(),
reverting to previous behavior when no support of alignof() is detected.
2021-12-02 11:20:01 -08:00
Nick TerrellandGitHub 21e28f5c24 Merge pull request #2891 from supperPants/dev
Fix typos
2021-12-02 13:53:33 -05:00
Yann ColletandGitHub a134737ad4 Merge pull request #2687 from sapiippo/dev
Makefile: fix build for mingw
2021-12-02 10:53:02 -08:00
Yann ColletandGitHub 550410d05d Merge pull request #2894 from facebook/tomerge2051
merge #2501
2021-12-02 10:52:46 -08:00
Yann Collet ffbdc8ac57 m68k CI tests on GA are now compulsory 2021-12-02 10:30:04 -08:00
Nick TerrellandGitHub b2fbd0a533 Merge pull request #2895 from kanavin/fix-repro
Makefile: sort all wildcard file list expansions
2021-12-02 13:20:15 -05:00
Alexander KanavinandAlexander Kanavin 1e514feec6 Makefile: sort all wildcard file list expansions
Otherwise the order is non-deterministic and breaks
reproducible builds.
2021-12-02 12:04:11 +01:00
Yann Collet 39dced092e fix align conditions for huf_compress 2021-12-01 23:02:00 -08:00