Commit Graph
9353 Commits
Author SHA1 Message Date
Nick TerrellandGitHub 859e0500ab Merge pull request #2915 from terrelln/oss-fuzz-build-fix
Fix oss-fuzz build
2021-12-08 15:32:49 -05:00
Felix HandteandGitHub 33562c55de Merge pull request #2912 from felixhandte/pkg-config-fix
Fix Up #2659; Build libzstd.pc Whenever Building the Lib on Unix
2021-12-08 15:22:56 -05:00
Nick Terrell aa7729c9f3 Fix oss-fuzz build
Disable assembly when dataflow sanitizer is enabled.

This regressed in PR #2893, which accidentally removed the check for
dataflow sanitizer.
2021-12-08 11:01:52 -08:00
Yann Collet fb3522a3fe fixed very minor cast warning under cygwin 2021-12-08 09:48:56 -08:00
W. Felix Handte 9f1dee8fa5 Fix Up #2659; Build libzstd.pc Whenever Building the Lib on Unix 2021-12-08 12:43:34 -05:00
Yann Collet 1c7d2c4dd5 updated xxHash to latest v0.8.1
with minor modifications directly embedded in source :
- does not compile XXH3
- namespace emulation (ZSTD_ prefix)

Incidentally fix #2824
2021-12-07 21:16:15 -08:00
binhdvoandGitHub 38dfc4699e Imply -q when stderr is not a tty (#2884)
* Imply -q when stderr is not a tty
2021-12-07 16:56:19 -05:00
Felix HandteandGitHub 9118ee04c2 Merge pull request #2659 from ericonr/pc
[lib] Fix libzstd.pc for lib-mt builds
2021-12-07 14:18:38 -05:00
Nick TerrellandGitHub b6b4c9a3da Merge pull request #2907 from Hello71/armv6-fix-legacy
Apply FORCE_MEMORY_ACCESS=1 to legacy
2021-12-06 15:41:22 -05:00
Nick TerrellandGitHub e7b0ae385e Merge pull request #2890 from 15596858998/dec_1201
fixbug CLI's -D fails when the argument is not a regular file
2021-12-06 13:18:15 -05:00
Alex Xu (Hello71) 3d773d7013 Apply FORCE_MEMORY_ACCESS=1 to legacy
See #2633, #2881.
2021-12-05 22:51:44 -05:00
15596858998 ba7e8f1051 add test case 2021-12-05 19:12:52 +08:00
Nick TerrellandGitHub 486472c453 Merge pull request #2893 from terrelln/issue-2789
[asm] Share portability macros and restrict ASM further
2021-12-03 14:07:30 -05:00
Felix HandteandGitHub d2c86ec898 Merge pull request #2897 from felixhandte/zstd-deprecated-avoid-deprecated
Avoid Using Deprecated Functions in Deprecated Code
2021-12-03 12:09:58 -05:00
Felix HandteandGitHub fd6c34f641 Merge pull request #2899 from felixhandte/cmake-disable-multithreading-android
Disable Multithreading in CMake Builds for Android
2021-12-03 12:09:39 -05:00
Nick Terrell c284569457 [asm] Share portability macros and restrict ASM further
Move portability macros to `lib/common/portability_macros.h`. This file
only contains platform/feature detection (e.g. 0/1 macros). This file is
shared between C and ASM code, so it cannot include any C code.

Rename `HUF_` ASM macros to be `ZSTD_` prefixed, and move to the new
header.

Restrict `ZSTD_ASM_SUPPORTED` to `__GNUC__`, because we need the GAS
assembler.

Finally, only include the ASM code if we are actually going to use it.
This disables it on all Windows platforms, which should resolve the
problem brought up in Issue #2789.
2021-12-02 16:58:04 -08:00
Nick TerrellandGitHub 647c1b6615 Merge pull request #2900 from terrelln/issue-2893-test
[CI] Add cmake windows build
2021-12-02 19:51:33 -05:00
Nick TerrellandGitHub 014bbb29f8 Merge pull request #2898 from terrelln/issue-2862
Improve zstd_opt build speed and size
2021-12-02 19:49:43 -05:00
Nick Terrell a74a36985a [CI] Add cmake windows build
Build on windows with cmake to ensure everything compiles.
2021-12-02 15:23:33 -08:00
Yann ColletandGitHub 1bf3d8a475 Merge pull request #2896 from facebook/m68k
Zstandard compiles and run on m68k cpus
2021-12-02 14:25:45 -08:00
W. Felix Handte 4a82bc9d00 Disable Multithreading in CMake Builds for Android 2021-12-02 17:23:42 -05:00
Nick Terrell e5bfaeede7 Improve zstd_opt build speed and size
Use the same trick as we did for zstd_lazy in PR #2828:
* Create one search function specialization for each (dictMode, mls).
* Select the search function pointer at the top of the match finder.

Additionally, we no longer inline `ZSTD_compressBlock_opt_generic` into
every function, since `dictMode` is no longer used as a template. Create
two specializations, for opt levels 0 and 2, and call one of the two
specializations.

Lastly, remove the hack that disabled inlining for zstd_opt for the
Linux Kernel, as we've gotten most of the benefit already.

Compilation time sees a ~4x reduction:

| Compiler | Flags                            | Dev Time (s) | PR Time (s) | Delta |
|----------|----------------------------------|--------------|-------------|-------|
| gcc      | -O3                              |         10.1 |         2.3 |  -77% |
| gcc      | -O3 -fsanitize=address,undefined |         61.1 |        10.2 |  -83% |
| clang    | -O3                              |          9.0 |         2.1 |  -76% |
| clang    | -O3 -fsanitize=address,undefined |         33.5 |         5.1 |  -84% |

Build size is reduced by 150KB - 200KB:

| Compiler | Dev libzstd.a Size (B) | PR libzstd.a Size (B) | Delta |
|----------|------------------------|-----------------------|-------|
| gcc      |                1327476 |               1177108 |  -11% |
| clang    |                1378324 |               1167780 |  -15% |

There is a <2% speed loss in all cases:

| Compiler | Level | Dev Speed (MB/s) | PR Speed (MB/s) | Delta  |
|----------|-------|------------------|-----------------|--------|
| gcc      |    16 |             4.78 |            4.72 | -1.25% |
| gcc      |    17 |             3.49 |            3.46 | -0.85% |
| gcc      |    18 |             2.92 |            2.86 | -2.04% |
| gcc      |    19 |             2.61 |            2.61 |  0.00% |
| clang    |    16 |             4.69 |            4.80 |  2.34% |
| clang    |    17 |             3.53 |            3.49 | -1.13% |
| clang    |    18 |             2.86 |            2.85 | -0.34% |
| clang    |    19 |             2.61 |            2.61 |  0.00% |

Fixes Issue #2862.
2021-12-02 14:19:41 -08:00
W. Felix Handte e688317652 Fix Include Path 2021-12-02 16:53:52 -05:00
Nick TerrellandGitHub 01ecd6ffc0 Merge pull request #2892 from terrelln/issue-2785
[CircleCI] Fix short-tests-0
2021-12-02 16:20:56 -05:00
W. Felix Handte d82d67d073 Migrate to FORWARD_IF_ERROR 2021-12-02 16:06:07 -05:00
Yann Collet 30b9db8ae4 changed macro name to ZSTD_ALIGNOF
for better consistency
2021-12-02 12:57:42 -08:00
Yann Collet 1d025d871b bound alignment backup to sizeof(void*) 2021-12-02 11:30:03 -08:00
W. Felix Handte a3ee9815c7 Avoid Using Deprecated Functions in Deprecated Code
`lib/deprecated` is no longer built by zstd's bundled build files. However,
users may try to build these files when they import the source tree into
their own build systems. And if they have `-Wdeprecated-declarations` on,
this can produce warnings.

This PR migrates these files away from using deprecated declarations.

This addresses #2767.
2021-12-02 14:25:33 -05:00
Yann Collet 80a13fd645 move the alignment macro to compiler.h
because mem.h is dropped in the Linux kernel.

Changed macro definition order (gcc/clang/msvc before c11)
due to a limitation in the kernel source builder.

Changed the backup to sizeof(),
reverting to previous behavior when no support of alignof() is detected.
2021-12-02 11:20:01 -08:00
Nick TerrellandGitHub 21e28f5c24 Merge pull request #2891 from supperPants/dev
Fix typos
2021-12-02 13:53:33 -05:00
Yann ColletandGitHub a134737ad4 Merge pull request #2687 from sapiippo/dev
Makefile: fix build for mingw
2021-12-02 10:53:02 -08:00
Yann ColletandGitHub 550410d05d Merge pull request #2894 from facebook/tomerge2051
merge #2501
2021-12-02 10:52:46 -08:00
Yann Collet ffbdc8ac57 m68k CI tests on GA are now compulsory 2021-12-02 10:30:04 -08:00
Nick TerrellandGitHub b2fbd0a533 Merge pull request #2895 from kanavin/fix-repro
Makefile: sort all wildcard file list expansions
2021-12-02 13:20:15 -05:00
Alexander KanavinandAlexander Kanavin 1e514feec6 Makefile: sort all wildcard file list expansions
Otherwise the order is non-deterministic and breaks
reproducible builds.
2021-12-02 12:04:11 +01:00
Yann Collet 39dced092e fix align conditions for huf_compress 2021-12-01 23:02:00 -08:00
Nick Terrell 91f5891dd0 [CircleCI] Fix short-tests-0
short-tests-0 were silently failing. I think because of the && make clean construction. Switch to ; instead.

Also fix all the test failures that were exposed.

`make all` is failing on CircleCI because it is missing Docker. Move that test
to GitHub actions, and switch the pedantic CircleCI test to `make allmost`.
2021-12-01 17:43:46 -08:00
Yann Collet e89e847820 added alignment test
and fix an incorrect alignment check in cwksp which was failing on m68k
2021-12-01 17:16:36 -08:00
Yann Collet a71eed360b removed lib/Makefile preamble
now included from libzstd.mk
2021-12-01 16:54:59 -08:00
Yann Collet 3f64b31585 Merge branch 'dev' into tomerge2051 2021-12-01 15:29:49 -08:00
Yann ColletandGitHub c8767a4c74 Merge pull request #2501 from skitt/hide-static-symbols
Explicitly hide static symbols
2021-12-01 15:26:42 -08:00
Nick TerrellandGitHub 9b97fdf74f Merge pull request #2887 from terrelln/issue-2815
[zdict] Remove ZDICT_CONTENTSIZE_MIN restriction for ZDICT_finalizeDictionary
2021-12-01 18:15:53 -05:00
Yann Collet 49d578763d reduce storage requirement
51 MB seems excessive for CI storate and considering the nature of the test.

(note : maybe we should consider using `/tmp` for files generated during tests,
as tmpfs is typically using RAM, thus preserving storage.)
2021-12-01 15:10:55 -08:00
Yann ColletandGitHub 3133d1e86e Merge pull request #2876 from 15596858998/dev
Solve the bug of extra output newline character
2021-12-01 15:10:08 -08:00
Yann ColletandGitHub 8031dc7a48 Merge pull request #2885 from yoniko/limit-level-32bit-systems
Limit `ZSTD_maxCLevel` to 21 for 32-bit binaries.
2021-12-01 14:19:16 -08:00
Nick TerrellandGitHub e34e58b458 Merge pull request #2886 from terrelln/issue-2865
[test] Test that the exec-stack bit isn't set on libzstd.so
2021-12-01 13:05:35 -05:00
Nick TerrellandGitHub 8f9f6442f5 Merge pull request #2888 from terrelln/issue-2814
[bmi2] Add lzcnt and bmi target attributes
2021-12-01 13:05:09 -05:00
Nick TerrellandGitHub 906c3e9ccb Merge pull request #2889 from terrelln/issue-2811
[contrib][pzstd] Fix build issue with gcc-5
2021-12-01 13:03:03 -05:00
supperPants c2260c1d45 Merge https://github.com/facebook/zstd into dev 2021-12-01 22:41:24 +08:00
supperPants d4713de5a3 Fix typos. 2021-12-01 22:36:21 +08:00