ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,335 Commits 17 Branches 77 Tags
c7269add7eaf028ed828d9af41e732cf01993aad
Commit Graph
10335 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick TerrellandNick Terrell c7269add7e [huf] Improve fast huffman decoding speed in linux kernel 
gcc in the linux kernel was not unrolling the inner loops of the Huffman
decoder, which was destroying decoding performance. The compiler was
generating crazy code with all sorts of branches. I suspect because of
Spectre mitigations, but I'm not certain. Once the loops were manually
unrolled, performance was restored.

Additionally, when gcc couldn't prove that the variable left shift in
the 4X2 decode loop wasn't greater than 63, it inserted checks to verify
it. To fix this, mask `entry.nbBits & 0x3F`, which allows gcc to eliete
this check. This is a no op, because `entry.nbBits` is guaranteed to be
less than 64.

Lastly, introduce the `HUF_DISABLE_FAST_DECODE` macro to disable the
fast C loops for Issue #3762. So if even after this change, there is a
performance regression, users can opt-out at compile time.
 2023-11-20 14:56:46 -05:00
Nick TerrellandNick Terrell e122fcbf58 [debug] Don't define g_debuglevel in the kernel 
We only use this constant when `DEBUGLEVEL>=2`, but we get
-Werror=pedantic errors for empty translation units, so still define it
except in kernel environments.

Backport from the kernel:

https://lore.kernel.org/lkml/20230616144400.172683-1-ben.dooks@codethink.co.uk/
 2023-11-17 09:54:10 -08:00
Nick TerrellandNick Terrell c2d470581e [linux] Remove usage of deprecated function 
ZSTD_resetDStream() is deprecated and replaced by ZSTD_DCtx_reset().
This removes deprecation warnings from the kernel build.

This change is a no-op, see the docs suggesting this replacement.

https://github.com/facebook/zstd/blob/fcbf2fde9ac7ce1562c7b3a394350e764bcb580f/lib/zstd.h#L2655-L2663
 2023-11-17 09:54:10 -08:00
Nick TerrellandNick Terrell a419265d30 [linux] Backport intptr_t removal 
Linux started providing intptr_t in <linux/types.h> so we no longer need
to define it here.

https://lkml.kernel.org/r/ed66b9e4-1fb7-45be-9bb9-d4bc291c691f@p183
 2023-11-17 09:54:10 -08:00
Yann ColletandGitHub fcbf2fde9a Merge pull request #3807 from gruenich/increase_minimum_cmake_version_3.5 
[cmake] Require CMake version 3.5 or newer
 2023-11-14 10:48:11 -08:00
Yann ColletandGitHub e19fe535f4 Merge pull request #3813 from elasota/overflow-clarification 
Clarify that a stream containing too many Huffman weights is invalid
 2023-11-13 00:03:42 -08:00
Yann ColletandGitHub 83adaecf41 Merge pull request #3812 from jondo2010/bazel_build_doc 
Add Bazel module instructions to README.md
 2023-11-13 00:03:10 -08:00
elasota e61e3ff152 Clarify that decoding too many Huffman weights is a failure condition 2023-11-08 20:06:58 -05:00
John Hughes 98d8ad27a2 Add Bazel module instructions to README.md 2023-11-08 09:08:21 +00:00
Christoph Grüninger f013b1b504 [cmake] Remove code for compatibility with CMake < 3.0 
The required version of CMake is now 3.5.
 2023-11-01 20:46:47 +01:00
Yann ColletandGitHub 1518570c62 Merge pull request #3806 from elasota/fmt-clarifications 
Correct FSE probability bit consumption in specification
 2023-11-01 10:59:54 -07:00
Christoph Grüninger 4502ca5f42 [cmake] Require CMake version 3.5 or newer 
More recent versions of CMake emit the following warning:
CMake Deprecation Warning at cmake/CMakeLists.txt:10 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.
 2023-10-31 23:30:43 +01:00
Yann ColletandGitHub fabce2a61c Merge pull request #3800 from facebook/dependabot/github_actions/actions/checkout-4.1.1 
Bump actions/checkout from 4.1.0 to 4.1.1
 2023-10-31 09:23:47 -07:00
Yann ColletandGitHub 81fdca0012 Merge pull request #3795 from Saverio976/doc/cmake-fetch-content 
Add doc on how to use it with cmake FetchContent
 2023-10-31 09:23:28 -07:00
elasota 324cce4996 Add definition of "log2sup" function 2023-10-31 11:45:10 -04:00
elasota b38d87b476 Clarify that the log2 of the largest possible symbol is the maximum number of bits consumed 2023-10-31 01:17:23 -04:00
Xavier Mitault 3c3845b9d8 Add target_include_directories because windows and macos need it for me 2023-10-23 17:38:09 +02:00
dependabot[bot]andGitHub af971cec65 Bump actions/checkout from 4.1.0 to 4.1.1 
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.0 to 4.1.1.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/8ade135a41bc03ea155e62e844d188df1ea18608...b4ffde65f46336ab88eb53be808477a3936bae11)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-10-23 05:56:05 +00:00
Yann ColletandGitHub 7567769a7e Merge pull request #3789 from facebook/fix_flexArray_fse 
solving flexArray issue #3785 in fse
 2023-10-20 12:40:13 -07:00
Yann Collet 24dabde507 revert to manually defining DTable 
thus avoiding the analyzer and ubsan to associate DTable to a size of 1.
 2023-10-18 22:45:57 -07:00
Yann Collet d988e00a7f baby-step towards solving flexArray issue #3785 
the flexArray in structure FSE_DecompressWksp
is just a way to derive a pointer easily,
without risk/complexity of calculating it manually.

Not sure if this change is good enough to avoid ubsan warnings though.
 2023-10-18 16:21:39 -07:00
Xavier Mitault e590c8a0e3 Add doc on how to use it with cmake FetchContent 2023-10-18 13:22:15 +02:00
Yann ColletandGitHub 69036dffe5 Merge pull request #3786 from facebook/fix_flexarray_cctx 
Remove FlexArray pattern from ZSTDMT
 2023-10-13 09:08:20 -07:00
Yann Collet 6bb1688c1a extended the fix to ZSTDMT's Buffer Pool 2023-10-08 00:25:17 -07:00
Yann Collet ea4027c003 removed unused macro constant 2023-10-07 23:32:22 -07:00
Yann Collet c87ad5bdb5 fixes suggested by @ebiggers 2023-10-07 23:29:42 -07:00
Yann Collet e8ff7d18eb removed FlexArray pattern from CCtxPool 
within ZSTDMT_.
This pattern is flagged by less forgiving variants of ubsan
notably used during compilation of the Linux Kernel.

There are 2 other places in the code where this pattern is used.
This fixes just one of them.
 2023-10-07 21:30:08 -07:00
Yann ColletandGitHub c692b8d12d Merge pull request #3752 from paulmenzel/fix-pzstd-makefile 
Fix pzstd Makefile to allow setting `DESTDIR` and `BINDIR` separately
 2023-10-07 19:32:22 -07:00
Yann ColletandGitHub 2b31cb0698 Merge pull request #3763 from dloidolt/fix_lib/README.md 
Fix a very small formatting typo in the lib/README.md file
 2023-10-07 19:31:36 -07:00
Yann ColletandGitHub 19317239ec Merge pull request #3772 from DimitriPapadopoulos/WIN32 
Do not test WIN32, instead test _WIN32
 2023-10-07 19:30:18 -07:00
Yann ColletandGitHub c1e588fcb4 Merge pull request #3771 from DimitriPapadopoulos/codespell 
Fix new typos found by codespell
 2023-10-07 19:29:41 -07:00
Yann ColletandGitHub 0e2cfebb63 Merge pull request #3777 from facebook/fix_x32 
fix x32 tests on Github CI
 2023-10-02 16:25:43 -07:00
Nick TerrellandNick Terrell 43118da8a7 Stop suppressing pointer-overflow UBSAN errors 
* Remove all pointer-overflow suppressions from our UBSAN builds/tests.
* Add `ZSTD_ALLOW_POINTER_OVERFLOW_ATTR` macro to suppress
  pointer-overflow at a per-function level. This is a superior approach
  because it also applies to users who build zstd with UBSAN.
* Add `ZSTD_wrappedPtr{Diff,Add,Sub}()` that use these suppressions.
  The end goal is to only tag these functions with
  `ZSTD_ALLOW_POINTER_OVERFLOW`. But we can start by annoting functions
  that rely on pointer overflow, and gradually transition to using
  these.
* Add `ZSTD_maybeNullPtrAdd()` to simplify pointer addition when the
  pointer may be `NULL`.
* Fix all the fuzzer issues that came up. I'm sure there will be a lot
  more, but these are the ones that came up within a few minutes of
  running the fuzzers, and while running GitHub CI.
 2023-09-28 17:35:05 -04:00
Nick TerrellandNick Terrell 3daed7017a Revert "Work around nullptr-with-nonzero-offset warning" 
This reverts commit c27fa39904.
 2023-09-28 17:35:05 -04:00
Yann Collet 2c17e05646 fix x32 tests on Github CI 
ubuntu-22.04 seems to have problems with x32 recently
switching to ubuntu-20.04 which seems to work fine so far

https://github.com/actions/runner-images/issues/8397
 2023-09-27 21:18:20 -07:00
Yann ColletandGitHub 7b762364cb Merge pull request #3774 from facebook/dependabot/github_actions/actions/checkout-4.1.0 
Bump actions/checkout from 4.0.0 to 4.1.0
 2023-09-24 23:34:06 -07:00
dependabot[bot]andGitHub d5cbae7c50 Bump actions/checkout from 4.0.0 to 4.1.0 
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.0.0 to 4.1.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/3df4ab11eba7bda6032a0b82a6bb43b11571feac...8ade135a41bc03ea155e62e844d188df1ea18608)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-09-25 05:18:07 +00:00
Dimitri Papadopoulos 585aaa0ed3 Do not test WIN32, instead test _WIN32 
To the best of my knowledge:
* `_WIN32` and `_WIN64` are defined by the compiler,
* `WIN32` and `WIN64` are defined by the user, to indicate whatever
  the user chooses them to indicate. They mean 32-bit and 64-bit Windows
  compilation by convention only.

See:
https://accu.org/journals/overload/24/132/wilson_2223/

Windows compilers in general, and MSVC in particular, have been defining
`_WIN32` and `_WIN64` for a long time, provably at least since Visual Studio
2015, and in practice as early as in the days of 16-bit Windows.

See:
https://learn.microsoft.com/en-us/cpp/preprocessor/predefined-macros?view=msvc-140
https://learn.microsoft.com/en-us/windows/win32/winprog64/the-tools

Tests used to be inconsistent, sometimes testing `_WIN32`, sometimes
`_WIN32` and `WIN32`. This brings consistency to Windows detection.
 2023-09-23 19:03:18 +02:00
Dimitri Papadopoulos fe34776c20 Fix new typos found by codespell 2023-09-23 18:56:01 +02:00
Nick TerrellandNick Terrell cdceb0fce5 Improve macro guards for ZSTD_assertValidSequence 
Refine the macro guards to define the functions exactly when they are
needed.

This fixes the chromium build with zstd.

Thanks to @GregTho for reporting!
 2023-09-22 16:36:14 -04:00
Dominik Loidolt 48b5a7bd8b Fix a very small formatting typo in the lib/README.md file 2023-09-19 16:22:47 +02:00
Yann ColletandGitHub 1c007e64b8 Merge pull request #3755 from facebook/estimate_doc 
added some documentation on ZSTD_estimate*Size() variants
 2023-09-18 22:32:40 -07:00
Yann Collet 3fc14e411b added some documentation on ZSTD_estimate*Size() variants 
as a follow up for #3747
 2023-09-13 11:35:19 -07:00
Yann ColletandGitHub 823b32ce43 Merge pull request #3749 from facebook/dependabot/github_actions/actions/checkout-4.0.0 
Bump actions/checkout from 3.5.3 to 4.0.0
 2023-09-12 13:58:08 -07:00
Yann ColletandGitHub 1ec201c7b9 Merge pull request #3745 from klausholstjacobsen/qnx-support 
Added qnx in the posix test section of platform.h
 2023-09-12 13:43:59 -07:00
Yann ColletandGitHub f2bc34d9b4 Merge pull request #3750 from facebook/dependabot/github_actions/actions/upload-artifact-3.1.3 
Bump actions/upload-artifact from 3.1.2 to 3.1.3
 2023-09-12 13:43:00 -07:00
Paul Menzel d55ebb5718 [pzstd]: Fix DESTDIR handling to allow setting BINDIR 
Currently, setting `BINDIR` and `DESTDIR` separately is not possible, so
the command below fails, as BINDIR is set explicitly:

    $ make -j80 install PREFIX=/usr EPREFIX=/usr BINDIR=/usr/bin SBINDIR=/usr/sbin LIBEXECDIR=/usr/libexec SYSCONFDIR=/etc SHAREDSTATEDIR=/var LOCALST ATEDIR=/var LIBDIR=/usr/lib INCLUDEDIR=/usr/include DATAROOTDIR=/usr/share DATADIR=/usr/share INFODIR=/usr/share/info LOCALEDIR=/usr/share/locale MAND IR=/usr/share/man DOCDIR=/usr/share/doc/zstd DESTDIR=/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/image -C contrib/pzstd DESTDIR=/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/image
    make: Entering directory
    '/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/source/contrib/pzstd' CFLAGS="  -I../../lib -I../../lib/common -I../../programs -I. -DNDEBUG -O3 -Wall -Wextra -Wno-deprecated-declarations   " LDFLAGS=" -O3 -Wall -Wextra -pedantic  " make -C ../../lib libzstd.a
    make[1]: Entering directory '/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/source/lib'
    make[1]: Leaving directory '/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/source/lib'
    g++ main.o ../../programs/util.o Options.o Pzstd.o SkippableFrame.o ../../lib/libzstd.a           -O3 -Wall -Wextra -pedantic    -pthread -o pzstd
    install -d -m 755 /usr/bin/
    install -m 755 pzstd /usr/bin/pzstd
    install: cannot create regular file '/usr/bin/pzstd': Permission denied
    make: *** [Makefile:116: install] Error 1

So, do not prefix `BINDIR` with `DESTDIR`, and adapt all paths for
installation. This is more common, and, for example, `programs/Makefile`
does the same.

Fixes: 8b4e84249b ("[pzstd] Fix Makefile")
 2023-09-12 19:15:25 +02:00
dependabot[bot]andGitHub d8b25cbf68 Bump actions/upload-artifact from 3.1.2 to 3.1.3 
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/0b7f8abb1508181956e8e162db84b466c27e18ce...a8a3f3ad30e3422c9c7b888a15615d19a852ae32)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-09-11 05:09:05 +00:00
dependabot[bot]andGitHub e0e309f27c Bump actions/checkout from 3.5.3 to 4.0.0 
Bumps [actions/checkout](https://github.com/actions/checkout) from 3.5.3 to 4.0.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/c85c95e3d7251135ab7dc9ce3241c5835cc595a9...3df4ab11eba7bda6032a0b82a6bb43b11571feac)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-09-11 05:09:02 +00:00
Yann ColletandGitHub 01dbbdf560 Merge pull request #3739 from JohanMabille/cmake 
Fixed zstd cmake shared build on windows
 2023-09-10 06:32:24 -07:00