1946 Commits
Author SHA1 Message Date
Elliot GorokhovskyandGitHub dc1f7b560b fix -Werror=pointer-arith in fuzzers (#3983) 2024-03-21 15:16:38 -04:00
Yann ColletandGitHub 6679d0ca7b Merge pull request #3982 from embg/fuzzer_readme
Document the process for adding a new fuzzer
2024-03-21 11:10:03 -07:00
Yann ColletandGitHub 1d3f664fce Merge pull request #3979 from yoniko/Werror-fuzz
Fail on errors when building fuzzers
2024-03-21 10:41:34 -07:00
Nick Terrell 731f4b70fc Fix & fuzz ZSTD_generateSequences
This function was seriously flawed:
* It didn't do output bounds checks
* It produced invalid sequences when an uncompressed or RLE block was emitted
* It produced invalid sequences when the block splitter was enabled
* It produced invalid sequences when ZSTD_c_targetCBlockSize was enabled

I've attempted to fix these issues, but this function is just a bad idea,
so I've marked it as deprecated and unsafe. We should replace it with
`ZSTD_extractSequences()` which operates on a compressed frame.
2024-03-21 07:18:05 -07:00
Elliot GorokhovskyandGitHub 741b87bbe1 Fuzzing and bugfixes for magicless-format decoding (#3976)
* fuzzing and bugfixes for magicless format

* reset dctx before each decompression

* do not memcmp empty buffers

* nit: decompressor errata
2024-03-20 19:22:34 -04:00
Elliot Gorokhovsky f62b2663b9 Add docs on how to add a new fuzzer 2024-03-19 14:05:23 -07:00
Yonatan Komornik 3487a60950 Fail on errors when building fuzzers
Fails on errors when building fuzzers with `fuzz.py` (adds `Werror`).
Currently allows `declaration-after-statement`, `c++-compat` and
`deprecated` as they are abundant in code (some fixes to
`declaration-after-statement` are presented in this commit).
2024-03-18 15:51:28 -07:00
Yonatan KomornikandGitHub 6a0052a409 Fix bugs in simple decompression fuzzer (#3978)
Fixes 2 issue in `simple_decompress.c`:
1. Wrong type used for storing the results of `ZSTD_findDecompressedSize` resulting in never matching to `ZSTD_CONTENTSIZE_ERROR` or `ZSTD_CONTENTSIZE_UNKNOWN`.

2. Experimental API is used (`ZSTD_findDecompressedSize`) without defining `ZSTD_STATIC_LINKING_ONLY`.
2024-03-18 15:36:40 -07:00
Elliot GorokhovskyandGitHub 7d970bd83c Implement one-shot fallback for magicless format (#3971) 2024-03-18 10:55:53 -04:00
W. Felix Handte 2215101cad Add a Few Tests 2024-03-13 16:32:04 -04:00
Yann ColletandGitHub 4bd911ae41 Merge pull request #3954 from facebook/zeroSeq_sample
Add the zeroSeq sample
2024-03-12 15:31:01 -07:00
Elliot GorokhovskyandGitHub f65b9e27ce Exercise ZSTD_findDecompressedSize() in the simple decompression fuzzer (#3959)
* Improve decompression fuzzer

* Fix legacy frame header fuzzer crash, add unit test
2024-03-12 17:07:06 -04:00
Elliot GorokhovskyandGitHub 92fbd42894 Export ZSTD_LEGACY_SUPPORT in tests/Makefile (#3955)
This doesn't affect most of the targets, but will help me sleep better at night knowing that future refactors won't break the legacy support.

Should have been included in https://github.com/facebook/zstd/pull/3943 but I noticed after that merged, so putting up a separate PR.
2024-03-12 14:36:54 -04:00
daniellerozenblitandGitHub 5a66afa051 Add common file extensions to --exclude-compressed (#3951) 2024-03-12 13:49:06 -04:00
Yann Collet 37ff4f91eb removed golden-decompression/.gitignore
replaced by an exclusion rule in tests/.gitignore
2024-03-12 10:47:27 -07:00
Yann Collet 0ae98ba215 add same .gitignore rule in golden-decompression/
as requested by @embg
2024-03-12 09:47:54 -07:00
Yann ColletandGitHub 8688c6ed92 Merge pull request #3942 from facebook/fix3719
Fix #3719 : mixing -c, -o and --rm
2024-03-12 09:44:52 -07:00
Yann ColletandGitHub ca0fc763e3 Merge pull request #3948 from facebook/tests_2927
add tests inspired from #2927
2024-03-12 09:43:59 -07:00
Yann Collet b39c76765b Add the zeroSeq sample
that should have been part of #3674
2024-03-12 09:28:25 -07:00
Yann ColletandGitHub fe7c645a7e Merge pull request #3937 from facebook/offset0
new method to deal with offset==0 erroneous edge case
2024-03-12 09:25:06 -07:00
Elliot GorokhovskyandGitHub e0872806df Use ZSTD_LEGACY_SUPPORT=5 in make test (#3943) 2024-03-12 10:08:26 -04:00
Yann Collet 74e856a195 add tests inspired from #2927
centered around -T# and --fast=# arguments
2024-03-11 17:57:57 -07:00
Yann Collet fbd9e628ae added tests 2024-03-11 12:17:34 -07:00
Yann Collet eb5f7a7fa2 produced golden sample for the offset==0 decoder test
is correctly detected as corrupted by new version,
and is accepted (changed into offset==1) by older version.

updated documentation accordingly, with an hexadecimal representation.
2024-03-09 00:33:44 -08:00
Yann Collet 2abe8d63e0 fix LLU->ULL
LLU is a correct prefix according to C99 & C11 standards (but not C90).
However, older versions of Visual Studio do not work with it.
Replace by ULL, which doesn't have this issue.

Fixes https://github.com/facebook/zstd/issues/3647
2024-03-04 00:16:01 -08:00
Yann ColletandGitHub e385c3dd46 Merge pull request #3753 from facebook/make2
minor Makefile refactoring
2024-03-03 19:13:00 -08:00
Yann Collet 6719794379 fixed some regressionTests
but not all
2024-02-23 18:48:29 -08:00
Yann Collet 3b40100058 fix long sequences (> 64 KB) 2024-02-23 15:35:12 -08:00
Yann Collet 6b11fc436c fix issue with incompressible sections 2024-02-23 14:53:56 -08:00
Yann Collet 1e240af30a fix datagen size control 2024-02-20 18:06:56 -08:00
Yann Collet 83598aa106 datagen generates lorem ipsum by default 2024-02-20 15:24:25 -08:00
Yann Collet 695d154cac fuzz: control debuglevel from Makefile
and make the compilation faster
2024-02-08 16:23:52 -08:00
Yann ColletandGitHub 06b5b37b8d Merge pull request #3895 from facebook/fix_3793
Improve compression of Arrays of Integers (High compression mode)
2024-02-05 20:12:24 -08:00
Yann Collet 887f5b62ae update compression results for regression tests 2024-02-05 01:27:22 -08:00
Yann Collet 641749fc09 fix uasan dictionary_stream_round_trip fuzz test 2024-02-05 00:36:10 -08:00
Yann Collet 9ae3bf5ee2 update compression results
good news: there are only improvements
2024-02-03 17:52:50 -08:00
Yann Collet a261375996 fix paramgrill Makefile recipe 2024-01-29 15:18:27 -08:00
Yann Collet 81f444f4f9 made playTests.sh more compatible with older versions of grep
replaced `\+` by `*`.
`\+` means `[1-N]`,
while `*` means `[0-N]`,
so it's not strictly equivalent
but `\+` happens to be badly supported on some flavors of grep,
and for the purpose of these tests, `*` is good enough.
2024-01-16 12:14:35 -08:00
Yann Collet e6f4b46493 playTests.sh does no longer needs grep -E
it makes the test script more portable across posix systems
because `grep -E` is not guaranteed
while `grep` is fairly common.
2024-01-15 11:16:46 -08:00
Yann ColletandGitHub 7f76d37044 Merge pull request #3850 from KapJI/better-errors
cli: better errors on argument parsing
2024-01-13 11:37:25 -08:00
Elliot GorokhovskyandGitHub c6cabf9441 Make offload API compatible with static CCtx (#3854)
* Add ZSTD_CCtxParams_registerSequenceProducer() to public API

* add unit test

* add docs to zstd.h

* nits

* Add ZSTDLIB_STATIC_API prefix

* Add asserts
2023-12-28 14:48:46 -05:00
Ruslan SayfutdinovandRuslan Sayfutdinov 8052cd0131 cli: better errors on arguent parsing 2023-12-18 13:59:33 +00:00
Yann Collet e8ff7d18eb removed FlexArray pattern from CCtxPool
within ZSTDMT_.
This pattern is flagged by less forgiving variants of ubsan
notably used during compilation of the Linux Kernel.

There are 2 other places in the code where this pattern is used.
This fixes just one of them.
2023-10-07 21:30:08 -07:00
Yann ColletandGitHub c1e588fcb4 Merge pull request #3771 from DimitriPapadopoulos/codespell
Fix new typos found by codespell
2023-10-07 19:29:41 -07:00
Nick TerrellandNick Terrell 43118da8a7 Stop suppressing pointer-overflow UBSAN errors
* Remove all pointer-overflow suppressions from our UBSAN builds/tests.
* Add `ZSTD_ALLOW_POINTER_OVERFLOW_ATTR` macro to suppress
  pointer-overflow at a per-function level. This is a superior approach
  because it also applies to users who build zstd with UBSAN.
* Add `ZSTD_wrappedPtr{Diff,Add,Sub}()` that use these suppressions.
  The end goal is to only tag these functions with
  `ZSTD_ALLOW_POINTER_OVERFLOW`. But we can start by annoting functions
  that rely on pointer overflow, and gradually transition to using
  these.
* Add `ZSTD_maybeNullPtrAdd()` to simplify pointer addition when the
  pointer may be `NULL`.
* Fix all the fuzzer issues that came up. I'm sure there will be a lot
  more, but these are the ones that came up within a few minutes of
  running the fuzzers, and while running GitHub CI.
2023-09-28 17:35:05 -04:00
Dimitri Papadopoulos fe34776c20 Fix new typos found by codespell 2023-09-23 18:56:01 +02:00
Yann Collet f4dbfce79c define LIB_SRCDIR and LIB_BINDIR 2023-09-12 13:46:03 -07:00
Yann ColletandGitHub 0fcb28c5d2 Merge pull request #3720 from QBos07/cygwin-msys2-support
Updated Makefiles for full MSYS2 and Cygwin installation and testing …
2023-08-22 16:29:34 -07:00
Yann Collet a07d7c4e29 added ZSTD_decompressDCtx() benchmark option to fullbench
useful to compare the difference between ZSTD_decompress
and ZSTD_decompressDCtx().
2023-08-16 10:43:39 -07:00
Quentin Boswank 78dbba76b8 Updated Makefiles for full MSYS2 and Cygwin installation and testing support.
They are Linux-like environments under Windows and have all the tools needed to support staged installation and testing.

Beware: this only affects the make build system.
2023-08-13 19:44:15 +02:00