Commit Graph
9696 Commits
Author SHA1 Message Date
Danielle Rozenblit 8bb833bb5a Merge branch 'null-buffer-decompress' of github.com:daniellerozenblit/zstd into null-buffer-decompress 2022-09-12 18:57:53 -07:00
Danielle Rozenblit e46b12e1b4 fix indentation 2022-09-12 18:56:59 -07:00
daniellerozenblitandGitHub f59f797aa8 Merge branch 'facebook:dev' into null-buffer-decompress 2022-09-12 14:54:36 -04:00
Danielle Rozenblit a1d89424c2 fuzzer error fix 2022-09-12 11:53:37 -07:00
daniellerozenblitandGitHub 74ece5037c Merge pull request #3262 from daniellerozenblit/sequence-bound
Add sequenceBound(srcSize) method
2022-09-09 17:40:42 -04:00
Danielle Rozenblit 1613caf8bd use ZSTD_sequenceBound in seqBench 2022-09-09 13:04:41 -07:00
daniellerozenblitandGitHub 88b49e923b Merge branch 'facebook:dev' into sequence-bound 2022-09-09 15:57:05 -04:00
Elliot GorokhovskyandGitHub 39ab02a71f Merge pull request #3257 from embg/seqBench2
Benchmark program for sequence compression API
2022-09-09 15:53:28 -04:00
Danielle Rozenblit aa82998821 add sequence bound function 2022-09-09 12:34:25 -07:00
Elliot GorokhovskyandGitHub 6600a05949 Merge pull request #3259 from DimitriPapadopoulos/codespell
Fix typos found by codespell
2022-09-09 11:15:05 -04:00
daniellerozenblitandGitHub c26aa5d0fa Merge pull request #3258 from daniellerozenblit/null-buffer-decompress
Fix undefined behavior in ZSTD_decompressStream()
2022-09-09 09:39:59 -04:00
Danielle Rozenblit a06e953db9 some additional comments, remove apt-get from clang jobs, better test titles 2022-09-08 18:30:07 -07:00
Dimitri Papadopoulos 0015308c0f Fix typos found by codespell 2022-09-08 23:17:00 +02:00
Danielle Rozenblit 3d7f9a90df skip flush operation in case where op is NULL 2022-09-08 13:53:13 -07:00
Danielle Rozenblit f3ddaaddd6 ternary operator instead of if statement 2022-09-08 12:59:49 -07:00
Danielle Rozenblit cf255cc5e0 remove 32-bit ubsan clang test (bug in clang that produces an error) 2022-09-08 12:04:57 -07:00
Elliot Gorokhovsky 61c79bf0d6 Benchmark program for sequence compression API 2022-09-08 09:20:50 -07:00
Danielle Rozenblit 66ed3df096 add clang jobs for ubsan in github workflow 2022-09-08 09:00:39 -07:00
Danielle Rozenblit 028842788b fix zero offset to nullpointer errors 2022-09-07 17:52:26 -07:00
Danielle Rozenblit fe22e8c538 revert change 2022-09-07 12:37:25 -07:00
Danielle Rozenblit ca78d101f7 do not recover pointer overflow for testing 2022-09-07 12:21:54 -07:00
Danielle Rozenblit bc7492cefa added zstreamtest_ubsan to make file + added ubsan zstreamtest job for CI tests 2022-09-07 11:54:17 -07:00
Danielle Rozenblit 282a955d33 added test that exposes zero offset to null pointer error when built with clang 2022-09-07 08:58:08 -07:00
Danielle Rozenblit 4a6783bbbb remove zstream_ubsan from git and add to gitignore 2022-09-07 07:40:43 -07:00
Danielle Rozenblit 69022ad886 null decompress buffer test and ubsan flag added 2022-09-06 14:34:55 -07:00
Yann ColletandGitHub 155d6a58a2 Merge pull request #3241 from wahern/wahern-combine-sh-faster
restore combine.sh bash performance while still sticking to POSIX
2022-08-16 16:09:43 -07:00
Elliot GorokhovskyandGitHub ef60302af9 Merge pull request #3230 from grossws/fix3229-docs
Add description for ZSTD_decompressStream and ZSTD_initDStream
2022-08-16 12:48:23 -04:00
Chris BurgessandGitHub 3b4e47092e Document pass-through behavior (#3242)
Adds documentation to help and man pages for legacy pass-through behavior
when force is set and destination is stdout. Documents --pass-through in
man pages
2022-08-15 10:29:54 -07:00
William Ahern 60fcc36af6 escape glob pattern special characters in subject string before generating search patterns in combine.sh list_has_item 2022-08-10 20:58:55 -07:00
William Ahern cca3544708 restore combine.sh bash performance while still sticking to POSIX 2022-08-10 16:51:17 -07:00
Yann ColletandGitHub 1b249cf075 Merge pull request #3235 from facebook/docTraining
[easy] added a few documentation words about dictionary training
2022-08-08 12:06:49 -07:00
Konstantin Gribov 1c847e2e32 Add description for ZSTD_decompressStream and ZSTD_initDStream
With that these functions become visible in generated docs.

Fixes #3229
2022-08-08 18:02:50 +03:00
Yann Collet 3f7a1b1328 added a few documentation words about dictionary training
partially answering questions such as #3233
which looks for guidance within `exmaples/`.
2022-08-05 17:09:22 +02:00
Nick TerrellandGitHub 03cc84fddb Add explicit --pass-through flag and default to enabled for *cat (#3223)
Fixes #3211.

Adds the `--[no-]pass-through` flag which enables/disables pass-through mode.

* `zstdcat`, `zcat`, and `gzcat` default to `--pass-through`.
  Pass-through mode can be disabled by passing `--no-pass-through`.
* All other binaries default to not setting pass-through mode.
  However, we preserve the legacy behavior of enabling pass-through
  mode when writing to stdout with `-f` set, unless pass-through
  mode is explicitly disabled with `--no-pass-through`.

Adds a new test for this behavior that should codify the behavior we want.
2022-08-04 17:15:59 -07:00
d0dcc9d775 fix issue #3144 (#3226)
* fix issue #3144

* add test case for verbose-wlog

Co-authored-by: zengyijing <yijingzeng@fb.com>
2022-08-04 13:51:14 -07:00
Yann ColletandGitHub e818fa8eb0 Merge pull request #3232 from facebook/fileiotypes_nomemh
fileio_types.h : avoid dependency on mem.h
2022-08-03 22:57:16 +02:00
Yann ColletandGitHub 9e90b180c5 Merge pull request #3231 from facebook/fileio_missingInclude
[easy] fixed missing include
2022-08-03 22:48:05 +02:00
Yann Collet 3dfcafacd7 fileio_types.h : avoid dependency on mem.h
fileio_types.h cannot be parsed by itself
because it relies on basic types defined in `lib/common/mem.h`.
As for #3231, it likely wasn't detected because `mem.h` was probably included before within target files.
But this is not proper.

A "easy" solution would be to add the missing include,
but each dependency should be considered "bad" by default,
and only allowed if it brings some tangible value.

In this case, since these types are only used to declare internal structure variables
which are effectively only flags,
I believe it's really not valuable to add a dependency on `mem.h` for this purpose
while the standard `int` type can do the same job.

I was expecting some compiler warnings following this change,
but it turns out we don't use `-Wconversion` by default on `zstd` source code,
so there is none.

Nevertheless, I enabled `-Wconversion` locally and proceeded to fix a few conversion warnings in the process.

Adding `-Wconversion` to the list of flags used for `zstd` is something I would be favorable over the long term,
but it cannot be done overnight,
because the nb of places where this warning is triggered is daunting.
Better progressively reduce the nb of triggered `-Wconversion` warnings before enabling this flag by default.
2022-08-03 21:39:35 +02:00
Yann Collet a925362534 minor : fixed missing include
I presume it was not detected before
because "fileio.h" is probably always included after "util.h".
2022-08-03 20:52:15 +02:00
Nick TerrellandGitHub a70ca2bd7d Fix off-by-one error in superblock mode (#3221)
Fixes #3212.

Long literal and match lengths had an off-by-one error in ZSTD_getSequenceLength.
Fix the off-by-one error, and add a golden compression test that catches the bug.
Also run all the golden tests in the cli-tests framework.
2022-08-03 11:28:39 -07:00
Felix HandteandGitHub 7e6278a706 Merge pull request #3196 from mileshu/dev
[T124890272] Mark 2 Obsolete Functions(ZSTD_copy*Ctx) Deprecated in Zstd
2022-08-02 12:34:04 -04:00
Miles Hu 201f2e339b Merge branch 'dev' of https://github.com/mileshu/zstd into dev 2022-08-01 22:52:47 -07:00
Miles HU c450f9f952 [T124890272] Mark 2 Obsolete Functions(ZSTD_copy*Ctx) Deprecated in Zstd
The discussion for this task is here: facebook/zstd#3128.

This task can probably be scoped to the first part: marking these functions deprecated.
We'll later look at removal when we roll out v1.6.0.
2022-08-01 22:45:52 -07:00
Nick TerrellandGitHub 0f4fd28a64 Deprecate ZSTD_getDecompressedSize() (#3225)
Fixes #3158.

Mark ZSTD_getDecompressedSize() as deprecated and replaced by ZSTD_getFrameContentSize().
2022-08-01 11:52:14 -07:00
Elliot GorokhovskyandGitHub 28ceb63503 Merge pull request #3220 from embg/issue3200
Disallow empty string as argument for --output-dir-flat and --output-dir-mirror
2022-08-01 14:04:57 -04:00
Qiongsi WuandGitHub 1b445c1c2e Fix hash4Ptr for big endian (#3227) 2022-08-01 10:41:24 -07:00
Yonatan KomornikandGitHub ae4670466c stdin multiple file fixes (#3222)
* Fixes for https://github.com/facebook/zstd/issues/3206 - bugs when handling stdin as part of multiple files.

* new line at end of multiple-files.sh
2022-07-29 16:13:07 -07:00
Elliot Gorokhovsky f9f27de91c Disallow empty output directory 2022-07-29 14:48:33 -07:00
Tom WangandGitHub d4a5bc4efc Add warning when multi-thread decompression is requested (#3208)
When user pass in argument for both decompression and multi-thread, print a warning message
to indicate that multi-threaded decompression is not supported.

* Add warning when multi-thread decompression is requested
* add test case for multi-threaded decoding warning
   Expectation is for -d -T0 we will not throw any warning,
   and see warning for any other -d -T(>1) inputs
2022-07-29 12:51:58 -07:00
Chris BurgessandGitHub 2b9fde932b Fix small file passthrough (#3215) 2022-07-29 12:22:46 -07:00