Commit Graph
2053 Commits
Author SHA1 Message Date
Dimitris Apostolou 4c4d403ecb Fix typos 2022-02-02 19:32:31 +02:00
Yonatan KomornikandGitHub cc0657f27d AsyncIO compression part 2 - added async read and asyncio to compression code (#3022)
* Compression asyncio:
- Added asyncio functionality for compression flow
- Added ReadPool for async reads, implemented in both comp and decomp flows
2022-01-31 15:43:41 -08:00
Nick TerrellandGitHub 0b70da6277 Merge pull request #3020 from terrelln/cli-tests
Add new CLI testing platform
2022-01-31 10:02:27 -08:00
Nick Terrell 8d65f87416 Fix static analysis false-positives
* It couldn't detect that the `fastCoverParams` can't be non-null, since it was just an assertion.
* It thought we were accesing `wksp->dtable` beyond the bounds because we were using it to set the `workSpace` value. Instead, compute the workspace size used in a different way.
2022-01-30 12:16:16 -08:00
Yann ColletandGitHub c9072dd60a Merge pull request #3027 from brailovich/dev
fix for -r on empty directory
2022-01-29 15:40:43 -08:00
Nick Terrell 495dcb839a [zstdcli] Fix option detection for --auto-threads
The option `--auto-threads` should still be accepted and parsed, even if
`ZSTD_MULTITHREAD` is not defined. It doesn't mean anything, but we
should still accept the option. Since we want scripts to be able to work
generically.

This bug was caught by tests I added to the new testing framework.
2022-01-27 13:56:59 -08:00
Nick Terrell 246982e782 [dibio] Fix assertion triggered by no inputs
Passing 0 inputs to `DiB_shuffle()` caused an assertion failure where
it should just return.

A test is added in a later commit, with the initial introduction of the
new testing framework.

Fixes #3007.
2022-01-27 13:56:59 -08:00
Nick Terrell e60eba58bf Print zlib/lz4/lzma library versions in verbose version output
Knowing the version of zlib/lz4/lzma we're linking against is very
useful for debugging issues with those libraries, so print it out in the
verbosity 4 version output.

Also print this information at the top of `playTests.sh`.
2022-01-26 18:25:58 -08:00
brailovichandGitHub beb4872241 Update zstdcli.c 2022-01-26 16:51:18 -08:00
brailovichandGitHub 4021b78437 fix for error message in recursive mode for an empty folder
-r on empty directory resulted in zstd waiting input from stdin. now zstd exits without error and prints a warning message explaining why no processing happened (no files or directories to process).
2022-01-24 17:42:21 -08:00
Yonatan KomornikandGitHub 70df5de1b2 AsyncIO compression part 1 - refactor of existing asyncio code (#3021)
* Refactored fileio.c:
- Extracted asyncio code to fileio_asyncio.c/.h
- Moved type definitions to fileio_types.h
- Moved common macro definitions needed by both fileio.c and fileio_asyncio.c to fileio_common.h

* Bugfix - rename fileio_asycio to fileio_asyncio

* Added copyrights & license to new files

* CR fixes
2022-01-24 14:43:02 -08:00
Yann Collet feaaf7a6b1 slightly shortened status and summary lines in very verbose mode 2022-01-21 21:38:35 -08:00
binhdvoandGitHub 17017ac8db Change zstdless behavior to align with zless (#2909)
* Change zstdless behavior to align with zless
2022-01-21 19:57:19 -05:00
Yann Collet 24318093cc slightly shortened compression status update line
to fit within 80 columns limit.
2022-01-21 14:08:46 -08:00
Yonatan KomornikandGitHub 8ab95f24da Merge pull request #2985 from yoniko/zstd-output-file-buffer
ZSTD CLI: Use buffered output
2022-01-21 13:57:05 -08:00
Yonatan KomornikandGitHub 1598e6c634 Async write for decompression (#2975)
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
2022-01-21 13:55:41 -08:00
Nick TerrellandGitHub ef1eafded3 Merge pull request #3009 from terrelln/asm-flags
[build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
2022-01-18 15:48:11 -08:00
Nick Terrell 8ea3d57de4 [build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
* Add `-Wa,--noexecstack` to both `ASFLAGS` and `CFLAGS`
* Pass `ASFLAGS` to `.S` compilation instead of `CFLAGS`

Fixes #3006.
2022-01-18 15:11:29 -08:00
Mike Gilbert 57a86d9ec6 Avoid updating timestamps when the destination is stdout
Fixes: 9cd6c1ff4d
Fixes: https://github.com/facebook/zstd/issues/2997
2022-01-13 16:47:18 -05:00
Felix HandteandGitHub f4a552a3fa Merge pull request #2987 from felixhandte/prepare-v1.5.2
Prepare v1.5.2
2022-01-11 17:47:39 -05:00
Yann ColletandGitHub 78633272a0 Merge pull request #2982 from terrelln/decompress-progress-0
Fix stderr progress logging for decompression
2022-01-11 10:15:15 -08:00
Yonatan Komornik df5013b463 ZSTD CLI: Use buffered output for improved performance 2022-01-07 17:10:13 -08:00
Nick Terrell 308a11b8e8 Fix stderr progress logging for decompression
When decompressing with `-q` and an output file, the progress bar was mistakenly printed. This is a minimal fix, with a larger refactor to be stacked on top of it.

Fixes #2967.
2022-01-07 15:09:56 -08:00
W. Felix Handte e1323744b6 Update Docs 2022-01-07 14:14:26 -05:00
W. Felix Handte 4bd96a61f1 Avoid xxHash Dependency by Inlining
xxHash symbols are present in `libzstd.so`, but they are local and therefore
unavailable outside the lib. There are two possible solutions to the problem.
We could make those symbols global, or we could remove the dependency.

This commit chooses the latter approach. I suppose this comes at the cost of
code size / build time. I'm open to comments on whether this is a good thing
to do, especially since this will apply even when we are statically linking
everything.
2022-01-06 00:25:26 -05:00
W. Felix Handte fc946d131b Remove Unused Include 2022-01-06 00:25:26 -05:00
W. Felix Handte 4620ce6a9a Makefiles: Add noexecstack Options to Compilation and Linking
Hopefully this marks the binary artifacts `noexecstack` even on platforms
where binaries default to true.
2022-01-05 15:12:31 -05:00
Yann Collet abc694f53e update man pages 2021-12-20 14:24:06 -08:00
Elliot GorokhovskyandGitHub c5f1e826ca Merge pull request #2925 from embg/dict_training_sample_limit_size
Allow user to specify memory limit for dictionary training
2021-12-15 15:58:17 -05:00
Elliot Gorokhovsky 71c0c07c19 Allow user to specify memory limit for dictionary training 2021-12-14 14:29:01 -05:00
Felix HandteandGitHub 0c26d98c0d Merge pull request #2910 from felixhandte/reject-irregular-dicts
Reject Irregular Dictionary Files
2021-12-09 11:44:37 -05:00
W. Felix Handte 9985e10fda Reject Irregular Dictionary Files
I hadn't seen #2890, so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.

Addresses #2874.
2021-12-08 16:17:04 -05:00
binhdvoandGitHub 38dfc4699e Imply -q when stderr is not a tty (#2884)
* Imply -q when stderr is not a tty
2021-12-07 16:56:19 -05:00
Nick TerrellandGitHub e7b0ae385e Merge pull request #2890 from 15596858998/dec_1201
fixbug CLI's -D fails when the argument is not a regular file
2021-12-06 13:18:15 -05:00
Yann ColletandGitHub a134737ad4 Merge pull request #2687 from sapiippo/dev
Makefile: fix build for mingw
2021-12-02 10:53:02 -08:00
Alexander KanavinandAlexander Kanavin 1e514feec6 Makefile: sort all wildcard file list expansions
Otherwise the order is non-deterministic and breaks
reproducible builds.
2021-12-02 12:04:11 +01:00
Yann ColletandGitHub 3133d1e86e Merge pull request #2876 from 15596858998/dev
Solve the bug of extra output newline character
2021-12-01 15:10:08 -08:00
15596858998 a8adfa7f67 fixbug CLI's -D fails when the argument is not a regular file 2021-12-01 21:47:35 +08:00
binhdvoandGitHub d7e1736375 Fix build for cygwin/bsd (#2882) 2021-11-29 14:11:39 -05:00
binhdvoandGitHub 7abebc847b Clarify documentation for -c (#2883) 2021-11-29 14:10:43 -05:00
15596858998 e315a047c7 更新 fileio.c
Solve the logic problem of wrong output of newline characters.
2021-11-23 22:08:23 +08:00
Dimitris Apostolou ebbd675998 Fix typos 2021-11-13 10:04:04 +02:00
Kevin Svetlitski 7fbd126e08 Suppress spurious unused parameter warning 2021-11-12 10:44:26 -08:00
Kevin Svetlitski 375e3aad6c Ensure formatting directives for displaying size_t are portable 2021-11-11 13:17:32 -08:00
Kevin Svetlitski 365c91194c Ensure print*CParams functions are only defined when used 2021-11-11 12:14:56 -08:00
Kevin Svetlitski df9b7755cb Fix const-ness of FIO_displayCompressionParameters 2021-11-10 17:25:27 -08:00
Kevin Svetlitski 63fe6198ed Display --zstd= subparameters in command-line ready form in verbose mode 2021-11-10 17:20:43 -08:00
Kevin Svetlitski 0665d4c1c2 Display command line parameters with concrete values in verbose mode 2021-11-05 12:01:20 -07:00
Kevin Svetlitski b3888193d9 Report memory required to decompress while compressing in verbose mode 2021-11-04 16:25:40 -07:00
Yann Collet 23dd28df67 minor improvements to benchmark display 2021-10-26 23:23:30 -07:00