Commit Graph
2043 Commits
Author SHA1 Message Date
Yonatan KomornikandGitHub 70df5de1b2 AsyncIO compression part 1 - refactor of existing asyncio code (#3021)
* Refactored fileio.c:
- Extracted asyncio code to fileio_asyncio.c/.h
- Moved type definitions to fileio_types.h
- Moved common macro definitions needed by both fileio.c and fileio_asyncio.c to fileio_common.h

* Bugfix - rename fileio_asycio to fileio_asyncio

* Added copyrights & license to new files

* CR fixes
2022-01-24 14:43:02 -08:00
Yann Collet feaaf7a6b1 slightly shortened status and summary lines in very verbose mode 2022-01-21 21:38:35 -08:00
binhdvoandGitHub 17017ac8db Change zstdless behavior to align with zless (#2909)
* Change zstdless behavior to align with zless
2022-01-21 19:57:19 -05:00
Yann Collet 24318093cc slightly shortened compression status update line
to fit within 80 columns limit.
2022-01-21 14:08:46 -08:00
Yonatan KomornikandGitHub 8ab95f24da Merge pull request #2985 from yoniko/zstd-output-file-buffer
ZSTD CLI: Use buffered output
2022-01-21 13:57:05 -08:00
Yonatan KomornikandGitHub 1598e6c634 Async write for decompression (#2975)
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
2022-01-21 13:55:41 -08:00
Nick TerrellandGitHub ef1eafded3 Merge pull request #3009 from terrelln/asm-flags
[build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
2022-01-18 15:48:11 -08:00
Nick Terrell 8ea3d57de4 [build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
* Add `-Wa,--noexecstack` to both `ASFLAGS` and `CFLAGS`
* Pass `ASFLAGS` to `.S` compilation instead of `CFLAGS`

Fixes #3006.
2022-01-18 15:11:29 -08:00
Mike Gilbert 57a86d9ec6 Avoid updating timestamps when the destination is stdout
Fixes: 9cd6c1ff4d
Fixes: https://github.com/facebook/zstd/issues/2997
2022-01-13 16:47:18 -05:00
Felix HandteandGitHub f4a552a3fa Merge pull request #2987 from felixhandte/prepare-v1.5.2
Prepare v1.5.2
2022-01-11 17:47:39 -05:00
Yann ColletandGitHub 78633272a0 Merge pull request #2982 from terrelln/decompress-progress-0
Fix stderr progress logging for decompression
2022-01-11 10:15:15 -08:00
Yonatan Komornik df5013b463 ZSTD CLI: Use buffered output for improved performance 2022-01-07 17:10:13 -08:00
Nick Terrell 308a11b8e8 Fix stderr progress logging for decompression
When decompressing with `-q` and an output file, the progress bar was mistakenly printed. This is a minimal fix, with a larger refactor to be stacked on top of it.

Fixes #2967.
2022-01-07 15:09:56 -08:00
W. Felix Handte e1323744b6 Update Docs 2022-01-07 14:14:26 -05:00
W. Felix Handte 4bd96a61f1 Avoid xxHash Dependency by Inlining
xxHash symbols are present in `libzstd.so`, but they are local and therefore
unavailable outside the lib. There are two possible solutions to the problem.
We could make those symbols global, or we could remove the dependency.

This commit chooses the latter approach. I suppose this comes at the cost of
code size / build time. I'm open to comments on whether this is a good thing
to do, especially since this will apply even when we are statically linking
everything.
2022-01-06 00:25:26 -05:00
W. Felix Handte fc946d131b Remove Unused Include 2022-01-06 00:25:26 -05:00
W. Felix Handte 4620ce6a9a Makefiles: Add noexecstack Options to Compilation and Linking
Hopefully this marks the binary artifacts `noexecstack` even on platforms
where binaries default to true.
2022-01-05 15:12:31 -05:00
Yann Collet abc694f53e update man pages 2021-12-20 14:24:06 -08:00
Elliot GorokhovskyandGitHub c5f1e826ca Merge pull request #2925 from embg/dict_training_sample_limit_size
Allow user to specify memory limit for dictionary training
2021-12-15 15:58:17 -05:00
Elliot Gorokhovsky 71c0c07c19 Allow user to specify memory limit for dictionary training 2021-12-14 14:29:01 -05:00
Felix HandteandGitHub 0c26d98c0d Merge pull request #2910 from felixhandte/reject-irregular-dicts
Reject Irregular Dictionary Files
2021-12-09 11:44:37 -05:00
W. Felix Handte 9985e10fda Reject Irregular Dictionary Files
I hadn't seen #2890, so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.

Addresses #2874.
2021-12-08 16:17:04 -05:00
binhdvoandGitHub 38dfc4699e Imply -q when stderr is not a tty (#2884)
* Imply -q when stderr is not a tty
2021-12-07 16:56:19 -05:00
Nick TerrellandGitHub e7b0ae385e Merge pull request #2890 from 15596858998/dec_1201
fixbug CLI's -D fails when the argument is not a regular file
2021-12-06 13:18:15 -05:00
Yann ColletandGitHub a134737ad4 Merge pull request #2687 from sapiippo/dev
Makefile: fix build for mingw
2021-12-02 10:53:02 -08:00
Alexander KanavinandAlexander Kanavin 1e514feec6 Makefile: sort all wildcard file list expansions
Otherwise the order is non-deterministic and breaks
reproducible builds.
2021-12-02 12:04:11 +01:00
Yann ColletandGitHub 3133d1e86e Merge pull request #2876 from 15596858998/dev
Solve the bug of extra output newline character
2021-12-01 15:10:08 -08:00
15596858998 a8adfa7f67 fixbug CLI's -D fails when the argument is not a regular file 2021-12-01 21:47:35 +08:00
binhdvoandGitHub d7e1736375 Fix build for cygwin/bsd (#2882) 2021-11-29 14:11:39 -05:00
binhdvoandGitHub 7abebc847b Clarify documentation for -c (#2883) 2021-11-29 14:10:43 -05:00
15596858998 e315a047c7 更新 fileio.c
Solve the logic problem of wrong output of newline characters.
2021-11-23 22:08:23 +08:00
Dimitris Apostolou ebbd675998 Fix typos 2021-11-13 10:04:04 +02:00
Kevin Svetlitski 7fbd126e08 Suppress spurious unused parameter warning 2021-11-12 10:44:26 -08:00
Kevin Svetlitski 375e3aad6c Ensure formatting directives for displaying size_t are portable 2021-11-11 13:17:32 -08:00
Kevin Svetlitski 365c91194c Ensure print*CParams functions are only defined when used 2021-11-11 12:14:56 -08:00
Kevin Svetlitski df9b7755cb Fix const-ness of FIO_displayCompressionParameters 2021-11-10 17:25:27 -08:00
Kevin Svetlitski 63fe6198ed Display --zstd= subparameters in command-line ready form in verbose mode 2021-11-10 17:20:43 -08:00
Kevin Svetlitski 0665d4c1c2 Display command line parameters with concrete values in verbose mode 2021-11-05 12:01:20 -07:00
Kevin Svetlitski b3888193d9 Report memory required to decompress while compressing in verbose mode 2021-11-04 16:25:40 -07:00
Yann Collet 23dd28df67 minor improvements to benchmark display 2021-10-26 23:23:30 -07:00
stanjo74andGitHub 52598d54e9 Limit train samples (#2809)
* Limit training samples size to 2GB

* simplified DISPLAYLEVEL() macro to use global vqriable instead of local.

* refactored training samples loading

* fixed compiler warning

* addressed comments from the pull request

* addressed @terrelln comments

* missed some fixes

* fixed type mismatch

* Fixed bug passing estimated number of samples rather insted of the loaded number of samples.
Changed unit conversion not to use bit-shifts.

* fixed a declaration after code

* fixed type conversion compile errors

* fixed more type castting

* fixed more type mismatching

* changed sizes type to size_t

* move type casting

* more type cast fixes
2021-10-04 17:47:52 -07:00
Yann ColletandGitHub 7868f38019 Merge pull request #2747 from Helflym/dev
Add AIX support in Makefile
2021-10-01 08:13:39 -07:00
Yann Collet 8150891939 regenerated zstdless.1 2021-09-22 14:48:51 -07:00
Yann Collet 3addf2f277 updated zstdgrep man page 2021-09-22 14:30:59 -07:00
Yann Collet 70b36c2308 update zstdgrep doc
to mention ripgrep alternative which transparently supports zstd-compressed files.
2021-09-22 14:30:06 -07:00
Yann Collet 999f8778af updated man pages
using ronn-ng
2021-09-22 14:18:24 -07:00
Yann Collet 2832bbbbdc emphasize usage of -r in documentation
notably as a way to overcome shell expansion limitations,
notably in a scenario of dictionary training.
2021-09-22 14:00:20 -07:00
senhuang42 b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
Nick Terrell 8bf699aa59 [build] Add support for ASM files in Make + CMake
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
  Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
  same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
  `make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.

The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.

The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
2021-09-17 14:13:53 -07:00
Yann Collet f58e63bee7 Merge branch 'dev' into opt_investigation 2021-09-12 01:42:49 -07:00