ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,656 Commits 17 Branches 77 Tags
dev
Commit Graph
2223 Commits
Author SHA1 Message Date
Sean Purcell 0f5c95af44 Disambiguate pledgedSrcSize == 0 
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
 2017-02-08 15:12:46 -08:00
Sean Purcell 4e709712e1 Decompressed size functions now handle multiframes and distinguish cases 
- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
 2017-02-08 14:50:10 -08:00
Przemyslaw Skibinski 93901fe85c remove redundant "ifeq ($(HAVE_ZLIB), 1)" 2017-02-08 21:11:18 +01:00
Przemyslaw Skibinski 4f9eaa7bb3 fixed gcc warnings 2017-02-08 18:08:09 +01:00
Przemyslaw Skibinski cb56306a50 .gz suffix for gzip compressed files 2017-02-08 17:37:14 +01:00
Przemyslaw Skibinski 02018c83cf added FIO_compressGzFrame 2017-02-08 16:54:23 +01:00
Przemyslaw Skibinski cfd4dc299a add "--format=gzip" option 2017-02-08 15:17:55 +01:00
Przemyslaw Skibinski ca20edd960 fixed zlib detection with MinGW 2017-02-08 14:32:49 +01:00
Przemyslaw Skibinski cdf5a7bd9f Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11 2017-02-08 13:49:35 +01:00
Przemyslaw Skibinski 40580ff669 added description of "--priority=rt" 2017-02-08 13:49:06 +01:00
Przemyslaw Skibinski 0665a359aa "--rt-prio" renamed to "--priority=rt" 2017-02-07 20:12:59 +01:00
Przemyslaw Skibinski d05014c739 added the "--rt-prio" option 2017-02-07 16:48:01 +01:00
Przemyslaw Skibinski 94abd6a26c SET_REALTIME_PRIORITY 2017-02-07 16:36:19 +01:00
Nick Terrell 7e3fc73795 Ensure <zlib.h> can be included in HAVE_ZLIB test 2017-02-06 11:54:31 -08:00
Nick Terrell 2cb8ee8784 Change zlib include to be a system include 2017-02-06 11:32:13 -08:00
Yann ColletandGitHub f6e9c4f1e1 Merge pull request #529 from facebook/gzstd_default 
Gzstd default
 2017-02-05 18:03:36 -08:00
cyan4973 613087c02b Silence zlib detection routine 
When it fails, $(CC) sends error message into stderr
redirected to /dev/null
 2017-02-04 23:36:12 -08:00
Nick Terrell b4016ff02f Add cover dictionary training to zstd.1 
Tested with `make install && man zstd` and visual inspection.
 2017-02-03 16:42:07 -08:00
Yann Collet 762ddeeb9e fixed zstdmt compilation under Windows minGW/MSYS2, by @inikep 2017-02-03 14:35:42 -08:00
Yann Collet 21eb80d485 remove zlib detection artefact 
result of compilation test is sent to /dev/null
 2017-02-03 14:34:41 -08:00
Yann Collet b02ac8d613 fixed pointer conversion warnings (C++) 
in gz module
 2017-02-03 08:43:06 -08:00
Yann Collet c2a4632789 release builds use less debug symbols and warnings 
release build are triggered through either `make`,
or their specific target `make zstd-release` and `make lib-release`.
 2017-02-02 20:54:41 -08:00
Yann Collet c3cba9d858 fixed silent conversion warnings in GZDECOMPRESS path 2017-02-02 17:12:50 -08:00
Yann Collet 030ac243a0 Changed Makefile to generate zstd with .gz support by default 
.gz support is detected by a runtime test.
 2017-02-02 16:49:34 -08:00
Yann Collet 3dc85bae66 minor : fixed zstd-frugal 
fixed a minor unused variable warning when compiling zstd-frugal target
 2017-02-02 16:19:09 -08:00
Yann ColletandGitHub 48bed91606 Merge pull request #527 from facebook/zstdmt 
zstdmt refinements
 2017-01-31 16:36:46 -08:00
Yann Collet 8d8513fb64 fixed C constant restrictions 2017-01-30 14:37:08 -08:00
Yann Collet 3672d06d06 zstdmt : section size is set to be a minimum of overlapSize 
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
 2017-01-30 13:35:45 -08:00
Yann Collet 6ccd37c8d4 cli : added advanced parameter overlapLog 
as a hidden (undocumented) parameter for now
 2017-01-30 13:07:24 -08:00
Yann Collet 6be2337c26 added command --block-size= 
for Multi-threading only.
alias : -B#
 2017-01-30 11:17:26 -08:00
Yann ColletandGitHub cc3d1bc262 Merge pull request #525 from terrelln/covermt 
Multithreaded COVER dictionary training
 2017-01-30 10:15:33 -08:00
Yann Collet 5d9b894e46 Fixed status display for zstdmt 
There is a large buffering effect when using zstdmt in MT mode.
Consequently, data is read first, pushed to workers,
and only later will the compressed result come out.
That means there is no longer immediate correlation
between amount of data read, and amount of data written.

This patch disables the displaying of % compression
when multi-threading is enabled.

It adds the displaying of total size when it can be determined
(it usually can be determined for files, but not for stdin)
so the user has a sense of "how far from the end" the compression compressed is.

There is no modification to decompression side,
since decompression is only single-threaded for now.
 2017-01-27 13:30:18 -08:00
Nick Terrell 2fe9126591 Add multithread support to COVER 2017-01-27 11:56:02 -08:00
Nick Terrell 9c018cc140 Add BUCK files for Nuclide support 2017-01-27 10:43:12 -08:00
Przemyslaw SkibinskiandNick Terrell eb2d23a90c improved #232 fix 2017-01-27 10:43:03 -08:00
Przemyslaw SkibinskiandNick Terrell 92a4dbf2e4 Fixed https://github.com/facebook/zstd/issues/232 2017-01-27 10:43:03 -08:00
Nick Terrell 83c387eb8e Fix zstdmt_compress.h include 2017-01-26 15:25:32 -08:00
Yann ColletandGitHub 717c65d690 Merge pull request #519 from inikep/dev11 
Dev11
 2017-01-26 14:23:44 -08:00
Yann Collet 8dafb1acf5 CLI : automatically set overlap size to max (windowSize) for max compression level 2017-01-25 17:01:13 -08:00
Yann Collet bb0027405a fixed zstdmt corruption issue when enabling overlapped sections 
see Asana board for detailed explanation on why and how to fix it
 2017-01-25 16:25:38 -08:00
Yann Collet 943cff9c37 fixed zstdmt cli freeze issue with large nb of threads 
fileio.c was continually pushing more content without giving a chance to flush compressed one.
It would block the job queue when input data was accumulated too fast (requiring to define many threads).
Fixed : fileio flushes whatever it can after each input attempt.
 2017-01-25 12:35:19 -08:00
Przemyslaw Skibinski 5022a18d51 improved #232 fix 2017-01-25 13:11:26 +01:00
Przemyslaw Skibinski 64fa2dbc5e Fixed https://github.com/facebook/zstd/issues/232 2017-01-25 13:02:33 +01:00
Yann Collet 512cbe8c10 zstdmt cli and API allow selection of section sizes 
By default, section sizes are 4x window size.
This new setting allow manual selection of section sizes.
The larger they are, the (slightly) better the compression ratio,
but also the higher the memory allocation cost,
and eventually the lesser the nb of possible threads,
since each section is compressed by a single thread.

It also introduces a prototype to set generic parameters,
ZSTDMT_setMTCtxParameter()

The idea is that it's possible to add enums
to extend the list of parameters that can be set this way.
This is more long-term oriented than a fixed-size struct.
Consider it as a test.
 2017-01-24 17:08:53 -08:00
Yann Collet f8804d1014 convert tabs to space 
joys of using multiple editors from multiple environments ...
 2017-01-20 17:23:19 -08:00
Yann Collet d7e3cb58c5 Resolved merge conflict dev+zstdmt 2017-01-20 16:44:50 -08:00
cyan4973 5fba09fa41 updated util's time for Windows compatibility 
Correctly measures time on Posix systems when running with
Multi-threading

Todo : check Windows measurement under multi-threading
 2017-01-20 12:57:31 -08:00
Yann Collet 458c8a94b4 minor refactoring : cleaner MT integration within bench 2017-01-19 17:44:15 -08:00
Yann Collet 500014af49 zstd cli can now compress using multi-threading 
added : command -T#
added : ZSTD_resetCStream() (zstdmt_compress)
added : FIO_setNbThreads()  (fileio)
 2017-01-19 17:04:28 -08:00
Yann Collet 0f984d94c4 changed MT enabling macro to ZSTD_MULTITHREAD 2017-01-19 14:05:07 -08:00