ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,624 Commits 17 Branches 77 Tags
cd9d0a7e6e5aa770b377d738eb94de6671c5487e
Commit Graph
4226 Commits
Author SHA1 Message Date
senhuang42 3327932609 Update ZSTD_getSequences function signature 2020-11-02 10:17:59 -05:00
senhuang42 7397d0102f Add new enum for different sequence formats for ingestion/extraction 2020-11-02 10:15:53 -05:00
Nick TerrellandGitHub 7205e609a9 Merge pull request #2354 from terrelln/stable-buffer 
Add ZSTD_c_stable{In,Out}Buffer and optimize when set
 2020-10-30 15:06:56 -07:00
senandGitHub c37c714ef1 Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api 
Refine external ZSTD_Sequence API
 2020-10-30 15:47:25 -04:00
Nick Terrell d4e021fe35 [lib] Avoid allocating the input buffer when ZSTD_c_stableInBuffer is set 
We don't use it when we have a stable input buffer, so don't allocate
it. I had to slightly modify `ZSTD_copyCCtx()` by storing the
`ZSTD_buffered_policy_e` in the `ZSTD_CCtx`, since `inBuffSize > 0` is
no longer the correct signal for the buffered mode.
 2020-10-30 10:55:34 -07:00
Nick Terrell 24f72789e2 [lib] Skip the input window buffer when ZSTD_c_stableInBuffer is set 
Compress directly from the `ZSTD_inBuffer`. We still allocate the input
buffer. A following commit will remove that allocation.
 2020-10-30 10:55:34 -07:00
Nick Terrell 6bd6b6f7d3 [cwksp] Return NULL when 0 bytes are requested 
This ensures that the buffer is never used.
 2020-10-30 10:55:34 -07:00
Nick Terrell fcf81cee5e [lib] Avoid allocating output buffer when ZSTD_c_stableOutBuffer is set 
We compress directly to the `ZSTD_outBuffer` so we don't need to
allocate it.
 2020-10-30 10:55:34 -07:00
Nick Terrell 6d5dc93d4e [lib] Compress directly into output when ZSTD_c_stableOutBuffer is set 
When we have a stable output buffer always compress directly into the
`ZSTD_outBuffer`. We are allowed to return `dstSizeTooSmall`.
 2020-10-30 10:55:34 -07:00
Nick Terrell 987cb4ca6a [lib] Take the shortcut when ZSTD_c_stableOutBuffer is set 
When we have a stable output buffer take the single-pass shortcut.
It is okay to return `dstSizeTooSmall` if the output buffer isn't
big enough, because we know it will never grow.
 2020-10-30 10:55:34 -07:00
Nick Terrell 809b2f2071 [lib] Set ZSTD_c_stable{In,Out}Buffer in ZSTD_compress2() 
Sets these parameters in ZSTD_compress2() then resets them to their
orignal values after the compression call.

An alternative design could be to add a flush mode `ZSTD_e_singlePass`
which implies `ZSTD_c_stable{In,Out}Buffer` but only for a single
compression call, by directly setting the applied parameters. I've opted
for the smaller change, but this is open for discussion.
 2020-10-30 10:55:34 -07:00
Nick Terrell c74be3f6de [lib] Validate buffers when ZSTD_c_stable{In,Out}Buffer is set 
Adds the validation of the input/output buffers only. They are still
unused.
 2020-10-30 10:55:34 -07:00
Nick Terrell e3e0775cc8 [API] Add ZSTD_c_stable{In,Out}Buffer parameters 
This commit adds the parameters and sets the value in the CCtxParams
but it does not do anything with the value.
 2020-10-30 10:54:39 -07:00
Nick Terrell e2581d9572 [lib] Set appliedParams in zstdmt mode 
Previously only `nbWorkers` was set. Set all parameters, because that is
what is expected. This is needed for the `ZSTD_c_stable{In,Out}Buffer`
parameters.
 2020-10-30 10:54:38 -07:00
senhuang42 f0da97642a Specify that getSequences() will always emit block boundary sequences 2020-10-30 12:31:17 -04:00
senhuang42 536e89c723 Sequence extractor should update CBlockState 2020-10-30 12:13:19 -04:00
senhuang42 32cac2627a Emit last literals of 0 size as well, to indicate block boundary 2020-10-29 16:41:17 -04:00
senhuang42 69bd5f0654 Correct literalsRead calculation to include longLength 2020-10-29 14:49:37 -04:00
senhuang42 59624f3163 Remove implicit typecast to appease appVeyor windows build 2020-10-28 16:25:09 -04:00
Yann Collet 09e3bb95d2 Merge branch 'dev' into libzstd_autoconf_full 2020-10-28 10:53:08 -07:00
Yann Collet 0adce4631d Merge branch 'libzstd_autoconf_full' of github.com:facebook/zstd into libzstd_autoconf_full 2020-10-28 10:25:55 -07:00
Yann Collet f6ecf1568f minor Makefile refactor 
hopefully improving readability
 2020-10-28 09:39:15 -07:00
senhuang42 3ed5d053d8 Clarify comments in zstd.h some more 2020-10-28 09:53:09 -04:00
Nick TerrellandGitHub 599ff58e08 Merge pull request #2339 from terrelln/zstdmt-stability 
Fix zstdmt stability issues and clean up the zstdmt code
 2020-10-27 19:43:13 -07:00
Yann Collet ceccd7ae2d Merge branch 'dev' into libzstd_autoconf_full 2020-10-27 15:45:30 -07:00
Yann ColletandGitHub 2d2507b9db Merge pull request #2374 from bket/portability 
'head -c BYTES' is non-portable
 2020-10-27 14:15:35 -07:00
senandGitHub 17b700d78a Merge pull request #2366 from senhuang42/enable_ldm_by_default 
Enable LDM by default if window size >= 128MB and strategy uses opt parser
 2020-10-27 14:59:28 -04:00
Nick TerrellandGitHub 0953645837 Merge pull request #2362 from senhuang42/fix_ldm_fuzz_issue 
Fix long distance matcher OSS-fuzz issue
 2020-10-27 11:13:03 -07:00
senhuang42 3163909d14 Remove unused variable position 2020-10-27 12:58:12 -04:00
senhuang42 dc448563e9 Add test compatibility with last literals in sequences 2020-10-27 12:35:28 -04:00
Björn Ketelaars 1f661b5f6b 'head -c BYTES' is non-portable 2020-10-27 16:55:23 +01:00
senhuang42 1d221ecc03 Add support for representing last literals in the extracted seqs 2020-10-27 11:19:48 -04:00
senhuang42 9171f920cd Improve documentation of seqStore_t 2020-10-27 10:50:22 -04:00
senhuang42 96b0ff7886 Improve documentation regarding various operations in copyBlockSequences 2020-10-27 10:36:06 -04:00
senhuang42 3a11c7eb03 Modify ZSTD_copyBlockSequences to agree with new API 2020-10-27 10:31:40 -04:00
senhuang42 761f40d1c6 Clarify and modify ZSTD_Sequence definition 2020-10-27 09:41:32 -04:00
Yann Collet 456db0c377 make install only rebuild binaries if they don't exist 
Now `make` followed by `make install` doesn't rebuild binaries

also : only generated target directories if they don't already exist
 2020-10-23 16:46:49 -07:00
Yann Collet a6ee614a44 make zstd is now differentiated from zstd-nomt 
avoids mixing object files using different flags
 2020-10-23 16:08:21 -07:00
Yann Collet 89b961ea46 simplified silent mode maintenance 2020-10-23 10:41:17 -07:00
Yann Collet ffe8d9e428 fix partial lib test 2020-10-23 10:27:12 -07:00
Yann Collet b5d4728713 simplified silent mode 2020-10-23 10:22:52 -07:00
Yann Collet a7ad05bf57 fixed building libzstd with manual BUILD_DIR 
and when HASH is not found
 2020-10-23 10:14:04 -07:00
Yann Collet d3f1a9b5bd fix partial-build test 
sometimes, the scope difference is solely determined by the list of source files,
not by the flags.
 2020-10-22 21:36:09 -07:00
Yann Collet a912ef0952 can integrate later dynamic flags changes 
for example `libzstd-mt` is `differentiated from `libzstd`
 2020-10-22 18:48:06 -07:00
Yann ColletandGitHub f90424da2d Merge pull request #2368 from facebook/progressive_libzstd 
faster rebuild of libzstd
 2020-10-22 17:36:56 -07:00
Yann Collet ce6cd07c33 updated build documentation 2020-10-22 12:31:23 -07:00
Yann Collet e3867fb735 fixed libzstd.dll compilation on mingw 
and zstd linking
 2020-10-22 11:52:19 -07:00
Yann Collet 494f7169ed fix directory creation for Windows' libzstd 2020-10-22 00:15:31 -07:00
Yann Collet dd24496951 programs/zstd also automatically generate object dir per conf 
same rules as lib/libzstd
can also be controlled via HASH and BUILD_DIR
 2020-10-21 23:38:33 -07:00
Yann Collet 0f8ee5c51e Merge branch 'dev' into libzstd_autoconf 2020-10-21 22:36:09 -07:00