ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,434 Commits 17 Branches 77 Tags
cbff372d105cb0593af24aba780481c83ef8d2ab
Commit Graph
1690 Commits
Author SHA1 Message Date
Yann Collet af3d9c506e added streaming test starting from non-0 pos 2022-01-26 10:31:25 -08:00
Yann Collet c1668a00d2 fix extended case combining stableInBuffer with continue() and flush() modes 2022-01-26 10:31:25 -08:00
Yann Collet 270f9bf005 better consistency in accessing @input 
as suggested by @terrelln.

Also : commented zstreamtest more
to ensure ZSTD_stableInBuffer is tested/
 2022-01-26 10:31:24 -08:00
Yann Collet 27d336b099 minor behavior refinements 
specifically, there is no obligation to start streaming compression with pos=0.
stableSrc mode is now compatible with this setup.
 2022-01-26 10:31:24 -08:00
Yann Collet 37b87add7a make stableSrc compatible with regular streaming API 
including flushStream().

Now the only condition is for `input.size` to continuously grow.
 2022-01-26 10:31:24 -08:00
Yann ColletandGitHub cc7d23bcec Merge pull request #2965 from facebook/offbase 
Converge sumtype (offset | repcode) numeric representation towards offBase
 2022-01-24 15:47:42 -08:00
binhdvoandGitHub 17017ac8db Change zstdless behavior to align with zless (#2909) 
* Change zstdless behavior to align with zless
 2022-01-21 19:57:19 -05:00
Yonatan KomornikandGitHub 1598e6c634 Async write for decompression (#2975) 
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
 2022-01-21 13:55:41 -08:00
Yann ColletandGitHub 71921e596f Merge pull request #2983 from facebook/minLitPricev2 
[opt] minor compression ratio improvement
 2022-01-20 16:02:31 -08:00
Nick Terrell 8ea3d57de4 [build][asm] Pass ASFLAGS to the assembler instead of CFLAGS 
* Add `-Wa,--noexecstack` to both `ASFLAGS` and `CFLAGS`
* Pass `ASFLAGS` to `.S` compilation instead of `CFLAGS`

Fixes #3006.
 2022-01-18 15:11:29 -08:00
Yann ColletandGitHub 05e884f288 Merge pull request #2956 from sunwire/fix-tar-test-cases 
Fix tar test cases
 2022-01-11 10:16:34 -08:00
Yann Collet 5595aec629 updated regression results 2022-01-07 15:08:06 -08:00
Nick Terrell c7b03c217c [license] Fix license header of huf_decompress_amd64.S 
* Add the license header for `huf_decompress_amd64.S`
* Add `.S` files to the `test-license.py` test
 2022-01-07 09:35:27 -08:00
Felix HandteandGitHub 7e679511a8 Merge pull request #2964 from felixhandte/noexecstack-all-archs 
Mark Huffman Decoder Assembly `noexecstack` on All Architectures
 2022-01-05 16:52:39 -05:00
W. Felix Handte ff5d1daf33 Clean Up Debugging Statements 2022-01-05 16:13:00 -05:00
W. Felix Handte 35208f702f Add Test Validating Stack is not Executable in playTests.sh 2022-01-04 16:57:35 -05:00
Yann Collet a0b9520e38 fullbench: added compress_freshCCtx scenario 2022-01-02 08:52:33 -08:00
Yann Collet 213dc6110f fixed fullbench freshCCtx scenario 2022-01-01 23:46:49 -08:00
Yann Collet 03903f5701 fixed minor compression difference in btlazy2 
subtle dependency on sumtype numeric representation
 2021-12-29 18:51:03 -08:00
Yann Collet 7a18d709ae updated all names to offBase convention 2021-12-29 17:30:43 -08:00
Yann Collet 8da414231d found a few more places which were dependent on seqStore offcode sumtype numeric representation 2021-12-28 17:03:24 -08:00
Yann Collet 681c81f06c abstracted storeSeq() sumtype numeric representation from decodecorpus.c 2021-12-28 11:58:33 -08:00
Yann Collet 435f5a2e6d fixed regression test assert 
optLdm->offset might be == 0 in invalid case.
Only use STORE_OFFSET() after validating it's a correct case.
 2021-12-28 09:55:31 -08:00
Paweł Marciniak 666372c7bf Fix tar test cases 2021-12-24 15:05:26 +01:00
Yann Collet 1aed962216 introduce macros STORE_OFFSET() and STORE_REPCODE() 
this meant to abstract the sumtype representation required
to transfert `offcode` to `ZSTD_storeSeq()`.

Unfortunately, the sumtype numeric representation is currently a leaky abstraction
that has permeated many other parts of the code,
especially within `zstd_lazy.c` and also within `zstd_opt.c` and `zstd_compress.c`.

While this PR makes a good job a transfering a large nb of call sites
to using the new macros, there are still a few sites where this transformation is more complex,
or where the numeric representation itself it used "as is".

One of the problematics area is the decision to use the numeric format of the sumtype
within the match finders of `zstd_lazy`.

This commit doesn't change the behavior, it only introduces and employes the macros,
but eventually the resulting code remains identical.

At target, if the numeric representation of the sumtype can be completely abstracted
and no other part of the code depends on it,
it will be possible to move it towards something slightly more efficient.
 2021-12-23 22:03:30 -08:00
Yann Collet aeff128331 change seqDef.offset into seqDef.offBase 
to better reflect the value stored in this field.
 2021-12-23 17:56:08 -08:00
Yann Collet e145b58cfd changed seqDef.matchLength into seqDef.mlBase 
since this is effectively what is stored in this field (== matchLength - MINMATCH).
This makes it clearer what needs to be done when reading from / writing to this field.
 2021-12-23 13:39:46 -08:00
Yann Collet b77fcac61f change ZSTD_storeSeq() interface to accept matchLength 
instead of mlBase.

This removes the need to do `- MINMATCH` at every call site.

The new interface contract is checked with an `assert()`.
 2021-12-23 12:03:33 -08:00
Yann ColletandGitHub a9e43b37d0 Revert "Limit ZSTD_maxCLevel to 21 for 32-bit binaries." 2021-12-20 11:43:14 -08:00
Yann Collet 01adddc3e0 update regression results 2021-12-16 20:43:23 -08:00
Yann Collet 7c7b9244d6 update regression results 2021-12-16 16:07:54 -08:00
Yann Collet 9a32492730 updated regression results.csv 2021-12-16 14:39:30 -08:00
Elliot GorokhovskyandGitHub c5f1e826ca Merge pull request #2925 from embg/dict_training_sample_limit_size 
Allow user to specify memory limit for dictionary training
 2021-12-15 15:58:17 -05:00
Elliot Gorokhovsky 71c0c07c19 Allow user to specify memory limit for dictionary training 2021-12-14 14:29:01 -05:00
Felix HandteandGitHub 5e2fede604 Merge pull request #2921 from felixhandte/neg-lvl-stagger-step 
Stagger Stepping in Negative Levels
 2021-12-14 14:13:57 -05:00
W. Felix Handte 450fca9704 Update Regression Tests w/ New Sizes 2021-12-13 17:29:32 -05:00
Nick TerrellandGitHub 3e2a70b6fb Merge pull request #2905 from 15596858998/dev_1205 
add test case
 2021-12-13 13:45:23 -08:00
zx123123andGitHub c69d13eb99 Update playTests.sh 2021-12-13 08:58:42 +08:00
Felix HandteandGitHub 0c26d98c0d Merge pull request #2910 from felixhandte/reject-irregular-dicts 
Reject Irregular Dictionary Files
 2021-12-09 11:44:37 -05:00
W. Felix Handte 9985e10fda Reject Irregular Dictionary Files 
I hadn't seen #2890, so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.

Addresses #2874.
 2021-12-08 16:17:04 -05:00
Yann Collet fb3522a3fe fixed very minor cast warning under cygwin 2021-12-08 09:48:56 -08:00
binhdvoandGitHub 38dfc4699e Imply -q when stderr is not a tty (#2884) 
* Imply -q when stderr is not a tty
 2021-12-07 16:56:19 -05:00
15596858998 ba7e8f1051 add test case 2021-12-05 19:12:52 +08:00
Nick TerrellandGitHub 01ecd6ffc0 Merge pull request #2892 from terrelln/issue-2785 
[CircleCI] Fix short-tests-0
 2021-12-02 16:20:56 -05:00
Alexander KanavinandAlexander Kanavin 1e514feec6 Makefile: sort all wildcard file list expansions 
Otherwise the order is non-deterministic and breaks
reproducible builds.
 2021-12-02 12:04:11 +01:00
Nick Terrell 91f5891dd0 [CircleCI] Fix short-tests-0 
short-tests-0 were silently failing. I think because of the && make clean construction. Switch to ; instead.

Also fix all the test failures that were exposed.

`make all` is failing on CircleCI because it is missing Docker. Move that test
to GitHub actions, and switch the pedantic CircleCI test to `make allmost`.
 2021-12-01 17:43:46 -08:00
Nick TerrellandGitHub 9b97fdf74f Merge pull request #2887 from terrelln/issue-2815 
[zdict] Remove ZDICT_CONTENTSIZE_MIN restriction for ZDICT_finalizeDictionary
 2021-12-01 18:15:53 -05:00
Yann Collet 49d578763d reduce storage requirement 
51 MB seems excessive for CI storate and considering the nature of the test.

(note : maybe we should consider using `/tmp` for files generated during tests,
as tmpfs is typically using RAM, thus preserving storage.)
 2021-12-01 15:10:55 -08:00
Yann ColletandGitHub 3133d1e86e Merge pull request #2876 from 15596858998/dev 
Solve the bug of extra output newline character
 2021-12-01 15:10:08 -08:00
Yann ColletandGitHub 8031dc7a48 Merge pull request #2885 from yoniko/limit-level-32bit-systems 
Limit `ZSTD_maxCLevel` to 21 for 32-bit binaries.
 2021-12-01 14:19:16 -08:00