ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,269 Commits 17 Branches 77 Tags
65404fe14ad292bcad07d6e1c6d6e96cdb799eb1
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Felix Handte 65404fe14a Merge pull request #2923 from IAL32/patch-1 
typo: Small spelling mistake in example
 2021-12-13 13:15:21 -05:00
Felix Handte 0c26d98c0d Merge pull request #2910 from felixhandte/reject-irregular-dicts 
Reject Irregular Dictionary Files
 2021-12-09 11:44:37 -05:00
W. Felix Handte 9985e10fda Reject Irregular Dictionary Files 
I hadn't seen #2890, so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.

Addresses #2874.
 2021-12-08 16:17:04 -05:00
Felix Handte 33562c55de Merge pull request #2912 from felixhandte/pkg-config-fix 
Fix Up #2659; Build libzstd.pc Whenever Building the Lib on Unix
 2021-12-08 15:22:56 -05:00
W. Felix Handte 9f1dee8fa5 Fix Up #2659; Build libzstd.pc Whenever Building the Lib on Unix 2021-12-08 12:43:34 -05:00
Felix Handte 9118ee04c2 Merge pull request #2659 from ericonr/pc 
[lib] Fix libzstd.pc for lib-mt builds
 2021-12-07 14:18:38 -05:00
Felix Handte d2c86ec898 Merge pull request #2897 from felixhandte/zstd-deprecated-avoid-deprecated 
Avoid Using Deprecated Functions in Deprecated Code
 2021-12-03 12:09:58 -05:00
Felix Handte fd6c34f641 Merge pull request #2899 from felixhandte/cmake-disable-multithreading-android 
Disable Multithreading in CMake Builds for Android
 2021-12-03 12:09:39 -05:00
W. Felix Handte 4a82bc9d00 Disable Multithreading in CMake Builds for Android 2021-12-02 17:23:42 -05:00
W. Felix Handte e688317652 Fix Include Path 2021-12-02 16:53:52 -05:00
W. Felix Handte d82d67d073 Migrate to FORWARD_IF_ERROR 2021-12-02 16:06:07 -05:00
W. Felix Handte a3ee9815c7 Avoid Using Deprecated Functions in Deprecated Code 
`lib/deprecated` is no longer built by zstd's bundled build files. However,
users may try to build these files when they import the source tree into
their own build systems. And if they have `-Wdeprecated-declarations` on,
this can produce warnings.

This PR migrates these files away from using deprecated declarations.

This addresses #2767.
 2021-12-02 14:25:33 -05:00
Felix Handte c2c6a4ab40 Merge pull request #2869 from felixhandte/oss-fuzz-fix-41005 
Determinism: Avoid Mapping Window into Reserved Indices during Reduction
 2021-11-18 10:11:48 -05:00
W. Felix Handte 66079085f0 Determinism: Avoid Mapping Window into Reserved Indices during Reduction 
PR #2850 attempted to fix a determinism bug that was uncovered by OSS-Fuzz. It
succeeded in addressing that source of non-determinism, but introduced a new
one: it was possible, when index reduction occurred, to map indices in the
window to the reserved value, which would cause them to be zeroed, potentially
altering parsing of the input.

This PR addresses this issue. It makes sure that the bottom of the window is
always `>= ZSTD_WINDOW_START_INDEX`.

I'm not sure if this makes #2850 redundant. I think it's probably still
valuable to have that protection as well.

Credit to OSS-Fuzz for discovering this issue.
 2021-11-17 18:09:18 -05:00
Felix Handte a071e00696 Merge pull request #2850 from felixhandte/oss-fuzz-fix-40829-for-real-this-time 
Fix Determinism Bug: Avoid Reducing Indices to Reserved Values
 2021-11-10 12:00:43 -05:00
W. Felix Handte 48572f52b1 Rewrite Fix to Still Auto-Vectorize 2021-11-09 12:17:03 -05:00
W. Felix Handte 61765cacd0 Avoid Reducing Indices to Reserved Values 
Previously, if an index was equal to `reducerValue + 1`, it would get remapped
during index reduction to 1 i.e. `ZSTD_DUBT_UNSORTED_MARK`. This can affect the
parsing of the input slightly, by causing tree nodes to be nullified when they
otherwise wouldn't be. This hardly matters from a correctness or efficiency
perspective, but it does impact determinism.

So this commit changes index reduction to avoid mapping indices to collide with
`ZSTD_DUBT_UNSORTED_MARK`.
 2021-11-08 20:03:52 -05:00
Felix Handte 23c1a2d260 Merge pull request #2774 from felixhandte/zstd-dfast-pipelined-single 
Pipelined Implementation of ZSTD_dfast
 2021-10-13 16:38:43 -04:00
W. Felix Handte 0bfc935add Convert Outer Control Structure to Loop 2021-10-12 13:34:17 -04:00
W. Felix Handte 79ca830766 Style: Add Comments to Variables and Move a Couple into the Loop 2021-10-05 16:18:09 -04:00
W. Felix Handte 168d0a3c89 Fix Flaky Test 
This test depended on `_extDict` and `_noDict` compressing identically, which
is not a guarantee we make, AFAIK.
 2021-10-05 16:18:09 -04:00
W. Felix Handte c2c32839dc Update results.csv 2021-10-05 16:18:00 -04:00
W. Felix Handte 62536ef7da Simplify DMS Implementation by Removing noDict Support 2021-10-05 14:54:37 -04:00
W. Felix Handte 051b473e7e Fall Back in _extDict to New _noDict Rather than Old Merged Impl 2021-10-05 14:54:37 -04:00
W. Felix Handte fcab4841aa Nit: Rename Function 2021-10-05 14:54:37 -04:00
W. Felix Handte 47fd762ecc Nit: Unnest Blocks that Don't Declare Anything 2021-10-05 14:54:37 -04:00
W. Felix Handte 2cdfad538c Search One Last Position 2021-10-05 14:54:37 -04:00
W. Felix Handte 6ae44c0db8 Advance Long Index Lookup (+0.5% Speed) 
This lookup can be advanced to before the short match check because either way
we will use it (in the next loop iter or in `_search_next_long`).
 2021-10-05 14:54:37 -04:00
W. Felix Handte 2ddef7c872 Write Back Advanced Hash in Long Matches as Well (+Ratio) 
Since we're now hashing the position ahead even if we find a long match and
don't search that next position, we can write it back into the hashtable even
in long matches. This seems to cost us no speed, and improves compression
ratio slightly!
 2021-10-05 14:54:37 -04:00
W. Felix Handte 39f2491bfc Use Look-Ahead Hash for Next Long Check after Short Match (+0.5% Speed) 
This costs a little ratio, unfortunately.
 2021-10-05 14:54:37 -04:00
W. Felix Handte db4e1b5479 Hash Long One Position Ahead (+2.5% Speed) 
Aside from maybe a latency win in the loop, this means that when we find a
short match, we've already done the hash we need to check the next long match.
 2021-10-05 14:54:37 -04:00
W. Felix Handte a1ac7205d0 Pull Match Found Stuff Out of the Loop 2021-10-05 14:54:37 -04:00
W. Felix Handte 072ffaad67 Extract Working Variables 2021-10-05 14:54:37 -04:00
W. Felix Handte 1bdf041071 Track Step Rather than Recalculating (+0.5% Speed) 2021-10-05 14:54:37 -04:00
W. Felix Handte 258c0623e1 Extract Single-Segment Variant of ZSTD_dfast 2021-10-05 14:54:37 -04:00
Felix Handte 8b7a19fcd4 Merge pull request #2805 from nolange/smaller_code_with_disabled_features 
Smaller code with disabled features
 2021-09-27 17:43:21 -04:00
Felix Handte d68aa19a2f Merge pull request #2749 from felixhandte/zstd-fast-pipelined 
Pipelined Implementation of ZSTD_fast (~+5% Speed)
 2021-09-09 17:05:30 -04:00
W. Felix Handte b0977e4ed2 Update results.csv 2021-09-01 14:45:00 -04:00
W. Felix Handte d6fd7761c9 Fix VS Build: Explicitly Cast to Narrow Ints 2021-09-01 14:15:04 -04:00
W. Felix Handte 98d3df326b Change Target Size in Fuzzer 
It's a bit strange, because this is hitting the dictionary special case where
the dictionary is contiguous with the input and still runs in the single-
segment path.

We should probably change that to hit the `extDict` path instead?
 2021-09-01 14:15:04 -04:00
W. Felix Handte 15e67bfa7e Deduplicate Implementations 
This removes the old `ZSTD_compressBlock_fast_generic()` and renames the new
`ZSTD_compressBlock_fast_generic_pipelined()` to replace it. This is
functionally a no-op.
 2021-09-01 14:15:04 -04:00
W. Felix Handte 64054dec44 Tweak Step 2021-09-01 14:15:04 -04:00
W. Felix Handte 24fcccd05c Unroll Loop Core; Reduce Frequency of Repcode Check & Step Calc (+>1% Speed) 
Unrolling the loop to handle 2 positions in each iteration allows us to reduce
the frequency of some operations that don't need to happen at every position.
One such operation is the step calculation, which is a very rough heuristic
anyways. It's fine if we do this a position later. The other operation is the
repcode check. But since the repcode check already tries expanding back one
position, we're really not missing much of importance by only trying it every
other position.

This commit also slightly reorders some operations.
 2021-09-01 14:15:04 -04:00
W. Felix Handte 57a100f6dc Add ip1 + 128 Prefetch; Tiny Cleanup 2021-09-01 14:15:04 -04:00
W. Felix Handte 991d660ea9 Nit: Only Store 2 Hash Variables 2021-09-01 14:15:04 -04:00
W. Felix Handte 8706bc115a Nit: Dedup idx0 and idx1 2021-09-01 14:15:04 -04:00
W. Felix Handte 7c24c3e6ce Give Up on Searching End of Block 
Amusingly, it seems to be a non-trivial performance hit to add in final
searches or even hash table insertions during cleanup. So let's not. It seems
to not make any meaningful difference in compression ratio.
 2021-09-01 14:15:03 -04:00
W. Felix Handte 35932ab2f1 Prefetch Input in Incompressible Sections (+0.25% Speed) 2021-09-01 14:15:03 -04:00
W. Felix Handte b092dd75b7 Shrink Pipeline from 4 Positions to 3 2021-09-01 14:15:03 -04:00
W. Felix Handte 387840af79 Re-Order Operations for Slightly Better Performance 2021-09-01 14:15:03 -04:00
W. Felix Handte bc768bccc0 Track Step Size Statefully, Rather than Recalculating Every Time 2021-09-01 14:15:03 -04:00
W. Felix Handte 80bc12b33a Initial Pipelined Implementation for ZSTD_fast 2021-09-01 14:15:03 -04:00
W. Felix Handte ab8aa49b8d Fix Benchmark Corruption Display 2021-09-01 14:15:03 -04:00
Felix Handte ae131282f5 Merge pull request #2742 from felixhandte/set-mtime 
Set mtime on Output Files
 2021-08-04 16:44:46 -04:00
W. Felix Handte a8f4612eab Remove sleep()s in Test; Replace with Artificial mtime 
This behavior of `touch` is standardized in POSIX, so it should be available.
 2021-08-04 15:56:46 -04:00
W. Felix Handte 9cd6c1ff4d Update mtime and atime for Written Files 2021-08-04 14:49:56 -04:00
W. Felix Handte a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
W. Felix Handte ead41bcb4e Add Test to Verify mtime is Copied to Destination 2021-08-03 17:22:58 -04:00
Felix Handte 495597699c Merge pull request #2729 from felixhandte/fix-ddss-load 
Fix DDSS Load
 2021-07-28 13:12:50 -04:00
W. Felix Handte da58821ff2 Fix DDSS Load 
This PR fixes an incorrect comparison in figuring out `minChain` in
`ZSTD_dedicatedDictSearch_lazy_loadDictionary()`. This incorrect comparison
had been masked by the fact that `idx` was always 1, until @terrelln changed
that in #2726.

Credit-to: OSS-Fuzz
 2021-07-27 11:49:44 -04:00
Felix Handte 628f65b79c Merge pull request #2714 from luisdallos/build-macros-typos 
lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros
 2021-06-18 16:16:44 -04:00
Felix Handte 67a25968f0 Merge pull request #2702 from felixhandte/human_size_output 
Format File Sizes Human-Readable in the CLI
 2021-06-10 16:53:43 -04:00
W. Felix Handte 8c00807bbc Whitespace Fixes to Improve Cross-Line Alignment 2021-06-10 13:28:38 -04:00
W. Felix Handte 94cf57bb13 Update Tests to Reflect New Formatting 2021-06-10 13:14:18 -04:00
W. Felix Handte 87e94e3e39 Convert Other Size Displays to Use Human-Readable Formatting 2021-06-10 12:57:46 -04:00
W. Felix Handte 2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB") 
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
 2021-06-10 12:53:07 -04:00
W. Felix Handte 9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte bc46b6efe4 Apply to Other Print Statement as Well 2021-06-10 12:53:07 -04:00
W. Felix Handte 7e0058848c Fix Whitespace 2021-06-10 12:53:07 -04:00
W. Felix Handte 93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte 464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte 9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting 
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
 2021-06-10 12:53:07 -04:00
Felix Handte 8a3bdfaa7b Merge pull request #2654 from wolfpld/dev 
Initialize "potentially uninitialized" pointers.
 2021-06-07 13:04:19 -04:00
W. Felix Handte 51708b2c62 Fix CircleCI Config to Fully Remove publish-github-release Job 2021-05-14 12:09:22 -04:00
Felix Handte 909925785a Merge pull request #2618 from felixhandte/single-file-build-mv 
Move Single-File Build Script from `contrib/` to `build/`
 2021-05-06 14:09:42 -04:00
W. Felix Handte 4ba49af665 Rewrite References to Location 2021-05-05 18:03:48 -04:00
Felix Handte b062d97520 Merge pull request #2525 from felixhandte/fix-file-permissions-again 
Improve Setting Permissions of Created Files
 2021-05-05 17:59:13 -04:00
W. Felix Handte 1d65917323 Move Single-File Build Script from contrib/ to build/ 2021-05-05 16:07:51 -04:00
W. Felix Handte 4f9c6fdb7f Attempt to Fix Windows Build Error 2021-05-05 13:13:56 -04:00
W. Felix Handte da61918c75 Also Pass Mode Bits in on Windows 
I think in some unix emulation environments on Windows, (cygwin?) mode bits
are somehow respected. So we might as well pass them in. Can't hurt.
 2021-05-05 13:10:34 -04:00
W. Felix Handte bea1b2ba70 rm -f in playTests.sh 2021-05-05 13:10:34 -04:00
W. Felix Handte 45c4918ccf Fix Build for Windows 2021-05-05 13:10:34 -04:00
W. Felix Handte 018ed6552a Attempt to Fix stat Format for BSDs 2021-05-05 13:10:34 -04:00
W. Felix Handte 1fb10ba831 Don't Block Removing File on Being Able to Read It 
`open()`'s mode bits are only applied to files that are created by the call.
If the output file already exists, but is not readable, the `fopen()` would
fail, preventing us from removing it, which would mean that the file would
not end up with the correct permission bits.

It's not clear to me why the `fopen()` is there at all. `UTIL_isRegularFile()`
should be sufficient, AFAICT.
 2021-05-05 13:10:34 -04:00
W. Felix Handte b87f97b3ea Create Files with Desired Permissions; Avoid chmod(); Remove UTIL_chmod() 2021-05-05 13:10:34 -04:00
W. Felix Handte 4e10ff15f5 Add Tests Checking File Permissions of Created Files 2021-05-05 13:10:34 -04:00
Felix Handte 2d10544b84 Merge pull request #2613 from felixhandte/allow-block-device 
Allow Reading from Block Devices with `--force`
 2021-05-05 13:06:32 -04:00
W. Felix Handte e58e9c7928 Add Test Case (Behind Flag); Run in GitHub Action 2021-05-04 18:43:39 -04:00
W. Felix Handte 33f3e293e8 Allow Reading from Block Devices with --force 2021-05-04 16:25:26 -04:00
Felix Handte da74f1c717 Merge pull request #2609 from felixhandte/md5sum-darwin 
Detect Presence of `md5` on Darwin
 2021-05-04 14:22:54 -04:00
W. Felix Handte ee122baacf Detect Presence of md5 on Darwin 
This fixes #2568.
 2021-05-04 12:33:19 -04:00
Felix Handte d1a6d080cf Merge pull request #2535 from felixhandte/gha-release-artifacts 
Add GitHub Action to Automatically Publish Release Tarballs
 2021-03-15 13:58:27 -04:00
W. Felix Handte d2b7f2e27a Allow a Passphrase on the Key 2021-03-15 12:48:53 -04:00
W. Felix Handte eed64d75f4 Maintain Artifact Name Backwards Compatibility 
When the tag is `v1.2.3`, name the artifacts `zstd-1.2.3.tar*` rather than
`zstd-v1.2.3.tar*`. When the tag doesn't match, use the full tag.
 2021-03-15 11:59:31 -04:00
W. Felix Handte a51511e7a7 Remove CircleCI Artifact Generation 2021-03-12 17:35:11 -05:00
W. Felix Handte 5d1fec8ce1 Add GitHub Action to Automatically Publish Release Tarballs 
This commit introduces a GitHub action that is triggered on release creation,
which creates the release tarball, compresses it, hashes it, signs it, and
attaches all of those files to the release.
 2021-03-12 12:33:58 -05:00
Felix Handte 74d65eaa92 Merge pull request #2514 from felixhandte/v1.4.9 
Prepare Codebase for v1.4.9 Release
 2021-03-02 11:41:55 -05:00
W. Felix Handte 3835957b2d Update CHANGELOG 2021-03-01 18:00:10 -05:00
W. Felix Handte 0f1a52b349 Documentation Rebuild 2021-03-01 17:57:02 -05:00