ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,025 Commits 17 Branches 77 Tags
fd94b9d1c9f2463858c6a2f6ba458f8df25f5faa
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Felix HandteandGitHub d68aa19a2f Merge pull request #2749 from felixhandte/zstd-fast-pipelined 
Pipelined Implementation of ZSTD_fast (~+5% Speed)
 2021-09-09 17:05:30 -04:00
W. Felix Handte b0977e4ed2 Update results.csv 2021-09-01 14:45:00 -04:00
W. Felix Handte d6fd7761c9 Fix VS Build: Explicitly Cast to Narrow Ints 2021-09-01 14:15:04 -04:00
W. Felix Handte 98d3df326b Change Target Size in Fuzzer 
It's a bit strange, because this is hitting the dictionary special case where
the dictionary is contiguous with the input and still runs in the single-
segment path.

We should probably change that to hit the `extDict` path instead?
 2021-09-01 14:15:04 -04:00
W. Felix Handte 15e67bfa7e Deduplicate Implementations 
This removes the old `ZSTD_compressBlock_fast_generic()` and renames the new
`ZSTD_compressBlock_fast_generic_pipelined()` to replace it. This is
functionally a no-op.
 2021-09-01 14:15:04 -04:00
W. Felix Handte 64054dec44 Tweak Step 2021-09-01 14:15:04 -04:00
W. Felix Handte 24fcccd05c Unroll Loop Core; Reduce Frequency of Repcode Check & Step Calc (+>1% Speed) 
Unrolling the loop to handle 2 positions in each iteration allows us to reduce
the frequency of some operations that don't need to happen at every position.
One such operation is the step calculation, which is a very rough heuristic
anyways. It's fine if we do this a position later. The other operation is the
repcode check. But since the repcode check already tries expanding back one
position, we're really not missing much of importance by only trying it every
other position.

This commit also slightly reorders some operations.
 2021-09-01 14:15:04 -04:00
W. Felix Handte 57a100f6dc Add ip1 + 128 Prefetch; Tiny Cleanup 2021-09-01 14:15:04 -04:00
W. Felix Handte 991d660ea9 Nit: Only Store 2 Hash Variables 2021-09-01 14:15:04 -04:00
W. Felix Handte 8706bc115a Nit: Dedup idx0 and idx1 2021-09-01 14:15:04 -04:00
W. Felix Handte 7c24c3e6ce Give Up on Searching End of Block 
Amusingly, it seems to be a non-trivial performance hit to add in final
searches or even hash table insertions during cleanup. So let's not. It seems
to not make any meaningful difference in compression ratio.
 2021-09-01 14:15:03 -04:00
W. Felix Handte 35932ab2f1 Prefetch Input in Incompressible Sections (+0.25% Speed) 2021-09-01 14:15:03 -04:00
W. Felix Handte b092dd75b7 Shrink Pipeline from 4 Positions to 3 2021-09-01 14:15:03 -04:00
W. Felix Handte 387840af79 Re-Order Operations for Slightly Better Performance 2021-09-01 14:15:03 -04:00
W. Felix Handte bc768bccc0 Track Step Size Statefully, Rather than Recalculating Every Time 2021-09-01 14:15:03 -04:00
W. Felix Handte 80bc12b33a Initial Pipelined Implementation for ZSTD_fast 2021-09-01 14:15:03 -04:00
W. Felix Handte ab8aa49b8d Fix Benchmark Corruption Display 2021-09-01 14:15:03 -04:00
Felix HandteandGitHub ae131282f5 Merge pull request #2742 from felixhandte/set-mtime 
Set mtime on Output Files
 2021-08-04 16:44:46 -04:00
W. Felix Handte a8f4612eab Remove sleep()s in Test; Replace with Artificial mtime 
This behavior of `touch` is standardized in POSIX, so it should be available.
 2021-08-04 15:56:46 -04:00
W. Felix Handte 9cd6c1ff4d Update mtime and atime for Written Files 2021-08-04 14:49:56 -04:00
W. Felix Handte a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
W. Felix Handte ead41bcb4e Add Test to Verify mtime is Copied to Destination 2021-08-03 17:22:58 -04:00
Felix HandteandGitHub 495597699c Merge pull request #2729 from felixhandte/fix-ddss-load 
Fix DDSS Load
 2021-07-28 13:12:50 -04:00
W. Felix Handte da58821ff2 Fix DDSS Load 
This PR fixes an incorrect comparison in figuring out `minChain` in
`ZSTD_dedicatedDictSearch_lazy_loadDictionary()`. This incorrect comparison
had been masked by the fact that `idx` was always 1, until @terrelln changed
that in #2726.

Credit-to: OSS-Fuzz
 2021-07-27 11:49:44 -04:00
Felix HandteandGitHub 628f65b79c Merge pull request #2714 from luisdallos/build-macros-typos 
lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros
 2021-06-18 16:16:44 -04:00
Felix HandteandGitHub 67a25968f0 Merge pull request #2702 from felixhandte/human_size_output 
Format File Sizes Human-Readable in the CLI
 2021-06-10 16:53:43 -04:00
W. Felix Handte 8c00807bbc Whitespace Fixes to Improve Cross-Line Alignment 2021-06-10 13:28:38 -04:00
W. Felix Handte 94cf57bb13 Update Tests to Reflect New Formatting 2021-06-10 13:14:18 -04:00
W. Felix Handte 87e94e3e39 Convert Other Size Displays to Use Human-Readable Formatting 2021-06-10 12:57:46 -04:00
W. Felix Handte 2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB") 
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
 2021-06-10 12:53:07 -04:00
W. Felix Handte 9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte bc46b6efe4 Apply to Other Print Statement as Well 2021-06-10 12:53:07 -04:00
W. Felix Handte 7e0058848c Fix Whitespace 2021-06-10 12:53:07 -04:00
W. Felix Handte 93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte 464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte 9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting 
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
 2021-06-10 12:53:07 -04:00
Felix HandteandGitHub 8a3bdfaa7b Merge pull request #2654 from wolfpld/dev 
Initialize "potentially uninitialized" pointers.
 2021-06-07 13:04:19 -04:00
W. Felix Handte 51708b2c62 Fix CircleCI Config to Fully Remove publish-github-release Job 2021-05-14 12:09:22 -04:00
Felix HandteandGitHub 909925785a Merge pull request #2618 from felixhandte/single-file-build-mv 
Move Single-File Build Script from `contrib/` to `build/`
 2021-05-06 14:09:42 -04:00
W. Felix Handte 4ba49af665 Rewrite References to Location 2021-05-05 18:03:48 -04:00
Felix HandteandGitHub b062d97520 Merge pull request #2525 from felixhandte/fix-file-permissions-again 
Improve Setting Permissions of Created Files
 2021-05-05 17:59:13 -04:00
W. Felix Handte 1d65917323 Move Single-File Build Script from contrib/ to build/ 2021-05-05 16:07:51 -04:00
W. Felix Handte 4f9c6fdb7f Attempt to Fix Windows Build Error 2021-05-05 13:13:56 -04:00
W. Felix Handte da61918c75 Also Pass Mode Bits in on Windows 
I think in some unix emulation environments on Windows, (cygwin?) mode bits
are somehow respected. So we might as well pass them in. Can't hurt.
 2021-05-05 13:10:34 -04:00
W. Felix Handte bea1b2ba70 rm -f in playTests.sh 2021-05-05 13:10:34 -04:00
W. Felix Handte 45c4918ccf Fix Build for Windows 2021-05-05 13:10:34 -04:00
W. Felix Handte 018ed6552a Attempt to Fix stat Format for BSDs 2021-05-05 13:10:34 -04:00
W. Felix Handte 1fb10ba831 Don't Block Removing File on Being Able to Read It 
`open()`'s mode bits are only applied to files that are created by the call.
If the output file already exists, but is not readable, the `fopen()` would
fail, preventing us from removing it, which would mean that the file would
not end up with the correct permission bits.

It's not clear to me why the `fopen()` is there at all. `UTIL_isRegularFile()`
should be sufficient, AFAICT.
 2021-05-05 13:10:34 -04:00
W. Felix Handte b87f97b3ea Create Files with Desired Permissions; Avoid chmod(); Remove UTIL_chmod() 2021-05-05 13:10:34 -04:00
W. Felix Handte 4e10ff15f5 Add Tests Checking File Permissions of Created Files 2021-05-05 13:10:34 -04:00
Felix HandteandGitHub 2d10544b84 Merge pull request #2613 from felixhandte/allow-block-device 
Allow Reading from Block Devices with `--force`
 2021-05-05 13:06:32 -04:00
W. Felix Handte e58e9c7928 Add Test Case (Behind Flag); Run in GitHub Action 2021-05-04 18:43:39 -04:00
W. Felix Handte 33f3e293e8 Allow Reading from Block Devices with --force 2021-05-04 16:25:26 -04:00
Felix HandteandGitHub da74f1c717 Merge pull request #2609 from felixhandte/md5sum-darwin 
Detect Presence of `md5` on Darwin
 2021-05-04 14:22:54 -04:00
W. Felix Handte ee122baacf Detect Presence of md5 on Darwin 
This fixes #2568.
 2021-05-04 12:33:19 -04:00
Felix HandteandGitHub d1a6d080cf Merge pull request #2535 from felixhandte/gha-release-artifacts 
Add GitHub Action to Automatically Publish Release Tarballs
 2021-03-15 13:58:27 -04:00
W. Felix Handte d2b7f2e27a Allow a Passphrase on the Key 2021-03-15 12:48:53 -04:00
W. Felix Handte eed64d75f4 Maintain Artifact Name Backwards Compatibility 
When the tag is `v1.2.3`, name the artifacts `zstd-1.2.3.tar*` rather than
`zstd-v1.2.3.tar*`. When the tag doesn't match, use the full tag.
 2021-03-15 11:59:31 -04:00
W. Felix Handte a51511e7a7 Remove CircleCI Artifact Generation 2021-03-12 17:35:11 -05:00
W. Felix Handte 5d1fec8ce1 Add GitHub Action to Automatically Publish Release Tarballs 
This commit introduces a GitHub action that is triggered on release creation,
which creates the release tarball, compresses it, hashes it, signs it, and
attaches all of those files to the release.
 2021-03-12 12:33:58 -05:00
Felix HandteandGitHub 74d65eaa92 Merge pull request #2514 from felixhandte/v1.4.9 
Prepare Codebase for v1.4.9 Release
 2021-03-02 11:41:55 -05:00
W. Felix Handte 3835957b2d Update CHANGELOG 2021-03-01 18:00:10 -05:00
W. Felix Handte 0f1a52b349 Documentation Rebuild 2021-03-01 17:57:02 -05:00
W. Felix Handte d7db928f72 Bump Library Version 1.4.8 -> 1.4.9 2021-03-01 17:45:30 -05:00
Felix HandteandGitHub aec1e8c715 Merge pull request #2513 from felixhandte/fix-2493 
Avoid Using `stat -c` on NetBSD
 2021-02-26 18:02:38 -05:00
Felix HandteandGitHub 45ee23f6a1 Merge pull request #2512 from felixhandte/fix-2509 
Detect `..` in Paths Correctly
 2021-02-26 18:02:18 -05:00
W. Felix Handte 221e4659cd Avoid Using stat -c on NetBSD 
Addresses #2493. I think. I don't have a NetBSD system to test on.
 2021-02-26 13:05:39 -05:00
W. Felix Handte 9b7f9d26d5 Cover These Edge Cases in Tests 2021-02-26 13:01:20 -05:00
W. Felix Handte 61db590ad8 Detect .. in Paths Correctly 
This commit addresses #2509.
 2021-02-26 12:29:42 -05:00
Felix HandteandGitHub a2adc6df9f Merge pull request #2495 from felixhandte/umask 
Use umask() to Constrain Created File Permissions
 2021-02-17 17:03:23 -05:00
W. Felix Handte a774c57973 Use umask() to Constrain Created File Permissions 
This commit addresses #2491.

Note that a downside of this solution is that it is global: `umask()` affects
all file creation calls in the process. I believe this is safe since
`fileio.c` functions should only ever be used in the zstd binary, and these
are (almost) the only files ever created by zstd, and AIUI they're only
created in a single thread. So we can get away with messing with global state.

Note that this doesn't change the permissions of files created by `dibio.c`.
I'm not sure what those should be...
 2021-02-17 15:27:39 -05:00
W. Felix Handte 927859f5e8 Also Update Man Page Documentation 2021-01-11 17:55:58 -05:00
W. Felix Handte 8b6a4b5b7c Allow Input From Console When --force is Passed 
Also update option flag documentation.
 2021-01-11 17:53:20 -05:00
Felix HandteandGitHub fe48742c69 Merge pull request #2422 from felixhandte/doc-update-repcodes 
Update Zstd Compression Format to Clarify Repcode Behavior
 2020-12-10 01:20:51 -05:00
W. Felix Handte 2d46d764cf Update Zstd Compression Format to Clarify Repcode Behavior 2020-12-09 20:03:58 -05:00
Felix HandteandGitHub f861e8c07b Merge pull request #2421 from felixhandte/pc-no-sed 
Don't Use Regexes to Build Pkg-Config File
 2020-12-09 18:58:17 -05:00
Felix HandteandGitHub bee9137409 Merge pull request #2419 from felixhandte/asan-dont-poison-static-allocs 
Don't ASAN-Poison Statically-Allocated Workspaces
 2020-12-09 16:50:52 -05:00
W. Felix Handte 9dab03db90 Create Enum to Represent Static/Dynamic Allocation Distinction in cwksp 2020-12-09 14:57:37 -05:00
W. Felix Handte db9e73cb07 Don't ASAN-Poison Statically-Allocated Workspaces 
Addresses #2286.
 2020-12-09 13:00:47 -05:00
W. Felix Handte a75f9ce3e9 Apply Same Strategy to CMake 2020-12-08 20:46:37 -05:00
W. Felix Handte b521183c74 Avoid Use of Regexes in Building Package-Config File 2020-12-08 20:10:05 -05:00
Felix HandteandGitHub 200c960f1d Merge pull request #2311 from felixhandte/ddss-fix-cparam-derivation 
Fix Compression Parameter Derivation Bugs Introduced by DDSS Changes
 2020-09-18 14:02:14 -04:00
Felix HandteandGitHub 5870c47bc0 Merge pull request #2313 from felixhandte/fuzzer-separate-long-tests 
Separate Long Tests in Fuzzer
 2020-09-18 12:03:00 -04:00
W. Felix Handte e991f9994d Run Long Tests in GitHub CI 2020-09-17 13:31:10 -04:00
W. Felix Handte 9398acb245 Move Last Two Long Tests in fuzzer.c into Separate --long-tests Section 2020-09-17 13:31:10 -04:00
W. Felix Handte f23a321781 Update Regression Test Results 2020-09-17 12:23:05 -04:00
W. Felix Handte 8930c6e551 Use ZSTD_CCtxParams_init() to Init CCtxParams, not memset() 
Even if the discrepancies are at the moment benign, it's probably better to
standardize on using the one true initializer, rather than trying (and failing)
to correctly duplicate its behavior.
 2020-09-17 12:15:33 -04:00
W. Felix Handte e8a44326fa Avoid Redundancy in ZSTD_initCDict_internal() Args; Don't Take CParams + CCtxParams 2020-09-17 12:08:36 -04:00
W. Felix Handte eee51a664a Fall Back if Derived CParams are Incompatible with DDSS; Refactor CDict Creation 
Rewrite ZSTD_createCDict_advanced() as a wrapper around
ZSTD_createCDict_advanced2(). Evaluate whether to use DDSS mode *after* fully
resolving cparams. If not, fall back.
 2020-09-15 18:01:08 -04:00
W. Felix Handte bc6521a6f6 Make ZSTD_createCDict_advanced2() cctxParams Arg Const 2020-09-15 14:06:10 -04:00
W. Felix Handte 26a96a5b35 Do More Complete CParams Deduction in Non-DDSS Path of ZSTD_createCDict_advanced2 
Call ZSTD_getCParamsFromCCtxParams() instead of ZSTD_getCParams_internal().
 2020-09-15 13:57:43 -04:00
W. Felix Handte a2af804129 Pull CParam Override Logic into Helper 2020-09-15 13:38:05 -04:00
Felix HandteandGitHub d903b552c8 Merge pull request #2295 from felixhandte/dedicated-dict-search-structure-chain 
DDSS for Lazy: Implement a Dedicated Dictionary Chain Table
 2020-09-11 01:39:23 -04:00
W. Felix Handte d6246d4a0f Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout 
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
 2020-09-10 23:35:42 -04:00
W. Felix Handte 6d3f816b3e Test Fewer Dictionary Sizes 2020-09-10 22:30:52 -04:00
W. Felix Handte b6df3fd438 Fix Debug Logging in 32-bit Build 2020-09-10 22:10:02 -04:00
W. Felix Handte c5fab8848a Document searchFuncs Table 2020-09-10 22:10:02 -04:00
W. Felix Handte 85a95840e4 Further Consolidate Dict Mode Checks 2020-09-10 22:10:02 -04:00
W. Felix Handte 032010fcc1 Improve Documentation Slightly 2020-09-10 22:10:02 -04:00