ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,813 Commits 17 Branches 77 Tags
a488ba114ec17ea1054b9057c26a046fc122b3b6
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Felix HandteandGitHub 909925785a Merge pull request #2618 from felixhandte/single-file-build-mv 
Move Single-File Build Script from `contrib/` to `build/`
 2021-05-06 14:09:42 -04:00
W. Felix Handte 4ba49af665 Rewrite References to Location 2021-05-05 18:03:48 -04:00
Felix HandteandGitHub b062d97520 Merge pull request #2525 from felixhandte/fix-file-permissions-again 
Improve Setting Permissions of Created Files
 2021-05-05 17:59:13 -04:00
W. Felix Handte 1d65917323 Move Single-File Build Script from contrib/ to build/ 2021-05-05 16:07:51 -04:00
W. Felix Handte 4f9c6fdb7f Attempt to Fix Windows Build Error 2021-05-05 13:13:56 -04:00
W. Felix Handte da61918c75 Also Pass Mode Bits in on Windows 
I think in some unix emulation environments on Windows, (cygwin?) mode bits
are somehow respected. So we might as well pass them in. Can't hurt.
 2021-05-05 13:10:34 -04:00
W. Felix Handte bea1b2ba70 rm -f in playTests.sh 2021-05-05 13:10:34 -04:00
W. Felix Handte 45c4918ccf Fix Build for Windows 2021-05-05 13:10:34 -04:00
W. Felix Handte 018ed6552a Attempt to Fix stat Format for BSDs 2021-05-05 13:10:34 -04:00
W. Felix Handte 1fb10ba831 Don't Block Removing File on Being Able to Read It 
`open()`'s mode bits are only applied to files that are created by the call.
If the output file already exists, but is not readable, the `fopen()` would
fail, preventing us from removing it, which would mean that the file would
not end up with the correct permission bits.

It's not clear to me why the `fopen()` is there at all. `UTIL_isRegularFile()`
should be sufficient, AFAICT.
 2021-05-05 13:10:34 -04:00
W. Felix Handte b87f97b3ea Create Files with Desired Permissions; Avoid chmod(); Remove UTIL_chmod() 2021-05-05 13:10:34 -04:00
W. Felix Handte 4e10ff15f5 Add Tests Checking File Permissions of Created Files 2021-05-05 13:10:34 -04:00
Felix HandteandGitHub 2d10544b84 Merge pull request #2613 from felixhandte/allow-block-device 
Allow Reading from Block Devices with `--force`
 2021-05-05 13:06:32 -04:00
W. Felix Handte e58e9c7928 Add Test Case (Behind Flag); Run in GitHub Action 2021-05-04 18:43:39 -04:00
W. Felix Handte 33f3e293e8 Allow Reading from Block Devices with --force 2021-05-04 16:25:26 -04:00
Felix HandteandGitHub da74f1c717 Merge pull request #2609 from felixhandte/md5sum-darwin 
Detect Presence of `md5` on Darwin
 2021-05-04 14:22:54 -04:00
W. Felix Handte ee122baacf Detect Presence of md5 on Darwin 
This fixes #2568.
 2021-05-04 12:33:19 -04:00
Felix HandteandGitHub d1a6d080cf Merge pull request #2535 from felixhandte/gha-release-artifacts 
Add GitHub Action to Automatically Publish Release Tarballs
 2021-03-15 13:58:27 -04:00
W. Felix Handte d2b7f2e27a Allow a Passphrase on the Key 2021-03-15 12:48:53 -04:00
W. Felix Handte eed64d75f4 Maintain Artifact Name Backwards Compatibility 
When the tag is `v1.2.3`, name the artifacts `zstd-1.2.3.tar*` rather than
`zstd-v1.2.3.tar*`. When the tag doesn't match, use the full tag.
 2021-03-15 11:59:31 -04:00
W. Felix Handte a51511e7a7 Remove CircleCI Artifact Generation 2021-03-12 17:35:11 -05:00
W. Felix Handte 5d1fec8ce1 Add GitHub Action to Automatically Publish Release Tarballs 
This commit introduces a GitHub action that is triggered on release creation,
which creates the release tarball, compresses it, hashes it, signs it, and
attaches all of those files to the release.
 2021-03-12 12:33:58 -05:00
Felix HandteandGitHub e4558ffd1d Merge pull request #2515 from facebook/dev 
ZStandard v1.4.9
 2021-03-02 17:20:57 -05:00
Felix HandteandGitHub 74d65eaa92 Merge pull request #2514 from felixhandte/v1.4.9 
Prepare Codebase for v1.4.9 Release
 2021-03-02 11:41:55 -05:00
W. Felix Handte 3835957b2d Update CHANGELOG 2021-03-01 18:00:10 -05:00
W. Felix Handte 0f1a52b349 Documentation Rebuild 2021-03-01 17:57:02 -05:00
W. Felix Handte d7db928f72 Bump Library Version 1.4.8 -> 1.4.9 2021-03-01 17:45:30 -05:00
Felix HandteandGitHub aec1e8c715 Merge pull request #2513 from felixhandte/fix-2493 
Avoid Using `stat -c` on NetBSD
 2021-02-26 18:02:38 -05:00
Felix HandteandGitHub 45ee23f6a1 Merge pull request #2512 from felixhandte/fix-2509 
Detect `..` in Paths Correctly
 2021-02-26 18:02:18 -05:00
W. Felix Handte 221e4659cd Avoid Using stat -c on NetBSD 
Addresses #2493. I think. I don't have a NetBSD system to test on.
 2021-02-26 13:05:39 -05:00
W. Felix Handte 9b7f9d26d5 Cover These Edge Cases in Tests 2021-02-26 13:01:20 -05:00
W. Felix Handte 61db590ad8 Detect .. in Paths Correctly 
This commit addresses #2509.
 2021-02-26 12:29:42 -05:00
Felix HandteandGitHub a2adc6df9f Merge pull request #2495 from felixhandte/umask 
Use umask() to Constrain Created File Permissions
 2021-02-17 17:03:23 -05:00
W. Felix Handte a774c57973 Use umask() to Constrain Created File Permissions 
This commit addresses #2491.

Note that a downside of this solution is that it is global: `umask()` affects
all file creation calls in the process. I believe this is safe since
`fileio.c` functions should only ever be used in the zstd binary, and these
are (almost) the only files ever created by zstd, and AIUI they're only
created in a single thread. So we can get away with messing with global state.

Note that this doesn't change the permissions of files created by `dibio.c`.
I'm not sure what those should be...
 2021-02-17 15:27:39 -05:00
W. Felix Handte 927859f5e8 Also Update Man Page Documentation 2021-01-11 17:55:58 -05:00
W. Felix Handte 8b6a4b5b7c Allow Input From Console When --force is Passed 
Also update option flag documentation.
 2021-01-11 17:53:20 -05:00
Felix HandteandGitHub fe48742c69 Merge pull request #2422 from felixhandte/doc-update-repcodes 
Update Zstd Compression Format to Clarify Repcode Behavior
 2020-12-10 01:20:51 -05:00
W. Felix Handte 2d46d764cf Update Zstd Compression Format to Clarify Repcode Behavior 2020-12-09 20:03:58 -05:00
Felix HandteandGitHub f861e8c07b Merge pull request #2421 from felixhandte/pc-no-sed 
Don't Use Regexes to Build Pkg-Config File
 2020-12-09 18:58:17 -05:00
Felix HandteandGitHub bee9137409 Merge pull request #2419 from felixhandte/asan-dont-poison-static-allocs 
Don't ASAN-Poison Statically-Allocated Workspaces
 2020-12-09 16:50:52 -05:00
W. Felix Handte 9dab03db90 Create Enum to Represent Static/Dynamic Allocation Distinction in cwksp 2020-12-09 14:57:37 -05:00
W. Felix Handte db9e73cb07 Don't ASAN-Poison Statically-Allocated Workspaces 
Addresses #2286.
 2020-12-09 13:00:47 -05:00
W. Felix Handte a75f9ce3e9 Apply Same Strategy to CMake 2020-12-08 20:46:37 -05:00
W. Felix Handte b521183c74 Avoid Use of Regexes in Building Package-Config File 2020-12-08 20:10:05 -05:00
Felix HandteandGitHub 200c960f1d Merge pull request #2311 from felixhandte/ddss-fix-cparam-derivation 
Fix Compression Parameter Derivation Bugs Introduced by DDSS Changes
 2020-09-18 14:02:14 -04:00
Felix HandteandGitHub 5870c47bc0 Merge pull request #2313 from felixhandte/fuzzer-separate-long-tests 
Separate Long Tests in Fuzzer
 2020-09-18 12:03:00 -04:00
W. Felix Handte e991f9994d Run Long Tests in GitHub CI 2020-09-17 13:31:10 -04:00
W. Felix Handte 9398acb245 Move Last Two Long Tests in fuzzer.c into Separate --long-tests Section 2020-09-17 13:31:10 -04:00
W. Felix Handte f23a321781 Update Regression Test Results 2020-09-17 12:23:05 -04:00
W. Felix Handte 8930c6e551 Use ZSTD_CCtxParams_init() to Init CCtxParams, not memset() 
Even if the discrepancies are at the moment benign, it's probably better to
standardize on using the one true initializer, rather than trying (and failing)
to correctly duplicate its behavior.
 2020-09-17 12:15:33 -04:00
W. Felix Handte e8a44326fa Avoid Redundancy in ZSTD_initCDict_internal() Args; Don't Take CParams + CCtxParams 2020-09-17 12:08:36 -04:00
W. Felix Handte eee51a664a Fall Back if Derived CParams are Incompatible with DDSS; Refactor CDict Creation 
Rewrite ZSTD_createCDict_advanced() as a wrapper around
ZSTD_createCDict_advanced2(). Evaluate whether to use DDSS mode *after* fully
resolving cparams. If not, fall back.
 2020-09-15 18:01:08 -04:00
W. Felix Handte bc6521a6f6 Make ZSTD_createCDict_advanced2() cctxParams Arg Const 2020-09-15 14:06:10 -04:00
W. Felix Handte 26a96a5b35 Do More Complete CParams Deduction in Non-DDSS Path of ZSTD_createCDict_advanced2 
Call ZSTD_getCParamsFromCCtxParams() instead of ZSTD_getCParams_internal().
 2020-09-15 13:57:43 -04:00
W. Felix Handte a2af804129 Pull CParam Override Logic into Helper 2020-09-15 13:38:05 -04:00
Felix HandteandGitHub d903b552c8 Merge pull request #2295 from felixhandte/dedicated-dict-search-structure-chain 
DDSS for Lazy: Implement a Dedicated Dictionary Chain Table
 2020-09-11 01:39:23 -04:00
W. Felix Handte d6246d4a0f Print More During Fuzzer Test to Avoid CI Killing it Due to Timeout 
This is kind of hacky. And maybe this test doesn't need to be permanently as
exhaustive as it is now. But while we're actively developing the DDSS, we
should ensure it's compatible across many different modes.
 2020-09-10 23:35:42 -04:00
W. Felix Handte 6d3f816b3e Test Fewer Dictionary Sizes 2020-09-10 22:30:52 -04:00
W. Felix Handte b6df3fd438 Fix Debug Logging in 32-bit Build 2020-09-10 22:10:02 -04:00
W. Felix Handte c5fab8848a Document searchFuncs Table 2020-09-10 22:10:02 -04:00
W. Felix Handte 85a95840e4 Further Consolidate Dict Mode Checks 2020-09-10 22:10:02 -04:00
W. Felix Handte 032010fcc1 Improve Documentation Slightly 2020-09-10 22:10:02 -04:00
W. Felix Handte 0faefbf1b3 Make DDSS Selection Override ForceCopy Directive 2020-09-10 22:10:02 -04:00
W. Felix Handte 2cc2b40a1b Test DDSS A Little More Thoroughly 2020-09-10 22:10:02 -04:00
W. Felix Handte efa33861f2 Attempt to Fix MSVC Warnings 2020-09-10 22:10:02 -04:00
W. Felix Handte ed43832770 Simplify Match Limit Checks 
Seems like a ~1.25% speedup.
 2020-09-10 22:10:02 -04:00
W. Felix Handte 06d240b8a7 Use All Available Space in the Hash Table to Extent Chain Table Reach 
Rather than restrict our temp chain table to 2 ** chainLog entries, this
commit uses all available space to reach further back to gather longer
chains to pack into the DDSS chain table.
 2020-09-10 22:10:02 -04:00
W. Felix Handte b2b0641ea0 Rewrite Table Fill to Retain Cache Entries Beyond Chain Window 2020-09-10 22:10:02 -04:00
W. Felix Handte 916238d9dc Avoid Malloc in Table Fill; Pack Tmp Structure into Hash Table 2020-09-10 22:10:02 -04:00
W. Felix Handte f42c5bddd9 Truncate Chain at Last Possible Attempt 
Make the chain table denser?
 2020-09-10 22:10:02 -04:00
W. Felix Handte 20a020edbc Prefetch Chain Table Matches 2020-09-10 22:10:02 -04:00
W. Felix Handte 9b9feb84f2 Lay Out Chain Table Chains Contiguously 
Rather than interleave all of the chain table entries, tying each entry's
position to the corresponding position in the input, this commit changes the
layout so that all the entries in a single chain are laid out next to each
other. The last entry in the hash table's bucket for this hash is now a packed
pointer of position + length of this chain.

This cannot be merged as written, since it allocates temporary memory inside
ZSTD_dedicatedDictSearch_lazy_loadDictionary().
 2020-09-10 22:10:02 -04:00
W. Felix Handte 66509c7bf4 Only Insert Positions Inside the Chain Window 2020-09-10 22:10:02 -04:00
W. Felix Handte 13c5ec3e41 Only Allow Dedicated Dict Search for Dicts Loaded in 1 Chunk 
The load algorithm requires we do it all in one go.
 2020-09-10 22:10:02 -04:00
Felix HandteandGitHub 005ceaa052 Merge pull request #2276 from felixhandte/dedicated-dict-search-structure 
DDSS For Lazy: Implement a Dedicated Dictionary Hash Table
 2020-09-10 22:09:25 -04:00
W. Felix Handte 07793547e6 Fix Bug: Only Use DDSS Insertion on CDict MatchStates 
Previously, if DDSS was enabled on a CCtx and a dictionary was inserted into
the CCtx, the CCtx MatchState would be filled as a DDSS struct, causing
segfaults etc. This changes the check to use whether the MatchState is marked
as using the DDSS (which is only ever set for CDict MatchStates), rather than
looking at the CCtxParams.
 2020-09-10 18:51:52 -04:00
W. Felix Handte d214d8c859 Shorten Dict Mode Conditionals in Order to Improve Readability 2020-09-10 18:51:52 -04:00
W. Felix Handte f49c1563ff Force-Inline ZSTD_insertAndFindFirstIndex_internal() 
Without this, gcc was declining to inline the function in `ZSTD_noDict` mode,
resulting in a ~10% slowdown.
 2020-09-10 18:51:52 -04:00
W. Felix Handte cab86b074f Clean Up Search Function Selection 2020-09-10 18:51:52 -04:00
W. Felix Handte 2ffbde0d95 Fix -Wshorten-64-to-32 Error 2020-09-10 18:51:52 -04:00
W. Felix Handte 7b5d2f72ea Adjust Working Context Table Sizes Back Down 2020-09-10 18:51:52 -04:00
W. Felix Handte c09454e28f Add Warning Comment to ZSTD_createCDict_advanced2() Declaration 2020-09-10 18:51:52 -04:00
W. Felix Handte d332f57897 Permit Matching Against Lowest Valid Position 
This comparison was previously faulty: the lowest valid position is itself
valid, and we should therefore be allowed to match against it.
 2020-09-10 18:51:52 -04:00
W. Felix Handte a3659fe1ef Make ZSTD_dedicatedDictSearch_getCParams Wrap ZSTD_getCParams 
Fixes up bounds-checking, and lets us clean up what is at the moment an
unnecessary duplication of the default cparams tables.
 2020-09-10 18:51:52 -04:00
W. Felix Handte b81f3a37f9 Easy: Fix Test 2020-09-10 18:51:52 -04:00
W. Felix Handte 7b9a755ac9 Remove Chain Limit on Hash Cache Entries; Slightly Improve Compression 
Entries in the hashTable chain cache aren't subject to the same aliasing that
the circular chain table is subject to. As such, we don't need to stop when we
cross the chain limit. We can delve deeper. :)
 2020-09-10 18:51:52 -04:00
W. Felix Handte e8b4011b52 Split Lookups in Hash Cache and Chain Table into Two Loops 
Sliiiight speedup.
 2020-09-10 18:51:52 -04:00
W. Felix Handte 9e83c782f8 Simplify DDS Hash Table Construction 
No need to walk the chainTable; we can just keep shifting the entries in the
hashTable.
 2020-09-10 18:51:52 -04:00
W. Felix Handte ad9f98ac3f Document the ZSTD_c_enableDedicatedDictSearch Parameter 2020-09-10 18:51:52 -04:00
W. Felix Handte 5390fee4f7 Rename and Move DD_BLOG Constant to ZSTD_LAZY_DDSS_BUCKET_LOG 2020-09-10 18:51:52 -04:00
W. Felix Handte 5e91ae27eb Prefetch First Batch of Match Positions; +11% Speed in Level 5 w/ 1 Dict 2020-09-10 18:51:52 -04:00
W. Felix Handte df386b3d8d Fix Off-By-One Error in Counting DDS Search Attempts 
This caused us to double-search the first position and fail to search the
last position in the chain, slowing down search and making it less effective.
 2020-09-10 18:51:52 -04:00
W. Felix Handte d46306087b Enable Dedicated Dict Search in the CLI 2020-09-10 18:51:52 -04:00
W. Felix Handte 914bfe7ee4 Init CCtx's Local Dict with CCtxParams 2020-09-10 18:51:52 -04:00
W. Felix Handte db2aa25252 Decision for Whether to Attach Should be Based on CDict Config, not CCtx 2020-09-10 18:51:52 -04:00
W. Felix Handte a494111385 Move Prefetch Before Insertion; Speed Up ~6% 2020-09-10 18:51:52 -04:00
W. Felix Handte eede46a47e Misc Refactor of DDS Search Code 2020-09-10 18:51:52 -04:00
W. Felix Handte f1b428fdac Rename enableDedicatedDictSearch to dedicatedDictSearch in MatchState 
This makes it clear that not only is the feature allowed here, we're actually
using it, as opposed to the CCtxParam field, in which it's enabled, but we may
or may not be using it.
 2020-09-10 18:51:52 -04:00
W. Felix Handte 41012193ad Always Init CDict's enableDedicatedDictSearch Field 2020-09-10 18:51:52 -04:00
W. Felix Handte 34b545acb0 Add a ZSTD_dedicatedDictSearch ZSTD_dictMode_e to Allow Const Propagation 
Speed +1.5%.
 2020-09-10 18:51:52 -04:00