ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,431 Commits 17 Branches 77 Tags
1e240af30a1d11ae45745c6c3e96307bad3771fd
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Felix HandteandGitHub 889392dac8 Merge pull request #3909 from felixhandte/security-md-notif-list-ad 
Advertise Availability of Security Vulnerability Notifications
 2024-02-13 12:38:18 -05:00
W. Felix Handte e13d099bf8 Advertise Availability of Security Vulnerability Notifications 2024-02-13 11:54:59 -05:00
W. Felix Handte b6805c54d6 Add SECURITY.md File 
This just adds a copy of the Meta default SECURITY.md that we can then modify.
 2024-02-13 11:50:55 -05:00
Felix HandteandGitHub 7afd9db8db Merge pull request #3893 from felixhandte/gh-actions-cet-test-update-dep 
Update Dependency in Intel CET Test; Re-Enable Test
 2024-02-01 14:49:37 -05:00
W. Felix Handte 04a6c8cbe2 Update Dependency in Intel CET Test; Re-Enable Test 
Motivated by #3884.
 2024-02-01 10:24:42 -08:00
Felix HandteandGitHub edb6e8f427 Merge pull request #3730 from facebook/dependabot/github_actions/github/codeql-action-2.21.4 
Bump github/codeql-action from 2.20.3 to 2.21.4
 2023-08-21 09:47:26 -07:00
Felix HandteandGitHub 3298a08076 Merge pull request #3725 from felixhandte/msan-unpoison-cwksp-on-free 
Unpoison Workspace Memory Before Custom-Free
 2023-08-16 10:18:31 -07:00
W. Felix Handte 9987d2f594 Unpoison Workspace Memory Before Freeing to Custom Free 
MSAN is hooked into the system malloc, but when the user provides a custom
allocator, it may not provide the same cleansing behavior. So if we leave
memory poisoned and return it to the user's allocator, where it is re-used
elsewhere, our poisoning can blow up in some other context.
 2023-08-16 12:09:12 -04:00
W. Felix Handte 5f5bdc1e5d Easy: Move Helper Functions Up 2023-08-16 12:08:52 -04:00
Felix HandteandGitHub ed6953772f Merge pull request #3623 from felixhandte/compile-out-compressors 
Allow Build-Time Exclusion of Individual Compression Strategies
 2023-05-22 17:38:10 -04:00
W. Felix Handte 1b65803fe7 Reorder Definitions in zstd_opt.c to Group Under Macro Guards (Slightly) 2023-05-22 12:41:48 -04:00
W. Felix Handte 59c7b2a492 Reorder Definitions in zstd_lazy.c to Group Under Macro Guards 2023-05-22 12:37:03 -04:00
W. Felix Handte 5490c75dda Also Allow/Document/Test Excluding dfast and Up 2023-05-04 12:31:41 -04:00
W. Felix Handte cc1ffe0bd6 Add Documentation to lib/README.md 2023-05-04 12:20:02 -04:00
W. Felix Handte eb9227935e Also Reorganize Zstd Opt Declarations 2023-05-04 12:18:58 -04:00
W. Felix Handte d09f195ceb Remove blockCompressor NULL Checks 2023-05-04 12:18:58 -04:00
W. Felix Handte b7add1dd67 Abort if Unsupported Parameters Used 2023-05-04 12:18:58 -04:00
W. Felix Handte f242f5be8f Re-Order Lazy Declarations; Minimize ifndefs 2023-05-04 12:18:58 -04:00
W. Felix Handte 698af84fcf Add CI Test for Excluding Matchfinders 2023-05-04 12:18:58 -04:00
W. Felix Handte bae174960b Add ZSTD_LIB_EXCLUDE_COMPRESSORS_DFAST_AND_UP Build Variable 2023-05-04 12:18:58 -04:00
W. Felix Handte 39b7946b95 Define Macros for Possibly-Present Functions; Use Them Rather than Ifdef Guards 2023-05-04 12:18:58 -04:00
W. Felix Handte b12e8cb3e7 Merge Ultra and Ultra2 Exclusion 
Ultra2 does not exist for dict compression, and so uses ultra. So ultra must
be present if ultra2 is.
 2023-05-04 12:18:58 -04:00
W. Felix Handte 6761e1c949 Tweak Ultra/Opt Guards 2023-05-04 12:18:58 -04:00
W. Felix Handte 16bbd7437c Avoid Ratio Regression Tests When Compressors are Excluded 2023-05-04 12:18:58 -04:00
W. Felix Handte 5a75956001 Adjust Strategy in CParams to Avoid Using Excluded Block Compressors 2023-05-04 12:18:58 -04:00
W. Felix Handte 50cdf84f58 Macro-Exclude Block Compressors from Declaration/Definition 2023-05-04 12:18:58 -04:00
W. Felix Handte 81b86a2024 NULL Out Block Compressor Table Entries When Excluded 
Don't check about excluding `ZSTD_fast`. It's always included so that we know
we can resolve downwards and hit a strategy that's present.
 2023-05-04 12:18:58 -04:00
W. Felix Handte cbf3e26316 Allow ZSTD_selectBlockCompressor() to Return NULL 
Return an error rather than segfaulting.
 2023-05-04 12:18:58 -04:00
Felix HandteandGitHub 7806d80338 Merge pull request #3634 from facebook/dependabot/github_actions/github/codeql-action-2.3.2 
Bump github/codeql-action from 2.3.0 to 2.3.2
 2023-05-01 08:06:07 -07:00
Felix HandteandGitHub 0f255ff4fa Merge pull request #3606 from facebook/dependabot/github_actions/github/codeql-action-2.2.11 
Bump github/codeql-action from 2.2.9 to 2.2.11
 2023-04-10 08:32:54 -07:00
Felix HandteandGitHub 2b71b79f98 Merge pull request #3591 from felixhandte/win-rel-artifact-name 
Rename/Restructure Windows Release Artifact
 2023-04-03 13:46:17 -07:00
W. Felix Handte fcaa422897 Rename/Restructure Windows Release Artifact 
https://github.com/facebook/zstd/releases/tag/v1.5.0 describes the structure
we want to adhere to. This commit tries to accomplish that automatically, so
we can avoid manual fixups on future releases.
 2023-04-03 15:35:57 -04:00
Felix HandteandGitHub 93da0416e8 Merge pull request #3574 from felixhandte/pzstd-max-cpp-std 
[contrib/pzstd] Detect and Select Maximum Available C++ Standard
 2023-03-27 16:44:07 -07:00
W. Felix Handte cbe0f0e435 Switch Strategies: Only Set -std=c++11 When Default is Older 2023-03-27 18:37:19 -04:00
W. Felix Handte 1b8bddc41e [contrib/pzstd] Detect and Select Maximum Available C++ Standard 
Rather than remove the flag entirely, as proposed in #3499, this commit uses
the newest C++ standard the compiler supports. This retains the selection of
using only standardized features (excluding GNU extensions) and keeps the
recency requirements of the codebase explicit.

Tested with various versions of `g++` and `clang++`.
 2023-03-27 11:24:47 -04:00
Felix HandteandGitHub d55a6483d7 Merge pull request #3542 from felixhandte/pin-moar-action-deps 
Pin Moar Action Dependencies
 2023-03-09 16:22:11 -08:00
W. Felix Handte cd9486031d Also Pin Dockerfile Dependency Hashes 2023-03-09 17:01:22 -05:00
Felix HandteandGitHub 283c228abe Merge pull request #3541 from felixhandte/fix-setvbuf-segfault 
Avoid Segfault Caused by Calling `setvbuf()` on Null File Pointer
 2023-03-09 13:54:11 -08:00
W. Felix Handte 1ec556238e Pin Moar Action Dependencies 
An offering to the Scorecard gods, may they have mercy on our souls.
 2023-03-09 12:54:07 -05:00
W. Felix Handte 957a0ae52d Add CLI Test 2023-03-09 12:48:11 -05:00
W. Felix Handte c4c3e11958 Avoid Calling setvbuf() on Null File Pointer 2023-03-09 12:47:40 -05:00
W. Felix Handte 50e8f55e7d Fix Python 3.6 Incompatibility in CLI Tests 2023-03-09 12:46:37 -05:00
Felix HandteandGitHub 1c42844668 Merge pull request #3479 from felixhandte/faster-file-ops 
Use `f`-variants of `chmod()` and `chown()`
 2023-02-16 13:07:34 -05:00
Felix HandteandGitHub 3c50854c05 Merge pull request #3511 from felixhandte/fix-release-artifact-upload-permission 
Fix Permissions on Publish Release Artifacts Job
 2023-02-15 13:35:04 -05:00
W. Felix Handte d54ad3c234 Fix Permissions on Publish Release Artifacts Job 
Publishing release artifacts requires the `contents` permission, as documented
by: https://docs.github.com/en/rest/overview/permissions-required-for-github-apps.
 2023-02-15 11:05:54 -05:00
W. Felix Handte f746c37d00 Use File Descriptor in Setting Stat on Output File 
Note that the `fd` is only valid while the file is still open. So we need to
move the setting calls to before we close the file. However! We cannot do so
with the `utime()` call (even though `futimens()` exists) because the follow-
ing `close()` call to the `fd` will reset the atime of the file. So it seems
the `utime()` call has to happen after the file is closed.
 2023-02-06 14:02:31 -08:00
W. Felix Handte a5a2418df4 Introduce Variants of Some Functions that Take Optional File Descriptors 
Somewhat surprisingly, calling `fchmod()` is non-trivially faster than calling
`chmod()`, and so on.

This commit introduces alternate variants to some common file util functions
that take an optional fd. If present, they call the `f`-variant of the
underlying function. Otherwise, they fall back to the regular filename-taking
version of the function.
 2023-02-06 13:55:34 -08:00
Felix HandteandGitHub 54a173e9fc Merge pull request #3477 from facebook/dependabot/github_actions/ilammy/msvc-dev-cmd-1.12.1 
Bump ilammy/msvc-dev-cmd from 1.12.0 to 1.12.1
 2023-02-06 13:04:21 -05:00
Felix HandteandGitHub 3d25502c2d Merge pull request #3432 from felixhandte/fix-perms 
Fix CLI Handling of Permissions and Ownership (Again)
 2023-01-20 19:19:05 -05:00
Felix HandteandGitHub 772229afd5 Merge pull request #3442 from felixhandte/pgo-tests 
Test PGO Builds
 2023-01-20 19:18:51 -05:00
W. Felix Handte aab3dd4312 Add PGO Build Jobs to CI 2023-01-20 18:37:04 -05:00
W. Felix Handte 87e169d05d Add Additional Flags to PGO Build 
In GCC, we can add a couple more flags to give us confidence that the profile
data is actually being found and used.

Also, my system for example doesn't have a binary installed under the name
`llvm-profdata`, but it does have, e.g., `llvm-profdata-13`, etc. So this
commit adds a variable that can be overridden.
 2023-01-20 17:32:49 -05:00
W. Felix Handte 7a8c8f3fe7 Easy: Print Mode as Octal in chmod() Trace 2023-01-18 11:57:54 -08:00
W. Felix Handte 0d2d460223 Mimic gzip chown(gid), chmod(), chown(uid) Behavior 
Avoids a race condition in which we unintentionally open up permissions to
the wrong group.
 2023-01-18 11:57:54 -08:00
W. Felix Handte 1e3eba65a6 Copy Permissions from Source File 2023-01-18 11:57:35 -08:00
W. Felix Handte 0382076af7 Re-Use stat_t in FIO_compressFilename_srcFile() 2023-01-18 11:33:07 -08:00
W. Felix Handte a5ed28f1fb Use Existing Src File Stat in *_dstFile() Funcs 
One fewer `stat()` call to make per operation!
 2023-01-17 14:08:22 -08:00
W. Felix Handte 5653f96776 Pass stat_t into *_dstFile() Functions 2023-01-17 14:05:15 -08:00
W. Felix Handte 2ad6855ac1 FIO_openSrcFile() Returns File Stat 2023-01-17 14:01:06 -08:00
Felix HandteandGitHub 23a356cdde Merge pull request #3424 from felixhandte/disable-asan-msan-poison-mingw 
Disable Custom ASAN/MSAN Poisoning on MinGW Builds
 2023-01-17 12:41:41 -05:00
W. Felix Handte d78fbedd96 Don't Even Declare Poisoning Functions if Poisoning is Disabled 
This guarantees that we won't accidentally forget to check the macro somewhere
where we use these functions.
 2023-01-13 11:56:48 -05:00
W. Felix Handte f10922a8fa Disable Custom ASAN/MSAN Poisoning on MinGW Builds 
Addresses #3240.
 2023-01-13 11:53:09 -05:00
Felix HandteandGitHub fd2eb8a68e Merge pull request #3402 from facebook/dependabot/github_actions/ossf/scorecard-action-2.1.2 
Bump ossf/scorecard-action from 2.1.0 to 2.1.2
 2023-01-12 13:28:04 -05:00
Felix HandteandGitHub f5ea3a196f Merge pull request #3397 from felixhandte/man-page-tweaks 
Man Page Tweaks, Edits, Formatting Fixes
 2022-12-22 14:49:59 -05:00
W. Felix Handte 11aba9b316 make man 2022-12-22 14:13:24 -05:00
W. Felix Handte 382026f096 Man Page Tweaks, Edits, Formatting Fixes 
This started as an application of the edits suggested in #3201 and expanded
from there.
 2022-12-22 14:13:17 -05:00
Felix HandteandGitHub 0694f1435d Merge pull request #3386 from felixhandte/pin-other-action-commit-hashes 
Pin Remaining Action Dependencies (Except OSS-Fuzz)
 2022-12-21 11:24:23 -05:00
W. Felix Handte 150aa23ef4 Pin Remaining Action Dependencies (Except OSS-Fuzz) 
The one that isn't pinned is the OSS-Fuzz builder and runner. They don't
offer tagged releases. I could pin to the current master commit, but I'm not
sure how desirable that is.
 2022-12-20 17:10:01 -05:00
Felix HandteandGitHub 0790e86c56 Merge pull request #3384 from felixhandte/pin-checkout-action-commit-hash 
Pin actions/checkout Dependency to Specific Commit Hash
 2022-12-20 16:08:03 -05:00
W. Felix Handte e3f2c8b11c Pin actions/checkout Dependency to Specific Commit Hash 
It's a bit silly, because if we can't trust GitHub, what are we doing here?
But OSSF complains about it, so let's fix it.
 2022-12-20 14:53:12 -05:00
Felix HandteandGitHub ea199876e2 Merge pull request #3378 from facebook/dependabot/github_actions/github/codeql-action-2.1.37 
Bump github/codeql-action from 1.0.26 to 2.1.37
 2022-12-20 14:17:33 -05:00
Felix HandteandGitHub 5f0543a508 Merge pull request #3377 from facebook/dependabot/github_actions/ossf/scorecard-action-2.1.0 
Bump ossf/scorecard-action from 2.0.6 to 2.1.0
 2022-12-20 14:16:10 -05:00
Felix HandteandGitHub 3a7e2eec21 Merge pull request #3340 from facebook/dependabot/github_actions/actions/upload-artifact-3 
Bump actions/upload-artifact from 1 to 3
 2022-12-20 14:15:43 -05:00
Felix HandteandGitHub f302ad8811 Merge pull request #3173 from felixhandte/update-copyright-company 
Update Copyright Comments
 2022-12-20 13:51:01 -05:00
W. Felix Handte 5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
 2022-12-20 12:52:34 -05:00
W. Felix Handte 7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022 
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
 2022-12-20 12:44:56 -05:00
W. Felix Handte 4c999cf968 Update test-license.py 2022-12-20 12:43:16 -05:00
W. Felix Handte 36d5c2f326 Update Copyright Year ('2021' -> 'present') 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f);
do
  sed -i 's/\-2021/-present/' $f;
done

g co HEAD -- .github/workflows/dev-short-tests.yml # fix bad match
```
 2022-12-20 12:42:50 -05:00
W. Felix Handte 8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms' 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
 2022-12-20 12:37:57 -05:00
W. Felix Handte a05cca6208 Manually Update VS Code Copyright Definitions 2022-12-20 12:36:06 -05:00
Felix HandteandGitHub bd4581c311 Merge pull request #3309 from joycebrum/dev 
Fix action error by upgrading Scorecard GHA to 2.0.6
 2022-11-01 17:53:37 -04:00
Felix HandteandGitHub 99d239de32 Merge pull request #3290 from felixhandte/ddict-dict-id-from-ddict 
Make ZSTD_getDictID_fromDDict() Read DictID from DDict
 2022-10-18 13:33:32 -04:00
W. Felix Handte d7841d150b Make ZSTD_getDictID_fromDDict() Read DictID from DDict 
Currently this function actually reads the dict ID from the dictionary's
header, via `ZSTD_getDictID_fromDict()`. But during decompression the decomp-
ressor actually compares the dict ID in the frame header with the dict ID in
the DDict. Now of course the dict ID in the dictionary contents and the dict
ID in the DDict struct *should* be the same. But in cases of memory corrupt-
ion, where they can drift out of sync, it's misleading for this function to
read it again from the dict buffer rather then return the dict ID that will
actually be used.

Also doing it this way avoids rechecking the magic and so on and so it is a
tiny bit more efficient.
 2022-10-14 22:53:03 -04:00
Felix HandteandGitHub 33273e1b9c Merge pull request #3277 from felixhandte/ossf-scorecard 
Enable OpenSSF Scorecard Action
 2022-10-03 14:33:21 -04:00
W. Felix Handte de9a450c00 Restrict from Running on Forks 2022-10-03 11:23:29 -04:00
Felix Handte 79729f8a2d Create scorecards.yml 
Addresses #3269.
 2022-09-30 14:30:22 -04:00
Felix HandteandGitHub 7e6278a706 Merge pull request #3196 from mileshu/dev 
[T124890272] Mark 2 Obsolete Functions(ZSTD_copy*Ctx) Deprecated in Zstd
 2022-08-02 12:34:04 -04:00
Felix HandteandGitHub 02ef78be58 Merge pull request #3184 from htnhan/features/list_verbose_to_show_dictionary_id 
zstd -lv <file> to show dictID
 2022-07-08 16:04:39 -04:00
Felix HandteandGitHub 8af64f4116 Merge pull request #3129 from felixhandte/zstd-fast-nodict-unconditional-ip1-table-write 
ZSTD_fast_noDict: Avoid Safety Check When Writing `ip1` into Table
 2022-05-11 17:04:02 -04:00
W. Felix Handte 1bc8019e10 Update results.csv 2022-05-11 10:27:35 -07:00
W. Felix Handte 1dd046a507 Fix Comments Slightly 2022-05-11 12:38:45 -04:00
W. Felix Handte cd1f582943 Hoist Hash Table Writes Up into Each Match Found Block 
Refactoring this way avoids the bad write in the case that `step > 4`, and
is a bit more straightforward. It also seems to perform better!
 2022-05-11 11:27:34 -04:00
W. Felix Handte 040986a4f4 ZSTD_fast_noDict: Minimize Checks When Writing Hash Table for ip1 
This commit avoids checking whether a hashtable write is safe in two of the
three match-found paths in `ZSTD_compressBlock_fast_noDict_generic`. This pro-
duces a ~0.5% speed-up in compression.

A comment in the code describes why we can skip this check in the other two
paths (the repcode check and the first match check in the unrolled loop).

A downside is that in the new position where we make this check, we have not
yet computed `mLength`. We therefore have to avoid writing *possibly* dangerous
positions, rather than the old check which only avoids writing *actually*
dangerous positions. This leads to a miniscule loss in ratio (remember that
this scenario can only been triggered in very negative levels or under incomp-
ressibility acceleration).
 2022-05-10 14:29:39 -07:00
Felix HandteandGitHub 8814aa5bfa Merge pull request #3059 from dirkmueller/gzip_fast_best 
Implement more gzip compatibility (#3037)
 2022-03-08 15:32:06 -05:00
Felix HandteandGitHub 4a35912b3b Merge pull request #3018 from felixhandte/gh-action-release-artifacts-trigger-fix 
Trigger Release Artifact Generation on Publish
 2022-01-20 21:08:59 -05:00
W. Felix Handte fa9cb4510a Trigger Release Artifact Generation on Publish 
We previously triggered release artifact generation on release creation. We
sometimes observed that the action failed to run. I hypothesized that we were
hitting rate limiting or something. I just stumbled across [this documentat-
ion](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release), which says:

> Note: Workflows are not triggered for the `created`, `edited`, or `deleted`
> activity types for draft releases. When you create your release through the
> GitHub browser UI, your release may automatically be saved as a draft.

This must have been what was happening. This commit therefore changes the
trigger to the `published` activity. This should be more reliable.

This does have the unfortunate side effect that artifacts won't be generated
or attached until *after* the release has been published, which is what I was
trying to avoid by using the `created` activity. Oh well.
 2022-01-20 17:36:28 -05:00
Felix HandteandGitHub 330c97d2bf Merge pull request #3015 from felixhandte/enable-cet-test-illegal-instruction 
Add GitHub Action Checking that Zstd Runs Successfully Under CET
 2022-01-20 16:18:42 -05:00
Felix HandteandGitHub c7e8315d88 Merge pull request #2994 from hjl-tools/hjl/cet-report/dev 
x86: Append -z cet-report=error to LDFLAGS
 2022-01-20 14:58:13 -05:00
W. Felix Handte 7cf80cb94c Add GitHub Action Checking that Zstd Runs Successfully Under CET 2022-01-20 11:25:52 -05:00
Felix HandteandGitHub 4dfc4eca9a Merge pull request #2992 from hjl-tools/hjl/cet/dev 
x86-64: Enable Intel CET
 2022-01-20 11:24:47 -05:00