ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,596 Commits 17 Branches 77 Tags
f1f1ae369a4cefd3474b3528e8d1847b18750605
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Felix HandteandGitHub 515c07a131 Merge pull request #3964 from felixhandte/promote-tgt-c-blk-size-to-stable 
Promote `ZSTD_c_targetCBlockSize` Parameter to Stable API
 2024-03-14 13:06:46 -04:00
Felix HandteandGitHub 490163ac73 Merge pull request #3963 from felixhandte/also-handle-hidden-files-output-dir-mirror 
Remove Erroneous Exclusion of Hidden Files and Folders in `--output-dir-mirror`
 2024-03-14 10:21:00 -04:00
Felix HandteandGitHub 04d91d5219 Merge pull request #3960 from felixhandte/use-utimensat-on-posix-2001 
Use `utimensat()` on FreeBSD
 2024-03-13 17:10:35 -04:00
W. Felix Handte 3613448fb8 Promote ZSTD_c_targetCBlockSize Parameter to Stable API 
This feature has demonstrated itself to be useful in web compression and we
want to encourage other folks to use it. But we currently make it difficult
to do so since it's locked away in the experimental API.

The API itself is really straightforward and I think it's fine to commit to
maintaining support / compatibility for this API even if in the future the
underlying implementation may continue to evolve.

Note that this commit changes its enum name and also its numeric value. Users
who respected the instructions of using the experimental API should be fine
with both of these changes since they should only have referred to it by the.

Conceivably someone could have done bad feature detection of this capability
by doing `#ifdef ZSTD_c_targetCBlockSize` which will now return false since
it's no longer a macro... but I think that's an acceptable hypothetical
breakage.
 2024-03-13 17:07:10 -04:00
W. Felix Handte 86b8e39a84 Remove Erroneous Exclusion of Hidden Files and Folders in --output-dir-mirror 2024-03-13 16:33:30 -04:00
W. Felix Handte 2215101cad Add a Few Tests 2024-03-13 16:32:04 -04:00
W. Felix Handte d6ee2d5d24 Use utimensat() on FreeBSD 
FreeBSD only claims to support POSIX 2001 [0]. But they do in fact support
`utimensat()`. This adds a specific check to opt them in to using it. This
value was selected by consulting [1].

See discussion on #3952.

Further addresses #3748.

[0] https://github.com/freebsd/freebsd-src/blob/937a0055858a098027f464abf0b2b1ec5d36748f/sys/sys/unistd.h#L96

[1] https://docs.freebsd.org/en/books/porters-handbook/versions/
 2024-03-13 13:26:21 -07:00
Felix HandteandGitHub a6ff1194d8 Merge pull request #3952 from felixhandte/stop-hardcoding-posix-version-on-bsd 
Stop Hardcoding the POSIX Version on BSDs
 2024-03-12 12:49:19 -04:00
W. Felix Handte f99a450ca4 Stop Hardcoding the POSIX Version on BSDs 
BSDs should all have a `unistd.h` header.
 2024-03-11 17:25:51 -04:00
Felix HandteandGitHub 889392dac8 Merge pull request #3909 from felixhandte/security-md-notif-list-ad 
Advertise Availability of Security Vulnerability Notifications
 2024-02-13 12:38:18 -05:00
W. Felix Handte e13d099bf8 Advertise Availability of Security Vulnerability Notifications 2024-02-13 11:54:59 -05:00
W. Felix Handte b6805c54d6 Add SECURITY.md File 
This just adds a copy of the Meta default SECURITY.md that we can then modify.
 2024-02-13 11:50:55 -05:00
Felix HandteandGitHub 7afd9db8db Merge pull request #3893 from felixhandte/gh-actions-cet-test-update-dep 
Update Dependency in Intel CET Test; Re-Enable Test
 2024-02-01 14:49:37 -05:00
W. Felix Handte 04a6c8cbe2 Update Dependency in Intel CET Test; Re-Enable Test 
Motivated by #3884.
 2024-02-01 10:24:42 -08:00
Felix HandteandGitHub edb6e8f427 Merge pull request #3730 from facebook/dependabot/github_actions/github/codeql-action-2.21.4 
Bump github/codeql-action from 2.20.3 to 2.21.4
 2023-08-21 09:47:26 -07:00
Felix HandteandGitHub 3298a08076 Merge pull request #3725 from felixhandte/msan-unpoison-cwksp-on-free 
Unpoison Workspace Memory Before Custom-Free
 2023-08-16 10:18:31 -07:00
W. Felix Handte 9987d2f594 Unpoison Workspace Memory Before Freeing to Custom Free 
MSAN is hooked into the system malloc, but when the user provides a custom
allocator, it may not provide the same cleansing behavior. So if we leave
memory poisoned and return it to the user's allocator, where it is re-used
elsewhere, our poisoning can blow up in some other context.
 2023-08-16 12:09:12 -04:00
W. Felix Handte 5f5bdc1e5d Easy: Move Helper Functions Up 2023-08-16 12:08:52 -04:00
Felix HandteandGitHub ed6953772f Merge pull request #3623 from felixhandte/compile-out-compressors 
Allow Build-Time Exclusion of Individual Compression Strategies
 2023-05-22 17:38:10 -04:00
W. Felix Handte 1b65803fe7 Reorder Definitions in zstd_opt.c to Group Under Macro Guards (Slightly) 2023-05-22 12:41:48 -04:00
W. Felix Handte 59c7b2a492 Reorder Definitions in zstd_lazy.c to Group Under Macro Guards 2023-05-22 12:37:03 -04:00
W. Felix Handte 5490c75dda Also Allow/Document/Test Excluding dfast and Up 2023-05-04 12:31:41 -04:00
W. Felix Handte cc1ffe0bd6 Add Documentation to lib/README.md 2023-05-04 12:20:02 -04:00
W. Felix Handte eb9227935e Also Reorganize Zstd Opt Declarations 2023-05-04 12:18:58 -04:00
W. Felix Handte d09f195ceb Remove blockCompressor NULL Checks 2023-05-04 12:18:58 -04:00
W. Felix Handte b7add1dd67 Abort if Unsupported Parameters Used 2023-05-04 12:18:58 -04:00
W. Felix Handte f242f5be8f Re-Order Lazy Declarations; Minimize ifndefs 2023-05-04 12:18:58 -04:00
W. Felix Handte 698af84fcf Add CI Test for Excluding Matchfinders 2023-05-04 12:18:58 -04:00
W. Felix Handte bae174960b Add ZSTD_LIB_EXCLUDE_COMPRESSORS_DFAST_AND_UP Build Variable 2023-05-04 12:18:58 -04:00
W. Felix Handte 39b7946b95 Define Macros for Possibly-Present Functions; Use Them Rather than Ifdef Guards 2023-05-04 12:18:58 -04:00
W. Felix Handte b12e8cb3e7 Merge Ultra and Ultra2 Exclusion 
Ultra2 does not exist for dict compression, and so uses ultra. So ultra must
be present if ultra2 is.
 2023-05-04 12:18:58 -04:00
W. Felix Handte 6761e1c949 Tweak Ultra/Opt Guards 2023-05-04 12:18:58 -04:00
W. Felix Handte 16bbd7437c Avoid Ratio Regression Tests When Compressors are Excluded 2023-05-04 12:18:58 -04:00
W. Felix Handte 5a75956001 Adjust Strategy in CParams to Avoid Using Excluded Block Compressors 2023-05-04 12:18:58 -04:00
W. Felix Handte 50cdf84f58 Macro-Exclude Block Compressors from Declaration/Definition 2023-05-04 12:18:58 -04:00
W. Felix Handte 81b86a2024 NULL Out Block Compressor Table Entries When Excluded 
Don't check about excluding `ZSTD_fast`. It's always included so that we know
we can resolve downwards and hit a strategy that's present.
 2023-05-04 12:18:58 -04:00
W. Felix Handte cbf3e26316 Allow ZSTD_selectBlockCompressor() to Return NULL 
Return an error rather than segfaulting.
 2023-05-04 12:18:58 -04:00
Felix HandteandGitHub 7806d80338 Merge pull request #3634 from facebook/dependabot/github_actions/github/codeql-action-2.3.2 
Bump github/codeql-action from 2.3.0 to 2.3.2
 2023-05-01 08:06:07 -07:00
Felix HandteandGitHub 0f255ff4fa Merge pull request #3606 from facebook/dependabot/github_actions/github/codeql-action-2.2.11 
Bump github/codeql-action from 2.2.9 to 2.2.11
 2023-04-10 08:32:54 -07:00
Felix HandteandGitHub 2b71b79f98 Merge pull request #3591 from felixhandte/win-rel-artifact-name 
Rename/Restructure Windows Release Artifact
 2023-04-03 13:46:17 -07:00
W. Felix Handte fcaa422897 Rename/Restructure Windows Release Artifact 
https://github.com/facebook/zstd/releases/tag/v1.5.0 describes the structure
we want to adhere to. This commit tries to accomplish that automatically, so
we can avoid manual fixups on future releases.
 2023-04-03 15:35:57 -04:00
Felix HandteandGitHub 93da0416e8 Merge pull request #3574 from felixhandte/pzstd-max-cpp-std 
[contrib/pzstd] Detect and Select Maximum Available C++ Standard
 2023-03-27 16:44:07 -07:00
W. Felix Handte cbe0f0e435 Switch Strategies: Only Set -std=c++11 When Default is Older 2023-03-27 18:37:19 -04:00
W. Felix Handte 1b8bddc41e [contrib/pzstd] Detect and Select Maximum Available C++ Standard 
Rather than remove the flag entirely, as proposed in #3499, this commit uses
the newest C++ standard the compiler supports. This retains the selection of
using only standardized features (excluding GNU extensions) and keeps the
recency requirements of the codebase explicit.

Tested with various versions of `g++` and `clang++`.
 2023-03-27 11:24:47 -04:00
Felix HandteandGitHub d55a6483d7 Merge pull request #3542 from felixhandte/pin-moar-action-deps 
Pin Moar Action Dependencies
 2023-03-09 16:22:11 -08:00
W. Felix Handte cd9486031d Also Pin Dockerfile Dependency Hashes 2023-03-09 17:01:22 -05:00
Felix HandteandGitHub 283c228abe Merge pull request #3541 from felixhandte/fix-setvbuf-segfault 
Avoid Segfault Caused by Calling `setvbuf()` on Null File Pointer
 2023-03-09 13:54:11 -08:00
W. Felix Handte 1ec556238e Pin Moar Action Dependencies 
An offering to the Scorecard gods, may they have mercy on our souls.
 2023-03-09 12:54:07 -05:00
W. Felix Handte 957a0ae52d Add CLI Test 2023-03-09 12:48:11 -05:00
W. Felix Handte c4c3e11958 Avoid Calling setvbuf() on Null File Pointer 2023-03-09 12:47:40 -05:00
W. Felix Handte 50e8f55e7d Fix Python 3.6 Incompatibility in CLI Tests 2023-03-09 12:46:37 -05:00
Felix HandteandGitHub 1c42844668 Merge pull request #3479 from felixhandte/faster-file-ops 
Use `f`-variants of `chmod()` and `chown()`
 2023-02-16 13:07:34 -05:00
Felix HandteandGitHub 3c50854c05 Merge pull request #3511 from felixhandte/fix-release-artifact-upload-permission 
Fix Permissions on Publish Release Artifacts Job
 2023-02-15 13:35:04 -05:00
W. Felix Handte d54ad3c234 Fix Permissions on Publish Release Artifacts Job 
Publishing release artifacts requires the `contents` permission, as documented
by: https://docs.github.com/en/rest/overview/permissions-required-for-github-apps.
 2023-02-15 11:05:54 -05:00
W. Felix Handte f746c37d00 Use File Descriptor in Setting Stat on Output File 
Note that the `fd` is only valid while the file is still open. So we need to
move the setting calls to before we close the file. However! We cannot do so
with the `utime()` call (even though `futimens()` exists) because the follow-
ing `close()` call to the `fd` will reset the atime of the file. So it seems
the `utime()` call has to happen after the file is closed.
 2023-02-06 14:02:31 -08:00
W. Felix Handte a5a2418df4 Introduce Variants of Some Functions that Take Optional File Descriptors 
Somewhat surprisingly, calling `fchmod()` is non-trivially faster than calling
`chmod()`, and so on.

This commit introduces alternate variants to some common file util functions
that take an optional fd. If present, they call the `f`-variant of the
underlying function. Otherwise, they fall back to the regular filename-taking
version of the function.
 2023-02-06 13:55:34 -08:00
Felix HandteandGitHub 54a173e9fc Merge pull request #3477 from facebook/dependabot/github_actions/ilammy/msvc-dev-cmd-1.12.1 
Bump ilammy/msvc-dev-cmd from 1.12.0 to 1.12.1
 2023-02-06 13:04:21 -05:00
Felix HandteandGitHub 3d25502c2d Merge pull request #3432 from felixhandte/fix-perms 
Fix CLI Handling of Permissions and Ownership (Again)
 2023-01-20 19:19:05 -05:00
Felix HandteandGitHub 772229afd5 Merge pull request #3442 from felixhandte/pgo-tests 
Test PGO Builds
 2023-01-20 19:18:51 -05:00
W. Felix Handte aab3dd4312 Add PGO Build Jobs to CI 2023-01-20 18:37:04 -05:00
W. Felix Handte 87e169d05d Add Additional Flags to PGO Build 
In GCC, we can add a couple more flags to give us confidence that the profile
data is actually being found and used.

Also, my system for example doesn't have a binary installed under the name
`llvm-profdata`, but it does have, e.g., `llvm-profdata-13`, etc. So this
commit adds a variable that can be overridden.
 2023-01-20 17:32:49 -05:00
W. Felix Handte 7a8c8f3fe7 Easy: Print Mode as Octal in chmod() Trace 2023-01-18 11:57:54 -08:00
W. Felix Handte 0d2d460223 Mimic gzip chown(gid), chmod(), chown(uid) Behavior 
Avoids a race condition in which we unintentionally open up permissions to
the wrong group.
 2023-01-18 11:57:54 -08:00
W. Felix Handte 1e3eba65a6 Copy Permissions from Source File 2023-01-18 11:57:35 -08:00
W. Felix Handte 0382076af7 Re-Use stat_t in FIO_compressFilename_srcFile() 2023-01-18 11:33:07 -08:00
W. Felix Handte a5ed28f1fb Use Existing Src File Stat in *_dstFile() Funcs 
One fewer `stat()` call to make per operation!
 2023-01-17 14:08:22 -08:00
W. Felix Handte 5653f96776 Pass stat_t into *_dstFile() Functions 2023-01-17 14:05:15 -08:00
W. Felix Handte 2ad6855ac1 FIO_openSrcFile() Returns File Stat 2023-01-17 14:01:06 -08:00
Felix HandteandGitHub 23a356cdde Merge pull request #3424 from felixhandte/disable-asan-msan-poison-mingw 
Disable Custom ASAN/MSAN Poisoning on MinGW Builds
 2023-01-17 12:41:41 -05:00
W. Felix Handte d78fbedd96 Don't Even Declare Poisoning Functions if Poisoning is Disabled 
This guarantees that we won't accidentally forget to check the macro somewhere
where we use these functions.
 2023-01-13 11:56:48 -05:00
W. Felix Handte f10922a8fa Disable Custom ASAN/MSAN Poisoning on MinGW Builds 
Addresses #3240.
 2023-01-13 11:53:09 -05:00
Felix HandteandGitHub fd2eb8a68e Merge pull request #3402 from facebook/dependabot/github_actions/ossf/scorecard-action-2.1.2 
Bump ossf/scorecard-action from 2.1.0 to 2.1.2
 2023-01-12 13:28:04 -05:00
Felix HandteandGitHub f5ea3a196f Merge pull request #3397 from felixhandte/man-page-tweaks 
Man Page Tweaks, Edits, Formatting Fixes
 2022-12-22 14:49:59 -05:00
W. Felix Handte 11aba9b316 make man 2022-12-22 14:13:24 -05:00
W. Felix Handte 382026f096 Man Page Tweaks, Edits, Formatting Fixes 
This started as an application of the edits suggested in #3201 and expanded
from there.
 2022-12-22 14:13:17 -05:00
Felix HandteandGitHub 0694f1435d Merge pull request #3386 from felixhandte/pin-other-action-commit-hashes 
Pin Remaining Action Dependencies (Except OSS-Fuzz)
 2022-12-21 11:24:23 -05:00
W. Felix Handte 150aa23ef4 Pin Remaining Action Dependencies (Except OSS-Fuzz) 
The one that isn't pinned is the OSS-Fuzz builder and runner. They don't
offer tagged releases. I could pin to the current master commit, but I'm not
sure how desirable that is.
 2022-12-20 17:10:01 -05:00
Felix HandteandGitHub 0790e86c56 Merge pull request #3384 from felixhandte/pin-checkout-action-commit-hash 
Pin actions/checkout Dependency to Specific Commit Hash
 2022-12-20 16:08:03 -05:00
W. Felix Handte e3f2c8b11c Pin actions/checkout Dependency to Specific Commit Hash 
It's a bit silly, because if we can't trust GitHub, what are we doing here?
But OSSF complains about it, so let's fix it.
 2022-12-20 14:53:12 -05:00
Felix HandteandGitHub ea199876e2 Merge pull request #3378 from facebook/dependabot/github_actions/github/codeql-action-2.1.37 
Bump github/codeql-action from 1.0.26 to 2.1.37
 2022-12-20 14:17:33 -05:00
Felix HandteandGitHub 5f0543a508 Merge pull request #3377 from facebook/dependabot/github_actions/ossf/scorecard-action-2.1.0 
Bump ossf/scorecard-action from 2.0.6 to 2.1.0
 2022-12-20 14:16:10 -05:00
Felix HandteandGitHub 3a7e2eec21 Merge pull request #3340 from facebook/dependabot/github_actions/actions/upload-artifact-3 
Bump actions/upload-artifact from 1 to 3
 2022-12-20 14:15:43 -05:00
Felix HandteandGitHub f302ad8811 Merge pull request #3173 from felixhandte/update-copyright-company 
Update Copyright Comments
 2022-12-20 13:51:01 -05:00
W. Felix Handte 5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
 2022-12-20 12:52:34 -05:00
W. Felix Handte 7f12f24cf4 Rewrite Copyright Date Ranges from -present to -2022 
Apparently it's better. Somehow.

```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do echo $f; sed -i 's/\-present/-2022/' $f; done

g co HEAD -- build/meson/
```
 2022-12-20 12:44:56 -05:00
W. Felix Handte 4c999cf968 Update test-license.py 2022-12-20 12:43:16 -05:00
W. Felix Handte 36d5c2f326 Update Copyright Year ('2021' -> 'present') 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f);
do
  sed -i 's/\-2021/-present/' $f;
done

g co HEAD -- .github/workflows/dev-short-tests.yml # fix bad match
```
 2022-12-20 12:42:50 -05:00
W. Felix Handte 8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms' 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
 2022-12-20 12:37:57 -05:00
W. Felix Handte a05cca6208 Manually Update VS Code Copyright Definitions 2022-12-20 12:36:06 -05:00
Felix HandteandGitHub bd4581c311 Merge pull request #3309 from joycebrum/dev 
Fix action error by upgrading Scorecard GHA to 2.0.6
 2022-11-01 17:53:37 -04:00
Felix HandteandGitHub 99d239de32 Merge pull request #3290 from felixhandte/ddict-dict-id-from-ddict 
Make ZSTD_getDictID_fromDDict() Read DictID from DDict
 2022-10-18 13:33:32 -04:00
W. Felix Handte d7841d150b Make ZSTD_getDictID_fromDDict() Read DictID from DDict 
Currently this function actually reads the dict ID from the dictionary's
header, via `ZSTD_getDictID_fromDict()`. But during decompression the decomp-
ressor actually compares the dict ID in the frame header with the dict ID in
the DDict. Now of course the dict ID in the dictionary contents and the dict
ID in the DDict struct *should* be the same. But in cases of memory corrupt-
ion, where they can drift out of sync, it's misleading for this function to
read it again from the dict buffer rather then return the dict ID that will
actually be used.

Also doing it this way avoids rechecking the magic and so on and so it is a
tiny bit more efficient.
 2022-10-14 22:53:03 -04:00
Felix HandteandGitHub 33273e1b9c Merge pull request #3277 from felixhandte/ossf-scorecard 
Enable OpenSSF Scorecard Action
 2022-10-03 14:33:21 -04:00
W. Felix Handte de9a450c00 Restrict from Running on Forks 2022-10-03 11:23:29 -04:00
Felix Handte 79729f8a2d Create scorecards.yml 
Addresses #3269.
 2022-09-30 14:30:22 -04:00
Felix HandteandGitHub 7e6278a706 Merge pull request #3196 from mileshu/dev 
[T124890272] Mark 2 Obsolete Functions(ZSTD_copy*Ctx) Deprecated in Zstd
 2022-08-02 12:34:04 -04:00
Felix HandteandGitHub 02ef78be58 Merge pull request #3184 from htnhan/features/list_verbose_to_show_dictionary_id 
zstd -lv <file> to show dictID
 2022-07-08 16:04:39 -04:00
Felix HandteandGitHub 8af64f4116 Merge pull request #3129 from felixhandte/zstd-fast-nodict-unconditional-ip1-table-write 
ZSTD_fast_noDict: Avoid Safety Check When Writing `ip1` into Table
 2022-05-11 17:04:02 -04:00
W. Felix Handte 1bc8019e10 Update results.csv 2022-05-11 10:27:35 -07:00
W. Felix Handte 1dd046a507 Fix Comments Slightly 2022-05-11 12:38:45 -04:00