ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,947 Commits 17 Branches 77 Tags
a610550e2c05cd08842e173bbeb830f596fdfaeb
Commit Graph
10947 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Victor ZhangandGitHub a610550e2c Merge pull request #4218 from facebook/externC 
Move #includes out of `extern "C"` blocks
 2025-01-07 10:06:08 -08:00
Yann ColletandGitHub f5e2996dd3 Merge pull request #4231 from sunpoet/dev 
Update MANDIR for FreeBSD
 2025-01-07 09:04:57 -08:00
Victor ZhangandGitHub 59c2e3335f Merge pull request #4230 from facebook/variedOutput 
Do not vary row matchfinder selection based on availability of SSE2/Neon
 2025-01-06 09:26:57 -08:00
Po-Chuan Hsieh 0fd521048d Update MANDIR for FreeBSD 
share/man became a valid path for manpage since Jan 2020. And we
converted the whole ports tree to share/man around last March.
 2025-01-05 22:26:08 +08:00
Victor Zhang d88651e604 Do not vary row matchfinder selection based on availability of SSE2/Neon 
Move towards a stronger guarantee of reproducibility by removing this small difference for machines without SSE2/Neon.
The SIMD behavior is now the default for all platforms.
 2025-01-03 09:35:18 -08:00
Nick TerrellandNick Terrell 1548bfc349 [opt] Fix too short of match getting generated 
The optimal parser with LDM enabled using minMatch > 3 could generate a match
length of 3 when minMatch >= 4. This is not allowed.

1. Fix the bug
2. Add validation logic to `ZSTD_buildSeqStore()` in debug mode for all block
   compressors that checks we never generate too short a match. This way we don't
   rely on the `generate_sequences` fuzzer to find this issue.

Credit to OSS-Fuzz
 2025-01-03 11:38:41 -05:00
Victor ZhangandGitHub 2759d9d52f Merge pull request #4229 from facebook/noFseek 
Support for libc variants without fseeko/ftello
 2025-01-02 22:42:00 -08:00
Victor Zhang dfb236b2aa chore: indentation alignment 2025-01-02 15:10:40 -08:00
Victor Zhang 6b046f5841 PR feedback 2025-01-02 15:05:58 -08:00
Victor Zhang 757e29e170 Oops 2025-01-02 14:17:24 -08:00
Victor Zhang 54c3d998a0 Support for libc variants without fseeko/ftello 
Some older Android libc implementations don't support `fseeko` or `ftello`.
This commit adds a new compile-time macro `LIBC_NO_FSEEKO` as well as a usage in CMake for old Android APIs.
 2025-01-02 14:02:10 -08:00
Yann ColletandGitHub f5d9d57fe6 Merge pull request #4228 from facebook/skippableMagicVariant 
ZSTD_getFrameHeader() for skippable frames
 2025-01-02 10:57:49 -08:00
Yann Collet 04a2a0219c update type names 
naming convention: Type names should start with a Capital letter (after the prefix)
 2024-12-29 14:25:33 -08:00
Yann ColletandGitHub cf5e53f618 Merge pull request #4227 from facebook/doc_skippableFrame 
clarify documentation for Frame-level methods invoked on a skippable frame
 2024-12-29 13:27:35 -08:00
Yann Collet a2ff6ea784 improve ZSTD_getFrameHeader on skippable frames 
now reports:
- the header size
- the magic variant (within @dictID field)
 2024-12-29 12:26:04 -08:00
Yann Collet f8a2b352d6 clarify doc on Frame-level methods when invoked on a skippable frame 
following discussion at #4226
 2024-12-29 02:41:09 -08:00
Yann ColletandGitHub 788926fe48 Merge pull request #4217 from facebook/ZSTD_compressSequencesAndLiterals 
ZSTD_compressSequencesAndLiterals
 2024-12-26 11:41:44 -08:00
Yann Collet 72ce56b527 fixed another invalid scenario 
compressSequencesAndLiterals() doesn't support sequence validation
 2024-12-23 21:15:50 -08:00
Yann Collet 7b294caf46 add one valid test case 
ZSTD_compressSequencesAndLiterals() may return a specific error code
when data to compress is non-compressible.
 2024-12-23 19:43:17 -08:00
Yann Collet f8725e80cc added fuzzer test for compressSequencesAndLiterals() 
piggy-backing onto existing compressSequences() fuzzer test
 2024-12-23 18:42:51 -08:00
Nick TerrellandNick Terrell f0937b83d9 [cmake] Fix -z noexecstack portability 
Summary:
Issue reported by @ryandesign and @MarcusCalhoun-Lopez.

CMake doesn't support spaces in flags. This caused older versions of gcc to
ignore the unknown flag "-z noexecstack" on MacOS since it was interpreted as a
single flag, not two separate flags. Then, during compilation it was treated as
"-z" "noexecstack", which was correctly forwarded to the linker. But the MacOS
linker does not support `-z noexecstack` so compilation failed.

The fix is to use `-Wl,-z,noexecstack`. This is never misinterpreted by a
compiler. However, not all compilers support this syntax to forward flags to the
linker. To fix this issue, we check if all the relevant `noexecstack` flags have
been successfully set, and if they haven't we disable assembly.

See also PR#4056 and PR#4061. I decided to go a different route because this is
simpler. It might not successfully set these flags on some compilers, but in that
case it also disables assembly, so they aren't required.

Test Plan:
```
mkdir build-cmake
cmake ../build/cmake/CMakeLists.txt
make -j
```

See that the linker flag is successfully detected & that assembly is enabled.

Run the same commands on MacOS which doesn't support `-Wl,-z,noexecstack` and see
that everything compiles and that `LD_FLAG_WL_Z_NOEXECSTACK` and
`ZSTD_HAS_NOEXECSTACK` are both false.
 2024-12-20 15:06:23 -08:00
Yann Collet 47cbfc87a9 restore invocation of ZSTD_entropyCompressSeqStore() 
in the ZSTD_compressSequences() pipeline
 2024-12-20 10:37:01 -08:00
Yann Collet 522adc34eb minor: use MEM_writeLE24() 
so that an empty frame needs only 3 bytes of dstCapacity.
 2024-12-20 10:37:01 -08:00
Yann Collet b7a9e69d8d added parameter litCapacity 
to ZSTD_compressSequencesAndLiterals()
to enforce the litCapacity >= litSize+8 condition.
 2024-12-20 10:37:01 -08:00
Yann Collet 76445bb379 add a check, to return an error if Sequence validation is enabled 
since ZSTD_compressSequencesAndLiterals() doesn't support it.
 2024-12-20 10:37:01 -08:00
Yann Collet 52a9bc6fca fixed minor error in preparation of one fullbench scenario 2024-12-20 10:37:00 -08:00
Yann Collet ab0f1798e8 ensure that srcSize is controlled 2024-12-20 10:37:00 -08:00
Yann Collet b339efff2b add dedicated error code for special case 
ZSTD_compressSequencesAndLiterals() cannot produce an uncompressed block
 2024-12-20 10:37:00 -08:00
Yann Collet a80f55f47d added a test for ZSTD_compressSequencesAndLiterals 
checks that srcSize is present in the frame header
and bounds the window size.
 2024-12-20 10:37:00 -08:00
Yann Collet 0a54f6f288 ZSTD_compressSequencesAndLiterals requires srcSize as parameter 
this makes it possible to adjust windowSize to its tightest.
 2024-12-20 10:37:00 -08:00
Yann Collet ad023b392f fixed minor error in one benchmark scenario 2024-12-20 10:37:00 -08:00
Yann Collet b7b4e86347 fixed minor conversion warning 2024-12-20 10:37:00 -08:00
Yann Collet 12c47d3262 improved speed of the Sequences converter 2024-12-20 10:37:00 -08:00
Yann Collet 95ad9e47ff added benchmark for ZSTD_convertBlockSequences_wBlockDelim() 2024-12-20 10:37:00 -08:00
Yann Collet d48e330ae1 change name to ZSTD_convertSequences*() 2024-12-20 10:37:00 -08:00
Yann Collet 61ac8311e0 attempt to silence Visual Studio warning about fopen() 2024-12-20 10:37:00 -08:00
Yann Collet f617e86b71 fixed incorrect assert 2024-12-20 10:37:00 -08:00
Yann Collet 47edd0acf4 removed fullbench-dll project from visual solutions 2024-12-20 10:37:00 -08:00
Yann Collet 6f8c1046d0 update Visual Studio solutions 2024-12-20 10:37:00 -08:00
Yann Collet f0d0d95234 added tests 
check that ZSTD_compressAndLiterals() also controls that the `srcSize` field is exact.
 2024-12-20 10:37:00 -08:00
Yann Collet 31b5ef2539 ZSTD_compressSequencesAndLiterals() now supports multi-blocks frames. 2024-12-20 10:36:59 -08:00
Yann Collet 5164d44dab change advanced parameter name: ZSTD_c_repcodeResolution 
and updated its documentation.
Note: older name ZSTD_c_searchForExternalRepcodes remains supported via #define
 2024-12-20 10:36:59 -08:00
Yann Collet ca8bd83373 minor: cleaner function parameter repcodeResolution 2024-12-20 10:36:59 -08:00
Yann Collet d2d0fdac42 updated documentation on validateSequence 2024-12-20 10:36:59 -08:00
Yann Collet 1f6d6815c3 optimization: instantiate specialized version without Sequence checking code 
results in +4% compression speed,
thanks to removal of branches in the hot loop.
 2024-12-20 10:36:59 -08:00
Yann Collet a288751de7 minor optimization: only track seqPos->posInSrc when validateSequences is enabled 
note: very minor saving, no performance impact
 2024-12-20 10:36:59 -08:00
Yann ColletandYann Collet f176514467 minor doc update 2024-12-20 10:36:59 -08:00
Yann ColletandYann Collet 1c8f5b0f11 minor optimization for ZSTD_compressSequencesAndLiterals() 
does not need to track and update internal `litPtr`.
note: does not measurably impact performance.
 2024-12-20 10:36:59 -08:00
Yann ColletandYann Collet 0a5c0807af minor conversion warning fix 2024-12-20 10:36:59 -08:00
Yann ColletandYann Collet f281497aef fullbench: new scenario: compressSequencesAndLiterals() 2024-12-20 10:36:59 -08:00