ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,081 Commits 17 Branches 77 Tags
613901b6d3ea198ac051f63e16c6ea9cd088a6e4
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 613901b6d3 modifies command echoing for macos / linux compatibility 2025-02-09 23:41:32 -08:00
Yann Collet 468e1453a5 disable --max in 32-bit mode 2025-02-09 23:16:14 -08:00
Yann Collet 1603cbe83e update test for 32-bit mode 
--max doesn't work in 32-bit mode, due to address space limitation
 2025-02-09 23:02:14 -08:00
Yann Collet 4bbf4a285d enable DYNAMIC_BMI2 by default on x86 (32-bit mode) 
so far was only enabled for x64 (64-bit mode)
 2025-01-19 08:11:59 -08:00
Yann Collet a556559841 no longer limit automated BMI2 detection to x64 
this was previously no triggered in x86 32-bit mode,
due to a limitation in `bitstream.h`, that was fixed in #4248.

Now, `bmi2` will be automatically detected and triggered
at compilation time, if the corresponding instruction set is enabled,
even in 32-bit mode.

Also: updated library documentation, to feature STATIC_BMI2 build variable
 2025-01-19 00:08:57 -08:00
Yann ColletandYann Collet 87f0a4fbe0 restore full equation 
do not solve the equation, even though some members cancel each other,
this is done for clarity,
we'll let the compiler do the resolution at compile time.
 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet 8bff69af86 Alignment instruction ZSTD_ALIGNED() in common/compiler.h 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet 2f3ee8b530 changed code compilation test to employ ZSTD_ARCH_X86_AVX2 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet debe3d20d9 removed unused branch 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet e3181cfd32 minor code doc update 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet aa2cdf964f added compilation-time checks to ensure AVX2 code is valid 
since it depends on a specific definition of ZSTD_Sequence structure.
 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet 57a4554192 removed unused variable 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet 4aaf9cefe9 fix minor conversion warning 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet db3d48823a no need for specialized variant 
the branch is not in the hot loop
 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet cd53924eff removed erroneous #includes 
that were automatically added by the editor without notification
 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet ed0a8b8be1 AVX2 version of ZSTD_get1BlockSummary() 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet b6a4d5a8ba minor +10% speed improvement for scalar ZSTD_get1BlockSummary() 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet 8eb2587432 added benchmark for get1BlockSummary() 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet bfc58f5ba2 generalize validation function 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet 8d62164589 control long length within AVX2 implementation 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet d1f0e5fb97 fullbench can run a verification function 
compressSequencesAndLiterals: fixed long lengths in scalar mode
 2025-01-15 17:11:27 -08:00
Yann ColletandYann Collet 886720442f initial implementation (incomplete) 
needs to take care of long lengths > 65535
 2025-01-15 17:11:27 -08:00
Yann Collet 56500044c4 bench: nb threads specified with -v 2025-01-12 12:01:57 -08:00
Yann Collet 04a2a0219c update type names 
naming convention: Type names should start with a Capital letter (after the prefix)
 2024-12-29 14:25:33 -08:00
Yann Collet a2ff6ea784 improve ZSTD_getFrameHeader on skippable frames 
now reports:
- the header size
- the magic variant (within @dictID field)
 2024-12-29 12:26:04 -08:00
Yann Collet f8a2b352d6 clarify doc on Frame-level methods when invoked on a skippable frame 
following discussion at #4226
 2024-12-29 02:41:09 -08:00
Yann Collet 72ce56b527 fixed another invalid scenario 
compressSequencesAndLiterals() doesn't support sequence validation
 2024-12-23 21:15:50 -08:00
Yann ColletandYann Collet f176514467 minor doc update 2024-12-20 10:36:59 -08:00
Yann ColletandYann Collet 1c8f5b0f11 minor optimization for ZSTD_compressSequencesAndLiterals() 
does not need to track and update internal `litPtr`.
note: does not measurably impact performance.
 2024-12-20 10:36:59 -08:00
Yann ColletandYann Collet 0a5c0807af minor conversion warning fix 2024-12-20 10:36:59 -08:00
Yann ColletandYann Collet f281497aef fullbench: new scenario: compressSequencesAndLiterals() 2024-12-20 10:36:59 -08:00
Yann ColletandYann Collet ac05ea89a5 fullbench: switch default generator to lorem ipsum 
which creates more "realistic" scenarios than former compressible noise.

The legacy data generator remains accessible,
it is triggered when requesting an explicit compressibility factor (-P#).
 2024-12-20 10:36:59 -08:00
Yann Collet 50ca9984ad minor: more accurate parameter 
just pass ldm_bucketLog, instead of the entire ldm* state
 2024-12-14 11:57:14 -08:00
Yann Collet b3035b36c6 blind fix for QNX 
following notification from @rainbowball.
fix #4186.

Note: there is currently no QNX compilation test in CI
so this is a "blind" fix,
and this target can be silently broken again in the future.
 2024-11-05 00:09:13 -08:00
Yann ColletandYann Collet 57239c4d3b fixed minor strict pedantic C90 issue 2024-10-23 11:50:57 -07:00
Yann ColletandYann Collet 4ce91cbf2b fixed workspace alignment on non 64-bit systems 2024-10-23 11:50:57 -07:00
Yann Collet f34bc9cee6 improve man page on benchmark mode 
update the man page in troff format,
and the README with latest `--help` content and complementary details about benchmark mode.

also: display level 0 when doing decompression benchmark
 2024-10-23 00:16:13 -07:00
Yann Collet 0079d515b1 Modify benchmark to only load sources once 
After a regrettable update,
the benchmark module ended up reloading sources for every compression level.

While the delay itself is likely torelable,
the main issue is that the `--quiet` mode now also displays a loading summary between each compression line.
This wasn't the original intention, which is to produce a compact view of all compressions.

This is fixed in this version,
where sources are loaded only once, for all compression levels,
and loading summary is only displayed once.
 2024-10-22 02:18:48 -07:00
Yann Collet fa1fcb08ab minor: better variable naming 2024-10-10 16:07:20 -07:00
Yann Collet 3e7c66acd1 added ascending order example 2024-10-09 01:06:24 -07:00
Yann Collet d45aee43f4 make __asm__ a __GNUC__ specific 2024-10-08 16:38:35 -07:00
Yann Collet 741b860fc1 store dummy bytes within ZSTD_match4Found_cmov() 
feels more logical, better contained
 2024-10-08 16:34:40 -07:00
Yann Collet 197c258a79 introduce memory barrier to force test order 
suggested by @terrelln
 2024-10-08 15:54:48 -07:00
Yann Collet 186b132495 made search strategy switchable 
between cmov and branch
and use a simple heuristic based on wlog to select between them.

note: performance is not good on clang (yet)
 2024-10-08 13:52:56 -07:00
Yann Collet 2cc600bab2 refactor search into an inline function 
for easier swapping with a parameter
 2024-10-08 11:10:48 -07:00
Yann Collet 1e7fa242f4 minor refactor zstd_fast 
make hot variables more local
 2024-10-07 11:22:40 -07:00
Yann Collet 8edd147686 fix missing fclose() 
fix #4151
 2024-10-01 09:52:45 -07:00
Yann Collet 0a68be83e7 updated setup-msys2 to v2.22.0 
following a warning in recent test reports

```
Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: msys2/setup-msys2@5beef6d11f. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
```
 2024-02-21 00:22:04 -08:00
Yann Collet 7170f51dd2 fix include order 2024-02-20 23:36:04 -08:00
Yann Collet 9e711c9360 fix Visual Studio datagen recipe 2024-02-20 22:59:58 -08:00
Yann Collet e62e15df19 fix clangbuild 
notably -Wconversion and -Wdocumentation
 2024-02-20 22:43:22 -08:00
Yann Collet 588dfbcc97 fix c89 compatibility by removing snprintf() 
note that this function has been in the code for a long while,
so why does it only start failing CI tests now ?
 2024-02-20 19:24:50 -08:00
Yann Collet b0e8580dc7 fix fuzz issue 5131069967892480 2024-02-08 16:38:20 -08:00
Yann Collet 695d154cac fuzz: control debuglevel from Makefile 
and make the compilation faster
 2024-02-08 16:23:52 -08:00
Yann Collet 887f5b62ae update compression results for regression tests 2024-02-05 01:27:22 -08:00
Yann Collet 6c35fb2e8c fix msan warnings 2024-02-05 01:21:06 -08:00
Yann Collet 641749fc09 fix uasan dictionary_stream_round_trip fuzz test 2024-02-05 00:36:10 -08:00
Yann Collet fe2e2ad36d use ZSTD_memcpy() 
which can be redirected in Linux kernel mode
 2024-02-03 19:57:38 -08:00
Yann Collet 0ae21d8c31 removed trace control 2024-02-03 19:32:59 -08:00
Yann Collet 5474edbe60 fixed wrong assert 
by introducing ZSTD_OPT_SIZE
 2024-02-03 19:31:53 -08:00
Yann Collet 9ae3bf5ee2 update compression results 
good news: there are only improvements
 2024-02-03 17:52:50 -08:00
Yann Collet e5af24c5fa fixed wrong assert 2024-02-03 17:48:29 -08:00
Yann Collet 8168a451e5 minor optimization, mostly for clarity 2024-02-03 17:26:47 -08:00
Yann Collet d31018e223 finally, a version that generalizes well 
While it's not always strictly a win,
it's a win for files that see a noticeably compression ratio increase,
while it's a very small noise for other files.

Downside is, this patch is less efficient for 32-bit arrays of integer
than the previous patch which was introducing losses for other files,
but it's still a net improvement on this scenario.
 2024-02-03 14:26:18 -08:00
Yann Collet 6bb1688c1a extended the fix to ZSTDMT's Buffer Pool 2023-10-08 00:25:17 -07:00
Yann Collet ea4027c003 removed unused macro constant 2023-10-07 23:32:22 -07:00
Yann Collet c87ad5bdb5 fixes suggested by @ebiggers 2023-10-07 23:29:42 -07:00
Yann Collet e8ff7d18eb removed FlexArray pattern from CCtxPool 
within ZSTDMT_.
This pattern is flagged by less forgiving variants of ubsan
notably used during compilation of the Linux Kernel.

There are 2 other places in the code where this pattern is used.
This fixes just one of them.
 2023-10-07 21:30:08 -07:00
Yann Collet 2c17e05646 fix x32 tests on Github CI 
ubuntu-22.04 seems to have problems with x32 recently
switching to ubuntu-20.04 which seems to work fine so far

https://github.com/actions/runner-images/issues/8397
 2023-09-27 21:18:20 -07:00
Yann ColletandYann Collet 9f58241dcc updated version number to v1.5.5 
also : updated man pages
 2023-03-31 23:02:08 -07:00
Yann Collet 0f77956bcc added a Clang-CL Windows test to CI 
If I understand correctly,
this should trigger the issue notified in #3569.
 2023-03-28 22:06:18 -07:00
Yann Collet 95ffc767f6 updated man pages 2023-02-09 14:40:39 -08:00
Yann Collet c5bf6b8b88 add requested check for legacy decoder v0.1 
which uses a different technique to store literals,
and therefore must check for potential overwrites.
 2023-02-07 14:47:16 -08:00
Yann Collet 9419747171 fix legacy decoders v0.4, v0.5 and v0.6 2023-02-07 14:02:12 -08:00
Yann Collet 6640377783 cmake build: fix nit 
reported by @jaimeMF in https://github.com/facebook/zstd/pull/3392#discussion_r1056643794
 2022-12-23 14:18:11 -08:00
Yann Collet 832c1a6a1c minor reformatting 
and minor reliability and maintenance changes
 2022-12-18 11:26:57 -08:00
Yann Collet d081d98ae7 Fix m68k CI tests on Github Actions 
seems there is a bug in the qemu version shipped with Ubuntu 22.04
 2022-12-13 12:36:53 -08:00
Yann Collet 15f3605135 removed gnu99 statement from meson recipe 2022-06-20 18:18:40 -07:00
Yann Collet eb726c6a20 updated man pages 
had to run the conversion script on Ubuntu, as it doesn't run correctly on macos anymore.
 2022-04-13 18:57:27 -07:00
Yann Collet cdee6a7dbd Merge branch 'dev' into fix44168 2022-01-31 17:31:55 -08:00
Yann ColletandYann Collet a66e8bb437 introduced LitHufLog constant 
which properly represents the maximum bit size of compressed literals (11) as defined in the specification.

To be preferred from HUF_TABLELOG_DEFAULT which represents the same value but by accident.

Name selected to keep the same convention as existing width definitions,
MLFSELog, LLFSELog and OffFSELog.
 2022-01-26 14:47:24 -08:00
Yann ColletandYann Collet 32a5d95dcb moved HufLog to lib/decompress 
it's only used to size decompression tables
 2022-01-26 14:47:24 -08:00
Yann ColletandYann Collet e9dd923fa4 only declare debug functions in debug mode 2022-01-26 14:47:24 -08:00
Yann ColletandYann Collet fc2ea97442 refactored fuzzer tests for sequence compression api 
add explicit delimiter mode to libfuzzer test
 2022-01-26 00:19:35 -08:00
Yann Collet 41153071a0 updated manual 2021-12-21 08:52:50 -08:00
Yann Collet abc694f53e update man pages 2021-12-20 14:24:06 -08:00
Yann Collet 2624652a32 Merge branch 'dev' into lazy_rebalance 2021-12-20 08:28:21 -08:00
Yann Collet 80a28f2078 update regression results 2021-12-20 07:54:57 -08:00
Yann Collet 01adddc3e0 update regression results 2021-12-16 20:43:23 -08:00
Yann Collet db1b408a2f rebalance lazy compression levels 2021-12-15 21:33:31 -08:00
Yann Collet 8150891939 regenerated zstdless.1 2021-09-22 14:48:51 -07:00
Yann Collet 3addf2f277 updated zstdgrep man page 2021-09-22 14:30:59 -07:00
Yann Collet 999f8778af updated man pages 
using ronn-ng
 2021-09-22 14:18:24 -07:00
Yann Collet fd94b9d1c9 Merge branch 'dev' into opt_investigation 2021-09-14 01:15:51 -07:00
Yann Collet b6b2855b80 updated regression tests 2021-09-12 10:22:35 -07:00
Yann Collet f58e63bee7 Merge branch 'dev' into opt_investigation 2021-09-12 01:42:49 -07:00
Yann Collet 640c5b1f77 fix automated_benchmarking 
make it able to process text output sent into either stdout or stderr
 2021-09-12 01:36:18 -07:00
Yann Collet ef78611c26 change update rate to 11/10/10/10 
better for larger blocks,
very small inefficiency on small block.
 2021-09-08 08:58:28 -07:00
Yann Collet b096a5c626 updated regression tests 2021-09-07 09:55:14 -07:00
Yann Collet 08ceda3dfc new statistics update policy 
small general compression ratio improvement for btopt+ strategies/
 2021-09-04 00:52:44 -07:00