ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,203 Commits 17 Branches 77 Tags
eb168a0afc062f49b3d2180db69ce380c5085509
Commit Graph
11203 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet eb168a0afc add --patch-apply command 
as an equivalent for `-d --patch-from`.
Requested by @sergeevabc in #2173.
 2025-03-25 14:50:39 -07:00
Yann ColletandGitHub 12ea5f6e30 Merge pull request #4314 from facebook/dependabot/github_actions/ossf/scorecard-action-2.4.1 
Bump ossf/scorecard-action from 2.4.0 to 2.4.1
 2025-03-24 18:33:51 -07:00
Yann ColletandGitHub ec40252066 Merge pull request #4345 from Cyan4973/block128 
[doc] minor clarification for maximum block size
 2025-03-24 18:33:01 -07:00
Yann ColletandGitHub 64dc08f47f Merge pull request #4344 from Cyan4973/assert1 
add an assert
 2025-03-24 18:32:37 -07:00
Yann Collet 334af72280 update zstd --fast benchmark result 
use --fast=4, instead of --fast=3,
to make it more representative of a lzo equivalent scenario.
 2025-03-23 12:06:58 -07:00
Yann Collet c59f7ac51e update benchmark result 
brotli uses -1 level, like all other compressors
 2025-03-23 12:02:51 -07:00
Yann Collet bea1e942f7 [doc] minor clarification for maximum block size 
closes #4339
 2025-03-22 23:52:45 -07:00
Yann Collet 2fec3989c1 add an assert 
to help static analyzers understand there is no overflow risk there.
 2025-03-22 18:23:31 -07:00
Yann ColletandGitHub 6775af5b2e Merge pull request #4341 from clan/pragma 
lib/zstd.h: don't put pragma after static
 2025-03-20 19:49:08 -07:00
Z. Liu cd8ca9d92e lib/zstd.h: move pragma before static 
otherwise will cause dev-python/zstandard build failed when compiling with
clang as reported at https://bugs.gentoo.org/950259

the root cause is pycparser, which is unfixed since reported 2.5 years
ago, :(

Signed-off-by: Z. Liu <zhixu.liu@gmail.com>
 2025-03-20 03:40:42 +00:00
Yann ColletandGitHub eca205fc78 Merge pull request #4336 from Cyan4973/qsort_v2 
fix #4312 - musl compilation compatibility
 2025-03-12 10:31:54 -07:00
Yann Collet 4d53e27144 removed OpenBSD specificity 2025-03-12 09:55:14 -07:00
Yann Collet ddcb41a282 updated documentation 2025-03-11 14:10:35 -07:00
Yann Collet a9b8fef2e8 add support for C11 Annex K qsort_s() 
standard defined re-entrant variant of qsort().
Unfortunately, Annex K is optional.
 2025-03-11 14:10:35 -07:00
Yann Collet dcf675886b re-design qsort() selection in cover 
centralizes auto detection tests,
then distribute the outcome in all the places where it's active.
 2025-03-11 14:10:35 -07:00
Yann ColletandYann Collet 51b6e79f65 fix #4312 
and upgraded the test so that it would fail, both at compile time and at run time, without the fix
 2025-03-11 14:10:35 -07:00
Yann ColletandYann Collet 9a57bdc0bd attempt to reduce length of long cli tests by invoking -T0 2025-03-11 14:10:35 -07:00
Nick TerrellandNick Terrell 68dfd14a8c [linux] Opt out of row based match finder for the kernel 
The row based match finder is slower without SIMD. We used to detect the
presence of SIMD to set the lower bound to 17, but that breaks
determinism. Instead, specifically opt into it for the kernel, because
it is one of the rare cases that doesn't have SIMD support.
 2025-03-11 16:18:59 -04:00
Yann ColletandGitHub b27ba56a6d Merge pull request #4335 from Cyan4973/dec_mt_warning 
warn when requesting decompression with multiple threads
 2025-03-11 11:51:15 -07:00
Yann Collet 19541b4d1a fix minor unused variable warning 2025-03-11 00:17:45 -07:00
Yann Collet c583c2c39e warn when requesting decompression with multiple threads 
restore #2918 fix
 2025-03-10 22:50:00 -07:00
Yann ColletandGitHub f9a6031963 Merge pull request #4334 from Cyan4973/fix4332 
fix #4332: setting ZSTD_NBTHREADS=0 via environment variable
 2025-03-10 21:12:24 -08:00
Yann Collet 3c4096c83e fixed ShellCheck warning 2025-03-10 19:11:44 -07:00
Yann Collet 2ff87aefac fix FreeBSD 
use an alias instead of a function

also: added more traces and updated version nb to v1.5.8
 2025-03-10 19:04:41 -07:00
Yann Collet c18374bb16 add test 
checks that ZSTD_NBTHREADS triggers the expected verbose message

Also: checked that the new test script fails on current `dev` branch, and is fixed by this branch
 2025-03-10 13:40:47 -07:00
Yann Collet 56e2ebf5c3 removed useless assert() 2025-03-10 09:54:06 -07:00
Yann Collet d5986f235f fix #4332: setting ZSTD_NBTHREADS=0 via environment variables 2025-03-10 00:12:34 -07:00
Nick TerrellandNick Terrell b16d193512 [test] Add tests for determinism 
Run compression & validate the compressed file matches a known checksum.

To update the output run:

```
make -C tests update-cli-tests
```
 2025-03-07 10:31:19 -05:00
Nick TerrellandNick Terrell 0de4991942 Add a method for checking if ZSTD was compiled with flags that impact determinism 2025-03-07 10:31:19 -05:00
Yann ColletandGitHub 99cf130cfc Merge pull request #4329 from Cyan4973/cmd_split 
New commands --split and --jobsize
 2025-03-05 07:46:04 -08:00
Nick TerrellandNick Terrell 190a620974 [zstd] Remove global variables in dictBuilder 
D50949782 fixed a race condition updating `g_displayLevel` by disabling display.
Instead of disabling display, delete the global variable and always "capture" a local `displayLevel` variable.
This also fixes `DISPLAYUPDATE()` by requiring the user to pass in the last update time as the first parameter.
 2025-03-05 10:35:01 -05:00
Nick TerrellandNick Terrell d5b84f5a27 [zstd] Backport D49756856 2025-03-05 10:35:01 -05:00
Yann ColletandGitHub 5e7d721235 Merge pull request #4327 from Cyan4973/win_artifacts 
Generate Zstandard binaries for Windows with lz4 support
 2025-03-04 16:08:01 -08:00
Yann Collet 5ae1cb9fa1 added a cli test for new command --jobsize 2025-03-04 15:24:43 -08:00
Yann Collet f5a0e047cb fix typo 2025-03-04 15:12:35 -08:00
Yann Collet 0b40c513fd update fileio to employ jobSize 2025-03-04 14:55:25 -08:00
Yann Collet 0298df50f9 update cli-tests 2025-03-04 14:43:06 -08:00
Yann Collet 9b8b414833 updated playTests.sh to employ the new commands 2025-03-04 13:22:01 -08:00
Yann Collet 19ffcf81b8 changed -B# command into --jobsize 
to reduce confusion with the term "block".

-B# remains supported for existing scripts,
but it's no longer documented, so it's effectively a hidden shortcut.
 2025-03-04 13:17:10 -08:00
Yann Collet fcfb3160dc convert benchmark unit to use the new naming scheme 
chunks instead of blocks
 2025-03-04 13:04:59 -08:00
Yann Collet d5dbdd6ece changed command --blocksize into --split 
to reduce confusion with the concept of "blocks" inside a Zstandard frame.

We are now talking about "independent chunks" being produced by a `split` operation.

updated documentation accordingly.

Note: old commands "-B#` and `--blocksize=#` remain supported,
to maintain compatibility with existing scripts.
 2025-03-04 12:53:23 -08:00
Yann Collet bceb8f2dbc attempt to no longer depend on system library presence for autodetection 
use explicit opt-in and path setting
 2025-03-03 17:20:43 -08:00
Yann Collet 9ec1a7c0ac update zstd build recipe to also support lz4 format 2025-03-03 15:36:38 -08:00
Yann ColletandGitHub 7df457a51d Merge pull request #4324 from facebook/dependabot/github_actions/actions/checkout-4 
Bump actions/checkout from 3 to 4
 2025-03-03 10:39:50 -08:00
Yann ColletandGitHub cab2a29c36 Update windows-artifacts.yml 
edit version comment
 2025-03-02 23:11:35 -08:00
Yann ColletandGitHub 3c6c472f01 Update scorecards.yml 
edit version comment
 2025-03-02 23:11:08 -08:00
Yann ColletandGitHub 46b2e05a33 Update publish-release-artifacts.yml 
fix version comment
 2025-03-02 23:10:09 -08:00
dependabot[bot]andGitHub 83db5376d7 Bump actions/checkout from 3 to 4 
Bumps [actions/checkout](https://github.com/actions/checkout) from 3 to 4.
- [Release notes](https://github.com/actions/checkout/releases)
- [Commits](https://github.com/actions/checkout/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2025-03-03 05:44:17 +00:00
Sergey SenozhatskyandNick Terrell 0727fc5ff9 lib: zstd: export API needed for dictionary support 
Patch series "zram: introduce custom comp backends API", v7.

This series introduces support for run-time compression algorithms tuning,
so users, for instance, can adjust compression/acceleration levels and
provide pre-trained compression/decompression dictionaries which certain
algorithms support.

At this point we stop supporting (old/deprecated) comp API.  We may add
new acomp API support in the future, but before that zram needs to undergo
some major rework (we are not ready for async compression).

Some benchmarks for reference (look at column #2)

*** init zstd
/sys/block/zram0/mm_stat
1750659072 504622188 514355200        0 514355200        1        0    34204    34204

*** init zstd dict=/home/ss/zstd-dict-amd64
/sys/block/zram0/mm_stat
1750650880 465908890 475398144        0 475398144        1        0    34185    34185

*** init zstd level=8 dict=/home/ss/zstd-dict-amd64
/sys/block/zram0/mm_stat
1750654976 430803319 439873536        0 439873536        1        0    34185    34185

*** init lz4
/sys/block/zram0/mm_stat
1750646784 664266564 677060608        0 677060608        1        0    34288    34288

*** init lz4 dict=/home/ss/lz4-dict-amd64
/sys/block/zram0/mm_stat
1750650880 619990300 632102912        0 632102912        1        0    34278    34278

*** init lz4hc
/sys/block/zram0/mm_stat
1750630400 609023822 621232128        0 621232128        1        0    34288    34288

*** init lz4hc dict=/home/ss/lz4-dict-amd64
/sys/block/zram0/mm_stat
1750659072 505133172 515231744        0 515231744        1        0    34278    34278

Recompress
init zram zstd (prio=0), zstd level=5 (prio 1), zstd with dict (prio 2)

*** zstd
/sys/block/zram0/mm_stat
1750982656 504630584 514269184        0 514269184        1        0    34204    34204

*** idle recompress priority=1 (zstd level=5)
/sys/block/zram0/mm_stat
1750982656 488645601 525438976        0 514269184        1        0    34204    34204

*** idle recompress priority=2 (zstd dict)
/sys/block/zram0/mm_stat
1750982656 460869640 517914624        0 514269184        1        0    34185    34204

This patch (of 24):

We need to export a number of API functions that enable advanced zstd
usage - C/D dictionaries, dictionaries sharing between contexts, etc.

Link: https://lkml.kernel.org/r/20240902105656.1383858-1-senozhatsky@chromium.org
Link: https://lkml.kernel.org/r/20240902105656.1383858-2-senozhatsky@chromium.org
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Cc: Nick Terrell <terrelln@fb.com>
Cc: Minchan Kim <minchan@kernel.org>
Cc: Sergey Senozhatsky <senozhatsky@chromium.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
 2025-02-28 17:09:46 -05:00
Yann ColletandGitHub a9337e6739 Merge pull request #4321 from Cyan4973/win_artifacts 
Automate Windows Artifact generation and publication
 2025-02-28 12:35:06 -08:00