ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,139 Commits 17 Branches 77 Tags
e6dccbf48246f4e2844251972fcc0946a5de5154
Commit Graph
10139 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
W. Felix Handte a5ed28f1fb Use Existing Src File Stat in *_dstFile() Funcs 
One fewer `stat()` call to make per operation!
 2023-01-17 14:08:22 -08:00
W. Felix Handte 5653f96776 Pass stat_t into *_dstFile() Functions 2023-01-17 14:05:15 -08:00
W. Felix Handte 2ad6855ac1 FIO_openSrcFile() Returns File Stat 2023-01-17 14:01:06 -08:00
Danielle Rozenblit 8353a4b095 fix maxBlockSize resolution + add test cases 2023-01-17 12:24:18 -08:00
Felix HandteandGitHub 23a356cdde Merge pull request #3424 from felixhandte/disable-asan-msan-poison-mingw 
Disable Custom ASAN/MSAN Poisoning on MinGW Builds
 2023-01-17 12:41:41 -05:00
Elliot GorokhovskyandGitHub 018b68f332 fix msys2 symlink breakage in CI (#3429) 2023-01-17 12:10:15 -05:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 		3add5ca3ef Bump github/codeql-action from 2.1.37 to 2.1.38 (#3428) 
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.1.37 to 2.1.38.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/959cbb7472c4d4ad70cdfe6f4976053fe48ab394...515828d97454b8354517688ddc5b48402b723750)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-01-16 17:22:19 -05:00
Elliot GorokhovskyandGitHub 5d8cfa6b96 Deprecate advanced streaming functions (#3408) 
* deprecate advanced streaming functions

* remove internal usage of the deprecated functions

* nit

* suppress warnings in tests/zstreamtest.c

* purge ZSTD_initDStream_usingDict

* nits

* c90 compat

* zstreamtest.c already disables deprecation warnings!

* fix initDStream() return value

* fix typo

* wasn't able to import private symbol properly, this commit works around that

* new strategy for zbuff

* undo zbuff deprecation warning changes

* move ZSTD_DISABLE_DEPRECATE_WARNINGS from .h to .c
 2023-01-13 14:51:47 -05:00
Yann Collet 2086e7396e missing #include for Windows 2023-01-13 11:38:27 -08:00
W. Felix Handte d78fbedd96 Don't Even Declare Poisoning Functions if Poisoning is Disabled 
This guarantees that we won't accidentally forget to check the macro somewhere
where we use these functions.
 2023-01-13 11:56:48 -05:00
W. Felix Handte f10922a8fa Disable Custom ASAN/MSAN Poisoning on MinGW Builds 
Addresses #3240.
 2023-01-13 11:53:09 -05:00
Danielle Rozenblit 14b8defb86 move ZSTD_BLOCKSIZE_MAX_MIN to static linking only section 2023-01-13 07:00:50 -08:00
Yann ColletandGitHub d5509080bc Merge pull request #3419 from facebook/fix3416 
fix root cause of #3416
 2023-01-13 00:21:08 -08:00
Yann Collet a2ef23dec0 restore support of clock_gettime() for POSIX systems 
This should notably allow posix systems with timespec_get()
to have access to a high resolution timer,
instead of falling back to C90's clock_t.
 2023-01-12 21:08:15 -08:00
Yann Collet bcfb7ad03c refactor timefn 
The timer storage type is no longer dependent on OS.
This will make it possible to re-enable posix precise timers
since the timer storage type will no longer be sensible to #include order.
See #3168 for details of pbs of previous interface.

Suggestion by @terrelln
 2023-01-12 19:24:31 -08:00
Nick TerrellandNick Terrell 5b266196a4 Add support for in-place decompression 
* Add a function and macro ZSTD_decompressionMargin() that computes the
  decompression margin for in-place decompression. The function computes
  a tight margin that works in all cases, and the macro computes an upper
  bound that will only work if flush isn't used.
* When doing in-place decompression, make sure that our output buffer
  doesn't overlap with the input buffer. This ensures that we don't
  decide to use the portion of the output buffer that overlaps the input
  buffer for temporary memory, like for literals.
* Add a simple unit test.
* Add in-place decompression to the simple_round_trip and
  stream_round_trip fuzzers. This should help verify that our margin stays
  correct.
 2023-01-12 16:28:08 -08:00
Yann Collet ac45e078a5 add explanation about new test 
as requested by @terrelln
 2023-01-12 15:49:01 -08:00
Yann Collet 796699c0bc fix root cause of #3416 
A minor change in 5434de0 changed a `<=` into a `<`,
and as an indirect consequence allowed compression attempt of literals when there are only 6 literals to compress
(previous limit was effectively 7 literals).

This is not in itself a problem, as the threshold is merely an heuristic,
but it emerged a bug that has always been there, and was just never triggered so far due to the previous limit.
This bug would make the literal compressor believes that all literals are the same symbol,
but for the exact case where nbLiterals==6, plus a pretty wild combination of other limit conditions,
this outcome could be false, resulting in data corruption.

Replaced the blind heuristic by an actual test for all limit cases,
so that even if the threshold is changed again in the future,
the detection of RLE mode will remain reliable.
 2023-01-12 15:41:08 -08:00
Yann ColletandGitHub 423500d1ae Merge pull request #3413 from facebook/timefn 
minor refactoring for timefn
 2023-01-12 15:34:00 -08:00
Danielle Rozenblit 06b096db47 additional tests and documentation updates + allow maxBlockSize to be set to 0 (goes to default) 2023-01-12 13:41:50 -08:00
Felix HandteandGitHub fd2eb8a68e Merge pull request #3402 from facebook/dependabot/github_actions/ossf/scorecard-action-2.1.2 
Bump ossf/scorecard-action from 2.1.0 to 2.1.2
 2023-01-12 13:28:04 -05:00
Danielle Rozenblit 53eb5a758c add simple test for maxBlockSize expected functionality 2023-01-12 08:55:39 -08:00
Elliot GorokhovskyandGitHub 4f7183d887 Completely overhaul Windows CI (#3410) 
* Overhaul windows CI

* upgrade setup-msbuild from v1.1.3 to v1.3

* remove cmake 2019 test

* fix 32-bit gcc mingw test

* merge conflict
 2023-01-11 16:29:23 -05:00
Danielle Rozenblit 1fffcfe01d update minimum threshold for max block size 2023-01-11 11:09:57 -08:00
Daniel KuteninandNick Terrell ca2ff788df Make the producer use the same amount of entropy 2023-01-11 10:09:19 -08:00
Daniel KuteninandNick Terrell 3ac0b91302 Fix fuzzing with ZSTD_MULTITHREAD 
At Google we fuzz zstd without ZSTD_MULTITHREAD but we want inputs to be as much as reproducible. It allows us to test new fuzzing methods for our fuzz team internally and have more horsepower to find bugs
 2023-01-11 10:09:19 -08:00
Yann ColletandGitHub 98ca8b4456 Merge pull request #3414 from facebook/dependabot/github_actions/actions/checkout-3.3.0 
Bump actions/checkout from 3.2.0 to 3.3.0
 2023-01-09 11:33:29 -08:00
Danielle Rozenblit fe08137d9a resolve max block value in cctx and use when calculating the max block size 2023-01-09 07:53:53 -08:00
dependabot[bot]andGitHub 59a536aa01 Bump actions/upload-artifact from 3.1.1 to 3.1.2 
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.1 to 3.1.2.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/83fd05a356d7e2593de66fc9913b3002723633cb...0b7f8abb1508181956e8e162db84b466c27e18ce)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-01-09 05:10:50 +00:00
dependabot[bot]andGitHub 6f17a5d8df Bump actions/checkout from 3.2.0 to 3.3.0 
Bumps [actions/checkout](https://github.com/actions/checkout) from 3.2.0 to 3.3.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/755da8c3cf115ac066823e79a1e1788f8940201b...ac593985615ec2ede58e132d2e21d2b1cbd6127c)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2023-01-09 05:10:46 +00:00
Yann Collet 8b130009e3 minor simplification refactoring for timefn 
`UTIL_getSpanTimeMicro()` can be factored in a generic way,
reducing OS-dependent code.
 2023-01-06 16:12:54 -08:00
Yann Collet 71dbe8f9d4 minor: fix conversion warnings 2023-01-04 20:00:04 -08:00
daniellerozenblitandGitHub d913417f72 Merge branch 'dev' into fuzz-max-block-size 2023-01-04 16:34:07 -05:00
Danielle Rozenblit 908e812733 initial commit 2023-01-04 13:01:54 -08:00
Yann ColletandGitHub 834fd07a99 Merge pull request #3391 from facebook/fix3228 
improve compression ratio of small alphabets
 2023-01-03 16:01:52 -08:00
Yann Collet c79fb4d78d update levels.sh test 
comparing level 19 to level 22 and expecting a stricter better result from level 22
is not that guaranteed,
because level 19 and 22 are very close to each other,
especially for small files,
so any noise in the final compression result
result in failing this test.

Level 22 could be compared to something much lower, like level 15,
But level 19 is required anyway, because there is a clamping test which depends on it.

Removed level 22, kept level 19
 2023-01-03 14:04:41 -08:00
Yann Collet ebba9ff425 update regression results 2023-01-03 14:04:23 -08:00
Yann Collet 5434de01e2 improve compression ratio of small alphabets 
fix #3328

In situations where the alphabet size is very small,
the evaluation of literal costs from the Optimal Parser is initially incorrect.
It takes some time to converge, during which compression is less efficient.
This is especially important for small files,
because there will not be enough data to converge,
so most of the parsing is selected based on incorrect metrics.

After this patch, the scenario ##3328 gets fixed,
delivering the expected 29 bytes compressed size (smallest known compressed size).
 2023-01-03 12:22:37 -08:00
daniellerozenblitandGitHub 1c818e3a0a Merge pull request #3302 from daniellerozenblit/optimal-huff-depth-speed 
Optimal huff depth speed improvements
 2023-01-03 12:51:51 -05:00
Danielle Rozenblit 87becc567d update regression results.csv 2023-01-03 08:41:40 -08:00
Danielle Rozenblit df714ddb0f implement suggestions 2023-01-03 07:20:21 -08:00
Yann ColletandGitHub 3248910432 Merge pull request #3248 from facebook/opt_comments1 
[easy] add a few comments to the optimal parser code base for improved clarity
 2022-12-28 18:03:57 -08:00
Yann Collet d07e72bb13 fixed incorrect assert 
commented Fweight instead
 2022-12-28 17:23:40 -08:00
Yann Collet 4a1a79a512 just add some comments to zstd_opt for improved clarity 2022-12-28 16:24:12 -08:00
Yann ColletandGitHub 9fbbd74871 Merge pull request #3400 from danlark1/dev 
Move deprecated annotation before static to allow C++ compilation for clang
 2022-12-28 15:50:26 -08:00
Yann ColletandGitHub bcbd395c1c Merge pull request #3395 from terrelln/2022-12-21-deprecated-test 
[tests] Remove deprecated function from longmatch.c test
 2022-12-28 15:49:50 -08:00
Yann Collet 00c85b28e7 update ZSTD_CCts_setCParams() inline documentation 
specify behavior when changing compression parameters during MT compression,
reported by @embg
 2022-12-28 15:08:18 -08:00
Yann ColletandGitHub 481a2e1010 Merge pull request #3403 from facebook/setCParams 
ZSTD_CCtx_setCParams
 2022-12-28 14:07:13 -08:00
Elliot GorokhovskyandGitHub 2a402626dd External matchfinder API (#3333) 
* First building commit with sample matchfinder

* Set up ZSTD_externalMatchCtx struct

* move seqBuffer to ZSTD_Sequence*

* support non-contiguous dictionary

* clean up parens

* add clearExternalMatchfinder, handle allocation errors

* Add useExternalMatchfinder cParam

* validate useExternalMatchfinder cParam

* Disable LDM + external matchfinder

* Check for static CCtx

* Validate mState and mStateDestructor

* Improve LDM check to cover both branches

* Error API with optional fallback

* handle RLE properly for external matchfinder

* nit

* Move to a CDict-like model for resource ownership

* Add hidden useExternalMatchfinder bool to CCtx_params_s

* Eliminate malloc, move to cwksp allocation

* Handle CCtx reset properly

* Ensure seqStore has enough space for external sequences

* fix capitalization

* Add DEBUGLOG statements

* Add compressionLevel param to matchfinder API

* fix c99 issues and add a param combination error code

* nits

* Test external matchfinder API

* C90 compat for simpleExternalMatchFinder

* Fix some @nocommits and an ASAN bug

* nit

* nit

* nits

* forward declare copySequencesToSeqStore functions in zstd_compress_internal.h

* nit

* nit

* nits

* Update copyright headers

* Fix CMake zstreamtest build

* Fix copyright headers (again)

* typo

* Add externalMatchfinder demo program to make contrib

* Reduce memory consumption for small blockSize

* ZSTD_postProcessExternalMatchFinderResult nits

* test sum(matchlen) + sum(litlen) == srcSize in debug builds

* refExternalMatchFinder -> registerExternalMatchFinder

* C90 nit

* zstreamtest nits

* contrib nits

* contrib nits

* allow block splitter + external matchfinder, refactor

* add windowSize param

* add contrib/externalMatchfinder/README.md

* docs

* go back to old RLE heuristic because of the first block issue

* fix initializer element is not a constant expression

* ref contrib from zstd.h

* extremely pedantic compiler warning fix, meson fix, typo fix

* Additional docs on API limitations

* minor nits

* Refactor maxNbSeq calculation into a helper function

* Fix copyright
 2022-12-28 16:45:14 -05:00
Yann Collet b17743e41b Signal parameter change during MT compression 2022-12-28 13:14:58 -08:00