ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,057 Commits 17 Branches 77 Tags
v1.5.4
Commit Graph
93 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Elliot GorokhovskyandGitHub ff42ed1582 Rename "External Matchfinder" to "Block-Level Sequence Producer" (#3484) 
* change "external matchfinder" to "external sequence producer"

* migrate contrib/ to new naming convention

* fix contrib build

* fix error message

* update debug strings

* fix def of invalid sequences in zstd.h

* nit

* update CHANGELOG

* fix .gitignore
 2023-02-09 17:01:17 -05:00
Elliot GorokhovskyandGitHub 31e41b3d5e Merge pull request #3471 from embg/fast_seq_parse 
Reduce external matchfinder API overhead by 25%
 2023-02-01 21:30:36 -05:00
Elliot Gorokhovsky 3fe5f1fbb9 assert externalRepSearch != ZSTD_ps_auto 2023-02-01 18:24:46 -08:00
Elliot Gorokhovsky 7f8189ca57 add ZSTD_c_fastExternalSequenceParsing cctxParam 2023-02-01 09:09:53 -08:00
Elliot GorokhovskyandGitHub 64052ef57d Guard against invalid sequences from external matchfinders (#3465) 2023-01-31 13:55:48 -05:00
Elliot GorokhovskyandGitHub 41682e6293 Merge pull request #3448 from facebook/embg-doc-fix 
Fix ZSTD_estimate* and ZSTD_initCStream() docs
 2023-01-23 15:04:53 -05:00
Elliot GorokhovskyandGitHub 3bfd3be5fb Fix ZSTD_estimate* and ZSTD_initCStream() docs 
Fix the following documentation bugs:
* Note that `ZSTD_estimate*` functions are not compatible with the external matchfinder API
* Note that `ZSTD_estimateCStreamSize_usingCCtxParams()` is not compatible with `nbWorkers >= 1`
* Remove incorrect warning that the legacy streaming API is incompatible with advanced parameters and/or dictionary compression
* Note that `ZSTD_initCStream()` is incompatible with dictionary compression
* Warn that
 2023-01-23 13:28:36 -05:00
Elliot GorokhovskyandGitHub 6aee603d0e Merge pull request #3446 from facebook/dependabot/github_actions/github/codeql-action-2.1.39 
Bump github/codeql-action from 2.1.38 to 2.1.39
 2023-01-23 11:55:13 -05:00
Elliot GorokhovskyandGitHub 64963dcbd6 Merge pull request #3437 from embg/fuzz_emf 
Fuzz the External Matchfinder API
 2023-01-20 11:42:59 -05:00
Elliot GorokhovskyGitHubNick Terrell
 		f593e54ee1 Enable if == 1 rather than if == 0 
Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
 2023-01-20 11:41:53 -05:00
Elliot Gorokhovsky 3f9f568aa6 Fuzz the external matchfinder API 2023-01-19 13:33:25 -08:00
Elliot GorokhovskyandGitHub bce0382c82 Bugfixes for the External Matchfinder API (#3433) 
* external matchfinder bugfixes + tests

* small doc fix
 2023-01-19 10:41:24 -05:00
Elliot GorokhovskyandGitHub 018b68f332 fix msys2 symlink breakage in CI (#3429) 2023-01-17 12:10:15 -05:00
Elliot GorokhovskyandGitHub 5d8cfa6b96 Deprecate advanced streaming functions (#3408) 
* deprecate advanced streaming functions

* remove internal usage of the deprecated functions

* nit

* suppress warnings in tests/zstreamtest.c

* purge ZSTD_initDStream_usingDict

* nits

* c90 compat

* zstreamtest.c already disables deprecation warnings!

* fix initDStream() return value

* fix typo

* wasn't able to import private symbol properly, this commit works around that

* new strategy for zbuff

* undo zbuff deprecation warning changes

* move ZSTD_DISABLE_DEPRECATE_WARNINGS from .h to .c
 2023-01-13 14:51:47 -05:00
Elliot GorokhovskyandGitHub 4f7183d887 Completely overhaul Windows CI (#3410) 
* Overhaul windows CI

* upgrade setup-msbuild from v1.1.3 to v1.3

* remove cmake 2019 test

* fix 32-bit gcc mingw test

* merge conflict
 2023-01-11 16:29:23 -05:00
Elliot GorokhovskyandGitHub 2a402626dd External matchfinder API (#3333) 
* First building commit with sample matchfinder

* Set up ZSTD_externalMatchCtx struct

* move seqBuffer to ZSTD_Sequence*

* support non-contiguous dictionary

* clean up parens

* add clearExternalMatchfinder, handle allocation errors

* Add useExternalMatchfinder cParam

* validate useExternalMatchfinder cParam

* Disable LDM + external matchfinder

* Check for static CCtx

* Validate mState and mStateDestructor

* Improve LDM check to cover both branches

* Error API with optional fallback

* handle RLE properly for external matchfinder

* nit

* Move to a CDict-like model for resource ownership

* Add hidden useExternalMatchfinder bool to CCtx_params_s

* Eliminate malloc, move to cwksp allocation

* Handle CCtx reset properly

* Ensure seqStore has enough space for external sequences

* fix capitalization

* Add DEBUGLOG statements

* Add compressionLevel param to matchfinder API

* fix c99 issues and add a param combination error code

* nits

* Test external matchfinder API

* C90 compat for simpleExternalMatchFinder

* Fix some @nocommits and an ASAN bug

* nit

* nit

* nits

* forward declare copySequencesToSeqStore functions in zstd_compress_internal.h

* nit

* nit

* nits

* Update copyright headers

* Fix CMake zstreamtest build

* Fix copyright headers (again)

* typo

* Add externalMatchfinder demo program to make contrib

* Reduce memory consumption for small blockSize

* ZSTD_postProcessExternalMatchFinderResult nits

* test sum(matchlen) + sum(litlen) == srcSize in debug builds

* refExternalMatchFinder -> registerExternalMatchFinder

* C90 nit

* zstreamtest nits

* contrib nits

* contrib nits

* allow block splitter + external matchfinder, refactor

* add windowSize param

* add contrib/externalMatchfinder/README.md

* docs

* go back to old RLE heuristic because of the first block issue

* fix initializer element is not a constant expression

* ref contrib from zstd.h

* extremely pedantic compiler warning fix, meson fix, typo fix

* Additional docs on API limitations

* minor nits

* Refactor maxNbSeq calculation into a helper function

* Fix copyright
 2022-12-28 16:45:14 -05:00
Elliot GorokhovskyandGitHub 588073c920 Merge pull request #3349 from embg/frame_header_fields 
Reserve two fields in ZSTD_frameHeader
 2022-12-14 10:58:35 -05:00
Elliot Gorokhovsky c43da3d605 Fix C90 compat 2022-12-13 18:01:32 -08:00
Elliot Gorokhovsky e1e82f74f1 Reserve two fields in ZSTD_frameHeader 2022-12-13 17:45:05 -08:00
Elliot GorokhovskyandGitHub a89e6b6812 Merge pull request #3331 from embg/issue3330 
Fix gcc-7 and gcc-8 CI breakages
 2022-12-02 18:34:37 -05:00
Elliot Gorokhovsky 3f0b912a80 Fix gcc-7 and gcc-8 CI breakages 2022-12-02 16:35:17 -05:00
Elliot GorokhovskyandGitHub 2a356737d8 Merge pull request #3321 from embg/ldmCparamFix 
Complete migration of ZSTD_c_enableLongDistanceMatching to ZSTD_paramSwitch_e framework
 2022-11-21 17:37:59 -05:00
Elliot Gorokhovsky bb3c01c853 Migrate other test usages of boolean LDM flag to paramSwitch enum 2022-11-21 16:20:38 -05:00
Elliot Gorokhovsky 3720910d06 Fix fuzzer failure 2022-11-21 16:09:04 -05:00
Elliot Gorokhovsky c8d870fe52 Improve LDM cparam validation logic 2022-11-21 15:39:18 -05:00
Elliot GorokhovskyandGitHub b1182fe242 Merge pull request #3300 from yiyuaner/dev 
Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable
 2022-11-02 15:28:02 -04:00
Elliot GorokhovskyandGitHub bb23f7b99b Merge pull request #3304 from GermanAizek/dev 
Rewrite check *bufStart condition
 2022-11-01 12:45:42 -04:00
Elliot GorokhovskyandGitHub 39ab02a71f Merge pull request #3257 from embg/seqBench2 
Benchmark program for sequence compression API
 2022-09-09 15:53:28 -04:00
Elliot GorokhovskyandGitHub 6600a05949 Merge pull request #3259 from DimitriPapadopoulos/codespell 
Fix typos found by codespell
 2022-09-09 11:15:05 -04:00
Elliot Gorokhovsky 61c79bf0d6 Benchmark program for sequence compression API 2022-09-08 09:20:50 -07:00
Elliot GorokhovskyandGitHub ef60302af9 Merge pull request #3230 from grossws/fix3229-docs 
Add description for ZSTD_decompressStream and ZSTD_initDStream
 2022-08-16 12:48:23 -04:00
Elliot GorokhovskyandGitHub 28ceb63503 Merge pull request #3220 from embg/issue3200 
Disallow empty string as argument for --output-dir-flat and --output-dir-mirror
 2022-08-01 14:04:57 -04:00
Elliot Gorokhovsky f9f27de91c Disallow empty output directory 2022-07-29 14:48:33 -07:00
Elliot Gorokhovsky e1873ad576 Fix buffer underflow for null dir1 2022-07-29 11:10:47 -07:00
Elliot GorokhovskyandGitHub e5db7c93f5 Merge pull request #3197 from embg/docstring_clarify 
Clarify benchmark chunking docstring
 2022-07-26 13:26:15 -04:00
Elliot GorokhovskyandGitHub bef1d9a831 Merge pull request #3209 from zhuhan0/dev 
[largeNbDicts] Second try at fixing decompression segfault to always create compressInstructions
 2022-07-26 13:19:38 -04:00
Elliot GorokhovskyandGitHub 466e13f722 Merge pull request #3205 from zhuhan0/dev 
[contrib][largeNbDicts] Fix decompression segfault; Add additional benchmark metrics
 2022-07-20 16:07:04 -04:00
Elliot Gorokhovsky 6d75b36b7f Clarify -B docstring 2022-07-14 00:22:21 -04:00
Elliot GorokhovskyandGitHub 3ef92cfcd4 Merge pull request #3180 from nocnokneo/MSVCBuildTests 
Fix ZSTD_BUILD_TESTS=ON with MSVC
 2022-07-05 13:13:34 -04:00
Elliot GorokhovskyandGitHub 5d2fb4288f Merge pull request #3179 from embg/1.5.3_bump 
Prepare v1.5.3
 2022-06-29 13:03:52 -07:00
Elliot Gorokhovsky bb3839a78c make -C programs zstd.1 2022-06-29 14:55:14 -04:00
Elliot Gorokhovsky 5c382bf110 1.5.3 version bump 2022-06-29 14:45:53 -04:00
Elliot GorokhovskyandGitHub e9d6fc867a Merge pull request #3177 from embg/dms_prefetch2 
Add prefetchCDictTables CCtxParam (+10-20% cold dict compression speed)
 2022-06-24 08:24:43 -07:00
Elliot Gorokhovsky cb9e341129 Nits 2022-06-23 16:59:21 -04:00
Elliot GorokhovskyandGitHub bb4a3c71ef Update README.md for fuzzers (#3174) 
* Update README.md for fuzzers

* Add ls corpora/*crash command

* nit

* Clarify wording and add Nick's command

* Minor clarification
 2022-06-22 21:02:07 -04:00
Elliot Gorokhovsky 747e06f4f6 Add tests 2022-06-22 17:05:23 -04:00
Elliot Gorokhovsky 6bd5ac6713 add prefetchCDictTables to largeNbDicts 2022-06-22 16:13:07 -04:00
Elliot Gorokhovsky 93b89fb24b Add docs 2022-06-22 16:13:07 -04:00
Elliot Gorokhovsky 2a128110d0 Add prefetchCDictTables CCtxParam 2022-06-22 16:13:07 -04:00
Elliot GorokhovskyandGitHub f6ef14329f "Short cache" optimization for level 1-4 DMS (+5-30% compression speed) (#3152) 
* first attempt at fast DMS short cache

* significant wins for some scenarios

* fix all clang regressions

* nits

* fix 1.5% gcc11 regression on hot 110Kdict scenario

* fix CI

* nit

* Add tags to doublefast hash table

* use tags in doublefast DMS

* Fix CI

* Clean up some hardcoded logic / constants

* Switch forCCtx to an enum

* nit

* add short cache to ip+1 long search

* Move tag size into hashLog

* Minor nits

* Truncate dictionaries greater than 16MB in short cache mode

* Helper function for tag comparison

* Cap short cache hashLog at 24 to prevent overflow

* size_t dictTagsMatch -> int dictTagsMatch

* nit

* Clean up and comment dictionary truncation

* Move ZSTD_tableFillPurpose_e next to ZSTD_dictTableLoadMethod_e

* Comment and expand helper functions

* Asserts and documentation

* nit
 2022-06-21 17:27:19 -04:00
Elliot GorokhovskyandGitHub b7b7edb3a3 Merge pull request #3161 from embg/largeNbDictsImprovements 
[contrib] largeNbDicts bugfix + improvements
 2022-06-15 07:39:50 -07:00
Elliot Gorokhovsky 2bbdc9f40e Fix FILE handle leak 2022-06-14 14:57:54 -07:00
Elliot Gorokhovsky f7ebbcd0cc Support advanced API so forceCopy/forceAttach works properly 2022-06-14 14:52:51 -07:00
Elliot Gorokhovsky e0c4863c5c largeNbDicts bugfix + improvements 2022-06-13 17:26:44 -07:00
Elliot GorokhovskyandGitHub b944db0c45 Merge pull request #3160 from danlark1/patch-1 
Fix big endian ARM NEON path
 2022-06-13 14:01:43 -04:00
Elliot GorokhovskyandGitHub f313a773a4 Merge pull request #3157 from embg/huge_dict_bugfix 
Bugfix for huge dictionaries
 2022-06-09 15:35:29 -04:00
Elliot Gorokhovsky 31bd6402c6 Bugfix for huge dictionaries 2022-06-09 11:39:30 -04:00
Elliot GorokhovskyandGitHub f349d18776 Merge pull request #3127 from embg/repcode_history 
Correct and clarify repcode offset history logic
 2022-05-12 13:50:15 -04:00
Elliot Gorokhovsky 3620a0a565 Nits 2022-05-12 12:53:15 -04:00
Elliot Gorokhovsky 22875ece61 Nits 2022-05-09 21:01:38 -04:00
Elliot Gorokhovsky 97aabc496e Correct and clarify repcode offset history logic 2022-05-09 21:01:38 -04:00
Elliot GorokhovskyandGitHub 8bf32de850 Merge pull request #3126 from embg/fix_freebsd_ci 
Unbreak FreeBSD CI
 2022-05-09 19:48:13 -04:00
Elliot Gorokhovsky 83049cb3fe Unbreak FreeBSD CI 2022-05-09 18:28:03 -04:00
Elliot GorokhovskyandGitHub 7915c1164e Merge pull request #3114 from embg/fast_extdict_pipeline2 
Software pipeline for ZSTD_compressBlock_fast_extDict
 2022-05-05 15:06:47 -04:00
Elliot Gorokhovsky 3be9a81e46 Update results.csv 2022-05-04 16:05:37 -04:00
Elliot Gorokhovsky ac371be27b Remove hasStep variant (not enough wins to justify the code size increase) 2022-04-28 18:06:24 -04:00
Elliot Gorokhovsky ce6b69f5c5 Final nit 2022-04-28 14:49:45 -04:00
Elliot Gorokhovsky 6a2e1f7c69 Revert "Hardcode repcode safety check, fix cosmetic nits" 
This reverts commit 518cb83833.
 2022-04-27 18:16:21 -04:00
Elliot Gorokhovsky 518cb83833 Hardcode repcode safety check, fix cosmetic nits 2022-04-26 17:54:25 -04:00
Elliot Gorokhovsky 809f652912 Optimize repcode predicate, hardcode hasStep == 0 scenario, cosmetic fixes 2022-04-20 14:40:52 -04:00
Elliot Gorokhovsky 2820efe7ec Nits 2022-04-19 11:39:52 -04:00
Elliot Gorokhovsky 3536262f70 Port noDict pipeline 2022-04-15 12:16:16 -04:00
Elliot GorokhovskyandGitHub 3e6bbdd847 Disable visual-2015 tests (#3106) 2022-03-31 12:26:20 -04:00
Elliot GorokhovskyandGitHub 64efba4c5e Software pipeline for ZSTD_compressBlock_fast_dictMatchState (#3086) 
* prefetch dict content inside loop

* ip0/ip1 pipeline

* add L2_4 prefetch to dms pipeline

* Remove L1 prefetch

* Remove L2 prefetching

* Reduce # of gotos

* Cosmetic fixes

* Check final position sometimes

* Track step size as in bc768bc

* Fix nits
 2022-03-17 12:35:11 -04:00
Elliot GorokhovskyandGitHub 0c386afbfd Fix CI failures by adding apt-get update to Github Actions (#3082) 
* Fix CI failures by adding apt-get update to Makefile

* Fix travis failure caused by apt-get update

* Move apt-get update from Makefile to Github Actions .yml

* Revert .travis.yml change

* Fix typo
 2022-03-03 14:52:06 -05:00
Elliot GorokhovskyandGitHub 87406b5f3b Merge pull request #3075 from TocarIP/tokarip/bzhi 
Use helper function for bit manipulations.
 2022-02-24 16:52:48 -05:00
Elliot GorokhovskyandGitHub 621d798988 Merge pull request #3045 from embg/asm_refactor 
Refactor bitwise intrinsics
 2022-02-17 13:08:08 -05:00
Elliot Gorokhovsky 71d9dab76f Replace XOR with subtraction for readability 2022-02-16 16:49:42 -05:00
Elliot Gorokhovsky 856c7dc51d Fix fuzzer.c nits and replace CLZ fallback 2022-02-16 11:40:05 -05:00
Elliot Gorokhovsky 00f2acba36 Add back check to prevent Win32 static analysis issues 2022-02-15 11:41:09 -05:00
Elliot Gorokhovsky 7c674a0919 Add tests for bitwise intrinsics 2022-02-14 18:36:33 -05:00
Elliot Gorokhovsky 6994a9f99c bits.h refactor and bugfix 2022-02-14 16:59:55 -05:00
Elliot Gorokhovsky 529cd7b821 Fix nits 2022-02-14 14:24:50 -05:00
Elliot Gorokhovsky 796182652d Pull out software fallbacks 2022-02-14 11:16:03 -05:00
Elliot Gorokhovsky db2f4a6532 Move bitwise builtins into bits.h 2022-02-14 11:16:03 -05:00
Elliot GorokhovskyandGitHub 970460f67d Merge pull request #3063 from embg/largeNbDicts 
Bugfix and new features for largeNbDicts benchmark
 2022-02-11 13:55:35 -05:00
Elliot Gorokhovsky 762898f5e4 Bugfix and new features for largeNbDicts benchmark 2022-02-11 13:15:16 -05:00
Elliot GorokhovskyandGitHub caf2fa170b Merge pull request #3061 from embg/debug_ci_windows 
Debug Windows CI failures
 2022-02-09 16:29:56 -05:00
Elliot Gorokhovsky 9caabc01c4 Replace "windows-latest" with "windows-2019" in CI workflows 2022-02-09 15:45:43 -05:00
Elliot Gorokhovsky f936dd89cb Minor lint fix 2022-01-20 11:54:43 -07:00
Elliot Gorokhovsky 8250faa01b Update CI documentation 2022-01-19 09:58:31 -07:00
Elliot Gorokhovsky 9b6dfedf0c Documentation and minor refactor to clarify MT memory management. 2022-01-18 09:43:05 -07:00
Elliot Gorokhovsky 71c0c07c19 Allow user to specify memory limit for dictionary training 2021-12-14 14:29:01 -05:00