 senhuang42
|
c54a25b666
|
Revert compressibility change
|
2020-11-02 11:38:58 -05:00 |
|
|
senhuang42
|
d4d0346b40
|
Update name of enum, clarify documentation
|
2020-11-02 11:38:17 -05:00 |
|
|
senhuang42
|
9102f30dbf
|
Update unit test
|
2020-11-02 11:30:31 -05:00 |
|
|
senhuang42
|
e6178f837f
|
Revert unnecessary seqCollector adjustment
|
2020-11-02 10:59:20 -05:00 |
|
|
senhuang42
|
e8501e00b8
|
Fix incorrect index increment in merge algorithm
|
2020-11-02 10:58:41 -05:00 |
|
|
senhuang42
|
a36fdada57
|
Add algorithm to remove all delimiters
|
2020-11-02 10:46:52 -05:00 |
|
|
senhuang42
|
435a3a0428
|
Update seqCollector definition
|
2020-11-02 10:19:26 -05:00 |
|
|
senhuang42
|
3327932609
|
Update ZSTD_getSequences function signature
|
2020-11-02 10:17:59 -05:00 |
|
|
senhuang42
|
7397d0102f
|
Add new enum for different sequence formats for ingestion/extraction
|
2020-11-02 10:15:53 -05:00 |
|
|
senandGitHub
|
f54109c589
|
Merge pull request #2378 from senhuang42/free_cress_ptr
[minor] Pass cRess_t by const ptr instead of by value
|
2020-11-01 20:33:25 -05:00 |
|
|
senandGitHub
|
c37c714ef1
|
Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api
Refine external ZSTD_Sequence API
|
2020-10-30 15:47:25 -04:00 |
|
|
senhuang42
|
f0da97642a
|
Specify that getSequences() will always emit block boundary sequences
|
2020-10-30 12:31:17 -04:00 |
|
|
senhuang42
|
536e89c723
|
Sequence extractor should update CBlockState
|
2020-10-30 12:13:19 -04:00 |
|
|
senhuang42
|
32cac2627a
|
Emit last literals of 0 size as well, to indicate block boundary
|
2020-10-29 16:41:17 -04:00 |
|
|
senhuang42
|
69bd5f0654
|
Correct literalsRead calculation to include longLength
|
2020-10-29 14:49:37 -04:00 |
|
|
senandGitHub
|
ff93440fc6
|
Merge pull request #2375 from senhuang42/ldm_oss_fuzz_testcase
Add a test case for LDM + opt parser with small uncompressible block
|
2020-10-29 09:32:05 -04:00 |
|
|
senhuang42
|
7198ebb213
|
Un-mix declarations and code
|
2020-10-28 18:51:03 -04:00 |
|
|
senhuang42
|
59624f3163
|
Remove implicit typecast to appease appVeyor windows build
|
2020-10-28 16:25:09 -04:00 |
|
|
senhuang42
|
60a52c29e6
|
Add check for allocation
|
2020-10-28 16:22:22 -04:00 |
|
|
senhuang42
|
3ed5d053d8
|
Clarify comments in zstd.h some more
|
2020-10-28 09:53:09 -04:00 |
|
|
senhuang42
|
169fc07aa1
|
Move test to appropriate location
|
2020-10-27 16:59:43 -04:00 |
|
|
senhuang42
|
db0b5d7d1e
|
Add test to fuzzer.c
|
2020-10-27 16:57:24 -04:00 |
|
|
senandGitHub
|
17b700d78a
|
Merge pull request #2366 from senhuang42/enable_ldm_by_default
Enable LDM by default if window size >= 128MB and strategy uses opt parser
|
2020-10-27 14:59:28 -04:00 |
|
|
senhuang42
|
3163909d14
|
Remove unused variable position
|
2020-10-27 12:58:12 -04:00 |
|
|
senhuang42
|
dc448563e9
|
Add test compatibility with last literals in sequences
|
2020-10-27 12:35:28 -04:00 |
|
|
senhuang42
|
1d221ecc03
|
Add support for representing last literals in the extracted seqs
|
2020-10-27 11:19:48 -04:00 |
|
|
senhuang42
|
9171f920cd
|
Improve documentation of seqStore_t
|
2020-10-27 10:50:22 -04:00 |
|
|
senhuang42
|
96b0ff7886
|
Improve documentation regarding various operations in copyBlockSequences
|
2020-10-27 10:36:06 -04:00 |
|
|
senhuang42
|
3a11c7eb03
|
Modify ZSTD_copyBlockSequences to agree with new API
|
2020-10-27 10:31:40 -04:00 |
|
|
senhuang42
|
761f40d1c6
|
Clarify and modify ZSTD_Sequence definition
|
2020-10-27 09:41:32 -04:00 |
|
|
senhuang42
|
8bdb32aebe
|
Add a function for LDM enable check
|
2020-10-20 13:46:02 -04:00 |
|
|
senhuang42
|
578e889ec1
|
Move ldm enable to compressStream2()
|
2020-10-20 13:04:45 -04:00 |
|
|
senhuang42
|
d28d8a1d72
|
Include LDM tables size for CCtx size estimation where relevant
|
2020-10-20 09:21:30 -04:00 |
|
|
senhuang42
|
81a2c02d8f
|
Move ldm no regression test to fuzzer longtests
|
2020-10-19 15:28:46 -04:00 |
|
|
senhuang42
|
b1c7fc5768
|
Add compatibility for multithreading
|
2020-10-19 12:07:06 -04:00 |
|
|
senhuang42
|
aad436da37
|
Document ldm enabled by default in zstd.h
|
2020-10-19 11:02:29 -04:00 |
|
|
senhuang42
|
df470e176b
|
Add unit test for no cctx requested params change
|
2020-10-19 10:52:41 -04:00 |
|
|
senhuang42
|
590f7f55f0
|
Add ldm enable condition in ZSTD_resetCCtx_internal
|
2020-10-19 10:26:17 -04:00 |
|
|
senhuang42
|
4d01979b62
|
Expose and call ZSTD_ldm_skipRawSeqStoreBytes()
|
2020-10-16 20:30:00 -04:00 |
|
|
senhuang42
|
adc53db6ed
|
Change make -j all check to make -j all && make check
|
2020-10-15 15:11:51 -04:00 |
|
|
senhuang42
|
42d037bdba
|
Add libregression build target, also fix make clean and .gitignore
|
2020-10-15 10:34:50 -04:00 |
|
|
senhuang42
|
f49926edf4
|
Change cycleLog adjustment to +3 from +4
|
2020-10-15 09:56:05 -04:00 |
|
|
senhuang42
|
ee84817fe7
|
Reset posInSequence when using ZSTD_referenceExternalSequences()
|
2020-10-14 22:06:08 -04:00 |
|
|
senhuang42
|
043b934ba2
|
Pass cRess_t by const const pointer
|
2020-10-14 20:19:46 -04:00 |
|
|
senhuang42
|
d0550bb18f
|
Clarify argument names, fix DEBUGLOG() statements
|
2020-10-14 15:45:43 -04:00 |
|
|
senhuang42
|
3f99c9b38d
|
Adjust match backwards count args
|
2020-10-14 15:23:03 -04:00 |
|
|
senhuang42
|
bf0d559449
|
Introduce, implement, and call ZSTD_ldm_countBackwardsMatch_2segments()
|
2020-10-14 12:58:06 -04:00 |
|
|
senhuang42
|
467e4383b0
|
Merge branch 'dev' of github.com:senhuang42/zstd into change_ldm_mt_config
|
2020-10-14 10:17:50 -04:00 |
|
|
senhuang42
|
a39614de1f
|
Fix incorrect usage of cycleLog() in --patch-from
|
2020-10-13 13:00:27 -04:00 |
|
|
senhuang42
|
354b5f1c0a
|
Use cycleLog instead of chainLog to determine LDM jobLog
|
2020-10-12 16:09:59 -04:00 |
|
|
senhuang42
|
d6911b86be
|
Require LDM matches to be strictly greater in length
|
2020-10-09 12:56:18 -04:00 |
|
|
senhuang42
|
b9c8033cde
|
Define kNullRawSeqStore for every file
|
2020-10-07 19:02:41 -04:00 |
|
|
senhuang42
|
a6165c1b28
|
Change matchState_t::ldmSeqStore to pointer
|
2020-10-07 14:13:57 -04:00 |
|
|
senhuang42
|
abce708a56
|
Move posInSequence correction to correct location
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0c515590d8
|
Replace offCode of largest match if ldm's offCode is superior
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0fac8e07e1
|
Refactor usage of ms->ldmSeqStore so that it is not modified during compressBlock(), and simplify skipRawSeqStoreBytes
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
a5500cf2af
|
Refactor separate ldm variables all into one struct
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0731b94e7c
|
Use kNullRawSeqStore constant in zstdmt_compress.c
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0325d878f2
|
Remove bubbling down matches with longer offCode and same matchLen
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
031b7ec15f
|
Disable LDM minMatch adjustment when using opt parser
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
ddf8a3f1b9
|
Enable inclusion of mid-flight LDMs in opt parser
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
88f72ed942
|
Correct incorrect offcode calculation
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
e96ea5d147
|
Fix static analyze fuzzer.c error
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
d8b43a4202
|
Add explicit conversion of size_t to U32
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
b8bfc4e63d
|
Add cSize regression test to fuzzer.c
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
c87d2e5866
|
Prefix new static ldm helpers with ZSTD_opt
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
429dec4f42
|
Add DEBUGLOG() calls in ldm helpers
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
10647924f1
|
Make function descriptions more accurate
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
1a687b3fcb
|
Improve documentation of relevant structs
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
37617e23d7
|
Correct matchLength calculation and remove unnecessary functions
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
7dee62c287
|
Reset ldmSeqStore after initStats_ultra() pass for btultra2
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0718aa70df
|
Refactor existing functions to use posInSequence
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
7348b40a87
|
Adjustments to ldm_calculateMatchRange() to calculate bounds correctly
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
a1ef2db5b2
|
Add ldm_calculateMatchRange() function
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
ef823e0299
|
Remove rawSeqStore.base and add rawSeqStore.posInSequence
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
cfd2aec1b7
|
Add unit tests into playTests.sh
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
4793ae3b84
|
Prevent duplicate LDMs from being inserted
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
65f9cfeeec
|
Add extra bounds check to prevent heap access after free ASAN error
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
bff5785fd5
|
Address mixed variables C90 warning
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
724b94ed18
|
ldm_getNextMatch fixed return values
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
ea92fb3a68
|
Cleanups, add comments and explanations
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
78da2e1808
|
Fixed sifting algorithm
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
6ccd97fc96
|
Fixed end of match boundary update issues
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
28394b64f2
|
Add proper bounds check on adding ldms
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
a2f2b58d04
|
Add a function ldm_voidSequences()
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
9c3c7cd20e
|
Fix function argument to getNextMatch()
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
c8b8572b38
|
Adjustments to no longer segfault on nci
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
f57c7e6bbf
|
Add base adjustment correction
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
5df9b5e05f
|
Add initial getNextMatch() in opt parser
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
f8ce7cabc3
|
Added more debugging
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
84009a076a
|
Add re-copying of ldmSeqStore after processing
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
42395a70c2
|
Add debug statements, flesh out functions
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
dd3dd199bb
|
Get zstd to build with new functions and callsites, fix arguments
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
766c4a8c28
|
Implement part of ldm_maybeAddLdm()
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
84777059d2
|
Implement ldm_getNextMatch()
|
2020-10-07 13:56:24 -04:00 |
|
|
senhuang42
|
28c74bf591
|
Implement basic splitSequence and skipSequence functions
|
2020-10-07 13:56:24 -04:00 |
|
|
senhuang42
|
634ab7830d
|
Flesh out required args for ldm_handleLdm()
|
2020-10-07 13:56:24 -04:00 |
|
|
senhuang42
|
db70761032
|
Add callsites to appropriate locations in ..opt_generic()
|
2020-10-07 13:56:24 -04:00 |
|
|
senhuang42
|
aea61e3c91
|
Add ldm helper function declarations into opt parser
|
2020-10-07 13:56:24 -04:00 |
|
|
senhuang42
|
35d9f488f5
|
Modify codepath to use opt parser exclusively if the compression level is high enough
|
2020-10-07 13:56:24 -04:00 |
|