 senhuang42
|
337fac216d
|
Add logic to handle ZSTD_sequenceRange
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
85822ddd53
|
Add last literals handling like getSequences()
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
2cff8df1a2
|
Pull block compression out of main compressSequences() function
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
cfced9344a
|
Implement ZSTD_updateSequenceRange
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
b116e1f211
|
Modify SequenceRange to have posInSequence
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
d99b675112
|
Add function definition for sequenceRange updater
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
74e95c05cc
|
Add ZSTD_SequenceRange to count ranges in array of ZSTD_Sequence
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
89f3848310
|
Add support for repcodes
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
3e930fd044
|
Code cleanup, add debuglog statments
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
086513b5b9
|
Implement first pass at compressSequences()
|
2020-11-16 10:49:16 -05:00 |
|
|
senhuang42
|
a9327b1e9b
|
Add initial function prototype for ZSTD_compressSequences_ext (to be renamed later)
|
2020-11-16 10:33:35 -05:00 |
|
|
sen
|
9736c497ce
|
Merge pull request #2393 from senhuang42/fix_sequence_extractions_api
Improve repcode handling in sequence extraction API
|
2020-11-15 18:29:52 -05:00 |
|
|
senhuang42
|
9d936d61d2
|
Reduce number of memcpy() calls
|
2020-11-13 19:43:30 -05:00 |
|
|
senhuang42
|
be4ac6c5bc
|
Use existing repcode update function to implement updates
|
2020-11-12 16:51:12 -05:00 |
|
|
senhuang42
|
674c9b9235
|
Add in proper block repcode histories
|
2020-11-12 15:34:37 -05:00 |
|
|
senhuang42
|
06c7f14066
|
Let block reps persist
|
2020-11-12 12:24:44 -05:00 |
|
|
senhuang42
|
396275068c
|
Fix incorrect repcode setting
|
2020-11-12 11:57:01 -05:00 |
|
|
senhuang42
|
1a8af0de73
|
Improve unit test
|
2020-11-12 11:09:09 -05:00 |
|
|
senhuang42
|
4d4fd2c55f
|
Overhaul repcode handling logic
|
2020-11-12 10:59:35 -05:00 |
|
|
sen
|
f62edf0fe9
|
Merge pull request #2381 from senhuang42/expand_sequence_extraction_api
Add enum to define ZSTD_Sequence type and update sequence extraction API
|
2020-11-06 13:00:31 -05:00 |
|
|
senhuang42
|
7d1dea070c
|
Update unit tests
|
2020-11-06 11:10:37 -05:00 |
|
|
senhuang42
|
779df995c6
|
Implement mergeGeneratedSequences()
|
2020-11-06 10:55:46 -05:00 |
|
|
senhuang42
|
51abd58208
|
Rename getSequences() to generateSequences()
|
2020-11-06 10:53:22 -05:00 |
|
|
senhuang42
|
261ea69661
|
Add new mergeGeneratedSequences() function
|
2020-11-06 10:52:34 -05:00 |
|
|
senhuang42
|
f782cac3d4
|
Change block delimiter removing to linear time approach
|
2020-11-02 17:06:20 -05:00 |
|
|
senhuang42
|
3c9b43da1d
|
Remove trailing comma
|
2020-11-02 11:53:58 -05:00 |
|
|
senhuang42
|
3434049c1f
|
Use ZSTD_memmove() instead of memmove()
|
2020-11-02 11:43:19 -05:00 |
|
|
senhuang42
|
c54a25b666
|
Revert compressibility change
|
2020-11-02 11:38:58 -05:00 |
|
|
senhuang42
|
d4d0346b40
|
Update name of enum, clarify documentation
|
2020-11-02 11:38:17 -05:00 |
|
|
senhuang42
|
9102f30dbf
|
Update unit test
|
2020-11-02 11:30:31 -05:00 |
|
|
senhuang42
|
e6178f837f
|
Revert unnecessary seqCollector adjustment
|
2020-11-02 10:59:20 -05:00 |
|
|
senhuang42
|
e8501e00b8
|
Fix incorrect index increment in merge algorithm
|
2020-11-02 10:58:41 -05:00 |
|
|
senhuang42
|
a36fdada57
|
Add algorithm to remove all delimiters
|
2020-11-02 10:46:52 -05:00 |
|
|
senhuang42
|
435a3a0428
|
Update seqCollector definition
|
2020-11-02 10:19:26 -05:00 |
|
|
senhuang42
|
3327932609
|
Update ZSTD_getSequences function signature
|
2020-11-02 10:17:59 -05:00 |
|
|
senhuang42
|
7397d0102f
|
Add new enum for different sequence formats for ingestion/extraction
|
2020-11-02 10:15:53 -05:00 |
|
|
sen
|
f54109c589
|
Merge pull request #2378 from senhuang42/free_cress_ptr
[minor] Pass cRess_t by const ptr instead of by value
|
2020-11-01 20:33:25 -05:00 |
|
|
sen
|
c37c714ef1
|
Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api
Refine external ZSTD_Sequence API
|
2020-10-30 15:47:25 -04:00 |
|
|
senhuang42
|
f0da97642a
|
Specify that getSequences() will always emit block boundary sequences
|
2020-10-30 12:31:17 -04:00 |
|
|
senhuang42
|
536e89c723
|
Sequence extractor should update CBlockState
|
2020-10-30 12:13:19 -04:00 |
|
|
senhuang42
|
32cac2627a
|
Emit last literals of 0 size as well, to indicate block boundary
|
2020-10-29 16:41:17 -04:00 |
|
|
senhuang42
|
69bd5f0654
|
Correct literalsRead calculation to include longLength
|
2020-10-29 14:49:37 -04:00 |
|
|
sen
|
ff93440fc6
|
Merge pull request #2375 from senhuang42/ldm_oss_fuzz_testcase
Add a test case for LDM + opt parser with small uncompressible block
|
2020-10-29 09:32:05 -04:00 |
|
|
senhuang42
|
7198ebb213
|
Un-mix declarations and code
|
2020-10-28 18:51:03 -04:00 |
|
|
senhuang42
|
59624f3163
|
Remove implicit typecast to appease appVeyor windows build
|
2020-10-28 16:25:09 -04:00 |
|
|
senhuang42
|
60a52c29e6
|
Add check for allocation
|
2020-10-28 16:22:22 -04:00 |
|
|
senhuang42
|
3ed5d053d8
|
Clarify comments in zstd.h some more
|
2020-10-28 09:53:09 -04:00 |
|
|
senhuang42
|
169fc07aa1
|
Move test to appropriate location
|
2020-10-27 16:59:43 -04:00 |
|
|
senhuang42
|
db0b5d7d1e
|
Add test to fuzzer.c
|
2020-10-27 16:57:24 -04:00 |
|
|
sen
|
17b700d78a
|
Merge pull request #2366 from senhuang42/enable_ldm_by_default
Enable LDM by default if window size >= 128MB and strategy uses opt parser
|
2020-10-27 14:59:28 -04:00 |
|
|
senhuang42
|
3163909d14
|
Remove unused variable position
|
2020-10-27 12:58:12 -04:00 |
|
|
senhuang42
|
dc448563e9
|
Add test compatibility with last literals in sequences
|
2020-10-27 12:35:28 -04:00 |
|
|
senhuang42
|
1d221ecc03
|
Add support for representing last literals in the extracted seqs
|
2020-10-27 11:19:48 -04:00 |
|
|
senhuang42
|
9171f920cd
|
Improve documentation of seqStore_t
|
2020-10-27 10:50:22 -04:00 |
|
|
senhuang42
|
96b0ff7886
|
Improve documentation regarding various operations in copyBlockSequences
|
2020-10-27 10:36:06 -04:00 |
|
|
senhuang42
|
3a11c7eb03
|
Modify ZSTD_copyBlockSequences to agree with new API
|
2020-10-27 10:31:40 -04:00 |
|
|
senhuang42
|
761f40d1c6
|
Clarify and modify ZSTD_Sequence definition
|
2020-10-27 09:41:32 -04:00 |
|
|
senhuang42
|
8bdb32aebe
|
Add a function for LDM enable check
|
2020-10-20 13:46:02 -04:00 |
|
|
senhuang42
|
578e889ec1
|
Move ldm enable to compressStream2()
|
2020-10-20 13:04:45 -04:00 |
|
|
senhuang42
|
d28d8a1d72
|
Include LDM tables size for CCtx size estimation where relevant
|
2020-10-20 09:21:30 -04:00 |
|
|
senhuang42
|
81a2c02d8f
|
Move ldm no regression test to fuzzer longtests
|
2020-10-19 15:28:46 -04:00 |
|
|
senhuang42
|
b1c7fc5768
|
Add compatibility for multithreading
|
2020-10-19 12:07:06 -04:00 |
|
|
senhuang42
|
aad436da37
|
Document ldm enabled by default in zstd.h
|
2020-10-19 11:02:29 -04:00 |
|
|
senhuang42
|
df470e176b
|
Add unit test for no cctx requested params change
|
2020-10-19 10:52:41 -04:00 |
|
|
senhuang42
|
590f7f55f0
|
Add ldm enable condition in ZSTD_resetCCtx_internal
|
2020-10-19 10:26:17 -04:00 |
|
|
senhuang42
|
4d01979b62
|
Expose and call ZSTD_ldm_skipRawSeqStoreBytes()
|
2020-10-16 20:30:00 -04:00 |
|
|
senhuang42
|
adc53db6ed
|
Change make -j all check to make -j all && make check
|
2020-10-15 15:11:51 -04:00 |
|
|
senhuang42
|
42d037bdba
|
Add libregression build target, also fix make clean and .gitignore
|
2020-10-15 10:34:50 -04:00 |
|
|
senhuang42
|
f49926edf4
|
Change cycleLog adjustment to +3 from +4
|
2020-10-15 09:56:05 -04:00 |
|
|
senhuang42
|
ee84817fe7
|
Reset posInSequence when using ZSTD_referenceExternalSequences()
|
2020-10-14 22:06:08 -04:00 |
|
|
senhuang42
|
043b934ba2
|
Pass cRess_t by const const pointer
|
2020-10-14 20:19:46 -04:00 |
|
|
senhuang42
|
d0550bb18f
|
Clarify argument names, fix DEBUGLOG() statements
|
2020-10-14 15:45:43 -04:00 |
|
|
senhuang42
|
3f99c9b38d
|
Adjust match backwards count args
|
2020-10-14 15:23:03 -04:00 |
|
|
senhuang42
|
bf0d559449
|
Introduce, implement, and call ZSTD_ldm_countBackwardsMatch_2segments()
|
2020-10-14 12:58:06 -04:00 |
|
|
senhuang42
|
467e4383b0
|
Merge branch 'dev' of github.com:senhuang42/zstd into change_ldm_mt_config
|
2020-10-14 10:17:50 -04:00 |
|
|
senhuang42
|
a39614de1f
|
Fix incorrect usage of cycleLog() in --patch-from
|
2020-10-13 13:00:27 -04:00 |
|
|
senhuang42
|
354b5f1c0a
|
Use cycleLog instead of chainLog to determine LDM jobLog
|
2020-10-12 16:09:59 -04:00 |
|
|
senhuang42
|
d6911b86be
|
Require LDM matches to be strictly greater in length
|
2020-10-09 12:56:18 -04:00 |
|
|
senhuang42
|
b9c8033cde
|
Define kNullRawSeqStore for every file
|
2020-10-07 19:02:41 -04:00 |
|
|
senhuang42
|
a6165c1b28
|
Change matchState_t::ldmSeqStore to pointer
|
2020-10-07 14:13:57 -04:00 |
|
|
senhuang42
|
abce708a56
|
Move posInSequence correction to correct location
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0c515590d8
|
Replace offCode of largest match if ldm's offCode is superior
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0fac8e07e1
|
Refactor usage of ms->ldmSeqStore so that it is not modified during compressBlock(), and simplify skipRawSeqStoreBytes
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
a5500cf2af
|
Refactor separate ldm variables all into one struct
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0731b94e7c
|
Use kNullRawSeqStore constant in zstdmt_compress.c
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0325d878f2
|
Remove bubbling down matches with longer offCode and same matchLen
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
031b7ec15f
|
Disable LDM minMatch adjustment when using opt parser
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
ddf8a3f1b9
|
Enable inclusion of mid-flight LDMs in opt parser
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
88f72ed942
|
Correct incorrect offcode calculation
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
e96ea5d147
|
Fix static analyze fuzzer.c error
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
d8b43a4202
|
Add explicit conversion of size_t to U32
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
b8bfc4e63d
|
Add cSize regression test to fuzzer.c
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
c87d2e5866
|
Prefix new static ldm helpers with ZSTD_opt
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
429dec4f42
|
Add DEBUGLOG() calls in ldm helpers
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
10647924f1
|
Make function descriptions more accurate
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
1a687b3fcb
|
Improve documentation of relevant structs
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
37617e23d7
|
Correct matchLength calculation and remove unnecessary functions
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
7dee62c287
|
Reset ldmSeqStore after initStats_ultra() pass for btultra2
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
0718aa70df
|
Refactor existing functions to use posInSequence
|
2020-10-07 13:56:25 -04:00 |
|
|
senhuang42
|
7348b40a87
|
Adjustments to ldm_calculateMatchRange() to calculate bounds correctly
|
2020-10-07 13:56:25 -04:00 |
|