ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,514 Commits 17 Branches 77 Tags
f77fd5ced0042bcccdb68f387c725e3ee142589c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Bimba ShresthaandGitHub 6b4a3e019f Merge pull request #2088 from bimbashrestha/bug 
[bug] Making compressStream2 fail when passing rawContent but claiming fullDict
 2020-04-23 14:16:56 -05:00
Bimba ShresthaandGitHub 861a73744d Merge pull request #2086 from bimbashrestha/quick3 
[bug] passing cParams.hashLog instead of comprParams->hashLog for ldm trigger
 2020-04-23 13:02:57 -05:00
Bimba Shrestha 1875f616ce passing dictContentType instead of rawContent every time 2020-04-21 22:29:35 -07:00
Bimba Shrestha f7a7409a49 adding fail test when passing wrong fullDict using refPrefix 2020-04-21 22:26:48 -07:00
Bimba Shrestha dba02245bf bash to shell conversion 2020-04-21 20:31:11 -07:00
Bimba Shrestha 659ff85cf5 changing to cParams.hashLog 2020-04-21 21:12:50 -05:00
Bimba Shrestha 0b107188b8 adding test for long mode trigger 2020-04-21 21:09:49 -05:00
Bimba ShresthaandGitHub 38a6d2a8f2 Merge pull request #2085 from bimbashrestha/quick2 
[doc-bug] Updating the recommended max chainlog for patch-from
 2020-04-20 14:09:23 -05:00
Bimba Shrestha b640802ff8 updating chainlog bound 2020-04-20 11:34:24 -07:00
Bimba ShresthaandGitHub c6311584aa Merge pull request #2083 from bimbashrestha/quick1 
[nit] Adding missing newline at end of patch-from recommendations
 2020-04-20 13:24:34 -05:00
Bimba Shrestha 587a20a12a adding newline 2020-04-20 10:25:58 -07:00
Bimba ShresthaandGitHub 5b0a452cac Adding --long support for --patch-from (#1959) 
* adding long support for patch-from

* adding refPrefix to dictionary_decompress

* adding refPrefix to dictionary_loader

* conversion nit

* triggering log mode on chainLog < fileLog and removing old threshold

* adding refPrefix to dictionary_round_trip

* adding docs

* adding enableldm + forceWindow test for dict

* separate patch-from logic into FIO_adjustParamsForPatchFromMode

* moving memLimit adjustment to outside ifdefs (need for decomp)

* removing refPrefix gate on dictionary_round_trip

* rebase on top of dev refPrefix change

* making sure refPrefx + ldm is < 1% of srcSize

* combining notes for patch-from

* moving memlimit logic inside fileio.c

* adding display for optimal parser and long mode trigger

* conversion nit

* fuzzer found heap-overflow fix

* another conversion nit

* moving FIO_adjustMemLimitForPatchFromMode outside ifndef

* making params immutable

* moving memLimit update before createDictBuffer call

* making maxSrcSize unsigned long long

* making dictSize and maxSrcSize params unsigned long long

* error on files larger than 4gb

* extend refPrefix test to include round trip

* conversion to size_t

* making sure ldm is at least 10x better

* removing break

* including zstd_compress_internal and removing redundant macros

* exposing ZSTD_cycleLog()

* using cycleLog instead of chainLog

* add some more docs about user optimizations

* formatting
 2020-04-17 15:58:53 -05:00
Bimba ShresthaandGitHub c0d4b2b5a3 Merge pull request #2075 from bimbashrestha/dict_fuzzer_ref 
[bug] handling case where prefix is NULL or 0 sized in refPrefix_advanced
 2020-04-07 17:37:19 -05:00
Bimba Shrestha 1658ae75cd handling nil case for refprefix 2020-04-07 14:41:53 -07:00
Bimba Shrestha 794f03459e adding refPrefix 2020-04-06 22:57:49 -07:00
Bimba ShresthaandGitHub 2847d1a606 Merge pull request #2069 from bimbashrestha/doversize 
[memory opt] Downsize decompression buffers when large for too long
 2020-04-06 16:06:37 -05:00
Bimba Shrestha 31e76f1ed4 adding test for dctx size reduction 2020-04-04 08:49:24 -07:00
Bimba ShresthaandGitHub ebc422b0b7 Merge pull request #2068 from bimbashrestha/quick 
[bug] Fixing Incorrect/inconsistent reported compression percentage for empty files
 2020-04-03 19:28:42 -05:00
Bimba Shrestha 0154866749 moving consts to zstd_internal and reusing them 2020-04-03 14:26:15 -07:00
Bimba Shrestha 0a172c5e43 converting to if 2020-04-03 14:21:24 -07:00
Bimba Shrestha 3a4c8cc9b3 adding dctx to function name 2020-04-03 14:14:46 -07:00
Bimba Shrestha ae47d50355 only computing sizes once 2020-04-03 14:12:23 -07:00
Bimba Shrestha a4cbe79ccb Using in and out size together 2020-04-03 14:09:21 -07:00
Bimba ShresthaandGitHub 009a92f749 Merge pull request #2067 from bimbashrestha/res-leak 
[bug] adding fclose before return
 2020-04-03 15:51:59 -05:00
Bimba Shrestha 1d267dc5d6 returning on null check 2020-04-03 13:39:02 -07:00
Bimba Shrestha d598c88fb3 adding fclose before return 2020-04-03 13:31:47 -07:00
Bimba Shrestha 936aa63ff1 adding oversized check on decompression 2020-04-03 13:25:32 -07:00
Bimba Shrestha 05574ec141 adding oversizeDuration to dctx and macros 2020-04-03 13:08:29 -07:00
Bimba Shrestha d0412f3aba no percentage on readsize == 0 2020-04-03 12:10:02 -07:00
Bimba Shrestha bf1856c26f removing max(1, ..) 2020-04-03 12:07:17 -07:00
Bimba ShresthaandGitHub 40414f984a Merge pull request #2036 from bimbashrestha/edist 
[contrib] Edit distance match finder
 2020-03-23 12:53:44 -05:00
Bimba ShresthaandGitHub 91a62bad47 Merge pull request #2047 from bimbashrestha/doc 
[doc] targetLen -> targetLength
 2020-03-23 08:27:54 -05:00
Bimba Shrestha 22e7123a9a targetLen -> targetLength 2020-03-23 06:25:08 -07:00
Bimba Shrestha dbdeca1412 Edit distance match finder 2020-03-13 09:39:33 -05:00
Bimba ShresthaandGitHub 66607d0eac Merge pull request #2033 from bimbashrestha/icc 
[opt] Small icc level 1 compression speed gain using #pragma vector
 2020-03-10 20:42:19 -05:00
Bimba ShresthaandGitHub 5c05d419e8 Merge pull request #2031 from bimbashrestha/bitscan 
_BitScan{Reverse,Forward} add check for undefined
 2020-03-10 16:22:25 -05:00
Bimba Shrestha a89c45bdbd Typo 2020-03-10 15:19:48 -05:00
Bimba Shrestha 43fc88f443 Adding comment and remvoing ivdep 2020-03-10 14:57:27 -05:00
Bimba ShresthaandGitHub 10f915f041 Merge pull request #2032 from bimbashrestha/no-size 
Adding --[no-]content-size to the cli
 2020-03-09 18:14:05 -05:00
Bimba Shrestha 9388dac2c0 Setting ZSTD_c_contentSizeFlag instead of UNKNOWN filesize 2020-03-09 15:40:18 -05:00
Bimba Shrestha 84d6bd4c7f Adding documentation 2020-03-09 14:50:49 -05:00
Bimba Shrestha fae64b3390 Adding test for --[no-]content-size 2020-03-09 14:44:38 -05:00
Bimba Shrestha 43874730f4 Adding --content-size 2020-03-09 14:19:05 -05:00
Bimba Shrestha 6d8e761ba3 Inverting content size prefs bool 2020-03-09 14:16:10 -05:00
Bimba Shrestha 03ee18d9c5 Adding (void)noContentSize 2020-03-09 14:00:52 -05:00
Bimba Shrestha 167244a64f --no-content-size cli 2020-03-09 13:07:29 -05:00
Bimba Shrestha dba3abc95a Missed returns 2020-03-05 12:20:59 -08:00
Bimba Shrestha a75e5f2ffc bitscan add undef check 2020-03-05 11:52:15 -08:00
Bimba ShresthaandGitHub bc0e10ae78 Merge pull request #2030 from bimbashrestha/quick 
Removing no-tree-vectorize for intel
 2020-03-05 10:41:06 -08:00
Bimba Shrestha 85d0efd619 Removing no-tree-vectorize for intel 2020-03-05 10:02:48 -08:00
Bimba Shrestha 4c72a1a9c2 adding vector to main loop 2020-03-05 09:55:38 -08:00
Bimba ShresthaandGitHub 11c1e5c9bc Merge pull request #2027 from bimbashrestha/travis-icc 
[ci] Adding icc make check to travis
 2020-03-03 17:01:47 -08:00
Bimba ShresthaandGitHub 9405af6fa0 Merge pull request #2026 from bimbashrestha/issues-temp 
Add issue templates to zstd
 2020-03-03 15:52:41 -08:00
Bimba Shrestha c7060bb711 Removing mmx compiler flags 2020-03-03 15:46:11 -08:00
Bimba Shrestha 1bc56dfa4c Adding icc make check to travis ci 2020-03-03 15:22:35 -08:00
Bimba ShresthaandGitHub fa9c0a4d78 Merge pull request #2025 from bimbashrestha/doc 
[doc] Fixing ZSTD_c_compressionLevel confusing note
 2020-03-03 13:43:29 -08:00
Bimba Shrestha cba46e9b7b Fixing ZSTD_c_compressionLevel confusing note 2020-03-03 13:12:02 -08:00
Bimba ShresthaandGitHub 8974906129 Merge pull request #2019 from bimbashrestha/quick 
--show-default-cparams Adding hashLog
 2020-02-27 14:15:34 -08:00
Bimba Shrestha 23165b387e Adding hashLog to --show-default-cparams 2020-02-27 13:56:56 -08:00
Bimba ShresthaandGitHub 0a00397e19 [cirrus-ci] Removing pkg -y update and using 11.3-snap instead of 11.2 (#2018) 
* Replacing with boilerplate from cirrus

* Adding task name and adding back freebsd 11.x

* Changing from 11.3 to freebsd-11-3-snap

* Rearanging the order of matrix and instance
 2020-02-27 13:53:03 -08:00
Bimba ShresthaandGitHub c9ce6db069 Merge pull request #1999 from bket/fix_have_lz4 
Fix forgotten portable header prefix
 2020-02-20 10:55:28 -08:00
Bimba ShresthaandGitHub b7b9d3fd34 Merge pull request #2002 from bimbashrestha/docs 
[doc] Adding notes about CI and an outline of collaboration workflow
 2020-02-20 10:54:04 -08:00
Bimba Shrestha 8f8c53d995 Remove redundant section and typo 2020-02-18 14:21:19 -08:00
Bimba Shrestha 042b28bd2c Outlining our current workflow 2020-02-18 13:03:52 -08:00
Bimba Shrestha 76c52b6b7c Adding steps for setting up travis and appveyor (and some general notes) 2020-02-16 11:31:17 -05:00
Bimba ShresthaandGitHub 06a57cf57e [build-issue] More portable header prefix usage (#) (#1987) 
* make 4.3 build issue fix

* Changing header name and adding comment
 2020-02-06 14:10:51 -08:00
Bimba ShresthaandGitHub f627e1a1ae Displaying level in --show-default-cparams (#1991) 
* Displaying level in --show-default-cparams

* Displaying actual enum value instead of level
 2020-02-06 13:51:43 -08:00
Bimba ShresthaandGitHub 6a4258a08a Removing symbols already in unit tests and adding some new unit tests for missing symbols (#1985) 
* Removing symbols that are not being tested

* Removing symbols used in zstdcli, fileio, dibio and benchzstd

* Removing symbols used in zbuff and add test-zbuff to travis

* Removing remaining symbols and adding unit tests instead

* Removing symbols test entirely
 2020-02-05 16:55:00 -08:00
Bimba ShresthaandGitHub 0cbafe3ce6 Merge pull request #1982 from bimbashrestha/quick 
Using appliedParams instead of "supplied" params in compressBegin()
 2020-02-03 16:48:21 -08:00
Bimba ShresthaandGitHub 075783c1f0 Merge pull request #1980 from bimbashrestha/params 
Adding --show-default-cparams to cli
 2020-02-03 14:43:55 -08:00
Bimba ShresthaandGitHub f1de05f348 Merge pull request #1981 from bimbashrestha/bug-fix 
Benchzstd multiple file decompression enable
 2020-02-03 14:40:26 -08:00
Bimba Shrestha d1389ac3ff Adding assert(NB_STRATEGIES == upperLimit) 2020-02-03 09:52:39 -08:00
Bimba Shrestha 80c26117a9 Line-wrapping 2020-02-03 09:38:16 -08:00
Bimba Shrestha ee8a712af3 Using appliedParams instead of supplied params 2020-01-31 15:49:07 -08:00
Bimba Shrestha f33baa21c6 Removing assert and changing ratio cSize 2020-01-31 11:54:14 -08:00
Bimba Shrestha 19473390d1 Adding (void)ZSTD_strategyMap 2020-01-31 10:54:02 -08:00
Bimba Shrestha 38d010eea0 Adding static assert and using it in cli 2020-01-31 10:47:17 -08:00
Bimba Shrestha 0fa6c90e89 Moving to same category as -v 2020-01-31 10:38:19 -08:00
Bimba Shrestha f4ae6c0b94 Moving documentation and elaborating 2020-01-30 17:54:42 -08:00
Bimba Shrestha 9d9cfa9ef9 Adding assert and using ZSTD_STRATEGY_MAX as ZSTD_NB_STRATEGIES 2020-01-30 17:54:01 -08:00
Bimba Shrestha 6ad2f0d7d6 Adding back one cast and using UTIL_FILESIZE_UNKNOWN to check unknown file size 2020-01-30 17:16:02 -08:00
Bimba Shrestha 6bf05a53ab Documenting in the man file 2020-01-30 17:11:38 -08:00
Bimba Shrestha c203341292 Removing more casts 2020-01-30 17:07:58 -08:00
Bimba Shrestha 2a0771cfdb Removing unnecessary cast 2020-01-30 17:05:33 -08:00
Bimba Shrestha b0ceab4dc0 Moving default cparams back, checking for unknown file size, using getCParams() 2020-01-30 16:52:07 -08:00
Bimba Shrestha b660ef87f8 Convertion nit U32 -> size_t 2020-01-30 14:29:04 -08:00
Bimba Shrestha 71754a26f8 Update comment 2020-01-30 14:14:27 -08:00
Bimba Shrestha 2f10019b92 Adding --show-default-cparams (show cparams before compressing 2020-01-30 14:12:03 -08:00
Bimba ShresthaandGitHub 3b10060b71 Merge pull request #1978 from bimbashrestha/quick 
[Typo][fix] make benchmarking typo fix
 2020-01-28 22:11:51 -08:00
Bimba Shrestha 9b049836c9 Typo baseline_build -> baseline_label 2020-01-28 21:39:20 -08:00
Bimba ShresthaandYann Collet 8fe562a770 [automated_benchmarking] Make arguments optional and add --dict argument (#1968) 
* Make arugments optional and add --dict argument

* Removing accidental print statement

* Change to more likely scenario for dictionary compression benchmark
 2020-01-28 11:29:43 -08:00
Bimba ShresthaandYann Collet f25a6e9f8f Adding new cli endpoint --patch-from= (#1940) 
* Adding new cli endpoint --diff-from=

* Appveyor conversion nit

* Using bool set trick instead of direct set

* Removing --diff-from and only leaving --diff-from=#

* Throwing error when both dictFileName vars are set

* Clean up syntax

* Renaming diff-from to patch-from

* Revering comma separated syntax clean up

* Updating playtests with patch-from

* Uncommenting accidentally commented

* Updating remaining docs and var names to be patch-from instead of diff-from

* Constifying

* Using existing log2 function and removing newly created one

* Argument order (moving prefs to end)

* Using comma separated syntax

* Moving to outside #ifndef
 2020-01-10 14:25:24 -08:00
Bimba ShresthaandYann Collet eb76f786bc [bench] Automated benchmarking script (#1906) 
* Initial revised automated benchmarking script

* Updating nb_iterations and making loop infinite

* Allowing benchmarking params to be changed from cli

* Renaming old speed test

* Removing numpy dependency for cli

* Change filename and benchmakr on pr level

* Moving build outside loop and adding iterations param

* Moving benchmarking to seperate travis ci test

* Fixing typo and using unused variable

* Added mode labels and updated README accordingly

* Adding new mode 'current' that compraes facebook:dev against current hash

* Typo

* Reverting previous accidental diff

* Typo

* Adding frequency config variable to prevent github from blacklisting

* Added new argument for frequency of fetching new prs

* Updating documentation
 2020-01-06 14:19:11 -08:00
Bimba ShresthaandYann Collet b1f53b1a10 [fuzz] Dividing by targetCBlockSize instead of blockSize for nbBlocks fit (#1936) 
* Adding fail logging for superblock flow

* Dividing by targetCBlockSize instead of blockSize

* Adding new const and using more acurate formula for nbBlocks

* Only do dstCapacity check if using superblock

* Remvoing disabling logic

* Updating test to make it catch more extreme case of previou bug

* Also updating comment

* Only taking compressEnd shortcut on non-superblock
 2020-01-03 16:53:51 -08:00
Bimba ShresthaandYann Collet ef1684f29a [doc] Adding -M# decompression operation modifier documentation (#1938) 
* Adding -M# operation modifier documentation

* Adding long name, typo and update desc
 2020-01-03 16:36:47 -08:00
Bimba Shrestha 56415efc76 Constifying, malloc check and naming nit 2019-12-17 17:16:51 -08:00
Bimba Shrestha 989ce13e19 One more type conversion 2019-12-13 16:50:21 -08:00
Bimba Shrestha 4399eed42e Adding explict cast to satisfy appveyor ci 2019-12-13 16:38:11 -08:00
Bimba Shrestha db5124ef6e More void* issues. Just replacing with BYTE* 2019-12-13 16:24:49 -08:00
Bimba Shrestha 49b2bf7106 'void* size issue' fix 2019-12-13 16:06:57 -08:00