ddidderr/zstd-rs
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,314 Commits 17 Branches 77 Tags
9fe50ed6239f86d569030744f6545feb89a6d933
Commit Graph
558 Commits
Author SHA1 Message Date
Yann Collet 9fe50ed623 fixed maximum windowLog for zstreamtest --newapi 
for compatibility with low memory VM
 2017-09-28 01:42:06 -07:00
Yann Collet 9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize 
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
 2017-09-28 01:25:40 -07:00
Yann Collet aa800c4793 reduced memory usage of zstreamtest --newapi 
to run on memory-constrained VM
 2017-09-27 18:00:15 -07:00
Yann Collet bc32b40b98 reduced zstreamtest --mt memory load 
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
 2017-09-27 17:27:38 -07:00
Yann Collet 02502191e5 separated --long tests between short and long tests 
A fast subset of these tests is now played in short test mode
 2017-09-27 15:48:06 -07:00
Yann Collet f9de54acfb reduced memory requirements for --long tests 
in new --long test section
 2017-09-27 15:38:27 -07:00
Yann Collet bdd0f6f046 improved make clean in tests/fuzz 2017-09-27 15:20:08 -07:00
Yann Collet 60059df051 shorter make test to avoid time out on travis CI 
Timed tests (fuzzer) are reduced
Long tests are shortened (less data generated)
 2017-09-27 15:16:27 -07:00
Yann Collet 91ffe71ec8 Merge pull request #864 from terrelln/long 
Increase maximum window size
 2017-09-26 16:26:29 -07:00
Nick Terrell 471aa385b3 [fuzz] Speed up round trip tests 
* Enforce smaller maximum values for parameters
* Adjust parameters to the source size

The memory usage is reduced by about 5x, which makes the fuzzers run at
least twice as fast, even more so with ASAN/MSAN enabled.
 2017-09-26 14:03:43 -07:00
Nick Terrell c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Yann Collet 8a4d0abc1c Merge pull request #861 from terrelln/zip 
[fuzz][CI] Set up regression tests
 2017-09-25 17:11:35 -07:00
Yann Collet 5d8fdd1641 Merge pull request #855 from terrelln/maxoff 
[libzstd] Increase MaxOff
 2017-09-25 16:34:29 -07:00
Nick Terrell 917a213254 [fuzz] Determine flags based on compiler version 2017-09-25 15:32:36 -07:00
Nick Terrell 11e21f23cb [fuzz] Mention the corpora in the README 2017-09-25 15:31:38 -07:00
Nick Terrell 6bb781e0f1 [fuzz] Add regressiontest targets 2017-09-25 15:31:33 -07:00
Nick Terrell bbe77212ef [libzstd] Increase MaxOff 2017-09-25 13:36:18 -07:00
Nick Terrell bfad5568b5 [fuzz] Make simple_round_trip compile cleanly 2017-09-25 13:28:45 -07:00
Nick Terrell 23199b6daf [fuzz] Fix fuzz.py env flags parsing 2017-09-25 13:28:18 -07:00
Nick Terrell 1c23b64049 [fuzz] fuzz.py can minimize and zip corpora 
* "minimize" minimizes the corpora into an output directory.
* "zip" zips up the minimized corpora, which are ready to deploy.
 2017-09-25 12:04:12 -07:00
Yann Collet cd3115b284 added control from frame content size at end of decompression 
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
 2017-09-21 16:21:10 -07:00
Nick Terrell 1fe762e236 [zstdcli] Fix LDM advanced options parsing 2017-09-18 14:49:35 -07:00
Yann Collet 31829cb057 Merge pull request #847 from terrelln/fuzzer 
[fuzzer] Fuzz long range matching & new API
 2017-09-15 12:09:00 -07:00
Nick Terrell 39357c41cb [fuzzer] Fuzz long range matching & new API 2017-09-14 14:48:08 -07:00
Yann Collet 218c09e5b3 Merge pull request #844 from terrelln/fuzzer 
Fuzzer
 2017-09-14 11:40:25 -07:00
Nick Terrell 9712d5ebe6 [fuzzer] Fix bugs in fuzz.py 2017-09-13 19:08:35 -07:00
Nick Terrell a6f08b4783 [fuzzer] Fix FUZZ_seed() 2017-09-13 18:41:32 -07:00
Nick Terrell 6c6412cef9 [fuzzer] Update README.md 2017-09-13 18:23:52 -07:00
Nick Terrell 6b8236cf7e [fuzz] Add fuzzing helper script 2017-09-13 17:45:21 -07:00
Nick Terrell b7e1522330 Add block fuzzers 2017-09-13 17:44:41 -07:00
Nick Terrell def3214d74 [fuzzer] Handle single empty directory 2017-09-13 17:44:30 -07:00
Yann Collet 739b620814 Merge pull request #842 from stellamplau/decodeCorpus-maxSize 
Add flag to limit max decompressed size in decodeCorpus
 2017-09-13 17:26:55 -07:00
Nick Terrell 8b6c80ada8 Update fuzzer Makefile 2017-09-13 16:16:57 -07:00
Nick Terrell 677c2cbf89 Update fuzzer sources 2017-09-13 16:16:57 -07:00
Stella Lau 963558a072 Fix implicit conversion error 2017-09-13 16:01:16 -07:00
Stella Lau 40bf0ced7d Add flag to limit max decompressed size in decodeCorpus 2017-09-13 15:16:56 -07:00
Yann Collet f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev 
Add long distance matcher
 2017-09-13 13:24:08 -07:00
Yann Collet be1f2dac5b Merge pull request #841 from facebook/utilTimeAPI 
modified util::time API (T19505791)
 2017-09-13 11:41:01 -07:00
Yann Collet a1bc08834f Merge pull request #840 from stellamplau/decodeCorpus-blocks 
Make decodecorpus generate raw compressed blocks
 2017-09-13 09:34:04 -07:00
Yann Collet c95c0c9725 modified util::time API 
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
 2017-09-12 18:12:46 -07:00
Stella Lau e89065506e Make decodecorpus generate raw compressed blocks 2017-09-12 17:18:45 -07:00
Stella Lau 3d8e313f64 Reduce ldm hash table size in test 2017-09-11 17:21:28 -07:00
Stella Lau eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Yann Collet b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream() 
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
 2017-09-09 14:37:28 -07:00
Yann Collet 058ed2ad33 ZSTD_decodingBufferSize_min() 
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
 2017-09-09 01:03:29 -07:00
Yann Collet 3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Stella Lau eeff55dfa8 Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev 2017-09-06 15:56:32 -07:00
Stella Lau 9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
Stella Lau af4068a697 Fix function name in tests/fuzz/regression_driver 2017-09-05 22:14:41 -07:00
Stella Lau 67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00