ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,540 Commits 17 Branches 77 Tags
d2c48042e6d44070ae5559cb2dfacec5ea791fc0
Commit Graph
6540 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
W. Felix Handte d2c48042e6 Switch Macro Guarding lstat() 2019-06-07 15:32:28 -04:00
W. Felix Handte b878bd08dd Clean Up Temp Files Produced By playTests.sh 2019-06-07 15:32:28 -04:00
Nick TerrellandGitHub d06c15cc89 Merge pull request #1635 from terrelln/opt-opt 
[libzstd] Optimize ZSTD_insertBt1() for repetitive data
 2019-06-06 13:15:23 -07:00
Nick Terrell f3800ba073 [regression] Update results.csv 2019-06-05 20:40:52 -07:00
Nick TerrellandNick Terrell cdb9481e38 [libzstd] Optimize ZSTD_insertBt1() for repetitive data 
We would only skip at most 192 bytes at a time before this diff.
This was added to optimize long matches and skip the middle of the
match. However, it doesn't handle the case of repetitive data.

This patch keeps the optimization, but also handles repetitive data
by taking the max of the two return values.

```
> for n in $(seq 9); do echo strategy=$n; dd status=none if=/dev/zero bs=1024k count=1000 | command time -f %U ./zstd --zstd=strategy=$n >/dev/null; done
strategy=1
0.27
strategy=2
0.23
strategy=3
0.27
strategy=4
0.43
strategy=5
0.56
strategy=6
0.43
strategy=7
0.34
strategy=8
0.34
strategy=9
0.35
```

At level 19 with multithreading the compressed size of `silesia.tar` regresses 300 bytes, and `enwik8` regresses 100 bytes.
In single threaded mode `enwik8` is also within 100 bytes, and I didn't test `silesia.tar`.

Fixes Issue #1634.
 2019-06-05 20:34:00 -07:00
Yann ColletandGitHub c0eb6c9c81 Merge pull request #1632 from ephiepark/dev 
CircleCI : use custom Docker Image with pre-installed dependencies
 2019-06-05 12:30:21 -07:00
Felix HandteandGitHub 015985c1f9 Merge pull request #1633 from yk-tanigawa/patch-1 
Update README.md
 2019-06-05 14:55:19 -04:00
Yosuke TanigawaandGitHub f46a3603c9 Update README.md 
Fix minor typo in the documentation.
 2019-06-05 09:11:21 -07:00
Ephraim Park 286394aa9b Merge branch 'dev' of https://github.com/ephiepark/zstd into dev 2019-06-04 21:14:20 -07:00
Ephraim Park c96535f67e circleci : remove install dependencies command from config.yml 2019-06-04 21:13:40 -07:00
Ephraim Park 3e9572bfa9 circleci : use custom docker image with installations 2019-06-04 21:13:40 -07:00
Ephraim ParkandGitHub f1bf281715 Merge pull request #2 from ephiepark/circleci 
CircleCI : use custom Docker Image with pre-installed dependencies
 2019-06-04 20:53:29 -07:00
Yann ColletandGitHub c6ca65ab78 Merge pull request #1631 from ephiepark/dev 
zstdcli : expose cpu load indicator for each file on -vv mode
 2019-06-04 15:14:35 -07:00
Yann ColletandGitHub 61d912f3ff Merge pull request #1629 from facebook/streamSize 
Added comments on I/O buffer sizes for streaming
 2019-06-04 12:38:43 -07:00
Ephraim Park a6a7f08eaf circleci : remove install dependencies command from config.yml 2019-06-04 12:18:33 -07:00
Ephraim Park a371df534d circleci : use custom docker image with installations 2019-06-04 11:05:42 -07:00
Yann Collet b3af1873a0 better title formatting for html documentation 
must pay attention to /** and /*! patterns.
 2019-06-04 10:35:40 -07:00
Yann Collet b5c98fbfd0 Added comments on I/O buffer sizes for streaming 
It seems this is still a confusing topic,
as in https://github.com/klauspost/compress/issues/109 .
 2019-06-04 10:26:16 -07:00
Ephraim Park e498bb64b6 zstdcli : align output message with previous message 2019-06-04 09:42:18 -07:00
Ephraim Park a38601f05a zstdcli : remove extra semicolon 2019-06-04 09:25:16 -07:00
Ephraim Park ef2c836955 zstdcli : trying to fix declaration after statement 2019-06-04 09:23:49 -07:00
Ephraim Park 5fe974261b zstdcli : moving cpu load calculation from FIO_compressFilename_dstFile to FIO_compressFilename_internal 2019-06-04 09:04:35 -07:00
Ephraim Park 2841c79e87 zstdcli : fixing mixed declarations and code error 2019-06-03 17:36:09 -07:00
Ephraim Park 43d463eccb zstdcli : moving start time and cpu initialization after potential prompt 2019-06-03 17:31:14 -07:00
Ephraim Park 5029ebce2f zstdcli : exposing cpu load indicator for each file on -vv mode 2019-06-03 14:38:40 -07:00
Yann ColletandGitHub b13a9207f9 Merge pull request #1623 from facebook/fullbench 
fullbench minor improvements
 2019-05-31 14:40:19 -07:00
Yann Collet 78056a3e9c consistency : changed constant name to kSampleSizeDefault 
as suggested by @terrelln.

added <assert.h> include
 2019-05-30 10:13:56 -07:00
Yann Collet ab3625cffd define CONTROL, to ensure check cannot be disabled 
assert() can be disabled with NDEBUG
 2019-05-29 16:18:22 -07:00
Yann Collet 904d4da239 fullbench : minor refactoring, for readability 2019-05-29 16:08:49 -07:00
Yann Collet ed38b645db fullbench: pass proper parameters in scenario 43 2019-05-29 15:26:06 -07:00
Yann ColletandGitHub b84274da0f Merge pull request #1622 from facebook/nextToUpdate3 
nextToUpdate3 does not need to be maintained outside of zstd_opt.c
 2019-05-28 18:26:21 -07:00
Yann Collet c63081623f fullbench 43: ensure context is freed after each usage 2019-05-28 17:27:52 -07:00
Yann Collet eb6b199038 fullbench: added streaming with fresh CCtx scenario 
worst case situation, where context must be recreated every time
but without knowledge of the input size
thus sizing the context for some large input.
 2019-05-28 16:50:49 -07:00
Yann Collet 4c4149452c fullbench can now select sample size with command -B# 2019-05-28 16:37:03 -07:00
Yann Collet 9719fd616c removed nextToUpdate3 from ZSTD_window 
it's now a local variable of ZSTD_compressBlock_opt()
 2019-05-28 16:18:12 -07:00
Yann Collet 33dabc8c80 get bt matches : made it a bit clearer which parameters are input and output 2019-05-28 16:11:32 -07:00
Yann Collet 327cf6fac1 nextToUpdate3 does not need to be maintained outside of zstd_opt.c 
It's re-synchronized with nextToUpdate at beginning of each block.
It only needs to be tracked from within zstd_opt block parser.

Made the logic clear, so that no code tried to maintain this variable.

An even better solution would be to make nextToUpdate3
an internal variable of ZSTD_compressBlock_opt_generic().
That would make it possible to remove it from ZSTD_matchState_t,
thus restricting its visibility to only where it's actually useful.

This would require deeper changes though,
since the matchState is the natural structure to transport parameters into and inside the parser.
 2019-05-28 15:26:52 -07:00
Yann Collet 6453f8158f complementary code comments 
on variables used / impacted during maxDist check
 2019-05-28 14:12:16 -07:00
Yann Collet 4baecdf72a added comments to better understand enforceMaxDist() 2019-05-28 13:15:48 -07:00
Felix HandteandGitHub 21bb78e908 Merge pull request #1618 from felixhandte/zstgrep-f-flag 
Handle `-f` Flag in `zstgrep`
 2019-05-24 20:59:11 -04:00
W. Felix Handte 6a0638048a Add Test 2019-05-24 17:21:44 -04:00
W. Felix Handte 61025d5b7d zstdgrep: Handle -f Flag 2019-05-24 16:55:59 -04:00
W. Felix Handte ff0be17cf7 Build Manual 2019-05-24 16:55:43 -04:00
Tyler-TranandNick Terrell cb47871a0a [dictBuilder] Be more specific than ERROR(generic) (#1616) 
* Specify errors at a finer granularity than `ERROR(generic)`.
* Add tests for bad parameters in the dictionary builder.
 2019-05-22 18:57:50 -07:00
Nick TerrellandGitHub b114477f2a Merge pull request #1613 from neheb/patch-1 
fileio: Disable BACKTRACE under uClibc
 2019-05-06 17:12:03 -07:00
Rosen PenevandGitHub 23b5ee1236 fileio: Disable BACKTRACE under uClibc 
uClibc does not support BACKTRACE. It also defines __GLIBC__ and minor as both 2.
This fixes compilation under such a setup.
 2019-05-06 15:02:47 -07:00
Nick TerrellandGitHub bfbef51afb Merge pull request #1610 from terrelln/pool-test 
[test] Fix poolTests deadlock and flakiness
 2019-05-02 15:13:50 -07:00
Nick Terrell 81f1f76479 [tests] Only sleep 10ms in poolTests 2019-05-02 14:35:25 -07:00
Nick Terrell 2953d45a0f [test] Fix poolTests deadlock and flakiness 2019-05-02 13:23:41 -07:00
lzutaoandNick Terrell 1d70bc3ba9 meson: Enable testing in CI (#1609) 
* tests: Fix shellcheck warnings in playTests.sh

* tests: Do not use ../programs which is relative to tests dirs

This commit fixes error when running playTests.sh in Meson.
Mesonbuild runs out of tree, so ./datagen not in `zstd/tests` dir,
it lies in <mesonbuilddir>/tests. This leads to ../programs invalid.

* tests: Replace relative paths for zstd/tests dir

* playTests: Set shell options explicitly, not in shebang

* playTests: Replace echo -e with printf

* meson: Fix test-zstd

Use std=gnu99 to build and test just like `make test`.

* meson: Fix legacy test

* meson: Enable testing in CI

Run build under release mode for faster test time.

* meson: Increase timeout time for test-zstream
 2019-05-02 12:35:37 -07:00