ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,845 Commits 17 Branches 77 Tags
f0937b83d9a32cb2b59f99bbc4db717ae6e83c9b
Commit Graph
97 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick Terrell e32e3e8662 Improve wildcopy performance across the board 2020-01-28 20:37:04 -08:00
Nick Terrell 009f388457 Fix playTests.sh for 32-bit mode 2020-01-17 14:20:44 -08:00
Nick Terrell fa48af545f Add --single-thread to zstd --help 2020-01-17 13:54:49 -08:00
Nick Terrell a11a9271d6 Fix lowLimit underflow in overflow correction 2020-01-17 12:10:18 -08:00
Nick Terrell f6d00c059f [util] Fix readLineFromFile on Cygwin 2020-01-13 14:37:22 -08:00
Nick Terrell 659e9f05cf Fix null pointer addition 2019-11-20 18:36:04 -08:00
Nick Terrell ac992b35a2 [cmake][tests] Add zstreamtest executable 2019-11-20 17:26:06 -08:00
Nick Terrell e68db76b4b Update .gitignore 2019-11-20 16:36:40 -08:00
Nick Terrell b1ec94e63c Fix ZSTD_f_zstd1_magicless for small data 
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
  take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
  taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.

Fixes #1813.
 2019-10-21 21:16:17 -07:00
Nick Terrell 0bc39bc3a0 [zstdmt] Don't memset the jobDescription 2019-10-18 15:05:51 -07:00
Nick Terrell 243824551f [threading] Add debug utilities 2019-10-18 15:05:34 -07:00
Nick Terrell 8c11f089a1 [fuzz] Increase output buffer size of stream_round_trip 
Fixes OSS-Fuzz crash.
Credit to OSS-Fuzz
 2019-10-18 13:39:08 -07:00
Nick Terrell 5cb7615f1f Add UNUSED_ATTR to ZSTD_storeSeq() 2019-09-20 21:37:13 -07:00
Nick Terrell 44c65da97e Remove literals overread in ZSTD_storeSeq() for ~neutral perf 2019-09-20 12:23:25 -07:00
Nick Terrell fde217df04 Fix bounds check in ZSTD_storeSeq() 2019-09-20 08:25:12 -07:00
Nick Terrell e068bd01df [tests] Fix decodecorpus 2019-09-20 01:09:47 -07:00
Nick Terrell ddab2a94e8 Pass iend into ZSTD_storeSeq() to allow ZSTD_wildcopy() 2019-09-20 00:56:20 -07:00
Nick Terrell cdad7fa512 Widen ZSTD_wildcopy to 32 bytes 2019-09-20 00:52:15 -07:00
Nick Terrell fbeaf6989e [libzstd] Improve advanced API docs 2019-09-15 12:41:24 -07:00
Nick Terrell f3800ba073 [regression] Update results.csv 2019-06-05 20:40:52 -07:00
Nick TerrellandGitHub 585b5a1039 Merge pull request #1600 from myzhang1029/long-tell 
Fix #1591 - Not building on MinGW-W64
 2019-04-24 21:05:25 -07:00
Nick TerrellandGitHub 3d673f3275 Merge pull request #1598 from terrelln/decode-seq 
[libzstd] Error if all sequence bits aren't consumed
 2019-04-23 15:44:20 -07:00
Nick TerrellandGitHub b758250eb1 Merge pull request #1595 from terrelln/legacy-fix 
[legacy] Fix ZSTDv0*_decodeSequence()
 2019-04-19 12:23:50 -07:00
Nick TerrellandGitHub 9ad7ea44ec Merge pull request #1594 from terrelln/legacy-fix 
[legacy] Fix Huffman jump table reads in v01 and v05
 2019-04-18 17:29:32 -07:00
Nick TerrellandGitHub a8db4bd3fe Merge pull request #1593 from terrelln/legacy-fix 
[fuzzer] Run fuzzers in legacy mode and fix legacy code
 2019-04-18 16:07:42 -07:00
Nick TerrellandGitHub af3531ee35 Merge pull request #1590 from terrelln/frame-info-fuzz 
[fuzzer] Fuzz frame info functions
 2019-04-17 12:13:49 -07:00
Nick TerrellandGitHub 83b51e9f88 Merge pull request #1586 from facebook/dev 
v1.4.0
 2019-04-16 15:37:24 -07:00
Nick TerrellandGitHub b85f7d7790 Merge pull request #1589 from terrelln/appveyorTest 
[libzstd] Require ZSTD_MULTITHREAD to create a ZSTDMT_CCtx
 2019-04-16 12:32:19 -07:00
Nick TerrellandGitHub a821aee3b1 Merge pull request #1587 from terrelln/release-prep 
[playTets] Don't pass status=none to dd
 2019-04-15 16:42:26 -07:00
Nick TerrellandGitHub d86c7ad458 Merge pull request #1585 from terrelln/release-prep 
Prepare for the zstd-1.4.0 release
 2019-04-15 14:41:50 -07:00
Nick TerrellandGitHub 13fddac46d Merge pull request #1583 from lzutao/meson 
Fix tests and meson build
 2019-04-12 11:13:20 -07:00
Nick TerrellandGitHub 9563fb4bfc Merge pull request #1580 from terrelln/dict-fuzz 
[libzstd] Fix decompression dictionary bugs and clean up initialization
 2019-04-10 19:29:09 -07:00
Nick TerrellandGitHub f86d4bd1d5 Merge pull request #1576 from terrelln/dict-fuzz 
Add new fuzzers and fix exposed bugs
 2019-04-10 11:29:15 -07:00
Nick TerrellandGitHub 5f6ca3c6ce Merge pull request #1578 from orip/r-flag-typo 
Fixed `-r` typo
 2019-04-10 11:19:02 -07:00
Nick TerrellandGitHub 1a90133b15 Merge pull request #1575 from terrelln/zstdmt 
[libzstd] Remove ZSTDMT from the shared library
 2019-04-08 16:51:40 -07:00
Nick Terrell 641e594309 [libzstd] Remove ZSTDMT from the shared object 
* Remove ZSTDMT from the shared object by default.
* Provide a macro `ZSTD_LEGACY_MULTITHREADED_API` to override it.
* Document it in `lib/README.md`.
 2019-04-07 18:47:52 -07:00
Nick TerrellandGitHub d5910a5d94 Merge pull request #1574 from terrelln/examples 
Stabilize ZSTD_getDictID_*() functions and clean up examples
 2019-04-05 23:29:32 -07:00
Nick TerrellandGitHub a63aaaa2cc Merge pull request #1573 from terrelln/regression 
[regression] Update results.csv for level 1 change
 2019-04-05 11:06:35 -07:00
Nick TerrellandGitHub dbc8a59a0a Merge pull request #1569 from terrelln/stable 
Stabilize the advanced API
 2019-04-05 10:47:48 -07:00
Nick TerrellandGitHub 72a3fbc0e4 Merge pull request #1562 from terrelln/2fast 
[libzstd] Speed up single segment zstd_fast by 5%
 2019-04-03 18:08:15 -07:00
Nick TerrellandGitHub 56261001ea Merge pull request #1567 from terrelln/examples2 
[examples] Update streaming_decompression.c
 2019-04-03 11:27:49 -07:00
Nick TerrellandGitHub 882ceb86bc Merge pull request #1566 from terrelln/examples 
[examples] Update multiple_streaming_compression.c
 2019-04-02 17:13:10 -07:00
Nick TerrellandGitHub 425ce5547c Merge pull request #1563 from terrelln/dms-sep 
[libzstd] Split out zstd_fast dict match state function
 2019-03-29 16:19:21 -06:00
Nick TerrellandGitHub 6625f3b390 Merge pull request #1561 from shakeelrao/fix-typo 
Update comments in zstd.h and fileio.c
 2019-03-28 23:42:16 -06:00
Nick TerrellandGitHub dcc6c7e9ae Merge pull request #1556 from terrelln/dictbuilder 
[cover] Improvements for small or homogeneous data
 2019-03-25 15:08:32 -07:00
Nick TerrellandGitHub 440f390cba Merge pull request #1557 from terrelln/examples 
[examples] Update streaming_compression to the new API
 2019-03-25 15:07:35 -07:00
Nick TerrellandGitHub 7186a50775 Merge pull request #1559 from shakeelrao/reject-dict 
[CLI] ensure dictionary and input file are different
 2019-03-25 15:06:58 -07:00
Nick TerrellandGitHub d97605ad85 Merge pull request #1558 from nehaljwani/fix-version-soversion-libzstd 
[libzstd] Specify soversion and version corectly for CMake build
 2019-03-23 13:32:39 -07:00
Nick TerrellandGitHub 0c7668cd06 Merge pull request #1555 from terrelln/load-dict 
[lib] Allow ZSTD_CCtx_loadDictionary() to be called before parameters are set
 2019-03-21 17:52:57 -07:00
Nick TerrellandGitHub 041bc0bdb7 Merge pull request #1551 from terrelln/stream-wrap 
Refactor old stream API to completely wrap the advanced API
 2019-03-21 16:10:00 -07:00
Nick TerrellandGitHub 0dd3588acc Merge pull request #1553 from shakeelrao/legacy-bound 
Add legacy support to decompressBound
 2019-03-19 10:53:12 -07:00
Nick TerrellandGitHub f52a7d8faa Merge pull request #1547 from shakeelrao/fix-error 
Fix incorrect error code in ZSTD_errorFrameSizeInfo
 2019-03-15 10:57:49 -07:00
Nick TerrellandGitHub 45139e9fd5 Merge pull request #1550 from terrelln/cparams-cdict 
[libzstd] Allow compression parameters to be set with a cdict
 2019-03-13 19:17:22 -07:00
Nick TerrellandGitHub ea3b81f02c Merge pull request #1549 from terrelln/ref-cdict-free 
[libzstd] Free local cdict when referencing cdict
 2019-03-13 16:08:54 -07:00
Nick TerrellandGitHub 13cc7c610e Merge pull request #1543 from shakeelrao/dev 
Provide a function to estimate decompressed size.
 2019-03-04 08:00:47 -08:00
Nick TerrellandGitHub 8b7fa49590 Merge pull request #1541 from terrelln/fuzz 
[fuzz] Add --enable-fuzzer for clang fuzzing
 2019-02-28 11:25:27 -08:00
Nick TerrellandGitHub be3bd70c57 Merge pull request #1532 from terrelln/cctx-params 
[libzstd] Rename ZSTD_CCtxParam_* to ZSTD_CCtxParams_*
 2019-02-20 10:46:46 -08:00
Nick TerrellandGitHub a36d9302ce Merge pull request #1531 from terrelln/appveyorTest 
Fixes for Windows release
 2019-02-19 19:42:07 -08:00
Nick TerrellandGitHub eb3a7a3827 Merge pull request #1530 from terrelln/param-order 
Clean up parameter code
 2019-02-19 14:40:10 -08:00
Nick TerrellandGitHub 54e9412ddd Merge pull request #1525 from terrelln/enable-huf-flag 
Fix optimal parser prices with uncompressed literals
 2019-02-15 20:21:08 -05:00
Nick TerrellandGitHub 4c870b70fc Merge pull request #1522 from terrelln/enable-huf-flag 
Add ZSTD_c_literalCompressionMode flag
 2019-02-13 19:02:32 -05:00
Nick TerrellandGitHub ae7380d2af Merge pull request #1519 from terrelln/play-bsd 
[cirrus] Run playTests.sh
 2019-02-13 16:21:14 -05:00
Nick TerrellandGitHub 32a98bafdc Merge pull request #1521 from bket/lstat 
Detection of symbolic links on OpenBSD
 2019-02-11 20:38:49 -05:00
Nick TerrellandGitHub a17369c7dc Merge pull request #1518 from bket/non-portable 
'head -c BYTES' is non portable
 2019-02-11 15:43:03 -05:00
Nick TerrellandGitHub 197a5737c8 Merge pull request #1516 from terrelln/dict-doc 
[zdict] Improve documentation
 2019-02-01 19:04:05 -05:00
Nick TerrellandGitHub 5c1148a6a5 Merge pull request #1515 from thatsafunnyname/patch-1 
-Wformat-security not needed with -Wformat=2
 2019-02-01 13:41:27 -05:00
Nick TerrellandGitHub fafa322456 Merge pull request #1511 from lwhsu/cirrus-ci 
Change Cirrus CI's badge to dev branch
 2019-01-30 16:53:13 -05:00
Nick TerrellandGitHub 79463a65e9 Merge pull request #1501 from lwhsu/cirrus-ci 
Add Cirrus-CI config and build status badge for testing on FreeBSD
 2019-01-30 14:15:02 -05:00
Nick TerrellandGitHub 87204a2afd Merge pull request #1467 from terrelln/regression 
[regression] Update results.csv
 2018-12-20 18:14:13 -08:00
Nick TerrellandGitHub 3a4634f2af Merge pull request #1459 from terrelln/destroy 
[zstdcli] Refuse to overwrite input file
 2018-12-18 17:03:54 -08:00
Nick TerrellandGitHub 7f6d11e1e2 Merge pull request #1450 from terrelln/no-progress 
[zstdcli] Add --no-progress flag
 2018-12-14 11:50:59 -08:00
Nick TerrellandGitHub 75fa3f2eb7 Merge pull request #1446 from terrelln/overflow 
[libzstd] Fix infinite loop in decompression
 2018-12-13 16:21:15 -08:00
Nick TerrellandGitHub 54ca4b3c5d Merge pull request #1445 from terrelln/regression 
[regression] add more methods
 2018-12-12 11:16:54 -08:00
Nick TerrellandGitHub c631fdec9f Merge pull request #1434 from terrelln/regression 
[regression] Add dictionary support
 2018-12-03 10:39:12 -08:00
Nick TerrellandGitHub 787532b91a Merge pull request #1418 from ryandesign/cpuid-i386 
Fix i386 build failure "Junk character 13"
 2018-11-16 11:01:46 -08:00
Nick TerrellandGitHub f15f1bfefb Merge pull request #1416 from terrelln/rsync 
- Factor out LDM's hash function for reuse
- Add rsyncable mode to zstdmt and expose it via the advanced API
- Fix `-B`/`--block-size` to actually set the job size
- Add rsyncable tests to `zstreamtest` and `playTests.sh`

Tested by:

```
> cat A.100MB B.100MB C.100MB D.100MB E.100MB | zstd --rsyncable -fo src/file.zst
/*stdin*\            : 48.22%   (524288000 => 252837782 bytes, src/file.zst)
> rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
Total bytes sent: 252868779
total size is 252837782  speedup is 1.00
> echo test > test
> cat test A.100MB test B.100MB test C.100MB test D.100MB test E.100MB | zstd --rsyncable -fo src/file.zst
/*stdin*\            : 48.23%   (524288025 => 252838025 bytes, src/unicorn.tar.zst)
> rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
Total bytes sent: 4605696
total size is 252838025  speedup is 53.60
```

Close #1155.
 2018-11-14 18:43:12 -08:00
Nick TerrellandGitHub ed39c0e04b Merge pull request #1382 from jonesz/ticket-compiler-rt-fuzzer 
Fix libFuzzer location in makefile.
 2018-10-22 10:48:56 -07:00
Nick TerrellandGitHub e06f91a169 Merge pull request #1343 from terrelln/faster-ci 
[zstreamtest] Reduce memory of newapi tests
 2018-09-27 19:29:02 -07:00
Nick TerrellandGitHub f884e9dd8e Merge pull request #1321 from bket/non-portable 
'head -c BYTES' is non-portable.
 2018-09-17 14:40:27 -07:00
Nick TerrellandGitHub 8f27e8cf3d Merge pull request #1322 from azat-archive/seekable-fixes-pull 
Fixes read write past end of input buffer.
 2018-09-17 11:04:51 -07:00
Nick TerrellandGitHub d492ef4e07 Merge pull request #1306 from jennifermliu/dev 
Change default f of FASTCOVER to 20
 2018-09-04 18:03:36 -07:00
Nick TerrellandGitHub 7a02df8dbe Merge pull request #1302 from jennifermliu/splitpoint 
Always use splitPoint=1.0 for non-optimize cover and fastcover and some minor changes on comments
 2018-09-04 16:59:41 -07:00
Nick TerrellandGitHub 1420129fda Merge pull request #1258 from jennifermliu/fastCover 
Use CDict instead of CCtx in analyzeEntropy
 2018-07-31 14:31:05 -07:00
Nick TerrellandGitHub fc685d6b94 Merge pull request #1257 from jennifermliu/fastCover 
Increment frequency for every dmer occurence within same sample
 2018-07-30 16:03:18 -07:00
Nick TerrellandGitHub 92633789f6 Merge pull request #1254 from jennifermliu/fastCover 
Save segmentFreqs in ctx instead of malloc and memset in SelectSegment
 2018-07-27 18:22:52 -07:00
Nick TerrellandGitHub 3abfbee215 Merge pull request #1250 from jennifermliu/fastCover 
Add Fast Cover Dictionary Builder
 2018-07-27 15:11:57 -07:00
Nick TerrellandGitHub 7974628587 Merge pull request #1252 from GeorgeLu97/bench-quiet-display-fix 
Fixes bug
 2018-07-26 16:12:15 -07:00
Nick TerrellandGitHub 77068a8447 Merge pull request #1246 from jennifermliu/benchmark 
Benchmark dictionary builders
 2018-07-20 18:09:31 -07:00
Nick TerrellandGitHub 2e47865d7d Merge pull request #1245 from terrelln/disable-arm 
Revert "attempt to re-enable arm64 tests"
 2018-07-19 15:32:53 -07:00
Nick TerrellandGitHub 4d1ad5cdb2 Merge pull request #1238 from jennifermliu/random 
Add random dictionary builder
 2018-07-19 13:52:15 -07:00
Nick TerrellandGitHub 45821fac0c Merge pull request #1225 from jennifermliu/dev 
Split samples when building dictionary for COVER
 2018-07-13 13:26:15 -07:00
Nick TerrellandGitHub aa2415402d Merge pull request #1235 from ldv-alt/dev 
tests: use /dev/zero instead of /dev/random on all systems except GNU/Hurd
 2018-07-12 16:56:54 -07:00
Nick TerrellandGitHub b426bcc097 [zstdmt] Fix jobsize bugs (#1205) 
[zstdmt] Fix jobsize bugs

* `ZSTDMT_serialState_reset()` should use `targetSectionSize`, not `jobSize` when sizing the seqstore.
  Add an assert that checks that we sized the seqstore using the right job size.
* `ZSTDMT_compressionJob()` should check if `rawSeqStore.seq == NULL`.
* `ZSTDMT_initCStream_internal()` should not adjust `mtctx->params.jobSize` (clamping to MIN/MAX is okay).
 2018-06-25 15:21:08 -07:00
Nick TerrellandGitHub 06b70179da Work around bug in zstd decoder (#1147) 
Work around bug in zstd decoder

Pull request #1144 exercised a new path in the zstd decoder that proved to
be buggy. Avoid the extremely rare bug by emitting an uncompressed block.
 2018-05-23 18:02:30 -07:00
Nick TerrellandGitHub 3c3f59e68f Enforce pledgeSrcSize whenever known (#1106) 
The test fails before the patch and passes after.

Fixes #1095.
 2018-04-12 16:02:03 -07:00
Nick TerrellandGitHub aa4dbd09a1 Pull job/overlap log logic into common function (#1055) 
Prepares for LDM integration by separating the job size and overlap logic
into helper functions.
 2018-03-19 15:56:36 -07:00
Nick TerrellandGitHub 90d38f6a53 Merge pull request #945 from terrelln/dev 
Fix cdict compressor repcodes
 2017-12-13 14:24:21 -08:00