ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
5,960 Commits 17 Branches 77 Tags
ef4df0df4af33a5fb822509933398e22cfde07d3
Commit Graph
5960 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Ryan Schmidt ef4df0df4a Fix i386 build failure "Junk character 13" 2018-11-16 02:16:21 -06:00
Nick TerrellandGitHub f15f1bfefb Merge pull request #1416 from terrelln/rsync 
- Factor out LDM's hash function for reuse
- Add rsyncable mode to zstdmt and expose it via the advanced API
- Fix `-B`/`--block-size` to actually set the job size
- Add rsyncable tests to `zstreamtest` and `playTests.sh`

Tested by:

```
> cat A.100MB B.100MB C.100MB D.100MB E.100MB | zstd --rsyncable -fo src/file.zst
/*stdin*\            : 48.22%   (524288000 => 252837782 bytes, src/file.zst)
> rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
Total bytes sent: 252868779
total size is 252837782  speedup is 1.00
> echo test > test
> cat test A.100MB test B.100MB test C.100MB test D.100MB test E.100MB | zstd --rsyncable -fo src/file.zst
/*stdin*\            : 48.23%   (524288025 => 252838025 bytes, src/unicorn.tar.zst)
> rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
Total bytes sent: 4605696
total size is 252838025  speedup is 53.60
```

Close #1155.
 2018-11-14 18:43:12 -08:00
Nick Terrell 71fe78cbf9 [tests] Add rsyncable tests 2018-11-14 17:00:29 -08:00
Nick Terrell 4316409427 [cli] Set the job size 2018-11-14 17:00:29 -08:00
Nick Terrell f9a671ac37 [cli] Add rsyncable 2018-11-14 17:00:16 -08:00
Nick Terrell b9693d3a49 [lib] Add rsyncable mode 
- Add rsyncable mode to multithreaded mode
- Factor out LDM's hash function for reuse
 2018-11-14 16:59:57 -08:00
Yann ColletandGitHub c584e84e68 Merge pull request #1415 from facebook/benchfn 
Separating benchfn and benchzstd
 2018-11-14 13:00:08 -08:00
Yann Collet 975abf8f5d circumvent gcc-8 strict aliasing warning 
because gcc considers that
`void* const *`  and `void**` are 2 different types ...
 2018-11-13 16:04:35 -08:00
Yann Collet 768a264200 Merge branch 'dev' of github.com:facebook/zstd into dev 2018-11-13 15:56:36 -08:00
Yann Collet 092c4abd4c bumped version number to v1.3.8 2018-11-13 15:53:38 -08:00
Yann Collet 3ba0d6dd27 fixed decode-only test condition 2018-11-13 14:15:12 -08:00
Yann Collet a29e3d1d01 fix debug trace 2018-11-13 13:27:22 -08:00
Yann Collet e4abd28769 fixed VS2010+ build script 2018-11-13 13:22:16 -08:00
Yann Collet f15312e884 fixed cmake build script 2018-11-13 13:16:41 -08:00
Yann Collet b830ccca5c changed benchfn api 
to use structure for function parameters
as it expresses much clearer than a long list of parameters,
since each parameter can now be named.
 2018-11-13 13:12:50 -08:00
Yann Collet 9867cdb847 benchfn can provided faulty return value 
with BMK_extract_returnValue()
 2018-11-13 12:01:17 -08:00
Yann Collet d38063f8ae separated bench module into benchfn and benchzstd 
it shall be possible to use benchfn
without any dependency on zstd.
 2018-11-13 11:01:59 -08:00
Yann ColletandGitHub f28af025d9 Merge pull request #1413 from felixhandte/attach-dict-fix-unsigned-compare 
Fix #1412: Perform Signed Comparison When Setting Attach Dict Param
 2018-11-12 17:53:11 -08:00
Yann Collet 626040ab53 changed PREFETCH() macro into PREFETCH_L2() 
which is more accurate
 2018-11-12 17:05:32 -08:00
W. Felix Handte 5faef4d378 Const 2018-11-12 14:48:42 -08:00
W. Felix Handte 2242330b26 Fix Fuzz Range 2018-11-12 13:01:14 -08:00
W. Felix Handte 2d9332eb21 Fix Types 2018-11-12 12:52:31 -08:00
W. Felix Handte 4127de5fa6 Switch Enum to Only Non-Negative Values, Update Comments 2018-11-12 12:47:47 -08:00
W. Felix Handte 596f7d1256 Fix #1412: Perform Signed Comparison When Setting Attach Dict Param 2018-11-12 12:07:57 -08:00
Yann ColletandGitHub 7b0c551bff Merge pull request #1411 from facebook/prefetch_dict 
Improves decompression speed when using cold dictionary
 2018-11-09 11:31:35 -08:00
Yann ColletandGitHub 1b4a9c518b Merge pull request #1410 from facebook/prefetch_dec 
improve long-range decoder speed
 2018-11-08 18:41:58 -08:00
Yann Collet 483759a3de Improves decompression speed when using cold dictionary 
by triggering the prefetching decoder path
(which used to be dedicated to long-range offsets only).

Figures on my laptop :
no content prefetch : ~300 MB/s (for reference)
full content prefetch : ~325 MB/s (before this patch)
new prefetch path : ~375 MB/s (after this patch)

The benchmark speed is already significant,
but another side-effect is that this version
prefetch less data into memory,
since it only prefetches what's needed, instead of the full dictionary.

This is supposed to help highly active environments
such as active databases,
that can't be properly measured in benchmark environment (too clean).

Also :
fixed the largeNbDict test program
which was working improperly when setting nbBlocks > nbFiles.
 2018-11-08 17:00:23 -08:00
Yann ColletandGitHub a801ba643c Merge pull request #1409 from terrelln/tsan-dict 
Add TSAN dictionary tests
 2018-11-08 13:43:01 -08:00
Yann Collet 20fb9e7f36 reduced assertion strength 
one limit case can apparently be generated during fuzzer tests
 2018-11-08 12:57:34 -08:00
Yann Collet f40c6b4395 added a simple test case for decode-only benchmark mode 
as it was broken and no one noticed
 2018-11-08 12:51:42 -08:00
Yann Collet 9126da5b5c improve long-range decoder speed 
on enwik9 at level 22 (which is almost a worst case scenario),
speed improves by +7% on my laptop (415 -> 445 MB/s)
 2018-11-08 12:47:46 -08:00
Yann Collet 8bed4012bd fixed decompression-only benchmark 2018-11-08 12:36:39 -08:00
Nick Terrell a40b6958c9 Run TSAN fuzztest 2018-11-08 10:59:01 -08:00
Nick Terrell 103d1ee7a4 Add multithreaded dictbuilder tests to fuzzer.c 2018-11-08 10:58:51 -08:00
Nick Terrell a8daa2d683 Signal before unlocking in pool.c 2018-11-08 10:45:53 -08:00
Yann ColletandGitHub 5b598bb203 Merge pull request #1408 from szreder/dev 
Prevent deadlock on malloc() failure.
 2018-11-08 10:43:40 -08:00
Bartosz Szreder 5c5c476338 Prevent deadlock on malloc() failure. 2018-11-08 10:29:31 +01:00
Yann ColletandGitHub f6eb12084d Merge pull request #1405 from pitrou/allow_cmake_debug_builds 
Allow creating debug builds with CMake
 2018-11-06 13:58:50 -08:00
Antoine Pitrou 2186fbcf67 Allow creating debug builds with CMake 
Only set the build type to Release if the user didn't select any.
See discussion in https://github.com/facebook/zstd/pull/1278#issuecomment-436327923
and below.
 2018-11-06 21:24:29 +01:00
Yann ColletandGitHub e0701d3c5d Merge pull request #1404 from facebook/T36302429 
fixed T36302471
 2018-11-06 11:53:20 -08:00
Yann ColletandGitHub af892e5f1c Merge pull request #1403 from facebook/assert_opt 
just add an assert() in ZSTD_insertBtAndGetAllMatches()
 2018-11-05 17:57:30 -08:00
Yann Collet 3e5cdf1b6a fixed T36302429 2018-11-05 17:50:30 -08:00
Yann Collet 2caa995558 just add an assert() in ZSTD_insertBtAndGetAllMatches() 
to express a condition on ll0 .
May help static analyzer as in #1397
 2018-11-05 17:13:32 -08:00
Yann ColletandGitHub 3a90229616 Merge pull request #1395 from facebook/decompressblock 
created zstd_decompress_block module
 2018-10-29 16:28:09 -07:00
Yann ColletandGitHub 2a0c9e0806 Merge pull request #1398 from bmwiedemann/pgo 
Fix building zstd-pgo
 2018-10-29 16:00:26 -07:00
Yann Collet acd75a1448 fixed a second memset() on NULL 
not sure why it only triggers now,
this code has been around for a while.

Introduced a new error code : dstBuffer_null,
I couldn't express anything even remotely similar with existing error codes set.
 2018-10-29 15:03:57 -07:00
Yann Collet 9c58098200 fixed memcpy() on NULL warning 
memcpy(NULL, src, 0) is undefined behavior.
 2018-10-29 13:57:37 -07:00
Yann ColletandGitHub ea966c8fb1 Merge pull request #1396 from facebook/huf_refactor 
refactor HUF_compress_internal for clarity
 2018-10-29 13:06:45 -07:00
Bernhard M. Wiedemann 572279e73e pgo: also optimize compression 
and clean *.o so that linking does not fail from code
that is partially augmented with coverage generation instructions
 2018-10-29 17:20:03 +01:00
Bernhard M. Wiedemann a38ad53ace zstd-pgo: do not make clean and zstd in parallel 
otherwise, there might not be a ./zstd to run
 2018-10-29 15:11:28 +01:00