 Paul Cruz
|
a19916425d
|
reworked adaptCompressionLevel to only account for completion information
|
2017-07-20 16:19:16 -07:00 |
|
|
Yann Collet
|
a90b16e150
|
Visual blind fix 2
|
2017-07-20 15:57:55 -07:00 |
|
|
Yann Collet
|
4542d7118c
|
Merge branch 'dev' of github.com:facebook/zstd into dev
|
2017-07-20 15:12:21 -07:00 |
|
|
Yann Collet
|
5e6c5203f3
|
fixed fuzzer test for non OS-X platforms
|
2017-07-20 15:11:56 -07:00 |
|
|
Yann ColletandGitHub
|
57251f52cf
|
Merge pull request #773 from terrelln/lk-v3
[linux] Update patches for v3
|
2017-07-20 14:47:26 -07:00 |
|
|
Nick Terrell
|
7d3ac0710d
|
[linux] Update patches for v3
|
2017-07-20 13:33:55 -07:00 |
|
|
Paul Cruz
|
7ab758a640
|
changed how completion is actually sampled
|
2017-07-20 10:53:51 -07:00 |
|
|
Yann Collet
|
b4d460f32c
|
pool.c : blindfix for Visual warnings
|
2017-07-20 01:13:14 -07:00 |
|
|
Stella Lau
|
13a01ffb27
|
Fix off-by-one in size calculations
|
2017-07-19 17:24:09 -07:00 |
|
|
Stella Lau
|
2427a154cb
|
Minor refactoring
|
2017-07-19 16:56:28 -07:00 |
|
|
Paul Cruz
|
dcf609f835
|
make adaptCompressionLevel oscillate less
|
2017-07-19 16:36:33 -07:00 |
|
|
Yann Collet
|
1ca1288689
|
added --memtest=# command to fuzzer
to jump directly to relevant test section
|
2017-07-19 16:01:16 -07:00 |
|
|
Paul Cruz
|
2a22c7915e
|
call ZSTD_compressBegin() once
|
2017-07-19 16:00:54 -07:00 |
|
|
Paul Cruz
|
6767abe652
|
fixing error when file size is multiple of job size (in which case, the srcSize of the last job is 0)
|
2017-07-19 14:54:15 -07:00 |
|
|
Stella Lau
|
030264ca51
|
Experiment with integrating ZSTD_count with findBestMatch
|
2017-07-19 14:14:26 -07:00 |
|
|
Yann Collet
|
3974d2b38a
|
blind fix for Windows Multithreading module
adds a fake 0 return value for mutex/cond init
|
2017-07-19 13:33:21 -07:00 |
|
|
Paul Cruz
|
42382c1216
|
added some debug statements, adjusted end condition
|
2017-07-19 13:30:07 -07:00 |
|
|
Paul Cruz
|
6945b3c43d
|
removed previous version of completion for compression
|
2017-07-19 11:51:50 -07:00 |
|
|
Paul Cruz
|
5a85c57e30
|
set up new calculations compression completion progress
|
2017-07-19 11:47:17 -07:00 |
|
|
Paul Cruz
|
f1ac518b59
|
split compression into smaller blocks
|
2017-07-19 11:23:40 -07:00 |
|
|
Paul Cruz
|
338951cd48
|
moved compression adapt to avoid warning
|
2017-07-19 10:23:46 -07:00 |
|
|
Paul Cruz
|
4497ecf297
|
change compression level only right before actually performing compression. When waiting, only update waiting statistics.
|
2017-07-19 10:14:00 -07:00 |
|
|
Paul Cruz
|
e11bf55d0b
|
added mechanism for measuring how much of a job has been created
|
2017-07-19 10:10:47 -07:00 |
|
|
Paul Cruz
|
559ea4ff25
|
split up read process into smaller chunks
|
2017-07-19 09:59:17 -07:00 |
|
|
Paul Cruz
|
6119cd2164
|
added additional print for help menu
|
2017-07-19 09:43:17 -07:00 |
|
|
Yann Collet
|
b71363b967
|
check pthread_*_init() success condition
|
2017-07-19 01:05:40 -07:00 |
|
|
Stella Lau
|
4352e09cb0
|
Avoid recounting match lengths with ZSTD_count
|
2017-07-18 18:35:25 -07:00 |
|
|
Stella Lau
|
1fa223859f
|
Switch to using ZSTD_count instead of function pointer
|
2017-07-18 18:05:10 -07:00 |
|
|
Paul Cruz
|
3d7f1afadd
|
changed createCCtx() to split into initialization and creation
|
2017-07-18 17:32:36 -07:00 |
|
|
Paul Cruz
|
2c4e4ddc50
|
added mutex for stats struct
|
2017-07-18 15:55:58 -07:00 |
|
|
Yann ColletandGitHub
|
44b0838253
|
Merge pull request #770 from terrelln/test-mode
[zstdcli] Fix -t in streaming mode
|
2017-07-18 15:40:59 -07:00 |
|
|
Paul Cruz
|
ad66faf16a
|
added progress check for filewriting, put important shared data behind mutex when being read from/written to
|
2017-07-18 15:23:11 -07:00 |
|
|
Yann ColletandGitHub
|
2aba13b770
|
Merge pull request #769 from terrelln/real-block-split
[libzstd] Fix bug in Huffman encoding
|
2017-07-18 14:58:26 -07:00 |
|
|
Nick Terrell
|
d0b27483ae
|
[zstdcli] Fix -t in streaming mode
|
2017-07-18 14:45:49 -07:00 |
|
|
Stella Lau
|
19258f51c1
|
Make the meaning of LDM_MEMORY_USAGE consistent across tables
|
2017-07-18 14:25:39 -07:00 |
|
|
Paul Cruz
|
a34bc30237
|
setting up basic readme
|
2017-07-18 13:31:02 -07:00 |
|
|
Paul Cruz
|
29c36cf051
|
rename completion variable, split up fwrite operations in order to track progress
|
2017-07-18 13:30:29 -07:00 |
|
|
Nick Terrell
|
cc1522351f
|
[libzstd] Fix bug in Huffman encoding
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.
The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.
Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s
Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
|
2017-07-18 13:20:40 -07:00 |
|
|
Paul Cruz
|
ae47eab2fd
|
changed test cases to use -s setting on the diffs
|
2017-07-18 12:58:50 -07:00 |
|
|
Yann ColletandGitHub
|
77d67fb167
|
Merge pull request #766 from terrelln/real-block-split
[libzstd] Pull optimal parser state out of seqStore_t
|
2017-07-18 08:26:24 -07:00 |
|
|
Yann ColletandGitHub
|
14c83b05c7
|
Merge pull request #765 from terrelln/real-block-split
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
|
2017-07-17 19:25:55 -07:00 |
|
|
Stella Lau
|
fc41a87964
|
Experiment with using a lag when hashing
|
2017-07-17 18:13:09 -07:00 |
|
|
Paul Cruz
|
5af04c57b0
|
change parameters for compression level adapt
|
2017-07-17 17:59:50 -07:00 |
|
|
Paul Cruz
|
b3c9e02bb6
|
added signal to other threads whenever error occurs
|
2017-07-17 15:34:58 -07:00 |
|
|
Nick Terrell
|
7a28b9e4a3
|
[libzstd] Pull optimal parser state out of seqStore_t
|
2017-07-17 15:29:11 -07:00 |
|
|
Stella Lau
|
a00e406231
|
Remove version archive
|
2017-07-17 15:17:32 -07:00 |
|
|
Stella Lau
|
15a041adbf
|
Add function to get valid entries only from table
|
2017-07-17 15:16:58 -07:00 |
|
|
Yann ColletandGitHub
|
3381bf4b84
|
Merge pull request #764 from terrelln/real-block-split
[libzstd] Refactor ZSTD_compressSequences()
|
2017-07-17 14:46:01 -07:00 |
|
|
Paul Cruz
|
6be22f1f84
|
swap buffers instead of copying memory over
|
2017-07-17 14:39:10 -07:00 |
|
|
Paul Cruz
|
708238e07e
|
open file outside of adaptCCtx, pass to the output thread
|
2017-07-17 14:01:13 -07:00 |
|