Paul Cruz
7ab758a640
changed how completion is actually sampled
2017-07-20 10:53:51 -07:00
Yann Collet
b4d460f32c
pool.c : blindfix for Visual warnings
2017-07-20 01:13:14 -07:00
Paul Cruz
dcf609f835
make adaptCompressionLevel oscillate less
2017-07-19 16:36:33 -07:00
Yann Collet
1ca1288689
added --memtest=# command to fuzzer
...
to jump directly to relevant test section
2017-07-19 16:01:16 -07:00
Paul Cruz
2a22c7915e
call ZSTD_compressBegin() once
2017-07-19 16:00:54 -07:00
Paul Cruz
6767abe652
fixing error when file size is multiple of job size (in which case, the srcSize of the last job is 0)
2017-07-19 14:54:15 -07:00
Yann Collet
3974d2b38a
blind fix for Windows Multithreading module
...
adds a fake 0 return value for mutex/cond init
2017-07-19 13:33:21 -07:00
Paul Cruz
42382c1216
added some debug statements, adjusted end condition
2017-07-19 13:30:07 -07:00
Paul Cruz
6945b3c43d
removed previous version of completion for compression
2017-07-19 11:51:50 -07:00
Paul Cruz
5a85c57e30
set up new calculations compression completion progress
2017-07-19 11:47:17 -07:00
Paul Cruz
f1ac518b59
split compression into smaller blocks
2017-07-19 11:23:40 -07:00
Paul Cruz
338951cd48
moved compression adapt to avoid warning
2017-07-19 10:23:46 -07:00
Paul Cruz
4497ecf297
change compression level only right before actually performing compression. When waiting, only update waiting statistics.
2017-07-19 10:14:00 -07:00
Paul Cruz
e11bf55d0b
added mechanism for measuring how much of a job has been created
2017-07-19 10:10:47 -07:00
Paul Cruz
559ea4ff25
split up read process into smaller chunks
2017-07-19 09:59:17 -07:00
Paul Cruz
6119cd2164
added additional print for help menu
2017-07-19 09:43:17 -07:00
Yann Collet
b71363b967
check pthread_*_init() success condition
2017-07-19 01:05:40 -07:00
Paul Cruz
3d7f1afadd
changed createCCtx() to split into initialization and creation
2017-07-18 17:32:36 -07:00
Paul Cruz
2c4e4ddc50
added mutex for stats struct
2017-07-18 15:55:58 -07:00
Yann Collet and GitHub
44b0838253
Merge pull request #770 from terrelln/test-mode
...
[zstdcli] Fix -t in streaming mode
2017-07-18 15:40:59 -07:00
Paul Cruz
ad66faf16a
added progress check for filewriting, put important shared data behind mutex when being read from/written to
2017-07-18 15:23:11 -07:00
Yann Collet and GitHub
2aba13b770
Merge pull request #769 from terrelln/real-block-split
...
[libzstd] Fix bug in Huffman encoding
2017-07-18 14:58:26 -07:00
Nick Terrell
d0b27483ae
[zstdcli] Fix -t in streaming mode
2017-07-18 14:45:49 -07:00
Paul Cruz
a34bc30237
setting up basic readme
2017-07-18 13:31:02 -07:00
Paul Cruz
29c36cf051
rename completion variable, split up fwrite operations in order to track progress
2017-07-18 13:30:29 -07:00
Nick Terrell
cc1522351f
[libzstd] Fix bug in Huffman encoding
...
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.
The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.
Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s
Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
2017-07-18 13:20:40 -07:00
Paul Cruz
ae47eab2fd
changed test cases to use -s setting on the diffs
2017-07-18 12:58:50 -07:00
Yann Collet and GitHub
77d67fb167
Merge pull request #766 from terrelln/real-block-split
...
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-18 08:26:24 -07:00
Yann Collet and GitHub
14c83b05c7
Merge pull request #765 from terrelln/real-block-split
...
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 19:25:55 -07:00
Paul Cruz
5af04c57b0
change parameters for compression level adapt
2017-07-17 17:59:50 -07:00
Paul Cruz
b3c9e02bb6
added signal to other threads whenever error occurs
2017-07-17 15:34:58 -07:00
Nick Terrell
7a28b9e4a3
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-17 15:29:11 -07:00
Yann Collet and GitHub
3381bf4b84
Merge pull request #764 from terrelln/real-block-split
...
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 14:46:01 -07:00
Paul Cruz
6be22f1f84
swap buffers instead of copying memory over
2017-07-17 14:39:10 -07:00
Paul Cruz
708238e07e
open file outside of adaptCCtx, pass to the output thread
2017-07-17 14:01:13 -07:00
Nick Terrell
e198230645
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 12:27:24 -07:00
Nick Terrell
634f012420
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 11:36:11 -07:00
Paul Cruz
044e40db5a
removed freeCCtx() calls from createCCtx() so that it is not called twice during errors
2017-07-17 11:19:23 -07:00
Paul Cruz
50ce4eaeb6
added error detection for pthread initialization, added compression completion measurement, fixed const values
2017-07-17 10:12:44 -07:00
Paul Cruz
1ab3f06f00
updated tests to use different seeds when executing different tests
2017-07-14 16:29:29 -07:00
Yann Collet and GitHub
fa3aa04ccd
Merge pull request #761 from paulcruz74/file-rename
...
renamed pool.c to poolTests.c
2017-07-14 09:09:45 -07:00
Yann Collet and GitHub
3841ee6fb3
Merge pull request #762 from facebook/errorCodes
...
pinned down error code enum values
2017-07-14 09:09:22 -07:00
Yann Collet
3b0cff3c33
fixed clang's -Wdocumentation
2017-07-13 18:58:30 -07:00
Yann Collet
2bd6440be0
pinned down error code enum values
...
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.
Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
2017-07-13 17:12:16 -07:00
Paul Cruz
0c8b9436b7
removed goto statements for the most part
2017-07-13 16:38:20 -07:00
Yann Collet
3502426fd4
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-13 15:49:19 -07:00
Yann Collet
6733c0777c
updated NEWS regarding #760
2017-07-13 15:34:44 -07:00
Paul Cruz
65a4ce2635
added tests for forced compression level
2017-07-13 14:57:24 -07:00
Paul Cruz
0d9665cef5
added additional tests for performance, allowed force compression level for testing purposes
2017-07-13 14:46:54 -07:00
Paul Cruz
9165e97fc6
added some tests for correctness, time, and compression ratio
2017-07-13 13:50:23 -07:00
Paul Cruz