Nick Terrell
abe12b3399
[libzstd] Fix bug in Huffman decompresser
...
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.
* Make the decompresser accept Huffman compressed literals whose size
increases.
* Add a test case that exposes the bug. The compressed file has to be
statically generated, since the compressor won't normally produce files
that expose the bug.
2017-08-07 12:37:48 -07:00
Yann Collet and GitHub
e1222544be
Merge pull request #753 from paulcruz74/adapt-approach-3
...
adaptive compression v1
2017-07-27 10:00:10 -07:00
Paul Cruz
2320e7378a
remove unused variable, add documentation for context fields
2017-07-26 17:02:47 -07:00
Paul Cruz
ab5a78547e
fix leaky abstraction regarding measuring completion
2017-07-26 16:40:05 -07:00
Paul Cruz
715f36ca81
added definitions for conversion constants, moved forced compression check to top of adaptCompressionLevel, used ZSTD_BLOCKSIZE_MAX
2017-07-26 15:52:15 -07:00
Paul Cruz
6c1c1242fc
set the window log value before performing compression
2017-07-26 14:29:59 -07:00
Paul Cruz
a959cc881a
moved reset of completion to right after wait
2017-07-26 10:34:48 -07:00
Paul Cruz
305d5ee70f
change to >= convergence counter
2017-07-26 10:20:29 -07:00
Paul Cruz
be92a38d6a
decrease completion requirements for change, move create thread wait, merge cases where compression thread should wait
2017-07-26 10:05:10 -07:00
Paul Cruz
0b18d21e03
building on readme, added another help tip in the menu
2017-07-25 17:47:02 -07:00
Paul Cruz
7cc74e0b27
adding more to readme
2017-07-25 16:55:16 -07:00
Paul Cruz
8dbb07d822
updated progress bar with better representation of time, added const
2017-07-25 16:03:43 -07:00
Paul Cruz
9a132707af
changing time units to seconds
2017-07-25 15:26:26 -07:00
Paul Cruz
31a9ed9883
updated const values, added more comments
2017-07-25 14:53:40 -07:00
Yann Collet and GitHub
6466fd3400
Merge pull request #776 from terrelln/dev
...
[libzstd] Fix CHECK_V_F() macro
2017-07-25 14:39:32 -07:00
Paul Cruz
5cfbf609a4
removed old debug statements no longer being used
2017-07-25 14:31:48 -07:00
Paul Cruz
0882cd1981
progress bar -- don't print num jobs, time elapsed shown in seconds
2017-07-25 14:26:55 -07:00
Paul Cruz
310c12d07e
moved debug statements to a compiler flag
2017-07-25 14:08:39 -07:00
Nick Terrell
ae20d413da
[libzstd] Fix CHECK_V_F macros
2017-07-25 12:52:01 -07:00
Paul Cruz
e02c79f833
started using decrease cooldown so that compression level would not decrease several times in a row
2017-07-25 11:16:27 -07:00
Paul Cruz
85d7c919f6
created independent function for controlling how completion relates to compression level change
2017-07-25 10:32:14 -07:00
Paul Cruz
6f1e260edd
added mechanism for getting rid of spikes
2017-07-25 10:01:10 -07:00
Paul Cruz
700758d676
added help statement for -p, switched it to hide progress bar now that progress bar is default
2017-07-24 16:26:20 -07:00
Paul Cruz
df3754b6ed
add quiet option, make progress bar default
2017-07-24 16:19:07 -07:00
Paul Cruz
4dc83ca64c
compression thread should take measurements independently based on whether or not the create/write thread will actually bottleneck performance
2017-07-24 15:14:58 -07:00
Paul Cruz
0ee3f8c2f8
adding more debug
2017-07-24 15:06:11 -07:00
Paul Cruz
8328f8192a
updating debug statements again
2017-07-24 14:40:23 -07:00
Paul Cruz
d3d759301f
changing position of endline for debug
2017-07-24 13:47:39 -07:00
Paul Cruz
e508f632d6
updated comments and debug statements
2017-07-24 11:01:36 -07:00
Paul Cruz
483d936b87
reduced competition for completion mutex by separating mutex use based on which values is updated
2017-07-23 14:09:16 -07:00
Paul Cruz
880f08d104
change how completion is measured in compression thread
2017-07-23 10:18:54 -07:00
Paul Cruz
08d9e42ec6
removed useless measurements
2017-07-21 18:02:55 -07:00
Paul Cruz
95bef759b3
switched over to model where reading only waits on compression thread
2017-07-21 17:49:39 -07:00
Paul Cruz
6455ec482c
taking the maximum of the completion level reads in order to determine which one was waiting more
2017-07-21 16:05:01 -07:00
Paul Cruz
05fe8dd47c
updating debug statements
2017-07-21 14:06:24 -07:00
Paul Cruz
db109f8fef
measure multiple completion levels during each wait
2017-07-21 13:38:24 -07:00
Paul Cruz
721c6a8b97
added bounding to compression level change
2017-07-21 09:30:24 -07:00
Paul Cruz
e929d3b787
added priority decision making for adapt compression level
2017-07-21 09:26:35 -07:00
Paul Cruz
9259c7afa4
semi working version that stabilizes
2017-07-20 18:45:33 -07:00
Yann Collet
38ba7002f2
fixed minor warning on unused variable in shell function
2017-07-20 18:39:04 -07:00
Paul Cruz
82e488770c
fixed bug where writeSize could be zero
2017-07-20 16:38:02 -07:00
Paul Cruz
a19916425d
reworked adaptCompressionLevel to only account for completion information
2017-07-20 16:19:16 -07:00
Yann Collet
a90b16e150
Visual blind fix 2
2017-07-20 15:57:55 -07:00
Yann Collet
4542d7118c
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-20 15:12:21 -07:00
Yann Collet
5e6c5203f3
fixed fuzzer test for non OS-X platforms
2017-07-20 15:11:56 -07:00
Yann Collet and GitHub
57251f52cf
Merge pull request #773 from terrelln/lk-v3
...
[linux] Update patches for v3
2017-07-20 14:47:26 -07:00
Nick Terrell
7d3ac0710d
[linux] Update patches for v3
2017-07-20 13:33:55 -07:00
Paul Cruz
7ab758a640
changed how completion is actually sampled
2017-07-20 10:53:51 -07:00
Yann Collet
b4d460f32c
pool.c : blindfix for Visual warnings
2017-07-20 01:13:14 -07:00
Paul Cruz
dcf609f835
make adaptCompressionLevel oscillate less
2017-07-19 16:36:33 -07:00
Nick Terrell