ddidderr/zstd-rs
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,021 Commits 17 Branches 77 Tags
5d5c895b18ea54c577fdc4d89c8fd334eba4e19a
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 5d5c895b18 fix initCStream_advanced() for fast strategies 
Compression ratio of fast strategies (levels 1 & 2)
was seriously reduced, due to accidental disabling of Literals compression.

Credit to @QrczakMK, which perfectly described the issue, and implementation details,
making the fix straightforward.

Example : initCStream with level 1 on synthetic sample P50 :
Before : 5,273,976 bytes
After  : 3,154,678 bytes
ZSTD_compress (for comparison) : 3,154,550

Fix #1787.

To follow : refactor the test which was supposed to catch this issue (and failed)
 2019-10-22 15:01:38 -07:00
Yann Collet 111b0c53b0 update documentation on deprecated functions 
mostly : note that these functions will soon generate deprecation warnings
 2019-10-22 13:51:18 -07:00
Yann Collet bad35bd307 turned off zlib tests during msan 
since the local zlib library is not msan-instrumented
 2019-10-21 12:55:39 -07:00
Yann Collet 632e07747c minor refactor of FIO_compressGzFrame() 
mostly narrowing variables lifetime and mutability.
 2019-10-21 12:14:59 -07:00
Yann Collet ff7bd16c0a clarifications for the FSE decoding table 
requested in #1782
 2019-10-18 17:48:12 -07:00
Yann Collet ed65210c9b Merge branch 'dev' of github.com:facebook/zstd into dev 2019-10-18 17:23:39 -07:00
Yann Collet ca73c218be added mode documentation on ZSTD_CLEVEL 
providing range restriction [1-19] explicitly
partially answering #1829
 2019-10-18 17:22:45 -07:00
Yann Collet 0492c57013 fixed visual defined test 2019-10-18 17:08:52 -07:00
Yann Collet f379637025 removed Visual warning 
for pointer casts
 2019-10-18 17:05:42 -07:00
Yann Collet 1bd6c15b6b fix function cast warning on Windows with gcc9 2019-10-18 15:45:57 -07:00
Yann Collet 58c59341d3 updated windows binary resources 2019-10-18 15:41:12 -07:00
Yann Collet 2ad75bb574 validated 'make test' for mingw environment 2019-10-18 15:37:07 -07:00
Yann Collet 85a016ed92 made make check faster 
for shorter feedback loop
 2019-10-18 15:21:50 -07:00
Yann Collet 0e154fc40e minor appveyor test script fix 2019-10-18 15:02:30 -07:00
Yann Collet 03ef7b73a7 attempt to run 'make check' tests on Appveyor 
for mingw builds
 2019-10-18 14:52:16 -07:00
Yann Collet 29e46ed0bd fix test on windows 
isDirectory() doesn't work on Windows
if directory name is followed by '/'
 2019-10-18 14:28:34 -07:00
Yann Collet ad86a5d0bc rewrite FIO_createFilename_fromOutDir() 2019-10-18 11:15:10 -07:00
Yann Collet caf40d0ae4 fix : no output file opened in test mode 
also : redistributed error code within fileio.c
for more precise diagnosis.
 2019-10-17 16:58:49 -07:00
Yann Collet 0a24d4ef18 removed regular file test on Windows 
since it does not work well on this platform
(tested with MinGW).

Note : could be an issue within UTIL_isRegularFile()
 2019-10-17 16:39:47 -07:00
Yann Collet 0ee360982d improved test mode -t 
The test mode do no longer open a file `/dev/null` nor write anything to output.

This is supposed to be more efficient than writing to `/dev/null`,
and more universal too : the previous method was failing on Windows.
 2019-10-17 16:09:53 -07:00
Yann Collet 1795133c45 refactored FIO_compressMultipleFilenames() prototype 
for consistency
 2019-10-17 15:32:03 -07:00
Yann Collet 7f86ae2867 fixed multiple implicit casts 2019-10-17 15:27:25 -07:00
Yann Collet ba7e2b6da7 tests: can override isTerminal with environment variable 2019-10-17 15:07:47 -07:00
Yann Collet 157479af0c fixed isSameFile() 2019-10-17 14:31:42 -07:00
Yann Collet 5b8e873357 fix harness test 2019-10-17 14:29:48 -07:00
Yann Collet a0c041612d fixed dict ptr init 2019-10-17 14:15:00 -07:00
Yann Collet b062b6fb2d minor refactoring of harness 2019-10-17 14:11:54 -07:00
Yann Collet bfd829f254 minor: DIFF determination 
use gdiff on SunOS
 2019-10-17 14:03:20 -07:00
Yann Collet 1a18f1484f force compression during tests 
to erase potentially remaining artifacts from previous runs
 2019-10-17 13:15:56 -07:00
Yann Collet a71256a2ee fix several cast 2019-10-17 11:01:20 -07:00
Yann Collet 000404311f fix incorrect dictName/FileName comparison on Windows 
inode identification does not seem to work on Windows,
even with on a msys2 posix layer.
 2019-10-17 10:56:14 -07:00
Yann Collet 8e715d57c5 Merge branch 'dev' of github.com:facebook/zstd into dev 2019-10-17 10:50:58 -07:00
Yann Collet 303261f659 ignore build artifact from educational decoder test 2019-10-17 10:50:39 -07:00
Yann Collet 25ce9ac401 removed UNALIGNED() macro from educational decoder 
as this name collides with existing macro in mingw64+clang9.
 2019-10-16 17:27:03 -07:00
Yann Collet 6323966e53 updated erroneous comments using ZSTD_dm_* 
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822)
 2019-10-16 16:14:04 -07:00
Yann Collet 2d5201b0ab removed wildcopy8() 
which is no longer used,
noticed by @davidbolvansky
 2019-10-16 14:51:33 -07:00
Yann Collet 6309be677c minor comments & refactoring 2019-10-15 16:09:18 -07:00
Yann Collet e0d413d648 fixed init warning 2019-10-04 15:09:52 -07:00
Yann Collet cadff8cdc4 zlibwrapper : fixed flag-variable name 
release doesn't use specific std/debug flags
 2019-10-02 17:41:19 -07:00
Yann Collet c69ed0f8d7 updated tests for zlibwrapper C90 strict compatibility 2019-10-02 17:32:19 -07:00
Yann Collet 69c9401932 simplified Makefile 
and fixed a few c++-compat issues
 2019-09-26 15:01:29 -07:00
Yann Collet 0582b27cae added c++-compat build flag 
ensure code can be compiled "as is" in C++ mode

also : restructured flags so that they can be individually changed / disabled on command line
 2019-09-26 13:08:25 -07:00
Yann Collet cb18fffe65 enforce C90 compatibility for zlibWrapper 2019-09-24 17:50:58 -07:00
Yann Collet ad2a2785f7 bump version number to v1.4.4 
so that future reports on `dev` branch use this number instead
 2019-09-24 15:15:33 -07:00
Yann Collet 243200e5bf minor refactor of ZSTD_fast 
- reduced variables lifetime
- more accurate code comments
 2019-09-17 14:02:57 -07:00
Yann Collet e0fb7e1eaf ignore dictionary artifacts 2019-09-12 09:39:15 -07:00
Yann Collet b9b9a1c8e9 fix education decoder test 
when `zstd` in not installed on local system
by allowing `ZSTD` variable to hold a custom location for the binary
 2019-09-10 09:36:02 -07:00
Yann Collet a3815d233c fix minor compilation condition 
harness.c is not designed to pass  -Wdeclaration-after-statement
 2019-09-06 16:51:16 -07:00
Yann Collet 2b0a271ed2 fix eductional decoder 
fix #1774
also :
- fix minor compilation warnings
- make sure the `test` is run during CI tests
 2019-09-06 14:30:13 -07:00
Yann Collet 64102f08da Merge branch 'dev' into decTest 2019-08-29 09:48:12 -07:00
Yann Collet e7ad20c9dc update test script 2019-08-28 13:46:18 -07:00
Yann Collet 4b3a8fe1c4 fix create_ script for sh 2019-08-28 13:23:48 -07:00
Yann Collet 8af941d2d7 Merge branch 'dev' into decTest 2019-08-28 12:17:29 -07:00
Yann Collet 416157df80 try to debug issues on CirrusCI 2019-08-27 16:38:21 -07:00
Yann Collet cd43f7d965 added single_file_decoder to contrib/ tests 2019-08-27 16:04:12 -07:00
Yann Collet f61e8a231f minor script renaming, for clarity 2019-08-27 16:01:39 -07:00
Yann Collet 517aeb89dc changed contrib project name for clarity 2019-08-27 15:50:47 -07:00
Yann Collet b12a5b9117 removed associated test 2019-08-27 14:52:16 -07:00
Yann Collet 5ed1b1e11d removed adaptive-compression 
the functionality is already integrated into `zstd` through `--adapt` command
 2019-08-27 14:47:40 -07:00
Yann Collet 97bb38635c number instead of nb 
suggested by @terrelln
 2019-08-17 08:04:42 +02:00
Yann Collet 1e07eb4d5c clarifications on the meaning of field Block_Size 
following comments from Intel's Smita Kumar.
 2019-08-16 15:15:25 +02:00
Yann Collet 782bfb858a fixed very minor inefficiency (nbSeq==127) 
The nbSeq "short" format (1-byte)
is compatible with any value < 128.

However, the code would cautiously only accept values < 127.
This is not an error, because the general 2-bytes format
is compatible with small values < 128.
Hence the inefficiency never triggered any warning.

Spotted by Intel's Smita Kumar.
 2019-08-15 16:41:34 +02:00
Yann Collet 01b2331ad1 bumped version number 
to v1.4.3
 2019-08-05 17:17:16 +02:00
Yann Collet facbe8b2c2 factored the logic selecting lowest match index 
as suggested by @terrelln
 2019-08-05 15:18:43 +02:00
Yann Collet 0b0b83e8f3 fix test 122 
it's an unsupported scenario.
 2019-08-03 16:51:26 +02:00
Yann Collet efe8496755 minor test refactoring 
just for clarity, for the currently failing unit test
 2019-08-02 19:31:19 +02:00
Yann Collet 387e20d4f0 fixed minor conversion warning in datagen 2019-08-02 18:02:54 +02:00
Yann Collet 37f47e51a8 fixed datagen 
to produce same content on both 32 and 64-bit platforms
by removing floating from literal table determination.

also : added checksum trace in compression control test,
so that it's easier to determine if test fails
as a consequence of compressing a different sample.
 2019-08-02 17:34:53 +02:00
Yann Collet d1927f0b39 regenerate sample to compress 
to reduce chances of differences between 32 and 64-bit fuzzer tests
 2019-08-02 15:31:00 +02:00
Yann Collet 98e7c344cd fixed strategies btopt+ 2019-08-02 14:42:53 +02:00
Yann Collet b4257b04e7 fixed strategy btlazy2 2019-08-02 14:26:26 +02:00
Yann Collet 5cf1b24aca fixed strategies greedy, lazy & lazy2 
restore dictionary compression ratio
 2019-08-02 14:21:39 +02:00
Yann Collet 2115292616 minor : fixed ptr arithmetic 
invalid on void ptr
 2019-08-01 17:12:26 +02:00
Yann Collet 810a9cac08 added efficiency test 
to detect gross CR variations after a patch.

Tests normal and dictionary compression.
 2019-08-01 16:59:22 +02:00
Yann Collet 98692c2838 fixed compression ratio regression when dictionary-compressing medium-size inputs at levels 1-3 2019-08-01 15:58:17 +02:00
Yann Collet 1f16245e9e updated man page 2019-07-24 16:04:37 -07:00
Yann Collet eaeb7f00b5 updated the _extDict variant of double fast 2019-07-12 14:17:17 -07:00
Yann Collet e8a7f5d3ce double-fast: changed the trade-off for a smaller positive change 
same number of complementary insertions, just organized differently
(long at `ip-2`, short at `ip-1`).
 2019-07-12 11:34:53 -07:00
Yann Collet d1327738c2 updated double_fast complementary insertion 
in a way which is more favorable to compression ratio,
though very slightly slower (~-1%).

More details in the PR.
 2019-07-11 15:25:22 -07:00
Yann Collet 2387d574cb updated .gitignore 2019-07-09 14:54:48 -07:00
Yann Collet 34a1a37246 updated .gitignore rule 2019-07-09 14:44:22 -07:00
Yann Collet b8ec4b0fd6 updated version number (to v1.4.1) 
also : added doc on context re-use, as suggested by @scherepanov at #1676
 2019-07-09 11:43:59 -07:00
Yann Collet 621adde3b2 changed naming to ZSTD_indexTooCloseToMax() 
Also : minor speed optimization :
shortcut to ZSTD_reset_matchState() rather than the full reset process.
It still needs to be completed with ZSTD_continueCCtx() for proper initialization.

Also : changed position of LDM hash tables in the context,
so that the "regular" hash tables can be at a predictable position,
hence allowing the shortcut to ZSTD_reset_matchState() without complex conditions.
 2019-06-24 14:39:29 -07:00
Yann Collet 45c9fbd6d9 prefer memset() rather than reduceIndex() when close to index range limit 
by disabling continue mode when index is close to limit.
 2019-06-21 16:19:21 -07:00
Yann Collet 944e2e9e12 benchfn : added macro macro CONTROL() 
like assert() but cannot be disabled.
proper separation of user contract errors (CONTROL())
and invariant verification (assert()).
 2019-06-21 15:58:55 -07:00
Yann Collet b3af1873a0 better title formatting for html documentation 
must pay attention to /** and /*! patterns.
 2019-06-04 10:35:40 -07:00
Yann Collet b5c98fbfd0 Added comments on I/O buffer sizes for streaming 
It seems this is still a confusing topic,
as in https://github.com/klauspost/compress/issues/109 .
 2019-06-04 10:26:16 -07:00
Yann Collet 80d6ccea79 removed UINT32_MAX 
apparently not guaranteed on all platforms,
replaced by UINT_MAX.
 2019-05-31 17:27:07 -07:00
Yann Collet fce4df3ab7 fixed wrong assert in double_fast 2019-05-31 17:06:28 -07:00
Yann Collet a968099038 minor code cleaning for new index invalidation strategy 2019-05-31 16:52:37 -07:00
Yann Collet d605f482c7 make double_fast compatible with new index invalidation strategy 2019-05-31 16:50:04 -07:00
Yann Collet a30febaeeb Made fast strategy compatible with new offset validation strategy 
fast mode does the same thing as before :
it pre-emptively invalidates any index that could lead to offset > maxDistance.
It's supposed to help speed.

But this logic is performed inside zstd_fast,
so that other strategies can select a different behavior.
 2019-05-31 16:34:55 -07:00
Yann Collet 58adb1059f extended exact window size to greedy/lazy modes 2019-05-31 16:08:48 -07:00
Yann Collet bc601bdc6d first implementation of small window size for btopt 
noticeably improves compression ratio
when window size is small (< 18).

enwik7	level 19

windowLog	`dev`	`smallwlog`	improvement
23	3.577	3.577	0.02%
22	3.536	3.538	0.06%
21	3.462	3.467	0.14%
20	3.364	3.377	0.39%
19	3.244	3.272	0.86%
18	3.110	3.166	1.80%
17	2.843	3.057	7.53%
16	2.724	2.943	8.04%
15	2.594	2.822	8.79%
14	2.456	2.686	9.36%
13	2.312	2.523	9.13%
12	2.162	2.361	9.20%
11	2.003	2.182	8.94%
 2019-05-31 15:55:12 -07:00
Yann Collet 78056a3e9c consistency : changed constant name to kSampleSizeDefault 
as suggested by @terrelln.

added <assert.h> include
 2019-05-30 10:13:56 -07:00
Yann Collet ab3625cffd define CONTROL, to ensure check cannot be disabled 
assert() can be disabled with NDEBUG
 2019-05-29 16:18:22 -07:00
Yann Collet 904d4da239 fullbench : minor refactoring, for readability 2019-05-29 16:08:49 -07:00
Yann Collet ed38b645db fullbench: pass proper parameters in scenario 43 2019-05-29 15:26:06 -07:00
Yann Collet c63081623f fullbench 43: ensure context is freed after each usage 2019-05-28 17:27:52 -07:00
Yann Collet eb6b199038 fullbench: added streaming with fresh CCtx scenario 
worst case situation, where context must be recreated every time
but without knowledge of the input size
thus sizing the context for some large input.
 2019-05-28 16:50:49 -07:00