ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,966 Commits 17 Branches 77 Tags
a71256a2ee4fc219a85ca0c93260c30434b026ba
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet a71256a2ee fix several cast 2019-10-17 11:01:20 -07:00
Yann Collet 000404311f fix incorrect dictName/FileName comparison on Windows 
inode identification does not seem to work on Windows,
even with on a msys2 posix layer.
 2019-10-17 10:56:14 -07:00
Yann Collet 8e715d57c5 Merge branch 'dev' of github.com:facebook/zstd into dev 2019-10-17 10:50:58 -07:00
Yann Collet 303261f659 ignore build artifact from educational decoder test 2019-10-17 10:50:39 -07:00
Yann Collet 25ce9ac401 removed UNALIGNED() macro from educational decoder 
as this name collides with existing macro in mingw64+clang9.
 2019-10-16 17:27:03 -07:00
Yann Collet 6323966e53 updated erroneous comments using ZSTD_dm_* 
instead of the current ZSTD_dct_*,
reported by @nigeltao (#1822)
 2019-10-16 16:14:04 -07:00
Yann Collet 2d5201b0ab removed wildcopy8() 
which is no longer used,
noticed by @davidbolvansky
 2019-10-16 14:51:33 -07:00
Yann Collet 6309be677c minor comments & refactoring 2019-10-15 16:09:18 -07:00
Yann Collet e0d413d648 fixed init warning 2019-10-04 15:09:52 -07:00
Yann Collet cadff8cdc4 zlibwrapper : fixed flag-variable name 
release doesn't use specific std/debug flags
 2019-10-02 17:41:19 -07:00
Yann Collet c69ed0f8d7 updated tests for zlibwrapper C90 strict compatibility 2019-10-02 17:32:19 -07:00
Yann Collet 69c9401932 simplified Makefile 
and fixed a few c++-compat issues
 2019-09-26 15:01:29 -07:00
Yann Collet 0582b27cae added c++-compat build flag 
ensure code can be compiled "as is" in C++ mode

also : restructured flags so that they can be individually changed / disabled on command line
 2019-09-26 13:08:25 -07:00
Yann Collet cb18fffe65 enforce C90 compatibility for zlibWrapper 2019-09-24 17:50:58 -07:00
Yann Collet ad2a2785f7 bump version number to v1.4.4 
so that future reports on `dev` branch use this number instead
 2019-09-24 15:15:33 -07:00
Yann Collet 243200e5bf minor refactor of ZSTD_fast 
- reduced variables lifetime
- more accurate code comments
 2019-09-17 14:02:57 -07:00
Yann Collet e0fb7e1eaf ignore dictionary artifacts 2019-09-12 09:39:15 -07:00
Yann Collet b9b9a1c8e9 fix education decoder test 
when `zstd` in not installed on local system
by allowing `ZSTD` variable to hold a custom location for the binary
 2019-09-10 09:36:02 -07:00
Yann Collet a3815d233c fix minor compilation condition 
harness.c is not designed to pass  -Wdeclaration-after-statement
 2019-09-06 16:51:16 -07:00
Yann Collet 2b0a271ed2 fix eductional decoder 
fix #1774
also :
- fix minor compilation warnings
- make sure the `test` is run during CI tests
 2019-09-06 14:30:13 -07:00
Yann Collet 64102f08da Merge branch 'dev' into decTest 2019-08-29 09:48:12 -07:00
Yann Collet e7ad20c9dc update test script 2019-08-28 13:46:18 -07:00
Yann Collet 4b3a8fe1c4 fix create_ script for sh 2019-08-28 13:23:48 -07:00
Yann Collet 8af941d2d7 Merge branch 'dev' into decTest 2019-08-28 12:17:29 -07:00
Yann Collet 416157df80 try to debug issues on CirrusCI 2019-08-27 16:38:21 -07:00
Yann Collet cd43f7d965 added single_file_decoder to contrib/ tests 2019-08-27 16:04:12 -07:00
Yann Collet f61e8a231f minor script renaming, for clarity 2019-08-27 16:01:39 -07:00
Yann Collet 517aeb89dc changed contrib project name for clarity 2019-08-27 15:50:47 -07:00
Yann Collet b12a5b9117 removed associated test 2019-08-27 14:52:16 -07:00
Yann Collet 5ed1b1e11d removed adaptive-compression 
the functionality is already integrated into `zstd` through `--adapt` command
 2019-08-27 14:47:40 -07:00
Yann Collet 97bb38635c number instead of nb 
suggested by @terrelln
 2019-08-17 08:04:42 +02:00
Yann Collet 1e07eb4d5c clarifications on the meaning of field Block_Size 
following comments from Intel's Smita Kumar.
 2019-08-16 15:15:25 +02:00
Yann Collet 782bfb858a fixed very minor inefficiency (nbSeq==127) 
The nbSeq "short" format (1-byte)
is compatible with any value < 128.

However, the code would cautiously only accept values < 127.
This is not an error, because the general 2-bytes format
is compatible with small values < 128.
Hence the inefficiency never triggered any warning.

Spotted by Intel's Smita Kumar.
 2019-08-15 16:41:34 +02:00
Yann Collet 01b2331ad1 bumped version number 
to v1.4.3
 2019-08-05 17:17:16 +02:00
Yann Collet facbe8b2c2 factored the logic selecting lowest match index 
as suggested by @terrelln
 2019-08-05 15:18:43 +02:00
Yann Collet 0b0b83e8f3 fix test 122 
it's an unsupported scenario.
 2019-08-03 16:51:26 +02:00
Yann Collet efe8496755 minor test refactoring 
just for clarity, for the currently failing unit test
 2019-08-02 19:31:19 +02:00
Yann Collet 387e20d4f0 fixed minor conversion warning in datagen 2019-08-02 18:02:54 +02:00
Yann Collet 37f47e51a8 fixed datagen 
to produce same content on both 32 and 64-bit platforms
by removing floating from literal table determination.

also : added checksum trace in compression control test,
so that it's easier to determine if test fails
as a consequence of compressing a different sample.
 2019-08-02 17:34:53 +02:00
Yann Collet d1927f0b39 regenerate sample to compress 
to reduce chances of differences between 32 and 64-bit fuzzer tests
 2019-08-02 15:31:00 +02:00
Yann Collet 98e7c344cd fixed strategies btopt+ 2019-08-02 14:42:53 +02:00
Yann Collet b4257b04e7 fixed strategy btlazy2 2019-08-02 14:26:26 +02:00
Yann Collet 5cf1b24aca fixed strategies greedy, lazy & lazy2 
restore dictionary compression ratio
 2019-08-02 14:21:39 +02:00
Yann Collet 2115292616 minor : fixed ptr arithmetic 
invalid on void ptr
 2019-08-01 17:12:26 +02:00
Yann Collet 810a9cac08 added efficiency test 
to detect gross CR variations after a patch.

Tests normal and dictionary compression.
 2019-08-01 16:59:22 +02:00
Yann Collet 98692c2838 fixed compression ratio regression when dictionary-compressing medium-size inputs at levels 1-3 2019-08-01 15:58:17 +02:00
Yann Collet 1f16245e9e updated man page 2019-07-24 16:04:37 -07:00
Yann Collet eaeb7f00b5 updated the _extDict variant of double fast 2019-07-12 14:17:17 -07:00
Yann Collet e8a7f5d3ce double-fast: changed the trade-off for a smaller positive change 
same number of complementary insertions, just organized differently
(long at `ip-2`, short at `ip-1`).
 2019-07-12 11:34:53 -07:00
Yann Collet d1327738c2 updated double_fast complementary insertion 
in a way which is more favorable to compression ratio,
though very slightly slower (~-1%).

More details in the PR.
 2019-07-11 15:25:22 -07:00
Yann Collet 2387d574cb updated .gitignore 2019-07-09 14:54:48 -07:00
Yann Collet 34a1a37246 updated .gitignore rule 2019-07-09 14:44:22 -07:00
Yann Collet b8ec4b0fd6 updated version number (to v1.4.1) 
also : added doc on context re-use, as suggested by @scherepanov at #1676
 2019-07-09 11:43:59 -07:00
Yann Collet 621adde3b2 changed naming to ZSTD_indexTooCloseToMax() 
Also : minor speed optimization :
shortcut to ZSTD_reset_matchState() rather than the full reset process.
It still needs to be completed with ZSTD_continueCCtx() for proper initialization.

Also : changed position of LDM hash tables in the context,
so that the "regular" hash tables can be at a predictable position,
hence allowing the shortcut to ZSTD_reset_matchState() without complex conditions.
 2019-06-24 14:39:29 -07:00
Yann Collet 45c9fbd6d9 prefer memset() rather than reduceIndex() when close to index range limit 
by disabling continue mode when index is close to limit.
 2019-06-21 16:19:21 -07:00
Yann Collet 944e2e9e12 benchfn : added macro macro CONTROL() 
like assert() but cannot be disabled.
proper separation of user contract errors (CONTROL())
and invariant verification (assert()).
 2019-06-21 15:58:55 -07:00
Yann Collet b3af1873a0 better title formatting for html documentation 
must pay attention to /** and /*! patterns.
 2019-06-04 10:35:40 -07:00
Yann Collet b5c98fbfd0 Added comments on I/O buffer sizes for streaming 
It seems this is still a confusing topic,
as in https://github.com/klauspost/compress/issues/109 .
 2019-06-04 10:26:16 -07:00
Yann Collet 80d6ccea79 removed UINT32_MAX 
apparently not guaranteed on all platforms,
replaced by UINT_MAX.
 2019-05-31 17:27:07 -07:00
Yann Collet fce4df3ab7 fixed wrong assert in double_fast 2019-05-31 17:06:28 -07:00
Yann Collet a968099038 minor code cleaning for new index invalidation strategy 2019-05-31 16:52:37 -07:00
Yann Collet d605f482c7 make double_fast compatible with new index invalidation strategy 2019-05-31 16:50:04 -07:00
Yann Collet a30febaeeb Made fast strategy compatible with new offset validation strategy 
fast mode does the same thing as before :
it pre-emptively invalidates any index that could lead to offset > maxDistance.
It's supposed to help speed.

But this logic is performed inside zstd_fast,
so that other strategies can select a different behavior.
 2019-05-31 16:34:55 -07:00
Yann Collet 58adb1059f extended exact window size to greedy/lazy modes 2019-05-31 16:08:48 -07:00
Yann Collet bc601bdc6d first implementation of small window size for btopt 
noticeably improves compression ratio
when window size is small (< 18).

enwik7	level 19

windowLog	`dev`	`smallwlog`	improvement
23	3.577	3.577	0.02%
22	3.536	3.538	0.06%
21	3.462	3.467	0.14%
20	3.364	3.377	0.39%
19	3.244	3.272	0.86%
18	3.110	3.166	1.80%
17	2.843	3.057	7.53%
16	2.724	2.943	8.04%
15	2.594	2.822	8.79%
14	2.456	2.686	9.36%
13	2.312	2.523	9.13%
12	2.162	2.361	9.20%
11	2.003	2.182	8.94%
 2019-05-31 15:55:12 -07:00
Yann Collet 78056a3e9c consistency : changed constant name to kSampleSizeDefault 
as suggested by @terrelln.

added <assert.h> include
 2019-05-30 10:13:56 -07:00
Yann Collet ab3625cffd define CONTROL, to ensure check cannot be disabled 
assert() can be disabled with NDEBUG
 2019-05-29 16:18:22 -07:00
Yann Collet 904d4da239 fullbench : minor refactoring, for readability 2019-05-29 16:08:49 -07:00
Yann Collet ed38b645db fullbench: pass proper parameters in scenario 43 2019-05-29 15:26:06 -07:00
Yann Collet c63081623f fullbench 43: ensure context is freed after each usage 2019-05-28 17:27:52 -07:00
Yann Collet eb6b199038 fullbench: added streaming with fresh CCtx scenario 
worst case situation, where context must be recreated every time
but without knowledge of the input size
thus sizing the context for some large input.
 2019-05-28 16:50:49 -07:00
Yann Collet 4c4149452c fullbench can now select sample size with command -B# 2019-05-28 16:37:03 -07:00
Yann Collet 9719fd616c removed nextToUpdate3 from ZSTD_window 
it's now a local variable of ZSTD_compressBlock_opt()
 2019-05-28 16:18:12 -07:00
Yann Collet 33dabc8c80 get bt matches : made it a bit clearer which parameters are input and output 2019-05-28 16:11:32 -07:00
Yann Collet 327cf6fac1 nextToUpdate3 does not need to be maintained outside of zstd_opt.c 
It's re-synchronized with nextToUpdate at beginning of each block.
It only needs to be tracked from within zstd_opt block parser.

Made the logic clear, so that no code tried to maintain this variable.

An even better solution would be to make nextToUpdate3
an internal variable of ZSTD_compressBlock_opt_generic().
That would make it possible to remove it from ZSTD_matchState_t,
thus restricting its visibility to only where it's actually useful.

This would require deeper changes though,
since the matchState is the natural structure to transport parameters into and inside the parser.
 2019-05-28 15:26:52 -07:00
Yann Collet 6453f8158f complementary code comments 
on variables used / impacted during maxDist check
 2019-05-28 14:12:16 -07:00
Yann Collet 4baecdf72a added comments to better understand enforceMaxDist() 2019-05-28 13:15:48 -07:00
Yann Collet 1e01560b83 fixed timespec_get() initialization bug on some targets 
not sure why, but msan fires an "unitialized variable" error
when time gets properly initialized by timespec_get().
Maybe in some cases, not all bytes of the structure are initialized ?
Or maybe msan fails to detect the initialization ?

Anyway, pre-initializing the variable before passing it to timespec_get() works.
 2019-04-11 13:46:30 -07:00
Yann Collet 058da605cb fixed minor conversion warning 2019-04-11 12:25:27 -07:00
Yann Collet f8e9bec73a fixed poolTests on Windows 
must use ZSTD_ prefix in front of pthread types
so that they get properly translated for Windows.
 2019-04-11 12:03:42 -07:00
Yann Collet 8f56fa2f58 fixed poolTests 
added poolTests to all
 2019-04-11 09:50:39 -07:00
Yann Collet 30c26ab726 fixed minor warning 
unused variable when assert() turned off in fileio.c
 2019-04-10 17:48:56 -07:00
Yann Collet fbdd30d68e fixed cmake build script for test programs 2019-04-10 17:47:01 -07:00
Yann Collet 526ec646b7 alternate static assert 
to circumvent Visual's C4804 warning
 2019-04-10 16:05:02 -07:00
Yann Collet 9703a59121 fixed minor conversion warning 2019-04-10 15:54:55 -07:00
Yann Collet 885476fb5b FreeBSD_11 specific fix 
C11 mandates the definition of timespec_get() and TIME_UTC.
However, FreeBSD11 announce C11 compliance, but does not provifr timespec_get(),
breaking link stage for benchfn.
Since it does not provide TIME_UTC either, which is also required by C11,
test this macro: this will automatically rule out FreeBSD 11 for this code path
(it will use the backup C90 path instead, based on clock_t).

The issue seeems fixed in FreeBSD 12.
 2019-04-10 15:22:18 -07:00
Yann Collet 2fa4f2e246 updated Visual projects 
added timefn
 2019-04-10 15:07:36 -07:00
Yann Collet 2c6b14ed22 fixed Windows header 
cmake build script: added timefn
 2019-04-10 14:54:13 -07:00
Yann Collet 3d346579d8 no more need for CLOCK_MONOTONIC 2019-04-10 14:16:39 -07:00
Yann Collet 36d2dfd846 moved C11 code path to timespec_get 2019-04-10 14:15:11 -07:00
Yann Collet 4765929271 fixed perror include 2019-04-10 14:04:11 -07:00
Yann Collet 70802cde6d fixed error message 
using stdlib's perror()
 2019-04-10 14:01:18 -07:00
Yann Collet 4b8185c7fc tried a blindfix for unix + c11 2019-04-10 13:26:27 -07:00
Yann Collet 59a7116cc2 benchfn dependencies reduced to only timefn 
benchfn used to rely on mem.h, and util,
which in turn relied on platform.h.
Using benchfn outside of zstd required to bring all these dependencies.

Now, dependency is reduced to timefn only.
This required to create a separate timefn from util,
and rewrite benchfn and timefn to no longer need mem.h.

Separating timefn from util has a wide effect accross the code base,
as usage of time functions is widespread.
A lot of build scripts had to be updated to also include timefn.
 2019-04-10 12:37:03 -07:00
Yann Collet 094c000904 Merge branch 'dev' into benchfn 2019-04-10 11:57:05 -07:00
Yann Collet 90c0462d63 minor presentation refactoring 
and removed some // comment style
 2019-04-10 10:03:06 -07:00
Yann Collet fb756156f5 fixed static size of benchFnState 
added a static assert to ensure condition is respected on target platform
 2019-01-30 17:59:07 -08:00
Yann Collet cee31bbd25 updated fullbench and paramgrill 
to use new benchfn
returning a double.
 2019-01-25 15:38:08 -08:00
Yann Collet b8701102e0 fixed benchzstd to use new version of benchfn 
returning a double type
 2019-01-25 15:11:50 -08:00
Yann Collet 8c3ba44ada Merge branch 'fileio_lz4' into benchfn 2019-01-25 14:53:56 -08:00