Commit Graph
100 Commits
Author SHA1 Message Date
Yann Collet 4cc5e2818a complete changelog with #2885 2021-12-09 09:53:45 -08:00
Yann Collet 3d738307b4 Update changelog for v1.5.1 2021-12-08 16:55:38 -08:00
Yann Collet 3ce265fea8 remove offending static assert lines
no idea why visual + clang-cl + appveyor don't like them,
I've not been able to reproduce the issue locally,
but these static assert are very unlikely to deliver a useful signal,
I can't imagine a situation where they will be wrong,
and if they are, then a ton of other things will be broken way before reaching that point.
2021-12-08 15:05:17 -08:00
Yann Collet 16241b7d26 altered copyright title 2021-12-08 13:18:41 -08:00
Yann Collet a9cd6164d7 removed declarations of XXH3 symbols when XXH_NO_XXH3 is defined
on top of implementations, which were already scoped out.
2021-12-08 12:56:16 -08:00
Yann Collet 27e706de88 replaces malloc / free / memcpy by Zstandard's version 2021-12-08 12:51:04 -08:00
Yann Collet fb3522a3fe fixed very minor cast warning under cygwin 2021-12-08 09:48:56 -08:00
Yann Collet 1c7d2c4dd5 updated xxHash to latest v0.8.1
with minor modifications directly embedded in source :
- does not compile XXH3
- namespace emulation (ZSTD_ prefix)

Incidentally fix #2824
2021-12-07 21:16:15 -08:00
Yann Collet 30b9db8ae4 changed macro name to ZSTD_ALIGNOF
for better consistency
2021-12-02 12:57:42 -08:00
Yann Collet 1d025d871b bound alignment backup to sizeof(void*) 2021-12-02 11:30:03 -08:00
Yann Collet 80a13fd645 move the alignment macro to compiler.h
because mem.h is dropped in the Linux kernel.

Changed macro definition order (gcc/clang/msvc before c11)
due to a limitation in the kernel source builder.

Changed the backup to sizeof(),
reverting to previous behavior when no support of alignof() is detected.
2021-12-02 11:20:01 -08:00
Yann Collet ffbdc8ac57 m68k CI tests on GA are now compulsory 2021-12-02 10:30:04 -08:00
Yann Collet 39dced092e fix align conditions for huf_compress 2021-12-01 23:02:00 -08:00
Yann Collet e89e847820 added alignment test
and fix an incorrect alignment check in cwksp which was failing on m68k
2021-12-01 17:16:36 -08:00
Yann Collet a71eed360b removed lib/Makefile preamble
now included from libzstd.mk
2021-12-01 16:54:59 -08:00
Yann Collet 3f64b31585 Merge branch 'dev' into tomerge2051 2021-12-01 15:29:49 -08:00
Yann Collet 49d578763d reduce storage requirement
51 MB seems excessive for CI storate and considering the nature of the test.

(note : maybe we should consider using `/tmp` for files generated during tests,
as tmpfs is typically using RAM, thus preserving storage.)
2021-12-01 15:10:55 -08:00
Yann Collet 23dd28df67 minor improvements to benchmark display 2021-10-26 23:23:30 -07:00
Yann Collet 2b2a5c449a fix minor cast warning 2021-10-26 08:38:17 -07:00
Yann Collet 518f06b281 added minimum for decoder buffer
also : introduced macro BOUNDED()
2021-10-26 08:21:31 -07:00
Yann Collet 082d6c6775 separate compression level tables into their own files
that's clearer than finding the tables somewhere in the middle of `compress.c`.

Also, down the line, it may potentially allows zstd to feature adjusted tables depending on target cpu.
2021-10-25 08:49:54 -07:00
Yann Collet 02be2a830f build macro ZSTD_DECODER_INTERNAL_BUFFER
just to make the topic more accessible for potential users.
2021-10-25 08:09:04 -07:00
Yann Collet 2ed14c2476 minor : fix comment
provide correct reasons to include zstd_internal.h
2021-09-26 08:44:18 -07:00
Yann Collet ecb2daeaa9 added link to RFC 2021-09-26 06:58:10 -07:00
Yann Collet 70a80b6be8 minor : makes the "stable" property of Zstandard format more prominent 2021-09-26 06:48:52 -07:00
Yann Collet fa2a4d77c7 constify MatchState* parameter when possible
turns out, it's possible to constify MatchState* parameter
in some parts of the binary tree algorithm,
making it a pure read-only parameter,
as opposed to a mutable state.

This is supposed to be helpful for both maintenance and the compiler.
2021-09-23 08:27:44 -07:00
Yann Collet 70b36c2308 update zstdgrep doc
to mention ripgrep alternative which transparently supports zstd-compressed files.
2021-09-22 14:30:06 -07:00
Yann Collet 2832bbbbdc emphasize usage of -r in documentation
notably as a way to overcome shell expansion limitations,
notably in a scenario of dictionary training.
2021-09-22 14:00:20 -07:00
Yann Collet 5449ede2e6 make automated-benchmarking faster
by employing parallel compilation of object files.
2021-09-08 15:12:28 -07:00
Yann Collet b7f46ebc23 use ZSTD_memcpy() for better portability
notably within kernel space
2021-09-08 14:45:53 -07:00
Yann Collet 4f0b1b9ee5 update regression tests 2021-09-08 14:37:42 -07:00
Yann Collet 7fce9a41b5 change update rate to 12/11/11/11
better for large files, and sources with relatively "stable" entropy,
like silesia.tar.
slightly worse for files with rapidly changing entropy,
like Calgary.tar/.

Updated small files tests in fuzzer
2021-09-08 14:05:57 -07:00
Yann Collet 42a3ed752a removed frequency booster for stat initialization of btultra2
used to be necessary to counter-balance the fixed-weight frequency update
which has been recently changed for an adaptive rate (targeting stable starting frequency stats).
2021-09-08 07:56:43 -07:00
Yann Collet 40e44bd56d updated regression tests 2021-09-01 13:26:39 -07:00
Yann Collet c1de65535f Merge branch 'dev' into qemu 2021-08-31 08:16:46 -07:00
Yann Collet 2de42174bb make ZSTD_HASHLOG3_MAX private
This is an implementation detail,
it doesn't belong to public space (zstd.h).
2021-08-20 09:52:42 -07:00
Yann Collet 139a2f1e0c remove invalid test
`--mt` is no longer supported by `zstreamtest`
(relevant API entry point has been removed from `libzstd`).

fix #2701
2021-06-24 09:07:55 -07:00
Yann Collet 61afa154cd improve tar compatibility
This patch is supposed to improve compatibility with less featured tar variants
"when the tar program used does not support historical options (without hyphen) nor the '-z' option."

Patch proposed by Antonio Diaz Diaz
2021-05-15 23:09:42 -07:00
Yann Collet 0b0b62d1cf minor mention of RFC8878
more recent update
2021-05-15 23:04:46 -07:00
Yann Collet 988beb3000 updated meson test
hopefully, bionic will have a more recent version of python
required to install meson.
2021-05-13 09:43:23 -07:00
Yann Collet 8fae35591e Merge branch 'dev' of github.com:facebook/zstd into dev 2021-05-12 13:12:30 -07:00
Yann Collet cb0cad9b79 reduce Max nb Workers to 64 in 32-bit mode
and restored limit to 256 when in 64-bit mode
(it was reduced to 200 to give more room for 32-bit).

This should fix test instability issues
using lot of threads in 32-bit environments.
2021-05-12 13:10:25 -07:00
Yann Collet 439e58d060 improved gcc-9 and gcc-10 decoding speed
the new alignment setting is better for gcc-9 and gcc-10
by about ~+5%.

Unfortunately, it's worse for essentially all other compilers.

Make the new alignment setting conditional to gcc-9+.
2021-05-08 00:01:01 -07:00
Yann Collet 6755baf940 update decoder hot loop alignment
This seems to bring an additional ~+1.2% decompression speed
on average across 10 compilers x 6 scenarios.
2021-05-07 15:18:16 -07:00
Yann Collet 4d9caa4928 Merge branch 'd_prefetch_refactor' of github.com:facebook/zstd into d_prefetch_refactor 2021-05-07 11:30:44 -07:00
Yann Collet 1db5947591 improve decompression speed of long variant by ~+5%
changed strategy,
now unconditionally prefetch the first 2 cache lines,
instead of cache lines corresponding to the first and last bytes of the match.

This better corresponds to cpu expectation,
which should auto-prefetch following cachelines on detecting the sequential nature of the read.

This is globally positive, by +5%,
though exact gains depend on compiler (from -2% to +15%).
The only negative counter-example is gcc-9.
2021-05-07 11:26:14 -07:00
Yann Collet 17b9e43c7d do not install g++ 2021-05-06 21:53:30 -07:00
Yann Collet ee425faaa7 Merge branch 'dev' into d_prefetch_refactor 2021-05-06 19:49:26 -07:00
Yann Collet f44c720fa8 added a paragraph on coding style 2021-05-06 18:40:25 -07:00
Yann Collet bd547232bc switch to clang 2021-05-06 16:07:44 -07:00
Yann Collet ee65162655 Merge branch 'dev' into fasterCygwin 2021-05-06 16:06:00 -07:00
Yann Collet 2e76bd7d10 attempt to make Appveyor's Cygwin test faster
Cygwin is the longest Appveyor test
Appveyor is typically the CI which finish last
2021-05-06 08:47:45 -07:00
Yann Collet df05b2ba7c fix --filelist compatibility with Windows cr+lf line ending 2021-05-05 18:01:55 -07:00
Yann Collet 9750f3c87b improved benchmark experience on Windows
benchmark results are not progressively displayed on Windows terminal.
For long benchmark sessions, nothing is displayed,
until the end, where everything is flushed.

Force display to be flushed after each update.
Updates happen roughtly every second, or even less,
so it's not a substantial workload.
2021-05-05 16:52:21 -07:00
Yann Collet 8cde167a27 Merge branch 'dev' into d_prefetch_refactor 2021-05-05 09:13:38 -07:00
Yann Collet 455fd1a067 updated documentation regarding minimum job size 2021-05-05 09:03:11 -07:00
Yann Collet 1026b9fa10 fix rsyncable mode 2021-05-04 15:59:27 -07:00
Yann Collet 8f86c29c06 allow jobSize to be as low as 512 KB
previous lower limit was 1 MB.

Note : by default, the lowest job size is 2 MB, achieved at level 1.
Even lower job sizes can be achieved by manipulating this value directly,
or manually modifying window sizes to lower amounts.

Updated unit test to ensure that this new limit works fine
(test would fail with previous 1 MB limit).
2021-05-04 11:02:55 -07:00
Yann Collet 8aafbd3604 Documented minimum version numbers
Any stable API entry point introduced after v1.0
should be documented with its minimum version number.

Since PR fixes this requirement
updating mostly new entry points since v1.4.0
and newly introduced ones for future v1.5.0.
2021-05-04 09:05:22 -07:00
Yann Collet 0f99a0d987 fix man page typo
fix #2553 detected by @jwilk
2021-03-24 05:55:04 -07:00
Yann Collet 9fb4a42c7b fix #2549 2021-03-20 17:29:41 -07:00
Yann Collet f5434663ea Refactor prefetching for the decoding loop
Following #2545,
I noticed that one field in `seq_t` is optional,
and only used in combination with prefetching.
(This may have contributed to static analyzer failure to detect correct initialization).

I then wondered if it would be possible to rewrite the code
so that this optional part is handled directly by the prefetching code
rather than delegated as an option into `ZSTD_decodeSequence()`.

This resulted into this refactoring exercise
where the prefetching responsibility is better isolated into its own function
and `ZSTD_decodeSequence()` is streamlined to contain strictly Sequence decoding operations.
Incidently, due to better code locality,
it reduces the need to send information around,
leading to simplified interface, and smaller state structures.
2021-03-19 15:48:17 -07:00
Yann Collet 2fa4c8c405 added code comments for new API ZSTD_seekTable 2021-03-03 22:54:04 -08:00
Yann Collet 6e390ced1f Merge branch 'seekTable' of github.com:facebook/zstd into seekTable 2021-03-03 22:44:38 -08:00
Yann Collet 16ec1cf355 added test case for seekTable API
and simple roundtrip test
2021-03-03 18:56:23 -08:00
Yann Collet 6c0bfc468c fixed wrong assert condition 2021-03-03 15:30:55 -08:00
Yann Collet a1d7b9d654 fixed gcc conversion warnings 2021-03-03 15:17:12 -08:00
Yann Collet 24d59a655d Merge branch 'dev' into seekTable 2021-03-03 15:08:40 -08:00
Yann Collet ac95a30455 various minor style fixes 2021-03-02 16:03:18 -08:00
Yann Collet 029f974ddc strengthen compilation flags 2021-03-02 15:43:52 -08:00
Yann Collet c7e42e147b fixed const guarantees
read-only objects are properly const-ified in parameters
2021-03-02 15:24:30 -08:00
Yann Collet a80b10f5e6 fix potential leak on exit 2021-03-02 15:03:37 -08:00
Yann Collet 3d91ab74a1 disabled icc tests on Github Actions 2021-02-23 13:25:08 -08:00
Yann Collet 08fef036ec update FreeBSD image to latest 12.2 2021-02-23 10:40:44 -08:00
Yann Collet f2c0312889 removing signed integer overflow exception from ubsan tests 2021-02-19 16:30:06 -08:00
Yann Collet 9b0772177c fix ubsan test errors
allows recovering from pointer overflow
2021-02-19 10:42:05 -08:00
Yann Collet b9748757b0 fixed minor cast warning 2021-02-05 09:55:54 -08:00
Yann Collet 134be2731a removed flackey gcc6 tests
from github actions.
replaced by equivalent "current" gcc tests when it makes sense
2021-02-05 09:25:33 -08:00
Yann Collet 0bad3e5c0f parallel make build on linux
fix #2474
2021-01-18 11:33:03 -08:00
Yann Collet 3324e87cff Added library version check 2021-01-07 10:37:27 -08:00
Yann Collet 2901b5e675 Merge branch 'dev' into cli-dll 2021-01-07 10:24:09 -08:00
Yann Collet cefdc023f7 The CLI can be linked to libzstd dynamic library
invoking target zstd-dll
2021-01-06 18:00:24 -08:00
Yann Collet 890d85bdb4 removed CLI dependency to legacy unsafe function
makint the CLI ons step closer to being linkable to the dynamic library
2021-01-06 16:19:42 -08:00
Yann Collet 9866148e22 removed redundant tests
clang v3.8 tests are either flacky or redundant,
prefer using clang-latest.
2021-01-06 15:40:20 -08:00
Yann Collet a0835b5bab Merge branch 'dev' into nomsanfuzz 2020-12-29 11:45:42 -08:00
Yann Collet ff2f888d56 fixed one more minor cast issue
can't use address calculation with `void*`
2020-12-29 11:44:37 -08:00
Yann Collet 3fd21d5438 Merge branch 'dev' into nomsanfuzz 2020-12-28 14:08:45 -08:00
Yann Collet 7f8be046b9 fixed minor warnings introduced in #2439 2020-12-28 14:07:31 -08:00
Yann Collet f1585fefa2 remove flackey msan ossfuzz test
while waiting for it to be fixed
2020-12-28 11:18:58 -08:00
Yann Collet f1225b186e try to fix 32-bit test on github actions
for some reasons, this test fails at _installing_ 32-bit dependencies
using the exact same command that actually works in other tests !!?

It's unclear why it fails repeateadly for this test only.
Try another way to install dependencies to fix that.
2020-12-21 15:47:20 -08:00
Yann Collet f2ac2b7bcf try to fix cross-compiler tests 2020-12-21 15:43:14 -08:00
Yann Collet 585196353d fix ppc64 build on circleci 2020-12-20 21:44:40 -08:00
Yann Collet 8233f55df8 verbose cross-compile tests on circleci
for better diagnosis
2020-12-20 21:31:58 -08:00
Yann Collet 52aa7f47dd updated clang+msan test 2020-12-20 18:29:36 -08:00
Yann Collet 9a9d3f76c4 fixed zstd+sanitizer build 2020-12-20 17:53:04 -08:00
Yann Collet f9884036c2 fixed zstd recipe 2020-12-20 17:19:23 -08:00
Yann Collet 9648bf027b try to keep libzstd.a "as is" once created
to be compatible with scenarios such as
`make -j allmost`
2020-12-20 17:10:57 -08:00
Yann Collet eacf1b3bc3 streamlined github action test 2020-12-20 15:03:59 -08:00
Yann Collet f78917cdd3 streamline zlibwrapper makefile
making better usage of default build rules
2020-12-20 12:53:30 -08:00
Yann Collet 7c495e8ea2 updated version number to v1.4.8 2020-12-18 15:52:11 -08:00