Commit Graph
8993 Commits
Author SHA1 Message Date
Yann Collet 7fce9a41b5 change update rate to 12/11/11/11
better for large files, and sources with relatively "stable" entropy,
like silesia.tar.
slightly worse for files with rapidly changing entropy,
like Calgary.tar/.

Updated small files tests in fuzzer
2021-09-08 14:05:57 -07:00
Yann Collet ef78611c26 change update rate to 11/10/10/10
better for larger blocks,
very small inefficiency on small block.
2021-09-08 08:58:28 -07:00
Yann Collet 42a3ed752a removed frequency booster for stat initialization of btultra2
used to be necessary to counter-balance the fixed-weight frequency update
which has been recently changed for an adaptive rate (targeting stable starting frequency stats).
2021-09-08 07:56:43 -07:00
Yann Collet b096a5c626 updated regression tests 2021-09-07 09:55:14 -07:00
Yann Collet 08ceda3dfc new statistics update policy
small general compression ratio improvement for btopt+ strategies/
2021-09-04 00:52:44 -07:00
Yann Collet 23a9368c45 new starting offcode table for zstd_opt 2021-09-03 17:41:42 -07:00
Yann Collet 27a8bbe265 new initializer for ll price 2021-09-03 16:07:31 -07:00
Yann Collet f0fc8cb3e1 Disable console notification by default within the library
As a library, the default shouldn't be to write anything on console.
`cover` and `fastcover` have a `g_displayLevel` variable to control this behavior.
It's now set to 0 (no display) by default.
Setting notification to a higher level should be an explicit operation by a console application.
2021-09-03 13:44:07 -07:00
Yann Collet eab692211e removed pretty-print of sizes in benchmark
This is less appropriate for this mode :
benchmark is about accuracy,
it's important to read the exact values.
2021-09-03 12:51:02 -07:00
senandGitHub 71076b7a01 Merge pull request #2763 from senhuang42/opt_compiletime
Improve compile speed and binary size in `opt`
2021-09-02 11:59:02 -04:00
Yann ColletandGitHub a8cf85ad0a Merge pull request #2762 from facebook/level13
minor rebalancing of level 13
2021-09-01 20:32:53 -07:00
senandGitHub 4d61f10e23 Merge pull request #2761 from senhuang42/fse_wksp_fix
Add 8 bytes to FSE_buildCTable wksp
2021-09-01 17:09:45 -04:00
Sen Huang d88c1d95ce Remove inlining for opt 2021-09-01 16:58:57 -04:00
Yann Collet 40e44bd56d updated regression tests 2021-09-01 13:26:39 -07:00
Yann Collet 70d89e5a12 minor rebalancing of level 13
This new setup is slighly better on `silesia.tar` :
Ratio : 3.649 -> 3.655
Speed : 11.9 MB/s -> 12.2 MB/s
At the cost of more memory : 24 MB -> 32 MB
The new memory budget is a reasonable interpolation between neighboring levels 12 and 14:
level 12 : 24 MB
level 13 : 32 MB (increased from 24 MB)
level 14 : 48 MB
Window size remains unaffected (4 MB)
2021-09-01 13:05:10 -07:00
senhuang42 414e24becf Add 8 bytes to FSE workspace 2021-09-01 15:56:33 -04:00
Yann ColletandGitHub 6715096611 Merge pull request #2758 from facebook/qemu
added qemu tests
2021-08-31 09:56:50 -07:00
Yann Collet c1de65535f Merge branch 'dev' into qemu 2021-08-31 08:16:46 -07:00
Yann ColletandGitHub 6933ac67d3 Merge pull request #2757 from facebook/transferGA
Reduce test time on TravisCI
2021-08-31 07:40:21 -07:00
Yann Collet 333ecf6865 add powerpc qemu emulation 2021-08-30 06:37:50 -07:00
Yann Collet 2b27d07d06 attempt at adding m68k qemu tests
with optional success (for the time being)
2021-08-29 21:39:06 -07:00
Yann Collet 1e5c90cb5b remove qemu tests
that are being transfered to GA in #2758.
This represents a saving of ~25mn of cpu time on TravisCI.
2021-08-29 20:54:18 -07:00
Yann Collet 74b4171fb8 fix alignment condition in FSE_buildCTable
2-bytes alignment is enough for 16-bit fields
2021-08-29 19:05:04 -07:00
Yann Collet f21977c5e6 fix playTests.sh when EXE_PREFIX not null 2021-08-29 17:20:12 -07:00
Yann Collet 18191c85c9 adding optional QEMU_SYS 2021-08-29 16:43:32 -07:00
Yann Collet 1c97ec73d7 added qemu tests
running zstd library on emulated targets
2021-08-29 16:28:41 -07:00
Yann Collet b341aa2f95 remove versions-compatibility test from GA
since it fails on Github Actions specifically.

The test is run on TravisCI for the time being.
Its duration has been reduced to ~6mn anyway.
2021-08-29 15:47:04 -07:00
Yann Collet 72bd2a83a0 reduce length of scanbuild static analyzer test
This was ~30mn, by far the longest run on travisCI.
That's because it re-analyzes multiple times the same files (library files notably).
It also performs actions that make no sense for the static analyzer purpose,
such as building the single-file library.

Reduced time spent in this test by reducing its scope :
just build the CLI, and obviously the library along it.
These are the only ones that really deserve to be analyzed.

Unfortunately, it still results in a number of false positives when using newer versions of scanbuild
(each version of scanbuild generates a different list of false positives).
These will have to be fixed before transfering to Github Actions.
2021-08-29 15:26:31 -07:00
Yann Collet 7f37b8a547 accelerate versionsCompatibilityTest
by allowing parallel build of units,
and reducing optimization levels.

Parallel build is only effective on "recent" versions of `zstd`,
as previously, the list of units was passed as a list of source files,
which is something neither `make` nor `gcc` can parallelize.
So its impact is mildly effective (-20%).

Reducing optimization level to `-O1` makes compilation much faster.
It also makes runtime slower,
but in this test, compilation time dominates run time.
The savings are very significant (-50%).

On my test system, it reduces the length of this test from 13mn to 5mn.
2021-08-29 14:48:11 -07:00
Yann Collet ef69539849 transferred inter-versions compatibility tests to GA 2021-08-29 11:53:56 -07:00
senandGitHub 1f3fc1936c Merge pull request #2753 from senhuang42/better_error_msg
[easy] Fix zstd bench error message
2021-08-23 20:37:48 -04:00
senhuang42 dce48f53df Fix benchzstd error message 2021-08-23 19:10:16 -04:00
Yann ColletandGitHub be82a0ab8f Merge pull request #2746 from eli-schwartz/meson-fixup
meson fixups
2021-08-23 15:57:47 -07:00
Yann ColletandGitHub 18a20b3ad7 Merge pull request #2752 from facebook/hashLog3max
make ZSTD_HASHLOG3_MAX private
2021-08-20 12:51:17 -07:00
Yann Collet 2de42174bb make ZSTD_HASHLOG3_MAX private
This is an implementation detail,
it doesn't belong to public space (zstd.h).
2021-08-20 09:52:42 -07:00
senandGitHub ae998544de Merge pull request #2750 from senhuang42/sb_compress
Improve branch misses on FSE symbol spreading
2021-08-20 12:47:24 -04:00
senhuang42 da095ed899 Improve branch misses on FSE symbol spreading 2021-08-18 10:22:22 -07:00
Eli Schwartz 9748608aeb meson: set the symbol visibility of the shared library to hidden
This matches the Makefile build. Due to one private xxhash symbol in use
by the program, it recompiles a private copy of xxhash.

Due to the test binaries making extensive (?) use of private symbols, it
doesn't even attempt to link to shared libzstd, and instead, all of the
original object files are added to libtestcommon itself for private
linkage. This, too, matches the Makefile build.

Ref. #2261
2021-08-09 22:03:17 -04:00
Eli Schwartz d95a3f5097 meson: fix warnings in build files
meson prefers that project-level options for Wall/Wextra/pedantic be
used, rather than hardcoding raw flags in add_project_arguments. If you
do the latter anyway, it raises a meson warning.

Set the default options for the project to use all this.

Also move the -Werror comment to the project default options with
appropriate format, but leave it commented out since it does not work.
2021-08-09 20:17:00 -04:00
Eli Schwartz ee3355df30 meson: link tests with a convenience library of sources used by multiple binaries
This saves repeatedly compiling them, resulting in 16 fewer compile
steps, in exchange for one `ar` step.
2021-08-09 20:15:44 -04:00
senandGitHub 6a258043f5 Merge pull request #2692 from senhuang42/rebalance_clevel
[RFC] Rebalance compression levels
2021-08-06 12:51:31 -04:00
Sen Huang 539b3aab9b Optimize 32-bit VecMask_next() 2021-08-04 17:14:58 -04:00
Felix HandteandGitHub ae131282f5 Merge pull request #2742 from felixhandte/set-mtime
Set mtime on Output Files
2021-08-04 16:44:46 -04:00
senhuang42 e411040ea1 Add 64 row entry support for lazy 2021-08-04 16:19:12 -04:00
senhuang42 31820e032c Rebalance clevels for lazy 2021-08-04 16:18:52 -04:00
senandGitHub f9b034086a Merge pull request #2732 from senhuang42/huf_sort_improvement
[HUF] Improve Huffman sorting algorithm
2021-08-04 16:07:56 -04:00
W. Felix Handte a8f4612eab Remove sleep()s in Test; Replace with Artificial mtime
This behavior of `touch` is standardized in POSIX, so it should be available.
2021-08-04 15:56:46 -04:00
W. Felix Handte 9cd6c1ff4d Update mtime and atime for Written Files 2021-08-04 14:49:56 -04:00
W. Felix Handte a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
senhuang42 aa1957477b Improve Huffman sorting algorithm 2021-08-04 12:43:34 -04:00