ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,782 Commits 17 Branches 77 Tags
13449d7ce1da1fcf037600e33afe4fa6bc85d12e
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 7ef6d7b36c deeper prefetching pipeline for decompressSequencesLong 
pipeline increased from 4 to 8 slots.
This change substantially improves decompression speed when there are long distance offsets.
example with enwik9 compressed at level 22 :
gcc-9 : 947 -> 1039 MB/s
clang-10: 884 -> 946 MB/s

I also checked the "cold dictionary" scenario,
and found a smaller benefit, around ~2%
(measurements are more noisy for this scenario).
 2021-05-05 10:04:03 -07:00
Yann Collet 713d4953f7 fixed gcc-7 conversion warning 2021-03-03 18:00:41 -08:00
Yann Collet 0d793a675a removed internal dependency from CLI 
ZSTD_cycleLog() is a very short function,
creating a rather large dependency onto libzstd's internal just for it is overkill.
Prefer duplicating this 2-lines function.

This PR makes the zstd CLI one step closer to being linkable to the dynamic library (see #2450)
More steps are still needed to reach this goal.
 2021-01-06 01:35:52 -08:00
Yann Collet fed1c62571 fix gcc10 warnings 
gcc10 doesn't like its own strncpy
 2020-11-30 04:44:37 -08:00
Yann Collet 95e74616d5 fix multiple minor conversion warnings 
unrelated to #2386, just cleaning up while I'm updating this file ...
 2020-11-06 09:57:05 -08:00
Yann Collet 2769e4d459 fix incorrect assert 
fix #2386, reported by @Neumann-A
 2020-11-06 09:44:04 -08:00
Yann Collet f6ecf1568f minor Makefile refactor 
hopefully improving readability
 2020-10-28 09:39:15 -07:00
Yann Collet 89b961ea46 simplified silent mode maintenance 2020-10-23 10:41:17 -07:00
Yann Collet ffe8d9e428 fix partial lib test 2020-10-23 10:27:12 -07:00
Yann Collet b5d4728713 simplified silent mode 2020-10-23 10:22:52 -07:00
Yann Collet a7ad05bf57 fixed building libzstd with manual BUILD_DIR 
and when HASH is not found
 2020-10-23 10:14:04 -07:00
Yann Collet 91a8cb9559 fix DEBUGLEVEL redefinition from tests/ 2020-10-22 00:20:40 -07:00
Yann Collet 494f7169ed fix directory creation for Windows' libzstd 2020-10-22 00:15:31 -07:00
Yann Collet ca75da8fa3 fix test 
DEBUGLEVEL redefinition
 2020-10-21 23:51:13 -07:00
Yann Collet dd24496951 programs/zstd also automatically generate object dir per conf 
same rules as lib/libzstd
can also be controlled via HASH and BUILD_DIR
 2020-10-21 23:38:33 -07:00
Yann Collet 01ecad2326 zstd in programs/ can also receive a DEBUGLEVEL value 
simplify tests : only set DEBUGLEVEL, not the flags directly
 2020-10-21 23:13:46 -07:00
Yann Collet 0f8ee5c51e Merge branch 'dev' into libzstd_autoconf 2020-10-21 22:36:09 -07:00
Yann Collet d0436b2a45 automatically detect configuration changes 
Makefile now automatically detects modifications of compilation flags,
and produce object files in directories dedicated to this compilation flags.
This makes it possible, for example, to compile libzstd with different DEBUGLEVEL.
Object files sharing the same configration will be generated into their dedicated directories.

Also : new compilation variables
- DEBUGLEVEL : select the debug level (assert & traces) inserted during compilation (default == 0 == release)
- HASH : select a hash function to differentiate configuration (default == md5sum)
- BUILD_DIR : skip the hash stage, store object files into manually specified directory
 2020-10-21 19:22:45 -07:00
Yann Collet 8a453a34c5 automatic %.h header dependency tracking 
also : BUILD_DIR can be manually specified
 2020-10-21 17:25:07 -07:00
Yann Collet 2b99bc29bf consolidated vpath 2020-10-21 04:01:01 -07:00
Yann Collet e8eb2939fe store %.o object files into obj/ 
both static and dynamic libraries have their own object directory
 2020-10-21 03:44:38 -07:00
Yann Collet 3e519be965 minor cleaning 2020-10-21 03:22:27 -07:00
Yann Collet 911dbdbb4b build libzstd.so from object files 
%.o object files generated for dynamic library
must be different from those generated for static library.

Due to this difference, %.o were so far only generated for the static library.
The dynamic library was rebuilt from %.c source.

This meant that, for every minor change, the entire dynamic library had to be rebuilt.

This is fixed in this PR :
only the modified %.c source get rebuilt.
 2020-10-20 22:19:57 -07:00
Yann Collet 684f8ea970 decouple zstd object files from lib/ 
That was a subtle one :
VPATH is affecting search for both %.c source and %.o object files.
This meant that, when an object file already exists in lib/,
it's used in programs/,
even though programs/ is supposed to generate its own %.o object files.

With the new vpath directive, this is no longer the case :
the search is only activated for %.c source files.
Now, local programs/%.o are always generated
even if equivalent ones are already created in lib/.

It more clearly guarantees that lib/ and programs/ can use different compilation directives
without mixing resulting %.o object files.
 2020-10-20 18:16:11 -07:00
Yann Collet a9f7e77d44 fixed travis tests 2020-10-20 17:08:14 -07:00
Yann Collet eb1959d926 fix .deps list 
and minor simplications
 2020-10-16 11:08:52 -07:00
Yann Collet c355d149de support verbose build output with V=1 2020-10-16 00:21:49 -07:00
Yann Collet b6b87c2b77 track header dependencies during build 
modifying a header file will now trigger recompilation of related *.c units
 2020-10-16 00:18:16 -07:00
Yann Collet 80cee8d3fe faster rebuild of zstd 
Building the zstd CLI costs time.
Some part of it is incompressible, leading to substantial iteration delay when testing code modifications.
That's mainly because all source files from the library must be rebuilt from source every time.

The main reason we don't build the CLI from library object files
is that we can't just build the object directly in the lib/ directory
(which they would by default)
since they use different compilation flags.
Specifically, the CLI enables multithreading, while the library doesn't (by default).

This is solved in this commit, by generating the object files locally.
Now, the CLI and the library can employ different sets of flags, without tripping over each other.
All library object files are generated directly into programs/ dir.
This works because no 2 source files have the same name.

Now, modifying a file doesn't require to recompile the entire lib, just the modified files.
The recipe is also compatible with `-j` parallel build, leading to large build time reductions on multi-core systems.
 2020-10-16 00:01:41 -07:00
Yann Collet 5618e000bd update fullbench for compressStream2() 
makes it possible to measure scenarios such as #2314
 2020-09-21 07:19:20 -07:00
Yann Collet 16b353b207 minor doc clarification regarding MT parameters 2020-07-11 02:16:52 -07:00
Yann Collet dd026ca505 re-inforced tests for initStaticCCtx 
ensure that `estimateCCtxSize()` works as intended.
 2020-05-09 11:30:45 -07:00
Yann Collet cf854f4660 support for verbose make 
A commonly accepted makefile idiom is V=1 or VERBOSE=1
to request the printing of all commands.

This is not "default" though, and must be manually added.

Example :
Before :
```
make libzstd
compiling dynamic library 1.4.5
creating versioned links

make libzstd V=1
compiling dynamic library 1.4.5
creating versioned links
```

After :
```
make libzstd
compiling dynamic library 1.4.5
creating versioned links

make libzstd V=1
compiling dynamic library 1.4.5
cc -DXXH_NAMESPACE=ZSTD_ -DZSTD_LEGACY_SUPPORT=5 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -O3 common/debug.c common/entropy_common.c common/error_private.c common/fse_decompress.c common/pool.c common/threading.c common/xxhash.c common/zstd_common.c compress/fse_compress.c compress/hist.c compress/huf_compress.c compress/zstd_compress.c compress/zstd_compress_literals.c compress/zstd_compress_sequences.c compress/zstd_compress_superblock.c compress/zstd_double_fast.c compress/zstd_fast.c compress/zstd_lazy.c compress/zstd_ldm.c compress/zstd_opt.c compress/zstdmt_compress.c decompress/huf_decompress.c decompress/zstd_ddict.c decompress/zstd_decompress.c decompress/zstd_decompress_block.c deprecated/zbuff_common.c deprecated/zbuff_compress.c deprecated/zbuff_decompress.c dictBuilder/cover.c dictBuilder/divsufsort.c dictBuilder/fastcover.c dictBuilder/zdict.c legacy/zstd_v05.c legacy/zstd_v06.c legacy/zstd_v07.c -shared -fPIC -fvisibility=hidden -Wl,-soname=libzstd.so.1 -o libzstd.so.1.4.5
creating versioned links
ln -sf libzstd.so.1.4.5 libzstd.so.1
ln -sf libzstd.so.1.4.5 libzstd.so
```
 2020-05-07 08:04:10 -07:00
Yann Collet 54144285fd small speed improvement for strategy fast 
gcc 9.3.0 :
kennedy : 459 -> 466
silesia : 360 -> 365
enwik8  : 267 -> 269

clang 10.0.0 :
kennedy : 436 -> 441
silesia : 364 -> 366
enwik8  : 271 -> 272
 2020-05-07 06:15:58 -07:00
Yann Collet 6f62a9caaa fixed zstd-nolegacy target 
when compiled as part of allVariants

(though I'm unsure why it was working before ...)
 2020-04-29 11:56:21 -07:00
Yann Collet 3746693b71 added ppc64le tests on travis 
on real hardware
 2019-12-15 01:30:12 -08:00
Yann Collet 52dc6ff00e travis: updated mingw cross compilation test 
- only compiles zstd CLI
- enforces zero warning tolerance policy

note : mingw compilation of the library complains (warning) about -fPIC flag.
It mostly ignores it, so it's not a big issue.
Still, it would need to be fixed if library cross-compilation becomes part of CI.
At this stage, it's unclear if this is a scenario.
 2019-11-27 14:39:43 -08:00
Yann Collet d6e32afa0a added mingw cross compilation tests 
based on a script in RocksDB project.

also : pushed qemu aarch64 tests to master/cron,
since there are already hardware aarch64 tests on dev.
 2019-11-27 12:44:02 -08:00
Yann Collet 54bd39abdf Revert "added clang test in travis" 
This reverts commit 4909a341db.
 2019-01-20 11:13:01 -08:00
Yann Collet 4909a341db added clang test in travis 2019-01-20 00:11:07 -08:00
Yann Collet 2a6aa6be5f updated clang tests 
target clangbuild
 2019-01-19 23:40:41 -08:00
Yann Collet a1394399b4 fixed minor conversion warnings in examples/ 2019-01-19 23:38:20 -08:00
cyan4973 e85b4c56b2 speed up appveyor tests 2018-07-30 17:13:13 +02:00
cyan4973 3f535007e4 fix %zu support under minGW 
and relevant test on Appveyor
 2018-07-30 16:56:18 +02:00
cyan4973 c738a2c795 ensure appveyor test fails due to formatting error 
to catch %zu incompatibility
 2018-07-30 16:44:20 +02:00
cyan4973 aade1e5904 Merge branch 'dev' into fix1241 2018-07-30 16:30:35 +02:00
cyan4973 3d4b09a5aa support %zu under mingw 2018-07-30 16:29:20 +02:00
cyan4973 7d1bc9cc8c fix minor conversion warning 2018-07-18 16:10:23 +02:00
cyan4973 0840d02ecf fix make test on Linux 
MOREFLAGS+= doesn't work on Linux
 2018-07-17 19:19:48 +02:00
cyan4973 b8a81a988c added a test to be played on travis 
to check a make variable construction
 2018-07-17 19:02:17 +02:00
cyan4973 9597b438e9 fix #1241 
Ensure that first input position is valid for a match
even during first usage of context
by starting reference at 1
(avoiding the problematic 0).
 2018-07-17 18:52:57 +02:00
cyan4973 53e1f0504e zstdmt debug traces compatibles with mingw 
since mingw does not have `sys/times.h`,
remove this path when detecting mingw compilation.
 2018-07-17 14:39:44 +02:00
cyan4973 f741fb8fcd minor fixes for MSYS2 compilation 2018-06-26 01:22:45 -07:00
cyan4973 ae0b7ffa0a made Visual Studio compatible with DEBUGLEVEL >= 2 2018-06-20 09:45:02 -07:00
cyan4973 b095bffac6 ignore Windows build/test artefacts 2018-05-01 05:45:46 -07:00
cyan4973 62487b5e76 fixed decoding bogus lz4 frame 
FIO would keep presenting data after an LZ4F decoding error
resulting in a NULL pointer dereference
when associated with older liblz4 version (< v1.8.1.2)
 2018-04-23 18:50:16 -07:00
cyan4973 4b26306cb8 blindfix : fullbench's one-time leak, detected by valgrind 2017-07-01 08:03:59 -07:00
cyan4973 c07e43c2b5 added --show-leak-kind=all to valgrind tests 2017-07-01 07:05:11 -07:00
cyan4973 21fdf97e00 Merge branch 'dev' into fullbench 2017-07-01 07:01:08 -07:00
cyan4973 1bafe393e4 fix : ZSTDMT_compressStream_generic() can accept NULL input 
also : converge implementations towards new version of ZSTDMT_compressStream_generic()
 2017-07-01 06:59:24 -07:00
cyan4973 b5bb7c6d95 fixed Visual compilation of fullbench-dll 2017-06-29 19:59:37 -07:00
Yann Collet f99c2c1a22 reduced --no-big-tests even more 
to pass tests on qemu-aarch64
 2017-06-21 23:35:58 -07:00
Yann Collet f5deae8a67 new api fuzzer tests : frame parameters are randonly set 2017-06-18 23:41:38 -07:00
Yann Collet ea3630a889 new api fuzzer set : dictionary is randomly set 2017-06-18 23:31:55 -07:00
Yann Collet 8dee0ec99e new api fuzzer tests : compression parameters are randomly set 2017-06-18 23:25:15 -07:00
cyan4973 c59162e053 minor fix for -Wdocumentation 2017-06-05 00:12:13 -07:00
cyan4973 8bcbf42617 fixed g++ prototype mismatch 2017-06-04 23:52:00 -07:00
cyan4973 51235393e3 fixed fullbench project for VS2010+ 2017-06-04 22:06:25 -07:00
Yann Collet 9a691e0f55 fixed visual warnings 2017-05-31 01:17:44 -07:00
Yann Collet b6dec4c3ae fixed minor cast warning 2017-05-27 17:09:06 -07:00
Yann Collet e071159101 mtctx->jobs allocate its own memory space 
to make ZSTDMT_CCtx_s size predictable
so that it can be included in CCtx
 2017-05-27 00:21:33 -07:00
Yann Collet b8136f019a static dctx is incompatible with legacy support 
documented, and runtime tested
 2017-05-27 00:03:08 -07:00
cyan4973 613087c02b Silence zlib detection routine 
When it fails, $(CC) sends error message into stderr
redirected to /dev/null
 2017-02-04 23:36:12 -08:00
cyan4973 6f31b76d1d removed gzstd test from travis 
this target does no longer exist
 2017-02-04 21:39:41 -08:00
cyan4973 f0ffa237da fixed VS2008 project 2017-01-20 15:24:06 -08:00
cyan4973 326575c3a3 fixed VS2010 project 2017-01-20 14:49:44 -08:00
cyan4973 2e3b659ae1 fixed minor warnings (Visual, conversion, doxygen) 2017-01-20 14:43:09 -08:00
cyan4973 5fba09fa41 updated util's time for Windows compatibility 
Correctly measures time on Posix systems when running with
Multi-threading

Todo : check Windows measurement under multi-threading
 2017-01-20 12:57:31 -08:00
Yann Collet 08611a9f64 removed make artefact 2016-12-07 14:04:39 +01:00
Yann Collet 9dfd0af164 ZBUFF_ as a wrapper to ZSTD streaming API. 2016-12-06 17:16:41 +01:00
Yann Collet a0d4031e7c fixed cmake build 2016-12-06 09:46:14 +01:00
Yann Collet 73ab42252b fixed zbuff in VS2005 projects 2016-12-06 09:08:43 +01:00
Yann Collet 2f902f946c fixed zlibwrapper use of xxh 2016-12-06 08:52:53 +01:00
Yann Collet f586bdfe23 fixed fuzzer test 2016-12-06 06:11:46 +01:00
Yann Collet 83543a7b26 updated NEWS 2016-09-28 00:15:03 +02:00
Yann Collet 47094ea66b added comment on filePos 2016-09-26 18:03:33 +02:00
Yann Collet cf409a7e2a fixed : init*_advanced() followed by reset() with different pledgedSrcSiz 2016-09-26 16:41:05 +02:00
Yann Collet 2f2639438a zstreamtest can fuzztest pledgedSrcSize 2016-09-26 14:06:08 +02:00
Yann Collet 58d5dfea54 zstreamtest uses ZSTD_reset?Stream 2016-09-25 01:34:03 +02:00
Yann Collet 21412bb3f6 Merge branch 'dev' of github.com:Cyan4973/zstd into dev 2016-09-22 15:57:56 +02:00
Yann Collet 51f4d566c2 small decompression speed boost for very small data 2016-09-22 15:57:28 +02:00
Yann Collet 47f3697f32 added error check 2016-09-20 11:59:12 +02:00
Yann Collet 86bdcd83c1 added error checking 2016-09-20 11:54:29 +02:00
Yann Collet 0704df3259 fixed cmake test 2016-09-19 16:55:35 +02:00
Yann Collet dbe70bad48 completed change from projects to build 2016-09-19 15:08:43 +02:00
Yann Collet 4c9a4c18a9 changed projects to build 2016-09-19 14:58:14 +02:00
Yann Collet 1eb2fdc74f bumped version number 2016-09-18 12:21:47 +02:00
Yann Collet 4ca3d4bc25 streaming compression example can handle situations where input buffer size is manually set to a small value. 2016-09-18 12:19:06 +02:00
Yann Collet 88aa179347 added comments on buffer sizes guarantees 2016-09-18 11:58:30 +02:00
Yann Collet 60038948e6 added -- command in help 2016-09-16 18:52:52 +02:00