Commit Graph
8215 Commits
Author SHA1 Message Date
Nick TerrellandGitHub 599ff58e08 Merge pull request #2339 from terrelln/zstdmt-stability
Fix zstdmt stability issues and clean up the zstdmt code
2020-10-27 19:43:13 -07:00
Yann ColletandGitHub 2d2507b9db Merge pull request #2374 from bket/portability
'head -c BYTES' is non-portable
2020-10-27 14:15:35 -07:00
senandGitHub 17b700d78a Merge pull request #2366 from senhuang42/enable_ldm_by_default
Enable LDM by default if window size >= 128MB and strategy uses opt parser
2020-10-27 14:59:28 -04:00
Nick TerrellandGitHub 0953645837 Merge pull request #2362 from senhuang42/fix_ldm_fuzz_issue
Fix long distance matcher OSS-fuzz issue
2020-10-27 11:13:03 -07:00
Björn Ketelaars 1f661b5f6b 'head -c BYTES' is non-portable 2020-10-27 16:55:23 +01:00
Yann ColletandGitHub 284d84ecf7 Merge pull request #2370 from facebook/libzstd_autoconf
Automatic configuration detection
2020-10-26 15:16:04 -07:00
Yann Collet d3f1a9b5bd fix partial-build test
sometimes, the scope difference is solely determined by the list of source files,
not by the flags.
2020-10-22 21:36:09 -07:00
Yann Collet 98787a3018 Merge branch 'dev' into libzstd_autoconf 2020-10-22 18:55:30 -07:00
Yann ColletandGitHub f90424da2d Merge pull request #2368 from facebook/progressive_libzstd
faster rebuild of libzstd
2020-10-22 17:36:56 -07:00
Yann Collet ce6cd07c33 updated build documentation 2020-10-22 12:31:23 -07:00
Yann Collet e3867fb735 fixed libzstd.dll compilation on mingw
and zstd linking
2020-10-22 11:52:19 -07:00
Yann Collet 91a8cb9559 fix DEBUGLEVEL redefinition from tests/ 2020-10-22 00:20:40 -07:00
Yann Collet 494f7169ed fix directory creation for Windows' libzstd 2020-10-22 00:15:31 -07:00
Yann Collet ca75da8fa3 fix test
DEBUGLEVEL redefinition
2020-10-21 23:51:13 -07:00
Yann Collet dd24496951 programs/zstd also automatically generate object dir per conf
same rules as lib/libzstd
can also be controlled via HASH and BUILD_DIR
2020-10-21 23:38:33 -07:00
Yann Collet 01ecad2326 zstd in programs/ can also receive a DEBUGLEVEL value
simplify tests : only set DEBUGLEVEL, not the flags directly
2020-10-21 23:13:46 -07:00
Yann Collet 0f8ee5c51e Merge branch 'dev' into libzstd_autoconf 2020-10-21 22:36:09 -07:00
Yann Collet d0436b2a45 automatically detect configuration changes
Makefile now automatically detects modifications of compilation flags,
and produce object files in directories dedicated to this compilation flags.
This makes it possible, for example, to compile libzstd with different DEBUGLEVEL.
Object files sharing the same configration will be generated into their dedicated directories.

Also : new compilation variables
- DEBUGLEVEL : select the debug level (assert & traces) inserted during compilation (default == 0 == release)
- HASH : select a hash function to differentiate configuration (default == md5sum)
- BUILD_DIR : skip the hash stage, store object files into manually specified directory
2020-10-21 19:22:45 -07:00
Yann Collet 8a453a34c5 automatic %.h header dependency tracking
also : BUILD_DIR can be manually specified
2020-10-21 17:25:07 -07:00
Yann ColletandGitHub 2224ec33ed Merge pull request #2367 from facebook/progressive_build
faster rebuild of zstd
2020-10-21 15:43:14 -07:00
Yann Collet 2b99bc29bf consolidated vpath 2020-10-21 04:01:01 -07:00
Yann Collet e8eb2939fe store %.o object files into obj/
both static and dynamic libraries have their own object directory
2020-10-21 03:44:38 -07:00
Yann Collet 3e519be965 minor cleaning 2020-10-21 03:22:27 -07:00
Yann Collet 911dbdbb4b build libzstd.so from object files
%.o object files generated for dynamic library
must be different from those generated for static library.

Due to this difference, %.o were so far only generated for the static library.
The dynamic library was rebuilt from %.c source.

This meant that, for every minor change, the entire dynamic library had to be rebuilt.

This is fixed in this PR :
only the modified %.c source get rebuilt.
2020-10-20 22:19:57 -07:00
Yann Collet 684f8ea970 decouple zstd object files from lib/
That was a subtle one :
VPATH is affecting search for both %.c source and %.o object files.
This meant that, when an object file already exists in lib/,
it's used in programs/,
even though programs/ is supposed to generate its own %.o object files.

With the new vpath directive, this is no longer the case :
the search is only activated for %.c source files.
Now, local programs/%.o are always generated
even if equivalent ones are already created in lib/.

It more clearly guarantees that lib/ and programs/ can use different compilation directives
without mixing resulting %.o object files.
2020-10-20 18:16:11 -07:00
Yann Collet a9f7e77d44 fixed travis tests 2020-10-20 17:08:14 -07:00
senhuang42 8bdb32aebe Add a function for LDM enable check 2020-10-20 13:46:02 -04:00
senhuang42 578e889ec1 Move ldm enable to compressStream2() 2020-10-20 13:04:45 -04:00
Nick TerrellandGitHub d6dae2000b Merge pull request #2365 from senhuang42/move_opt_parser_test_to_long_tests
Move ldm + opt parser no regression test to long tests
2020-10-20 11:34:36 -04:00
senhuang42 d28d8a1d72 Include LDM tables size for CCtx size estimation where relevant 2020-10-20 09:21:30 -04:00
senhuang42 81a2c02d8f Move ldm no regression test to fuzzer longtests 2020-10-19 15:28:46 -04:00
senhuang42 b1c7fc5768 Add compatibility for multithreading 2020-10-19 12:07:06 -04:00
senhuang42 aad436da37 Document ldm enabled by default in zstd.h 2020-10-19 11:02:29 -04:00
senhuang42 df470e176b Add unit test for no cctx requested params change 2020-10-19 10:52:41 -04:00
senhuang42 590f7f55f0 Add ldm enable condition in ZSTD_resetCCtx_internal 2020-10-19 10:26:17 -04:00
senhuang42 4d01979b62 Expose and call ZSTD_ldm_skipRawSeqStoreBytes() 2020-10-16 20:30:00 -04:00
Yann Collet de985b1c1a fix appveyor test
was mixing `*.o` object files from different compilation settings
2020-10-16 14:31:40 -07:00
Nick TerrellandGitHub 9327a77c76 Merge pull request #2359 from senhuang42/fuzz_makefile_cleanup
[minor] Fix some /tests/fuzz Makefile annoyances
2020-10-16 13:53:34 -07:00
Yann ColletandGitHub a0ec50c2dc Merge pull request #2355 from senhuang42/change_ldm_mt_config
Reduce --long mode MT jobsize at higher levels
2020-10-16 13:35:50 -07:00
Yann ColletandGitHub d380f803b9 Merge pull request #2357 from senhuang42/fix_ldm_nd_behavior
Fix nondeterministic LDM behavior in multithreading
2020-10-16 13:35:28 -07:00
Yann Collet 314c7df170 minor : change test order
to reduce a warning with `clang` on Windows
2020-10-16 13:26:47 -07:00
Yann Collet eb1959d926 fix .deps list
and minor simplications
2020-10-16 11:08:52 -07:00
Yann Collet c355d149de support verbose build output with V=1 2020-10-16 00:21:49 -07:00
Yann Collet b6b87c2b77 track header dependencies during build
modifying a header file will now trigger recompilation of related *.c units
2020-10-16 00:18:16 -07:00
Yann Collet 80cee8d3fe faster rebuild of zstd
Building the zstd CLI costs time.
Some part of it is incompressible, leading to substantial iteration delay when testing code modifications.
That's mainly because all source files from the library must be rebuilt from source every time.

The main reason we don't build the CLI from library object files
is that we can't just build the object directly in the lib/ directory
(which they would by default)
since they use different compilation flags.
Specifically, the CLI enables multithreading, while the library doesn't (by default).

This is solved in this commit, by generating the object files locally.
Now, the CLI and the library can employ different sets of flags, without tripping over each other.
All library object files are generated directly into programs/ dir.
This works because no 2 source files have the same name.

Now, modifying a file doesn't require to recompile the entire lib, just the modified files.
The recipe is also compatible with `-j` parallel build, leading to large build time reductions on multi-core systems.
2020-10-16 00:01:41 -07:00
Yann ColletandGitHub ca885a198a Merge pull request #2361 from senhuang42/fix_travis_minimal_decompressor_macros_test
[travis CI] Change make -j all check to make -j all && make check
2020-10-15 15:35:57 -07:00
senhuang42 adc53db6ed Change make -j all check to make -j all && make check 2020-10-15 15:11:51 -04:00
Yann ColletandGitHub 056f4c9145 Merge pull request #2360 from facebook/zwrap_parallel_build
fix zwrapbench parallel build
2020-10-15 11:54:15 -07:00
Yann Collet abf5c0655c fix zlibwrap parallel build
previous recipe would build object files directly within programs/
which could be in competition with other local builds happening in programs/ at the same time.

fixed by generating the relevant object file locally.
2020-10-15 10:56:37 -07:00
senhuang42 42d037bdba Add libregression build target, also fix make clean and .gitignore 2020-10-15 10:34:50 -04:00