Commit Graph
3537 Commits
Author SHA1 Message Date
Yann Collet ceccd7ae2d Merge branch 'dev' into libzstd_autoconf_full 2020-10-27 15:45:30 -07:00
Yann ColletandGitHub 2d2507b9db Merge pull request #2374 from bket/portability
'head -c BYTES' is non-portable
2020-10-27 14:15:35 -07:00
senandGitHub 17b700d78a Merge pull request #2366 from senhuang42/enable_ldm_by_default
Enable LDM by default if window size >= 128MB and strategy uses opt parser
2020-10-27 14:59:28 -04:00
Nick TerrellandGitHub 0953645837 Merge pull request #2362 from senhuang42/fix_ldm_fuzz_issue
Fix long distance matcher OSS-fuzz issue
2020-10-27 11:13:03 -07:00
Björn Ketelaars 1f661b5f6b 'head -c BYTES' is non-portable 2020-10-27 16:55:23 +01:00
Yann Collet 456db0c377 make install only rebuild binaries if they don't exist
Now `make` followed by `make install` doesn't rebuild binaries

also : only generated target directories if they don't already exist
2020-10-23 16:46:49 -07:00
Yann Collet a6ee614a44 make zstd is now differentiated from zstd-nomt
avoids mixing object files using different flags
2020-10-23 16:08:21 -07:00
Yann Collet 89b961ea46 simplified silent mode maintenance 2020-10-23 10:41:17 -07:00
Yann Collet ffe8d9e428 fix partial lib test 2020-10-23 10:27:12 -07:00
Yann Collet b5d4728713 simplified silent mode 2020-10-23 10:22:52 -07:00
Yann Collet a7ad05bf57 fixed building libzstd with manual BUILD_DIR
and when HASH is not found
2020-10-23 10:14:04 -07:00
Yann Collet d3f1a9b5bd fix partial-build test
sometimes, the scope difference is solely determined by the list of source files,
not by the flags.
2020-10-22 21:36:09 -07:00
Yann Collet a912ef0952 can integrate later dynamic flags changes
for example `libzstd-mt` is `differentiated from `libzstd`
2020-10-22 18:48:06 -07:00
Yann Collet ce6cd07c33 updated build documentation 2020-10-22 12:31:23 -07:00
Yann Collet e3867fb735 fixed libzstd.dll compilation on mingw
and zstd linking
2020-10-22 11:52:19 -07:00
Yann Collet 494f7169ed fix directory creation for Windows' libzstd 2020-10-22 00:15:31 -07:00
Yann Collet dd24496951 programs/zstd also automatically generate object dir per conf
same rules as lib/libzstd
can also be controlled via HASH and BUILD_DIR
2020-10-21 23:38:33 -07:00
Yann Collet 0f8ee5c51e Merge branch 'dev' into libzstd_autoconf 2020-10-21 22:36:09 -07:00
Yann Collet d0436b2a45 automatically detect configuration changes
Makefile now automatically detects modifications of compilation flags,
and produce object files in directories dedicated to this compilation flags.
This makes it possible, for example, to compile libzstd with different DEBUGLEVEL.
Object files sharing the same configration will be generated into their dedicated directories.

Also : new compilation variables
- DEBUGLEVEL : select the debug level (assert & traces) inserted during compilation (default == 0 == release)
- HASH : select a hash function to differentiate configuration (default == md5sum)
- BUILD_DIR : skip the hash stage, store object files into manually specified directory
2020-10-21 19:22:45 -07:00
Yann Collet 8a453a34c5 automatic %.h header dependency tracking
also : BUILD_DIR can be manually specified
2020-10-21 17:25:07 -07:00
Yann ColletandGitHub 2224ec33ed Merge pull request #2367 from facebook/progressive_build
faster rebuild of zstd
2020-10-21 15:43:14 -07:00
Yann Collet 2b99bc29bf consolidated vpath 2020-10-21 04:01:01 -07:00
Yann Collet e8eb2939fe store %.o object files into obj/
both static and dynamic libraries have their own object directory
2020-10-21 03:44:38 -07:00
Yann Collet 3e519be965 minor cleaning 2020-10-21 03:22:27 -07:00
Yann Collet 911dbdbb4b build libzstd.so from object files
%.o object files generated for dynamic library
must be different from those generated for static library.

Due to this difference, %.o were so far only generated for the static library.
The dynamic library was rebuilt from %.c source.

This meant that, for every minor change, the entire dynamic library had to be rebuilt.

This is fixed in this PR :
only the modified %.c source get rebuilt.
2020-10-20 22:19:57 -07:00
senhuang42 8bdb32aebe Add a function for LDM enable check 2020-10-20 13:46:02 -04:00
senhuang42 578e889ec1 Move ldm enable to compressStream2() 2020-10-20 13:04:45 -04:00
senhuang42 d28d8a1d72 Include LDM tables size for CCtx size estimation where relevant 2020-10-20 09:21:30 -04:00
senhuang42 b1c7fc5768 Add compatibility for multithreading 2020-10-19 12:07:06 -04:00
senhuang42 aad436da37 Document ldm enabled by default in zstd.h 2020-10-19 11:02:29 -04:00
senhuang42 590f7f55f0 Add ldm enable condition in ZSTD_resetCCtx_internal 2020-10-19 10:26:17 -04:00
senhuang42 4d01979b62 Expose and call ZSTD_ldm_skipRawSeqStoreBytes() 2020-10-16 20:30:00 -04:00
Yann ColletandGitHub a0ec50c2dc Merge pull request #2355 from senhuang42/change_ldm_mt_config
Reduce --long mode MT jobsize at higher levels
2020-10-16 13:35:50 -07:00
Yann Collet 314c7df170 minor : change test order
to reduce a warning with `clang` on Windows
2020-10-16 13:26:47 -07:00
senhuang42 f49926edf4 Change cycleLog adjustment to +3 from +4 2020-10-15 09:56:05 -04:00
senhuang42 ee84817fe7 Reset posInSequence when using ZSTD_referenceExternalSequences() 2020-10-14 22:06:08 -04:00
senhuang42 d0550bb18f Clarify argument names, fix DEBUGLOG() statements 2020-10-14 15:45:43 -04:00
senhuang42 3f99c9b38d Adjust match backwards count args 2020-10-14 15:23:03 -04:00
senhuang42 bf0d559449 Introduce, implement, and call ZSTD_ldm_countBackwardsMatch_2segments() 2020-10-14 12:58:06 -04:00
senhuang42 467e4383b0 Merge branch 'dev' of github.com:senhuang42/zstd into change_ldm_mt_config 2020-10-14 10:17:50 -04:00
Nick TerrellandGitHub 8c46c1d851 Merge pull request #2356 from bsdimp/neon
aarch64: use __ARM_NEON instead of __aarch64__ to control use of neon
2020-10-13 15:42:46 -07:00
Yann ColletandGitHub 1283935ac2 Merge pull request #2281 from likema/fix-aix-51
Fix building on AIX 5.1
2020-10-13 13:09:33 -07:00
Yann ColletandGitHub f5d5cd3b40 Merge pull request #2341 from senhuang42/ldm_optimized_for_opt_parser
Integrate long distance matches into optimal parser
2020-10-13 13:09:07 -07:00
Warner Losh 43c0054405 aarch64: use __ARM_NEON instead of __aarch64__ to control use of neon
There are compilation environments in aarch64 where NEON isn't
available. While these environments could define ZSTD_NO_INTRINSICS,
it's more fail-safe to use the more specific symbol to know if NEON
extensions are available.

__ARM_NEON is the proper symbol, defined in ARM C Language Extensions
Release 2.1 (https://developer.arm.com/documentation/ihi0053/d/). Some
sources suggest __ARM_NEON__, but that's the obsolete spelling from
prior versions of the standard.

Signed-off-by: Warner Losh <imp@bsdimp.com>
2020-10-13 12:12:46 -06:00
Nick Terrell 7e6f91ed84 [minor] Improve docs and add an assert in response to review 2020-10-12 16:43:17 -07:00
senhuang42 354b5f1c0a Use cycleLog instead of chainLog to determine LDM jobLog 2020-10-12 16:09:59 -04:00
Nick Terrell d5c688e8ae Fix ZSTD_adjustCParams_internal() to handle dictionary logic
Pass in the `ZSTD_cParamMode_e` to select how we define our cparams.
Based on the mode we either take the `dictSize` into account or we set
it to `0`. See the documentation for `ZSTD_cParamMode_e`.

Some of the modes currently share the same behavior. But they have
distinct modes because they are drastically different cases. E.g.
compression + reprocessing the dictionary and creating a cdict.

Additionally, when downsizing the hashLog and chainLog take the
(adjusted) dictionary size into account, since the size of the
dictionary gets added onto the window size.

Adds a simple test to ensure that we aren't downsizing too far.
2020-10-12 12:50:04 -07:00
Nick Terrell fadaab8c7c [minor improvement] Pass 0 as the content size in the DDS
The DDS structure can't be copied into the working tables like the DMS.
So it doesn't need to account for the source size when sizing its
parameters, just the dictionary size.
2020-10-12 12:47:21 -07:00
Nick Terrell 48ef15fb47 [minor improvement] Pass dictSize when selecting parameters
When selecting parameters in streaming compression with a dictionary use
the dictionary size to select the parameters.
2020-10-12 12:47:19 -07:00
Nick Terrell 012818df99 [refactor] Remove ZSTD_resetCStream_internal()
This function is only called in one place. It isn't a logical separation
of duties, and it was only obsfucating the code now, so inline it.
2020-10-12 12:46:10 -07:00