Commit Graph
9330 Commits
Author SHA1 Message Date
Yann Collet f92ec5ea54 change the offset|repcode sumtype format to match offBase
directly at ZSTD_storeSeq() interface.

In the process, remove ZSTD_REP_MOVE.

This makes it possible, in future commits,
to update and effectively simplify the naming scheme
to properly label the updated processing pipeline :
offset | repcode => offBase => offCode + offBits
2021-12-29 12:03:36 -08:00
Yann Collet ad7c9fc11e use ZSTD_memcpy(), for proper redirection within Linux Kernel 2021-12-28 17:41:47 -08:00
Yann Collet 8da414231d found a few more places which were dependent on seqStore offcode sumtype numeric representation 2021-12-28 17:03:24 -08:00
Yann Collet de9f52e945 regroup all mentions of ZSTD_REP_MOVE within zstd_compress_internal.h 2021-12-28 13:47:57 -08:00
Yann Collet a34ccad9a6 fixed minor conversion warnings 2021-12-28 13:21:22 -08:00
Yann Collet 92a08eec72 abstracted storeSeq() sumtype numeric representation from zstd_lazy.c 2021-12-28 12:23:39 -08:00
Yann Collet e909fa627f abstracted storeSeq() sumtype numeric representation from zstd_opt.c 2021-12-28 12:14:33 -08:00
Yann Collet 681c81f06c abstracted storeSeq() sumtype numeric representation from decodecorpus.c 2021-12-28 11:58:33 -08:00
Yann Collet 6fa640ef70 separate newRep() from updateRep()
the new contracts seems to make more sense :
updateRep() updates an array of repeat offsets _in place_,
while newRep() generates a new structure with the updated repeat-offset array.

Most callers are actually expecting the in-place variant,
and a limited sub-section, in `zstd_opt.c` mainly, prefer `newRep()`.
2021-12-28 11:52:33 -08:00
Yann Collet 321583ccf5 fixed minor typecast warnings 2021-12-28 11:38:21 -08:00
Yann Collet b7630a474b abstracted usage of offBase sumtype within zstd_lazy.c 2021-12-28 10:59:47 -08:00
Yann Collet 435f5a2e6d fixed regression test assert
optLdm->offset might be == 0 in invalid case.
Only use STORE_OFFSET() after validating it's a correct case.
2021-12-28 09:55:31 -08:00
Yann Collet 2068889146 created STORED_*() macros
to act on values stored / expressed in the sumtype numeric representation required by `storedSeq()`.

This makes it possible to abstract away this representation by using the macros to extract these values.

First user : ZSTD_updateRep() .
2021-12-28 06:59:07 -08:00
Yann Collet 1aed962216 introduce macros STORE_OFFSET() and STORE_REPCODE()
this meant to abstract the sumtype representation required
to transfert `offcode` to `ZSTD_storeSeq()`.

Unfortunately, the sumtype numeric representation is currently a leaky abstraction
that has permeated many other parts of the code,
especially within `zstd_lazy.c` and also within `zstd_opt.c` and `zstd_compress.c`.

While this PR makes a good job a transfering a large nb of call sites
to using the new macros, there are still a few sites where this transformation is more complex,
or where the numeric representation itself it used "as is".

One of the problematics area is the decision to use the numeric format of the sumtype
within the match finders of `zstd_lazy`.

This commit doesn't change the behavior, it only introduces and employes the macros,
but eventually the resulting code remains identical.

At target, if the numeric representation of the sumtype can be completely abstracted
and no other part of the code depends on it,
it will be possible to move it towards something slightly more efficient.
2021-12-23 22:03:30 -08:00
Yann Collet bec7bbb5a4 Merge branch 'dev' into seqStore_off 2021-12-23 18:03:17 -08:00
Yann Collet aeff128331 change seqDef.offset into seqDef.offBase
to better reflect the value stored in this field.
2021-12-23 17:56:08 -08:00
Yann Collet 75525fcb9f library optimization flag can be selected on command line again
`CFLAGS=-O0 make`
will now use `-O0` instead of enforcing `-O3`
which used to be the behavior before introduction of `libzstd.mk`.

This should result in faster tests,
since a few tests depend on this capability for faster roundtrips.
2021-12-23 17:43:12 -08:00
Yann Collet e145b58cfd changed seqDef.matchLength into seqDef.mlBase
since this is effectively what is stored in this field (== matchLength - MINMATCH).
This makes it clearer what needs to be done when reading from / writing to this field.
2021-12-23 13:39:46 -08:00
Yann Collet b77fcac61f change ZSTD_storeSeq() interface to accept matchLength
instead of mlBase.

This removes the need to do `- MINMATCH` at every call site.

The new interface contract is checked with an `assert()`.
2021-12-23 12:03:33 -08:00
Yann ColletandGitHub eebf24bd2a Merge pull request #2947 from MehdiChinoune/patch-1
Fix zstd-static output name with MINGW/Clang
2021-12-22 11:13:40 -08:00
مهدي شينون (Mehdi Chinoune)andMehdi Chinoune 14a0eaf73b Fix zstd-static output name with MINGW/Clang 2021-12-22 10:16:41 +01:00
Yann Collet 41153071a0 updated manual 2021-12-21 08:52:50 -08:00
Yann Collet abc694f53e update man pages 2021-12-20 14:24:06 -08:00
Yann Collet 1219b7f3e3 updated changelog for v1.5.1 2021-12-20 12:28:27 -08:00
Yann ColletandGitHub 24960134d9 Merge pull request #2940 from facebook/revert-2885-limit-level-32bit-systems
Revert "Limit `ZSTD_maxCLevel` to 21 for 32-bit binaries."
2021-12-20 12:27:28 -08:00
Yann ColletandGitHub a9e43b37d0 Revert "Limit ZSTD_maxCLevel to 21 for 32-bit binaries." 2021-12-20 11:43:14 -08:00
Yann ColletandGitHub 85bd1e663b Merge pull request #2934 from facebook/lazy_rebalance
rebalance lazy compression levels
2021-12-20 11:37:44 -08:00
Yann Collet 2624652a32 Merge branch 'dev' into lazy_rebalance 2021-12-20 08:28:21 -08:00
Yann Collet 80a28f2078 update regression results 2021-12-20 07:54:57 -08:00
Yann Collet 9fc7f38e12 added target update_regressionResults
to automate the creation of updated tests/regression/results.csv
2021-12-16 21:46:14 -08:00
Yann Collet 01adddc3e0 update regression results 2021-12-16 20:43:23 -08:00
Yann Collet 7c7b9244d6 update regression results 2021-12-16 16:07:54 -08:00
Yann Collet 47466d2192 Merge branch 'dev' into lazy_rebalance 2021-12-16 14:46:21 -08:00
Yann Collet 9a32492730 updated regression results.csv 2021-12-16 14:39:30 -08:00
Yann Collet 925649b2d0 fixed version number for fast modes 2021-12-16 11:52:43 -08:00
Yann Collet f829c32258 forgot the chainlog is effectively a "fake" value with rowHash
the only value which makes sense is `hashlog-1`
as it mimics the real memory usage.
2021-12-16 11:37:40 -08:00
Yann ColletandGitHub 5c96146db3 Merge pull request #2931 from facebook/fix_rowlog
fixed incorrect rowlog initialization
2021-12-16 09:48:12 -08:00
Yann Collet 5c63096269 updated benchmark for v1.5.1
answers #2764
2021-12-16 04:46:21 -08:00
Yann Collet 8a724f91cb minor changelog update 2021-12-16 01:07:15 -08:00
Yann Collet 7a347769dc minor changelog update 2021-12-15 21:45:30 -08:00
Yann Collet db1b408a2f rebalance lazy compression levels 2021-12-15 21:33:31 -08:00
Yann Collet c8d6067615 fixed incorrect rowlog initialization
the variable has only very limited usage,
being only used once at the beginning of the block for prefetching only,
hence the error had no impact on compression ratio.
2021-12-15 14:37:05 -08:00
Elliot GorokhovskyandGitHub c5f1e826ca Merge pull request #2925 from embg/dict_training_sample_limit_size
Allow user to specify memory limit for dictionary training
2021-12-15 15:58:17 -05:00
Yann Collet 622fee6140 minor changelog update, for clarity 2021-12-15 11:56:11 -08:00
Yann ColletandGitHub eaf786242d Merge pull request #2929 from facebook/sse_row_lazy
simplify SSE implementation of row_lazy match finder
2021-12-15 11:47:15 -08:00
Felix HandteandGitHub d5933a4009 Merge pull request #2930 from nolange/reduce_tables_to_8bit
Reduce tables to 8bit
2021-12-15 12:43:25 -05:00
Norbert Lange 2fbb1d10c1 Reduce bit tables to 8bit
This saves some 1.7Kb in rodata section (x86_64, zstd tool),
while assembler code stays the same except
the type of a few load/extend instructions.

Should not have negative performance implications.
2021-12-14 23:47:57 +01:00
Norbert Lange 99923dfc1a Add typedefs for 8bit (un)signed
To make code more expressive, add U8 and S8 typedefs
2021-12-14 23:47:57 +01:00
binhdvoandGitHub 64205b7832 Fix performance degradation with -m32 (#2926) 2021-12-14 15:53:50 -05:00
Elliot Gorokhovsky 71c0c07c19 Allow user to specify memory limit for dictionary training 2021-12-14 14:29:01 -05:00