External matchfinder API (#3333)
* First building commit with sample matchfinder * Set up ZSTD_externalMatchCtx struct * move seqBuffer to ZSTD_Sequence* * support non-contiguous dictionary * clean up parens * add clearExternalMatchfinder, handle allocation errors * Add useExternalMatchfinder cParam * validate useExternalMatchfinder cParam * Disable LDM + external matchfinder * Check for static CCtx * Validate mState and mStateDestructor * Improve LDM check to cover both branches * Error API with optional fallback * handle RLE properly for external matchfinder * nit * Move to a CDict-like model for resource ownership * Add hidden useExternalMatchfinder bool to CCtx_params_s * Eliminate malloc, move to cwksp allocation * Handle CCtx reset properly * Ensure seqStore has enough space for external sequences * fix capitalization * Add DEBUGLOG statements * Add compressionLevel param to matchfinder API * fix c99 issues and add a param combination error code * nits * Test external matchfinder API * C90 compat for simpleExternalMatchFinder * Fix some @nocommits and an ASAN bug * nit * nit * nits * forward declare copySequencesToSeqStore functions in zstd_compress_internal.h * nit * nit * nits * Update copyright headers * Fix CMake zstreamtest build * Fix copyright headers (again) * typo * Add externalMatchfinder demo program to make contrib * Reduce memory consumption for small blockSize * ZSTD_postProcessExternalMatchFinderResult nits * test sum(matchlen) + sum(litlen) == srcSize in debug builds * refExternalMatchFinder -> registerExternalMatchFinder * C90 nit * zstreamtest nits * contrib nits * contrib nits * allow block splitter + external matchfinder, refactor * add windowSize param * add contrib/externalMatchfinder/README.md * docs * go back to old RLE heuristic because of the first block issue * fix initializer element is not a constant expression * ref contrib from zstd.h * extremely pedantic compiler warning fix, meson fix, typo fix * Additional docs on API limitations * minor nits * Refactor maxNbSeq calculation into a helper function * Fix copyright
This commit is contained in:
Elliot Gorokhovsky
+154
-2
@@ -478,6 +478,7 @@ typedef enum {
|
||||
* ZSTD_c_useBlockSplitter
|
||||
* ZSTD_c_useRowMatchFinder
|
||||
* ZSTD_c_prefetchCDictTables
|
||||
* ZSTD_c_enableMatchFinderFallback
|
||||
* Because they are not stable, it's necessary to define ZSTD_STATIC_LINKING_ONLY to access them.
|
||||
* note : never ever use experimentalParam? names directly;
|
||||
* also, the enums values themselves are unstable and can still change.
|
||||
@@ -497,7 +498,8 @@ typedef enum {
|
||||
ZSTD_c_experimentalParam13=1010,
|
||||
ZSTD_c_experimentalParam14=1011,
|
||||
ZSTD_c_experimentalParam15=1012,
|
||||
ZSTD_c_experimentalParam16=1013
|
||||
ZSTD_c_experimentalParam16=1013,
|
||||
ZSTD_c_experimentalParam17=1014
|
||||
} ZSTD_cParameter;
|
||||
|
||||
typedef struct {
|
||||
@@ -560,7 +562,7 @@ typedef enum {
|
||||
* They will be used to compress next frame.
|
||||
* Resetting session never fails.
|
||||
* - The parameters : changes all parameters back to "default".
|
||||
* This removes any reference to any dictionary too.
|
||||
* This also removes any reference to any dictionary or external matchfinder.
|
||||
* Parameters can only be changed between 2 sessions (i.e. no compression is currently ongoing)
|
||||
* otherwise the reset fails, and function returns an error value (which can be tested using ZSTD_isError())
|
||||
* - Both : similar to resetting the session, followed by resetting parameters.
|
||||
@@ -2044,6 +2046,20 @@ ZSTDLIB_STATIC_API size_t ZSTD_CCtx_refPrefix_advanced(ZSTD_CCtx* cctx, const vo
|
||||
*/
|
||||
#define ZSTD_c_prefetchCDictTables ZSTD_c_experimentalParam16
|
||||
|
||||
/* ZSTD_c_enableMatchFinderFallback
|
||||
* Allowed values are 0 (disable) and 1 (enable). The default setting is 0.
|
||||
*
|
||||
* Controls whether zstd will fall back to an internal matchfinder if an
|
||||
* external matchfinder is registered and returns an error code. This fallback is
|
||||
* block-by-block: the internal matchfinder will only be called for blocks where
|
||||
* the external matchfinder returns an error code. Fallback compression will
|
||||
* follow any other cParam settings, such as compression level, the same as in a
|
||||
* normal (fully-internal) compression operation.
|
||||
*
|
||||
* The user is strongly encouraged to read the full external matchfinder API
|
||||
* documentation (below) before setting this parameter. */
|
||||
#define ZSTD_c_enableMatchFinderFallback ZSTD_c_experimentalParam17
|
||||
|
||||
/*! ZSTD_CCtx_getParameter() :
|
||||
* Get the requested compression parameter value, selected by enum ZSTD_cParameter,
|
||||
* and store it into int* value.
|
||||
@@ -2676,6 +2692,142 @@ ZSTDLIB_STATIC_API size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_
|
||||
ZSTDLIB_STATIC_API size_t ZSTD_insertBlock (ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); /**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression. */
|
||||
|
||||
|
||||
/* ********************** EXTERNAL MATCHFINDER API **********************
|
||||
*
|
||||
* *** OVERVIEW ***
|
||||
* This API allows users to replace the zstd internal block-level matchfinder
|
||||
* with an external matchfinder function. Potential applications of the API
|
||||
* include hardware-accelerated matchfinders and matchfinders specialized to
|
||||
* particular types of data.
|
||||
*
|
||||
* See contrib/externalMatchfinder for an example program employing the
|
||||
* external matchfinder API.
|
||||
*
|
||||
* *** USAGE ***
|
||||
* The user is responsible for implementing a function of type
|
||||
* ZSTD_externalMatchFinder_F. For each block, zstd will pass the following
|
||||
* arguments to the user-provided function:
|
||||
*
|
||||
* - externalMatchState: a pointer to a user-managed state for the external
|
||||
* matchfinder.
|
||||
*
|
||||
* - outSeqs, outSeqsCapacity: an output buffer for sequences produced by the
|
||||
* external matchfinder. outSeqsCapacity is guaranteed >=
|
||||
* ZSTD_sequenceBound(srcSize). The memory backing outSeqs is managed by
|
||||
* the CCtx.
|
||||
*
|
||||
* - src, srcSize: an input buffer which the external matchfinder must parse
|
||||
* into sequences. srcSize is guaranteed to be <= ZSTD_BLOCKSIZE_MAX.
|
||||
*
|
||||
* - dict, dictSize: a history buffer, which may be empty, which the external
|
||||
* matchfinder may reference as it produces sequences for the src buffer.
|
||||
* Currently, zstd will always pass dictSize == 0 into external matchfinders,
|
||||
* but this will change in the future.
|
||||
*
|
||||
* - compressionLevel: a signed integer representing the zstd compression level
|
||||
* set by the user for the current operation. The external matchfinder may
|
||||
* choose to use this information to change its compression strategy and
|
||||
* speed/ratio tradeoff. Note: The compression level does not reflect zstd
|
||||
* parameters set through the advanced API.
|
||||
*
|
||||
* - windowSize: a size_t representing the maximum allowed offset for external
|
||||
* sequences. Note that sequence offsets are sometimes allowed to exceed the
|
||||
* windowSize if a dictionary is present, see doc/zstd_compression_format.md
|
||||
* for details.
|
||||
*
|
||||
* The user-provided function shall return a size_t representing the number of
|
||||
* sequences written to outSeqs. This return value will be treated as an error
|
||||
* code if it is greater than outSeqsCapacity. The return value must be non-zero
|
||||
* if srcSize is non-zero. The ZSTD_EXTERNAL_MATCHFINDER_ERROR macro is provided
|
||||
* for convenience, but any value greater than outSeqsCapacity will be treated as
|
||||
* an error code.
|
||||
*
|
||||
* If the user-provided function does not return an error code, the sequences
|
||||
* written to outSeqs must be a valid parse of the src buffer. Data corruption may
|
||||
* occur if the parse is not valid. A parse is defined to be valid if the
|
||||
* following conditions hold:
|
||||
* - The sum of matchLengths and literalLengths is equal to srcSize.
|
||||
* - All sequences in the parse have matchLength != 0, except for the final
|
||||
* sequence. matchLength is not constrained for the final sequence.
|
||||
* - All offsets respect the windowSize parameter as specified in
|
||||
* doc/zstd_compression_format.md.
|
||||
*
|
||||
* zstd will only validate these conditions (and fail compression if they do not
|
||||
* hold) if the ZSTD_c_validateSequences cParam is enabled. Note that sequence
|
||||
* validation has a performance cost.
|
||||
*
|
||||
* If the user-provided function returns an error, zstd will either fall back
|
||||
* to an internal matchfinder or fail the compression operation. The user can
|
||||
* choose between the two behaviors by setting the
|
||||
* ZSTD_c_enableMatchFinderFallback cParam. Fallback compression will follow any
|
||||
* other cParam settings, such as compression level, the same as in a normal
|
||||
* compression operation.
|
||||
*
|
||||
* The user shall instruct zstd to use a particular ZSTD_externalMatchFinder_F
|
||||
* function by calling ZSTD_registerExternalMatchFinder(cctx, externalMatchState,
|
||||
* externalMatchFinder). This setting will persist until the next parameter reset
|
||||
* of the CCtx.
|
||||
*
|
||||
* The externalMatchState must be initialized by the user before calling
|
||||
* ZSTD_registerExternalMatchFinder. The user is responsible for destroying the
|
||||
* externalMatchState.
|
||||
*
|
||||
* *** LIMITATIONS ***
|
||||
* External matchfinders are compatible with all zstd compression APIs. There are
|
||||
* only two limitations.
|
||||
*
|
||||
* First, the ZSTD_c_enableLongDistanceMatching cParam is not supported.
|
||||
* COMPRESSION WILL FAIL if it is enabled and the user tries to compress with an
|
||||
* external matchfinder.
|
||||
* - Note that ZSTD_c_enableLongDistanceMatching is auto-enabled by default in
|
||||
* some cases (see its documentation for details). Users must explicitly set
|
||||
* ZSTD_c_enableLongDistanceMatching to ZSTD_ps_disable in such cases if an
|
||||
* external matchfinder is registered.
|
||||
* - As of this writing, ZSTD_c_enableLongDistanceMatching is disabled by default
|
||||
* whenever ZSTD_c_windowLog < 128MB, but that's subject to change. Users should
|
||||
* check the docs on ZSTD_c_enableLongDistanceMatching whenever the external
|
||||
* matchfinder API is used in conjunction with advanced settings (like windowLog).
|
||||
*
|
||||
* Second, history buffers are not supported. Concretely, zstd will always pass
|
||||
* dictSize == 0 to the external matchfinder (for now). This has two implications:
|
||||
* - Dictionaries are not supported. Compression will *not* fail if the user
|
||||
* references a dictionary, but the dictionary won't have any effect.
|
||||
* - Stream history is not supported. All compression APIs, including streaming
|
||||
* APIs, work with the external matchfinder, but the external matchfinder won't
|
||||
* receive any history from the previous block. Each block is an independent chunk.
|
||||
*
|
||||
* Long-term, we plan to overcome both limitations. There is no technical blocker to
|
||||
* overcoming them. It is purely a question of engineering effort.
|
||||
*/
|
||||
|
||||
#define ZSTD_EXTERNAL_MATCHFINDER_ERROR ((size_t)(-1))
|
||||
|
||||
typedef size_t ZSTD_externalMatchFinder_F (
|
||||
void* externalMatchState,
|
||||
ZSTD_Sequence* outSeqs, size_t outSeqsCapacity,
|
||||
const void* src, size_t srcSize,
|
||||
const void* dict, size_t dictSize,
|
||||
int compressionLevel,
|
||||
size_t windowSize
|
||||
);
|
||||
|
||||
/*! ZSTD_registerExternalMatchFinder() :
|
||||
* Instruct zstd to use an external matchfinder function.
|
||||
*
|
||||
* The externalMatchState must be initialized by the caller, and the caller is
|
||||
* responsible for managing its lifetime. This parameter is sticky across
|
||||
* compressions. It will remain set until the user explicitly resets compression
|
||||
* parameters.
|
||||
*
|
||||
* The user is strongly encouraged to read the full API documentation (above)
|
||||
* before calling this function. */
|
||||
ZSTDLIB_STATIC_API void
|
||||
ZSTD_registerExternalMatchFinder(
|
||||
ZSTD_CCtx* cctx,
|
||||
void* externalMatchState,
|
||||
ZSTD_externalMatchFinder_F* externalMatchFinder
|
||||
);
|
||||
|
||||
#endif /* ZSTD_H_ZSTD_STATIC_LINKING_ONLY */
|
||||
|
||||
#if defined (__cplusplus)
|
||||
|
||||
Reference in New Issue
Block a user