3758 Commits
Author SHA1 Message Date
Nick Terrell b53da1f6f4 Add extra space to match linux kernel 2022-02-22 19:58:18 -08:00
Nick Terrell 693aa7bad2 Add comment to unused variable suppression 2021-11-17 13:46:09 -08:00
Nick Terrell b9302410bf [linux-kernel] Don't inline function in zstd_opt.c
The optimal parser is unlikely to be used in the linux kernel in
practice. There is no reason these functions should be force inlined,
since we aren't gaining anything, and are losing build size.

| Compiler | Before (Bytes) | After (Bytes) | Delta (Bytes) |
|----------|----------------|---------------|---------------|
| gcc-11   |        1142090 |        952754 |       -189336 |
| clang-12 |        1228402 |        976290 |       -252112 |

This is a temporary solution pending the resolution of Issue #2862 in
the `dev` branch.
2021-11-15 20:37:58 -08:00
Nick Terrell 0118fe65ff Fix unused variable warning
`litLengthSum` is unused when asserts are disabled. Already fixed in `dev` by
PR#2838. Found by the Kernel test robot [0].

[0] https://lore.kernel.org/linux-mm/202111120312.833wII4i-lkp@intel.com/T/

Reported-by: kernel test robot <lkp@intel.com>
2021-11-15 16:55:59 -08:00
Nick Terrell 608bacf6cb Backport zstd patch from LKML
Credit to Nathan Chancellor for the bug fix and Nick Desaulniers for the
bug report.

Link: ClangBuiltLinux/linux#1486
Link: https://lore.kernel.org/all/20211021202353.2356400-1-nathan@kernel.org/
2021-11-11 12:18:35 -08:00
Nick Terrell c0c38ba1db [binary-tree] Fix underflow of nbCompares
Fix underflow of `nbCompares` by switching to an `int` and comparing
`nbCompares > 0`. This is a minimal fix, because I don't want to change
the logic. These loops seem to be doing `nbCompares + 1` comparisons.

The bug was reported by Dan Carpenter and found by Smatch static
checker.

https://lore.kernel.org/all/20211008063704.GA5370@kili/
2021-10-11 15:57:35 -07:00
Nick Terrell 5a3e16f0c2 [ldm] Fix ZSTD_c_ldmHashRateLog bounds check
There is no minimum value check, so the parameter could be negative.
Switch to the standard pattern of using `BOUNDCHECK()`.

The bug was reported by Dan Carpenter and found by Smatch static
checker.

https://lore.kernel.org/all/20211008063704.GA5370@kili/
2021-10-11 15:43:57 -07:00
Nick Terrell 2c94f9fc61 [nit] Fix buggy indentation
The bug was reported by Dan Carpenter and found by Smatch static
checker.

https://lore.kernel.org/all/20211008063704.GA5370@kili/
2021-10-11 15:43:33 -07:00
Nick Terrell 695181c2e0 [multiple-ddicts] Fix NULL checks
The bug was reported by Dan Carpenter and found by Smatch static
checker.

https://lore.kernel.org/all/20211008063704.GA5370@kili/
2021-10-11 15:42:45 -07:00
Nick Terrell 20821a46f4 [lib] Make lib compatible with -Wfall-through excepting legacy
Switch to a macro `ZSTD_FALLTHROUGH;` instead of a comment. On supported
compilers this uses an attribute, otherwise it becomes a comment.

This is necessary to be compatible with clang's `-Wfall-through`, and
gcc's `-Wfall-through=2` which don't support comments. Without this the
linux build emits a bunch of warnings.
2021-09-23 11:54:14 -07:00
Nick Terrell de9de869a3 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:37:39 -07:00
Nick Terrell c1a244d534 Merge pull request #2539 from terrelln/linux-kernel-fixes
Fixes for the next linux kernel patch version
2021-03-24 16:29:38 -07:00
senandGitHub c48889f097 Merge pull request #2538 from senhuang42/monotonicity_test
Add memory monotonicity test over srcSize
2021-03-22 16:54:34 -04:00
Nick TerrellandGitHub ebc2dfa821 Merge pull request #2524 from terrelln/huf-stack-reduction
[huf] Reduce stack usage of HUF_readDTableX2 by ~972 bytes
2021-03-22 12:37:54 -07:00
Nick Terrell 634bfd339f [FSE] Clean up workspace using dynamically sized struct 2021-03-22 11:07:07 -07:00
Sen Huang dff4a0e867 Make ZSTD_estimateCCtxSize_internal() loop through all srcSize parameter sets as well 2021-03-21 16:15:31 -07:00
Nick Terrell 756bd59322 [huf][fse] Clean up workspaces
* Move `counting` to a struct in `FSE_decompress_wksp_body()`
* Fix error code in `FSE_decompress_wksp_body()`
* Rename a variable in `HUF_ReadDTableX2_Workspace`
2021-03-17 16:50:37 -07:00
Sen Huang 77ae664ba6 Fix ZSTD_dedicatedDictSearch_isSupported() requirements 2021-03-16 17:36:05 -07:00
Nick Terrell ea288e0d8e [lib] Bump zstd version number 2021-03-16 11:47:27 -07:00
Nick Terrell cd1551d261 [lib][tracing] Add ZSTD_NO_TRACE macro
When defined, it disables tracing, and avoids including the header.
2021-03-16 11:47:27 -07:00
Yann ColletandGitHub 3d6c903056 Merge pull request #2521 from animalize/doc_free
doc: ZSTD_free*() functions accept NULL pointer
2021-03-06 21:33:28 -08:00
Nick Terrell 3b1aba42cc [fse] Reduce stack usage of FSE_decompress_wksp() by 512 bytes
* Move `counting` into the workspace
* Inrease `HUF_DECOMPRESS_WORKSPACE_SIZE` by 512 bytes
2021-03-05 13:24:27 -08:00
Nick Terrell 0f18059a4e [huf] Reduce stack usage of HUF_readDTableX2 by ~460 bytes
* Use `HUF_readStats_wksp()`
* Use workspace in `HUF_fillDTableX2*()`
* Clean up workspace usage to use a workspace struct
2021-03-05 12:39:46 -08:00
Nick TerrellandGitHub b5fd348a85 Merge pull request #2523 from terrelln/huf-stack-reduction
Add HUF_writeCTable_wksp() function
2021-03-05 12:35:09 -08:00
Nick Terrell 5df2a21f1e Add HUF_writeCTable_wksp() function
This saves ~700 bytes of stack space in HUF_writeCTable.
2021-03-05 10:29:18 -08:00
Nick TerrellandGitHub e50f88ca4c Merge pull request #2522 from terrelln/stack-reduction
Reduce stack usage of ZSTD_buildCTable()
2021-03-04 20:55:58 -08:00
Nick Terrell 27498ff00f Reduce stack usage of ZSTD_buildCTable()
It is a stack high-point for some compression strategies and has an easy
fix. This moves the normalized count into the entropy workspace.
2021-03-04 16:12:11 -08:00
animalize 0933775d79 doc: ZSTD_free*() functions accept NULL pointer 2021-03-04 12:02:52 +08:00
Yann Collet 029f974ddc strengthen compilation flags 2021-03-02 15:43:52 -08:00
W. Felix Handte d7db928f72 Bump Library Version 1.4.8 -> 1.4.9 2021-03-01 17:45:30 -05:00
Yann ColletandGitHub 61b63e9060 Merge pull request #2492 from niacat/dev
Use standard md5 tool on NetBSD.
2021-02-24 16:38:10 -08:00
Nick Terrell 7736549bea [bug-fix] Make simple single-pass functions ignore advanced parameters
The simple compression functions are intended to ignore the advanced
parameters, but they were accidentally using them. All the
`ZSTD_parameters` were set correctly, but any extra parameters were
used as-is. E.g. `ZSTD_c_format`.

This PR makes all the simple single-pass functions listed below ignore
the advanced parameters, as intended.

* `ZSTD_compressCCtx()`
* `ZSTD_compress_usingDict()`
* `ZSTD_compress_usingCDict()`
* `ZSTD_compress_advanced()`
* `ZSTD_compress_usingCDict_advanced()`

It also adds a test case that ensures that each of these functions
ignore the advanced parameters.
2021-02-12 19:11:23 -08:00
Nick Terrell c62eb05964 [lib] Set appliedParams.compressionLevel correctly
Forward the correct compressionLevel to the appliedParams in all cases.
It was already correct for the advanced API, so only the old single-pass
functions needed to be fixed.

This compression level is unused by the library, but is set so that the
tracing framework can consume it.
2021-02-12 15:00:14 -08:00
Nick Terrell f520f6dfbe [trace] Minor fixes found during integration
* Mark `ZSTD_CCtx_getParameter()` as const
* Add `extern "C"` guards to `zstd_trace.h`
2021-02-11 16:20:04 -08:00
Yann ColletandGitHub 8884cb887d Merge pull request #2483 from mpu/ldmgear
New algorithms for the long distance matcher
2021-02-11 08:38:23 -08:00
nia 74f85818a6 Use standard md5 tool on NetBSD.
This avoids a GNU coreutils dependency.

-n is used to match the output format of coreutils:
http://man.netbsd.org/md5.1
2021-02-11 10:50:11 +01:00
Quentin Carbonneaux 552efcac2d relocate large arrays from the stack to ldmState_t 2021-02-10 16:16:54 +01:00
Nick Terrell e59c9459a5 [trace] Keep track of a uint64_t tracing context
The most common information that you want to track between begin() and
end() is the timestamp of the begin function, so you can measure the
duration of the (de)compression call. Allow the tracing library to put
this information inside the `ZSTD_TraceCtx`, so it doesn't need to keep
a global map in this case. If a single uint64_t is not enough, the
tracing library can return a unique identifier (like the context
pointer) instead, and use it as a key in a map.

This keeps the simple case simple.
2021-02-09 11:37:05 -08:00
Quentin Carbonneaux e2ad174d73 fix some compiler warnings 2021-02-08 20:19:16 +01:00
Nick Terrell 54a4998a80 Add basic tracing functionality 2021-02-05 16:28:52 -08:00
Nick Terrell f9b1e711ba [zstd] Fix NULL pointer addition in ZSTD_checkContinuity()
Don't start a new section when `dstSize == 0` to avoid NULL
pointer addition.
2021-02-05 12:18:06 -08:00
Yann Collet b9748757b0 fixed minor cast warning 2021-02-05 09:55:54 -08:00
Quentin Carbonneaux 874a590e5c deal safely with short inputs in ZSTD_ldm_generateSequences
The fuzzer CI found this bug.
2021-02-04 11:15:24 +01:00
Quentin Carbonneaux 9f327c02fd new core ldm algorithm 2021-02-03 22:24:07 +01:00
Quentin Carbonneaux aee3dc877f fix a variable name to reflect its nature 2021-01-22 02:24:19 -08:00
Quentin Carbonneaux d6e3de77dc fix warning and remove one more occurrence of makeEntryAndInsertByTag 2021-01-20 01:39:16 -08:00
Quentin Carbonneaux e0d5eca8fa fix forgotten numTagBits in getTagMask 2021-01-20 00:54:20 -08:00
Quentin Carbonneaux 1e65711ca5 a couple performance improvement changes for ldm 2021-01-20 00:54:20 -08:00
Yann Collet 0bad3e5c0f parallel make build on linux
fix #2474
2021-01-18 11:33:03 -08:00
yumeyaoandGitHub 821d9acd17 Fix visibility of symbols in .so (#2441)
Fix visibility of symbols in .so and add an alias for renamed API
2021-01-08 14:27:31 -08:00