ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,691 Commits 17 Branches 77 Tags
f80dec66b0fdeeb0bb08ab4739a2549476e8cb4e
Commit Graph
8691 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Bimba Shrestha df9e5b6f4c adding 2020-05-07 22:07:40 -05:00
Yann ColletandGitHub efc656c9a6 Merge pull request #2114 from facebook/verbose 
support for verbose make
 2020-05-07 13:16:33 -07:00
Yann ColletandGitHub 1afe57cff7 Merge pull request #2112 from facebook/cfast 
small speed improvement for strategy fast
 2020-05-07 13:13:34 -07:00
caoyzhandNick Terrell 969ba4f2b9 Change the modification of ZSTD_wildcopy() 2020-05-07 13:10:46 -07:00
caoyzhandNick Terrell a7e34ff693 revert ZSTD_reduceTable_internal()'s modificatiion 2020-05-07 13:10:46 -07:00
caoyzhandNick Terrell 9e802ede9c Modify indent of comments 2020-05-07 13:10:46 -07:00
caoyzhandNick Terrell 7f75f05e84 Change "arm_neon.h" to system include <arm_neon.h> 2020-05-07 13:10:46 -07:00
caoyzhandNick Terrell b2e56f7f7f Optimize compression by using neon function. 2020-05-07 13:10:46 -07:00
Mingye WangandGitHub 37d10b6c75 programs/README.md: update help text 
I noticed that the help is outdated as there is no mention of --adapt. Y'all should probably put "update help text" into your release checklist.
 2020-05-08 02:52:18 +08:00
Nick TerrellandNick Terrell 45c66dd298 [zdict] Stabilize ZDICT_finalizeDictionary() 2020-05-07 10:37:01 -07:00
Bimba ShresthaandGitHub 625924774e Merge pull request #2115 from bimbashrestha/docs 
[doc] Adding static analysis section and note
 2020-05-07 12:03:33 -05:00
Bimba Shrestha 1b1cc3d1b7 adding some static-anslysis docs 2020-05-07 10:38:06 -05:00
Yann Collet cf854f4660 support for verbose make 
A commonly accepted makefile idiom is V=1 or VERBOSE=1
to request the printing of all commands.

This is not "default" though, and must be manually added.

Example :
Before :
```
make libzstd
compiling dynamic library 1.4.5
creating versioned links

make libzstd V=1
compiling dynamic library 1.4.5
creating versioned links
```

After :
```
make libzstd
compiling dynamic library 1.4.5
creating versioned links

make libzstd V=1
compiling dynamic library 1.4.5
cc -DXXH_NAMESPACE=ZSTD_ -DZSTD_LEGACY_SUPPORT=5 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat  -O3 common/debug.c common/entropy_common.c common/error_private.c common/fse_decompress.c common/pool.c common/threading.c common/xxhash.c common/zstd_common.c compress/fse_compress.c compress/hist.c compress/huf_compress.c compress/zstd_compress.c compress/zstd_compress_literals.c compress/zstd_compress_sequences.c compress/zstd_compress_superblock.c compress/zstd_double_fast.c compress/zstd_fast.c compress/zstd_lazy.c compress/zstd_ldm.c compress/zstd_opt.c compress/zstdmt_compress.c decompress/huf_decompress.c decompress/zstd_ddict.c decompress/zstd_decompress.c decompress/zstd_decompress_block.c deprecated/zbuff_common.c deprecated/zbuff_compress.c deprecated/zbuff_decompress.c dictBuilder/cover.c dictBuilder/divsufsort.c dictBuilder/fastcover.c dictBuilder/zdict.c legacy/zstd_v05.c legacy/zstd_v06.c legacy/zstd_v07.c -shared -fPIC -fvisibility=hidden -Wl,-soname=libzstd.so.1 -o libzstd.so.1.4.5
creating versioned links
ln -sf libzstd.so.1.4.5 libzstd.so.1
ln -sf libzstd.so.1.4.5 libzstd.so
```
 2020-05-07 08:04:10 -07:00
Mark Dittmer 9b8f337357 [contrib] Support seek table-only API 
Memory constrained use cases that manage multiple archives benefit from
retaining multiple archive seek tables without retaining a ZSTD_seekable
instance for each.

* New opaque type for seek table: ZSTD_seekTable.
* ZSTD_seekable_copySeekTable() supports copying seek table out of a
  ZSTD_seekable.
* ZSTD_seekTable_[eachSeekTableOp]() defines seek table API that mirrors
  existing seek table operations.
* Existing ZSTD_seekable_[eachSeekTableOp]() retained; they delegate to
  ZSTD_seekTable the variant.

These changes allow the above-mentioned use cases to initialize a
ZSTD_seekable, extract its ZSTD_seekTable, then throw the ZSTD_seekable
away to save memory. Standard ZSTD operations can then be used to
decompress frames based on seek table offsets.

The copy and delegate patterns are intended to minimize impact on
existing code and clients. Using copy instead of move for the infrequent
operation extracting a seek table ensures that the extraction does not
render the ZSTD_seekable useless. Delegating to *new* seek
table-oriented APIs ensures that this is not a breaking change for
existing clients while supporting all meaningful operations that depend
only on seek table data.
 2020-05-07 09:31:43 -04:00
Yann Collet 54144285fd small speed improvement for strategy fast 
gcc 9.3.0 :
kennedy : 459 -> 466
silesia : 360 -> 365
enwik8  : 267 -> 269

clang 10.0.0 :
kennedy : 436 -> 441
silesia : 364 -> 366
enwik8  : 271 -> 272
 2020-05-07 06:15:58 -07:00
Nick TerrellandNick Terrell 5717bd39ee [lib] Fix NULL pointer dereference 
When the output buffer is `NULL` with size 0, but the frame content size
is non-zero, we will write to the NULL pointer because our bounds check
underflowed.

This was exposed by a recent PR that allowed an empty frame into the
single-pass shortcut in streaming mode.

* Fix the bug.
* Fix another NULL dereference in zstd-v1.
* Overflow checks in 32-bit mode.
* Add a dedicated test.
* Expose the bug in the dedicated simple_decompress fuzzer.
* Switch all mallocs in fuzzers to return NULL for size=0.
* Fix a new timeout in a fuzzer.

Neither clang nor gcc show a decompression speed regression on x86-64.
On x86-32 clang is slightly positive and gcc loses 2.5% of speed.

Credit to OSS-Fuzz.
 2020-05-06 12:09:02 -07:00
Felix HandteandGitHub ad8dbae1b7 Merge pull request #2103 from felixhandte/relative-includes 
Migrate Includes to Relative Paths
 2020-05-06 09:42:23 -07:00
Bimba ShresthaandGitHub e7df0d41bb Merge pull request #2095 from bimbashrestha/grep 
[bugs] zstdgrep/grep inconsistencies
 2020-05-06 11:18:15 -05:00
Yann ColletandGitHub d116f24070 Merge pull request #2109 from facebook/fasterCygwin 
faster cygwin test
 2020-05-05 20:50:40 -07:00
Bimba Shrestha 250184adf6 adding tests back 2020-05-05 16:51:06 -07:00
Yann ColletandGitHub e09cdf9080 Merge pull request #2108 from facebook/hufc_wconv 
some more conversion fixes …
 2020-05-05 16:38:17 -07:00
Yann Collet 4f755bdafc faster cygwin test 
The cygwin test on Appveyor is way too long (>30 mn).
The main issue is the very long zstreamtest, because no time out has been programmed.
Ensures that both fuzzer and zstreamtest receive a 30sec slot.
Should reduce cygwin tests duration by -30mn.
 2020-05-05 14:08:35 -07:00
W. Felix Handte 5163778c02 Yay It Works, Clean Up Logging A Bit 2020-05-05 15:27:27 -04:00
W. Felix Handte 005ffd3bdb Fix Sed Regex Escaping for Extended Regex Rules...... 2020-05-05 14:54:04 -04:00
W. Felix Handte bdfb50f7c4 Uhhh It's Still Failing, Let's Try Printing Debug Info? 2020-05-05 13:58:22 -04:00
Yann Collet ef7d98e55a Merge branch 'dev' into hufc_wconv 2020-05-05 10:23:00 -07:00
W. Felix Handte 229cbd056e Fix Fallbacks to Work on FreeBSD 2020-05-05 13:18:45 -04:00
Yann Collet c29fd7cd8b some more conversion warnings 
hunting down some static analyzer warnings
 2020-05-05 10:16:59 -07:00
Bimba Shrestha 6ff16b4246 adding help text 2020-05-05 10:03:57 -07:00
Bimba Shrestha a06a508bf3 fixing inconsistencies 2020-05-05 09:56:15 -07:00
Yann ColletandGitHub 7120d97cb3 fix minor conversion warnings (#2106) 2020-05-05 09:08:05 -07:00
W. Felix Handte 6381b7ee3f Add Fallback When realpath Doesn't Support --relative-to 2020-05-04 18:59:47 -04:00
W. Felix Handte c7cba8e87c Mark zstddeclib-in.c Not Executable 2020-05-04 18:59:26 -04:00
Yann Collet c1b836f4c3 fix minor conversion warnings 2020-05-04 14:43:09 -07:00
Felix HandteandGitHub 8b327149a8 Merge pull request #1976 from felixhandte/minimal-lib-target 
Add Minification Variable to `lib/Makefile`
 2020-05-04 12:42:56 -07:00
W. Felix Handte b48f6c7d26 Try to Fix Single File Library Combiner Script to Handle Relative Includes 2020-05-04 15:20:26 -04:00
W. Felix Handte 7b75d772b1 Remove Useless Assignment in Makefile 2020-05-04 15:20:26 -04:00
W. Felix Handte 7dcca6bc64 Also Move programs/ Directory to Relative Includes 2020-05-04 15:20:26 -04:00
W. Felix Handte 6028827fee Rewrite Include Paths to be Relative 
Addresses #1998.
 2020-05-04 15:20:26 -04:00
Bimba ShresthaandGitHub 0301ef5d04 [bench] Extending largeNbDicts to compression (#2089) 
* adding cdict_collection_t

* adding shuffleCDictionaries()

* adding compressInstructions

* adding compress()

* integrating compression into bench()

* copy paste error fix

* static analyzer uninit value complaint fix

* changing to control

* removing assert

* changing to control

* moving memcpy to seperate function

* fixing static analyzer complaint

* another hacky solution attempt

* Copying createbuffer logic
 2020-05-04 10:42:22 -07:00
Felix HandteandGitHub 7e9aabd652 Merge pull request #2099 from felixhandte/compile-under-pedantic 
Compile Under `-pedantic -Werror` and `-std=c90`
 2020-05-04 10:07:13 -07:00
W. Felix Handte 0b81035101 Test Flag in Travis CI 2020-05-04 11:22:07 -04:00
W. Felix Handte fa5e01c467 Add Space-Optimized Helper Variable to Lib Makefile 
This diff reorganizes the `lib/Makefile` to extract various settings that a
user would normally invoke together (supposing that they were aware of them)
if they were trying to build the smallest `libzstd` possible. It collects
these settings under a master setting `ZSTD_LIB_MIN_SIZE`.

Also document this new option.
 2020-05-04 11:19:25 -04:00
Felix HandteandGitHub 816ed80774 Merge pull request #1984 from MeghnaM/1636-Reduce-stack-usage-of-HUF_sort 
Reduce stack usage of HUF_sort()
 2020-05-04 08:15:31 -07:00
W. Felix Handte 2cf72d56a6 Try to Fix MSVC Error 
It's complaining about the `memcpy`s, saying:

"warning C4090: 'function': different 'const' qualifiers"

Let's try explicitly casting to the argument types...
 2020-05-04 10:59:15 -04:00
W. Felix Handte 6528fa088e Do a -pedantic Build in CircleCI 2020-05-04 10:59:15 -04:00
W. Felix Handte dacbcd2cc1 Fix Up Some Pointer Handling in Tests 2020-05-04 10:59:15 -04:00
W. Felix Handte 3764859060 Switch Helper Declaration to Not Force Inline 
It was causing build issues in ANSI mode.
 2020-05-04 10:59:15 -04:00
W. Felix Handte c7da66c9cf Purge C++-Style Comments (// ...), Make Compilation Succeed Under C90 2020-05-04 10:59:15 -04:00
W. Felix Handte 952427aebf Avoid inline Keyword in C90 
Previously we would use it for all gcc-like compilations, even when a
restrictive mode that disallowed it had been selected.
 2020-05-04 10:59:15 -04:00