ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,656 Commits 17 Branches 77 Tags
dev
Commit Graph
10656 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet d1327738c2 updated double_fast complementary insertion 
in a way which is more favorable to compression ratio,
though very slightly slower (~-1%).

More details in the PR.
 2019-07-11 15:25:22 -07:00
Yann ColletandGitHub b01c1c679f Merge pull request #1675 from ephiepark/dev 
Factor out the logic to build sequences
 2019-07-10 13:32:31 -07:00
Yann Collet 2387d574cb updated .gitignore 2019-07-09 14:54:48 -07:00
Yann Collet 34a1a37246 updated .gitignore rule 2019-07-09 14:44:22 -07:00
Yann ColletandGitHub 8eda16c9f6 Merge pull request #1677 from LeeYoung624/gitignore_fix 
fix gitignore errors
 2019-07-09 14:36:38 -07:00
Yann Collet b8ec4b0fd6 updated version number (to v1.4.1) 
also : added doc on context re-use, as suggested by @scherepanov at #1676
 2019-07-09 11:43:59 -07:00
LeeYoung624 654cb9d439 fix gitignore errors 2019-07-09 21:08:13 +08:00
Yann ColletandGitHub 096714d1b8 Merge pull request #1671 from ephiepark/dev 
Adding targetCBlockSize param
 2019-07-03 17:47:44 -07:00
Ephraim Park f57ac7b09e Factor out the logic to build sequences 2019-07-03 15:42:38 -07:00
Ephraim Park 9007701670 Adding targetCBlockSize param 2019-07-03 15:41:52 -07:00
Nick TerrellandGitHub e962f07d19 [fuzz] Add a compression fuzzer with randomly sized output buffer (#1670) 2019-07-02 22:05:07 -07:00
Nick TerrellandGitHub 6c92ba774e ZSTD_compressSequences_internal assert op <= oend (#1667) 
When we wrote one byte beyond the end of the buffer for RLE
blocks back in 1.3.7, we would then have `op > oend`. That is
a problem when we use `oend - op` for the size of the destination
buffer, and allows further writes beyond the end of the buffer for
the rest of the function. Lets assert that it doesn't happen.
 2019-07-02 15:45:47 -07:00
Yann ColletandGitHub 857e608b51 Merge pull request #1658 from facebook/memset 
memset() rather than reduceIndex()
 2019-07-01 15:01:43 -07:00
Yann ColletandGitHub 4d611ca405 Merge pull request #1664 from ephiepark/dev 
decodecorpus
 2019-07-01 14:13:49 -07:00
Ephraim ParkandGitHub 01f5b5d918 Merge pull request #7 from ephiepark/decodecorpus 
reflect code review comments
 2019-07-01 10:18:59 -07:00
Ephraim Park 28309520c0 reflect code review comments 2019-07-01 10:17:30 -07:00
Yann ColletandGitHub 068cc18927 Merge pull request #1665 from lzutao/meson-deprecated-warnings 
meson: Fix deprecated build warnings on build options
 2019-06-30 11:36:35 -07:00
Lzu Tao 132a1ad291 meson: Always build gen_html on build machine 
Because we use gen_html as a generator instead of a binary to
run on host machine.
 2019-06-29 02:22:39 +07:00
Lzu Tao 8e590a1af3 meson: Fix deprecated build warnings on build options 
Meson now reserves the `build_` prefix for options
 2019-06-29 02:22:39 +07:00
Lzu Tao 4d156854fe meson: Beautify travis config 2019-06-29 01:42:17 +07:00
Ephraim ParkandGitHub 9f4d71f48d Merge pull request #6 from ephiepark/decodecorpus 
Add test case for short bistream
 2019-06-27 17:38:50 -07:00
Ephraim Park 01e8384000 Add test case for short bistream 2019-06-27 17:37:39 -07:00
Tyler-TranandNick Terrell c55d2e7ba3 Adding shrinking flag for cover and fastcover (#1656) 
* Changed ERROR(GENERIC) excluding inits

* editing git ignore

* Edited init functions to size_t returns

* moved declarations earlier

* resolved issues with changes to init functions

* fixed style and an error check

* attempting to add tests that might trigger changes

* added && die to cases expecting to fail

* resolved no die on expected failed command

* fixed accel to be incorrect value

* Adding an automated shrinking option

* Fixing build

* finalizing fixes

* fix?

* Removing added comment in cover.h

* Styling fixes

* Merging with fb dev

* removing megic number for default regression

* Requested revisions

* fixing support for fast cover

* fixing casting errors

* parenthesis fix

* fixing some build nits

* resolving travis ci syntax

* might resolve all compilation issues

* removed unused variable

* remodeling the selectDict function

* fixing bad memory access

* fixing error checks

* fixed erroring check in selectDict

* fixing mixed declarations

* modify mixed declaration

* fixing nits and adding test cases

* Adding requested changes + fixed bug for error checking

* switched double comparison from != to <

* fixed declaration typing

* refactoring COVER_best_finish() and changing shrinkDict

* removing the const's

* modifying ZDICT_optimizeTrainFromBuffer_cover functions

* fixing potential bad memcpy

* fixing the error function for dict size
 2019-06-27 16:26:57 -07:00
Ephraim ParkandGitHub 36d0bc2617 Merge pull request #5 from ephiepark/decodecorpus 
Decodecorpus
 2019-06-27 15:02:48 -07:00
Ephraim Park c7c1ba3a19 Fix a constraint stricter than the spec 2019-06-26 16:43:37 -07:00
Ephraim Park 734eff70b8 enable repeat mode on rle 2019-06-26 16:39:00 -07:00
Yann Collet 621adde3b2 changed naming to ZSTD_indexTooCloseToMax() 
Also : minor speed optimization :
shortcut to ZSTD_reset_matchState() rather than the full reset process.
It still needs to be completed with ZSTD_continueCCtx() for proper initialization.

Also : changed position of LDM hash tables in the context,
so that the "regular" hash tables can be at a predictable position,
hence allowing the shortcut to ZSTD_reset_matchState() without complex conditions.
 2019-06-24 14:39:29 -07:00
Nick TerrellandGitHub 9038579ab2 Merge pull request #1659 from terrelln/big-dict 
Fix data corruption in niche use case
 2019-06-24 12:40:58 -07:00
Nick Terrell 8e16b872e5 [tests] Add tests for big dictionaries 2019-06-21 17:58:24 -07:00
Yann Collet 45c9fbd6d9 prefer memset() rather than reduceIndex() when close to index range limit 
by disabling continue mode when index is close to limit.
 2019-06-21 16:19:21 -07:00
Yann Collet 944e2e9e12 benchfn : added macro macro CONTROL() 
like assert() but cannot be disabled.
proper separation of user contract errors (CONTROL())
and invariant verification (assert()).
 2019-06-21 15:58:55 -07:00
Nick Terrell 674534a700 [zstd] Fix data corruption in niche use case 
* Extract the overflow correction into a helper function.
* Load the dictionary `ZSTD_CHUNKSIZE_MAX = 512 MB` bytes at a time
  and overflow correct between each chunk.

Data corruption could happen when all these conditions are true:

* You are using multithreading mode
* Your overlap size is >= 512 MB (implies window size >= 512 MB)
* You are using a strategy >= ZSTD_btlazy
* You are compressing more than 4 GB

The problem is that when loading a large dictionary we don't do
overflow correction. We can only load 512 MB at a time, and may
need to do overflow correction before each chunk.
 2019-06-21 15:47:31 -07:00
Nick Terrell 4156060ca4 [zstdmt] Update assert to use ZSTD_WINDOWLOG_MAX 2019-06-21 15:39:33 -07:00
Nick Terrell 95e2b430ea [opt] Add asserts for corruption in ZSTD_updateTree() 2019-06-21 15:22:29 -07:00
Yann ColletandGitHub b26818fe73 Merge pull request #1626 from LeeYoung624/dev 
add cmake lz4 support
 2019-06-19 13:35:33 -07:00
Nick TerrellandGitHub aeb86b109a Merge pull request #1655 from terrelln/regression-test 
[regression] Update results for small wlog patch PR#1624
 2019-06-17 11:57:41 -07:00
Nick Terrell 06d5e5ff5a [regression] Update results for small wlog patch PR#1624 2019-06-17 10:40:33 -07:00
Yann ColletandGitHub 837ff9d206 Merge pull request #1647 from LeeYoung624/cmake_bug_fix 
CMake bug fix: didn't install zstdless and zstdgrep.
 2019-06-14 17:32:05 -07:00
Yann ColletandGitHub 1166782eb7 Merge pull request #1640 from felixhandte/lstat-macro-guard 
Protect lstat() With Better Macro Guard
 2019-06-14 17:30:03 -07:00
Yann ColletandGitHub 9af909bf35 Merge pull request #1624 from facebook/smallwlog 
Improves compression ratio for small windowLog
 2019-06-14 17:28:21 -07:00
Yann ColletandGitHub 018df9c9b1 Merge pull request #1650 from scharan/RemoveExportsForStaticLibrary 
Remove ZSTD_DLL_EXPORT=1 for static lib
 2019-06-14 17:09:00 -07:00
Yann ColletandGitHub b0592024e2 Merge pull request #1644 from chungy/chmod_600 
[programs] set chmod 600 after opening destination file
 2019-06-13 17:25:31 -07:00
Sai Charan Koduru f547dca9e3 Remove ZSTD_DLL_EXPORT=1 for static lib 
As a principle, static libs should not dllexport methods, that should only be used when building DLLs.

Case in point: when static libs with dllexport directives are linked into DLLs created with a .def file, the VC++ compiler exports the dllexported methods into the DLL, in addition to the exports listed in the .def file. This will result in undesired link dependencies and is not the correct thing to do.
 2019-06-12 12:26:36 -07:00
LeeYoung624 a4c0c27410 Install zstdless & zstdgrep as 'PROGRAMS' in CMake 2019-06-12 09:57:28 +08:00
Felix HandteandGitHub 344f942573 Merge pull request #1620 from michaelforney/test-no-threads 
Skip --adapt and --rsyncable tests when built without thread support
 2019-06-11 15:05:25 -04:00
Nick TerrellandGitHub 81276c4ee4 Merge pull request #1646 from terrelln/oss-fuzz 
[fuzz] Remove max_len from the options
 2019-06-10 22:13:16 -07:00
LeeandGitHub bc400ed13b Merge pull request #4 from facebook/dev 
merge
 2019-06-11 11:25:26 +08:00
Yann ColletandGitHub c601622aa7 Merge pull request #1619 from j301scott/dev 
CMake: Check for existing custom target 'uninstall'
 2019-06-10 11:56:49 -07:00
Nick Terrell 6810dd6191 [fuzz] Remove max_len from the options 2019-06-10 11:05:45 -07:00
W. Felix Handte d8970199f8 Add Contbuild Test for C99 Build 2019-06-10 11:22:45 -04:00