ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,362 Commits 17 Branches 77 Tags
1416bc0f07b266e4c682f44968af1745dcce064c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 1416bc0f07 erase existence of a buffer when it's sent out of the pool 
In some complex scenario,
the buffer would be freed because it's too large,
another buffer would be allocated, but fail,
trigger an error,
and the general buffer pool would then be freed,
where the definition of the already freed buffer would be found
(beyond total index, but still), and freed again, resulting in double-free error.
 2017-09-29 16:27:47 -07:00
Yann Collet e963800e27 zstdmt : fixed : buffer dst0 wasn't properly set to null after usage 
now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources()
wether previous compression was completed correctly or not.
 2017-09-28 23:01:31 -07:00
Yann Collet 754ae5cc0b removed ZSTDMT_waitForAllJobsCompleted() from ZSTDMT_freeCCtx() 
as per @terrelln comment
 2017-09-28 20:45:31 -07:00
Yann Collet e0065cf660 make test : removed zstreamtest unit tests for variants 
slightly reduced time to create dictionary
at beginning of unit tests
 2017-09-28 18:34:38 -07:00
Yann Collet 47c6a95d07 zstreamtest : run unit tests only during "normal" session 
not during --mt, --newapi and --opaque
this avoids running them 4x during `make test`
 2017-09-28 18:27:22 -07:00
Yann Collet 86b4fe5b45 adjustCParams : restored previous behavior 
unknowns srcSize presumed small if there is a dictionary (dictSize>0)
and presumed large otherwise.
 2017-09-28 18:14:28 -07:00
Yann Collet e4ec427720 Merge branch 'dev' into shorterTests 
fixed conflicts
 2017-09-28 12:19:28 -07:00
Yann Collet bbef058ae6 zstreamtest --newapi : reduced maximum allocated memory 2017-09-28 11:48:45 -07:00
Yann Collet 8074261d00 zstdmt : move on when not enough memory for a new input buffer 
just continue operations without input forward progress,
instead of an error that stops current compression session.
 2017-09-28 11:46:19 -07:00
Yann Collet 2cd15dd9a4 fixed minor Visual conversion warning 2017-09-28 02:33:41 -07:00
Yann Collet 377abcc02c zstdmt : better behavior when freeing a context right after a memory allocation error 
wait for all jobs to be completed, so that freeing can happen safely
 2017-09-28 02:23:44 -07:00
Yann Collet d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet 9fe50ed623 fixed maximum windowLog for zstreamtest --newapi 
for compatibility with low memory VM
 2017-09-28 01:42:06 -07:00
Yann Collet 9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize 
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
 2017-09-28 01:25:40 -07:00
Yann Collet aa800c4793 reduced memory usage of zstreamtest --newapi 
to run on memory-constrained VM
 2017-09-27 18:00:15 -07:00
Yann Collet bc32b40b98 reduced zstreamtest --mt memory load 
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
 2017-09-27 17:27:38 -07:00
Yann Collet 54a827fff0 Merge branch 'dev' into newFormats 
Fixed conflicts in zstdmt_compress.c
 2017-09-27 16:39:40 -07:00
Yann Collet 02502191e5 separated --long tests between short and long tests 
A fast subset of these tests is now played in short test mode
 2017-09-27 15:48:06 -07:00
Yann Collet f9de54acfb reduced memory requirements for --long tests 
in new --long test section
 2017-09-27 15:38:27 -07:00
Yann Collet bdd0f6f046 improved make clean in tests/fuzz 2017-09-27 15:20:08 -07:00
Yann Collet 60059df051 shorter make test to avoid time out on travis CI 
Timed tests (fuzzer) are reduced
Long tests are shortened (less data generated)
 2017-09-27 15:16:27 -07:00
Yann Collet ea1f50bf73 removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal() 
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.

Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.

A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
 2017-09-27 13:51:05 -07:00
Yann Collet 60ca44b545 switched name to cxxtest 2017-09-27 12:24:13 -07:00
Yann Collet c994932788 fixed ZSTD_format_e value validation 2017-09-27 12:22:22 -07:00
Yann Collet 763f8b5e45 Change c++ test to use CXX and CXXFLAGS environment variables 
Fix OS-X warning on compiling .c files with clang++
Also changed test name from gpptest to cpptest, since it's no longer g++ specific
 2017-09-27 12:09:52 -07:00
Yann Collet ecf1778e23 updated ZSTD_format_e value validation 
also updated manual
 2017-09-27 11:19:21 -07:00
Yann Collet 9416195221 changed error code when pos<=size condition is not respected 
Now pointing towards src_size or dst_size,
instead of error_GENERIC.
 2017-09-27 10:35:56 -07:00
Yann Collet d56a350402 removed unsupported formats 2017-09-27 10:29:31 -07:00
Yann Collet bfabd1d4dc fixed zstreamtest decoding error 
same error (wrong output buffer size) was present
on --mt and --new_api tests.
 2017-09-27 01:01:11 -07:00
Yann Collet ca306c1c84 fixed a bug in zstreamtest 
decoder output buffer would receive a wrong size.

In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.

This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
 2017-09-27 00:39:41 -07:00
Yann Collet cd53ac831b fixed DCtx initialization error 
now relying on initialization of dctx->format first
 2017-09-26 18:26:09 -07:00
Yann Collet 4791561c4a silence minor gcc warning -Wempty-body 
also silence fuzz test artefacts
 2017-09-26 17:57:38 -07:00
Yann Collet c0dd960363 switch assert() position 2017-09-26 15:36:57 -07:00
Yann Collet 319c699991 created ZSTD_startingInputLength() 
as suggested by @terrelln
 2017-09-26 15:36:14 -07:00
Yann Collet 8d1e97ea9c minor fixes following @terrelln comments 2017-09-26 15:06:30 -07:00
Yann Collet df4e9bba25 fixed constant errors for gcc in c99 mode 
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
 2017-09-26 14:31:06 -07:00
Yann Collet 9f0b8dfbe9 Merge branch 'dev' into newFormats 2017-09-26 14:22:39 -07:00
Yann Collet 3095ca8c56 fixed minor conversion warnings for g++ on Linux 
U64 is not considered equivalent to unsigned long long
 2017-09-26 13:53:50 -07:00
Yann Collet 56f1f0e3dd write summary for --list on multiple files 2017-09-26 11:21:36 -07:00
Yann Collet 52a1d1c6dc added ZSTD_DCtx_reset() 2017-09-25 16:56:48 -07:00
Yann Collet f2a913862c added ZSTD_decompress_generic_simpleArgs() 2017-09-25 15:46:34 -07:00
Yann Collet 6ee05a02b8 added ZSTD_decompress_generic() 
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
 2017-09-25 15:41:48 -07:00
Yann Collet b8d4a3887f introduced constant ZSTD_frameIdSize 
within zstd_internal.h
This is the size of magic number.

Avoids using `4` directly in source code, which is a bit less meaningful.
 2017-09-25 15:26:18 -07:00
Yann Collet 044fb4c057 implemented magic-less frame decoder 2017-09-25 15:12:09 -07:00
Yann Collet 62568c9a42 added capability to generate magic-less frames 
decoder not implemented yet
 2017-09-25 14:26:26 -07:00
Yann Collet 96f0cde31a minor function rename 
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
 2017-09-24 16:47:02 -07:00
Yann Collet 7c3dea42ce added prototypes for advanced parameters for decompression API 
required to decode custom formats
 2017-09-24 15:57:29 -07:00
Yann Collet e60f48c549 Merge branch 'dev' into newFormats 2017-09-24 14:33:37 -07:00
Yann Collet cd3115b284 added control from frame content size at end of decompression 
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
 2017-09-21 16:21:10 -07:00
Yann Collet 645563583e Merge branch 'dev' into newFormats 2017-09-21 16:08:06 -07:00
Yann Collet f97c2dbd39 created ZSTD_format declaration 2017-09-21 16:07:29 -07:00
Yann Collet bdc83f523b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 15:50:14 -07:00
Yann Collet 9c1908a3cd added streaming_memory_usage example 2017-09-18 15:49:59 -07:00
Yann Collet cb8b471e8b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 14:48:23 -07:00
Yann Collet 7d1ff3817b fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream() 
previous result was over-estimated
by counting streaming buffers twice
 2017-09-18 14:47:34 -07:00
Yann Collet 92889709f9 fix #851 : sudo zstd -t file.zst changes /dev/null permissions 
reported by @mike155
 2017-09-18 13:41:54 -07:00
Yann Collet 539b91ee9b minor : added assert in bt 2017-09-16 23:41:58 -07:00
Yann Collet 1722055799 add comment on using -B# to split input file for dictionary training 2017-09-15 16:23:50 -07:00
Yann Collet c68d17f2da ensures that sampleSizes table is large enough 
as recommended by @terrelln
 2017-09-15 15:31:31 -07:00
Yann Collet 25a60488dd fixed 64-to-32 conversion warnings 2017-09-15 11:55:13 -07:00
Yann Collet a9694231ca fixed minor conversion warning 2017-09-15 10:16:26 -07:00
Yann Collet 086b9597d9 added ability to split input files for dictionary training 
using command -B#
This is the same behavior as benchmark module,
which can also split input into arbitrary size blocks, using -B#.
 2017-09-14 16:45:10 -07:00
Yann Collet 77c137b3ae minor comment refactor 2017-09-14 15:12:57 -07:00
Yann Collet 335780c427 fixed too strong alignment assert in ZSTD_initStaticCCtx() 
64-bits fields are only 32-bits aligned on 32-bits CPU
 2017-09-13 16:35:29 -07:00
Yann Collet 8f26dc3f9c blindfix for Visual 
LARGE_INTEGER is not an integer :
https://msdn.microsoft.com/en-us/library/windows/desktop/aa383713(v=vs.85).aspx

Do not take any risk with the structure definition :
use int init = 0; like Mac code
 2017-09-12 21:21:17 -07:00
Yann Collet bc41c7f0eb fixed minor prototype warning 2017-09-12 19:32:26 -07:00
Yann Collet c95c0c9725 modified util::time API 
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
 2017-09-12 18:12:46 -07:00
Yann Collet 3306bcb0e6 fix #820 : GCC v3.x 32-bits doesn't define 64-bits intrinsic 
resulting in undefined symbol error.
Push the requirement to GCC 4 for now.
Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead.
__LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits)
Might be better, but seems also more complex, hence more prone to side effects.
Keeping the simple solution for now (just rely on __GNUC__)
 2017-09-11 15:17:31 -07:00
Yann Collet f325ee4e84 fixed pass-through warning 2017-09-11 14:37:03 -07:00
Yann Collet 0d6ecc72a3 makes it possible to compile libzstd in single-thread mode without zstdmt_compress.c (#819) 2017-09-11 14:09:34 -07:00
Yann Collet ce31004f20 fix following suggestions by @terrelln 2017-09-11 13:12:52 -07:00
Yann Collet b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream() 
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
 2017-09-09 14:37:28 -07:00
Yann Collet 058ed2ad33 ZSTD_decodingBufferSize_min() 
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.

also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
 2017-09-09 01:03:29 -07:00
Yann Collet 3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Yann Collet baa37c3362 programs/Makefile : better support for GNU conventions 
see https://www.gnu.org/prep/standards/html_node/Command-Variables.html
 2017-09-06 16:53:59 -07:00
Yann Collet 3a12531a3d lib/Makefile : better support for GNU conventions 
see https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html
 2017-09-06 16:35:49 -07:00
Yann Collet 1c7b914cdf update README on BUCK file 2017-09-06 16:23:39 -07:00
Yann Collet 36374cc3b4 update and clarify lib/README 2017-09-06 16:15:18 -07:00
Yann Collet ed7ace38e9 updated NEWS 
fix for 32-bits decoder
 2017-09-01 11:58:37 -07:00
Yann Collet 8a5c0c98ae restored 32-bits decoder ability to decode long offsets (>32 MB, levels 21+) 2017-09-01 11:56:57 -07:00
Yann Collet 0558850735 bench stops immediately on decoding error 2017-09-01 11:46:15 -07:00
Yann Collet 36aa8b5999 improved decoding speed 2017-09-01 11:40:59 -07:00
Yann Collet d963daa6a9 fixed minor warning (empty translation unit) 2017-09-01 00:12:07 -07:00
Yann Collet 3704507774 fixed decompression bug reported by @Etsukata (#828) 2017-09-01 00:05:37 -07:00
Yann Collet 0c314cde4b updated zstd API manual 
for new CCtxParams object
 2017-08-31 18:28:19 -07:00
Yann Collet 369c29dd1a fixed impact of merge conflict for longRange 2017-08-31 18:25:56 -07:00
Yann Collet c7818fc676 Merge branch 'modTests' into dev 
fixed conflict
 2017-08-31 17:00:16 -07:00
Yann Collet 4299c27132 improved console log of utils.h 
removed a warning when compiling on Windows
 2017-08-31 16:58:47 -07:00
Yann Collet d791aef80e blind attempt at removing gcc dependency 
from appveyor's mingw builds, for #815
 2017-08-31 15:24:17 -07:00
Yann Collet 4e81dc04f9 Merge branch 'dev' into modTests 2017-08-31 15:14:13 -07:00
Yann Collet 3982edf813 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-31 15:13:48 -07:00
Yann Collet e0cecd8736 fixed poolTests 
needs more dependencies from zstd for custom allocators and error codes
 2017-08-31 15:13:31 -07:00
Yann Collet 179b161dc1 fixed poolTest 
needs more dependencies from zstd for custom allocators and error codes
 2017-08-31 15:02:14 -07:00
Yann Collet 8e298382a8 changed target allarch into allzstd 
allzstd contains only zstd-related tests.
allmost = allzstd + zwrapper tests (which require zlib)
 2017-08-31 14:30:52 -07:00
Yann Collet b0cb081dc8 last batch of header files changed to reflect new license (#825) 
only remains to update contrib/linux-kernel (@terrelln)
 2017-08-31 12:20:50 -07:00
Yann Collet e21384fffb fixed more file headers after license change (#825) 2017-08-31 12:11:57 -07:00
Yann Collet e9dc204f42 fixed a bunch of headers after license change (#825) 2017-08-31 11:24:54 -07:00
Yann Collet 9023898f1e updated NEWS 2017-08-30 17:44:12 -07:00
Yann Collet 394bdd7db9 changed license for examples 
intentionnally this time
 2017-08-29 09:24:11 -07:00
Yann Collet 6b2b6a9bd5 fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously 
can only happen if windowLog>=27  (level 22 --ultra)
 2017-08-22 12:09:21 -07:00