ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,429 Commits 17 Branches 77 Tags
f4340f46b2387bc8de7d5320c0b83bb1499933ad
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet f4340f46b2 make clean before zstd-lz4 compatibility tests 
to ensure zstd is rebuilt with lz4 support
 2017-10-08 02:41:28 -07:00
Yann Collet d29bf941b9 added cxxtest to master branch 
C++ compilation will be tested every night
 2017-10-08 00:12:17 -07:00
Yann Collet 7f6a783862 fixed a small error in decodeCorpus 
a compressed block must be strictly smaller than its decompressed size.
 2017-10-07 15:19:52 -07:00
Yann Collet 2b2f3c6f64 Merge branch 'dev' for v1.3.2 2017-10-06 12:13:27 -07:00
Yann Collet 4252621e26 playtests: do not use cat on large files 
some target have limitation making cat incompatible with large files
(namely debian hurd-i386)
 2017-10-05 20:21:59 -07:00
Yann Collet 67478f4cb0 fixed minor conversion warnings for printf 
in debug mode
 2017-10-02 17:28:57 -07:00
Yann Collet 9b166d2291 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-10-02 16:34:26 -07:00
Yann Collet 3b27ed41fd Merge branch 'srcSize' into dev 2017-10-02 16:34:14 -07:00
Yann Collet 7e00df4a49 bumped version number 
and updated NEWS in anticipation for release
 2017-10-02 16:27:25 -07:00
Yann Collet 4946993f87 removed isRegularFile parameter 
no longer useful : size of src is determined for each file.
 2017-10-02 12:29:25 -07:00
Yann Collet 7f580f9ee8 interruption handler and variable are static 2017-10-02 11:39:05 -07:00
Yann Collet 0d58aaf6f0 /contrib: fixed license header 
removed last reference to PATENTS file
 2017-10-02 02:07:17 -07:00
Yann Collet fe5444bc66 removed the statement for all versions of Visual Studio 2017-10-02 02:02:16 -07:00
Yann Collet 51d82d5516 same error in Visual Studio 2012 ... 2017-10-02 01:12:40 -07:00
Yann Collet ed7ae4c9bd The issue also impacts Visual Studio 2010 2017-10-02 00:45:28 -07:00
Yann Collet 6e7ba3df2f added (void)sig to avoid compilers complaining that sig is not used. 2017-10-02 00:19:47 -07:00
Yann Collet 82bc200f82 conditionnally removed invocation that generates a buggy warning with Visual Studio 2008 2017-10-02 00:02:24 -07:00
Yann Collet 5e4ad557bc Merge branch 'dev' of github.com:facebook/zstd into dev 2017-10-01 15:33:57 -07:00
Yann Collet bd18095edc blindfix for Visual : minor casting issue 
should not happen since SIGIGN is provided by <signal.h>,
so it should work "ouf of the box"
 2017-10-01 15:32:48 -07:00
Yann Collet 00fc1ba8dd cli: add Ctrl-C support, requested by @mike155 in #854 
Now, pressing Ctrl-C during compression or decompression
will erase operation artefact (unfinished destination file)
before leaving execution.
 2017-10-01 12:10:26 -07:00
Yann Collet 5db19b8685 added comment on ZSTD_COMPRESSBOUND() 
as requested by @terrelln
 2017-10-01 11:32:38 -07:00
Yann Collet 6e930c13d1 Merge branch 'dev' into compressBound 2017-10-01 11:24:02 -07:00
Yann Collet 76ac0b2d99 macro compatible with scenario where windowSize = 1024 (minimum) 2017-09-30 15:34:44 -07:00
Yann Collet dc404119e5 ZSTD_adjustCParams_internal : minor optimization 2017-09-30 15:02:40 -07:00
Yann Collet ee1ed78fcb fix proper naming on FSE_createCTable() arguments in fse.h 2017-09-30 11:08:50 -07:00
Yann Collet 5b10345b26 added ZSTD_COMPRESSBOUND() as a macro 
ZSTD_compressBound() works fine, but is only useful for dynamic allocation.
For static allocation, only a macro can provide the amount during compilation time.
 2017-09-29 23:17:41 -07:00
Yann Collet 8afb151c9b cli: fixed wrong initialization in MT mode 
It's not good to mix old and new API
ZSTD_resetCStream() doesn't just set pledgedSrcSize :
it also sets the CCtx for a single thread compression.

Problem is, when 2+ threads are defined in cctx->requestedParams,
ZSTD_compress_generic() will want to start MT compression,
since initialization is supposed to have already happened (thanks to ZSTD_resetCStream())
except that the underlying ZSTDMT_CCtx* object is not created,
resulting in a segfault.

This is an invalid construction
(correct one is to use ZSTD_CCtx_setPledgedSrcSize()).
I haven't found a nice way to mitigate this impact if someone makes the same mistake.

At some point, removing the old API to keep only the new API within fileio.c will limit these risks.
 2017-09-29 22:14:37 -07:00
Yann Collet fbd5ab7027 minor fix : no longer use fake srcSize during resource creation 
srcSize is read and provided at each file, not at resource creation.
This used to be useful with older API, because it could not re-adapt parameters between sessions.

At some point, it will be better to remove the old code, and only keep the new_api.
It works fine by now.
 2017-09-29 19:40:27 -07:00
Yann Collet db1668a43b fix : srcSize written in frame header when multiple files compressed 
This information used to be disabled when nbFiles>1.
It was badly initialized later in the code, resulting in an error.
 2017-09-29 18:05:18 -07:00
Yann Collet 1416bc0f07 erase existence of a buffer when it's sent out of the pool 
In some complex scenario,
the buffer would be freed because it's too large,
another buffer would be allocated, but fail,
trigger an error,
and the general buffer pool would then be freed,
where the definition of the already freed buffer would be found
(beyond total index, but still), and freed again, resulting in double-free error.
 2017-09-29 16:27:47 -07:00
Yann Collet 8afcc80e07 decode more data before triggering error 
fixes #874 :
when a frame is not properly terminated by a "last block" signal,
zstd -d used to detect it immediately and error out.
This version will decode and flush the last block, and only then issue an error.
 2017-09-29 15:54:09 -07:00
Yann Collet e963800e27 zstdmt : fixed : buffer dst0 wasn't properly set to null after usage 
now it's possible to unconditionnally invoke ZSTD_releaseAllJobRessources()
wether previous compression was completed correctly or not.
 2017-09-28 23:01:31 -07:00
Yann Collet 754ae5cc0b removed ZSTDMT_waitForAllJobsCompleted() from ZSTDMT_freeCCtx() 
as per @terrelln comment
 2017-09-28 20:45:31 -07:00
Yann Collet e0065cf660 make test : removed zstreamtest unit tests for variants 
slightly reduced time to create dictionary
at beginning of unit tests
 2017-09-28 18:34:38 -07:00
Yann Collet 47c6a95d07 zstreamtest : run unit tests only during "normal" session 
not during --mt, --newapi and --opaque
this avoids running them 4x during `make test`
 2017-09-28 18:27:22 -07:00
Yann Collet 86b4fe5b45 adjustCParams : restored previous behavior 
unknowns srcSize presumed small if there is a dictionary (dictSize>0)
and presumed large otherwise.
 2017-09-28 18:14:28 -07:00
Yann Collet b93598d6a4 zstdmt : reduced maximum nb of threads 
to avoid memory address space issues on 32-bits systems
(see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=876416#17)
 2017-09-28 13:49:12 -07:00
Yann Collet e4ec427720 Merge branch 'dev' into shorterTests 
fixed conflicts
 2017-09-28 12:19:28 -07:00
Yann Collet bbef058ae6 zstreamtest --newapi : reduced maximum allocated memory 2017-09-28 11:48:45 -07:00
Yann Collet 8074261d00 zstdmt : move on when not enough memory for a new input buffer 
just continue operations without input forward progress,
instead of an error that stops current compression session.
 2017-09-28 11:46:19 -07:00
Yann Collet 2cd15dd9a4 fixed minor Visual conversion warning 2017-09-28 02:33:41 -07:00
Yann Collet 377abcc02c zstdmt : better behavior when freeing a context right after a memory allocation error 
wait for all jobs to be completed, so that freeing can happen safely
 2017-09-28 02:23:44 -07:00
Yann Collet d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet 9fe50ed623 fixed maximum windowLog for zstreamtest --newapi 
for compatibility with low memory VM
 2017-09-28 01:42:06 -07:00
Yann Collet 9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize 
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
 2017-09-28 01:25:40 -07:00
Yann Collet aa800c4793 reduced memory usage of zstreamtest --newapi 
to run on memory-constrained VM
 2017-09-27 18:00:15 -07:00
Yann Collet bc32b40b98 reduced zstreamtest --mt memory load 
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
 2017-09-27 17:27:38 -07:00
Yann Collet 54a827fff0 Merge branch 'dev' into newFormats 
Fixed conflicts in zstdmt_compress.c
 2017-09-27 16:39:40 -07:00
Yann Collet 02502191e5 separated --long tests between short and long tests 
A fast subset of these tests is now played in short test mode
 2017-09-27 15:48:06 -07:00
Yann Collet f9de54acfb reduced memory requirements for --long tests 
in new --long test section
 2017-09-27 15:38:27 -07:00
Yann Collet bdd0f6f046 improved make clean in tests/fuzz 2017-09-27 15:20:08 -07:00
Yann Collet 60059df051 shorter make test to avoid time out on travis CI 
Timed tests (fuzzer) are reduced
Long tests are shortened (less data generated)
 2017-09-27 15:16:27 -07:00
Yann Collet ea1f50bf73 removed ZSTD_decompressBegin() from ZSTD_initDCtx_internal() 
It does not feel "right" from a dependency perspective.
ZSTD_initDCtx_internal() is triggered once, on DCtx creation,
while ZSTD_decompressBegin() is invoked at the beginning of each new frame,
and is also a user-facing prototype.

Downside : a DCtx must be init before first usage !
This was always the intention by the way, and is documented as such.
This stage is automatically done within ZSTD_decompress() and variants,
and also within ZSTD_decompressStream().
Only ZSTD_decompressContinue() is impacted,
it must be preceded by a ZSTD_decompressBegin(), as detailed in doc.

A test has been fixed, to no longer rely on undocumented assumption that ZSTD_decompressBegin() is invoked during init.
 2017-09-27 13:51:05 -07:00
Yann Collet 60ca44b545 switched name to cxxtest 2017-09-27 12:24:13 -07:00
Yann Collet c994932788 fixed ZSTD_format_e value validation 2017-09-27 12:22:22 -07:00
Yann Collet 763f8b5e45 Change c++ test to use CXX and CXXFLAGS environment variables 
Fix OS-X warning on compiling .c files with clang++
Also changed test name from gpptest to cpptest, since it's no longer g++ specific
 2017-09-27 12:09:52 -07:00
Yann Collet ecf1778e23 updated ZSTD_format_e value validation 
also updated manual
 2017-09-27 11:19:21 -07:00
Yann Collet 9416195221 changed error code when pos<=size condition is not respected 
Now pointing towards src_size or dst_size,
instead of error_GENERIC.
 2017-09-27 10:35:56 -07:00
Yann Collet d56a350402 removed unsupported formats 2017-09-27 10:29:31 -07:00
Yann Collet bfabd1d4dc fixed zstreamtest decoding error 
same error (wrong output buffer size) was present
on --mt and --new_api tests.
 2017-09-27 01:01:11 -07:00
Yann Collet ca306c1c84 fixed a bug in zstreamtest 
decoder output buffer would receive a wrong size.

In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.

This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
 2017-09-27 00:39:41 -07:00
Yann Collet b0a3ad96d3 reduced OS-X tests as they time out regularly 2017-09-26 19:41:11 -07:00
Yann Collet cd53ac831b fixed DCtx initialization error 
now relying on initialization of dctx->format first
 2017-09-26 18:26:09 -07:00
Yann Collet 4791561c4a silence minor gcc warning -Wempty-body 
also silence fuzz test artefacts
 2017-09-26 17:57:38 -07:00
Yann Collet c0dd960363 switch assert() position 2017-09-26 15:36:57 -07:00
Yann Collet 319c699991 created ZSTD_startingInputLength() 
as suggested by @terrelln
 2017-09-26 15:36:14 -07:00
Yann Collet 8d1e97ea9c minor fixes following @terrelln comments 2017-09-26 15:06:30 -07:00
Yann Collet df4e9bba25 fixed constant errors for gcc in c99 mode 
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
 2017-09-26 14:31:06 -07:00
Yann Collet 9f0b8dfbe9 Merge branch 'dev' into newFormats 2017-09-26 14:22:39 -07:00
Yann Collet 3095ca8c56 fixed minor conversion warnings for g++ on Linux 
U64 is not considered equivalent to unsigned long long
 2017-09-26 13:53:50 -07:00
Yann Collet 56f1f0e3dd write summary for --list on multiple files 2017-09-26 11:21:36 -07:00
Yann Collet 52a1d1c6dc added ZSTD_DCtx_reset() 2017-09-25 16:56:48 -07:00
Yann Collet f2a913862c added ZSTD_decompress_generic_simpleArgs() 2017-09-25 15:46:34 -07:00
Yann Collet 6ee05a02b8 added ZSTD_decompress_generic() 
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
 2017-09-25 15:41:48 -07:00
Yann Collet b8d4a3887f introduced constant ZSTD_frameIdSize 
within zstd_internal.h
This is the size of magic number.

Avoids using `4` directly in source code, which is a bit less meaningful.
 2017-09-25 15:26:18 -07:00
Yann Collet 044fb4c057 implemented magic-less frame decoder 2017-09-25 15:12:09 -07:00
Yann Collet 62568c9a42 added capability to generate magic-less frames 
decoder not implemented yet
 2017-09-25 14:26:26 -07:00
Yann Collet 96f0cde31a minor function rename 
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
 2017-09-24 16:47:02 -07:00
Yann Collet 7c3dea42ce added prototypes for advanced parameters for decompression API 
required to decode custom formats
 2017-09-24 15:57:29 -07:00
Yann Collet e60f48c549 Merge branch 'dev' into newFormats 2017-09-24 14:33:37 -07:00
Yann Collet cd3115b284 added control from frame content size at end of decompression 
adding check at end of single-pass ZSTD_decompressFrame().
Check within ZSTD_decompressContinue() was already added in a previous patch : b3f33ccfb3
 2017-09-21 16:21:10 -07:00
Yann Collet 645563583e Merge branch 'dev' into newFormats 2017-09-21 16:08:06 -07:00
Yann Collet f97c2dbd39 created ZSTD_format declaration 2017-09-21 16:07:29 -07:00
Yann Collet bdc83f523b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 15:50:14 -07:00
Yann Collet 9c1908a3cd added streaming_memory_usage example 2017-09-18 15:49:59 -07:00
Yann Collet cb8b471e8b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-09-18 14:48:23 -07:00
Yann Collet 7d1ff3817b fix ZSTD_sizeof_CCtx() / ZSTD_sizeof_CStream() 
previous result was over-estimated
by counting streaming buffers twice
 2017-09-18 14:47:34 -07:00
Yann Collet 92889709f9 fix #851 : sudo zstd -t file.zst changes /dev/null permissions 
reported by @mike155
 2017-09-18 13:41:54 -07:00
Yann Collet 539b91ee9b minor : added assert in bt 2017-09-16 23:41:58 -07:00
Yann Collet 1722055799 add comment on using -B# to split input file for dictionary training 2017-09-15 16:23:50 -07:00
Yann Collet c68d17f2da ensures that sampleSizes table is large enough 
as recommended by @terrelln
 2017-09-15 15:31:31 -07:00
Yann Collet 25a60488dd fixed 64-to-32 conversion warnings 2017-09-15 11:55:13 -07:00
Yann Collet a9694231ca fixed minor conversion warning 2017-09-15 10:16:26 -07:00
Yann Collet 086b9597d9 added ability to split input files for dictionary training 
using command -B#
This is the same behavior as benchmark module,
which can also split input into arbitrary size blocks, using -B#.
 2017-09-14 16:45:10 -07:00
Yann Collet 77c137b3ae minor comment refactor 2017-09-14 15:12:57 -07:00
Yann Collet 335780c427 fixed too strong alignment assert in ZSTD_initStaticCCtx() 
64-bits fields are only 32-bits aligned on 32-bits CPU
 2017-09-13 16:35:29 -07:00
Yann Collet 8f26dc3f9c blindfix for Visual 
LARGE_INTEGER is not an integer :
https://msdn.microsoft.com/en-us/library/windows/desktop/aa383713(v=vs.85).aspx

Do not take any risk with the structure definition :
use int init = 0; like Mac code
 2017-09-12 21:21:17 -07:00
Yann Collet bc41c7f0eb fixed minor prototype warning 2017-09-12 19:32:26 -07:00
Yann Collet c95c0c9725 modified util::time API 
for easier invocation.
- no longer expose frequency timer :
it's either useless, or stored internally in a static variable (init is only necessary once).
- UTIL_getTime() provides result by function return.
 2017-09-12 18:12:46 -07:00
Yann Collet 3306bcb0e6 fix #820 : GCC v3.x 32-bits doesn't define 64-bits intrinsic 
resulting in undefined symbol error.
Push the requirement to GCC 4 for now.
Another solution, proposed by @NWilson, is to use __LONG_MAX__ instead.
__LONG_MAX__ is a GCC-specific constant, which value is supposed to depend on underlying target hardware (32/64 bits)
Might be better, but seems also more complex, hence more prone to side effects.
Keeping the simple solution for now (just rely on __GNUC__)
 2017-09-11 15:17:31 -07:00