ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,560 Commits 17 Branches 77 Tags
31af8290d155091f99c42da7b3787964ca4b37d7
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 31af8290d1 ZSTD_estimateCCtx_advanced() 
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.

ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
 2017-06-26 15:52:39 -07:00
Yann Collet 6122af9cf5 --no-big-tests for msan 
msan tests require too much RAM for 2 GB VM
 2017-06-23 00:37:15 -07:00
Yann Collet cca1ed14db Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-06-23 00:13:03 -07:00
Yann Collet 32f546c763 zstreamtest : disabled multi-threading tests with --newapi --no-big-tests 
--no-big-tests is typically used in combination with qemu-user-static
qemu-user-static allocated 4 GB of RAM upfront.
On 2 GB VM, this can degenerate into a crash.
It's not a problem as long as memory is not used.
But with multi-threading enabled, memory fragmentation kicks in,
so the amoung of RAM effectively touched increases,
and can pass beyond the 2 GB limit of the VM.

In single-threaded mode, there is no such issue :
memory requirement is smaller, and remains well-located,
so very little fragmentation is expected.

This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests`
work fine on Travis CI.
 2017-06-23 00:09:02 -07:00
Yann Collet 49f8459d11 fixed minor cast warning 2017-06-21 18:43:39 -07:00
Yann Collet 4d3bdcf130 reduced CLevelMax for --no-big-tests 2017-06-21 18:20:02 -07:00
Yann Collet bfc2f00080 --no-big-tests for zstreamtest 
Apply --no-big-tests for tsan tests
 2017-06-21 17:57:14 -07:00
Yann Collet ecb0f46866 add controls over streaming buffers 2017-06-21 17:25:01 -07:00
Yann Collet 224e7a1053 added --list command contrib by @paulcruz74 2017-06-21 15:58:36 -07:00
Yann Collet dce789281b fixed : decompression of skippable frames in streaming mode 2017-06-21 15:53:42 -07:00
Yann Collet 991f9dfcde switched fileio.c to ZSTD_DEBUG trigger macro 2017-06-21 15:16:13 -07:00
Yann Collet 204b6b7ef6 fixed streaming buffered allocation with CDict compression 2017-06-21 15:13:00 -07:00
Yann Collet 1e4129b27b fixed dangling pointer risk, detected by @terrelln 2017-06-21 13:26:10 -07:00
Yann Collet 83095970e6 free cdictLocal faster, suggested by @terrelln 2017-06-21 12:26:40 -07:00
Yann Collet c16748b137 added test cases for new command --list 2017-06-21 12:09:53 -07:00
Yann Collet 7bd1a2900e added ZSTD_dictMode_e to control dictionary loading mode 2017-06-21 11:50:33 -07:00
Yann Collet e51d51bdf7 fixed memcpy() overlap 2017-06-20 17:44:55 -07:00
Yann Collet 466f92eaa6 removed one useless streaming compression stage, detected by @terrelln 2017-06-20 16:25:29 -07:00
Yann Collet c3bce24ef4 fixed potential dangling pointer, detected by @terrelln 2017-06-20 16:09:11 -07:00
Yann Collet 78b8234554 fixed comments, following suggestion by @terrelln 2017-06-20 14:26:48 -07:00
Yann Collet b44ab82f7a ensure new ZSTD_strategy starts at value 1 2017-06-20 14:11:49 -07:00
Yann Collet ff8f83bd47 fixed fuzzer test 2017-06-20 12:17:32 -07:00
Yann Collet c08e649e95 first implementation of bench.c with new API ZSTD_compress_generic() 
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.

Also : fixed a long standing bug in ZSTDMT streaming API.
 2017-06-19 18:25:35 -07:00
Yann Collet 008d44ad66 fix attempt : fullbench VS2008 project 2017-06-19 17:45:43 -07:00
Yann Collet 695a0a3449 fixed IA64 compilation error, by @mcmilk 2017-06-19 15:27:30 -07:00
Yann Collet fe234bf48b fix attempts : fullbench for VS2008 2017-06-19 15:23:19 -07:00
Yann Collet bf99150be3 update new api presentation in zstd.h and manual 2017-06-19 12:56:25 -07:00
Yann Collet c7dcf0f379 updated manual 2017-06-19 12:03:25 -07:00
Yann Collet 688952062b minor declaration statement warning fix 2017-06-19 11:53:42 -07:00
Yann Collet d7a3bffba9 new api : setting compression parameters is refused if a dictionary is already loaded 2017-06-19 11:53:01 -07:00
Yann Collet ed1d039127 newapi fuzzer tests : random flush orders in main loop 2017-06-19 11:07:33 -07:00
Yann Collet bd05b19713 removed macro CLAMP from decodecorpus 
never used,
and would duplicate with CLAMP from zstd_compress.c
 2017-06-16 18:16:27 -07:00
Yann Collet 01743a36e7 fuzzer tests for new API 2017-06-16 17:56:41 -07:00
Yann Collet 381e66cfbd added ZSTD_clampCParams() 
now ZSTD_adjustCParams() is always successful,
it always produces a valid CParams
 2017-06-16 17:34:54 -07:00
Yann Collet aee916e37c fixed +/-1 error for pledgedSrcSizePlusOne 2017-06-16 17:02:35 -07:00
Yann Collet d3de3d51a3 fix attempt 2 : Visual sign conversion warning 2017-06-16 16:51:33 -07:00
Yann Collet 944be54774 fixed attempt : minor Visual sign conversion warning 2017-06-16 14:05:01 -07:00
Yann Collet b26728c9c8 added ZSTD_startNewCompression() 2017-06-16 14:00:46 -07:00
Yann Collet a0ba849fe6 changed frameContentSize field to pledgedSrcSizePlusOne 
pledgedSrcSize is proper : it's a promise, not yet fulfilled.
It will be controlled at the end.

PlusOne is meant to have 0 (default) == unknown
 2017-06-16 13:29:17 -07:00
Yann Collet 2cf7755da7 fix : pledgedSrcSize correctly reset to unknown in "continue" mode 2017-06-16 12:34:41 -07:00
Yann Collet 9e73f2f320 fix : correctly reset pledgedSrcSize to unknown status 
when starting a new compression with an existing context
 2017-06-16 12:24:01 -07:00
Yann Collet 33873f0e74 fixed : new advanced AIP : setting nbThreads to the same value > 1 2017-06-16 12:04:21 -07:00
Yann Collet 559ee82e90 fixed : calling ZSTD_compress_generic() to end-flush a stream in multiple steps 2017-06-16 11:58:21 -07:00
Yann Collet bd18c885a3 added ZSTD_CCtx_reset 2017-06-16 10:17:50 -07:00
Yann Collet cc9f9b7f4c protection : ZSTD_CONTENTSIZE_UNKNOWN automatically disables contentSizeFlag 2017-06-15 18:17:34 -07:00
Yann Collet 05ae4b2190 added protection : MT incompatible with Static allocation 2017-06-15 18:03:34 -07:00
Yann Collet bb0aaf9579 minor man update on -B# option in benchmark mode 2017-06-12 05:19:15 -07:00
Yann Collet f129fd3970 disabled MT code path when ZSTD_MULTITHREAD is not defined 2017-06-11 18:46:09 -07:00
Yann Collet 9e6a2eaab6 added MT support to NEWAPI 2017-06-11 18:39:46 -07:00
Yann Collet 23aace9778 added control stage to MT mode 2017-06-11 18:32:36 -07:00
Yann Collet f35e2de61c linked newAPI to ZSTDMT 2017-06-05 18:32:48 -07:00
Yann Collet 8c910d2097 updated ZSTDMT streaming API 
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
 2017-06-03 01:15:02 -07:00
Yann Collet 58e8d793e1 made debug definitions common within zstd_internal.h 2017-06-02 18:20:48 -07:00
Yann Collet 86bd83ef12 completed NEWS for v1.3.0 2017-06-02 17:43:55 -07:00
Yann Collet 257a7226d8 updated NEWS for v1.3.0 2017-06-02 17:35:11 -07:00
Yann Collet 8ddf4c22d5 fixed missing initialization 2017-06-02 17:16:49 -07:00
Yann Collet 33a7e679e5 significant zlib wrapper code refactoring 
code indentation
variable scope and names
constify

Only coding style changes.
The logic should remain the same.
 2017-06-02 17:10:49 -07:00
Yann Collet 4effccbf56 zlib_wrapper's uncompress() uses ZSTD_isFrame() for routing 
more generic and safer than using own routing for magic number comparison
 2017-06-02 14:27:11 -07:00
Yann Collet dcb7535352 ensure zlibwrapper uses ZSTD_malloc() and ZSTD_free() 
which is compatible with { NULL, NULL, NULL }
 2017-06-02 14:01:21 -07:00
Yann Collet b877e834b1 minor indent 2017-06-02 13:47:11 -07:00
Yann Collet 6056e4c3eb added POOL_sizeof() for single-thread 2017-06-02 11:36:47 -07:00
Yann Collet c35e535002 added support for multithreading parameters 2017-06-01 18:44:06 -07:00
Yann Collet c4a5a21c5c created ZSTDMT_sizeof_CCtx() and POOL_sizeof() 
required by ZSTD_sizeofCCtx() while adding a ZSTDMT_CCtx*
 2017-06-01 17:56:14 -07:00
Yann Collet cd2892fd1e protected impossible switch(){default:} with assert(0) 
can be converted into assume(0) in some future
 2017-06-01 09:44:54 -07:00
Yann Collet 06589fe516 Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-05-31 10:03:20 -07:00
Yann Collet 18ab5affa5 fixed visual warning 2017-05-31 09:59:22 -07:00
Yann Collet 01b1549f83 finally converted ZSTD_compressStream_generic() to use {in,ou}Buffer 
replacing the older read/write variables from ZBUFF_* era.
Mostly to help code readability.

Fixed relevant callers.
 2017-05-30 18:10:26 -07:00
Yann Collet c4f46b94ce ZSTD_createCCtx_advanced() now uses ZSTD_calloc() 
initially uses calloc() instead of memset().

Performance improvement is unlikely measurable,
since ZSTD_CCtx is now very small,
with all tables transferred into workSpace.
 2017-05-30 17:45:37 -07:00
Yann Collet deee6e523f expose ZSTD_compress_generic_simpleArgs() 
which is a binding towards ZSTD_compress_generic()
using only integral types for arguments.
 2017-05-30 17:42:00 -07:00
Yann Collet ae728a43b8 removed defaultCustomMem 
now ZSTD_customCMem is promoted as new default.

Advantages : ZSTD_customCMem = { NULL, NULL, NULL},
so it's natural default after a memset.

ZSTD_customCMem is public constant
(defaultCustomMem was private only).

Also : makes it possible to introduce ZSTD_calloc(),
which can now default to stdlib's calloc()
when it detects system default.

Fixed zlibwrapper which depended on defaultCustomMem.
 2017-05-30 17:11:39 -07:00
Yann Collet 5bcef1ada2 removed mtctx->cstream 
use the first cctx in pool when ZSTDMT is used in single-thread mode
now that cctx and cstream are the same object.
 2017-05-30 16:37:19 -07:00
Yann Collet beb62b15a8 Merge branch 'dev' into advancedAPI2 
Fixed conflic in zstd_decompress.c
 2017-05-30 16:18:57 -07:00
Yann Collet 44e45e8423 added ZSTDMT_createCCtx_advanced() 
make it possible to use custom allocators
 2017-05-30 16:12:06 -07:00
Yann Collet f45ca527a1 Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-05-30 10:02:03 -07:00
Yann Collet 7028cbd7fd fixed a few code comments : ZSTD_getFrameParams => ZSTD_getFrameHeader 2017-05-25 18:29:08 -07:00
Yann Collet cdf7e82222 Added ZSTD_initStaticCDict() 2017-05-25 18:05:49 -07:00
Yann Collet 57827f906f added ZSTD_initStaticDDict() 2017-05-25 15:44:06 -07:00
Yann Collet 25989e361c updated ZSTD_estimate?DictSize() to pass parameter byReference 
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
 2017-05-25 15:07:37 -07:00
Yann Collet 0fdc71c3dc added ZSTD_initStaticDCtx() 2017-05-24 17:41:41 -07:00
Yann Collet ba183005d3 merged DStream's inBuff and outBuff into a single buffer 
Saves one malloc().
Also : makes it easier to implement static allocation
 2017-05-24 15:42:24 -07:00
Yann Collet 2e4db3e531 fixed performance regression with ZSTD_decompress() on small files 
memset() was a quick fix to initialization problems,
but initialize too much space (tables, buffers)
which show up in decompression speed of ZSTD_decompress()
since it needs to recreate DCtx at each invocation.

Fixed by only initialization relevant pointers and size fields.
 2017-05-24 13:15:19 -07:00
Yann Collet 11ea2f7fda Merged ZSTD_DCtx and ZSTD_DStream objects 
They are now the same object.
It's recommended to keep both types in source code
as previous versions of library (<v1.3.0)
still need this differentiation.
 2017-05-23 16:19:43 -07:00
Yann Collet b81f19ffce move MEM_readMINMATCH() into zstd_opt.h 
which is its only user.
Use case too narrow to belong to mem.h.
renamed to ZSTD_readMINMATCH()
 2017-05-23 15:41:55 -07:00
Yann Collet c7fe262dc9 added ZSTD_initStaticCCtx() 
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
 2017-05-23 13:20:41 -07:00
Yann Collet 5ac72b417c Buffered are now allocated inside workSpace 2017-05-23 11:18:24 -07:00
Yann Collet 1880337c30 Simplifier compression call graph 
Everything converge towards ZSTD_compressBegin_internal
which delegated to ZSTD_copyCCtx_internal if cdict!=NULL.

This simplifies routing which was previously depending on cdict.
 2017-05-22 18:21:51 -07:00
Yann Collet b0739bcf8f simplified reset by removing full-reset policy 
this was meant to be applied prior to dictionary loading.
But effectively, it seems redundant with later loading stage,
so it can be skipped safely.
 2017-05-22 17:45:15 -07:00
Yann Collet 1ad7c82eb5 Implemented separation between requested and applied parameters 
first version to pass cli tests with -DZSTD_NEWAPI
 2017-05-22 17:06:04 -07:00
Yann Collet 24de7b0346 Implemented ZSTD_CCtx_refCDict() 2017-05-22 13:05:45 -07:00
Yann Collet ee970398b2 Merge branch 'dev' into advancedAPI2 2017-05-22 12:33:56 -07:00
Yann Collet 8b21ec42a9 ZSTD_compress_generic() can handle dictionary compression 2017-05-19 19:46:15 -07:00
Yann Collet 60a557e7fd fixing symbol.c test 
I believe it would be better to rely on fuzzer tests compiled with dll
 2017-05-19 11:27:43 -07:00
Yann Collet 334a288d0d ZSTD_CCtx_setParameter() only works during initialization stage 
and generate a stage_wrong error otherwise.
 2017-05-19 11:04:41 -07:00
Yann Collet 48855fa0d2 fixed declaration-after-statement warning 2017-05-19 10:56:11 -07:00
Yann Collet fa3671eac7 changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX 
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
 2017-05-19 10:51:30 -07:00
Yann Collet 009d604e00 ZSTD_compress_generic() supports multiple successive frames 
also : clarified streaming API implementation
 2017-05-19 10:17:59 -07:00
Yann Collet 6d4fef36de Added ZSTD_compress_generic() 
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
 2017-05-17 18:36:15 -07:00
Yann Collet 23c256e44b removed useless variable from CCtx 
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
 2017-05-16 18:10:11 -07:00
Yann Collet 906a8f269a Merge branch 'dev' into advancedAPI2 2017-05-16 17:33:08 -07:00
Yann Collet e12ae02357 minor coding style update 
reduced some long lines
 2017-05-16 17:32:33 -07:00