ddidderr/zstd-rs
1
0 Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
3,351 Commits 17 Branches 77 Tags
0fdc71c3dc1e8cbef7416218c46d39ad0631381a
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 0fdc71c3dc added ZSTD_initStaticDCtx() 2017-05-24 17:41:41 -07:00
Yann Collet ba183005d3 merged DStream's inBuff and outBuff into a single buffer 
Saves one malloc().
Also : makes it easier to implement static allocation
 2017-05-24 15:42:24 -07:00
Yann Collet 2e4db3e531 fixed performance regression with ZSTD_decompress() on small files 
memset() was a quick fix to initialization problems,
but initialize too much space (tables, buffers)
which show up in decompression speed of ZSTD_decompress()
since it needs to recreate DCtx at each invocation.

Fixed by only initialization relevant pointers and size fields.
 2017-05-24 13:15:19 -07:00
Yann Collet 11ea2f7fda Merged ZSTD_DCtx and ZSTD_DStream objects 
They are now the same object.
It's recommended to keep both types in source code
as previous versions of library (<v1.3.0)
still need this differentiation.
 2017-05-23 16:19:43 -07:00
Yann Collet b81f19ffce move MEM_readMINMATCH() into zstd_opt.h 
which is its only user.
Use case too narrow to belong to mem.h.
renamed to ZSTD_readMINMATCH()
 2017-05-23 15:41:55 -07:00
Yann Collet c7fe262dc9 added ZSTD_initStaticCCtx() 
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
 2017-05-23 13:20:41 -07:00
Yann Collet 5ac72b417c Buffered are now allocated inside workSpace 2017-05-23 11:18:24 -07:00
Yann Collet 1880337c30 Simplifier compression call graph 
Everything converge towards ZSTD_compressBegin_internal
which delegated to ZSTD_copyCCtx_internal if cdict!=NULL.

This simplifies routing which was previously depending on cdict.
 2017-05-22 18:21:51 -07:00
Yann Collet b0739bcf8f simplified reset by removing full-reset policy 
this was meant to be applied prior to dictionary loading.
But effectively, it seems redundant with later loading stage,
so it can be skipped safely.
 2017-05-22 17:45:15 -07:00
Yann Collet 1ad7c82eb5 Implemented separation between requested and applied parameters 
first version to pass cli tests with -DZSTD_NEWAPI
 2017-05-22 17:06:04 -07:00
Yann Collet 24de7b0346 Implemented ZSTD_CCtx_refCDict() 2017-05-22 13:05:45 -07:00
Yann Collet ee970398b2 Merge branch 'dev' into advancedAPI2 2017-05-22 12:33:56 -07:00
Yann Collet 8b21ec42a9 ZSTD_compress_generic() can handle dictionary compression 2017-05-19 19:46:15 -07:00
Yann Collet 60a557e7fd fixing symbol.c test 
I believe it would be better to rely on fuzzer tests compiled with dll
 2017-05-19 11:27:43 -07:00
Yann Collet 334a288d0d ZSTD_CCtx_setParameter() only works during initialization stage 
and generate a stage_wrong error otherwise.
 2017-05-19 11:04:41 -07:00
Yann Collet 48855fa0d2 fixed declaration-after-statement warning 2017-05-19 10:56:11 -07:00
Yann Collet fa3671eac7 changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX 
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
 2017-05-19 10:51:30 -07:00
Yann Collet 009d604e00 ZSTD_compress_generic() supports multiple successive frames 
also : clarified streaming API implementation
 2017-05-19 10:17:59 -07:00
Yann Collet 6d4fef36de Added ZSTD_compress_generic() 
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
 2017-05-17 18:36:15 -07:00
Yann Collet 23c256e44b removed useless variable from CCtx 
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
 2017-05-16 18:10:11 -07:00
Yann Collet 906a8f269a Merge branch 'dev' into advancedAPI2 2017-05-16 17:33:08 -07:00
Yann Collet e12ae02357 minor coding style update 
reduced some long lines
 2017-05-16 17:32:33 -07:00
Yann Collet 9f95e445ab minor comment clarifications 2017-05-16 17:26:43 -07:00
Yann Collet 0bdb575c31 Merge branch 'dev' into advancedAPI2 2017-05-16 16:32:29 -07:00
Yann Collet 7101434ec9 pedantic : added one error check 
on a function which (today) never fails.
But who knows, maybe tomorrow ...
 2017-05-16 16:28:24 -07:00
Yann Collet bfff8999c5 added prototype ZSTD_versionString() 2017-05-16 16:12:23 -07:00
Yann Collet 4eff8136aa added prototype ZSTD_decompressBegin_usingDDict (#700) 2017-05-16 16:05:27 -07:00
Yann Collet 2d4d31c18a removed gcc compilation flag -Wbad-function-cast 
It makes it more difficult to directly cast the result of a function,
requiring to store the result in an intermediate variable.
It does not necessarily help readability,
and this restriction can be difficult to overcome in some constructions,
like some macros.

also : fixed minor Visual conversion warnings in datagencli.c
 2017-05-16 11:34:38 -07:00
Yann Collet 46ac9ade9b fixed vla warning on linux 2017-05-15 18:15:08 -07:00
Yann Collet 133f0aee54 fixed redundant declarations in legacy v0.5 and v0.7 decoders 
triggered by new flag -Wredundant-decls
 2017-05-15 17:44:04 -07:00
Yann Collet 83d0c764dc added several compilation flags 2017-05-15 17:15:46 -07:00
Yann Collet a5ffe3d370 pushed enum values for strategy by one (ZSTD_fast==1) 
this makes it possible to use `0` to mean:
"do not change strategy"
 2017-05-12 16:29:19 -07:00
Yann Collet add66f816d changed macro LOADCPARAMS by static function ZSTD_cLevelToCParams() 
for improved compiler checks.
Also : ensure most parameters can receive value "0"
to mean "do not change".
 2017-05-12 16:01:15 -07:00
Yann Collet b0edb7fb0e added ZSTD_CCtx_setParameter() 2017-05-12 15:31:53 -07:00
Yann Collet ef738c1b23 better error code when compressing using NULL CDict 
which is not allowed (but detected, and generates an error).
 2017-05-12 13:55:25 -07:00
Yann Collet db8e21d5a0 made ZSTD_compress_generic() definition accessible 
note that the implementation is not done yet.
 2017-05-12 13:46:49 -07:00
Yann Collet 33eb7ac6b6 updated Advanced API proposal 
only declarations in zstd.h
 2017-05-12 12:36:11 -07:00
Yann Collet 30ab64e21d added test for ZSTD_estimateCStreamSize() 2017-05-10 11:30:19 -07:00
Yann Collet cef02d9317 changed variable name, for clarity 
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
 2017-05-10 11:14:08 -07:00
Yann Collet 669346fe8b fixed ZSTD_estimateCStreamSize() 
https://github.com/facebook/zstd/pull/689#discussion_r115637721
 2017-05-10 11:08:00 -07:00
Yann Collet 6fb2f24132 shortened ZSTD_createCStream_Advanced() 
https://github.com/facebook/zstd/pull/689#discussion_r115637613
 2017-05-10 11:06:06 -07:00
Yann Collet 461a9cc7c6 fixed symbols test 2017-05-09 16:53:09 -07:00
Yann Collet f16f4497ca added ZSTD_estimateDStreamSize() 2017-05-09 16:18:17 -07:00
Yann Collet 542c9dfcf8 changed name frameParams into frameHeader 
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()

The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.

ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
 2017-05-09 15:46:07 -07:00
Yann Collet 5a36c069e7 regroup memory usage function declarations 
in a single paragraph in zstd.h, for clarity
 2017-05-09 15:11:30 -07:00
Yann Collet fa8dadb294 separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize() 
for clarity
 2017-05-08 18:24:16 -07:00
Yann Collet 51652522a2 bumped version number 2017-05-08 17:52:46 -07:00
Yann Collet a1d6704d7f added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize() 
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
 2017-05-08 17:51:49 -07:00
Yann Collet 7855366598 Updated ZSTD_freeCCtx() 
which can also contain streaming buffers now.
Redirected ZSTD_freeCStream() towards it.
 2017-05-08 17:15:00 -07:00
Yann Collet fc5145955a updated ZSTD_estimateCCtxSize() 
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).

Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
 2017-05-08 17:07:59 -07:00
Yann Collet 791d744279 Updated ZSTD_sizeof_CCtx() 
can now contain buffers if object used as CStream.
ZSTD_sizeof_CStream() is now just a thin wrapper of ZSTD_sizeof_CCtx().
 2017-05-08 16:17:30 -07:00
Yann Collet 0be6fd3429 merged CCtx and CStream as a single same object 
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
 2017-05-08 16:08:01 -07:00
Yann Collet 01a1abfdb5 cli : -d and -t do not stop after a failed decompression 
The problematic srcfile will be named on console/log,
but decompression/test will continue onto next file in the list.
 2017-05-05 19:15:24 -07:00
Yann Collet 11bff3fbd3 added dev branch CircleCI badge 2017-05-05 15:55:03 -07:00
Yann Collet 3d55e1fbee added dev branch Appveyor badge 2017-05-05 15:48:42 -07:00
Yann Collet 25b2833396 keep dev branch status only 
master branch status is removed, due to misattribution during cron jobs
 2017-05-05 15:41:23 -07:00
Yann Collet 36153af5ec creates a binary archive without the programs directory 
also improves compression ratio to -mx9
 2017-05-04 17:40:40 -07:00
Yann Collet 9203dab529 Appveyor build artefact creates zipped cli binary 2017-05-04 17:30:37 -07:00
Yann Collet c21a9c3596 removed zstdmt generation from Appveyor artefact build 2017-05-04 17:26:58 -07:00
Yann Collet a00e9599f1 removed -g from DEBUGFLAGS 
It inflates binary sizes, which is negative for the Windows build.
It also makes it impossible to check if 2 different source codes
get nonetheless compiled to the same binary,
since checksum will be different, due to integrated source code.
 2017-05-04 17:24:29 -07:00
Yann Collet b668badf00 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-05-03 14:33:39 -07:00
Yann Collet c08e56861e updated dict graphs to 2D mode 2017-05-03 14:33:28 -07:00
Yann Collet 710497d8ea updated programs/README.md, to introduce compilation variables 
make it possible to enable/disable features individually
 2017-05-02 17:18:24 -07:00
Yann Collet f47284fec0 reorganized Makefile for multiple targets 2017-05-02 16:55:57 -07:00
Yann Collet 3791d2105d added xzstd4 target 
support for all formats, xz/lzma/lz4 included
 2017-05-02 16:38:37 -07:00
Yann Collet 2e63a877f3 fixed xzstd 
--format=xz was missing a break, making the execution continue into lz4 error message
 2017-05-02 15:40:42 -07:00
Yann Collet 705de8eba4 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-05-02 14:39:42 -07:00
Yann Collet 01a71739b0 updated DSpeed chart to remove 3D effect (#589) 2017-05-02 14:39:03 -07:00
Yann Collet 7aada3ca44 updated NEWS for v1.2.0 2017-05-02 12:14:12 -07:00
Yann Collet 606c04c228 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-05-02 12:13:52 -07:00
Yann Collet b184589c4c minor code refactoring for clarity 2017-05-01 11:35:47 -07:00
Yann Collet 33c38b0925 fixed const in prototype, that Visual doesn't accept 2017-05-01 11:12:30 -07:00
Yann Collet f39a6731ec sync bitstream.h from fse library 2017-05-01 09:56:03 -07:00
Yann Collet 667bd9723d Merge branch 'dev' of github.com:facebook/zstd into dev 2017-04-28 17:00:46 -07:00
Yann Collet 202082f285 sync bitstream from FSE project 
add assert into unsafe *_fast() variants
 2017-04-28 17:00:31 -07:00
Yann Collet 89f50deec7 minor code refactoring 
clearer tables
 2017-04-28 16:52:36 -07:00
Yann Collet 68a7d3d49a added HUF_PUBLIC_API macro to huf.h 
to make it possible to control symbol visibility.
Also : better separation and comments between "public" and "static" sections
 2017-04-28 12:46:48 -07:00
Yann Collet 7bd68e194d minor release note update 2017-04-28 08:54:13 -07:00
Yann Collet 29297c6751 Changed default level 18 (large input) 
Previous -18 : 4.7 MB/s, R:3.833
New -18 : 5.1 MB/s. R:3.825

It's a better fit within -17 (6.8 MB/s) and -19 (4.0 MB/s)
The new level 18 also uses significantly less memory.
And, it makes a good transition between level 17 (mml5)
and level 19 (mml3).
Up to now, there was no level with mml4.

(note : minmatch setting can have a large impact on some (specific) datasets)
 2017-04-27 17:44:01 -07:00
Yann Collet b636133932 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-04-27 17:29:52 -07:00
Yann Collet 36e79a1a1f tests : -g is part of DEBUGFLAGS 2017-04-27 17:29:40 -07:00
Yann Collet c6915429f2 shortened Appveyor release tests 
fuzzer supports time suffix `s` for "seconds"
 2017-04-27 16:24:53 -07:00
Yann Collet a92cbb7004 Added a secondary test, checking dictID presence after setting noDictIdFLag=1 2017-04-27 15:08:56 -07:00
Yann Collet 7d283cdfa2 added test for ZSTD_initCStream_usingCDict_advanced() 2017-04-27 14:48:34 -07:00
Yann Collet 2f73427d35 added test for ZSTD_compress_usingCDict_advanced() 2017-04-27 14:39:39 -07:00
Yann Collet d3694e6c70 removed C4204 2017-04-27 14:29:35 -07:00
Yann Collet 7321345fd2 fixed another VS2008 init error 2017-04-27 14:19:34 -07:00
Yann Collet 1c3ab0c77f fixed init error on Visual 2008 2017-04-27 12:57:11 -07:00
Yann Collet 8b669535f8 bumped version number to v1.2.0 2017-04-27 12:50:20 -07:00
Yann Collet 5a804456bc updated NEWS 2017-04-27 12:49:22 -07:00
Yann Collet 77bf59ef50 added ZSTD_initCStream_usingCDict_advanced() 2017-04-27 11:43:04 -07:00
Yann Collet f4bd857d81 created ZSTD_compress_usingCDict_advanced() 2017-04-27 11:31:55 -07:00
Yann Collet 69a54d138a fixed compilation warning : declaration-after-statement 2017-04-27 01:11:26 -07:00
Yann Collet 31533bacce Changed ZSTD_createCDict_advanced() 
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.

Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
 2017-04-27 00:29:04 -07:00
Yann Collet 768df129d2 changed ZSTD_compressBegin_usingCDict() 
No longer takes `pledgedSrcSize` as argument
this is in line with similar functions ZSTD_compress_usingCDict()
and ZSTD_initCStream_usingCDict().
 2017-04-26 15:42:10 -07:00
Yann Collet e42afbc6fa Comply with suggested comments by @terrelln 
created FSE_CTABLE_SIZE() and FSE_DTABLE_SIZE()
 2017-04-26 11:39:35 -07:00
Yann Collet 230d7acc7d cli : add support for --threads=# command 
updated documentation
add relevant test case
 2017-04-21 11:38:13 -07:00
Yann Collet d0b1846cf4 ignore more cmake build artefacts 2017-04-21 10:59:36 -07:00
Yann Collet 7271203bdb transferred entropy scratch space from CCtx into workSpace 
Saved 6 KB
 2017-04-20 23:21:19 -07:00
Yann Collet a408645f50 made some room for entropy scratch space 2017-04-20 23:09:39 -07:00