ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,100 Commits 17 Branches 77 Tags
aecf3b479c45affa9fd8ead068e9160253a8ec5c
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 4912fc2acc updated NEWS for v1.3.1 2017-08-20 11:45:58 -07:00
Yann Collet e8d35cc5e9 minor formulation change, recommended by @ulikunitz 2017-08-20 10:39:20 -07:00
Yann Collet d6394cc4c3 fixed test-zstd-nolegacy 2017-08-20 10:15:44 -07:00
Yann Collet 7db552676e reduced pool queue to 0 to save memory 
fixed : pool performance when jobs are fires fast and queueSize==0
 2017-08-19 15:07:54 -07:00
Yann Collet 1c108c811e cli : Display supported formats on -vV command 
Requested and inspired by patch from @ib (#771)
 2017-08-19 13:33:50 -07:00
Yann Collet d0d06e421f added alternative representation for huffman bistream 2017-08-19 12:26:09 -07:00
Yann Collet 8b12812147 fix #803 : wrong example in huffman bitstream section, reported by @ulikunitz 2017-08-19 12:17:57 -07:00
Yann Collet 2ecd34ee5e fixed unused variables warnings 2017-08-19 01:23:49 -07:00
Yann Collet 23706fb743 updated doc on compilation variables 2017-08-19 01:14:36 -07:00
Yann Collet 9203003d5f fixed zstd-nolegacy 
and added it to allVariants for CI testings
 2017-08-19 01:01:53 -07:00
Yann Collet 4b387729b6 fixed zstd-small 
and added it to shortest for CI tests
 2017-08-19 00:48:29 -07:00
Yann Collet 7729ab83bb Merge branch 'dev' into variants 2017-08-19 00:37:06 -07:00
Yann Collet 32e943b3ef Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-19 00:36:37 -07:00
Yann Collet 288e263ec4 added zstd-decompress to the list of variants tested 2017-08-18 18:39:39 -07:00
Yann Collet 166645e7b3 fixed zstd-compress 
file-information is dependent on decompression functions.
it should only be enabled when ZSTD_NODECOMPRESS is not set.

also : added zstd-compress compilation test into `make shortest`
 2017-08-18 18:30:41 -07:00
Yann Collet f207b39f55 blindfix for Windows conversion warning 
long type is 32-bits on Windows 64,
while it's 64-bits on Unix.
64-to-32 shortening conversion for long is a specific Windows issue.
 2017-08-18 17:06:12 -07:00
Yann Collet 32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Yann Collet 4f73b3b55d added GPLv2 license 
and removed PATENTS clause
 2017-08-18 16:32:08 -07:00
Yann Collet 88d2f72df9 fixed --list command in presence of special blocks 
block type RLE is special, compressed size is always 1.
block type 3 is "reserved", aka not supported.
 2017-08-18 16:18:20 -07:00
Yann Collet c523c93b26 improved and fixed --list command, original patches by @ib (#772) 
accepts all skippable frame identifiers.
display in MB or KB, depending on frame size.
fixed combination of skippable and zstd frames.
 2017-08-18 15:57:53 -07:00
Yann Collet 3f54d788e9 removed --list from cli help (-h), reported by Agostino Sarubbo (@asarubbo) (#800) 
redundant with shorter -l.
both -l and --list do the same thing, and are documented in man page.
 2017-08-18 15:15:31 -07:00
Yann Collet bbddff480d Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-18 15:12:30 -07:00
Yann Collet 0ab9d55e7a Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-11 10:17:49 -07:00
Yann Collet e94fd71bbd Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-27 13:49:46 -07:00
Yann Collet b72808a49b changed zstream test time 
Used to be 3x5mn, resulting in time out issues (make test)
Now set to 3x2mn
 2017-07-21 11:45:42 -07:00
Yann Collet ceda7a9a58 minor Makefile refactor 2017-07-21 11:44:39 -07:00
Yann Collet 38ba7002f2 fixed minor warning on unused variable in shell function 2017-07-20 18:39:04 -07:00
Yann Collet a90b16e150 Visual blind fix 2 2017-07-20 15:57:55 -07:00
Yann Collet 4542d7118c Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-20 15:12:21 -07:00
Yann Collet 5e6c5203f3 fixed fuzzer test for non OS-X platforms 2017-07-20 15:11:56 -07:00
Yann Collet b4d460f32c pool.c : blindfix for Visual warnings 2017-07-20 01:13:14 -07:00
Yann Collet 1ca1288689 added --memtest=# command to fuzzer 
to jump directly to relevant test section
 2017-07-19 16:01:16 -07:00
Yann Collet 3974d2b38a blind fix for Windows Multithreading module 
adds a fake 0 return value for mutex/cond init
 2017-07-19 13:33:21 -07:00
Yann Collet b71363b967 check pthread_*_init() success condition 2017-07-19 01:05:40 -07:00
Yann Collet 3b0cff3c33 fixed clang's -Wdocumentation 2017-07-13 18:58:30 -07:00
Yann Collet 2bd6440be0 pinned down error code enum values 
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.

Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
 2017-07-13 17:12:16 -07:00
Yann Collet 3502426fd4 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-13 15:49:19 -07:00
Yann Collet 6733c0777c updated NEWS regarding #760 2017-07-13 15:34:44 -07:00
Yann Collet 3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) 
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
 2017-07-13 10:17:23 -07:00
Yann Collet 132e6efd76 switched ZSTDMT_compress_advanced() last argument to overlapLog 
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
 2017-07-13 02:22:58 -07:00
Yann Collet 4e77f7761d clarified comment on ZSTD_p_contentSizeFlag 2017-07-13 02:09:07 -07:00
Yann Collet 33cecaaaca Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-12 14:38:36 -07:00
Yann Collet 8ef666c325 slightly increased buffer pool, to cover normal "full load" scenarios 
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
 2017-07-12 14:23:34 -07:00
Yann Collet 052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum 
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
 2017-07-11 17:18:26 -07:00
Yann Collet 2a62f48bf4 release input buffers from inside worker thread 
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
 2017-07-11 15:56:40 -07:00
Yann Collet 57236184af buffer pool : all buffers have same size 
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
 2017-07-11 15:17:25 -07:00
Yann Collet 34b2b95631 zstdmt : intermediate outBuffer allocated from within worker 
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
 2017-07-11 14:59:10 -07:00
Yann Collet 16261e6951 buffer pool can be invoked from multiple threads 2017-07-11 14:14:07 -07:00
Yann Collet ef0ff7fe7f zstdmt: removed margin for improved memory usage 2017-07-11 08:54:29 -07:00
Yann Collet 4616fad18b improved ZSTDMT_compress() memory usage 
does not need the input buffer for streaming operations

also : reduced a few tests time length
 2017-07-10 17:16:41 -07:00
Yann Collet 670b1fc547 optimized memory usage for ZSTDMT_compress() 
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.

In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).

To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
 2017-07-10 16:30:55 -07:00
Yann Collet 3510efb02d fix : custom allocator correctly propagated to child contexts 2017-07-10 14:21:40 -07:00
Yann Collet ee3423d709 extended fuzzer MT memory tests 2017-07-10 14:09:16 -07:00
Yann Collet 88da8f1816 fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter() 
also : compile fuzzer with MT enabled
 2017-07-10 14:02:33 -07:00
Yann Collet f9524cf366 added --memtest to fuzzer 2017-07-10 13:48:41 -07:00
Yann Collet e32fb0c1fe added ZSTD_sizeof_CCtx() test 2017-07-10 12:29:57 -07:00
Yann Collet 40156a4967 bumped version nb to v1.3.1 2017-07-08 04:55:09 -07:00
Yann Collet 0f4fc6c20a fixed several conversion warnings 2017-07-07 17:13:12 -07:00
Yann Collet ed0243a63c removed zbufftest from list of all tests 2017-07-07 16:16:14 -07:00
Yann Collet 9bde061a0b fixed minor Visual compilation limitation 2017-07-07 16:14:17 -07:00
Yann Collet 593d517ebf fixed minor cast warning 2017-07-07 16:09:47 -07:00
Yann Collet 842644e42e target gpptest uses CXX environment variable 2017-07-07 15:55:41 -07:00
Yann Collet d4dcace34b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-07 15:53:20 -07:00
Yann Collet ead4dd48f6 new field frameHeader.headerSize 2017-07-07 15:51:24 -07:00
Yann Collet 46396523c0 ZSTD_getFrameHeader : control of windowSize limits is delegated to caller 
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
 2017-07-07 15:32:12 -07:00
Yann Collet 990449b89d new field : ZSTD_frameHeader.frameType 
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
 2017-07-07 15:21:35 -07:00
Yann Collet e622330a3b extended frameHeader.windowSize to unsigned long long 2017-07-07 14:19:01 -07:00
Yann Collet 7758ed8458 fixed fullbench, part 2 2017-07-06 02:48:00 -07:00
Yann Collet 9b2c1acfc0 fixed fullbench 2017-07-06 02:22:57 -07:00
Yann Collet f04deff4fc fixed #718, reported by @GregSlazinski, solution suggested by @mcmilk 2017-07-06 01:42:46 -07:00
Yann Collet d75c0e71c4 minor code refactoring 2017-07-05 18:10:07 -07:00
Yann Collet 3345a91964 cli : use new advanced API by default 2017-07-05 17:34:15 -07:00
Yann Collet 49af41820d clarified status of zstdmt_compress.h API 2017-07-05 17:21:13 -07:00
Yann Collet 27e883371d fixed wrong assert() condition 
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
 2017-07-04 19:33:16 -07:00
Yann Collet 2cb9774f5e more precise estimation of amount to flush at end of stream (single thread mode) 
also : can use DEBUGLEVEL variable in /tests
 2017-07-04 12:39:26 -07:00
Yann Collet 6383372dec fixed : 0-copy in NULL is UB 2017-07-04 10:36:41 -07:00
Yann Collet 2de2396a36 refactor ZSTDMT_compress() 2017-07-03 16:23:36 -07:00
Yann Collet 2084b041f4 fixed comments 2017-07-03 15:52:19 -07:00
Yann Collet 5a77361595 fixed wrong function name in comment 2017-07-03 15:21:24 -07:00
Yann Collet b8280fe929 minor code refactoring 2017-07-03 15:14:55 -07:00
Yann Collet 368b974387 fixed gz error reporting 2017-07-03 13:47:46 -07:00
Yann Collet c9f21c8a71 gzip : only display inflateEnd error when there is no error before 2017-07-03 13:45:09 -07:00
Yann Collet 6c35112d08 gzip decoding does no longer exit() on invalid input 2017-07-03 13:24:50 -07:00
Yann Collet e97ff3b54a decoding malformed lz4 frames does no longer exit() 2017-07-03 11:27:29 -07:00
Yann Collet c34185df13 fileio : decoding malformed lzma frame does no longer exit() 
makes it possible to continue decoding file list
 2017-07-03 10:27:16 -07:00
Yann Collet 2485f88bf8 fixed legacy version init bug 2017-07-01 09:09:34 -07:00
Yann Collet 58bd0e70fc fixed : dictionary compression with new advanced API in Multi-threading mode 2017-06-30 16:01:02 -07:00
Yann Collet d8b33a598d Optimized ZSTDMT single-pass mode speed on large sources 
by ensuring job sizes remain "not too large"
 2017-06-30 15:44:57 -07:00
Yann Collet d5c046c609 implemented shortcut for zstd_compress_generic() in MT mode 
added ZSTDMT_compress_advanced() API
 2017-06-30 14:51:01 -07:00
Yann Collet e7e5a8cef7 made fullbench compatible with multi-threading 
fullbench 61/62 measure speed of ZSTD_compress_generic with 2 threads
 2017-06-29 18:56:24 -07:00
Yann Collet afb0aca739 zstreamtest : big tests are only enabled in 64-bits mode 
to avoid requesting too much memory in 32-bits mode during MT tests
 2017-06-29 18:19:09 -07:00
Yann Collet a3d9926c40 compression optimization opportunity 
switch to single-pass mode directly into output buffer
when outputSize >= ZSTD_compressBound(inputSize).
Speed gains observed with fullbench (~+15% on level 1)
 2017-06-29 14:44:49 -07:00
Yann Collet 2e84bec9ac updated fullbench to also measure ZSTD_compress_generic() 
will make it possible to visualize
optimization opportunity for ZSTD_e_end
 2017-06-29 13:03:10 -07:00
Yann Collet 97f2bf66da minor : fix typo 2017-06-29 11:31:40 -07:00
Yann Collet 7303ed5d74 minor : specify compression level in tags 2017-06-29 11:30:31 -07:00
Yann Collet acbef3decd ZSTD_getFrameContentSize() is promoted to "stable" status 2017-06-29 05:19:51 -07:00
Yann Collet 037466245f refactor ZSTD_check_compressionLevel_monotonicIncrease_memoryBudget() 
use less macro statements
the initial version was meant to work with STATIC_ASSERT
but since it doesn't work and needs assert()
it's possible to rewrite it using normally compiled code
which is better for compiler.

Downside : the error message is less precise.
There is a DEBUGLOG(3,) to compensate.
 2017-06-28 20:24:08 -07:00
Yann Collet 2bf428df45 Merge branch 'advancedAPI2' into refPrefix 2017-06-28 16:35:49 -07:00
Yann Collet 62f7efc04a added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job 2017-06-28 16:25:13 -07:00
Yann Collet 1ca76039af fixed -Wdeclaration-after-statement 2017-06-28 15:40:21 -07:00