ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,061 Commits 17 Branches 77 Tags
4f73b3b55d83b082aad180816a5d336eff33a243
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 4f73b3b55d added GPLv2 license 
and removed PATENTS clause
 2017-08-18 16:32:08 -07:00
Yann Collet 88d2f72df9 fixed --list command in presence of special blocks 
block type RLE is special, compressed size is always 1.
block type 3 is "reserved", aka not supported.
 2017-08-18 16:18:20 -07:00
Yann Collet c523c93b26 improved and fixed --list command, original patches by @ib (#772) 
accepts all skippable frame identifiers.
display in MB or KB, depending on frame size.
fixed combination of skippable and zstd frames.
 2017-08-18 15:57:53 -07:00
Yann Collet 3f54d788e9 removed --list from cli help (-h), reported by Agostino Sarubbo (@asarubbo) (#800) 
redundant with shorter -l.
both -l and --list do the same thing, and are documented in man page.
 2017-08-18 15:15:31 -07:00
Yann Collet bbddff480d Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-18 15:12:30 -07:00
Yann Collet 0ab9d55e7a Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-11 10:17:49 -07:00
Yann Collet e94fd71bbd Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-27 13:49:46 -07:00
Yann Collet ceda7a9a58 minor Makefile refactor 2017-07-21 11:44:39 -07:00
Yann Collet 38ba7002f2 fixed minor warning on unused variable in shell function 2017-07-20 18:39:04 -07:00
Yann Collet a90b16e150 Visual blind fix 2 2017-07-20 15:57:55 -07:00
Yann Collet 4542d7118c Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-20 15:12:21 -07:00
Yann Collet 5e6c5203f3 fixed fuzzer test for non OS-X platforms 2017-07-20 15:11:56 -07:00
Yann Collet b4d460f32c pool.c : blindfix for Visual warnings 2017-07-20 01:13:14 -07:00
Yann Collet 1ca1288689 added --memtest=# command to fuzzer 
to jump directly to relevant test section
 2017-07-19 16:01:16 -07:00
Yann Collet 3974d2b38a blind fix for Windows Multithreading module 
adds a fake 0 return value for mutex/cond init
 2017-07-19 13:33:21 -07:00
Yann Collet b71363b967 check pthread_*_init() success condition 2017-07-19 01:05:40 -07:00
Yann Collet 3b0cff3c33 fixed clang's -Wdocumentation 2017-07-13 18:58:30 -07:00
Yann Collet 2bd6440be0 pinned down error code enum values 
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.

Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
 2017-07-13 17:12:16 -07:00
Yann Collet 3502426fd4 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-13 15:49:19 -07:00
Yann Collet 6733c0777c updated NEWS regarding #760 2017-07-13 15:34:44 -07:00
Yann Collet 3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) 
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
 2017-07-13 10:17:23 -07:00
Yann Collet 132e6efd76 switched ZSTDMT_compress_advanced() last argument to overlapLog 
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
 2017-07-13 02:22:58 -07:00
Yann Collet 4e77f7761d clarified comment on ZSTD_p_contentSizeFlag 2017-07-13 02:09:07 -07:00
Yann Collet 33cecaaaca Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-12 14:38:36 -07:00
Yann Collet 8ef666c325 slightly increased buffer pool, to cover normal "full load" scenarios 
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
 2017-07-12 14:23:34 -07:00
Yann Collet 052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum 
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
 2017-07-11 17:18:26 -07:00
Yann Collet 2a62f48bf4 release input buffers from inside worker thread 
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
 2017-07-11 15:56:40 -07:00
Yann Collet 57236184af buffer pool : all buffers have same size 
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
 2017-07-11 15:17:25 -07:00
Yann Collet 34b2b95631 zstdmt : intermediate outBuffer allocated from within worker 
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
 2017-07-11 14:59:10 -07:00
Yann Collet 16261e6951 buffer pool can be invoked from multiple threads 2017-07-11 14:14:07 -07:00
Yann Collet ef0ff7fe7f zstdmt: removed margin for improved memory usage 2017-07-11 08:54:29 -07:00
Yann Collet 4616fad18b improved ZSTDMT_compress() memory usage 
does not need the input buffer for streaming operations

also : reduced a few tests time length
 2017-07-10 17:16:41 -07:00
Yann Collet 670b1fc547 optimized memory usage for ZSTDMT_compress() 
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.

In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).

To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
 2017-07-10 16:30:55 -07:00
Yann Collet 3510efb02d fix : custom allocator correctly propagated to child contexts 2017-07-10 14:21:40 -07:00
Yann Collet ee3423d709 extended fuzzer MT memory tests 2017-07-10 14:09:16 -07:00
Yann Collet 88da8f1816 fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter() 
also : compile fuzzer with MT enabled
 2017-07-10 14:02:33 -07:00
Yann Collet f9524cf366 added --memtest to fuzzer 2017-07-10 13:48:41 -07:00
Yann Collet e32fb0c1fe added ZSTD_sizeof_CCtx() test 2017-07-10 12:29:57 -07:00
Yann Collet 40156a4967 bumped version nb to v1.3.1 2017-07-08 04:55:09 -07:00
Yann Collet 0f4fc6c20a fixed several conversion warnings 2017-07-07 17:13:12 -07:00
Yann Collet ed0243a63c removed zbufftest from list of all tests 2017-07-07 16:16:14 -07:00
Yann Collet 9bde061a0b fixed minor Visual compilation limitation 2017-07-07 16:14:17 -07:00
Yann Collet 593d517ebf fixed minor cast warning 2017-07-07 16:09:47 -07:00
Yann Collet 842644e42e target gpptest uses CXX environment variable 2017-07-07 15:55:41 -07:00
Yann Collet d4dcace34b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-07 15:53:20 -07:00
Yann Collet ead4dd48f6 new field frameHeader.headerSize 2017-07-07 15:51:24 -07:00
Yann Collet 46396523c0 ZSTD_getFrameHeader : control of windowSize limits is delegated to caller 
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
 2017-07-07 15:32:12 -07:00
Yann Collet 990449b89d new field : ZSTD_frameHeader.frameType 
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
 2017-07-07 15:21:35 -07:00
Yann Collet e622330a3b extended frameHeader.windowSize to unsigned long long 2017-07-07 14:19:01 -07:00
Yann Collet 7758ed8458 fixed fullbench, part 2 2017-07-06 02:48:00 -07:00
Yann Collet 9b2c1acfc0 fixed fullbench 2017-07-06 02:22:57 -07:00
Yann Collet f04deff4fc fixed #718, reported by @GregSlazinski, solution suggested by @mcmilk 2017-07-06 01:42:46 -07:00
Yann Collet d75c0e71c4 minor code refactoring 2017-07-05 18:10:07 -07:00
Yann Collet 3345a91964 cli : use new advanced API by default 2017-07-05 17:34:15 -07:00
Yann Collet 49af41820d clarified status of zstdmt_compress.h API 2017-07-05 17:21:13 -07:00
Yann Collet 27e883371d fixed wrong assert() condition 
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
 2017-07-04 19:33:16 -07:00
Yann Collet 2cb9774f5e more precise estimation of amount to flush at end of stream (single thread mode) 
also : can use DEBUGLEVEL variable in /tests
 2017-07-04 12:39:26 -07:00
Yann Collet 6383372dec fixed : 0-copy in NULL is UB 2017-07-04 10:36:41 -07:00
Yann Collet 2de2396a36 refactor ZSTDMT_compress() 2017-07-03 16:23:36 -07:00
Yann Collet 2084b041f4 fixed comments 2017-07-03 15:52:19 -07:00
Yann Collet 5a77361595 fixed wrong function name in comment 2017-07-03 15:21:24 -07:00
Yann Collet b8280fe929 minor code refactoring 2017-07-03 15:14:55 -07:00
Yann Collet 368b974387 fixed gz error reporting 2017-07-03 13:47:46 -07:00
Yann Collet c9f21c8a71 gzip : only display inflateEnd error when there is no error before 2017-07-03 13:45:09 -07:00
Yann Collet 6c35112d08 gzip decoding does no longer exit() on invalid input 2017-07-03 13:24:50 -07:00
Yann Collet e97ff3b54a decoding malformed lz4 frames does no longer exit() 2017-07-03 11:27:29 -07:00
Yann Collet c34185df13 fileio : decoding malformed lzma frame does no longer exit() 
makes it possible to continue decoding file list
 2017-07-03 10:27:16 -07:00
Yann Collet 2485f88bf8 fixed legacy version init bug 2017-07-01 09:09:34 -07:00
Yann Collet 58bd0e70fc fixed : dictionary compression with new advanced API in Multi-threading mode 2017-06-30 16:01:02 -07:00
Yann Collet d8b33a598d Optimized ZSTDMT single-pass mode speed on large sources 
by ensuring job sizes remain "not too large"
 2017-06-30 15:44:57 -07:00
Yann Collet d5c046c609 implemented shortcut for zstd_compress_generic() in MT mode 
added ZSTDMT_compress_advanced() API
 2017-06-30 14:51:01 -07:00
Yann Collet e7e5a8cef7 made fullbench compatible with multi-threading 
fullbench 61/62 measure speed of ZSTD_compress_generic with 2 threads
 2017-06-29 18:56:24 -07:00
Yann Collet afb0aca739 zstreamtest : big tests are only enabled in 64-bits mode 
to avoid requesting too much memory in 32-bits mode during MT tests
 2017-06-29 18:19:09 -07:00
Yann Collet a3d9926c40 compression optimization opportunity 
switch to single-pass mode directly into output buffer
when outputSize >= ZSTD_compressBound(inputSize).
Speed gains observed with fullbench (~+15% on level 1)
 2017-06-29 14:44:49 -07:00
Yann Collet 2e84bec9ac updated fullbench to also measure ZSTD_compress_generic() 
will make it possible to visualize
optimization opportunity for ZSTD_e_end
 2017-06-29 13:03:10 -07:00
Yann Collet 97f2bf66da minor : fix typo 2017-06-29 11:31:40 -07:00
Yann Collet 7303ed5d74 minor : specify compression level in tags 2017-06-29 11:30:31 -07:00
Yann Collet acbef3decd ZSTD_getFrameContentSize() is promoted to "stable" status 2017-06-29 05:19:51 -07:00
Yann Collet 037466245f refactor ZSTD_check_compressionLevel_monotonicIncrease_memoryBudget() 
use less macro statements
the initial version was meant to work with STATIC_ASSERT
but since it doesn't work and needs assert()
it's possible to rewrite it using normally compiled code
which is better for compiler.

Downside : the error message is less precise.
There is a DEBUGLOG(3,) to compensate.
 2017-06-28 20:24:08 -07:00
Yann Collet 2bf428df45 Merge branch 'advancedAPI2' into refPrefix 2017-06-28 16:35:49 -07:00
Yann Collet 62f7efc04a added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job 2017-06-28 16:25:13 -07:00
Yann Collet 1ca76039af fixed -Wdeclaration-after-statement 2017-06-28 15:40:21 -07:00
Yann Collet 813535105b added function to control monotonic memory budget increase of ZSTD_defaultCParameters[0] 
It's a runtime test, based on assert(),
played once, on first ZSTD_getCParams() usage,
when ZSTD_DEBUG is enabled.
 2017-06-28 15:34:56 -07:00
Yann Collet adbe74a8ac adjusted compression levels to guarantee a monotonically increasing memory budget 2017-06-28 13:22:37 -07:00
Yann Collet 33a6639039 fixed ZSTD_refPrefix with Multithread-enabled CCtx 2017-06-28 11:09:43 -07:00
Yann Collet 1fd2df9c2c fixed zstreamtest 
modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary()
but it is still allowed after ZSTD_refPrefix()
 2017-06-28 09:57:59 -07:00
Yann Collet 2e4274262d controlled dictMode 2017-06-27 17:09:12 -07:00
Yann Collet b7372933b8 implemented ZSTD_refPrefix() 2017-06-27 15:49:12 -07:00
Yann Collet 7d3816183f exposed ZSTD_MAGIC_DICTIONARY in zstd.h 
makes it easier to explain ZSTD_dictMode
 2017-06-27 13:50:34 -07:00
Yann Collet fecc721fd9 added parameter ZSTD_p_refDictContent 2017-06-27 11:46:39 -07:00
Yann Collet c7fb884eea fixed minor conversion warning 2017-06-26 18:02:23 -07:00
Yann Collet dde10b23fe refactored ZSTD_estimateDStreamSize() 
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
 2017-06-26 17:44:26 -07:00
Yann Collet 09ae03a570 ZSTD_estimateCDictSize_advanced() 
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
 2017-06-26 16:47:32 -07:00
Yann Collet 0c9a915a28 ZSTD_estimateCStreamSize_advanced() 2017-06-26 16:02:25 -07:00
Yann Collet 31af8290d1 ZSTD_estimateCCtx_advanced() 
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.

ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
 2017-06-26 15:52:39 -07:00
Yann Collet 6122af9cf5 --no-big-tests for msan 
msan tests require too much RAM for 2 GB VM
 2017-06-23 00:37:15 -07:00
Yann Collet cca1ed14db Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2 2017-06-23 00:13:03 -07:00
Yann Collet 32f546c763 zstreamtest : disabled multi-threading tests with --newapi --no-big-tests 
--no-big-tests is typically used in combination with qemu-user-static
qemu-user-static allocated 4 GB of RAM upfront.
On 2 GB VM, this can degenerate into a crash.
It's not a problem as long as memory is not used.
But with multi-threading enabled, memory fragmentation kicks in,
so the amoung of RAM effectively touched increases,
and can pass beyond the 2 GB limit of the VM.

In single-threaded mode, there is no such issue :
memory requirement is smaller, and remains well-located,
so very little fragmentation is expected.

This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests`
work fine on Travis CI.
 2017-06-23 00:09:02 -07:00
Yann Collet 49f8459d11 fixed minor cast warning 2017-06-21 18:43:39 -07:00
Yann Collet 4d3bdcf130 reduced CLevelMax for --no-big-tests 2017-06-21 18:20:02 -07:00