ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,203 Commits 17 Branches 77 Tags
1c7b914cdfcc63cf75f40455c4a1593393feb365
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann Collet 1c7b914cdf update README on BUCK file 2017-09-06 16:23:39 -07:00
Yann Collet 36374cc3b4 update and clarify lib/README 2017-09-06 16:15:18 -07:00
Yann Collet ed7ace38e9 updated NEWS 
fix for 32-bits decoder
 2017-09-01 11:58:37 -07:00
Yann Collet 8a5c0c98ae restored 32-bits decoder ability to decode long offsets (>32 MB, levels 21+) 2017-09-01 11:56:57 -07:00
Yann Collet 0558850735 bench stops immediately on decoding error 2017-09-01 11:46:15 -07:00
Yann Collet 36aa8b5999 improved decoding speed 2017-09-01 11:40:59 -07:00
Yann Collet d963daa6a9 fixed minor warning (empty translation unit) 2017-09-01 00:12:07 -07:00
Yann Collet 3704507774 fixed decompression bug reported by @Etsukata (#828) 2017-09-01 00:05:37 -07:00
Yann Collet 0c314cde4b updated zstd API manual 
for new CCtxParams object
 2017-08-31 18:28:19 -07:00
Yann Collet 369c29dd1a fixed impact of merge conflict for longRange 2017-08-31 18:25:56 -07:00
Yann Collet c7818fc676 Merge branch 'modTests' into dev 
fixed conflict
 2017-08-31 17:00:16 -07:00
Yann Collet 4299c27132 improved console log of utils.h 
removed a warning when compiling on Windows
 2017-08-31 16:58:47 -07:00
Yann Collet d791aef80e blind attempt at removing gcc dependency 
from appveyor's mingw builds, for #815
 2017-08-31 15:24:17 -07:00
Yann Collet 4e81dc04f9 Merge branch 'dev' into modTests 2017-08-31 15:14:13 -07:00
Yann Collet 3982edf813 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-31 15:13:48 -07:00
Yann Collet e0cecd8736 fixed poolTests 
needs more dependencies from zstd for custom allocators and error codes
 2017-08-31 15:13:31 -07:00
Yann Collet 179b161dc1 fixed poolTest 
needs more dependencies from zstd for custom allocators and error codes
 2017-08-31 15:02:14 -07:00
Yann Collet 8e298382a8 changed target allarch into allzstd 
allzstd contains only zstd-related tests.
allmost = allzstd + zwrapper tests (which require zlib)
 2017-08-31 14:30:52 -07:00
Yann Collet b0cb081dc8 last batch of header files changed to reflect new license (#825) 
only remains to update contrib/linux-kernel (@terrelln)
 2017-08-31 12:20:50 -07:00
Yann Collet e21384fffb fixed more file headers after license change (#825) 2017-08-31 12:11:57 -07:00
Yann Collet e9dc204f42 fixed a bunch of headers after license change (#825) 2017-08-31 11:24:54 -07:00
Yann Collet 9023898f1e updated NEWS 2017-08-30 17:44:12 -07:00
Yann Collet 394bdd7db9 changed license for examples 
intentionnally this time
 2017-08-29 09:24:11 -07:00
Yann Collet 6b2b6a9bd5 fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously 
can only happen if windowLog>=27  (level 22 --ultra)
 2017-08-22 12:09:21 -07:00
Yann Collet 232d62b637 fixed a few headers that were too hastily copy/pasted during last license change 2017-08-21 11:24:32 -07:00
Yann Collet 4912fc2acc updated NEWS for v1.3.1 2017-08-20 11:45:58 -07:00
Yann Collet e8d35cc5e9 minor formulation change, recommended by @ulikunitz 2017-08-20 10:39:20 -07:00
Yann Collet d6394cc4c3 fixed test-zstd-nolegacy 2017-08-20 10:15:44 -07:00
Yann Collet 7db552676e reduced pool queue to 0 to save memory 
fixed : pool performance when jobs are fires fast and queueSize==0
 2017-08-19 15:07:54 -07:00
Yann Collet 1c108c811e cli : Display supported formats on -vV command 
Requested and inspired by patch from @ib (#771)
 2017-08-19 13:33:50 -07:00
Yann Collet d0d06e421f added alternative representation for huffman bistream 2017-08-19 12:26:09 -07:00
Yann Collet 8b12812147 fix #803 : wrong example in huffman bitstream section, reported by @ulikunitz 2017-08-19 12:17:57 -07:00
Yann Collet 2ecd34ee5e fixed unused variables warnings 2017-08-19 01:23:49 -07:00
Yann Collet 23706fb743 updated doc on compilation variables 2017-08-19 01:14:36 -07:00
Yann Collet 9203003d5f fixed zstd-nolegacy 
and added it to allVariants for CI testings
 2017-08-19 01:01:53 -07:00
Yann Collet 4b387729b6 fixed zstd-small 
and added it to shortest for CI tests
 2017-08-19 00:48:29 -07:00
Yann Collet 7729ab83bb Merge branch 'dev' into variants 2017-08-19 00:37:06 -07:00
Yann Collet 32e943b3ef Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-19 00:36:37 -07:00
Yann Collet 288e263ec4 added zstd-decompress to the list of variants tested 2017-08-18 18:39:39 -07:00
Yann Collet 166645e7b3 fixed zstd-compress 
file-information is dependent on decompression functions.
it should only be enabled when ZSTD_NODECOMPRESS is not set.

also : added zstd-compress compilation test into `make shortest`
 2017-08-18 18:30:41 -07:00
Yann Collet f207b39f55 blindfix for Windows conversion warning 
long type is 32-bits on Windows 64,
while it's 64-bits on Unix.
64-to-32 shortening conversion for long is a specific Windows issue.
 2017-08-18 17:06:12 -07:00
Yann Collet 32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Yann Collet 4f73b3b55d added GPLv2 license 
and removed PATENTS clause
 2017-08-18 16:32:08 -07:00
Yann Collet 88d2f72df9 fixed --list command in presence of special blocks 
block type RLE is special, compressed size is always 1.
block type 3 is "reserved", aka not supported.
 2017-08-18 16:18:20 -07:00
Yann Collet c523c93b26 improved and fixed --list command, original patches by @ib (#772) 
accepts all skippable frame identifiers.
display in MB or KB, depending on frame size.
fixed combination of skippable and zstd frames.
 2017-08-18 15:57:53 -07:00
Yann Collet 3f54d788e9 removed --list from cli help (-h), reported by Agostino Sarubbo (@asarubbo) (#800) 
redundant with shorter -l.
both -l and --list do the same thing, and are documented in man page.
 2017-08-18 15:15:31 -07:00
Yann Collet bbddff480d Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-18 15:12:30 -07:00
Yann Collet 0ab9d55e7a Merge branch 'dev' of github.com:facebook/zstd into dev 2017-08-11 10:17:49 -07:00
Yann Collet e94fd71bbd Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-27 13:49:46 -07:00
Yann Collet ceda7a9a58 minor Makefile refactor 2017-07-21 11:44:39 -07:00
Yann Collet 38ba7002f2 fixed minor warning on unused variable in shell function 2017-07-20 18:39:04 -07:00
Yann Collet a90b16e150 Visual blind fix 2 2017-07-20 15:57:55 -07:00
Yann Collet 4542d7118c Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-20 15:12:21 -07:00
Yann Collet 5e6c5203f3 fixed fuzzer test for non OS-X platforms 2017-07-20 15:11:56 -07:00
Yann Collet b4d460f32c pool.c : blindfix for Visual warnings 2017-07-20 01:13:14 -07:00
Yann Collet 1ca1288689 added --memtest=# command to fuzzer 
to jump directly to relevant test section
 2017-07-19 16:01:16 -07:00
Yann Collet 3974d2b38a blind fix for Windows Multithreading module 
adds a fake 0 return value for mutex/cond init
 2017-07-19 13:33:21 -07:00
Yann Collet b71363b967 check pthread_*_init() success condition 2017-07-19 01:05:40 -07:00
Yann Collet 3b0cff3c33 fixed clang's -Wdocumentation 2017-07-13 18:58:30 -07:00
Yann Collet 2bd6440be0 pinned down error code enum values 
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.

Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
 2017-07-13 17:12:16 -07:00
Yann Collet 3502426fd4 Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-13 15:49:19 -07:00
Yann Collet 6733c0777c updated NEWS regarding #760 2017-07-13 15:34:44 -07:00
Yann Collet 3a60efd3a9 policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX (#760) 
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
 2017-07-13 10:17:23 -07:00
Yann Collet 132e6efd76 switched ZSTDMT_compress_advanced() last argument to overlapLog 
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
 2017-07-13 02:22:58 -07:00
Yann Collet 4e77f7761d clarified comment on ZSTD_p_contentSizeFlag 2017-07-13 02:09:07 -07:00
Yann Collet 33cecaaaca Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-12 14:38:36 -07:00
Yann Collet 8ef666c325 slightly increased buffer pool, to cover normal "full load" scenarios 
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
 2017-07-12 14:23:34 -07:00
Yann Collet 052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum 
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
 2017-07-11 17:18:26 -07:00
Yann Collet 2a62f48bf4 release input buffers from inside worker thread 
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
 2017-07-11 15:56:40 -07:00
Yann Collet 57236184af buffer pool : all buffers have same size 
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
 2017-07-11 15:17:25 -07:00
Yann Collet 34b2b95631 zstdmt : intermediate outBuffer allocated from within worker 
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
 2017-07-11 14:59:10 -07:00
Yann Collet 16261e6951 buffer pool can be invoked from multiple threads 2017-07-11 14:14:07 -07:00
Yann Collet ef0ff7fe7f zstdmt: removed margin for improved memory usage 2017-07-11 08:54:29 -07:00
Yann Collet 4616fad18b improved ZSTDMT_compress() memory usage 
does not need the input buffer for streaming operations

also : reduced a few tests time length
 2017-07-10 17:16:41 -07:00
Yann Collet 670b1fc547 optimized memory usage for ZSTDMT_compress() 
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.

In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).

To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
 2017-07-10 16:30:55 -07:00
Yann Collet 3510efb02d fix : custom allocator correctly propagated to child contexts 2017-07-10 14:21:40 -07:00
Yann Collet ee3423d709 extended fuzzer MT memory tests 2017-07-10 14:09:16 -07:00
Yann Collet 88da8f1816 fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter() 
also : compile fuzzer with MT enabled
 2017-07-10 14:02:33 -07:00
Yann Collet f9524cf366 added --memtest to fuzzer 2017-07-10 13:48:41 -07:00
Yann Collet e32fb0c1fe added ZSTD_sizeof_CCtx() test 2017-07-10 12:29:57 -07:00
Yann Collet 40156a4967 bumped version nb to v1.3.1 2017-07-08 04:55:09 -07:00
Yann Collet 0f4fc6c20a fixed several conversion warnings 2017-07-07 17:13:12 -07:00
Yann Collet ed0243a63c removed zbufftest from list of all tests 2017-07-07 16:16:14 -07:00
Yann Collet 9bde061a0b fixed minor Visual compilation limitation 2017-07-07 16:14:17 -07:00
Yann Collet 593d517ebf fixed minor cast warning 2017-07-07 16:09:47 -07:00
Yann Collet 842644e42e target gpptest uses CXX environment variable 2017-07-07 15:55:41 -07:00
Yann Collet d4dcace34b Merge branch 'dev' of github.com:facebook/zstd into dev 2017-07-07 15:53:20 -07:00
Yann Collet ead4dd48f6 new field frameHeader.headerSize 2017-07-07 15:51:24 -07:00
Yann Collet 46396523c0 ZSTD_getFrameHeader : control of windowSize limits is delegated to caller 
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
 2017-07-07 15:32:12 -07:00
Yann Collet 990449b89d new field : ZSTD_frameHeader.frameType 
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
 2017-07-07 15:21:35 -07:00
Yann Collet e622330a3b extended frameHeader.windowSize to unsigned long long 2017-07-07 14:19:01 -07:00
Yann Collet 7758ed8458 fixed fullbench, part 2 2017-07-06 02:48:00 -07:00
Yann Collet 9b2c1acfc0 fixed fullbench 2017-07-06 02:22:57 -07:00
Yann Collet f04deff4fc fixed #718, reported by @GregSlazinski, solution suggested by @mcmilk 2017-07-06 01:42:46 -07:00
Yann Collet d75c0e71c4 minor code refactoring 2017-07-05 18:10:07 -07:00
Yann Collet 3345a91964 cli : use new advanced API by default 2017-07-05 17:34:15 -07:00
Yann Collet 49af41820d clarified status of zstdmt_compress.h API 2017-07-05 17:21:13 -07:00
Yann Collet 27e883371d fixed wrong assert() condition 
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
 2017-07-04 19:33:16 -07:00
Yann Collet 2cb9774f5e more precise estimation of amount to flush at end of stream (single thread mode) 
also : can use DEBUGLEVEL variable in /tests
 2017-07-04 12:39:26 -07:00
Yann Collet 6383372dec fixed : 0-copy in NULL is UB 2017-07-04 10:36:41 -07:00