Commit Graph
1208 Commits
Author SHA1 Message Date
Yann Collet 5b726dbe4d fix gcc-arm warning "suggest braces around empty body" 2017-01-12 17:46:46 +01:00
Yann Collet ad9f6bd123 zstdmt : fix : resources properly collected even when early fail
In previous version, main function would return early when detecting a job error.
Late threads resources were therefore not collected back into pools.
New version just register the error, but continue the collecting process.
All buffers and context should be released back to pool before leaving main function.
2017-01-12 03:06:35 +01:00
Yann Collet b05c4828ea zstdmt : correctly check for cctx and buffer allocation
Result from getBuffer and getCCtx could be NULL when allocation fails.
Now correctly checks : job creation stop and last job reports an allocation error.
releaseBuffer and releaseCCtx are now also compatible with NULL input.

Identified a new potential issue :
when early job fails, later jobs are not collected for resource retrieval.
2017-01-12 02:01:28 +01:00
Yann Collet 107bcbbbc2 zstdmt : changed internal naming from frame to chunk
Since the result of mt compression is a single frame,
changed naming, which implied the concatenation of multiple frames.

minor : ensures that content size is written in header
2017-01-12 01:25:46 +01:00
Yann Collet 5eb749e734 ZSTDMT_compress() creates a single frame
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()

As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().

This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
2017-01-11 18:21:25 +01:00
Yann Collet 04cbc36499 minor refactor (release CCtx 1st) and comment clarification 2017-01-11 16:08:08 +01:00
Yann Collet 085179bb78 fixed ZSTDMT_createCCtx() : checked inner objects are properly created 2017-01-11 15:58:05 +01:00
Yann Collet 8ce1cc2bec improved ZSTD_createCCtxPool() cancellation
use ZSTD_freeCCtxPool() to release the partially created pool.
avoids to duplicate logic.

Also : identified a new difficult corner case :
when freeing the Pool, all CCtx should be previously released back to the pool.
Otherwise, it means some CCtx are still in use.
There is currently no clear policy on what to do in such a case.
Note : it's supposed to never happen.
Since pool creation/usage is static, it has no external user,
which limits risks.
2017-01-11 15:44:26 +01:00
Yann Collet 47557ba2b2 fixed ZSTDMT_createCCtxPool() when inner CCtx creation fails 2017-01-11 15:35:56 +01:00
Yann Collet 6334b04d61 compile object files, for faster recompilation 2017-01-02 03:22:18 +01:00
Yann Collet f1cb55192c fixed linux warnings 2017-01-02 01:11:55 +01:00
Yann Collet 0ec6a95ba1 minor fixes 2017-01-02 00:49:42 +01:00
Yann Collet 2ec635a162 use pthread_cond to send signals between threads 2017-01-01 17:31:33 +01:00
Nick Terrell bb13387d7d Fix pool for threading.h 2016-12-31 19:10:47 -05:00
Nick Terrell 4204e03e77 Add threading.h condition variables 2016-12-31 19:10:29 -05:00
Yann Collet 3b9d434356 extended ZSTDMT code support for non-MT systems and WIN32 (preliminary) 2016-12-31 16:32:19 +01:00
Yann Collet c8efc1c874 simplified Buffer Pool 2016-12-31 14:45:33 +01:00
Yann Collet 3b29dbd9e8 new zstdmt version using generic treadpool 2016-12-31 06:04:25 +01:00
Yann Collet c6a6417458 bench correctly measures time for multi-threaded compression (posix only) 2016-12-31 03:31:26 +01:00
Yann ColletandGitHub f765a375a5 Merge pull request #504 from terrelln/thread-pool
[zstdmt] Add thread pool
2016-12-30 15:31:49 +01:00
Nick Terrell e777a5be6b Add a thread pool for ZSTDMT and COVER 2016-12-29 23:39:44 -08:00
Yann Collet e70912c72b Changed : input divided into roughly equal parts.
Debug : can measure time waiting for mutexes to unlock.
2016-12-29 01:24:01 +01:00
Yann Collet 6c0ed9483a compression threads use ZSTD_compressCCtx() 2016-12-28 17:08:28 +01:00
Yann ColletandGitHub 8d7432914f Merge pull request #503 from inikep/dev11
Dev11
2016-12-28 16:50:39 +01:00
Yann Collet ce9e1452fd protect buffer pool with a mutex 2016-12-28 15:31:19 +01:00
Przemyslaw Skibinski 75f3a3a335 changed default PREFIX and MANDIR 2016-12-28 12:32:41 +01:00
Yann Collet 3d93f2fce7 first zstdmt sketch 2016-12-27 07:19:36 +01:00
Yann Collet 39c105c605 Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-23 22:25:31 +01:00
Yann Collet aca113f4f5 fixed ZSTD_sizeof_?Dict() 2016-12-23 22:25:03 +01:00
Yann ColletandGitHub c07d2e3a31 Merge pull request #499 from inikep/dev11
improved *BSD and Solaris compatibility
2016-12-23 21:32:03 +01:00
Przemyslaw Skibinski 63b0014b96 BSD: improved "make install" 2016-12-23 10:05:49 +01:00
Nick Terrell 1b5d4a7d53 ZDICT_finalizeDictionary() flipped comparison 2016-12-22 18:14:57 -08:00
Nick Terrell bcbe77e994 ZDICT_finalizeDictionary() flipped comparison
`ZDICT_finalizeDictionary()` had a flipped comparison.
I also allowed `dictBufferCapacity == dictContentSize`.
It might be the case that the user wants to fill the dictionary
completely up, and then let zstd take exactly the space it needs
for the entropy tables.
2016-12-22 18:01:14 -08:00
Nick Terrell 78a0072d5a Fix failing test due to deprecation warning 2016-12-22 17:36:16 -08:00
Yann Collet d76d1a9ef0 added ZDICT_finalizeDictionary() 2016-12-22 20:18:43 +01:00
Przemyslaw Skibinski b999170311 Solaris: working "make -C lib install" 2016-12-22 20:14:37 +01:00
Yann Collet ba75e9d8c3 fix : zlib wrapper compile in gnu90 mode 2016-12-21 19:57:18 +01:00
Yann Collet 0819abe3c1 added ZSTD_createDDict_byReference() body 2016-12-21 19:25:15 +01:00
Yann Collet 4e5eea61a8 added ZSTD_createDDict_byReference() 2016-12-21 16:44:35 +01:00
Yann Collet 8333106b8a Merge branch 'dev' of github.com:facebook/zstd into dev 2016-12-21 16:44:24 +01:00
Yann ColletandGitHub 0d7e84899f Merge pull request #489 from inikep/v112
improved detection of POSIX
2016-12-21 16:42:46 +01:00
Yann Collet 1f57c2ed32 added : ZSTD_createCDict_byReference() 2016-12-21 16:20:11 +01:00
Przemyslaw Skibinski 2f6ccee6af platform.h: removed Compiler Options 2016-12-21 13:23:34 +01:00
Przemyslaw Skibinski 5736db219e fix basic types redefinition 2016-12-21 09:26:00 +01:00
Nick Terrell 8157a4c3cc Fix dictionary loading bug causing an MSAN failure
Offset rep codes must be in the range `[1, dictSize)`.
Fix dictionary loading to reject `0` as a offset rep code.
2016-12-20 10:47:52 -08:00
Przemyslaw Skibinski f8046b8e72 Merge remote-tracking branch 'refs/remotes/facebook/dev' into v112
# Conflicts:
#	appveyor.yml
2016-12-19 08:20:26 +01:00
Yann Collet d564faa3c6 fix : ZSTD_initCStream_srcSize() correctly set srcSize in frame header 2016-12-18 21:39:15 +01:00
Yann Collet 1496c3dc47 Fix : size estimation when some samples are very large 2016-12-18 11:58:23 +01:00
Yann Collet d46ecb58a5 added dll compilation tests 2016-12-17 16:28:12 +01:00
Nick Terrell 8de46ab51a Export all API functions 2016-12-16 13:27:30 -08:00