Yann Collet
34b2b95631
zstdmt : intermediate outBuffer allocated from within worker
...
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
2017-07-11 14:59:10 -07:00
Yann Collet
16261e6951
buffer pool can be invoked from multiple threads
2017-07-11 14:14:07 -07:00
Yann Collet and GitHub
fe80d45807
Merge pull request #755 from jacquesg/solaris
...
Solaris fix
2017-07-11 12:11:44 -07:00
Paul Cruz
34afb9b23e
changed to using ZSTD_compressBegin_usingDict() and fixed strange issue with ZSTD_compressContinue()
2017-07-11 11:50:00 -07:00
Jacques Germishuys
c325c8db82
fix missing symbol 'nanosleep' for Solaris
2017-07-11 19:26:10 +02:00
Paul Cruz
7ec5928626
fixed an error where -c argument wasn't working for single files
2017-07-11 10:23:25 -07:00
Stella Lau
f6c5d07fe2
Save v3
2017-07-11 09:23:44 -07:00
Yann Collet
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
Stella Lau
6c3673f4c3
Add rolling hash
2017-07-10 22:27:43 -07:00
Paul Cruz
f918545491
made some progress on improving compression ratio, but problems exist with speed limits, and for some reason higher compression levels are really slow
2017-07-10 18:16:42 -07:00
Yann Collet
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
Yann Collet
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
Paul Cruz
01fc7c4244
changed how the detection of the last job works
2017-07-10 16:27:58 -07:00
Paul Cruz
c36552ef8a
dst buffer should use ZSTD_compressBound to determine how much space it needs
2017-07-10 16:10:19 -07:00
Paul Cruz
7aa36df6df
fixed memory leak that was happening when creating jobs
2017-07-10 16:03:09 -07:00
Stella Lau
ef2b728316
Clean up and refactor compress function
2017-07-10 15:48:47 -07:00
Paul Cruz
e410d63d45
made input buffer an internal part of the compression context
2017-07-10 15:37:14 -07:00
Yann Collet
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
Yann Collet
ee3423d709
extended fuzzer MT memory tests
2017-07-10 14:09:16 -07:00
Yann Collet
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
Yann Collet
f9524cf366
added --memtest to fuzzer
2017-07-10 13:48:41 -07:00
Stella Lau
e4155b11d7
Add warning flags to makefile and clean up code to remove warnings
2017-07-10 13:08:19 -07:00
Stella Lau
10a71d9f1c
Add compression context
2017-07-10 12:38:27 -07:00
Yann Collet
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
Paul Cruz
89190ef07d
renamed pool.c to poolTests.c
2017-07-10 11:32:30 -07:00
Paul Cruz
cc7f8e4d71
small changes
2017-07-10 11:10:11 -07:00
Paul Cruz
7e09b508ff
changed name
2017-07-10 11:05:37 -07:00
Paul Cruz
ed72ea5438
removed single from Makefile
2017-07-10 10:58:03 -07:00
Paul Cruz
ced3ec5714
removed scripts
2017-07-10 10:53:02 -07:00
Paul Cruz
82f0d64bee
removed single.c
2017-07-10 10:51:50 -07:00
Paul Cruz
62ebbabd32
updated error checking in each thread
2017-07-10 09:36:22 -07:00
Stella Lau
ae9cf235d6
Add LDM_DCtx
2017-07-10 07:38:09 -07:00
Stella Lau
5432214ee3
Minor refactoring
2017-07-10 06:50:49 -07:00
Stella Lau
b94b468e84
Merge branch 'ldm' of https://github.com/stellamplau/zstd into ldm
2017-07-10 06:32:46 -07:00
Stella Lau
474e06ac5b
Minor refactoring
2017-07-10 06:32:29 -07:00
Stella Lau
eb280cd568
Add folder for old versions
2017-07-10 06:32:05 -07:00
Stella Lau
719ccdc5a5
Update mainfile
2017-07-09 22:45:54 -07:00
Yann Collet
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
Yann Collet
0f4fc6c20a
fixed several conversion warnings
2017-07-07 17:13:12 -07:00
Stella Lau
acdeb9f302
Add compression statistics
2017-07-07 17:09:28 -07:00
Paul Cruz
c3ae23d459
added ability to compress without specifying out filename
2017-07-07 17:07:05 -07:00
Yann Collet
ed0243a63c
removed zbufftest from list of all tests
2017-07-07 16:16:14 -07:00
Yann Collet
9bde061a0b
fixed minor Visual compilation limitation
2017-07-07 16:14:17 -07:00
Yann Collet
593d517ebf
fixed minor cast warning
2017-07-07 16:09:47 -07:00
Paul Cruz
7163ffafde
playing around with adapt param
2017-07-07 15:56:00 -07:00
Yann Collet
842644e42e
target gpptest uses CXX environment variable
2017-07-07 15:55:41 -07:00
Yann Collet
d4dcace34b
Merge branch 'dev' of github.com:facebook/zstd into dev
2017-07-07 15:53:20 -07:00
Yann Collet
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
Paul Cruz
1c9d6b2c6b
rewrote time elapsed with UTIL
2017-07-07 15:42:20 -07:00
Yann Collet
46396523c0
ZSTD_getFrameHeader : control of windowSize limits is delegated to caller
...
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
2017-07-07 15:32:12 -07:00
Yann Collet