Stella Lau
583dda17a8
Update rolling hash
2017-07-11 18:13:26 -07:00
Yann Collet
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
Paul Cruz
0a401852c4
added debug statement
2017-07-11 16:50:50 -07:00
Yann Collet
2a62f48bf4
release input buffers from inside worker thread
...
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
2017-07-11 15:56:40 -07:00
Paul Cruz
72a183efad
changed dictionary size, added debugging statements
2017-07-11 15:49:52 -07:00
Yann Collet
57236184af
buffer pool : all buffers have same size
...
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
2017-07-11 15:17:25 -07:00
Paul Cruz
7c54e09347
updated DEBUG statements
2017-07-11 15:15:41 -07:00
Paul Cruz
a3c077b8c6
added error message, updated copying dictionary into the input buffer
2017-07-11 15:00:52 -07:00
Yann Collet
34b2b95631
zstdmt : intermediate outBuffer allocated from within worker
...
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
2017-07-11 14:59:10 -07:00
Yann Collet
16261e6951
buffer pool can be invoked from multiple threads
2017-07-11 14:14:07 -07:00
Yann Collet and GitHub
fe80d45807
Merge pull request #755 from jacquesg/solaris
...
Solaris fix
2017-07-11 12:11:44 -07:00
Paul Cruz
34afb9b23e
changed to using ZSTD_compressBegin_usingDict() and fixed strange issue with ZSTD_compressContinue()
2017-07-11 11:50:00 -07:00
Jacques Germishuys
c325c8db82
fix missing symbol 'nanosleep' for Solaris
2017-07-11 19:26:10 +02:00
Paul Cruz
7ec5928626
fixed an error where -c argument wasn't working for single files
2017-07-11 10:23:25 -07:00
Stella Lau
f6c5d07fe2
Save v3
2017-07-11 09:23:44 -07:00
Yann Collet
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
Stella Lau
6c3673f4c3
Add rolling hash
2017-07-10 22:27:43 -07:00
Paul Cruz
f918545491
made some progress on improving compression ratio, but problems exist with speed limits, and for some reason higher compression levels are really slow
2017-07-10 18:16:42 -07:00
Yann Collet
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
Yann Collet
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
Paul Cruz
01fc7c4244
changed how the detection of the last job works
2017-07-10 16:27:58 -07:00
Paul Cruz
c36552ef8a
dst buffer should use ZSTD_compressBound to determine how much space it needs
2017-07-10 16:10:19 -07:00
Paul Cruz
7aa36df6df
fixed memory leak that was happening when creating jobs
2017-07-10 16:03:09 -07:00
Stella Lau
ef2b728316
Clean up and refactor compress function
2017-07-10 15:48:47 -07:00
Paul Cruz
e410d63d45
made input buffer an internal part of the compression context
2017-07-10 15:37:14 -07:00
Yann Collet
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
Yann Collet
ee3423d709
extended fuzzer MT memory tests
2017-07-10 14:09:16 -07:00
Yann Collet
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
Yann Collet
f9524cf366
added --memtest to fuzzer
2017-07-10 13:48:41 -07:00
Stella Lau
e4155b11d7
Add warning flags to makefile and clean up code to remove warnings
2017-07-10 13:08:19 -07:00
Stella Lau
10a71d9f1c
Add compression context
2017-07-10 12:38:27 -07:00
Yann Collet
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
Paul Cruz
89190ef07d
renamed pool.c to poolTests.c
2017-07-10 11:32:30 -07:00
Paul Cruz
cc7f8e4d71
small changes
2017-07-10 11:10:11 -07:00
Paul Cruz
7e09b508ff
changed name
2017-07-10 11:05:37 -07:00
Paul Cruz
ed72ea5438
removed single from Makefile
2017-07-10 10:58:03 -07:00
Paul Cruz
ced3ec5714
removed scripts
2017-07-10 10:53:02 -07:00
Paul Cruz
82f0d64bee
removed single.c
2017-07-10 10:51:50 -07:00
Paul Cruz
62ebbabd32
updated error checking in each thread
2017-07-10 09:36:22 -07:00
Stella Lau
ae9cf235d6
Add LDM_DCtx
2017-07-10 07:38:09 -07:00
Stella Lau
5432214ee3
Minor refactoring
2017-07-10 06:50:49 -07:00
Stella Lau
b94b468e84
Merge branch 'ldm' of https://github.com/stellamplau/zstd into ldm
2017-07-10 06:32:46 -07:00
Stella Lau
474e06ac5b
Minor refactoring
2017-07-10 06:32:29 -07:00
Stella Lau
eb280cd568
Add folder for old versions
2017-07-10 06:32:05 -07:00
Stella Lau
719ccdc5a5
Update mainfile
2017-07-09 22:45:54 -07:00
Yann Collet
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
Yann Collet
0f4fc6c20a
fixed several conversion warnings
2017-07-07 17:13:12 -07:00
Stella Lau
acdeb9f302
Add compression statistics
2017-07-07 17:09:28 -07:00
Paul Cruz
c3ae23d459
added ability to compress without specifying out filename
2017-07-07 17:07:05 -07:00
Yann Collet
ed0243a63c
removed zbufftest from list of all tests
2017-07-07 16:16:14 -07:00
Stella Lau