55 Commits
Author SHA1 Message Date
Yann Collet e6f4b46493 playTests.sh does no longer needs grep -E
it makes the test script more portable across posix systems
because `grep -E` is not guaranteed
while `grep` is fairly common.
2024-01-15 11:16:46 -08:00
Yonatan KomornikandGitHub c78f434aa4 Fix zstd-dll build missing dependencies (#3496)
* Fixes zstd-dll build (https://github.com/facebook/zstd/issues/3492):
- Adds pool.o and threading.o dependency to the zstd-dll target
- Moves custom allocation functions into header to avoid needing to add dependency on common.o
- Adds test target for zstd-dll
- Adds github workflow that buildis zstd-dll
2023-02-12 12:32:31 -08:00
Yann Collet ea684c335a added c89 build test to CI 2023-01-19 14:59:30 -08:00
W. Felix Handte 5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
W. Felix Handte 8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
Yonatan Komornik 500f02eb66 Fixes two bugs in the Windows thread / pthread translation layer
1. If threads are resized the threads' `ZSTD_pthread_t` might move
while the worker still holds a pointer into it (see more details in #3120).
2. The join operation was waiting for a thread and then return its `thread.arg`
as a return value, but since the `ZSTD_pthread_t thread` was passed by value it
would have a stale `arg` that wouldn't match the thread's actual return value.

This fix changes the `ZSTD_pthread_join` API and removes support for returning
a value. This means that we are diverging from the `pthread_join` API and this
is no longer just an alias.
In the future, if needed, we could return a Windows thread's return value using
`GetExitCodeThread`, but as this path wouldn't be excised in any case, it's
preferable to not add it right now.
2022-12-17 13:38:02 -08:00
Yann Collet b7d55cfa0d fix issue #3119
fix segfault error when running zstreamtest with MALLOC_PERTURB_
2022-10-12 23:04:23 -07:00
Yonatan KomornikandGitHub 1598e6c634 Async write for decompression (#2975)
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
2022-01-21 13:55:41 -08:00
Yann Collet 6211bfee5e fixed backup prototype for POOL_sizeof() 2021-12-30 14:33:21 -08:00
Yann Collet b1978d60ee POOL_sizeof() only needs a const read-only reference 2021-12-30 14:08:51 -08:00
Nick TerrellandNick Terrell a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
Nick Terrell 66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
Martin Liska b684900a4a Allow external creation of POOLs that can be shared. 2020-10-07 12:44:33 +02:00
Nick Terrell c465f24457 ZSTD_ prefix mem{cpy,move,set},malloc,calloc,free 2020-08-26 12:26:03 -07:00
Nick Terrell a686d306d2 Rename ZSTD_{malloc,calloc,free} to ZSTD_custom{Malloc,Calloc,Free} 2020-08-26 12:25:08 -07:00
Nick Terrell 80f577baa2 Move standard includes to zstd_deps.h 2020-08-26 12:25:08 -07:00
yoshihitoh c6548eac8e Rename static vars to avoid redefinition error. 2020-06-29 10:51:50 +09:00
Nick Terrell ac58c8d720 Fix copyright and license lines
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
2020-03-26 17:02:06 -07:00
Nick Terrell 243824551f [threading] Add debug utilities 2019-10-18 15:05:34 -07:00
Nick Terrell a8daa2d683 Signal before unlocking in pool.c 2018-11-08 10:45:53 -08:00
Yann Collet fbd5dfc1b1 changed POOL_resize() return type to int
return is now just en error code.
This guarantee that `ctx` remains valid after POOL_resize().
Gets rid of internal POOL_free() operation.
2018-06-22 12:14:59 -07:00
Yann Collet 243cd9d8bb add a cond_broadcast after resize
to make sure all threads (notably newly available threads)
get awaken to immediately process potential items in the queue.
2018-06-21 18:04:58 -07:00
Yann Collet 818e72b4d5 added extended POOL test
abrupt end + downsizing with running jobs remaining in queue.

also : POOL_resize() requires numThreads >= 1
2018-06-21 14:58:59 -07:00
Yann Collet 6de249c1c6 fixed: bug when counting nb of active threads
when queueSize > 1

also : added a test in testpool.c
       verifying resizing is effective.
2018-06-20 18:28:49 -07:00
Yann Collet 6b48eb12c0 change control of threadLimit
now limits maximum nb of active threads
even when queueSize > 1.
2018-06-20 14:35:39 -07:00
Yann Collet 62469c9f41 fixed wrong size in pthread struct transfer 2018-06-19 20:14:03 -07:00
Yann Collet 166901dc72 reduced POOL_resize() restriction
It's not necessary to ensure that no job is ongoing.
The pool is only expanded, existing threads are preserved.
In case of error, the only option is to return NULL and terminate the thread pool anyway.
2018-06-19 18:07:18 -07:00
Yann Collet 4567c57199 finalized POOL_resize()
POOL_ctx* POOL_resize(POOL_ctx* ctx, size_t numThreads)

The function may fail, and returns a NULL pointer in this case.
2018-06-19 16:03:12 -07:00
Yann Collet 1c714fda3f introduced POOL_resize()
not complete yet :
finalize behavior in case of unfinished expansion
2018-06-18 20:46:39 -07:00
Yann Collet 997e4d0ccd added POOL_tryAdd() 2018-01-18 14:39:51 -08:00
Yann Collet 61e5a1adfc removed direct call to malloc() from pool.c 2017-10-31 17:43:24 -07:00
Yann Collet 86b4fe5b45 adjustCParams : restored previous behavior
unknowns srcSize presumed small if there is a dictionary (dictSize>0)
and presumed large otherwise.
2017-09-28 18:14:28 -07:00
Nick Terrell 6c41adfb28 [libzstd] pthread function prefixed with ZSTD_
* `sed -i 's/pthread_/ZSTD_pthread_/g' lib/{,common,compress,decompress,dictBuilder}/*.[hc]`
* Fix up `lib/common/threading.[hc]`
* `sed -i s/PTHREAD_MUTEX_LOCK/ZSTD_PTHREAD_MUTEX_LOCK/g lib/compress/zstdmt_compress.c`
2017-09-27 11:48:48 -07:00
Yann Collet 3128e03be6 updated license header
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
Nick Terrell 423b133568 [POOL] Allow free on NULL when multithreading is disabled 2017-09-05 11:18:13 -07:00
Nick Terrell 02033be08c [pool] Visual Studios disallows empty structs 2017-08-28 17:19:01 -07:00
Nick Terrell 26dc040a7b [pool] Accept custom allocators 2017-08-24 17:01:41 -07:00
Nick Terrell 89dc856cae [pool] Fix formatting 2017-08-24 16:48:32 -07:00
Yann Collet 7db552676e reduced pool queue to 0 to save memory
fixed : pool performance when jobs are fires fast and queueSize==0
2017-08-19 15:07:54 -07:00
Yann Collet 32fb407c9d updated a bunch of headers
for the new license
2017-08-18 16:52:05 -07:00
Stella Lau 73ba58955f Signal after finishing job when queueSize=0 2017-08-01 20:12:06 -07:00
Stella Lau 1d76da1d87 Replace marker with queueEmpty variable and update pool.h comment 2017-08-01 12:30:16 -07:00
Stella Lau 5adceeed01 Allow queueSize=0 in pool.c and update poolTests 2017-07-31 10:10:16 -07:00
Yann Collet a90b16e150 Visual blind fix 2 2017-07-20 15:57:55 -07:00
Yann Collet 16261e6951 buffer pool can be invoked from multiple threads 2017-07-11 14:14:07 -07:00
Yann Collet 6056e4c3eb added POOL_sizeof() for single-thread 2017-06-02 11:36:47 -07:00
Yann Collet c4a5a21c5c created ZSTDMT_sizeof_CCtx() and POOL_sizeof()
required by ZSTD_sizeofCCtx() while adding a ZSTDMT_CCtx*
2017-06-01 17:56:14 -07:00
Nick Terrell e628eaf87a Fix pool.c threading.h import 2017-01-26 15:29:10 -08:00
cyan4973 2e3b659ae1 fixed minor warnings (Visual, conversion, doxygen) 2017-01-20 14:43:09 -08:00
Yann Collet 0f984d94c4 changed MT enabling macro to ZSTD_MULTITHREAD 2017-01-19 14:05:07 -08:00