ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,521 Commits 17 Branches 77 Tags
6a8fba9e5fb8be200df3b251a827be7115b53edb
Commit Graph
9521 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Nick TerrellandGitHub 6a8fba9e5f Merge pull request #3092 from terrelln/2022-03-09-decoder-errata-doc 
[doc] Add decompressor errata document
 2022-03-09 15:10:29 -08:00
Nick Terrell 696fa2524a [doc] Add decompressor errata document 
Add a document that lists the known bugs in the decoder where valid
frames are rejected, along with the version that they were fixed.
 2022-03-09 14:40:41 -08:00
Felix HandteandGitHub 8814aa5bfa Merge pull request #3059 from dirkmueller/gzip_fast_best 
Implement more gzip compatibility (#3037)
 2022-03-08 15:32:06 -05:00
Yann ColletandGitHub db104f6e83 Merge pull request #3079 from niamster/cmake-compat-5 
build:cmake: enable ZSTD legacy support by default
 2022-03-08 00:21:23 -08:00
Yann ColletandGitHub 930b5cecaa Merge pull request #3060 from rex4539/typos 
Fix typos
 2022-03-05 14:21:58 -08:00
Dimitris Apostolou cf1894b324 Fix typos 2022-03-05 23:47:25 +02:00
Xi RuoyaoandGitHub d109cef201 fix the assertion in readLinesFromFile (#3084) 
* fix the assertion in readLinesFromFile

When the file is not terminated by endline, readLineFromFile will append
a '\0' for the last line.  In this case pos + lineLength == dstCapacity.

* test: don't print very long text garbage
 2022-03-04 14:56:44 -05:00
Yann ColletandGitHub e8448a3d61 Merge pull request #3080 from TocarIP/tokarip/static_bmi2 
Enable STATIC_BMI2 for gcc/clang
 2022-03-03 12:49:40 -08:00
Ilya Tokar 7c3d1cb3ab Enable STATIC_BMI2 for gcc/clang 
Some usage (e.g. BIT_getLowerBit) uses it without checking for MSVC,
so enabling for clang gives a small performance boost.
 2022-03-03 15:03:54 -05:00
Elliot GorokhovskyandGitHub 0c386afbfd Fix CI failures by adding apt-get update to Github Actions (#3082) 
* Fix CI failures by adding apt-get update to Makefile

* Fix travis failure caused by apt-get update

* Move apt-get update from Makefile to Github Actions .yml

* Revert .travis.yml change

* Fix typo
 2022-03-03 14:52:06 -05:00
Nick TerrellandGitHub 41665679bf Merge pull request #3081 from terrelln/dict-load-zero 
[programs] Fix infinite loop when empty input is passed to trainer
 2022-03-02 12:58:51 -08:00
Nick Terrell da737c7ab8 [programs] Fix infinite loop when empty input is passed to trainer 
When an empty input file was passed to the dictionary trainer, it would infinite loop.
The added test case exposes the bug, and is fixed with this PR.
 2022-03-02 11:05:26 -08:00
Dmytro Milinevskyi 03bba1b0bf build:cmake: enable ZSTD legacy support by default 2022-03-01 18:29:47 +01:00
Elliot GorokhovskyandGitHub 87406b5f3b Merge pull request #3075 from TocarIP/tokarip/bzhi 
Use helper function for bit manipulations.
 2022-02-24 16:52:48 -05:00
Ilya Tokar 0178c12dd9 Use helper function for bit manipulations. 
We already have BIT_getLowerBits, so use it. Benefits are 2fold:
1) Somewhat cleaner code
2) We are now using bzhi instructions, when available. Performance
delta is too small for microbenchmarks, but avoiding load still helps
larger applications, by reducing data cache pressure.
 2022-02-23 17:59:56 -05:00
Elliot GorokhovskyandGitHub 621d798988 Merge pull request #3045 from embg/asm_refactor 
Refactor bitwise intrinsics
 2022-02-17 13:08:08 -05:00
Elliot Gorokhovsky 71d9dab76f Replace XOR with subtraction for readability 2022-02-16 16:49:42 -05:00
Elliot Gorokhovsky 856c7dc51d Fix fuzzer.c nits and replace CLZ fallback 2022-02-16 11:40:05 -05:00
Elliot Gorokhovsky 00f2acba36 Add back check to prevent Win32 static analysis issues 2022-02-15 11:41:09 -05:00
Elliot Gorokhovsky 7c674a0919 Add tests for bitwise intrinsics 2022-02-14 18:36:33 -05:00
Elliot Gorokhovsky 6994a9f99c bits.h refactor and bugfix 2022-02-14 16:59:55 -05:00
Elliot Gorokhovsky 529cd7b821 Fix nits 2022-02-14 14:24:50 -05:00
Elliot Gorokhovsky 796182652d Pull out software fallbacks 2022-02-14 11:16:03 -05:00
Elliot Gorokhovsky db2f4a6532 Move bitwise builtins into bits.h 2022-02-14 11:16:03 -05:00
Elliot GorokhovskyandGitHub 970460f67d Merge pull request #3063 from embg/largeNbDicts 
Bugfix and new features for largeNbDicts benchmark
 2022-02-11 13:55:35 -05:00
Elliot Gorokhovsky 762898f5e4 Bugfix and new features for largeNbDicts benchmark 2022-02-11 13:15:16 -05:00
Yann ColletandGitHub 3202c75148 Merge pull request #3054 from ooosssososos/patch-1 
[trace] Add aarch64 to supported architectures for zstd_trace
 2022-02-11 08:13:09 -08:00
Dirk Müller e653e97f77 Implement more gzip compatibility (#3037) 
-n --no-name is the current behavior already, so we can implement
this as a noop.

--best is an alias for -9 in gzip

add basic cli tests.
 2022-02-09 23:09:18 +01:00
Elliot GorokhovskyandGitHub caf2fa170b Merge pull request #3061 from embg/debug_ci_windows 
Debug Windows CI failures
 2022-02-09 16:29:56 -05:00
Elliot Gorokhovsky 9caabc01c4 Replace "windows-latest" with "windows-2019" in CI workflows 2022-02-09 15:45:43 -05:00
binhdvoGitHubBinh Vo
 		936ae8a3a0 Move zstdgrep and zstdless tests to cli-tests (#3057) 
* Move zstdgrep and zstdless tests to cli-tests

Co-authored-by: Binh Vo <binhvo@fb.com>
 2022-02-09 11:21:33 -05:00
Nick TerrellandGitHub 7e364e8828 Merge pull request #3055 from terrelln/cli-test-symlinks 
[cli-tests] Fix zstd symlinks
 2022-02-07 16:16:50 -08:00
Nick Terrell 169f8c11ff [cli-tests] Fix zstd symlinks 
The zstd symlinks, notably `zstdcat`, weren't working as expected
because only the `tests/cli-tests/bin/zstd` wrapper was symlinked. We
still invoked `zstd` with the name `zstd`. The fix is to create a
directory of zstd symlinks in `tests/cli-tests/bin/symlinks` for each
name that zstd recognizes. And when `tets/cli-tests/bin/zstd` is
invoked, it selects the correct symlink to call.

See the test `zstd-cli/zstdcat.sh` for an example of how it would work.
 2022-02-07 15:20:42 -08:00
Oscar ShiandGitHub fede1d3abe [trace] Add aarch64 to supported architectures for zstd_trace 
Arm Toolchain should support weak symbols
 2022-02-07 14:41:07 -08:00
Nick TerrellandGitHub b848c167ab Merge pull request #3050 from shadchin/patch-1 
Select legacy level for cmake
 2022-02-04 11:23:19 -08:00
Yann ColletandGitHub fcef19966c Merge pull request #3042 from u1f35c/fix2968 
Fix required decompression memory usage reported by -vv + --long
 2022-02-04 07:50:58 -08:00
Yann ColletandGitHub be0a4b4df6 Merge pull request #3046 from rex4539/typos 
Fix typos
 2022-02-04 07:50:25 -08:00
Alexander ShadchinandGitHub 317bd108fe Select legacy level for cmake 2022-02-04 14:24:58 +03:00
Yonatan KomornikandGitHub 4bba97b4cb Macos playtest envvars fix (#3035) 
* playtests.sh: fix for a bug in macos' /bin/sh that persists temporary env vars when introduced before function calls
* cli-tests/run.py: Do not use existing ZSTD* envvars
 2022-02-03 18:42:20 -08:00
binhdvoandGitHub b9566fc558 Add rails for huffman table log calculation (#3047) 2022-02-02 15:12:48 -05:00
Dimitris Apostolou 4c4d403ecb Fix typos 2022-02-02 19:32:31 +02:00
Jonathan McDowell 470eb8330a Fix required decompression memory usage reported by -vv + --long 
The use of --long alters the window size internally in the underlying
library (lib/compress/zstd_compress.c:ZSTD_getCParamsFromCCtxParams),
which changes the memory required for decompression. This means that the
reported requirement from the zstd binary when -vv is specified is
incorrect.

A full fix for this would be to add an API call to be able to retrieve
the required decompression memory from the library, but as a
lighterweight fix we can just take account of the fact we've enabled
long mode and update our verbose output appropriately.

Fixes #2968
 2022-02-02 12:39:27 +00:00
Yann ColletandGitHub 529a5879bf Merge pull request #3040 from facebook/fix44239 
fix 44239
 2022-02-01 13:33:01 -08:00
Yann Collet cad9f8d5f9 fix 44239 
credit to oss-fuzz

This issue could happen when using the new Sequence Compression API in Explicit Delimiter Mode
with a too small dstCapacity.
In which case, there was one place where the buffer size wasn't checked.
 2022-02-01 10:49:38 -08:00
Yann ColletandGitHub 26c36fafb9 Merge pull request #3036 from facebook/fix44168 
fixed bug 44168
 2022-02-01 10:10:32 -08:00
Yann Collet cdee6a7dbd Merge branch 'dev' into fix44168 2022-01-31 17:31:55 -08:00
Yonatan KomornikandGitHub 4b24ebdcf3 Travis CI: fix by installing pip compatible with python 3.6 (#3041) 
Pip install script no longer supports python3.6 by default, switched to a script that does.
 2022-01-31 16:49:49 -08:00
Yann Collet 9a758ce520 update sequence_compression_api fuzzer test 
to check for under-sized dstCapacity.
 2022-01-31 16:17:11 -08:00
Yonatan KomornikandGitHub cc0657f27d AsyncIO compression part 2 - added async read and asyncio to compression code (#3022) 
* Compression asyncio:
- Added asyncio functionality for compression flow
- Added ReadPool for async reads, implemented in both comp and decomp flows
 2022-01-31 15:43:41 -08:00
Nick TerrellandGitHub 0b70da6277 Merge pull request #3020 from terrelln/cli-tests 
Add new CLI testing platform
 2022-01-31 10:02:27 -08:00