ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,595 Commits 17 Branches 77 Tags
v1.5.6
Commit Graph
1946 Commits
Author SHA1 Message Date
Yann Collet 7f37b8a547 accelerate versionsCompatibilityTest 
by allowing parallel build of units,
and reducing optimization levels.

Parallel build is only effective on "recent" versions of `zstd`,
as previously, the list of units was passed as a list of source files,
which is something neither `make` nor `gcc` can parallelize.
So its impact is mildly effective (-20%).

Reducing optimization level to `-O1` makes compilation much faster.
It also makes runtime slower,
but in this test, compilation time dominates run time.
The savings are very significant (-50%).

On my test system, it reduces the length of this test from 13mn to 5mn.
 2021-08-29 14:48:11 -07:00
Clément Chigot 6ef6cd7999 test: avoid /dev/full on AIX 2021-08-13 10:25:50 +02:00
Clément Chigot 399849e236 Makefile: add AIX support 
For lib, AIX linker doesn't allow --soname.
 2021-08-13 10:25:14 +02:00
senandGitHub 6a258043f5 Merge pull request #2692 from senhuang42/rebalance_clevel 
[RFC] Rebalance compression levels
 2021-08-06 12:51:31 -04:00
Sen Huang 539b3aab9b Optimize 32-bit VecMask_next() 2021-08-04 17:14:58 -04:00
Felix HandteandGitHub ae131282f5 Merge pull request #2742 from felixhandte/set-mtime 
Set mtime on Output Files
 2021-08-04 16:44:46 -04:00
senhuang42 e411040ea1 Add 64 row entry support for lazy 2021-08-04 16:19:12 -04:00
W. Felix Handte a8f4612eab Remove sleep()s in Test; Replace with Artificial mtime 
This behavior of `touch` is standardized in POSIX, so it should be available.
 2021-08-04 15:56:46 -04:00
senhuang42 aa1957477b Improve Huffman sorting algorithm 2021-08-04 12:43:34 -04:00
W. Felix Handte ead41bcb4e Add Test to Verify mtime is Copied to Destination 2021-08-03 17:22:58 -04:00
Nick Terrell d8a0797268 [fuzz] Add Huffman round trip fuzzer 
* Add a Huffman round trip fuzzer
* Fix two minor bugs in Huffman that aren't exposed in zstd
  - Incorrect weight comparison (weights are allowed to be equal to
    table log).
  - HUF_compress1X_usingCTable_internal() can return compressed
    size >= source size, so the assert that `cSize <= 65535` isn't
    correct, and it needs to be checked instead.
 2021-08-03 08:10:06 -07:00
Nick Terrell 46f2710562 [HUF] Improve Huffman encoding speed 
Improve Huffman encoding speed by 20% for gcc and 10% for clang.

| Compiler |     Benchmark     | Config  |   Dataset   | Ratio | Speed MB/s (dev) | Speed MB/s (huf-cspeed) | Speed MB/s (huf-cspeed - dev) |
|----------|-------------------|---------|-------------|-------|------------------|-------------------------|-------------------------------|
| gcc      | compress          | level_1 | enwik7      | 2.43  | 253.70           | 258.72                  | 2.0%                          |
| gcc      | compress          | level_1 | silesia     | 2.88  | 341.90           | 348.15                  | 1.8%                          |
| gcc      | compress_literals | level_1 | enwik7      | 1.49  | 761.83           | 912.76                  | 19.8%                         |
| gcc      | compress_literals | level_1 | silesia     | 1.28  | 754.83           | 902.37                  | 19.5%                         |
| gcc      | compress_literals | level_7 | enwik7      | 1.29  | 502.81           | 552.79                  | 9.9%                          |
| gcc      | compress_literals | level_7 | silesia     | 1.11  | 675.97           | 776.44                  | 14.9%                         |
| clang    | compress          | level_1 | enwik7      | 2.43  | 277.54           | 280.98                  | 1.2%                          |
| clang    | compress          | level_1 | silesia     | 2.88  | 369.98           | 375.46                  | 1.5%                          |
| clang    | compress_literals | level_1 | enwik7      | 1.49  | 828.83           | 918.41                  | 10.8%                         |
| clang    | compress_literals | level_1 | silesia     | 1.28  | 815.81           | 905.41                  | 11.0%                         |
| clang    | compress_literals | level_7 | enwik7      | 1.29  | 533.13           | 553.30                  | 3.8%                          |
| clang    | compress_literals | level_7 | silesia     | 1.11  | 714.52           | 775.38                  | 8.5%                          |
 2021-07-27 15:10:35 -07:00
senandGitHub d90bc0e0b6 Merge pull request #2720 from senhuang42/remove_folder 
Remove folder when done with test
 2021-06-30 14:37:21 -04:00
senhuang42 f5f6cc2e48 Remove folder when done with test 2021-06-30 13:09:00 -04:00
Binh Vo 6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
Binh Vo 9d9f7680f8 Add API for fetching skippable frame content 2021-06-14 16:01:28 -04:00
binhdvoandGitHub 89127e5ee2 Merge pull request #2705 from binhdvo/bootcamp 
Add support for negative values in advanced flags
 2021-06-11 14:08:23 -04:00
Binh Vo 6fad35c6a1 Add support for negative levels in --adapt=min and --adapt=max" 2021-06-11 12:13:09 -04:00
W. Felix Handte 94cf57bb13 Update Tests to Reflect New Formatting 2021-06-10 13:14:18 -04:00
binhdvoandGitHub 325952f878 Revert "Add support for --long-param flag, fix #2104" 2021-06-09 15:35:43 -04:00
binhdvoandGitHub 78e16b15f1 Merge pull request #2703 from binhdvo/bootcamp 
Add support for --long-param flag, fix #2104
 2021-06-09 15:03:38 -04:00
Binh Vo 6583fa3f0a Add support for --long-param flag 2021-06-09 14:07:52 -04:00
Yann ColletandGitHub 05d70903a6 Merge pull request #2698 from binhdvo/bootcamp 
Fix --progress flag to properly control progress display and default …
 2021-06-09 09:56:01 -07:00
Binh Vo d2f31b6627 Fix --progress flag to properly control progress display and default progress display on when using -v 2021-06-08 17:24:38 -04:00
Sen Huang 2ff5c7b59f Add no intrinsics fuzztest, rowhash compression size test, and S390X to travis 2021-06-07 00:54:53 -07:00
senhuang42 939276cd0c Add ldm and block splitter auto-enable to old api 2021-05-24 13:09:32 -04:00
Nick Terrell c468e1b9cb [test][regression] Update results.csv 
Changing the repcode search slightly, and changing when zstd is in
ext-dict vs. prefix mode in edge cases, slightly changes the compressed
results.
 2021-05-18 19:46:37 -07:00
Yann Collet 61afa154cd improve tar compatibility 
This patch is supposed to improve compatibility with less featured tar variants
"when the tar program used does not support historical options (without hyphen) nor the '-z' option."

Patch proposed by Antonio Diaz Diaz
 2021-05-15 23:09:42 -07:00
Nick Terrell 725c5e4e38 [fuzz] Add determinism fuzzing to simple & dictionary round trip 
Compress the input twice in the `simple_round_trip` and
`dictionary_round_trip` fuzzers with exactly the same parameters, but
reusing the context. Then ensure that the compressed output is
identical.
 2021-05-13 17:05:59 -07:00
senandGitHub 06718087f8 Remove deprecate flag for vcx (#2647) 2021-05-13 19:33:48 -04:00
senandGitHub 40def70387 Add source level deprecation warning disabling to certain tests/utils (#2645) 2021-05-13 14:41:21 -04:00
senandGitHub b35c250bf3 Remove const data members in threadpooltest payload (#2639) 2021-05-12 12:56:57 -04:00
senandGitHub d8d6e48a0a Add threadPool unit tests to fuzzer.c (#2604) 2021-05-07 11:13:44 -04:00
senandGitHub 698f261b35 [1.5.0] Deprecate some functions (#2582) 
* Add deprecated macro to zstd.h, mark certain functions as deprecated

* Remove ZSTD_compress.c dependencies on deprecated functions
 2021-05-06 17:59:32 -04:00
Nick TerrellandGitHub 207e33bb61 Merge pull request #2616 from terrelln/deterministic-dict 
[lib] Add ZSTD_c_deterministicRefPrefix
 2021-05-06 11:09:22 -07:00
Nick TerrellandGitHub fc8330b885 Merge pull request #2621 from terrelln/regression-test 
[test][regression] Update results.csv
 2021-05-06 10:32:07 -07:00
senandGitHub d6be7659b0 Add seekable roundtrip fuzzer (#2617) 2021-05-06 10:08:21 -04:00
Nick TerrellandGitHub d2925de98a Merge pull request #2615 from terrelln/stack-space 
[lib] Move some ZSTD_CCtx_params off the stack
 2021-05-05 19:43:39 -07:00
Nick Terrell ce615d7fba [test][regression] Update results.csv 
The LDM change in PR #2602 changed the algorithm slightly.
The compressed size is generally positive, and when it is worse,
it is only a few bytes.
 2021-05-05 19:00:36 -07:00
Nick Terrell 172b4b6ac4 [lib] Add ZSTD_c_deterministicRefPrefix 
This flag forces zstd to always load the prefix in ext-dict mode, even
if it happens to be contiguous, to force determinism. It also applies to
dictionaries that are re-processed.

A determinism test case is also added, which fails without
`ZSTD_c_deterministicRefPrefix` and passes with it set.

Question: Should this be the default behavior? It isn't in this PR.
 2021-05-05 18:49:56 -07:00
Nick Terrell eb7e74ccb7 [tests] Set DEBUGLEVEL=2 by default 
This allows us to quickly check for compile errors in debug log
messages, which are compiled out when `DEBUGLEVEL < 2`.
 2021-05-05 13:29:06 -07:00
W. Felix Handte bea1b2ba70 rm -f in playTests.sh 2021-05-05 13:10:34 -04:00
W. Felix Handte 018ed6552a Attempt to Fix stat Format for BSDs 2021-05-05 13:10:34 -04:00
W. Felix Handte 1fb10ba831 Don't Block Removing File on Being Able to Read It 
`open()`'s mode bits are only applied to files that are created by the call.
If the output file already exists, but is not readable, the `fopen()` would
fail, preventing us from removing it, which would mean that the file would
not end up with the correct permission bits.

It's not clear to me why the `fopen()` is there at all. `UTIL_isRegularFile()`
should be sufficient, AFAICT.
 2021-05-05 13:10:34 -04:00
W. Felix Handte 4e10ff15f5 Add Tests Checking File Permissions of Created Files 2021-05-05 13:10:34 -04:00
Felix HandteandGitHub 2d10544b84 Merge pull request #2613 from felixhandte/allow-block-device 
Allow Reading from Block Devices with `--force`
 2021-05-05 13:06:32 -04:00
Yann ColletandGitHub c077f257b4 Merge pull request #2611 from facebook/smallerJobs 
allow jobSize to be as low as 512 KB
 2021-05-05 00:03:29 -07:00
Nick Terrell 0b88c2582c [test] Add large dict/data --patch-from test 
Dictionary size must be > `ZSTD_CHUNKSIZE_MAX`.
 2021-05-04 17:31:32 -07:00
W. Felix Handte e58e9c7928 Add Test Case (Behind Flag); Run in GitHub Action 2021-05-04 18:43:39 -04:00
Yann Collet 8f86c29c06 allow jobSize to be as low as 512 KB 
previous lower limit was 1 MB.

Note : by default, the lowest job size is 2 MB, achieved at level 1.
Even lower job sizes can be achieved by manipulating this value directly,
or manually modifying window sizes to lower amounts.

Updated unit test to ensure that this new limit works fine
(test would fail with previous 1 MB limit).
 2021-05-04 11:02:55 -07:00