ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,145 Commits 17 Branches 77 Tags
04734ee84a10466b4ada0edaa086f7a0a948c5fe
Commit Graph
2003 Commits
Author SHA1 Message Date
Yann Collet 23dd28df67 minor improvements to benchmark display 2021-10-26 23:23:30 -07:00
stanjo74andGitHub 52598d54e9 Limit train samples (#2809) 
* Limit training samples size to 2GB

* simplified DISPLAYLEVEL() macro to use global vqriable instead of local.

* refactored training samples loading

* fixed compiler warning

* addressed comments from the pull request

* addressed @terrelln comments

* missed some fixes

* fixed type mismatch

* Fixed bug passing estimated number of samples rather insted of the loaded number of samples.
Changed unit conversion not to use bit-shifts.

* fixed a declaration after code

* fixed type conversion compile errors

* fixed more type castting

* fixed more type mismatching

* changed sizes type to size_t

* move type casting

* more type cast fixes
 2021-10-04 17:47:52 -07:00
Yann ColletandGitHub 7868f38019 Merge pull request #2747 from Helflym/dev 
Add AIX support in Makefile
 2021-10-01 08:13:39 -07:00
Yann Collet 8150891939 regenerated zstdless.1 2021-09-22 14:48:51 -07:00
Yann Collet 3addf2f277 updated zstdgrep man page 2021-09-22 14:30:59 -07:00
Yann Collet 70b36c2308 update zstdgrep doc 
to mention ripgrep alternative which transparently supports zstd-compressed files.
 2021-09-22 14:30:06 -07:00
Yann Collet 999f8778af updated man pages 
using ronn-ng
 2021-09-22 14:18:24 -07:00
Yann Collet 2832bbbbdc emphasize usage of -r in documentation 
notably as a way to overcome shell expansion limitations,
notably in a scenario of dictionary training.
 2021-09-22 14:00:20 -07:00
senhuang42 b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
Nick Terrell 8bf699aa59 [build] Add support for ASM files in Make + CMake 
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
  Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
  same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
  `make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.

The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.

The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
 2021-09-17 14:13:53 -07:00
Yann Collet f58e63bee7 Merge branch 'dev' into opt_investigation 2021-09-12 01:42:49 -07:00
Yann Collet 640c5b1f77 fix automated_benchmarking 
make it able to process text output sent into either stdout or stderr
 2021-09-12 01:36:18 -07:00
Felix HandteandGitHub d68aa19a2f Merge pull request #2749 from felixhandte/zstd-fast-pipelined 
Pipelined Implementation of ZSTD_fast (~+5% Speed)
 2021-09-09 17:05:30 -04:00
senhuang42 30fe49af4e Fix patch-from help msg typo 2021-09-07 10:08:35 -04:00
Yann Collet 08ceda3dfc new statistics update policy 
small general compression ratio improvement for btopt+ strategies/
 2021-09-04 00:52:44 -07:00
Yann Collet eab692211e removed pretty-print of sizes in benchmark 
This is less appropriate for this mode :
benchmark is about accuracy,
it's important to read the exact values.
 2021-09-03 12:51:02 -07:00
W. Felix Handte ab8aa49b8d Fix Benchmark Corruption Display 2021-09-01 14:15:03 -04:00
Yann Collet f21977c5e6 fix playTests.sh when EXE_PREFIX not null 2021-08-29 17:20:12 -07:00
senhuang42 dce48f53df Fix benchzstd error message 2021-08-23 19:10:16 -04:00
Clément Chigot 399849e236 Makefile: add AIX support 
For lib, AIX linker doesn't allow --soname.
 2021-08-13 10:25:14 +02:00
W. Felix Handte 9cd6c1ff4d Update mtime and atime for Written Files 2021-08-04 14:49:56 -04:00
W. Felix Handte a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
makise-homura a5f518ae27 Change zstdcli's main() declaration due to -Wmain on some compilers 2021-07-14 19:55:47 +03:00
makise-homura d4ad02c721 Add support for MCST LCC compiler 2021-07-10 03:57:06 +03:00
Binh Vo 6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
binhdvoandGitHub 89127e5ee2 Merge pull request #2705 from binhdvo/bootcamp 
Add support for negative values in advanced flags
 2021-06-11 14:08:23 -04:00
Binh Vo 6fad35c6a1 Add support for negative levels in --adapt=min and --adapt=max" 2021-06-11 12:13:09 -04:00
W. Felix Handte 8c00807bbc Whitespace Fixes to Improve Cross-Line Alignment 2021-06-10 13:28:38 -04:00
W. Felix Handte 87e94e3e39 Convert Other Size Displays to Use Human-Readable Formatting 2021-06-10 12:57:46 -04:00
W. Felix Handte 2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB") 
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
 2021-06-10 12:53:07 -04:00
W. Felix Handte 9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte bc46b6efe4 Apply to Other Print Statement as Well 2021-06-10 12:53:07 -04:00
W. Felix Handte 7e0058848c Fix Whitespace 2021-06-10 12:53:07 -04:00
W. Felix Handte 93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte 464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte 9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting 
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 8e0a9695d7 Attempt to fix a failing test with help from @aqrit 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 1eb852854b Some fixes to address things @felixhandte found 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 376a2730a8 Try enabling the BIG strings now the unsigned long long is in effect 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 20b9b00b41 Try unsigned long long 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 64385ef7cb Update humanSize() to skip the big numbers (it requires 64 bit) 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 1ef6f3d079 Use unsigned long instead to help with some tests 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte e5fc830795 human_size() should use size_t 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 35576e63ce Convert tabs to spaces 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 77001f00fb Use human_size() on the "multiple files compressed" output also 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 894698d3b6 Use human_size() in the benchmark output also 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 4e0d9f1cc8 Move the variable declarations to the top 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte eefdbcd93a Make the variable types match 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte b6b23dfe64 Convert names to CamelCase 2021-06-10 12:53:07 -04:00