ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,138 Commits 17 Branches 77 Tags
d99ed4740df23bd634cde0ee89207a9dda2c9130
Commit Graph
6138 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Yann ColletandGitHub d99ed4740d Merge pull request #1454 from lzutao/readme 
README: Explain a little bit about Meson build
 2018-12-19 10:43:27 -08:00
Lzu Tao c4b9b8aa52 README: Explain a bit more about Meson build [skip ci] 
- Increase one heading level. Now headings is bigger and prettier.
 2018-12-19 12:52:30 +07:00
Nick TerrellandGitHub 3a4634f2af Merge pull request #1459 from terrelln/destroy 
[zstdcli] Refuse to overwrite input file
 2018-12-18 17:03:54 -08:00
Nick Terrell 7abd6acebf Also compare device thanks to @felixhandte 2018-12-18 15:40:27 -08:00
Nick Terrell cd2c8defad [zstdcli] Refuse to overwrite input file 
Compare the input and output files by their inode number and
refuse to open the output file if the input file is the same.

This doesn't work when (de)compressing multiple files to a single
file, but that is a very uncommon use case, mostly used for
benchmarking by me.

Fixes #1422.
 2018-12-18 15:29:54 -08:00
Yann ColletandGitHub d0e15f8d32 Merge pull request #1458 from terrelln/estimate 
[libzstd] Fix estimate with negative levels
 2018-12-18 15:12:21 -08:00
Yann ColletandGitHub 04baecaeed Merge pull request #1457 from facebook/btultra2.1 
btultra2 and very small input
 2018-12-18 14:46:55 -08:00
Nick Terrell d7def456d8 [libzstd] Fix estimate with negative levels 
* Fix `ZSTD_estimateCCtxSize()` with negative levels.
* Fix `ZSTD_estimateCStreamSize()` with negative levels.
* Add a unit test to test for this error.
 2018-12-18 14:24:49 -08:00
Yann Collet ef984e7307 fix debug levels 
as reported by @terrelln.
2 is reserved for temporary usage only.
 2018-12-18 13:40:07 -08:00
Yann Collet 635783da12 btultra2 and very small srcSize 
When srcSize is small,
the nb of symbols produced is likely too small to warrant dedicated probability tables.
In which case, predefined distribution tables will be used instead.

There is a cheap algorithm in btultra initialization :
it presumes default distribution will be used if srcSize <= 1024.

btultra2 now uses the same threshold to shut down probability estimation,
since measured frequencies won't be used at entropy stage,
and therefore relying on them to determine sequence cost is misleading,
resulting in worse compression ratios.

This fixes btultra2 performance issue on very small input.

Note that, a proper way should be
to determine which symbol is going to use predefined probaility
and which symbol is going to use dynamic ones.
But the current algorithm is unable to make a "per-symbol" decision.
So this will require significant modifications.
 2018-12-18 12:32:58 -08:00
Yann ColletandGitHub 517d8c984c Merge pull request #1449 from facebook/ovlog_def 
overlapLog default values
 2018-12-18 09:45:53 -08:00
Yann ColletandGitHub 345af8abc7 Merge pull request #1456 from yijinfb/example-ZSTD_compressCCtx 
Add multiple_simple_compression.c (demonstrates the use of ZSTD_compressCCtx(...)) to examples directory.
 2018-12-18 09:24:18 -08:00
Yi Jin bc4dc606de break loadFile_orDie() into 2: loadFile_orDie() loads file into a pre-allocated memory buffer, mallocAndLoadFile_orDie() allocates memory first, then calls loadFile_orDie() 2018-12-17 16:54:55 -08:00
Yann Collet 373ff8b983 play around with rescale weights 2018-12-17 15:48:34 -08:00
Yi Jin 452689678b bug fix in loadFile_orDie(): exit with error code if filesize is bigger than provided buffer 2018-12-16 22:44:27 -08:00
Yi Jin a951ee9fe1 comment out printf() to make output terse 2018-12-16 21:27:52 -08:00
Yi Jin 595401e4c7 edit README.md and remove simple_compressionCCtx.c 2018-12-16 21:09:21 -08:00
Yi Jin ea4859ec71 add multiple_simple_compression.c 2018-12-16 20:45:43 -08:00
Yi Jin cce0842c60 Factor out allocMemory_orDIe(...) in simple_compressionCCtx.c, and amend Makefile to account for dependency on utils.h 2018-12-16 15:36:28 -08:00
Yi Jin 0d14153cfb formatting 2018-12-14 22:10:31 -08:00
Yi Jin 04d06ad885 refactor utils.h and implement simple_compressionCCtx.c 2018-12-14 18:12:05 -08:00
Yann ColletandGitHub 5bdbd997ae Merge pull request #1452 from lzutao/meson_getversion 
meson: Remove unused sys import
 2018-12-14 16:50:41 -08:00
Nick TerrellandGitHub 7f6d11e1e2 Merge pull request #1450 from terrelln/no-progress 
[zstdcli] Add --no-progress flag
 2018-12-14 11:50:59 -08:00
Nick Terrell bdfcaecc0a [zstdcli] Add --no-progress flag 
The `--no-progress` flag disables zstd's progress bars, but leaves
the summary.

I've added simple tests to `playTests.sh` to make sure the parsing
works.
 2018-12-14 11:50:25 -08:00
Yann ColletandGitHub d4698424ce Merge pull request #1447 from lzutao/meson_symlink_soversion 
meson: More accurate Windows build support
 2018-12-14 09:18:37 -08:00
Yi Jin 193fbd30f2 WIP 2018-12-13 22:51:35 -08:00
Lzu Tao 066cfc069b meson: Remove unused sys import 2018-12-14 11:03:04 +07:00
Yann Collet 96adc846c5 fixed tests 
with correct pointer type
 2018-12-13 16:50:19 -08:00
Yann Collet 8be145a8c1 fixed default job size 2018-12-13 16:38:08 -08:00
Nick TerrellandGitHub 75fa3f2eb7 Merge pull request #1446 from terrelln/overflow 
[libzstd] Fix infinite loop in decompression
 2018-12-13 16:21:15 -08:00
Yann Collet 62180b27d5 zstdmt parameter getter/setter use int 2018-12-13 15:47:34 -08:00
Yi Jin 7f975c0c28 create simple_compressionCCtx.c out of simple_compression.c, to be worked on 2018-12-13 15:42:55 -08:00
Nick Terrell aaea4ef924 [libzstd] Fix infinite loop in decompression 
When we switched `ZSTD_SKIPPABLEHEADERSIZE` to a macro, the places where we do:

    MEM_readLE32(ptr) + ZSTD_SKIPPABLEHEADERSIZE

can now overflow `(unsigned)-8` to `0` and we infinite loop. We now check
the frame size and reject sizes that overflow a U32.

Note that this bug never made it into a release, and was only in the dev branch
for a few days.

Credit to OSS-Fuzz
 2018-12-13 15:13:19 -08:00
Yann Collet 34f01e600f fixed multiple conversions 
from 64-bit to 32-bit
 2018-12-13 14:02:22 -08:00
Yann ColletandGitHub 4acf139e9f Merge pull request #1448 from lzutao/travis_reformat 
travis: Use script instead of Cmd env
 2018-12-13 12:04:10 -08:00
Lzu Tao b3be899469 travis: Use script instead of Cmd env 2018-12-14 02:06:14 +07:00
Lzu Tao 3ee5504fb2 Simplify logic by setting default value for MESON_INSTALL_DESTDIR_PREFIX 2018-12-13 18:07:01 +07:00
Lzu Tao ce22f76668 meson: Update man1 extension on meson 0.49.0 2018-12-13 14:58:17 +07:00
Lzu Tao abfde03cb5 meson: Update meson symlink script usage 2018-12-13 14:58:17 +07:00
Lzu Tao fa2fc274fd meson: Correct support for building on Windows 
Let soversion base on version if not set. For example, if version is 3.6.0
and soversion is not defined, it is set to 3.
 2018-12-13 14:58:17 +07:00
Lzu Tao 67babb6d23 Replace many os.path methods with pathlib one's 
Use MESON_INSTALL_DESTDIR_PREFIX variable instead of DESTDIR.
 2018-12-13 14:58:17 +07:00
Yann Collet 1993f5d412 fixed ovlog tests 
and updated man page
 2018-12-12 21:09:14 -08:00
Yann ColletandGitHub fbcae274a4 Merge pull request #1444 from facebook/btultra2 
btultra2
 2018-12-12 21:09:01 -08:00
Yann Collet f2f86d369b Merge branch 'btultra2' into ovlog_def 2018-12-12 20:58:14 -08:00
Yann Collet 9a92ed401d updated compression results.csv 
and fixed nit
 2018-12-12 20:30:09 -08:00
Yann Collet 9792acda3b Merge branch 'dev' into btultra2 2018-12-12 20:18:27 -08:00
Nick TerrellandGitHub 54ca4b3c5d Merge pull request #1445 from terrelln/regression 
[regression] add more methods
 2018-12-12 11:16:54 -08:00
Yann Collet c313a85ee2 removed exception code for overlapLog level 22 
CLI used to set overlapLog at value 9 when level == 22.
This is no longer necessary (handled internally within library)
 2018-12-11 18:13:06 -08:00
Yann Collet 7bb8dfc62f new overlapLog default values 
varies between 6 and 9, depending on strategy
 2018-12-11 18:10:29 -08:00
Yann Collet eee789b7ea continued: changed to overlapLog 
in deeper code layer.
for consistency.
 2018-12-11 17:41:42 -08:00