ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,485 Commits 17 Branches 77 Tags
7b94de78ff2c3634cd7ef32f0e16f4722c03578f
Commit Graph
5043 Commits
Author SHA1 Message Date
Yann Collet 2366a87ddc fix minor visual conversion warning 2024-10-24 13:38:12 -07:00
Yann Collet 326c45bb8e complete sample11 with reduced fingerprint size 2024-10-24 13:17:56 -07:00
Yann Collet ca6e55cbf5 reduce splitBlock arguments 2024-10-24 13:17:56 -07:00
Yann Collet 94d7b07425 organize specialization at recordFingerprint level 2024-10-24 13:17:56 -07:00
Yann Collet 566763fdc9 new variant, sampling by 11 2024-10-24 13:17:56 -07:00
Yann Collet 90095f056d apply limit conditions for all splitting strategies 
instead of just for blind split.

This is in anticipation of adversarial input,
that would intentionally target the sampling pattern of the split detector.

Note that, even without this protection, splitting can never expand beyond ZSTD_COMPRESSBOUND(),
because this upper limit uses a 1KB block size worst case scenario,
and splitting never creates blocks thath small.

The protection is more to ensure that data is not expanded by more than 3-bytes per 128 KB full block,
which is a much stricter limit.
 2024-10-24 11:36:56 -07:00
Yann Collet c80645a055 stricter limits to ensure expansion factor with blind-split strategy 
issue reported by @terrelln
 2024-10-23 14:55:10 -07:00
Yann Collet 7d3e5e3ba1 split all full 128 KB blocks 
this helps make the streaming behavior more consistent,
since it does no longer depend on having more data presented on the input.

suggested by @terrelln
 2024-10-23 14:18:48 -07:00
Yann Collet b68ddce818 rewrite fingerprint storage to no longer need 64-bit members 
so that it can be stored using standard alignment requirement (sizeof(void*)).

Distance function still requires 64-bit signed multiplication though,
so it won't change the issue regarding the bug in ubsan for clang 32-bit on github ci.
 2024-10-23 11:50:57 -07:00
Yann ColletandYann Collet 57239c4d3b fixed minor strict pedantic C90 issue 2024-10-23 11:50:57 -07:00
Yann Collet 18b1e67223 fixed extraneous return 
strict C90 compliance test
 2024-10-23 11:50:57 -07:00
Yann Collet 0be334d208 fixes static state allocation check 
detected by @felixhandte
 2024-10-23 11:50:57 -07:00
Yann Collet 06b7cfabf8 rewrote ZSTD_cwksp_initialAllocStart() to be easier to read 
following a discussion with @felixhandte
 2024-10-23 11:50:57 -07:00
Yann Collet 16450d0732 rewrite penalty update 
suggested by @terrelln
 2024-10-23 11:50:57 -07:00
Yann Collet 1ec5f9f1f6 changed loop exit condition so that there is no need to assert() within the loop. 2024-10-23 11:50:57 -07:00
Yann Collet 4662f6e646 renamed: FingerPrint => Fingerprint 
suggested by @terrelln
 2024-10-23 11:50:57 -07:00
Yann Collet ea85dc7af6 conservatively estimate over-splitting in presence of incompressible loss 
ensure data can never be expanded by more than 3 bytes per full block.
 2024-10-23 11:50:57 -07:00
Yann Collet 5ae34e4c96 ensure lastBlock is correctly determined 
reported by @terrelln
 2024-10-23 11:50:57 -07:00
Yann Collet 7bad787d8b made ZSTD_isPower2() an inline function 2024-10-23 11:50:57 -07:00
Yann Collet a167571db5 added a faster block splitter variant 
that samples 1 in 5 positions.

This variant is fast enough for lazy2 and btlazy2,
but it's less good in combination with post-splitter at higher levels (>= btopt).
 2024-10-23 11:50:57 -07:00
Yann Collet 1c62e714ab minor split optimization 
let's fill the initial stats directly into target fingerprint
 2024-10-23 11:50:57 -07:00
Yann ColletandYann Collet 4ce91cbf2b fixed workspace alignment on non 64-bit systems 2024-10-23 11:50:57 -07:00
Yann Collet cae8d13294 splitter workspace is now provided by ZSTD_CCtx* 2024-10-23 11:50:56 -07:00
Yann Collet 4685eafa81 fix alignment test 
for non 64-bit systems
 2024-10-23 11:50:56 -07:00
Yann Collet 73a6653653 ZSTD_splitBlock_4k() uses externally provided workspace 
ideally, this workspace would be provided from the ZSTD_CCtx* state
 2024-10-23 11:50:56 -07:00
Yann Collet 31d48e9ffa fixing minor formatting issue in 32-bit mode with logs enabled 2024-10-23 11:50:56 -07:00
Yann Collet 6dc52122e6 fixed c90 comment style 2024-10-23 11:50:56 -07:00
Yann Collet 20c3d176cd fix assert 2024-10-23 11:50:56 -07:00
Yann Collet 0d4b520657 only split full blocks 
short term simplification
 2024-10-23 11:50:56 -07:00
Yann Collet 8b3887f579 fixed kernel build 2024-10-23 11:50:56 -07:00
Yann Collet f83ed087f6 fixed RLE detection test 2024-10-23 11:50:56 -07:00
Yann Collet 83a3402a92 fix overlap write scenario in presence of incompressible data 2024-10-23 11:50:56 -07:00
Yann Collet fa147cbb4d more ZSTD_memset() to apply 2024-10-23 11:50:56 -07:00
Yann Collet 6021b6663a minor C++-ism 
though I really wonder if this is a property worth maintaining.
 2024-10-23 11:50:56 -07:00
Yann Collet e2d7d08888 use ZSTD_memset() 
for better portability on Linux kernel
 2024-10-23 11:50:56 -07:00
Yann Collet 586ca96fec do not use new as variable name 2024-10-23 11:50:56 -07:00
Yann Collet 9e52789962 fixed strict C90 semantic 2024-10-23 11:50:56 -07:00
Yann Collet a5bce4ae84 XP: add a pre-splitter 
instead of ingesting only full blocks, make an analysis of data, and infer where to split.
 2024-10-23 11:50:56 -07:00
Yann Collet 47d4f5662d rewrite code in the manner suggested by @terrelln 2024-10-17 09:37:23 -07:00
Yann Collet 6326775166 slightly improved compression ratio at levels 3 & 4 
The compression ratio benefits are small but consistent, i.e. always positive.
On `silesia.tar` corpus, this modification saves ~75 KB at level 3.
The measured speed cost is negligible, i.e. below noise level, between 0 and -1%.
 2024-10-17 09:37:23 -07:00
Yann Collet c2abfc5ba4 minor improvement to level 3 dictionary compression ratio 2024-10-15 17:58:33 -07:00
Yann Collet e63896eb58 small dictionary compression speed improvement 
not as good as small-blocks improvement,
but generally positive.
 2024-10-15 17:48:35 -07:00
Yann ColletandGitHub 8c38bda935 Merge pull request #4165 from facebook/cspeed_cmov 
Improve compression speed on small blocks
 2024-10-11 16:20:19 -07:00
Yann Collet 8e5823b65c rename variable name 
findMatch -> matchFound
since it's a test, as opposed to an active search operation.
suggested by @terrelln
 2024-10-11 15:38:12 -07:00
Yann Collet 83de00316c fixed parameter ordering in dfast 
noticed by @terrelln
 2024-10-11 15:36:15 -07:00
Yann Collet fa1fcb08ab minor: better variable naming 2024-10-10 16:07:20 -07:00
Yann Collet d45aee43f4 make __asm__ a __GNUC__ specific 2024-10-08 16:38:35 -07:00
Yann Collet 741b860fc1 store dummy bytes within ZSTD_match4Found_cmov() 
feels more logical, better contained
 2024-10-08 16:34:40 -07:00
Yann Collet 197c258a79 introduce memory barrier to force test order 
suggested by @terrelln
 2024-10-08 15:54:48 -07:00
Yann Collet 186b132495 made search strategy switchable 
between cmov and branch
and use a simple heuristic based on wlog to select between them.

note: performance is not good on clang (yet)
 2024-10-08 13:52:56 -07:00