std/lzma: add new package

This commit is contained in:
Nigel Tao committed 2023-12-12 23:41:26 +11:00
1 parent 42d3b4d692
commit 57f01ec3f6
9 files changed
+3720

No files matched your search

+331
View File
@@ -0,0 +1,331 @@
# LZMA
Lempel–Ziv–Markov chain algorithm (LZMA) is a general purpose compression
format, using range coding and Lempel-Ziv style length-distance
back-references. Various descriptions of the file format are at:
- [LZMA-SDK reference implementation](https://raw.githubusercontent.com/jljusten/LZMA-SDK/781863cdf592da3e97420f50de5dac056ad352a5/CPP/7zip/Bundles/LzmaSpec/LzmaSpec.cpp)
- [LZMA-SDK specification](https://raw.githubusercontent.com/jljusten/LZMA-SDK/781863cdf592da3e97420f50de5dac056ad352a5/DOC/lzma-specification.txt)
- [Lzip Manual - Stream Format](https://www.nongnu.org/lzip/manual/lzip_manual.html#Stream-format)
- [Wikipedia (LZMA)](https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Markov_chain_algorithm)
## Byms and Packets
The compressed stream is a stream of "byms" (binary symbols, either 0 or 1;
yeah, it's an awkward term, but less ambiguous than using "bits" for both
compressed and uncompressed things). Byms are conceptually very similar to bits
(binary digits, ubiquitous in computing, a 0.5 KiB file holds 512 bytes, which
is 4096 bits), but are encoded using an adaptive binary range-coder so that N
byms usually compresses to fewer than N bits.
The stream is divided into packets, each packet describing either a single
byte, or an LZ77 sequence with its length and distance implicitly or explicitly
encoded. Each part of each packet is modeled with independent contexts, so the
probability predictions for each bym are correlated with the values of that bym
(and related byms from the same field) in previous packets of the same type.
There are 7 types of packets:
Symbols Meaning
0 ,literal LITERAL byte (8_byms)
1,0 ,len ,dist MATCH
1,1,0,0 SHORTREP len = 1, dist = Most Recently Used
1,1,0,1 ,len LONGREP[0] dist = Most Recently Used
1,1,1,0 ,len LONGREP[1] dist = 2nd Most Recently Used
1,1,1,1,0 ,len LONGREP[2] dist = 3rd Most Recently Used
1,1,1,1,1 ,len LONGREP[3] dist = 4th Most Recently Used
The length encoding:
Symbols Value
0 ,3_byms Ranges from 2 ..= 9
1,0 ,3_byms Ranges from 10 ..= 17
1,1 ,8_byms Ranges from 18 ..= 273
The distance encoding starts with a 6-bym "Slot" value, which determines how
many further byms are needed. For small Slot values, there are up to 5 extra
byms, all context-encoded like the rest of LZMA. For large Slot values, there
are N extra byms. The first (N - 4) of them are encoded with a fixed 50%
probability and the remaining 4 byms are context-encoded. The largest encodable
distance-biased-by-1 is `0xFFFF_FFFF`, a (2 + 26 + 4) bit number.
Slot (decimal) Distance (binary), biased by 1 Extra byms
0 0 0
1 1 0
2 10 0
3 11 0
4 10 x 1
5 11 x 1
6 10 xx 2
7 11 xx 2
8 10 xxx 3
9 11 xxx 4
10 10 xxxx 5
11 11 xxxx 4
12 10 xxxxx 5
13 11 xxxxx 5
14 10 yy zzzz 2+4
15 11 yy zzzz 2+4
16 10 yyy zzzz 3+4
17 11 yyy zzzz 3+4
18 10 yyyy zzzz 4+4
... ... ...
61 11 yyyyyyyyyyyyyyyyyyyyyyyyy zzzz 25+4
62 10 yyyyyyyyyyyyyyyyyyyyyyyyyy zzzz 26+4
63 11 yyyyyyyyyyyyyyyyyyyyyyyyyy zzzz 26+4
"xxxx" means up-to-5 byms are encoded with a "reverse" binary tree. Each Slot
has its own binary tree probabilities.
"yyyy" means up-to-26 byms. Each has a fixed 50% probability.
"zzzz" means four byms encoded with a "reverse" binary tree. All Slots use the
same "align" binary tree probabilities.
"Biased by 1" means that slot=2 implies biasedDistance=2 so distance=3. A
biasedDistance of `0xFFFF_FFFF` means the End of Stream. Otherwise, the
corrected (unbiased) distance ranges in `[1 ..= 0xFFFF_FFFF]`.
Distance is also capped by the dictionary size, stated in the LZMA header.
## State
In addition to the hundreds of contextual probabilities (see below), there's
also a "State" variable with 12 possible values. The initial State is zero.
Each packet causes a state transtition. The left-most column is the current
State, the other columns hold the next State, depending on the packet:
State LITERAL MATCH LONGREP SHORTREP
0 0 7 8 9
1 0 7 8 9
2 0 7 8 9
3 0 7 8 9
4 1 7 8 9
5 2 7 8 9
6 3 7 8 9
7 4 10 11 11
8 5 10 11 11
9 6 10 11 11
10 4 10 11 11
11 5 10 11 11
Equivalently, but looking backwards instead of forwards, each State embodies
the 1st, 2nd, 3rd and 4th MRUP (Most Recently Used Packet). The ? question mark
means every possible packet. Some States (2, 5 and 11) have two rows - multiple
possible histories could lead to that State:
State 1MRUP 2MRUP 3MRUP 4MRUP
0 LITERAL LITERAL LITERAL ?
1 LITERAL LITERAL MATCH ?
2a LITERAL LITERAL LONGREP ?
2b LITERAL LITERAL SHORTREP NON-LITERAL
3 LITERAL LITERAL SHORTREP LITERAL
4 LITERAL MATCH ? ?
5a LITERAL LONGREP ? ?
5b LITERAL SHORTREP NON-LITERAL ?
6 LITERAL SHORTREP LITERAL ?
7 MATCH LITERAL ? ?
8 LONGREP LITERAL ? ?
9 SHORTREP LITERAL ? ?
10 MATCH NON-LITERAL ? ?
11a LONGREP NON-LITERAL ? ?
11b SHORTREP NON-LITERAL ? ?
A State's 12 possible values can also be aggregated into whether the State is
less than or at least 7: whether the last packet was a LITERAL or a NON-LITERAL
(a Lempel-Ziv copy-from-history). When decoding a LITERAL after a NON-LITERAL
(as opposed to a LITERAL after a LITERAL, i.e. having `decodeLiteral()` branch
on `State < 7`), there is information in the next historical byte after the
previous NON-LITERAL packet's copy-source. That byte almost always does not
equal the about-to-be-decoded literal byte, because if it did equal, the copy
would have been longer.
## Algorithm Overview
The core loop's body (with AlgOveNN line numbers for others to reference) looks
conceptually like this:
AlgOve00 if decodeTheNextBym() == 0 {
AlgOve01 // Decode a LITERAL.
AlgOve02 literal = decodeLiteral()
AlgOve03 emitLiteral(literal)
AlgOve04 continue
AlgOve..
AlgOve20 } else if decodeTheNextBym() == 0 {
AlgOve21 // Decode a MATCH.
AlgOve22 len = decodeLen()
AlgOve23 slot = decodeSlot(min(len-2, 3))
AlgOve24 distBiasedBy1 = decodeDistBiasedBy1(slot)
AlgOve25 if distBiasedBy1 == 0xFFFF_FFFF {
AlgOve26 break
AlgOve27 }
AlgOve28 mrud = (1 + distBiasedBy1, mrud[0], mrud[1], mrud[2])
AlgOve29 goto labelDoTheLZCopy
AlgOve..
AlgOve40 } else if decodeTheNextBym() == 0 {
AlgOve41 if decodeTheNextBym() == 0 {
AlgOve42 // Decode a SHORTREP.
AlgOve43 len = 1
AlgOve44 goto labelDoTheLZCopy
AlgOve45 }
AlgOve46 // Decode a LONGREP[0].
AlgOve..
AlgOve60 } else if decodeTheNextBym() == 0 {
AlgOve61 // Decode a LONGREP[1].
AlgOve62 mrud = (mrud[1], mrud[0], mrud[2], mrud[3])
AlgOve63 } else if decodeTheNextBym() == 0 {
AlgOve64 // Decode a LONGREP[2].
AlgOve65 mrud = (mrud[2], mrud[0], mrud[1], mrud[3])
AlgOve66 } else {
AlgOve67 // Decode a LONGREP[3].
AlgOve68 mrud = (mrud[3], mrud[0], mrud[1], mrud[2])
AlgOve69 }
AlgOve..
AlgOve80 len = decodeLen()
AlgOve..
AlgOve90 labelDoTheLZCopy:
AlgOve91 // mrud[0] has been set to what will be (after the emitCopy
AlgOve92 // call) the most recently used distance. mrud[1] is the 2nd
AlgOve93 // most recently used, mrud[2] is the 3rd, mrud[3] is the 4th.
AlgOve94 emitCopy(len, mrud[0])
Conceptually means that some details have been elided. The `State` variable
needs updating after each iteration, but that's not shown. `decodeTheNextBym()`
also depends not only on the `State` but also which line number you're at. The
probabilities used for "is it a LITERAL or NON-LITERAL" are different from the
probabilities used for "is it a LONGREP[2] or LONGREP[3]".
`decodeLen` expands to:
AlgOveNN.0 if decodeTheNextBym() == 0 {
AlgOveNN.1 // Decode a low length.
AlgOveNN.2 len = decodeMultipleByms(3) + 2
AlgOveNN.3 } else if decodeTheNextBym() == 0 {
AlgOveNN.4 // Decode a middle length.
AlgOveNN.5 len = decodeMultipleByms(3) + 10
AlgOveNN.6 } else {
AlgOveNN.7 // Decode a high length.
AlgOveNN.8 len = decodeMultipleByms(8) + 18
AlgOveNN.9 }
## Range Coding and `decodeTheNextBym`
The codec also tracks two `uint32` variables, called `bits` and `range`, that
represent an interval (a position and width) on the number line between zero
and one. The literal bitstream (the 0s and 1s in the file, read in MSB to LSB
order) is translated into a `bym` stream, where each `bym` is either 0 or 1. If
both `bym` values have equal (50%) probability then the `bym` stream is
basically the same as the bitstream. If symbols have unequal probability then
we can achieve compression: the literal bitstream can be shorter (in terms of
number of bits) then the `bym` stream.
Specifically, if `prob` is the probability (scaled so that `(1 << 11) = 2048`
means 100% probability and so `1024` means 50% probability) that the next `bym`
is 0, and `range` is at least `(1 << 24)`, then (to over-simplify for now)
`decodeTheNextBym()` is:
threshold = (range >> 11) * prob
if bits < threshold {
bym = 0
} else {
bym = 1
}
The probability `prob` depends on the `State` variable as well as additional
context, such as the high bits of the most recent decoded byte and the low bits
of the number of decoded bytes. Implementations maintain an array with hundreds
of contextual probabilities (each a `uint16` in the range `1 ..= 2047`),
indexed by a `prob_index` variable (calculated from the `State` and other
factors). Expanding (marked by ¶) the over-simplified pseudo-code above:
prob = probs_array[prob_index] // ¶
threshold = (range >> 11) * prob
if bits < threshold {
bym = 0
} else {
bym = 1
}
LZMA probabilties also adapt. Producing a 0 (or 1) `bym` increases (or
decreases) the relevant contextual probability: the prediction that, the next
time we're in the same context, that next `bym` is 0. Expanding again:
prob = probs_array[prob_index]
threshold = (range >> 11) * prob
if bits < threshold {
bym = 0
prob += (2048 - prob) >> 5 // ¶
probs_array[prob_index] = prob // ¶
} else {
bym = 1
prob -= prob >> 5 // ¶
probs_array[prob_index] = prob // ¶
}
We also need to update the `bits` and `range`. The `range` shrinks every time
we decode a `bym` and, if small enough, then grows by reading the next source
byte. Expanding one last time, `decodeTheNextBym()` (which also updates `bits`,
`range` and the relevant contextual probability) is:
prob = probs_array[prob_index]
threshold = (range >> 11) * prob
if bits < threshold {
bym = 0
range = threshold // ¶
prob += (2048 - prob) >> 5
probs_array[prob_index] = prob
} else {
bym = 1
bits -= threshold // ¶
range -= threshold // ¶
prob -= prob >> 5
probs_array[prob_index] = prob
}
if (range >> 24) == 0 { // ¶
bits = (bits << 8) | src.read_byte() // ¶
range <<= 8 // ¶
} // ¶
## Binary Trees and `decodeMultipleByms`
Decoding N byms (e.g. decoding a literal, a length, a slot or a distance) is
basically N sequential calls to `decodeTheNextBym`, using `((1 << N) - 1)`
different probabilties. This can be visualized as a balanced binary tree with
`((1 << N) - 1)` branch nodes and `(1 << N)` leaf nodes. Each branch node has
its own probability for taking one or the other of its two child nodes.
- There's 1 probability for the first bym.
- There's 2 probabilities for the second bym. The one that is used depends on
the value of the first bym.
- There's 4 probabilities for the third bym. The one that is used depends on
the values of the first two byms.
- Etc.
- There's `(1 << (N - 1))` probabilities for the Nth bym. The one that is used
depends on the values of all previous byms.
- There's `(1 << N)` leaf nodes, one for each possible N-bit value.
These probabilities can be flattened to a `(1 << N)` element array, whose first
element is unused (or available for repurposing, such as for what the reference
implementation calls the "choice" probabilities). For example, if N is 3 then
the 8-element array could be labeled as `u1223333`, where `u` is unused, `1`
marks the probability for the first bym, `2` marks the probabilities for the
second bym, etc.
Algorithmically, we loop N times, walking the tree from the root to a leaf. The
final N-bit value is captured in the tree node number (after applying a bit
mask or, equivalently, a subtraction):
tree_node = 1
while tree_node < (1 << N) {
bym = decodeTheNextBym() // Using prob_index = tree_node.
tree_node = (tree_node << 1) | bym
}
tree_node &= ((1 << N) - 1) // Equivalent to "tree_node -= (1 << N)".
That's a "forward" binary tree, where the bits occur in MSB to LSB order. A
"reverse" binary tree uses the opposite order.
+835
View File
@@ -0,0 +1,835 @@
// Copyright 2023 The Wuffs Authors.
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.
//
// SPDX-License-Identifier: Apache-2.0 OR MIT
// --------
// The AlgOveNN line numbers refer to the ./README.md file.
//
// Most LZMA C/C++ implementations use macros for the "decodeTheNextBym()" code
// fragment in that "Algorithm Overview" section, repeated multiple times in
// the decoding algorithm. Wuffs does not have macros (or inline functions), so
// its LZMA implementation looks relatively verbose and repetitive (even though
// it's equivalent to the C/C++ code, at the machine code level).
pri func decoder.decode_bitstream_slow?(dst: base.io_writer, src: base.io_reader) {
var c8 : base.u8
var bits : base.u32
var range : base.u32
var state : base.u32[..= 11]
// repN is what the "Algorithm Overview" calls (mrud[N] - 1). The minimum
// distance is 1.
var rep0 : base.u32[..= 0xFFFF_FFFE]
var rep1 : base.u32[..= 0xFFFF_FFFE]
var rep2 : base.u32[..= 0xFFFF_FFFE]
var rep3 : base.u32[..= 0xFFFF_FFFE]
var reptmp : base.u32[..= 0xFFFF_FFFE]
var rep : base.u32
var pos : base.u64
var lc : base.u32[..= 8]
var lp_mask : base.u64[..= 15]
var pb_mask : base.u64[..= 15]
var prob : base.u32
var threshold : base.u32
var tree_node : base.u32[..= 0x1FF]
var prev_byte : base.u8
var match_byte : base.u8
var match_cusp : base.u32[..= 0xFFFF]
var match_bit : base.u32[..= 1]
var len_state : base.u32[..= 3] // Equals min(len-2, 3) when in a MATCH.
var slot : base.u32[..= 0x7F]
var len : base.u32[..= 273]
var num_extra_bits : base.u32[..= 30]
var dist_extra_bits : base.u32
var high_bit_was_on : base.u32
var i : base.u32
var index_ao00 : base.u32[..= (12 << 4) - 1]
var index_ao41 : base.u32[..= (12 << 4) - 1]
var index_lit : base.u32[..= (1 << 4) - 1]
var index_len : base.u32[..= (1 << 4) - 1]
var index_small_dist_base : base.u32
var index_small_dist_extra : base.u32
var index_small_dist : base.u32[..= 127]
var index_large_dist : base.u32[..= 15]
// Initialize the bitstream.
c8 = args.src.read_u8?()
if c8 <> 0x00 {
return "#bad code"
}
bits = args.src.read_u32be?()
if bits == 0xFFFF_FFFF {
return "#bad code"
}
range = 0xFFFF_FFFF
lc = this.lc
lp_mask = ((1 as base.u64) << this.lp) - 1
pb_mask = ((1 as base.u64) << this.pb) - 1
// "outer" is the core loop discussed in README.md's "Algorithm Overview".
while.outer pos < this.decoded_length {
// AlgOve00 if decodeTheNextBym() == 0
index_ao00 = (state << 4) | ((pos & pb_mask) as base.u32)
prob = this.probs_ao00[index_ao00] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_ao00[index_ao00] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve01 // Decode a LITERAL.
// AlgOve02 literal = decodeLiteral()
index_lit = 15 & (
(((pos & lp_mask) as base.u32) << lc) |
((prev_byte as base.u32) >> (8 - lc)))
tree_node = 1
if state >= 7 {
assert tree_node < 0x100
while true,
pre tree_node < 0x100,
{
match_bit = (match_byte >> 7) as base.u32
match_byte ~mod<<= 1
prob = this.probs_lit[index_lit][0x100 + (match_bit << 8) + tree_node] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_lit[index_lit][0x100 + (match_bit << 8) + tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1)
if (tree_node >= 0x100) or (match_bit <> 0) {
break
}
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_lit[index_lit][0x100 + (match_bit << 8) + tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1) | 1
if (tree_node >= 0x100) or (match_bit == 0) {
break
}
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
}
while tree_node < 0x100 {
prob = this.probs_lit[index_lit][tree_node] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_lit[index_lit][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1)
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_lit[index_lit][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1) | 1
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
// AlgOve03 emitLiteral(literal)
// AlgOve04 continue
prev_byte = (tree_node & 0xFF) as base.u8
args.dst.write_u8?(a: prev_byte)
pos ~mod+= 1
state = STATE_TRANSITION_LITERAL[state] as base.u32
continue.outer
}
// AlgOve00a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_ao00[index_ao00] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
while.goto_do_the_lz_copy true,
post len >= 1,
{{
// AlgOve20 if decodeTheNextBym() == 0
prob = this.probs_ao20[state] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_ao20[state] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve21 // Decode a MATCH.
while.goto_have_len true,
post len >= 1,
{{
// AlgOve22 len = decodeLen()
// AlgOve22.0 if decodeTheNextBym() == 0
prob = this.probs_match_len_low[0][0] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_match_len_low[0][0] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve22.1 // Decode a low length.
// AlgOve22.2 len = decodeMultipleByms(3) + 2
index_len = (pos & pb_mask) as base.u32
tree_node = 1
while tree_node < 0x08 {
prob = this.probs_match_len_low[index_len][tree_node] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_match_len_low[index_len][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1)
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_match_len_low[index_len][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1) | 1
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
len_state = CLAMP_NO_MORE_THAN_3[tree_node & 0x07] as base.u32
len = (tree_node & 0x07) + 2
assert len >= 1
break.goto_have_len // goto AlgOve23.
}
// AlgOve22.0a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_match_len_low[0][0] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve22.3 if decodeTheNextBym() == 0
prob = this.probs_match_len_mid[0][0] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_match_len_mid[0][0] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve22.4 // Decode a middle length.
// AlgOve22.5 len = decodeMultipleByms(3) + 10
index_len = (pos & pb_mask) as base.u32
tree_node = 1
while tree_node < 0x08 {
prob = this.probs_match_len_mid[index_len][tree_node] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_match_len_mid[index_len][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1)
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_match_len_mid[index_len][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1) | 1
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
len = (tree_node & 0x07) + 10
assert len >= 1
len_state = 3
break.goto_have_len // goto AlgOve23.
}
// AlgOve22.3a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_match_len_mid[0][0] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve22.7 // Decode a high length.
// AlgOve22.8 len = decodeMultipleByms(8) + 18
tree_node = 1
while tree_node < 0x100 {
prob = this.probs_match_len_high[0][tree_node] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_match_len_high[0][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1)
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_match_len_high[0][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1) | 1
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
len = (tree_node & 0xFF) + 18
assert len >= 1
len_state = 3
break.goto_have_len
}} endwhile.goto_have_len
// AlgOve23 slot = decodeSlot(min(len-2, 3))
slot = 1
while slot < 0x40,
inv len >= 1,
{
prob = this.probs_slot[len_state][slot] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_slot[len_state][slot] = (prob & 0xFFFF) as base.u16
slot = (slot << 1)
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_slot[len_state][slot] = (prob & 0xFFFF) as base.u16
slot = (slot << 1) | 1
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
slot &= 0x3F
// AlgOve24 distBiasedBy1 = decodeDistBiasedBy1(slot)
rep = slot
if slot < 4 {
// No-op.
} else if slot < 14 {
num_extra_bits = (slot >> 1) - 1
rep = (2 | (slot & 1)) << num_extra_bits
// Read num_extra_bits "xxxx" bits.
index_small_dist_base = rep ~mod- slot
index_small_dist_extra = 1
dist_extra_bits = 0
i = 0
while i < num_extra_bits,
inv len >= 1,
{
assert i < 30 via "a < b: a < c; c <= b"(c: num_extra_bits)
index_small_dist = (index_small_dist_base ~mod+ index_small_dist_extra) & 127
prob = this.probs_small_dist[index_small_dist] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_small_dist[index_small_dist] = (prob & 0xFFFF) as base.u16
index_small_dist_extra = (index_small_dist_extra ~mod<< 1)
i += 1
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_small_dist[index_small_dist] = (prob & 0xFFFF) as base.u16
index_small_dist_extra = (index_small_dist_extra ~mod<< 1) | 1
dist_extra_bits |= (1 as base.u32) << i
i += 1
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
rep ~mod+= dist_extra_bits
} else {
num_extra_bits = (slot >> 1) - 1
assert num_extra_bits >= 6
assert num_extra_bits <= 30
rep = (2 | (slot & 1)) << num_extra_bits
// Read (num_extra_bits - 4) "yyyy" bits.
dist_extra_bits = 0
while true,
pre num_extra_bits > 4,
inv len >= 1,
post num_extra_bits > 0,
post num_extra_bits <= 4,
{
range >>= 1
bits ~mod-= range
high_bit_was_on = 0 ~mod- (bits >> 31)
bits ~mod+= range & high_bit_was_on
dist_extra_bits = (dist_extra_bits ~mod<< 1) | ((high_bit_was_on ~mod+ 1) & 1)
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
num_extra_bits -= 1
if num_extra_bits <= 4 {
break
}
} endwhile
dist_extra_bits ~mod<<= 4
// Read 4 "zzzz" bits.
index_large_dist = 1
while true,
pre num_extra_bits > 0,
pre num_extra_bits <= 4,
inv len >= 1,
{
prob = this.probs_large_dist[index_large_dist] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_large_dist[index_large_dist] = (prob & 0xFFFF) as base.u16
index_large_dist = 15 & ((index_large_dist ~mod<< 1))
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_large_dist[index_large_dist] = (prob & 0xFFFF) as base.u16
index_large_dist = 15 & ((index_large_dist ~mod<< 1) | 1)
dist_extra_bits |= (1 as base.u32) << (4 - num_extra_bits)
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
num_extra_bits -= 1
if num_extra_bits <= 0 {
break
}
} endwhile
rep ~mod+= dist_extra_bits
}
// AlgOve25 if distBiasedBy1 == 0xFFFF_FFFF
if rep >= 0xFFFF_FFFF {
// AlgOve26 break
break.outer
}
// AlgOve28 mrud = (1 + distBiasedBy1, mrud[0], mrud[1], mrud[2])
rep3 = rep2
rep2 = rep1
rep1 = rep0
rep0 = rep
assert len >= 1
state = STATE_TRANSITION_MATCH[state] as base.u32
break.goto_do_the_lz_copy // goto AlgOve90.
}
// AlgOve20a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_ao20[state] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve40 if decodeTheNextBym() == 0
prob = this.probs_ao40[state] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_ao40[state] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve41 if decodeTheNextBym() == 0
index_ao41 = (state << 4) | ((pos & pb_mask) as base.u32)
prob = this.probs_ao41[index_ao41] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_ao41[index_ao41] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve42 // Decode a SHORTREP.
// AlgOve43 len = 1
len = 1
assert len >= 1
state = STATE_TRANSITION_SHORTREP[state] as base.u32
break.goto_do_the_lz_copy // goto AlgOve90.
}
// AlgOve41a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_ao41[index_ao41] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve46 // Decode a LONGREP[0].
} else {
// AlgOve40a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_ao40[state] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve60 if decodeTheNextBym() == 0
prob = this.probs_ao60[state] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_ao60[state] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve61 // Decode a LONGREP[1].
// AlgOve62 mrud = (mrud[1], mrud[0], mrud[2], mrud[3])
reptmp = rep1
rep1 = rep0
rep0 = reptmp
} else {
// AlgOve60a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_ao60[state] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve63 if decodeTheNextBym() == 0
prob = this.probs_ao63[state] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_ao63[state] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve64 // Decode a LONGREP[2].
// AlgOve65 mrud = (mrud[2], mrud[0], mrud[1], mrud[3])
reptmp = rep2
rep2 = rep1
rep1 = rep0
rep0 = reptmp
} else {
// AlgOve63a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_ao63[state] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve67 // Decode a LONGREP[3].
// AlgOve68 mrud = (mrud[3], mrud[0], mrud[1], mrud[2])
reptmp = rep3
rep3 = rep2
rep2 = rep1
rep1 = rep0
rep0 = reptmp
}
}
}
// AlgOve80 len = decodeLen()
while.goto_after_decode_len true,
post len >= 1,
{{
// AlgOve80.0 if decodeTheNextBym() == 0
prob = this.probs_longrep_len_low[0][0] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_longrep_len_low[0][0] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve80.1 // Decode a low length.
// AlgOve80.2 len = decodeMultipleByms(3) + 2
index_len = (pos & pb_mask) as base.u32
tree_node = 1
while tree_node < 0x08 {
prob = this.probs_longrep_len_low[index_len][tree_node] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_longrep_len_low[index_len][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1)
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_longrep_len_low[index_len][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1) | 1
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
len = (tree_node & 0x07) + 2
assert len >= 1
state = STATE_TRANSITION_LONGREP[state] as base.u32
break.goto_after_decode_len
}
// AlgOve80.0a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_longrep_len_low[0][0] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve80.3 if decodeTheNextBym() == 0
prob = this.probs_longrep_len_mid[0][0] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_longrep_len_mid[0][0] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve80.4 // Decode a middle length.
// AlgOve80.5 len = decodeMultipleByms(3) + 10
index_len = (pos & pb_mask) as base.u32
tree_node = 1
while tree_node < 0x08 {
prob = this.probs_longrep_len_mid[index_len][tree_node] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_longrep_len_mid[index_len][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1)
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_longrep_len_mid[index_len][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1) | 1
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
len = (tree_node & 0x07) + 10
assert len >= 1
state = STATE_TRANSITION_LONGREP[state] as base.u32
break.goto_after_decode_len
}
// AlgOve80.3a if decodeTheNextBym() == 0 // If-false branch.
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_longrep_len_mid[0][0] = (prob & 0xFFFF) as base.u16
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
// AlgOve80.7 // Decode a high length.
// AlgOve80.8 len = decodeMultipleByms(8) + 18
tree_node = 1
while tree_node < 0x100 {
prob = this.probs_longrep_len_high[0][tree_node] as base.u32
threshold = (range >> 11) ~mod* prob
if bits < threshold {
range = threshold
prob ~mod+= (2048 ~mod- prob) >> 5
this.probs_longrep_len_high[0][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1)
} else {
bits ~mod-= threshold
range ~mod-= threshold
prob ~mod-= prob >> 5
this.probs_longrep_len_high[0][tree_node] = (prob & 0xFFFF) as base.u16
tree_node = (tree_node << 1) | 1
}
if (range >> 24) == 0 {
c8 = args.src.read_u8?()
bits = (bits ~mod<< 8) | (c8 as base.u32)
range ~mod<<= 8
}
} endwhile
len = (tree_node & 0xFF) + 18
assert len >= 1
state = STATE_TRANSITION_LONGREP[state] as base.u32
break.goto_after_decode_len
}} endwhile.goto_after_decode_len
break.goto_do_the_lz_copy
}} endwhile.goto_do_the_lz_copy
// AlgOve90 labelDoTheLZCopy:
// AlgOve94 emitCopy(len, mrud[0])
assert len >= 1
if (((rep0 + 1) as base.u64) > pos) or
(((rep0 + 1) as base.u64) > (this.dict_size as base.u64)) {
return "#bad distance"
}
pos ~mod+= len as base.u64
while (len as base.u64) > args.dst.length(),
inv len >= 1,
post (len as base.u64) <= args.dst.length(),
{
yield? base."$short write"
} endwhile
if ((rep0 + 1) as base.u64) > args.dst.history_length() {
return base."#insufficient history"
}
assert (rep0 + 1) >= 1
match_cusp = args.dst.limited_copy_u32_from_history_fast_return_cusp!(
up_to: len, distance: (rep0 + 1))
match_byte = (match_cusp >> 8) as base.u8
prev_byte = (match_cusp & 0xFF) as base.u8
} endwhile.outer
if bits <> 0 {
return "#bad bitstream trailer"
} else if (this.decoded_length <> 0xFFFF_FFFF_FFFF_FFFF) and
(this.decoded_length <> pos) {
return "#bad decoded length"
}
}
+324
View File
@@ -0,0 +1,324 @@
// Copyright 2023 The Wuffs Authors.
//
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
// https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
// <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
// option. This file may not be copied, modified, or distributed
// except according to those terms.
//
// SPDX-License-Identifier: Apache-2.0 OR MIT
pub status "#bad bitstream trailer"
pub status "#bad code"
pub status "#bad decoded length"
pub status "#bad distance"
pub status "#bad header"
pub status "#truncated input"
pub status "#unsupported decoded length"
pub status "#unsupported properties"
pub const DECODER_HISTORY_RETAIN_LENGTH_MAX_INCL_WORST_CASE : base.u64 = 0xFFFF_FFFF
pub const DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE : base.u64 = 0
pub struct decoder? implements base.io_transformer(
// In theory, lc ranges in [0 ..= 8]. In practice, to keep the
// worst-case size of the statically allocated probs_lit array
// managable, we only support configurations where ((lc + lp) <= 4) and
// otherwise return "#unsupported properties".
//
// This matches "LZMA2's limited property condition" described at:
// https://github.com/jljusten/LZMA-SDK/blob/781863cdf592da3e97420f50de5dac056ad352a5/DOC/lzma-specification.txt#L192
lc : base.u32[..= 4], // The number of Literal Context bits. Default: 3.
lp : base.u32[..= 4], // The number of Literal Position bits. Default: 0.
pb : base.u32[..= 4], // The number of Position Bits. Default: 2.
dict_size : base.u32,
decoded_length : base.u64,
util : base.utility,
) + (
// 12 is the number of states. 4 is the pb inclusive maximum.
//
// The AlgOveNN line numbers refer to the ./README.md file.
probs_ao00 : array[12 << 4] base.u16, // AlgOve00.
probs_ao20 : array[12] base.u16, // AlgOve20.
probs_ao40 : array[12] base.u16, // AlgOve40.
probs_ao41 : array[12 << 4] base.u16, // AlgOve41.
probs_ao60 : array[12] base.u16, // AlgOve60.
probs_ao63 : array[12] base.u16, // AlgOve63.
// AlgOve22 and what "decodeLen expands to".
probs_match_len_low : array[1 << 4] array[1 << 3] base.u16,
probs_match_len_mid : array[1 << 4] array[1 << 3] base.u16,
probs_match_len_high : array[1 << 0] array[1 << 8] base.u16,
// AlgOve80 and what "decodeLen expands to".
probs_longrep_len_low : array[1 << 4] array[1 << 3] base.u16,
probs_longrep_len_mid : array[1 << 4] array[1 << 3] base.u16,
probs_longrep_len_high : array[1 << 0] array[1 << 8] base.u16,
// AlgOve23. The first index is min(len-2, 3). There are then 64
// possible slots, so a 6-bit binary tree.
probs_slot : array[4] array[1 << 6] base.u16,
// This holds the "xxxx" binary trees, packed per below. The first
// element and last 13 elements are unused. The index_small_dist_base
// constant is set to (Low - 1) and index_small_dist_extra variable
// starts at 1. (index_small_dist_base + index_small_dist_extra) ranges
// in Low .. High during the "decode a binary tree" loop.
//
// Slot Low .. High H-L NumberOfXByms
// 4 1 .. 2 1 1
// 5 2 .. 3 1 1
// 6 3 .. 6 3 2
// 7 6 .. 9 3 2
// 8 9 .. 16 7 3
// 9 16 .. 23 7 3
// 10 23 .. 38 15 4
// 11 38 .. 53 15 4
// 12 53 .. 84 31 5
// 13 84 .. 115 31 5
probs_small_dist : array[128] base.u16,
// This holds the "zzzz" binary tree.
probs_large_dist : array[1 << 4] base.u16,
// 4 is the (lc + lp) inclusive maximum. The 0x300 holds three 8-bit
// binary trees. The first is for (state < 7), when the previous packet
// was a LITERAL. The other two trees are for (state >= 7), when it was
// an LZ-copy. Which tree gets used depends on the bits in the next
// historical byte after the LZ-copy-source, up until the point where
// those bits don't match the currently decoded literal byte, when it
// drops back to the first tree.
probs_lit : array[1 << 4] array[0x300] base.u16,
)
pub func decoder.get_quirk(key: base.u32) base.u64 {
return 0
}
pub func decoder.set_quirk!(key: base.u32, value: base.u64) base.status {
return base."#unsupported option"
}
pub func decoder.history_retain_length() base.optional_u63 {
if this.dict_size == 0 {
return this.util.make_optional_u63(has_value: false, value: 0)
}
return this.util.make_optional_u63(has_value: true, value: this.dict_size as base.u64)
}
pub func decoder.workbuf_len() base.range_ii_u64 {
return this.util.make_range_ii_u64(min_incl: 0, max_incl: 0)
}
pub func decoder.transform_io?(dst: base.io_writer, src: base.io_reader, workbuf: slice base.u8) {
var status : base.status
while true {
status =? this.do_transform_io?(dst: args.dst, src: args.src, workbuf: args.workbuf)
if (status == base."$short read") and args.src.is_closed() {
return "#truncated input"
}
yield? status
} endwhile
}
pri func decoder.do_transform_io?(dst: base.io_writer, src: base.io_reader, workbuf: slice base.u8) {
this.decode_properties?(src: args.src)
this.initialize_probs!()
this.decode_bitstream_slow?(dst: args.dst, src: args.src)
}
pri func decoder.decode_properties?(src: base.io_reader) {
var c8 : base.u8
var c32 : base.u32
var lc : base.u32[..= 8]
var lp : base.u32[..= 4]
var pb : base.u32[..= 4]
c8 = args.src.read_u8?()
if c8 >= 225 {
return "#bad header"
}
lc = (c8 % 9) as base.u32
c8 /= 9
lp = (c8 % 5) as base.u32
pb = (c8 / 5) as base.u32
if (lc + lp) > 4 {
return "#unsupported properties"
}
this.lc = lc.min(no_more_than: 4)
this.lp = lp
this.pb = pb
c32 = args.src.read_u32le?()
this.dict_size = c32.max(no_less_than: 4096)
this.decoded_length = args.src.read_u64le?()
if (this.decoded_length >= 0x8000_0000_0000_0000) and
(this.decoded_length <> 0xFFFF_FFFF_FFFF_FFFF) {
return "#unsupported decoded length"
}
}
// initialize_probs sets all of the probs_foo[i][j] values to 1024.
pri func decoder.initialize_probs!() {
var i : base.u32
var j : base.u32
i = 0
while i < (12 << 4) {
this.probs_ao00[i] = 1024
i += 1
} endwhile
i = 0
while i < 12 {
this.probs_ao20[i] = 1024
i += 1
} endwhile
i = 0
while i < 12 {
this.probs_ao40[i] = 1024
i += 1
} endwhile
i = 0
while i < (12 << 4) {
this.probs_ao41[i] = 1024
i += 1
} endwhile
i = 0
while i < 12 {
this.probs_ao60[i] = 1024
i += 1
} endwhile
i = 0
while i < 12 {
this.probs_ao63[i] = 1024
i += 1
} endwhile
i = 0
while i < 16 {
j = 0
while j < 8,
inv i < 16,
{
this.probs_match_len_low[i][j] = 1024
j += 1
} endwhile
i += 1
} endwhile
i = 0
while i < 16 {
j = 0
while j < 8,
inv i < 16,
{
this.probs_match_len_mid[i][j] = 1024
j += 1
} endwhile
i += 1
} endwhile
i = 0
while i < 256 {
this.probs_match_len_high[0][i] = 1024
i += 1
} endwhile
i = 0
while i < 16 {
j = 0
while j < 8,
inv i < 16,
{
this.probs_longrep_len_low[i][j] = 1024
j += 1
} endwhile
i += 1
} endwhile
i = 0
while i < 16 {
j = 0
while j < 8,
inv i < 16,
{
this.probs_longrep_len_mid[i][j] = 1024
j += 1
} endwhile
i += 1
} endwhile
i = 0
while i < 256 {
this.probs_longrep_len_high[0][i] = 1024
i += 1
} endwhile
i = 0
while i < 4 {
j = 0
while j < 64,
inv i < 4,
{
this.probs_slot[i][j] = 1024
j += 1
} endwhile
i += 1
} endwhile
i = 0
while i < 128 {
this.probs_small_dist[i] = 1024
i += 1
} endwhile
i = 0
while i < 16 {
this.probs_large_dist[i] = 1024
i += 1
} endwhile
i = 0
while i < 16 {
j = 0
while j < 0x300,
inv i < 16,
{
this.probs_lit[i][j] = 1024
j += 1
} endwhile
i += 1
} endwhile
}
pri const STATE_TRANSITION_LITERAL : roarray[12] base.u8[..= 11] = [
0x0, 0x0, 0x0, 0x0, 0x1, 0x2, 0x3, 0x4, 0x5, 0x6, 0x4, 0x5,
]
pri const STATE_TRANSITION_MATCH : roarray[12] base.u8[..= 11] = [
0x7, 0x7, 0x7, 0x7, 0x7, 0x7, 0x7, 0xA, 0xA, 0xA, 0xA, 0xA,
]
pri const STATE_TRANSITION_LONGREP : roarray[12] base.u8[..= 11] = [
0x8, 0x8, 0x8, 0x8, 0x8, 0x8, 0x8, 0xB, 0xB, 0xB, 0xB, 0xB,
]
pri const STATE_TRANSITION_SHORTREP : roarray[12] base.u8[..= 11] = [
0x9, 0x9, 0x9, 0x9, 0x9, 0x9, 0x9, 0xB, 0xB, 0xB, 0xB, 0xB,
]
pri const CLAMP_NO_MORE_THAN_3 : roarray[8] base.u8[..= 3] = [
0, 1, 2, 3, 3, 3, 3, 3,
]