mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-22 08:47:28 +01:00
Rename std/flate to std/deflate
This commit is contained in:
1 parent
d168e6977b
commit
1388f84e5c
20 files changed
+590
-571
No files matched your search
@@ -0,0 +1,540 @@
|
||||
# Flate
|
||||
|
||||
Flate (also known as DEFLATE) is a general purpose compression format, using
|
||||
Huffman codes and Lempel-Ziv style length-distance back-references. It is
|
||||
specified in [RFC 1951](https://www.ietf.org/rfc/rfc1951.txt).
|
||||
|
||||
Gzip ([RFC 1952](https://www.ietf.org/rfc/rfc1952.txt)) and zlib ([RFC
|
||||
1950](https://www.ietf.org/rfc/rfc1950.txt)) are two thin wrappers (with
|
||||
similar purpose, but different wire formats) around the raw flate encoding.
|
||||
Gzip (the file format) is used by the `gzip` and `tar` command line tools and
|
||||
by the HTTP network protocol. Zlib is used by the ELF executable and PNG image
|
||||
file formats.
|
||||
|
||||
[Zip](https://support.pkware.com/display/PKZIP/APPNOTE) (also known as PKZIP)
|
||||
is another wrapper, similar to combining gzip with tar, that can compress
|
||||
multiple files into a single archive. Zip is widely used by the ECMA Office
|
||||
Open XML format, the OASIS Open Document Format for Office Applications and the
|
||||
Java JAR format.
|
||||
|
||||
Wrangling those formats that build on flate (gzip, zip and zlib) is not
|
||||
provided by this package. For zlib, look at the `std/zlib` package instead. The
|
||||
other formats are TODO.
|
||||
|
||||
For example, look at `test/testdata/romeo.txt*`. First, the uncompressed text:
|
||||
|
||||
$ xxd test/testdata/romeo.txt
|
||||
00000000: 526f 6d65 6f20 616e 6420 4a75 6c69 6574 Romeo and Juliet
|
||||
00000010: 0a45 7863 6572 7074 2066 726f 6d20 4163 .Excerpt from Ac
|
||||
etc
|
||||
00000390: 740a 536f 2073 7475 6d62 6c65 7374 206f t.So stumblest o
|
||||
000003a0: 6e20 6d79 2063 6f75 6e73 656c 3f0a n my counsel?.
|
||||
|
||||
The raw flate encoding:
|
||||
|
||||
$ xxd test/testdata/romeo.txt.flate
|
||||
00000000: 4d53 c16e db30 0cbd f32b d853 2e46 0e3d MS.n.0...+.S.F.=
|
||||
00000010: 2e87 20ed 0234 c5ba 0049 861e 861d 149b .. ..4...I......
|
||||
etc
|
||||
00000200: 7d13 8ffd b9a3 24bb 68f4 eb30 7a59 610d }.....$.h..0zYa.
|
||||
00000210: 7f01 ..
|
||||
|
||||
The gzip format wraps a variable length header (in this case, 20 bytes) and 8
|
||||
byte footer around the raw flate data. The header contains the NUL-terminated C
|
||||
string name of the original, uncompressed file, "romeo.txt", amongst other
|
||||
data. The footer contains a 4 byte CRC32 checksum and a 4 byte length of the
|
||||
uncompressed file (0x3ae = 942 bytes).
|
||||
|
||||
$ xxd test/testdata/romeo.txt.gz
|
||||
00000000: 1f8b 0808 26d8 5d59 0003 726f 6d65 6f2e ....&.]Y..romeo.
|
||||
00000010: 7478 7400 4d53 c16e db30 0cbd f32b d853 txt.MS.n.0...+.S
|
||||
etc
|
||||
00000210: de5d 2c67 7d13 8ffd b9a3 24bb 68f4 eb30 .],g}.....$.h..0
|
||||
00000220: 7a59 610d 7f01 ef07 e5ab ae03 0000 zYa...........
|
||||
|
||||
The zlib format wraps a 2 byte header and 4 byte footer around the raw flate
|
||||
data. The footer contains a 4 byte Adler32 checksum. TODO: move this to
|
||||
std/zlib/README.md.
|
||||
|
||||
$ xxd test/testdata/romeo.txt.zlib
|
||||
00000000: 789c 4d53 c16e db30 0cbd f32b d853 2e46 x.MS.n.0...+.S.F
|
||||
00000010: 0e3d 2e87 20ed 0234 c5ba 0049 861e 861d .=.. ..4...I....
|
||||
etc
|
||||
00000200: 2c67 7d13 8ffd b9a3 24bb 68f4 eb30 7a59 ,g}.....$.h..0zY
|
||||
00000210: 610d 7f01 57bb 3ede a...W.>.
|
||||
|
||||
|
||||
# Wire Format Worked Example
|
||||
|
||||
Consider `test/testdata/romeo.txt.flate`. The relevant spec is RFC 1951.
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000000 0x0000 M 0x4D 0b_0100_1101
|
||||
000001 0x0001 S 0x53 0b_0101_0011
|
||||
000002 0x0002 . 0xC1 0b_1100_0001
|
||||
000003 0x0003 n 0x6E 0b_0110_1110
|
||||
000004 0x0004 . 0xDB 0b_1101_1011
|
||||
000005 0x0005 0 0x30 0b_0011_0000
|
||||
000006 0x0006 . 0x0C 0b_0000_1100
|
||||
000007 0x0007 . 0xBD 0b_1011_1101
|
||||
000008 0x0008 . 0xF3 0b_1111_0011
|
||||
000009 0x0009 + 0x2B 0b_0010_1011
|
||||
000010 0x000A . 0xD8 0b_1101_1000
|
||||
000011 0x000B S 0x53 0b_0101_0011
|
||||
000012 0x000C . 0x2E 0b_0010_1110
|
||||
000013 0x000D F 0x46 0b_0100_0110
|
||||
000014 0x000E . 0x0E 0b_0000_1110
|
||||
000015 0x000F = 0x3D 0b_0011_1101
|
||||
000016 0x0010 . 0x2E 0b_0010_1110
|
||||
000017 0x0011 . 0x87 0b_1000_0111
|
||||
000018 0x0012 0x20 0b_0010_0000
|
||||
000019 0x0013 . 0xED 0b_1110_1101
|
||||
etc
|
||||
000522 0x020A . 0xEB 0b_1110_1011
|
||||
000523 0x020B 0 0x30 0b_0011_0000
|
||||
000524 0x020C z 0x7A 0b_0111_1010
|
||||
000525 0x020D Y 0x59 0b_0101_1001
|
||||
000526 0x020E a 0x61 0b_0110_0001
|
||||
000527 0x020F . 0x0D 0b_0000_1101
|
||||
000528 0x0210 . 0x7F 0b_0111_1111
|
||||
000529 0x0211 . 0x01 0b_0000_0001
|
||||
|
||||
Starting at the top:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000000 0x0000 M 0x4D 0b_0100_1101
|
||||
|
||||
As per the RFC section 3.2.3. Details of block format:
|
||||
|
||||
- BFINAL is the first (LSB) bit, 0b1. This block is the final block.
|
||||
- BTYPE is the next two bits in natural (MSB to LSB) order, 0b10, which means
|
||||
a block that's compressed with dynamic Huffman codes.
|
||||
|
||||
There are 3 block types: uncompressed, compressed with fixed Huffman codes and
|
||||
compressed with dynamic Huffman codes. The first type is straightforward:
|
||||
containing a uint16 length `N` (and its complement, for error detection) and
|
||||
then `N` literal bytes. The second type is the same as the third type but with
|
||||
built-in `lcode` and `dcode` tables (see below). The third type is the
|
||||
interesting part of the flate format, and its deconstruction continues below.
|
||||
|
||||
|
||||
## Dynamic Huffman Blocks
|
||||
|
||||
The bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000000 0x0000 M 0x4D 0b_0100_1...
|
||||
000001 0x0001 S 0x53 0b_0101_0011
|
||||
000002 0x0002 . 0xC1 0b_1100_0001
|
||||
|
||||
As per the RFC section 3.2.7. Compression with dynamic Huffman codes, three
|
||||
variables (5, 5 and 4 bits) are read in the same natural (MSB to LSB) order:
|
||||
|
||||
- HLIT = 0b01001 = 9
|
||||
- HDIST = 0b10011 = 19
|
||||
- HCLEN = 0b1010 = 10
|
||||
|
||||
Adding the implicit biases give:
|
||||
|
||||
nlit = 9 + 257 = 266
|
||||
ndist = 19 + 1 = 20
|
||||
nclen = 10 + 4 = 14
|
||||
|
||||
|
||||
## Code Length Code Lengths
|
||||
|
||||
The bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000002 0x0002 . 0xC1 0b_1100_000.
|
||||
000003 0x0003 n 0x6E 0b_0110_1110
|
||||
000004 0x0004 . 0xDB 0b_1101_1011
|
||||
000005 0x0005 0 0x30 0b_0011_0000
|
||||
000006 0x0006 . 0x0C 0b_0000_1100
|
||||
000007 0x0007 . 0xBD 0b_1011_1101
|
||||
|
||||
There are 19 possible code length code lengths (this is not a typo), but the
|
||||
wire format order is shuffled (in the idiosyncratic order: 16, 17, 18, 0, 8,
|
||||
..., 15) so that later elements are more likely to be zero (and hence
|
||||
compressible). There are `nclen` (in this case, 14) explicit code length code
|
||||
lengths, 3 bits each.
|
||||
|
||||
clcode_lengths[16] = 0b000 = 0
|
||||
clcode_lengths[17] = 0b100 = 4
|
||||
clcode_lengths[18] = 0b101 = 5
|
||||
clcode_lengths[ 0] = 0b011 = 3
|
||||
clcode_lengths[ 8] = 0b011 = 3
|
||||
clcode_lengths[ 7] = 0b011 = 3
|
||||
clcode_lengths[ 9] = 0b011 = 3
|
||||
clcode_lengths[ 6] = 0b011 = 3
|
||||
clcode_lengths[10] = 0b000 = 0
|
||||
clcode_lengths[ 5] = 0b011 = 3
|
||||
clcode_lengths[11] = 0b000 = 0
|
||||
clcode_lengths[ 4] = 0b011 = 3
|
||||
clcode_lengths[12] = 0b000 = 0
|
||||
clcode_lengths[ 3] = 0b101 = 5
|
||||
|
||||
The remaining (19 - `nclen`) = 5 entries are implicitly zero:
|
||||
|
||||
clcode_lengths[13] = 0
|
||||
clcode_lengths[ 2] = 0
|
||||
clcode_lengths[14] = 0
|
||||
clcode_lengths[ 1] = 0
|
||||
clcode_lengths[15] = 0
|
||||
|
||||
Undoing the shuffle gives:
|
||||
|
||||
clcode_lengths[ 0] = 3
|
||||
clcode_lengths[ 1] = 0
|
||||
clcode_lengths[ 2] = 0
|
||||
clcode_lengths[ 3] = 5
|
||||
clcode_lengths[ 4] = 3
|
||||
clcode_lengths[ 5] = 3
|
||||
clcode_lengths[ 6] = 3
|
||||
clcode_lengths[ 7] = 3
|
||||
clcode_lengths[ 8] = 3
|
||||
clcode_lengths[ 9] = 3
|
||||
clcode_lengths[10] = 0
|
||||
clcode_lengths[11] = 0
|
||||
clcode_lengths[12] = 0
|
||||
clcode_lengths[13] = 0
|
||||
clcode_lengths[14] = 0
|
||||
clcode_lengths[15] = 0
|
||||
clcode_lengths[16] = 0
|
||||
clcode_lengths[17] = 4
|
||||
clcode_lengths[18] = 5
|
||||
|
||||
For `clcode`s, there are 7 + 1 + 2 = 10 non-zero entries: 7 3-bit codes, 1
|
||||
4-bit code and 2 5-bit codes. The canonical Huffman encoding's map from bits to
|
||||
`clcode`s is:
|
||||
|
||||
bits clcode
|
||||
0b000 0
|
||||
0b001 4
|
||||
0b010 5
|
||||
0b011 6
|
||||
0b100 7
|
||||
0b101 8
|
||||
0b110 9
|
||||
0b1110 17
|
||||
0b11110 3
|
||||
0b11111 18
|
||||
|
||||
Call that Huffman table `H-CL`.
|
||||
|
||||
|
||||
## Lcodes and Dcodes
|
||||
|
||||
The bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000007 0x0007 . 0xBD 0b_1011_1...
|
||||
000008 0x0008 . 0xF3 0b_1111_0011
|
||||
000009 0x0009 + 0x2B 0b_0010_1011
|
||||
000010 0x000A . 0xD8 0b_1101_1000
|
||||
000011 0x000B S 0x53 0b_0101_0011
|
||||
000012 0x000C . 0x2E 0b_0010_1110
|
||||
000013 0x000D F 0x46 0b_0100_0110
|
||||
|
||||
When decoding via a Huffman table, bits are read in LSB to MSB order,
|
||||
right-to-left in this debugging output. Extra bits, if any, are then read in
|
||||
the other, natural order (MSB to LSB). Decoding via `H-CL` gives:
|
||||
|
||||
bits clcode
|
||||
0b1110 17, 3 extra bits = 0b111 = 7
|
||||
0b001 4
|
||||
0b11111 18, 7 extra bits = 0b0001010 = 10
|
||||
0b001 4
|
||||
0b101 8
|
||||
0b1110 17, 3 extra bits = 0b010 = 2
|
||||
0b100 7
|
||||
0b1110 17, 3 extra bits = 0b001 = 1
|
||||
0b011 6
|
||||
0b000 0
|
||||
|
||||
Still in the RFC section 3.2.7., this means:
|
||||
|
||||
lcode_lengths has 3 + 7 = 10 consecutive zeroes
|
||||
lcode_lengths[ 10] = 4
|
||||
lcode_lengths has 11 + 10 = 21 consecutive zeroes
|
||||
lcode_lengths[ 32] = 4
|
||||
lcode_lengths[ 33] = 8
|
||||
lcode_lengths has 3 + 2 = 5 consecutive zeroes
|
||||
lcode_lengths[ 39] = 7
|
||||
lcode_lengths has 3 + 1 = 4 consecutive zeroes
|
||||
lcode_lengths[ 44] = 6
|
||||
lcode_lengths[ 45] = 0
|
||||
|
||||
After decoding the first 10 code lengths, the bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000013 0x000D F 0x46 0b_01.._....
|
||||
|
||||
Continuing to read all `nlit` (266) `lcode` lengths and `ndist` (20) `dcode`
|
||||
lengths (zeroes are not shown here):
|
||||
|
||||
lcode_lengths[ 10] = 4
|
||||
lcode_lengths[ 32] = 4
|
||||
lcode_lengths[ 33] = 8
|
||||
lcode_lengths[ 39] = 7
|
||||
lcode_lengths[ 44] = 6
|
||||
lcode_lengths[ 46] = 7
|
||||
lcode_lengths[ 50] = 8
|
||||
lcode_lengths[ 58] = 9
|
||||
lcode_lengths[ 59] = 7
|
||||
lcode_lengths[ 63] = 7
|
||||
etc
|
||||
lcode_lengths[264] = 9
|
||||
lcode_lengths[265] = 9
|
||||
|
||||
and
|
||||
|
||||
dcode_lengths[ 5] = 6
|
||||
dcode_lengths[ 6] = 5
|
||||
dcode_lengths[ 8] = 4
|
||||
dcode_lengths[ 9] = 5
|
||||
dcode_lengths[10] = 3
|
||||
etc
|
||||
dcode_lengths[18] = 3
|
||||
dcode_lengths[19] = 6
|
||||
|
||||
The `H-CL` table is no longer used from here onwards.
|
||||
|
||||
For `lcode`s, there are 6 + 9 + 10 + 16 + 10 + 12 = 63 non-zero entries: 6
|
||||
4-bit codes, 9 5-bit codes, 10 6-bit codes, 16 7-bit codes, 10 8-bit codes and
|
||||
12 9-bit codes. The canonical Huffman encoding's map from bits to `lcode`
|
||||
values is:
|
||||
|
||||
bits lcode
|
||||
0b0000 10
|
||||
0b0001 32
|
||||
0b0010 101
|
||||
0b0011 116
|
||||
0b0100 257
|
||||
0b0101 259
|
||||
0b01100 97
|
||||
0b01101 104
|
||||
0b01110 105
|
||||
0b01111 110
|
||||
0b10000 111
|
||||
0b10001 114
|
||||
0b10010 115
|
||||
0b10011 258
|
||||
0b10100 260
|
||||
0b101010 44
|
||||
0b101011 99
|
||||
0b101100 100
|
||||
0b101101 102
|
||||
0b101110 108
|
||||
0b101111 109
|
||||
0b110000 112
|
||||
0b110001 117
|
||||
0b110010 119
|
||||
0b110011 121
|
||||
0b1101000 39
|
||||
0b1101001 46
|
||||
0b1101010 59
|
||||
0b1101011 63
|
||||
0b1101100 65
|
||||
0b1101101 69
|
||||
0b1101110 73
|
||||
0b1101111 79
|
||||
0b1110000 82
|
||||
0b1110001 83
|
||||
0b1110010 84
|
||||
0b1110011 98
|
||||
0b1110100 103
|
||||
0b1110101 261
|
||||
0b1110110 262
|
||||
0b1110111 263
|
||||
0b11110000 33
|
||||
0b11110001 50
|
||||
0b11110010 66
|
||||
0b11110011 67
|
||||
0b11110100 72
|
||||
0b11110101 74
|
||||
0b11110110 77
|
||||
0b11110111 87
|
||||
0b11111000 107
|
||||
0b11111001 118
|
||||
0b111110100 58
|
||||
0b111110101 68
|
||||
0b111110110 76
|
||||
0b111110111 78
|
||||
0b111111000 85
|
||||
0b111111001 91
|
||||
0b111111010 93
|
||||
0b111111011 120
|
||||
0b111111100 122
|
||||
0b111111101 256
|
||||
0b111111110 264
|
||||
0b111111111 265
|
||||
|
||||
Call that Huffman table `H-L`.
|
||||
|
||||
For `dcode`s, there are 5 + 4 + 3 + 2 = 14 non-zero entries: 5 3-bit codes, 4
|
||||
4-bit codes, 3 5-bit codes and 2 6-bit codes. The canonical Huffman encoding's
|
||||
map from bits to `dcode` values is:
|
||||
|
||||
bits dcode
|
||||
0b000 10
|
||||
0b001 14
|
||||
0b010 16
|
||||
0b011 17
|
||||
0b100 18
|
||||
0b1010 8
|
||||
0b1011 12
|
||||
0b1100 13
|
||||
0b1101 15
|
||||
0b11100 6
|
||||
0b11101 9
|
||||
0b11110 11
|
||||
0b111110 5
|
||||
0b111111 19
|
||||
|
||||
Call that Huffman table `H-D`.
|
||||
|
||||
|
||||
## Decoding Literals
|
||||
|
||||
The bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000052 0x0034 . 0xE7 0b_1..._....
|
||||
000053 0x0035 C 0x43 0b_0100_0011
|
||||
000054 0x0036 . 0xE8 0b_1110_1000
|
||||
000055 0x0037 ) 0x29 0b_0010_1001
|
||||
000056 0x0038 . 0xA0 0b_1010_0000
|
||||
000057 0x0039 . 0xF1 0b_1111_0001
|
||||
|
||||
Decoding from that bit stream via `H-L` gives some literal bytes (where the
|
||||
`lcode` value is < 256):
|
||||
|
||||
bits lcode
|
||||
0b1110000 82 (ASCII 'R')
|
||||
0b10000 111 (ASCII 'o')
|
||||
0b101111 109 (ASCII 'm')
|
||||
0b0010 101 (ASCII 'e')
|
||||
0b10000 111 (ASCII 'o')
|
||||
0b0001 32 (ASCII ' ')
|
||||
0b01100 97 (ASCII 'a')
|
||||
0b01111 110 (ASCII 'n')
|
||||
|
||||
|
||||
## Decoding Back-References
|
||||
|
||||
This continues, up until we get to the "O Romeo, Romeo! wherefore art thou
|
||||
Romeo?" line.
|
||||
|
||||
The bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000089 0x0059 . 0xC1 0b_11.._....
|
||||
000090 0x005A . 0x1E 0b_0001_1110
|
||||
000091 0x005B . 0x0F 0b_0000_1111
|
||||
000092 0x005C . 0xF9 0b_1111_1001
|
||||
|
||||
Decoding from that bit stream via `H-L` gives:
|
||||
|
||||
bits lcode
|
||||
0b1101111 79 (ASCII 'O')
|
||||
0b0001 32 (ASCII ' ')
|
||||
0b1110000 82 (ASCII 'R')
|
||||
0b10011 258
|
||||
|
||||
This 258 is our first non-literal `lcode`, the start of a length-distance
|
||||
back-reference. We first need to decode the length. The RFC section 3.2.5.
|
||||
Compressed blocks (length and distance codes) says that an `lcode` of 258 means
|
||||
that length=4 with no extra bits.
|
||||
|
||||
The bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000092 0x005C . 0xF9 0b_111._....
|
||||
000093 0x005D Y 0x59 0b_0101_1001
|
||||
|
||||
We next decode via `H-D` instead of `H-L`.
|
||||
|
||||
bits dcode
|
||||
0b11110 11
|
||||
|
||||
Again, from section 3.2.5., a `dcode` of 11 means a distance in the range [49,
|
||||
64], 49 plus 4 extra bits. The next 4 bits are 0b0110=6, so the distance is 55.
|
||||
We therefore copy length=4 bytes from distance=55 bytes ago: "omeo".
|
||||
|
||||
The bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000093 0x005D Y 0x59 0b_01.._....
|
||||
000094 0x005E U 0x55 0b_0101_0101
|
||||
000095 0x005F > 0x3E 0b_0011_1110
|
||||
|
||||
We go back to decoding via `H-L`, which gives
|
||||
|
||||
bits lcode
|
||||
0b101010 44 (ASCII ',')
|
||||
0b10100 260
|
||||
|
||||
This is another non-literal. Section 3.2.5. says that an `lcode` of 260 means
|
||||
length=6 with no extra bits.
|
||||
|
||||
The bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000095 0x005F > 0x3E 0b_0011_111.
|
||||
|
||||
Decode with `H-D`.
|
||||
|
||||
bits dcode
|
||||
0b111110 5
|
||||
|
||||
Again, from section 3.2.5., a `dcode` of 5 means a distance in the range [7,
|
||||
8], 7 plus 1 extra bit. The next 1 bits are 0b0=0, so the distance is 7. We
|
||||
therefore copy length=6 bytes from distance=7 bytes ago: " Romeo".
|
||||
|
||||
The bit stream is now at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000096 0x0060 . 0x0F 0b_0000_1111
|
||||
|
||||
We go back to decoding via `H-L`, which gives
|
||||
|
||||
bits lcode
|
||||
0b11110000 33 (ASCII '!')
|
||||
|
||||
And on we go.
|
||||
|
||||
|
||||
## End of Block
|
||||
|
||||
The block finishes with the bit stream at:
|
||||
|
||||
offset xoffset ASCII hex binary
|
||||
000522 0x020A . 0xEB 0b_1110_101.
|
||||
000523 0x020B 0 0x30 0b_0011_0000
|
||||
000524 0x020C z 0x7A 0b_0111_1010
|
||||
000525 0x020D Y 0x59 0b_0101_1001
|
||||
000526 0x020E a 0x61 0b_0110_0001
|
||||
000527 0x020F . 0x0D 0b_0000_1101
|
||||
000528 0x0210 . 0x7F 0b_0111_1111
|
||||
000529 0x0211 . 0x01 0b_0000_0001
|
||||
|
||||
The decoding is:
|
||||
|
||||
bits lcode
|
||||
0b101011 99 (ASCII 'c')
|
||||
0b10000 111 (ASCII 'o')
|
||||
0b110001 117 (ASCII 'u')
|
||||
0b01111 110 (ASCII 'n')
|
||||
0b0100 257 (length=3 + 0_extra_bits...)
|
||||
|
||||
bits dcode
|
||||
0b1101 15 (distance=193 + 6_extra_bits, 0b000010 in MSB to LSB order;
|
||||
length=3, distance=195: copy "sel")
|
||||
|
||||
bits lcode
|
||||
0b1101011 63 (ASCII '?')
|
||||
0b0000 10 (ASCII '\n')
|
||||
0b111111101 256 (end of block)
|
||||
|
||||
In other words, the block ends with "counsel?\n".
|
||||
@@ -0,0 +1,54 @@
|
||||
// Copyright 2017 The Wuffs Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
// code_order is defined in the RFC section 3.2.7.
|
||||
pri const code_order[19] u8[..18] = $(
|
||||
16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15,
|
||||
)
|
||||
|
||||
// reverse8 reverses the bits in a byte.
|
||||
pri const reverse8[256] u8 = $(
|
||||
0x00, 0x80, 0x40, 0xC0, 0x20, 0xA0, 0x60, 0xE0, // 0x00 - 0x07
|
||||
0x10, 0x90, 0x50, 0xD0, 0x30, 0xB0, 0x70, 0xF0, // 0x08 - 0x0F
|
||||
0x08, 0x88, 0x48, 0xC8, 0x28, 0xA8, 0x68, 0xE8, // 0x10 - 0x17
|
||||
0x18, 0x98, 0x58, 0xD8, 0x38, 0xB8, 0x78, 0xF8, // 0x18 - 0x1F
|
||||
0x04, 0x84, 0x44, 0xC4, 0x24, 0xA4, 0x64, 0xE4, // 0x20 - 0x27
|
||||
0x14, 0x94, 0x54, 0xD4, 0x34, 0xB4, 0x74, 0xF4, // 0x28 - 0x2F
|
||||
0x0C, 0x8C, 0x4C, 0xCC, 0x2C, 0xAC, 0x6C, 0xEC, // 0x30 - 0x37
|
||||
0x1C, 0x9C, 0x5C, 0xDC, 0x3C, 0xBC, 0x7C, 0xFC, // 0x38 - 0x3F
|
||||
0x02, 0x82, 0x42, 0xC2, 0x22, 0xA2, 0x62, 0xE2, // 0x40 - 0x47
|
||||
0x12, 0x92, 0x52, 0xD2, 0x32, 0xB2, 0x72, 0xF2, // 0x48 - 0x4F
|
||||
0x0A, 0x8A, 0x4A, 0xCA, 0x2A, 0xAA, 0x6A, 0xEA, // 0x50 - 0x57
|
||||
0x1A, 0x9A, 0x5A, 0xDA, 0x3A, 0xBA, 0x7A, 0xFA, // 0x58 - 0x5F
|
||||
0x06, 0x86, 0x46, 0xC6, 0x26, 0xA6, 0x66, 0xE6, // 0x60 - 0x67
|
||||
0x16, 0x96, 0x56, 0xD6, 0x36, 0xB6, 0x76, 0xF6, // 0x68 - 0x6F
|
||||
0x0E, 0x8E, 0x4E, 0xCE, 0x2E, 0xAE, 0x6E, 0xEE, // 0x70 - 0x77
|
||||
0x1E, 0x9E, 0x5E, 0xDE, 0x3E, 0xBE, 0x7E, 0xFE, // 0x78 - 0x7F
|
||||
0x01, 0x81, 0x41, 0xC1, 0x21, 0xA1, 0x61, 0xE1, // 0x80 - 0x87
|
||||
0x11, 0x91, 0x51, 0xD1, 0x31, 0xB1, 0x71, 0xF1, // 0x88 - 0x8F
|
||||
0x09, 0x89, 0x49, 0xC9, 0x29, 0xA9, 0x69, 0xE9, // 0x90 - 0x97
|
||||
0x19, 0x99, 0x59, 0xD9, 0x39, 0xB9, 0x79, 0xF9, // 0x98 - 0x9F
|
||||
0x05, 0x85, 0x45, 0xC5, 0x25, 0xA5, 0x65, 0xE5, // 0xA0 - 0xA7
|
||||
0x15, 0x95, 0x55, 0xD5, 0x35, 0xB5, 0x75, 0xF5, // 0xA8 - 0xAF
|
||||
0x0D, 0x8D, 0x4D, 0xCD, 0x2D, 0xAD, 0x6D, 0xED, // 0xB0 - 0xB7
|
||||
0x1D, 0x9D, 0x5D, 0xDD, 0x3D, 0xBD, 0x7D, 0xFD, // 0xB8 - 0xBF
|
||||
0x03, 0x83, 0x43, 0xC3, 0x23, 0xA3, 0x63, 0xE3, // 0xC0 - 0xC7
|
||||
0x13, 0x93, 0x53, 0xD3, 0x33, 0xB3, 0x73, 0xF3, // 0xC8 - 0xCF
|
||||
0x0B, 0x8B, 0x4B, 0xCB, 0x2B, 0xAB, 0x6B, 0xEB, // 0xD0 - 0xD7
|
||||
0x1B, 0x9B, 0x5B, 0xDB, 0x3B, 0xBB, 0x7B, 0xFB, // 0xD8 - 0xDF
|
||||
0x07, 0x87, 0x47, 0xC7, 0x27, 0xA7, 0x67, 0xE7, // 0xE0 - 0xE7
|
||||
0x17, 0x97, 0x57, 0xD7, 0x37, 0xB7, 0x77, 0xF7, // 0xE8 - 0xEF
|
||||
0x0F, 0x8F, 0x4F, 0xCF, 0x2F, 0xAF, 0x6F, 0xEF, // 0xF0 - 0xF7
|
||||
0x1F, 0x9F, 0x5F, 0xDF, 0x3F, 0xBF, 0x7F, 0xFF, // 0xF8 - 0xFF
|
||||
)
|
||||
@@ -0,0 +1,757 @@
|
||||
// Copyright 2017 The Wuffs Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
packageid "flte"
|
||||
|
||||
// TODO: rename "struct flate_decoder" to just "struct decoder", to avoid
|
||||
// stutter in the fully qualified C name such as "wuffs_flate__flate_decoder".
|
||||
|
||||
pub error "bad Huffman code (over-subscribed)"
|
||||
pub error "bad Huffman code (under-subscribed)"
|
||||
pub error "bad Huffman code length count"
|
||||
pub error "bad Huffman code length repetition"
|
||||
pub error "bad Huffman code"
|
||||
pub error "bad Huffman minimum code length"
|
||||
pub error "bad distance"
|
||||
pub error "bad distance code count"
|
||||
pub error "bad flate block"
|
||||
pub error "bad literal/length code count"
|
||||
pub error "inconsistent stored block length"
|
||||
pub error "internal error: inconsistent Huffman decoder state"
|
||||
pub error "internal error: inconsistent Huffman end_of_block"
|
||||
pub error "internal error: inconsistent distance"
|
||||
pub error "internal error: inconsistent n_bits"
|
||||
pub error "missing end-of-block code"
|
||||
pub error "no Huffman codes"
|
||||
|
||||
// The next two tables were created by script/print-flate-magic-numbers.go.
|
||||
//
|
||||
// The u32 values' meanings are the same as the flate_decoder.huffs u32 values.
|
||||
// In particular, bit 30 indicates a base number + extra bits, bits 23-8 are
|
||||
// the base number and bits 7-4 are the number of those extra bits.
|
||||
//
|
||||
// Some trailing elements are 0x08000000. Bit 27 indicates an invalid value.
|
||||
|
||||
pri const lcode_magic_numbers[32] u32 = $(
|
||||
0x40000300, 0x40000400, 0x40000500, 0x40000600, 0x40000700, 0x40000800, 0x40000900, 0x40000A00,
|
||||
0x40000B10, 0x40000D10, 0x40000F10, 0x40001110, 0x40001320, 0x40001720, 0x40001B20, 0x40001F20,
|
||||
0x40002330, 0x40002B30, 0x40003330, 0x40003B30, 0x40004340, 0x40005340, 0x40006340, 0x40007340,
|
||||
0x40008350, 0x4000A350, 0x4000C350, 0x4000E350, 0x40010200, 0x08000000, 0x08000000, 0x08000000,
|
||||
)
|
||||
|
||||
pri const dcode_magic_numbers[32] u32 = $(
|
||||
0x40000100, 0x40000200, 0x40000300, 0x40000400, 0x40000510, 0x40000710, 0x40000920, 0x40000D20,
|
||||
0x40001130, 0x40001930, 0x40002140, 0x40003140, 0x40004150, 0x40006150, 0x40008160, 0x4000C160,
|
||||
0x40010170, 0x40018170, 0x40020180, 0x40030180, 0x40040190, 0x40060190, 0x400801A0, 0x400C01A0,
|
||||
0x401001B0, 0x401801B0, 0x402001C0, 0x403001C0, 0x404001D0, 0x406001D0, 0x08000000, 0x08000000,
|
||||
)
|
||||
|
||||
pub struct flate_decoder?(
|
||||
// These fields yield src's bits in Least Significant Bits order.
|
||||
bits u32,
|
||||
n_bits u32,
|
||||
|
||||
// huffs and n_huffs_bits are the lookup tables for Huffman decodings.
|
||||
//
|
||||
// There are up to 2 Huffman decoders active at any one time. As per this
|
||||
// package's README.md:
|
||||
// - huffs[0] is used for clcode and lcode.
|
||||
// - huffs[1] is used for dcode.
|
||||
//
|
||||
// The initial table key is the low n_huffs_bits of the flate_decoder.bits
|
||||
// field. Keys longer than 9 bits require a two step lookup, the first step
|
||||
// examines the low 9 bits, the second step examines the remaining bits.
|
||||
// Two steps are required at most, as keys are at most 15 bits long.
|
||||
//
|
||||
// Using flate_decoder.bits's low n_huffs_bits as a table key is valid even
|
||||
// if flate_decoder.n_bits is less than n_huffs_bits, because the immediate
|
||||
// next step after indexing the table by the key is to compare
|
||||
// flate_decoder.n_bits to the table value's number of flate_decoder.bits
|
||||
// to consume. If it compares less, then more source bytes are read and the
|
||||
// table lookup re-tried.
|
||||
//
|
||||
// The table value is:
|
||||
// - bit 31 indicates the symbol is a literal.
|
||||
// - bit 30 indicates the symbol is a base number + extra bits.
|
||||
// - bit 29 indicates the symbol is the end-of-block.
|
||||
// - bit 28 indicates a redirect to another part of the table.
|
||||
// - bit 27 indicates an invalid value.
|
||||
// - bits 26-24 are zero.
|
||||
// - bits 23- 8 is the redirect offset, literal (in bits 15-8) or base number.
|
||||
// - bits 7- 4 is the redirected table's bits or number of extra bits.
|
||||
// - bits 3- 0 are the number of flate_decoder.bits to consume.
|
||||
//
|
||||
// Exactly one of the eight bits 31-24 should be set.
|
||||
//
|
||||
// TODO: replace the magic "big enough" 1234 with something more
|
||||
// principled, perhaps discovered via an exhaustive search.
|
||||
huffs[2][1234] u32,
|
||||
n_huffs_bits[2] u32[..9],
|
||||
|
||||
// history holds up to the last 32KiB of decoded output, if the decoding
|
||||
// was incomplete (e.g. due to a short read or write). RFC 1951 (DEFLATE)
|
||||
// gives the maximum distance in a length-distance back-reference as 32768,
|
||||
// or 0x8000.
|
||||
//
|
||||
// It is a ringbuffer, so that the most distant byte in the decoding isn't
|
||||
// necessarily history[0]. The ringbuffer is full (i.e. it holds 32KiB of
|
||||
// history) if and only if history_index >= 0x8000.
|
||||
//
|
||||
// history[history_index & 0x7FFF] is where the next byte of decoded output
|
||||
// will be written.
|
||||
history[0x8000] u8, // 32 KiB.
|
||||
history_index u32,
|
||||
|
||||
// code_lengths is used to pass out-of-band data to init_huff.
|
||||
//
|
||||
// code_lengths[in.n_codes0 + i] holds the number of bits in the i'th code.
|
||||
code_lengths[320] u8[..15],
|
||||
|
||||
// end_of_block is whether decode_huffman_xxx saw an end-of-block code.
|
||||
//
|
||||
// TODO: can decode_huffman_xxx signal this in band instead of out of band?
|
||||
end_of_block bool,
|
||||
)
|
||||
|
||||
pub func flate_decoder.decode?(dst writer1, src reader1)() {
|
||||
while true {
|
||||
in.dst.mark()
|
||||
var z status = try this.decode_blocks?(dst:in.dst, src:in.src)
|
||||
if not z.is_suspension() {
|
||||
return z
|
||||
}
|
||||
// TODO: should "since_mark" be "since_mark!", as the return value lets
|
||||
// you modify the state of in.dst, so future mutations (via the slice)
|
||||
// can change the veracity of any in.dst assertions?
|
||||
var written[] u8 = in.dst.since_mark()
|
||||
// Append written, the decoded output, to the history ringbuffer.
|
||||
if written.length() >= 0x8000 {
|
||||
// If written is longer than the ringbuffer, we can ignore the
|
||||
// previous value of history_index, as we will overwrite the whole
|
||||
// ringbuffer.
|
||||
written = written.suffix(up_to:0x8000)
|
||||
this.history[:].copy_from_slice(s:written)
|
||||
this.history_index = 0x8000
|
||||
} else {
|
||||
// Otherwise, append written to the history ringbuffer starting at
|
||||
// the previous history_index (modulo 0x8000).
|
||||
var n_copied u64 = this.history[this.history_index & 0x7FFF:].copy_from_slice(s:written)
|
||||
if n_copied < written.length() {
|
||||
// a_slice.copy_from(s:b_slice) returns the minimum of the two
|
||||
// slice lengths. If that value is less than b_slice.length(),
|
||||
// then not all of b_slice was copied.
|
||||
//
|
||||
// In terms of the history ringbuffer, that means that we have
|
||||
// to wrap around and copy the remainder of written over the
|
||||
// start of the history ringbuffer.
|
||||
written = written[n_copied:]
|
||||
n_copied = this.history[:].copy_from_slice(s:written)
|
||||
// Set history_index (modulo 0x8000) to the length of this
|
||||
// remainder. The &0x7FFF is redundant, but proves to the
|
||||
// compiler that the conversion to u32 will not overflow. The
|
||||
// +0x8000 is to maintain that the history ringbuffer is full
|
||||
// if and only if history_index >= 0x8000.
|
||||
this.history_index = ((n_copied & 0x7FFF) as u32) + 0x8000
|
||||
} else {
|
||||
// We didn't need to wrap around.
|
||||
var already_full u32[..0x8000]
|
||||
if this.history_index >= 0x8000 {
|
||||
already_full = 0x8000
|
||||
}
|
||||
this.history_index = (this.history_index & 0x7FFF) + ((n_copied & 0x7FFF) as u32) + already_full
|
||||
}
|
||||
}
|
||||
yield z
|
||||
}
|
||||
}
|
||||
|
||||
pri func flate_decoder.decode_blocks?(dst writer1, src reader1)() {
|
||||
var final u32
|
||||
while final == 0 {
|
||||
while this.n_bits < 3,
|
||||
post this.n_bits >= 3,
|
||||
{
|
||||
this.bits |= (in.src.read_u8?() as u32) << this.n_bits
|
||||
this.n_bits += 8
|
||||
}
|
||||
final = this.bits & 0x01
|
||||
var type u32 = (this.bits >> 1) & 0x03
|
||||
this.bits >>= 3
|
||||
this.n_bits -= 3
|
||||
|
||||
if type == 0 {
|
||||
this.decode_uncompressed?(dst:in.dst, src:in.src)
|
||||
continue
|
||||
} else if type == 1 {
|
||||
this.init_fixed_huffman?()
|
||||
} else if type == 2 {
|
||||
this.init_dynamic_huffman?(src:in.src)
|
||||
} else {
|
||||
return error "bad flate block"
|
||||
}
|
||||
|
||||
this.end_of_block = false
|
||||
this.decode_huffman_fast?(dst:in.dst, src:in.src)
|
||||
if this.end_of_block {
|
||||
continue
|
||||
}
|
||||
this.decode_huffman_slow?(dst:in.dst, src:in.src)
|
||||
if this.end_of_block {
|
||||
continue
|
||||
}
|
||||
return error "internal error: inconsistent Huffman end_of_block"
|
||||
}
|
||||
}
|
||||
|
||||
// decode_uncompressed decodes an uncompresed block as per the RFC section
|
||||
// 3.2.4.
|
||||
pri func flate_decoder.decode_uncompressed?(dst writer1, src reader1)() {
|
||||
// TODO: make this "if" into a function invariant?
|
||||
//
|
||||
// Ditto for decode_huffman_slow and decode_huffman_fast.
|
||||
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
|
||||
return error "internal error: inconsistent n_bits"
|
||||
}
|
||||
this.n_bits = 0
|
||||
this.bits = 0
|
||||
|
||||
var length u32 = in.src.read_u32le?()
|
||||
if (length.low_bits(n:16) + length.high_bits(n:16)) != 0xFFFF {
|
||||
return error "inconsistent stored block length"
|
||||
}
|
||||
length = length.low_bits(n:16)
|
||||
while true {
|
||||
var n_copied u32 = in.dst.copy_from_reader32(r:in.src, length:length)
|
||||
if length <= n_copied {
|
||||
length = 0
|
||||
break
|
||||
}
|
||||
length -= n_copied
|
||||
if in.dst.available() == 0 {
|
||||
// TODO: "closed for writes" instead?
|
||||
yield suspension "short write"
|
||||
} else {
|
||||
yield suspension "short read"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// init_fixed_huffman initializes this.huffs as per the RFC section 3.2.6.
|
||||
pri func flate_decoder.init_fixed_huffman?()() {
|
||||
var i u32
|
||||
while i < 144 {
|
||||
this.code_lengths[i] = 8
|
||||
i += 1
|
||||
}
|
||||
while i < 256 {
|
||||
this.code_lengths[i] = 9
|
||||
i += 1
|
||||
}
|
||||
while i < 280 {
|
||||
this.code_lengths[i] = 7
|
||||
i += 1
|
||||
}
|
||||
while i < 288 {
|
||||
this.code_lengths[i] = 8
|
||||
i += 1
|
||||
}
|
||||
while i < 320 {
|
||||
this.code_lengths[i] = 5
|
||||
i += 1
|
||||
}
|
||||
this.init_huff?(which:0, n_codes0:0, n_codes1:288, base_symbol:257)
|
||||
this.init_huff?(which:1, n_codes0:288, n_codes1:320, base_symbol:0)
|
||||
}
|
||||
|
||||
// init_dynamic_huffman initializes this.huffs as per the RFC section 3.2.7.
|
||||
pri func flate_decoder.init_dynamic_huffman?(src reader1)() {
|
||||
var bits u32 = this.bits
|
||||
var n_bits u32 = this.n_bits
|
||||
while n_bits < 14,
|
||||
post n_bits >= 14,
|
||||
{
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
var n_lit u32[257..288] = bits.low_bits(n:5) + 257
|
||||
if n_lit > 286 {
|
||||
return error "bad literal/length code count"
|
||||
}
|
||||
bits >>= 5
|
||||
var n_dist u32[1..32] = bits.low_bits(n:5) + 1
|
||||
if n_dist > 30 {
|
||||
return error "bad distance code count"
|
||||
}
|
||||
bits >>= 5
|
||||
var n_clen u32[4..19] = bits.low_bits(n:4) + 4
|
||||
bits >>= 4
|
||||
n_bits -= 14
|
||||
|
||||
// Read the clcode Huffman table: H-CL.
|
||||
var i u32
|
||||
while i < n_clen {
|
||||
while n_bits < 3,
|
||||
inv i < n_clen,
|
||||
post n_bits >= 3,
|
||||
{
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
assert i < 19 via "a < b: a < c; c <= b"(c:n_clen)
|
||||
this.code_lengths[code_order[i]] = (bits & 0x07) as u8
|
||||
bits >>= 3
|
||||
n_bits -= 3
|
||||
i += 1
|
||||
}
|
||||
while i < 19 {
|
||||
this.code_lengths[code_order[i]] = 0
|
||||
i += 1
|
||||
}
|
||||
this.init_huff?(which:0, n_codes0:0, n_codes1:19, base_symbol:0xFFF)
|
||||
|
||||
// Decode the code lengths for the next two Huffman tables.
|
||||
var mask u32[..511] = ((1 as u32) << this.n_huffs_bits[0]) - 1
|
||||
i = 0
|
||||
while i < (n_lit + n_dist) {
|
||||
assert i < (288 + 32) via "a < (b + c): a < (b0 + c0); b0 <= b; c0 <= c"(b0:n_lit, c0:n_dist)
|
||||
|
||||
// Decode a clcode symbol from H-CL.
|
||||
var table_entry u32
|
||||
while true,
|
||||
inv i < 320,
|
||||
{
|
||||
table_entry = this.huffs[0][bits & mask]
|
||||
var table_entry_n_bits u32[..15] = table_entry & 15
|
||||
if n_bits >= table_entry_n_bits {
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
break
|
||||
}
|
||||
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
// For H-CL, there should be no redirections and all symbols should be
|
||||
// literals.
|
||||
if (table_entry >> 24) != 0x80 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
table_entry = (table_entry >> 8) & 0xFF
|
||||
|
||||
// Write a literal code length.
|
||||
if table_entry < 16 {
|
||||
this.code_lengths[i] = table_entry as u8
|
||||
i += 1
|
||||
continue
|
||||
}
|
||||
|
||||
// Write a repeated code length.
|
||||
var n_extra_bits u32[..7]
|
||||
var rep_symbol u8[..15]
|
||||
var rep_count u32
|
||||
if table_entry == 16 {
|
||||
n_extra_bits = 2
|
||||
if i <= 0 {
|
||||
return error "bad Huffman code length repetition"
|
||||
}
|
||||
rep_symbol = this.code_lengths[i - 1]
|
||||
rep_count = 3
|
||||
assert rep_count <= 11
|
||||
} else if table_entry == 17 {
|
||||
n_extra_bits = 3
|
||||
rep_symbol = 0
|
||||
rep_count = 3
|
||||
assert rep_count <= 11
|
||||
} else if table_entry == 18 {
|
||||
n_extra_bits = 7
|
||||
rep_symbol = 0
|
||||
rep_count = 11
|
||||
assert rep_count <= 11
|
||||
} else {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
while n_bits < n_extra_bits,
|
||||
inv i < 320,
|
||||
inv rep_count <= 11,
|
||||
post n_bits >= n_extra_bits,
|
||||
{
|
||||
assert n_bits < 7 via "a < b: a < c; c <= b"(c:n_extra_bits)
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
rep_count += bits.low_bits(n:n_extra_bits)
|
||||
bits >>= n_extra_bits
|
||||
n_bits -= n_extra_bits
|
||||
|
||||
while rep_count > 0 {
|
||||
// TODO: hoist this check up one level?
|
||||
if i >= (n_lit + n_dist) {
|
||||
return error "bad Huffman code length count"
|
||||
}
|
||||
assert i < (288 + 32) via "a < (b + c): a < (b0 + c0); b0 <= b; c0 <= c"(b0:n_lit, c0:n_dist)
|
||||
this.code_lengths[i] = rep_symbol
|
||||
i += 1
|
||||
rep_count -= 1
|
||||
}
|
||||
}
|
||||
|
||||
if i != (n_lit + n_dist) {
|
||||
return error "bad Huffman code length count"
|
||||
}
|
||||
if this.code_lengths[256] == 0 {
|
||||
return error "missing end-of-block code"
|
||||
}
|
||||
|
||||
this.init_huff?(which:0, n_codes0:0, n_codes1:n_lit, base_symbol:257)
|
||||
this.init_huff?(which:1, n_codes0:n_lit, n_codes1:n_lit + n_dist, base_symbol:0)
|
||||
|
||||
this.bits = bits
|
||||
this.n_bits = n_bits
|
||||
}
|
||||
|
||||
// TODO: make named constants for 15, 19, 319, etc.
|
||||
|
||||
// TODO: skip generating WUFFS_COROUTINE_SUSPENSION_POINTs for *callers* of
|
||||
// this function, as it can only return errors, not suspensions.
|
||||
|
||||
pri func flate_decoder.init_huff?(which u32[..1], n_codes0 u32[..320], n_codes1 u32[..320], base_symbol u32)() {
|
||||
// For the clcode example in this package's README.md:
|
||||
// - n_codes0 = 0
|
||||
// - n_codes1 = 19
|
||||
// - code_lengths[ 0] = 3
|
||||
// - code_lengths[ 1] = 0
|
||||
// - code_lengths[ 2] = 0
|
||||
// - code_lengths[ 3] = 5
|
||||
// - code_lengths[ 4] = 3
|
||||
// - code_lengths[ 5] = 3
|
||||
// - code_lengths[ 6] = 3
|
||||
// - code_lengths[ 7] = 3
|
||||
// - code_lengths[ 8] = 3
|
||||
// - code_lengths[ 9] = 3
|
||||
// - code_lengths[10] = 0
|
||||
// - code_lengths[11] = 0
|
||||
// - code_lengths[12] = 0
|
||||
// - code_lengths[13] = 0
|
||||
// - code_lengths[14] = 0
|
||||
// - code_lengths[15] = 0
|
||||
// - code_lengths[16] = 0
|
||||
// - code_lengths[17] = 4
|
||||
// - code_lengths[18] = 5
|
||||
|
||||
// Calculate counts.
|
||||
//
|
||||
// For the clcode example in this package's README.md:
|
||||
// - counts[0] = 9
|
||||
// - counts[1] = 0
|
||||
// - counts[2] = 0
|
||||
// - counts[3] = 7
|
||||
// - counts[4] = 1
|
||||
// - counts[5] = 2
|
||||
// - all other counts elements are 0.
|
||||
var counts[16] u16[..320]
|
||||
var i u32 = in.n_codes0
|
||||
while i < in.n_codes1 {
|
||||
assert i < 320 via "a < b: a < c; c <= b"(c:in.n_codes1)
|
||||
// TODO: this if should be unnecessary. Have some way to assert that,
|
||||
// for all j, counts[j] <= i, and thus counts[j]++ will not overflow.
|
||||
if counts[this.code_lengths[i]] >= 320 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
counts[this.code_lengths[i]] += 1
|
||||
i += 1
|
||||
}
|
||||
if ((counts[0] as u32) + in.n_codes0) == in.n_codes1 {
|
||||
return error "no Huffman codes"
|
||||
}
|
||||
|
||||
// Check that the Huffman code completely covers all possible input bits.
|
||||
var remaining u32 = 1 // There is 1 possible 0-bit code.
|
||||
i = 1
|
||||
while i <= 15 {
|
||||
if remaining > (1 << 30) {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
// Each iteration doubles the number of possible remaining codes.
|
||||
remaining <<= 1
|
||||
if remaining < (counts[i] as u32) {
|
||||
return error "bad Huffman code (over-subscribed)"
|
||||
}
|
||||
remaining -= counts[i] as u32
|
||||
i += 1
|
||||
}
|
||||
if remaining != 0 {
|
||||
// TODO: when is a degenerate Huffman table valid?
|
||||
return error "bad Huffman code (under-subscribed)"
|
||||
}
|
||||
|
||||
// Calculate offsets and n_symbols.
|
||||
//
|
||||
// For the clcode example in this package's README.md:
|
||||
// - offsets[0] = 0
|
||||
// - offsets[1] = 0
|
||||
// - offsets[2] = 0
|
||||
// - offsets[3] = 0
|
||||
// - offsets[4] = 7
|
||||
// - offsets[5] = 8
|
||||
// - offsets[6] = 10
|
||||
// - all other offsets elements are 10.
|
||||
// - n_symbols = 10
|
||||
var offsets[16] u16[..320]
|
||||
var n_symbols u32[..320]
|
||||
i = 1
|
||||
while i <= 15 {
|
||||
offsets[i] = n_symbols as u16
|
||||
var count u32[..320] = counts[i] as u32
|
||||
if n_symbols > (320 - count) {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
assert (n_symbols + count) <= 320 via "(a + b) <= c: a <= (c - b)"()
|
||||
// TODO: change this to n_symbols += count, once the proof engine's
|
||||
// bounds checking can handle it.
|
||||
n_symbols = n_symbols + count
|
||||
i += 1
|
||||
}
|
||||
if n_symbols > 288 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
// Calculate symbols.
|
||||
//
|
||||
// For the clcode example in this package's README.md:
|
||||
// - symbols[0] = 0
|
||||
// - symbols[1] = 4
|
||||
// - symbols[2] = 5
|
||||
// - symbols[3] = 6
|
||||
// - symbols[4] = 7
|
||||
// - symbols[5] = 8
|
||||
// - symbols[6] = 9
|
||||
// - symbols[7] = 17
|
||||
// - symbols[8] = 3
|
||||
// - symbols[9] = 18
|
||||
//
|
||||
// As a (local variable) side effect, offsets' values will be updated:
|
||||
// - offsets[3] = 7, formerly 0
|
||||
// - offsets[4] = 8, formerly 7
|
||||
// - offsets[5] = 10, formerly 8
|
||||
var symbols[320] u16[..319]
|
||||
i = in.n_codes0
|
||||
while i < in.n_codes1,
|
||||
inv n_symbols <= 288,
|
||||
{
|
||||
assert i < 320 via "a < b: a < c; c <= b"(c:in.n_codes1)
|
||||
// TODO: this if check should be unnecessary.
|
||||
if i < in.n_codes0 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
if this.code_lengths[i] != 0 {
|
||||
if offsets[this.code_lengths[i]] >= 320 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
symbols[offsets[this.code_lengths[i]]] = (i - in.n_codes0) as u16
|
||||
offsets[this.code_lengths[i]] += 1
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
|
||||
// Calculate min_cl and max_cl.
|
||||
//
|
||||
// For the clcode example in this package's README.md:
|
||||
// - min_cl = 3
|
||||
// - max_cl = 5
|
||||
var min_cl u32[1..9] = 1
|
||||
while true,
|
||||
inv n_symbols <= 288,
|
||||
{
|
||||
if counts[min_cl] != 0 {
|
||||
break
|
||||
}
|
||||
if min_cl >= 9 {
|
||||
return error "bad Huffman minimum code length"
|
||||
}
|
||||
min_cl += 1
|
||||
}
|
||||
var max_cl u32[1..15] = 15
|
||||
while true,
|
||||
inv n_symbols <= 288,
|
||||
{
|
||||
if counts[max_cl] != 0 {
|
||||
break
|
||||
}
|
||||
if max_cl <= 1 {
|
||||
// TODO: when is a degenerate Huffman table valid?
|
||||
return error "no Huffman codes"
|
||||
}
|
||||
max_cl -= 1
|
||||
}
|
||||
if max_cl <= 9 {
|
||||
this.n_huffs_bits[in.which] = max_cl
|
||||
} else {
|
||||
this.n_huffs_bits[in.which] = 9
|
||||
}
|
||||
|
||||
// Calculate this.huffs[in.which].
|
||||
//
|
||||
// For the clcode example in this package's README.md:
|
||||
// - this.huffs[0][0b..000] = 0x80000003 (literal, symbols[0]=0x00, code_length=3)
|
||||
// - this.huffs[0][0b..100] = 0x80000403 (literal, symbols[1]=0x04, code_length=3)
|
||||
// - this.huffs[0][0b..010] = 0x80000503 (literal, symbols[2]=0x05, code_length=3)
|
||||
// - this.huffs[0][0b..110] = 0x80000603 (literal, symbols[3]=0x06, code_length=3)
|
||||
// - this.huffs[0][0b..001] = 0x80000703 (literal, symbols[4]=0x07, code_length=3)
|
||||
// - this.huffs[0][0b..101] = 0x80000803 (literal, symbols[5]=0x08, code_length=3)
|
||||
// - this.huffs[0][0b..011] = 0x80000903 (literal, symbols[6]=0x09, code_length=3)
|
||||
// - this.huffs[0][0b.0111] = 0x80001104 (literal, symbols[7]=0x11, code_length=4)
|
||||
// - this.huffs[0][0b01111] = 0x80000305 (literal, symbols[8]=0x03, code_length=5)
|
||||
// - this.huffs[0][0b11111] = 0x80001805 (literal, symbols[9]=0x18, code_length=5)
|
||||
i = 0
|
||||
if (n_symbols != (offsets[max_cl] as u32)) or (n_symbols != (offsets[15] as u32)) {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
if (in.n_codes0 + (symbols[0] as u32)) >= 320 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
var initial_high_bits u32 = 1 << 9
|
||||
if max_cl < 9 {
|
||||
initial_high_bits = (1 as u32) << max_cl
|
||||
}
|
||||
var prev_cl u32[..15] = this.code_lengths[in.n_codes0 + (symbols[0] as u32)] as u32
|
||||
var prev_redirect_key u32 = 0xFFFFFFFF
|
||||
var top u32[..1234]
|
||||
var next_top u32[..1234] = 512
|
||||
var code u32
|
||||
var key u32
|
||||
var value u32
|
||||
while true,
|
||||
pre code < (1 << 15),
|
||||
pre i < 288,
|
||||
inv n_symbols <= 288,
|
||||
{
|
||||
if (in.n_codes0 + (symbols[i] as u32)) >= 320 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
var cl u32[..15] = this.code_lengths[in.n_codes0 + (symbols[i] as u32)] as u32
|
||||
if cl > prev_cl {
|
||||
code <<= cl - prev_cl
|
||||
if code >= (1 << 15) {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
}
|
||||
// For the remainder of this loop body, prev_cl is the original code
|
||||
// length, cl is possibly clipped by 9, if in the 2nd-level table.
|
||||
prev_cl = cl
|
||||
|
||||
key = code
|
||||
if cl > 9 {
|
||||
// TODO: we shouldn't need a temporary variable.
|
||||
var tmp u32[..6] = cl - 9
|
||||
cl = tmp
|
||||
assert cl <= 9 via "a <= b: a == c; c <= b"(c:tmp)
|
||||
|
||||
var redirect_key u32[..511] = (key >> tmp) & 511
|
||||
key = key.low_bits(n:tmp)
|
||||
if prev_redirect_key != redirect_key {
|
||||
prev_redirect_key = redirect_key
|
||||
|
||||
// Calculate the number of bits needed for the 2nd level table.
|
||||
// This computation is similar to "check that the Huffman code
|
||||
// completely covers all possible input bits" above.
|
||||
remaining = (1 as u32) << cl
|
||||
var j u32 = prev_cl
|
||||
while j <= 15,
|
||||
inv cl <= 9,
|
||||
inv code < (1 << 15),
|
||||
inv i < 288,
|
||||
inv n_symbols <= 288,
|
||||
{
|
||||
if remaining <= (counts[j] as u32) {
|
||||
break
|
||||
}
|
||||
remaining -= counts[j] as u32
|
||||
if remaining > (1 << 30) {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
remaining <<= 1
|
||||
j += 1
|
||||
}
|
||||
if (j <= 9) or (15 < j) {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
tmp = j - 9
|
||||
initial_high_bits = (1 as u32) << tmp
|
||||
|
||||
top = next_top
|
||||
if (top + ((1 as u32) << tmp)) > 1234 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
next_top = top + ((1 as u32) << tmp)
|
||||
|
||||
redirect_key = (reverse8[redirect_key >> 1] as u32) | ((redirect_key & 1) << 8)
|
||||
this.huffs[in.which][redirect_key] = 0x10000009 | (top << 8) | (tmp << 4)
|
||||
}
|
||||
}
|
||||
if (key >= (1 << 9)) or (counts[prev_cl] <= 0) {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
counts[prev_cl] -= 1
|
||||
|
||||
var reversed_key u32[..511] = (reverse8[key >> 1] as u32) | ((key & 1) << 8)
|
||||
reversed_key >>= 9 - cl
|
||||
|
||||
var symbol u32[..319] = symbols[i] as u32
|
||||
if symbol == 256 {
|
||||
// End-of-block.
|
||||
value = 0x20000000 | cl
|
||||
} else if (symbol < 256) and (in.which == 0) {
|
||||
// Literal.
|
||||
value = 0x80000000 | (symbol << 8) | cl
|
||||
} else if symbol >= in.base_symbol {
|
||||
// Base number + extra bits.
|
||||
symbol -= in.base_symbol
|
||||
if in.which == 0 {
|
||||
value = lcode_magic_numbers[symbol & 31] | cl
|
||||
} else {
|
||||
value = dcode_magic_numbers[symbol & 31] | cl
|
||||
}
|
||||
} else {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
// The table uses log2(initial_high_bits) bits, but reversed_key only
|
||||
// has cl bits. We duplicate the key-value pair across all possible
|
||||
// values of the high (log2(initial_high_bits) - cl) bits.
|
||||
var high_bits u32 = initial_high_bits
|
||||
var delta u32 = (1 as u32) << cl
|
||||
while high_bits >= delta,
|
||||
inv code < (1 << 15),
|
||||
inv i < 288,
|
||||
inv n_symbols <= 288,
|
||||
{
|
||||
high_bits -= delta
|
||||
if (top + ((high_bits | reversed_key) & 511)) >= 1234 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
this.huffs[in.which][top + ((high_bits | reversed_key) & 511)] = value
|
||||
}
|
||||
|
||||
i += 1
|
||||
if i >= n_symbols {
|
||||
break
|
||||
}
|
||||
assert i < 288 via "a < b: a < c; c <= b"(c:n_symbols)
|
||||
code += 1
|
||||
if code >= (1 << 15) {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,384 @@
|
||||
// Copyright 2017 The Wuffs Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
// TODO: describe how the xxx_fast version differs from the xxx_slow one, the
|
||||
// assumptions that xxx_fast makes, and how that makes it fast.
|
||||
pri func flate_decoder.decode_huffman_fast?(dst writer1, src reader1)() {
|
||||
// When editing this function, consider making the equivalent change to the
|
||||
// decode_huffman_slow function. Keep the diff between the two
|
||||
// decode_huffman_*.wuffs files as small as possible, while retaining both
|
||||
// correctness and performance.
|
||||
|
||||
// TODO: make this a func pre-condition.
|
||||
if not in.dst.is_marked() {
|
||||
return error "bad argument"
|
||||
}
|
||||
|
||||
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
|
||||
return error "internal error: inconsistent n_bits"
|
||||
}
|
||||
|
||||
var bits u32 = this.bits
|
||||
var n_bits u32 = this.n_bits
|
||||
|
||||
var table_entry u32
|
||||
var table_entry_n_bits u32[..15]
|
||||
var lmask u32[..511] = ((1 as u32) << this.n_huffs_bits[0]) - 1
|
||||
var dmask u32[..511] = ((1 as u32) << this.n_huffs_bits[1]) - 1
|
||||
|
||||
// Check up front, on each iteration, that we have enough buffer space to
|
||||
// both read (12 bytes) and write (258 bytes) as much as we need to. Doing
|
||||
// this check once (per iteration), up front, removes the need to check
|
||||
// (and possibly suspend the coroutine) multiple times inside the loop
|
||||
// body, so it's faster overall.
|
||||
//
|
||||
// For writing, a literal code obviously corresponds to writing 1 byte, and
|
||||
// 258 is the maximum length in a length-distance pair, as specified in the
|
||||
// RFC section 3.2.5. Compressed blocks (length and distance codes).
|
||||
//
|
||||
// For reading, strictly speaking, we only need 6 bytes (48 bits) of
|
||||
// available input, because the H-L length/literal code is up to 15 bits
|
||||
// plus up to 5 extra bits, the H-D distance code is up to 15 bits plus up
|
||||
// to 13 extra bits and 15 + 5 + 15 + 13 == 48. However, in.src.available()
|
||||
// assertions are made in terms of bytes, not bits.
|
||||
//
|
||||
// In theory, we could write assertions that mixed bytes and bits, such as:
|
||||
//
|
||||
// assert ((in.src.available() >= 6) and (n_bits >= 5)) or
|
||||
// ((in.src.available() >= 4) and (n_bits >= 21))
|
||||
//
|
||||
// but that looks clumsy and makes the proofs more complicated. Instead, we
|
||||
// are conservative and work in terms of bytes only. In bytes, each code
|
||||
// requires either one (direct) or two (redirect) lookups, up to 2 bytes
|
||||
// for each. There are also up to 13 extra bits, or 2 bytes. Thus, each
|
||||
// code requires up to 2 + 2 + 2 == 6 bytes, and there are two codes (one
|
||||
// from H-L and one from H-D). Therefore, we check for at least 12 bytes.
|
||||
while:loop(in.dst.available() >= 258) and (in.src.available() >= 12),
|
||||
inv in.dst.is_marked(),
|
||||
{
|
||||
// Ensure that we have at least 15 bits of input.
|
||||
if n_bits < 15 {
|
||||
assert n_bits >= 0 // TODO: this shouldn't be necessary.
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
assert n_bits >= 15
|
||||
} else {
|
||||
assert in.src.available() >= 10
|
||||
}
|
||||
// These assertions are redundant, but are listed explicitly for
|
||||
// clarity. They must hold regardless of which branch was taken for the
|
||||
// if statement above.
|
||||
assert in.src.available() >= 10
|
||||
assert n_bits >= 15
|
||||
|
||||
// Decode an lcode symbol from H-L.
|
||||
table_entry = this.huffs[0][bits & lmask]
|
||||
table_entry_n_bits = table_entry & 0x0F
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
|
||||
if (table_entry >> 31) != 0 {
|
||||
// Literal.
|
||||
in.dst.write_u8?(x:(table_entry >> 8) & 0xFF)
|
||||
continue:loop
|
||||
} else if (table_entry >> 30) != 0 {
|
||||
// No-op; code continues past the if-else chain.
|
||||
assert in.src.available() >= 8
|
||||
} else if (table_entry >> 29) != 0 {
|
||||
// End of block.
|
||||
this.end_of_block = true
|
||||
break:loop
|
||||
} else if (table_entry >> 28) != 0 {
|
||||
// Redirect.
|
||||
|
||||
// Ensure that we have at least 15 bits of input.
|
||||
if n_bits < 15 {
|
||||
assert n_bits >= 0 // TODO: this shouldn't be necessary.
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
assert n_bits >= 15
|
||||
} else {
|
||||
assert in.src.available() >= 8
|
||||
}
|
||||
// Once again, redundant but explicit assertions.
|
||||
assert in.src.available() >= 8
|
||||
assert n_bits >= 15
|
||||
|
||||
var redir_top u32[..0xFFFF] = (table_entry >> 8) & 0xFFFF
|
||||
var redir_mask u32[..0x7FFF] = ((1 as u32) << ((table_entry >> 4) & 0x0F)) - 1
|
||||
if (redir_top + (bits & redir_mask)) >= 1234 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
table_entry = this.huffs[0][redir_top + (bits & redir_mask)]
|
||||
table_entry_n_bits = table_entry & 0x0F
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
|
||||
if (table_entry >> 31) != 0 {
|
||||
// Literal.
|
||||
in.dst.write_u8?(x:(table_entry >> 8) & 0xFF)
|
||||
continue:loop
|
||||
} else if (table_entry >> 30) != 0 {
|
||||
// No-op; code continues past the if-else chain.
|
||||
} else if (table_entry >> 29) != 0 {
|
||||
// End of block.
|
||||
this.end_of_block = true
|
||||
break:loop
|
||||
} else if (table_entry >> 28) != 0 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
} else if (table_entry >> 27) != 0 {
|
||||
return error "bad Huffman code"
|
||||
} else {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
// Once again, redundant but explicit assertions.
|
||||
assert in.src.available() >= 8
|
||||
|
||||
} else if (table_entry >> 27) != 0 {
|
||||
return error "bad Huffman code"
|
||||
} else {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
// length = base number + extra bits.
|
||||
var length u32[..0x7FFF] = (table_entry >> 8) & 0x7FFF
|
||||
table_entry_n_bits = (table_entry >> 4) & 0x0F
|
||||
if table_entry_n_bits > 0 {
|
||||
// Ensure that we have at least 15 bits of input.
|
||||
if n_bits < 15 {
|
||||
assert n_bits >= 0 // TODO: this shouldn't be necessary.
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
assert n_bits >= 15
|
||||
} else {
|
||||
assert in.src.available() >= 6
|
||||
}
|
||||
// Once again, redundant but explicit assertions.
|
||||
assert in.src.available() >= 6
|
||||
assert n_bits >= 15
|
||||
|
||||
length = (length + bits.low_bits(n:table_entry_n_bits)) & 0x7FFF
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
|
||||
} else {
|
||||
assert in.src.available() >= 6
|
||||
}
|
||||
if length > 258 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
// Ensure that we have at least 15 bits of input.
|
||||
if n_bits < 15 {
|
||||
assert n_bits >= 0 // TODO: this shouldn't be necessary.
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
assert n_bits >= 15
|
||||
} else {
|
||||
assert in.src.available() >= 4
|
||||
}
|
||||
// Once again, redundant but explicit assertions.
|
||||
assert in.src.available() >= 4
|
||||
assert n_bits >= 15
|
||||
|
||||
// Decode a dcode symbol from H-D.
|
||||
table_entry = this.huffs[1][bits & dmask]
|
||||
table_entry_n_bits = table_entry & 15
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
|
||||
// Check for a redirect.
|
||||
if (table_entry >> 28) == 1 {
|
||||
// Ensure that we have at least 15 bits of input.
|
||||
if n_bits < 15 {
|
||||
assert n_bits >= 0 // TODO: this shouldn't be necessary.
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
assert n_bits >= 15
|
||||
} else {
|
||||
assert in.src.available() >= 2
|
||||
}
|
||||
// Once again, redundant but explicit assertions.
|
||||
assert in.src.available() >= 2
|
||||
assert n_bits >= 15
|
||||
|
||||
redir_top = (table_entry >> 8) & 0xFFFF
|
||||
redir_mask = ((1 as u32) << ((table_entry >> 4) & 0x0F)) - 1
|
||||
if (redir_top + (bits & redir_mask)) >= 1234 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
table_entry = this.huffs[1][redir_top + (bits & redir_mask)]
|
||||
table_entry_n_bits = table_entry & 0x0F
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
} else {
|
||||
assert in.src.available() >= 2
|
||||
}
|
||||
|
||||
// For H-D, all symbols should be base number + extra bits.
|
||||
if (table_entry >> 24) != 0x40 {
|
||||
if (table_entry >> 24) == 0x08 {
|
||||
return error "bad Huffman code"
|
||||
}
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
// distance = base number + extra bits.
|
||||
var distance u32[..0x7FFF] = (table_entry >> 8) & 0x7FFF
|
||||
table_entry_n_bits = (table_entry >> 4) & 0x0F
|
||||
if table_entry_n_bits > 0 {
|
||||
// Ensure that we have at least 15 bits of input.
|
||||
if n_bits < 15 {
|
||||
assert n_bits >= 0 // TODO: this shouldn't be necessary.
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
assert n_bits >= 15
|
||||
}
|
||||
// Once again, redundant but explicit assertions.
|
||||
assert n_bits >= 15
|
||||
|
||||
distance = (distance + bits.low_bits(n:table_entry_n_bits)) & 0x7FFF
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
}
|
||||
if distance <= 0 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
var n_copied u32
|
||||
// The "while true { etc; break }" is a redundant version of "etc", but
|
||||
// its presence minimizes the diff between decode_huffman_fast and
|
||||
// decode_huffman_slow.
|
||||
while true,
|
||||
pre length <= 258,
|
||||
pre distance > 0,
|
||||
pre in.dst.available() >= 258,
|
||||
inv in.dst.is_marked(),
|
||||
{
|
||||
// Copy from this.history.
|
||||
if (distance as u64) > in.dst.since_mark().length() {
|
||||
// Set (hlen, hdist) to be the length-distance pair to copy
|
||||
// from this.history, and (length, distance) to be the
|
||||
// remaining length-distance pair to copy from in.dst.
|
||||
var hlen u32[..0x7FFF]
|
||||
var hdist u32[..0x7FFF] = ((distance as u64) - in.dst.since_mark().length()) as u32
|
||||
if length > hdist {
|
||||
length -= hdist
|
||||
hlen = hdist
|
||||
// TODO: this if check should be redundant.
|
||||
if length > 258 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
} else {
|
||||
hlen = length
|
||||
length = 0
|
||||
assert length <= 258
|
||||
}
|
||||
if this.history_index < hdist {
|
||||
return error "bad distance"
|
||||
}
|
||||
// Re-purpose the hdist variable as the this.history index to
|
||||
// start copying from.
|
||||
hdist = (this.history_index - hdist) & 0x7FFF
|
||||
|
||||
// Copy from hdist to the end of this.history, and then
|
||||
// possibly wrap around the this.history ringbuffer.
|
||||
//
|
||||
// The "while true { etc; break }" is a redundant version of
|
||||
// "etc", but its presence minimizes the diff between
|
||||
// decode_huffman_fast and decode_huffman_slow.
|
||||
while true,
|
||||
inv length <= 258,
|
||||
inv distance > 0,
|
||||
inv in.dst.available() >= 258,
|
||||
inv in.dst.is_marked(),
|
||||
{
|
||||
// TODO: copy_from_slice32 should probably update the
|
||||
// "in.dst.available() >= 258" fact. It should certainly
|
||||
// invalidate any fact that was "<=" instead of ">=".
|
||||
n_copied = in.dst.copy_from_slice32(s:this.history[hdist:], length:hlen)
|
||||
if hlen <= n_copied {
|
||||
break
|
||||
}
|
||||
hlen -= n_copied
|
||||
in.dst.copy_from_slice32(s:this.history[:], length:hlen)
|
||||
break
|
||||
}
|
||||
|
||||
if length == 0 {
|
||||
// No need to copy from in.dst.
|
||||
continue:loop
|
||||
}
|
||||
|
||||
if (distance as u64) > in.dst.since_mark().length() {
|
||||
return error "internal error: inconsistent distance"
|
||||
}
|
||||
}
|
||||
// Once again, redundant but explicit assertions.
|
||||
assert length <= 258
|
||||
assert distance > 0
|
||||
assert (distance as u64) <= in.dst.since_mark().length()
|
||||
assert in.dst.available() >= 258
|
||||
assert in.dst.is_marked()
|
||||
|
||||
// We can therefore prove:
|
||||
assert (length as u64) <= 258
|
||||
assert (length as u64) <= in.dst.available() via "a <= b: a <= c; c <= b"(c:258)
|
||||
|
||||
// Copy from in.dst.
|
||||
in.dst.copy_from_history32(distance:distance, length:length)
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Ensure n_bits < 8 by rewindng in.src, if we loaded too many of its bytes
|
||||
// into the bits variable.
|
||||
//
|
||||
// Note that we can unconditionally call unread_u8 (without resulting in an
|
||||
// "invalid I/O operation" error code) only because this whole function can
|
||||
// never suspend, as all of its I/O operations were checked beforehand for
|
||||
// sufficient buffer space. Otherwise, resuming from the suspension could
|
||||
// mean that the (possibly different) in.src is no longer rewindable, even
|
||||
// if conceptually, this function was responsible for reading the bytes we
|
||||
// want to rewind.
|
||||
//
|
||||
// TODO: have the compiler enforce (via some explicit syntax) that this
|
||||
// function is ProvenNotToSuspend?
|
||||
while n_bits >= 8,
|
||||
post n_bits < 8,
|
||||
{
|
||||
n_bits -= 8
|
||||
in.src.unread_u8?()
|
||||
}
|
||||
|
||||
this.bits = bits & (((1 as u32) << n_bits) - 1)
|
||||
this.n_bits = n_bits
|
||||
|
||||
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
|
||||
return error "internal error: inconsistent n_bits"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
// Copyright 2017 The Wuffs Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// https://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
pri func flate_decoder.decode_huffman_slow?(dst writer1, src reader1)() {
|
||||
// When editing this function, consider making the equivalent change to the
|
||||
// decode_huffman_fast function. Keep the diff between the two
|
||||
// decode_huffman_*.wuffs files as small as possible, while retaining both
|
||||
// correctness and performance.
|
||||
|
||||
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
|
||||
return error "internal error: inconsistent n_bits"
|
||||
}
|
||||
|
||||
var bits u32 = this.bits
|
||||
var n_bits u32 = this.n_bits
|
||||
|
||||
var table_entry u32
|
||||
var table_entry_n_bits u32[..15]
|
||||
var lmask u32[..511] = ((1 as u32) << this.n_huffs_bits[0]) - 1
|
||||
var dmask u32[..511] = ((1 as u32) << this.n_huffs_bits[1]) - 1
|
||||
|
||||
while:loop true {
|
||||
// Decode an lcode symbol from H-L.
|
||||
while true {
|
||||
table_entry = this.huffs[0][bits & lmask]
|
||||
table_entry_n_bits = table_entry & 0x0F
|
||||
if n_bits >= table_entry_n_bits {
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
break
|
||||
}
|
||||
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
|
||||
if (table_entry >> 31) != 0 {
|
||||
// Literal.
|
||||
in.dst.write_u8?(x:(table_entry >> 8) & 0xFF)
|
||||
continue:loop
|
||||
} else if (table_entry >> 30) != 0 {
|
||||
// No-op; code continues past the if-else chain.
|
||||
} else if (table_entry >> 29) != 0 {
|
||||
// End of block.
|
||||
this.end_of_block = true
|
||||
break:loop
|
||||
} else if (table_entry >> 28) != 0 {
|
||||
// Redirect.
|
||||
var redir_top u32[..0xFFFF] = (table_entry >> 8) & 0xFFFF
|
||||
var redir_mask u32[..0x7FFF] = ((1 as u32) << ((table_entry >> 4) & 0x0F)) - 1
|
||||
while true {
|
||||
if (redir_top + (bits & redir_mask)) >= 1234 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
table_entry = this.huffs[0][redir_top + (bits & redir_mask)]
|
||||
table_entry_n_bits = table_entry & 0x0F
|
||||
if n_bits >= table_entry_n_bits {
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
break
|
||||
}
|
||||
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
|
||||
if (table_entry >> 31) != 0 {
|
||||
// Literal.
|
||||
in.dst.write_u8?(x:(table_entry >> 8) & 0xFF)
|
||||
continue:loop
|
||||
} else if (table_entry >> 30) != 0 {
|
||||
// No-op; code continues past the if-else chain.
|
||||
} else if (table_entry >> 29) != 0 {
|
||||
// End of block.
|
||||
this.end_of_block = true
|
||||
break:loop
|
||||
} else if (table_entry >> 28) != 0 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
} else if (table_entry >> 27) != 0 {
|
||||
return error "bad Huffman code"
|
||||
} else {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
} else if (table_entry >> 27) != 0 {
|
||||
return error "bad Huffman code"
|
||||
} else {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
// length = base number + extra bits.
|
||||
var length u32[..0x7FFF] = (table_entry >> 8) & 0x7FFF
|
||||
table_entry_n_bits = (table_entry >> 4) & 0x0F
|
||||
if table_entry_n_bits > 0 {
|
||||
while n_bits < table_entry_n_bits,
|
||||
post n_bits >= table_entry_n_bits,
|
||||
{
|
||||
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
length = (length + bits.low_bits(n:table_entry_n_bits)) & 0x7FFF
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
}
|
||||
|
||||
// Decode a dcode symbol from H-D.
|
||||
while true {
|
||||
table_entry = this.huffs[1][bits & dmask]
|
||||
table_entry_n_bits = table_entry & 15
|
||||
if n_bits >= table_entry_n_bits {
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
break
|
||||
}
|
||||
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
// Check for a redirect.
|
||||
if (table_entry >> 28) == 1 {
|
||||
redir_top = (table_entry >> 8) & 0xFFFF
|
||||
redir_mask = ((1 as u32) << ((table_entry >> 4) & 0x0F)) - 1
|
||||
while true {
|
||||
if (redir_top + (bits & redir_mask)) >= 1234 {
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
table_entry = this.huffs[1][redir_top + (bits & redir_mask)]
|
||||
table_entry_n_bits = table_entry & 0x0F
|
||||
if n_bits >= table_entry_n_bits {
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
break
|
||||
}
|
||||
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
}
|
||||
|
||||
// For H-D, all symbols should be base number + extra bits.
|
||||
if (table_entry >> 24) != 0x40 {
|
||||
if (table_entry >> 24) == 0x08 {
|
||||
return error "bad Huffman code"
|
||||
}
|
||||
return error "internal error: inconsistent Huffman decoder state"
|
||||
}
|
||||
|
||||
// distance = base number + extra bits.
|
||||
var distance u32[..0x7FFF] = (table_entry >> 8) & 0x7FFF
|
||||
table_entry_n_bits = (table_entry >> 4) & 0x0F
|
||||
if table_entry_n_bits > 0 {
|
||||
while n_bits < table_entry_n_bits,
|
||||
post n_bits >= table_entry_n_bits,
|
||||
{
|
||||
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
|
||||
bits |= (in.src.read_u8?() as u32) << n_bits
|
||||
n_bits += 8
|
||||
}
|
||||
distance = (distance + bits.low_bits(n:table_entry_n_bits)) & 0x7FFF
|
||||
bits >>= table_entry_n_bits
|
||||
n_bits -= table_entry_n_bits
|
||||
}
|
||||
|
||||
var n_copied u32
|
||||
while true {
|
||||
// Copy from this.history.
|
||||
if (distance as u64) > in.dst.since_mark().length() {
|
||||
// Set (hlen, hdist) to be the length-distance pair to copy
|
||||
// from this.history, and (length, distance) to be the
|
||||
// remaining length-distance pair to copy from in.dst.
|
||||
var hlen u32[..0x7FFF]
|
||||
var hdist u32[..0x7FFF] = ((distance as u64) - in.dst.since_mark().length()) as u32
|
||||
if length > hdist {
|
||||
length -= hdist
|
||||
hlen = hdist
|
||||
} else {
|
||||
hlen = length
|
||||
length = 0
|
||||
}
|
||||
if this.history_index < hdist {
|
||||
return error "bad distance"
|
||||
}
|
||||
// Re-purpose the hdist variable as the this.history index to
|
||||
// start copying from.
|
||||
hdist = (this.history_index - hdist) & 0x7FFF
|
||||
|
||||
// Copy from hdist to the end of this.history.
|
||||
while true {
|
||||
n_copied = in.dst.copy_from_slice32(s:this.history[hdist:], length:hlen)
|
||||
if hlen <= n_copied {
|
||||
hlen = 0
|
||||
break
|
||||
}
|
||||
if n_copied > 0 {
|
||||
hlen -= n_copied
|
||||
hdist = (hdist ~+ n_copied) & 0x7FFF
|
||||
if hdist == 0 {
|
||||
// Wrap around the this.history ringbuffer.
|
||||
break
|
||||
}
|
||||
}
|
||||
// TODO: "closed for writes" instead?
|
||||
yield suspension "short write"
|
||||
}
|
||||
// Copy from the start of this.history, if we wrapped around.
|
||||
if hlen > 0 {
|
||||
while true {
|
||||
n_copied = in.dst.copy_from_slice32(s:this.history[hdist:], length:hlen)
|
||||
if hlen <= n_copied {
|
||||
hlen = 0
|
||||
break
|
||||
}
|
||||
hlen -= n_copied
|
||||
hdist = (hdist + (n_copied & 0x7FFF)) & 0x7FFF
|
||||
// TODO: "closed for writes" instead?
|
||||
yield suspension "short write"
|
||||
}
|
||||
}
|
||||
|
||||
if length == 0 {
|
||||
// No need to copy from in.dst.
|
||||
continue:loop
|
||||
}
|
||||
}
|
||||
|
||||
// Copy from in.dst.
|
||||
n_copied = in.dst.copy_from_history32(distance:distance, length:length)
|
||||
if length <= n_copied {
|
||||
length = 0
|
||||
break
|
||||
}
|
||||
length -= n_copied
|
||||
// TODO: "closed for writes" instead?
|
||||
yield suspension "short write"
|
||||
}
|
||||
}
|
||||
|
||||
// TODO: "assert n_bits < 8"? What about (bits >> n_bits)?
|
||||
|
||||
this.bits = bits
|
||||
this.n_bits = n_bits
|
||||
|
||||
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
|
||||
return error "internal error: inconsistent n_bits"
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user