Rename std/flate to std/deflate

This commit is contained in:
Nigel Tao committed 2017-12-04 10:27:48 +11:00
1 parent d168e6977b
commit 1388f84e5c
20 files changed
+590 -571

No files matched your search

+540
View File
@@ -0,0 +1,540 @@
# Flate
Flate (also known as DEFLATE) is a general purpose compression format, using
Huffman codes and Lempel-Ziv style length-distance back-references. It is
specified in [RFC 1951](https://www.ietf.org/rfc/rfc1951.txt).
Gzip ([RFC 1952](https://www.ietf.org/rfc/rfc1952.txt)) and zlib ([RFC
1950](https://www.ietf.org/rfc/rfc1950.txt)) are two thin wrappers (with
similar purpose, but different wire formats) around the raw flate encoding.
Gzip (the file format) is used by the `gzip` and `tar` command line tools and
by the HTTP network protocol. Zlib is used by the ELF executable and PNG image
file formats.
[Zip](https://support.pkware.com/display/PKZIP/APPNOTE) (also known as PKZIP)
is another wrapper, similar to combining gzip with tar, that can compress
multiple files into a single archive. Zip is widely used by the ECMA Office
Open XML format, the OASIS Open Document Format for Office Applications and the
Java JAR format.
Wrangling those formats that build on flate (gzip, zip and zlib) is not
provided by this package. For zlib, look at the `std/zlib` package instead. The
other formats are TODO.
For example, look at `test/testdata/romeo.txt*`. First, the uncompressed text:
$ xxd test/testdata/romeo.txt
00000000: 526f 6d65 6f20 616e 6420 4a75 6c69 6574 Romeo and Juliet
00000010: 0a45 7863 6572 7074 2066 726f 6d20 4163 .Excerpt from Ac
etc
00000390: 740a 536f 2073 7475 6d62 6c65 7374 206f t.So stumblest o
000003a0: 6e20 6d79 2063 6f75 6e73 656c 3f0a n my counsel?.
The raw flate encoding:
$ xxd test/testdata/romeo.txt.flate
00000000: 4d53 c16e db30 0cbd f32b d853 2e46 0e3d MS.n.0...+.S.F.=
00000010: 2e87 20ed 0234 c5ba 0049 861e 861d 149b .. ..4...I......
etc
00000200: 7d13 8ffd b9a3 24bb 68f4 eb30 7a59 610d }.....$.h..0zYa.
00000210: 7f01 ..
The gzip format wraps a variable length header (in this case, 20 bytes) and 8
byte footer around the raw flate data. The header contains the NUL-terminated C
string name of the original, uncompressed file, "romeo.txt", amongst other
data. The footer contains a 4 byte CRC32 checksum and a 4 byte length of the
uncompressed file (0x3ae = 942 bytes).
$ xxd test/testdata/romeo.txt.gz
00000000: 1f8b 0808 26d8 5d59 0003 726f 6d65 6f2e ....&.]Y..romeo.
00000010: 7478 7400 4d53 c16e db30 0cbd f32b d853 txt.MS.n.0...+.S
etc
00000210: de5d 2c67 7d13 8ffd b9a3 24bb 68f4 eb30 .],g}.....$.h..0
00000220: 7a59 610d 7f01 ef07 e5ab ae03 0000 zYa...........
The zlib format wraps a 2 byte header and 4 byte footer around the raw flate
data. The footer contains a 4 byte Adler32 checksum. TODO: move this to
std/zlib/README.md.
$ xxd test/testdata/romeo.txt.zlib
00000000: 789c 4d53 c16e db30 0cbd f32b d853 2e46 x.MS.n.0...+.S.F
00000010: 0e3d 2e87 20ed 0234 c5ba 0049 861e 861d .=.. ..4...I....
etc
00000200: 2c67 7d13 8ffd b9a3 24bb 68f4 eb30 7a59 ,g}.....$.h..0zY
00000210: 610d 7f01 57bb 3ede a...W.>.
# Wire Format Worked Example
Consider `test/testdata/romeo.txt.flate`. The relevant spec is RFC 1951.
offset xoffset ASCII hex binary
000000 0x0000 M 0x4D 0b_0100_1101
000001 0x0001 S 0x53 0b_0101_0011
000002 0x0002 . 0xC1 0b_1100_0001
000003 0x0003 n 0x6E 0b_0110_1110
000004 0x0004 . 0xDB 0b_1101_1011
000005 0x0005 0 0x30 0b_0011_0000
000006 0x0006 . 0x0C 0b_0000_1100
000007 0x0007 . 0xBD 0b_1011_1101
000008 0x0008 . 0xF3 0b_1111_0011
000009 0x0009 + 0x2B 0b_0010_1011
000010 0x000A . 0xD8 0b_1101_1000
000011 0x000B S 0x53 0b_0101_0011
000012 0x000C . 0x2E 0b_0010_1110
000013 0x000D F 0x46 0b_0100_0110
000014 0x000E . 0x0E 0b_0000_1110
000015 0x000F = 0x3D 0b_0011_1101
000016 0x0010 . 0x2E 0b_0010_1110
000017 0x0011 . 0x87 0b_1000_0111
000018 0x0012 0x20 0b_0010_0000
000019 0x0013 . 0xED 0b_1110_1101
etc
000522 0x020A . 0xEB 0b_1110_1011
000523 0x020B 0 0x30 0b_0011_0000
000524 0x020C z 0x7A 0b_0111_1010
000525 0x020D Y 0x59 0b_0101_1001
000526 0x020E a 0x61 0b_0110_0001
000527 0x020F . 0x0D 0b_0000_1101
000528 0x0210 . 0x7F 0b_0111_1111
000529 0x0211 . 0x01 0b_0000_0001
Starting at the top:
offset xoffset ASCII hex binary
000000 0x0000 M 0x4D 0b_0100_1101
As per the RFC section 3.2.3. Details of block format:
- BFINAL is the first (LSB) bit, 0b1. This block is the final block.
- BTYPE is the next two bits in natural (MSB to LSB) order, 0b10, which means
a block that's compressed with dynamic Huffman codes.
There are 3 block types: uncompressed, compressed with fixed Huffman codes and
compressed with dynamic Huffman codes. The first type is straightforward:
containing a uint16 length `N` (and its complement, for error detection) and
then `N` literal bytes. The second type is the same as the third type but with
built-in `lcode` and `dcode` tables (see below). The third type is the
interesting part of the flate format, and its deconstruction continues below.
## Dynamic Huffman Blocks
The bit stream is now at:
offset xoffset ASCII hex binary
000000 0x0000 M 0x4D 0b_0100_1...
000001 0x0001 S 0x53 0b_0101_0011
000002 0x0002 . 0xC1 0b_1100_0001
As per the RFC section 3.2.7. Compression with dynamic Huffman codes, three
variables (5, 5 and 4 bits) are read in the same natural (MSB to LSB) order:
- HLIT = 0b01001 = 9
- HDIST = 0b10011 = 19
- HCLEN = 0b1010 = 10
Adding the implicit biases give:
nlit = 9 + 257 = 266
ndist = 19 + 1 = 20
nclen = 10 + 4 = 14
## Code Length Code Lengths
The bit stream is now at:
offset xoffset ASCII hex binary
000002 0x0002 . 0xC1 0b_1100_000.
000003 0x0003 n 0x6E 0b_0110_1110
000004 0x0004 . 0xDB 0b_1101_1011
000005 0x0005 0 0x30 0b_0011_0000
000006 0x0006 . 0x0C 0b_0000_1100
000007 0x0007 . 0xBD 0b_1011_1101
There are 19 possible code length code lengths (this is not a typo), but the
wire format order is shuffled (in the idiosyncratic order: 16, 17, 18, 0, 8,
..., 15) so that later elements are more likely to be zero (and hence
compressible). There are `nclen` (in this case, 14) explicit code length code
lengths, 3 bits each.
clcode_lengths[16] = 0b000 = 0
clcode_lengths[17] = 0b100 = 4
clcode_lengths[18] = 0b101 = 5
clcode_lengths[ 0] = 0b011 = 3
clcode_lengths[ 8] = 0b011 = 3
clcode_lengths[ 7] = 0b011 = 3
clcode_lengths[ 9] = 0b011 = 3
clcode_lengths[ 6] = 0b011 = 3
clcode_lengths[10] = 0b000 = 0
clcode_lengths[ 5] = 0b011 = 3
clcode_lengths[11] = 0b000 = 0
clcode_lengths[ 4] = 0b011 = 3
clcode_lengths[12] = 0b000 = 0
clcode_lengths[ 3] = 0b101 = 5
The remaining (19 - `nclen`) = 5 entries are implicitly zero:
clcode_lengths[13] = 0
clcode_lengths[ 2] = 0
clcode_lengths[14] = 0
clcode_lengths[ 1] = 0
clcode_lengths[15] = 0
Undoing the shuffle gives:
clcode_lengths[ 0] = 3
clcode_lengths[ 1] = 0
clcode_lengths[ 2] = 0
clcode_lengths[ 3] = 5
clcode_lengths[ 4] = 3
clcode_lengths[ 5] = 3
clcode_lengths[ 6] = 3
clcode_lengths[ 7] = 3
clcode_lengths[ 8] = 3
clcode_lengths[ 9] = 3
clcode_lengths[10] = 0
clcode_lengths[11] = 0
clcode_lengths[12] = 0
clcode_lengths[13] = 0
clcode_lengths[14] = 0
clcode_lengths[15] = 0
clcode_lengths[16] = 0
clcode_lengths[17] = 4
clcode_lengths[18] = 5
For `clcode`s, there are 7 + 1 + 2 = 10 non-zero entries: 7 3-bit codes, 1
4-bit code and 2 5-bit codes. The canonical Huffman encoding's map from bits to
`clcode`s is:
bits clcode
0b000 0
0b001 4
0b010 5
0b011 6
0b100 7
0b101 8
0b110 9
0b1110 17
0b11110 3
0b11111 18
Call that Huffman table `H-CL`.
## Lcodes and Dcodes
The bit stream is now at:
offset xoffset ASCII hex binary
000007 0x0007 . 0xBD 0b_1011_1...
000008 0x0008 . 0xF3 0b_1111_0011
000009 0x0009 + 0x2B 0b_0010_1011
000010 0x000A . 0xD8 0b_1101_1000
000011 0x000B S 0x53 0b_0101_0011
000012 0x000C . 0x2E 0b_0010_1110
000013 0x000D F 0x46 0b_0100_0110
When decoding via a Huffman table, bits are read in LSB to MSB order,
right-to-left in this debugging output. Extra bits, if any, are then read in
the other, natural order (MSB to LSB). Decoding via `H-CL` gives:
bits clcode
0b1110 17, 3 extra bits = 0b111 = 7
0b001 4
0b11111 18, 7 extra bits = 0b0001010 = 10
0b001 4
0b101 8
0b1110 17, 3 extra bits = 0b010 = 2
0b100 7
0b1110 17, 3 extra bits = 0b001 = 1
0b011 6
0b000 0
Still in the RFC section 3.2.7., this means:
lcode_lengths has 3 + 7 = 10 consecutive zeroes
lcode_lengths[ 10] = 4
lcode_lengths has 11 + 10 = 21 consecutive zeroes
lcode_lengths[ 32] = 4
lcode_lengths[ 33] = 8
lcode_lengths has 3 + 2 = 5 consecutive zeroes
lcode_lengths[ 39] = 7
lcode_lengths has 3 + 1 = 4 consecutive zeroes
lcode_lengths[ 44] = 6
lcode_lengths[ 45] = 0
After decoding the first 10 code lengths, the bit stream is now at:
offset xoffset ASCII hex binary
000013 0x000D F 0x46 0b_01.._....
Continuing to read all `nlit` (266) `lcode` lengths and `ndist` (20) `dcode`
lengths (zeroes are not shown here):
lcode_lengths[ 10] = 4
lcode_lengths[ 32] = 4
lcode_lengths[ 33] = 8
lcode_lengths[ 39] = 7
lcode_lengths[ 44] = 6
lcode_lengths[ 46] = 7
lcode_lengths[ 50] = 8
lcode_lengths[ 58] = 9
lcode_lengths[ 59] = 7
lcode_lengths[ 63] = 7
etc
lcode_lengths[264] = 9
lcode_lengths[265] = 9
and
dcode_lengths[ 5] = 6
dcode_lengths[ 6] = 5
dcode_lengths[ 8] = 4
dcode_lengths[ 9] = 5
dcode_lengths[10] = 3
etc
dcode_lengths[18] = 3
dcode_lengths[19] = 6
The `H-CL` table is no longer used from here onwards.
For `lcode`s, there are 6 + 9 + 10 + 16 + 10 + 12 = 63 non-zero entries: 6
4-bit codes, 9 5-bit codes, 10 6-bit codes, 16 7-bit codes, 10 8-bit codes and
12 9-bit codes. The canonical Huffman encoding's map from bits to `lcode`
values is:
bits lcode
0b0000 10
0b0001 32
0b0010 101
0b0011 116
0b0100 257
0b0101 259
0b01100 97
0b01101 104
0b01110 105
0b01111 110
0b10000 111
0b10001 114
0b10010 115
0b10011 258
0b10100 260
0b101010 44
0b101011 99
0b101100 100
0b101101 102
0b101110 108
0b101111 109
0b110000 112
0b110001 117
0b110010 119
0b110011 121
0b1101000 39
0b1101001 46
0b1101010 59
0b1101011 63
0b1101100 65
0b1101101 69
0b1101110 73
0b1101111 79
0b1110000 82
0b1110001 83
0b1110010 84
0b1110011 98
0b1110100 103
0b1110101 261
0b1110110 262
0b1110111 263
0b11110000 33
0b11110001 50
0b11110010 66
0b11110011 67
0b11110100 72
0b11110101 74
0b11110110 77
0b11110111 87
0b11111000 107
0b11111001 118
0b111110100 58
0b111110101 68
0b111110110 76
0b111110111 78
0b111111000 85
0b111111001 91
0b111111010 93
0b111111011 120
0b111111100 122
0b111111101 256
0b111111110 264
0b111111111 265
Call that Huffman table `H-L`.
For `dcode`s, there are 5 + 4 + 3 + 2 = 14 non-zero entries: 5 3-bit codes, 4
4-bit codes, 3 5-bit codes and 2 6-bit codes. The canonical Huffman encoding's
map from bits to `dcode` values is:
bits dcode
0b000 10
0b001 14
0b010 16
0b011 17
0b100 18
0b1010 8
0b1011 12
0b1100 13
0b1101 15
0b11100 6
0b11101 9
0b11110 11
0b111110 5
0b111111 19
Call that Huffman table `H-D`.
## Decoding Literals
The bit stream is now at:
offset xoffset ASCII hex binary
000052 0x0034 . 0xE7 0b_1..._....
000053 0x0035 C 0x43 0b_0100_0011
000054 0x0036 . 0xE8 0b_1110_1000
000055 0x0037 ) 0x29 0b_0010_1001
000056 0x0038 . 0xA0 0b_1010_0000
000057 0x0039 . 0xF1 0b_1111_0001
Decoding from that bit stream via `H-L` gives some literal bytes (where the
`lcode` value is < 256):
bits lcode
0b1110000 82 (ASCII 'R')
0b10000 111 (ASCII 'o')
0b101111 109 (ASCII 'm')
0b0010 101 (ASCII 'e')
0b10000 111 (ASCII 'o')
0b0001 32 (ASCII ' ')
0b01100 97 (ASCII 'a')
0b01111 110 (ASCII 'n')
## Decoding Back-References
This continues, up until we get to the "O Romeo, Romeo! wherefore art thou
Romeo?" line.
The bit stream is now at:
offset xoffset ASCII hex binary
000089 0x0059 . 0xC1 0b_11.._....
000090 0x005A . 0x1E 0b_0001_1110
000091 0x005B . 0x0F 0b_0000_1111
000092 0x005C . 0xF9 0b_1111_1001
Decoding from that bit stream via `H-L` gives:
bits lcode
0b1101111 79 (ASCII 'O')
0b0001 32 (ASCII ' ')
0b1110000 82 (ASCII 'R')
0b10011 258
This 258 is our first non-literal `lcode`, the start of a length-distance
back-reference. We first need to decode the length. The RFC section 3.2.5.
Compressed blocks (length and distance codes) says that an `lcode` of 258 means
that length=4 with no extra bits.
The bit stream is now at:
offset xoffset ASCII hex binary
000092 0x005C . 0xF9 0b_111._....
000093 0x005D Y 0x59 0b_0101_1001
We next decode via `H-D` instead of `H-L`.
bits dcode
0b11110 11
Again, from section 3.2.5., a `dcode` of 11 means a distance in the range [49,
64], 49 plus 4 extra bits. The next 4 bits are 0b0110=6, so the distance is 55.
We therefore copy length=4 bytes from distance=55 bytes ago: "omeo".
The bit stream is now at:
offset xoffset ASCII hex binary
000093 0x005D Y 0x59 0b_01.._....
000094 0x005E U 0x55 0b_0101_0101
000095 0x005F > 0x3E 0b_0011_1110
We go back to decoding via `H-L`, which gives
bits lcode
0b101010 44 (ASCII ',')
0b10100 260
This is another non-literal. Section 3.2.5. says that an `lcode` of 260 means
length=6 with no extra bits.
The bit stream is now at:
offset xoffset ASCII hex binary
000095 0x005F > 0x3E 0b_0011_111.
Decode with `H-D`.
bits dcode
0b111110 5
Again, from section 3.2.5., a `dcode` of 5 means a distance in the range [7,
8], 7 plus 1 extra bit. The next 1 bits are 0b0=0, so the distance is 7. We
therefore copy length=6 bytes from distance=7 bytes ago: " Romeo".
The bit stream is now at:
offset xoffset ASCII hex binary
000096 0x0060 . 0x0F 0b_0000_1111
We go back to decoding via `H-L`, which gives
bits lcode
0b11110000 33 (ASCII '!')
And on we go.
## End of Block
The block finishes with the bit stream at:
offset xoffset ASCII hex binary
000522 0x020A . 0xEB 0b_1110_101.
000523 0x020B 0 0x30 0b_0011_0000
000524 0x020C z 0x7A 0b_0111_1010
000525 0x020D Y 0x59 0b_0101_1001
000526 0x020E a 0x61 0b_0110_0001
000527 0x020F . 0x0D 0b_0000_1101
000528 0x0210 . 0x7F 0b_0111_1111
000529 0x0211 . 0x01 0b_0000_0001
The decoding is:
bits lcode
0b101011 99 (ASCII 'c')
0b10000 111 (ASCII 'o')
0b110001 117 (ASCII 'u')
0b01111 110 (ASCII 'n')
0b0100 257 (length=3 + 0_extra_bits...)
bits dcode
0b1101 15 (distance=193 + 6_extra_bits, 0b000010 in MSB to LSB order;
length=3, distance=195: copy "sel")
bits lcode
0b1101011 63 (ASCII '?')
0b0000 10 (ASCII '\n')
0b111111101 256 (end of block)
In other words, the block ends with "counsel?\n".
+54
View File
@@ -0,0 +1,54 @@
// Copyright 2017 The Wuffs Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// code_order is defined in the RFC section 3.2.7.
pri const code_order[19] u8[..18] = $(
16, 17, 18, 0, 8, 7, 9, 6, 10, 5, 11, 4, 12, 3, 13, 2, 14, 1, 15,
)
// reverse8 reverses the bits in a byte.
pri const reverse8[256] u8 = $(
0x00, 0x80, 0x40, 0xC0, 0x20, 0xA0, 0x60, 0xE0, // 0x00 - 0x07
0x10, 0x90, 0x50, 0xD0, 0x30, 0xB0, 0x70, 0xF0, // 0x08 - 0x0F
0x08, 0x88, 0x48, 0xC8, 0x28, 0xA8, 0x68, 0xE8, // 0x10 - 0x17
0x18, 0x98, 0x58, 0xD8, 0x38, 0xB8, 0x78, 0xF8, // 0x18 - 0x1F
0x04, 0x84, 0x44, 0xC4, 0x24, 0xA4, 0x64, 0xE4, // 0x20 - 0x27
0x14, 0x94, 0x54, 0xD4, 0x34, 0xB4, 0x74, 0xF4, // 0x28 - 0x2F
0x0C, 0x8C, 0x4C, 0xCC, 0x2C, 0xAC, 0x6C, 0xEC, // 0x30 - 0x37
0x1C, 0x9C, 0x5C, 0xDC, 0x3C, 0xBC, 0x7C, 0xFC, // 0x38 - 0x3F
0x02, 0x82, 0x42, 0xC2, 0x22, 0xA2, 0x62, 0xE2, // 0x40 - 0x47
0x12, 0x92, 0x52, 0xD2, 0x32, 0xB2, 0x72, 0xF2, // 0x48 - 0x4F
0x0A, 0x8A, 0x4A, 0xCA, 0x2A, 0xAA, 0x6A, 0xEA, // 0x50 - 0x57
0x1A, 0x9A, 0x5A, 0xDA, 0x3A, 0xBA, 0x7A, 0xFA, // 0x58 - 0x5F
0x06, 0x86, 0x46, 0xC6, 0x26, 0xA6, 0x66, 0xE6, // 0x60 - 0x67
0x16, 0x96, 0x56, 0xD6, 0x36, 0xB6, 0x76, 0xF6, // 0x68 - 0x6F
0x0E, 0x8E, 0x4E, 0xCE, 0x2E, 0xAE, 0x6E, 0xEE, // 0x70 - 0x77
0x1E, 0x9E, 0x5E, 0xDE, 0x3E, 0xBE, 0x7E, 0xFE, // 0x78 - 0x7F
0x01, 0x81, 0x41, 0xC1, 0x21, 0xA1, 0x61, 0xE1, // 0x80 - 0x87
0x11, 0x91, 0x51, 0xD1, 0x31, 0xB1, 0x71, 0xF1, // 0x88 - 0x8F
0x09, 0x89, 0x49, 0xC9, 0x29, 0xA9, 0x69, 0xE9, // 0x90 - 0x97
0x19, 0x99, 0x59, 0xD9, 0x39, 0xB9, 0x79, 0xF9, // 0x98 - 0x9F
0x05, 0x85, 0x45, 0xC5, 0x25, 0xA5, 0x65, 0xE5, // 0xA0 - 0xA7
0x15, 0x95, 0x55, 0xD5, 0x35, 0xB5, 0x75, 0xF5, // 0xA8 - 0xAF
0x0D, 0x8D, 0x4D, 0xCD, 0x2D, 0xAD, 0x6D, 0xED, // 0xB0 - 0xB7
0x1D, 0x9D, 0x5D, 0xDD, 0x3D, 0xBD, 0x7D, 0xFD, // 0xB8 - 0xBF
0x03, 0x83, 0x43, 0xC3, 0x23, 0xA3, 0x63, 0xE3, // 0xC0 - 0xC7
0x13, 0x93, 0x53, 0xD3, 0x33, 0xB3, 0x73, 0xF3, // 0xC8 - 0xCF
0x0B, 0x8B, 0x4B, 0xCB, 0x2B, 0xAB, 0x6B, 0xEB, // 0xD0 - 0xD7
0x1B, 0x9B, 0x5B, 0xDB, 0x3B, 0xBB, 0x7B, 0xFB, // 0xD8 - 0xDF
0x07, 0x87, 0x47, 0xC7, 0x27, 0xA7, 0x67, 0xE7, // 0xE0 - 0xE7
0x17, 0x97, 0x57, 0xD7, 0x37, 0xB7, 0x77, 0xF7, // 0xE8 - 0xEF
0x0F, 0x8F, 0x4F, 0xCF, 0x2F, 0xAF, 0x6F, 0xEF, // 0xF0 - 0xF7
0x1F, 0x9F, 0x5F, 0xDF, 0x3F, 0xBF, 0x7F, 0xFF, // 0xF8 - 0xFF
)
+757
View File
@@ -0,0 +1,757 @@
// Copyright 2017 The Wuffs Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
packageid "flte"
// TODO: rename "struct flate_decoder" to just "struct decoder", to avoid
// stutter in the fully qualified C name such as "wuffs_flate__flate_decoder".
pub error "bad Huffman code (over-subscribed)"
pub error "bad Huffman code (under-subscribed)"
pub error "bad Huffman code length count"
pub error "bad Huffman code length repetition"
pub error "bad Huffman code"
pub error "bad Huffman minimum code length"
pub error "bad distance"
pub error "bad distance code count"
pub error "bad flate block"
pub error "bad literal/length code count"
pub error "inconsistent stored block length"
pub error "internal error: inconsistent Huffman decoder state"
pub error "internal error: inconsistent Huffman end_of_block"
pub error "internal error: inconsistent distance"
pub error "internal error: inconsistent n_bits"
pub error "missing end-of-block code"
pub error "no Huffman codes"
// The next two tables were created by script/print-flate-magic-numbers.go.
//
// The u32 values' meanings are the same as the flate_decoder.huffs u32 values.
// In particular, bit 30 indicates a base number + extra bits, bits 23-8 are
// the base number and bits 7-4 are the number of those extra bits.
//
// Some trailing elements are 0x08000000. Bit 27 indicates an invalid value.
pri const lcode_magic_numbers[32] u32 = $(
0x40000300, 0x40000400, 0x40000500, 0x40000600, 0x40000700, 0x40000800, 0x40000900, 0x40000A00,
0x40000B10, 0x40000D10, 0x40000F10, 0x40001110, 0x40001320, 0x40001720, 0x40001B20, 0x40001F20,
0x40002330, 0x40002B30, 0x40003330, 0x40003B30, 0x40004340, 0x40005340, 0x40006340, 0x40007340,
0x40008350, 0x4000A350, 0x4000C350, 0x4000E350, 0x40010200, 0x08000000, 0x08000000, 0x08000000,
)
pri const dcode_magic_numbers[32] u32 = $(
0x40000100, 0x40000200, 0x40000300, 0x40000400, 0x40000510, 0x40000710, 0x40000920, 0x40000D20,
0x40001130, 0x40001930, 0x40002140, 0x40003140, 0x40004150, 0x40006150, 0x40008160, 0x4000C160,
0x40010170, 0x40018170, 0x40020180, 0x40030180, 0x40040190, 0x40060190, 0x400801A0, 0x400C01A0,
0x401001B0, 0x401801B0, 0x402001C0, 0x403001C0, 0x404001D0, 0x406001D0, 0x08000000, 0x08000000,
)
pub struct flate_decoder?(
// These fields yield src's bits in Least Significant Bits order.
bits u32,
n_bits u32,
// huffs and n_huffs_bits are the lookup tables for Huffman decodings.
//
// There are up to 2 Huffman decoders active at any one time. As per this
// package's README.md:
// - huffs[0] is used for clcode and lcode.
// - huffs[1] is used for dcode.
//
// The initial table key is the low n_huffs_bits of the flate_decoder.bits
// field. Keys longer than 9 bits require a two step lookup, the first step
// examines the low 9 bits, the second step examines the remaining bits.
// Two steps are required at most, as keys are at most 15 bits long.
//
// Using flate_decoder.bits's low n_huffs_bits as a table key is valid even
// if flate_decoder.n_bits is less than n_huffs_bits, because the immediate
// next step after indexing the table by the key is to compare
// flate_decoder.n_bits to the table value's number of flate_decoder.bits
// to consume. If it compares less, then more source bytes are read and the
// table lookup re-tried.
//
// The table value is:
// - bit 31 indicates the symbol is a literal.
// - bit 30 indicates the symbol is a base number + extra bits.
// - bit 29 indicates the symbol is the end-of-block.
// - bit 28 indicates a redirect to another part of the table.
// - bit 27 indicates an invalid value.
// - bits 26-24 are zero.
// - bits 23- 8 is the redirect offset, literal (in bits 15-8) or base number.
// - bits 7- 4 is the redirected table's bits or number of extra bits.
// - bits 3- 0 are the number of flate_decoder.bits to consume.
//
// Exactly one of the eight bits 31-24 should be set.
//
// TODO: replace the magic "big enough" 1234 with something more
// principled, perhaps discovered via an exhaustive search.
huffs[2][1234] u32,
n_huffs_bits[2] u32[..9],
// history holds up to the last 32KiB of decoded output, if the decoding
// was incomplete (e.g. due to a short read or write). RFC 1951 (DEFLATE)
// gives the maximum distance in a length-distance back-reference as 32768,
// or 0x8000.
//
// It is a ringbuffer, so that the most distant byte in the decoding isn't
// necessarily history[0]. The ringbuffer is full (i.e. it holds 32KiB of
// history) if and only if history_index >= 0x8000.
//
// history[history_index & 0x7FFF] is where the next byte of decoded output
// will be written.
history[0x8000] u8, // 32 KiB.
history_index u32,
// code_lengths is used to pass out-of-band data to init_huff.
//
// code_lengths[in.n_codes0 + i] holds the number of bits in the i'th code.
code_lengths[320] u8[..15],
// end_of_block is whether decode_huffman_xxx saw an end-of-block code.
//
// TODO: can decode_huffman_xxx signal this in band instead of out of band?
end_of_block bool,
)
pub func flate_decoder.decode?(dst writer1, src reader1)() {
while true {
in.dst.mark()
var z status = try this.decode_blocks?(dst:in.dst, src:in.src)
if not z.is_suspension() {
return z
}
// TODO: should "since_mark" be "since_mark!", as the return value lets
// you modify the state of in.dst, so future mutations (via the slice)
// can change the veracity of any in.dst assertions?
var written[] u8 = in.dst.since_mark()
// Append written, the decoded output, to the history ringbuffer.
if written.length() >= 0x8000 {
// If written is longer than the ringbuffer, we can ignore the
// previous value of history_index, as we will overwrite the whole
// ringbuffer.
written = written.suffix(up_to:0x8000)
this.history[:].copy_from_slice(s:written)
this.history_index = 0x8000
} else {
// Otherwise, append written to the history ringbuffer starting at
// the previous history_index (modulo 0x8000).
var n_copied u64 = this.history[this.history_index & 0x7FFF:].copy_from_slice(s:written)
if n_copied < written.length() {
// a_slice.copy_from(s:b_slice) returns the minimum of the two
// slice lengths. If that value is less than b_slice.length(),
// then not all of b_slice was copied.
//
// In terms of the history ringbuffer, that means that we have
// to wrap around and copy the remainder of written over the
// start of the history ringbuffer.
written = written[n_copied:]
n_copied = this.history[:].copy_from_slice(s:written)
// Set history_index (modulo 0x8000) to the length of this
// remainder. The &0x7FFF is redundant, but proves to the
// compiler that the conversion to u32 will not overflow. The
// +0x8000 is to maintain that the history ringbuffer is full
// if and only if history_index >= 0x8000.
this.history_index = ((n_copied & 0x7FFF) as u32) + 0x8000
} else {
// We didn't need to wrap around.
var already_full u32[..0x8000]
if this.history_index >= 0x8000 {
already_full = 0x8000
}
this.history_index = (this.history_index & 0x7FFF) + ((n_copied & 0x7FFF) as u32) + already_full
}
}
yield z
}
}
pri func flate_decoder.decode_blocks?(dst writer1, src reader1)() {
var final u32
while final == 0 {
while this.n_bits < 3,
post this.n_bits >= 3,
{
this.bits |= (in.src.read_u8?() as u32) << this.n_bits
this.n_bits += 8
}
final = this.bits & 0x01
var type u32 = (this.bits >> 1) & 0x03
this.bits >>= 3
this.n_bits -= 3
if type == 0 {
this.decode_uncompressed?(dst:in.dst, src:in.src)
continue
} else if type == 1 {
this.init_fixed_huffman?()
} else if type == 2 {
this.init_dynamic_huffman?(src:in.src)
} else {
return error "bad flate block"
}
this.end_of_block = false
this.decode_huffman_fast?(dst:in.dst, src:in.src)
if this.end_of_block {
continue
}
this.decode_huffman_slow?(dst:in.dst, src:in.src)
if this.end_of_block {
continue
}
return error "internal error: inconsistent Huffman end_of_block"
}
}
// decode_uncompressed decodes an uncompresed block as per the RFC section
// 3.2.4.
pri func flate_decoder.decode_uncompressed?(dst writer1, src reader1)() {
// TODO: make this "if" into a function invariant?
//
// Ditto for decode_huffman_slow and decode_huffman_fast.
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
return error "internal error: inconsistent n_bits"
}
this.n_bits = 0
this.bits = 0
var length u32 = in.src.read_u32le?()
if (length.low_bits(n:16) + length.high_bits(n:16)) != 0xFFFF {
return error "inconsistent stored block length"
}
length = length.low_bits(n:16)
while true {
var n_copied u32 = in.dst.copy_from_reader32(r:in.src, length:length)
if length <= n_copied {
length = 0
break
}
length -= n_copied
if in.dst.available() == 0 {
// TODO: "closed for writes" instead?
yield suspension "short write"
} else {
yield suspension "short read"
}
}
}
// init_fixed_huffman initializes this.huffs as per the RFC section 3.2.6.
pri func flate_decoder.init_fixed_huffman?()() {
var i u32
while i < 144 {
this.code_lengths[i] = 8
i += 1
}
while i < 256 {
this.code_lengths[i] = 9
i += 1
}
while i < 280 {
this.code_lengths[i] = 7
i += 1
}
while i < 288 {
this.code_lengths[i] = 8
i += 1
}
while i < 320 {
this.code_lengths[i] = 5
i += 1
}
this.init_huff?(which:0, n_codes0:0, n_codes1:288, base_symbol:257)
this.init_huff?(which:1, n_codes0:288, n_codes1:320, base_symbol:0)
}
// init_dynamic_huffman initializes this.huffs as per the RFC section 3.2.7.
pri func flate_decoder.init_dynamic_huffman?(src reader1)() {
var bits u32 = this.bits
var n_bits u32 = this.n_bits
while n_bits < 14,
post n_bits >= 14,
{
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
var n_lit u32[257..288] = bits.low_bits(n:5) + 257
if n_lit > 286 {
return error "bad literal/length code count"
}
bits >>= 5
var n_dist u32[1..32] = bits.low_bits(n:5) + 1
if n_dist > 30 {
return error "bad distance code count"
}
bits >>= 5
var n_clen u32[4..19] = bits.low_bits(n:4) + 4
bits >>= 4
n_bits -= 14
// Read the clcode Huffman table: H-CL.
var i u32
while i < n_clen {
while n_bits < 3,
inv i < n_clen,
post n_bits >= 3,
{
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
assert i < 19 via "a < b: a < c; c <= b"(c:n_clen)
this.code_lengths[code_order[i]] = (bits & 0x07) as u8
bits >>= 3
n_bits -= 3
i += 1
}
while i < 19 {
this.code_lengths[code_order[i]] = 0
i += 1
}
this.init_huff?(which:0, n_codes0:0, n_codes1:19, base_symbol:0xFFF)
// Decode the code lengths for the next two Huffman tables.
var mask u32[..511] = ((1 as u32) << this.n_huffs_bits[0]) - 1
i = 0
while i < (n_lit + n_dist) {
assert i < (288 + 32) via "a < (b + c): a < (b0 + c0); b0 <= b; c0 <= c"(b0:n_lit, c0:n_dist)
// Decode a clcode symbol from H-CL.
var table_entry u32
while true,
inv i < 320,
{
table_entry = this.huffs[0][bits & mask]
var table_entry_n_bits u32[..15] = table_entry & 15
if n_bits >= table_entry_n_bits {
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
break
}
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
// For H-CL, there should be no redirections and all symbols should be
// literals.
if (table_entry >> 24) != 0x80 {
return error "internal error: inconsistent Huffman decoder state"
}
table_entry = (table_entry >> 8) & 0xFF
// Write a literal code length.
if table_entry < 16 {
this.code_lengths[i] = table_entry as u8
i += 1
continue
}
// Write a repeated code length.
var n_extra_bits u32[..7]
var rep_symbol u8[..15]
var rep_count u32
if table_entry == 16 {
n_extra_bits = 2
if i <= 0 {
return error "bad Huffman code length repetition"
}
rep_symbol = this.code_lengths[i - 1]
rep_count = 3
assert rep_count <= 11
} else if table_entry == 17 {
n_extra_bits = 3
rep_symbol = 0
rep_count = 3
assert rep_count <= 11
} else if table_entry == 18 {
n_extra_bits = 7
rep_symbol = 0
rep_count = 11
assert rep_count <= 11
} else {
return error "internal error: inconsistent Huffman decoder state"
}
while n_bits < n_extra_bits,
inv i < 320,
inv rep_count <= 11,
post n_bits >= n_extra_bits,
{
assert n_bits < 7 via "a < b: a < c; c <= b"(c:n_extra_bits)
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
rep_count += bits.low_bits(n:n_extra_bits)
bits >>= n_extra_bits
n_bits -= n_extra_bits
while rep_count > 0 {
// TODO: hoist this check up one level?
if i >= (n_lit + n_dist) {
return error "bad Huffman code length count"
}
assert i < (288 + 32) via "a < (b + c): a < (b0 + c0); b0 <= b; c0 <= c"(b0:n_lit, c0:n_dist)
this.code_lengths[i] = rep_symbol
i += 1
rep_count -= 1
}
}
if i != (n_lit + n_dist) {
return error "bad Huffman code length count"
}
if this.code_lengths[256] == 0 {
return error "missing end-of-block code"
}
this.init_huff?(which:0, n_codes0:0, n_codes1:n_lit, base_symbol:257)
this.init_huff?(which:1, n_codes0:n_lit, n_codes1:n_lit + n_dist, base_symbol:0)
this.bits = bits
this.n_bits = n_bits
}
// TODO: make named constants for 15, 19, 319, etc.
// TODO: skip generating WUFFS_COROUTINE_SUSPENSION_POINTs for *callers* of
// this function, as it can only return errors, not suspensions.
pri func flate_decoder.init_huff?(which u32[..1], n_codes0 u32[..320], n_codes1 u32[..320], base_symbol u32)() {
// For the clcode example in this package's README.md:
// - n_codes0 = 0
// - n_codes1 = 19
// - code_lengths[ 0] = 3
// - code_lengths[ 1] = 0
// - code_lengths[ 2] = 0
// - code_lengths[ 3] = 5
// - code_lengths[ 4] = 3
// - code_lengths[ 5] = 3
// - code_lengths[ 6] = 3
// - code_lengths[ 7] = 3
// - code_lengths[ 8] = 3
// - code_lengths[ 9] = 3
// - code_lengths[10] = 0
// - code_lengths[11] = 0
// - code_lengths[12] = 0
// - code_lengths[13] = 0
// - code_lengths[14] = 0
// - code_lengths[15] = 0
// - code_lengths[16] = 0
// - code_lengths[17] = 4
// - code_lengths[18] = 5
// Calculate counts.
//
// For the clcode example in this package's README.md:
// - counts[0] = 9
// - counts[1] = 0
// - counts[2] = 0
// - counts[3] = 7
// - counts[4] = 1
// - counts[5] = 2
// - all other counts elements are 0.
var counts[16] u16[..320]
var i u32 = in.n_codes0
while i < in.n_codes1 {
assert i < 320 via "a < b: a < c; c <= b"(c:in.n_codes1)
// TODO: this if should be unnecessary. Have some way to assert that,
// for all j, counts[j] <= i, and thus counts[j]++ will not overflow.
if counts[this.code_lengths[i]] >= 320 {
return error "internal error: inconsistent Huffman decoder state"
}
counts[this.code_lengths[i]] += 1
i += 1
}
if ((counts[0] as u32) + in.n_codes0) == in.n_codes1 {
return error "no Huffman codes"
}
// Check that the Huffman code completely covers all possible input bits.
var remaining u32 = 1 // There is 1 possible 0-bit code.
i = 1
while i <= 15 {
if remaining > (1 << 30) {
return error "internal error: inconsistent Huffman decoder state"
}
// Each iteration doubles the number of possible remaining codes.
remaining <<= 1
if remaining < (counts[i] as u32) {
return error "bad Huffman code (over-subscribed)"
}
remaining -= counts[i] as u32
i += 1
}
if remaining != 0 {
// TODO: when is a degenerate Huffman table valid?
return error "bad Huffman code (under-subscribed)"
}
// Calculate offsets and n_symbols.
//
// For the clcode example in this package's README.md:
// - offsets[0] = 0
// - offsets[1] = 0
// - offsets[2] = 0
// - offsets[3] = 0
// - offsets[4] = 7
// - offsets[5] = 8
// - offsets[6] = 10
// - all other offsets elements are 10.
// - n_symbols = 10
var offsets[16] u16[..320]
var n_symbols u32[..320]
i = 1
while i <= 15 {
offsets[i] = n_symbols as u16
var count u32[..320] = counts[i] as u32
if n_symbols > (320 - count) {
return error "internal error: inconsistent Huffman decoder state"
}
assert (n_symbols + count) <= 320 via "(a + b) <= c: a <= (c - b)"()
// TODO: change this to n_symbols += count, once the proof engine's
// bounds checking can handle it.
n_symbols = n_symbols + count
i += 1
}
if n_symbols > 288 {
return error "internal error: inconsistent Huffman decoder state"
}
// Calculate symbols.
//
// For the clcode example in this package's README.md:
// - symbols[0] = 0
// - symbols[1] = 4
// - symbols[2] = 5
// - symbols[3] = 6
// - symbols[4] = 7
// - symbols[5] = 8
// - symbols[6] = 9
// - symbols[7] = 17
// - symbols[8] = 3
// - symbols[9] = 18
//
// As a (local variable) side effect, offsets' values will be updated:
// - offsets[3] = 7, formerly 0
// - offsets[4] = 8, formerly 7
// - offsets[5] = 10, formerly 8
var symbols[320] u16[..319]
i = in.n_codes0
while i < in.n_codes1,
inv n_symbols <= 288,
{
assert i < 320 via "a < b: a < c; c <= b"(c:in.n_codes1)
// TODO: this if check should be unnecessary.
if i < in.n_codes0 {
return error "internal error: inconsistent Huffman decoder state"
}
if this.code_lengths[i] != 0 {
if offsets[this.code_lengths[i]] >= 320 {
return error "internal error: inconsistent Huffman decoder state"
}
symbols[offsets[this.code_lengths[i]]] = (i - in.n_codes0) as u16
offsets[this.code_lengths[i]] += 1
}
i += 1
}
// Calculate min_cl and max_cl.
//
// For the clcode example in this package's README.md:
// - min_cl = 3
// - max_cl = 5
var min_cl u32[1..9] = 1
while true,
inv n_symbols <= 288,
{
if counts[min_cl] != 0 {
break
}
if min_cl >= 9 {
return error "bad Huffman minimum code length"
}
min_cl += 1
}
var max_cl u32[1..15] = 15
while true,
inv n_symbols <= 288,
{
if counts[max_cl] != 0 {
break
}
if max_cl <= 1 {
// TODO: when is a degenerate Huffman table valid?
return error "no Huffman codes"
}
max_cl -= 1
}
if max_cl <= 9 {
this.n_huffs_bits[in.which] = max_cl
} else {
this.n_huffs_bits[in.which] = 9
}
// Calculate this.huffs[in.which].
//
// For the clcode example in this package's README.md:
// - this.huffs[0][0b..000] = 0x80000003 (literal, symbols[0]=0x00, code_length=3)
// - this.huffs[0][0b..100] = 0x80000403 (literal, symbols[1]=0x04, code_length=3)
// - this.huffs[0][0b..010] = 0x80000503 (literal, symbols[2]=0x05, code_length=3)
// - this.huffs[0][0b..110] = 0x80000603 (literal, symbols[3]=0x06, code_length=3)
// - this.huffs[0][0b..001] = 0x80000703 (literal, symbols[4]=0x07, code_length=3)
// - this.huffs[0][0b..101] = 0x80000803 (literal, symbols[5]=0x08, code_length=3)
// - this.huffs[0][0b..011] = 0x80000903 (literal, symbols[6]=0x09, code_length=3)
// - this.huffs[0][0b.0111] = 0x80001104 (literal, symbols[7]=0x11, code_length=4)
// - this.huffs[0][0b01111] = 0x80000305 (literal, symbols[8]=0x03, code_length=5)
// - this.huffs[0][0b11111] = 0x80001805 (literal, symbols[9]=0x18, code_length=5)
i = 0
if (n_symbols != (offsets[max_cl] as u32)) or (n_symbols != (offsets[15] as u32)) {
return error "internal error: inconsistent Huffman decoder state"
}
if (in.n_codes0 + (symbols[0] as u32)) >= 320 {
return error "internal error: inconsistent Huffman decoder state"
}
var initial_high_bits u32 = 1 << 9
if max_cl < 9 {
initial_high_bits = (1 as u32) << max_cl
}
var prev_cl u32[..15] = this.code_lengths[in.n_codes0 + (symbols[0] as u32)] as u32
var prev_redirect_key u32 = 0xFFFFFFFF
var top u32[..1234]
var next_top u32[..1234] = 512
var code u32
var key u32
var value u32
while true,
pre code < (1 << 15),
pre i < 288,
inv n_symbols <= 288,
{
if (in.n_codes0 + (symbols[i] as u32)) >= 320 {
return error "internal error: inconsistent Huffman decoder state"
}
var cl u32[..15] = this.code_lengths[in.n_codes0 + (symbols[i] as u32)] as u32
if cl > prev_cl {
code <<= cl - prev_cl
if code >= (1 << 15) {
return error "internal error: inconsistent Huffman decoder state"
}
}
// For the remainder of this loop body, prev_cl is the original code
// length, cl is possibly clipped by 9, if in the 2nd-level table.
prev_cl = cl
key = code
if cl > 9 {
// TODO: we shouldn't need a temporary variable.
var tmp u32[..6] = cl - 9
cl = tmp
assert cl <= 9 via "a <= b: a == c; c <= b"(c:tmp)
var redirect_key u32[..511] = (key >> tmp) & 511
key = key.low_bits(n:tmp)
if prev_redirect_key != redirect_key {
prev_redirect_key = redirect_key
// Calculate the number of bits needed for the 2nd level table.
// This computation is similar to "check that the Huffman code
// completely covers all possible input bits" above.
remaining = (1 as u32) << cl
var j u32 = prev_cl
while j <= 15,
inv cl <= 9,
inv code < (1 << 15),
inv i < 288,
inv n_symbols <= 288,
{
if remaining <= (counts[j] as u32) {
break
}
remaining -= counts[j] as u32
if remaining > (1 << 30) {
return error "internal error: inconsistent Huffman decoder state"
}
remaining <<= 1
j += 1
}
if (j <= 9) or (15 < j) {
return error "internal error: inconsistent Huffman decoder state"
}
tmp = j - 9
initial_high_bits = (1 as u32) << tmp
top = next_top
if (top + ((1 as u32) << tmp)) > 1234 {
return error "internal error: inconsistent Huffman decoder state"
}
next_top = top + ((1 as u32) << tmp)
redirect_key = (reverse8[redirect_key >> 1] as u32) | ((redirect_key & 1) << 8)
this.huffs[in.which][redirect_key] = 0x10000009 | (top << 8) | (tmp << 4)
}
}
if (key >= (1 << 9)) or (counts[prev_cl] <= 0) {
return error "internal error: inconsistent Huffman decoder state"
}
counts[prev_cl] -= 1
var reversed_key u32[..511] = (reverse8[key >> 1] as u32) | ((key & 1) << 8)
reversed_key >>= 9 - cl
var symbol u32[..319] = symbols[i] as u32
if symbol == 256 {
// End-of-block.
value = 0x20000000 | cl
} else if (symbol < 256) and (in.which == 0) {
// Literal.
value = 0x80000000 | (symbol << 8) | cl
} else if symbol >= in.base_symbol {
// Base number + extra bits.
symbol -= in.base_symbol
if in.which == 0 {
value = lcode_magic_numbers[symbol & 31] | cl
} else {
value = dcode_magic_numbers[symbol & 31] | cl
}
} else {
return error "internal error: inconsistent Huffman decoder state"
}
// The table uses log2(initial_high_bits) bits, but reversed_key only
// has cl bits. We duplicate the key-value pair across all possible
// values of the high (log2(initial_high_bits) - cl) bits.
var high_bits u32 = initial_high_bits
var delta u32 = (1 as u32) << cl
while high_bits >= delta,
inv code < (1 << 15),
inv i < 288,
inv n_symbols <= 288,
{
high_bits -= delta
if (top + ((high_bits | reversed_key) & 511)) >= 1234 {
return error "internal error: inconsistent Huffman decoder state"
}
this.huffs[in.which][top + ((high_bits | reversed_key) & 511)] = value
}
i += 1
if i >= n_symbols {
break
}
assert i < 288 via "a < b: a < c; c <= b"(c:n_symbols)
code += 1
if code >= (1 << 15) {
return error "internal error: inconsistent Huffman decoder state"
}
}
}
+384
View File
@@ -0,0 +1,384 @@
// Copyright 2017 The Wuffs Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// TODO: describe how the xxx_fast version differs from the xxx_slow one, the
// assumptions that xxx_fast makes, and how that makes it fast.
pri func flate_decoder.decode_huffman_fast?(dst writer1, src reader1)() {
// When editing this function, consider making the equivalent change to the
// decode_huffman_slow function. Keep the diff between the two
// decode_huffman_*.wuffs files as small as possible, while retaining both
// correctness and performance.
// TODO: make this a func pre-condition.
if not in.dst.is_marked() {
return error "bad argument"
}
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
return error "internal error: inconsistent n_bits"
}
var bits u32 = this.bits
var n_bits u32 = this.n_bits
var table_entry u32
var table_entry_n_bits u32[..15]
var lmask u32[..511] = ((1 as u32) << this.n_huffs_bits[0]) - 1
var dmask u32[..511] = ((1 as u32) << this.n_huffs_bits[1]) - 1
// Check up front, on each iteration, that we have enough buffer space to
// both read (12 bytes) and write (258 bytes) as much as we need to. Doing
// this check once (per iteration), up front, removes the need to check
// (and possibly suspend the coroutine) multiple times inside the loop
// body, so it's faster overall.
//
// For writing, a literal code obviously corresponds to writing 1 byte, and
// 258 is the maximum length in a length-distance pair, as specified in the
// RFC section 3.2.5. Compressed blocks (length and distance codes).
//
// For reading, strictly speaking, we only need 6 bytes (48 bits) of
// available input, because the H-L length/literal code is up to 15 bits
// plus up to 5 extra bits, the H-D distance code is up to 15 bits plus up
// to 13 extra bits and 15 + 5 + 15 + 13 == 48. However, in.src.available()
// assertions are made in terms of bytes, not bits.
//
// In theory, we could write assertions that mixed bytes and bits, such as:
//
// assert ((in.src.available() >= 6) and (n_bits >= 5)) or
// ((in.src.available() >= 4) and (n_bits >= 21))
//
// but that looks clumsy and makes the proofs more complicated. Instead, we
// are conservative and work in terms of bytes only. In bytes, each code
// requires either one (direct) or two (redirect) lookups, up to 2 bytes
// for each. There are also up to 13 extra bits, or 2 bytes. Thus, each
// code requires up to 2 + 2 + 2 == 6 bytes, and there are two codes (one
// from H-L and one from H-D). Therefore, we check for at least 12 bytes.
while:loop(in.dst.available() >= 258) and (in.src.available() >= 12),
inv in.dst.is_marked(),
{
// Ensure that we have at least 15 bits of input.
if n_bits < 15 {
assert n_bits >= 0 // TODO: this shouldn't be necessary.
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
assert n_bits >= 15
} else {
assert in.src.available() >= 10
}
// These assertions are redundant, but are listed explicitly for
// clarity. They must hold regardless of which branch was taken for the
// if statement above.
assert in.src.available() >= 10
assert n_bits >= 15
// Decode an lcode symbol from H-L.
table_entry = this.huffs[0][bits & lmask]
table_entry_n_bits = table_entry & 0x0F
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
if (table_entry >> 31) != 0 {
// Literal.
in.dst.write_u8?(x:(table_entry >> 8) & 0xFF)
continue:loop
} else if (table_entry >> 30) != 0 {
// No-op; code continues past the if-else chain.
assert in.src.available() >= 8
} else if (table_entry >> 29) != 0 {
// End of block.
this.end_of_block = true
break:loop
} else if (table_entry >> 28) != 0 {
// Redirect.
// Ensure that we have at least 15 bits of input.
if n_bits < 15 {
assert n_bits >= 0 // TODO: this shouldn't be necessary.
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
assert n_bits >= 15
} else {
assert in.src.available() >= 8
}
// Once again, redundant but explicit assertions.
assert in.src.available() >= 8
assert n_bits >= 15
var redir_top u32[..0xFFFF] = (table_entry >> 8) & 0xFFFF
var redir_mask u32[..0x7FFF] = ((1 as u32) << ((table_entry >> 4) & 0x0F)) - 1
if (redir_top + (bits & redir_mask)) >= 1234 {
return error "internal error: inconsistent Huffman decoder state"
}
table_entry = this.huffs[0][redir_top + (bits & redir_mask)]
table_entry_n_bits = table_entry & 0x0F
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
if (table_entry >> 31) != 0 {
// Literal.
in.dst.write_u8?(x:(table_entry >> 8) & 0xFF)
continue:loop
} else if (table_entry >> 30) != 0 {
// No-op; code continues past the if-else chain.
} else if (table_entry >> 29) != 0 {
// End of block.
this.end_of_block = true
break:loop
} else if (table_entry >> 28) != 0 {
return error "internal error: inconsistent Huffman decoder state"
} else if (table_entry >> 27) != 0 {
return error "bad Huffman code"
} else {
return error "internal error: inconsistent Huffman decoder state"
}
// Once again, redundant but explicit assertions.
assert in.src.available() >= 8
} else if (table_entry >> 27) != 0 {
return error "bad Huffman code"
} else {
return error "internal error: inconsistent Huffman decoder state"
}
// length = base number + extra bits.
var length u32[..0x7FFF] = (table_entry >> 8) & 0x7FFF
table_entry_n_bits = (table_entry >> 4) & 0x0F
if table_entry_n_bits > 0 {
// Ensure that we have at least 15 bits of input.
if n_bits < 15 {
assert n_bits >= 0 // TODO: this shouldn't be necessary.
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
assert n_bits >= 15
} else {
assert in.src.available() >= 6
}
// Once again, redundant but explicit assertions.
assert in.src.available() >= 6
assert n_bits >= 15
length = (length + bits.low_bits(n:table_entry_n_bits)) & 0x7FFF
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
} else {
assert in.src.available() >= 6
}
if length > 258 {
return error "internal error: inconsistent Huffman decoder state"
}
// Ensure that we have at least 15 bits of input.
if n_bits < 15 {
assert n_bits >= 0 // TODO: this shouldn't be necessary.
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
assert n_bits >= 15
} else {
assert in.src.available() >= 4
}
// Once again, redundant but explicit assertions.
assert in.src.available() >= 4
assert n_bits >= 15
// Decode a dcode symbol from H-D.
table_entry = this.huffs[1][bits & dmask]
table_entry_n_bits = table_entry & 15
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
// Check for a redirect.
if (table_entry >> 28) == 1 {
// Ensure that we have at least 15 bits of input.
if n_bits < 15 {
assert n_bits >= 0 // TODO: this shouldn't be necessary.
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
assert n_bits >= 15
} else {
assert in.src.available() >= 2
}
// Once again, redundant but explicit assertions.
assert in.src.available() >= 2
assert n_bits >= 15
redir_top = (table_entry >> 8) & 0xFFFF
redir_mask = ((1 as u32) << ((table_entry >> 4) & 0x0F)) - 1
if (redir_top + (bits & redir_mask)) >= 1234 {
return error "internal error: inconsistent Huffman decoder state"
}
table_entry = this.huffs[1][redir_top + (bits & redir_mask)]
table_entry_n_bits = table_entry & 0x0F
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
} else {
assert in.src.available() >= 2
}
// For H-D, all symbols should be base number + extra bits.
if (table_entry >> 24) != 0x40 {
if (table_entry >> 24) == 0x08 {
return error "bad Huffman code"
}
return error "internal error: inconsistent Huffman decoder state"
}
// distance = base number + extra bits.
var distance u32[..0x7FFF] = (table_entry >> 8) & 0x7FFF
table_entry_n_bits = (table_entry >> 4) & 0x0F
if table_entry_n_bits > 0 {
// Ensure that we have at least 15 bits of input.
if n_bits < 15 {
assert n_bits >= 0 // TODO: this shouldn't be necessary.
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
assert n_bits >= 15
}
// Once again, redundant but explicit assertions.
assert n_bits >= 15
distance = (distance + bits.low_bits(n:table_entry_n_bits)) & 0x7FFF
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
}
if distance <= 0 {
return error "internal error: inconsistent Huffman decoder state"
}
var n_copied u32
// The "while true { etc; break }" is a redundant version of "etc", but
// its presence minimizes the diff between decode_huffman_fast and
// decode_huffman_slow.
while true,
pre length <= 258,
pre distance > 0,
pre in.dst.available() >= 258,
inv in.dst.is_marked(),
{
// Copy from this.history.
if (distance as u64) > in.dst.since_mark().length() {
// Set (hlen, hdist) to be the length-distance pair to copy
// from this.history, and (length, distance) to be the
// remaining length-distance pair to copy from in.dst.
var hlen u32[..0x7FFF]
var hdist u32[..0x7FFF] = ((distance as u64) - in.dst.since_mark().length()) as u32
if length > hdist {
length -= hdist
hlen = hdist
// TODO: this if check should be redundant.
if length > 258 {
return error "internal error: inconsistent Huffman decoder state"
}
} else {
hlen = length
length = 0
assert length <= 258
}
if this.history_index < hdist {
return error "bad distance"
}
// Re-purpose the hdist variable as the this.history index to
// start copying from.
hdist = (this.history_index - hdist) & 0x7FFF
// Copy from hdist to the end of this.history, and then
// possibly wrap around the this.history ringbuffer.
//
// The "while true { etc; break }" is a redundant version of
// "etc", but its presence minimizes the diff between
// decode_huffman_fast and decode_huffman_slow.
while true,
inv length <= 258,
inv distance > 0,
inv in.dst.available() >= 258,
inv in.dst.is_marked(),
{
// TODO: copy_from_slice32 should probably update the
// "in.dst.available() >= 258" fact. It should certainly
// invalidate any fact that was "<=" instead of ">=".
n_copied = in.dst.copy_from_slice32(s:this.history[hdist:], length:hlen)
if hlen <= n_copied {
break
}
hlen -= n_copied
in.dst.copy_from_slice32(s:this.history[:], length:hlen)
break
}
if length == 0 {
// No need to copy from in.dst.
continue:loop
}
if (distance as u64) > in.dst.since_mark().length() {
return error "internal error: inconsistent distance"
}
}
// Once again, redundant but explicit assertions.
assert length <= 258
assert distance > 0
assert (distance as u64) <= in.dst.since_mark().length()
assert in.dst.available() >= 258
assert in.dst.is_marked()
// We can therefore prove:
assert (length as u64) <= 258
assert (length as u64) <= in.dst.available() via "a <= b: a <= c; c <= b"(c:258)
// Copy from in.dst.
in.dst.copy_from_history32(distance:distance, length:length)
break
}
}
// Ensure n_bits < 8 by rewindng in.src, if we loaded too many of its bytes
// into the bits variable.
//
// Note that we can unconditionally call unread_u8 (without resulting in an
// "invalid I/O operation" error code) only because this whole function can
// never suspend, as all of its I/O operations were checked beforehand for
// sufficient buffer space. Otherwise, resuming from the suspension could
// mean that the (possibly different) in.src is no longer rewindable, even
// if conceptually, this function was responsible for reading the bytes we
// want to rewind.
//
// TODO: have the compiler enforce (via some explicit syntax) that this
// function is ProvenNotToSuspend?
while n_bits >= 8,
post n_bits < 8,
{
n_bits -= 8
in.src.unread_u8?()
}
this.bits = bits & (((1 as u32) << n_bits) - 1)
this.n_bits = n_bits
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
return error "internal error: inconsistent n_bits"
}
}
+258
View File
@@ -0,0 +1,258 @@
// Copyright 2017 The Wuffs Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pri func flate_decoder.decode_huffman_slow?(dst writer1, src reader1)() {
// When editing this function, consider making the equivalent change to the
// decode_huffman_fast function. Keep the diff between the two
// decode_huffman_*.wuffs files as small as possible, while retaining both
// correctness and performance.
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
return error "internal error: inconsistent n_bits"
}
var bits u32 = this.bits
var n_bits u32 = this.n_bits
var table_entry u32
var table_entry_n_bits u32[..15]
var lmask u32[..511] = ((1 as u32) << this.n_huffs_bits[0]) - 1
var dmask u32[..511] = ((1 as u32) << this.n_huffs_bits[1]) - 1
while:loop true {
// Decode an lcode symbol from H-L.
while true {
table_entry = this.huffs[0][bits & lmask]
table_entry_n_bits = table_entry & 0x0F
if n_bits >= table_entry_n_bits {
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
break
}
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
if (table_entry >> 31) != 0 {
// Literal.
in.dst.write_u8?(x:(table_entry >> 8) & 0xFF)
continue:loop
} else if (table_entry >> 30) != 0 {
// No-op; code continues past the if-else chain.
} else if (table_entry >> 29) != 0 {
// End of block.
this.end_of_block = true
break:loop
} else if (table_entry >> 28) != 0 {
// Redirect.
var redir_top u32[..0xFFFF] = (table_entry >> 8) & 0xFFFF
var redir_mask u32[..0x7FFF] = ((1 as u32) << ((table_entry >> 4) & 0x0F)) - 1
while true {
if (redir_top + (bits & redir_mask)) >= 1234 {
return error "internal error: inconsistent Huffman decoder state"
}
table_entry = this.huffs[0][redir_top + (bits & redir_mask)]
table_entry_n_bits = table_entry & 0x0F
if n_bits >= table_entry_n_bits {
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
break
}
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
if (table_entry >> 31) != 0 {
// Literal.
in.dst.write_u8?(x:(table_entry >> 8) & 0xFF)
continue:loop
} else if (table_entry >> 30) != 0 {
// No-op; code continues past the if-else chain.
} else if (table_entry >> 29) != 0 {
// End of block.
this.end_of_block = true
break:loop
} else if (table_entry >> 28) != 0 {
return error "internal error: inconsistent Huffman decoder state"
} else if (table_entry >> 27) != 0 {
return error "bad Huffman code"
} else {
return error "internal error: inconsistent Huffman decoder state"
}
} else if (table_entry >> 27) != 0 {
return error "bad Huffman code"
} else {
return error "internal error: inconsistent Huffman decoder state"
}
// length = base number + extra bits.
var length u32[..0x7FFF] = (table_entry >> 8) & 0x7FFF
table_entry_n_bits = (table_entry >> 4) & 0x0F
if table_entry_n_bits > 0 {
while n_bits < table_entry_n_bits,
post n_bits >= table_entry_n_bits,
{
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
length = (length + bits.low_bits(n:table_entry_n_bits)) & 0x7FFF
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
}
// Decode a dcode symbol from H-D.
while true {
table_entry = this.huffs[1][bits & dmask]
table_entry_n_bits = table_entry & 15
if n_bits >= table_entry_n_bits {
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
break
}
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
// Check for a redirect.
if (table_entry >> 28) == 1 {
redir_top = (table_entry >> 8) & 0xFFFF
redir_mask = ((1 as u32) << ((table_entry >> 4) & 0x0F)) - 1
while true {
if (redir_top + (bits & redir_mask)) >= 1234 {
return error "internal error: inconsistent Huffman decoder state"
}
table_entry = this.huffs[1][redir_top + (bits & redir_mask)]
table_entry_n_bits = table_entry & 0x0F
if n_bits >= table_entry_n_bits {
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
break
}
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
}
// For H-D, all symbols should be base number + extra bits.
if (table_entry >> 24) != 0x40 {
if (table_entry >> 24) == 0x08 {
return error "bad Huffman code"
}
return error "internal error: inconsistent Huffman decoder state"
}
// distance = base number + extra bits.
var distance u32[..0x7FFF] = (table_entry >> 8) & 0x7FFF
table_entry_n_bits = (table_entry >> 4) & 0x0F
if table_entry_n_bits > 0 {
while n_bits < table_entry_n_bits,
post n_bits >= table_entry_n_bits,
{
assert n_bits < 15 via "a < b: a < c; c <= b"(c:table_entry_n_bits)
bits |= (in.src.read_u8?() as u32) << n_bits
n_bits += 8
}
distance = (distance + bits.low_bits(n:table_entry_n_bits)) & 0x7FFF
bits >>= table_entry_n_bits
n_bits -= table_entry_n_bits
}
var n_copied u32
while true {
// Copy from this.history.
if (distance as u64) > in.dst.since_mark().length() {
// Set (hlen, hdist) to be the length-distance pair to copy
// from this.history, and (length, distance) to be the
// remaining length-distance pair to copy from in.dst.
var hlen u32[..0x7FFF]
var hdist u32[..0x7FFF] = ((distance as u64) - in.dst.since_mark().length()) as u32
if length > hdist {
length -= hdist
hlen = hdist
} else {
hlen = length
length = 0
}
if this.history_index < hdist {
return error "bad distance"
}
// Re-purpose the hdist variable as the this.history index to
// start copying from.
hdist = (this.history_index - hdist) & 0x7FFF
// Copy from hdist to the end of this.history.
while true {
n_copied = in.dst.copy_from_slice32(s:this.history[hdist:], length:hlen)
if hlen <= n_copied {
hlen = 0
break
}
if n_copied > 0 {
hlen -= n_copied
hdist = (hdist ~+ n_copied) & 0x7FFF
if hdist == 0 {
// Wrap around the this.history ringbuffer.
break
}
}
// TODO: "closed for writes" instead?
yield suspension "short write"
}
// Copy from the start of this.history, if we wrapped around.
if hlen > 0 {
while true {
n_copied = in.dst.copy_from_slice32(s:this.history[hdist:], length:hlen)
if hlen <= n_copied {
hlen = 0
break
}
hlen -= n_copied
hdist = (hdist + (n_copied & 0x7FFF)) & 0x7FFF
// TODO: "closed for writes" instead?
yield suspension "short write"
}
}
if length == 0 {
// No need to copy from in.dst.
continue:loop
}
}
// Copy from in.dst.
n_copied = in.dst.copy_from_history32(distance:distance, length:length)
if length <= n_copied {
length = 0
break
}
length -= n_copied
// TODO: "closed for writes" instead?
yield suspension "short write"
}
}
// TODO: "assert n_bits < 8"? What about (bits >> n_bits)?
this.bits = bits
this.n_bits = n_bits
if (this.n_bits >= 8) or ((this.bits >> this.n_bits) != 0) {
return error "internal error: inconsistent n_bits"
}
}