mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-19 08:38:14 +01:00
std/brotli: large-window mode, custom dictionaries and a fuzzer
Completes the decoder's feature set relative to libbrotlidec v1.2.1: - QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the large-window WBITS escape, window bits up to 30, the 544-symbol distance alphabet and 64-bit distance arithmetic, with workbuf_len growing to (1 << 30) + 542 when the quirk is set. - decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw (LZ77 prefix) and serialized shared dictionaries: compound-dictionary distance mapping, the shared-dictionary parser (custom word lists, transform lists including the SHIFT transforms, context-dependent dictionary selection, prefix dictionaries) and all of its validation. Wuffs structs cannot hold slices, so the dictionary bytes are copied into the front of the caller's workbuf; the layout is documented in the README. Serialized dictionaries are accepted as in a BROTLI_EXPERIMENTAL build of libbrotli. - fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and random I/O limits. Tests: new golden vectors for large windows (including a 17 MiB far-distance stream generated in the test), raw and serialized dictionaries, nine more malformed inputs, and mimic comparisons that attach the same dictionaries to libbrotlidec. Differential runs over 2.7 million mutated large-window and dictionary streams found no divergence from libbrotlidec. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
This commit is contained in:
43 files changed
+4253
-460
No files matched your search
+160
-13
@@ -49,8 +49,10 @@ decoder does not read any further than that.
|
||||
|
||||
## Workbuf Layout
|
||||
|
||||
`workbuf_len` is `(1 << 24) + 542` bytes, both minimum and maximum: the
|
||||
largest (non-large-window) brotli window size plus some slack. The slack
|
||||
Without the large-window quirk and without attached dictionaries (see the
|
||||
"Large Window" and "Dictionaries" sections below), `workbuf_len` is `(1 << 24)
|
||||
+ 542` bytes, both minimum and maximum: the largest (non-large-window) brotli
|
||||
window size plus some slack. The slack
|
||||
mirrors the C code's `kRingBufferWriteAheadSlack`, which its fast paths use to
|
||||
write (and discard) a few bytes past the ring buffer's end. This
|
||||
implementation does not yet have such fast paths, but it still reserves the
|
||||
@@ -101,15 +103,18 @@ a corresponding Wuffs status:
|
||||
| `BROTLI_DECODER_ERROR_FORMAT_PADDING_2` | `"#bad final padding"` |
|
||||
| `BROTLI_DECODER_ERROR_FORMAT_DISTANCE` | `"#bad distance"` |
|
||||
| `BROTLI_DECODER_ERROR_FORMAT_BLOCK_SWITCH` | `"#bad block switch"` |
|
||||
| `BROTLI_DECODER_ERROR_COMPOUND_DICTIONARY` | `"#bad compound dictionary"` |
|
||||
| `BROTLI_DECODER_NEEDS_MORE_INPUT` (with closed `src`) | `"#truncated input"` |
|
||||
|
||||
Notes:
|
||||
|
||||
- `"#unsupported large window"` is for the large-window escape `WBITS` value
|
||||
(`0010001`). libbrotlidec (with `BROTLI_DECODER_PARAM_LARGE_WINDOW` not set,
|
||||
the default) rejects it with `BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS`. For
|
||||
standard (non-large-window) brotli, no other `WBITS` value is invalid, so
|
||||
`"#bad window bits"` is not currently produced.
|
||||
(`0010001`) when the `QUIRK_LARGE_WINDOW` quirk is not set. libbrotlidec
|
||||
(with `BROTLI_DECODER_PARAM_LARGE_WINDOW` not set, the default) rejects it
|
||||
with `BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS`. For standard
|
||||
(non-large-window) brotli, no other `WBITS` value is invalid. With the quirk
|
||||
set, `"#bad window bits"` is for a large-window header with a non-zero
|
||||
reserved bit or with window bits outside of `[10 ..= 30]`.
|
||||
- Like the C code, which error an overrun metablock produces (`"#bad block
|
||||
length"` or `"#bad metablock length"`) can depend on how the input is split
|
||||
across `transform_io` calls. When the C code runs out of input, it first
|
||||
@@ -118,8 +123,9 @@ Notes:
|
||||
running out of input.
|
||||
- `"#bad block switch"` needs more than `(1 << 24)` commands in one metablock.
|
||||
It is implemented but not tested.
|
||||
- The remaining libbrotlidec error codes do not apply: `COMPOUND_DICTIONARY`
|
||||
and `DICTIONARY_NOT_SET` (custom dictionaries are not supported),
|
||||
- The remaining libbrotlidec error codes do not apply: `DICTIONARY_NOT_SET`
|
||||
(it needs a word list without data, which libbrotlidec only has when built
|
||||
with `BROTLI_EXTERNAL_DICTIONARY_DATA` and not given that data),
|
||||
`INVALID_ARGUMENTS`, the `ALLOC_*` codes (this decoder does not allocate)
|
||||
and `UNREACHABLE`.
|
||||
|
||||
@@ -145,10 +151,151 @@ invariants for memory safety but Wuffs requires explicit bounds checks.
|
||||
|
||||
## TODO
|
||||
|
||||
- Large-window brotli (`BROTLI_DECODER_PARAM_LARGE_WINDOW`, window bits up to
|
||||
30). The window-related fields are sized for 24 bits.
|
||||
- Custom, shared and compound dictionaries.
|
||||
- Metadata callbacks (metadata blocks are currently skipped).
|
||||
- Metadata callbacks (metadata blocks are currently skipped). See the
|
||||
"Metadata Blocks" section below.
|
||||
- Fast paths.
|
||||
- Fuzzers.
|
||||
- An encoder.
|
||||
|
||||
|
||||
## Large Window
|
||||
|
||||
Large-window brotli (an incompatible extension of RFC 7932, with window sizes
|
||||
up to `(1 << 30)` bytes) is enabled by the `QUIRK_LARGE_WINDOW` quirk. It
|
||||
mirrors libbrotlidec's `BROTLI_DECODER_PARAM_LARGE_WINDOW` parameter: it must
|
||||
be set before decoding starts (afterwards, `set_quirk` returns `"#bad call
|
||||
sequence"`, like `BrotliDecoderSetParameter` failing once the decoder has
|
||||
left its `BROTLI_STATE_UNINITED` state). With the quirk set:
|
||||
|
||||
- a stream that starts with the large-window `WBITS` escape is decoded as a
|
||||
large-window stream: the escape is followed by a reserved bit (that must be
|
||||
zero) and 6 window bits (that must be in `[10 ..= 30]`), otherwise the
|
||||
decoder fails with `"#bad window bits"`.
|
||||
- a stream that does not start with that escape is decoded as standard brotli,
|
||||
exactly as without the quirk.
|
||||
- `workbuf_len` is `(1 << 30) + 542` bytes (plus any dictionaries), the
|
||||
largest large-window window size plus slack. As before, the workbuf only
|
||||
needs to be as long as the stream's actual window size plus 542 bytes (plus
|
||||
any dictionaries). Without the quirk, it is `(1 << 24) + 542` bytes (plus
|
||||
any dictionaries), as before.
|
||||
|
||||
For a large-window stream, like the C code, the distance alphabet's
|
||||
`alphabet_size_max` uses `BROTLI_LARGE_MAX_DISTANCE_BITS` (62) and its
|
||||
`alphabet_size_limit` comes from `BrotliCalculateDistanceCodeLimit` (at most
|
||||
544 symbols, so that no distance exceeds `BROTLI_MAX_ALLOWED_DISTANCE`). Up to
|
||||
29 extra distance bits are read (the C code's `BrotliSafeReadBits32`) and the
|
||||
distance is calculated with 64-bit arithmetic, like the C code's
|
||||
`brotli_reg_t`. The Huffman table stride for distance trees is sized for 544
|
||||
symbols in both modes.
|
||||
|
||||
The `DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE` constant is now the overall
|
||||
worst case: `(1 << 30) + 542` plus the maximum total dictionary length
|
||||
(`0x7FFF_FFFF`). The new `DECODER_WORKBUF_LEN_STANDARD` and
|
||||
`DECODER_WORKBUF_LEN_LARGE_WINDOW` constants give the window-only values.
|
||||
|
||||
|
||||
## Dictionaries
|
||||
|
||||
`decoder.add_dictionary!(dict_type: base.u32, dict: roslice base.u8, workbuf:
|
||||
slice base.u8) base.status` mirrors libbrotlidec's
|
||||
`BrotliDecoderAttachDictionary`. `dict_type` is `DICTIONARY_TYPE_RAW` (0, the
|
||||
C code's `BROTLI_SHARED_DICTIONARY_RAW`) or `DICTIONARY_TYPE_SERIALIZED` (1,
|
||||
`BROTLI_SHARED_DICTIONARY_SERIALIZED`). It must be called before decoding
|
||||
starts. It can be called multiple times.
|
||||
|
||||
Wuffs structs cannot hold pointers or slices, so `add_dictionary` copies the
|
||||
dictionary bytes into the workbuf. Every `add_dictionary` and `transform_io`
|
||||
call must be passed the same workbuf (or at least one whose first bytes hold
|
||||
the same dictionary bytes). The `add_dictionary` workbuf only needs to be long
|
||||
enough to hold all of the dictionaries added so far. Call `workbuf_len` after
|
||||
the last `add_dictionary` call: it grows by each dictionary's length.
|
||||
|
||||
The workbuf layout is:
|
||||
|
||||
[0 .. D) the attached dictionaries, concatenated, where D is
|
||||
the total length of all successfully added
|
||||
dictionaries (raw or serialized, in full).
|
||||
[D .. D + W + 542) the history ring buffer (W = 1 << WBITS) and slack.
|
||||
|
||||
The dictionaries come first because their position should not depend on the
|
||||
stream's window size, which is only known once decoding has started.
|
||||
|
||||
Raw dictionaries are LZ77 prefix dictionaries. Like the C code (its
|
||||
`BrotliDecoderCompoundDictionary` and `AttachCompoundDictionary`,
|
||||
`InitializeCompoundDictionaryCopy` and `CopyFromCompoundDictionary`
|
||||
functions), at most 15 such "compound dictionary" chunks (raw dictionaries
|
||||
plus serialized dictionaries' LZ77 prefixes) can be attached. A backward
|
||||
reference distance `d` beyond the current maximum LZ77 distance `m` (but
|
||||
within `BROTLI_MAX_ALLOWED_DISTANCE`) refers to the compound dictionary when
|
||||
`d - m - 1` is less than the compound dictionary's total size `T`, at address
|
||||
`T - (d - m)`. A copy that would extend past the compound dictionary's end
|
||||
fails with `"#bad compound dictionary"`. Otherwise, the static dictionary's
|
||||
address is `d - m - 1 - T`.
|
||||
|
||||
Serialized dictionaries are parsed and validated like
|
||||
`BrotliSharedDictionaryAttach` (`DecodeSharedDictionary`,
|
||||
`DryParseDictionary`, `ParseDictionary`, `ParseWordList`,
|
||||
`ParseTransformsList` and `ParsePrefixSuffixTable` in
|
||||
`c/common/shared_dictionary.c`): the `0x91 0x00` magic, an optional LZ77
|
||||
prefix (at most 1073741823 bytes), up to 64 custom word lists (word lengths 4
|
||||
to 31, at most 15 size bits per length), up to 64 custom transform lists (up
|
||||
to 256 prefix/suffix stringlets, up to 255 transforms, transform types up to
|
||||
`BROTLI_TRANSFORM_SHIFT_ALL`, with 16-bit parameters for the
|
||||
`BROTLI_TRANSFORM_SHIFT_FIRST` and `BROTLI_TRANSFORM_SHIFT_ALL` types), up to
|
||||
64 (word list, transform list) combinations and an optional 64-entry context
|
||||
map. The parsed metadata lives in fixed-size decoder fields (`wl_etc`, `tl_etc`
|
||||
and `sd_etc`), with offsets into the copied bytes in the workbuf. Like the C
|
||||
code:
|
||||
|
||||
- only one attached serialized dictionary can have custom word lists or
|
||||
transform lists (otherwise `"#unsupported multiple custom static
|
||||
dictionaries"`), and
|
||||
- attaching a serialized dictionary without custom word lists or transform
|
||||
lists (only an LZ77 prefix) resets the static dictionary to the built-in
|
||||
(RFC 7932) one, even if an earlier serialized dictionary had custom lists.
|
||||
|
||||
Static dictionary references then mirror `ProcessCommandsInternal`: the word
|
||||
list and transform list are selected by the literal context (`BROTLI_CONTEXT(p1,
|
||||
p2, context_lookup)`) through the context map (if enabled), with the C code's
|
||||
fall back to the other (word list, transform list) combinations when the
|
||||
address is out of range, and words are transformed like
|
||||
`BrotliTransformDictionaryWord` (including `Shift` for the shift transforms).
|
||||
Transformed words can be up to 541 bytes long.
|
||||
|
||||
`add_dictionary` statuses (the C code just returns `BROTLI_FALSE`):
|
||||
|
||||
| Condition | Wuffs status |
|
||||
| -------------------------------------------------------------- | ----------------------------------------------------- |
|
||||
| decoding has started | `base."#bad call sequence"` |
|
||||
| unknown `dict_type` | `base."#bad argument"` |
|
||||
| more than 15 compound dictionary chunks | `"#too many compound dictionaries"` |
|
||||
| invalid serialized dictionary | `"#bad shared dictionary"` |
|
||||
| a second serialized dictionary with custom lists | `"#unsupported multiple custom static dictionaries"` |
|
||||
| more than `0x7FFF_FFFF` bytes of dictionaries in total | `"#too much dictionary data"` |
|
||||
| `workbuf` shorter than the dictionaries so far plus this one | `base."#bad workbuf length"` |
|
||||
|
||||
Differences from the C code:
|
||||
|
||||
- On failure, `add_dictionary` leaves the decoder unchanged. libbrotlidec's
|
||||
`BrotliDecoderAttachDictionary` can fail after partially updating its shared
|
||||
dictionary (e.g. after counting a serialized dictionary's LZ77 prefix or
|
||||
reading its number of word lists), which can affect later attach calls.
|
||||
- libbrotlicommon only parses serialized dictionaries when built with
|
||||
`BROTLI_EXPERIMENTAL`, which is not the default. A default build rejects
|
||||
every serialized dictionary. This package always accepts them, matching an
|
||||
experimental build. The mimic tests skip serialized dictionary cases when
|
||||
the linked libbrotlidec rejects them; they pass, with no skips, when linked
|
||||
against a `BROTLI_EXPERIMENTAL` build.
|
||||
- The C code holds the compound dictionary's total size in an `int`. This
|
||||
package rejects more than `0x7FFF_FFFF` bytes of dictionaries in total
|
||||
(counting serialized dictionaries in full), where the C code would overflow.
|
||||
- `BROTLI_DECODER_ERROR_DICTIONARY_NOT_SET` cannot happen (see above).
|
||||
|
||||
|
||||
## Metadata Blocks
|
||||
|
||||
libbrotlidec can report metadata blocks' contents through callbacks
|
||||
(`BrotliDecoderSetMetadataCallbacks`). Wuffs code cannot call back into its
|
||||
caller, so there is no equivalent: metadata blocks are skipped (validated and
|
||||
consumed but not reported), exactly as libbrotlidec does without callbacks,
|
||||
so the decoded output is the same. A future quirk could expose metadata
|
||||
blocks, e.g. by suspending with a status when one starts.
|
||||
+234
-40
@@ -20,12 +20,12 @@
|
||||
//
|
||||
// See README.md for how C error codes map to Wuffs statuses.
|
||||
//
|
||||
// TODO: support large-window brotli (BROTLI_DECODER_PARAM_LARGE_WINDOW),
|
||||
// custom / attached / shared dictionaries, metadata callbacks and fast paths.
|
||||
// TODO: support metadata callbacks and fast paths.
|
||||
|
||||
pub status "#bad block length"
|
||||
pub status "#bad block switch"
|
||||
pub status "#bad code length code lengths"
|
||||
pub status "#bad compound dictionary"
|
||||
pub status "#bad context map repeat"
|
||||
pub status "#bad dictionary"
|
||||
pub status "#bad distance"
|
||||
@@ -36,12 +36,16 @@ pub status "#bad Huffman space"
|
||||
pub status "#bad metablock length"
|
||||
pub status "#bad padding"
|
||||
pub status "#bad reserved bit"
|
||||
pub status "#bad shared dictionary"
|
||||
pub status "#bad simple Huffman alphabet"
|
||||
pub status "#bad simple Huffman symbol"
|
||||
pub status "#bad transform"
|
||||
pub status "#bad window bits"
|
||||
pub status "#too many compound dictionaries"
|
||||
pub status "#too much dictionary data"
|
||||
pub status "#truncated input"
|
||||
pub status "#unsupported large window"
|
||||
pub status "#unsupported multiple custom static dictionaries"
|
||||
|
||||
pri status "#internal error: inconsistent bit reader"
|
||||
pri status "#internal error: inconsistent Huffman table"
|
||||
@@ -49,10 +53,33 @@ pri status "#internal error: inconsistent ring buffer"
|
||||
|
||||
pub const DECODER_DST_HISTORY_RETAIN_LENGTH_MAX_INCL_WORST_CASE : base.u64 = 0
|
||||
|
||||
// DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE is the largest (standard, not
|
||||
// large-window) brotli window size, (1 << 24) bytes, plus WORKBUF_SLACK. The
|
||||
// workbuf holds the history ring buffer.
|
||||
pub const DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE : base.u64 = 0x100_021E // 0x100_0000 + 542.
|
||||
// DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE is the largest workbuf_len value:
|
||||
// DICT_TOTAL_MAX bytes of attached dictionaries plus the largest
|
||||
// (large-window) brotli window size, (1 << 30) bytes, plus WORKBUF_SLACK.
|
||||
//
|
||||
// Without attached dictionaries and without the QUIRK_LARGE_WINDOW quirk,
|
||||
// workbuf_len is DECODER_WORKBUF_LEN_STANDARD, which is much smaller.
|
||||
pub const DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE : base.u64 = 0xC000_021D // 0x7FFF_FFFF + 0x4000_0000 + 542.
|
||||
|
||||
// DECODER_WORKBUF_LEN_STANDARD is the largest standard (not large-window)
|
||||
// brotli window size, (1 << 24) bytes, plus WORKBUF_SLACK. This is the
|
||||
// workbuf_len when no dictionaries are attached and the QUIRK_LARGE_WINDOW
|
||||
// quirk is not set.
|
||||
pub const DECODER_WORKBUF_LEN_STANDARD : base.u64 = 0x100_021E // 0x100_0000 + 542.
|
||||
|
||||
// DECODER_WORKBUF_LEN_LARGE_WINDOW is the largest large-window brotli window
|
||||
// size, (1 << 30) bytes, plus WORKBUF_SLACK.
|
||||
pub const DECODER_WORKBUF_LEN_LARGE_WINDOW : base.u64 = 0x4000_021E // 0x4000_0000 + 542.
|
||||
|
||||
// DICTIONARY_TYPE_RAW and DICTIONARY_TYPE_SERIALIZED are the values for
|
||||
// decoder.add_dictionary's dict_type argument. They mirror the C code's
|
||||
// BROTLI_SHARED_DICTIONARY_RAW and BROTLI_SHARED_DICTIONARY_SERIALIZED.
|
||||
pub const DICTIONARY_TYPE_RAW : base.u32 = 0
|
||||
pub const DICTIONARY_TYPE_SERIALIZED : base.u32 = 1
|
||||
|
||||
// DICT_TOTAL_MAX is the maximum total length of all attached dictionaries.
|
||||
// The C code holds the compound dictionaries' total size in an int.
|
||||
pri const DICT_TOTAL_MAX : base.u64 = 0x7FFF_FFFF
|
||||
|
||||
// WORKBUF_SLACK mirrors kRingBufferWriteAheadSlack in decode.c. The C code's
|
||||
// fast paths can write (and then discard) up to this many bytes past the end
|
||||
@@ -70,7 +97,9 @@ pri const WORKBUF_SLACK : base.u64 = 542
|
||||
// or the BROTLI_HUFFMAN_MAX_SIZE_ETC constants in huffman.h. There are at most
|
||||
// 256 trees in each of the literal, insert-and-copy and distance groups.
|
||||
//
|
||||
// TODO: large-window brotli needs a larger distance stride.
|
||||
// The distance stride is for the largest distance alphabet_size_limit, which
|
||||
// is 544 (for large-window brotli, BrotliCalculateDistanceCodeLimit with
|
||||
// NPOSTFIX = 3 and NDIRECT = 120), not 520 (for standard brotli).
|
||||
|
||||
pri const LIT_HUFFS_BASE : base.u32 = 0
|
||||
pri const LIT_HUFFS_STRIDE : base.u32 = 632 // 256 + 376.
|
||||
@@ -79,26 +108,26 @@ pri const CMD_HUFFS_BASE : base.u32 = 161792 // 0 + (256 * 632).
|
||||
pri const CMD_HUFFS_STRIDE : base.u32 = 1080 // 704 + 376.
|
||||
|
||||
pri const DIST_HUFFS_BASE : base.u32 = 438272 // 161792 + (256 * 1080).
|
||||
pri const DIST_HUFFS_STRIDE : base.u32 = 896 // 520 + 376.
|
||||
pri const DIST_HUFFS_STRIDE : base.u32 = 920 // 544 + 376.
|
||||
|
||||
// BLOCK_TYPE_HUFFS_STRIDE is BROTLI_HUFFMAN_MAX_SIZE_258.
|
||||
pri const BLOCK_TYPE_HUFFS_BASE : base.u32 = 667648 // 438272 + (256 * 896).
|
||||
pri const BLOCK_TYPE_HUFFS_BASE : base.u32 = 673792 // 438272 + (256 * 920).
|
||||
pri const BLOCK_TYPE_HUFFS_STRIDE : base.u32 = 632
|
||||
|
||||
// BLOCK_LEN_HUFFS_STRIDE is BROTLI_HUFFMAN_MAX_SIZE_26.
|
||||
pri const BLOCK_LEN_HUFFS_BASE : base.u32 = 669544 // 667648 + (3 * 632).
|
||||
pri const BLOCK_LEN_HUFFS_BASE : base.u32 = 675688 // 673792 + (3 * 632).
|
||||
pri const BLOCK_LEN_HUFFS_STRIDE : base.u32 = 396
|
||||
|
||||
// The context map table's size is BROTLI_HUFFMAN_MAX_SIZE_272.
|
||||
pri const CONTEXT_MAP_HUFFS_BASE : base.u32 = 670732 // 669544 + (3 * 396).
|
||||
pri const CONTEXT_MAP_HUFFS_BASE : base.u32 = 676876 // 675688 + (3 * 396).
|
||||
|
||||
// The code length code (and the code length code length code) tables are 5
|
||||
// (and 4) bits wide, replicated to fill 256 entries, the same as the 8-bit
|
||||
// root tables for every other code.
|
||||
pri const CODE_LENGTH_HUFFS_BASE : base.u32 = 671378 // 670732 + 646.
|
||||
pri const CODE_LENGTH_CODE_HUFFS_BASE : base.u32 = 671634 // 671378 + 256.
|
||||
pri const CODE_LENGTH_HUFFS_BASE : base.u32 = 677522 // 676876 + 646.
|
||||
pri const CODE_LENGTH_CODE_HUFFS_BASE : base.u32 = 677778 // 677522 + 256.
|
||||
|
||||
pri const HUFFS_LEN : base.u32 = 671890 // 671634 + 256.
|
||||
pri const HUFFS_LEN : base.u32 = 678034 // 677778 + 256.
|
||||
|
||||
// The two context maps live in one decoder.context_maps array. The literal
|
||||
// context map (up to 256 block types times 64 contexts) comes first, then the
|
||||
@@ -136,11 +165,12 @@ pub struct decoder? implements base.io_transformer(
|
||||
// rb_flushed is how many of the ring buffer's bytes (in the current
|
||||
// trip around the ring) have been written to args.dst.
|
||||
//
|
||||
// TODO: large-window brotli allows up to 30 window_bits.
|
||||
window_bits : base.u32[..= 24],
|
||||
window_size : base.u32[..= 0x100_0000],
|
||||
max_backward_distance : base.u32[..= 0x100_0000],
|
||||
rb_size : base.u32[..= 0x100_0000],
|
||||
// window_bits is at most 24 for standard brotli and at most 30 for
|
||||
// large-window brotli.
|
||||
window_bits : base.u32[..= 30],
|
||||
window_size : base.u32[..= 0x4000_0000],
|
||||
max_backward_distance : base.u32[..= 0x4000_0000],
|
||||
rb_size : base.u32[..= 0x4000_0000],
|
||||
pos : base.u32,
|
||||
rb_flushed : base.u32,
|
||||
|
||||
@@ -168,9 +198,12 @@ pub struct decoder? implements base.io_transformer(
|
||||
|
||||
distance_postfix_bits : base.u32[..= 3],
|
||||
num_direct_distance_codes : base.u32[..= 120],
|
||||
distance_alphabet_size : base.u32[..= 520],
|
||||
num_literal_htrees : base.u32[..= 256],
|
||||
num_dist_htrees : base.u32[..= 256],
|
||||
// distance_alphabet_size is the C code's
|
||||
// distance_hgroup.alphabet_size_limit: at most 520 for standard
|
||||
// brotli and at most 544 for large-window brotli.
|
||||
distance_alphabet_size : base.u32[..= 544],
|
||||
num_literal_htrees : base.u32[..= 256],
|
||||
num_dist_htrees : base.u32[..= 256],
|
||||
|
||||
// The C code's context_map_slice and dist_context_map_slice, as
|
||||
// offsets into decoder.context_maps, plus the context_lookup mode.
|
||||
@@ -197,9 +230,71 @@ pub struct decoder? implements base.io_transformer(
|
||||
insert_length : base.u32,
|
||||
copy_length : base.u32,
|
||||
|
||||
// quirk_large_window is the QUIRK_LARGE_WINDOW quirk (the C code's
|
||||
// BROTLI_DECODER_PARAM_LARGE_WINDOW parameter). large_window is
|
||||
// whether the stream is a large-window stream (the C code's
|
||||
// s->large_window after DecodeWindowBits), which needs both the quirk
|
||||
// and the large-window WBITS escape.
|
||||
quirk_large_window : base.bool,
|
||||
large_window : base.bool,
|
||||
|
||||
// started is whether the C code's s->state would no longer be
|
||||
// BROTLI_STATE_UNINITED. Quirks and dictionaries can only be set or
|
||||
// added before then.
|
||||
started : base.bool,
|
||||
|
||||
// dict_total is the total length of all attached dictionaries, which
|
||||
// live at the start of the workbuf: args.workbuf[.. dict_total]. The
|
||||
// ring buffer follows: args.workbuf[dict_total ..].
|
||||
dict_total : base.u64[..= 0x7FFF_FFFF],
|
||||
|
||||
// The compound dictionary (the C code's BrotliDecoderCompoundDictionary
|
||||
// in state.h). It has num_chunks chunks. The i'th chunk starts at
|
||||
// args.workbuf[chunk_starts[i]] and spans compound dictionary
|
||||
// addresses [chunk_offsets[i] .. chunk_offsets[i + 1]].
|
||||
// chunk_offsets[num_chunks] is the C code's total_size.
|
||||
num_chunks : base.u32[..= 15],
|
||||
chunk_starts : array[15] base.u32,
|
||||
chunk_offsets : array[16] base.u32,
|
||||
|
||||
// The shared dictionary's static dictionary part (the C code's
|
||||
// BrotliSharedDictionary in shared_dictionary_internal.h). The C
|
||||
// code's words[i] and transforms[i] are given by sd_words[i] and
|
||||
// sd_transforms[i]: 0 means the built-in (RFC 7932) word list or
|
||||
// transform list and (1 + j) means the j'th custom list (see the
|
||||
// wl_etc and tl_etc fields). sd_context_based, sd_context_map and
|
||||
// sd_num_dictionaries are the C code's context_based, context_map and
|
||||
// num_dictionaries (except that the initial zero value means one).
|
||||
// sd_num_word_lists and sd_num_transform_lists are the C code's
|
||||
// num_word_lists and num_transform_lists.
|
||||
sd_context_based : base.bool,
|
||||
sd_num_dictionaries : base.u32[..= 64],
|
||||
sd_num_word_lists : base.u32[..= 64],
|
||||
sd_num_transform_lists : base.u32[..= 64],
|
||||
sd_words : array[64] base.u8[..= 64],
|
||||
sd_transforms : array[64] base.u8[..= 64],
|
||||
sd_context_map : array[64] base.u8[..= 63],
|
||||
|
||||
// The prefix (LZ77 dictionary chunk) of a serialized shared
|
||||
// dictionary, as found by parse_shared_dictionary: its position
|
||||
// within that dictionary's bytes and its length.
|
||||
sd_prefix_pos : base.u64,
|
||||
sd_prefix_size : base.u64,
|
||||
|
||||
// More staging fields for parse_shared_dictionary. See also the
|
||||
// stage_etc arrays.
|
||||
stage_num_word_lists : base.u32[..= 64],
|
||||
stage_num_transform_lists : base.u32[..= 64],
|
||||
stage_num_dictionaries : base.u32[..= 64],
|
||||
stage_context_based : base.bool,
|
||||
|
||||
// parse_pos is where parse_shared_dictionary's helper functions
|
||||
// store the new parsing position.
|
||||
parse_pos : base.u64,
|
||||
|
||||
util : base.utility,
|
||||
) + (
|
||||
huffs : array[671890] base.u16, // HUFFS_LEN.
|
||||
huffs : array[678034] base.u16, // HUFFS_LEN.
|
||||
|
||||
context_maps : array[17408] base.u8, // 16384 + 1024.
|
||||
context_modes : array[256] base.u8,
|
||||
@@ -221,16 +316,52 @@ pub struct decoder? implements base.io_transformer(
|
||||
dist_offset : array[1128] base.u32,
|
||||
|
||||
// transformed_word holds a transformed static dictionary word: a
|
||||
// prefix (up to 8 bytes), a word (up to 24 bytes) and a suffix (up to
|
||||
// 8 bytes).
|
||||
transformed_word : array[64] base.u8,
|
||||
// prefix (up to 255 bytes), a word (up to 31 bytes) and a suffix (up
|
||||
// to 255 bytes). Built-in transforms' prefixes and suffixes are at
|
||||
// most 8 bytes and built-in words are at most 24 bytes.
|
||||
transformed_word : array[1024] base.u8,
|
||||
|
||||
// Custom word lists (the C code's BrotliDictionary words_instances).
|
||||
// wl_data is the offset (within the workbuf) of the list's words.
|
||||
wl_size_bits : array[64] array[32] base.u8,
|
||||
wl_offsets : array[64] array[32] base.u32,
|
||||
wl_data : array[64] base.u32,
|
||||
|
||||
// Custom transform lists (the C code's BrotliTransforms
|
||||
// transforms_instances). tl_prefix_suffix, tl_transforms and
|
||||
// tl_params are offsets (within the workbuf) of the C code's
|
||||
// prefix_suffix, transforms and params arrays. tl_map is the C code's
|
||||
// prefix_suffix_map.
|
||||
tl_prefix_suffix : array[64] base.u32,
|
||||
tl_map : array[64] array[256] base.u16,
|
||||
tl_num_transforms : array[64] base.u32,
|
||||
tl_transforms : array[64] base.u32,
|
||||
tl_params : array[64] base.u32,
|
||||
|
||||
// Staging areas for parse_shared_dictionary. They are copied to the
|
||||
// sd_etc fields only if the whole serialized dictionary is valid.
|
||||
stage_words : array[64] base.u8,
|
||||
stage_transforms : array[64] base.u8,
|
||||
stage_context_map : array[64] base.u8,
|
||||
)
|
||||
|
||||
pub func decoder.get_quirk(key: base.u32) base.u64 {
|
||||
if (args.key == QUIRK_LARGE_WINDOW) and this.quirk_large_window {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// set_quirk mirrors BrotliDecoderSetParameter in decode.c, which fails once
|
||||
// decoding has started.
|
||||
pub func decoder.set_quirk!(key: base.u32, value: base.u64) base.status {
|
||||
if args.key == QUIRK_LARGE_WINDOW {
|
||||
if this.started {
|
||||
return base."#bad call sequence"
|
||||
}
|
||||
this.quirk_large_window = args.value > 0
|
||||
return ok
|
||||
}
|
||||
return base."#unsupported option"
|
||||
}
|
||||
|
||||
@@ -238,10 +369,19 @@ pub func decoder.dst_history_retain_length() base.optional_u63 {
|
||||
return this.util.make_optional_u63(has_value: true, value: 0)
|
||||
}
|
||||
|
||||
// workbuf_len is the total length of the dictionaries added so far (see
|
||||
// add_dictionary) plus the largest window size (for the QUIRK_LARGE_WINDOW
|
||||
// quirk's current value) plus WORKBUF_SLACK.
|
||||
pub func decoder.workbuf_len() base.range_ii_u64 {
|
||||
var n : base.u64[..= 0x4000_021E]
|
||||
|
||||
n = DECODER_WORKBUF_LEN_STANDARD
|
||||
if this.quirk_large_window {
|
||||
n = DECODER_WORKBUF_LEN_LARGE_WINDOW
|
||||
}
|
||||
return this.util.make_range_ii_u64(
|
||||
min_incl: DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE,
|
||||
max_incl: DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE)
|
||||
min_incl: n + this.dict_total,
|
||||
max_incl: n + this.dict_total)
|
||||
}
|
||||
|
||||
// transform_io wraps do_transform_io, which decodes into the ring buffer (the
|
||||
@@ -252,17 +392,31 @@ pub func decoder.workbuf_len() base.range_ii_u64 {
|
||||
// write" when it needs every decoded byte flushed before it can wrap around
|
||||
// the ring buffer. If this function can flush all of them, it resumes
|
||||
// do_transform_io immediately instead of suspending.
|
||||
//
|
||||
// The first this.dict_total bytes of args.workbuf hold the attached
|
||||
// dictionaries (see add_dictionary). The ring buffer follows them.
|
||||
pub func decoder.transform_io?(dst: base.io_writer, src: base.io_reader, workbuf: slice base.u8) {
|
||||
var status : base.status
|
||||
|
||||
while true {
|
||||
status =? this.do_transform_io?(src: args.src, workbuf: args.workbuf)
|
||||
this.flush!(dst: args.dst, workbuf: args.workbuf)
|
||||
if this.dict_total > args.workbuf.length() {
|
||||
return base."#bad workbuf length"
|
||||
}
|
||||
status =? this.do_transform_io?(src: args.src,
|
||||
workbuf: args.workbuf[this.dict_total ..],
|
||||
dictbuf: args.workbuf[.. this.dict_total])
|
||||
if this.dict_total > args.workbuf.length() {
|
||||
return base."#bad workbuf length"
|
||||
}
|
||||
this.flush!(dst: args.dst, workbuf: args.workbuf[this.dict_total ..])
|
||||
|
||||
if status.is_ok() {
|
||||
while not this.is_fully_flushed() {
|
||||
yield? base."$short write"
|
||||
this.flush!(dst: args.dst, workbuf: args.workbuf)
|
||||
if this.dict_total > args.workbuf.length() {
|
||||
return base."#bad workbuf length"
|
||||
}
|
||||
this.flush!(dst: args.dst, workbuf: args.workbuf[this.dict_total ..])
|
||||
}
|
||||
return ok
|
||||
|
||||
@@ -310,7 +464,7 @@ pri func decoder.flush!(dst: base.io_writer, workbuf: slice base.u8) {
|
||||
|
||||
// Mirrors the BROTLI_STATE_ETC state machine in BrotliDecoderDecompressStream
|
||||
// in decode.c.
|
||||
pri func decoder.do_transform_io?(src: base.io_reader, workbuf: slice base.u8) {
|
||||
pri func decoder.do_transform_io?(src: base.io_reader, workbuf: slice base.u8, dictbuf: roslice base.u8) {
|
||||
var w : base.u64
|
||||
var aligned : base.bool
|
||||
|
||||
@@ -358,7 +512,7 @@ pri func decoder.do_transform_io?(src: base.io_reader, workbuf: slice base.u8) {
|
||||
// BROTLI_STATE_UNCOMPRESSED.
|
||||
this.copy_uncompressed?(src: args.src, workbuf: args.workbuf)
|
||||
} else {
|
||||
this.decode_compressed_metablock?(src: args.src, workbuf: args.workbuf)
|
||||
this.decode_compressed_metablock?(src: args.src, workbuf: args.workbuf, dictbuf: args.dictbuf)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -386,6 +540,7 @@ pri func decoder.initialize_codes!() {
|
||||
|
||||
this.bits = 0
|
||||
this.n_bits = 0
|
||||
this.large_window = false
|
||||
this.pos = 0
|
||||
this.rb_size = 0
|
||||
this.rb_flushed = 0
|
||||
@@ -452,6 +607,23 @@ pri func decoder.read_bits?(src: base.io_reader, n: base.u32[..= 24]) {
|
||||
this.n_bits = n_bits - n
|
||||
}
|
||||
|
||||
// read_bits32 is like read_bits but args.n can be up to 32. Mirrors
|
||||
// BrotliSafeReadBits32 in bit_reader.h.
|
||||
pri func decoder.read_bits32?(src: base.io_reader, n: base.u32[..= 32]) {
|
||||
var n : base.u32[..= 32]
|
||||
var lo : base.u32
|
||||
|
||||
n = args.n
|
||||
if n <= 24 {
|
||||
this.read_bits?(src: args.src, n: n)
|
||||
return ok
|
||||
}
|
||||
this.read_bits?(src: args.src, n: 16)
|
||||
lo = this.bits_value & 0xFFFF
|
||||
this.read_bits?(src: args.src, n: n - 16)
|
||||
this.bits_value = lo | (this.bits_value ~mod<< 16)
|
||||
}
|
||||
|
||||
// read_symbol reads one Huffman-encoded symbol, decoded by the table at
|
||||
// this.huffs[args.table ..], and stores it in this.symbol. It reads input
|
||||
// bytes only when needed. Mirrors SafeReadSymbol and SafeDecodeSymbol in
|
||||
@@ -521,9 +693,11 @@ pri func decoder.jump_to_byte_boundary!() base.bool {
|
||||
return pad == 0
|
||||
}
|
||||
|
||||
// Mirrors DecodeWindowBits in decode.c.
|
||||
// Mirrors DecodeWindowBits in decode.c, and the BROTLI_STATE_UNINITED and
|
||||
// BROTLI_STATE_LARGE_WINDOW_BITS states of BrotliDecoderDecompressStream.
|
||||
pri func decoder.decode_window_bits?(src: base.io_reader) {
|
||||
var n : base.u32[..= 7]
|
||||
var w : base.u32[..= 63]
|
||||
|
||||
this.read_bits?(src: args.src, n: 1)
|
||||
if this.bits_value == 0 {
|
||||
@@ -537,12 +711,30 @@ pri func decoder.decode_window_bits?(src: base.io_reader) {
|
||||
this.read_bits?(src: args.src, n: 3)
|
||||
n = this.bits_value & 7
|
||||
if n == 1 {
|
||||
// The C code would read a "Large Window Brotli" header, if
|
||||
// BROTLI_DECODER_PARAM_LARGE_WINDOW was set, or otherwise fail
|
||||
// with BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS.
|
||||
//
|
||||
// TODO: support large-window brotli.
|
||||
return "#unsupported large window"
|
||||
// The "Large Window Brotli" escape. Without the
|
||||
// QUIRK_LARGE_WINDOW quirk (the C code's
|
||||
// BROTLI_DECODER_PARAM_LARGE_WINDOW), the C code fails with
|
||||
// BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS.
|
||||
if not this.quirk_large_window {
|
||||
return "#unsupported large window"
|
||||
}
|
||||
this.read_bits?(src: args.src, n: 1)
|
||||
if this.bits_value == 1 {
|
||||
return "#bad window bits"
|
||||
}
|
||||
// The C code is now in the BROTLI_STATE_LARGE_WINDOW_BITS
|
||||
// state.
|
||||
this.started = true
|
||||
this.large_window = true
|
||||
this.read_bits?(src: args.src, n: 6)
|
||||
w = this.bits_value & 63
|
||||
if (w < 10) or (30 < w) {
|
||||
return "#bad window bits"
|
||||
}
|
||||
this.window_bits = w
|
||||
this.window_size = (1 as base.u32) << w
|
||||
this.max_backward_distance = this.window_size - 16
|
||||
return ok
|
||||
} else if n <> 0 {
|
||||
this.window_bits = 8 + n
|
||||
} else {
|
||||
@@ -553,6 +745,8 @@ pri func decoder.decode_window_bits?(src: base.io_reader) {
|
||||
if this.window_bits < 10 {
|
||||
return "#bad window bits"
|
||||
}
|
||||
// The C code is now in the BROTLI_STATE_INITIALIZE state.
|
||||
this.started = true
|
||||
this.window_size = (1 as base.u32) << this.window_bits
|
||||
// Maximum distance, see section 9.1. of the spec. BROTLI_WINDOW_GAP is 16.
|
||||
this.max_backward_distance = this.window_size - 16
|
||||
|
||||
@@ -0,0 +1,955 @@
|
||||
// Copyright 2026 The Wuffs Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
|
||||
// https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
|
||||
// <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
|
||||
// option. This file may not be copied, modified, or distributed
|
||||
// except according to those terms.
|
||||
//
|
||||
// SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
|
||||
// --------
|
||||
|
||||
// add_dictionary attaches a dictionary, like BrotliDecoderAttachDictionary in
|
||||
// decode.c. It must be called before decoding starts (before the first
|
||||
// transform_io call that consumes any input).
|
||||
//
|
||||
// args.dict_type is DICTIONARY_TYPE_RAW (a raw LZ77 prefix dictionary) or
|
||||
// DICTIONARY_TYPE_SERIALIZED (a serialized shared dictionary).
|
||||
//
|
||||
// Wuffs structs cannot hold pointers, so the dictionary's bytes are copied
|
||||
// into the workbuf: they are appended to the dictionaries added so far, which
|
||||
// live at the start of the workbuf. args.workbuf must therefore be at least
|
||||
// as long as the total length of all dictionaries added so far, including
|
||||
// this one, and every later add_dictionary and transform_io call must be
|
||||
// passed the same workbuf (or at least one whose prefix holds the same
|
||||
// bytes). After the last add_dictionary call, workbuf_len gives the workbuf
|
||||
// length that transform_io needs.
|
||||
//
|
||||
// On failure, the decoder's state is unchanged. The C code can leave its
|
||||
// state partially updated when BrotliDecoderAttachDictionary fails.
|
||||
pub func decoder.add_dictionary!(dict_type: base.u32, dict: roslice base.u8, workbuf: slice base.u8) base.status {
|
||||
var status : base.status
|
||||
var n : base.u64
|
||||
var lo : base.u64
|
||||
var hi : base.u64
|
||||
var i : base.u32
|
||||
|
||||
if this.started {
|
||||
return base."#bad call sequence"
|
||||
}
|
||||
n = args.dict.length()
|
||||
|
||||
if args.dict_type == DICTIONARY_TYPE_RAW {
|
||||
// Mirrors the BROTLI_SHARED_DICTIONARY_RAW case of
|
||||
// BrotliSharedDictionaryAttach in shared_dictionary.c and
|
||||
// AttachCompoundDictionary in decode.c.
|
||||
if this.num_chunks >= 15 {
|
||||
return "#too many compound dictionaries"
|
||||
}
|
||||
} else if args.dict_type == DICTIONARY_TYPE_SERIALIZED {
|
||||
// Mirrors DecodeSharedDictionary in shared_dictionary.c.
|
||||
status = this.parse_shared_dictionary!(dict: args.dict, base_offset: this.dict_total)
|
||||
if not status.is_ok() {
|
||||
return status
|
||||
}
|
||||
} else {
|
||||
return base."#bad argument"
|
||||
}
|
||||
|
||||
// Copy the dictionary bytes into the workbuf.
|
||||
if n > DICT_TOTAL_MAX {
|
||||
return "#too much dictionary data"
|
||||
}
|
||||
lo = this.dict_total
|
||||
hi = lo + n
|
||||
if hi > DICT_TOTAL_MAX {
|
||||
return "#too much dictionary data"
|
||||
} else if hi > args.workbuf.length() {
|
||||
return base."#bad workbuf length"
|
||||
}
|
||||
if lo > hi {
|
||||
return "#too much dictionary data"
|
||||
}
|
||||
args.workbuf[lo .. hi].copy_from_slice!(s: args.dict)
|
||||
|
||||
// Commit.
|
||||
if args.dict_type == DICTIONARY_TYPE_RAW {
|
||||
this.add_chunk!(start: lo, size: n)
|
||||
|
||||
} else {
|
||||
if this.sd_prefix_size > 0 {
|
||||
this.add_chunk!(start: lo ~sat+ this.sd_prefix_pos, size: this.sd_prefix_size)
|
||||
}
|
||||
// Mirrors the end of ParseDictionary in shared_dictionary.c. If the
|
||||
// serialized dictionary has no custom word lists or transform lists,
|
||||
// the static dictionary is reset to the built-in one, even if an
|
||||
// earlier serialized dictionary had custom lists.
|
||||
this.sd_num_word_lists = this.stage_num_word_lists
|
||||
this.sd_num_transform_lists = this.stage_num_transform_lists
|
||||
if (this.stage_num_word_lists > 0) or (this.stage_num_transform_lists > 0) {
|
||||
this.sd_context_based = this.stage_context_based
|
||||
this.sd_num_dictionaries = this.stage_num_dictionaries
|
||||
i = 0
|
||||
while i < 64 {
|
||||
this.sd_words[i] = this.stage_words[i].min(no_more_than: 64)
|
||||
this.sd_transforms[i] = this.stage_transforms[i].min(no_more_than: 64)
|
||||
this.sd_context_map[i] = this.stage_context_map[i] & 0x3F
|
||||
i += 1
|
||||
}
|
||||
} else {
|
||||
this.sd_context_based = false
|
||||
this.sd_num_dictionaries = 1
|
||||
this.sd_words[0] = 0
|
||||
this.sd_transforms[0] = 0
|
||||
}
|
||||
}
|
||||
|
||||
if hi <= DICT_TOTAL_MAX {
|
||||
this.dict_total = hi
|
||||
}
|
||||
return ok
|
||||
}
|
||||
|
||||
// add_chunk adds a chunk to the compound dictionary. Mirrors
|
||||
// AttachCompoundDictionary in decode.c.
|
||||
pri func decoder.add_chunk!(start: base.u64, size: base.u64) {
|
||||
var k : base.u32[..= 15]
|
||||
|
||||
k = this.num_chunks
|
||||
if k >= 15 {
|
||||
return nothing
|
||||
}
|
||||
this.chunk_starts[k] = (args.start & 0xFFFF_FFFF) as base.u32
|
||||
this.chunk_offsets[k + 1] = this.chunk_offsets[k] ~mod+ ((args.size & 0xFFFF_FFFF) as base.u32)
|
||||
this.num_chunks = k + 1
|
||||
}
|
||||
|
||||
// parse_shared_dictionary parses and validates a serialized shared
|
||||
// dictionary. Mirrors DecodeSharedDictionary, DryParseDictionary and
|
||||
// ParseDictionary in shared_dictionary.c.
|
||||
//
|
||||
// It does not change the decoder's sd_etc fields, only the stage_etc fields
|
||||
// (and the wl_etc and tl_etc fields, but only when there are no custom word
|
||||
// lists or transform lists in use). The caller commits them.
|
||||
//
|
||||
// args.base_offset is where the serialized dictionary's bytes will be copied
|
||||
// to, within the workbuf.
|
||||
pri func decoder.parse_shared_dictionary!(dict: roslice base.u8, base_offset: base.u64) base.status {
|
||||
var pos : base.u64
|
||||
var chunk_size : base.u64
|
||||
var prefix_pos : base.u64
|
||||
var num_prefix : base.u32[..= 1]
|
||||
var nwl : base.u32[..= 0xFF]
|
||||
var ntl : base.u32[..= 0xFF]
|
||||
var ndict : base.u32[..= 0xFF]
|
||||
var is_custom : base.bool
|
||||
var has_custom : base.bool
|
||||
var i : base.u32
|
||||
var v : base.u32[..= 0xFF]
|
||||
var v2 : base.u32[..= 0xFF]
|
||||
var status : base.status
|
||||
|
||||
// Check the magic header bytes.
|
||||
if args.dict.length() < 2 {
|
||||
return "#bad shared dictionary"
|
||||
} else if (args.dict[0] <> 0x91) or (args.dict[1] <> 0x00) {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
|
||||
// DryParseDictionary.
|
||||
pos = 2
|
||||
// LZ77_DICTIONARY_LENGTH. Mirrors ReadVarint32.
|
||||
chunk_size = this.read_varint32!(dict: args.dict, pos: pos)
|
||||
if chunk_size >= 0x1_0000_0000 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
pos = this.parse_pos
|
||||
prefix_pos = pos
|
||||
num_prefix = 0
|
||||
if chunk_size <> 0 {
|
||||
// "This limitation is not specified but the 32-bit Brotli decoder for
|
||||
// now".
|
||||
if chunk_size > 1073_741823 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
num_prefix = 1
|
||||
if (pos ~sat+ chunk_size) > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
pos ~sat+= chunk_size
|
||||
}
|
||||
// NUM_WORD_LISTS and NUM_TRANSFORM_LISTS.
|
||||
if (pos ~sat+ 2) > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
is_custom = false
|
||||
if pos < args.dict.length() {
|
||||
if args.dict[pos] > 0 {
|
||||
is_custom = true
|
||||
}
|
||||
}
|
||||
if (pos ~sat+ 1) < args.dict.length() {
|
||||
if args.dict[pos ~sat+ 1] > 0 {
|
||||
is_custom = true
|
||||
}
|
||||
}
|
||||
|
||||
if (num_prefix + this.num_chunks) > 15 {
|
||||
return "#too many compound dictionaries"
|
||||
}
|
||||
// "Cannot combine different static dictionaries, only prefix
|
||||
// dictionaries".
|
||||
has_custom = (this.sd_num_word_lists > 0) or (this.sd_num_transform_lists > 0)
|
||||
if has_custom and is_custom {
|
||||
return "#unsupported multiple custom static dictionaries"
|
||||
}
|
||||
|
||||
// ParseDictionary. The LZ77_DICTIONARY_LENGTH bytes were checked above.
|
||||
|
||||
// NUM_WORD_LISTS.
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
nwl = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if nwl > 64 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
i = 0
|
||||
while i < nwl,
|
||||
inv nwl <= 64,
|
||||
{
|
||||
assert i < 64 via "a < b: a < c; c <= b"(c: nwl)
|
||||
status = this.parse_word_list!(dict: args.dict, pos: pos, base_offset: args.base_offset, list: i)
|
||||
if not status.is_ok() {
|
||||
return status
|
||||
}
|
||||
pos = this.parse_pos
|
||||
i += 1
|
||||
}
|
||||
|
||||
// NUM_TRANSFORM_LISTS.
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
ntl = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if ntl > 64 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
i = 0
|
||||
while i < ntl,
|
||||
inv ntl <= 64,
|
||||
{
|
||||
assert i < 64 via "a < b: a < c; c <= b"(c: ntl)
|
||||
status = this.parse_transforms_list!(dict: args.dict, pos: pos, base_offset: args.base_offset, list: i)
|
||||
if not status.is_ok() {
|
||||
return status
|
||||
}
|
||||
pos = this.parse_pos
|
||||
i += 1
|
||||
}
|
||||
|
||||
this.stage_num_word_lists = nwl.min(no_more_than: 64)
|
||||
this.stage_num_transform_lists = ntl.min(no_more_than: 64)
|
||||
this.stage_num_dictionaries = 1
|
||||
this.stage_context_based = false
|
||||
i = 0
|
||||
while i < 64 {
|
||||
this.stage_words[i] = 0
|
||||
this.stage_transforms[i] = 0
|
||||
this.stage_context_map[i] = 0
|
||||
i += 1
|
||||
}
|
||||
|
||||
if (nwl <> 0) or (ntl <> 0) {
|
||||
// NUM_DICTIONARIES.
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
ndict = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if (ndict == 0) or (ndict > 64) {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.stage_num_dictionaries = ndict.min(no_more_than: 64)
|
||||
i = 0
|
||||
while i < ndict,
|
||||
inv ndict <= 64,
|
||||
{
|
||||
assert i < 64 via "a < b: a < c; c <= b"(c: ndict)
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
v = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if v > nwl {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
v2 = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if v2 > ntl {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
// Index nwl (or ntl) means the built-in list, which this.sd_words
|
||||
// (or this.sd_transforms) represents as 0.
|
||||
if v == nwl {
|
||||
this.stage_words[i] = 0
|
||||
} else {
|
||||
this.stage_words[i] = ((v & 0x3F) + 1) as base.u8
|
||||
}
|
||||
if v2 == ntl {
|
||||
this.stage_transforms[i] = 0
|
||||
} else {
|
||||
this.stage_transforms[i] = ((v2 & 0x3F) + 1) as base.u8
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
|
||||
// CONTEXT_ENABLED. Mirrors ReadBool.
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
v = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if v > 1 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.stage_context_based = v == 1
|
||||
|
||||
// CONTEXT_MAP.
|
||||
if this.stage_context_based {
|
||||
i = 0
|
||||
while i < 64 {
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
v = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if v >= ndict {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.stage_context_map[i] = (v & 0x3F) as base.u8
|
||||
i += 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
this.sd_prefix_pos = prefix_pos
|
||||
this.sd_prefix_size = chunk_size
|
||||
return ok
|
||||
}
|
||||
|
||||
// read_varint32 reads a varint at args.dict[args.pos ..]. It returns a value
|
||||
// of 0x1_0000_0000 or more on failure. On success, it returns the value and
|
||||
// stores the new position in this.parse_pos. Mirrors ReadVarint32 in
|
||||
// shared_dictionary.c.
|
||||
pri func decoder.read_varint32!(dict: roslice base.u8, pos: base.u64) base.u64 {
|
||||
var pos : base.u64
|
||||
var num : base.u32[..= 4]
|
||||
var b : base.u64
|
||||
var result : base.u64
|
||||
|
||||
pos = args.pos
|
||||
result = 0
|
||||
num = 0
|
||||
while true {
|
||||
if pos >= args.dict.length() {
|
||||
return 0x1_0000_0000
|
||||
}
|
||||
b = args.dict[pos] as base.u64
|
||||
pos ~sat+= 1
|
||||
if (num == 4) and (b > 15) {
|
||||
return 0x1_0000_0000
|
||||
}
|
||||
result |= (b & 127) << (7 * num)
|
||||
if b < 128 {
|
||||
break
|
||||
}
|
||||
if num >= 4 {
|
||||
return 0x1_0000_0000
|
||||
}
|
||||
num += 1
|
||||
}
|
||||
this.parse_pos = pos
|
||||
return result & 0xFFFF_FFFF
|
||||
}
|
||||
|
||||
// parse_word_list parses the args.list'th custom word list, at
|
||||
// args.dict[args.pos ..]. On success, it stores the new position in
|
||||
// this.parse_pos. Mirrors ParseWordList and BrotliSizeBitsToOffsets in
|
||||
// shared_dictionary.c.
|
||||
pri func decoder.parse_word_list!(dict: roslice base.u8, pos: base.u64, base_offset: base.u64, list: base.u32[..= 63]) base.status {
|
||||
var pos : base.u64
|
||||
var len : base.u32
|
||||
var v : base.u8
|
||||
var offset : base.u32
|
||||
var wpos : base.u64
|
||||
|
||||
pos = args.pos
|
||||
if (pos ~sat+ 28) > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
len = 0
|
||||
while len < 32 {
|
||||
v = 0
|
||||
if len >= 4 {
|
||||
if (pos ~sat+ ((len - 4) as base.u64)) < args.dict.length() {
|
||||
v = args.dict[pos ~sat+ ((len - 4) as base.u64)]
|
||||
}
|
||||
// BROTLI_MAX_SIZE_BITS.
|
||||
if v > 15 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
}
|
||||
this.wl_size_bits[args.list][len] = v
|
||||
len += 1
|
||||
}
|
||||
pos ~sat+= 28
|
||||
|
||||
offset = 0
|
||||
len = 0
|
||||
while len < 32 {
|
||||
this.wl_offsets[args.list][len] = offset
|
||||
v = this.wl_size_bits[args.list][len]
|
||||
if v <> 0 {
|
||||
offset ~mod+= len << (v & 15)
|
||||
}
|
||||
len += 1
|
||||
}
|
||||
|
||||
wpos = args.base_offset ~sat+ pos
|
||||
this.wl_data[args.list] = (wpos & 0xFFFF_FFFF) as base.u32
|
||||
pos ~sat+= offset as base.u64
|
||||
if pos > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.parse_pos = pos
|
||||
return ok
|
||||
}
|
||||
|
||||
// parse_transforms_list parses the args.list'th custom transform list, at
|
||||
// args.dict[args.pos ..]. On success, it stores the new position in
|
||||
// this.parse_pos. Mirrors ParseTransformsList and ParsePrefixSuffixTable
|
||||
// in shared_dictionary.c.
|
||||
pri func decoder.parse_transforms_list!(dict: roslice base.u8, pos: base.u64, base_offset: base.u64, list: base.u32[..= 63]) base.status {
|
||||
var pos : base.u64
|
||||
var data_length : base.u64
|
||||
var ps_start : base.u64
|
||||
var offset : base.u64
|
||||
var stringlet_len : base.u64
|
||||
var count : base.u32[..= 256]
|
||||
var nt : base.u32[..= 0xFF]
|
||||
var t : base.u32
|
||||
var tpos : base.u64
|
||||
var j : base.u64
|
||||
var prefix_id : base.u32[..= 0xFF]
|
||||
var t_type : base.u32[..= 0xFF]
|
||||
var suffix_id : base.u32[..= 0xFF]
|
||||
var has_params : base.bool
|
||||
var p0 : base.u32
|
||||
var p1 : base.u32
|
||||
|
||||
pos = args.pos
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
|
||||
// ParsePrefixSuffixTable. PREFIX_SUFFIX_LENGTH.
|
||||
if (pos ~sat+ 2) > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
data_length = (this.byte_at(s: args.dict, i: pos) as base.u64) |
|
||||
((this.byte_at(s: args.dict, i: pos ~sat+ 1) as base.u64) << 8)
|
||||
pos ~sat+= 2
|
||||
// It must at least have space for the NUL terminator.
|
||||
if data_length < 1 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
ps_start = pos
|
||||
if (pos ~sat+ data_length) >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
offset = 0
|
||||
count = 0
|
||||
while true {
|
||||
// STRING_LENGTH.
|
||||
j = ps_start ~sat+ offset
|
||||
stringlet_len = 0
|
||||
if j < args.dict.length() {
|
||||
stringlet_len = args.dict[j] as base.u64
|
||||
}
|
||||
if count >= 256 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.tl_map[args.list][count] = (offset & 0xFFFF) as base.u16
|
||||
count += 1
|
||||
offset ~sat+= 1
|
||||
if stringlet_len == 0 {
|
||||
if offset == data_length {
|
||||
break
|
||||
}
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
if count > 255 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
offset ~sat+= stringlet_len
|
||||
if offset >= data_length {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
}
|
||||
pos ~sat+= data_length
|
||||
this.tl_prefix_suffix[args.list] = ((args.base_offset ~sat+ ps_start) & 0xFFFF_FFFF) as base.u32
|
||||
|
||||
// NUM_TRANSFORMS. The "(pos + data_length) >= length" check above means
|
||||
// that this is in bounds.
|
||||
nt = 0
|
||||
if pos < args.dict.length() {
|
||||
nt = args.dict[pos] as base.u32
|
||||
}
|
||||
pos ~sat+= 1
|
||||
this.tl_num_transforms[args.list] = nt
|
||||
tpos = pos
|
||||
this.tl_transforms[args.list] = ((args.base_offset ~sat+ tpos) & 0xFFFF_FFFF) as base.u32
|
||||
pos ~sat+= (nt as base.u64) * 3
|
||||
if pos > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
|
||||
// Check for errors and read extra parameters.
|
||||
has_params = false
|
||||
t = 0
|
||||
while t < nt {
|
||||
j = tpos ~sat+ ((t as base.u64) * 3)
|
||||
prefix_id = this.byte_at(s: args.dict, i: j)
|
||||
t_type = this.byte_at(s: args.dict, i: j ~sat+ 1)
|
||||
suffix_id = this.byte_at(s: args.dict, i: j ~sat+ 2)
|
||||
if prefix_id >= count {
|
||||
return "#bad shared dictionary"
|
||||
} else if t_type >= 23 {
|
||||
// BROTLI_NUM_TRANSFORM_TYPES.
|
||||
return "#bad shared dictionary"
|
||||
} else if suffix_id >= count {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
// BROTLI_TRANSFORM_SHIFT_FIRST or BROTLI_TRANSFORM_SHIFT_ALL.
|
||||
if (t_type == 21) or (t_type == 22) {
|
||||
has_params = true
|
||||
}
|
||||
t ~mod+= 1
|
||||
}
|
||||
|
||||
this.tl_params[args.list] = 0
|
||||
if has_params {
|
||||
this.tl_params[args.list] = ((args.base_offset ~sat+ pos) & 0xFFFF_FFFF) as base.u32
|
||||
j = pos
|
||||
pos ~sat+= (nt as base.u64) * 2
|
||||
if pos > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
t = 0
|
||||
while t < nt {
|
||||
t_type = this.byte_at(s: args.dict, i: tpos ~sat+ (((t as base.u64) * 3) + 1))
|
||||
if (t_type <> 21) and (t_type <> 22) {
|
||||
p0 = this.byte_at(s: args.dict, i: j ~sat+ ((t as base.u64) * 2))
|
||||
p1 = this.byte_at(s: args.dict, i: j ~sat+ (((t as base.u64) * 2) + 1))
|
||||
if (p0 <> 0) or (p1 <> 0) {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
}
|
||||
t ~mod+= 1
|
||||
}
|
||||
}
|
||||
this.parse_pos = pos
|
||||
return ok
|
||||
}
|
||||
|
||||
// --------
|
||||
|
||||
// copy_dictionary_word writes a transformed static dictionary word to
|
||||
// this.transformed_word and returns its length. It returns 0xFFFF_FFFE for a
|
||||
// BROTLI_DECODER_ERROR_FORMAT_DICTIONARY error and 0xFFFF_FFFF for a
|
||||
// BROTLI_DECODER_ERROR_FORMAT_TRANSFORM error. args.address is the C code's
|
||||
// (distance_code - max_distance - 1 - compound_dictionary_size) and args.ctx
|
||||
// is the C code's BROTLI_CONTEXT(p1, p2, s->context_lookup).
|
||||
//
|
||||
// Mirrors the static dictionary part of ProcessCommandsInternal in decode.c,
|
||||
// including its selection of a word list and transform list from the shared
|
||||
// dictionary.
|
||||
pri func decoder.copy_dictionary_word!(dictbuf: roslice base.u8, address: base.u32, len: base.u32[..= 31], ctx: base.u32[..= 63]) base.u32 {
|
||||
var dict_id : base.u32[..= 63]
|
||||
var wsel : base.u32[..= 64]
|
||||
var tsel : base.u32[..= 64]
|
||||
var shift : base.u32[..= 24]
|
||||
var nt : base.u32[..= 255]
|
||||
var address : base.u32
|
||||
var transform_idx : base.u32
|
||||
var word_idx : base.u32
|
||||
var offset : base.u32
|
||||
var dist_remaining : base.u32
|
||||
var d2 : base.u32
|
||||
var w2 : base.u32[..= 64]
|
||||
var t2 : base.u32[..= 64]
|
||||
var shift2 : base.u32[..= 24]
|
||||
var num : base.u32
|
||||
|
||||
dict_id = 0
|
||||
if this.sd_context_based {
|
||||
dict_id = (this.sd_context_map[args.ctx] & 0x3F) as base.u32
|
||||
}
|
||||
wsel = this.sd_words[dict_id] as base.u32
|
||||
tsel = this.sd_transforms[dict_id] as base.u32
|
||||
shift = this.word_list_size_bits(wsel: wsel, len: args.len)
|
||||
nt = this.transform_list_length(tsel: tsel)
|
||||
address = args.address
|
||||
transform_idx = address >> shift
|
||||
|
||||
// If the distance is out of bound, select a next static dictionary if
|
||||
// there exist multiple.
|
||||
if ((transform_idx >= nt) or (shift == 0)) and (this.sd_num_dictionaries > 1) {
|
||||
dist_remaining = address ~mod- ((((1 as base.u32) << shift) & 0xFFFF_FFFE) ~mod* nt)
|
||||
d2 = 0
|
||||
while d2 < this.sd_num_dictionaries {
|
||||
assert d2 < 64 via "a < b: a < c; c <= b"(c: this.sd_num_dictionaries)
|
||||
w2 = this.sd_words[d2] as base.u32
|
||||
shift2 = this.word_list_size_bits(wsel: w2, len: args.len)
|
||||
if (d2 <> dict_id) and (shift2 <> 0) {
|
||||
t2 = this.sd_transforms[d2] as base.u32
|
||||
num = (((1 as base.u32) << shift2) & 0xFFFF_FFFE) ~mod* this.transform_list_length(tsel: t2)
|
||||
if dist_remaining < num {
|
||||
wsel = w2
|
||||
tsel = t2
|
||||
address = dist_remaining
|
||||
shift = shift2
|
||||
nt = this.transform_list_length(tsel: tsel)
|
||||
transform_idx = address >> shift
|
||||
break
|
||||
}
|
||||
dist_remaining -= num
|
||||
}
|
||||
d2 += 1
|
||||
}
|
||||
}
|
||||
|
||||
if shift == 0 {
|
||||
return 0xFFFF_FFFE
|
||||
} else if transform_idx >= nt {
|
||||
return 0xFFFF_FFFF
|
||||
}
|
||||
word_idx = address & (((1 as base.u32) << shift) - 1)
|
||||
offset = this.word_list_offset(wsel: wsel, len: args.len) ~mod+ (word_idx ~mod* args.len)
|
||||
num = this.transform_dictionary_word!(dictbuf: args.dictbuf,
|
||||
wsel: wsel,
|
||||
tsel: tsel,
|
||||
offset: offset,
|
||||
len: args.len,
|
||||
transform_idx: transform_idx.min(no_more_than: 254))
|
||||
return num
|
||||
}
|
||||
|
||||
// word_list_size_bits returns the C code's size_bits_by_length[args.len] for
|
||||
// the word list selected by args.wsel: 0 for the built-in list or (1 + j) for
|
||||
// the j'th custom list.
|
||||
pri func decoder.word_list_size_bits(wsel: base.u32[..= 64], len: base.u32[..= 31]) base.u32[..= 24] {
|
||||
if args.wsel == 0 {
|
||||
return DICTIONARY_SIZE_BITS_BY_LENGTH[args.len]
|
||||
}
|
||||
return (this.wl_size_bits[(args.wsel - 1) & 63][args.len] & 15) as base.u32
|
||||
}
|
||||
|
||||
// word_list_offset returns the C code's offsets_by_length[args.len] for the
|
||||
// word list selected by args.wsel.
|
||||
pri func decoder.word_list_offset(wsel: base.u32[..= 64], len: base.u32[..= 31]) base.u32 {
|
||||
if args.wsel == 0 {
|
||||
return DICTIONARY_OFFSETS_BY_LENGTH[args.len]
|
||||
}
|
||||
return this.wl_offsets[(args.wsel - 1) & 63][args.len]
|
||||
}
|
||||
|
||||
// transform_list_length returns the C code's num_transforms for the
|
||||
// transform list selected by args.tsel: 0 for the built-in list or (1 + j)
|
||||
// for the j'th custom list.
|
||||
pri func decoder.transform_list_length(tsel: base.u32[..= 64]) base.u32[..= 255] {
|
||||
if args.tsel == 0 {
|
||||
return NUM_TRANSFORMS
|
||||
}
|
||||
return this.tl_num_transforms[(args.tsel - 1) & 63].min(no_more_than: 255)
|
||||
}
|
||||
|
||||
// transform_dictionary_word writes the transformed word to
|
||||
// this.transformed_word and returns its length. Mirrors
|
||||
// BrotliTransformDictionaryWord in transform.c.
|
||||
pri func decoder.transform_dictionary_word!(dictbuf: roslice base.u8, wsel: base.u32[..= 64], tsel: base.u32[..= 64], offset: base.u32, len: base.u32[..= 31], transform_idx: base.u32[..= 254]) base.u32 {
|
||||
var idx : base.u32
|
||||
var t : base.u32[..= 120]
|
||||
var k : base.u32[..= 63]
|
||||
var tpos : base.u64
|
||||
var prefix_id : base.u32[..= 0xFF]
|
||||
var t_type : base.u32[..= 0xFF]
|
||||
var suffix_id : base.u32[..= 0xFF]
|
||||
var param : base.u32
|
||||
var word_start : base.u32
|
||||
var word_len : base.u32
|
||||
var skip : base.u32[..= 9]
|
||||
var i : base.u32
|
||||
var j : base.u64
|
||||
var remaining : base.u32
|
||||
var step : base.u32
|
||||
|
||||
// Look up the transform's prefix_id, type and suffix_id.
|
||||
k = 0
|
||||
tpos = 0
|
||||
if args.tsel == 0 {
|
||||
t = args.transform_idx.min(no_more_than: 120)
|
||||
prefix_id = TRANSFORM_PREFIX_ID[t] as base.u32
|
||||
t_type = TRANSFORM_TYPE[t] as base.u32
|
||||
suffix_id = TRANSFORM_SUFFIX_ID[t] as base.u32
|
||||
} else {
|
||||
k = (args.tsel - 1) & 63
|
||||
tpos = (this.tl_transforms[k] as base.u64) + ((args.transform_idx as base.u64) * 3)
|
||||
prefix_id = this.byte_at(s: args.dictbuf, i: tpos)
|
||||
t_type = this.byte_at(s: args.dictbuf, i: tpos + 1)
|
||||
suffix_id = this.byte_at(s: args.dictbuf, i: tpos + 2)
|
||||
}
|
||||
|
||||
// The prefix.
|
||||
idx = this.append_affix!(dictbuf: args.dictbuf, tsel: args.tsel, id: prefix_id, idx: 0)
|
||||
|
||||
// The word, possibly with its first or last few bytes omitted.
|
||||
word_len = args.len
|
||||
skip = 0
|
||||
if t_type <= 9 {
|
||||
// BROTLI_TRANSFORM_OMIT_LAST_N.
|
||||
if t_type < args.len {
|
||||
word_len = args.len ~mod- t_type
|
||||
} else {
|
||||
word_len = 0
|
||||
}
|
||||
} else if (12 <= t_type) and (t_type <= 20) {
|
||||
// BROTLI_TRANSFORM_OMIT_FIRST_N.
|
||||
skip = t_type - 11
|
||||
if skip < args.len {
|
||||
word_len = args.len ~mod- skip
|
||||
} else {
|
||||
word_len = 0
|
||||
}
|
||||
}
|
||||
word_start = idx
|
||||
i = 0
|
||||
while i < word_len {
|
||||
if idx >= 1024 {
|
||||
break
|
||||
}
|
||||
if args.wsel == 0 {
|
||||
j = ((args.offset ~mod+ skip) ~mod+ i) as base.u64
|
||||
if j < 122784 {
|
||||
this.transformed_word[idx] = DICTIONARY[j]
|
||||
}
|
||||
} else {
|
||||
j = (this.wl_data[(args.wsel - 1) & 63] as base.u64) +
|
||||
(((args.offset ~mod+ skip) ~mod+ i) as base.u64)
|
||||
if j < args.dictbuf.length() {
|
||||
this.transformed_word[idx] = args.dictbuf[j]
|
||||
}
|
||||
}
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
|
||||
// Upper-casing and shifting. The C code's ToUpperCase can modify bytes
|
||||
// after the word's end, but those bytes are either overwritten by the
|
||||
// suffix or are past the transformed word's length, so we don't bother.
|
||||
if t_type == 10 {
|
||||
// BROTLI_TRANSFORM_UPPERCASE_FIRST.
|
||||
this.to_upper_case!(p: word_start, end: idx)
|
||||
} else if t_type == 11 {
|
||||
// BROTLI_TRANSFORM_UPPERCASE_ALL.
|
||||
i = word_start
|
||||
while i < idx {
|
||||
step = this.to_upper_case!(p: i, end: idx)
|
||||
i ~mod+= step
|
||||
}
|
||||
} else if (t_type == 21) or (t_type == 22) {
|
||||
// BROTLI_TRANSFORM_SHIFT_FIRST or BROTLI_TRANSFORM_SHIFT_ALL. Only
|
||||
// custom transforms have these types, and their parameters exist.
|
||||
j = (this.tl_params[k] as base.u64) + ((args.transform_idx as base.u64) * 2)
|
||||
param = this.byte_at(s: args.dictbuf, i: j) |
|
||||
(this.byte_at(s: args.dictbuf, i: j + 1) << 8)
|
||||
if t_type == 21 {
|
||||
this.shift!(p: word_start, word_len: idx ~mod- word_start, param: param)
|
||||
} else {
|
||||
i = word_start
|
||||
remaining = idx ~mod- word_start
|
||||
while remaining > 0 {
|
||||
step = this.shift!(p: i, word_len: remaining, param: param)
|
||||
if step >= remaining {
|
||||
break
|
||||
}
|
||||
i ~mod+= step
|
||||
remaining ~mod-= step
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The suffix.
|
||||
idx = this.append_affix!(dictbuf: args.dictbuf, tsel: args.tsel, id: suffix_id, idx: idx)
|
||||
return idx
|
||||
}
|
||||
|
||||
// append_affix appends the args.id'th prefix or suffix of the transform list
|
||||
// selected by args.tsel to this.transformed_word, starting at args.idx. It
|
||||
// returns the new length.
|
||||
pri func decoder.append_affix!(dictbuf: roslice base.u8, tsel: base.u32[..= 64], id: base.u32[..= 0xFF], idx: base.u32) base.u32 {
|
||||
var idx : base.u32
|
||||
var affix : base.u32[..= 216]
|
||||
var affix_len : base.u32
|
||||
var i : base.u32
|
||||
var j : base.u32
|
||||
var k : base.u32[..= 63]
|
||||
var pos : base.u64
|
||||
|
||||
idx = args.idx
|
||||
if args.tsel == 0 {
|
||||
affix = PREFIX_SUFFIX_MAP[args.id.min(no_more_than: 49)]
|
||||
affix_len = PREFIX_SUFFIX[affix] as base.u32
|
||||
i = 0
|
||||
while i < affix_len {
|
||||
j = (affix ~mod+ 1) ~mod+ i
|
||||
if (j >= 217) or (idx >= 1024) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = PREFIX_SUFFIX[j]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
k = (args.tsel - 1) & 63
|
||||
pos = (this.tl_prefix_suffix[k] as base.u64) + (this.tl_map[k][args.id] as base.u64)
|
||||
affix_len = 0
|
||||
if pos < args.dictbuf.length() {
|
||||
affix_len = args.dictbuf[pos] as base.u32
|
||||
}
|
||||
i = 0
|
||||
while i < affix_len {
|
||||
pos ~mod+= 1
|
||||
if (pos >= args.dictbuf.length()) or (idx >= 1024) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = args.dictbuf[pos]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
// shift applies a shift transform to the (UTF-8 encoded) character at
|
||||
// this.transformed_word[args.p ..], which is args.word_len bytes long. It
|
||||
// returns the step: the number of bytes in the character. Mirrors Shift in
|
||||
// transform.c.
|
||||
pri func decoder.shift!(p: base.u32, word_len: base.u32, param: base.u32) base.u32 {
|
||||
var p : base.u32[..= 1019]
|
||||
var scalar : base.u32
|
||||
var c0 : base.u32
|
||||
var c1 : base.u32
|
||||
var c2 : base.u32
|
||||
var c3 : base.u32
|
||||
|
||||
if args.p > 1019 {
|
||||
return 1
|
||||
}
|
||||
p = args.p
|
||||
// Limited sign extension: scalar < (1 << 24).
|
||||
scalar = (args.param & 0x7FFF) ~mod+ (0x100_0000 ~mod- (args.param & 0x8000))
|
||||
c0 = this.transformed_word[p] as base.u32
|
||||
if c0 < 0x80 {
|
||||
// 1-byte rune / 0sssssss / 7 bit scalar (ASCII).
|
||||
scalar ~mod+= c0
|
||||
this.transformed_word[p] = (scalar & 0x7F) as base.u8
|
||||
return 1
|
||||
} else if c0 < 0xC0 {
|
||||
// Continuation / 10AAAAAA.
|
||||
return 1
|
||||
} else if c0 < 0xE0 {
|
||||
// 2-byte rune / 110sssss AAssssss / 11 bit scalar.
|
||||
if args.word_len < 2 {
|
||||
return 1
|
||||
}
|
||||
c1 = this.transformed_word[p + 1] as base.u32
|
||||
scalar ~mod+= (c1 & 0x3F) | ((c0 & 0x1F) << 6)
|
||||
this.transformed_word[p + 0] = (0xC0 | ((scalar >> 6) & 0x1F)) as base.u8
|
||||
this.transformed_word[p + 1] = ((c1 & 0xC0) | (scalar & 0x3F)) as base.u8
|
||||
return 2
|
||||
} else if c0 < 0xF0 {
|
||||
// 3-byte rune / 1110ssss AAssssss BBssssss / 16 bit scalar.
|
||||
if args.word_len < 3 {
|
||||
return args.word_len
|
||||
}
|
||||
c1 = this.transformed_word[p + 1] as base.u32
|
||||
c2 = this.transformed_word[p + 2] as base.u32
|
||||
scalar ~mod+= (c2 & 0x3F) | ((c1 & 0x3F) << 6) | ((c0 & 0x0F) << 12)
|
||||
this.transformed_word[p + 0] = (0xE0 | ((scalar >> 12) & 0x0F)) as base.u8
|
||||
this.transformed_word[p + 1] = ((c1 & 0xC0) | ((scalar >> 6) & 0x3F)) as base.u8
|
||||
this.transformed_word[p + 2] = ((c2 & 0xC0) | (scalar & 0x3F)) as base.u8
|
||||
return 3
|
||||
} else if c0 < 0xF8 {
|
||||
// 4-byte rune / 11110sss AAssssss BBssssss CCssssss / 21 bit scalar.
|
||||
if args.word_len < 4 {
|
||||
return args.word_len
|
||||
}
|
||||
c1 = this.transformed_word[p + 1] as base.u32
|
||||
c2 = this.transformed_word[p + 2] as base.u32
|
||||
c3 = this.transformed_word[p + 3] as base.u32
|
||||
scalar ~mod+= (c3 & 0x3F) | ((c2 & 0x3F) << 6) | ((c1 & 0x3F) << 12) | ((c0 & 0x07) << 18)
|
||||
this.transformed_word[p + 0] = (0xF0 | ((scalar >> 18) & 0x07)) as base.u8
|
||||
this.transformed_word[p + 1] = ((c1 & 0xC0) | ((scalar >> 12) & 0x3F)) as base.u8
|
||||
this.transformed_word[p + 2] = ((c2 & 0xC0) | ((scalar >> 6) & 0x3F)) as base.u8
|
||||
this.transformed_word[p + 3] = ((c3 & 0xC0) | (scalar & 0x3F)) as base.u8
|
||||
return 4
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
// Mirrors ToUpperCase in transform.c, except that it does not modify bytes at
|
||||
// or after args.end. It returns the step: the number of bytes in the
|
||||
// (overly simplified model of a) UTF-8 encoded character.
|
||||
pri func decoder.to_upper_case!(p: base.u32, end: base.u32) base.u32 {
|
||||
var c : base.u8
|
||||
|
||||
if args.p >= 1024 {
|
||||
return 1
|
||||
}
|
||||
c = this.transformed_word[args.p]
|
||||
if c < 0xC0 {
|
||||
if ('a' <= c) and (c <= 'z') {
|
||||
this.transformed_word[args.p] = c ^ 32
|
||||
}
|
||||
return 1
|
||||
}
|
||||
// An overly simplified uppercasing model for UTF-8.
|
||||
if c < 0xE0 {
|
||||
if ((args.p + 1) < args.end) and ((args.p + 1) < 1024) {
|
||||
this.transformed_word[args.p + 1] ^= 32
|
||||
}
|
||||
return 2
|
||||
}
|
||||
// An arbitrary transform for three byte characters.
|
||||
if ((args.p + 2) < args.end) and ((args.p + 2) < 1024) {
|
||||
this.transformed_word[args.p + 2] ^= 5
|
||||
}
|
||||
return 3
|
||||
}
|
||||
|
||||
// byte_at returns args.s[args.i], or 0 if args.i is out of bounds.
|
||||
pri func decoder.byte_at(s: roslice base.u8, i: base.u64) base.u32[..= 0xFF] {
|
||||
if args.i < args.s.length() {
|
||||
return args.s[args.i] as base.u32
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -135,7 +135,7 @@ pri func decoder.read_simple_huffman_code?(src: base.io_reader, alphabet_size_ma
|
||||
// symbol, 1 means 2 symbols, 2 means 3 symbols, 3 means 4 symbols with
|
||||
// lengths [2, 2, 2, 2], 4 means 4 symbols with lengths [1, 2, 3, 3].
|
||||
pri func decoder.build_simple_huffman_table!(hoff: base.u32, num_symbols: base.u32[..= 4]) base.status {
|
||||
var t : base.u32[..= 671634]
|
||||
var t : base.u32[..= 677778]
|
||||
var v : array[4] base.u32[..= 0x3FF]
|
||||
var tmp : base.u32[..= 0x3FF]
|
||||
var i : base.u32
|
||||
|
||||
+156
-188
@@ -13,17 +13,18 @@
|
||||
// Mirrors the BROTLI_STATE_HUFFMAN_CODE_0 through
|
||||
// BROTLI_STATE_BEFORE_COMPRESSED_METABLOCK_BODY states of
|
||||
// BrotliDecoderDecompressStream in decode.c, followed by ProcessCommands.
|
||||
pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slice base.u8) {
|
||||
var t : base.u32
|
||||
var n : base.u32[..= 256]
|
||||
var n0 : base.u32[..= 256]
|
||||
var n2 : base.u32[..= 256]
|
||||
var bits : base.u32
|
||||
var npostfix : base.u32[..= 3]
|
||||
var ndirect : base.u32[..= 120]
|
||||
var alphabet : base.u32[..= 520]
|
||||
var i : base.u32
|
||||
var k : base.u32
|
||||
pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slice base.u8, dictbuf: roslice base.u8) {
|
||||
var t : base.u32
|
||||
var n : base.u32[..= 256]
|
||||
var n0 : base.u32[..= 256]
|
||||
var n2 : base.u32[..= 256]
|
||||
var bits : base.u32
|
||||
var npostfix : base.u32[..= 3]
|
||||
var ndirect : base.u32[..= 120]
|
||||
var alphabet_max : base.u32[..= 1128]
|
||||
var alphabet : base.u32[..= 544]
|
||||
var i : base.u32
|
||||
var k : base.u32
|
||||
|
||||
// BROTLI_STATE_HUFFMAN_CODE_0 ..= BROTLI_STATE_HUFFMAN_CODE_3.
|
||||
t = 0
|
||||
@@ -75,8 +76,19 @@ pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slic
|
||||
this.decode_context_map?(src: args.src, size: n2 << 2, offset: DIST_CONTEXT_MAP_BASE)
|
||||
this.num_dist_htrees = this.bits_value.min(no_more_than: 256)
|
||||
|
||||
// BROTLI_DISTANCE_ALPHABET_SIZE(npostfix, ndirect, BROTLI_MAX_DISTANCE_BITS).
|
||||
alphabet = 16 + ndirect + ((24 as base.u32) << (npostfix + 1))
|
||||
if this.large_window {
|
||||
// BROTLI_DISTANCE_ALPHABET_SIZE(npostfix, ndirect,
|
||||
// BROTLI_LARGE_MAX_DISTANCE_BITS) and the max_alphabet_size of
|
||||
// BrotliCalculateDistanceCodeLimit(BROTLI_MAX_ALLOWED_DISTANCE,
|
||||
// npostfix, ndirect).
|
||||
alphabet_max = 16 + ndirect + ((62 as base.u32) << (npostfix + 1))
|
||||
alphabet = this.calculate_distance_code_limit(npostfix: npostfix, ndirect: ndirect)
|
||||
} else {
|
||||
// BROTLI_DISTANCE_ALPHABET_SIZE(npostfix, ndirect,
|
||||
// BROTLI_MAX_DISTANCE_BITS).
|
||||
alphabet = 16 + ndirect + ((24 as base.u32) << (npostfix + 1))
|
||||
alphabet_max = alphabet
|
||||
}
|
||||
this.distance_alphabet_size = alphabet
|
||||
|
||||
// BROTLI_STATE_TREE_GROUP. Mirrors HuffmanTreeGroupDecode, three times.
|
||||
@@ -104,7 +116,7 @@ pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slic
|
||||
while k < this.num_dist_htrees {
|
||||
assert k < 256 via "a < b: a < c; c <= b"(c: this.num_dist_htrees)
|
||||
this.read_huffman_code?(src: args.src,
|
||||
alphabet_size_max: alphabet,
|
||||
alphabet_size_max: alphabet_max,
|
||||
alphabet_size_limit: alphabet,
|
||||
hoff: DIST_HUFFS_BASE + (k * DIST_HUFFS_STRIDE),
|
||||
table_size_max: DIST_HUFFS_STRIDE)
|
||||
@@ -117,7 +129,50 @@ pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slic
|
||||
this.htree_command = 0
|
||||
this.calculate_distance_lut!()
|
||||
|
||||
this.process_commands?(src: args.src, workbuf: args.workbuf)
|
||||
this.process_commands?(src: args.src, workbuf: args.workbuf, dictbuf: args.dictbuf)
|
||||
}
|
||||
|
||||
// Mirrors BrotliCalculateDistanceCodeLimit in constants.h, with a
|
||||
// max_distance of BROTLI_MAX_ALLOWED_DISTANCE (0x7FFF_FFFC). It returns the
|
||||
// max_alphabet_size, which is at most 544.
|
||||
pri func decoder.calculate_distance_code_limit(npostfix: base.u32[..= 3], ndirect: base.u32[..= 120]) base.u32[..= 544] {
|
||||
var offset : base.u32
|
||||
var ndistbits : base.u32
|
||||
var tmp : base.u32
|
||||
var half : base.u32[..= 1]
|
||||
var group : base.u32
|
||||
var postfix : base.u32[..= 7]
|
||||
var ret : base.u32
|
||||
|
||||
// The max_distance <= ndirect case cannot happen.
|
||||
//
|
||||
// offset is (forbidden_distance - ndirect - 1), where forbidden_distance
|
||||
// is (max_distance + 1), after removing the postfix and "head-start".
|
||||
offset = ((0x7FFF_FFFC - args.ndirect) >> args.npostfix) + 4
|
||||
postfix = ((1 as base.u32) << args.npostfix) - 1
|
||||
// Calculate the number of distance bits (a poor man's log2floor).
|
||||
tmp = offset / 2
|
||||
ndistbits = 0
|
||||
while (tmp <> 0) and (ndistbits < 32) {
|
||||
ndistbits += 1
|
||||
tmp >>= 1
|
||||
}
|
||||
// One bit is covered with subrange addressing ("half").
|
||||
ndistbits ~mod-= 1
|
||||
// Find the subrange.
|
||||
half = (offset >> (ndistbits & 31)) & 1
|
||||
// Calculate the "group" part of the dcode.
|
||||
group = ((ndistbits ~mod- 1) ~mod<< 1) | half
|
||||
if group == 0 {
|
||||
return args.ndirect + 16
|
||||
}
|
||||
// Decrement "group", so it is the last permitted "group".
|
||||
group -= 1
|
||||
ret = (((group ~mod<< args.npostfix) | postfix) ~mod+ args.ndirect) ~mod+ 17
|
||||
if ret > 544 {
|
||||
return 544
|
||||
}
|
||||
return ret
|
||||
}
|
||||
|
||||
// Mirrors DecodeContextMap in decode.c. The context map is written to
|
||||
@@ -310,7 +365,7 @@ pri func decoder.prepare_literal_decoding!() {
|
||||
pri func decoder.calculate_distance_lut!() {
|
||||
var npostfix : base.u32[..= 3]
|
||||
var ndirect : base.u32[..= 120]
|
||||
var limit : base.u32[..= 520]
|
||||
var limit : base.u32[..= 544]
|
||||
var postfix : base.u32[..= 8]
|
||||
var i : base.u32
|
||||
var j : base.u32
|
||||
@@ -361,7 +416,7 @@ pri func decoder.calculate_distance_lut!() {
|
||||
}
|
||||
|
||||
// Mirrors ProcessCommandsInternal in decode.c.
|
||||
pri func decoder.process_commands?(src: base.io_reader, workbuf: slice base.u8) {
|
||||
pri func decoder.process_commands?(src: base.io_reader, workbuf: slice base.u8, dictbuf: roslice base.u8) {
|
||||
var cmd_code : base.u32[..= 703]
|
||||
var insert_length : base.u32
|
||||
var p1 : base.u32[..= 0xFF]
|
||||
@@ -375,6 +430,10 @@ pri func decoder.process_commands?(src: base.io_reader, workbuf: slice base.u8)
|
||||
var n : base.u32
|
||||
var k : base.u32
|
||||
var b : base.u8
|
||||
var compound_size : base.u32
|
||||
var address : base.u32
|
||||
var chunk : base.u32[..= 14]
|
||||
var j : base.u64
|
||||
|
||||
while true {
|
||||
// CommandBegin.
|
||||
@@ -460,39 +519,84 @@ pri func decoder.process_commands?(src: base.io_reader, workbuf: slice base.u8)
|
||||
distance = this.distance_code
|
||||
|
||||
if distance > max_distance {
|
||||
// Apply a static dictionary reference, since the distance is
|
||||
// larger than the max LZ77 distance.
|
||||
// Apply a compound dictionary or static dictionary reference,
|
||||
// since the distance is larger than the max LZ77 distance.
|
||||
//
|
||||
// The maximum allowed distance is BROTLI_MAX_ALLOWED_DISTANCE.
|
||||
if distance > 0x7FFF_FFFC {
|
||||
return "#bad distance"
|
||||
}
|
||||
// The C code checks for SHARED_BROTLI_MIN_DICTIONARY_WORD_LENGTH
|
||||
// (4) and SHARED_BROTLI_MAX_DICTIONARY_WORD_LENGTH (31) but the
|
||||
// RFC 7932 dictionary's size_bits_by_length is zero for lengths
|
||||
// 25 ..= 31, which also fails with "#bad dictionary".
|
||||
//
|
||||
// TODO: support custom dictionaries.
|
||||
if (i < 4) or (24 < i) {
|
||||
return "#bad dictionary"
|
||||
}
|
||||
this.dist_rb_idx ~mod+= this.distance_context
|
||||
n = this.copy_dictionary_word!(
|
||||
address: (distance ~mod- max_distance) ~mod- 1,
|
||||
len: i.min(no_more_than: 24))
|
||||
if n == 0xFFFF_FFFF {
|
||||
return "#bad transform"
|
||||
} else if (n == 0) and (distance <= 120) {
|
||||
return "#bad transform"
|
||||
}
|
||||
this.mb_remaining ~mod-= n
|
||||
k = 0
|
||||
while k < n {
|
||||
this.put_byte!(workbuf: args.workbuf, b: this.transformed_word[k & 63])
|
||||
k ~mod+= 1
|
||||
if this.pos == this.rb_size {
|
||||
this.ring_event?()
|
||||
compound_size = this.chunk_offsets[this.num_chunks]
|
||||
address = (distance - max_distance) - 1
|
||||
|
||||
if address < compound_size {
|
||||
// Mirrors InitializeCompoundDictionaryCopy and
|
||||
// CopyFromCompoundDictionary in decode.c. This address is the
|
||||
// C code's (compound_dictionary_size - (s->distance_code -
|
||||
// s->max_distance)).
|
||||
address = compound_size ~mod- (address + 1)
|
||||
if ((address as base.u64) + (i as base.u64)) > (compound_size as base.u64) {
|
||||
return "#bad compound dictionary"
|
||||
}
|
||||
// Update the recent distances cache.
|
||||
this.dist_rb[this.dist_rb_idx & 3] = distance
|
||||
this.dist_rb_idx ~mod+= 1
|
||||
this.mb_remaining ~mod-= i
|
||||
chunk = 0
|
||||
while i > 0 {
|
||||
while chunk < 14 {
|
||||
if address < this.chunk_offsets[chunk + 1] {
|
||||
break
|
||||
}
|
||||
chunk += 1
|
||||
}
|
||||
j = (this.chunk_starts[chunk] as base.u64) +
|
||||
((address ~mod- this.chunk_offsets[chunk]) as base.u64)
|
||||
b = 0
|
||||
if j < args.dictbuf.length() {
|
||||
b = args.dictbuf[j]
|
||||
}
|
||||
this.put_byte!(workbuf: args.workbuf, b: b)
|
||||
address ~mod+= 1
|
||||
i ~mod-= 1
|
||||
if this.pos == this.rb_size {
|
||||
this.ring_event?()
|
||||
}
|
||||
}
|
||||
|
||||
} else if (4 <= i) and (i <= 31) {
|
||||
// SHARED_BROTLI_MIN_DICTIONARY_WORD_LENGTH and
|
||||
// SHARED_BROTLI_MAX_DICTIONARY_WORD_LENGTH.
|
||||
p1 = this.get_byte(workbuf: args.workbuf, distance: 1) as base.u32
|
||||
p2 = this.get_byte(workbuf: args.workbuf, distance: 2) as base.u32
|
||||
mode_offset = this.context_mode << 9
|
||||
ctx = (CONTEXT_LOOKUP[mode_offset | p1] |
|
||||
CONTEXT_LOOKUP[mode_offset | 0x100 | p2]) as base.u32
|
||||
// Compensate double distance-ring-buffer roll.
|
||||
this.dist_rb_idx ~mod+= this.distance_context
|
||||
n = this.copy_dictionary_word!(dictbuf: args.dictbuf,
|
||||
address: address ~mod- compound_size,
|
||||
len: i.min(no_more_than: 31),
|
||||
ctx: ctx)
|
||||
if n == 0xFFFF_FFFE {
|
||||
return "#bad dictionary"
|
||||
} else if n == 0xFFFF_FFFF {
|
||||
return "#bad transform"
|
||||
} else if (n == 0) and (distance <= 120) {
|
||||
return "#bad transform"
|
||||
}
|
||||
this.mb_remaining ~mod-= n
|
||||
k = 0
|
||||
while k < n {
|
||||
this.put_byte!(workbuf: args.workbuf, b: this.transformed_word[k & 1023])
|
||||
k ~mod+= 1
|
||||
if this.pos == this.rb_size {
|
||||
this.ring_event?()
|
||||
}
|
||||
}
|
||||
|
||||
} else {
|
||||
return "#bad dictionary"
|
||||
}
|
||||
|
||||
} else {
|
||||
@@ -543,12 +647,17 @@ pri func decoder.read_distance?(src: base.io_reader) {
|
||||
if (code >= this.distance_alphabet_size) or (code >= 1128) {
|
||||
return "#internal error: inconsistent Huffman table"
|
||||
}
|
||||
this.read_bits?(src: args.src, n: this.dist_extra_bits[code].min(no_more_than: 24) as base.u32)
|
||||
// The C code uses BrotliSafeReadBits32, as large-window brotli can have
|
||||
// more than 24 extra bits.
|
||||
this.read_bits32?(src: args.src, n: this.dist_extra_bits[code].min(no_more_than: 32) as base.u32)
|
||||
if code >= 1128 {
|
||||
return "#internal error: inconsistent Huffman table"
|
||||
}
|
||||
this.distance_code = this.dist_offset[code] ~mod+
|
||||
(this.bits_value ~mod<< this.distance_postfix_bits)
|
||||
// The C code calculates this with brotli_reg_t (64-bit) arithmetic and
|
||||
// then casts to int. BrotliCalculateDistanceCodeLimit means that the
|
||||
// result is less than BROTLI_MAX_ALLOWED_DISTANCE.
|
||||
this.distance_code = (((this.dist_offset[code] as base.u64) ~mod+
|
||||
((this.bits_value as base.u64) ~mod<< this.distance_postfix_bits)) & 0xFFFF_FFFF) as base.u32
|
||||
}
|
||||
|
||||
// Mirrors TakeDistanceFromRingBuffer in decode.c.
|
||||
@@ -585,144 +694,3 @@ pri func decoder.take_distance_from_ring_buffer!(code: base.u32[..= 15]) {
|
||||
this.distance_code = v - 3
|
||||
}
|
||||
}
|
||||
|
||||
// copy_dictionary_word writes a transformed static dictionary word to
|
||||
// this.transformed_word and returns its length. It returns 0xFFFF_FFFF if the
|
||||
// transform index is out of range. Mirrors part of ProcessCommandsInternal in
|
||||
// decode.c and BrotliTransformDictionaryWord in transform.c.
|
||||
pri func decoder.copy_dictionary_word!(address: base.u32, len: base.u32[..= 24]) base.u32 {
|
||||
var shift : base.u32[..= 24]
|
||||
var word_idx : base.u32
|
||||
var transform_idx : base.u32
|
||||
var offset : base.u32
|
||||
var t : base.u32[..= 120]
|
||||
var t_type : base.u32[..= 22]
|
||||
var affix : base.u32[..= 216]
|
||||
var affix_len : base.u32
|
||||
var idx : base.u32
|
||||
var word_start : base.u32
|
||||
var word_len : base.u32
|
||||
var skip : base.u32[..= 9]
|
||||
var i : base.u32
|
||||
var j : base.u32
|
||||
var step : base.u32
|
||||
|
||||
shift = DICTIONARY_SIZE_BITS_BY_LENGTH[args.len]
|
||||
word_idx = args.address & (((1 as base.u32) << shift) - 1)
|
||||
transform_idx = args.address >> shift
|
||||
if transform_idx >= NUM_TRANSFORMS {
|
||||
return 0xFFFF_FFFF
|
||||
}
|
||||
t = transform_idx.min(no_more_than: 120)
|
||||
offset = DICTIONARY_OFFSETS_BY_LENGTH[args.len] ~mod+ (word_idx ~mod* args.len)
|
||||
|
||||
// The prefix.
|
||||
idx = 0
|
||||
affix = PREFIX_SUFFIX_MAP[TRANSFORM_PREFIX_ID[t]]
|
||||
affix_len = PREFIX_SUFFIX[affix] as base.u32
|
||||
i = 0
|
||||
while i < affix_len {
|
||||
j = (affix ~mod+ 1) ~mod+ i
|
||||
if (j >= 217) or (idx >= 64) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = PREFIX_SUFFIX[j]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
|
||||
// The word, possibly with its first or last few bytes omitted.
|
||||
t_type = TRANSFORM_TYPE[t] as base.u32
|
||||
word_len = args.len
|
||||
skip = 0
|
||||
if t_type <= 9 {
|
||||
// BROTLI_TRANSFORM_OMIT_LAST_N.
|
||||
if t_type < args.len {
|
||||
word_len = args.len ~mod- t_type
|
||||
} else {
|
||||
word_len = 0
|
||||
}
|
||||
} else if (12 <= t_type) and (t_type <= 20) {
|
||||
// BROTLI_TRANSFORM_OMIT_FIRST_N.
|
||||
skip = t_type - 11
|
||||
if skip < args.len {
|
||||
word_len = args.len ~mod- skip
|
||||
} else {
|
||||
word_len = 0
|
||||
}
|
||||
}
|
||||
word_start = idx
|
||||
i = 0
|
||||
while i < word_len {
|
||||
j = (offset ~mod+ skip) ~mod+ i
|
||||
if (j >= 122784) or (idx >= 64) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = DICTIONARY[j]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
|
||||
// Upper-casing. The C code's ToUpperCase can modify bytes after the
|
||||
// word's end, but those bytes are either overwritten by the suffix or are
|
||||
// past the transformed word's length, so we don't bother.
|
||||
if t_type == 10 {
|
||||
// BROTLI_TRANSFORM_UPPERCASE_FIRST.
|
||||
this.to_upper_case!(p: word_start, end: idx)
|
||||
} else if t_type == 11 {
|
||||
// BROTLI_TRANSFORM_UPPERCASE_ALL.
|
||||
i = word_start
|
||||
while i < idx {
|
||||
step = this.to_upper_case!(p: i, end: idx)
|
||||
i ~mod+= step
|
||||
}
|
||||
}
|
||||
// TODO: support BROTLI_TRANSFORM_SHIFT_FIRST and _SHIFT_ALL (used only by
|
||||
// custom dictionaries' transforms).
|
||||
|
||||
// The suffix.
|
||||
affix = PREFIX_SUFFIX_MAP[TRANSFORM_SUFFIX_ID[t]]
|
||||
affix_len = PREFIX_SUFFIX[affix] as base.u32
|
||||
i = 0
|
||||
while i < affix_len {
|
||||
j = (affix ~mod+ 1) ~mod+ i
|
||||
if (j >= 217) or (idx >= 64) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = PREFIX_SUFFIX[j]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
|
||||
return idx
|
||||
}
|
||||
|
||||
// Mirrors ToUpperCase in transform.c, except that it does not modify bytes at
|
||||
// or after args.end. It returns the step: the number of bytes in the
|
||||
// (overly simplified model of a) UTF-8 encoded character.
|
||||
pri func decoder.to_upper_case!(p: base.u32, end: base.u32) base.u32 {
|
||||
var c : base.u8
|
||||
|
||||
if args.p >= 64 {
|
||||
return 1
|
||||
}
|
||||
c = this.transformed_word[args.p]
|
||||
if c < 0xC0 {
|
||||
if ('a' <= c) and (c <= 'z') {
|
||||
this.transformed_word[args.p] = c ^ 32
|
||||
}
|
||||
return 1
|
||||
}
|
||||
// An overly simplified uppercasing model for UTF-8.
|
||||
if c < 0xE0 {
|
||||
if ((args.p + 1) < args.end) and ((args.p + 1) < 64) {
|
||||
this.transformed_word[args.p + 1] ^= 32
|
||||
}
|
||||
return 2
|
||||
}
|
||||
// An arbitrary transform for three byte characters.
|
||||
if ((args.p + 2) < args.end) and ((args.p + 2) < 64) {
|
||||
this.transformed_word[args.p + 2] ^= 5
|
||||
}
|
||||
return 3
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
// Copyright 2026 The Wuffs Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
|
||||
// https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
|
||||
// <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
|
||||
// option. This file may not be copied, modified, or distributed
|
||||
// except according to those terms.
|
||||
//
|
||||
// SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
|
||||
// --------
|
||||
|
||||
// Quirks are discussed in (/doc/note/quirks.md).
|
||||
//
|
||||
// The base38 encoding of "brot" is 0x0A_ADE4. Left shifting by 10 gives
|
||||
// 0x2AB7_9000.
|
||||
pri const QUIRKS_BASE : base.u32 = 0x2AB7_9000
|
||||
|
||||
// --------
|
||||
|
||||
// When this quirk is set, a positive value enables large-window brotli
|
||||
// (https://datatracker.ietf.org/doc/draft-vandevenne-shared-brotli-format/),
|
||||
// with window sizes up to (1 << 30) bytes. Zero means standard (RFC 7932)
|
||||
// brotli. It mirrors the C code's BROTLI_DECODER_PARAM_LARGE_WINDOW.
|
||||
//
|
||||
// It must be set before decoding starts. Setting it also increases the
|
||||
// workbuf_len.
|
||||
//
|
||||
// Without this quirk, a stream that starts with the large-window WBITS escape
|
||||
// is rejected with "#unsupported large window". With this quirk, a stream that
|
||||
// does not start with that escape is decoded as standard brotli.
|
||||
pub const QUIRK_LARGE_WINDOW : base.u32 = 0x2AB7_9000 | 0x00
|
||||
Reference in new issue
Block a user