mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-21 00:47:28 +01:00
Merge branch 'm1c-decoder-extensions' into claude/project-thread-0izcgq
Combines the encoder (quality 0 and 1) with the decoder's large-window mode and custom dictionaries. The test and mimic files take both sides' new sections; the fuzzer gains the std/softfloat module that the encoder now pulls into std/brotli. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
This commit is contained in:
42 files changed
+4220
-461
No files matched your search
+162
-14
@@ -49,8 +49,10 @@ decoder does not read any further than that.
|
||||
|
||||
## Workbuf Layout
|
||||
|
||||
`workbuf_len` is `(1 << 24) + 542` bytes, both minimum and maximum: the
|
||||
largest (non-large-window) brotli window size plus some slack. The slack
|
||||
Without the large-window quirk and without attached dictionaries (see the
|
||||
"Large Window" and "Dictionaries" sections below), `workbuf_len` is `(1 << 24)
|
||||
+ 542` bytes, both minimum and maximum: the largest (non-large-window) brotli
|
||||
window size plus some slack. The slack
|
||||
mirrors the C code's `kRingBufferWriteAheadSlack`, which its fast paths use to
|
||||
write (and discard) a few bytes past the ring buffer's end. This
|
||||
implementation does not yet have such fast paths, but it still reserves the
|
||||
@@ -101,15 +103,18 @@ a corresponding Wuffs status:
|
||||
| `BROTLI_DECODER_ERROR_FORMAT_PADDING_2` | `"#bad final padding"` |
|
||||
| `BROTLI_DECODER_ERROR_FORMAT_DISTANCE` | `"#bad distance"` |
|
||||
| `BROTLI_DECODER_ERROR_FORMAT_BLOCK_SWITCH` | `"#bad block switch"` |
|
||||
| `BROTLI_DECODER_ERROR_COMPOUND_DICTIONARY` | `"#bad compound dictionary"` |
|
||||
| `BROTLI_DECODER_NEEDS_MORE_INPUT` (with closed `src`) | `"#truncated input"` |
|
||||
|
||||
Notes:
|
||||
|
||||
- `"#unsupported large window"` is for the large-window escape `WBITS` value
|
||||
(`0010001`). libbrotlidec (with `BROTLI_DECODER_PARAM_LARGE_WINDOW` not set,
|
||||
the default) rejects it with `BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS`. For
|
||||
standard (non-large-window) brotli, no other `WBITS` value is invalid, so
|
||||
`"#bad window bits"` is not currently produced.
|
||||
(`0010001`) when the `QUIRK_LARGE_WINDOW` quirk is not set. libbrotlidec
|
||||
(with `BROTLI_DECODER_PARAM_LARGE_WINDOW` not set, the default) rejects it
|
||||
with `BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS`. For standard
|
||||
(non-large-window) brotli, no other `WBITS` value is invalid. With the quirk
|
||||
set, `"#bad window bits"` is for a large-window header with a non-zero
|
||||
reserved bit or with window bits outside of `[10 ..= 30]`.
|
||||
- Like the C code, which error an overrun metablock produces (`"#bad block
|
||||
length"` or `"#bad metablock length"`) can depend on how the input is split
|
||||
across `transform_io` calls. When the C code runs out of input, it first
|
||||
@@ -118,8 +123,9 @@ Notes:
|
||||
running out of input.
|
||||
- `"#bad block switch"` needs more than `(1 << 24)` commands in one metablock.
|
||||
It is implemented but not tested.
|
||||
- The remaining libbrotlidec error codes do not apply: `COMPOUND_DICTIONARY`
|
||||
and `DICTIONARY_NOT_SET` (custom dictionaries are not supported),
|
||||
- The remaining libbrotlidec error codes do not apply: `DICTIONARY_NOT_SET`
|
||||
(it needs a word list without data, which libbrotlidec only has when built
|
||||
with `BROTLI_EXTERNAL_DICTIONARY_DATA` and not given that data),
|
||||
`INVALID_ARGUMENTS`, the `ALLOC_*` codes (this decoder does not allocate)
|
||||
and `UNREACHABLE`.
|
||||
|
||||
@@ -145,13 +151,11 @@ invariants for memory safety but Wuffs requires explicit bounds checks.
|
||||
|
||||
## TODO
|
||||
|
||||
- Large-window brotli (`BROTLI_DECODER_PARAM_LARGE_WINDOW`, window bits up to
|
||||
30). The window-related fields are sized for 24 bits.
|
||||
- Custom, shared and compound dictionaries.
|
||||
- Metadata callbacks (metadata blocks are currently skipped).
|
||||
- Metadata callbacks (metadata blocks are currently skipped). See the
|
||||
"Metadata Blocks" section below.
|
||||
- Fast paths.
|
||||
- Fuzzers.
|
||||
- An encoder.
|
||||
- Encoder quality levels 2 to 11, encoder dictionaries and large-window
|
||||
encoding (see the "Encoder" section below).
|
||||
|
||||
|
||||
## Encoder
|
||||
@@ -324,3 +328,147 @@ some synthetic inputs).
|
||||
- Large-window encoding (only used at qualities 3 and above).
|
||||
- Custom and shared dictionaries (`BrotliEncoderAttachPreparedDictionary`).
|
||||
- Changing parameters mid-stream.
|
||||
|
||||
|
||||
## Large Window
|
||||
|
||||
Large-window brotli (an incompatible extension of RFC 7932, with window sizes
|
||||
up to `(1 << 30)` bytes) is enabled by the `QUIRK_LARGE_WINDOW` quirk. It
|
||||
mirrors libbrotlidec's `BROTLI_DECODER_PARAM_LARGE_WINDOW` parameter: it must
|
||||
be set before decoding starts (afterwards, `set_quirk` returns `"#bad call
|
||||
sequence"`, like `BrotliDecoderSetParameter` failing once the decoder has
|
||||
left its `BROTLI_STATE_UNINITED` state). With the quirk set:
|
||||
|
||||
- a stream that starts with the large-window `WBITS` escape is decoded as a
|
||||
large-window stream: the escape is followed by a reserved bit (that must be
|
||||
zero) and 6 window bits (that must be in `[10 ..= 30]`), otherwise the
|
||||
decoder fails with `"#bad window bits"`.
|
||||
- a stream that does not start with that escape is decoded as standard brotli,
|
||||
exactly as without the quirk.
|
||||
- `workbuf_len` is `(1 << 30) + 542` bytes (plus any dictionaries), the
|
||||
largest large-window window size plus slack. As before, the workbuf only
|
||||
needs to be as long as the stream's actual window size plus 542 bytes (plus
|
||||
any dictionaries). Without the quirk, it is `(1 << 24) + 542` bytes (plus
|
||||
any dictionaries), as before.
|
||||
|
||||
For a large-window stream, like the C code, the distance alphabet's
|
||||
`alphabet_size_max` uses `BROTLI_LARGE_MAX_DISTANCE_BITS` (62) and its
|
||||
`alphabet_size_limit` comes from `BrotliCalculateDistanceCodeLimit` (at most
|
||||
544 symbols, so that no distance exceeds `BROTLI_MAX_ALLOWED_DISTANCE`). Up to
|
||||
29 extra distance bits are read (the C code's `BrotliSafeReadBits32`) and the
|
||||
distance is calculated with 64-bit arithmetic, like the C code's
|
||||
`brotli_reg_t`. The Huffman table stride for distance trees is sized for 544
|
||||
symbols in both modes.
|
||||
|
||||
The `DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE` constant is now the overall
|
||||
worst case: `(1 << 30) + 542` plus the maximum total dictionary length
|
||||
(`0x7FFF_FFFF`). The new `DECODER_WORKBUF_LEN_STANDARD` and
|
||||
`DECODER_WORKBUF_LEN_LARGE_WINDOW` constants give the window-only values.
|
||||
|
||||
|
||||
## Dictionaries
|
||||
|
||||
`decoder.add_dictionary!(dict_type: base.u32, dict: roslice base.u8, workbuf:
|
||||
slice base.u8) base.status` mirrors libbrotlidec's
|
||||
`BrotliDecoderAttachDictionary`. `dict_type` is `DICTIONARY_TYPE_RAW` (0, the
|
||||
C code's `BROTLI_SHARED_DICTIONARY_RAW`) or `DICTIONARY_TYPE_SERIALIZED` (1,
|
||||
`BROTLI_SHARED_DICTIONARY_SERIALIZED`). It must be called before decoding
|
||||
starts. It can be called multiple times.
|
||||
|
||||
Wuffs structs cannot hold pointers or slices, so `add_dictionary` copies the
|
||||
dictionary bytes into the workbuf. Every `add_dictionary` and `transform_io`
|
||||
call must be passed the same workbuf (or at least one whose first bytes hold
|
||||
the same dictionary bytes). The `add_dictionary` workbuf only needs to be long
|
||||
enough to hold all of the dictionaries added so far. Call `workbuf_len` after
|
||||
the last `add_dictionary` call: it grows by each dictionary's length.
|
||||
|
||||
The workbuf layout is:
|
||||
|
||||
[0 .. D) the attached dictionaries, concatenated, where D is
|
||||
the total length of all successfully added
|
||||
dictionaries (raw or serialized, in full).
|
||||
[D .. D + W + 542) the history ring buffer (W = 1 << WBITS) and slack.
|
||||
|
||||
The dictionaries come first because their position should not depend on the
|
||||
stream's window size, which is only known once decoding has started.
|
||||
|
||||
Raw dictionaries are LZ77 prefix dictionaries. Like the C code (its
|
||||
`BrotliDecoderCompoundDictionary` and `AttachCompoundDictionary`,
|
||||
`InitializeCompoundDictionaryCopy` and `CopyFromCompoundDictionary`
|
||||
functions), at most 15 such "compound dictionary" chunks (raw dictionaries
|
||||
plus serialized dictionaries' LZ77 prefixes) can be attached. A backward
|
||||
reference distance `d` beyond the current maximum LZ77 distance `m` (but
|
||||
within `BROTLI_MAX_ALLOWED_DISTANCE`) refers to the compound dictionary when
|
||||
`d - m - 1` is less than the compound dictionary's total size `T`, at address
|
||||
`T - (d - m)`. A copy that would extend past the compound dictionary's end
|
||||
fails with `"#bad compound dictionary"`. Otherwise, the static dictionary's
|
||||
address is `d - m - 1 - T`.
|
||||
|
||||
Serialized dictionaries are parsed and validated like
|
||||
`BrotliSharedDictionaryAttach` (`DecodeSharedDictionary`,
|
||||
`DryParseDictionary`, `ParseDictionary`, `ParseWordList`,
|
||||
`ParseTransformsList` and `ParsePrefixSuffixTable` in
|
||||
`c/common/shared_dictionary.c`): the `0x91 0x00` magic, an optional LZ77
|
||||
prefix (at most 1073741823 bytes), up to 64 custom word lists (word lengths 4
|
||||
to 31, at most 15 size bits per length), up to 64 custom transform lists (up
|
||||
to 256 prefix/suffix stringlets, up to 255 transforms, transform types up to
|
||||
`BROTLI_TRANSFORM_SHIFT_ALL`, with 16-bit parameters for the
|
||||
`BROTLI_TRANSFORM_SHIFT_FIRST` and `BROTLI_TRANSFORM_SHIFT_ALL` types), up to
|
||||
64 (word list, transform list) combinations and an optional 64-entry context
|
||||
map. The parsed metadata lives in fixed-size decoder fields (`wl_etc`, `tl_etc`
|
||||
and `sd_etc`), with offsets into the copied bytes in the workbuf. Like the C
|
||||
code:
|
||||
|
||||
- only one attached serialized dictionary can have custom word lists or
|
||||
transform lists (otherwise `"#unsupported multiple custom static
|
||||
dictionaries"`), and
|
||||
- attaching a serialized dictionary without custom word lists or transform
|
||||
lists (only an LZ77 prefix) resets the static dictionary to the built-in
|
||||
(RFC 7932) one, even if an earlier serialized dictionary had custom lists.
|
||||
|
||||
Static dictionary references then mirror `ProcessCommandsInternal`: the word
|
||||
list and transform list are selected by the literal context (`BROTLI_CONTEXT(p1,
|
||||
p2, context_lookup)`) through the context map (if enabled), with the C code's
|
||||
fall back to the other (word list, transform list) combinations when the
|
||||
address is out of range, and words are transformed like
|
||||
`BrotliTransformDictionaryWord` (including `Shift` for the shift transforms).
|
||||
Transformed words can be up to 541 bytes long.
|
||||
|
||||
`add_dictionary` statuses (the C code just returns `BROTLI_FALSE`):
|
||||
|
||||
| Condition | Wuffs status |
|
||||
| -------------------------------------------------------------- | ----------------------------------------------------- |
|
||||
| decoding has started | `base."#bad call sequence"` |
|
||||
| unknown `dict_type` | `base."#bad argument"` |
|
||||
| more than 15 compound dictionary chunks | `"#too many compound dictionaries"` |
|
||||
| invalid serialized dictionary | `"#bad shared dictionary"` |
|
||||
| a second serialized dictionary with custom lists | `"#unsupported multiple custom static dictionaries"` |
|
||||
| more than `0x7FFF_FFFF` bytes of dictionaries in total | `"#too much dictionary data"` |
|
||||
| `workbuf` shorter than the dictionaries so far plus this one | `base."#bad workbuf length"` |
|
||||
|
||||
Differences from the C code:
|
||||
|
||||
- On failure, `add_dictionary` leaves the decoder unchanged. libbrotlidec's
|
||||
`BrotliDecoderAttachDictionary` can fail after partially updating its shared
|
||||
dictionary (e.g. after counting a serialized dictionary's LZ77 prefix or
|
||||
reading its number of word lists), which can affect later attach calls.
|
||||
- libbrotlicommon only parses serialized dictionaries when built with
|
||||
`BROTLI_EXPERIMENTAL`, which is not the default. A default build rejects
|
||||
every serialized dictionary. This package always accepts them, matching an
|
||||
experimental build. The mimic tests skip serialized dictionary cases when
|
||||
the linked libbrotlidec rejects them; they pass, with no skips, when linked
|
||||
against a `BROTLI_EXPERIMENTAL` build.
|
||||
- The C code holds the compound dictionary's total size in an `int`. This
|
||||
package rejects more than `0x7FFF_FFFF` bytes of dictionaries in total
|
||||
(counting serialized dictionaries in full), where the C code would overflow.
|
||||
- `BROTLI_DECODER_ERROR_DICTIONARY_NOT_SET` cannot happen (see above).
|
||||
|
||||
|
||||
## Metadata Blocks
|
||||
|
||||
libbrotlidec can report metadata blocks' contents through callbacks
|
||||
(`BrotliDecoderSetMetadataCallbacks`). Wuffs code cannot call back into its
|
||||
caller, so there is no equivalent: metadata blocks are skipped (validated and
|
||||
consumed but not reported), exactly as libbrotlidec does without callbacks,
|
||||
so the decoded output is the same. A future quirk could expose metadata
|
||||
blocks, e.g. by suspending with a status when one starts.
|
||||
+234
-40
@@ -20,12 +20,12 @@
|
||||
//
|
||||
// See README.md for how C error codes map to Wuffs statuses.
|
||||
//
|
||||
// TODO: support large-window brotli (BROTLI_DECODER_PARAM_LARGE_WINDOW),
|
||||
// custom / attached / shared dictionaries, metadata callbacks and fast paths.
|
||||
// TODO: support metadata callbacks and fast paths.
|
||||
|
||||
pub status "#bad block length"
|
||||
pub status "#bad block switch"
|
||||
pub status "#bad code length code lengths"
|
||||
pub status "#bad compound dictionary"
|
||||
pub status "#bad context map repeat"
|
||||
pub status "#bad dictionary"
|
||||
pub status "#bad distance"
|
||||
@@ -36,12 +36,16 @@ pub status "#bad Huffman space"
|
||||
pub status "#bad metablock length"
|
||||
pub status "#bad padding"
|
||||
pub status "#bad reserved bit"
|
||||
pub status "#bad shared dictionary"
|
||||
pub status "#bad simple Huffman alphabet"
|
||||
pub status "#bad simple Huffman symbol"
|
||||
pub status "#bad transform"
|
||||
pub status "#bad window bits"
|
||||
pub status "#too many compound dictionaries"
|
||||
pub status "#too much dictionary data"
|
||||
pub status "#truncated input"
|
||||
pub status "#unsupported large window"
|
||||
pub status "#unsupported multiple custom static dictionaries"
|
||||
|
||||
pri status "#internal error: inconsistent bit reader"
|
||||
pri status "#internal error: inconsistent Huffman table"
|
||||
@@ -49,10 +53,33 @@ pri status "#internal error: inconsistent ring buffer"
|
||||
|
||||
pub const DECODER_DST_HISTORY_RETAIN_LENGTH_MAX_INCL_WORST_CASE : base.u64 = 0
|
||||
|
||||
// DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE is the largest (standard, not
|
||||
// large-window) brotli window size, (1 << 24) bytes, plus WORKBUF_SLACK. The
|
||||
// workbuf holds the history ring buffer.
|
||||
pub const DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE : base.u64 = 0x100_021E // 0x100_0000 + 542.
|
||||
// DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE is the largest workbuf_len value:
|
||||
// DICT_TOTAL_MAX bytes of attached dictionaries plus the largest
|
||||
// (large-window) brotli window size, (1 << 30) bytes, plus WORKBUF_SLACK.
|
||||
//
|
||||
// Without attached dictionaries and without the QUIRK_LARGE_WINDOW quirk,
|
||||
// workbuf_len is DECODER_WORKBUF_LEN_STANDARD, which is much smaller.
|
||||
pub const DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE : base.u64 = 0xC000_021D // 0x7FFF_FFFF + 0x4000_0000 + 542.
|
||||
|
||||
// DECODER_WORKBUF_LEN_STANDARD is the largest standard (not large-window)
|
||||
// brotli window size, (1 << 24) bytes, plus WORKBUF_SLACK. This is the
|
||||
// workbuf_len when no dictionaries are attached and the QUIRK_LARGE_WINDOW
|
||||
// quirk is not set.
|
||||
pub const DECODER_WORKBUF_LEN_STANDARD : base.u64 = 0x100_021E // 0x100_0000 + 542.
|
||||
|
||||
// DECODER_WORKBUF_LEN_LARGE_WINDOW is the largest large-window brotli window
|
||||
// size, (1 << 30) bytes, plus WORKBUF_SLACK.
|
||||
pub const DECODER_WORKBUF_LEN_LARGE_WINDOW : base.u64 = 0x4000_021E // 0x4000_0000 + 542.
|
||||
|
||||
// DICTIONARY_TYPE_RAW and DICTIONARY_TYPE_SERIALIZED are the values for
|
||||
// decoder.add_dictionary's dict_type argument. They mirror the C code's
|
||||
// BROTLI_SHARED_DICTIONARY_RAW and BROTLI_SHARED_DICTIONARY_SERIALIZED.
|
||||
pub const DICTIONARY_TYPE_RAW : base.u32 = 0
|
||||
pub const DICTIONARY_TYPE_SERIALIZED : base.u32 = 1
|
||||
|
||||
// DICT_TOTAL_MAX is the maximum total length of all attached dictionaries.
|
||||
// The C code holds the compound dictionaries' total size in an int.
|
||||
pri const DICT_TOTAL_MAX : base.u64 = 0x7FFF_FFFF
|
||||
|
||||
// WORKBUF_SLACK mirrors kRingBufferWriteAheadSlack in decode.c. The C code's
|
||||
// fast paths can write (and then discard) up to this many bytes past the end
|
||||
@@ -70,7 +97,9 @@ pri const WORKBUF_SLACK : base.u64 = 542
|
||||
// or the BROTLI_HUFFMAN_MAX_SIZE_ETC constants in huffman.h. There are at most
|
||||
// 256 trees in each of the literal, insert-and-copy and distance groups.
|
||||
//
|
||||
// TODO: large-window brotli needs a larger distance stride.
|
||||
// The distance stride is for the largest distance alphabet_size_limit, which
|
||||
// is 544 (for large-window brotli, BrotliCalculateDistanceCodeLimit with
|
||||
// NPOSTFIX = 3 and NDIRECT = 120), not 520 (for standard brotli).
|
||||
|
||||
pri const LIT_HUFFS_BASE : base.u32 = 0
|
||||
pri const LIT_HUFFS_STRIDE : base.u32 = 632 // 256 + 376.
|
||||
@@ -79,26 +108,26 @@ pri const CMD_HUFFS_BASE : base.u32 = 161792 // 0 + (256 * 632).
|
||||
pri const CMD_HUFFS_STRIDE : base.u32 = 1080 // 704 + 376.
|
||||
|
||||
pri const DIST_HUFFS_BASE : base.u32 = 438272 // 161792 + (256 * 1080).
|
||||
pri const DIST_HUFFS_STRIDE : base.u32 = 896 // 520 + 376.
|
||||
pri const DIST_HUFFS_STRIDE : base.u32 = 920 // 544 + 376.
|
||||
|
||||
// BLOCK_TYPE_HUFFS_STRIDE is BROTLI_HUFFMAN_MAX_SIZE_258.
|
||||
pri const BLOCK_TYPE_HUFFS_BASE : base.u32 = 667648 // 438272 + (256 * 896).
|
||||
pri const BLOCK_TYPE_HUFFS_BASE : base.u32 = 673792 // 438272 + (256 * 920).
|
||||
pri const BLOCK_TYPE_HUFFS_STRIDE : base.u32 = 632
|
||||
|
||||
// BLOCK_LEN_HUFFS_STRIDE is BROTLI_HUFFMAN_MAX_SIZE_26.
|
||||
pri const BLOCK_LEN_HUFFS_BASE : base.u32 = 669544 // 667648 + (3 * 632).
|
||||
pri const BLOCK_LEN_HUFFS_BASE : base.u32 = 675688 // 673792 + (3 * 632).
|
||||
pri const BLOCK_LEN_HUFFS_STRIDE : base.u32 = 396
|
||||
|
||||
// The context map table's size is BROTLI_HUFFMAN_MAX_SIZE_272.
|
||||
pri const CONTEXT_MAP_HUFFS_BASE : base.u32 = 670732 // 669544 + (3 * 396).
|
||||
pri const CONTEXT_MAP_HUFFS_BASE : base.u32 = 676876 // 675688 + (3 * 396).
|
||||
|
||||
// The code length code (and the code length code length code) tables are 5
|
||||
// (and 4) bits wide, replicated to fill 256 entries, the same as the 8-bit
|
||||
// root tables for every other code.
|
||||
pri const CODE_LENGTH_HUFFS_BASE : base.u32 = 671378 // 670732 + 646.
|
||||
pri const CODE_LENGTH_CODE_HUFFS_BASE : base.u32 = 671634 // 671378 + 256.
|
||||
pri const CODE_LENGTH_HUFFS_BASE : base.u32 = 677522 // 676876 + 646.
|
||||
pri const CODE_LENGTH_CODE_HUFFS_BASE : base.u32 = 677778 // 677522 + 256.
|
||||
|
||||
pri const HUFFS_LEN : base.u32 = 671890 // 671634 + 256.
|
||||
pri const HUFFS_LEN : base.u32 = 678034 // 677778 + 256.
|
||||
|
||||
// The two context maps live in one decoder.context_maps array. The literal
|
||||
// context map (up to 256 block types times 64 contexts) comes first, then the
|
||||
@@ -136,11 +165,12 @@ pub struct decoder? implements base.io_transformer(
|
||||
// rb_flushed is how many of the ring buffer's bytes (in the current
|
||||
// trip around the ring) have been written to args.dst.
|
||||
//
|
||||
// TODO: large-window brotli allows up to 30 window_bits.
|
||||
window_bits : base.u32[..= 24],
|
||||
window_size : base.u32[..= 0x100_0000],
|
||||
max_backward_distance : base.u32[..= 0x100_0000],
|
||||
rb_size : base.u32[..= 0x100_0000],
|
||||
// window_bits is at most 24 for standard brotli and at most 30 for
|
||||
// large-window brotli.
|
||||
window_bits : base.u32[..= 30],
|
||||
window_size : base.u32[..= 0x4000_0000],
|
||||
max_backward_distance : base.u32[..= 0x4000_0000],
|
||||
rb_size : base.u32[..= 0x4000_0000],
|
||||
pos : base.u32,
|
||||
rb_flushed : base.u32,
|
||||
|
||||
@@ -168,9 +198,12 @@ pub struct decoder? implements base.io_transformer(
|
||||
|
||||
distance_postfix_bits : base.u32[..= 3],
|
||||
num_direct_distance_codes : base.u32[..= 120],
|
||||
distance_alphabet_size : base.u32[..= 520],
|
||||
num_literal_htrees : base.u32[..= 256],
|
||||
num_dist_htrees : base.u32[..= 256],
|
||||
// distance_alphabet_size is the C code's
|
||||
// distance_hgroup.alphabet_size_limit: at most 520 for standard
|
||||
// brotli and at most 544 for large-window brotli.
|
||||
distance_alphabet_size : base.u32[..= 544],
|
||||
num_literal_htrees : base.u32[..= 256],
|
||||
num_dist_htrees : base.u32[..= 256],
|
||||
|
||||
// The C code's context_map_slice and dist_context_map_slice, as
|
||||
// offsets into decoder.context_maps, plus the context_lookup mode.
|
||||
@@ -197,9 +230,71 @@ pub struct decoder? implements base.io_transformer(
|
||||
insert_length : base.u32,
|
||||
copy_length : base.u32,
|
||||
|
||||
// quirk_large_window is the QUIRK_LARGE_WINDOW quirk (the C code's
|
||||
// BROTLI_DECODER_PARAM_LARGE_WINDOW parameter). large_window is
|
||||
// whether the stream is a large-window stream (the C code's
|
||||
// s->large_window after DecodeWindowBits), which needs both the quirk
|
||||
// and the large-window WBITS escape.
|
||||
quirk_large_window : base.bool,
|
||||
large_window : base.bool,
|
||||
|
||||
// started is whether the C code's s->state would no longer be
|
||||
// BROTLI_STATE_UNINITED. Quirks and dictionaries can only be set or
|
||||
// added before then.
|
||||
started : base.bool,
|
||||
|
||||
// dict_total is the total length of all attached dictionaries, which
|
||||
// live at the start of the workbuf: args.workbuf[.. dict_total]. The
|
||||
// ring buffer follows: args.workbuf[dict_total ..].
|
||||
dict_total : base.u64[..= 0x7FFF_FFFF],
|
||||
|
||||
// The compound dictionary (the C code's BrotliDecoderCompoundDictionary
|
||||
// in state.h). It has num_chunks chunks. The i'th chunk starts at
|
||||
// args.workbuf[chunk_starts[i]] and spans compound dictionary
|
||||
// addresses [chunk_offsets[i] .. chunk_offsets[i + 1]].
|
||||
// chunk_offsets[num_chunks] is the C code's total_size.
|
||||
num_chunks : base.u32[..= 15],
|
||||
chunk_starts : array[15] base.u32,
|
||||
chunk_offsets : array[16] base.u32,
|
||||
|
||||
// The shared dictionary's static dictionary part (the C code's
|
||||
// BrotliSharedDictionary in shared_dictionary_internal.h). The C
|
||||
// code's words[i] and transforms[i] are given by sd_words[i] and
|
||||
// sd_transforms[i]: 0 means the built-in (RFC 7932) word list or
|
||||
// transform list and (1 + j) means the j'th custom list (see the
|
||||
// wl_etc and tl_etc fields). sd_context_based, sd_context_map and
|
||||
// sd_num_dictionaries are the C code's context_based, context_map and
|
||||
// num_dictionaries (except that the initial zero value means one).
|
||||
// sd_num_word_lists and sd_num_transform_lists are the C code's
|
||||
// num_word_lists and num_transform_lists.
|
||||
sd_context_based : base.bool,
|
||||
sd_num_dictionaries : base.u32[..= 64],
|
||||
sd_num_word_lists : base.u32[..= 64],
|
||||
sd_num_transform_lists : base.u32[..= 64],
|
||||
sd_words : array[64] base.u8[..= 64],
|
||||
sd_transforms : array[64] base.u8[..= 64],
|
||||
sd_context_map : array[64] base.u8[..= 63],
|
||||
|
||||
// The prefix (LZ77 dictionary chunk) of a serialized shared
|
||||
// dictionary, as found by parse_shared_dictionary: its position
|
||||
// within that dictionary's bytes and its length.
|
||||
sd_prefix_pos : base.u64,
|
||||
sd_prefix_size : base.u64,
|
||||
|
||||
// More staging fields for parse_shared_dictionary. See also the
|
||||
// stage_etc arrays.
|
||||
stage_num_word_lists : base.u32[..= 64],
|
||||
stage_num_transform_lists : base.u32[..= 64],
|
||||
stage_num_dictionaries : base.u32[..= 64],
|
||||
stage_context_based : base.bool,
|
||||
|
||||
// parse_pos is where parse_shared_dictionary's helper functions
|
||||
// store the new parsing position.
|
||||
parse_pos : base.u64,
|
||||
|
||||
util : base.utility,
|
||||
) + (
|
||||
huffs : array[671890] base.u16, // HUFFS_LEN.
|
||||
huffs : array[678034] base.u16, // HUFFS_LEN.
|
||||
|
||||
context_maps : array[17408] base.u8, // 16384 + 1024.
|
||||
context_modes : array[256] base.u8,
|
||||
@@ -221,16 +316,52 @@ pub struct decoder? implements base.io_transformer(
|
||||
dist_offset : array[1128] base.u32,
|
||||
|
||||
// transformed_word holds a transformed static dictionary word: a
|
||||
// prefix (up to 8 bytes), a word (up to 24 bytes) and a suffix (up to
|
||||
// 8 bytes).
|
||||
transformed_word : array[64] base.u8,
|
||||
// prefix (up to 255 bytes), a word (up to 31 bytes) and a suffix (up
|
||||
// to 255 bytes). Built-in transforms' prefixes and suffixes are at
|
||||
// most 8 bytes and built-in words are at most 24 bytes.
|
||||
transformed_word : array[1024] base.u8,
|
||||
|
||||
// Custom word lists (the C code's BrotliDictionary words_instances).
|
||||
// wl_data is the offset (within the workbuf) of the list's words.
|
||||
wl_size_bits : array[64] array[32] base.u8,
|
||||
wl_offsets : array[64] array[32] base.u32,
|
||||
wl_data : array[64] base.u32,
|
||||
|
||||
// Custom transform lists (the C code's BrotliTransforms
|
||||
// transforms_instances). tl_prefix_suffix, tl_transforms and
|
||||
// tl_params are offsets (within the workbuf) of the C code's
|
||||
// prefix_suffix, transforms and params arrays. tl_map is the C code's
|
||||
// prefix_suffix_map.
|
||||
tl_prefix_suffix : array[64] base.u32,
|
||||
tl_map : array[64] array[256] base.u16,
|
||||
tl_num_transforms : array[64] base.u32,
|
||||
tl_transforms : array[64] base.u32,
|
||||
tl_params : array[64] base.u32,
|
||||
|
||||
// Staging areas for parse_shared_dictionary. They are copied to the
|
||||
// sd_etc fields only if the whole serialized dictionary is valid.
|
||||
stage_words : array[64] base.u8,
|
||||
stage_transforms : array[64] base.u8,
|
||||
stage_context_map : array[64] base.u8,
|
||||
)
|
||||
|
||||
pub func decoder.get_quirk(key: base.u32) base.u64 {
|
||||
if (args.key == QUIRK_LARGE_WINDOW) and this.quirk_large_window {
|
||||
return 1
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// set_quirk mirrors BrotliDecoderSetParameter in decode.c, which fails once
|
||||
// decoding has started.
|
||||
pub func decoder.set_quirk!(key: base.u32, value: base.u64) base.status {
|
||||
if args.key == QUIRK_LARGE_WINDOW {
|
||||
if this.started {
|
||||
return base."#bad call sequence"
|
||||
}
|
||||
this.quirk_large_window = args.value > 0
|
||||
return ok
|
||||
}
|
||||
return base."#unsupported option"
|
||||
}
|
||||
|
||||
@@ -238,10 +369,19 @@ pub func decoder.dst_history_retain_length() base.optional_u63 {
|
||||
return this.util.make_optional_u63(has_value: true, value: 0)
|
||||
}
|
||||
|
||||
// workbuf_len is the total length of the dictionaries added so far (see
|
||||
// add_dictionary) plus the largest window size (for the QUIRK_LARGE_WINDOW
|
||||
// quirk's current value) plus WORKBUF_SLACK.
|
||||
pub func decoder.workbuf_len() base.range_ii_u64 {
|
||||
var n : base.u64[..= 0x4000_021E]
|
||||
|
||||
n = DECODER_WORKBUF_LEN_STANDARD
|
||||
if this.quirk_large_window {
|
||||
n = DECODER_WORKBUF_LEN_LARGE_WINDOW
|
||||
}
|
||||
return this.util.make_range_ii_u64(
|
||||
min_incl: DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE,
|
||||
max_incl: DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE)
|
||||
min_incl: n + this.dict_total,
|
||||
max_incl: n + this.dict_total)
|
||||
}
|
||||
|
||||
// transform_io wraps do_transform_io, which decodes into the ring buffer (the
|
||||
@@ -252,17 +392,31 @@ pub func decoder.workbuf_len() base.range_ii_u64 {
|
||||
// write" when it needs every decoded byte flushed before it can wrap around
|
||||
// the ring buffer. If this function can flush all of them, it resumes
|
||||
// do_transform_io immediately instead of suspending.
|
||||
//
|
||||
// The first this.dict_total bytes of args.workbuf hold the attached
|
||||
// dictionaries (see add_dictionary). The ring buffer follows them.
|
||||
pub func decoder.transform_io?(dst: base.io_writer, src: base.io_reader, workbuf: slice base.u8) {
|
||||
var status : base.status
|
||||
|
||||
while true {
|
||||
status =? this.do_transform_io?(src: args.src, workbuf: args.workbuf)
|
||||
this.flush!(dst: args.dst, workbuf: args.workbuf)
|
||||
if this.dict_total > args.workbuf.length() {
|
||||
return base."#bad workbuf length"
|
||||
}
|
||||
status =? this.do_transform_io?(src: args.src,
|
||||
workbuf: args.workbuf[this.dict_total ..],
|
||||
dictbuf: args.workbuf[.. this.dict_total])
|
||||
if this.dict_total > args.workbuf.length() {
|
||||
return base."#bad workbuf length"
|
||||
}
|
||||
this.flush!(dst: args.dst, workbuf: args.workbuf[this.dict_total ..])
|
||||
|
||||
if status.is_ok() {
|
||||
while not this.is_fully_flushed() {
|
||||
yield? base."$short write"
|
||||
this.flush!(dst: args.dst, workbuf: args.workbuf)
|
||||
if this.dict_total > args.workbuf.length() {
|
||||
return base."#bad workbuf length"
|
||||
}
|
||||
this.flush!(dst: args.dst, workbuf: args.workbuf[this.dict_total ..])
|
||||
}
|
||||
return ok
|
||||
|
||||
@@ -310,7 +464,7 @@ pri func decoder.flush!(dst: base.io_writer, workbuf: slice base.u8) {
|
||||
|
||||
// Mirrors the BROTLI_STATE_ETC state machine in BrotliDecoderDecompressStream
|
||||
// in decode.c.
|
||||
pri func decoder.do_transform_io?(src: base.io_reader, workbuf: slice base.u8) {
|
||||
pri func decoder.do_transform_io?(src: base.io_reader, workbuf: slice base.u8, dictbuf: roslice base.u8) {
|
||||
var w : base.u64
|
||||
var aligned : base.bool
|
||||
|
||||
@@ -358,7 +512,7 @@ pri func decoder.do_transform_io?(src: base.io_reader, workbuf: slice base.u8) {
|
||||
// BROTLI_STATE_UNCOMPRESSED.
|
||||
this.copy_uncompressed?(src: args.src, workbuf: args.workbuf)
|
||||
} else {
|
||||
this.decode_compressed_metablock?(src: args.src, workbuf: args.workbuf)
|
||||
this.decode_compressed_metablock?(src: args.src, workbuf: args.workbuf, dictbuf: args.dictbuf)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -386,6 +540,7 @@ pri func decoder.initialize_codes!() {
|
||||
|
||||
this.bits = 0
|
||||
this.n_bits = 0
|
||||
this.large_window = false
|
||||
this.pos = 0
|
||||
this.rb_size = 0
|
||||
this.rb_flushed = 0
|
||||
@@ -452,6 +607,23 @@ pri func decoder.read_bits?(src: base.io_reader, n: base.u32[..= 24]) {
|
||||
this.n_bits = n_bits - n
|
||||
}
|
||||
|
||||
// read_bits32 is like read_bits but args.n can be up to 32. Mirrors
|
||||
// BrotliSafeReadBits32 in bit_reader.h.
|
||||
pri func decoder.read_bits32?(src: base.io_reader, n: base.u32[..= 32]) {
|
||||
var n : base.u32[..= 32]
|
||||
var lo : base.u32
|
||||
|
||||
n = args.n
|
||||
if n <= 24 {
|
||||
this.read_bits?(src: args.src, n: n)
|
||||
return ok
|
||||
}
|
||||
this.read_bits?(src: args.src, n: 16)
|
||||
lo = this.bits_value & 0xFFFF
|
||||
this.read_bits?(src: args.src, n: n - 16)
|
||||
this.bits_value = lo | (this.bits_value ~mod<< 16)
|
||||
}
|
||||
|
||||
// read_symbol reads one Huffman-encoded symbol, decoded by the table at
|
||||
// this.huffs[args.table ..], and stores it in this.symbol. It reads input
|
||||
// bytes only when needed. Mirrors SafeReadSymbol and SafeDecodeSymbol in
|
||||
@@ -521,9 +693,11 @@ pri func decoder.jump_to_byte_boundary!() base.bool {
|
||||
return pad == 0
|
||||
}
|
||||
|
||||
// Mirrors DecodeWindowBits in decode.c.
|
||||
// Mirrors DecodeWindowBits in decode.c, and the BROTLI_STATE_UNINITED and
|
||||
// BROTLI_STATE_LARGE_WINDOW_BITS states of BrotliDecoderDecompressStream.
|
||||
pri func decoder.decode_window_bits?(src: base.io_reader) {
|
||||
var n : base.u32[..= 7]
|
||||
var w : base.u32[..= 63]
|
||||
|
||||
this.read_bits?(src: args.src, n: 1)
|
||||
if this.bits_value == 0 {
|
||||
@@ -537,12 +711,30 @@ pri func decoder.decode_window_bits?(src: base.io_reader) {
|
||||
this.read_bits?(src: args.src, n: 3)
|
||||
n = this.bits_value & 7
|
||||
if n == 1 {
|
||||
// The C code would read a "Large Window Brotli" header, if
|
||||
// BROTLI_DECODER_PARAM_LARGE_WINDOW was set, or otherwise fail
|
||||
// with BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS.
|
||||
//
|
||||
// TODO: support large-window brotli.
|
||||
return "#unsupported large window"
|
||||
// The "Large Window Brotli" escape. Without the
|
||||
// QUIRK_LARGE_WINDOW quirk (the C code's
|
||||
// BROTLI_DECODER_PARAM_LARGE_WINDOW), the C code fails with
|
||||
// BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS.
|
||||
if not this.quirk_large_window {
|
||||
return "#unsupported large window"
|
||||
}
|
||||
this.read_bits?(src: args.src, n: 1)
|
||||
if this.bits_value == 1 {
|
||||
return "#bad window bits"
|
||||
}
|
||||
// The C code is now in the BROTLI_STATE_LARGE_WINDOW_BITS
|
||||
// state.
|
||||
this.started = true
|
||||
this.large_window = true
|
||||
this.read_bits?(src: args.src, n: 6)
|
||||
w = this.bits_value & 63
|
||||
if (w < 10) or (30 < w) {
|
||||
return "#bad window bits"
|
||||
}
|
||||
this.window_bits = w
|
||||
this.window_size = (1 as base.u32) << w
|
||||
this.max_backward_distance = this.window_size - 16
|
||||
return ok
|
||||
} else if n <> 0 {
|
||||
this.window_bits = 8 + n
|
||||
} else {
|
||||
@@ -553,6 +745,8 @@ pri func decoder.decode_window_bits?(src: base.io_reader) {
|
||||
if this.window_bits < 10 {
|
||||
return "#bad window bits"
|
||||
}
|
||||
// The C code is now in the BROTLI_STATE_INITIALIZE state.
|
||||
this.started = true
|
||||
this.window_size = (1 as base.u32) << this.window_bits
|
||||
// Maximum distance, see section 9.1. of the spec. BROTLI_WINDOW_GAP is 16.
|
||||
this.max_backward_distance = this.window_size - 16
|
||||
|
||||
@@ -0,0 +1,955 @@
|
||||
// Copyright 2026 The Wuffs Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or
|
||||
// https://www.apache.org/licenses/LICENSE-2.0> or the MIT license
|
||||
// <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your
|
||||
// option. This file may not be copied, modified, or distributed
|
||||
// except according to those terms.
|
||||
//
|
||||
// SPDX-License-Identifier: Apache-2.0 OR MIT
|
||||
|
||||
// --------
|
||||
|
||||
// add_dictionary attaches a dictionary, like BrotliDecoderAttachDictionary in
|
||||
// decode.c. It must be called before decoding starts (before the first
|
||||
// transform_io call that consumes any input).
|
||||
//
|
||||
// args.dict_type is DICTIONARY_TYPE_RAW (a raw LZ77 prefix dictionary) or
|
||||
// DICTIONARY_TYPE_SERIALIZED (a serialized shared dictionary).
|
||||
//
|
||||
// Wuffs structs cannot hold pointers, so the dictionary's bytes are copied
|
||||
// into the workbuf: they are appended to the dictionaries added so far, which
|
||||
// live at the start of the workbuf. args.workbuf must therefore be at least
|
||||
// as long as the total length of all dictionaries added so far, including
|
||||
// this one, and every later add_dictionary and transform_io call must be
|
||||
// passed the same workbuf (or at least one whose prefix holds the same
|
||||
// bytes). After the last add_dictionary call, workbuf_len gives the workbuf
|
||||
// length that transform_io needs.
|
||||
//
|
||||
// On failure, the decoder's state is unchanged. The C code can leave its
|
||||
// state partially updated when BrotliDecoderAttachDictionary fails.
|
||||
pub func decoder.add_dictionary!(dict_type: base.u32, dict: roslice base.u8, workbuf: slice base.u8) base.status {
|
||||
var status : base.status
|
||||
var n : base.u64
|
||||
var lo : base.u64
|
||||
var hi : base.u64
|
||||
var i : base.u32
|
||||
|
||||
if this.started {
|
||||
return base."#bad call sequence"
|
||||
}
|
||||
n = args.dict.length()
|
||||
|
||||
if args.dict_type == DICTIONARY_TYPE_RAW {
|
||||
// Mirrors the BROTLI_SHARED_DICTIONARY_RAW case of
|
||||
// BrotliSharedDictionaryAttach in shared_dictionary.c and
|
||||
// AttachCompoundDictionary in decode.c.
|
||||
if this.num_chunks >= 15 {
|
||||
return "#too many compound dictionaries"
|
||||
}
|
||||
} else if args.dict_type == DICTIONARY_TYPE_SERIALIZED {
|
||||
// Mirrors DecodeSharedDictionary in shared_dictionary.c.
|
||||
status = this.parse_shared_dictionary!(dict: args.dict, base_offset: this.dict_total)
|
||||
if not status.is_ok() {
|
||||
return status
|
||||
}
|
||||
} else {
|
||||
return base."#bad argument"
|
||||
}
|
||||
|
||||
// Copy the dictionary bytes into the workbuf.
|
||||
if n > DICT_TOTAL_MAX {
|
||||
return "#too much dictionary data"
|
||||
}
|
||||
lo = this.dict_total
|
||||
hi = lo + n
|
||||
if hi > DICT_TOTAL_MAX {
|
||||
return "#too much dictionary data"
|
||||
} else if hi > args.workbuf.length() {
|
||||
return base."#bad workbuf length"
|
||||
}
|
||||
if lo > hi {
|
||||
return "#too much dictionary data"
|
||||
}
|
||||
args.workbuf[lo .. hi].copy_from_slice!(s: args.dict)
|
||||
|
||||
// Commit.
|
||||
if args.dict_type == DICTIONARY_TYPE_RAW {
|
||||
this.add_chunk!(start: lo, size: n)
|
||||
|
||||
} else {
|
||||
if this.sd_prefix_size > 0 {
|
||||
this.add_chunk!(start: lo ~sat+ this.sd_prefix_pos, size: this.sd_prefix_size)
|
||||
}
|
||||
// Mirrors the end of ParseDictionary in shared_dictionary.c. If the
|
||||
// serialized dictionary has no custom word lists or transform lists,
|
||||
// the static dictionary is reset to the built-in one, even if an
|
||||
// earlier serialized dictionary had custom lists.
|
||||
this.sd_num_word_lists = this.stage_num_word_lists
|
||||
this.sd_num_transform_lists = this.stage_num_transform_lists
|
||||
if (this.stage_num_word_lists > 0) or (this.stage_num_transform_lists > 0) {
|
||||
this.sd_context_based = this.stage_context_based
|
||||
this.sd_num_dictionaries = this.stage_num_dictionaries
|
||||
i = 0
|
||||
while i < 64 {
|
||||
this.sd_words[i] = this.stage_words[i].min(no_more_than: 64)
|
||||
this.sd_transforms[i] = this.stage_transforms[i].min(no_more_than: 64)
|
||||
this.sd_context_map[i] = this.stage_context_map[i] & 0x3F
|
||||
i += 1
|
||||
}
|
||||
} else {
|
||||
this.sd_context_based = false
|
||||
this.sd_num_dictionaries = 1
|
||||
this.sd_words[0] = 0
|
||||
this.sd_transforms[0] = 0
|
||||
}
|
||||
}
|
||||
|
||||
if hi <= DICT_TOTAL_MAX {
|
||||
this.dict_total = hi
|
||||
}
|
||||
return ok
|
||||
}
|
||||
|
||||
// add_chunk adds a chunk to the compound dictionary. Mirrors
|
||||
// AttachCompoundDictionary in decode.c.
|
||||
pri func decoder.add_chunk!(start: base.u64, size: base.u64) {
|
||||
var k : base.u32[..= 15]
|
||||
|
||||
k = this.num_chunks
|
||||
if k >= 15 {
|
||||
return nothing
|
||||
}
|
||||
this.chunk_starts[k] = (args.start & 0xFFFF_FFFF) as base.u32
|
||||
this.chunk_offsets[k + 1] = this.chunk_offsets[k] ~mod+ ((args.size & 0xFFFF_FFFF) as base.u32)
|
||||
this.num_chunks = k + 1
|
||||
}
|
||||
|
||||
// parse_shared_dictionary parses and validates a serialized shared
|
||||
// dictionary. Mirrors DecodeSharedDictionary, DryParseDictionary and
|
||||
// ParseDictionary in shared_dictionary.c.
|
||||
//
|
||||
// It does not change the decoder's sd_etc fields, only the stage_etc fields
|
||||
// (and the wl_etc and tl_etc fields, but only when there are no custom word
|
||||
// lists or transform lists in use). The caller commits them.
|
||||
//
|
||||
// args.base_offset is where the serialized dictionary's bytes will be copied
|
||||
// to, within the workbuf.
|
||||
pri func decoder.parse_shared_dictionary!(dict: roslice base.u8, base_offset: base.u64) base.status {
|
||||
var pos : base.u64
|
||||
var chunk_size : base.u64
|
||||
var prefix_pos : base.u64
|
||||
var num_prefix : base.u32[..= 1]
|
||||
var nwl : base.u32[..= 0xFF]
|
||||
var ntl : base.u32[..= 0xFF]
|
||||
var ndict : base.u32[..= 0xFF]
|
||||
var is_custom : base.bool
|
||||
var has_custom : base.bool
|
||||
var i : base.u32
|
||||
var v : base.u32[..= 0xFF]
|
||||
var v2 : base.u32[..= 0xFF]
|
||||
var status : base.status
|
||||
|
||||
// Check the magic header bytes.
|
||||
if args.dict.length() < 2 {
|
||||
return "#bad shared dictionary"
|
||||
} else if (args.dict[0] <> 0x91) or (args.dict[1] <> 0x00) {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
|
||||
// DryParseDictionary.
|
||||
pos = 2
|
||||
// LZ77_DICTIONARY_LENGTH. Mirrors ReadVarint32.
|
||||
chunk_size = this.read_varint32!(dict: args.dict, pos: pos)
|
||||
if chunk_size >= 0x1_0000_0000 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
pos = this.parse_pos
|
||||
prefix_pos = pos
|
||||
num_prefix = 0
|
||||
if chunk_size <> 0 {
|
||||
// "This limitation is not specified but the 32-bit Brotli decoder for
|
||||
// now".
|
||||
if chunk_size > 1073_741823 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
num_prefix = 1
|
||||
if (pos ~sat+ chunk_size) > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
pos ~sat+= chunk_size
|
||||
}
|
||||
// NUM_WORD_LISTS and NUM_TRANSFORM_LISTS.
|
||||
if (pos ~sat+ 2) > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
is_custom = false
|
||||
if pos < args.dict.length() {
|
||||
if args.dict[pos] > 0 {
|
||||
is_custom = true
|
||||
}
|
||||
}
|
||||
if (pos ~sat+ 1) < args.dict.length() {
|
||||
if args.dict[pos ~sat+ 1] > 0 {
|
||||
is_custom = true
|
||||
}
|
||||
}
|
||||
|
||||
if (num_prefix + this.num_chunks) > 15 {
|
||||
return "#too many compound dictionaries"
|
||||
}
|
||||
// "Cannot combine different static dictionaries, only prefix
|
||||
// dictionaries".
|
||||
has_custom = (this.sd_num_word_lists > 0) or (this.sd_num_transform_lists > 0)
|
||||
if has_custom and is_custom {
|
||||
return "#unsupported multiple custom static dictionaries"
|
||||
}
|
||||
|
||||
// ParseDictionary. The LZ77_DICTIONARY_LENGTH bytes were checked above.
|
||||
|
||||
// NUM_WORD_LISTS.
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
nwl = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if nwl > 64 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
i = 0
|
||||
while i < nwl,
|
||||
inv nwl <= 64,
|
||||
{
|
||||
assert i < 64 via "a < b: a < c; c <= b"(c: nwl)
|
||||
status = this.parse_word_list!(dict: args.dict, pos: pos, base_offset: args.base_offset, list: i)
|
||||
if not status.is_ok() {
|
||||
return status
|
||||
}
|
||||
pos = this.parse_pos
|
||||
i += 1
|
||||
}
|
||||
|
||||
// NUM_TRANSFORM_LISTS.
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
ntl = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if ntl > 64 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
i = 0
|
||||
while i < ntl,
|
||||
inv ntl <= 64,
|
||||
{
|
||||
assert i < 64 via "a < b: a < c; c <= b"(c: ntl)
|
||||
status = this.parse_transforms_list!(dict: args.dict, pos: pos, base_offset: args.base_offset, list: i)
|
||||
if not status.is_ok() {
|
||||
return status
|
||||
}
|
||||
pos = this.parse_pos
|
||||
i += 1
|
||||
}
|
||||
|
||||
this.stage_num_word_lists = nwl.min(no_more_than: 64)
|
||||
this.stage_num_transform_lists = ntl.min(no_more_than: 64)
|
||||
this.stage_num_dictionaries = 1
|
||||
this.stage_context_based = false
|
||||
i = 0
|
||||
while i < 64 {
|
||||
this.stage_words[i] = 0
|
||||
this.stage_transforms[i] = 0
|
||||
this.stage_context_map[i] = 0
|
||||
i += 1
|
||||
}
|
||||
|
||||
if (nwl <> 0) or (ntl <> 0) {
|
||||
// NUM_DICTIONARIES.
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
ndict = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if (ndict == 0) or (ndict > 64) {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.stage_num_dictionaries = ndict.min(no_more_than: 64)
|
||||
i = 0
|
||||
while i < ndict,
|
||||
inv ndict <= 64,
|
||||
{
|
||||
assert i < 64 via "a < b: a < c; c <= b"(c: ndict)
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
v = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if v > nwl {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
v2 = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if v2 > ntl {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
// Index nwl (or ntl) means the built-in list, which this.sd_words
|
||||
// (or this.sd_transforms) represents as 0.
|
||||
if v == nwl {
|
||||
this.stage_words[i] = 0
|
||||
} else {
|
||||
this.stage_words[i] = ((v & 0x3F) + 1) as base.u8
|
||||
}
|
||||
if v2 == ntl {
|
||||
this.stage_transforms[i] = 0
|
||||
} else {
|
||||
this.stage_transforms[i] = ((v2 & 0x3F) + 1) as base.u8
|
||||
}
|
||||
i += 1
|
||||
}
|
||||
|
||||
// CONTEXT_ENABLED. Mirrors ReadBool.
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
v = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if v > 1 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.stage_context_based = v == 1
|
||||
|
||||
// CONTEXT_MAP.
|
||||
if this.stage_context_based {
|
||||
i = 0
|
||||
while i < 64 {
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
v = args.dict[pos] as base.u32
|
||||
pos ~sat+= 1
|
||||
if v >= ndict {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.stage_context_map[i] = (v & 0x3F) as base.u8
|
||||
i += 1
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
this.sd_prefix_pos = prefix_pos
|
||||
this.sd_prefix_size = chunk_size
|
||||
return ok
|
||||
}
|
||||
|
||||
// read_varint32 reads a varint at args.dict[args.pos ..]. It returns a value
|
||||
// of 0x1_0000_0000 or more on failure. On success, it returns the value and
|
||||
// stores the new position in this.parse_pos. Mirrors ReadVarint32 in
|
||||
// shared_dictionary.c.
|
||||
pri func decoder.read_varint32!(dict: roslice base.u8, pos: base.u64) base.u64 {
|
||||
var pos : base.u64
|
||||
var num : base.u32[..= 4]
|
||||
var b : base.u64
|
||||
var result : base.u64
|
||||
|
||||
pos = args.pos
|
||||
result = 0
|
||||
num = 0
|
||||
while true {
|
||||
if pos >= args.dict.length() {
|
||||
return 0x1_0000_0000
|
||||
}
|
||||
b = args.dict[pos] as base.u64
|
||||
pos ~sat+= 1
|
||||
if (num == 4) and (b > 15) {
|
||||
return 0x1_0000_0000
|
||||
}
|
||||
result |= (b & 127) << (7 * num)
|
||||
if b < 128 {
|
||||
break
|
||||
}
|
||||
if num >= 4 {
|
||||
return 0x1_0000_0000
|
||||
}
|
||||
num += 1
|
||||
}
|
||||
this.parse_pos = pos
|
||||
return result & 0xFFFF_FFFF
|
||||
}
|
||||
|
||||
// parse_word_list parses the args.list'th custom word list, at
|
||||
// args.dict[args.pos ..]. On success, it stores the new position in
|
||||
// this.parse_pos. Mirrors ParseWordList and BrotliSizeBitsToOffsets in
|
||||
// shared_dictionary.c.
|
||||
pri func decoder.parse_word_list!(dict: roslice base.u8, pos: base.u64, base_offset: base.u64, list: base.u32[..= 63]) base.status {
|
||||
var pos : base.u64
|
||||
var len : base.u32
|
||||
var v : base.u8
|
||||
var offset : base.u32
|
||||
var wpos : base.u64
|
||||
|
||||
pos = args.pos
|
||||
if (pos ~sat+ 28) > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
len = 0
|
||||
while len < 32 {
|
||||
v = 0
|
||||
if len >= 4 {
|
||||
if (pos ~sat+ ((len - 4) as base.u64)) < args.dict.length() {
|
||||
v = args.dict[pos ~sat+ ((len - 4) as base.u64)]
|
||||
}
|
||||
// BROTLI_MAX_SIZE_BITS.
|
||||
if v > 15 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
}
|
||||
this.wl_size_bits[args.list][len] = v
|
||||
len += 1
|
||||
}
|
||||
pos ~sat+= 28
|
||||
|
||||
offset = 0
|
||||
len = 0
|
||||
while len < 32 {
|
||||
this.wl_offsets[args.list][len] = offset
|
||||
v = this.wl_size_bits[args.list][len]
|
||||
if v <> 0 {
|
||||
offset ~mod+= len << (v & 15)
|
||||
}
|
||||
len += 1
|
||||
}
|
||||
|
||||
wpos = args.base_offset ~sat+ pos
|
||||
this.wl_data[args.list] = (wpos & 0xFFFF_FFFF) as base.u32
|
||||
pos ~sat+= offset as base.u64
|
||||
if pos > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.parse_pos = pos
|
||||
return ok
|
||||
}
|
||||
|
||||
// parse_transforms_list parses the args.list'th custom transform list, at
|
||||
// args.dict[args.pos ..]. On success, it stores the new position in
|
||||
// this.parse_pos. Mirrors ParseTransformsList and ParsePrefixSuffixTable
|
||||
// in shared_dictionary.c.
|
||||
pri func decoder.parse_transforms_list!(dict: roslice base.u8, pos: base.u64, base_offset: base.u64, list: base.u32[..= 63]) base.status {
|
||||
var pos : base.u64
|
||||
var data_length : base.u64
|
||||
var ps_start : base.u64
|
||||
var offset : base.u64
|
||||
var stringlet_len : base.u64
|
||||
var count : base.u32[..= 256]
|
||||
var nt : base.u32[..= 0xFF]
|
||||
var t : base.u32
|
||||
var tpos : base.u64
|
||||
var j : base.u64
|
||||
var prefix_id : base.u32[..= 0xFF]
|
||||
var t_type : base.u32[..= 0xFF]
|
||||
var suffix_id : base.u32[..= 0xFF]
|
||||
var has_params : base.bool
|
||||
var p0 : base.u32
|
||||
var p1 : base.u32
|
||||
|
||||
pos = args.pos
|
||||
if pos >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
|
||||
// ParsePrefixSuffixTable. PREFIX_SUFFIX_LENGTH.
|
||||
if (pos ~sat+ 2) > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
data_length = (this.byte_at(s: args.dict, i: pos) as base.u64) |
|
||||
((this.byte_at(s: args.dict, i: pos ~sat+ 1) as base.u64) << 8)
|
||||
pos ~sat+= 2
|
||||
// It must at least have space for the NUL terminator.
|
||||
if data_length < 1 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
ps_start = pos
|
||||
if (pos ~sat+ data_length) >= args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
offset = 0
|
||||
count = 0
|
||||
while true {
|
||||
// STRING_LENGTH.
|
||||
j = ps_start ~sat+ offset
|
||||
stringlet_len = 0
|
||||
if j < args.dict.length() {
|
||||
stringlet_len = args.dict[j] as base.u64
|
||||
}
|
||||
if count >= 256 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
this.tl_map[args.list][count] = (offset & 0xFFFF) as base.u16
|
||||
count += 1
|
||||
offset ~sat+= 1
|
||||
if stringlet_len == 0 {
|
||||
if offset == data_length {
|
||||
break
|
||||
}
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
if count > 255 {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
offset ~sat+= stringlet_len
|
||||
if offset >= data_length {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
}
|
||||
pos ~sat+= data_length
|
||||
this.tl_prefix_suffix[args.list] = ((args.base_offset ~sat+ ps_start) & 0xFFFF_FFFF) as base.u32
|
||||
|
||||
// NUM_TRANSFORMS. The "(pos + data_length) >= length" check above means
|
||||
// that this is in bounds.
|
||||
nt = 0
|
||||
if pos < args.dict.length() {
|
||||
nt = args.dict[pos] as base.u32
|
||||
}
|
||||
pos ~sat+= 1
|
||||
this.tl_num_transforms[args.list] = nt
|
||||
tpos = pos
|
||||
this.tl_transforms[args.list] = ((args.base_offset ~sat+ tpos) & 0xFFFF_FFFF) as base.u32
|
||||
pos ~sat+= (nt as base.u64) * 3
|
||||
if pos > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
|
||||
// Check for errors and read extra parameters.
|
||||
has_params = false
|
||||
t = 0
|
||||
while t < nt {
|
||||
j = tpos ~sat+ ((t as base.u64) * 3)
|
||||
prefix_id = this.byte_at(s: args.dict, i: j)
|
||||
t_type = this.byte_at(s: args.dict, i: j ~sat+ 1)
|
||||
suffix_id = this.byte_at(s: args.dict, i: j ~sat+ 2)
|
||||
if prefix_id >= count {
|
||||
return "#bad shared dictionary"
|
||||
} else if t_type >= 23 {
|
||||
// BROTLI_NUM_TRANSFORM_TYPES.
|
||||
return "#bad shared dictionary"
|
||||
} else if suffix_id >= count {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
// BROTLI_TRANSFORM_SHIFT_FIRST or BROTLI_TRANSFORM_SHIFT_ALL.
|
||||
if (t_type == 21) or (t_type == 22) {
|
||||
has_params = true
|
||||
}
|
||||
t ~mod+= 1
|
||||
}
|
||||
|
||||
this.tl_params[args.list] = 0
|
||||
if has_params {
|
||||
this.tl_params[args.list] = ((args.base_offset ~sat+ pos) & 0xFFFF_FFFF) as base.u32
|
||||
j = pos
|
||||
pos ~sat+= (nt as base.u64) * 2
|
||||
if pos > args.dict.length() {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
t = 0
|
||||
while t < nt {
|
||||
t_type = this.byte_at(s: args.dict, i: tpos ~sat+ (((t as base.u64) * 3) + 1))
|
||||
if (t_type <> 21) and (t_type <> 22) {
|
||||
p0 = this.byte_at(s: args.dict, i: j ~sat+ ((t as base.u64) * 2))
|
||||
p1 = this.byte_at(s: args.dict, i: j ~sat+ (((t as base.u64) * 2) + 1))
|
||||
if (p0 <> 0) or (p1 <> 0) {
|
||||
return "#bad shared dictionary"
|
||||
}
|
||||
}
|
||||
t ~mod+= 1
|
||||
}
|
||||
}
|
||||
this.parse_pos = pos
|
||||
return ok
|
||||
}
|
||||
|
||||
// --------
|
||||
|
||||
// copy_dictionary_word writes a transformed static dictionary word to
|
||||
// this.transformed_word and returns its length. It returns 0xFFFF_FFFE for a
|
||||
// BROTLI_DECODER_ERROR_FORMAT_DICTIONARY error and 0xFFFF_FFFF for a
|
||||
// BROTLI_DECODER_ERROR_FORMAT_TRANSFORM error. args.address is the C code's
|
||||
// (distance_code - max_distance - 1 - compound_dictionary_size) and args.ctx
|
||||
// is the C code's BROTLI_CONTEXT(p1, p2, s->context_lookup).
|
||||
//
|
||||
// Mirrors the static dictionary part of ProcessCommandsInternal in decode.c,
|
||||
// including its selection of a word list and transform list from the shared
|
||||
// dictionary.
|
||||
pri func decoder.copy_dictionary_word!(dictbuf: roslice base.u8, address: base.u32, len: base.u32[..= 31], ctx: base.u32[..= 63]) base.u32 {
|
||||
var dict_id : base.u32[..= 63]
|
||||
var wsel : base.u32[..= 64]
|
||||
var tsel : base.u32[..= 64]
|
||||
var shift : base.u32[..= 24]
|
||||
var nt : base.u32[..= 255]
|
||||
var address : base.u32
|
||||
var transform_idx : base.u32
|
||||
var word_idx : base.u32
|
||||
var offset : base.u32
|
||||
var dist_remaining : base.u32
|
||||
var d2 : base.u32
|
||||
var w2 : base.u32[..= 64]
|
||||
var t2 : base.u32[..= 64]
|
||||
var shift2 : base.u32[..= 24]
|
||||
var num : base.u32
|
||||
|
||||
dict_id = 0
|
||||
if this.sd_context_based {
|
||||
dict_id = (this.sd_context_map[args.ctx] & 0x3F) as base.u32
|
||||
}
|
||||
wsel = this.sd_words[dict_id] as base.u32
|
||||
tsel = this.sd_transforms[dict_id] as base.u32
|
||||
shift = this.word_list_size_bits(wsel: wsel, len: args.len)
|
||||
nt = this.transform_list_length(tsel: tsel)
|
||||
address = args.address
|
||||
transform_idx = address >> shift
|
||||
|
||||
// If the distance is out of bound, select a next static dictionary if
|
||||
// there exist multiple.
|
||||
if ((transform_idx >= nt) or (shift == 0)) and (this.sd_num_dictionaries > 1) {
|
||||
dist_remaining = address ~mod- ((((1 as base.u32) << shift) & 0xFFFF_FFFE) ~mod* nt)
|
||||
d2 = 0
|
||||
while d2 < this.sd_num_dictionaries {
|
||||
assert d2 < 64 via "a < b: a < c; c <= b"(c: this.sd_num_dictionaries)
|
||||
w2 = this.sd_words[d2] as base.u32
|
||||
shift2 = this.word_list_size_bits(wsel: w2, len: args.len)
|
||||
if (d2 <> dict_id) and (shift2 <> 0) {
|
||||
t2 = this.sd_transforms[d2] as base.u32
|
||||
num = (((1 as base.u32) << shift2) & 0xFFFF_FFFE) ~mod* this.transform_list_length(tsel: t2)
|
||||
if dist_remaining < num {
|
||||
wsel = w2
|
||||
tsel = t2
|
||||
address = dist_remaining
|
||||
shift = shift2
|
||||
nt = this.transform_list_length(tsel: tsel)
|
||||
transform_idx = address >> shift
|
||||
break
|
||||
}
|
||||
dist_remaining -= num
|
||||
}
|
||||
d2 += 1
|
||||
}
|
||||
}
|
||||
|
||||
if shift == 0 {
|
||||
return 0xFFFF_FFFE
|
||||
} else if transform_idx >= nt {
|
||||
return 0xFFFF_FFFF
|
||||
}
|
||||
word_idx = address & (((1 as base.u32) << shift) - 1)
|
||||
offset = this.word_list_offset(wsel: wsel, len: args.len) ~mod+ (word_idx ~mod* args.len)
|
||||
num = this.transform_dictionary_word!(dictbuf: args.dictbuf,
|
||||
wsel: wsel,
|
||||
tsel: tsel,
|
||||
offset: offset,
|
||||
len: args.len,
|
||||
transform_idx: transform_idx.min(no_more_than: 254))
|
||||
return num
|
||||
}
|
||||
|
||||
// word_list_size_bits returns the C code's size_bits_by_length[args.len] for
|
||||
// the word list selected by args.wsel: 0 for the built-in list or (1 + j) for
|
||||
// the j'th custom list.
|
||||
pri func decoder.word_list_size_bits(wsel: base.u32[..= 64], len: base.u32[..= 31]) base.u32[..= 24] {
|
||||
if args.wsel == 0 {
|
||||
return DICTIONARY_SIZE_BITS_BY_LENGTH[args.len]
|
||||
}
|
||||
return (this.wl_size_bits[(args.wsel - 1) & 63][args.len] & 15) as base.u32
|
||||
}
|
||||
|
||||
// word_list_offset returns the C code's offsets_by_length[args.len] for the
|
||||
// word list selected by args.wsel.
|
||||
pri func decoder.word_list_offset(wsel: base.u32[..= 64], len: base.u32[..= 31]) base.u32 {
|
||||
if args.wsel == 0 {
|
||||
return DICTIONARY_OFFSETS_BY_LENGTH[args.len]
|
||||
}
|
||||
return this.wl_offsets[(args.wsel - 1) & 63][args.len]
|
||||
}
|
||||
|
||||
// transform_list_length returns the C code's num_transforms for the
|
||||
// transform list selected by args.tsel: 0 for the built-in list or (1 + j)
|
||||
// for the j'th custom list.
|
||||
pri func decoder.transform_list_length(tsel: base.u32[..= 64]) base.u32[..= 255] {
|
||||
if args.tsel == 0 {
|
||||
return NUM_TRANSFORMS
|
||||
}
|
||||
return this.tl_num_transforms[(args.tsel - 1) & 63].min(no_more_than: 255)
|
||||
}
|
||||
|
||||
// transform_dictionary_word writes the transformed word to
|
||||
// this.transformed_word and returns its length. Mirrors
|
||||
// BrotliTransformDictionaryWord in transform.c.
|
||||
pri func decoder.transform_dictionary_word!(dictbuf: roslice base.u8, wsel: base.u32[..= 64], tsel: base.u32[..= 64], offset: base.u32, len: base.u32[..= 31], transform_idx: base.u32[..= 254]) base.u32 {
|
||||
var idx : base.u32
|
||||
var t : base.u32[..= 120]
|
||||
var k : base.u32[..= 63]
|
||||
var tpos : base.u64
|
||||
var prefix_id : base.u32[..= 0xFF]
|
||||
var t_type : base.u32[..= 0xFF]
|
||||
var suffix_id : base.u32[..= 0xFF]
|
||||
var param : base.u32
|
||||
var word_start : base.u32
|
||||
var word_len : base.u32
|
||||
var skip : base.u32[..= 9]
|
||||
var i : base.u32
|
||||
var j : base.u64
|
||||
var remaining : base.u32
|
||||
var step : base.u32
|
||||
|
||||
// Look up the transform's prefix_id, type and suffix_id.
|
||||
k = 0
|
||||
tpos = 0
|
||||
if args.tsel == 0 {
|
||||
t = args.transform_idx.min(no_more_than: 120)
|
||||
prefix_id = TRANSFORM_PREFIX_ID[t] as base.u32
|
||||
t_type = TRANSFORM_TYPE[t] as base.u32
|
||||
suffix_id = TRANSFORM_SUFFIX_ID[t] as base.u32
|
||||
} else {
|
||||
k = (args.tsel - 1) & 63
|
||||
tpos = (this.tl_transforms[k] as base.u64) + ((args.transform_idx as base.u64) * 3)
|
||||
prefix_id = this.byte_at(s: args.dictbuf, i: tpos)
|
||||
t_type = this.byte_at(s: args.dictbuf, i: tpos + 1)
|
||||
suffix_id = this.byte_at(s: args.dictbuf, i: tpos + 2)
|
||||
}
|
||||
|
||||
// The prefix.
|
||||
idx = this.append_affix!(dictbuf: args.dictbuf, tsel: args.tsel, id: prefix_id, idx: 0)
|
||||
|
||||
// The word, possibly with its first or last few bytes omitted.
|
||||
word_len = args.len
|
||||
skip = 0
|
||||
if t_type <= 9 {
|
||||
// BROTLI_TRANSFORM_OMIT_LAST_N.
|
||||
if t_type < args.len {
|
||||
word_len = args.len ~mod- t_type
|
||||
} else {
|
||||
word_len = 0
|
||||
}
|
||||
} else if (12 <= t_type) and (t_type <= 20) {
|
||||
// BROTLI_TRANSFORM_OMIT_FIRST_N.
|
||||
skip = t_type - 11
|
||||
if skip < args.len {
|
||||
word_len = args.len ~mod- skip
|
||||
} else {
|
||||
word_len = 0
|
||||
}
|
||||
}
|
||||
word_start = idx
|
||||
i = 0
|
||||
while i < word_len {
|
||||
if idx >= 1024 {
|
||||
break
|
||||
}
|
||||
if args.wsel == 0 {
|
||||
j = ((args.offset ~mod+ skip) ~mod+ i) as base.u64
|
||||
if j < 122784 {
|
||||
this.transformed_word[idx] = DICTIONARY[j]
|
||||
}
|
||||
} else {
|
||||
j = (this.wl_data[(args.wsel - 1) & 63] as base.u64) +
|
||||
(((args.offset ~mod+ skip) ~mod+ i) as base.u64)
|
||||
if j < args.dictbuf.length() {
|
||||
this.transformed_word[idx] = args.dictbuf[j]
|
||||
}
|
||||
}
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
|
||||
// Upper-casing and shifting. The C code's ToUpperCase can modify bytes
|
||||
// after the word's end, but those bytes are either overwritten by the
|
||||
// suffix or are past the transformed word's length, so we don't bother.
|
||||
if t_type == 10 {
|
||||
// BROTLI_TRANSFORM_UPPERCASE_FIRST.
|
||||
this.to_upper_case!(p: word_start, end: idx)
|
||||
} else if t_type == 11 {
|
||||
// BROTLI_TRANSFORM_UPPERCASE_ALL.
|
||||
i = word_start
|
||||
while i < idx {
|
||||
step = this.to_upper_case!(p: i, end: idx)
|
||||
i ~mod+= step
|
||||
}
|
||||
} else if (t_type == 21) or (t_type == 22) {
|
||||
// BROTLI_TRANSFORM_SHIFT_FIRST or BROTLI_TRANSFORM_SHIFT_ALL. Only
|
||||
// custom transforms have these types, and their parameters exist.
|
||||
j = (this.tl_params[k] as base.u64) + ((args.transform_idx as base.u64) * 2)
|
||||
param = this.byte_at(s: args.dictbuf, i: j) |
|
||||
(this.byte_at(s: args.dictbuf, i: j + 1) << 8)
|
||||
if t_type == 21 {
|
||||
this.shift!(p: word_start, word_len: idx ~mod- word_start, param: param)
|
||||
} else {
|
||||
i = word_start
|
||||
remaining = idx ~mod- word_start
|
||||
while remaining > 0 {
|
||||
step = this.shift!(p: i, word_len: remaining, param: param)
|
||||
if step >= remaining {
|
||||
break
|
||||
}
|
||||
i ~mod+= step
|
||||
remaining ~mod-= step
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The suffix.
|
||||
idx = this.append_affix!(dictbuf: args.dictbuf, tsel: args.tsel, id: suffix_id, idx: idx)
|
||||
return idx
|
||||
}
|
||||
|
||||
// append_affix appends the args.id'th prefix or suffix of the transform list
|
||||
// selected by args.tsel to this.transformed_word, starting at args.idx. It
|
||||
// returns the new length.
|
||||
pri func decoder.append_affix!(dictbuf: roslice base.u8, tsel: base.u32[..= 64], id: base.u32[..= 0xFF], idx: base.u32) base.u32 {
|
||||
var idx : base.u32
|
||||
var affix : base.u32[..= 216]
|
||||
var affix_len : base.u32
|
||||
var i : base.u32
|
||||
var j : base.u32
|
||||
var k : base.u32[..= 63]
|
||||
var pos : base.u64
|
||||
|
||||
idx = args.idx
|
||||
if args.tsel == 0 {
|
||||
affix = PREFIX_SUFFIX_MAP[args.id.min(no_more_than: 49)]
|
||||
affix_len = PREFIX_SUFFIX[affix] as base.u32
|
||||
i = 0
|
||||
while i < affix_len {
|
||||
j = (affix ~mod+ 1) ~mod+ i
|
||||
if (j >= 217) or (idx >= 1024) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = PREFIX_SUFFIX[j]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
k = (args.tsel - 1) & 63
|
||||
pos = (this.tl_prefix_suffix[k] as base.u64) + (this.tl_map[k][args.id] as base.u64)
|
||||
affix_len = 0
|
||||
if pos < args.dictbuf.length() {
|
||||
affix_len = args.dictbuf[pos] as base.u32
|
||||
}
|
||||
i = 0
|
||||
while i < affix_len {
|
||||
pos ~mod+= 1
|
||||
if (pos >= args.dictbuf.length()) or (idx >= 1024) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = args.dictbuf[pos]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
return idx
|
||||
}
|
||||
|
||||
// shift applies a shift transform to the (UTF-8 encoded) character at
|
||||
// this.transformed_word[args.p ..], which is args.word_len bytes long. It
|
||||
// returns the step: the number of bytes in the character. Mirrors Shift in
|
||||
// transform.c.
|
||||
pri func decoder.shift!(p: base.u32, word_len: base.u32, param: base.u32) base.u32 {
|
||||
var p : base.u32[..= 1019]
|
||||
var scalar : base.u32
|
||||
var c0 : base.u32
|
||||
var c1 : base.u32
|
||||
var c2 : base.u32
|
||||
var c3 : base.u32
|
||||
|
||||
if args.p > 1019 {
|
||||
return 1
|
||||
}
|
||||
p = args.p
|
||||
// Limited sign extension: scalar < (1 << 24).
|
||||
scalar = (args.param & 0x7FFF) ~mod+ (0x100_0000 ~mod- (args.param & 0x8000))
|
||||
c0 = this.transformed_word[p] as base.u32
|
||||
if c0 < 0x80 {
|
||||
// 1-byte rune / 0sssssss / 7 bit scalar (ASCII).
|
||||
scalar ~mod+= c0
|
||||
this.transformed_word[p] = (scalar & 0x7F) as base.u8
|
||||
return 1
|
||||
} else if c0 < 0xC0 {
|
||||
// Continuation / 10AAAAAA.
|
||||
return 1
|
||||
} else if c0 < 0xE0 {
|
||||
// 2-byte rune / 110sssss AAssssss / 11 bit scalar.
|
||||
if args.word_len < 2 {
|
||||
return 1
|
||||
}
|
||||
c1 = this.transformed_word[p + 1] as base.u32
|
||||
scalar ~mod+= (c1 & 0x3F) | ((c0 & 0x1F) << 6)
|
||||
this.transformed_word[p + 0] = (0xC0 | ((scalar >> 6) & 0x1F)) as base.u8
|
||||
this.transformed_word[p + 1] = ((c1 & 0xC0) | (scalar & 0x3F)) as base.u8
|
||||
return 2
|
||||
} else if c0 < 0xF0 {
|
||||
// 3-byte rune / 1110ssss AAssssss BBssssss / 16 bit scalar.
|
||||
if args.word_len < 3 {
|
||||
return args.word_len
|
||||
}
|
||||
c1 = this.transformed_word[p + 1] as base.u32
|
||||
c2 = this.transformed_word[p + 2] as base.u32
|
||||
scalar ~mod+= (c2 & 0x3F) | ((c1 & 0x3F) << 6) | ((c0 & 0x0F) << 12)
|
||||
this.transformed_word[p + 0] = (0xE0 | ((scalar >> 12) & 0x0F)) as base.u8
|
||||
this.transformed_word[p + 1] = ((c1 & 0xC0) | ((scalar >> 6) & 0x3F)) as base.u8
|
||||
this.transformed_word[p + 2] = ((c2 & 0xC0) | (scalar & 0x3F)) as base.u8
|
||||
return 3
|
||||
} else if c0 < 0xF8 {
|
||||
// 4-byte rune / 11110sss AAssssss BBssssss CCssssss / 21 bit scalar.
|
||||
if args.word_len < 4 {
|
||||
return args.word_len
|
||||
}
|
||||
c1 = this.transformed_word[p + 1] as base.u32
|
||||
c2 = this.transformed_word[p + 2] as base.u32
|
||||
c3 = this.transformed_word[p + 3] as base.u32
|
||||
scalar ~mod+= (c3 & 0x3F) | ((c2 & 0x3F) << 6) | ((c1 & 0x3F) << 12) | ((c0 & 0x07) << 18)
|
||||
this.transformed_word[p + 0] = (0xF0 | ((scalar >> 18) & 0x07)) as base.u8
|
||||
this.transformed_word[p + 1] = ((c1 & 0xC0) | ((scalar >> 12) & 0x3F)) as base.u8
|
||||
this.transformed_word[p + 2] = ((c2 & 0xC0) | ((scalar >> 6) & 0x3F)) as base.u8
|
||||
this.transformed_word[p + 3] = ((c3 & 0xC0) | (scalar & 0x3F)) as base.u8
|
||||
return 4
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
// Mirrors ToUpperCase in transform.c, except that it does not modify bytes at
|
||||
// or after args.end. It returns the step: the number of bytes in the
|
||||
// (overly simplified model of a) UTF-8 encoded character.
|
||||
pri func decoder.to_upper_case!(p: base.u32, end: base.u32) base.u32 {
|
||||
var c : base.u8
|
||||
|
||||
if args.p >= 1024 {
|
||||
return 1
|
||||
}
|
||||
c = this.transformed_word[args.p]
|
||||
if c < 0xC0 {
|
||||
if ('a' <= c) and (c <= 'z') {
|
||||
this.transformed_word[args.p] = c ^ 32
|
||||
}
|
||||
return 1
|
||||
}
|
||||
// An overly simplified uppercasing model for UTF-8.
|
||||
if c < 0xE0 {
|
||||
if ((args.p + 1) < args.end) and ((args.p + 1) < 1024) {
|
||||
this.transformed_word[args.p + 1] ^= 32
|
||||
}
|
||||
return 2
|
||||
}
|
||||
// An arbitrary transform for three byte characters.
|
||||
if ((args.p + 2) < args.end) and ((args.p + 2) < 1024) {
|
||||
this.transformed_word[args.p + 2] ^= 5
|
||||
}
|
||||
return 3
|
||||
}
|
||||
|
||||
// byte_at returns args.s[args.i], or 0 if args.i is out of bounds.
|
||||
pri func decoder.byte_at(s: roslice base.u8, i: base.u64) base.u32[..= 0xFF] {
|
||||
if args.i < args.s.length() {
|
||||
return args.s[args.i] as base.u32
|
||||
}
|
||||
return 0
|
||||
}
|
||||
@@ -135,7 +135,7 @@ pri func decoder.read_simple_huffman_code?(src: base.io_reader, alphabet_size_ma
|
||||
// symbol, 1 means 2 symbols, 2 means 3 symbols, 3 means 4 symbols with
|
||||
// lengths [2, 2, 2, 2], 4 means 4 symbols with lengths [1, 2, 3, 3].
|
||||
pri func decoder.build_simple_huffman_table!(hoff: base.u32, num_symbols: base.u32[..= 4]) base.status {
|
||||
var t : base.u32[..= 671634]
|
||||
var t : base.u32[..= 677778]
|
||||
var v : array[4] base.u32[..= 0x3FF]
|
||||
var tmp : base.u32[..= 0x3FF]
|
||||
var i : base.u32
|
||||
|
||||
+156
-188
@@ -13,17 +13,18 @@
|
||||
// Mirrors the BROTLI_STATE_HUFFMAN_CODE_0 through
|
||||
// BROTLI_STATE_BEFORE_COMPRESSED_METABLOCK_BODY states of
|
||||
// BrotliDecoderDecompressStream in decode.c, followed by ProcessCommands.
|
||||
pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slice base.u8) {
|
||||
var t : base.u32
|
||||
var n : base.u32[..= 256]
|
||||
var n0 : base.u32[..= 256]
|
||||
var n2 : base.u32[..= 256]
|
||||
var bits : base.u32
|
||||
var npostfix : base.u32[..= 3]
|
||||
var ndirect : base.u32[..= 120]
|
||||
var alphabet : base.u32[..= 520]
|
||||
var i : base.u32
|
||||
var k : base.u32
|
||||
pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slice base.u8, dictbuf: roslice base.u8) {
|
||||
var t : base.u32
|
||||
var n : base.u32[..= 256]
|
||||
var n0 : base.u32[..= 256]
|
||||
var n2 : base.u32[..= 256]
|
||||
var bits : base.u32
|
||||
var npostfix : base.u32[..= 3]
|
||||
var ndirect : base.u32[..= 120]
|
||||
var alphabet_max : base.u32[..= 1128]
|
||||
var alphabet : base.u32[..= 544]
|
||||
var i : base.u32
|
||||
var k : base.u32
|
||||
|
||||
// BROTLI_STATE_HUFFMAN_CODE_0 ..= BROTLI_STATE_HUFFMAN_CODE_3.
|
||||
t = 0
|
||||
@@ -75,8 +76,19 @@ pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slic
|
||||
this.decode_context_map?(src: args.src, size: n2 << 2, offset: DIST_CONTEXT_MAP_BASE)
|
||||
this.num_dist_htrees = this.bits_value.min(no_more_than: 256)
|
||||
|
||||
// BROTLI_DISTANCE_ALPHABET_SIZE(npostfix, ndirect, BROTLI_MAX_DISTANCE_BITS).
|
||||
alphabet = 16 + ndirect + ((24 as base.u32) << (npostfix + 1))
|
||||
if this.large_window {
|
||||
// BROTLI_DISTANCE_ALPHABET_SIZE(npostfix, ndirect,
|
||||
// BROTLI_LARGE_MAX_DISTANCE_BITS) and the max_alphabet_size of
|
||||
// BrotliCalculateDistanceCodeLimit(BROTLI_MAX_ALLOWED_DISTANCE,
|
||||
// npostfix, ndirect).
|
||||
alphabet_max = 16 + ndirect + ((62 as base.u32) << (npostfix + 1))
|
||||
alphabet = this.calculate_distance_code_limit(npostfix: npostfix, ndirect: ndirect)
|
||||
} else {
|
||||
// BROTLI_DISTANCE_ALPHABET_SIZE(npostfix, ndirect,
|
||||
// BROTLI_MAX_DISTANCE_BITS).
|
||||
alphabet = 16 + ndirect + ((24 as base.u32) << (npostfix + 1))
|
||||
alphabet_max = alphabet
|
||||
}
|
||||
this.distance_alphabet_size = alphabet
|
||||
|
||||
// BROTLI_STATE_TREE_GROUP. Mirrors HuffmanTreeGroupDecode, three times.
|
||||
@@ -104,7 +116,7 @@ pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slic
|
||||
while k < this.num_dist_htrees {
|
||||
assert k < 256 via "a < b: a < c; c <= b"(c: this.num_dist_htrees)
|
||||
this.read_huffman_code?(src: args.src,
|
||||
alphabet_size_max: alphabet,
|
||||
alphabet_size_max: alphabet_max,
|
||||
alphabet_size_limit: alphabet,
|
||||
hoff: DIST_HUFFS_BASE + (k * DIST_HUFFS_STRIDE),
|
||||
table_size_max: DIST_HUFFS_STRIDE)
|
||||
@@ -117,7 +129,50 @@ pri func decoder.decode_compressed_metablock?(src: base.io_reader, workbuf: slic
|
||||
this.htree_command = 0
|
||||
this.calculate_distance_lut!()
|
||||
|
||||
this.process_commands?(src: args.src, workbuf: args.workbuf)
|
||||
this.process_commands?(src: args.src, workbuf: args.workbuf, dictbuf: args.dictbuf)
|
||||
}
|
||||
|
||||
// Mirrors BrotliCalculateDistanceCodeLimit in constants.h, with a
|
||||
// max_distance of BROTLI_MAX_ALLOWED_DISTANCE (0x7FFF_FFFC). It returns the
|
||||
// max_alphabet_size, which is at most 544.
|
||||
pri func decoder.calculate_distance_code_limit(npostfix: base.u32[..= 3], ndirect: base.u32[..= 120]) base.u32[..= 544] {
|
||||
var offset : base.u32
|
||||
var ndistbits : base.u32
|
||||
var tmp : base.u32
|
||||
var half : base.u32[..= 1]
|
||||
var group : base.u32
|
||||
var postfix : base.u32[..= 7]
|
||||
var ret : base.u32
|
||||
|
||||
// The max_distance <= ndirect case cannot happen.
|
||||
//
|
||||
// offset is (forbidden_distance - ndirect - 1), where forbidden_distance
|
||||
// is (max_distance + 1), after removing the postfix and "head-start".
|
||||
offset = ((0x7FFF_FFFC - args.ndirect) >> args.npostfix) + 4
|
||||
postfix = ((1 as base.u32) << args.npostfix) - 1
|
||||
// Calculate the number of distance bits (a poor man's log2floor).
|
||||
tmp = offset / 2
|
||||
ndistbits = 0
|
||||
while (tmp <> 0) and (ndistbits < 32) {
|
||||
ndistbits += 1
|
||||
tmp >>= 1
|
||||
}
|
||||
// One bit is covered with subrange addressing ("half").
|
||||
ndistbits ~mod-= 1
|
||||
// Find the subrange.
|
||||
half = (offset >> (ndistbits & 31)) & 1
|
||||
// Calculate the "group" part of the dcode.
|
||||
group = ((ndistbits ~mod- 1) ~mod<< 1) | half
|
||||
if group == 0 {
|
||||
return args.ndirect + 16
|
||||
}
|
||||
// Decrement "group", so it is the last permitted "group".
|
||||
group -= 1
|
||||
ret = (((group ~mod<< args.npostfix) | postfix) ~mod+ args.ndirect) ~mod+ 17
|
||||
if ret > 544 {
|
||||
return 544
|
||||
}
|
||||
return ret
|
||||
}
|
||||
|
||||
// Mirrors DecodeContextMap in decode.c. The context map is written to
|
||||
@@ -310,7 +365,7 @@ pri func decoder.prepare_literal_decoding!() {
|
||||
pri func decoder.calculate_distance_lut!() {
|
||||
var npostfix : base.u32[..= 3]
|
||||
var ndirect : base.u32[..= 120]
|
||||
var limit : base.u32[..= 520]
|
||||
var limit : base.u32[..= 544]
|
||||
var postfix : base.u32[..= 8]
|
||||
var i : base.u32
|
||||
var j : base.u32
|
||||
@@ -361,7 +416,7 @@ pri func decoder.calculate_distance_lut!() {
|
||||
}
|
||||
|
||||
// Mirrors ProcessCommandsInternal in decode.c.
|
||||
pri func decoder.process_commands?(src: base.io_reader, workbuf: slice base.u8) {
|
||||
pri func decoder.process_commands?(src: base.io_reader, workbuf: slice base.u8, dictbuf: roslice base.u8) {
|
||||
var cmd_code : base.u32[..= 703]
|
||||
var insert_length : base.u32
|
||||
var p1 : base.u32[..= 0xFF]
|
||||
@@ -375,6 +430,10 @@ pri func decoder.process_commands?(src: base.io_reader, workbuf: slice base.u8)
|
||||
var n : base.u32
|
||||
var k : base.u32
|
||||
var b : base.u8
|
||||
var compound_size : base.u32
|
||||
var address : base.u32
|
||||
var chunk : base.u32[..= 14]
|
||||
var j : base.u64
|
||||
|
||||
while true {
|
||||
// CommandBegin.
|
||||
@@ -460,39 +519,84 @@ pri func decoder.process_commands?(src: base.io_reader, workbuf: slice base.u8)
|
||||
distance = this.distance_code
|
||||
|
||||
if distance > max_distance {
|
||||
// Apply a static dictionary reference, since the distance is
|
||||
// larger than the max LZ77 distance.
|
||||
// Apply a compound dictionary or static dictionary reference,
|
||||
// since the distance is larger than the max LZ77 distance.
|
||||
//
|
||||
// The maximum allowed distance is BROTLI_MAX_ALLOWED_DISTANCE.
|
||||
if distance > 0x7FFF_FFFC {
|
||||
return "#bad distance"
|
||||
}
|
||||
// The C code checks for SHARED_BROTLI_MIN_DICTIONARY_WORD_LENGTH
|
||||
// (4) and SHARED_BROTLI_MAX_DICTIONARY_WORD_LENGTH (31) but the
|
||||
// RFC 7932 dictionary's size_bits_by_length is zero for lengths
|
||||
// 25 ..= 31, which also fails with "#bad dictionary".
|
||||
//
|
||||
// TODO: support custom dictionaries.
|
||||
if (i < 4) or (24 < i) {
|
||||
return "#bad dictionary"
|
||||
}
|
||||
this.dist_rb_idx ~mod+= this.distance_context
|
||||
n = this.copy_dictionary_word!(
|
||||
address: (distance ~mod- max_distance) ~mod- 1,
|
||||
len: i.min(no_more_than: 24))
|
||||
if n == 0xFFFF_FFFF {
|
||||
return "#bad transform"
|
||||
} else if (n == 0) and (distance <= 120) {
|
||||
return "#bad transform"
|
||||
}
|
||||
this.mb_remaining ~mod-= n
|
||||
k = 0
|
||||
while k < n {
|
||||
this.put_byte!(workbuf: args.workbuf, b: this.transformed_word[k & 63])
|
||||
k ~mod+= 1
|
||||
if this.pos == this.rb_size {
|
||||
this.ring_event?()
|
||||
compound_size = this.chunk_offsets[this.num_chunks]
|
||||
address = (distance - max_distance) - 1
|
||||
|
||||
if address < compound_size {
|
||||
// Mirrors InitializeCompoundDictionaryCopy and
|
||||
// CopyFromCompoundDictionary in decode.c. This address is the
|
||||
// C code's (compound_dictionary_size - (s->distance_code -
|
||||
// s->max_distance)).
|
||||
address = compound_size ~mod- (address + 1)
|
||||
if ((address as base.u64) + (i as base.u64)) > (compound_size as base.u64) {
|
||||
return "#bad compound dictionary"
|
||||
}
|
||||
// Update the recent distances cache.
|
||||
this.dist_rb[this.dist_rb_idx & 3] = distance
|
||||
this.dist_rb_idx ~mod+= 1
|
||||
this.mb_remaining ~mod-= i
|
||||
chunk = 0
|
||||
while i > 0 {
|
||||
while chunk < 14 {
|
||||
if address < this.chunk_offsets[chunk + 1] {
|
||||
break
|
||||
}
|
||||
chunk += 1
|
||||
}
|
||||
j = (this.chunk_starts[chunk] as base.u64) +
|
||||
((address ~mod- this.chunk_offsets[chunk]) as base.u64)
|
||||
b = 0
|
||||
if j < args.dictbuf.length() {
|
||||
b = args.dictbuf[j]
|
||||
}
|
||||
this.put_byte!(workbuf: args.workbuf, b: b)
|
||||
address ~mod+= 1
|
||||
i ~mod-= 1
|
||||
if this.pos == this.rb_size {
|
||||
this.ring_event?()
|
||||
}
|
||||
}
|
||||
|
||||
} else if (4 <= i) and (i <= 31) {
|
||||
// SHARED_BROTLI_MIN_DICTIONARY_WORD_LENGTH and
|
||||
// SHARED_BROTLI_MAX_DICTIONARY_WORD_LENGTH.
|
||||
p1 = this.get_byte(workbuf: args.workbuf, distance: 1) as base.u32
|
||||
p2 = this.get_byte(workbuf: args.workbuf, distance: 2) as base.u32
|
||||
mode_offset = this.context_mode << 9
|
||||
ctx = (CONTEXT_LOOKUP[mode_offset | p1] |
|
||||
CONTEXT_LOOKUP[mode_offset | 0x100 | p2]) as base.u32
|
||||
// Compensate double distance-ring-buffer roll.
|
||||
this.dist_rb_idx ~mod+= this.distance_context
|
||||
n = this.copy_dictionary_word!(dictbuf: args.dictbuf,
|
||||
address: address ~mod- compound_size,
|
||||
len: i.min(no_more_than: 31),
|
||||
ctx: ctx)
|
||||
if n == 0xFFFF_FFFE {
|
||||
return "#bad dictionary"
|
||||
} else if n == 0xFFFF_FFFF {
|
||||
return "#bad transform"
|
||||
} else if (n == 0) and (distance <= 120) {
|
||||
return "#bad transform"
|
||||
}
|
||||
this.mb_remaining ~mod-= n
|
||||
k = 0
|
||||
while k < n {
|
||||
this.put_byte!(workbuf: args.workbuf, b: this.transformed_word[k & 1023])
|
||||
k ~mod+= 1
|
||||
if this.pos == this.rb_size {
|
||||
this.ring_event?()
|
||||
}
|
||||
}
|
||||
|
||||
} else {
|
||||
return "#bad dictionary"
|
||||
}
|
||||
|
||||
} else {
|
||||
@@ -543,12 +647,17 @@ pri func decoder.read_distance?(src: base.io_reader) {
|
||||
if (code >= this.distance_alphabet_size) or (code >= 1128) {
|
||||
return "#internal error: inconsistent Huffman table"
|
||||
}
|
||||
this.read_bits?(src: args.src, n: this.dist_extra_bits[code].min(no_more_than: 24) as base.u32)
|
||||
// The C code uses BrotliSafeReadBits32, as large-window brotli can have
|
||||
// more than 24 extra bits.
|
||||
this.read_bits32?(src: args.src, n: this.dist_extra_bits[code].min(no_more_than: 32) as base.u32)
|
||||
if code >= 1128 {
|
||||
return "#internal error: inconsistent Huffman table"
|
||||
}
|
||||
this.distance_code = this.dist_offset[code] ~mod+
|
||||
(this.bits_value ~mod<< this.distance_postfix_bits)
|
||||
// The C code calculates this with brotli_reg_t (64-bit) arithmetic and
|
||||
// then casts to int. BrotliCalculateDistanceCodeLimit means that the
|
||||
// result is less than BROTLI_MAX_ALLOWED_DISTANCE.
|
||||
this.distance_code = (((this.dist_offset[code] as base.u64) ~mod+
|
||||
((this.bits_value as base.u64) ~mod<< this.distance_postfix_bits)) & 0xFFFF_FFFF) as base.u32
|
||||
}
|
||||
|
||||
// Mirrors TakeDistanceFromRingBuffer in decode.c.
|
||||
@@ -585,144 +694,3 @@ pri func decoder.take_distance_from_ring_buffer!(code: base.u32[..= 15]) {
|
||||
this.distance_code = v - 3
|
||||
}
|
||||
}
|
||||
|
||||
// copy_dictionary_word writes a transformed static dictionary word to
|
||||
// this.transformed_word and returns its length. It returns 0xFFFF_FFFF if the
|
||||
// transform index is out of range. Mirrors part of ProcessCommandsInternal in
|
||||
// decode.c and BrotliTransformDictionaryWord in transform.c.
|
||||
pri func decoder.copy_dictionary_word!(address: base.u32, len: base.u32[..= 24]) base.u32 {
|
||||
var shift : base.u32[..= 24]
|
||||
var word_idx : base.u32
|
||||
var transform_idx : base.u32
|
||||
var offset : base.u32
|
||||
var t : base.u32[..= 120]
|
||||
var t_type : base.u32[..= 22]
|
||||
var affix : base.u32[..= 216]
|
||||
var affix_len : base.u32
|
||||
var idx : base.u32
|
||||
var word_start : base.u32
|
||||
var word_len : base.u32
|
||||
var skip : base.u32[..= 9]
|
||||
var i : base.u32
|
||||
var j : base.u32
|
||||
var step : base.u32
|
||||
|
||||
shift = DICTIONARY_SIZE_BITS_BY_LENGTH[args.len]
|
||||
word_idx = args.address & (((1 as base.u32) << shift) - 1)
|
||||
transform_idx = args.address >> shift
|
||||
if transform_idx >= NUM_TRANSFORMS {
|
||||
return 0xFFFF_FFFF
|
||||
}
|
||||
t = transform_idx.min(no_more_than: 120)
|
||||
offset = DICTIONARY_OFFSETS_BY_LENGTH[args.len] ~mod+ (word_idx ~mod* args.len)
|
||||
|
||||
// The prefix.
|
||||
idx = 0
|
||||
affix = PREFIX_SUFFIX_MAP[TRANSFORM_PREFIX_ID[t]]
|
||||
affix_len = PREFIX_SUFFIX[affix] as base.u32
|
||||
i = 0
|
||||
while i < affix_len {
|
||||
j = (affix ~mod+ 1) ~mod+ i
|
||||
if (j >= 217) or (idx >= 64) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = PREFIX_SUFFIX[j]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
|
||||
// The word, possibly with its first or last few bytes omitted.
|
||||
t_type = TRANSFORM_TYPE[t] as base.u32
|
||||
word_len = args.len
|
||||
skip = 0
|
||||
if t_type <= 9 {
|
||||
// BROTLI_TRANSFORM_OMIT_LAST_N.
|
||||
if t_type < args.len {
|
||||
word_len = args.len ~mod- t_type
|
||||
} else {
|
||||
word_len = 0
|
||||
}
|
||||
} else if (12 <= t_type) and (t_type <= 20) {
|
||||
// BROTLI_TRANSFORM_OMIT_FIRST_N.
|
||||
skip = t_type - 11
|
||||
if skip < args.len {
|
||||
word_len = args.len ~mod- skip
|
||||
} else {
|
||||
word_len = 0
|
||||
}
|
||||
}
|
||||
word_start = idx
|
||||
i = 0
|
||||
while i < word_len {
|
||||
j = (offset ~mod+ skip) ~mod+ i
|
||||
if (j >= 122784) or (idx >= 64) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = DICTIONARY[j]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
|
||||
// Upper-casing. The C code's ToUpperCase can modify bytes after the
|
||||
// word's end, but those bytes are either overwritten by the suffix or are
|
||||
// past the transformed word's length, so we don't bother.
|
||||
if t_type == 10 {
|
||||
// BROTLI_TRANSFORM_UPPERCASE_FIRST.
|
||||
this.to_upper_case!(p: word_start, end: idx)
|
||||
} else if t_type == 11 {
|
||||
// BROTLI_TRANSFORM_UPPERCASE_ALL.
|
||||
i = word_start
|
||||
while i < idx {
|
||||
step = this.to_upper_case!(p: i, end: idx)
|
||||
i ~mod+= step
|
||||
}
|
||||
}
|
||||
// TODO: support BROTLI_TRANSFORM_SHIFT_FIRST and _SHIFT_ALL (used only by
|
||||
// custom dictionaries' transforms).
|
||||
|
||||
// The suffix.
|
||||
affix = PREFIX_SUFFIX_MAP[TRANSFORM_SUFFIX_ID[t]]
|
||||
affix_len = PREFIX_SUFFIX[affix] as base.u32
|
||||
i = 0
|
||||
while i < affix_len {
|
||||
j = (affix ~mod+ 1) ~mod+ i
|
||||
if (j >= 217) or (idx >= 64) {
|
||||
break
|
||||
}
|
||||
this.transformed_word[idx] = PREFIX_SUFFIX[j]
|
||||
idx ~mod+= 1
|
||||
i ~mod+= 1
|
||||
}
|
||||
|
||||
return idx
|
||||
}
|
||||
|
||||
// Mirrors ToUpperCase in transform.c, except that it does not modify bytes at
|
||||
// or after args.end. It returns the step: the number of bytes in the
|
||||
// (overly simplified model of a) UTF-8 encoded character.
|
||||
pri func decoder.to_upper_case!(p: base.u32, end: base.u32) base.u32 {
|
||||
var c : base.u8
|
||||
|
||||
if args.p >= 64 {
|
||||
return 1
|
||||
}
|
||||
c = this.transformed_word[args.p]
|
||||
if c < 0xC0 {
|
||||
if ('a' <= c) and (c <= 'z') {
|
||||
this.transformed_word[args.p] = c ^ 32
|
||||
}
|
||||
return 1
|
||||
}
|
||||
// An overly simplified uppercasing model for UTF-8.
|
||||
if c < 0xE0 {
|
||||
if ((args.p + 1) < args.end) and ((args.p + 1) < 64) {
|
||||
this.transformed_word[args.p + 1] ^= 32
|
||||
}
|
||||
return 2
|
||||
}
|
||||
// An arbitrary transform for three byte characters.
|
||||
if ((args.p + 2) < args.end) and ((args.p + 2) < 64) {
|
||||
this.transformed_word[args.p + 2] ^= 5
|
||||
}
|
||||
return 3
|
||||
}
|
||||
Reference in new issue
Block a user