Completes the decoder's feature set relative to libbrotlidec v1.2.1: - QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the large-window WBITS escape, window bits up to 30, the 544-symbol distance alphabet and 64-bit distance arithmetic, with workbuf_len growing to (1 << 30) + 542 when the quirk is set. - decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw (LZ77 prefix) and serialized shared dictionaries: compound-dictionary distance mapping, the shared-dictionary parser (custom word lists, transform lists including the SHIFT transforms, context-dependent dictionary selection, prefix dictionaries) and all of its validation. Wuffs structs cannot hold slices, so the dictionary bytes are copied into the front of the caller's workbuf; the layout is documented in the README. Serialized dictionaries are accepted as in a BROTLI_EXPERIMENTAL build of libbrotli. - fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and random I/O limits. Tests: new golden vectors for large windows (including a 17 MiB far-distance stream generated in the test), raw and serialized dictionaries, nine more malformed inputs, and mimic comparisons that attach the same dictionaries to libbrotlidec. Differential runs over 2.7 million mutated large-window and dictionary streams found no divergence from libbrotlidec. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
Brotli
Brotli is a general purpose compression format, combining LZ77 style length-distance back-references, Huffman coding, second order context modeling and a built-in static dictionary. It is specified by RFC 7932. The reference implementation is at github.com/google/brotli.
This package (std/brotli) is a decoder. It aims for exact parity with the
reference implementation's decoder (libbrotlidec), version v1.2.1, and in
particular its c/dec/decode.c file: the same output for valid input and the
same verdict (and the same error code, mapped as below) for invalid input. The
Wuffs code mirrors the C code's logic and validation checks, function by
function, and comments like "Mirrors ReadHuffmanCode in decode.c" name the
corresponding C function. The C code's explicit state machine (its
BROTLI_STATE_ETC enums) is replaced by Wuffs coroutines.
Format Overview
A brotli stream starts with a 1, 4 or 7 bit WBITS header, giving the window
size (the maximum back-reference distance, plus 16). It is followed by a
sequence of metablocks. Each metablock header gives its uncompressed length
(MLEN) and whether it is the last metablock. A metablock is one of:
- a metadata block, whose payload is skipped (it does not produce output),
- an uncompressed block, whose payload is byte-aligned and copied verbatim, or
- a compressed block.
A compressed metablock header defines, for each of three block categories
(literals, insert-and-copy commands and distances), the number of block types
and the prefix codes for block type switches and block lengths. It then gives
the distance parameters (NPOSTFIX and NDIRECT), the literal context modes,
the literal and distance context maps (run-length and move-to-front encoded)
and finally the groups of prefix (Huffman) codes. Each prefix code is either a
"simple" code (up to 4 symbols) or a "complex" code (code lengths that are
themselves prefix coded).
The metablock body is a sequence of commands. Each command inserts zero or more literal bytes (decoded with a prefix code chosen by the context map, from the previous two bytes and the current literal block type) and then copies bytes, either from earlier in the output (an LZ77 back-reference, possibly re-using one of the last four distances) or, for distances beyond the maximum possible back-reference, a transformed word from the static dictionary.
The final metablock is followed by zero padding up to a byte boundary. The decoder does not read any further than that.
Workbuf Layout
Without the large-window quirk and without attached dictionaries (see the
"Large Window" and "Dictionaries" sections below), workbuf_len is `(1 << 24)
- 542
bytes, both minimum and maximum: the largest (non-large-window) brotli window size plus some slack. The slack mirrors the C code'skRingBufferWriteAheadSlack, which its fast paths use to write (and discard) a few bytes past the ring buffer's end. This implementation does not yet have such fast paths, but it still reserves the slack, so that adding them later will not change theworkbuf_lenAPI. A workbuf shorter than the stream's window size plus 542 bytes is rejected withbase."#bad workbuf length"`.
The first (1 << WBITS) bytes of the workbuf are the history ring buffer.
Decoded bytes are written to the ring buffer and then copied to the dst
io_writer. Before the ring buffer wraps around, all of its bytes must have
been copied to dst, so transform_io can suspend with $short write.
The C code's ring buffer (s->ringbuffer_size) can start smaller than the
window size (its "canny ring buffer allocation") and grows over successive
metablocks. The Wuffs ring buffer is always (1 << WBITS) bytes long, but the
decoder still tracks the C code's (virtual) ring buffer size, as it decides
when the C code calls WriteRingBuffer. That in turn decides whether some
invalid input (an overrun metablock) is reported as BLOCK_LENGTH_1 or
BLOCK_LENGTH_2, and when the C code's max_distance becomes fixed.
Everything else (the Huffman tables, context maps, etc.) lives in the decoder struct, not the workbuf. The Huffman tables use the same 8-bit root table and second level table layout as the C code, sized per tree as the C code allocates them.
Status Mapping
Every BROTLI_DECODER_ERROR_FORMAT_* error code (and running out of input) has
a corresponding Wuffs status:
| libbrotlidec error code | Wuffs status |
|---|---|
BROTLI_DECODER_ERROR_FORMAT_EXUBERANT_NIBBLE |
"#bad exuberant nibble" |
BROTLI_DECODER_ERROR_FORMAT_RESERVED |
"#bad reserved bit" |
BROTLI_DECODER_ERROR_FORMAT_EXUBERANT_META_NIBBLE |
"#bad exuberant metadata nibble" |
BROTLI_DECODER_ERROR_FORMAT_SIMPLE_HUFFMAN_ALPHABET |
"#bad simple Huffman alphabet" |
BROTLI_DECODER_ERROR_FORMAT_SIMPLE_HUFFMAN_SAME |
"#bad simple Huffman symbol" |
BROTLI_DECODER_ERROR_FORMAT_CL_SPACE |
"#bad code length code lengths" |
BROTLI_DECODER_ERROR_FORMAT_HUFFMAN_SPACE |
"#bad Huffman space" |
BROTLI_DECODER_ERROR_FORMAT_CONTEXT_MAP_REPEAT |
"#bad context map repeat" |
BROTLI_DECODER_ERROR_FORMAT_BLOCK_LENGTH_1 |
"#bad block length" |
BROTLI_DECODER_ERROR_FORMAT_BLOCK_LENGTH_2 |
"#bad metablock length" |
BROTLI_DECODER_ERROR_FORMAT_TRANSFORM |
"#bad transform" |
BROTLI_DECODER_ERROR_FORMAT_DICTIONARY |
"#bad dictionary" |
BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS |
"#bad window bits" or "#unsupported large window" |
BROTLI_DECODER_ERROR_FORMAT_PADDING_1 |
"#bad padding" |
BROTLI_DECODER_ERROR_FORMAT_PADDING_2 |
"#bad final padding" |
BROTLI_DECODER_ERROR_FORMAT_DISTANCE |
"#bad distance" |
BROTLI_DECODER_ERROR_FORMAT_BLOCK_SWITCH |
"#bad block switch" |
BROTLI_DECODER_ERROR_COMPOUND_DICTIONARY |
"#bad compound dictionary" |
BROTLI_DECODER_NEEDS_MORE_INPUT (with closed src) |
"#truncated input" |
Notes:
"#unsupported large window"is for the large-window escapeWBITSvalue (0010001) when theQUIRK_LARGE_WINDOWquirk is not set. libbrotlidec (withBROTLI_DECODER_PARAM_LARGE_WINDOWnot set, the default) rejects it withBROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS. For standard (non-large-window) brotli, no otherWBITSvalue is invalid. With the quirk set,"#bad window bits"is for a large-window header with a non-zero reserved bit or with window bits outside of[10 ..= 30].- Like the C code, which error an overrun metablock produces (
"#bad block length"or"#bad metablock length") can depend on how the input is split acrosstransform_iocalls. When the C code runs out of input, it first tries to flush its output (WriteRingBuffer) and that checks for an overrun metablock. A Wuffs$short readsuspension corresponds to the C code's running out of input. "#bad block switch"needs more than(1 << 24)commands in one metablock. It is implemented but not tested.- The remaining libbrotlidec error codes do not apply:
DICTIONARY_NOT_SET(it needs a word list without data, which libbrotlidec only has when built withBROTLI_EXTERNAL_DICTIONARY_DATAand not given that data),INVALID_ARGUMENTS, theALLOC_*codes (this decoder does not allocate) andUNREACHABLE.
The "#internal error: etc" statuses should never happen. They are
consistency checks, generally where the C code relies on (unchecked)
invariants for memory safety but Wuffs requires explicit bounds checks.
Differences from the C Code
- The bit reader reads input bytes only when it needs them (like the C code's
"safe"
BrotliSafeReadBitsandSafeReadSymbolfunctions), so thattransform_iocan suspend at any point. There is no "fast path" (like the C code'sProcessCommands, as opposed toSafeProcessCommands). Both paths produce the same output and errors. - LZ77 copies and dictionary words are written to the ring buffer one byte at
a time, with ring buffer events (the C code's
WriteRingBufferandWrapRingBuffercalls) at the same positions as the C code. - On error, the Wuffs decoder may have already written more output to
dstthan libbrotlidec would have (libbrotlidec does not flush on error), but what both have written agrees.
TODO
- Metadata callbacks (metadata blocks are currently skipped). See the "Metadata Blocks" section below.
- Fast paths.
- An encoder.
Large Window
Large-window brotli (an incompatible extension of RFC 7932, with window sizes
up to (1 << 30) bytes) is enabled by the QUIRK_LARGE_WINDOW quirk. It
mirrors libbrotlidec's BROTLI_DECODER_PARAM_LARGE_WINDOW parameter: it must
be set before decoding starts (afterwards, set_quirk returns "#bad call sequence", like BrotliDecoderSetParameter failing once the decoder has
left its BROTLI_STATE_UNINITED state). With the quirk set:
- a stream that starts with the large-window
WBITSescape is decoded as a large-window stream: the escape is followed by a reserved bit (that must be zero) and 6 window bits (that must be in[10 ..= 30]), otherwise the decoder fails with"#bad window bits". - a stream that does not start with that escape is decoded as standard brotli, exactly as without the quirk.
workbuf_lenis(1 << 30) + 542bytes (plus any dictionaries), the largest large-window window size plus slack. As before, the workbuf only needs to be as long as the stream's actual window size plus 542 bytes (plus any dictionaries). Without the quirk, it is(1 << 24) + 542bytes (plus any dictionaries), as before.
For a large-window stream, like the C code, the distance alphabet's
alphabet_size_max uses BROTLI_LARGE_MAX_DISTANCE_BITS (62) and its
alphabet_size_limit comes from BrotliCalculateDistanceCodeLimit (at most
544 symbols, so that no distance exceeds BROTLI_MAX_ALLOWED_DISTANCE). Up to
29 extra distance bits are read (the C code's BrotliSafeReadBits32) and the
distance is calculated with 64-bit arithmetic, like the C code's
brotli_reg_t. The Huffman table stride for distance trees is sized for 544
symbols in both modes.
The DECODER_WORKBUF_LEN_MAX_INCL_WORST_CASE constant is now the overall
worst case: (1 << 30) + 542 plus the maximum total dictionary length
(0x7FFF_FFFF). The new DECODER_WORKBUF_LEN_STANDARD and
DECODER_WORKBUF_LEN_LARGE_WINDOW constants give the window-only values.
Dictionaries
decoder.add_dictionary!(dict_type: base.u32, dict: roslice base.u8, workbuf: slice base.u8) base.status mirrors libbrotlidec's
BrotliDecoderAttachDictionary. dict_type is DICTIONARY_TYPE_RAW (0, the
C code's BROTLI_SHARED_DICTIONARY_RAW) or DICTIONARY_TYPE_SERIALIZED (1,
BROTLI_SHARED_DICTIONARY_SERIALIZED). It must be called before decoding
starts. It can be called multiple times.
Wuffs structs cannot hold pointers or slices, so add_dictionary copies the
dictionary bytes into the workbuf. Every add_dictionary and transform_io
call must be passed the same workbuf (or at least one whose first bytes hold
the same dictionary bytes). The add_dictionary workbuf only needs to be long
enough to hold all of the dictionaries added so far. Call workbuf_len after
the last add_dictionary call: it grows by each dictionary's length.
The workbuf layout is:
[0 .. D) the attached dictionaries, concatenated, where D is
the total length of all successfully added
dictionaries (raw or serialized, in full).
[D .. D + W + 542) the history ring buffer (W = 1 << WBITS) and slack.
The dictionaries come first because their position should not depend on the stream's window size, which is only known once decoding has started.
Raw dictionaries are LZ77 prefix dictionaries. Like the C code (its
BrotliDecoderCompoundDictionary and AttachCompoundDictionary,
InitializeCompoundDictionaryCopy and CopyFromCompoundDictionary
functions), at most 15 such "compound dictionary" chunks (raw dictionaries
plus serialized dictionaries' LZ77 prefixes) can be attached. A backward
reference distance d beyond the current maximum LZ77 distance m (but
within BROTLI_MAX_ALLOWED_DISTANCE) refers to the compound dictionary when
d - m - 1 is less than the compound dictionary's total size T, at address
T - (d - m). A copy that would extend past the compound dictionary's end
fails with "#bad compound dictionary". Otherwise, the static dictionary's
address is d - m - 1 - T.
Serialized dictionaries are parsed and validated like
BrotliSharedDictionaryAttach (DecodeSharedDictionary,
DryParseDictionary, ParseDictionary, ParseWordList,
ParseTransformsList and ParsePrefixSuffixTable in
c/common/shared_dictionary.c): the 0x91 0x00 magic, an optional LZ77
prefix (at most 1073741823 bytes), up to 64 custom word lists (word lengths 4
to 31, at most 15 size bits per length), up to 64 custom transform lists (up
to 256 prefix/suffix stringlets, up to 255 transforms, transform types up to
BROTLI_TRANSFORM_SHIFT_ALL, with 16-bit parameters for the
BROTLI_TRANSFORM_SHIFT_FIRST and BROTLI_TRANSFORM_SHIFT_ALL types), up to
64 (word list, transform list) combinations and an optional 64-entry context
map. The parsed metadata lives in fixed-size decoder fields (wl_etc, tl_etc
and sd_etc), with offsets into the copied bytes in the workbuf. Like the C
code:
- only one attached serialized dictionary can have custom word lists or
transform lists (otherwise
"#unsupported multiple custom static dictionaries"), and - attaching a serialized dictionary without custom word lists or transform lists (only an LZ77 prefix) resets the static dictionary to the built-in (RFC 7932) one, even if an earlier serialized dictionary had custom lists.
Static dictionary references then mirror ProcessCommandsInternal: the word
list and transform list are selected by the literal context (BROTLI_CONTEXT(p1, p2, context_lookup)) through the context map (if enabled), with the C code's
fall back to the other (word list, transform list) combinations when the
address is out of range, and words are transformed like
BrotliTransformDictionaryWord (including Shift for the shift transforms).
Transformed words can be up to 541 bytes long.
add_dictionary statuses (the C code just returns BROTLI_FALSE):
| Condition | Wuffs status |
|---|---|
| decoding has started | base."#bad call sequence" |
unknown dict_type |
base."#bad argument" |
| more than 15 compound dictionary chunks | "#too many compound dictionaries" |
| invalid serialized dictionary | "#bad shared dictionary" |
| a second serialized dictionary with custom lists | "#unsupported multiple custom static dictionaries" |
more than 0x7FFF_FFFF bytes of dictionaries in total |
"#too much dictionary data" |
workbuf shorter than the dictionaries so far plus this one |
base."#bad workbuf length" |
Differences from the C code:
- On failure,
add_dictionaryleaves the decoder unchanged. libbrotlidec'sBrotliDecoderAttachDictionarycan fail after partially updating its shared dictionary (e.g. after counting a serialized dictionary's LZ77 prefix or reading its number of word lists), which can affect later attach calls. - libbrotlicommon only parses serialized dictionaries when built with
BROTLI_EXPERIMENTAL, which is not the default. A default build rejects every serialized dictionary. This package always accepts them, matching an experimental build. The mimic tests skip serialized dictionary cases when the linked libbrotlidec rejects them; they pass, with no skips, when linked against aBROTLI_EXPERIMENTALbuild. - The C code holds the compound dictionary's total size in an
int. This package rejects more than0x7FFF_FFFFbytes of dictionaries in total (counting serialized dictionaries in full), where the C code would overflow. BROTLI_DECODER_ERROR_DICTIONARY_NOT_SETcannot happen (see above).
Metadata Blocks
libbrotlidec can report metadata blocks' contents through callbacks
(BrotliDecoderSetMetadataCallbacks). Wuffs code cannot call back into its
caller, so there is no equivalent: metadata blocks are skipped (validated and
consumed but not reported), exactly as libbrotlidec does without callbacks,
so the decoded output is the same. A future quirk could expose metadata
blocks, e.g. by suspending with a status when one starts.