Adds a brotli encoder (std/brotli's encoder struct, a base.io_transformer) whose output is byte-identical to libbrotlienc v1.2.1 for quality levels 0 and 1, mirroring encode.c's BrotliEncoderCompressStream(Fast), compress_fragment.c and compress_fragment_two_pass.c, plus the entropy coding and bit stream helpers they need. Every double operation goes through std/softfloat, following the C evaluation order exactly. Parameters are quirks (encode_quirks.wuffs) that mirror BrotliEncoderSetParameter, sanitized at the first transform_io call. Each transform_io resumption corresponds to one C call with available_in equal to the readable src length; a closed src is FINISH; request_flush! and request_emit_metadata! make the next resumption a FLUSH or EMIT_METADATA call. Qualities 2 to 11 return "#unsupported encoder quality" for now. Encoder tables are generated by script/print-brotli-tables.go -encoder. Tests: round trips through the Wuffs decoder over the corpus and synthetic inputs; with -mimic, byte-exact comparisons against libbrotlienc over quality × window × mode × input chunking × flush schedule × metadata × size hint. script/manual-test-brotli-encoder-parity.c runs the full grid: 86,268 configurations, zero mismatches. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
Brotli
Brotli is a general purpose compression format, combining LZ77 style length-distance back-references, Huffman coding, second order context modeling and a built-in static dictionary. It is specified by RFC 7932. The reference implementation is at github.com/google/brotli.
This package (std/brotli) is a decoder. It aims for exact parity with the
reference implementation's decoder (libbrotlidec), version v1.2.1, and in
particular its c/dec/decode.c file: the same output for valid input and the
same verdict (and the same error code, mapped as below) for invalid input. The
Wuffs code mirrors the C code's logic and validation checks, function by
function, and comments like "Mirrors ReadHuffmanCode in decode.c" name the
corresponding C function. The C code's explicit state machine (its
BROTLI_STATE_ETC enums) is replaced by Wuffs coroutines.
Format Overview
A brotli stream starts with a 1, 4 or 7 bit WBITS header, giving the window
size (the maximum back-reference distance, plus 16). It is followed by a
sequence of metablocks. Each metablock header gives its uncompressed length
(MLEN) and whether it is the last metablock. A metablock is one of:
- a metadata block, whose payload is skipped (it does not produce output),
- an uncompressed block, whose payload is byte-aligned and copied verbatim, or
- a compressed block.
A compressed metablock header defines, for each of three block categories
(literals, insert-and-copy commands and distances), the number of block types
and the prefix codes for block type switches and block lengths. It then gives
the distance parameters (NPOSTFIX and NDIRECT), the literal context modes,
the literal and distance context maps (run-length and move-to-front encoded)
and finally the groups of prefix (Huffman) codes. Each prefix code is either a
"simple" code (up to 4 symbols) or a "complex" code (code lengths that are
themselves prefix coded).
The metablock body is a sequence of commands. Each command inserts zero or more literal bytes (decoded with a prefix code chosen by the context map, from the previous two bytes and the current literal block type) and then copies bytes, either from earlier in the output (an LZ77 back-reference, possibly re-using one of the last four distances) or, for distances beyond the maximum possible back-reference, a transformed word from the static dictionary.
The final metablock is followed by zero padding up to a byte boundary. The decoder does not read any further than that.
Workbuf Layout
workbuf_len is (1 << 24) + 542 bytes, both minimum and maximum: the
largest (non-large-window) brotli window size plus some slack. The slack
mirrors the C code's kRingBufferWriteAheadSlack, which its fast paths use to
write (and discard) a few bytes past the ring buffer's end. This
implementation does not yet have such fast paths, but it still reserves the
slack, so that adding them later will not change the workbuf_len API. A
workbuf shorter than the stream's window size plus 542 bytes is rejected with
base."#bad workbuf length".
The first (1 << WBITS) bytes of the workbuf are the history ring buffer.
Decoded bytes are written to the ring buffer and then copied to the dst
io_writer. Before the ring buffer wraps around, all of its bytes must have
been copied to dst, so transform_io can suspend with $short write.
The C code's ring buffer (s->ringbuffer_size) can start smaller than the
window size (its "canny ring buffer allocation") and grows over successive
metablocks. The Wuffs ring buffer is always (1 << WBITS) bytes long, but the
decoder still tracks the C code's (virtual) ring buffer size, as it decides
when the C code calls WriteRingBuffer. That in turn decides whether some
invalid input (an overrun metablock) is reported as BLOCK_LENGTH_1 or
BLOCK_LENGTH_2, and when the C code's max_distance becomes fixed.
Everything else (the Huffman tables, context maps, etc.) lives in the decoder struct, not the workbuf. The Huffman tables use the same 8-bit root table and second level table layout as the C code, sized per tree as the C code allocates them.
Status Mapping
Every BROTLI_DECODER_ERROR_FORMAT_* error code (and running out of input) has
a corresponding Wuffs status:
| libbrotlidec error code | Wuffs status |
|---|---|
BROTLI_DECODER_ERROR_FORMAT_EXUBERANT_NIBBLE |
"#bad exuberant nibble" |
BROTLI_DECODER_ERROR_FORMAT_RESERVED |
"#bad reserved bit" |
BROTLI_DECODER_ERROR_FORMAT_EXUBERANT_META_NIBBLE |
"#bad exuberant metadata nibble" |
BROTLI_DECODER_ERROR_FORMAT_SIMPLE_HUFFMAN_ALPHABET |
"#bad simple Huffman alphabet" |
BROTLI_DECODER_ERROR_FORMAT_SIMPLE_HUFFMAN_SAME |
"#bad simple Huffman symbol" |
BROTLI_DECODER_ERROR_FORMAT_CL_SPACE |
"#bad code length code lengths" |
BROTLI_DECODER_ERROR_FORMAT_HUFFMAN_SPACE |
"#bad Huffman space" |
BROTLI_DECODER_ERROR_FORMAT_CONTEXT_MAP_REPEAT |
"#bad context map repeat" |
BROTLI_DECODER_ERROR_FORMAT_BLOCK_LENGTH_1 |
"#bad block length" |
BROTLI_DECODER_ERROR_FORMAT_BLOCK_LENGTH_2 |
"#bad metablock length" |
BROTLI_DECODER_ERROR_FORMAT_TRANSFORM |
"#bad transform" |
BROTLI_DECODER_ERROR_FORMAT_DICTIONARY |
"#bad dictionary" |
BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS |
"#bad window bits" or "#unsupported large window" |
BROTLI_DECODER_ERROR_FORMAT_PADDING_1 |
"#bad padding" |
BROTLI_DECODER_ERROR_FORMAT_PADDING_2 |
"#bad final padding" |
BROTLI_DECODER_ERROR_FORMAT_DISTANCE |
"#bad distance" |
BROTLI_DECODER_ERROR_FORMAT_BLOCK_SWITCH |
"#bad block switch" |
BROTLI_DECODER_NEEDS_MORE_INPUT (with closed src) |
"#truncated input" |
Notes:
"#unsupported large window"is for the large-window escapeWBITSvalue (0010001). libbrotlidec (withBROTLI_DECODER_PARAM_LARGE_WINDOWnot set, the default) rejects it withBROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS. For standard (non-large-window) brotli, no otherWBITSvalue is invalid, so"#bad window bits"is not currently produced.- Like the C code, which error an overrun metablock produces (
"#bad block length"or"#bad metablock length") can depend on how the input is split acrosstransform_iocalls. When the C code runs out of input, it first tries to flush its output (WriteRingBuffer) and that checks for an overrun metablock. A Wuffs$short readsuspension corresponds to the C code's running out of input. "#bad block switch"needs more than(1 << 24)commands in one metablock. It is implemented but not tested.- The remaining libbrotlidec error codes do not apply:
COMPOUND_DICTIONARYandDICTIONARY_NOT_SET(custom dictionaries are not supported),INVALID_ARGUMENTS, theALLOC_*codes (this decoder does not allocate) andUNREACHABLE.
The "#internal error: etc" statuses should never happen. They are
consistency checks, generally where the C code relies on (unchecked)
invariants for memory safety but Wuffs requires explicit bounds checks.
Differences from the C Code
- The bit reader reads input bytes only when it needs them (like the C code's
"safe"
BrotliSafeReadBitsandSafeReadSymbolfunctions), so thattransform_iocan suspend at any point. There is no "fast path" (like the C code'sProcessCommands, as opposed toSafeProcessCommands). Both paths produce the same output and errors. - LZ77 copies and dictionary words are written to the ring buffer one byte at
a time, with ring buffer events (the C code's
WriteRingBufferandWrapRingBuffercalls) at the same positions as the C code. - On error, the Wuffs decoder may have already written more output to
dstthan libbrotlidec would have (libbrotlidec does not flush on error), but what both have written agrees.
TODO
- Large-window brotli (
BROTLI_DECODER_PARAM_LARGE_WINDOW, window bits up to 30). The window-related fields are sized for 24 bits. - Custom, shared and compound dictionaries.
- Metadata callbacks (metadata blocks are currently skipped).
- Fast paths.
- Fuzzers.
- An encoder.
Encoder
This package also has an encoder, which implements the base.io_transformer
interface. It aims for byte-exact parity with the reference implementation's
encoder (libbrotlienc), version v1.2.1: for the same parameters and the same
sequence of operations, it produces exactly the same bytes. Like the decoder,
the Wuffs code mirrors the C code function by function, with comments like
"Mirrors BrotliCompressFragmentFast in compress_fragment.c". Its floating
point arithmetic (the C code's doubles) uses std/softfloat, so that the
results are bit-exact regardless of the compiler and CPU.
Only qualities 0 and 1 (the C code's FAST_ONE_PASS_COMPRESSION_QUALITY and
FAST_TWO_PASS_COMPRESSION_QUALITY, implemented by compress_fragment.c and
compress_fragment_two_pass.c) are implemented. For qualities 2 to 11
(including the default quality, 11), transform_io returns
"#unsupported encoder quality".
The source files are:
encode_brotli.wuffs: theencoderstruct, its public methods and theencode.cstreaming logic (BrotliEncoderCompressStream,BrotliEncoderCompressStreamFast,ProcessMetadata,EnsureInitialized,InjectBytePaddingBlock, etc).encode_quirks.wuffs: the encoder's quirk keys.encode_bit_writer.wuffs:write_bits.h,fast_log.h,find_match_length.hand smallbrotli_bit_stream.chelpers.encode_entropy.wuffs:entropy_encode.c(Huffman tree construction) and thebrotli_bit_stream.cfunctions that store Huffman codes.encode_fragment_fast.wuffs:compress_fragment.c(quality 0).encode_fragment_two_pass.wuffs:compress_fragment_two_pass.c(quality 1).common_encoder_tables.wuffs: tables generated bygo run script/print-brotli-tables.go -encoderfrom the C code.
Encoder API
In addition to the base.io_transformer methods, the encoder has:
request_flush!() status: the nexttransform_iocall acts like the C code'sBROTLI_OPERATION_FLUSH(instead ofPROCESSorFINISH) until the flush is complete. It returns"#bad call sequence"while a metadata request is pending.request_emit_metadata!(length: u64) status: the nexttransform_iocall acts likeBROTLI_OPERATION_EMIT_METADATA, with the nextlengthbytes ofsrcas the metadata block's contents.lengthcan be at most1 << 24(otherwise"#bad argument"). It returns"#bad call sequence"while another request is pending.max_compressed_size(input_size: u64) u64: mirrorsBrotliEncoderMaxCompressedSize(0 means overflow).
dst_history_retain_length is 0: the encoder never re-reads dst.
Encoder Quirks
Each quirk mirrors one BrotliEncoderParameter, with the value semantics of
BrotliEncoderSetParameter. Values are stored as is (and must fit in a u32,
otherwise set_quirk returns "#bad argument"). Like the C code, they are
sanitized (SanitizeParams, ComputeLgBlock, ChooseDistanceParams) at the
first transform_io call, after which set_quirk returns
"#bad call sequence".
| Quirk | C parameter |
|---|---|
QUIRK_ENCODER_QUALITY |
BROTLI_PARAM_QUALITY (default 11) |
QUIRK_LGWIN |
BROTLI_PARAM_LGWIN (default 22) |
QUIRK_LGBLOCK |
BROTLI_PARAM_LGBLOCK |
QUIRK_MODE |
BROTLI_PARAM_MODE |
QUIRK_DISABLE_LITERAL_CONTEXT_MODELING |
BROTLI_PARAM_DISABLE_LITERAL_CONTEXT_MODELING (0 or 1) |
QUIRK_SIZE_HINT |
BROTLI_PARAM_SIZE_HINT |
QUIRK_LARGE_WINDOW (shared with the decoder) |
BROTLI_PARAM_LARGE_WINDOW |
QUIRK_NPOSTFIX |
BROTLI_PARAM_NPOSTFIX |
QUIRK_NDIRECT |
BROTLI_PARAM_NDIRECT |
QUIRK_STREAM_OFFSET |
BROTLI_PARAM_STREAM_OFFSET (at most 1 << 30) |
The quality quirk is not named QUIRK_QUALITY because that name would clash
with base.QUIRK_QUALITY. At qualities 0 and 1, only the quality, lgwin and
stream offset affect the output (lgblock, mode, size hint, large window,
npostfix, ndirect and literal context modeling are stored, sanitized and then
ignored, as in the C code).
Encoder Workbuf Layout
At qualities 0 and 1, the workbuf is the C code's metablock storage (its
GetBrotliStorage buffer): workbuf_len is (2 << lgwin) + 503 + 16 bytes
(the storage for a 1 << lgwin byte block plus 16 bytes of slack for 8-byte
bit writes), where lgwin is the sanitized value. A shorter workbuf still works
if every input block is shorter: a block of n bytes needs (2 * n) + 519
bytes, otherwise transform_io returns "#bad workbuf length". At other
qualities, workbuf_len is 0.
The hash table (up to 1 << 17 entries), the quality 1 command and literal
buffers (1 << 17 entries each) and the Huffman scratch space are arrays in
the encoder struct (which is therefore more than 1 MiB). They do not need
zero-initializing: WUFFS_INITIALIZE__LEAVE_INTERNAL_BUFFERS_UNINITIALIZED is
fine. Unlike the C code, the input block is not copied: the encoder compresses
directly from src.
Encoder Operation Mapping
Each transform_io call (initial or resumed) corresponds to one or more
BrotliEncoderCompressStream calls, with src's readable bytes as the C
code's available_in and dst's writable bytes as available_out. The
operation is:
EMIT_METADATAifrequest_emit_metadatais pending (until the metadata block is complete),- otherwise
FLUSHifrequest_flushis pending (until the flush is complete), - otherwise
FINISHifsrcis closed, - otherwise
PROCESS.
transform_io suspends with "$short write" when there is pending output
(the C code's BrotliEncoderHasMoreOutput) but dst is full, suspends with
"$short read" when all of src is consumed (and, for a flush, the flush is
complete) and returns ok once the stream is finished (the C code's
BrotliEncoderIsFinished). So these two loops produce the same bytes:
- C: for each chunk, call
BrotliEncoderCompressStream(op)until all of the chunk is consumed and there is no more output (or, forFINISH, until finished). - Wuffs: for each chunk, call
request_flush(forFLUSH) orrequest_emit_metadata(len)(forEMIT_METADATA), makesrchold exactly that chunk (closed forFINISH) and calltransform_iountil it returns"$short read"(or ok forFINISH), giving it moredstroom after each"$short write".
At qualities 0 and 1 the C code compresses each call's available input
immediately (in blocks of at most 1 << lgwin bytes), so the output depends
on how the input is split into chunks. The test programs' encode drivers
(wuffs_brotli_encode_plan in test/c/std/brotli.c and
mimic_brotli_encode in test/c/mimiclib/brotli.c) run the same "plan" of
(operation, length) steps with each library.
Encoder Testing
wuffs test -mimic std/brotli checks byte-exact parity with libbrotlienc
over a grid of qualities, lgwins, modes, input chunkings, flushes, metadata
blocks, size hints and other parameters, as well as round trips through the
decoder. script/manual-test-brotli-encoder-parity.c runs a larger grid
(about 86 thousand configurations over every file in test/data/brotli and
some synthetic inputs).
Encoder Differences from the C Code
- The input block is not copied (see the workbuf layout above), and output is
always staged in the workbuf (or a small internal buffer) and then copied to
dst. The C code sometimes writes directly tonext_out. The bytes are the same. - Wuffs' C code generator does not support
slice base.u16orslice base.u32arguments (and mishandlescopy_from_sliceonbase.u16arrays), so some functions (e.g.create_huffman_tree,convert_bit_depths_to_symbols) take a selector naming one of the encoder's arrays instead of a slice. - Like the C code, there is no fallback to an uncompressed stream (that is
only in the C code's one-shot
BrotliEncoderCompress).
Encoder TODO
- Qualities 2 to 11 (the C code's
EncodeData, hashers, block splitting, context modeling, Zopfli, etc). - Large-window encoding (only used at qualities 3 and above).
- Custom and shared dictionaries (
BrotliEncoderAttachPreparedDictionary). - Changing parameters mid-stream.