mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-22 00:47:28 +01:00
Adds a brotli encoder (std/brotli's encoder struct, a base.io_transformer) whose output is byte-identical to libbrotlienc v1.2.1 for quality levels 0 and 1, mirroring encode.c's BrotliEncoderCompressStream(Fast), compress_fragment.c and compress_fragment_two_pass.c, plus the entropy coding and bit stream helpers they need. Every double operation goes through std/softfloat, following the C evaluation order exactly. Parameters are quirks (encode_quirks.wuffs) that mirror BrotliEncoderSetParameter, sanitized at the first transform_io call. Each transform_io resumption corresponds to one C call with available_in equal to the readable src length; a closed src is FINISH; request_flush! and request_emit_metadata! make the next resumption a FLUSH or EMIT_METADATA call. Qualities 2 to 11 return "#unsupported encoder quality" for now. Encoder tables are generated by script/print-brotli-tables.go -encoder. Tests: round trips through the Wuffs decoder over the corpus and synthetic inputs; with -mimic, byte-exact comparisons against libbrotlienc over quality × window × mode × input chunking × flush schedule × metadata × size hint. script/manual-test-brotli-encoder-parity.c runs the full grid: 86,268 configurations, zero mismatches. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
327 lines
17 KiB
Markdown
327 lines
17 KiB
Markdown
# Brotli
|
|
|
|
Brotli is a general purpose compression format, combining LZ77 style
|
|
length-distance back-references, Huffman coding, second order context modeling
|
|
and a built-in static dictionary. It is specified by [RFC
|
|
7932](https://www.rfc-editor.org/rfc/rfc7932). The reference implementation is
|
|
at [github.com/google/brotli](https://github.com/google/brotli).
|
|
|
|
This package (`std/brotli`) is a decoder. It aims for exact parity with the
|
|
reference implementation's decoder (libbrotlidec), version v1.2.1, and in
|
|
particular its `c/dec/decode.c` file: the same output for valid input and the
|
|
same verdict (and the same error code, mapped as below) for invalid input. The
|
|
Wuffs code mirrors the C code's logic and validation checks, function by
|
|
function, and comments like "Mirrors ReadHuffmanCode in decode.c" name the
|
|
corresponding C function. The C code's explicit state machine (its
|
|
`BROTLI_STATE_ETC` enums) is replaced by Wuffs coroutines.
|
|
|
|
|
|
## Format Overview
|
|
|
|
A brotli stream starts with a 1, 4 or 7 bit `WBITS` header, giving the window
|
|
size (the maximum back-reference distance, plus 16). It is followed by a
|
|
sequence of metablocks. Each metablock header gives its uncompressed length
|
|
(`MLEN`) and whether it is the last metablock. A metablock is one of:
|
|
|
|
- a metadata block, whose payload is skipped (it does not produce output),
|
|
- an uncompressed block, whose payload is byte-aligned and copied verbatim, or
|
|
- a compressed block.
|
|
|
|
A compressed metablock header defines, for each of three block categories
|
|
(literals, insert-and-copy commands and distances), the number of block types
|
|
and the prefix codes for block type switches and block lengths. It then gives
|
|
the distance parameters (`NPOSTFIX` and `NDIRECT`), the literal context modes,
|
|
the literal and distance context maps (run-length and move-to-front encoded)
|
|
and finally the groups of prefix (Huffman) codes. Each prefix code is either a
|
|
"simple" code (up to 4 symbols) or a "complex" code (code lengths that are
|
|
themselves prefix coded).
|
|
|
|
The metablock body is a sequence of commands. Each command inserts zero or more
|
|
literal bytes (decoded with a prefix code chosen by the context map, from the
|
|
previous two bytes and the current literal block type) and then copies bytes,
|
|
either from earlier in the output (an LZ77 back-reference, possibly re-using
|
|
one of the last four distances) or, for distances beyond the maximum possible
|
|
back-reference, a transformed word from the static dictionary.
|
|
|
|
The final metablock is followed by zero padding up to a byte boundary. The
|
|
decoder does not read any further than that.
|
|
|
|
|
|
## Workbuf Layout
|
|
|
|
`workbuf_len` is `(1 << 24) + 542` bytes, both minimum and maximum: the
|
|
largest (non-large-window) brotli window size plus some slack. The slack
|
|
mirrors the C code's `kRingBufferWriteAheadSlack`, which its fast paths use to
|
|
write (and discard) a few bytes past the ring buffer's end. This
|
|
implementation does not yet have such fast paths, but it still reserves the
|
|
slack, so that adding them later will not change the `workbuf_len` API. A
|
|
workbuf shorter than the stream's window size plus 542 bytes is rejected with
|
|
`base."#bad workbuf length"`.
|
|
|
|
The first `(1 << WBITS)` bytes of the workbuf are the history ring buffer.
|
|
Decoded bytes are written to the ring buffer and then copied to the `dst`
|
|
`io_writer`. Before the ring buffer wraps around, all of its bytes must have
|
|
been copied to `dst`, so `transform_io` can suspend with `$short write`.
|
|
|
|
The C code's ring buffer (`s->ringbuffer_size`) can start smaller than the
|
|
window size (its "canny ring buffer allocation") and grows over successive
|
|
metablocks. The Wuffs ring buffer is always `(1 << WBITS)` bytes long, but the
|
|
decoder still tracks the C code's (virtual) ring buffer size, as it decides
|
|
when the C code calls `WriteRingBuffer`. That in turn decides whether some
|
|
invalid input (an overrun metablock) is reported as `BLOCK_LENGTH_1` or
|
|
`BLOCK_LENGTH_2`, and when the C code's `max_distance` becomes fixed.
|
|
|
|
Everything else (the Huffman tables, context maps, etc.) lives in the decoder
|
|
struct, not the workbuf. The Huffman tables use the same 8-bit root table and
|
|
second level table layout as the C code, sized per tree as the C code
|
|
allocates them.
|
|
|
|
|
|
## Status Mapping
|
|
|
|
Every `BROTLI_DECODER_ERROR_FORMAT_*` error code (and running out of input) has
|
|
a corresponding Wuffs status:
|
|
|
|
| libbrotlidec error code | Wuffs status |
|
|
| ----------------------------------------------------- | ----------------------------------------------------- |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_EXUBERANT_NIBBLE` | `"#bad exuberant nibble"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_RESERVED` | `"#bad reserved bit"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_EXUBERANT_META_NIBBLE` | `"#bad exuberant metadata nibble"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_SIMPLE_HUFFMAN_ALPHABET` | `"#bad simple Huffman alphabet"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_SIMPLE_HUFFMAN_SAME` | `"#bad simple Huffman symbol"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_CL_SPACE` | `"#bad code length code lengths"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_HUFFMAN_SPACE` | `"#bad Huffman space"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_CONTEXT_MAP_REPEAT` | `"#bad context map repeat"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_BLOCK_LENGTH_1` | `"#bad block length"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_BLOCK_LENGTH_2` | `"#bad metablock length"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_TRANSFORM` | `"#bad transform"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_DICTIONARY` | `"#bad dictionary"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS` | `"#bad window bits"` or `"#unsupported large window"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_PADDING_1` | `"#bad padding"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_PADDING_2` | `"#bad final padding"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_DISTANCE` | `"#bad distance"` |
|
|
| `BROTLI_DECODER_ERROR_FORMAT_BLOCK_SWITCH` | `"#bad block switch"` |
|
|
| `BROTLI_DECODER_NEEDS_MORE_INPUT` (with closed `src`) | `"#truncated input"` |
|
|
|
|
Notes:
|
|
|
|
- `"#unsupported large window"` is for the large-window escape `WBITS` value
|
|
(`0010001`). libbrotlidec (with `BROTLI_DECODER_PARAM_LARGE_WINDOW` not set,
|
|
the default) rejects it with `BROTLI_DECODER_ERROR_FORMAT_WINDOW_BITS`. For
|
|
standard (non-large-window) brotli, no other `WBITS` value is invalid, so
|
|
`"#bad window bits"` is not currently produced.
|
|
- Like the C code, which error an overrun metablock produces (`"#bad block
|
|
length"` or `"#bad metablock length"`) can depend on how the input is split
|
|
across `transform_io` calls. When the C code runs out of input, it first
|
|
tries to flush its output (`WriteRingBuffer`) and that checks for an overrun
|
|
metablock. A Wuffs `$short read` suspension corresponds to the C code's
|
|
running out of input.
|
|
- `"#bad block switch"` needs more than `(1 << 24)` commands in one metablock.
|
|
It is implemented but not tested.
|
|
- The remaining libbrotlidec error codes do not apply: `COMPOUND_DICTIONARY`
|
|
and `DICTIONARY_NOT_SET` (custom dictionaries are not supported),
|
|
`INVALID_ARGUMENTS`, the `ALLOC_*` codes (this decoder does not allocate)
|
|
and `UNREACHABLE`.
|
|
|
|
The `"#internal error: etc"` statuses should never happen. They are
|
|
consistency checks, generally where the C code relies on (unchecked)
|
|
invariants for memory safety but Wuffs requires explicit bounds checks.
|
|
|
|
|
|
## Differences from the C Code
|
|
|
|
- The bit reader reads input bytes only when it needs them (like the C code's
|
|
"safe" `BrotliSafeReadBits` and `SafeReadSymbol` functions), so that
|
|
`transform_io` can suspend at any point. There is no "fast path" (like the C
|
|
code's `ProcessCommands`, as opposed to `SafeProcessCommands`). Both paths
|
|
produce the same output and errors.
|
|
- LZ77 copies and dictionary words are written to the ring buffer one byte at
|
|
a time, with ring buffer events (the C code's `WriteRingBuffer` and
|
|
`WrapRingBuffer` calls) at the same positions as the C code.
|
|
- On error, the Wuffs decoder may have already written more output to `dst`
|
|
than libbrotlidec would have (libbrotlidec does not flush on error), but
|
|
what both have written agrees.
|
|
|
|
|
|
## TODO
|
|
|
|
- Large-window brotli (`BROTLI_DECODER_PARAM_LARGE_WINDOW`, window bits up to
|
|
30). The window-related fields are sized for 24 bits.
|
|
- Custom, shared and compound dictionaries.
|
|
- Metadata callbacks (metadata blocks are currently skipped).
|
|
- Fast paths.
|
|
- Fuzzers.
|
|
- An encoder.
|
|
|
|
|
|
## Encoder
|
|
|
|
This package also has an `encoder`, which implements the `base.io_transformer`
|
|
interface. It aims for byte-exact parity with the reference implementation's
|
|
encoder (libbrotlienc), version v1.2.1: for the same parameters and the same
|
|
sequence of operations, it produces exactly the same bytes. Like the decoder,
|
|
the Wuffs code mirrors the C code function by function, with comments like
|
|
"Mirrors BrotliCompressFragmentFast in compress_fragment.c". Its floating
|
|
point arithmetic (the C code's `double`s) uses `std/softfloat`, so that the
|
|
results are bit-exact regardless of the compiler and CPU.
|
|
|
|
Only qualities 0 and 1 (the C code's `FAST_ONE_PASS_COMPRESSION_QUALITY` and
|
|
`FAST_TWO_PASS_COMPRESSION_QUALITY`, implemented by `compress_fragment.c` and
|
|
`compress_fragment_two_pass.c`) are implemented. For qualities 2 to 11
|
|
(including the default quality, 11), `transform_io` returns
|
|
`"#unsupported encoder quality"`.
|
|
|
|
The source files are:
|
|
|
|
- `encode_brotli.wuffs`: the `encoder` struct, its public methods and the
|
|
`encode.c` streaming logic (`BrotliEncoderCompressStream`,
|
|
`BrotliEncoderCompressStreamFast`, `ProcessMetadata`, `EnsureInitialized`,
|
|
`InjectBytePaddingBlock`, etc).
|
|
- `encode_quirks.wuffs`: the encoder's quirk keys.
|
|
- `encode_bit_writer.wuffs`: `write_bits.h`, `fast_log.h`,
|
|
`find_match_length.h` and small `brotli_bit_stream.c` helpers.
|
|
- `encode_entropy.wuffs`: `entropy_encode.c` (Huffman tree construction) and
|
|
the `brotli_bit_stream.c` functions that store Huffman codes.
|
|
- `encode_fragment_fast.wuffs`: `compress_fragment.c` (quality 0).
|
|
- `encode_fragment_two_pass.wuffs`: `compress_fragment_two_pass.c` (quality
|
|
1).
|
|
- `common_encoder_tables.wuffs`: tables generated by `go run
|
|
script/print-brotli-tables.go -encoder` from the C code.
|
|
|
|
|
|
### Encoder API
|
|
|
|
In addition to the `base.io_transformer` methods, the encoder has:
|
|
|
|
- `request_flush!() status`: the next `transform_io` call acts like the C
|
|
code's `BROTLI_OPERATION_FLUSH` (instead of `PROCESS` or `FINISH`) until
|
|
the flush is complete. It returns `"#bad call sequence"` while a metadata
|
|
request is pending.
|
|
- `request_emit_metadata!(length: u64) status`: the next `transform_io` call
|
|
acts like `BROTLI_OPERATION_EMIT_METADATA`, with the next `length` bytes of
|
|
`src` as the metadata block's contents. `length` can be at most `1 << 24`
|
|
(otherwise `"#bad argument"`). It returns `"#bad call sequence"` while
|
|
another request is pending.
|
|
- `max_compressed_size(input_size: u64) u64`: mirrors
|
|
`BrotliEncoderMaxCompressedSize` (0 means overflow).
|
|
|
|
`dst_history_retain_length` is 0: the encoder never re-reads `dst`.
|
|
|
|
|
|
### Encoder Quirks
|
|
|
|
Each quirk mirrors one `BrotliEncoderParameter`, with the value semantics of
|
|
`BrotliEncoderSetParameter`. Values are stored as is (and must fit in a u32,
|
|
otherwise `set_quirk` returns `"#bad argument"`). Like the C code, they are
|
|
sanitized (`SanitizeParams`, `ComputeLgBlock`, `ChooseDistanceParams`) at the
|
|
first `transform_io` call, after which `set_quirk` returns
|
|
`"#bad call sequence"`.
|
|
|
|
| Quirk | C parameter |
|
|
| ---------------------------------------- | ---------------------------------------------- |
|
|
| `QUIRK_ENCODER_QUALITY` | `BROTLI_PARAM_QUALITY` (default 11) |
|
|
| `QUIRK_LGWIN` | `BROTLI_PARAM_LGWIN` (default 22) |
|
|
| `QUIRK_LGBLOCK` | `BROTLI_PARAM_LGBLOCK` |
|
|
| `QUIRK_MODE` | `BROTLI_PARAM_MODE` |
|
|
| `QUIRK_DISABLE_LITERAL_CONTEXT_MODELING` | `BROTLI_PARAM_DISABLE_LITERAL_CONTEXT_MODELING` (0 or 1) |
|
|
| `QUIRK_SIZE_HINT` | `BROTLI_PARAM_SIZE_HINT` |
|
|
| `QUIRK_LARGE_WINDOW` (shared with the decoder) | `BROTLI_PARAM_LARGE_WINDOW` |
|
|
| `QUIRK_NPOSTFIX` | `BROTLI_PARAM_NPOSTFIX` |
|
|
| `QUIRK_NDIRECT` | `BROTLI_PARAM_NDIRECT` |
|
|
| `QUIRK_STREAM_OFFSET` | `BROTLI_PARAM_STREAM_OFFSET` (at most `1 << 30`) |
|
|
|
|
The quality quirk is not named `QUIRK_QUALITY` because that name would clash
|
|
with `base.QUIRK_QUALITY`. At qualities 0 and 1, only the quality, lgwin and
|
|
stream offset affect the output (lgblock, mode, size hint, large window,
|
|
npostfix, ndirect and literal context modeling are stored, sanitized and then
|
|
ignored, as in the C code).
|
|
|
|
|
|
### Encoder Workbuf Layout
|
|
|
|
At qualities 0 and 1, the workbuf is the C code's metablock storage (its
|
|
`GetBrotliStorage` buffer): `workbuf_len` is `(2 << lgwin) + 503 + 16` bytes
|
|
(the storage for a `1 << lgwin` byte block plus 16 bytes of slack for 8-byte
|
|
bit writes), where lgwin is the sanitized value. A shorter workbuf still works
|
|
if every input block is shorter: a block of `n` bytes needs `(2 * n) + 519`
|
|
bytes, otherwise `transform_io` returns `"#bad workbuf length"`. At other
|
|
qualities, `workbuf_len` is 0.
|
|
|
|
The hash table (up to `1 << 17` entries), the quality 1 command and literal
|
|
buffers (`1 << 17` entries each) and the Huffman scratch space are arrays in
|
|
the `encoder` struct (which is therefore more than 1 MiB). They do not need
|
|
zero-initializing: `WUFFS_INITIALIZE__LEAVE_INTERNAL_BUFFERS_UNINITIALIZED` is
|
|
fine. Unlike the C code, the input block is not copied: the encoder compresses
|
|
directly from `src`.
|
|
|
|
|
|
### Encoder Operation Mapping
|
|
|
|
Each `transform_io` call (initial or resumed) corresponds to one or more
|
|
`BrotliEncoderCompressStream` calls, with `src`'s readable bytes as the C
|
|
code's `available_in` and `dst`'s writable bytes as `available_out`. The
|
|
operation is:
|
|
|
|
- `EMIT_METADATA` if `request_emit_metadata` is pending (until the metadata
|
|
block is complete),
|
|
- otherwise `FLUSH` if `request_flush` is pending (until the flush is
|
|
complete),
|
|
- otherwise `FINISH` if `src` is closed,
|
|
- otherwise `PROCESS`.
|
|
|
|
`transform_io` suspends with `"$short write"` when there is pending output
|
|
(the C code's `BrotliEncoderHasMoreOutput`) but `dst` is full, suspends with
|
|
`"$short read"` when all of `src` is consumed (and, for a flush, the flush is
|
|
complete) and returns ok once the stream is finished (the C code's
|
|
`BrotliEncoderIsFinished`). So these two loops produce the same bytes:
|
|
|
|
- C: for each chunk, call `BrotliEncoderCompressStream(op)` until all of the
|
|
chunk is consumed and there is no more output (or, for `FINISH`, until
|
|
finished).
|
|
- Wuffs: for each chunk, call `request_flush` (for `FLUSH`) or
|
|
`request_emit_metadata(len)` (for `EMIT_METADATA`), make `src` hold exactly
|
|
that chunk (closed for `FINISH`) and call `transform_io` until it returns
|
|
`"$short read"` (or ok for `FINISH`), giving it more `dst` room after each
|
|
`"$short write"`.
|
|
|
|
At qualities 0 and 1 the C code compresses each call's available input
|
|
immediately (in blocks of at most `1 << lgwin` bytes), so the output depends
|
|
on how the input is split into chunks. The test programs' encode drivers
|
|
(`wuffs_brotli_encode_plan` in `test/c/std/brotli.c` and
|
|
`mimic_brotli_encode` in `test/c/mimiclib/brotli.c`) run the same "plan" of
|
|
(operation, length) steps with each library.
|
|
|
|
|
|
### Encoder Testing
|
|
|
|
`wuffs test -mimic std/brotli` checks byte-exact parity with libbrotlienc
|
|
over a grid of qualities, lgwins, modes, input chunkings, flushes, metadata
|
|
blocks, size hints and other parameters, as well as round trips through the
|
|
decoder. `script/manual-test-brotli-encoder-parity.c` runs a larger grid
|
|
(about 86 thousand configurations over every file in `test/data/brotli` and
|
|
some synthetic inputs).
|
|
|
|
|
|
### Encoder Differences from the C Code
|
|
|
|
- The input block is not copied (see the workbuf layout above), and output is
|
|
always staged in the workbuf (or a small internal buffer) and then copied to
|
|
`dst`. The C code sometimes writes directly to `next_out`. The bytes are the
|
|
same.
|
|
- Wuffs' C code generator does not support `slice base.u16` or
|
|
`slice base.u32` arguments (and mishandles `copy_from_slice` on `base.u16`
|
|
arrays), so some functions (e.g. `create_huffman_tree`,
|
|
`convert_bit_depths_to_symbols`) take a selector naming one of the
|
|
encoder's arrays instead of a slice.
|
|
- Like the C code, there is no fallback to an uncompressed stream (that is
|
|
only in the C code's one-shot `BrotliEncoderCompress`).
|
|
|
|
|
|
### Encoder TODO
|
|
|
|
- Qualities 2 to 11 (the C code's `EncodeData`, hashers, block splitting,
|
|
context modeling, Zopfli, etc).
|
|
- Large-window encoding (only used at qualities 3 and above).
|
|
- Custom and shared dictionaries (`BrotliEncoderAttachPreparedDictionary`).
|
|
- Changing parameters mid-stream.
|