std/brotli: document large-window encoding and short workbufs

This commit is contained in:
Claude committed 2026-10-27 02:40:28 +00:00
1 parent 0b9cfddf3c
commit eb5ae76130
2 files changed
+126 -24

No files matched your search

+5 -1
View File
@@ -32,4 +32,8 @@ encoder has `request_flush`, `request_emit_metadata` and
## Implementations
- [std/brotli](/std/brotli/README.md#encoder) (qualities 0 to 9 only).
- [std/brotli](/std/brotli/README.md#encoder) (qualities 0 to 9 only,
including large-window brotli, with windows of up to 1 GiB, at qualities 3
to 9). Its `workbuf_len` is the worst case for the parameters (more than
2 GiB for large-window brotli at lgwin 30), but a shorter workbuf works for
short inputs: see the README's "Encoder Workbuf Layout" section.
+121 -23
View File
@@ -154,8 +154,8 @@ invariants for memory safety but Wuffs requires explicit bounds checks.
- Metadata callbacks (metadata blocks are currently skipped). See the
"Metadata Blocks" section below.
- Fast paths.
- Encoder quality levels 10 and 11, encoder dictionaries and large-window
encoding at qualities 3 and above (see the "Encoder" section below).
- Encoder quality levels 10 and 11 and encoder dictionaries (see the
"Encoder" section below).
## Encoder
@@ -178,11 +178,11 @@ qualities 2 to 9 (the C code's `EncodeData` path: a ring buffer, hashers,
2, `BrotliStoreMetaBlockTrivial` at quality 3 and, at qualities 4 to 9, the
literal context modeling heuristics, `BrotliBuildMetaBlockGreedy`,
`BrotliOptimizeHistograms` and `BrotliStoreMetaBlock`). For qualities 10 and
11 (including the default quality, 11), and for large-window brotli at
qualities 3 and above, `transform_io` returns
11 (including the default quality, 11), `transform_io` returns
`"#unsupported encoder quality"` (and `workbuf_len` is 0). Large-window
brotli works at quality 2 (and is ignored at qualities 0 and 1, like the C
code).
brotli (see "Encoder Large Window" below) works at qualities 3 to 9 with
lgwin 10 to 30. Like the C code, qualities 0 to 2 ignore the large-window
quirk.
The source files are:
@@ -213,6 +213,10 @@ The source files are:
H54, H5, H6, H58, H68, H40, H41 and H42) and `SearchInStaticDictionary`
(with `static_dict.c`'s matching), as one set of functions selected by the
hasher type at run time.
- `encode_hasher_composite.wuffs`: the large-window hashers:
`hash_composite_inc.h` (H35, H55 and H65, which combine H3, H54 or H6 with
a rolling hasher) and `hash_rolling_inc.h` (`HROLLING_FAST` and
`HROLLING`).
- `encode_backward_references.wuffs`: `backward_references.c` and
`backward_references_inc.h` (`BrotliCreateBackwardReferences`),
`command.h` and `ComputeDistanceCode`.
@@ -283,7 +287,7 @@ first `transform_io` call, after which `set_quirk` returns
| `QUIRK_MODE` | `BROTLI_PARAM_MODE` |
| `QUIRK_DISABLE_LITERAL_CONTEXT_MODELING` | `BROTLI_PARAM_DISABLE_LITERAL_CONTEXT_MODELING` (0 or 1) |
| `QUIRK_SIZE_HINT` | `BROTLI_PARAM_SIZE_HINT` |
| `QUIRK_LARGE_WINDOW` (shared with the decoder) | `BROTLI_PARAM_LARGE_WINDOW` |
| `QUIRK_LARGE_WINDOW` (shared with the decoder) | `BROTLI_PARAM_LARGE_WINDOW` (0 or 1) |
| `QUIRK_NPOSTFIX` | `BROTLI_PARAM_NPOSTFIX` |
| `QUIRK_NDIRECT` | `BROTLI_PARAM_NDIRECT` |
| `QUIRK_STREAM_OFFSET` | `BROTLI_PARAM_STREAM_OFFSET` (at most `1 << 30`) |
@@ -293,7 +297,8 @@ with `base.QUIRK_QUALITY`. At qualities 0 and 1, only the quality, lgwin and
stream offset affect the output (lgblock, mode, size hint, large window,
npostfix, ndirect and literal context modeling are stored, sanitized and then
ignored, as in the C code). At qualities 2 to 9, they have the same effect as
in the C code: lgblock (at qualities 4 and above; the C code's
in the C code: large window (see "Encoder Large Window" below), lgblock (at
qualities 4 and above; the C code's
`ComputeLgBlock` fixes it to 14 below that), npostfix, ndirect and the mode
(only `BROTLI_MODE_FONT` matters, through `ChooseDistanceParams`, at
qualities 4 and above), disabling literal context modeling (at qualities 5
@@ -319,17 +324,15 @@ code's `ComputeRbBits`), `max_len` is `1 << min(rb_bits, 24)` (the C code's
1. The metablock storage: `(2 * max_len) + 503 + 16` bytes, like at
qualities 0 and 1 but for the largest metablock.
2. The ring buffer (the C code's `RingBuffer.buffer_`): `2 + (1 << rb_bits) +
(1 << lgblock) + 7` bytes. The first 2 bytes are the copy of the last two
bytes that precede `data_`, the tail (`1 << lgblock` bytes) mirrors the
start of the buffer and the 7 trailing bytes are the C code's slack for
8-byte loads.
3. The hasher's memory (the C code's `common->extra[0 .. 3]`): the sum of
2. The hasher's memory (the C code's `common->extra[0 .. 3]`): the sum of
`FN(HashMemAllocInBytes)` for the hasher that `ChooseHasher` picks. When
the size hint quirk is unset (zero), that choice depends on the input's
length (through `UpdateSizeHint`), so this is the larger of the two
possible choices (for a size hint below or at least 1 MiB).
4. The commands: `16 * cmds_cap` bytes, in the C struct order: `insert_len_`
possible choices (for a size hint below or at least 1 MiB). With lgwin
above 24 (large window), the composite hashers H35, H55 and H65 add their
rolling hasher's table (`(1 << 24)` u32le values, 64 MiB) after H3's,
H54's or H6's memory.
3. The commands: `16 * cmds_cap` bytes, in the C struct order: `insert_len_`
(u32le), `copy_len_` (u32le, the copy length in the low 25 bits and
`copy_code - copy_len` in the high 7 bits), `dist_extra_` (u32le),
`cmd_prefix_` (u16le) and `dist_prefix_` (u16le). `cmds_cap` is
@@ -339,7 +342,7 @@ code's `ComputeRbBits`), `max_len` is `1 << min(rb_bits, 24)` (the C code's
input block) fewer than that many commands and, like the C code's
`num_commands_ + bytes / 2 + 1` allocation, each input block of `n` bytes
adds at most `(n / 2) + 1` commands.
5. At qualities 4 and above, `BrotliBuildMetaBlockGreedy`'s `MetaBlockSplit`
4. At qualities 4 and above, `BrotliBuildMetaBlockGreedy`'s `MetaBlockSplit`
and `BrotliStoreMetaBlock`'s scratch space:
- the block types (u8) and then the block lengths (u32le) of the literal,
command and distance block splits, with room for `(max_len / 512) + 1`,
@@ -355,13 +358,38 @@ code's `ComputeRbBits`), `max_len` is `1 << min(rb_bits, 24)` (the C code's
- `EncodeContextMap`'s `rle_symbols` (256 * 64 u32le values) and
- the block encoders' depths (u8) and then bits (u16le) for up to 256
histograms per category.
5. The ring buffer (the C code's `RingBuffer.buffer_`): `2 + (1 << rb_bits) +
(1 << lgblock) + 7` bytes. The first 2 bytes are the copy of the last two
bytes that precede `data_`, the tail (`1 << lgblock` bytes) mirrors the
start of the buffer and the 7 trailing bytes are the C code's slack for
8-byte loads.
Unlike the C code, which allocates its storage and command buffers as it
goes (for the actual metablock and block sizes), `workbuf_len` is the worst
case for the sanitized parameters, so that one workbuf works for any input.
At qualities 2 to 9, `transform_io` returns `"#bad workbuf length"` if the
workbuf is shorter than that. Some examples (in MiB, with the default
lgblock and an unset size hint):
Unlike the C code, which allocates its storage, ring buffer and command
buffers as it goes (for the actual metablock and block sizes, and the actual
input length), `workbuf_len` is the worst case for the sanitized parameters,
so that one workbuf works for any input. A shorter workbuf still works for
short inputs, because the ring buffer is the last region and the encoder
only touches its first `2 + n + 7` bytes for an input of `n` bytes, as long
as the ring buffer never wraps around. Precisely, at qualities 2 to 9:
- `transform_io` returns `"#bad workbuf length"` if the workbuf is shorter
than `workbuf_len - R + 18`, where `R` is the ring buffer region's length
(`2 + (1 << rb_bits) + (1 << lgblock) + 7`), i.e. if it does not reach 16
bytes into the ring buffer's data.
- If any `transform_io` call gets a workbuf shorter than `workbuf_len`, then
from then on (for the rest of the stream), before copying input into the
ring buffer, `transform_io` returns `"#bad workbuf length"` (without
consuming that input) if the total input so far (not counting metadata)
would be `n` bytes with `n + 16 > (1 << rb_bits)` (the ring buffer would
wrap, or its tail would be needed) or with the workbuf shorter than
`workbuf_len - R + 2 + n + 16`.
So, for a total input of `n` bytes with `n + 16 <= (1 << rb_bits)`, a
workbuf of `workbuf_len - R + n + 18` bytes produces the same output as one
of `workbuf_len` bytes. The parity test checks this with exactly that length
(and that one byte less fails). This matters most for large-window brotli,
whose ring buffer is up to 2 GiB. Some examples of `workbuf_len` (in MiB,
with the default lgblock and an unset size hint):
| quality | lgwin 10 | lgwin 16 | lgwin 18 | lgwin 22 | lgwin 24 |
| ------- | -------- | -------- | -------- | -------- | -------- |
@@ -372,6 +400,21 @@ lgblock and an unset size hint):
| 7 | 4.4 | 4.4 | 13.8 | 51.4 | 107.5 |
| 9 | 5.1 | 5.1 | 39.5 | 77.1 | 133.2 |
With the large-window quirk (at qualities 3 and above; lgwin up to 24 gives
the same numbers as without it), larger lgwin values double the ring buffer
for each extra bit, and lgwin above 24 adds the composite hashers' 64 MiB
rolling hash table (at quality 4 and at qualities 5 to 9 with lgwin of at
least 19, only for a size hint of at least 1 MiB, or an unset one, as
`ChooseHasher` otherwise keeps H3, H5 or H58):
| quality | lgwin 25 | lgwin 26 | lgwin 28 | lgwin 30 |
| ------- | -------- | -------- | -------- | -------- |
| 3 | 160.6 | 224.6 | 608.6 | 2144.6 |
| 4 | 199.4 | 263.4 | 647.4 | 2183.4 |
| 5 | 197.5 | 261.5 | 645.5 | 2181.5 |
| 7 | 203.5 | 267.5 | 651.5 | 2187.5 |
| 9 | 229.2 | 293.2 | 677.2 | 2213.2 |
Fixed-size state (the hash table and command and literal buffers of
qualities 0 and 1, the Huffman scratch space, the greedy block splitters'
state, the block encoders' block type and length codes, etc) lives in the `encoder` struct (which is therefore
@@ -429,6 +472,45 @@ encoder makes the same `EncodeData` calls, at the same points. One
emitting the flint, and its callers then call it again.
### Encoder Large Window
The `QUIRK_LARGE_WINDOW` quirk mirrors `BROTLI_PARAM_LARGE_WINDOW`
(large-window brotli is an incompatible extension of RFC 7932, decodable by
this package's decoder with the same quirk, see "Large Window" above, and by
libbrotlidec with `BROTLI_DECODER_PARAM_LARGE_WINDOW`). Like the C code's
`SanitizeParams`, `EnsureInitialized` and `ChooseHasher`:
- At qualities 0 to 2 (the C code's `MAX_QUALITY_FOR_STATIC_ENTROPY_CODES`
and below), the quirk is ignored: lgwin is clamped to 24 and the output is
standard brotli.
- At qualities 3 and above, lgwin is clamped to `[10 ..= 30]` (instead of
`[10 ..= 24]`) and the stream starts with the 14-bit large-window header
(`EncodeWindowBits`), even for lgwin 24 and below. The distance parameters
(`BrotliInitDistanceParams` with `large_window`) allow distances up to
`BROTLI_MAX_ALLOWED_DISTANCE` with an alphabet of up to
`BROTLI_LARGE_MAX_DISTANCE_BITS` (62) distance bits, so the distance
Huffman codes are stored with a larger alphabet size than without the
quirk, even when no distance is large. Backward distances are at most
`(1 << lgwin) - 16`, so their extra bits (at most 30) fit in a command's
32-bit `dist_extra_`, as in the C code.
- lgwin above 24 changes the hasher (only at qualities 3 to 9; the C code's
qualities 10 and 11 use H10 regardless): H3 becomes H35 (H3 plus
`HROLLING_FAST`), H54 becomes H55 (H54 plus `HROLLING_FAST`) and H6 and
H68 become H65 (H6, not H68, plus `HROLLING`). H5, H58 and H40 to H42 stay
as they are, so qualities 5 to 9 only use H65 when the size hint (set or
estimated by `UpdateSizeHint`) is at least 1 MiB (and lgwin is at least 19,
which it is). The rolling hashers find matches of at least 32 bytes at any
distance in the window (`HROLLING_FAST` only at positions and distances
that are multiples of 4) and do not search the static dictionary.
- lgblock is unaffected (it is at most 24), but `rb_bits` (and so the ring
buffer) grows with lgwin. Metablocks are still at most `1 << 24` bytes.
Positions (the C code's `size_t` and the `WrapPosition`'ed `uint32_t`
values) and distances are `base.u64` throughout the encoder, so nothing is
truncated for windows up to `1 << 30`. The rolling hashers' table holds
32-bit positions, like the C code's.
### Encoder Testing
`wuffs test -mimic std/brotli` checks byte-exact parity with libbrotlienc
@@ -462,6 +544,22 @@ libbrotlienc v1.2.1 on x86-64.
`BROTLI_MAX_SIMD_QUALITY` is undefined (no 64-bit count-trailing-zeros
builtin) or 7 (MSVC on x86-64) choose differently and so produce
different (but valid) output at those qualities.
- The composite hashers (H35, H55 and H65) are not separate hasher types
here: `hasher_type` is their first hasher (H3, H54 or H6) and a non-zero
`hr_jump` (4 for `HROLLING_FAST`, 1 for `HROLLING`) adds the rolling
hasher, whose functions run after the first hasher's, like the C code's
composite `FN(Prepare)`, `FN(StitchToPreviousBlock)` and
`FN(FindLongestMatch)`. The C code's composite hasher defers its inner
hashers' `FN(Initialize)` to its first `FN(Prepare)`. Here, the first
hasher's `FN(Initialize)` runs in `hasher_setup` (it does not touch the
hasher's memory, so this is equivalent) and the rolling hasher's at the
first `hasher_prepare`. The C code's composite gives its first hasher a
copy of its `HasherCommon` (with zeroed static dictionary statistics);
here, the first hasher uses the encoder's statistics, which the composite
itself never uses, so the result is the same.
- The workbuf's regions are in a different order than the C code's
allocations and the ring buffer is fully reserved from the start (see the
workbuf layout above). This does not change the output.
- `BrotliBuildMetaBlockGreedy`'s two kinds of block splitter (one for a
single histogram per block type, `metablock_inc.h`, and one for several
literal contexts, `ContextBlockSplitter` in `metablock.c`) are one