std/brotli: document the encoder at qualities 2 to 9

This commit is contained in:
Claude committed 2026-10-27 01:20:26 +00:00
1 parent 314b20e832
commit b9f9d6ce65
3 files changed
+178 -37

No files matched your search

+1 -1
View File
@@ -32,4 +32,4 @@ encoder has `request_flush`, `request_emit_metadata` and
## Implementations
- [std/brotli](/std/brotli/README.md#encoder) (qualities 0 and 1 only).
- [std/brotli](/std/brotli/README.md#encoder) (qualities 0 to 9 only).
+169 -29
View File
@@ -154,8 +154,8 @@ invariants for memory safety but Wuffs requires explicit bounds checks.
- Metadata callbacks (metadata blocks are currently skipped). See the
"Metadata Blocks" section below.
- Fast paths.
- Encoder quality levels 2 to 11, encoder dictionaries and large-window
encoding (see the "Encoder" section below).
- Encoder quality levels 10 and 11, encoder dictionaries and large-window
encoding at qualities 3 and above (see the "Encoder" section below).
## Encoder
@@ -169,11 +169,20 @@ the Wuffs code mirrors the C code function by function, with comments like
point arithmetic (the C code's `double`s) uses `std/softfloat`, so that the
results are bit-exact regardless of the compiler and CPU.
Only qualities 0 and 1 (the C code's `FAST_ONE_PASS_COMPRESSION_QUALITY` and
`FAST_TWO_PASS_COMPRESSION_QUALITY`, implemented by `compress_fragment.c` and
`compress_fragment_two_pass.c`) are implemented. For qualities 2 to 11
(including the default quality, 11), `transform_io` returns
`"#unsupported encoder quality"`.
Qualities 0 to 9 are implemented: qualities 0 and 1 (the C code's
`FAST_ONE_PASS_COMPRESSION_QUALITY` and `FAST_TWO_PASS_COMPRESSION_QUALITY`,
implemented by `compress_fragment.c` and `compress_fragment_two_pass.c`) and
qualities 2 to 9 (the C code's `EncodeData` path: a ring buffer, hashers,
`BrotliCreateBackwardReferences` with static dictionary matching, and
`WriteMetaBlockInternal`, which picks `BrotliStoreMetaBlockFast` at quality
2, `BrotliStoreMetaBlockTrivial` at quality 3 and, at qualities 4 to 9, the
literal context modeling heuristics, `BrotliBuildMetaBlockGreedy`,
`BrotliOptimizeHistograms` and `BrotliStoreMetaBlock`). For qualities 10 and
11 (including the default quality, 11), and for large-window brotli at
qualities 3 and above, `transform_io` returns
`"#unsupported encoder quality"` (and `workbuf_len` is 0). Large-window
brotli works at quality 2 (and is ignored at qualities 0 and 1, like the C
code).
The source files are:
@@ -195,11 +204,41 @@ The source files are:
- `encode_histogram.wuffs`: `histogram.h`, `histogram_inc.h` and
`histogram.c` (histogram helpers and `BrotliBuildHistogramsWithContext`),
also as `bit_cost` methods.
- `encode_data.wuffs`: the rest of `encode.c` that qualities 2 to 9 use
(`EncodeData`, `WriteMetaBlockInternal`, `ShouldCompress`,
`DecideOverLiteralContextModeling`, `ChooseContextMap`, `UpdateSizeHint`,
`ExtendLastCommand`, etc), `ringbuffer.h`, the `quality.h` helpers and the
workbuf layout.
- `encode_hasher.wuffs`: `hash.h` and the `hash_*_inc.h` hashers (H2, H3, H4,
H54, H5, H6, H58, H68, H40, H41 and H42) and `SearchInStaticDictionary`
(with `static_dict.c`'s matching), as one set of functions selected by the
hasher type at run time.
- `encode_backward_references.wuffs`: `backward_references.c` and
`backward_references_inc.h` (`BrotliCreateBackwardReferences`),
`command.h` and `ComputeDistanceCode`.
- `encode_bit_stream.wuffs`: `brotli_bit_stream.c` (`StoreMetaBlockHeader`,
`BrotliStoreUncompressedMetaBlock`, `BrotliStoreMetaBlockTrivial`,
`BrotliStoreMetaBlockFast`, `BrotliStoreMetaBlock`, block split codes,
`EncodeContextMap` with move-to-front and run-length coding,
`BuildAndStoreEntropyCodes`, etc).
- `encode_metablock_greedy.wuffs`: `BrotliBuildMetaBlockGreedy` (and its
greedy block splitters, `metablock_inc.h`) and `BrotliOptimizeHistograms`
for its output.
- `encode_util.wuffs`: little-endian workbuf accessors.
- `encode_block_splitter.wuffs`, `encode_cluster.wuffs` and
`encode_metablock.wuffs`: the `metablock_builder` struct, with
`block_splitter.c`, `cluster.c` and `BrotliBuildMetaBlock` (which only
quality 10 and above use, and so the encoder does not call it yet). The
encoder uses some of its smaller functions (`BrotliInitDistanceParams`,
`PrefixEncodeCopyDistance`, `CommandRestoreDistanceCode` and
`BrotliOptimizeHuffmanCountsForRle`).
- `encode_tester.wuffs`: TEST ONLY. The `metablock_tester` struct exposes
the internal functions above to C test programs. It is not part of the
encoder API.
- `common_encoder_tables.wuffs`: tables generated by `go run
script/print-brotli-tables.go -encoder` from the C code.
script/print-brotli-tables.go -encoder` from the C code (including the
static dictionary hash tables of `dictionary_hash.c` and the
`static_dict_lut.h` buckets and words).
### Encoder API
@@ -247,7 +286,14 @@ The quality quirk is not named `QUIRK_QUALITY` because that name would clash
with `base.QUIRK_QUALITY`. At qualities 0 and 1, only the quality, lgwin and
stream offset affect the output (lgblock, mode, size hint, large window,
npostfix, ndirect and literal context modeling are stored, sanitized and then
ignored, as in the C code).
ignored, as in the C code). At qualities 2 to 9, they have the same effect as
in the C code: lgblock (at qualities 4 and above; the C code's
`ComputeLgBlock` fixes it to 14 below that), npostfix, ndirect and the mode
(only `BROTLI_MODE_FONT` matters, through `ChooseDistanceParams`, at
qualities 4 and above), disabling literal context modeling (at qualities 5
and above) and the size hint (the hasher choice at qualities 4 and above and
the complex static context map at qualities 5 and above). An unset (zero)
size hint is replaced by the C code's `UpdateSizeHint` estimate.
### Encoder Workbuf Layout
@@ -257,15 +303,77 @@ At qualities 0 and 1, the workbuf is the C code's metablock storage (its
(the storage for a `1 << lgwin` byte block plus 16 bytes of slack for 8-byte
bit writes), where lgwin is the sanitized value. A shorter workbuf still works
if every input block is shorter: a block of `n` bytes needs `(2 * n) + 519`
bytes, otherwise `transform_io` returns `"#bad workbuf length"`. At other
qualities, `workbuf_len` is 0.
bytes, otherwise `transform_io` returns `"#bad workbuf length"`.
The hash table (up to `1 << 17` entries), the quality 1 command and literal
buffers (`1 << 17` entries each) and the Huffman scratch space are arrays in
the `encoder` struct (which is therefore more than 1 MiB). They do not need
zero-initializing: `WUFFS_INITIALIZE__LEAVE_INTERNAL_BUFFERS_UNINITIALIZED` is
fine. Unlike the C code, the input block is not copied: the encoder compresses
directly from `src`.
At qualities 2 to 9, the workbuf holds everything whose size depends on the
input or on the parameters, in this order (see `compute_layout_offset` in
`encode_data.wuffs`). Here `rb_bits` is `1 + max(lgwin, lgblock)` (the C
code's `ComputeRbBits`), `max_len` is `1 << min(rb_bits, 24)` (the C code's
`MaxMetablockSize`) and `cmds_cap` is the commands capacity (see below):
1. The metablock storage: `(2 * max_len) + 503 + 16` bytes, like at
qualities 0 and 1 but for the largest metablock.
2. The ring buffer (the C code's `RingBuffer.buffer_`): `2 + (1 << rb_bits) +
(1 << lgblock) + 7` bytes. The first 2 bytes are the copy of the last two
bytes that precede `data_`, the tail (`1 << lgblock` bytes) mirrors the
start of the buffer and the 7 trailing bytes are the C code's slack for
8-byte loads.
3. The hasher's memory (the C code's `common->extra[0 .. 3]`): the sum of
`FN(HashMemAllocInBytes)` for the hasher that `ChooseHasher` picks. When
the size hint quirk is unset (zero), that choice depends on the input's
length (through `UpdateSizeHint`), so this is the larger of the two
possible choices (for a size hint below or at least 1 MiB).
4. The commands: `16 * cmds_cap` bytes, in the C struct order: `insert_len_`
(u32le), `copy_len_` (u32le, the copy length in the low 25 bits and
`copy_code - copy_len` in the high 7 bits), `dist_extra_` (u32le),
`cmd_prefix_` (u16le) and `dist_prefix_` (u16le). `cmds_cap` is
`min(prior + ((1 << lgblock) / 2) + 1, (max_len / 2) + 1)`, where `prior`
is `(max_len / 8) - 1` (and at most `MAX_NUM_DELAYED_SYMBOLS - 1`, which
is `0x2FFE`, below quality 4): `EncodeData` keeps (merges with the next
input block) fewer than that many commands and, like the C code's
`num_commands_ + bytes / 2 + 1` allocation, each input block of `n` bytes
adds at most `(n / 2) + 1` commands.
5. At qualities 4 and above, `BrotliBuildMetaBlockGreedy`'s `MetaBlockSplit`
and `BrotliStoreMetaBlock`'s scratch space:
- the block types (u8) and then the block lengths (u32le) of the literal,
command and distance block splits, with room for `(max_len / 512) + 1`,
`(cmds_cap / 1024) + 1` and `(cmds_cap / 512) + 1` blocks (the C
code's `max_num_blocks`, given that a metablock has at most `max_len`
literals and `cmds_cap` commands),
- the histograms (u32le counts, without the C code's `total_count_` and
`bit_cost_` fields, which are not needed here): up to 260 literal
histograms (`(256 / 13) + 1` block types of 13 contexts), 257 command
histograms and 257 distance histograms of 256, 704 and 544 counts,
- the block splitters' `combined_histo` (2 * 13 * 256 u32le counts),
- the literal context map (256 * 64 bytes),
- `EncodeContextMap`'s `rle_symbols` (256 * 64 u32le values) and
- the block encoders' depths (u8) and then bits (u16le) for up to 256
histograms per category.
Unlike the C code, which allocates its storage and command buffers as it
goes (for the actual metablock and block sizes), `workbuf_len` is the worst
case for the sanitized parameters, so that one workbuf works for any input.
At qualities 2 to 9, `transform_io` returns `"#bad workbuf length"` if the
workbuf is shorter than that. Some examples (in MiB, with the default
lgblock and an unset size hint):
| quality | lgwin 10 | lgwin 16 | lgwin 18 | lgwin 22 | lgwin 24 |
| ------- | -------- | -------- | -------- | -------- | -------- |
| 0, 1 | 0.0 | 0.1 | 0.5 | 8.0 | 32.0 |
| 2, 3 | 0.5 | 1.0 | 2.1 | 24.6 | 64.6 |
| 4 | 7.9 | 7.9 | 9.8 | 47.3 | 103.4 |
| 5 | 4.4 | 4.4 | 7.0 | 45.9 | 102.0 |
| 7 | 4.4 | 4.4 | 13.8 | 51.4 | 107.5 |
| 9 | 5.1 | 5.1 | 39.5 | 77.1 | 133.2 |
Fixed-size state (the hash table and command and literal buffers of
qualities 0 and 1, the Huffman scratch space, the greedy block splitters'
state, the block encoders' block type and length codes, etc) lives in the `encoder` struct (which is therefore
about 2.5 MiB). It does not need zero-initializing:
`WUFFS_INITIALIZE__LEAVE_INTERNAL_BUFFERS_UNINITIALIZED` is fine. At
qualities 0 and 1, the input block is not copied: the encoder compresses
directly from `src`. At qualities 2 to 9, like the C code, the input is
copied into the ring buffer.
### Encoder Operation Mapping
@@ -304,23 +412,53 @@ on how the input is split into chunks. The test programs' encode drivers
`mimic_brotli_encode` in `test/c/mimiclib/brotli.c`) run the same "plan" of
(operation, length) steps with each library.
At qualities 2 to 9 the C code copies the input into its ring buffer and
only compresses it (`EncodeData`) when an input block (`1 << lgblock` bytes)
is full, or on a flush or finish, so the output depends on the chunking only
through the flushes (and the `UpdateSizeHint` estimate, which depends on how
much input is available when the first block is compressed). The Wuffs
encoder makes the same `EncodeData` calls, at the same points. One
`transform_io` call can correspond to several C calls: with a stream offset
(the C code's "flint"), a C call can return early, with input left, after
emitting the flint, and its callers then call it again.
### Encoder Testing
`wuffs test -mimic std/brotli` checks byte-exact parity with libbrotlienc
over a grid of qualities, lgwins, modes, input chunkings, flushes, metadata
blocks, size hints and other parameters, as well as round trips through the
decoder. `script/manual-test-brotli-encoder-parity.c` runs a larger grid
(about 86 thousand configurations over every file in `test/data/brotli` and
some synthetic inputs).
over a grid of qualities (0 to 9), lgwins, modes, input chunkings, flushes,
metadata blocks, size hints and other parameters (lgblock, npostfix, ndirect,
disabling literal context modeling, stream offset, large window at quality 2),
as well as round trips through the decoder.
`script/manual-test-brotli-encoder-parity.c` runs a larger grid over every
file in `test/data/brotli`, some larger files in `test/data` (up to about
400 KiB) and some synthetic inputs (up to 1 MiB). Its `-q=LO-HI` and
`-shard=K/N` options select the qualities and split the inputs over several
processes. For qualities 0 and 1 that is about 86 thousand configurations
and for qualities 2 to 9 several hundred thousand, all byte-exact.
### Encoder Differences from the C Code
- The input block is not copied (see the workbuf layout above), and output is
always staged in the workbuf (or a small internal buffer) and then copied to
`dst`. The C code sometimes writes directly to `next_out`. The bytes are the
same.
- At qualities 0 and 1, the input block is not copied (see the workbuf
layout above). Output is always staged in the workbuf (or a small internal
buffer) and then copied to `dst`. The C code sometimes writes directly to
`next_out`. The bytes are the same.
- The C code instantiates each hasher (and the block splitters) from
templates. Here, each template is one set of functions, parameterized at
run time (e.g. by the hasher type). The H58 and H68 hashers' SIMD tag
matching (`GetMatchingTagMask` in `matching_tag_mask.h`) is computed one
tag at a time, which finds the same candidates in the same order. Like the
x86-64 GCC reference build (where `BROTLI_MAX_SIMD_QUALITY` is 6, as in
any build with a 64-bit `__builtin_ctzll`), qualities 5 and 6 with lgwin
above 16 use H58 or H68 instead of H5 or H6. libbrotlienc builds where
`BROTLI_MAX_SIMD_QUALITY` is undefined (no 64-bit count-trailing-zeros
builtin) or 7 (MSVC on x86-64) choose differently and so produce
different (but valid) output at those qualities.
- `BrotliBuildMetaBlockGreedy`'s two kinds of block splitter (one for a
single histogram per block type, `metablock_inc.h`, and one for several
literal contexts, `ContextBlockSplitter` in `metablock.c`) are one
implementation here, as they make the same decisions with one context.
- Wuffs' C code generator does not support `slice base.u16` or
`slice base.u32` arguments (and mishandles `copy_from_slice` on `base.u16`
arrays), so some functions (e.g. `create_huffman_tree`,
@@ -332,9 +470,11 @@ some synthetic inputs).
### Encoder TODO
- Qualities 2 to 11 (the C code's `EncodeData`, hashers, block splitting,
context modeling, Zopfli, etc).
- Large-window encoding (only used at qualities 3 and above).
- Qualities 10 and 11 (Zopfli, the H10 binary tree hasher and
`BrotliBuildMetaBlock`, which is already ported in `encode_metablock.wuffs`).
- Large-window encoding at qualities 3 and above (it works at quality 2).
- `workbuf_len` is a worst case. A future option could take the input length
(like the C code's size hint) to give a tighter bound.
- Custom and shared dictionaries (`BrotliEncoderAttachPreparedDictionary`).
- Changing parameters mid-stream.
+8 -7
View File
@@ -19,14 +19,15 @@
// The C code's floating point (double) arithmetic is done with the
// std/softfloat package, which is bit-exact. FastLog2's log2 matches glibc's.
//
// Only qualities 0 and 1 (the C code's BrotliCompressFragmentFast and
// BrotliCompressFragmentTwoPass) are implemented so far. See README.md for
// the API, including how the C code's FLUSH and EMIT_METADATA operations map
// to the request_flush and request_emit_metadata methods.
// Qualities 0 and 1 (the C code's BrotliCompressFragmentFast and
// BrotliCompressFragmentTwoPass) and qualities 2 to 9 (the C code's
// EncodeData, see encode_data.wuffs) are implemented so far. See README.md
// for the API, including how the C code's FLUSH and EMIT_METADATA operations
// map to the request_flush and request_emit_metadata methods.
//
// TODO: qualities 2 to 11 (the C code's EncodeData, WriteMetaBlockInternal,
// hashers, block splitting, context modeling and Zopfli), large-window
// brotli at those qualities and custom (prepared or shared) dictionaries.
// TODO: qualities 10 and 11 (Zopfli and BrotliBuildMetaBlock), large-window
// brotli at qualities 3 and above and custom (prepared or shared)
// dictionaries.
use "std/softfloat"