std/brotli: document compound dictionaries at qualities 10 and 11

Describe the Zopfli path's dictionary lookups, the extra quality 11
BackwardMatch storage in workbuf_len and drop the TODO items.
This commit is contained in:
Claude committed 2026-10-27 09:08:15 +00:00
1 parent ec99ba2a01
commit 6568bc1b75
2 files changed
+34 -12

No files matched your search

+2 -2
View File
@@ -46,7 +46,7 @@ first `transform_io` call.
`min_incl` is a smaller practical bound), but a shorter workbuf works for
short inputs: see the README's "Encoder Workbuf Layout" section. Up to 15
raw dictionaries can be attached (like libbrotlienc's
`BrotliEncoderAttachPreparedDictionary`) at qualities 0 to 9, see the
README's "Encoder Dictionaries" section. They live at the start of the
`BrotliEncoderAttachPreparedDictionary`; like the C code, qualities 0
and 1 ignore them), see the README's "Encoder Dictionaries" section. They live at the start of the
workbuf, with their prepared hash tables, and `workbuf_len` grows
accordingly.
+32 -10
View File
@@ -154,8 +154,6 @@ invariants for memory safety but Wuffs requires explicit bounds checks.
- Metadata callbacks (metadata blocks are currently skipped). See the
"Metadata Blocks" section below.
- Fast paths.
- Compound dictionary matches at encoder qualities 10 and 11 (see the
"Encoder" section below).
## Encoder
@@ -186,10 +184,7 @@ hasher, `BrotliCreateZopfliBackwardReferences` at quality 10 and
below) works at qualities 3 to 11 with lgwin 10 to 30. Like the C code,
qualities 0 to 2 ignore the large-window quirk.
Raw (LZ77 prefix) dictionaries (see "Encoder Dictionaries" below) work at
qualities 0 to 9, and, like the C code, qualities 0 and 1 ignore them. At
qualities 10 and 11, `transform_io` returns `"#unsupported encoder quality"`
if a dictionary is attached (the Zopfli path's dictionary lookups are not
implemented yet).
all qualities, and, like the C code, qualities 0 and 1 ignore them.
The source files are:
@@ -517,6 +512,19 @@ literals) and:
- `M` is the `metablock_builder`'s `workbuf_len_worst_case(L, cmds_cap)`
(`max_incl`) or `workbuf_len_practical(L, cmds_cap)` (`min_incl`).
With raw dictionaries attached (see "Encoder Dictionaries" below), both
`min_incl` and `max_incl` also include the dictionaries' `D` bytes, and, at
quality 11, `Z` has room for 256 more `BackwardMatch`es (the C code's
`shadow_matches` in `BrotliCreateHqZopfliBackwardReferences`: scratch space
past the current position's matches, where the binary tree's and the
compound dictionary's matches are found before `MergeMatches` merges them)
and, for `max_incl`, up to 164 `BackwardMatch`es per byte (64 more, the
C code's `match_limit`, from the compound dictionary). At quality 10, the
C code's matches buffer (`2 * (MAX_NUM_MATCHES_H10 + 64)` entries, for one
position) already has room for its `lz_matches_offset` of
`MAX_NUM_MATCHES_H10 + 128`, so `Z` is unchanged. In practice `M` is the
larger of the two, so only `D` changes the table below.
`workbuf_len_practical` is `workbuf_len_worst_case` with these caps:
`FindBlocks` makes at most `(k / 16) + 1` blocks for `k` literals, commands
or distances (instead of one block per symbol), `ClusterBlocks`' first phase
@@ -554,7 +562,11 @@ words, at most the C code's `max_distance` check in `FindAllMatchesH10` and
less than `(1 << 24) + (1 << 17)`, plus 15 for the distance code), which
satisfies the `metablock_builder`'s precondition (distance codes at most the
distance params' `max_distance`, which is at least `(1 << 26) - 4` without
large-window brotli).
large-window brotli). With raw dictionaries attached, the static dictionary
words start `gap` further away and compound dictionary matches can be
further away too, but explicit distances are still at most `max_distance`
(`FindAllMatchesH10` and `FindAllCompoundDictionaryMatches` skip longer
ones, and distance cache hits use short distance codes), as in the C code.
### Encoder Operation Mapping
@@ -725,6 +737,19 @@ The compound dictionary then affects compression like in the C code:
that are within `params.dist.max_distance` (`BROTLI_MAX_DISTANCE`, or
`BROTLI_MAX_ALLOWED_DISTANCE` for large-window brotli) and that do not
cross a chunk's end.
- At qualities 10 and 11 (`BrotliCreateZopfliBackwardReferences` and
`BrotliCreateHqZopfliBackwardReferences` in `backward_references_hq.c`),
for each position, `LookupAllCompoundDictionaryMatches` finds up to 64
matches (of length 4 or more, each longer than the previous one, over the
chunks in order) in the dictionaries, which `MergeMatches` merges (by
length, then distance) with the H10 binary tree's and static dictionary's
matches. The binary tree's static dictionary matches start at
`dictionary_start + gap`, and `UpdateNodes`, `EvaluateNode` (via
`ComputeDistanceShortcut`) and `BrotliZopfliCreateCommands` treat
distances up to `dictionary_start + gap` as non-static-dictionary ones
(which update the distance cache). `UpdateNodes` also tries the distance
cache's distances that land in the compound dictionary (finding the chunk
and matching up to its end).
- With `QUIRK_STREAM_OFFSET`, `dictionary_start` is `min(position +
stream_offset, max_backward_limit)`, as in the C code (whose output then
continues another stream and can only be decoded after it).
@@ -1018,9 +1043,6 @@ metablock) for as long as it needs the results.
- Speed: at qualities 10 and 11 the Wuffs encoder is several times slower
than libbrotlienc, mostly because of the soft float arithmetic in the
Zopfli cost model, the block splitter and histogram clustering.
- Compound dictionary matches at qualities 10 and 11 (the Zopfli path's
`FindCompoundDictionaryMatch` lookups); attaching a dictionary at those
qualities currently returns `"#unsupported encoder quality"`.
- Serialized (shared) dictionaries, which only an experimental
(`BROTLI_EXPERIMENTAL`) libbrotlienc build supports.
- Changing parameters mid-stream.