diff --git a/doc/std/compression-encoders.md b/doc/std/compression-encoders.md index f7c64521..c47ea32b 100644 --- a/doc/std/compression-encoders.md +++ b/doc/std/compression-encoders.md @@ -46,7 +46,7 @@ first `transform_io` call. `min_incl` is a smaller practical bound), but a shorter workbuf works for short inputs: see the README's "Encoder Workbuf Layout" section. Up to 15 raw dictionaries can be attached (like libbrotlienc's - `BrotliEncoderAttachPreparedDictionary`) at qualities 0 to 9, see the - README's "Encoder Dictionaries" section. They live at the start of the + `BrotliEncoderAttachPreparedDictionary`; like the C code, qualities 0 + and 1 ignore them), see the README's "Encoder Dictionaries" section. They live at the start of the workbuf, with their prepared hash tables, and `workbuf_len` grows accordingly. diff --git a/std/brotli/README.md b/std/brotli/README.md index 28ff1e1b..df046b52 100644 --- a/std/brotli/README.md +++ b/std/brotli/README.md @@ -154,8 +154,6 @@ invariants for memory safety but Wuffs requires explicit bounds checks. - Metadata callbacks (metadata blocks are currently skipped). See the "Metadata Blocks" section below. - Fast paths. -- Compound dictionary matches at encoder qualities 10 and 11 (see the - "Encoder" section below). ## Encoder @@ -186,10 +184,7 @@ hasher, `BrotliCreateZopfliBackwardReferences` at quality 10 and below) works at qualities 3 to 11 with lgwin 10 to 30. Like the C code, qualities 0 to 2 ignore the large-window quirk. Raw (LZ77 prefix) dictionaries (see "Encoder Dictionaries" below) work at -qualities 0 to 9, and, like the C code, qualities 0 and 1 ignore them. At -qualities 10 and 11, `transform_io` returns `"#unsupported encoder quality"` -if a dictionary is attached (the Zopfli path's dictionary lookups are not -implemented yet). +all qualities, and, like the C code, qualities 0 and 1 ignore them. The source files are: @@ -517,6 +512,19 @@ literals) and: - `M` is the `metablock_builder`'s `workbuf_len_worst_case(L, cmds_cap)` (`max_incl`) or `workbuf_len_practical(L, cmds_cap)` (`min_incl`). +With raw dictionaries attached (see "Encoder Dictionaries" below), both +`min_incl` and `max_incl` also include the dictionaries' `D` bytes, and, at +quality 11, `Z` has room for 256 more `BackwardMatch`es (the C code's +`shadow_matches` in `BrotliCreateHqZopfliBackwardReferences`: scratch space +past the current position's matches, where the binary tree's and the +compound dictionary's matches are found before `MergeMatches` merges them) +and, for `max_incl`, up to 164 `BackwardMatch`es per byte (64 more, the +C code's `match_limit`, from the compound dictionary). At quality 10, the +C code's matches buffer (`2 * (MAX_NUM_MATCHES_H10 + 64)` entries, for one +position) already has room for its `lz_matches_offset` of +`MAX_NUM_MATCHES_H10 + 128`, so `Z` is unchanged. In practice `M` is the +larger of the two, so only `D` changes the table below. + `workbuf_len_practical` is `workbuf_len_worst_case` with these caps: `FindBlocks` makes at most `(k / 16) + 1` blocks for `k` literals, commands or distances (instead of one block per symbol), `ClusterBlocks`' first phase @@ -554,7 +562,11 @@ words, at most the C code's `max_distance` check in `FindAllMatchesH10` and less than `(1 << 24) + (1 << 17)`, plus 15 for the distance code), which satisfies the `metablock_builder`'s precondition (distance codes at most the distance params' `max_distance`, which is at least `(1 << 26) - 4` without -large-window brotli). +large-window brotli). With raw dictionaries attached, the static dictionary +words start `gap` further away and compound dictionary matches can be +further away too, but explicit distances are still at most `max_distance` +(`FindAllMatchesH10` and `FindAllCompoundDictionaryMatches` skip longer +ones, and distance cache hits use short distance codes), as in the C code. ### Encoder Operation Mapping @@ -725,6 +737,19 @@ The compound dictionary then affects compression like in the C code: that are within `params.dist.max_distance` (`BROTLI_MAX_DISTANCE`, or `BROTLI_MAX_ALLOWED_DISTANCE` for large-window brotli) and that do not cross a chunk's end. +- At qualities 10 and 11 (`BrotliCreateZopfliBackwardReferences` and + `BrotliCreateHqZopfliBackwardReferences` in `backward_references_hq.c`), + for each position, `LookupAllCompoundDictionaryMatches` finds up to 64 + matches (of length 4 or more, each longer than the previous one, over the + chunks in order) in the dictionaries, which `MergeMatches` merges (by + length, then distance) with the H10 binary tree's and static dictionary's + matches. The binary tree's static dictionary matches start at + `dictionary_start + gap`, and `UpdateNodes`, `EvaluateNode` (via + `ComputeDistanceShortcut`) and `BrotliZopfliCreateCommands` treat + distances up to `dictionary_start + gap` as non-static-dictionary ones + (which update the distance cache). `UpdateNodes` also tries the distance + cache's distances that land in the compound dictionary (finding the chunk + and matching up to its end). - With `QUIRK_STREAM_OFFSET`, `dictionary_start` is `min(position + stream_offset, max_backward_limit)`, as in the C code (whose output then continues another stream and can only be decoded after it). @@ -1018,9 +1043,6 @@ metablock) for as long as it needs the results. - Speed: at qualities 10 and 11 the Wuffs encoder is several times slower than libbrotlienc, mostly because of the soft float arithmetic in the Zopfli cost model, the block splitter and histogram clustering. -- Compound dictionary matches at qualities 10 and 11 (the Zopfli path's - `FindCompoundDictionaryMatch` lookups); attaching a dictionary at those - qualities currently returns `"#unsupported encoder quality"`. - Serialized (shared) dictionaries, which only an experimental (`BROTLI_EXPERIMENTAL`) libbrotlienc build supports. - Changing parameters mid-stream.