mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-17 16:32:11 +01:00
std/brotli: document the quality 10 and 11 encoder
This commit is contained in:
3 files changed
+148
-27
No files matched your search
@@ -25,6 +25,11 @@ In Wuffs syntax, the `base.io_transformer` methods are:
|
||||
- `transform_io?(dst: io_writer, src: io_reader, workbuf: slice u8)`
|
||||
- `workbuf_len() range_ii_u64`
|
||||
|
||||
A workbuf of `workbuf_len().max_incl` bytes always suffices. Some encoders
|
||||
(e.g. brotli at qualities 10 and 11) have a smaller `min_incl`, which is
|
||||
enough for realistic inputs: with a workbuf shorter than `max_incl`, an
|
||||
unusual input can make `transform_io` return `"#bad workbuf length"`.
|
||||
|
||||
Encoders may have further, format-specific methods. For example, the brotli
|
||||
encoder has `request_flush`, `request_emit_metadata` and
|
||||
`max_compressed_size`.
|
||||
@@ -32,4 +37,4 @@ encoder has `request_flush`, `request_emit_metadata` and
|
||||
|
||||
## Implementations
|
||||
|
||||
- [std/brotli](/std/brotli/README.md#encoder) (qualities 0 to 9 only).
|
||||
- [std/brotli](/std/brotli/README.md#encoder) (qualities 0 to 11, without encoder dictionaries).
|
||||
+135
-19
@@ -154,8 +154,8 @@ invariants for memory safety but Wuffs requires explicit bounds checks.
|
||||
- Metadata callbacks (metadata blocks are currently skipped). See the
|
||||
"Metadata Blocks" section below.
|
||||
- Fast paths.
|
||||
- Encoder quality levels 10 and 11, encoder dictionaries and large-window
|
||||
encoding at qualities 3 and above (see the "Encoder" section below).
|
||||
- Encoder dictionaries and large-window encoding at qualities 3 and above
|
||||
(see the "Encoder" section below).
|
||||
|
||||
|
||||
## Encoder
|
||||
@@ -169,7 +169,7 @@ the Wuffs code mirrors the C code function by function, with comments like
|
||||
point arithmetic (the C code's `double`s) uses `std/softfloat`, so that the
|
||||
results are bit-exact regardless of the compiler and CPU.
|
||||
|
||||
Qualities 0 to 9 are implemented: qualities 0 and 1 (the C code's
|
||||
All qualities (0 to 11) are implemented: qualities 0 and 1 (the C code's
|
||||
`FAST_ONE_PASS_COMPRESSION_QUALITY` and `FAST_TWO_PASS_COMPRESSION_QUALITY`,
|
||||
implemented by `compress_fragment.c` and `compress_fragment_two_pass.c`) and
|
||||
qualities 2 to 9 (the C code's `EncodeData` path: a ring buffer, hashers,
|
||||
@@ -177,10 +177,14 @@ qualities 2 to 9 (the C code's `EncodeData` path: a ring buffer, hashers,
|
||||
`WriteMetaBlockInternal`, which picks `BrotliStoreMetaBlockFast` at quality
|
||||
2, `BrotliStoreMetaBlockTrivial` at quality 3 and, at qualities 4 to 9, the
|
||||
literal context modeling heuristics, `BrotliBuildMetaBlockGreedy`,
|
||||
`BrotliOptimizeHistograms` and `BrotliStoreMetaBlock`). For qualities 10 and
|
||||
11 (including the default quality, 11), and for large-window brotli at
|
||||
qualities 3 and above, `transform_io` returns
|
||||
`"#unsupported encoder quality"` (and `workbuf_len` is 0). Large-window
|
||||
`BrotliOptimizeHistograms` and `BrotliStoreMetaBlock`) and qualities 10 and
|
||||
11 (the default is 11: the same `EncodeData` path with the H10 binary tree
|
||||
hasher, `BrotliCreateZopfliBackwardReferences` at quality 10 and
|
||||
`BrotliCreateHqZopfliBackwardReferences` at quality 11, and then
|
||||
`ChooseContextMode`, `BrotliBuildMetaBlock`, `BrotliOptimizeHistograms` and
|
||||
`BrotliStoreMetaBlock`). For large-window brotli at qualities 3 and above,
|
||||
`transform_io` returns `"#unsupported encoder quality"` (and `workbuf_len`
|
||||
is 0). Large-window
|
||||
brotli works at quality 2 (and is ignored at qualities 0 and 1, like the C
|
||||
code).
|
||||
|
||||
@@ -204,7 +208,7 @@ The source files are:
|
||||
- `encode_histogram.wuffs`: `histogram.h`, `histogram_inc.h` and
|
||||
`histogram.c` (histogram helpers and `BrotliBuildHistogramsWithContext`),
|
||||
also as `bit_cost` methods.
|
||||
- `encode_data.wuffs`: the rest of `encode.c` that qualities 2 to 9 use
|
||||
- `encode_data.wuffs`: the rest of `encode.c` that qualities 2 to 11 use
|
||||
(`EncodeData`, `WriteMetaBlockInternal`, `ShouldCompress`,
|
||||
`DecideOverLiteralContextModeling`, `ChooseContextMap`, `UpdateSizeHint`,
|
||||
`ExtendLastCommand`, etc), `ringbuffer.h`, the `quality.h` helpers and the
|
||||
@@ -221,6 +225,25 @@ The source files are:
|
||||
`BrotliStoreMetaBlockFast`, `BrotliStoreMetaBlock`, block split codes,
|
||||
`EncodeContextMap` with move-to-front and run-length coding,
|
||||
`BuildAndStoreEntropyCodes`, etc).
|
||||
- `encode_hash_to_binary_tree.wuffs`: `hash_to_binary_tree_inc.h` (the H10
|
||||
hasher of qualities 10 and 11, including `StoreRange`,
|
||||
`StitchToPreviousBlock` and `FindAllMatches`).
|
||||
- `encode_static_dict.wuffs`: `static_dict.c`'s
|
||||
`BrotliFindAllStaticDictionaryMatches` (every transform), with the
|
||||
`static_dict_lut.h` tables.
|
||||
- `encode_literal_cost.wuffs`: `literal_cost.c`
|
||||
(`BrotliEstimateBitCostsForLiterals`, with `utf8_util.c`'s
|
||||
`BrotliIsMostlyUTF8`), in f32 arithmetic via `std/softfloat`.
|
||||
- `encode_zopfli.wuffs`: `backward_references_hq.c` (the `ZopfliNode` array,
|
||||
`ZopfliCostModel`, `StartPosQueue`, `UpdateNodes`,
|
||||
`ZopfliComputeShortestPath`, `ZopfliCreateCommands`,
|
||||
`BrotliCreateZopfliBackwardReferences` and
|
||||
`BrotliCreateHqZopfliBackwardReferences`), with the C code's f32 costs as
|
||||
binary32 bit patterns.
|
||||
- `encode_store_hq.wuffs`: the quality 10 and 11 parts of
|
||||
`WriteMetaBlockInternal` (`ChooseContextMode`, the `metablock_builder` call
|
||||
and `BrotliStoreMetaBlock` with any literal context mode and context maps)
|
||||
and the quality 10 and 11 workbuf sizing.
|
||||
- `encode_metablock_greedy.wuffs`: `BrotliBuildMetaBlockGreedy` (and its
|
||||
greedy block splitters, `metablock_inc.h`) and `BrotliOptimizeHistograms`
|
||||
for its output.
|
||||
@@ -234,10 +257,10 @@ The source files are:
|
||||
- `encode_metablock.wuffs`: the `metablock_builder` struct and `metablock.c`
|
||||
(`BrotliInitDistanceParams`, `BrotliBuildMetaBlock`,
|
||||
`BrotliOptimizeHistograms`). See "Encoder Metablock Builder" below.
|
||||
Only quality 10 and above use `BrotliBuildMetaBlock`, so the encoder does
|
||||
not call it yet, but it uses some of the smaller functions
|
||||
(`BrotliInitDistanceParams`, `PrefixEncodeCopyDistance`,
|
||||
`CommandRestoreDistanceCode` and `BrotliOptimizeHuffmanCountsForRle`).
|
||||
Only qualities 10 and 11 use `BrotliBuildMetaBlock` (lower qualities use
|
||||
some of the smaller functions: `BrotliInitDistanceParams`,
|
||||
`PrefixEncodeCopyDistance`, `CommandRestoreDistanceCode` and
|
||||
`BrotliOptimizeHuffmanCountsForRle`).
|
||||
- `encode_tester.wuffs`: TEST ONLY. The `metablock_tester` struct exposes
|
||||
the internal functions above to C test programs. It is not part of the
|
||||
encoder API.
|
||||
@@ -372,6 +395,9 @@ lgblock and an unset size hint):
|
||||
| 7 | 4.4 | 4.4 | 13.8 | 51.4 | 107.5 |
|
||||
| 9 | 5.1 | 5.1 | 39.5 | 77.1 | 133.2 |
|
||||
|
||||
At qualities 10 and 11, `workbuf_len` is a range instead (see the next
|
||||
section).
|
||||
|
||||
Fixed-size state (the hash table and command and literal buffers of
|
||||
qualities 0 and 1, the Huffman scratch space, the greedy block splitters'
|
||||
state, the block encoders' block type and length codes, etc) lives in the `encoder` struct (which is therefore
|
||||
@@ -382,6 +408,82 @@ directly from `src`. At qualities 2 to 9, like the C code, the input is
|
||||
copied into the ring buffer.
|
||||
|
||||
|
||||
### Qualities 10 and 11 Workbuf Length
|
||||
|
||||
At qualities 10 and 11, the workbuf has the same regions as at quality 9
|
||||
(with `cmds_cap` computed the same way), except that:
|
||||
|
||||
- region 3 is the H10 hasher: `4 << 17` bytes of buckets (u32le) and then
|
||||
the binary tree forest (`8 << lgwin` bytes, u32le), and
|
||||
- region 5 keeps only the literal context map, `rle_symbols` and the block
|
||||
encoders' depths and bits, followed by a variable region, which runs from
|
||||
there to the end of the workbuf (however long that is).
|
||||
|
||||
The variable region holds, at different times, the Zopfli scratch space
|
||||
(while `EncodeData` computes the backward references: `(n + 1)` 16-byte
|
||||
`ZopfliNode`s, `(n + 2)` f32 literal costs and, at quality 10, room for one
|
||||
position's matches or, at quality 11, `n` u32 match counts followed by the
|
||||
8-byte `BackwardMatch`es of the whole input block, for an input block of `n`
|
||||
bytes) and then the `metablock_builder`'s workbuf (see "Encoder Metablock
|
||||
Builder" below), whose results the metablock store then reads.
|
||||
|
||||
The C code allocates these as it goes, for the actual input, and their worst
|
||||
cases are huge (see `workbuf_len_worst_case` below), so `workbuf_len` returns
|
||||
a range whose `max_incl` is the C code's worst case for the sanitized
|
||||
parameters (saturating at `0xFFFF_FFFF_FFFF_FFFF`) and whose `min_incl` is a
|
||||
practical bound. Both have the same fixed regions, followed by a variable
|
||||
region of `max(Z, M)` bytes, where `L` is the most literals that a metablock
|
||||
can hold (`min(max_len, (max_len / 8) + (1 << lgblock))`, since `EncodeData`
|
||||
merges another input block only while there are fewer than `max_len / 8`
|
||||
literals) and:
|
||||
|
||||
- `Z` is the Zopfli scratch space for `n = 1 << lgblock` bytes, with up to
|
||||
100 `BackwardMatch`es per byte (`max_incl`: up to 2 from the short
|
||||
distance scan, 64 from the binary tree and 34 from the static dictionary)
|
||||
or 16 per byte (`min_incl`), and
|
||||
- `M` is the `metablock_builder`'s `workbuf_len_worst_case(L, cmds_cap)`
|
||||
(`max_incl`) or `workbuf_len_practical(L, cmds_cap)` (`min_incl`).
|
||||
|
||||
`workbuf_len_practical` is `workbuf_len_worst_case` with these caps:
|
||||
`FindBlocks` makes at most `(k / 16) + 1` blocks for `k` literals, commands
|
||||
or distances (instead of one block per symbol), `ClusterBlocks`' first phase
|
||||
makes at most `(k / 64) + 64` clusters and there are at most that many (and
|
||||
at most 256) block types. Over 142 files from `test/data` (including
|
||||
`harvesters.bmp`, 3 MiB, and other images, text, JSON and binaries) and some
|
||||
synthetic inputs, at qualities 10 and 11 and lgwins 16 and 24, the C
|
||||
algorithm's actual maxima were one block per 83 literals (per 85 commands,
|
||||
per 154 distances), one first-phase cluster per 286 literals (per 915
|
||||
commands) and about 7.4 `BackwardMatch`es per byte (for random text over a
|
||||
2-letter alphabet; at most 2.3 for the files), so the caps have a margin of
|
||||
at least 4x. The largest `metablock_builder` workbuf used was 43.7 MB, for
|
||||
2.1 MiB of literals.
|
||||
|
||||
This is one documented deviation from the C code: with a workbuf of at least
|
||||
`min_incl` but less than `max_incl` bytes, an input that needs more than the
|
||||
variable region holds makes `transform_io` return `"#bad workbuf length"`
|
||||
(where the C code would allocate more memory). A workbuf of `max_incl` bytes
|
||||
never fails. In MiB (with the default lgblock, which is
|
||||
`min(18, lgwin)` for lgwin above 16 and 16 otherwise), `min_incl` and
|
||||
`max_incl` are:
|
||||
|
||||
| quality | lgwin 10 | lgwin 16 | lgwin 18 | lgwin 22 | lgwin 24 |
|
||||
| ------- | ------------- | ------------- | ------------- | -------------- | -------------- |
|
||||
| 10 | 91.7 .. 288.0 | 92.2 .. 288.5 | 128.7 .. 914.2 | 403.0 .. 5116.1 | 775.5 .. 9678.1 |
|
||||
| 11 | 91.7 .. 288.0 | 92.2 .. 288.5 | 128.7 .. 914.2 | 403.0 .. 5116.1 | 775.5 .. 9678.1 |
|
||||
|
||||
(Quality 11's extra `BackwardMatch` storage fits in the space that the
|
||||
`metablock_builder` needs anyway, so the two rows are equal. With lgblock
|
||||
24, `min_incl` is about 2.1 GiB at quality 10 and 2.6 GiB at quality 11.)
|
||||
|
||||
At qualities 10 and 11, every command's distance code is less than `1 <<
|
||||
25` (distances are at most `(1 << lgwin) - 16` or, for static dictionary
|
||||
words, at most the C code's `max_distance` check in `FindAllMatchesH10` and
|
||||
less than `(1 << 24) + (1 << 17)`, plus 15 for the distance code), which
|
||||
satisfies the `metablock_builder`'s precondition (distance codes at most the
|
||||
distance params' `max_distance`, which is at least `(1 << 26) - 4` without
|
||||
large-window brotli).
|
||||
|
||||
|
||||
### Encoder Operation Mapping
|
||||
|
||||
Each `transform_io` call (initial or resumed) corresponds to one or more
|
||||
@@ -432,7 +534,7 @@ emitting the flint, and its callers then call it again.
|
||||
### Encoder Testing
|
||||
|
||||
`wuffs test -mimic std/brotli` checks byte-exact parity with libbrotlienc
|
||||
over a grid of qualities (0 to 9), lgwins, modes, input chunkings, flushes,
|
||||
over a grid of qualities (0 to 11), lgwins, modes, input chunkings, flushes,
|
||||
metadata blocks, size hints and other parameters (lgblock, npostfix, ndirect,
|
||||
disabling literal context modeling, stream offset, large window at quality 2),
|
||||
as well as round trips through the decoder.
|
||||
@@ -473,6 +575,13 @@ libbrotlienc v1.2.1 on x86-64.
|
||||
encoder's arrays instead of a slice.
|
||||
- Like the C code, there is no fallback to an uncompressed stream (that is
|
||||
only in the C code's one-shot `BrotliEncoderCompress`).
|
||||
- At qualities 10 and 11, a workbuf between `workbuf_len`'s `min_incl` and
|
||||
`max_incl` can be too short for an unusual input, which then fails with
|
||||
`"#bad workbuf length"` (see "Qualities 10 and 11 Workbuf Length").
|
||||
- At qualities 10 and 11, the C code's f32 costs (`ZopfliNode` costs, the
|
||||
cost model and literal costs) are computed with `std/softfloat`'s binary32
|
||||
operations, rounding each operation to f32 like an SSE2 build (whose
|
||||
`float` arithmetic has no excess precision).
|
||||
|
||||
|
||||
### Encoder Metablock Builder
|
||||
@@ -480,8 +589,9 @@ libbrotlienc v1.2.1 on x86-64.
|
||||
The `metablock_builder` struct (a `pri struct`, embeddable in the encoder
|
||||
with one field, e.g. `mb : metablock_builder` in the private data section)
|
||||
implements the quality 10 and 11 metablock construction (`BrotliBuildMetaBlock`)
|
||||
and `BrotliOptimizeHistograms`. The encoder has one (its `mb` field) but
|
||||
only calls its smaller helpers so far (see the source file list above). Its
|
||||
and `BrotliOptimizeHistograms`. The encoder has one (its `mb` field) and
|
||||
uses it at qualities 10 and 11 (and its smaller helpers at lower
|
||||
qualities). Its
|
||||
fixed-size state (about 0.8 MiB: FindBlocks' cost arrays and a 64Ki-entry
|
||||
FastLog2 cache in its `bit_cost`) lives in the struct; everything that the C
|
||||
code allocates on the heap lives in a caller-supplied workbuf.
|
||||
@@ -556,6 +666,10 @@ The exact signatures are:
|
||||
of the port. `wb_high` records what a call actually used: over the parity
|
||||
harness's inputs, at most about 42 MB (for 1.3 MiB of literals split into
|
||||
256 literal block types).
|
||||
- `workbuf_len_practical(num_literals: u64, num_commands: u64) u64` is the
|
||||
same bound with caps on the numbers of blocks and clusters (see
|
||||
"Qualities 10 and 11 Workbuf Length"), which the encoder uses for its
|
||||
`workbuf_len().min_incl`.
|
||||
|
||||
The MetaBlockSplit result is described by these fields, indexed by category
|
||||
(0 = literal, 1 = command, 2 = distance), with offsets into `wb`:
|
||||
@@ -596,11 +710,13 @@ metablock) for as long as it needs the results.
|
||||
|
||||
### Encoder TODO
|
||||
|
||||
- Qualities 10 and 11 (Zopfli, the H10 binary tree hasher and
|
||||
`BrotliBuildMetaBlock`, which is already ported in `encode_metablock.wuffs`).
|
||||
- Large-window encoding at qualities 3 and above (it works at quality 2).
|
||||
- `workbuf_len` is a worst case. A future option could take the input length
|
||||
(like the C code's size hint) to give a tighter bound.
|
||||
- `workbuf_len` is a worst case (or, at qualities 10 and 11, a practical
|
||||
bound). A future option could take the input length (like the C code's
|
||||
size hint) to give a tighter bound.
|
||||
- Speed: at qualities 10 and 11 the Wuffs encoder is several times slower
|
||||
than libbrotlienc, mostly because of the soft float arithmetic in the
|
||||
block splitter and histogram clustering.
|
||||
- Custom and shared dictionaries (`BrotliEncoderAttachPreparedDictionary`).
|
||||
- Changing parameters mid-stream.
|
||||
|
||||
|
||||
@@ -20,14 +20,14 @@
|
||||
// std/softfloat package, which is bit-exact. FastLog2's log2 matches glibc's.
|
||||
//
|
||||
// Qualities 0 and 1 (the C code's BrotliCompressFragmentFast and
|
||||
// BrotliCompressFragmentTwoPass) and qualities 2 to 9 (the C code's
|
||||
// EncodeData, see encode_data.wuffs) are implemented so far. See README.md
|
||||
// for the API, including how the C code's FLUSH and EMIT_METADATA operations
|
||||
// map to the request_flush and request_emit_metadata methods.
|
||||
// BrotliCompressFragmentTwoPass) and qualities 2 to 11 (the C code's
|
||||
// EncodeData, see encode_data.wuffs, with Zopfli and BrotliBuildMetaBlock at
|
||||
// qualities 10 and 11) are implemented. See README.md for the API, including
|
||||
// how the C code's FLUSH and EMIT_METADATA operations map to the
|
||||
// request_flush and request_emit_metadata methods.
|
||||
//
|
||||
// TODO: qualities 10 and 11 (Zopfli and BrotliBuildMetaBlock), large-window
|
||||
// brotli at qualities 3 and above and custom (prepared or shared)
|
||||
// dictionaries.
|
||||
// TODO: large-window brotli at qualities 3 and above and custom (prepared or
|
||||
// shared) dictionaries.
|
||||
|
||||
use "std/softfloat"
|
||||
|
||||
|
||||
Reference in new issue
Block a user