mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-19 08:38:14 +01:00
std/brotli: document the encoder at qualities 2 to 9
This commit is contained in:
3 files changed
+178
-37
No files matched your search
@@ -32,4 +32,4 @@ encoder has `request_flush`, `request_emit_metadata` and
|
||||
|
||||
## Implementations
|
||||
|
||||
- [std/brotli](/std/brotli/README.md#encoder) (qualities 0 and 1 only).
|
||||
- [std/brotli](/std/brotli/README.md#encoder) (qualities 0 to 9 only).
|
||||
+169
-29
@@ -154,8 +154,8 @@ invariants for memory safety but Wuffs requires explicit bounds checks.
|
||||
- Metadata callbacks (metadata blocks are currently skipped). See the
|
||||
"Metadata Blocks" section below.
|
||||
- Fast paths.
|
||||
- Encoder quality levels 2 to 11, encoder dictionaries and large-window
|
||||
encoding (see the "Encoder" section below).
|
||||
- Encoder quality levels 10 and 11, encoder dictionaries and large-window
|
||||
encoding at qualities 3 and above (see the "Encoder" section below).
|
||||
|
||||
|
||||
## Encoder
|
||||
@@ -169,11 +169,20 @@ the Wuffs code mirrors the C code function by function, with comments like
|
||||
point arithmetic (the C code's `double`s) uses `std/softfloat`, so that the
|
||||
results are bit-exact regardless of the compiler and CPU.
|
||||
|
||||
Only qualities 0 and 1 (the C code's `FAST_ONE_PASS_COMPRESSION_QUALITY` and
|
||||
`FAST_TWO_PASS_COMPRESSION_QUALITY`, implemented by `compress_fragment.c` and
|
||||
`compress_fragment_two_pass.c`) are implemented. For qualities 2 to 11
|
||||
(including the default quality, 11), `transform_io` returns
|
||||
`"#unsupported encoder quality"`.
|
||||
Qualities 0 to 9 are implemented: qualities 0 and 1 (the C code's
|
||||
`FAST_ONE_PASS_COMPRESSION_QUALITY` and `FAST_TWO_PASS_COMPRESSION_QUALITY`,
|
||||
implemented by `compress_fragment.c` and `compress_fragment_two_pass.c`) and
|
||||
qualities 2 to 9 (the C code's `EncodeData` path: a ring buffer, hashers,
|
||||
`BrotliCreateBackwardReferences` with static dictionary matching, and
|
||||
`WriteMetaBlockInternal`, which picks `BrotliStoreMetaBlockFast` at quality
|
||||
2, `BrotliStoreMetaBlockTrivial` at quality 3 and, at qualities 4 to 9, the
|
||||
literal context modeling heuristics, `BrotliBuildMetaBlockGreedy`,
|
||||
`BrotliOptimizeHistograms` and `BrotliStoreMetaBlock`). For qualities 10 and
|
||||
11 (including the default quality, 11), and for large-window brotli at
|
||||
qualities 3 and above, `transform_io` returns
|
||||
`"#unsupported encoder quality"` (and `workbuf_len` is 0). Large-window
|
||||
brotli works at quality 2 (and is ignored at qualities 0 and 1, like the C
|
||||
code).
|
||||
|
||||
The source files are:
|
||||
|
||||
@@ -195,11 +204,41 @@ The source files are:
|
||||
- `encode_histogram.wuffs`: `histogram.h`, `histogram_inc.h` and
|
||||
`histogram.c` (histogram helpers and `BrotliBuildHistogramsWithContext`),
|
||||
also as `bit_cost` methods.
|
||||
- `encode_data.wuffs`: the rest of `encode.c` that qualities 2 to 9 use
|
||||
(`EncodeData`, `WriteMetaBlockInternal`, `ShouldCompress`,
|
||||
`DecideOverLiteralContextModeling`, `ChooseContextMap`, `UpdateSizeHint`,
|
||||
`ExtendLastCommand`, etc), `ringbuffer.h`, the `quality.h` helpers and the
|
||||
workbuf layout.
|
||||
- `encode_hasher.wuffs`: `hash.h` and the `hash_*_inc.h` hashers (H2, H3, H4,
|
||||
H54, H5, H6, H58, H68, H40, H41 and H42) and `SearchInStaticDictionary`
|
||||
(with `static_dict.c`'s matching), as one set of functions selected by the
|
||||
hasher type at run time.
|
||||
- `encode_backward_references.wuffs`: `backward_references.c` and
|
||||
`backward_references_inc.h` (`BrotliCreateBackwardReferences`),
|
||||
`command.h` and `ComputeDistanceCode`.
|
||||
- `encode_bit_stream.wuffs`: `brotli_bit_stream.c` (`StoreMetaBlockHeader`,
|
||||
`BrotliStoreUncompressedMetaBlock`, `BrotliStoreMetaBlockTrivial`,
|
||||
`BrotliStoreMetaBlockFast`, `BrotliStoreMetaBlock`, block split codes,
|
||||
`EncodeContextMap` with move-to-front and run-length coding,
|
||||
`BuildAndStoreEntropyCodes`, etc).
|
||||
- `encode_metablock_greedy.wuffs`: `BrotliBuildMetaBlockGreedy` (and its
|
||||
greedy block splitters, `metablock_inc.h`) and `BrotliOptimizeHistograms`
|
||||
for its output.
|
||||
- `encode_util.wuffs`: little-endian workbuf accessors.
|
||||
- `encode_block_splitter.wuffs`, `encode_cluster.wuffs` and
|
||||
`encode_metablock.wuffs`: the `metablock_builder` struct, with
|
||||
`block_splitter.c`, `cluster.c` and `BrotliBuildMetaBlock` (which only
|
||||
quality 10 and above use, and so the encoder does not call it yet). The
|
||||
encoder uses some of its smaller functions (`BrotliInitDistanceParams`,
|
||||
`PrefixEncodeCopyDistance`, `CommandRestoreDistanceCode` and
|
||||
`BrotliOptimizeHuffmanCountsForRle`).
|
||||
- `encode_tester.wuffs`: TEST ONLY. The `metablock_tester` struct exposes
|
||||
the internal functions above to C test programs. It is not part of the
|
||||
encoder API.
|
||||
- `common_encoder_tables.wuffs`: tables generated by `go run
|
||||
script/print-brotli-tables.go -encoder` from the C code.
|
||||
script/print-brotli-tables.go -encoder` from the C code (including the
|
||||
static dictionary hash tables of `dictionary_hash.c` and the
|
||||
`static_dict_lut.h` buckets and words).
|
||||
|
||||
|
||||
### Encoder API
|
||||
@@ -247,7 +286,14 @@ The quality quirk is not named `QUIRK_QUALITY` because that name would clash
|
||||
with `base.QUIRK_QUALITY`. At qualities 0 and 1, only the quality, lgwin and
|
||||
stream offset affect the output (lgblock, mode, size hint, large window,
|
||||
npostfix, ndirect and literal context modeling are stored, sanitized and then
|
||||
ignored, as in the C code).
|
||||
ignored, as in the C code). At qualities 2 to 9, they have the same effect as
|
||||
in the C code: lgblock (at qualities 4 and above; the C code's
|
||||
`ComputeLgBlock` fixes it to 14 below that), npostfix, ndirect and the mode
|
||||
(only `BROTLI_MODE_FONT` matters, through `ChooseDistanceParams`, at
|
||||
qualities 4 and above), disabling literal context modeling (at qualities 5
|
||||
and above) and the size hint (the hasher choice at qualities 4 and above and
|
||||
the complex static context map at qualities 5 and above). An unset (zero)
|
||||
size hint is replaced by the C code's `UpdateSizeHint` estimate.
|
||||
|
||||
|
||||
### Encoder Workbuf Layout
|
||||
@@ -257,15 +303,77 @@ At qualities 0 and 1, the workbuf is the C code's metablock storage (its
|
||||
(the storage for a `1 << lgwin` byte block plus 16 bytes of slack for 8-byte
|
||||
bit writes), where lgwin is the sanitized value. A shorter workbuf still works
|
||||
if every input block is shorter: a block of `n` bytes needs `(2 * n) + 519`
|
||||
bytes, otherwise `transform_io` returns `"#bad workbuf length"`. At other
|
||||
qualities, `workbuf_len` is 0.
|
||||
bytes, otherwise `transform_io` returns `"#bad workbuf length"`.
|
||||
|
||||
The hash table (up to `1 << 17` entries), the quality 1 command and literal
|
||||
buffers (`1 << 17` entries each) and the Huffman scratch space are arrays in
|
||||
the `encoder` struct (which is therefore more than 1 MiB). They do not need
|
||||
zero-initializing: `WUFFS_INITIALIZE__LEAVE_INTERNAL_BUFFERS_UNINITIALIZED` is
|
||||
fine. Unlike the C code, the input block is not copied: the encoder compresses
|
||||
directly from `src`.
|
||||
At qualities 2 to 9, the workbuf holds everything whose size depends on the
|
||||
input or on the parameters, in this order (see `compute_layout_offset` in
|
||||
`encode_data.wuffs`). Here `rb_bits` is `1 + max(lgwin, lgblock)` (the C
|
||||
code's `ComputeRbBits`), `max_len` is `1 << min(rb_bits, 24)` (the C code's
|
||||
`MaxMetablockSize`) and `cmds_cap` is the commands capacity (see below):
|
||||
|
||||
1. The metablock storage: `(2 * max_len) + 503 + 16` bytes, like at
|
||||
qualities 0 and 1 but for the largest metablock.
|
||||
2. The ring buffer (the C code's `RingBuffer.buffer_`): `2 + (1 << rb_bits) +
|
||||
(1 << lgblock) + 7` bytes. The first 2 bytes are the copy of the last two
|
||||
bytes that precede `data_`, the tail (`1 << lgblock` bytes) mirrors the
|
||||
start of the buffer and the 7 trailing bytes are the C code's slack for
|
||||
8-byte loads.
|
||||
3. The hasher's memory (the C code's `common->extra[0 .. 3]`): the sum of
|
||||
`FN(HashMemAllocInBytes)` for the hasher that `ChooseHasher` picks. When
|
||||
the size hint quirk is unset (zero), that choice depends on the input's
|
||||
length (through `UpdateSizeHint`), so this is the larger of the two
|
||||
possible choices (for a size hint below or at least 1 MiB).
|
||||
4. The commands: `16 * cmds_cap` bytes, in the C struct order: `insert_len_`
|
||||
(u32le), `copy_len_` (u32le, the copy length in the low 25 bits and
|
||||
`copy_code - copy_len` in the high 7 bits), `dist_extra_` (u32le),
|
||||
`cmd_prefix_` (u16le) and `dist_prefix_` (u16le). `cmds_cap` is
|
||||
`min(prior + ((1 << lgblock) / 2) + 1, (max_len / 2) + 1)`, where `prior`
|
||||
is `(max_len / 8) - 1` (and at most `MAX_NUM_DELAYED_SYMBOLS - 1`, which
|
||||
is `0x2FFE`, below quality 4): `EncodeData` keeps (merges with the next
|
||||
input block) fewer than that many commands and, like the C code's
|
||||
`num_commands_ + bytes / 2 + 1` allocation, each input block of `n` bytes
|
||||
adds at most `(n / 2) + 1` commands.
|
||||
5. At qualities 4 and above, `BrotliBuildMetaBlockGreedy`'s `MetaBlockSplit`
|
||||
and `BrotliStoreMetaBlock`'s scratch space:
|
||||
- the block types (u8) and then the block lengths (u32le) of the literal,
|
||||
command and distance block splits, with room for `(max_len / 512) + 1`,
|
||||
`(cmds_cap / 1024) + 1` and `(cmds_cap / 512) + 1` blocks (the C
|
||||
code's `max_num_blocks`, given that a metablock has at most `max_len`
|
||||
literals and `cmds_cap` commands),
|
||||
- the histograms (u32le counts, without the C code's `total_count_` and
|
||||
`bit_cost_` fields, which are not needed here): up to 260 literal
|
||||
histograms (`(256 / 13) + 1` block types of 13 contexts), 257 command
|
||||
histograms and 257 distance histograms of 256, 704 and 544 counts,
|
||||
- the block splitters' `combined_histo` (2 * 13 * 256 u32le counts),
|
||||
- the literal context map (256 * 64 bytes),
|
||||
- `EncodeContextMap`'s `rle_symbols` (256 * 64 u32le values) and
|
||||
- the block encoders' depths (u8) and then bits (u16le) for up to 256
|
||||
histograms per category.
|
||||
|
||||
Unlike the C code, which allocates its storage and command buffers as it
|
||||
goes (for the actual metablock and block sizes), `workbuf_len` is the worst
|
||||
case for the sanitized parameters, so that one workbuf works for any input.
|
||||
At qualities 2 to 9, `transform_io` returns `"#bad workbuf length"` if the
|
||||
workbuf is shorter than that. Some examples (in MiB, with the default
|
||||
lgblock and an unset size hint):
|
||||
|
||||
| quality | lgwin 10 | lgwin 16 | lgwin 18 | lgwin 22 | lgwin 24 |
|
||||
| ------- | -------- | -------- | -------- | -------- | -------- |
|
||||
| 0, 1 | 0.0 | 0.1 | 0.5 | 8.0 | 32.0 |
|
||||
| 2, 3 | 0.5 | 1.0 | 2.1 | 24.6 | 64.6 |
|
||||
| 4 | 7.9 | 7.9 | 9.8 | 47.3 | 103.4 |
|
||||
| 5 | 4.4 | 4.4 | 7.0 | 45.9 | 102.0 |
|
||||
| 7 | 4.4 | 4.4 | 13.8 | 51.4 | 107.5 |
|
||||
| 9 | 5.1 | 5.1 | 39.5 | 77.1 | 133.2 |
|
||||
|
||||
Fixed-size state (the hash table and command and literal buffers of
|
||||
qualities 0 and 1, the Huffman scratch space, the greedy block splitters'
|
||||
state, the block encoders' block type and length codes, etc) lives in the `encoder` struct (which is therefore
|
||||
about 2.5 MiB). It does not need zero-initializing:
|
||||
`WUFFS_INITIALIZE__LEAVE_INTERNAL_BUFFERS_UNINITIALIZED` is fine. At
|
||||
qualities 0 and 1, the input block is not copied: the encoder compresses
|
||||
directly from `src`. At qualities 2 to 9, like the C code, the input is
|
||||
copied into the ring buffer.
|
||||
|
||||
|
||||
### Encoder Operation Mapping
|
||||
@@ -304,23 +412,53 @@ on how the input is split into chunks. The test programs' encode drivers
|
||||
`mimic_brotli_encode` in `test/c/mimiclib/brotli.c`) run the same "plan" of
|
||||
(operation, length) steps with each library.
|
||||
|
||||
At qualities 2 to 9 the C code copies the input into its ring buffer and
|
||||
only compresses it (`EncodeData`) when an input block (`1 << lgblock` bytes)
|
||||
is full, or on a flush or finish, so the output depends on the chunking only
|
||||
through the flushes (and the `UpdateSizeHint` estimate, which depends on how
|
||||
much input is available when the first block is compressed). The Wuffs
|
||||
encoder makes the same `EncodeData` calls, at the same points. One
|
||||
`transform_io` call can correspond to several C calls: with a stream offset
|
||||
(the C code's "flint"), a C call can return early, with input left, after
|
||||
emitting the flint, and its callers then call it again.
|
||||
|
||||
|
||||
### Encoder Testing
|
||||
|
||||
`wuffs test -mimic std/brotli` checks byte-exact parity with libbrotlienc
|
||||
over a grid of qualities, lgwins, modes, input chunkings, flushes, metadata
|
||||
blocks, size hints and other parameters, as well as round trips through the
|
||||
decoder. `script/manual-test-brotli-encoder-parity.c` runs a larger grid
|
||||
(about 86 thousand configurations over every file in `test/data/brotli` and
|
||||
some synthetic inputs).
|
||||
over a grid of qualities (0 to 9), lgwins, modes, input chunkings, flushes,
|
||||
metadata blocks, size hints and other parameters (lgblock, npostfix, ndirect,
|
||||
disabling literal context modeling, stream offset, large window at quality 2),
|
||||
as well as round trips through the decoder.
|
||||
`script/manual-test-brotli-encoder-parity.c` runs a larger grid over every
|
||||
file in `test/data/brotli`, some larger files in `test/data` (up to about
|
||||
400 KiB) and some synthetic inputs (up to 1 MiB). Its `-q=LO-HI` and
|
||||
`-shard=K/N` options select the qualities and split the inputs over several
|
||||
processes. For qualities 0 and 1 that is about 86 thousand configurations
|
||||
and for qualities 2 to 9 several hundred thousand, all byte-exact.
|
||||
|
||||
|
||||
### Encoder Differences from the C Code
|
||||
|
||||
- The input block is not copied (see the workbuf layout above), and output is
|
||||
always staged in the workbuf (or a small internal buffer) and then copied to
|
||||
`dst`. The C code sometimes writes directly to `next_out`. The bytes are the
|
||||
same.
|
||||
- At qualities 0 and 1, the input block is not copied (see the workbuf
|
||||
layout above). Output is always staged in the workbuf (or a small internal
|
||||
buffer) and then copied to `dst`. The C code sometimes writes directly to
|
||||
`next_out`. The bytes are the same.
|
||||
- The C code instantiates each hasher (and the block splitters) from
|
||||
templates. Here, each template is one set of functions, parameterized at
|
||||
run time (e.g. by the hasher type). The H58 and H68 hashers' SIMD tag
|
||||
matching (`GetMatchingTagMask` in `matching_tag_mask.h`) is computed one
|
||||
tag at a time, which finds the same candidates in the same order. Like the
|
||||
x86-64 GCC reference build (where `BROTLI_MAX_SIMD_QUALITY` is 6, as in
|
||||
any build with a 64-bit `__builtin_ctzll`), qualities 5 and 6 with lgwin
|
||||
above 16 use H58 or H68 instead of H5 or H6. libbrotlienc builds where
|
||||
`BROTLI_MAX_SIMD_QUALITY` is undefined (no 64-bit count-trailing-zeros
|
||||
builtin) or 7 (MSVC on x86-64) choose differently and so produce
|
||||
different (but valid) output at those qualities.
|
||||
- `BrotliBuildMetaBlockGreedy`'s two kinds of block splitter (one for a
|
||||
single histogram per block type, `metablock_inc.h`, and one for several
|
||||
literal contexts, `ContextBlockSplitter` in `metablock.c`) are one
|
||||
implementation here, as they make the same decisions with one context.
|
||||
- Wuffs' C code generator does not support `slice base.u16` or
|
||||
`slice base.u32` arguments (and mishandles `copy_from_slice` on `base.u16`
|
||||
arrays), so some functions (e.g. `create_huffman_tree`,
|
||||
@@ -332,9 +470,11 @@ some synthetic inputs).
|
||||
|
||||
### Encoder TODO
|
||||
|
||||
- Qualities 2 to 11 (the C code's `EncodeData`, hashers, block splitting,
|
||||
context modeling, Zopfli, etc).
|
||||
- Large-window encoding (only used at qualities 3 and above).
|
||||
- Qualities 10 and 11 (Zopfli, the H10 binary tree hasher and
|
||||
`BrotliBuildMetaBlock`, which is already ported in `encode_metablock.wuffs`).
|
||||
- Large-window encoding at qualities 3 and above (it works at quality 2).
|
||||
- `workbuf_len` is a worst case. A future option could take the input length
|
||||
(like the C code's size hint) to give a tighter bound.
|
||||
- Custom and shared dictionaries (`BrotliEncoderAttachPreparedDictionary`).
|
||||
- Changing parameters mid-stream.
|
||||
|
||||
|
||||
@@ -19,14 +19,15 @@
|
||||
// The C code's floating point (double) arithmetic is done with the
|
||||
// std/softfloat package, which is bit-exact. FastLog2's log2 matches glibc's.
|
||||
//
|
||||
// Only qualities 0 and 1 (the C code's BrotliCompressFragmentFast and
|
||||
// BrotliCompressFragmentTwoPass) are implemented so far. See README.md for
|
||||
// the API, including how the C code's FLUSH and EMIT_METADATA operations map
|
||||
// to the request_flush and request_emit_metadata methods.
|
||||
// Qualities 0 and 1 (the C code's BrotliCompressFragmentFast and
|
||||
// BrotliCompressFragmentTwoPass) and qualities 2 to 9 (the C code's
|
||||
// EncodeData, see encode_data.wuffs) are implemented so far. See README.md
|
||||
// for the API, including how the C code's FLUSH and EMIT_METADATA operations
|
||||
// map to the request_flush and request_emit_metadata methods.
|
||||
//
|
||||
// TODO: qualities 2 to 11 (the C code's EncodeData, WriteMetaBlockInternal,
|
||||
// hashers, block splitting, context modeling and Zopfli), large-window
|
||||
// brotli at those qualities and custom (prepared or shared) dictionaries.
|
||||
// TODO: qualities 10 and 11 (Zopfli and BrotliBuildMetaBlock), large-window
|
||||
// brotli at qualities 3 and above and custom (prepared or shared)
|
||||
// dictionaries.
|
||||
|
||||
use "std/softfloat"
|
||||
|
||||
|
||||
Reference in new issue
Block a user