Mirror the Zopfli path's compound dictionary code in
backward_references_hq.c and hash.h: LookupAllCompoundDictionaryMatches,
FindAllCompoundDictionaryMatches, MergeMatches, the lz_matches_offset
and shadow_matches scratch space, the gap in ComputeDistanceShortcut,
UpdateNodes (including its compound dictionary last distance branch)
and BrotliZopfliCreateCommands, and the dictionary distance passed to
FindAllMatchesH10.
Remove the quality 10 and 11 dictionary gate (and its now unused
status), size the quality 11 variable region for the shadow matches
and turn the gate's test into a byte-exact check. The mimic dictionary
grid, big and rejections tests now also cover qualities 10 and 11.
At qualities 10 and 11, attaching a dictionary makes transform_io
return "#unsupported encoder quality" until the Zopfli path's compound
dictionary lookups are ported.
Add mimic_brotli_encode_with_dictionaries to the mimic library and
encoder dictionary tests: the add_dictionary API, its workbuf
accounting and rejection cases (matching libbrotlienc's default build),
a parity grid over qualities 0 to 9 with several dictionary sets, stream
offsets, large window, flushes and metadata, and dictionaries of 2 to
33 MiB, each round tripped through both decoders.
Document the encoder's dictionary API, workbuf layout and differences
from the C code.
Put the q10/q11 variable region before the ring buffer (now the last
region), growing it (and moving the ring buffer) at the first
transform_io call to use a workbuf longer than workbuf_len's minimum.
Large window now also works at qualities 10 and 11.
workbuf_len's minimum now assumes at most (n / 16) + 1 blocks per
category and (n / 64) + 64 first-phase clusters (measured maxima are
about 1/83 and 1/258 of n), and 16 BackwardMatches per byte at
quality 11 (measured maximum about 7.4). The maximum stays the C
worst case. Add q10/q11 mimic tests and workbuf_len checks, and use
the minimum in the tests' encode plan driver.
Move the ring buffer to the end of the workbuf layout (at qualities 2 to
9). A workbuf shorter than workbuf_len now works if it holds the input
so far (plus 16 bytes of slack) in the ring buffer region and the input
stays short enough for the ring buffer never to wrap. This matters most
for large-window brotli, where workbuf_len at lgwin 30 is more than
2 GiB, mostly for a ring buffer that short inputs never touch.
The parity harness now also checks its large-window configurations with
the shortest such workbuf (and that one byte less fails).
Port H10 (hash_to_binary_tree_inc.h), literal_cost.c,
backward_references_hq.c, the static dictionary match finder
(static_dict.c) and the quality 10+ parts of the metablock writer,
using the metablock_builder. The variable-size Zopfli and
metablock_builder scratch lives in a new workbuf layout region.
Check large-window encoding at qualities 3 to 9 (lgwin 18 to 30, with
the H35, H55 and H65 composite hashers) against libbrotlienc, with round
trips through Wuffs' decoder (with the large-window quirk) and
libbrotlidec, over some small files and a 46 MiB input with repeats at
distances of about 20 and 40 MiB. Also check the large-window
workbuf_len and allow encoder workbufs of up to 3 GiB in the test
driver.
Bound the number of block types by the number of symbols in
workbuf_len_worst_case, document the metablock_builder interface and its
workbuf layout in README.md, and add deterministic bit cost and
build_meta_block tests (with golden values from libbrotlienc v1.2.1) to
test/c/std/brotli.c.
Combines the encoder (quality 0 and 1) with the decoder's large-window
mode and custom dictionaries. The test and mimic files take both sides'
new sections; the fuzzer gains the std/softfloat module that the
encoder now pulls into std/brotli.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
Adds a brotli encoder (std/brotli's encoder struct, a base.io_transformer)
whose output is byte-identical to libbrotlienc v1.2.1 for quality levels 0
and 1, mirroring encode.c's BrotliEncoderCompressStream(Fast),
compress_fragment.c and compress_fragment_two_pass.c, plus the entropy
coding and bit stream helpers they need. Every double operation goes
through std/softfloat, following the C evaluation order exactly.
Parameters are quirks (encode_quirks.wuffs) that mirror
BrotliEncoderSetParameter, sanitized at the first transform_io call.
Each transform_io resumption corresponds to one C call with available_in
equal to the readable src length; a closed src is FINISH; request_flush!
and request_emit_metadata! make the next resumption a FLUSH or
EMIT_METADATA call. Qualities 2 to 11 return "#unsupported encoder
quality" for now. Encoder tables are generated by
script/print-brotli-tables.go -encoder.
Tests: round trips through the Wuffs decoder over the corpus and synthetic
inputs; with -mimic, byte-exact comparisons against libbrotlienc over
quality × window × mode × input chunking × flush schedule × metadata ×
size hint. script/manual-test-brotli-encoder-parity.c runs the full
grid: 86,268 configurations, zero mismatches.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
Completes the decoder's feature set relative to libbrotlidec v1.2.1:
- QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the
large-window WBITS escape, window bits up to 30, the 544-symbol distance
alphabet and 64-bit distance arithmetic, with workbuf_len growing to
(1 << 30) + 542 when the quirk is set.
- decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw
(LZ77 prefix) and serialized shared dictionaries: compound-dictionary
distance mapping, the shared-dictionary parser (custom word lists,
transform lists including the SHIFT transforms, context-dependent
dictionary selection, prefix dictionaries) and all of its validation.
Wuffs structs cannot hold slices, so the dictionary bytes are copied
into the front of the caller's workbuf; the layout is documented in
the README. Serialized dictionaries are accepted as in a
BROTLI_EXPERIMENTAL build of libbrotli.
- fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and
random I/O limits.
Tests: new golden vectors for large windows (including a 17 MiB
far-distance stream generated in the test), raw and serialized
dictionaries, nine more malformed inputs, and mimic comparisons that
attach the same dictionaries to libbrotlidec. Differential runs over
2.7 million mutated large-window and dictionary streams found no
divergence from libbrotlidec.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
This is the first milestone of a byte-exact rewrite of brotli (reference
implementation tag v1.2.1) in Wuffs. The decoder mirrors c/dec/decode.c
function by function, including its validation checks: the same output for
valid input and the same verdict, with each BROTLI_DECODER_ERROR_FORMAT_*
code mapped to a Wuffs status (see std/brotli/README.md). The C state
machine is replaced by Wuffs coroutines, so decoding works with arbitrarily
small src and dst buffers.
The history window lives in the caller's workbuf. Huffman table groups,
context maps and the static dictionary scratch space live in the decoder
struct. The static dictionary, transforms, context lookup tables and prefix
code tables in common_tables.wuffs are generated from the brotli source by
script/print-brotli-tables.go.
Tests: golden tests on the upstream tests/testdata vectors and on streams
produced by the v1.2.1 CLI at several qualities and window sizes, 1-byte
I/O limits, truncation at every prefix, malformed inputs per error code,
trailing data and workbuf boundaries. With -mimic, every vector and every
malformed input is also compared against libbrotlidec, including the exact
error code. A differential run over about a million mutated inputs found
no divergence.
Not yet supported (left as TODOs): large-window brotli, custom/shared
dictionaries, metadata callbacks and fast paths.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
Wuffs the Language has no floating point types, but some reference
implementations (notably the brotli encoder) make decisions with double and
float arithmetic and libm's log2. Porting them with byte-identical output
needs bit-exact floating point results.
This package implements f64 and f32 add, sub, mul, div, fma, comparisons,
min/max, integer conversions and f64<->f32 conversions on bit patterns,
with round-to-nearest-even, subnormals, infinities and NaNs, following the
structure of Berkeley SoftFloat 3. It also ports glibc 2.39's x86-64
__log2_fma log2 implementation (the variant selected on CPUs with FMA and
AVX2), following the shipped machine code's fused operations.
Verification: test/c/std/softfloat.c checks every operation against the C
compiler's native arithmetic on edge cases and random operands. The
script/manual-test-softfloat-*.c programs run 10^8 random operations per
operation group and compare log2_u64 against libm for every integer in
[1, 2^32) plus 10^7 larger values; all with zero mismatches.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
"wuffs test -mimic std/webp" now passes.
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.
Updates PR #168
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications. There
were many stylistic changes but also:
- decoder.workbuf_len was fixed to account for all of the partitions,
not just the first two (the first partition and the first 'other'
partition). The PR's vp8 code, after an update to handle multiple
other partitions (which then implied random access to the partitioned
source bytes), assumed that enough workbuf was available (which it
often is, test/c/std/webp.c tests pass the entire g_work_slice_u8).
But strictly speaking, this isn't guaranteed unless workbuf_len
reports (after decode_image_config is called) what's needed.
- Calculating that workbuf_len (at decode_image_config time) lead to
introducing a new base.QUIRK_SOURCE_LENGTH quirk, to pass that
contextual information from std/webp to std/vp8. This calculation
belongs in std/vp8, not std/webp (or a future std/webm).
- Added support (and a test) for the case when the source bytes are
streamed and some (but not yet all) of the partition data in args.src
can be copied to args.workbuf.
Updates PR #168