Mirror BrotliEncoderPrepareDictionary (BROTLI_SHARED_DICTIONARY_RAW),
CreatePreparedDictionary and BrotliEncoderAttachPreparedDictionary as
encoder.add_dictionary, with the prepared hash tables (slot offsets,
heads and items) in the workbuf after the dictionary bytes, and the
compound dictionary search (FindCompoundDictionaryMatch and
LookupCompoundDictionaryMatch) for the hashers whose
CreateBackwardReferences variant enables it, the compound dictionary
gap for static dictionary distances and the compound branch of
ExtendLastCommand.
Extend the encoder parity script with a -dict grid that also checks
round trips through libbrotlidec and the Wuffs decoder.
Move the ring buffer to the end of the workbuf layout (at qualities 2 to
9). A workbuf shorter than workbuf_len now works if it holds the input
so far (plus 16 bytes of slack) in the ring buffer region and the input
stays short enough for the ring buffer never to wrap. This matters most
for large-window brotli, where workbuf_len at lgwin 30 is more than
2 GiB, mostly for a ring buffer that short inputs never touch.
The parity harness now also checks its large-window configurations with
the shortest such workbuf (and that one byte less fails).
The -big option checks synthetic inputs of 3 to 70 MiB with repeats at
distances of up to 40 MiB (beyond the 16 MiB non-large-window limit),
with the usual grid up to 12 MiB and a few one-shot and streaming
large-window configurations above that.
Port the composite hashers H35, H55 and H65 (hash_composite_inc.h) and
the rolling hashers HROLLING_FAST and HROLLING (hash_rolling_inc.h),
which ChooseHasher picks for lgwin above 24, and drop the
"#unsupported encoder quality" gate for the large-window quirk at
qualities 3 to 9. The workbuf's hasher region now includes the rolling
hasher's 64 MiB table when a composite hasher can be chosen.
Extend the encoder parity harness with a large-window grid (lgwin 10 to
30 at qualities 2 to 9) and a -lw option that runs only that grid.
Clamp the synthetic commands' distance codes to the distance params'
max_distance, as the encoder does. Larger (invalid) distances made the C
code's ComputeDistanceCost write past the end of its distance histogram,
which showed up as a spurious mismatch for a large-window configuration.
Give the metablock builder an upper bound on the workbuf length that
build_meta_block needs, expose it through the metablock_tester and check
it against the high water mark in the parity harness. Add shard options
(-largeoff, -edge, -manytypes) and a stress phase that reaches 256 block
types to the harness.
Add the metablock_builder struct, mirroring BrotliSplitBlock (with its
MyRand sampling, InitialEntropyCodes, RefineEntropyCodes, FindBlocks,
RemapBlockIds, BuildBlockHistograms and ClusterBlocks), the histogram
clustering functions (BrotliCompareAndPushToQueue,
BrotliHistogramCombine, BrotliHistogramBitCostDistance,
BrotliHistogramRemap, BrotliHistogramReindex, BrotliClusterHistograms),
BrotliInitDistanceParams, BrotliBuildMetaBlock (including the distance
parameter search) and BrotliOptimizeHistograms (with
BrotliOptimizeHuffmanCountsForRle). All memory that the C code
allocates lives in a caller-supplied workbuf, carved out by a bump
allocator.
bit_cost gets an optional 64Ki-entry FastLog2 cache (filled by the
same softfloat log2), which makes the clustering several times faster.
Extend the metablock_tester and the parity harness to compare
SplitBlock, ClusterHistograms*, BuildMetaBlock and OptimizeHistograms
results with libbrotlienc v1.2.1.
Add the bit_cost struct (encode_bit_cost.wuffs, encode_histogram.wuffs):
BrotliBitsEntropy, BrotliPopulationCost{Literal,Command,Distance},
FastLog2, the histogram helpers and BrotliBuildHistogramsWithContext,
with doubles computed by std/softfloat in the C code's operation order.
Histograms and commands are little-endian byte slices with the C structs'
memory layout.
Add a test-only metablock_tester struct and the
script/manual-test-brotli-metablock-parity.c harness, which compares
BitsEntropy and PopulationCost* bit for bit with libbrotlienc v1.2.1
(200000 random histograms: 0 mismatches).
Adds a brotli encoder (std/brotli's encoder struct, a base.io_transformer)
whose output is byte-identical to libbrotlienc v1.2.1 for quality levels 0
and 1, mirroring encode.c's BrotliEncoderCompressStream(Fast),
compress_fragment.c and compress_fragment_two_pass.c, plus the entropy
coding and bit stream helpers they need. Every double operation goes
through std/softfloat, following the C evaluation order exactly.
Parameters are quirks (encode_quirks.wuffs) that mirror
BrotliEncoderSetParameter, sanitized at the first transform_io call.
Each transform_io resumption corresponds to one C call with available_in
equal to the readable src length; a closed src is FINISH; request_flush!
and request_emit_metadata! make the next resumption a FLUSH or
EMIT_METADATA call. Qualities 2 to 11 return "#unsupported encoder
quality" for now. Encoder tables are generated by
script/print-brotli-tables.go -encoder.
Tests: round trips through the Wuffs decoder over the corpus and synthetic
inputs; with -mimic, byte-exact comparisons against libbrotlienc over
quality × window × mode × input chunking × flush schedule × metadata ×
size hint. script/manual-test-brotli-encoder-parity.c runs the full
grid: 86,268 configurations, zero mismatches.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
This is the first milestone of a byte-exact rewrite of brotli (reference
implementation tag v1.2.1) in Wuffs. The decoder mirrors c/dec/decode.c
function by function, including its validation checks: the same output for
valid input and the same verdict, with each BROTLI_DECODER_ERROR_FORMAT_*
code mapped to a Wuffs status (see std/brotli/README.md). The C state
machine is replaced by Wuffs coroutines, so decoding works with arbitrarily
small src and dst buffers.
The history window lives in the caller's workbuf. Huffman table groups,
context maps and the static dictionary scratch space live in the decoder
struct. The static dictionary, transforms, context lookup tables and prefix
code tables in common_tables.wuffs are generated from the brotli source by
script/print-brotli-tables.go.
Tests: golden tests on the upstream tests/testdata vectors and on streams
produced by the v1.2.1 CLI at several qualities and window sizes, 1-byte
I/O limits, truncation at every prefix, malformed inputs per error code,
trailing data and workbuf boundaries. With -mimic, every vector and every
malformed input is also compared against libbrotlidec, including the exact
error code. A differential run over about a million mutated inputs found
no divergence.
Not yet supported (left as TODOs): large-window brotli, custom/shared
dictionaries, metadata callbacks and fast paths.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
Wuffs the Language has no floating point types, but some reference
implementations (notably the brotli encoder) make decisions with double and
float arithmetic and libm's log2. Porting them with byte-identical output
needs bit-exact floating point results.
This package implements f64 and f32 add, sub, mul, div, fma, comparisons,
min/max, integer conversions and f64<->f32 conversions on bit patterns,
with round-to-nearest-even, subnormals, infinities and NaNs, following the
structure of Berkeley SoftFloat 3. It also ports glibc 2.39's x86-64
__log2_fma log2 implementation (the variant selected on CPUs with FMA and
AVX2), following the shipped machine code's fused operations.
Verification: test/c/std/softfloat.c checks every operation against the C
compiler's native arithmetic on edge cases and random operands. The
script/manual-test-softfloat-*.c programs run 10^8 random operations per
operation group and compare log2_u64 against libm for every integer in
[1, 2^32) plus 10^7 larger values; all with zero mismatches.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
It changes from " 0-9?a-z" to ".0-9a-z~". The default or placeholder
changes from " " to "." so it's now always visible. The wildcard changes
from "?" to "~" so it's now sorted last.