334 Commits
Author SHA1 Message Date
Claude fbae955a74 std/brotli: speed up qualities 0 to 9 and the decoder
Encoder (output stays byte-exact with libbrotlienc v1.2.1):
- H58/H68 (q5-6): match the tags 8 at a time (SWAR, like
  GetMatchingTagMask's non-SSE2 code) and visit the matches with a de
  Bruijn ctz; inline the bucket entry check into the H5/H6 and H58/H68
  loops, caching the 4 bytes compared at best_len; a dedicated H58/H68
  store and hash.
- find_last_distances caches the current byte at best_len.
- ctz64 and a branch-free log2_floor_non_zero (de Bruijn tables), and
  find_match_length_with_limit uses ctz.
- load32le, load64le, peek_u16, poke_u16 and poke_u32 use one bounds
  check each.

Decoder (same output and statuses):
- The literal fast path keeps pos and the counts in local variables.
- Copies of at most 16 bytes, with a distance of at least 16, copy 16
  bytes (like the C code; the window gap keeps the extra bytes unread).

script/manual-test-brotli-float.c also checks log2_floor_non_zero (every
u32) and ctz64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZ978BikvkegL2JUkqoB1e
2026-10-28 08:50:15 +00:00
Claude 044437a8ec std/brotli: speed up the encoder at qualities 10 and 11, byte-exactly
- encode_float.wuffs: package-private fast paths for the f32/f64 add, sub,
  mul, compare and int conversion calls in the encoder's hot loops. Calling
  std/softfloat is a cross-package public call (a receiver check, never
  inlined). Each fast path returns exactly the softfloat result and falls
  back to it for the uncommon cases. script/manual-test-brotli-float.c
  cross-checks them against hardware float and double arithmetic.
- Zopfli update_nodes: compute a length's cost only when its copy code
  changes, hoist the distance cache values and the continuation byte, and
  use a table for small int to f32 conversions.
- FindBlocks: memoize (block_switch_bitcost + insert_cost) for capped costs
  and skip the subtraction for costs at or above a per-byte threshold
  (S(c) = c - min_cost is monotone, so such costs would be capped anyway).
- BrotliPopulationCost: per-call memo of each count's term and depth, and
  skip the zero terms of the entropy sums.
- std/softfloat: a 64-bit f32_add_mags.

Output is unchanged (mimic tests on gcc and clang, parity sweeps).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZ978BikvkegL2JUkqoB1e
2026-10-27 23:21:31 +00:00
Claude bc1b5fcd49 std/brotli: add QUIRK_REPORT_METADATA
The decoder can now report metadata blocks, like libbrotlidec's
BrotliDecoderSetMetadataCallbacks: transform_io returns
"@metadata reported", metadata_length gives the block's length and the
block's contents are written to dst. The differential fuzzer compares
the reported metadata with libbrotlidec's callbacks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UZ978Fjdsyig4v3wYAZRg5
2026-10-27 17:06:08 +00:00
Claude 8dcecc67af std/brotli: document the uninitialized ring buffer tail reads
libbrotlienc can read the uninitialized tail of a ring buffer that it grew
after a short first write, and the Wuffs encoder reads the corresponding
workbuf bytes, so both outputs can depend on memory contents (and agree
when the memory does). The fuzzer reruns a mismatch with zeroed memory for
both encoders and counts it as uninit_tail when the outputs then agree and
Wuffs' output decodes.
2026-10-27 16:47:03 +00:00
Claude 73de2fe182 Merge claude/project-thread-0izcgq (speed work) 2026-10-27 15:13:58 +00:00
Claude a856d133d9 script: hash the seed in the brotli fuzzer
Seeding SplitMix64 with seed * 0x9E3779B97F4A7C15 made consecutive seeds
generate the same sequence shifted by one number, so shards with
consecutive seeds repeated each other's cases.
2026-10-27 14:01:14 +00:00
Claude 8a8df6667a std/brotli: document and test a libbrotlienc metadata stale pointer bug
At qualities 0 and 1, a metadata block whose body went through the C
encoder's tiny_buf_ leaves next_out_ pointing at remaining_metadata_bytes_,
and a later flush's padding bits overwrite it, so every later
BrotliEncoderCompressStream call fails. Wuffs produces the same bytes and
does not mirror the failure. The fuzzer recognizes the corrupted C state.
2026-10-27 13:25:23 +00:00
Claude 0e0674d62b script: no Wuffs-only zero-length dst calls with a stream offset in the brotli fuzzer's API mode 2026-10-27 12:57:24 +00:00
Claude 8ce9c3f552 script: fuzz encoder dictionaries at qualities 10 and 11 2026-10-27 12:41:05 +00:00
Claude 24807c63d1 Merge compound dictionaries at qualities 10 and 11 into m7a-speed 2026-10-27 12:40:14 +00:00
Claude c5c1c00ba6 Merge claude/project-thread-0izcgq (q10/q11 dictionaries) 2026-10-27 12:38:06 +00:00
Claude 0c238497fb script: accept the stream offset flush failure in the brotli fuzzer when both encoders agree 2026-10-27 10:57:15 +00:00
Claude 4dff581dd0 script: check the softfloat comparisons in manual-test-softfloat-sweep.c
Add f32_compare and f64_compare, which run the cmp, eq, ne, lt, le, gt,
ge, min and max functions on every pair of 594 structured values
against the hardware's comparison operators.
2026-10-27 10:43:02 +00:00
Claude 1d1c338981 std/softfloat: add fast paths for f32 and f64 add, sub and mul
f32_add, f32_sub and f32_mul now compute the result directly in binary32
when both operands are normal and the result is normal (or overflows),
instead of always going through f64. f64_add and f64_sub get a similar
fast path for normal operands. Every other case takes the previous code
path, so results (including NaN payloads) are unchanged.

Add script/manual-test-softfloat-sweep.c, which checks the f32 and f64
add, sub, mul and div operations over every pair of operand exponents
with many mantissa patterns, and the f32 conversions, against the
hardware and (bit-for-bit, NaNs included) the via-f64 definitions.
2026-10-27 09:26:09 +00:00
Claude d89864223e script: add replay and API edge case modes to the brotli fuzzer 2026-10-27 09:12:22 +00:00
Claude f6798c006d script: use the smallest dictionary grid at q10/q11 for big inputs
Like the regular grid, qualities 10 and 11 use the smallest dictionary
configuration set for inputs longer than 64 KiB.
2026-10-27 09:08:15 +00:00
Claude ef6ac20062 script: add the encoder mode to the differential brotli fuzzer 2026-10-27 09:00:11 +00:00
Claude 7191bab983 script: add a differential brotli fuzzer (decoder mode) 2026-10-27 08:54:18 +00:00
Claude 3f60ead4fc script: allow encoder parity workbufs of up to 12 GiB, link libbrotlidec 2026-10-27 06:10:02 +00:00
Claude ed7afaeab9 Merge claude/project-thread-0izcgq (encoder raw dictionaries)
At qualities 10 and 11, attaching a dictionary makes transform_io
return "#unsupported encoder quality" until the Zopfli path's compound
dictionary lookups are ported.
2026-10-27 06:06:39 +00:00
Claude 36a4380062 std/brotli: test encoder dictionaries and document them
Add mimic_brotli_encode_with_dictionaries to the mimic library and
encoder dictionary tests: the add_dictionary API, its workbuf
accounting and rejection cases (matching libbrotlienc's default build),
a parity grid over qualities 0 to 9 with several dictionary sets, stream
offsets, large window, flushes and metadata, and dictionaries of 2 to
33 MiB, each round tripped through both decoders.

Document the encoder's dictionary API, workbuf layout and differences
from the C code.
2026-10-27 05:28:37 +00:00
Claude aabdeaf4b0 std/brotli: add encoder raw (prefix) dictionaries
Mirror BrotliEncoderPrepareDictionary (BROTLI_SHARED_DICTIONARY_RAW),
CreatePreparedDictionary and BrotliEncoderAttachPreparedDictionary as
encoder.add_dictionary, with the prepared hash tables (slot offsets,
heads and items) in the workbuf after the dictionary bytes, and the
compound dictionary search (FindCompoundDictionaryMatch and
LookupCompoundDictionaryMatch) for the hashers whose
CreateBackwardReferences variant enables it, the compound dictionary
gap for static dictionary distances and the compound branch of
ExtendLastCommand.

Extend the encoder parity script with a -dict grid that also checks
round trips through libbrotlidec and the Wuffs decoder.
2026-10-27 04:54:39 +00:00
Claude 7d58caf91c Merge claude/project-thread-0izcgq (large window at qualities 3 to 9)
Put the q10/q11 variable region before the ring buffer (now the last
region), growing it (and moving the ring buffer) at the first
transform_io call to use a workbuf longer than workbuf_len's minimum.
Large window now also works at qualities 10 and 11.
2026-10-27 04:46:52 +00:00
Claude 2236afbd96 script: add a -oneshot option to the encoder parity harness 2026-10-27 03:13:45 +00:00
Claude 3669090209 script: use a smaller encoder parity grid for big q10/q11 inputs 2026-10-27 03:09:29 +00:00
Claude bef6ffc1bd std/brotli: fix the Zopfli starting distance cache order
Also run the encoder parity harness at qualities up to 11, using the
workbuf_len minimum.
2026-10-27 02:43:46 +00:00
Claude 0b9cfddf3c std/brotli: let a shorter encoder workbuf work for short inputs
Move the ring buffer to the end of the workbuf layout (at qualities 2 to
9). A workbuf shorter than workbuf_len now works if it holds the input
so far (plus 16 bytes of slack) in the ring buffer region and the input
stays short enough for the ring buffer never to wrap. This matters most
for large-window brotli, where workbuf_len at lgwin 30 is more than
2 GiB, mostly for a ring buffer that short inputs never touch.

The parity harness now also checks its large-window configurations with
the shortest such workbuf (and that one byte less fails).
2026-10-27 02:36:23 +00:00
Claude 152ad1d2dd Merge branch 'claude/project-thread-0izcgq' into m5a-zopfli 2026-10-27 02:36:09 +00:00
Claude 361091fee6 std/brotli: add quality 10 and 11 (Zopfli) encoder, untested
Port H10 (hash_to_binary_tree_inc.h), literal_cost.c,
backward_references_hq.c, the static dictionary match finder
(static_dict.c) and the quality 10+ parts of the metablock writer,
using the metablock_builder. The variable-size Zopfli and
metablock_builder scratch lives in a new workbuf layout region.
2026-10-27 02:36:05 +00:00
Claude fd89148c0b Merge branch 'claude/project-thread-0izcgq' into m5b-large-window 2026-10-27 02:30:38 +00:00
Claude 1560aeff5a Merge branch 'm4b-metablock' into claude/project-thread-0izcgq 2026-10-27 02:24:52 +00:00
Claude 75b09b6700 script: add -big inputs with far repeats to the encoder parity test
The -big option checks synthetic inputs of 3 to 70 MiB with repeats at
distances of up to 40 MiB (beyond the 16 MiB non-large-window limit),
with the usual grid up to 12 MiB and a few one-shot and streaming
large-window configurations above that.
2026-10-27 02:21:31 +00:00
Claude 8b0c3204a0 std/brotli: add large-window encoding at qualities 3 to 9
Port the composite hashers H35, H55 and H65 (hash_composite_inc.h) and
the rolling hashers HROLLING_FAST and HROLLING (hash_rolling_inc.h),
which ChooseHasher picks for lgwin above 24, and drop the
"#unsupported encoder quality" gate for the large-window quirk at
qualities 3 to 9. The workbuf's hasher region now includes the rolling
hasher's 64 MiB table when a composite hasher can be chosen.

Extend the encoder parity harness with a large-window grid (lgwin 10 to
30 at qualities 2 to 9) and a -lw option that runs only that grid.
2026-10-27 02:13:21 +00:00
Claude 132445a5e6 script: keep metablock parity harness distances valid
Clamp the synthetic commands' distance codes to the distance params'
max_distance, as the encoder does. Larger (invalid) distances made the C
code's ComputeDistanceCost write past the end of its distance histogram,
which showed up as a spurious mismatch for a large-window configuration.
2026-10-27 01:26:30 +00:00
Claude abecc768e5 Merge branch 'm4b-metablock' into m4a-encoder-core
# Conflicts:
#	release/c/wuffs-unsupported-snapshot.c
2026-10-27 01:10:29 +00:00
Claude e66b315284 std/brotli: fix the UTF-8 prefix LUT; extend the encoder parity test to qualities 2 to 9 2026-10-27 01:04:57 +00:00
Claude 9ad7ae6d69 std/brotli: add metablock_builder.workbuf_len_worst_case
Give the metablock builder an upper bound on the workbuf length that
build_meta_block needs, expose it through the metablock_tester and check
it against the high water mark in the parity harness. Add shard options
(-largeoff, -edge, -manytypes) and a stress phase that reaches 256 block
types to the harness.
2026-10-27 01:03:11 +00:00
Claude 09d4b23d09 std/brotli: WIP encoder qualities 2 to 9 (EncodeData, hashers, greedy metablocks) 2026-10-27 00:58:17 +00:00
Claude 202b406248 std/brotli: port block_splitter.c, cluster.c and metablock.c
Add the metablock_builder struct, mirroring BrotliSplitBlock (with its
MyRand sampling, InitialEntropyCodes, RefineEntropyCodes, FindBlocks,
RemapBlockIds, BuildBlockHistograms and ClusterBlocks), the histogram
clustering functions (BrotliCompareAndPushToQueue,
BrotliHistogramCombine, BrotliHistogramBitCostDistance,
BrotliHistogramRemap, BrotliHistogramReindex, BrotliClusterHistograms),
BrotliInitDistanceParams, BrotliBuildMetaBlock (including the distance
parameter search) and BrotliOptimizeHistograms (with
BrotliOptimizeHuffmanCountsForRle). All memory that the C code
allocates lives in a caller-supplied workbuf, carved out by a bump
allocator.

bit_cost gets an optional 64Ki-entry FastLog2 cache (filled by the
same softfloat log2), which makes the clustering several times faster.

Extend the metablock_tester and the parity harness to compare
SplitBlock, ClusterHistograms*, BuildMetaBlock and OptimizeHistograms
results with libbrotlienc v1.2.1.
2026-10-27 00:52:42 +00:00
Claude 88c32bc668 std/brotli: port bit_cost.c and histogram.c for the encoder
Add the bit_cost struct (encode_bit_cost.wuffs, encode_histogram.wuffs):
BrotliBitsEntropy, BrotliPopulationCost{Literal,Command,Distance},
FastLog2, the histogram helpers and BrotliBuildHistogramsWithContext,
with doubles computed by std/softfloat in the C code's operation order.
Histograms and commands are little-endian byte slices with the C structs'
memory layout.

Add a test-only metablock_tester struct and the
script/manual-test-brotli-metablock-parity.c harness, which compares
BitsEntropy and PopulationCost* bit for bit with libbrotlienc v1.2.1
(200000 random histograms: 0 mismatches).
2026-10-27 00:16:37 +00:00
Claude ca1ba9d134 std/brotli: add an encoder for quality levels 0 and 1
Adds a brotli encoder (std/brotli's encoder struct, a base.io_transformer)
whose output is byte-identical to libbrotlienc v1.2.1 for quality levels 0
and 1, mirroring encode.c's BrotliEncoderCompressStream(Fast),
compress_fragment.c and compress_fragment_two_pass.c, plus the entropy
coding and bit stream helpers they need. Every double operation goes
through std/softfloat, following the C evaluation order exactly.

Parameters are quirks (encode_quirks.wuffs) that mirror
BrotliEncoderSetParameter, sanitized at the first transform_io call.
Each transform_io resumption corresponds to one C call with available_in
equal to the readable src length; a closed src is FINISH; request_flush!
and request_emit_metadata! make the next resumption a FLUSH or
EMIT_METADATA call. Qualities 2 to 11 return "#unsupported encoder
quality" for now. Encoder tables are generated by
script/print-brotli-tables.go -encoder.

Tests: round trips through the Wuffs decoder over the corpus and synthetic
inputs; with -mimic, byte-exact comparisons against libbrotlienc over
quality × window × mode × input chunking × flush schedule × metadata ×
size hint. script/manual-test-brotli-encoder-parity.c runs the full
grid: 86,268 configurations, zero mismatches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:55:56 +00:00
Claude 4978207531 std/brotli: add a brotli decoder with parity to libbrotlidec v1.2.1
This is the first milestone of a byte-exact rewrite of brotli (reference
implementation tag v1.2.1) in Wuffs. The decoder mirrors c/dec/decode.c
function by function, including its validation checks: the same output for
valid input and the same verdict, with each BROTLI_DECODER_ERROR_FORMAT_*
code mapped to a Wuffs status (see std/brotli/README.md). The C state
machine is replaced by Wuffs coroutines, so decoding works with arbitrarily
small src and dst buffers.

The history window lives in the caller's workbuf. Huffman table groups,
context maps and the static dictionary scratch space live in the decoder
struct. The static dictionary, transforms, context lookup tables and prefix
code tables in common_tables.wuffs are generated from the brotli source by
script/print-brotli-tables.go.

Tests: golden tests on the upstream tests/testdata vectors and on streams
produced by the v1.2.1 CLI at several qualities and window sizes, 1-byte
I/O limits, truncation at every prefix, malformed inputs per error code,
trailing data and workbuf boundaries. With -mimic, every vector and every
malformed input is also compared against libbrotlidec, including the exact
error code. A differential run over about a million mutated inputs found
no divergence.

Not yet supported (left as TODOs): large-window brotli, custom/shared
dictionaries, metadata callbacks and fast paths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 22:46:34 +00:00
Claude f33e9f3377 std/softfloat: add bit-exact IEEE 754 binary64/binary32 arithmetic
Wuffs the Language has no floating point types, but some reference
implementations (notably the brotli encoder) make decisions with double and
float arithmetic and libm's log2. Porting them with byte-identical output
needs bit-exact floating point results.

This package implements f64 and f32 add, sub, mul, div, fma, comparisons,
min/max, integer conversions and f64<->f32 conversions on bit patterns,
with round-to-nearest-even, subnormals, infinities and NaNs, following the
structure of Berkeley SoftFloat 3. It also ports glibc 2.39's x86-64
__log2_fma log2 implementation (the variant selected on CPUs with FMA and
AVX2), following the shipped machine code's fused operations.

Verification: test/c/std/softfloat.c checks every operation against the C
compiler's native arithmetic on edge cases and random operands. The
script/manual-test-softfloat-*.c programs run 10^8 random operations per
operation group and compare log2_u64 against libm for every integer in
[1, 2^32) plus 10^7 larger values; all with zero mismatches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 22:46:32 +00:00
Nigel Tao c677bef7fd std/png: parse cICP chunk
Updates #180
2026-10-18 18:13:59 +11:00
Nigel Tao 1ca418ef16 snippet/uncompng: support APNG (Animated PNG) 2026-09-15 10:42:51 +10:00
Nigel Tao 2f7e714ab0 script: add print-png-chunks.go 2026-08-12 21:10:36 +10:00
Nigel Tao b89fd2fa03 script: undo print-nia-checksums.sh comment
This was accidentally commented out by "std/vp8: blend VP8X ALPH chunk",
commit d9cf0751ca.
2026-08-10 17:38:42 +10:00
Nigel Tao aad70fdfa5 script: print-riff-chunks.go handles WEBP/ANMF 2026-08-10 16:55:56 +10:00
Nigel Tao c4df79d82e script: add print-riff-chunks.go 2026-08-07 16:33:25 +10:00
Nigel Tao 5d5fb1d6d9 base38: add "BCP 47 as base38" example 2026-08-06 14:57:00 +10:00