Commit Graph
1268 Commits
Author SHA1 Message Date
Claude 152ad1d2dd Merge branch 'claude/project-thread-0izcgq' into m5a-zopfli 2026-10-27 02:36:09 +00:00
Claude 361091fee6 std/brotli: add quality 10 and 11 (Zopfli) encoder, untested
Port H10 (hash_to_binary_tree_inc.h), literal_cost.c,
backward_references_hq.c, the static dictionary match finder
(static_dict.c) and the quality 10+ parts of the metablock writer,
using the metablock_builder. The variable-size Zopfli and
metablock_builder scratch lives in a new workbuf layout region.
2026-10-27 02:36:05 +00:00
Claude 1560aeff5a Merge branch 'm4b-metablock' into claude/project-thread-0izcgq 2026-10-27 02:24:52 +00:00
Claude 8409ec2a5d std/brotli: record the encoder parity test's configuration count 2026-10-27 01:50:02 +00:00
Claude b10d57f20f Merge branch 'm4b-metablock' into m4a-encoder-core
# Conflicts:
#	std/brotli/README.md
2026-10-27 01:26:59 +00:00
Claude 1bd0d84299 std/brotli: document build_meta_block's distance precondition 2026-10-27 01:26:46 +00:00
Claude b9f9d6ce65 std/brotli: document the encoder at qualities 2 to 9 2026-10-27 01:20:26 +00:00
Claude 314b20e832 std/brotli: use metablock_builder's distance and RLE helpers in the encoder
This removes the encoder's copies of PrefixEncodeCopyDistance,
CommandRestoreDistanceCode, BrotliInitDistanceParams and
BrotliOptimizeHuffmanCountsForRle. The shared BrotliInitDistanceParams
also handles large-window distance params (used at quality 2).
2026-10-27 01:15:03 +00:00
Claude 5393d92234 std/brotli: document the metablock builder and add golden tests
Bound the number of block types by the number of symbols in
workbuf_len_worst_case, document the metablock_builder interface and its
workbuf layout in README.md, and add deterministic bit cost and
build_meta_block tests (with golden values from libbrotlienc v1.2.1) to
test/c/std/brotli.c.
2026-10-27 01:12:51 +00:00
Claude abecc768e5 Merge branch 'm4b-metablock' into m4a-encoder-core
# Conflicts:
#	release/c/wuffs-unsupported-snapshot.c
2026-10-27 01:10:29 +00:00
Claude e66b315284 std/brotli: fix the UTF-8 prefix LUT; extend the encoder parity test to qualities 2 to 9 2026-10-27 01:04:57 +00:00
Claude 9ad7ae6d69 std/brotli: add metablock_builder.workbuf_len_worst_case
Give the metablock builder an upper bound on the workbuf length that
build_meta_block needs, expose it through the metablock_tester and check
it against the high water mark in the parity harness. Add shard options
(-largeoff, -edge, -manytypes) and a stress phase that reaches 256 block
types to the harness.
2026-10-27 01:03:11 +00:00
Claude 09d4b23d09 std/brotli: WIP encoder qualities 2 to 9 (EncodeData, hashers, greedy metablocks) 2026-10-27 00:58:17 +00:00
Claude 202b406248 std/brotli: port block_splitter.c, cluster.c and metablock.c
Add the metablock_builder struct, mirroring BrotliSplitBlock (with its
MyRand sampling, InitialEntropyCodes, RefineEntropyCodes, FindBlocks,
RemapBlockIds, BuildBlockHistograms and ClusterBlocks), the histogram
clustering functions (BrotliCompareAndPushToQueue,
BrotliHistogramCombine, BrotliHistogramBitCostDistance,
BrotliHistogramRemap, BrotliHistogramReindex, BrotliClusterHistograms),
BrotliInitDistanceParams, BrotliBuildMetaBlock (including the distance
parameter search) and BrotliOptimizeHistograms (with
BrotliOptimizeHuffmanCountsForRle). All memory that the C code
allocates lives in a caller-supplied workbuf, carved out by a bump
allocator.

bit_cost gets an optional 64Ki-entry FastLog2 cache (filled by the
same softfloat log2), which makes the clustering several times faster.

Extend the metablock_tester and the parity harness to compare
SplitBlock, ClusterHistograms*, BuildMetaBlock and OptimizeHistograms
results with libbrotlienc v1.2.1.
2026-10-27 00:52:42 +00:00
Claude 88c32bc668 std/brotli: port bit_cost.c and histogram.c for the encoder
Add the bit_cost struct (encode_bit_cost.wuffs, encode_histogram.wuffs):
BrotliBitsEntropy, BrotliPopulationCost{Literal,Command,Distance},
FastLog2, the histogram helpers and BrotliBuildHistogramsWithContext,
with doubles computed by std/softfloat in the C code's operation order.
Histograms and commands are little-endian byte slices with the C structs'
memory layout.

Add a test-only metablock_tester struct and the
script/manual-test-brotli-metablock-parity.c harness, which compares
BitsEntropy and PopulationCost* bit for bit with libbrotlienc v1.2.1
(200000 random histograms: 0 mismatches).
2026-10-27 00:16:37 +00:00
Claude d74d28bef9 Merge branch 'm1c-decoder-extensions' into claude/project-thread-0izcgq
Combines the encoder (quality 0 and 1) with the decoder's large-window
mode and custom dictionaries. The test and mimic files take both sides'
new sections; the fuzzer gains the std/softfloat module that the
encoder now pulls into std/brotli.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:59:14 +00:00
Claude ca1ba9d134 std/brotli: add an encoder for quality levels 0 and 1
Adds a brotli encoder (std/brotli's encoder struct, a base.io_transformer)
whose output is byte-identical to libbrotlienc v1.2.1 for quality levels 0
and 1, mirroring encode.c's BrotliEncoderCompressStream(Fast),
compress_fragment.c and compress_fragment_two_pass.c, plus the entropy
coding and bit stream helpers they need. Every double operation goes
through std/softfloat, following the C evaluation order exactly.

Parameters are quirks (encode_quirks.wuffs) that mirror
BrotliEncoderSetParameter, sanitized at the first transform_io call.
Each transform_io resumption corresponds to one C call with available_in
equal to the readable src length; a closed src is FINISH; request_flush!
and request_emit_metadata! make the next resumption a FLUSH or
EMIT_METADATA call. Qualities 2 to 11 return "#unsupported encoder
quality" for now. Encoder tables are generated by
script/print-brotli-tables.go -encoder.

Tests: round trips through the Wuffs decoder over the corpus and synthetic
inputs; with -mimic, byte-exact comparisons against libbrotlienc over
quality × window × mode × input chunking × flush schedule × metadata ×
size hint. script/manual-test-brotli-encoder-parity.c runs the full
grid: 86,268 configurations, zero mismatches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:55:56 +00:00
Claude b9be8feaee std/brotli: large-window mode, custom dictionaries and a fuzzer
Completes the decoder's feature set relative to libbrotlidec v1.2.1:

- QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the
  large-window WBITS escape, window bits up to 30, the 544-symbol distance
  alphabet and 64-bit distance arithmetic, with workbuf_len growing to
  (1 << 30) + 542 when the quirk is set.
- decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw
  (LZ77 prefix) and serialized shared dictionaries: compound-dictionary
  distance mapping, the shared-dictionary parser (custom word lists,
  transform lists including the SHIFT transforms, context-dependent
  dictionary selection, prefix dictionaries) and all of its validation.
  Wuffs structs cannot hold slices, so the dictionary bytes are copied
  into the front of the caller's workbuf; the layout is documented in
  the README. Serialized dictionaries are accepted as in a
  BROTLI_EXPERIMENTAL build of libbrotli.
- fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and
  random I/O limits.

Tests: new golden vectors for large windows (including a 17 MiB
far-distance stream generated in the test), raw and serialized
dictionaries, nine more malformed inputs, and mimic comparisons that
attach the same dictionaries to libbrotlidec. Differential runs over
2.7 million mutated large-window and dictionary streams found no
divergence from libbrotlidec.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:45:06 +00:00
Claude 4978207531 std/brotli: add a brotli decoder with parity to libbrotlidec v1.2.1
This is the first milestone of a byte-exact rewrite of brotli (reference
implementation tag v1.2.1) in Wuffs. The decoder mirrors c/dec/decode.c
function by function, including its validation checks: the same output for
valid input and the same verdict, with each BROTLI_DECODER_ERROR_FORMAT_*
code mapped to a Wuffs status (see std/brotli/README.md). The C state
machine is replaced by Wuffs coroutines, so decoding works with arbitrarily
small src and dst buffers.

The history window lives in the caller's workbuf. Huffman table groups,
context maps and the static dictionary scratch space live in the decoder
struct. The static dictionary, transforms, context lookup tables and prefix
code tables in common_tables.wuffs are generated from the brotli source by
script/print-brotli-tables.go.

Tests: golden tests on the upstream tests/testdata vectors and on streams
produced by the v1.2.1 CLI at several qualities and window sizes, 1-byte
I/O limits, truncation at every prefix, malformed inputs per error code,
trailing data and workbuf boundaries. With -mimic, every vector and every
malformed input is also compared against libbrotlidec, including the exact
error code. A differential run over about a million mutated inputs found
no divergence.

Not yet supported (left as TODOs): large-window brotli, custom/shared
dictionaries, metadata callbacks and fast paths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 22:46:34 +00:00
Claude f33e9f3377 std/softfloat: add bit-exact IEEE 754 binary64/binary32 arithmetic
Wuffs the Language has no floating point types, but some reference
implementations (notably the brotli encoder) make decisions with double and
float arithmetic and libm's log2. Porting them with byte-identical output
needs bit-exact floating point results.

This package implements f64 and f32 add, sub, mul, div, fma, comparisons,
min/max, integer conversions and f64<->f32 conversions on bit patterns,
with round-to-nearest-even, subnormals, infinities and NaNs, following the
structure of Berkeley SoftFloat 3. It also ports glibc 2.39's x86-64
__log2_fma log2 implementation (the variant selected on CPUs with FMA and
AVX2), following the shipped machine code's fused operations.

Verification: test/c/std/softfloat.c checks every operation against the C
compiler's native arithmetic on edge cases and random operands. The
script/manual-test-softfloat-*.c programs run 10^8 random operations per
operation group and compare log2_u64 against libm for every integer in
[1, 2^32) plus 10^7 larger values; all with zero mismatches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 22:46:32 +00:00
Nigel Tao c677bef7fd std/png: parse cICP chunk
Updates #180
2026-10-18 18:13:59 +11:00
Nigel Tao 466cf6f15d std/webp: tweak workbuf_len determination 2026-08-10 09:23:05 +10:00
Nigel Tao d9cf0751ca std/vp8: blend VP8X ALPH chunk 2026-08-06 11:05:26 +10:00
Nigel Tao 5f398130e8 std/vp8: fix QUIRK_WIDTH_AND_HEIGHT height 2026-08-01 11:09:04 +10:00
Nigel Tao e7f6c79752 std/vp8: decode VP8X ALPH chunk 2026-07-29 15:58:52 +10:00
Nigel Tao 2a97911e02 std/webp: factor out decode_vp8l.wuffs 2026-07-29 15:58:50 +10:00
Nigel Tao 6e9e2e00c2 std/vp8: accept VP8X (but ignore any ALPH chunk) 2026-07-20 21:24:35 +10:00
Nigel Tao e3bda4efc9 std/webp: ignore alpha_is_used bit per the spec 2026-07-16 21:49:31 +10:00
Nigel Tao 8515165bee std/webp: treat this.width as constant after set 2026-07-16 21:49:31 +10:00
Nigel Tao 183beab481 std/webp: relax VP8L color transform validity
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.

Updates PR #168
2026-07-16 21:49:31 +10:00
Nigel Tao 3fce6225c9 std/vp8: add comments about reconstruction etc 2026-07-09 17:08:58 +10:00
Nigel Tao 46ac36bd7b std/handsum: tweak scaleAndBiasChromaDown 2026-07-08 10:23:10 +10:00
Nigel Tao 0e985fe880 std/jpeg: add link to Ekrem Fidan's blog post 2026-07-07 17:07:15 +10:00
Nigel Tao 67591a3570 std/vp8: simplify swizzle_one_row_of_macroblocks
Prior to 381e186f "std/vp8: swizzle_ycck once, not once per MB row", we
swizzled one row of macroblocks at a time. After 381e186f, we swizzled
the whole image in one pass, but kept the option of, in the future,
amending the swizzle_ycck implementation to allow ((y_min_incl > 0) and
(ycc_upsampling == FANCY_ETC)) at the same time.

However, 381e186f didn't show a performance loss so large that it seems
worth complicating the swizzle_ycck implementation in that way. So we
just drop, in this commit, the option of that amendment.
2026-07-03 21:58:33 +10:00
Nigel Tao b2a2486096 std/vp8: use YCC_UPSAMPLING__FANCY_LIKE_LIBWEBP
Updates PR #168
2026-07-03 21:47:34 +10:00
Nigel Tao 381e186f30 std/vp8: swizzle_ycck once, not once per MB row
This prepares us to change std/vp8's YCC upsampling method from "basic"
to "fancy, like libwebp". Wuffs' current implementation of "fancy"
requires the initial Y coordinate to be zero, as fancy upsampling
special cases the top row of the image and it's simpler if the top row
(row zero) is always presented in the upsampling call.

This is not an optimization or pessimization per se, but for the record,
the performance impact is between negligible and a small improvement.

name                                               old speed      new speed      delta

wuffs_webp_lossy_decode_image_19k_8bpp/clang14     33.7MB/s ± 0%  34.1MB/s ± 0%  +1.28%  (p=0.008 n=5+5)
wuffs_webp_lossy_decode_image_40k_24bpp/clang14     106MB/s ± 0%   106MB/s ± 0%  +0.09%  (p=0.008 n=5+5)
wuffs_webp_lossy_decode_image_77k_24bpp/clang14     129MB/s ± 0%   130MB/s ± 0%  +0.54%  (p=0.008 n=5+5)
wuffs_webp_lossy_decode_image_552k_32bpp/clang14    207MB/s ± 0%   211MB/s ± 0%  +1.90%  (p=0.008 n=5+5)
wuffs_webp_lossy_decode_image_4002k_24bpp/clang14   137MB/s ± 0%   138MB/s ± 0%  +0.40%  (p=0.008 n=5+5)

wuffs_webp_lossy_decode_image_19k_8bpp/gcc12       32.9MB/s ± 0%  32.9MB/s ± 0%  +0.05%  (p=0.008 n=5+5)
wuffs_webp_lossy_decode_image_40k_24bpp/gcc12      98.4MB/s ± 0%  98.8MB/s ± 0%  +0.42%  (p=0.008 n=5+5)
wuffs_webp_lossy_decode_image_77k_24bpp/gcc12       125MB/s ± 0%   125MB/s ± 0%  +0.17%  (p=0.008 n=5+5)
wuffs_webp_lossy_decode_image_552k_32bpp/gcc12      211MB/s ± 0%   214MB/s ± 0%  +1.21%  (p=0.008 n=5+5)
wuffs_webp_lossy_decode_image_4002k_24bpp/gcc12     137MB/s ± 0%   137MB/s ± 0%    ~     (p=0.095 n=5+5)
2026-07-03 21:47:26 +10:00
Nigel Tao 85a0283cfc base: tweak swizzle_ycck ycc_upsampling parameter
This moves from a 2-valued "basic (box) or fancy (triangle)" upsampling
parameter to a 3-valued one, which can now represent the fact that
libjpeg's and libwebp's fancy (triangle) upsampling is ever-so-slightly
different (see script/script/replicate-libwebp-chroma-upsampling.go and
its comments).

This is not an optimization or pessimization per se, but for the record,
the performance impact is mixed.

name                                             old speed     new speed     delta

wuffs_jpeg_decode_19k_8bpp/clang14               317MB/s ± 0%  314MB/s ± 0%  -0.99%  (p=0.008 n=5+5)
wuffs_jpeg_decode_30k_24bpp_progressive/clang14  318MB/s ± 0%  328MB/s ± 0%  +3.13%  (p=0.008 n=5+5)
wuffs_jpeg_decode_30k_24bpp_sequential/clang14   879MB/s ± 0%  871MB/s ± 0%  -0.83%  (p=0.008 n=5+5)
wuffs_jpeg_decode_77k_24bpp/clang14              719MB/s ± 0%  713MB/s ± 0%  -0.89%  (p=0.008 n=5+5)
wuffs_jpeg_decode_552k_24bpp_420/clang14         779MB/s ± 0%  771MB/s ± 0%  -1.01%  (p=0.008 n=5+5)
wuffs_jpeg_decode_552k_24bpp_444/clang14         902MB/s ± 0%  890MB/s ± 0%  -1.32%  (p=0.008 n=5+5)
wuffs_jpeg_decode_4002k_24bpp/clang14            784MB/s ± 0%  779MB/s ± 0%  -0.66%  (p=0.008 n=5+5)

wuffs_jpeg_decode_19k_8bpp/gcc12                 358MB/s ± 0%  362MB/s ± 0%  +0.93%  (p=0.008 n=5+5)
wuffs_jpeg_decode_30k_24bpp_progressive/gcc12    305MB/s ± 0%  311MB/s ± 0%  +1.91%  (p=0.008 n=5+5)
wuffs_jpeg_decode_30k_24bpp_sequential/gcc12     912MB/s ± 0%  918MB/s ± 0%  +0.73%  (p=0.016 n=4+5)
wuffs_jpeg_decode_77k_24bpp/gcc12                749MB/s ± 0%  763MB/s ± 0%  +1.83%  (p=0.008 n=5+5)
wuffs_jpeg_decode_552k_24bpp_420/gcc12           813MB/s ± 0%  824MB/s ± 0%  +1.37%  (p=0.008 n=5+5)
wuffs_jpeg_decode_552k_24bpp_444/gcc12           969MB/s ± 0%  984MB/s ± 0%  +1.62%  (p=0.008 n=5+5)
wuffs_jpeg_decode_4002k_24bpp/gcc12              815MB/s ± 0%  832MB/s ± 0%  +2.06%  (p=0.008 n=5+5)
2026-07-03 21:38:28 +10:00
Nigel Tao 0e06bcc9bc std/jpeg: add Further Reading README.md section 2026-06-29 22:05:13 +10:00
Nigel Tao 79bf624c32 std/vp8: add README.md 2026-06-29 16:09:59 +10:00
Nigel Tao 3ba284b67e std: link to "Making Software - Image Compression" 2026-06-22 17:46:11 +10:00
Nigel Tao e9ebad85a7 std/vp8: fix skip for all-zero DCT coefficients
"wuffs test -mimic std/webp" now passes.

This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.

Updates PR #168
2026-06-22 15:23:11 +10:00
Nigel Tao 9e274bee24 std/vp8: use YCC_MODEL__BT_601_STUDIO_RANGE
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.

Updates PR #168
2026-06-22 10:42:05 +10:00
Nigel Tao 9a027b8016 base: add WUFFS_BASE__YCC_MODEL__ETC 2026-06-20 18:19:45 +10:00
Nigel Tao ecca245226 std/vp8: fix luma_mode == 4 HZNC tracking
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.

Updates PR #168
2026-06-19 19:05:15 +10:00
Nigel Tao cae59b4ce8 std/vp8: fix luma subblock prediction modes
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.

Updates PR #168
2026-06-19 19:04:31 +10:00
Nigel Tao 24fae427c1 std/vp8: swizzle to the dst pixel buffer
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.

Updates PR #168
2026-06-19 08:45:56 +10:00
Nigel Tao 214b63c275 std/vp8: rename decode_first_partition 2026-06-18 21:19:58 +10:00
Nigel Tao 7b69762670 std/vp8: implement loop filters
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.

Updates PR #168
2026-06-18 17:22:40 +10:00
Nigel Tao 487922dab4 std/vp8: fix prediction modes and offsets
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.

Updates PR #168
2026-06-17 16:13:49 +10:00
Nigel Tao 68d13b8740 std/vp8: implement IDCT/IWHT
This commit, part of a series, is inspired by pull request PR #168 by
Damian Kaczmarek (with AI assistance), with some modifications.

Updates PR #168
2026-06-17 16:11:36 +10:00