Commit Graph
301 Commits
Author SHA1 Message Date
Claude 361091fee6 std/brotli: add quality 10 and 11 (Zopfli) encoder, untested
Port H10 (hash_to_binary_tree_inc.h), literal_cost.c,
backward_references_hq.c, the static dictionary match finder
(static_dict.c) and the quality 10+ parts of the metablock writer,
using the metablock_builder. The variable-size Zopfli and
metablock_builder scratch lives in a new workbuf layout region.
2026-10-27 02:36:05 +00:00
Claude abecc768e5 Merge branch 'm4b-metablock' into m4a-encoder-core
# Conflicts:
#	release/c/wuffs-unsupported-snapshot.c
2026-10-27 01:10:29 +00:00
Claude e66b315284 std/brotli: fix the UTF-8 prefix LUT; extend the encoder parity test to qualities 2 to 9 2026-10-27 01:04:57 +00:00
Claude 9ad7ae6d69 std/brotli: add metablock_builder.workbuf_len_worst_case
Give the metablock builder an upper bound on the workbuf length that
build_meta_block needs, expose it through the metablock_tester and check
it against the high water mark in the parity harness. Add shard options
(-largeoff, -edge, -manytypes) and a stress phase that reaches 256 block
types to the harness.
2026-10-27 01:03:11 +00:00
Claude 09d4b23d09 std/brotli: WIP encoder qualities 2 to 9 (EncodeData, hashers, greedy metablocks) 2026-10-27 00:58:17 +00:00
Claude 202b406248 std/brotli: port block_splitter.c, cluster.c and metablock.c
Add the metablock_builder struct, mirroring BrotliSplitBlock (with its
MyRand sampling, InitialEntropyCodes, RefineEntropyCodes, FindBlocks,
RemapBlockIds, BuildBlockHistograms and ClusterBlocks), the histogram
clustering functions (BrotliCompareAndPushToQueue,
BrotliHistogramCombine, BrotliHistogramBitCostDistance,
BrotliHistogramRemap, BrotliHistogramReindex, BrotliClusterHistograms),
BrotliInitDistanceParams, BrotliBuildMetaBlock (including the distance
parameter search) and BrotliOptimizeHistograms (with
BrotliOptimizeHuffmanCountsForRle). All memory that the C code
allocates lives in a caller-supplied workbuf, carved out by a bump
allocator.

bit_cost gets an optional 64Ki-entry FastLog2 cache (filled by the
same softfloat log2), which makes the clustering several times faster.

Extend the metablock_tester and the parity harness to compare
SplitBlock, ClusterHistograms*, BuildMetaBlock and OptimizeHistograms
results with libbrotlienc v1.2.1.
2026-10-27 00:52:42 +00:00
Claude 88c32bc668 std/brotli: port bit_cost.c and histogram.c for the encoder
Add the bit_cost struct (encode_bit_cost.wuffs, encode_histogram.wuffs):
BrotliBitsEntropy, BrotliPopulationCost{Literal,Command,Distance},
FastLog2, the histogram helpers and BrotliBuildHistogramsWithContext,
with doubles computed by std/softfloat in the C code's operation order.
Histograms and commands are little-endian byte slices with the C structs'
memory layout.

Add a test-only metablock_tester struct and the
script/manual-test-brotli-metablock-parity.c harness, which compares
BitsEntropy and PopulationCost* bit for bit with libbrotlienc v1.2.1
(200000 random histograms: 0 mismatches).
2026-10-27 00:16:37 +00:00
Claude ca1ba9d134 std/brotli: add an encoder for quality levels 0 and 1
Adds a brotli encoder (std/brotli's encoder struct, a base.io_transformer)
whose output is byte-identical to libbrotlienc v1.2.1 for quality levels 0
and 1, mirroring encode.c's BrotliEncoderCompressStream(Fast),
compress_fragment.c and compress_fragment_two_pass.c, plus the entropy
coding and bit stream helpers they need. Every double operation goes
through std/softfloat, following the C evaluation order exactly.

Parameters are quirks (encode_quirks.wuffs) that mirror
BrotliEncoderSetParameter, sanitized at the first transform_io call.
Each transform_io resumption corresponds to one C call with available_in
equal to the readable src length; a closed src is FINISH; request_flush!
and request_emit_metadata! make the next resumption a FLUSH or
EMIT_METADATA call. Qualities 2 to 11 return "#unsupported encoder
quality" for now. Encoder tables are generated by
script/print-brotli-tables.go -encoder.

Tests: round trips through the Wuffs decoder over the corpus and synthetic
inputs; with -mimic, byte-exact comparisons against libbrotlienc over
quality × window × mode × input chunking × flush schedule × metadata ×
size hint. script/manual-test-brotli-encoder-parity.c runs the full
grid: 86,268 configurations, zero mismatches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:55:56 +00:00
Claude 4978207531 std/brotli: add a brotli decoder with parity to libbrotlidec v1.2.1
This is the first milestone of a byte-exact rewrite of brotli (reference
implementation tag v1.2.1) in Wuffs. The decoder mirrors c/dec/decode.c
function by function, including its validation checks: the same output for
valid input and the same verdict, with each BROTLI_DECODER_ERROR_FORMAT_*
code mapped to a Wuffs status (see std/brotli/README.md). The C state
machine is replaced by Wuffs coroutines, so decoding works with arbitrarily
small src and dst buffers.

The history window lives in the caller's workbuf. Huffman table groups,
context maps and the static dictionary scratch space live in the decoder
struct. The static dictionary, transforms, context lookup tables and prefix
code tables in common_tables.wuffs are generated from the brotli source by
script/print-brotli-tables.go.

Tests: golden tests on the upstream tests/testdata vectors and on streams
produced by the v1.2.1 CLI at several qualities and window sizes, 1-byte
I/O limits, truncation at every prefix, malformed inputs per error code,
trailing data and workbuf boundaries. With -mimic, every vector and every
malformed input is also compared against libbrotlidec, including the exact
error code. A differential run over about a million mutated inputs found
no divergence.

Not yet supported (left as TODOs): large-window brotli, custom/shared
dictionaries, metadata callbacks and fast paths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 22:46:34 +00:00
Claude f33e9f3377 std/softfloat: add bit-exact IEEE 754 binary64/binary32 arithmetic
Wuffs the Language has no floating point types, but some reference
implementations (notably the brotli encoder) make decisions with double and
float arithmetic and libm's log2. Porting them with byte-identical output
needs bit-exact floating point results.

This package implements f64 and f32 add, sub, mul, div, fma, comparisons,
min/max, integer conversions and f64<->f32 conversions on bit patterns,
with round-to-nearest-even, subnormals, infinities and NaNs, following the
structure of Berkeley SoftFloat 3. It also ports glibc 2.39's x86-64
__log2_fma log2 implementation (the variant selected on CPUs with FMA and
AVX2), following the shipped machine code's fused operations.

Verification: test/c/std/softfloat.c checks every operation against the C
compiler's native arithmetic on edge cases and random operands. The
script/manual-test-softfloat-*.c programs run 10^8 random operations per
operation group and compare log2_u64 against libm for every integer in
[1, 2^32) plus 10^7 larger values; all with zero mismatches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 22:46:32 +00:00
Nigel Tao c677bef7fd std/png: parse cICP chunk
Updates #180
2026-10-18 18:13:59 +11:00
Nigel Tao 1ca418ef16 snippet/uncompng: support APNG (Animated PNG) 2026-09-15 10:42:51 +10:00
Nigel Tao 2f7e714ab0 script: add print-png-chunks.go 2026-08-12 21:10:36 +10:00
Nigel Tao b89fd2fa03 script: undo print-nia-checksums.sh comment
This was accidentally commented out by "std/vp8: blend VP8X ALPH chunk",
commit d9cf0751ca.
2026-08-10 17:38:42 +10:00
Nigel Tao aad70fdfa5 script: print-riff-chunks.go handles WEBP/ANMF 2026-08-10 16:55:56 +10:00
Nigel Tao c4df79d82e script: add print-riff-chunks.go 2026-08-07 16:33:25 +10:00
Nigel Tao 5d5fb1d6d9 base38: add "BCP 47 as base38" example 2026-08-06 14:57:00 +10:00
Nigel Tao d9cf0751ca std/vp8: blend VP8X ALPH chunk 2026-08-06 11:05:26 +10:00
Nigel Tao e3b5a7d506 script: add replicate-libwebp-chroma-upsampling.go 2026-06-30 16:09:45 +10:00
Nigel Tao ad039b5d3a snippet: add basic-handsum-decode.c 2026-06-29 14:28:48 +10:00
Nigel Tao 15a1cbf055 enumerated-media-types: add some Android values 2026-06-23 11:30:07 +10:00
Nigel Tao e0a44a0c40 script: add Magic Ring Buffer comment 2026-06-23 10:47:26 +10:00
Nigel Tao e7dd8f6520 script: add handsum-tilify.go 2026-05-10 21:17:50 +10:00
Nigel Tao 43a6814dfe std/handsum: add new package 2025-02-24 11:18:24 +11:00
Nigel Tao 71c6c97b86 script: fix WUFFS_BASE__FOURCC__TARGA typo 2024-11-06 22:37:33 +11:00
Nigel Tao c5e610cd5f script: add print-jpeg-markers -x flag 2024-10-27 11:51:34 +10:30
Nigel Tao 85af5a4bed std/tga: rename to std/targa 2024-09-08 09:00:10 +10:00
Nigel Tao 8c854ae702 std/thumbhash: implement do_decode_frame 2024-09-06 12:35:53 +10:00
Nigel Tao b8740d8881 std/thumbhash: add new package 2024-09-03 20:07:37 +10:00
Nigel Tao bd42fa7af2 base38: reassign alphabet and numbers
It changes from " 0-9?a-z" to ".0-9a-z~". The default or placeholder
changes from " " to "." so it's now always visible. The wildcard changes
from "?" to "~" so it's now sorted last.
2024-08-26 12:30:20 +10:00
Nigel Tao 6a9347843b std/jpeg: ignore too-short COM or APPn markers 2024-08-25 13:46:16 +10:00
Nigel Tao 408ab25f04 script: have confirm-matches-djpeg say "too large"
+Differ (large)
2024-08-25 13:05:59 +10:00
Nigel Tao 819742cfac std/jpeg: add "#short SOS bitstream" 2024-08-24 17:01:28 +10:00
Nigel Tao a824c55c3d std/jpeg: allow DHT markers before SOF 2024-08-24 16:13:31 +10:00
Nigel Tao 43df015669 std/etc2: add new package 2024-08-17 15:31:29 +10:00
Nigel Tao d06700f00b std/qoi: add new package
Fixes #63
2024-08-08 23:49:17 +10:00
Nigel Tao 5e0b2ae0d5 script: add print-json-token-etc usage comments 2024-07-27 09:41:00 +10:00
Nigel Tao a7fb1e8d69 script: avoid gcc -Wstringop-overread for strnlen 2024-07-27 09:33:39 +10:00
Nigel Tao 09f4500e19 std/webp: add new package (and vp8 placeholder) 2024-05-10 13:52:07 +10:00
Nigel Tao 92ff96359a std/crc32: re-write x86_sse42 implementation
name                                                       old speed      new speed      delta

wuffs_crc32_ieee_10k/clang14                               13.5GB/s ± 0%  21.9GB/s ± 0%  +62.09%  (p=0.016 n=5+4)
wuffs_crc32_ieee_100k/clang14                              22.4GB/s ± 9%  29.7GB/s ± 0%  +32.76%  (p=0.016 n=5+4)

wuffs_crc32_ieee_10k/gcc12                                 14.2GB/s ± 2%  22.2GB/s ± 0%  +56.92%  (p=0.008 n=5+5)
wuffs_crc32_ieee_100k/gcc12                                21.3GB/s ± 3%  29.6GB/s ± 1%  +39.18%  (p=0.008 n=5+5)

wuffs_gzip_decode_10k/clang14                               366MB/s ± 0%   372MB/s ± 0%   +1.41%  (p=0.008 n=5+5)
wuffs_gzip_decode_100k/clang14                              482MB/s ± 0%   494MB/s ± 0%   +2.57%  (p=0.008 n=5+5)

wuffs_gzip_decode_10k/gcc12                                 398MB/s ± 0%   418MB/s ± 0%   +5.19%  (p=0.008 n=5+5)
wuffs_gzip_decode_100k/gcc12                                510MB/s ± 0%   537MB/s ± 0%   +5.21%  (p=0.008 n=5+5)

wuffs_png_decode_image_19k_8bpp/clang14                     263MB/s ± 0%   264MB/s ± 0%   +0.39%  (p=0.008 n=5+5)
wuffs_png_decode_image_40k_24bpp/clang14                    297MB/s ± 0%   298MB/s ± 0%   +0.16%  (p=0.008 n=5+5)
wuffs_png_decode_image_77k_8bpp/clang14                     932MB/s ± 0%   945MB/s ± 0%   +1.33%  (p=0.008 n=5+5)
wuffs_png_decode_image_552k_32bpp_ignore_checksum/clang14   831MB/s ± 0%   833MB/s ± 0%   +0.25%  (p=0.008 n=5+5)
wuffs_png_decode_image_552k_32bpp_verify_checksum/clang14   799MB/s ± 0%   802MB/s ± 0%   +0.37%  (p=0.008 n=5+5)
wuffs_png_decode_image_4002k_24bpp/clang14                  306MB/s ± 0%   307MB/s ± 0%   +0.56%  (p=0.008 n=5+5)

wuffs_png_decode_image_19k_8bpp/gcc12                       283MB/s ± 0%   270MB/s ± 0%   -4.84%  (p=0.008 n=5+5)
wuffs_png_decode_image_40k_24bpp/gcc12                      330MB/s ± 0%   329MB/s ± 0%   -0.29%  (p=0.016 n=5+4)
wuffs_png_decode_image_77k_8bpp/gcc12                       992MB/s ± 0%   957MB/s ± 0%   -3.56%  (p=0.008 n=5+5)
wuffs_png_decode_image_552k_32bpp_ignore_checksum/gcc12     920MB/s ± 0%   908MB/s ± 0%   -1.30%  (p=0.008 n=5+5)
wuffs_png_decode_image_552k_32bpp_verify_checksum/gcc12     875MB/s ± 0%   867MB/s ± 0%   -0.95%  (p=0.008 n=5+5)
wuffs_png_decode_image_4002k_24bpp/gcc12                    341MB/s ± 0%   342MB/s ± 0%   +0.25%  (p=0.008 n=5+5)
2024-04-23 23:10:30 +10:00
Nigel Tao e541e0cf79 std/crc64: re-write x86_sse42 implementation
name                           old speed      new speed       delta

wuffs_crc64_ecma_10k/clang14   5.84GB/s ± 0%  23.43GB/s ± 0%  +301.23%  (p=0.008 n=5+5)
wuffs_crc64_ecma_100k/clang14  6.18GB/s ± 0%  30.23GB/s ± 1%  +389.45%  (p=0.008 n=5+5)

wuffs_crc64_ecma_10k/gcc12     5.87GB/s ± 1%  24.02GB/s ± 0%  +309.21%  (p=0.008 n=5+5)
wuffs_crc64_ecma_100k/gcc12    6.21GB/s ± 0%  30.68GB/s ± 0%  +394.22%  (p=0.008 n=5+5)

$ time example-mzcat < linux-6.8.2.tar.xz > /dev/null
Before: user 0m7.604s
After:  user 0m7.432s
2024-04-22 14:05:16 +10:00
Nigel Tao e1f803324b test: use s/100/300/ digits of pi for crcNN tests 2024-04-22 09:13:22 +10:00
Nigel Tao 58d2a80691 std/crc64: fix x86_sse42 k1' magic number 2024-04-21 22:43:34 +10:00
Nigel Tao 77541659f3 std/crc64: tweak x86 to be more like std/crc32 2024-04-21 22:05:36 +10:00
Nigel Tao f32bfe95c6 std/crc64: optimize for x86+SSE4.2
name                           old speed      new speed      delta

wuffs_crc64_ecma_10k/clang14   1.78GB/s ± 0%  5.79GB/s ± 0%  +225.48%  (p=0.008 n=5+5)
wuffs_crc64_ecma_100k/clang14  1.78GB/s ± 0%  6.09GB/s ± 0%  +241.67%  (p=0.008 n=5+5)

wuffs_crc64_ecma_10k/gcc12     2.05GB/s ± 1%  5.79GB/s ± 1%  +182.80%  (p=0.008 n=5+5)
wuffs_crc64_ecma_100k/gcc12    2.01GB/s ± 0%  6.09GB/s ± 0%  +202.64%  (p=0.008 n=5+5)

wuffs_xz_decode_100k/clang14   57.9MB/s ± 1%  60.2MB/s ± 0%    +4.11%  (p=0.008 n=5+5)

wuffs_xz_decode_100k/gcc12     57.0MB/s ± 0%  57.8MB/s ± 1%    +1.39%  (p=0.008 n=5+5)

$ time example-mzcat < linux-6.8.2.tar.xz > /dev/null
Before: user 0m8.122s
After:  user 0m8.115s
2024-04-02 00:17:43 +11:00
Nigel Tao 4e20ba6876 example: add output-crc32-digest flag 2024-02-15 13:40:09 +11:00
Nigel Tao 764169165a script: reduce os.Write perms, s/0666/0644/ 2024-02-09 22:16:10 +11:00
Nigel Tao 5a7299b5c2 script: have print-*-checksums.sh show exit codes 2023-12-26 22:21:13 +11:00
Nigel Tao 33db93f617 script: add print-mzcat-checksums.sh 2023-12-26 10:56:22 +11:00
Nigel Tao 920ac3e5c5 std/crc64: use slicing-by-M algorithm
name                           old speed     new speed      delta

wuffs_crc64_ecma_10k/clang11   485MB/s ± 0%  1774MB/s ± 0%  +265.68%  (p=0.008 n=5+5)
wuffs_crc64_ecma_100k/clang11  485MB/s ± 0%  1777MB/s ± 0%  +266.41%  (p=0.008 n=5+5)

wuffs_crc64_ecma_10k/gcc10     478MB/s ± 0%  1858MB/s ± 0%  +288.52%  (p=0.016 n=4+5)
wuffs_crc64_ecma_100k/gcc10    480MB/s ± 1%  1857MB/s ± 0%  +287.21%  (p=0.008 n=5+5)
2023-10-15 00:08:56 +11:00