Commit Graph
4059 Commits
Author SHA1 Message Date
Claude 9ad7ae6d69 std/brotli: add metablock_builder.workbuf_len_worst_case
Give the metablock builder an upper bound on the workbuf length that
build_meta_block needs, expose it through the metablock_tester and check
it against the high water mark in the parity harness. Add shard options
(-largeoff, -edge, -manytypes) and a stress phase that reaches 256 block
types to the harness.
2026-10-27 01:03:11 +00:00
Claude 202b406248 std/brotli: port block_splitter.c, cluster.c and metablock.c
Add the metablock_builder struct, mirroring BrotliSplitBlock (with its
MyRand sampling, InitialEntropyCodes, RefineEntropyCodes, FindBlocks,
RemapBlockIds, BuildBlockHistograms and ClusterBlocks), the histogram
clustering functions (BrotliCompareAndPushToQueue,
BrotliHistogramCombine, BrotliHistogramBitCostDistance,
BrotliHistogramRemap, BrotliHistogramReindex, BrotliClusterHistograms),
BrotliInitDistanceParams, BrotliBuildMetaBlock (including the distance
parameter search) and BrotliOptimizeHistograms (with
BrotliOptimizeHuffmanCountsForRle). All memory that the C code
allocates lives in a caller-supplied workbuf, carved out by a bump
allocator.

bit_cost gets an optional 64Ki-entry FastLog2 cache (filled by the
same softfloat log2), which makes the clustering several times faster.

Extend the metablock_tester and the parity harness to compare
SplitBlock, ClusterHistograms*, BuildMetaBlock and OptimizeHistograms
results with libbrotlienc v1.2.1.
2026-10-27 00:52:42 +00:00
Claude 88c32bc668 std/brotli: port bit_cost.c and histogram.c for the encoder
Add the bit_cost struct (encode_bit_cost.wuffs, encode_histogram.wuffs):
BrotliBitsEntropy, BrotliPopulationCost{Literal,Command,Distance},
FastLog2, the histogram helpers and BrotliBuildHistogramsWithContext,
with doubles computed by std/softfloat in the C code's operation order.
Histograms and commands are little-endian byte slices with the C structs'
memory layout.

Add a test-only metablock_tester struct and the
script/manual-test-brotli-metablock-parity.c harness, which compares
BitsEntropy and PopulationCost* bit for bit with libbrotlienc v1.2.1
(200000 random histograms: 0 mismatches).
2026-10-27 00:16:37 +00:00
Claude d74d28bef9 Merge branch 'm1c-decoder-extensions' into claude/project-thread-0izcgq
Combines the encoder (quality 0 and 1) with the decoder's large-window
mode and custom dictionaries. The test and mimic files take both sides'
new sections; the fuzzer gains the std/softfloat module that the
encoder now pulls into std/brotli.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:59:14 +00:00
Claude ca1ba9d134 std/brotli: add an encoder for quality levels 0 and 1
Adds a brotli encoder (std/brotli's encoder struct, a base.io_transformer)
whose output is byte-identical to libbrotlienc v1.2.1 for quality levels 0
and 1, mirroring encode.c's BrotliEncoderCompressStream(Fast),
compress_fragment.c and compress_fragment_two_pass.c, plus the entropy
coding and bit stream helpers they need. Every double operation goes
through std/softfloat, following the C evaluation order exactly.

Parameters are quirks (encode_quirks.wuffs) that mirror
BrotliEncoderSetParameter, sanitized at the first transform_io call.
Each transform_io resumption corresponds to one C call with available_in
equal to the readable src length; a closed src is FINISH; request_flush!
and request_emit_metadata! make the next resumption a FLUSH or
EMIT_METADATA call. Qualities 2 to 11 return "#unsupported encoder
quality" for now. Encoder tables are generated by
script/print-brotli-tables.go -encoder.

Tests: round trips through the Wuffs decoder over the corpus and synthetic
inputs; with -mimic, byte-exact comparisons against libbrotlienc over
quality × window × mode × input chunking × flush schedule × metadata ×
size hint. script/manual-test-brotli-encoder-parity.c runs the full
grid: 86,268 configurations, zero mismatches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:55:56 +00:00
Claude b9be8feaee std/brotli: large-window mode, custom dictionaries and a fuzzer
Completes the decoder's feature set relative to libbrotlidec v1.2.1:

- QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the
  large-window WBITS escape, window bits up to 30, the 544-symbol distance
  alphabet and 64-bit distance arithmetic, with workbuf_len growing to
  (1 << 30) + 542 when the quirk is set.
- decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw
  (LZ77 prefix) and serialized shared dictionaries: compound-dictionary
  distance mapping, the shared-dictionary parser (custom word lists,
  transform lists including the SHIFT transforms, context-dependent
  dictionary selection, prefix dictionaries) and all of its validation.
  Wuffs structs cannot hold slices, so the dictionary bytes are copied
  into the front of the caller's workbuf; the layout is documented in
  the README. Serialized dictionaries are accepted as in a
  BROTLI_EXPERIMENTAL build of libbrotli.
- fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and
  random I/O limits.

Tests: new golden vectors for large windows (including a 17 MiB
far-distance stream generated in the test), raw and serialized
dictionaries, nine more malformed inputs, and mimic comparisons that
attach the same dictionaries to libbrotlidec. Differential runs over
2.7 million mutated large-window and dictionary streams found no
divergence from libbrotlidec.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:45:06 +00:00
Claude 4978207531 std/brotli: add a brotli decoder with parity to libbrotlidec v1.2.1
This is the first milestone of a byte-exact rewrite of brotli (reference
implementation tag v1.2.1) in Wuffs. The decoder mirrors c/dec/decode.c
function by function, including its validation checks: the same output for
valid input and the same verdict, with each BROTLI_DECODER_ERROR_FORMAT_*
code mapped to a Wuffs status (see std/brotli/README.md). The C state
machine is replaced by Wuffs coroutines, so decoding works with arbitrarily
small src and dst buffers.

The history window lives in the caller's workbuf. Huffman table groups,
context maps and the static dictionary scratch space live in the decoder
struct. The static dictionary, transforms, context lookup tables and prefix
code tables in common_tables.wuffs are generated from the brotli source by
script/print-brotli-tables.go.

Tests: golden tests on the upstream tests/testdata vectors and on streams
produced by the v1.2.1 CLI at several qualities and window sizes, 1-byte
I/O limits, truncation at every prefix, malformed inputs per error code,
trailing data and workbuf boundaries. With -mimic, every vector and every
malformed input is also compared against libbrotlidec, including the exact
error code. A differential run over about a million mutated inputs found
no divergence.

Not yet supported (left as TODOs): large-window brotli, custom/shared
dictionaries, metadata callbacks and fast paths.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 22:46:34 +00:00
Claude f33e9f3377 std/softfloat: add bit-exact IEEE 754 binary64/binary32 arithmetic
Wuffs the Language has no floating point types, but some reference
implementations (notably the brotli encoder) make decisions with double and
float arithmetic and libm's log2. Porting them with byte-identical output
needs bit-exact floating point results.

This package implements f64 and f32 add, sub, mul, div, fma, comparisons,
min/max, integer conversions and f64<->f32 conversions on bit patterns,
with round-to-nearest-even, subnormals, infinities and NaNs, following the
structure of Berkeley SoftFloat 3. It also ports glibc 2.39's x86-64
__log2_fma log2 implementation (the variant selected on CPUs with FMA and
AVX2), following the shipped machine code's fused operations.

Verification: test/c/std/softfloat.c checks every operation against the C
compiler's native arithmetic on edge cases and random operands. The
script/manual-test-softfloat-*.c programs run 10^8 random operations per
operation group and compare log2_u64 against libm for every integer in
[1, 2^32) plus 10^7 larger values; all with zero mismatches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 22:46:32 +00:00
Nigel Tao 4be865da25 base: fix parse_number_f64("0.000...0001e30001")
Updates PR #184
2026-10-21 12:24:09 +11:00
Nigel Tao d54d330601 base: let parse_number_xxx return "arg too long"
Updates PR #184
2026-10-21 12:04:44 +11:00
Nigel Tao 7f70dda95c release: wuffs gen -version=0.4.0-alpha.12 v0.4.0-alpha.12 2026-10-20 14:35:26 +11:00
Nigel Tao 0d8b52336e spec/suitar: fix normalization link 2026-10-20 14:33:59 +11:00
Nigel Tao 0baab7fe72 release: spin in v0.3.6
This copies wuffs-v0.3.c from the v0.3 branch to the main branch. The
main branch is currently working on an unstable v0.4.
spin-in-v0.3.6
2026-10-20 14:32:33 +11:00
Nigel Tao c99f5af9fe base: fix parse_number_f64, 801 digits before '.'
Fixes #183
2026-10-20 14:09:09 +11:00
Nigel Tao fb8926614b A+C: add sabith saheb 2026-10-20 13:30:30 +11:00
Nigel Tao 2fa1d20d76 Merge pull request #182 from Medios-lgtm/parse-number-slice-bounds
base: don't read past the end of the slice in parse_number_{u64,f64}
2026-10-20 13:28:38 +11:00
sabith saheb 9d41525a2f base: don't read past the end of the slice in parse_number_{u64,f64} 2026-10-19 12:59:04 +05:30
Nigel Tao 2a6876f8bb release: wuffs gen -version=0.4.0-alpha.11 v0.4.0-alpha.11 2026-10-18 18:17:12 +11:00
Nigel Tao c677bef7fd std/png: parse cICP chunk
Updates #180
2026-10-18 18:13:59 +11:00
Nigel Tao ab24b06450 doc: add lz4, etc to enumerated-media-types.txt 2026-10-18 14:47:55 +11:00
Nigel Tao 1f513d8023 cgen: tweak arm_neon_utility.make_u16x4_multiple
The -O3 codegen is the same (https://godbolt.org/z/5ecWrEq77) but this
avoids compound literals, not part of C++11, and can trigger "compound
literals are a C99-specific feature [-Werror,-Wc99-extensions]".

Fixes #181
2026-10-06 22:30:32 +11:00
Nigel Tao 89b7cc7a36 base: add WUFFS_BASE__MIME_TYPE__ETC 2026-10-05 16:55:53 +11:00
Nigel Tao b2e6da3024 lib/suitar: allow implicitly-sized entries 2026-09-29 22:31:20 +10:00
Nigel Tao 72045c951e doc: add rar, zip to enumerated-media-types.txt 2026-09-21 09:43:35 +10:00
Nigel Tao f31d952b62 snippet: avoid some -Wunused-function warnings 2026-09-16 20:49:39 +10:00
Nigel Tao fbb65550ac snippet/uncompng: fix encode_apng return value 2026-09-16 20:39:10 +10:00
Nigel Tao 5f95a0d49a example: inspect SECCOMP_MODE_STRICT return value 2026-09-16 20:32:27 +10:00
Nigel Tao e0f1efc4a6 example/gifplayer: add NIE support 2026-09-15 22:30:23 +10:00
Nigel Tao 56250f1f27 example/gifplayer: refactor 2026-09-15 22:28:50 +10:00
Nigel Tao b1adeeccb1 lib/suitar: allow Write to a sparse entry 2026-09-15 15:31:04 +10:00
Nigel Tao b77e1a67de lib/suitar: make some Reader.Next errors sticky
Updates PR #179
2026-09-15 14:02:10 +10:00
Nigel Tao c944c491c4 lib/suitar: tweak some style nits
Updates PR #179
2026-09-15 14:00:21 +10:00
Nigel Tao 4abc399eaa A+C: add rootkiller6788 2026-09-15 10:44:18 +10:00
Nigel Tao 901b740c00 spec/suitar: elaborate on duplicate file names 2026-09-15 10:42:51 +10:00
Nigel Tao 3d46062176 cgen: note that base.io_buffer__struct is linear 2026-09-15 10:42:51 +10:00
Nigel Tao 1ca418ef16 snippet/uncompng: support APNG (Animated PNG) 2026-09-15 10:42:51 +10:00
Nigel Tao 742e8b78d9 lib/suitar: reject names ending in "/." or "/.." 2026-09-15 10:42:51 +10:00
Nigel Tao 5049107b90 Merge pull request #179 from rootkiller6788/fix-suitar-writer
lib/suitar: fix Writer block-boundary handling and directory headers
2026-09-15 10:35:54 +10:00
rootkiller6788 193d0354b4 lib/suitar: test chunked writes and directory entries
TestWriterChunked writes one file several times, splitting the contents
over Write calls differently each time, and checks the archive bytes come
out identical. The capped writer makes a runaway Write fail the test
instead of hanging it.

TestWriterDir writes a directory and then a file inside it, and reads
both back.
2026-09-12 23:48:25 +08:00
rootkiller6788 26ce8600a1 lib/suitar: let WriteHeader write a directory entry
WriteHeader copied h into w.header before calling w.Write to emit the
GNUTYPE_LONGNAME block holding the name. Write only accepts TypeReg and
TypeGNUSparse, so a TypeDir header failed that call and left the Writer
permanently in an error state - every later entry on the same Writer
failed too.

Set w.header after the name block has gone out instead.
2026-09-12 23:48:18 +08:00
rootkiller6788 05a8209fa7 lib/suitar: reset bIndex after flushing a full block
Write left w.bIndex at 512 once a full block had been written out, so the
next loop iteration took the "partial block" branch, copied zero bytes and
went round again forever, writing the same 512 bytes over and over. Calling
Write with sizes that aren't multiples of 512 is enough to hit this.

It also duplicated the last data block when a Write call happened to end
exactly on a block boundary, because flush then saw a non-zero bIndex and
wrote the block out a second time.
2026-09-12 23:48:01 +08:00
Nigel Tao 0f214ba59c lib/suitar: add trailing EOF blocks
Both /usr/bin/tar and the Go standard library don't seem to need this,
but some other TAR readers might expect it.
2026-09-03 08:31:06 +10:00
Nigel Tao 504f8485f4 lib/suitar: add new Go package
$ TZ=UTC tar -tvf test/data/archive.dense.suitar
-rw-r--r-- nobody/nobody     0 2020-02-05 06:10 artificial/0.bytes
-rw-r--r-- nobody/nobody   853 2020-08-12 14:04 github-tags.json
-rwxr-xr-x nobody/nobody   693 2021-05-03 06:19 hello.sh
-rw-r--r-- nobody/nobody   104 2021-05-03 06:16 non-ascii/αβ.txt
-rw-r--r-- nobody/nobody   151 2021-05-03 06:23 non-ascii/😻.txt
-rw-r--r-- nobody/nobody   208 2020-02-05 06:10 pjw-thumbnail.png
-rw-r--r-- nobody/nobody   942 2020-02-05 06:10 romeo.txt
-rw-r--r-- nobody/nobody   558 2020-02-05 06:10 romeo.txt.gz
2026-09-01 21:55:46 +10:00
Nigel Tao ce863060d6 lib/uncompng: support APNG (Animated PNG) 2026-08-25 10:17:48 +10:00
Nigel Tao 2f7e714ab0 script: add print-png-chunks.go 2026-08-12 21:10:36 +10:00
Nigel Tao e772ccd47a lib/uncompng: rearrange Encode argument order 2026-08-11 09:52:32 +10:00
Nigel Tao 1230506e7c test/data: add "VP8X with animation" test image 2026-08-10 17:40:43 +10:00
Nigel Tao b89fd2fa03 script: undo print-nia-checksums.sh comment
This was accidentally commented out by "std/vp8: blend VP8X ALPH chunk",
commit d9cf0751ca.
2026-08-10 17:38:42 +10:00
Nigel Tao aad70fdfa5 script: print-riff-chunks.go handles WEBP/ANMF 2026-08-10 16:55:56 +10:00
Nigel Tao 466cf6f15d std/webp: tweak workbuf_len determination 2026-08-10 09:23:05 +10:00