Commit Graph
14 Commits
Author SHA1 Message Date
Claude f6798c006d script: use the smallest dictionary grid at q10/q11 for big inputs
Like the regular grid, qualities 10 and 11 use the smallest dictionary
configuration set for inputs longer than 64 KiB.
2026-10-27 09:08:15 +00:00
Claude 3f60ead4fc script: allow encoder parity workbufs of up to 12 GiB, link libbrotlidec 2026-10-27 06:10:02 +00:00
Claude ed7afaeab9 Merge claude/project-thread-0izcgq (encoder raw dictionaries)
At qualities 10 and 11, attaching a dictionary makes transform_io
return "#unsupported encoder quality" until the Zopfli path's compound
dictionary lookups are ported.
2026-10-27 06:06:39 +00:00
Claude 36a4380062 std/brotli: test encoder dictionaries and document them
Add mimic_brotli_encode_with_dictionaries to the mimic library and
encoder dictionary tests: the add_dictionary API, its workbuf
accounting and rejection cases (matching libbrotlienc's default build),
a parity grid over qualities 0 to 9 with several dictionary sets, stream
offsets, large window, flushes and metadata, and dictionaries of 2 to
33 MiB, each round tripped through both decoders.

Document the encoder's dictionary API, workbuf layout and differences
from the C code.
2026-10-27 05:28:37 +00:00
Claude aabdeaf4b0 std/brotli: add encoder raw (prefix) dictionaries
Mirror BrotliEncoderPrepareDictionary (BROTLI_SHARED_DICTIONARY_RAW),
CreatePreparedDictionary and BrotliEncoderAttachPreparedDictionary as
encoder.add_dictionary, with the prepared hash tables (slot offsets,
heads and items) in the workbuf after the dictionary bytes, and the
compound dictionary search (FindCompoundDictionaryMatch and
LookupCompoundDictionaryMatch) for the hashers whose
CreateBackwardReferences variant enables it, the compound dictionary
gap for static dictionary distances and the compound branch of
ExtendLastCommand.

Extend the encoder parity script with a -dict grid that also checks
round trips through libbrotlidec and the Wuffs decoder.
2026-10-27 04:54:39 +00:00
Claude 7d58caf91c Merge claude/project-thread-0izcgq (large window at qualities 3 to 9)
Put the q10/q11 variable region before the ring buffer (now the last
region), growing it (and moving the ring buffer) at the first
transform_io call to use a workbuf longer than workbuf_len's minimum.
Large window now also works at qualities 10 and 11.
2026-10-27 04:46:52 +00:00
Claude 2236afbd96 script: add a -oneshot option to the encoder parity harness 2026-10-27 03:13:45 +00:00
Claude 3669090209 script: use a smaller encoder parity grid for big q10/q11 inputs 2026-10-27 03:09:29 +00:00
Claude bef6ffc1bd std/brotli: fix the Zopfli starting distance cache order
Also run the encoder parity harness at qualities up to 11, using the
workbuf_len minimum.
2026-10-27 02:43:46 +00:00
Claude 0b9cfddf3c std/brotli: let a shorter encoder workbuf work for short inputs
Move the ring buffer to the end of the workbuf layout (at qualities 2 to
9). A workbuf shorter than workbuf_len now works if it holds the input
so far (plus 16 bytes of slack) in the ring buffer region and the input
stays short enough for the ring buffer never to wrap. This matters most
for large-window brotli, where workbuf_len at lgwin 30 is more than
2 GiB, mostly for a ring buffer that short inputs never touch.

The parity harness now also checks its large-window configurations with
the shortest such workbuf (and that one byte less fails).
2026-10-27 02:36:23 +00:00
Claude 75b09b6700 script: add -big inputs with far repeats to the encoder parity test
The -big option checks synthetic inputs of 3 to 70 MiB with repeats at
distances of up to 40 MiB (beyond the 16 MiB non-large-window limit),
with the usual grid up to 12 MiB and a few one-shot and streaming
large-window configurations above that.
2026-10-27 02:21:31 +00:00
Claude 8b0c3204a0 std/brotli: add large-window encoding at qualities 3 to 9
Port the composite hashers H35, H55 and H65 (hash_composite_inc.h) and
the rolling hashers HROLLING_FAST and HROLLING (hash_rolling_inc.h),
which ChooseHasher picks for lgwin above 24, and drop the
"#unsupported encoder quality" gate for the large-window quirk at
qualities 3 to 9. The workbuf's hasher region now includes the rolling
hasher's 64 MiB table when a composite hasher can be chosen.

Extend the encoder parity harness with a large-window grid (lgwin 10 to
30 at qualities 2 to 9) and a -lw option that runs only that grid.
2026-10-27 02:13:21 +00:00
Claude e66b315284 std/brotli: fix the UTF-8 prefix LUT; extend the encoder parity test to qualities 2 to 9 2026-10-27 01:04:57 +00:00
Claude ca1ba9d134 std/brotli: add an encoder for quality levels 0 and 1
Adds a brotli encoder (std/brotli's encoder struct, a base.io_transformer)
whose output is byte-identical to libbrotlienc v1.2.1 for quality levels 0
and 1, mirroring encode.c's BrotliEncoderCompressStream(Fast),
compress_fragment.c and compress_fragment_two_pass.c, plus the entropy
coding and bit stream helpers they need. Every double operation goes
through std/softfloat, following the C evaluation order exactly.

Parameters are quirks (encode_quirks.wuffs) that mirror
BrotliEncoderSetParameter, sanitized at the first transform_io call.
Each transform_io resumption corresponds to one C call with available_in
equal to the readable src length; a closed src is FINISH; request_flush!
and request_emit_metadata! make the next resumption a FLUSH or
EMIT_METADATA call. Qualities 2 to 11 return "#unsupported encoder
quality" for now. Encoder tables are generated by
script/print-brotli-tables.go -encoder.

Tests: round trips through the Wuffs decoder over the corpus and synthetic
inputs; with -mimic, byte-exact comparisons against libbrotlienc over
quality × window × mode × input chunking × flush schedule × metadata ×
size hint. script/manual-test-brotli-encoder-parity.c runs the full
grid: 86,268 configurations, zero mismatches.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:55:56 +00:00