Completes the decoder's feature set relative to libbrotlidec v1.2.1:
- QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the
large-window WBITS escape, window bits up to 30, the 544-symbol distance
alphabet and 64-bit distance arithmetic, with workbuf_len growing to
(1 << 30) + 542 when the quirk is set.
- decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw
(LZ77 prefix) and serialized shared dictionaries: compound-dictionary
distance mapping, the shared-dictionary parser (custom word lists,
transform lists including the SHIFT transforms, context-dependent
dictionary selection, prefix dictionaries) and all of its validation.
Wuffs structs cannot hold slices, so the dictionary bytes are copied
into the front of the caller's workbuf; the layout is documented in
the README. Serialized dictionaries are accepted as in a
BROTLI_EXPERIMENTAL build of libbrotli.
- fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and
random I/O limits.
Tests: new golden vectors for large windows (including a 17 MiB
far-distance stream generated in the test), raw and serialized
dictionaries, nine more malformed inputs, and mimic comparisons that
attach the same dictionaries to libbrotlidec. Differential runs over
2.7 million mutated large-window and dictionary streams found no
divergence from libbrotlidec.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
This is the first milestone of a byte-exact rewrite of brotli (reference
implementation tag v1.2.1) in Wuffs. The decoder mirrors c/dec/decode.c
function by function, including its validation checks: the same output for
valid input and the same verdict, with each BROTLI_DECODER_ERROR_FORMAT_*
code mapped to a Wuffs status (see std/brotli/README.md). The C state
machine is replaced by Wuffs coroutines, so decoding works with arbitrarily
small src and dst buffers.
The history window lives in the caller's workbuf. Huffman table groups,
context maps and the static dictionary scratch space live in the decoder
struct. The static dictionary, transforms, context lookup tables and prefix
code tables in common_tables.wuffs are generated from the brotli source by
script/print-brotli-tables.go.
Tests: golden tests on the upstream tests/testdata vectors and on streams
produced by the v1.2.1 CLI at several qualities and window sizes, 1-byte
I/O limits, truncation at every prefix, malformed inputs per error code,
trailing data and workbuf boundaries. With -mimic, every vector and every
malformed input is also compared against libbrotlidec, including the exact
error code. A differential run over about a million mutated inputs found
no divergence.
Not yet supported (left as TODOs): large-window brotli, custom/shared
dictionaries, metadata callbacks and fast paths.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
It changes from " 0-9?a-z" to ".0-9a-z~". The default or placeholder
changes from " " to "." so it's now always visible. The wildcard changes
from "?" to "~" so it's now sorted last.
lo128.dat and hi128.dat were constructed manually, then the
256.bytes.two-concatenated-streams.xz file was constructed by:
xz --keep lo128.dat
xz --keep hi128.dat
cat lo128.dat.xz hi128.dat.xz > 256.bytes.two-concatenated-streams.xz
It was created wih a custom xz binary (built from the source at
https://github.com/tukaani-project/xz), with this command:
xz --compress --keep --stdout enwik5 > enwik5.lzma2-chunk-max-1-13.xz
after applying this patch to xz's src/liblzma/lzma/lzma2_encoder.h:
-#define LZMA2_CHUNK_MAX (UINT32_C(1) << 16)
+#define LZMA2_CHUNK_MAX (UINT32_C(1) << 13)
The resultant enwik5.lzma2-chunk-max-1-13.xz file has multiple LZMA2
chunks, unlike the existing enwik5.xz (created with the normal chunk
limit). Wuffs' std/xz currently does not decode this new file properly.
SPDX is an ISO standard for describing licenses in a computer readable
format, which is useful when applying license compliance tools and the
like to source code.
In addition to ensuring that the Wuffs C distribution is properly
annotated to the benefit of projects using Wuffs and SPDX-supporting
tools, add it everywhere else where there's a license in this repo for
consistency, too.
The LICENSE has changed from a single license (Apache2) to a dual
license (Apache2 or MIT, at your option).
If you've previously used Wuffs under the Apache2 license, you are able
to keep doing so. No further action required by you.
The two files listed below (previous releases) are exceptions, as they
are older snapshots, frozen in time. Their license has not changed:
release/c/wuffs-v0.2.c
release/c/wuffs-v0.3.c
Every name in the AUTHORS file, other than Google Inc., has signed
Google's CLA (Contributor License Agreement).