Files
wuffs/test/data/brotli/README.md
T
Claude b9be8feaee std/brotli: large-window mode, custom dictionaries and a fuzzer
Completes the decoder's feature set relative to libbrotlidec v1.2.1:

- QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the
  large-window WBITS escape, window bits up to 30, the 544-symbol distance
  alphabet and 64-bit distance arithmetic, with workbuf_len growing to
  (1 << 30) + 542 when the quirk is set.
- decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw
  (LZ77 prefix) and serialized shared dictionaries: compound-dictionary
  distance mapping, the shared-dictionary parser (custom word lists,
  transform lists including the SHIFT transforms, context-dependent
  dictionary selection, prefix dictionaries) and all of its validation.
  Wuffs structs cannot hold slices, so the dictionary bytes are copied
  into the front of the caller's workbuf; the layout is documented in
  the README. Serialized dictionaries are accepted as in a
  BROTLI_EXPERIMENTAL build of libbrotli.
- fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and
  random I/O limits.

Tests: new golden vectors for large windows (including a 17 MiB
far-distance stream generated in the test), raw and serialized
dictionaries, nine more malformed inputs, and mimic comparisons that
attach the same dictionaries to libbrotlidec. Differential runs over
2.7 million mutated large-window and dictionary streams found no
divergence from libbrotlidec.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:45:06 +00:00

84 lines
4.6 KiB
Markdown

# Brotli Test Data
Files without a `.br` suffix, and their `.compressed` (and `.compressed.NN`)
counterparts, were copied from the `tests/testdata` directory of the [reference
brotli implementation](https://github.com/google/brotli) at tag v1.2.1. That
repository is licensed under the MIT license: "Copyright (c) 2009, 2010,
2013-2016 by the Brotli Authors".
Some of the upstream files were not copied:
- `lcet10.txt`, `mapsdatazrh`, `plrabn12.txt` and `zerosukkanooa`, because their
decompressed forms are larger than 256 KiB.
- `bb.binast` and `random_chunks`, because they have no `.compressed` form.
The `*.qNwM.br` files were generated by the brotli v1.2.1 command line tool,
from `../romeo.txt`, `../256.bytes` and `alice29.txt`, at quality level N and
with window bits M:
brotli -q N -w M -o romeo.txt.qNwM.br ../romeo.txt
The small window (`w10` and `w16`) `alice29.txt` files exercise the history
ring buffer wrapping around.
The `invalid.*.br` files are malformed. Apart from `invalid.truncated.br` (the
first 20 bytes of `romeo.txt.q11w22.br`), they were found by randomly mutating
the other (valid) files and keeping, for each libbrotlidec v1.2.1 error code,
the smallest mutated file that produced that error code. The file name
mentions that error code, e.g. `invalid.cl-space.br` produces
`BROTLI_DECODER_ERROR_FORMAT_CL_SPACE`.
## Large Window and Dictionary Test Data
These files exercise the `QUIRK_LARGE_WINDOW` quirk and `add_dictionary`. They
were generated with brotli v1.2.1: the command line tool where it suffices
and otherwise a small C program calling `BrotliEncoderSetParameter` (with
`BROTLI_PARAM_LARGE_WINDOW`), `BrotliEncoderPrepareDictionary` and
`BrotliEncoderAttachPreparedDictionary`, linked against a libbrotlienc built
with `BROTLI_EXPERIMENTAL` (needed for serialized shared dictionaries). Every
file's decompressed form was checked with libbrotlidec (also built with
`BROTLI_EXPERIMENTAL`).
- `*.qNlwM.br` are large-window streams (quality N, window bits M), e.g.
`brotli -q 11 --large_window=26 -o alice29.txt.q11lw26.br alice29.txt`.
- `far17m.q9lw25.br` is a large-window (25 window bits) quality 9 stream whose
decompressed form (17 MiB plus 128 KiB, generated by the test program
instead of being checked in) is 64 KiB of xorshift32 output, 17 MiB of
zeroes and the same 64 KiB again. The second copy refers back more than (1
<< 24) bytes. A standard (24 window bits) encoding is twice as large.
- `*.raw-romeo.br` use `../romeo.txt` as a raw (LZ77 prefix) dictionary, e.g.
`brotli -q 11 -w 22 -D ../romeo.txt -o romeo.txt.q11w22.raw-romeo.br
../romeo.txt`.
- `*.raw-chunks.br` use several raw dictionaries (compound dictionary chunks):
`alice29.txt`'s bytes `[0 .. 20000)`, `[20000 .. 20001)`, `[20001 .. 20001)`
(empty) and `[20001 .. 60000)`, or `../romeo.txt`'s bytes `[0 .. 300)`, `[300
.. 500)` and `[500 .. 942)`.
- `shared-dictionary.*.bin` are serialized shared dictionaries (the format
parsed by `BrotliSharedDictionaryAttach` in brotli's
`c/common/shared_dictionary.c`), built by a script:
- `full` has a 500 byte LZ77 prefix (the start of `../romeo.txt`), two
custom word lists (common words from `alice29.txt` and `asyoulik.txt`),
two custom transform lists (including `BROTLI_TRANSFORM_SHIFT_FIRST` and
`BROTLI_TRANSFORM_SHIFT_ALL` transforms), three (word list, transform
list) combinations (the third is the built-in pair) and a context map.
- `words` has one custom word list, the built-in transforms and no prefix.
- `prefix` has only an LZ77 prefix (the start of `asyoulik.txt`).
- `utf8` has a word list of UTF-8 (and invalid UTF-8) words and shift
transforms with various parameters.
- `shared-dictionary.text` is made of `shared-dictionary.full.bin` words,
transformed by its transforms. `*.sd-full.br` (and `*.sd-words.br`) were
compressed with `shared-dictionary.full.bin` (and `.words.bin`) attached.
`asyoulik.txt.q11w18.sd-prefix-raw.br` was compressed with
`shared-dictionary.prefix.bin` and then `../romeo.txt`'s first 300 bytes
(raw) attached.
- `romeo-mutated.sd-utf8.br` is a mutated `romeo.txt.q5w10.br` that decodes
(to `romeo-mutated.sd-utf8.txt`) without error when
`shared-dictionary.utf8.bin` is attached, and whose dictionary references
apply shift transforms to 1, 2, 3 and 4 byte UTF-8 sequences.
- `invalid.lw-*.br`, `invalid.raw-*.br` and `invalid.sd-*.br` are malformed.
They were found by mutating the files above, like the other `invalid.*.br`
files, with the large-window quirk set (`lw`), or the raw (`raw`, the
`romeo.txt` chunks) or serialized (`sd`, `shared-dictionary.full.bin`)
dictionaries attached.