Files
wuffs/test/data/brotli/README.md
T
Claude b9be8feaee std/brotli: large-window mode, custom dictionaries and a fuzzer
Completes the decoder's feature set relative to libbrotlidec v1.2.1:

- QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the
  large-window WBITS escape, window bits up to 30, the 544-symbol distance
  alphabet and 64-bit distance arithmetic, with workbuf_len growing to
  (1 << 30) + 542 when the quirk is set.
- decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw
  (LZ77 prefix) and serialized shared dictionaries: compound-dictionary
  distance mapping, the shared-dictionary parser (custom word lists,
  transform lists including the SHIFT transforms, context-dependent
  dictionary selection, prefix dictionaries) and all of its validation.
  Wuffs structs cannot hold slices, so the dictionary bytes are copied
  into the front of the caller's workbuf; the layout is documented in
  the README. Serialized dictionaries are accepted as in a
  BROTLI_EXPERIMENTAL build of libbrotli.
- fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and
  random I/O limits.

Tests: new golden vectors for large windows (including a 17 MiB
far-distance stream generated in the test), raw and serialized
dictionaries, nine more malformed inputs, and mimic comparisons that
attach the same dictionaries to libbrotlidec. Differential runs over
2.7 million mutated large-window and dictionary streams found no
divergence from libbrotlidec.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
2026-10-26 23:45:06 +00:00

4.6 KiB

Brotli Test Data

Files without a .br suffix, and their .compressed (and .compressed.NN) counterparts, were copied from the tests/testdata directory of the reference brotli implementation at tag v1.2.1. That repository is licensed under the MIT license: "Copyright (c) 2009, 2010, 2013-2016 by the Brotli Authors".

Some of the upstream files were not copied:

  • lcet10.txt, mapsdatazrh, plrabn12.txt and zerosukkanooa, because their decompressed forms are larger than 256 KiB.
  • bb.binast and random_chunks, because they have no .compressed form.

The *.qNwM.br files were generated by the brotli v1.2.1 command line tool, from ../romeo.txt, ../256.bytes and alice29.txt, at quality level N and with window bits M:

brotli -q N -w M -o romeo.txt.qNwM.br ../romeo.txt

The small window (w10 and w16) alice29.txt files exercise the history ring buffer wrapping around.

The invalid.*.br files are malformed. Apart from invalid.truncated.br (the first 20 bytes of romeo.txt.q11w22.br), they were found by randomly mutating the other (valid) files and keeping, for each libbrotlidec v1.2.1 error code, the smallest mutated file that produced that error code. The file name mentions that error code, e.g. invalid.cl-space.br produces BROTLI_DECODER_ERROR_FORMAT_CL_SPACE.

Large Window and Dictionary Test Data

These files exercise the QUIRK_LARGE_WINDOW quirk and add_dictionary. They were generated with brotli v1.2.1: the command line tool where it suffices and otherwise a small C program calling BrotliEncoderSetParameter (with BROTLI_PARAM_LARGE_WINDOW), BrotliEncoderPrepareDictionary and BrotliEncoderAttachPreparedDictionary, linked against a libbrotlienc built with BROTLI_EXPERIMENTAL (needed for serialized shared dictionaries). Every file's decompressed form was checked with libbrotlidec (also built with BROTLI_EXPERIMENTAL).

  • *.qNlwM.br are large-window streams (quality N, window bits M), e.g. brotli -q 11 --large_window=26 -o alice29.txt.q11lw26.br alice29.txt.
  • far17m.q9lw25.br is a large-window (25 window bits) quality 9 stream whose decompressed form (17 MiB plus 128 KiB, generated by the test program instead of being checked in) is 64 KiB of xorshift32 output, 17 MiB of zeroes and the same 64 KiB again. The second copy refers back more than (1 << 24) bytes. A standard (24 window bits) encoding is twice as large.
  • *.raw-romeo.br use ../romeo.txt as a raw (LZ77 prefix) dictionary, e.g. brotli -q 11 -w 22 -D ../romeo.txt -o romeo.txt.q11w22.raw-romeo.br ../romeo.txt.
  • *.raw-chunks.br use several raw dictionaries (compound dictionary chunks): alice29.txt's bytes [0 .. 20000), [20000 .. 20001), [20001 .. 20001) (empty) and [20001 .. 60000), or ../romeo.txt's bytes [0 .. 300), [300 .. 500) and [500 .. 942).
  • shared-dictionary.*.bin are serialized shared dictionaries (the format parsed by BrotliSharedDictionaryAttach in brotli's c/common/shared_dictionary.c), built by a script:
    • full has a 500 byte LZ77 prefix (the start of ../romeo.txt), two custom word lists (common words from alice29.txt and asyoulik.txt), two custom transform lists (including BROTLI_TRANSFORM_SHIFT_FIRST and BROTLI_TRANSFORM_SHIFT_ALL transforms), three (word list, transform list) combinations (the third is the built-in pair) and a context map.
    • words has one custom word list, the built-in transforms and no prefix.
    • prefix has only an LZ77 prefix (the start of asyoulik.txt).
    • utf8 has a word list of UTF-8 (and invalid UTF-8) words and shift transforms with various parameters.
  • shared-dictionary.text is made of shared-dictionary.full.bin words, transformed by its transforms. *.sd-full.br (and *.sd-words.br) were compressed with shared-dictionary.full.bin (and .words.bin) attached. asyoulik.txt.q11w18.sd-prefix-raw.br was compressed with shared-dictionary.prefix.bin and then ../romeo.txt's first 300 bytes (raw) attached.
  • romeo-mutated.sd-utf8.br is a mutated romeo.txt.q5w10.br that decodes (to romeo-mutated.sd-utf8.txt) without error when shared-dictionary.utf8.bin is attached, and whose dictionary references apply shift transforms to 1, 2, 3 and 4 byte UTF-8 sequences.
  • invalid.lw-*.br, invalid.raw-*.br and invalid.sd-*.br are malformed. They were found by mutating the files above, like the other invalid.*.br files, with the large-window quirk set (lw), or the raw (raw, the romeo.txt chunks) or serialized (sd, shared-dictionary.full.bin) dictionaries attached.