Completes the decoder's feature set relative to libbrotlidec v1.2.1: - QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the large-window WBITS escape, window bits up to 30, the 544-symbol distance alphabet and 64-bit distance arithmetic, with workbuf_len growing to (1 << 30) + 542 when the quirk is set. - decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw (LZ77 prefix) and serialized shared dictionaries: compound-dictionary distance mapping, the shared-dictionary parser (custom word lists, transform lists including the SHIFT transforms, context-dependent dictionary selection, prefix dictionaries) and all of its validation. Wuffs structs cannot hold slices, so the dictionary bytes are copied into the front of the caller's workbuf; the layout is documented in the README. Serialized dictionaries are accepted as in a BROTLI_EXPERIMENTAL build of libbrotli. - fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and random I/O limits. Tests: new golden vectors for large windows (including a 17 MiB far-distance stream generated in the test), raw and serialized dictionaries, nine more malformed inputs, and mimic comparisons that attach the same dictionaries to libbrotlidec. Differential runs over 2.7 million mutated large-window and dictionary streams found no divergence from libbrotlidec. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
4.6 KiB
Brotli Test Data
Files without a .br suffix, and their .compressed (and .compressed.NN)
counterparts, were copied from the tests/testdata directory of the reference
brotli implementation at tag v1.2.1. That
repository is licensed under the MIT license: "Copyright (c) 2009, 2010,
2013-2016 by the Brotli Authors".
Some of the upstream files were not copied:
lcet10.txt,mapsdatazrh,plrabn12.txtandzerosukkanooa, because their decompressed forms are larger than 256 KiB.bb.binastandrandom_chunks, because they have no.compressedform.
The *.qNwM.br files were generated by the brotli v1.2.1 command line tool,
from ../romeo.txt, ../256.bytes and alice29.txt, at quality level N and
with window bits M:
brotli -q N -w M -o romeo.txt.qNwM.br ../romeo.txt
The small window (w10 and w16) alice29.txt files exercise the history
ring buffer wrapping around.
The invalid.*.br files are malformed. Apart from invalid.truncated.br (the
first 20 bytes of romeo.txt.q11w22.br), they were found by randomly mutating
the other (valid) files and keeping, for each libbrotlidec v1.2.1 error code,
the smallest mutated file that produced that error code. The file name
mentions that error code, e.g. invalid.cl-space.br produces
BROTLI_DECODER_ERROR_FORMAT_CL_SPACE.
Large Window and Dictionary Test Data
These files exercise the QUIRK_LARGE_WINDOW quirk and add_dictionary. They
were generated with brotli v1.2.1: the command line tool where it suffices
and otherwise a small C program calling BrotliEncoderSetParameter (with
BROTLI_PARAM_LARGE_WINDOW), BrotliEncoderPrepareDictionary and
BrotliEncoderAttachPreparedDictionary, linked against a libbrotlienc built
with BROTLI_EXPERIMENTAL (needed for serialized shared dictionaries). Every
file's decompressed form was checked with libbrotlidec (also built with
BROTLI_EXPERIMENTAL).
*.qNlwM.brare large-window streams (quality N, window bits M), e.g.brotli -q 11 --large_window=26 -o alice29.txt.q11lw26.br alice29.txt.far17m.q9lw25.bris a large-window (25 window bits) quality 9 stream whose decompressed form (17 MiB plus 128 KiB, generated by the test program instead of being checked in) is 64 KiB of xorshift32 output, 17 MiB of zeroes and the same 64 KiB again. The second copy refers back more than (1 << 24) bytes. A standard (24 window bits) encoding is twice as large.*.raw-romeo.bruse../romeo.txtas a raw (LZ77 prefix) dictionary, e.g.brotli -q 11 -w 22 -D ../romeo.txt -o romeo.txt.q11w22.raw-romeo.br ../romeo.txt.*.raw-chunks.bruse several raw dictionaries (compound dictionary chunks):alice29.txt's bytes[0 .. 20000),[20000 .. 20001),[20001 .. 20001)(empty) and[20001 .. 60000), or../romeo.txt's bytes[0 .. 300),[300 .. 500)and[500 .. 942).shared-dictionary.*.binare serialized shared dictionaries (the format parsed byBrotliSharedDictionaryAttachin brotli'sc/common/shared_dictionary.c), built by a script:fullhas a 500 byte LZ77 prefix (the start of../romeo.txt), two custom word lists (common words fromalice29.txtandasyoulik.txt), two custom transform lists (includingBROTLI_TRANSFORM_SHIFT_FIRSTandBROTLI_TRANSFORM_SHIFT_ALLtransforms), three (word list, transform list) combinations (the third is the built-in pair) and a context map.wordshas one custom word list, the built-in transforms and no prefix.prefixhas only an LZ77 prefix (the start ofasyoulik.txt).utf8has a word list of UTF-8 (and invalid UTF-8) words and shift transforms with various parameters.
shared-dictionary.textis made ofshared-dictionary.full.binwords, transformed by its transforms.*.sd-full.br(and*.sd-words.br) were compressed withshared-dictionary.full.bin(and.words.bin) attached.asyoulik.txt.q11w18.sd-prefix-raw.brwas compressed withshared-dictionary.prefix.binand then../romeo.txt's first 300 bytes (raw) attached.romeo-mutated.sd-utf8.bris a mutatedromeo.txt.q5w10.brthat decodes (toromeo-mutated.sd-utf8.txt) without error whenshared-dictionary.utf8.binis attached, and whose dictionary references apply shift transforms to 1, 2, 3 and 4 byte UTF-8 sequences.invalid.lw-*.br,invalid.raw-*.brandinvalid.sd-*.brare malformed. They were found by mutating the files above, like the otherinvalid.*.brfiles, with the large-window quirk set (lw), or the raw (raw, theromeo.txtchunks) or serialized (sd,shared-dictionary.full.bin) dictionaries attached.