mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-18 08:32:11 +01:00
Completes the decoder's feature set relative to libbrotlidec v1.2.1: - QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the large-window WBITS escape, window bits up to 30, the 544-symbol distance alphabet and 64-bit distance arithmetic, with workbuf_len growing to (1 << 30) + 542 when the quirk is set. - decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw (LZ77 prefix) and serialized shared dictionaries: compound-dictionary distance mapping, the shared-dictionary parser (custom word lists, transform lists including the SHIFT transforms, context-dependent dictionary selection, prefix dictionaries) and all of its validation. Wuffs structs cannot hold slices, so the dictionary bytes are copied into the front of the caller's workbuf; the layout is documented in the README. Serialized dictionaries are accepted as in a BROTLI_EXPERIMENTAL build of libbrotli. - fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and random I/O limits. Tests: new golden vectors for large windows (including a 17 MiB far-distance stream generated in the test), raw and serialized dictionaries, nine more malformed inputs, and mimic comparisons that attach the same dictionaries to libbrotlidec. Differential runs over 2.7 million mutated large-window and dictionary streams found no divergence from libbrotlidec. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
84 lines
4.6 KiB
Markdown
84 lines
4.6 KiB
Markdown
# Brotli Test Data
|
|
|
|
Files without a `.br` suffix, and their `.compressed` (and `.compressed.NN`)
|
|
counterparts, were copied from the `tests/testdata` directory of the [reference
|
|
brotli implementation](https://github.com/google/brotli) at tag v1.2.1. That
|
|
repository is licensed under the MIT license: "Copyright (c) 2009, 2010,
|
|
2013-2016 by the Brotli Authors".
|
|
|
|
Some of the upstream files were not copied:
|
|
|
|
- `lcet10.txt`, `mapsdatazrh`, `plrabn12.txt` and `zerosukkanooa`, because their
|
|
decompressed forms are larger than 256 KiB.
|
|
- `bb.binast` and `random_chunks`, because they have no `.compressed` form.
|
|
|
|
The `*.qNwM.br` files were generated by the brotli v1.2.1 command line tool,
|
|
from `../romeo.txt`, `../256.bytes` and `alice29.txt`, at quality level N and
|
|
with window bits M:
|
|
|
|
brotli -q N -w M -o romeo.txt.qNwM.br ../romeo.txt
|
|
|
|
The small window (`w10` and `w16`) `alice29.txt` files exercise the history
|
|
ring buffer wrapping around.
|
|
|
|
The `invalid.*.br` files are malformed. Apart from `invalid.truncated.br` (the
|
|
first 20 bytes of `romeo.txt.q11w22.br`), they were found by randomly mutating
|
|
the other (valid) files and keeping, for each libbrotlidec v1.2.1 error code,
|
|
the smallest mutated file that produced that error code. The file name
|
|
mentions that error code, e.g. `invalid.cl-space.br` produces
|
|
`BROTLI_DECODER_ERROR_FORMAT_CL_SPACE`.
|
|
|
|
|
|
## Large Window and Dictionary Test Data
|
|
|
|
These files exercise the `QUIRK_LARGE_WINDOW` quirk and `add_dictionary`. They
|
|
were generated with brotli v1.2.1: the command line tool where it suffices
|
|
and otherwise a small C program calling `BrotliEncoderSetParameter` (with
|
|
`BROTLI_PARAM_LARGE_WINDOW`), `BrotliEncoderPrepareDictionary` and
|
|
`BrotliEncoderAttachPreparedDictionary`, linked against a libbrotlienc built
|
|
with `BROTLI_EXPERIMENTAL` (needed for serialized shared dictionaries). Every
|
|
file's decompressed form was checked with libbrotlidec (also built with
|
|
`BROTLI_EXPERIMENTAL`).
|
|
|
|
- `*.qNlwM.br` are large-window streams (quality N, window bits M), e.g.
|
|
`brotli -q 11 --large_window=26 -o alice29.txt.q11lw26.br alice29.txt`.
|
|
- `far17m.q9lw25.br` is a large-window (25 window bits) quality 9 stream whose
|
|
decompressed form (17 MiB plus 128 KiB, generated by the test program
|
|
instead of being checked in) is 64 KiB of xorshift32 output, 17 MiB of
|
|
zeroes and the same 64 KiB again. The second copy refers back more than (1
|
|
<< 24) bytes. A standard (24 window bits) encoding is twice as large.
|
|
- `*.raw-romeo.br` use `../romeo.txt` as a raw (LZ77 prefix) dictionary, e.g.
|
|
`brotli -q 11 -w 22 -D ../romeo.txt -o romeo.txt.q11w22.raw-romeo.br
|
|
../romeo.txt`.
|
|
- `*.raw-chunks.br` use several raw dictionaries (compound dictionary chunks):
|
|
`alice29.txt`'s bytes `[0 .. 20000)`, `[20000 .. 20001)`, `[20001 .. 20001)`
|
|
(empty) and `[20001 .. 60000)`, or `../romeo.txt`'s bytes `[0 .. 300)`, `[300
|
|
.. 500)` and `[500 .. 942)`.
|
|
- `shared-dictionary.*.bin` are serialized shared dictionaries (the format
|
|
parsed by `BrotliSharedDictionaryAttach` in brotli's
|
|
`c/common/shared_dictionary.c`), built by a script:
|
|
- `full` has a 500 byte LZ77 prefix (the start of `../romeo.txt`), two
|
|
custom word lists (common words from `alice29.txt` and `asyoulik.txt`),
|
|
two custom transform lists (including `BROTLI_TRANSFORM_SHIFT_FIRST` and
|
|
`BROTLI_TRANSFORM_SHIFT_ALL` transforms), three (word list, transform
|
|
list) combinations (the third is the built-in pair) and a context map.
|
|
- `words` has one custom word list, the built-in transforms and no prefix.
|
|
- `prefix` has only an LZ77 prefix (the start of `asyoulik.txt`).
|
|
- `utf8` has a word list of UTF-8 (and invalid UTF-8) words and shift
|
|
transforms with various parameters.
|
|
- `shared-dictionary.text` is made of `shared-dictionary.full.bin` words,
|
|
transformed by its transforms. `*.sd-full.br` (and `*.sd-words.br`) were
|
|
compressed with `shared-dictionary.full.bin` (and `.words.bin`) attached.
|
|
`asyoulik.txt.q11w18.sd-prefix-raw.br` was compressed with
|
|
`shared-dictionary.prefix.bin` and then `../romeo.txt`'s first 300 bytes
|
|
(raw) attached.
|
|
- `romeo-mutated.sd-utf8.br` is a mutated `romeo.txt.q5w10.br` that decodes
|
|
(to `romeo-mutated.sd-utf8.txt`) without error when
|
|
`shared-dictionary.utf8.bin` is attached, and whose dictionary references
|
|
apply shift transforms to 1, 2, 3 and 4 byte UTF-8 sequences.
|
|
- `invalid.lw-*.br`, `invalid.raw-*.br` and `invalid.sd-*.br` are malformed.
|
|
They were found by mutating the files above, like the other `invalid.*.br`
|
|
files, with the large-window quirk set (`lw`), or the raw (`raw`, the
|
|
`romeo.txt` chunks) or serialized (`sd`, `shared-dictionary.full.bin`)
|
|
dictionaries attached.
|