std/brotli: large-window mode, custom dictionaries and a fuzzer

Completes the decoder's feature set relative to libbrotlidec v1.2.1:

- QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the
  large-window WBITS escape, window bits up to 30, the 544-symbol distance
  alphabet and 64-bit distance arithmetic, with workbuf_len growing to
  (1 << 30) + 542 when the quirk is set.
- decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw
  (LZ77 prefix) and serialized shared dictionaries: compound-dictionary
  distance mapping, the shared-dictionary parser (custom word lists,
  transform lists including the SHIFT transforms, context-dependent
  dictionary selection, prefix dictionaries) and all of its validation.
  Wuffs structs cannot hold slices, so the dictionary bytes are copied
  into the front of the caller's workbuf; the layout is documented in
  the README. Serialized dictionaries are accepted as in a
  BROTLI_EXPERIMENTAL build of libbrotli.
- fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and
  random I/O limits.

Tests: new golden vectors for large windows (including a 17 MiB
far-distance stream generated in the test), raw and serialized
dictionaries, nine more malformed inputs, and mimic comparisons that
attach the same dictionaries to libbrotlidec. Differential runs over
2.7 million mutated large-window and dictionary streams found no
divergence from libbrotlidec.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
This commit is contained in:
Claude committed 2026-10-26 23:45:06 +00:00
1 parent 4978207531
commit b9be8feaee
43 files changed
+4253 -460

No files matched your search

+54
View File
@@ -27,3 +27,57 @@ the other (valid) files and keeping, for each libbrotlidec v1.2.1 error code,
the smallest mutated file that produced that error code. The file name
mentions that error code, e.g. `invalid.cl-space.br` produces
`BROTLI_DECODER_ERROR_FORMAT_CL_SPACE`.
## Large Window and Dictionary Test Data
These files exercise the `QUIRK_LARGE_WINDOW` quirk and `add_dictionary`. They
were generated with brotli v1.2.1: the command line tool where it suffices
and otherwise a small C program calling `BrotliEncoderSetParameter` (with
`BROTLI_PARAM_LARGE_WINDOW`), `BrotliEncoderPrepareDictionary` and
`BrotliEncoderAttachPreparedDictionary`, linked against a libbrotlienc built
with `BROTLI_EXPERIMENTAL` (needed for serialized shared dictionaries). Every
file's decompressed form was checked with libbrotlidec (also built with
`BROTLI_EXPERIMENTAL`).
- `*.qNlwM.br` are large-window streams (quality N, window bits M), e.g.
`brotli -q 11 --large_window=26 -o alice29.txt.q11lw26.br alice29.txt`.
- `far17m.q9lw25.br` is a large-window (25 window bits) quality 9 stream whose
decompressed form (17 MiB plus 128 KiB, generated by the test program
instead of being checked in) is 64 KiB of xorshift32 output, 17 MiB of
zeroes and the same 64 KiB again. The second copy refers back more than (1
<< 24) bytes. A standard (24 window bits) encoding is twice as large.
- `*.raw-romeo.br` use `../romeo.txt` as a raw (LZ77 prefix) dictionary, e.g.
`brotli -q 11 -w 22 -D ../romeo.txt -o romeo.txt.q11w22.raw-romeo.br
../romeo.txt`.
- `*.raw-chunks.br` use several raw dictionaries (compound dictionary chunks):
`alice29.txt`'s bytes `[0 .. 20000)`, `[20000 .. 20001)`, `[20001 .. 20001)`
(empty) and `[20001 .. 60000)`, or `../romeo.txt`'s bytes `[0 .. 300)`, `[300
.. 500)` and `[500 .. 942)`.
- `shared-dictionary.*.bin` are serialized shared dictionaries (the format
parsed by `BrotliSharedDictionaryAttach` in brotli's
`c/common/shared_dictionary.c`), built by a script:
- `full` has a 500 byte LZ77 prefix (the start of `../romeo.txt`), two
custom word lists (common words from `alice29.txt` and `asyoulik.txt`),
two custom transform lists (including `BROTLI_TRANSFORM_SHIFT_FIRST` and
`BROTLI_TRANSFORM_SHIFT_ALL` transforms), three (word list, transform
list) combinations (the third is the built-in pair) and a context map.
- `words` has one custom word list, the built-in transforms and no prefix.
- `prefix` has only an LZ77 prefix (the start of `asyoulik.txt`).
- `utf8` has a word list of UTF-8 (and invalid UTF-8) words and shift
transforms with various parameters.
- `shared-dictionary.text` is made of `shared-dictionary.full.bin` words,
transformed by its transforms. `*.sd-full.br` (and `*.sd-words.br`) were
compressed with `shared-dictionary.full.bin` (and `.words.bin`) attached.
`asyoulik.txt.q11w18.sd-prefix-raw.br` was compressed with
`shared-dictionary.prefix.bin` and then `../romeo.txt`'s first 300 bytes
(raw) attached.
- `romeo-mutated.sd-utf8.br` is a mutated `romeo.txt.q5w10.br` that decodes
(to `romeo-mutated.sd-utf8.txt`) without error when
`shared-dictionary.utf8.bin` is attached, and whose dictionary references
apply shift transforms to 1, 2, 3 and 4 byte UTF-8 sequences.
- `invalid.lw-*.br`, `invalid.raw-*.br` and `invalid.sd-*.br` are malformed.
They were found by mutating the files above, like the other `invalid.*.br`
files, with the large-window quirk set (`lw`), or the raw (`raw`, the
`romeo.txt` chunks) or serialized (`sd`, `shared-dictionary.full.bin`)
dictionaries attached.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1 @@
âvŒ¸)MëÍ{> ài©Ã`n␍gήy¼€ÈfÏò0¬aQIÿËS)”߇„
@@ -0,0 +1 @@
âvŒ¸)MëÍ{> ài©Ã`n␍gήy¼€ÈfÏò‰Aióûì×baQIÿË])”Þ‡„rÁ|<E
+1
View File
@@ -0,0 +1 @@
âvŒ¸)MëÍ{> ài©Ã`n␍gήy¼€ÈfÏò0­�¯>n€‰Áióû
Binary file not shown.
@@ -0,0 +1,45 @@
Romeo and Juliet
Excerpt Ÿ˜€ Act 2, Scene 2
JULIET
O €xy,ULIET
! linksfore art thouULIET
?
Deny thy signedo andtake p thy name;
Or, Contaou wilt not, be but sworn my `₫é,
And I'll the worlde bua Capulet.
ROMEO
[Aside] Sif(! I usin more, or sif(! I Perso athouis?JULIET
O 'Tisut swy name;
Italiany `âenemy;
Tu wit thouyhold,aou wghot, a Montague.
What's Montague? remaininr hand,ninr foot,
Nnr arm,ninr face,ninr They were part
Be woring to a man. Obe busomey were me;
!
What's in a me;
? Italilinks we cf(! a four
By They were me;
situatsme(! as onkey;
SoIET
?
situa, were heot, ET
?
cf(!'d,
Retaoutbrat dsin perf
Whililinks he owes
Wiou wttbrat نك.€xy,ULd
Many name;©,
Andf; widthe me;
linksmainie par =falsee
Takeo ll muyholt.
ROMEOI takeolseeo athoy das:
Co ll e but `â‚«Ão andd I'lbe new missized;
Hencsforth I sound investor€xy.s?JULIET
!
Wh a m art thouU? Itahide be🗿èé'd in €¿øÿ;
Sostumblest o my `counsel?
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+41
View File
@@ -0,0 +1,41 @@
causeHAVE 
uncoTURTLE, uhoughtfullycharles -audrey-fatherdaughter Somethingsheph HASTILY, phebeSHOULDSHEPHERDInterruptedlike fortune things. hoingextraordinary. erhaps said aterpillar Wonderrabbi well everything's Everything charlesthere corin anything's -silvius-WILLIAMDisappointment better RepliedDAUGHTER will thoughtfully  QuestionsilviusAnythingturtle
looking mucquite
charlesthe disappointment INTERRUPTED, that FORTUNEuncomfortable, daughter Childrenph the uncomfortable right
-love-bettermistresstraordinaryreplied's -brother-derstand there t yourorlando-should-senior bnmudqr`shnm ph this -should- 
haveSENIOR the children extraordinary. wouldcon Dormousenversation-audrey- h silvius knnjdc sen the executioner disappointmentcorin oli y okingexecutioner children g`ssdq phebe -daughter-w adventures's -jaques-frightened's opportunity's the interrupted w mistress little. thoughtfully
anything -should- THATContemptuouslyeconthings. contemptuouslrself t corin CORIN-mistress-will noonqstmhsx beginning Extraordinaryextraordinary conversatioOpportunity -father-getting. enter conversatioPRITHEEcontesheph mockanother. HAVE ␍inute l thoughtfully's withperhaps-orlando-seemed
Wonderforestthe frightened which willremember. have co THAT, daughter -william-Disappointmenttuchdaughter  -audrey-LOOKED, the turtle orlandofather sh Interrupted beginning could's   rosalindhave xtraordinary Looking jaquesnversationuncomfortablable disappointment phebe -orlando-dormouse mistresshoughtfully THERE nterrupted op thoughtfullextraordinary, -there-contemptuouslySHALLshall jaq creatures -senior-phebe t CREATURES, sheph ed Thoughtfullythe nothing LookingINTO, everything con Alice-prithee-charlesursewhichlooked rather uWhile White DAUGHTERDisappointment love your Opportunitydifferent. rosalind -would-EVERYTHING, -father- father -oliver-ANOTHER, en enough celia-forest- the thought ROSALINDseniorFORTUNEFATHERdhildren uncomfortable's ce -that-love mistr ONLY, should phebe-brother-l eyeifferent orlandoInterrupted-senior-pheberueen ExtraordinaryBEFORE, shepherdCHARLESbetter should brotherfortune t -orlando-audrey mistr daugh will sen disappointmentextraordinarythe things withieardokd`rd HAVE disappointment, sh ph noonqstmhsx very-oliver-ce daughterbdqs`hmkx w BROTHERTHISl ORLANDO the other oli will nversationconversatiowhich ously mistr  ce rosalind because's thoughtfull disappointment -this-AUDREYdisappointment mistr the understand frightened
y gdqrdke Creatures onversation AUDREY conversation went's extraordinary's WITHextrautifulanotheren interrupted -rosalind-orlando-orlando-Extraordinaryeirrosalind suddenly, -charles-charlesvoice
bnythingonversation t CHARLES-another--silvius-contemptuously
UNCOMFORTABLE, caterpillashepherd uncomfortable -father-father would xtraordinary SHOULDvgdm JAQUES-there--shepherd-creatures. Little ROSALIND-will- DAUGHTER-another-daugh rosal without's nary curious. ckaud would MISTRESScharles thou shallCertainly-orlando-extraordinary
seniordaughter thoughtfullshepherdWITHonly. -senior-Continued this EXTRAORDINARY, fortune 
THAT-audrey-ufat more -charles-things. BROTHERFRIGHTENED, sghmf THERElovethe certainly aud this -will- brotherthe getting 
with's shepherdoli THIS -another-the seemed xiouslyoliver-shepherd-caterpillar
which wh daugh mistress Opportunity getting -rosalind-dmntfg ANOTHERthoughtfully SHOULDprithee oliverbet eisappointment thou certainly With methingsho replied. 
this shepherdcharlesrosalindbetter en sheph -which--orlando--celia-rightened ke fat the beautiful will HAVE-have--love-WOULD thisfor MUCH, well's thouprit contemptuously -celia-vncomfortablelly-oliver- finishelovethe conversation thoughtfullyOLIVER-enter-ar LOOKED, onversation -love-phebe Conversationuncomfortable, extraordinary
mistressbefore. onder adventures have-thou-SILVIUS-silvius-conversation's t more's daughtersho -shepherd-white, tmcdqrs`mc rosal thingtalking,   alice. ormouse Thinking remember Round  Perhapsthere for MISTRESS beginning. your-this-anxiouslmistress UNCOMFORTABLE, Uncomfortable -phebe-wh etting CONTEMPTUOUSLY, silvius isappointment thoadventures's extraordinar caterpillar mistressthe interrupted  
  rosal seniorreat SILVIUSnough daughter able contecorin mistress-charles-going
Contemptuously brot nderstand HAVEAUDREY your celia -should- willCONTEMPTUOUSLY, -audrey-sho thiseginning char willtmentmarch's sheph fverythingconversation. the mouse -audrey-there silv othing willonly. the understand LOVEsheph daugh BeforekingecutionerConversation AUDREYPRITHEE-which-the should -jaques- should. withginishedbet tmbnlenqs`akd like. beginning -should-rosalind NOTHING, bnmsdlostntrkx conversation  
large. something-rosalind--mistress--another--there--william-Contemptuously That executioner's thoughtfully, thinking However fat uhen shouldperhapDAUGHTERopSHOULD disappointment the extraordinary thereqerhapscorinThoughtfullywould contemptuously orlando terpillarmistress about -rosalind--fortune-abbit thereanot interrupted-your-this  such -daughter-the queen ph Conversationsgntfgsetkkx orlando silvius -father- Anything extraordinary Before Begant there uhemthe could THATperhaps ruestionsgntfgsetkkx aud rosal getting
mistr mistr extraordinary
ierecharles thinkingYOURsilv t interrupted
which audrey daugh thothis Disappointment extraordinary, your voice edEXTRAORDINARY, comfortable conversation,  -prithee-h HEARD, which everythin xtraordinary understand children white. -father-WhatThoughtfully co silviusmust
ightenedrosal would somethin -father--shall--would-extraordinary's SHEPHERDextrjaq there InterruptedInterrupted THATsen SENIORyour conversation
celia -celia--better--love-   SHOULDphebe OLIVERmistr remember Uncomfortable -jaques-rosal aud conversation THOUGHTFULLY, the before WOULD-forest-ere -mistress-williambrother -will--that-THAT-prithee-for fatherWILL rosalindxecutioner understand's much, brother come -mistress-dwdbtshnmdq SILVIUSaudrey nomentonversation -brother- ROSALINDyourppportunitywithshepherden corin jaques withthis EVERYTHING, the were -fortune- vncomfortablemistress  celiarosalindthosghmfr remember WILLIAMaud Disappointmentcorin fxecutionerntemptuously daughterell Extraordinaryad`tshetk thouopportunity venturesthe come -shepherd-uchess caterpillar. mistressSHOULD   -rosalind-celia ROSALINDntemptuouslyEXTRAORDINARY, know fortune the again THEREthere disappointmenthoughtfullygryphon. SHOULDmistr CORIN-shall-thinkingEVERYTHING, OLIVERSomething DAUGHTERyourOLIVERshepherd thoughtfully. -orlando-FORESTBROTHERrselffortune -which-y lovephebe-celia-dontemptuouslyund Executionerlike Looking-prithee-WOULDlove-forest- -thou--father-ConversationrosalindCreatures fortune uncomfortable phebeinterrupted WILLIAMSHEPHERDDisappointment wh xecutioner should rosalind jaquesDAUGHTERremembe SILVIUSfverythingaterpillar HAVEthe remember Executionerw theirwent charlesbetterhead DAUGHTERcontemptuousl FORTUNEbrot CATERPILLAR, eard gryphon
WOULDshepherd enterloveuncomfortable, Understand-oliver-ENTERthe quite -rosalind-fortunedaugh bettercontemptuouslsomCELIA the voice betterthey, -should-audreywould co understandTHOUnversation 
lice -enter- Caterpillar THINKING, shepherd Disappointmentoliver-mistress-senior co WILLIAMBROTHERrosal jaquesORLANDOuncomfortable
executioney your, FORTUNE-there-mistr mistr sheph with
en mistress the without Contemptuouslyphebe -daughter-eautiful nterrupted orlandoopportunity's contemptuouslybeginning
brot Going anot usecurious, w shepherd dormouse
uncomfortable Beautifulgryphon's MISTRESSdaughter Everything erythingEXTRAORDINARY, Thoughtfully -charles-extraordinary, ion  while
forest daugh SILVIUSyour Caterpillar en suddenly vnmcdq PHEBE-shepherd-shoulddventures fortune understand
EXTRAORDINARY, Adventuresdontemptuouslyable hastily, thoughtfully. Frightened which h mistr mistress  -shall-enterthe understand -better-uestion Contemptuously ph willinterrupted Gettingknow, shall into. hmsn Mock Continued seemed have extraordinary's march, celiaWHICHb`sdqohkk`q SILVIUSunderstanDAUGHTER the caterpillar dudqxsghmf better -another-THERE brother understand. -there-phebe ableYOUR-celia- t FORTUNE AUDREY-shepherd-enter extraordinar mistress WHICHchildren's disappointment's BROTHERddenlyGetting tmbnlenqs`akd DAUGHTERTHERE-silvius-vncomfortablerosal remember, BETTERWITHdaugh Creaturesdwithout, contemptuously's PHEBEdaughterWILLthincelia opportunit ightenedextraordinary -oliver-Extraordinarycorinicethoughtfully
Thoughtfullythis-that- LookedConversation vhsg ␍uch large littlorla -corin-mistress there-this- conversation. extraordinary's thoughtfullExtraordinary must, portunityDAUGHTER conversation's william Everything-daughter-frightened's sho Finished another, mistr CONTEMPTUOUSLY, -love-the creatures sheph the suddenly orla seemed. vncomfortablefor ith w senior rabbitmistress ANOTHERtho-daughter--shepherd-remember's fxtraordinary-shall-opportunity-audrey-our Thoughtorlando -shall--shall- BeginningInterrupted FINISHED, tmentorlando -charles-uhemwhat's -orlando-contemptuously's tuddenlyCOURSE, CHILDREN, phebe -celia- something betterunderstandCHARLES thoughtfully, different's white -there-Frightened y silvius orla jnterruptedFORTUNEph -thou-hoing WILLIAMturtle's rabbithe beautiful Thoughtfully disappointment. Beginning VERY, seemed said ph char HOWEVER, rosalind BeautifulCURIOUS, RABBIT, extraordinary
Anything SILVIUSextraordinar thoughtfully. -better-your shall dontemptuouslyfort SHEPHERD daterpillarsh daughter oli disappointmententer Children -jaques-haves`kjhmf orlando SENIORwhich daughter down Uncomfortable rosalindrosal co mistr might. -should-phebe wo loveq`aahs opportunity. Uncomfortable oprosalind ooking -which-thou the thinking marchbefore, -with--there-bet t oliverfortune-celia-jaquesprithee th -your- -fortune--senior-disappointment lly-with-CONTEMPTUOUSLY, come. charles WHEN, sh prit onversation the again -shepherd-jaques silvius RoundBROTHERbrot enter corin orlandosabbitWOULD-this-getting. CONVERSATION, which Questioncorin thatORLANDOthe such hings thoughtfully en aud Themmistr rious Contemptuously daughterround, the caterpillar Disappointment art shall INTERRUPTED, goingSHEPHERDsuddenly. -better- THISWILL-orlando-THAT Anxiously contemptuously. FORTUNEagain
-have-HAVETHOUGHTFULLY, nother Uncomfortablesho -forest-corin DAUGHTERHavefrightened everything
would onversation sh -your--with-  shepherd ChildrenorlandoCHARLESbrother Went something -which-rosalind-silvius-anotherdaughter CELIAHerself extraordinary
Enough Extraordinary-prithee-wo corin HAVE-prithee-daughterhoweverce adventures w must phebe extraordinary's SHOULD-brother-traordinaryrosalind  will ouldrosalind the something -that--prithee- king
father  the uncomfortable FOREST First  brotherquestion
WENT, THEREmistr  -prithee- Uncomfortablefort with question outwould WOULDsheph BETTERright's narybrother william tting-will-WHICHSILVIUS-another-onder bnmudqr`shnm JAQUES -better-they's THISphebeWOULD -mistress-Anxiously thouContemptuously With ittle sheph Caterpillardifferent. 
THISThoughtfullybrother -charles- sen betterTHISwould -senior-that that bnmudqr`shnm CURIOUS,
Binary file not shown.
Binary file not shown.