mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-18 16:32:11 +01:00
std/brotli: large-window mode, custom dictionaries and a fuzzer
Completes the decoder's feature set relative to libbrotlidec v1.2.1: - QUIRK_LARGE_WINDOW mirrors BROTLI_DECODER_PARAM_LARGE_WINDOW: the large-window WBITS escape, window bits up to 30, the 544-symbol distance alphabet and 64-bit distance arithmetic, with workbuf_len growing to (1 << 30) + 542 when the quirk is set. - decoder.add_dictionary! mirrors BrotliDecoderAttachDictionary for raw (LZ77 prefix) and serialized shared dictionaries: compound-dictionary distance mapping, the shared-dictionary parser (custom word lists, transform lists including the SHIFT transforms, context-dependent dictionary selection, prefix dictionaries) and all of its validation. Wuffs structs cannot hold slices, so the dictionary bytes are copied into the front of the caller's workbuf; the layout is documented in the README. Serialized dictionaries are accepted as in a BROTLI_EXPERIMENTAL build of libbrotli. - fuzz/c/std/brotli_fuzzer.c, exercising the quirk, dictionaries and random I/O limits. Tests: new golden vectors for large windows (including a 17 MiB far-distance stream generated in the test), raw and serialized dictionaries, nine more malformed inputs, and mimic comparisons that attach the same dictionaries to libbrotlidec. Differential runs over 2.7 million mutated large-window and dictionary streams found no divergence from libbrotlidec. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011NoosQNe7DKWnqDCBeqw1p
This commit is contained in:
43 files changed
+4253
-460
No files matched your search
@@ -27,3 +27,57 @@ the other (valid) files and keeping, for each libbrotlidec v1.2.1 error code,
|
||||
the smallest mutated file that produced that error code. The file name
|
||||
mentions that error code, e.g. `invalid.cl-space.br` produces
|
||||
`BROTLI_DECODER_ERROR_FORMAT_CL_SPACE`.
|
||||
|
||||
|
||||
## Large Window and Dictionary Test Data
|
||||
|
||||
These files exercise the `QUIRK_LARGE_WINDOW` quirk and `add_dictionary`. They
|
||||
were generated with brotli v1.2.1: the command line tool where it suffices
|
||||
and otherwise a small C program calling `BrotliEncoderSetParameter` (with
|
||||
`BROTLI_PARAM_LARGE_WINDOW`), `BrotliEncoderPrepareDictionary` and
|
||||
`BrotliEncoderAttachPreparedDictionary`, linked against a libbrotlienc built
|
||||
with `BROTLI_EXPERIMENTAL` (needed for serialized shared dictionaries). Every
|
||||
file's decompressed form was checked with libbrotlidec (also built with
|
||||
`BROTLI_EXPERIMENTAL`).
|
||||
|
||||
- `*.qNlwM.br` are large-window streams (quality N, window bits M), e.g.
|
||||
`brotli -q 11 --large_window=26 -o alice29.txt.q11lw26.br alice29.txt`.
|
||||
- `far17m.q9lw25.br` is a large-window (25 window bits) quality 9 stream whose
|
||||
decompressed form (17 MiB plus 128 KiB, generated by the test program
|
||||
instead of being checked in) is 64 KiB of xorshift32 output, 17 MiB of
|
||||
zeroes and the same 64 KiB again. The second copy refers back more than (1
|
||||
<< 24) bytes. A standard (24 window bits) encoding is twice as large.
|
||||
- `*.raw-romeo.br` use `../romeo.txt` as a raw (LZ77 prefix) dictionary, e.g.
|
||||
`brotli -q 11 -w 22 -D ../romeo.txt -o romeo.txt.q11w22.raw-romeo.br
|
||||
../romeo.txt`.
|
||||
- `*.raw-chunks.br` use several raw dictionaries (compound dictionary chunks):
|
||||
`alice29.txt`'s bytes `[0 .. 20000)`, `[20000 .. 20001)`, `[20001 .. 20001)`
|
||||
(empty) and `[20001 .. 60000)`, or `../romeo.txt`'s bytes `[0 .. 300)`, `[300
|
||||
.. 500)` and `[500 .. 942)`.
|
||||
- `shared-dictionary.*.bin` are serialized shared dictionaries (the format
|
||||
parsed by `BrotliSharedDictionaryAttach` in brotli's
|
||||
`c/common/shared_dictionary.c`), built by a script:
|
||||
- `full` has a 500 byte LZ77 prefix (the start of `../romeo.txt`), two
|
||||
custom word lists (common words from `alice29.txt` and `asyoulik.txt`),
|
||||
two custom transform lists (including `BROTLI_TRANSFORM_SHIFT_FIRST` and
|
||||
`BROTLI_TRANSFORM_SHIFT_ALL` transforms), three (word list, transform
|
||||
list) combinations (the third is the built-in pair) and a context map.
|
||||
- `words` has one custom word list, the built-in transforms and no prefix.
|
||||
- `prefix` has only an LZ77 prefix (the start of `asyoulik.txt`).
|
||||
- `utf8` has a word list of UTF-8 (and invalid UTF-8) words and shift
|
||||
transforms with various parameters.
|
||||
- `shared-dictionary.text` is made of `shared-dictionary.full.bin` words,
|
||||
transformed by its transforms. `*.sd-full.br` (and `*.sd-words.br`) were
|
||||
compressed with `shared-dictionary.full.bin` (and `.words.bin`) attached.
|
||||
`asyoulik.txt.q11w18.sd-prefix-raw.br` was compressed with
|
||||
`shared-dictionary.prefix.bin` and then `../romeo.txt`'s first 300 bytes
|
||||
(raw) attached.
|
||||
- `romeo-mutated.sd-utf8.br` is a mutated `romeo.txt.q5w10.br` that decodes
|
||||
(to `romeo-mutated.sd-utf8.txt`) without error when
|
||||
`shared-dictionary.utf8.bin` is attached, and whose dictionary references
|
||||
apply shift transforms to 1, 2, 3 and 4 byte UTF-8 sequences.
|
||||
- `invalid.lw-*.br`, `invalid.raw-*.br` and `invalid.sd-*.br` are malformed.
|
||||
They were found by mutating the files above, like the other `invalid.*.br`
|
||||
files, with the large-window quirk set (`lw`), or the raw (`raw`, the
|
||||
`romeo.txt` chunks) or serialized (`sd`, `shared-dictionary.full.bin`)
|
||||
dictionaries attached.
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1 @@
|
||||
âvŒ¸)MëÍ{> ài©Ã`n␍gήy¼€ÈfÏò0¬aQIÿËS)”߇„
|
||||
@@ -0,0 +1 @@
|
||||
âvŒ¸)MëÍ{> ài©Ã`n␍gήy¼€ÈfÏò‰Aióûì×baQIÿË])”Þ‡„rÁ|<E
|
||||
@@ -0,0 +1 @@
|
||||
âvŒ¸)MëÍ{> ài©Ã`n␍gήy¼€ÈfÏò0�¯>n€‰Áióû
|
||||
Binary file not shown.
@@ -0,0 +1,45 @@
|
||||
Romeo and Juliet
|
||||
Excerpt Ÿ˜€ Act 2, Scene 2
|
||||
|
||||
JULIET
|
||||
O €xy,ULIET
|
||||
! linksfore art thouULIET
|
||||
?
|
||||
Deny thy signedo andtake p thy name;
|
||||
Or, Contaou wilt not, be but sworn my `₫é,
|
||||
And I'll the worlde bua Capulet.
|
||||
|
||||
ROMEO
|
||||
[Aside] Sif(! I usin more, or sif(! I Perso athouis?JULIET
|
||||
O 'Tisut swy name;
|
||||
Italiany `âenemy;
|
||||
Tu wit thouyhold,aou wghot, a Montague.
|
||||
What's Montague? remaininr hand,ninr foot,
|
||||
Nnr arm,ninr face,ninr They were part
|
||||
Be woring to a man. Obe busomey were me;
|
||||
!
|
||||
What's in a me;
|
||||
? Italilinks we cf(! a four
|
||||
By They were me;
|
||||
situatsme(! as onkey;
|
||||
SoIET
|
||||
?
|
||||
situa, were heot, ET
|
||||
?
|
||||
cf(!'d,
|
||||
Retaoutbrat dsin perf
|
||||
|
||||
Whililinks he owes
|
||||
Wiou wttbrat نك.€xy,ULd
|
||||
|
||||
Many name;©,
|
||||
Andf; widthe me;
|
||||
linksmainie par =falsee
|
||||
Takeo ll muyholt.
|
||||
|
||||
ROMEOI takeolseeo athoy das:
|
||||
Co ll e but `â‚«Ão andd I'lbe new missized;
|
||||
Hencsforth I sound investor€xy.s?JULIET
|
||||
!
|
||||
Wh a m art thouU? Itahide be🗿èé'd in €¿øÿ;
|
||||
Sostumblest o my `counsel?
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -0,0 +1,41 @@
|
||||
causeHAVE
|
||||
uncoTURTLE, uhoughtfullycharles -audrey-fatherdaughter Somethingsheph HASTILY, phebeSHOULDSHEPHERDInterruptedlike fortune things. hoingextraordinary. erhaps said aterpillar Wonderrabbi well everything's Everything charlesthere corin anything's -silvius-WILLIAMDisappointment better RepliedDAUGHTER will thoughtfully QuestionsilviusAnythingturtle
|
||||
looking mucquite
|
||||
charlesthe disappointment INTERRUPTED, that FORTUNEuncomfortable, daughter Childrenph the uncomfortable right
|
||||
-love-bettermistresstraordinaryreplied's -brother-derstand there t yourorlando-should-senior bnmudqr`shnm ph this -should-
|
||||
haveSENIOR the children extraordinary. wouldcon Dormousenversation-audrey- h silvius knnjdc sen the executioner disappointmentcorin oli y okingexecutioner children g`ssdq phebe -daughter-w adventures's -jaques-frightened's opportunity's the interrupted w mistress little. thoughtfully
|
||||
anything -should- THATContemptuouslyeconthings. contemptuouslrself t corin CORIN-mistress-will noonqstmhsx beginning Extraordinaryextraordinary conversatioOpportunity -father-getting. enter conversatioPRITHEEcontesheph mockanother. HAVE ␍inute l thoughtfully's withperhaps-orlando-seemed
|
||||
Wonderforestthe frightened which willremember. have co THAT, daughter -william-Disappointmenttuchdaughter -audrey-LOOKED, the turtle orlandofather sh Interrupted beginning could's rosalindhave xtraordinary Looking jaquesnversationuncomfortablable disappointment phebe -orlando-dormouse mistresshoughtfully THERE nterrupted op thoughtfullextraordinary, -there-contemptuouslySHALLshall jaq creatures -senior-phebe t CREATURES, sheph ed Thoughtfullythe nothing LookingINTO, everything con Alice-prithee-charlesursewhichlooked rather uWhile White DAUGHTERDisappointment love your Opportunitydifferent. rosalind -would-EVERYTHING, -father- father -oliver-ANOTHER, en enough celia-forest- the thought ROSALINDseniorFORTUNEFATHERdhildren uncomfortable's ce -that-love mistr ONLY, should phebe-brother-l eyeifferent orlandoInterrupted-senior-pheberueen ExtraordinaryBEFORE, shepherdCHARLESbetter should brotherfortune t -orlando-audrey mistr daugh will sen disappointmentextraordinarythe things withieardokd`rd HAVE disappointment, sh ph noonqstmhsx very-oliver-ce daughterbdqs`hmkx w BROTHERTHISl ORLANDO the other oli will nversationconversatiowhich ously mistr ce rosalind because's thoughtfull disappointment -this-AUDREYdisappointment mistr the understand frightened
|
||||
y gdqrdke Creatures onversation AUDREY conversation went's extraordinary's WITHextrautifulanotheren interrupted -rosalind-orlando-orlando-Extraordinaryeirrosalind suddenly, -charles-charlesvoice
|
||||
bnythingonversation t CHARLES-another--silvius-contemptuously
|
||||
UNCOMFORTABLE, caterpillashepherd uncomfortable -father-father would xtraordinary SHOULDvgdm JAQUES-there--shepherd-creatures. Little ROSALIND-will- DAUGHTER-another-daugh rosal without's nary curious. ckaud would MISTRESScharles thou shallCertainly-orlando-extraordinary
|
||||
seniordaughter thoughtfullshepherdWITHonly. -senior-Continued this EXTRAORDINARY, fortune
|
||||
THAT-audrey-ufat more -charles-things. BROTHERFRIGHTENED, sghmf THERElovethe certainly aud this -will- brotherthe getting
|
||||
with's shepherdoli THIS -another-the seemed xiouslyoliver-shepherd-caterpillar
|
||||
which wh daugh mistress Opportunity getting -rosalind-dmntfg ANOTHERthoughtfully SHOULDprithee oliverbet eisappointment thou certainly With methingsho replied.
|
||||
this shepherdcharlesrosalindbetter en sheph -which--orlando--celia-rightened ke fat the beautiful will HAVE-have--love-WOULD thisfor MUCH, well's thouprit contemptuously -celia-vncomfortablelly-oliver- finishelovethe conversation thoughtfullyOLIVER-enter-ar LOOKED, onversation -love-phebe Conversationuncomfortable, extraordinary
|
||||
mistressbefore. onder adventures have-thou-SILVIUS-silvius-conversation's t more's daughtersho -shepherd-white, tmcdqrs`mc rosal thingtalking, alice. ormouse Thinking remember Round Perhapsthere for MISTRESS beginning. your-this-anxiouslmistress UNCOMFORTABLE, Uncomfortable -phebe-wh etting CONTEMPTUOUSLY, silvius isappointment thoadventures's extraordinar caterpillar mistressthe interrupted
|
||||
rosal seniorreat SILVIUSnough daughter able contecorin mistress-charles-going
|
||||
Contemptuously brot nderstand HAVEAUDREYyour celia -should- willCONTEMPTUOUSLY, -audrey-sho thiseginning char willtmentmarch's sheph fverythingconversation. the mouse -audrey-there silv othing willonly. the understand LOVEsheph daugh BeforekingecutionerConversation AUDREYPRITHEE-which-the should -jaques- should. withginishedbet tmbnlenqs`akd like. beginning -should-rosalind NOTHING, bnmsdlostntrkx conversation
|
||||
large. something-rosalind--mistress--another--there--william-Contemptuously That executioner's thoughtfully, thinking However fat uhen shouldperhapDAUGHTERopSHOULD disappointment the extraordinary thereqerhapscorinThoughtfullywould contemptuously orlando terpillarmistress about -rosalind--fortune-abbit thereanot interrupted-your-this such -daughter-the queen ph Conversationsgntfgsetkkx orlando silvius -father- Anything extraordinary Before Begant there uhemthe could THATperhaps ruestionsgntfgsetkkx aud rosal getting
|
||||
mistr mistr extraordinary
|
||||
ierecharles thinkingYOURsilv t interrupted
|
||||
which audrey daugh thothis Disappointment extraordinary, your voice edEXTRAORDINARY, comfortable conversation, -prithee-h HEARD, which everythin xtraordinary understand children white. -father-WhatThoughtfully co silviusmust
|
||||
ightenedrosal would somethin -father--shall--would-extraordinary's SHEPHERDextrjaq there InterruptedInterrupted THATsen SENIORyour conversation
|
||||
celia -celia--better--love- SHOULDphebe OLIVERmistr remember Uncomfortable -jaques-rosal aud conversation THOUGHTFULLY, the before WOULD-forest-ere -mistress-williambrother -will--that-THAT-prithee-for fatherWILL rosalindxecutioner understand's much, brother come -mistress-dwdbtshnmdq SILVIUSaudrey nomentonversation -brother- ROSALINDyourppportunitywithshepherden corin jaqueswiththis EVERYTHING, the were -fortune- vncomfortablemistress celiarosalindthosghmfr remember WILLIAMaud Disappointmentcorin fxecutionerntemptuously daughterell Extraordinaryad`tshetk thouopportunity venturesthe come -shepherd-uchess caterpillar. mistressSHOULD -rosalind-celia ROSALINDntemptuouslyEXTRAORDINARY, know fortune the again THEREthere disappointmenthoughtfullygryphon. SHOULDmistr CORIN-shall-thinkingEVERYTHING, OLIVERSomething DAUGHTERyourOLIVERshepherd thoughtfully. -orlando-FORESTBROTHERrselffortune -which-y lovephebe-celia-dontemptuouslyund Executionerlike Looking-prithee-WOULDlove-forest- -thou--father-ConversationrosalindCreatures fortune uncomfortable phebeinterrupted WILLIAMSHEPHERDDisappointment wh xecutioner should rosalind jaquesDAUGHTERremembe SILVIUSfverythingaterpillar HAVEthe remember Executionerw theirwent charlesbetterhead DAUGHTERcontemptuousl FORTUNEbrot CATERPILLAR, eard gryphon
|
||||
WOULDshepherd enterloveuncomfortable, Understand-oliver-ENTERthe quite -rosalind-fortunedaugh bettercontemptuouslsomCELIA the voice betterthey, -should-audreywould co understandTHOUnversation
|
||||
lice -enter- Caterpillar THINKING, shepherd Disappointmentoliver-mistress-senior co WILLIAMBROTHERrosal jaquesORLANDOuncomfortable
|
||||
executioney your, FORTUNE-there-mistr mistr sheph with
|
||||
en mistress the without Contemptuouslyphebe -daughter-eautiful nterrupted orlandoopportunity's contemptuouslybeginning
|
||||
brot Going anot usecurious, w shepherd dormouse
|
||||
uncomfortable Beautifulgryphon's MISTRESSdaughter Everything erythingEXTRAORDINARY, Thoughtfully -charles-extraordinary, ion while
|
||||
forest daugh SILVIUSyour Caterpillar en suddenly vnmcdq PHEBE-shepherd-shoulddventures fortune understand
|
||||
EXTRAORDINARY, Adventuresdontemptuouslyable hastily, thoughtfully. Frightened which h mistr mistress -shall-enterthe understand -better-uestion Contemptuously ph willinterrupted Gettingknow, shall into. hmsn Mock Continued seemed have extraordinary's march, celiaWHICHb`sdqohkk`q SILVIUSunderstanDAUGHTER the caterpillar dudqxsghmf better -another-THERE brother understand. -there-phebe ableYOUR-celia- t FORTUNEAUDREY-shepherd-enter extraordinar mistress WHICHchildren's disappointment's BROTHERddenlyGetting tmbnlenqs`akd DAUGHTERTHERE-silvius-vncomfortablerosal remember, BETTERWITHdaugh Creaturesdwithout, contemptuously's PHEBEdaughterWILLthincelia opportunit ightenedextraordinary -oliver-Extraordinarycorinicethoughtfully
|
||||
Thoughtfullythis-that- LookedConversation vhsg ␍uch large littlorla -corin-mistress there-this- conversation. extraordinary's thoughtfullExtraordinary must, portunityDAUGHTER conversation's william Everything-daughter-frightened's sho Finished another, mistr CONTEMPTUOUSLY, -love-the creatures sheph the suddenly orla seemed. vncomfortablefor ith w senior rabbitmistress ANOTHERtho-daughter--shepherd-remember's fxtraordinary-shall-opportunity-audrey-our Thoughtorlando -shall--shall- BeginningInterrupted FINISHED, tmentorlando -charles-uhemwhat's -orlando-contemptuously's tuddenlyCOURSE, CHILDREN, phebe -celia- something betterunderstandCHARLES thoughtfully, different's white -there-Frightened y silvius orla jnterruptedFORTUNEph -thou-hoing WILLIAMturtle's rabbithe beautiful Thoughtfully disappointment. Beginning VERY, seemed said ph char HOWEVER, rosalind BeautifulCURIOUS, RABBIT, extraordinary
|
||||
Anything SILVIUSextraordinar thoughtfully. -better-your shall dontemptuouslyfort SHEPHERD daterpillarsh daughter oli disappointmententer Children -jaques-haves`kjhmf orlando SENIORwhich daughter down Uncomfortable rosalindrosal co mistr might. -should-phebe wo loveq`aahs opportunity. Uncomfortable oprosalindooking -which-thou the thinking marchbefore, -with--there-bet t oliverfortune-celia-jaquesprithee th -your- -fortune--senior-disappointment lly-with-CONTEMPTUOUSLY, come. charles WHEN, sh prit onversation the again -shepherd-jaques silvius RoundBROTHERbrot enter corin orlandosabbitWOULD-this-getting. CONVERSATION, which Questioncorin thatORLANDOthe such hings thoughtfully en aud Themmistr rious Contemptuously daughterround, the caterpillar Disappointment art shall INTERRUPTED, goingSHEPHERDsuddenly. -better- THISWILL-orlando-THATAnxiously contemptuously. FORTUNEagain
|
||||
-have-HAVETHOUGHTFULLY, nother Uncomfortablesho -forest-corin DAUGHTERHavefrightened everything
|
||||
would onversation sh -your--with- shepherd ChildrenorlandoCHARLESbrother Went something -which-rosalind-silvius-anotherdaughter CELIAHerself extraordinary
|
||||
Enough Extraordinary-prithee-wo corin HAVE-prithee-daughterhoweverce adventures w must phebe extraordinary's SHOULD-brother-traordinaryrosalind will ouldrosalind the something -that--prithee- king
|
||||
father the uncomfortable FOREST First brotherquestion
|
||||
WENT, THEREmistr -prithee- Uncomfortablefort with question outwould WOULDsheph BETTERright's narybrother william tting-will-WHICHSILVIUS-another-onder bnmudqr`shnm JAQUES-better-they's THISphebeWOULD -mistress-Anxiously thouContemptuously Withittle sheph Caterpillardifferent.
|
||||
THISThoughtfullybrother -charles- sen betterTHISwould -senior-that that bnmudqr`shnm CURIOUS,
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Reference in new issue
Block a user