Describe the Zopfli path's dictionary lookups, the extra quality 11 BackwardMatch storage in workbuf_len and drop the TODO items.
2.5 KiB
Compression Encoders
Compression encoders read from one input stream (an io_reader called src)
and write to an output stream (an io_writer called dst). Like
compression decoders, they implement the
base.io_transformer interface, whose key method is transform_io. It
incrementally compresses the source data.
This method is a coroutine, and does not require
either all of the input or all of the output to fit in a single contiguous
buffer. It suspends with the $short_read status when it has consumed all of
src (and src is not closed) and with $short_write when dst is full. It
returns ok once src is closed and all of the compressed data has been
written. Encoder parameters (e.g. the compression quality) are set with
set_quirk before the first transform_io call.
API Listing
In Wuffs syntax, the base.io_transformer methods are:
dst_history_retain_length() u64get_quirk(key: u32) u64set_quirk!(key: u32, value: u64) statustransform_io?(dst: io_writer, src: io_reader, workbuf: slice u8)workbuf_len() range_ii_u64
A workbuf of workbuf_len().max_incl bytes always suffices. Some encoders
(e.g. brotli at qualities 10 and 11) have a smaller min_incl, which is
enough for realistic inputs: with a workbuf shorter than max_incl, an
unusual input can make transform_io return "#bad workbuf length".
Encoders may have further, format-specific methods. For example, the brotli
encoder has request_flush, request_emit_metadata, max_compressed_size
and, for raw (LZ77 prefix) dictionaries, add_dictionary and
add_dictionary_workbuf_len. Like quirks, dictionaries are added before the
first transform_io call.
Implementations
- std/brotli (qualities 0 to 11,
including large-window brotli, with windows of up to 1 GiB, at qualities 3
to 11). Its
workbuf_lenis the worst case for the parameters (more than 2 GiB for large-window brotli at lgwin 30; at qualities 10 and 11,min_inclis a smaller practical bound), but a shorter workbuf works for short inputs: see the README's "Encoder Workbuf Layout" section. Up to 15 raw dictionaries can be attached (like libbrotlienc'sBrotliEncoderAttachPreparedDictionary; like the C code, qualities 0 and 1 ignore them), see the README's "Encoder Dictionaries" section. They live at the start of the workbuf, with their prepared hash tables, andworkbuf_lengrows accordingly.