mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-22 08:47:28 +01:00
Implement raczstd dictionaries
This commit is contained in:
1 parent
3b466277dc
commit
2d8648ae35
3 files changed
+276
-41
No files matched your search
+24
-29
@@ -486,6 +486,27 @@ if they are `Branch Node`s.
|
||||
# Specific Codecs
|
||||
|
||||
|
||||
## Common Dictionary Format
|
||||
|
||||
Unless otherwise noted, codecs use this common dictionary format.
|
||||
|
||||
If a `Leaf Node`'s `Secondary CRange` is empty then there is no dictionary.
|
||||
Otherwise, the `Secondary CRange` must be at least 8 bytes long:
|
||||
|
||||
- 4 byte little-endian `uint32_t` `Dictionary Length`. The high 2 bits are
|
||||
reserved and must be zero. The maximum (inclusive) `Dictionary Length` is
|
||||
therefore `((1 << 30) - 1)`, or `1_073_741_823` bytes.
|
||||
- `Dictionary Length` bytes `Dictionary`.
|
||||
- 4 byte little-endian `uint32_t` `Dictionary Checksum`, the Checksum` is
|
||||
CRC-32 IEEE checksum over the `Dictionary`'s bytes (excluding both the 4
|
||||
byte prefix and the 4 byte suffix).
|
||||
- Padding (ignored).
|
||||
|
||||
The `Leaf TTag` must be `0xFF`. All other `Leaf TTag` values (below `0xC0`) are
|
||||
reserved. The empty `Tertiary CRange` is ignored. The `Leaf STag` value is also
|
||||
ignored, other than deriving the `Secondary CRange`.
|
||||
|
||||
|
||||
## RAC + Zeroes
|
||||
|
||||
The `CRange`s are ignored. The `DRange` is filled with NUL bytes (`memset` to
|
||||
@@ -525,35 +546,9 @@ TODO.
|
||||
## RAC + Zstandard
|
||||
|
||||
The `CFile` data in the `Leaf Node`'s `Primary CRange` is decompressed as
|
||||
Zstandard (RFC 8478), possibly referencing a LZ77 prefix dictionary.
|
||||
|
||||
If a `Leaf Node`'s `Secondary CRange` is empty then there is no dictionary.
|
||||
Otherwise, the `Secondary CRange` must be at least 12 bytes long:
|
||||
|
||||
- 4 byte little-endian `uint32_t` `Dictionary ID`.
|
||||
- 8 byte little-endian `uint64_t` `Dictionary Length`.
|
||||
- `Dictionary Length` bytes `Dictionary`.
|
||||
- Padding (ignored).
|
||||
|
||||
The `Dictionary ID` is per section 3.1.1.1.3 in the RFC, although in RAC it
|
||||
always occupies 4 bytes, even if Zstandard itself could represent it in fewer
|
||||
bytes. In the terminology of the RFC section 5, the `Dictionary` bytes are "raw
|
||||
content" dictionaries.
|
||||
|
||||
As per the RFC section 3.1.1.1.2, the Zstandard format allows for a window size
|
||||
up to 3.75 TiB, which is more than the maximum `uint32_t` value. The
|
||||
`Dictionary Length` field here is therefore 8 bytes, not 4. However, the RFC
|
||||
acknowledges that "a decoder is allowed to reject a compressed frame that
|
||||
requests a memory size beyond decoder's authorized range", and likewise, a
|
||||
RAC + Zstandard decoder is allowed to reject a `Dictionary Length` that it
|
||||
considers too large. The RFC goes on to say "it's recommended for decoders to
|
||||
support values of Window Size up to 8 MiB and for encoders not to generate
|
||||
frames requiring a Window Size larger than 8 MiB", and that recommendation
|
||||
similarly applies to RAC + Zstandard.
|
||||
|
||||
The `Leaf TTag` must be `0xFF`. All other `Leaf TTag` values (below `0xC0`) are
|
||||
reserved. The empty `Tertiary CRange` is ignored. The `Leaf STag` value is also
|
||||
ignored, other than deriving the `Secondary CRange`.
|
||||
Zstandard (RFC 8478), possibly referencing a dictionary wrapped in RAC's common
|
||||
dictionary format, described above. After unwrapping, the dictionary's bytes
|
||||
can be either a "raw" or "trained" dictionary, as per RFC 8478 section 5.
|
||||
|
||||
|
||||
# Examples
|
||||
|
||||
Reference in new issue
Block a user