Implement raczstd dictionaries

This commit is contained in:
Nigel Tao committed 2019-10-23 20:10:07 +11:00
1 parent 3b466277dc
commit 2d8648ae35
3 files changed
+276 -41

No files matched your search

+24 -29
View File
@@ -486,6 +486,27 @@ if they are `Branch Node`s.
# Specific Codecs
## Common Dictionary Format
Unless otherwise noted, codecs use this common dictionary format.
If a `Leaf Node`'s `Secondary CRange` is empty then there is no dictionary.
Otherwise, the `Secondary CRange` must be at least 8 bytes long:
- 4 byte little-endian `uint32_t` `Dictionary Length`. The high 2 bits are
reserved and must be zero. The maximum (inclusive) `Dictionary Length` is
therefore `((1 << 30) - 1)`, or `1_073_741_823` bytes.
- `Dictionary Length` bytes `Dictionary`.
- 4 byte little-endian `uint32_t` `Dictionary Checksum`, the Checksum` is
CRC-32 IEEE checksum over the `Dictionary`'s bytes (excluding both the 4
byte prefix and the 4 byte suffix).
- Padding (ignored).
The `Leaf TTag` must be `0xFF`. All other `Leaf TTag` values (below `0xC0`) are
reserved. The empty `Tertiary CRange` is ignored. The `Leaf STag` value is also
ignored, other than deriving the `Secondary CRange`.
## RAC + Zeroes
The `CRange`s are ignored. The `DRange` is filled with NUL bytes (`memset` to
@@ -525,35 +546,9 @@ TODO.
## RAC + Zstandard
The `CFile` data in the `Leaf Node`'s `Primary CRange` is decompressed as
Zstandard (RFC 8478), possibly referencing a LZ77 prefix dictionary.
If a `Leaf Node`'s `Secondary CRange` is empty then there is no dictionary.
Otherwise, the `Secondary CRange` must be at least 12 bytes long:
- 4 byte little-endian `uint32_t` `Dictionary ID`.
- 8 byte little-endian `uint64_t` `Dictionary Length`.
- `Dictionary Length` bytes `Dictionary`.
- Padding (ignored).
The `Dictionary ID` is per section 3.1.1.1.3 in the RFC, although in RAC it
always occupies 4 bytes, even if Zstandard itself could represent it in fewer
bytes. In the terminology of the RFC section 5, the `Dictionary` bytes are "raw
content" dictionaries.
As per the RFC section 3.1.1.1.2, the Zstandard format allows for a window size
up to 3.75 TiB, which is more than the maximum `uint32_t` value. The
`Dictionary Length` field here is therefore 8 bytes, not 4. However, the RFC
acknowledges that "a decoder is allowed to reject a compressed frame that
requests a memory size beyond decoder's authorized range", and likewise, a
RAC + Zstandard decoder is allowed to reject a `Dictionary Length` that it
considers too large. The RFC goes on to say "it's recommended for decoders to
support values of Window Size up to 8 MiB and for encoders not to generate
frames requiring a Window Size larger than 8 MiB", and that recommendation
similarly applies to RAC + Zstandard.
The `Leaf TTag` must be `0xFF`. All other `Leaf TTag` values (below `0xC0`) are
reserved. The empty `Tertiary CRange` is ignored. The `Leaf STag` value is also
ignored, other than deriving the `Secondary CRange`.
Zstandard (RFC 8478), possibly referencing a dictionary wrapped in RAC's common
dictionary format, described above. After unwrapping, the dictionary's bytes
can be either a "raw" or "trained" dictionary, as per RFC 8478 section 5.
# Examples