From 9980a6fef4b2b595b64707b6df389afdf2ea059d Mon Sep 17 00:00:00 2001 From: Nigel Tao Date: Sat, 9 Mar 2019 13:23:46 +1100 Subject: [PATCH] Add a Random Access Compression (RAC) spec --- doc/spec/README.md | 1 + doc/spec/rac-spec.md | 498 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 499 insertions(+) create mode 100644 doc/spec/rac-spec.md diff --git a/doc/spec/README.md b/doc/spec/README.md index 591c7af5..518d6e9f 100644 --- a/doc/spec/README.md +++ b/doc/spec/README.md @@ -2,3 +2,4 @@ This directory contains various specifications, whether intrinsically related to Wuffs or stand-alone formats that were invented alongside Wuffs' development. - [Naïve Image Formats: NIE, NII, NIA](./nie-spec.md) + - [Random Access Compression: RAC](./rac-spec.md) diff --git a/doc/spec/rac-spec.md b/doc/spec/rac-spec.md new file mode 100644 index 00000000..deef9f70 --- /dev/null +++ b/doc/spec/rac-spec.md @@ -0,0 +1,498 @@ +# Random Access Compression: RAC + +Status: Draft (as of March 2019). There is no compatibility guarantee yet. + + +## Overview + +The goal of the RAC file format is to compress a source file (the "decompressed +file") such that, starting from the compressed file, it is possible to +reconstruct the half-open byte range `[di .. dj)` of the decompressed file +without always having to first decompress all of `[0 .. di)`. + +Conceptually, the decompressed file is partitioned into non-overlapping chunks. +Each compressed chunk can be decompressed independently (although possibly +sharing additional context, such as a LZ77 prefix dictionary). A RAC file also +contains a hierarchical index of those chunks. + +RAC is a container format, and while it supports common compression codecs like +Zlib, Brotli and Zstandard, it is not tied to any particular compression codec. + +Non-goals for version 1 include: + + - Filesystem metadata like file names, file sizes and modification times. + - Multiple source files. + +There is the capability (see reserved `TTag`s, below) but no promise to address +these in a future RAC version. There might not be a need to, as other designs +such as EROFS ([Extendable Read-Only File +System](https://lkml.org/lkml/2018/5/31/306)) already exist. + +Non-goals in general include: + + - Encryption. + - Streaming decompression. + + +### Related Work + +RAC differs from most other archive formats, whether established like +[Tar](https://en.wikipedia.org/wiki/Tar_%28computing%29) and +[Zip](https://en.wikipedia.org/wiki/Zip_%28file_format%29) or newer like +[śiva](https://blog.sourced.tech/post/siva/). Even if those formats let you +seek inside an archive, seek points are named by strings (i.e. file names) and +are positioned only at source file boundaries. You cannot seek to a relative +offset (in what RAC calls `DSpace`) within a source file. They also often don't +support shared compression dictionaries embedded within an archive. + +RAC differs from the [Riegeli](https://github.com/google/riegeli) +sequence-of-records format. A RAC file arguably contains what Riegeli calls +records (which are not files, as they don't have names) and both formats +support efficient seeking, but Riegeli seeks to a point in what RAC calls +`CSpace`, whereas RAC seeks to a point in `DSpace`. + +RAC differs from the [LevelDB +Table](https://github.com/google/leveldb/blob/master/doc/table_format.md) +format, even if the LevelDB Table string keys are re-purposed to encode +integer-valued `DSpace` seek points. A LevelDB Table, by itself, does not +support shared compression dictionaries. + +A LevelDB Table index is also flat, unlike RAC's hierarchical index. In both +cases, a maliciously written index could contain multiple entries for the same +key, and different decoders could therefore produce different output for the +same source file - a potential security vulnerability. For LevelDB Tables, +verifying an index key's uniqueness requires scanning every key in the file, +which can be relatively expensive, and is therefore not done in practice. In +comparison, RAC's "Branch Node Validation" process (see below) ensures that +exactly one `Leaf Node` contains any given byte offset `di` in the decompressed +file, and the RAC index's hierarchical nature places a scalable upper bound on +the scanning cost of verifying that, based on that portion of the index tree +visited for any given request `[di .. dj)`. + + +### Glossary + + - `CBias` is the delta added to a `CPointer` to produce a `COffset`. + - `DBias` is the delta added to a `DPointer` to produce a `DOffset`. + - `CFile` is the compressed file. + - `DFile` is the decompressed file. + - `CFileSize` is the size of the `CFile`. + - `DFileSize` is the size of the `DFile`. + - `COffset` is a byte offset in `CSpace`. + - `DOffset` is a byte offset in `DSpace`. + - `CPointer` is a relative `COffset`, prior to bias-correction. + - `DPointer` is a relative `DOffset`, prior to bias-correction. + - `CRange` is a `Range` in `CSpace`. + - `DRange` is a `Range` in `DSpace`. + - `CSpace` means that byte offsets refer to the `CFile`. + - `DSpace` means that byte offsets refer to the `DFile`. + +`Range` is a pair of byte offsets `[i .. j)`, in either `CSpace` or `DSpace`. +It is half-open, containing every byte offset `x` such that `(i <= x)` and `(x +< j)`. The size of a `Range` equals `(j - i)` if `(j > i)`, otherwise it equals +zero. Having `(j <= i)` is valid, and means an empty `Range`: one with zero +size. There are many valid `[i .. j)` pairs that mean an empty range. + +All bytes are 8 bits and unless explicitly specified otherwise, all fixed-size +unsigned integers (e.g. `uint32_t`, `uint64_t`) are encoded little-endian. +Within those unsigned integers, bit 0 is the least significant bit and e.g. bit +31 is the most significant bit of a `uint32_t`. + +The maximum supported `CFileSize` and the maximum supported `DFileSize` are the +same number: `0xFFFFFFFFFFFFFF`, which is `((1 << 56) - 1)`. + + +## File Structure + +A RAC file (the `CFile`) must be at least 4 bytes long, and start with the 3 +byte `Magic` (see below), so that no valid RAC file can also be e.g. a valid +JPEG file. The fourth byte is examined in the process described by the "Root +Node at the CFile Start" section, below. + +The `CFile` contains a tree of `Node`s. Each `Node` is either a `Branch Node` +(pointing to between 1 and 255 child `Node`s) or a `Leaf Node`. There must be +at least one `Branch Node`, called the `Root Node`. Parsing a `CFile` requires +knowing the `CFileSize` in order to identify the `Root Node`, which is either +at the start or the end of the `CFile`. + +Each `Node` has a `DRange`. An empty `DRange` means that the `Node` contains +metadata or other decompression context such as a shared dictionary. + +Each `Leaf Node` also has 3 `CRange`s (`Primary`, `Secondary` and `Tertiary`), +any or all of which may be empty. The contents of the `CFile`, within those +`CRange`s, are decompressed according to the `Codec` (see below) to reconstruct +that part of the `DFile` within the `Leaf Node`'s `DRange`. + + +## Branch Nodes + +A `Branch Node`'s encoding in the `CFile` has a variable byte size, between 32 +and 4096 inclusive, depending on its number of children. Specifically, it +occupies `((Arity * 16) + 16)` bytes, grouped into 8 byte segments (but not +necessarily 8 byte aligned), starting at a `COffset` called its `Branch +COffset`: + + +-+-+-+-+-+-+-+-+ + |0|1|2|3|4|5|6|7| + +-+-+-+-+-+-+-+-+ + |Magic|A|Che|V|T| Magic, Arity, Checksum, Version, TTag[0] + | DPtr[1] |T| DPtr[1], TTag[1] + | DPtr[2] |T| DPtr[2], TTag[2] + | ... |.| ..., ... + | DPtr[A-2] |T| DPtr[Arity-2], TTag[Arity-2] + | DPtr[A-1] |T| DPtr[Arity-1], TTag[Arity-1] + | DPtr[A] |C| DPtr[Arity] a.k.a. DPtrMax, Codec + | CPtr[0] |S| CPtr[0], STag[0] + | CPtr[1] |S| CPtr[1], STag[1] + | CPtr[2] |S| CPtr[2], STag[2] + | ... |.| ..., ... + | CPtr[A-2] |S| CPtr[Arity-2], STag[Arity-2] + | CPtr[A-1] |S| CPtr[Arity-1], STag[Arity-1] + | CPtr[A] |A| CPtr[Arity] a.k.a. CPtrMax, Arity + +-+-+-+-+-+-+-+-+ + +For the `(XPtr | Other)` 8 byte fields, the `XPtr` occupies the low 56 bits (as +a little-endian `uint64_t`) and the `Other` occupies the high 8 bits. + +The `CPtr` and `DPtr` values are what is explicitly written in the `CFile`'s +bytes. These are added to a `Branch Node`'s implicit `Branch CBias` and `Branch +DBias` values to give the implicit `COff` and `DOff` values: `COff[i]` and +`DOff[i]` are defined to be `(Branch_CBias + CPtr[i])` and `(Branch_DBias + +DPtr[i])`. + +`CPtrMax` is another name for `CPtr[Arity]`, and `COffMax` is defined to be +`(Branch_CBias + CPtrMax)`. Likewise for `DPtrMax` and `DOffMax`. + +The `DPtr[0]` value is implicit, and always equals zero, so that `DOff[0]` +always equals the `Branch DBias`. + + - For the `Root Node`, the `DPtrMax` also sets the `DFileSize`. The `Branch + CBias` and `Branch DBias` are both zero. The `Branch COffset` is determined + by the "Root Node" section below. + - For a child `Branch Node`, the `Branch COffset`, `Branch CBias` and `Branch + DBias` are given by the parent `Branch Node`. See the "Search Within a + Branch Node" section below. + + +### Magic + +`Magic` is the three bytes `"\x72\xC3\x63"`, which is invalid UTF-8 but is +`"rÃc"` in ISO 8859-1. The tilde isn't particularly meaningful, other than +`"rÃc"` being a nonsensical word (with nonsensical capitalization) that is +unlikely to appear in other files. + +Every `Branch Node` must start with these `Magic` bytes, not just a `Branch +Node` positioned at the start of the `CFile`. + + +### Arity + +`Arity` is the `Branch Node`'s number of children. Zero is invalid. + +The `Arity` byte is given twice: the fourth byte and the final byte of the +`Branch Node`. The two values must match. + +The repetition lets a RAC reader determine the size of the `Branch Node` data +(as the size depends on the `Arity`), given either its start or its end offset +in `CSpace`. For almost all `Branch Node`s, we will know its start offset (its +`Branch COffset`), but for a `Root Node` at the end of a `CFile`, we will only +know its end offset. + + +### Checksum + +`Checksum` is a checksum of the `Branch Node`'s bytes. It is not a checksum of +the `CFile` or `DFile` contents pointed to by a `Branch Node`. Content +checksums are a `Codec`-specific consideration. + +The little-endian `uint16_t` `Checksum` value is the low 16 bits XOR'ed with +the high 16 bits of the `uint32_t` CRC-32 IEEE checksum of the `((Arity * 16) + +10)` bytes immediately after the `Checksum`. The 4 bytes immediately before the +`Checksum` are not considered: the `Magic` bytes have only one valid value and +the `Arity` byte near the start is replicated by the `Arity` byte at the end. + + +### Version + +`Version` must have the value `0x01`, indicating version 1 of the RAC format. + + +### COffs and DOffs, STags and TTags + +For every `a` in the half-open range `[0 .. Arity)`, the `a`'th child `Node` +has two tags, `STag[a]` and `TTag[a]`, and a `DRange` of `[DOff[a] .. +DOff[a+1])`. The `DOff` values must be non-decreasing: see the "Branch Node +Validation" section below. + +A `TTag[a]` of `0xFE` means that child is a `Branch Node`. A `TTag[a]` in the +half-open range `[0xC0 .. 0xFE)` is reserved. Otherwise, the child is a `Leaf +Node`. + +A child `Branch Node`'s `SubBranch COffset` is defined to be `COff[a]`. Its +`COff[a+1]` value is ignored. Its `SubBranch DBias` and `SubBranch DOffMax` are +defined to be `DOff[a]` and `DOff[a+1]. + + - When `(STag[a] < Arity)`, it is a `CBiasing Branch Node`. The `SubBranch + CBias` is defined to be `(Branch_CBias + CPtr[STag[a]])`. + - When `(STag[a] >= Arity)`, it is a `CNeutral Branch Node`. The `SubBranch + CBias` is defined to be `(Branch_CBias)`. + +A child `Leaf Node`'s `STag[a]` and `TTag[a]` values are also called its `Leaf +STag` and `Leaf TTag`. It also has: + + - A `Primary CRange`, equal to `[COff[a] .. COff[a+1])`. Recall that it is + valid for a `Range`'s upper bound to be less than its lower bound, meaning + an empty `Range`. + - A `Secondary CRange`, equal to `[COff[STag[a]] .. COff[STag[a]+1])` when + `(STag[a] < Arity)`. If `(STag[a] >= Arity)` then the `Secondary CRange` is + the canonical empty range `[0 .. 0)`. + - A `Tertiary CRange`, similar to the `Secondary CRange`, but using `TTag[a]` + instead of `STag[a]`. + +Note that, since `Arity` is at most 255, an `STag[a]` of `0xFF` always results +in a `CNeutral Branch Node` or an empty `Secondary CRange`. Likewise, a +`TTag[a]` of `0xFF` always results in an empty `Tertiary CRange`. + + +### COffMax + +`COffMax` is an inclusive upper bound on every `COff` in a `Branch Node` and in +its descendent `Branch Node`s. A child `Branch Node` must not have a larger +`COffMax` than the parent `Branch Node`'s `COffMax`, and the `Root Node`'s +`COffMax` must equal the `CFileSize`. See the "Branch Node Validation" section +below. + +A RAC file can therefore be incrementally modified, if the RAC writer only +appends new `CFile` bytes and does not re-write existing `CFile` bytes, so that +the `CFileSize` increases. Even if the old (smaller) RAC file's `Root Node` was +at the `CFile` start, the new (larger) `CFileSize` means that those starting +bytes are an obsolete `Root Node` (but still a valid `Branch Node`). The new +`Root Node` is therefore located at the end of the new RAC file. + +Concatenating RAC files (concatenating in `DSpace`) involves concatenating the +RAC files in `CSpace` and then appending a new `Root Node` with `CBiasing +Branch Node`s pointing to each source RAC file's `Root Node`. + + +### Codec + +`Codec`s define specializations of RAC, such as "RAC + Zlib" or "RAC + Brotli". +It is valid for a "RAC + Zstandard" only decoder to reject a "RAC + Brotli" +file, even if it is a valid RAC file. Recall that RAC is just a container, and +not tied to any particular compression codec. For the `Codec` byte in a `Branch +Node`: + + - `0x00` is reserved. + - `0x01` means "RAC + Zlib". + - `0x02` means "RAC + Brotli". + - `0x04` means "RAC + ZStandard". + - Any other value less than 0x08 means that all of this `Branch Node`'s + children must be `Branch Node`s and not `Leaf Node`s and that no child's + `Codec` byte can have a bit set that is not set in this `Codec` byte. + - All other values, 0x08 or greater, are reserved. + + +### Branch Node Validation + +The first time that a RAC reader visits any particular `Branch Node`, it must +check that the `Magic` matches, the two `Arity` values match and are non-zero, +the computed checksum matches the listed `Checksum` and that the RAC reader +accepts the `Version` and the `Codec`. + +It must also check that all of its `DOff` values are sorted: `(DOff[a] <= +DOff[a+1])` for every `a` in the half-open range `[0 .. Arity)`. By induction, +this means that all of its `DOff` values do not exceed `DOffMax`, and again by +induction, therefore do not exceed `DFileSize`. + +It must also check that all of its `COff` values do not exceed `COffMax` (and +again by induction, therefore do not exceed `CFileSize`). Other than that, +`COff` values do not have to be sorted: successive `Node`s (in `DSpace`) can be +out of order (in `CSpace`), allowing for incrementally modified RAC files. + +For the `Root Node`, its `COffMax` must equal the `CFileSize`. Recall that +parsing a `CFile` requires knowing the `CFileSize`, and also that a `Root +Node`'s `Branch CBias` is zero, so its `COffMax` equals its `CPtrMax`. + +For a child `Branch Node`, its `Codec` bits must be a subset of its parent's +`Codec` bits, its `COffMax` must be less than or equal to its parent's +`COffMax`, and its `DOffMax` must equal its parent's `SubBranch DOffMax`. The +`DOffMax` condition is equivalent to checking that the parent and child agree +on the child's size in `DSpace`. The parent states that it is its `(DPtr[a+1] - +DPtr[a])` and the child states that it is its `DPtrMax`. + +One conservative way to check `Branch Node`s' validity on first visit is to +check them on every visit, as validating any particular `Branch Node` is +idempotent, but other ways are acceptable. + + +## Root Node + +The `Root Node` might be at the start of the `CFile`, as this might optimize +alignment of `Branch Node`s and of `CRange`s. All `Branch Node`s' sizes are +multiples of 16 bytes, and a maximal `Branch Node` is exactly 4096 bytes. + +The `Root Node` might be at the end of the `CFile`, as this allows one-pass +(streaming) encoding of a RAC file. It also allows appending to, concatenating +or incrementally modifying existing RAC files relatively cheaply. + +To find the `Root Node`, first look for a valid `Root Node` at the `CFile` +start. If and only if that fails, look at the `CFile` end. If that also fails, +it is not a valid RAC file. + + +### Root Node at the CFile Start + +The fourth byte of the `CFile` gives the `Arity`, assuming the `Root Node` is +at the `CFile` start. If it is zero, fail over to the `CFile` end. A RAC writer +that does not want to place the `Root Node` at the `CFile` start should +intentionally write a zero to the fourth byte. + +Otherwise, that `Arity` defines the size in bytes of any starting `Root Node`: +`((Arity * 16) + 16)`. If the `CFileSize` is less than this size, fail over to +the `CFile` end. + +If those starting bytes form a valid `Root Node` (as per the "Branch Node +Validation" section), including having a `CPtrMax` equal to the `CFileSize`, +then we have indeed found our `Root Node`: its `Branch COffset` is zero. +Otherwise, fail over to the `CFile` end. + + +### Root Node at the CFile End + +If there is no valid `Root Node` at the `CFile` start then the last byte of the +`CFile` gives the `Root Node`'s `Arity`. This does not necessarily need to +match the fourth byte of the `CFile`. + +As before, that `Arity` defines the size in bytes of any ending `Root Node`: +`((Arity * 16) + 16)`. If the `CFileSize` is less than this size, it is not a +valid RAC file. + +If those ending bytes form a valid `Root Node` (as per the "Branch Node +Validation" section), including having a `CPtrMax` equal to the `CFileSize`, +then we have indeed found our `Root Node`: its `Branch COffset` is the +`CFileSize` minus the size implied by the `Arity`. Otherwise, it is not a valid +RAC file. + + +## DRange Reconstruction + +To reconstruct the `DRange` `[di .. dj)`, if that `DRange` is empty then the +request is trivially satisfied. + +Otherwise, if `(dj > DFileSize)` then reject the request. + +Otherwise, start at the `Root Node` and continue to the next section to find +the `Leaf Node` containing the `DOffset` `di`. + + +### Search Within a Branch Node + +Load (and validate) the `Branch Node` given its `Branch COffset`, `Branch +CBias` and `Branch DBias`. + +Find the largest child index `a` such that `(a < Arity)` and `(DOff[a] <= di)` +and `(DOff[a+1] > di)`, then examine `TTag[a]` to see if the child is a `Leaf +Node`. If so, skip to the next section. + +For a `Branch Node` child, let `CRemaining` equal this `Branch Node`'s (the +parents') `COffMax` minus the `SubBranch COffset`. It invalid for `CRemaining` +to be less than 4, or to be less than the child's size implied by the child's +`Arity` byte at a `COffset` equal to `(SubBranch_COffset + 3)`. + +The `SubBranch COffset`, `SubBranch CBias` and `SubBranch DBias` from the +parent `Branch Node` become the `Branch COffset`, `Branch CBias` and `Branch +DBias` of the child `Branch Node`. In order to rule out infinite loops, at +least one of these two conditions must hold: + + - The child's `Branch COffset` is less than the parent's `Branch COffset`. + - The child's `DPtrMax` is less than the parent's `DPtrMax`. + +It is valid for one of those conditions to hold between a parent-child pair and +the other condition to hold between the next parent-child pair. + +Repeat this "Search Within a Branch Node" section with the child `Branch Node`. + + +### Decompressing a Leaf Node + +If a `Leaf Node`'s `DRange` is empty, decompression is a no-op and skip the +rest of this section. + +Otherwise, decompression combines the `Primary CRange`, `Secondary CRange` and +`Tertiary CRange` slices of the `CFile`, and the `Leaf STag` and `Leaf TTag` +values, in a `Codec`-specific way to produce decompressed data. + +There are two general principles, although specific `Codec`s can overrule: + + - The `Codec` may produce fewer bytes than the `DRange` size. In that case, + the remaining bytes (in `DSpace`) are set to NUL (`memset` to zero). + - The `Codec` may consume fewer bytes than each `CRange` size, and the + compressed data typically contains an explicit "end of data" marker. In + that case, the remaining bytes (in `CSpace`) are ignored. Padding allows + `COff` values to optionally be aligned. + +It is invalid to produce more bytes than the `DRange` size or to consume more +bytes than each `CRange` size. + + +### Continuing the Reconstruction + +If decompressing that `Leaf Node` did not fully reconstruct `[di .. dj)`, +advance through the `Node` tree in depth first search order, decompressing +every `Leaf Node` along the way, until we have gone up to or beyond `dj`. + +During that traversal, `Node`s with an empty `DRange` should be skipped, even +if they are `Branch Node`s. + + +# Specific Codecs + + +## RAC + Zlib + +The `CFile` data in the `Leaf Node`'s `Primary CRange` is decompressed as Zlib +(RFC 1950), possibly referencing a LZ77 prefix dictionary (what the RFC calls a +"preset dictionary"). + +If a `Leaf Node`'s `Secondary CRange` is empty then there is no dictionary. +Otherwise, the `Secondary CRange` must be at least 6 bytes long: + + - 2 byte little-endian `uint16_t` `Dictionary Length`. + - `Dictionary Length` bytes `Dictionary`. + - 4 byte big-endian `uint32_t` `Dictionary Checksum`. + - Padding (ignored). + +The `Dictionary Checksum` is Zlib's Adler32 checksum over the `Dictionary`'s +bytes (excluding the initial 2 bytes for the `Dictionary Length`). The checksum +is stored big-endian, like Zlib's other checksums, and its 4 byte value must +match the `DICTID` (in RFC 1950 terminology) given in the `Primary CRange`'s +Zlib-formatted data. + +The `Tertiary CRange` and `Leaf TTag` values are ignored. The `Leaf STag` value +is also ignored, other than deriving the `Secondary CRange`. + + +## RAC + Brotli + +TODO. + + +## RAC + Zstandard + +TODO. + + +# Examples + +TODO. + + +# Acknowledgements + +I (Nigel Tao) thank Robert Obryk, Sanjay Ghemawat and Sean Klein for their +review. + + +--- + +Updated on March 2019.