diff --git a/doc/spec/rac-spec.md b/doc/spec/rac-spec.md index 3aff1b91..a14fef27 100644 --- a/doc/spec/rac-spec.md +++ b/doc/spec/rac-spec.md @@ -1,6 +1,6 @@ # Random Access Compression: RAC -Status: Draft (as of August 2019). There is no compatibility guarantee yet. +Status: Draft (as of September 2019). There is no compatibility guarantee yet. ## Overview @@ -32,10 +32,11 @@ Non-goals for version 1 include: - Filesystem metadata like file names, file sizes and modification times. - Multiple source files. -There is the capability (see reserved `TTag`s, below) but no promise to address -these in a future RAC version. There might not be a need to, as other designs -such as [SquashFS](http://squashfs.sourceforge.net/) and EROFS ([Extendable -Read-Only File System](https://lkml.org/lkml/2018/5/31/306)) already exist. +There is the capability (see attributes and reserved `TTag`s, below) but no +promise to address these in a future RAC version. There might not be a need to, +as other designs such as [SquashFS](http://squashfs.sourceforge.net/) and EROFS +([Extendable Read-Only File System](https://lkml.org/lkml/2018/5/31/306)) +already exist. Non-goals in general include: @@ -88,8 +89,10 @@ at least one `Branch Node`, called the `Root Node`. Parsing a `CFile` requires knowing the `CFileSize` in order to identify the `Root Node`, which is either at the start or the end of the `CFile`. -Each `Node` has a `DRange`. An empty `DRange` means that the `Node` contains -metadata or other decompression context such as a shared dictionary. +Each `Node` has a `DRange`, which can be empty. `Branch Node`s can also have +attributes: elements that aren't child `Node`s, which must have an empty +`DRange`. Empty elements contain metadata or other decompression context such +as a shared dictionary. Each `Leaf Node` also has 3 `CRange`s (`Primary`, `Secondary` and `Tertiary`), any or all of which may be empty. The contents of the `CFile`, within those @@ -114,7 +117,7 @@ COffset`: | ... |0|.| ..., Reserved (0), ... | DPtr[A-2] |0|T| DPtr[Arity-2], Reserved (0), TTag[Arity-2] | DPtr[A-1] |0|T| DPtr[Arity-1], Reserved (0), TTag[Arity-1] - | DPtr[A] |0|C| DPtr[Arity] a.k.a. DPtrMax, Reserved (0), Codec + | DPtr[A] |0|C| DPtr[Arity] a.k.a. DPtrMax, Reserved (0), CodecByte | CPtr[0] |L|S| CPtr[0], CLen[0], STag[0] | CPtr[1] |L|S| CPtr[1], CLen[1], STag[1] | CPtr[2] |L|S| CPtr[2], CLen[2], STag[2] @@ -164,7 +167,8 @@ Node` positioned at the start of the `CFile`. ### Arity -`Arity` is the `Branch Node`'s number of children. Zero is invalid. +`Arity` is the `Branch Node`'s number of elements: the number of child `Node`s +plus the number of attributes. Having zero children is invalid. The `Arity` byte is given twice: the fourth byte and the final byte of the `Branch Node`. The two values must match. @@ -196,14 +200,19 @@ The `Reserved (0)` bytes must have the value `0x00`. ### COffs and DOffs, STags and TTags -For every `a` in the half-open range `[0 .. Arity)`, the `a`'th child `Node` -has two tags, `STag[a]` and `TTag[a]`, and a `DRange` of `[DOff[a] .. -DOff[a+1])`. The `DOff` values must be non-decreasing: see the "Branch Node -Validation" section below. +For every `a` in the half-open range `[0 .. Arity)`, the `a`'th element has two +tags, `STag[a]` and `TTag[a]`, and a `DRange` of `[DOff[a] .. DOff[a+1])`. The +`DOff` values must be non-decreasing: see the "Branch Node Validation" section +below. -A `TTag[a]` of `0xFE` means that child is a `Branch Node`. A `TTag[a]` in the -half-open range `[0xC0 .. 0xFE)` is reserved. Otherwise, the child is a `Leaf -Node`. +A `TTag[a]` of `0xFE` means that that element is a `Branch Node` child. + +A `TTag[a]` of `0xFD` means that there is no child, but is instead a `Codec +Element` attribute, whose `DRange` must be empty, and the rest of this section +does not apply: the `STag` is ignored. + +A `TTag[a]` in the half-open range `[0xC0 .. 0xFD)` is reserved. Otherwise, the +element is a `Leaf Node` child. A child `Branch Node`'s `SubBranch COffset` is defined to be `COff[a]`. Its `SubBranch DBias` and `SubBranch DOffMax` are defined to be `DOff[a]` and @@ -272,47 +281,78 @@ values. `Codec`s define specializations of RAC, such as "RAC + Zlib" or "RAC + Brotli". It is valid for a "RAC + Zstandard" only decoder to reject a "RAC + Brotli" file, even if it is a valid RAC file. Recall that RAC is just a container, and -not tied to any particular compression codec. For the `Codec` byte in a `Branch -Node`: +not tied to any particular compression codec. + +There are two categories of `Codec`s: the high `0x80` bit of the `Codec Byte` +being `0` or `1` denotes a `Short` or `Long` codec respectively. `Short Codec`s +are represented by a single byte (the `Codec Byte`). `Long Codec`s use that +`Codec Byte` to locate 7 additional bytes: a `Codec Element`. + +For both `Short Codec`s and `Long Codec`s, the second-highest `0x40` bit of the +`Codec Byte` is the `Mix Bit`. A `Mix Bit` of `0` means that all of the +`Node`'s descendents have exactly the same `Codec` (not just the same `Codec +Byte`; `Short Codec`s and `Long Codec`s are not considered exactly the same +even if they represent the same compression algorithm). A `Mix Bit` of `1` +means that descendents may have a different `Codec`. + +For `Short Codec`s, the remaining low 6 bits correspond to a specific +compression algorithm: - `0x00` means "RAC + Zeroes". - `0x01` means "RAC + Zlib". - `0x02` means "RAC + Brotli". - `0x04` means "RAC + ZStandard". - - Any other value less than 0x08 means that all of this `Branch Node`'s - children must be `Branch Node`s and not `Leaf Node`s and that no child's - `Codec` byte can have a bit set that is not set in this `Codec` byte. - - All other values, 0x08 or greater, are reserved. + - All other values are reserved. + +For `Long Codec`s, the remaining low 6 bits of the `Codec Byte` define a number +`c64`. The lowest index `i` out of `(c64 + (64 * 0))`, `(c64 + (64 * 1))`, +`(c64 + (64 * 2))` and `(c64 + (64 * 3))` such that `TTag[i]` is `0xFD` locates +the 7 bytes that identifies the `Codec`. The location is what would otherwise +occupy the `i`th element's `CPtr | CLen` space. It is invalid for no such `i` +to exist. + +`Long Codec` values are not reserved by this specification, other than 7 NUL +bytes also means "RAC + Zeroes". Users may define their own values for +arbitrary compression algorithms. Maintaining a centralized registry mapping +values to algorithms is out of scope of this specification, although we suggest +a convention that the 7 bytes form a human-readable (ASCII) string, padded with +trailing NUL bytes. For example, a hypothetical "Middle Out version 2" +compression algorithm that typically used the ".mdo2" file extension might be +represented by the 7 bytes `"mdo2\x00\x00\x00"`. ### Branch Node Validation The first time that a RAC reader visits any particular `Branch Node`, it must check that the `Magic` matches, the two `Arity` values match and are non-zero, -the computed checksum matches the listed `Checksum` and that the RAC reader -accepts the `Version` and the `Codec`. +there is at least one child `Node` (not just non-`Node` attributes), the +computed checksum matches the listed `Checksum` and that the RAC reader accepts +the `Version`. For `Long Codec`s, there must exist an `0xFD` `TTag` as per the +previous section. It must also check that all of its `DOff` values are sorted: `(DOff[a] <= DOff[a+1])` for every `a` in the half-open range `[0 .. Arity)`. By induction, this means that all of its `DOff` values do not exceed `DOffMax`, and again by induction, therefore do not exceed `DFileSize`. -It must also check that all of its `COff` values do not exceed `COffMax` (and -again by induction, therefore do not exceed `CFileSize`). Other than that, -`COff` values do not have to be sorted: successive `Node`s (in `DSpace`) can be -out of order (in `CSpace`), allowing for incrementally modified RAC files. +It must also check that, other than `Codec Element` attributes, all of its +`COff` values do not exceed `COffMax` (and again by induction, therefore do not +exceed `CFileSize`). Other than that, `COff` values do not have to be sorted: +successive `Node`s (in `DSpace`) can be out of order (in `CSpace`), allowing +for incrementally modified RAC files. For the `Root Node`, its `COffMax` must equal the `CFileSize`. Recall that parsing a `CFile` requires knowing the `CFileSize`, and also that a `Root Node`'s `Branch CBias` is zero, so its `COffMax` equals its `CPtrMax`. -For a child `Branch Node`, its `Codec` bits must be a subset of its parent's -`Codec` bits, its `Version` must be less than or equal to its parent's -`Version`, its `COffMax` must be less than or equal to its parent's `COffMax`, -and its `DOffMax` must equal its parent's `SubBranch DOffMax`. The `DOffMax` -condition is equivalent to checking that the parent and child agree on the -child's size in `DSpace`. The parent states that it is its `(DPtr[a+1] - -DPtr[a])` and the child states that it is its `DPtrMax`. +For a child `Branch Node`, if the parent's `Codec` does not have the `Mix Bit` +set then the child's `Codec` must equal its parent's. Furthermore, its +`Version` must be less than or equal to its parent's `Version`, its `COffMax` +must be less than or equal to its parent's `COffMax`, and its `DOffMax` must +equal its parent's `SubBranch DOffMax`. The `DOffMax` condition is equivalent +to checking that the parent and child agree on the child's size in `DSpace`. +The parent states that it is its `(DPtr[a+1] - DPtr[a])` and the child states +that it is its `DPtrMax`. One conservative way to check `Branch Node`s' validity on first visit is to check them on every visit, as validating any particular `Branch Node` is @@ -618,12 +658,13 @@ Re-formatting to highlight the groups-of-8-bytes structure and reprise the 00000104 b4 00 00 00 00 00 04 00 | CPtr[2] |L|S| 0000010c 14 01 00 00 00 00 01 03 | CPtr[A] |V|A| -Its `Codec` is `0x01`, "RAC + Zlib", its Version is `0x01` and its `Arity` is -`0x03`. The `DPtr` values are `0x0000` (implicit), `0x0000`, `0x0023` and -`0x0029`. The `CPtr` values are `0x009F`, `0x0000`, `0x00B4` and `0x0114` (the -size of the whole RAC file). Note that the `CPtr` values are not sorted. The -last two children's `TTag`s are `0xFE` and `0xFE` and their `STag`s are `0x01` -and `0x00`, which means that they are both `CBiasing Branch Node`s. +Its `CodecByte` (and therefore its `Codec`) is `0x01`, "RAC + Zlib", its +Version is `0x01` and its `Arity` is `0x03`. The `DPtr` values are `0x0000` +(implicit), `0x0000`, `0x0023` and `0x0029`. The `CPtr` values are `0x009F`, +`0x0000`, `0x00B4` and `0x0114` (the size of the whole RAC file). Note that the +`CPtr` values are not sorted. The last two children's `TTag`s are `0xFE` and +`0xFE` and their `STag`s are `0x01` and `0x00`, which means that they are both +`CBiasing Branch Node`s. # Reference Implementation @@ -652,4 +693,4 @@ review. --- -Updated on August 2019. +Updated on September 2019. diff --git a/lib/rac/chunk_reader.go b/lib/rac/chunk_reader.go index 28dc0eda..f0b1fd13 100644 --- a/lib/rac/chunk_reader.go +++ b/lib/rac/chunk_reader.go @@ -25,12 +25,14 @@ func u48LE(b []byte) int64 { _ = b[7] // Early bounds check to guarantee safety of reads below. u := uint64(b[0]) | uint64(b[1])<<8 | uint64(b[2])<<16 | uint64(b[3])<<24 | uint64(b[4])<<32 | uint64(b[5])<<40 | uint64(b[6])<<48 | uint64(b[7])<<56 - return int64(u & 0xFFFFFFFFFFFF) + const mask = (1 << 48) - 1 + return int64(u & mask) } -// bitwiseSubset returns whether every 1-bit in inner is also set in outer. -func bitwiseSubset(outer Codec, inner Codec) bool { - return outer == (outer | inner) +func u64LE(b []byte) uint64 { + _ = b[7] // Early bounds check to guarantee safety of reads below. + return uint64(b[0]) | uint64(b[1])<<8 | uint64(b[2])<<16 | uint64(b[3])<<24 | + uint64(b[4])<<32 | uint64(b[5])<<40 | uint64(b[6])<<48 | uint64(b[7])<<56 } // Range is the half-open range [low, high). It is invalid for low to be @@ -78,11 +80,40 @@ func nodeSize(arity uint8) int { // true. type rNode [4096]byte -func (b *rNode) arity() int { return int(b[3]) } -func (b *rNode) codec() Codec { return Codec(b[(8*int(b[3]))+7]) } -func (b *rNode) cPtrMax() int64 { return u48LE(b[(16*int(b[3]))+8:]) } -func (b *rNode) dPtrMax() int64 { return u48LE(b[8*int(b[3]):]) } -func (b *rNode) version() uint8 { return b[(16*int(b[3]))+14] } +func (b *rNode) arity() int { return int(b[3]) } +func (b *rNode) codecHasMixBit() bool { return b[(8*int(b[3]))+7]&0x40 != 0 } +func (b *rNode) cPtrMax() int64 { return u48LE(b[(16*int(b[3]))+8:]) } +func (b *rNode) dPtrMax() int64 { return u48LE(b[8*int(b[3]):]) } +func (b *rNode) version() uint8 { return b[(16*int(b[3]))+14] } + +// codec returns the 1-byte (short) or 7-byte (long) codec as a valid Codec, or +// CodecInvalid. +// +// The (uint64) Codec value returned does not have the Mix Bit (the 62nd bit) +// set, regardless of whether it's set in the rNode's bytes. +func (b *rNode) codec() Codec { + arity := int(b[3]) + cByte := b[(8*arity)+7] + + // Look for a short codec. + if (cByte & 0x80) == 0 { + return Codec(cByte&0x3F) << 56 + } + cByte &= 0x3F + + // Look for a long codec. + const highBit, low56Bits = 0x8000000000000000, 0x00FFFFFFFFFFFFFF + for j := uint8(0); j < 4; j++ { + i := int(cByte | (j << 6)) + if (i < arity) && (b.tTag(i) == 0xFD) { + base := (8 * arity) + 8 + codec := Codec(u64LE(b[(8*i)+base:])) + return (codec & low56Bits) | highBit + } + } + + return CodecInvalid +} func (b *rNode) cLen(i int) uint8 { base := (8 * int(b[3])) + 14 @@ -178,37 +209,49 @@ func (b *rNode) valid() bool { } // Check that the "Reserved (0)" bytes are zero and that the TTag values - // aren't in the reserved range [0xC0, 0xFE). + // aren't in the reserved range [0xC0, 0xFD). + hasChildren := false for i := 0; i < arity; i++ { if b[(8*i)+6] != 0 { return false } - if tTag := b[(8*i)+7]; (0xC0 <= tTag) && (tTag < 0xFE) { + if tTag := b[(8*i)+7]; (0xC0 <= tTag) && (tTag < 0xFD) { return false + } else if tTag != 0xFD { + hasChildren = true } } - if b[(8*arity)+6] != 0 { + if !hasChildren || (b[(8*arity)+6] != 0) { return false } // Check that the DPtr values are non-decreasing. The first DPtr value is // implicitly zero. - prev := u48LE(b[8*1:]) - for i := 2; i <= arity; i++ { + prev := int64(0) + for i := 1; i <= arity; i++ { curr := u48LE(b[8*i:]) if curr < prev { return false + } else if curr != prev { + if tTag := b[(8*i)+7]; tTag == 0xFD { + return false + } } prev = curr } - // Check that no CPtr value exceeds CPtrMax (the final CPtr value). + // Check that no CPtr value exceeds CPtrMax (the final CPtr value), other + // than 0xFD Codec Entries. base := (8 * arity) + 8 cPtrMax := u48LE(b[size-8:]) for i := 0; i < arity; i++ { - if cPtr := u48LE(b[(8*i)+base:]); cPtr > cPtrMax { - return false + if cPtr := u48LE(b[(8*i)+base:]); cPtr <= cPtrMax { + continue } + if tTag := b[(8*i)+7]; tTag == 0xFD { + continue + } + return false } // Check the the version is non-zero. @@ -225,7 +268,8 @@ func (b *rNode) valid() bool { // Further checking of the codec, version, COffMax and DOffMax requires // more context, and is done in loadAndValidate. - return true + + return b.codec().Valid() } // ChunkReader parses a RAC file. @@ -431,7 +475,7 @@ func (r *ChunkReader) load(cOffset int64, arity uint8) error { } func (r *ChunkReader) loadAndValidate(cOffset int64, - parentCodec Codec, parentVersion uint8, parentCOffMax int64, + parentCodec Codec, parentCodecHasMixBit bool, parentVersion uint8, parentCOffMax int64, childCBias int64, childDSize int64) error { if (cOffset < 0) || ((r.CompressedSize - 4) < cOffset) { @@ -467,7 +511,7 @@ func (r *ChunkReader) loadAndValidate(cOffset int64, // Validate the parent and child codec, version, COffMax and DOffMax. childVersion := r.currNode.version() - if !bitwiseSubset(parentCodec, r.currNode.codec()) || + if !parentChildCodecsValid(parentCodec, r.currNode.codec(), parentCodecHasMixBit) || (parentVersion < childVersion) || (parentCOffMax < (childCBias + r.currNode.cPtrMax())) || (childDSize != r.currNode.dPtrMax()) { @@ -553,6 +597,7 @@ func (r *ChunkReader) resolveSeekPosition() error { } parentCodec := r.currNode.codec() + parentCodecHasMixBit := r.currNode.codecHasMixBit() parentVersion := r.currNode.version() parentCOffMax := cBias + r.currNode.cPtrMax() childCOffset := r.currNode.cOff(i, cBias) @@ -564,7 +609,7 @@ func (r *ChunkReader) resolveSeekPosition() error { childDSize := r.currNode.dSize(i) if err := r.loadAndValidate(childCOffset, - parentCodec, parentVersion, parentCOffMax, + parentCodec, parentCodecHasMixBit, parentVersion, parentCOffMax, childCBias, childDSize); err != nil { return err } diff --git a/lib/rac/chunk_writer.go b/lib/rac/chunk_writer.go index f665b239..f07442ea 100644 --- a/lib/rac/chunk_writer.go +++ b/lib/rac/chunk_writer.go @@ -15,6 +15,7 @@ package rac import ( + "errors" "hash/crc32" "io" "sort" @@ -346,6 +347,9 @@ func (w *ChunkWriter) AddChunk( dRangeSize uint64, codec Codec, primary []byte, secondary OptResource, tertiary OptResource) error { + if w.err != nil { + return w.err + } if dRangeSize == 0 { return nil } @@ -361,6 +365,15 @@ func (w *ChunkWriter) AddChunk( if err := w.initialize(); err != nil { return err } + if len(w.leafNodes) == 0 { + if !codec.Valid() { + return errInvalidCodec + } + w.codec = codec + } else if w.codec != codec { + w.err = errors.New("rac: TODO: support writing multiple Codecs") + return w.err + } if err := w.write(primary); err != nil { return err } @@ -413,7 +426,7 @@ func (w *ChunkWriter) Close() error { _, err := w.Writer.Write(emptyRACFile[:]) return err } - rootNode := gather(w.leafNodes) + rootNode := gather(w.leafNodes, w.codec.isLong()) indexSize := rootNode.calcEncodedSize(0, w.IndexLocation == IndexLocationAtEnd) nw := &nodeWriter{ @@ -505,6 +518,9 @@ func (n *wNode) calcEncodedSize(accumulator uint64, rootAndIsAtEnd bool) (newAcc if arity == 0 { return accumulator } + if n.codec.isLong() { + arity++ + } size := (arity * 16) + 16 cLength := calcCLength(size) @@ -552,12 +568,25 @@ func (w *nodeWriter) writeIndex(n *wNode, rootAndIsAtEnd bool) error { } } + if !n.codec.Valid() { + return errors.New("rac: TODO: support writing multiple Codecs") + } + buf, dPtr := w.buffer[:], uint64(0) arity := uint64(len(n.children) + len(n.resources)) + if n.codec.isLong() { + arity++ + } if arity > 0xFF { return errInternalArityIsTooLarge } + // Codec Element 'DPtr|Reserved0|TTag', if present. + if n.codec.isLong() { + putU64LE(buf, 0xFD00000000000000) + buf = buf[8:] + } + // DPtr's. We write resources before regular children (non-resources), so // that any TTag that refers to a resource always avoids the [0xC0, 0xFD] // reserved zone. The ratio of resources to regulars is at most 2:1, as @@ -577,10 +606,18 @@ func (w *nodeWriter) writeIndex(n *wNode, rootAndIsAtEnd bool) error { } buf = buf[8*len(n.children):] - // DPtrMax. - putU64LE(buf, dPtr|(uint64(n.codec)<<56)) + // DPtrMax and the CodecByte. By construction, a Long Codec's 'c64' value + // is always 0, as the Codec Element is always in position 0. + codecHighByte := uint64(n.codec) & 0xFF00000000000000 + putU64LE(buf, dPtr|codecHighByte) buf = buf[8:] + // Codec Element 'CPtr|CLen|STag', if present. + if n.codec.isLong() { + putU64LE(buf, uint64(n.codec)&0x00FFFFFFFFFFFFFF) + buf = buf[8:] + } + // CPtr's. While the RAC format allows otherwise, this nodeWriter always // keeps the CBias at zero, so that a CPtr equals a COffset. for i, res := range n.resources { @@ -670,13 +707,22 @@ func resourceToTag(resources []int, r OptResource) uint64 { // that the root node always directly contains the first and last leaf nodes as // children, as those leaves presumably contain the most commonly accessed // parts of the decompressed file. -func gather(nodes []wNode) wNode { +// +// If doing this TODO, we'd also have to change the "codec = codecMixBit | +// CodecZeroes" line below, as it assumes that no branch nodes have both branch +// node children and leaf node children. +func gather(nodes []wNode, codecIsLong bool) wNode { if len(nodes) == 0 { panic("gather: no nodes") } resources := map[OptResource]bool{} + arityBudget := 0xFF + if codecIsLong { + arityBudget = 0xFE + } + for { i, j, arity, newNodes := 0, 0, 0, []wNode(nil) for ; j < len(nodes); j++ { @@ -685,7 +731,7 @@ func gather(nodes []wNode) wNode { new2 := (o.secondary != 0) && !resources[o.secondary] new3 := (o.tertiary != 0) && !resources[o.tertiary] arity += 1 + btoi(new2) + btoi(new3) - if arity <= 0xFF { + if arity <= arityBudget { if new2 { resources[o.secondary] = true } @@ -727,9 +773,19 @@ func gather(nodes []wNode) wNode { func makeBranch(children []wNode, resMap map[OptResource]bool) wNode { dRangeSize, codec := uint64(0), Codec(0) - for _, c := range children { + for i, c := range children { dRangeSize += c.dRangeSize - codec |= c.codec + if i == 0 { + codec = c.codec + } else if codec != c.codec { + // We construct the node tree so that a branch node's children are + // either all branch nodes or all leaf nodes. If they are all leaf + // nodes (with the same Codec), or all branch nodes with the same + // Codec, the parent uses the same Codec. If they are all branch + // nodes, with different Codecs, we set the Mix bit and since it + // has no leaf nodes, we might as well use CodecZeroes. + codec = codecMixBit | CodecZeroes + } } resList := []int(nil) diff --git a/lib/rac/rac.go b/lib/rac/rac.go index 8933ced8..093fa9ef 100644 --- a/lib/rac/rac.go +++ b/lib/rac/rac.go @@ -39,14 +39,71 @@ const ( // Codec is the compression codec for the RAC file. // +// For leaf nodes, there are two categories of valid Codecs: Short and Long. A +// Short Codec's uint64 value's high 2 bits and low 56 bits must be zero. A +// Long Codec's uint64 value's high 8 bits must be one and then 7 zeroes. +// Symbolically, Short and Long match: +// - 0b00??????_00000000_00000000_00000000_00000000_00000000_00000000_00000000 +// - 0b10000000_????????_????????_????????_????????_????????_????????_???????? +// +// In terms of the RAC file format, a Short Codec fits in a single byte: the +// Codec Byte in the middle of a branch node. A Long Codec uses that Codec Byte +// to locate 7 other bytes, a location which would otherwise form a "CPtr|CLen" +// value. When converting from the 7 bytes in the wire format to this Go type +// (a uint64 value), they are read little-endian: the "CPtr" bytes are the low +// 6 bytes, the "CLen" is the second-highest byte and the highest byte is +// hard-coded to 0x80. +// +// For example, the 7 bytes "mdo2\x00\x00\x00" would correspond to a Codec +// value of 0x8000_0000_326F_646D. +// +// The Mix Bit is not part of the uint64 representation. Neither is a Long +// Codec's 'c64' index. This package's exported API deals with leaf nodes. +// Branch nodes' wire formats are internal implementation details. +// // See the RAC specification for further discussion. -type Codec uint8 +type Codec uint64 + +func (c Codec) isLong() bool { return int64(c) < 0 } +func (c Codec) isShort() bool { return int64(c) >= 0 } + +// Valid returns whether c matches the Short or Long pattern. +func (c Codec) Valid() bool { + if (c >> 63) == 0 { + return ((c << 8) == 0) && ((c >> 62) == 0) + } + return (c >> 56) == 0x80 +} + +func (c Codec) name() string { + if c.isShort() && c.Valid() { + switch c >> 56 { + case 0: + return "Zeroes" + case 1: + return "Zlib" + case 2: + return "Brotli" + case 4: + return "Zstandard" + } + } + return "" +} + +func parentChildCodecsValid(parent Codec, child Codec, parentHasMixBit bool) bool { + return (parent == child) || parentHasMixBit +} const ( - CodecZeroes = Codec(0x00) - CodecZlib = Codec(0x01) - CodecBrotli = Codec(0x02) - CodecZstandard = Codec(0x04) + CodecZeroes = Codec(0x00 << 56) + CodecZlib = Codec(0x01 << 56) + CodecBrotli = Codec(0x02 << 56) + CodecZstandard = Codec(0x04 << 56) + + codecMixBit = Codec(1 << 62) + codecLongZeroes = Codec(1 << 63) + CodecInvalid = Codec((1 << 64) - 1) ) // IndexLocation is whether the index is at the start or end of the RAC file. diff --git a/lib/rac/rac_test.go b/lib/rac/rac_test.go index a351d81e..7daa197c 100644 --- a/lib/rac/rac_test.go +++ b/lib/rac/rac_test.go @@ -594,3 +594,53 @@ func TestReaderZeroes(t *testing.T) { t.Fatalf("got %q, want %q", got, want) } } + +func TestLongCodec(t *testing.T) { + const codec = Codec(0x80000000326F646D) // "mdo2" backwards, with a high bit. + buf := &bytes.Buffer{} + w := &ChunkWriter{ + Writer: buf, + IndexLocation: IndexLocationAtStart, + TempFile: &bytes.Buffer{}, + } + if err := w.AddChunk(0x66, codec, []byte{0xAA, 0xBB}, 0, 0); err != nil { + t.Fatalf("AddChunk: %v", err) + } + if err := w.AddChunk(0x77, codec, []byte{0xCC}, 0, 0); err != nil { + t.Fatalf("AddChunk: %v", err) + } + if err := w.Close(); err != nil { + t.Fatalf("Close: %v", err) + } + + encoded := buf.Bytes() + gotHexDump := hex.Dump(encoded) + + const wantHexDump = "" + + "00000000 72 c3 63 03 3d c9 00 fd 00 00 00 00 00 00 00 ff |r.c.=...........|\n" + + "00000010 66 00 00 00 00 00 00 ff dd 00 00 00 00 00 00 80 |f...............|\n" + + "00000020 6d 64 6f 32 00 00 00 00 40 00 00 00 00 00 01 ff |mdo2....@.......|\n" + + "00000030 42 00 00 00 00 00 01 ff 43 00 00 00 00 00 01 03 |B.......C.......|\n" + + "00000040 aa bb cc |...|\n" + + if gotHexDump != wantHexDump { + t.Fatalf("\ngot:\n%s\nwant:\n%s", gotHexDump, wantHexDump) + } + + r := &ChunkReader{ + ReadSeeker: bytes.NewReader(encoded), + CompressedSize: int64(len(encoded)), + } + for i := 0; i < 2; i++ { + c, err := r.NextChunk() + if err != nil { + t.Fatalf("i=%d: %v", i, err) + } + if got, want := c.Codec, codec; got != want { + t.Fatalf("i=%d: Codec: got 0x%X, want 0x%X", i, got, want) + } + } + if _, err := r.NextChunk(); err != io.EOF { + t.Fatalf("got %v, want %v", err, io.EOF) + } +} diff --git a/lib/rac/reader.go b/lib/rac/reader.go index d64bb920..dc50e17b 100644 --- a/lib/rac/reader.go +++ b/lib/rac/reader.go @@ -368,7 +368,7 @@ func (r *Reader) nextChunk() error { return r.err } - if chunk.Codec == 0 { + if (chunk.Codec == CodecZeroes) || (chunk.Codec == codecLongZeroes) { r.currChunk.N = 0 r.dRange = chunk.DRange r.zeroes = zeroesReader(r.dRange.Size()) @@ -384,16 +384,12 @@ func (r *Reader) nextChunk() error { } } if codecReader == nil { - name := "" - switch chunk.Codec { - case CodecZlib: - name = " (Zlib)" - case CodecBrotli: - name = " (Brotli)" - case CodecZstandard: - name = " (Zstandard)" + name0, name1, name2 := "", chunk.Codec.name(), "" + if name1 != "" { + name0, name2 = " (", ")" } - r.err = fmt.Errorf("rac: no matching CodecReader for Codec 0x%02X%s", chunk.Codec, name) + r.err = fmt.Errorf("rac: no matching CodecReader for Codec 0x%X%s%s%s", + chunk.Codec, name0, name1, name2) return r.err } diff --git a/lib/raczlib/raczlib_test.go b/lib/raczlib/raczlib_test.go index aba447da..66bfd549 100644 --- a/lib/raczlib/raczlib_test.go +++ b/lib/raczlib/raczlib_test.go @@ -161,7 +161,7 @@ func TestReaderConcatenation(t *testing.T) { buf[0x37] = 0x00 // CBiasing with COff[0], which is encLen0. // Codec and Version. - buf[0x1F] = byte(rac.CodecZlib) + buf[0x1F] = byte(rac.CodecZlib >> 56) buf[0x3E] = 0x01 // Checksum.