mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-18 00:32:11 +01:00
Re-do RAC codec values, from 8 bits to 56 bits
This commit is contained in:
1 parent
e7c7e7bcf9
commit
4006157f9f
7 files changed
+331
-86
No files matched your search
+83
-42
@@ -1,6 +1,6 @@
|
||||
# Random Access Compression: RAC
|
||||
|
||||
Status: Draft (as of August 2019). There is no compatibility guarantee yet.
|
||||
Status: Draft (as of September 2019). There is no compatibility guarantee yet.
|
||||
|
||||
|
||||
## Overview
|
||||
@@ -32,10 +32,11 @@ Non-goals for version 1 include:
|
||||
- Filesystem metadata like file names, file sizes and modification times.
|
||||
- Multiple source files.
|
||||
|
||||
There is the capability (see reserved `TTag`s, below) but no promise to address
|
||||
these in a future RAC version. There might not be a need to, as other designs
|
||||
such as [SquashFS](http://squashfs.sourceforge.net/) and EROFS ([Extendable
|
||||
Read-Only File System](https://lkml.org/lkml/2018/5/31/306)) already exist.
|
||||
There is the capability (see attributes and reserved `TTag`s, below) but no
|
||||
promise to address these in a future RAC version. There might not be a need to,
|
||||
as other designs such as [SquashFS](http://squashfs.sourceforge.net/) and EROFS
|
||||
([Extendable Read-Only File System](https://lkml.org/lkml/2018/5/31/306))
|
||||
already exist.
|
||||
|
||||
Non-goals in general include:
|
||||
|
||||
@@ -88,8 +89,10 @@ at least one `Branch Node`, called the `Root Node`. Parsing a `CFile` requires
|
||||
knowing the `CFileSize` in order to identify the `Root Node`, which is either
|
||||
at the start or the end of the `CFile`.
|
||||
|
||||
Each `Node` has a `DRange`. An empty `DRange` means that the `Node` contains
|
||||
metadata or other decompression context such as a shared dictionary.
|
||||
Each `Node` has a `DRange`, which can be empty. `Branch Node`s can also have
|
||||
attributes: elements that aren't child `Node`s, which must have an empty
|
||||
`DRange`. Empty elements contain metadata or other decompression context such
|
||||
as a shared dictionary.
|
||||
|
||||
Each `Leaf Node` also has 3 `CRange`s (`Primary`, `Secondary` and `Tertiary`),
|
||||
any or all of which may be empty. The contents of the `CFile`, within those
|
||||
@@ -114,7 +117,7 @@ COffset`:
|
||||
| ... |0|.| ..., Reserved (0), ...
|
||||
| DPtr[A-2] |0|T| DPtr[Arity-2], Reserved (0), TTag[Arity-2]
|
||||
| DPtr[A-1] |0|T| DPtr[Arity-1], Reserved (0), TTag[Arity-1]
|
||||
| DPtr[A] |0|C| DPtr[Arity] a.k.a. DPtrMax, Reserved (0), Codec
|
||||
| DPtr[A] |0|C| DPtr[Arity] a.k.a. DPtrMax, Reserved (0), CodecByte
|
||||
| CPtr[0] |L|S| CPtr[0], CLen[0], STag[0]
|
||||
| CPtr[1] |L|S| CPtr[1], CLen[1], STag[1]
|
||||
| CPtr[2] |L|S| CPtr[2], CLen[2], STag[2]
|
||||
@@ -164,7 +167,8 @@ Node` positioned at the start of the `CFile`.
|
||||
|
||||
### Arity
|
||||
|
||||
`Arity` is the `Branch Node`'s number of children. Zero is invalid.
|
||||
`Arity` is the `Branch Node`'s number of elements: the number of child `Node`s
|
||||
plus the number of attributes. Having zero children is invalid.
|
||||
|
||||
The `Arity` byte is given twice: the fourth byte and the final byte of the
|
||||
`Branch Node`. The two values must match.
|
||||
@@ -196,14 +200,19 @@ The `Reserved (0)` bytes must have the value `0x00`.
|
||||
|
||||
### COffs and DOffs, STags and TTags
|
||||
|
||||
For every `a` in the half-open range `[0 .. Arity)`, the `a`'th child `Node`
|
||||
has two tags, `STag[a]` and `TTag[a]`, and a `DRange` of `[DOff[a] ..
|
||||
DOff[a+1])`. The `DOff` values must be non-decreasing: see the "Branch Node
|
||||
Validation" section below.
|
||||
For every `a` in the half-open range `[0 .. Arity)`, the `a`'th element has two
|
||||
tags, `STag[a]` and `TTag[a]`, and a `DRange` of `[DOff[a] .. DOff[a+1])`. The
|
||||
`DOff` values must be non-decreasing: see the "Branch Node Validation" section
|
||||
below.
|
||||
|
||||
A `TTag[a]` of `0xFE` means that child is a `Branch Node`. A `TTag[a]` in the
|
||||
half-open range `[0xC0 .. 0xFE)` is reserved. Otherwise, the child is a `Leaf
|
||||
Node`.
|
||||
A `TTag[a]` of `0xFE` means that that element is a `Branch Node` child.
|
||||
|
||||
A `TTag[a]` of `0xFD` means that there is no child, but is instead a `Codec
|
||||
Element` attribute, whose `DRange` must be empty, and the rest of this section
|
||||
does not apply: the `STag` is ignored.
|
||||
|
||||
A `TTag[a]` in the half-open range `[0xC0 .. 0xFD)` is reserved. Otherwise, the
|
||||
element is a `Leaf Node` child.
|
||||
|
||||
A child `Branch Node`'s `SubBranch COffset` is defined to be `COff[a]`. Its
|
||||
`SubBranch DBias` and `SubBranch DOffMax` are defined to be `DOff[a]` and
|
||||
@@ -272,47 +281,78 @@ values.
|
||||
`Codec`s define specializations of RAC, such as "RAC + Zlib" or "RAC + Brotli".
|
||||
It is valid for a "RAC + Zstandard" only decoder to reject a "RAC + Brotli"
|
||||
file, even if it is a valid RAC file. Recall that RAC is just a container, and
|
||||
not tied to any particular compression codec. For the `Codec` byte in a `Branch
|
||||
Node`:
|
||||
not tied to any particular compression codec.
|
||||
|
||||
There are two categories of `Codec`s: the high `0x80` bit of the `Codec Byte`
|
||||
being `0` or `1` denotes a `Short` or `Long` codec respectively. `Short Codec`s
|
||||
are represented by a single byte (the `Codec Byte`). `Long Codec`s use that
|
||||
`Codec Byte` to locate 7 additional bytes: a `Codec Element`.
|
||||
|
||||
For both `Short Codec`s and `Long Codec`s, the second-highest `0x40` bit of the
|
||||
`Codec Byte` is the `Mix Bit`. A `Mix Bit` of `0` means that all of the
|
||||
`Node`'s descendents have exactly the same `Codec` (not just the same `Codec
|
||||
Byte`; `Short Codec`s and `Long Codec`s are not considered exactly the same
|
||||
even if they represent the same compression algorithm). A `Mix Bit` of `1`
|
||||
means that descendents may have a different `Codec`.
|
||||
|
||||
For `Short Codec`s, the remaining low 6 bits correspond to a specific
|
||||
compression algorithm:
|
||||
|
||||
- `0x00` means "RAC + Zeroes".
|
||||
- `0x01` means "RAC + Zlib".
|
||||
- `0x02` means "RAC + Brotli".
|
||||
- `0x04` means "RAC + ZStandard".
|
||||
- Any other value less than 0x08 means that all of this `Branch Node`'s
|
||||
children must be `Branch Node`s and not `Leaf Node`s and that no child's
|
||||
`Codec` byte can have a bit set that is not set in this `Codec` byte.
|
||||
- All other values, 0x08 or greater, are reserved.
|
||||
- All other values are reserved.
|
||||
|
||||
For `Long Codec`s, the remaining low 6 bits of the `Codec Byte` define a number
|
||||
`c64`. The lowest index `i` out of `(c64 + (64 * 0))`, `(c64 + (64 * 1))`,
|
||||
`(c64 + (64 * 2))` and `(c64 + (64 * 3))` such that `TTag[i]` is `0xFD` locates
|
||||
the 7 bytes that identifies the `Codec`. The location is what would otherwise
|
||||
occupy the `i`th element's `CPtr | CLen` space. It is invalid for no such `i`
|
||||
to exist.
|
||||
|
||||
`Long Codec` values are not reserved by this specification, other than 7 NUL
|
||||
bytes also means "RAC + Zeroes". Users may define their own values for
|
||||
arbitrary compression algorithms. Maintaining a centralized registry mapping
|
||||
values to algorithms is out of scope of this specification, although we suggest
|
||||
a convention that the 7 bytes form a human-readable (ASCII) string, padded with
|
||||
trailing NUL bytes. For example, a hypothetical "Middle Out version 2"
|
||||
compression algorithm that typically used the ".mdo2" file extension might be
|
||||
represented by the 7 bytes `"mdo2\x00\x00\x00"`.
|
||||
|
||||
|
||||
### Branch Node Validation
|
||||
|
||||
The first time that a RAC reader visits any particular `Branch Node`, it must
|
||||
check that the `Magic` matches, the two `Arity` values match and are non-zero,
|
||||
the computed checksum matches the listed `Checksum` and that the RAC reader
|
||||
accepts the `Version` and the `Codec`.
|
||||
there is at least one child `Node` (not just non-`Node` attributes), the
|
||||
computed checksum matches the listed `Checksum` and that the RAC reader accepts
|
||||
the `Version`. For `Long Codec`s, there must exist an `0xFD` `TTag` as per the
|
||||
previous section.
|
||||
|
||||
It must also check that all of its `DOff` values are sorted: `(DOff[a] <=
|
||||
DOff[a+1])` for every `a` in the half-open range `[0 .. Arity)`. By induction,
|
||||
this means that all of its `DOff` values do not exceed `DOffMax`, and again by
|
||||
induction, therefore do not exceed `DFileSize`.
|
||||
|
||||
It must also check that all of its `COff` values do not exceed `COffMax` (and
|
||||
again by induction, therefore do not exceed `CFileSize`). Other than that,
|
||||
`COff` values do not have to be sorted: successive `Node`s (in `DSpace`) can be
|
||||
out of order (in `CSpace`), allowing for incrementally modified RAC files.
|
||||
It must also check that, other than `Codec Element` attributes, all of its
|
||||
`COff` values do not exceed `COffMax` (and again by induction, therefore do not
|
||||
exceed `CFileSize`). Other than that, `COff` values do not have to be sorted:
|
||||
successive `Node`s (in `DSpace`) can be out of order (in `CSpace`), allowing
|
||||
for incrementally modified RAC files.
|
||||
|
||||
For the `Root Node`, its `COffMax` must equal the `CFileSize`. Recall that
|
||||
parsing a `CFile` requires knowing the `CFileSize`, and also that a `Root
|
||||
Node`'s `Branch CBias` is zero, so its `COffMax` equals its `CPtrMax`.
|
||||
|
||||
For a child `Branch Node`, its `Codec` bits must be a subset of its parent's
|
||||
`Codec` bits, its `Version` must be less than or equal to its parent's
|
||||
`Version`, its `COffMax` must be less than or equal to its parent's `COffMax`,
|
||||
and its `DOffMax` must equal its parent's `SubBranch DOffMax`. The `DOffMax`
|
||||
condition is equivalent to checking that the parent and child agree on the
|
||||
child's size in `DSpace`. The parent states that it is its `(DPtr[a+1] -
|
||||
DPtr[a])` and the child states that it is its `DPtrMax`.
|
||||
For a child `Branch Node`, if the parent's `Codec` does not have the `Mix Bit`
|
||||
set then the child's `Codec` must equal its parent's. Furthermore, its
|
||||
`Version` must be less than or equal to its parent's `Version`, its `COffMax`
|
||||
must be less than or equal to its parent's `COffMax`, and its `DOffMax` must
|
||||
equal its parent's `SubBranch DOffMax`. The `DOffMax` condition is equivalent
|
||||
to checking that the parent and child agree on the child's size in `DSpace`.
|
||||
The parent states that it is its `(DPtr[a+1] - DPtr[a])` and the child states
|
||||
that it is its `DPtrMax`.
|
||||
|
||||
One conservative way to check `Branch Node`s' validity on first visit is to
|
||||
check them on every visit, as validating any particular `Branch Node` is
|
||||
@@ -618,12 +658,13 @@ Re-formatting to highlight the groups-of-8-bytes structure and reprise the
|
||||
00000104 b4 00 00 00 00 00 04 00 | CPtr[2] |L|S|
|
||||
0000010c 14 01 00 00 00 00 01 03 | CPtr[A] |V|A|
|
||||
|
||||
Its `Codec` is `0x01`, "RAC + Zlib", its Version is `0x01` and its `Arity` is
|
||||
`0x03`. The `DPtr` values are `0x0000` (implicit), `0x0000`, `0x0023` and
|
||||
`0x0029`. The `CPtr` values are `0x009F`, `0x0000`, `0x00B4` and `0x0114` (the
|
||||
size of the whole RAC file). Note that the `CPtr` values are not sorted. The
|
||||
last two children's `TTag`s are `0xFE` and `0xFE` and their `STag`s are `0x01`
|
||||
and `0x00`, which means that they are both `CBiasing Branch Node`s.
|
||||
Its `CodecByte` (and therefore its `Codec`) is `0x01`, "RAC + Zlib", its
|
||||
Version is `0x01` and its `Arity` is `0x03`. The `DPtr` values are `0x0000`
|
||||
(implicit), `0x0000`, `0x0023` and `0x0029`. The `CPtr` values are `0x009F`,
|
||||
`0x0000`, `0x00B4` and `0x0114` (the size of the whole RAC file). Note that the
|
||||
`CPtr` values are not sorted. The last two children's `TTag`s are `0xFE` and
|
||||
`0xFE` and their `STag`s are `0x01` and `0x00`, which means that they are both
|
||||
`CBiasing Branch Node`s.
|
||||
|
||||
|
||||
# Reference Implementation
|
||||
@@ -652,4 +693,4 @@ review.
|
||||
|
||||
---
|
||||
|
||||
Updated on August 2019.
|
||||
Updated on September 2019.
|
||||
+66
-21
@@ -25,12 +25,14 @@ func u48LE(b []byte) int64 {
|
||||
_ = b[7] // Early bounds check to guarantee safety of reads below.
|
||||
u := uint64(b[0]) | uint64(b[1])<<8 | uint64(b[2])<<16 | uint64(b[3])<<24 |
|
||||
uint64(b[4])<<32 | uint64(b[5])<<40 | uint64(b[6])<<48 | uint64(b[7])<<56
|
||||
return int64(u & 0xFFFFFFFFFFFF)
|
||||
const mask = (1 << 48) - 1
|
||||
return int64(u & mask)
|
||||
}
|
||||
|
||||
// bitwiseSubset returns whether every 1-bit in inner is also set in outer.
|
||||
func bitwiseSubset(outer Codec, inner Codec) bool {
|
||||
return outer == (outer | inner)
|
||||
func u64LE(b []byte) uint64 {
|
||||
_ = b[7] // Early bounds check to guarantee safety of reads below.
|
||||
return uint64(b[0]) | uint64(b[1])<<8 | uint64(b[2])<<16 | uint64(b[3])<<24 |
|
||||
uint64(b[4])<<32 | uint64(b[5])<<40 | uint64(b[6])<<48 | uint64(b[7])<<56
|
||||
}
|
||||
|
||||
// Range is the half-open range [low, high). It is invalid for low to be
|
||||
@@ -78,11 +80,40 @@ func nodeSize(arity uint8) int {
|
||||
// true.
|
||||
type rNode [4096]byte
|
||||
|
||||
func (b *rNode) arity() int { return int(b[3]) }
|
||||
func (b *rNode) codec() Codec { return Codec(b[(8*int(b[3]))+7]) }
|
||||
func (b *rNode) cPtrMax() int64 { return u48LE(b[(16*int(b[3]))+8:]) }
|
||||
func (b *rNode) dPtrMax() int64 { return u48LE(b[8*int(b[3]):]) }
|
||||
func (b *rNode) version() uint8 { return b[(16*int(b[3]))+14] }
|
||||
func (b *rNode) arity() int { return int(b[3]) }
|
||||
func (b *rNode) codecHasMixBit() bool { return b[(8*int(b[3]))+7]&0x40 != 0 }
|
||||
func (b *rNode) cPtrMax() int64 { return u48LE(b[(16*int(b[3]))+8:]) }
|
||||
func (b *rNode) dPtrMax() int64 { return u48LE(b[8*int(b[3]):]) }
|
||||
func (b *rNode) version() uint8 { return b[(16*int(b[3]))+14] }
|
||||
|
||||
// codec returns the 1-byte (short) or 7-byte (long) codec as a valid Codec, or
|
||||
// CodecInvalid.
|
||||
//
|
||||
// The (uint64) Codec value returned does not have the Mix Bit (the 62nd bit)
|
||||
// set, regardless of whether it's set in the rNode's bytes.
|
||||
func (b *rNode) codec() Codec {
|
||||
arity := int(b[3])
|
||||
cByte := b[(8*arity)+7]
|
||||
|
||||
// Look for a short codec.
|
||||
if (cByte & 0x80) == 0 {
|
||||
return Codec(cByte&0x3F) << 56
|
||||
}
|
||||
cByte &= 0x3F
|
||||
|
||||
// Look for a long codec.
|
||||
const highBit, low56Bits = 0x8000000000000000, 0x00FFFFFFFFFFFFFF
|
||||
for j := uint8(0); j < 4; j++ {
|
||||
i := int(cByte | (j << 6))
|
||||
if (i < arity) && (b.tTag(i) == 0xFD) {
|
||||
base := (8 * arity) + 8
|
||||
codec := Codec(u64LE(b[(8*i)+base:]))
|
||||
return (codec & low56Bits) | highBit
|
||||
}
|
||||
}
|
||||
|
||||
return CodecInvalid
|
||||
}
|
||||
|
||||
func (b *rNode) cLen(i int) uint8 {
|
||||
base := (8 * int(b[3])) + 14
|
||||
@@ -178,37 +209,49 @@ func (b *rNode) valid() bool {
|
||||
}
|
||||
|
||||
// Check that the "Reserved (0)" bytes are zero and that the TTag values
|
||||
// aren't in the reserved range [0xC0, 0xFE).
|
||||
// aren't in the reserved range [0xC0, 0xFD).
|
||||
hasChildren := false
|
||||
for i := 0; i < arity; i++ {
|
||||
if b[(8*i)+6] != 0 {
|
||||
return false
|
||||
}
|
||||
if tTag := b[(8*i)+7]; (0xC0 <= tTag) && (tTag < 0xFE) {
|
||||
if tTag := b[(8*i)+7]; (0xC0 <= tTag) && (tTag < 0xFD) {
|
||||
return false
|
||||
} else if tTag != 0xFD {
|
||||
hasChildren = true
|
||||
}
|
||||
}
|
||||
if b[(8*arity)+6] != 0 {
|
||||
if !hasChildren || (b[(8*arity)+6] != 0) {
|
||||
return false
|
||||
}
|
||||
|
||||
// Check that the DPtr values are non-decreasing. The first DPtr value is
|
||||
// implicitly zero.
|
||||
prev := u48LE(b[8*1:])
|
||||
for i := 2; i <= arity; i++ {
|
||||
prev := int64(0)
|
||||
for i := 1; i <= arity; i++ {
|
||||
curr := u48LE(b[8*i:])
|
||||
if curr < prev {
|
||||
return false
|
||||
} else if curr != prev {
|
||||
if tTag := b[(8*i)+7]; tTag == 0xFD {
|
||||
return false
|
||||
}
|
||||
}
|
||||
prev = curr
|
||||
}
|
||||
|
||||
// Check that no CPtr value exceeds CPtrMax (the final CPtr value).
|
||||
// Check that no CPtr value exceeds CPtrMax (the final CPtr value), other
|
||||
// than 0xFD Codec Entries.
|
||||
base := (8 * arity) + 8
|
||||
cPtrMax := u48LE(b[size-8:])
|
||||
for i := 0; i < arity; i++ {
|
||||
if cPtr := u48LE(b[(8*i)+base:]); cPtr > cPtrMax {
|
||||
return false
|
||||
if cPtr := u48LE(b[(8*i)+base:]); cPtr <= cPtrMax {
|
||||
continue
|
||||
}
|
||||
if tTag := b[(8*i)+7]; tTag == 0xFD {
|
||||
continue
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Check the the version is non-zero.
|
||||
@@ -225,7 +268,8 @@ func (b *rNode) valid() bool {
|
||||
|
||||
// Further checking of the codec, version, COffMax and DOffMax requires
|
||||
// more context, and is done in loadAndValidate.
|
||||
return true
|
||||
|
||||
return b.codec().Valid()
|
||||
}
|
||||
|
||||
// ChunkReader parses a RAC file.
|
||||
@@ -431,7 +475,7 @@ func (r *ChunkReader) load(cOffset int64, arity uint8) error {
|
||||
}
|
||||
|
||||
func (r *ChunkReader) loadAndValidate(cOffset int64,
|
||||
parentCodec Codec, parentVersion uint8, parentCOffMax int64,
|
||||
parentCodec Codec, parentCodecHasMixBit bool, parentVersion uint8, parentCOffMax int64,
|
||||
childCBias int64, childDSize int64) error {
|
||||
|
||||
if (cOffset < 0) || ((r.CompressedSize - 4) < cOffset) {
|
||||
@@ -467,7 +511,7 @@ func (r *ChunkReader) loadAndValidate(cOffset int64,
|
||||
|
||||
// Validate the parent and child codec, version, COffMax and DOffMax.
|
||||
childVersion := r.currNode.version()
|
||||
if !bitwiseSubset(parentCodec, r.currNode.codec()) ||
|
||||
if !parentChildCodecsValid(parentCodec, r.currNode.codec(), parentCodecHasMixBit) ||
|
||||
(parentVersion < childVersion) ||
|
||||
(parentCOffMax < (childCBias + r.currNode.cPtrMax())) ||
|
||||
(childDSize != r.currNode.dPtrMax()) {
|
||||
@@ -553,6 +597,7 @@ func (r *ChunkReader) resolveSeekPosition() error {
|
||||
}
|
||||
|
||||
parentCodec := r.currNode.codec()
|
||||
parentCodecHasMixBit := r.currNode.codecHasMixBit()
|
||||
parentVersion := r.currNode.version()
|
||||
parentCOffMax := cBias + r.currNode.cPtrMax()
|
||||
childCOffset := r.currNode.cOff(i, cBias)
|
||||
@@ -564,7 +609,7 @@ func (r *ChunkReader) resolveSeekPosition() error {
|
||||
childDSize := r.currNode.dSize(i)
|
||||
|
||||
if err := r.loadAndValidate(childCOffset,
|
||||
parentCodec, parentVersion, parentCOffMax,
|
||||
parentCodec, parentCodecHasMixBit, parentVersion, parentCOffMax,
|
||||
childCBias, childDSize); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
+63
-7
@@ -15,6 +15,7 @@
|
||||
package rac
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"hash/crc32"
|
||||
"io"
|
||||
"sort"
|
||||
@@ -346,6 +347,9 @@ func (w *ChunkWriter) AddChunk(
|
||||
dRangeSize uint64, codec Codec, primary []byte,
|
||||
secondary OptResource, tertiary OptResource) error {
|
||||
|
||||
if w.err != nil {
|
||||
return w.err
|
||||
}
|
||||
if dRangeSize == 0 {
|
||||
return nil
|
||||
}
|
||||
@@ -361,6 +365,15 @@ func (w *ChunkWriter) AddChunk(
|
||||
if err := w.initialize(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(w.leafNodes) == 0 {
|
||||
if !codec.Valid() {
|
||||
return errInvalidCodec
|
||||
}
|
||||
w.codec = codec
|
||||
} else if w.codec != codec {
|
||||
w.err = errors.New("rac: TODO: support writing multiple Codecs")
|
||||
return w.err
|
||||
}
|
||||
if err := w.write(primary); err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -413,7 +426,7 @@ func (w *ChunkWriter) Close() error {
|
||||
_, err := w.Writer.Write(emptyRACFile[:])
|
||||
return err
|
||||
}
|
||||
rootNode := gather(w.leafNodes)
|
||||
rootNode := gather(w.leafNodes, w.codec.isLong())
|
||||
indexSize := rootNode.calcEncodedSize(0, w.IndexLocation == IndexLocationAtEnd)
|
||||
|
||||
nw := &nodeWriter{
|
||||
@@ -505,6 +518,9 @@ func (n *wNode) calcEncodedSize(accumulator uint64, rootAndIsAtEnd bool) (newAcc
|
||||
if arity == 0 {
|
||||
return accumulator
|
||||
}
|
||||
if n.codec.isLong() {
|
||||
arity++
|
||||
}
|
||||
size := (arity * 16) + 16
|
||||
cLength := calcCLength(size)
|
||||
|
||||
@@ -552,12 +568,25 @@ func (w *nodeWriter) writeIndex(n *wNode, rootAndIsAtEnd bool) error {
|
||||
}
|
||||
}
|
||||
|
||||
if !n.codec.Valid() {
|
||||
return errors.New("rac: TODO: support writing multiple Codecs")
|
||||
}
|
||||
|
||||
buf, dPtr := w.buffer[:], uint64(0)
|
||||
arity := uint64(len(n.children) + len(n.resources))
|
||||
if n.codec.isLong() {
|
||||
arity++
|
||||
}
|
||||
if arity > 0xFF {
|
||||
return errInternalArityIsTooLarge
|
||||
}
|
||||
|
||||
// Codec Element 'DPtr|Reserved0|TTag', if present.
|
||||
if n.codec.isLong() {
|
||||
putU64LE(buf, 0xFD00000000000000)
|
||||
buf = buf[8:]
|
||||
}
|
||||
|
||||
// DPtr's. We write resources before regular children (non-resources), so
|
||||
// that any TTag that refers to a resource always avoids the [0xC0, 0xFD]
|
||||
// reserved zone. The ratio of resources to regulars is at most 2:1, as
|
||||
@@ -577,10 +606,18 @@ func (w *nodeWriter) writeIndex(n *wNode, rootAndIsAtEnd bool) error {
|
||||
}
|
||||
buf = buf[8*len(n.children):]
|
||||
|
||||
// DPtrMax.
|
||||
putU64LE(buf, dPtr|(uint64(n.codec)<<56))
|
||||
// DPtrMax and the CodecByte. By construction, a Long Codec's 'c64' value
|
||||
// is always 0, as the Codec Element is always in position 0.
|
||||
codecHighByte := uint64(n.codec) & 0xFF00000000000000
|
||||
putU64LE(buf, dPtr|codecHighByte)
|
||||
buf = buf[8:]
|
||||
|
||||
// Codec Element 'CPtr|CLen|STag', if present.
|
||||
if n.codec.isLong() {
|
||||
putU64LE(buf, uint64(n.codec)&0x00FFFFFFFFFFFFFF)
|
||||
buf = buf[8:]
|
||||
}
|
||||
|
||||
// CPtr's. While the RAC format allows otherwise, this nodeWriter always
|
||||
// keeps the CBias at zero, so that a CPtr equals a COffset.
|
||||
for i, res := range n.resources {
|
||||
@@ -670,13 +707,22 @@ func resourceToTag(resources []int, r OptResource) uint64 {
|
||||
// that the root node always directly contains the first and last leaf nodes as
|
||||
// children, as those leaves presumably contain the most commonly accessed
|
||||
// parts of the decompressed file.
|
||||
func gather(nodes []wNode) wNode {
|
||||
//
|
||||
// If doing this TODO, we'd also have to change the "codec = codecMixBit |
|
||||
// CodecZeroes" line below, as it assumes that no branch nodes have both branch
|
||||
// node children and leaf node children.
|
||||
func gather(nodes []wNode, codecIsLong bool) wNode {
|
||||
if len(nodes) == 0 {
|
||||
panic("gather: no nodes")
|
||||
}
|
||||
|
||||
resources := map[OptResource]bool{}
|
||||
|
||||
arityBudget := 0xFF
|
||||
if codecIsLong {
|
||||
arityBudget = 0xFE
|
||||
}
|
||||
|
||||
for {
|
||||
i, j, arity, newNodes := 0, 0, 0, []wNode(nil)
|
||||
for ; j < len(nodes); j++ {
|
||||
@@ -685,7 +731,7 @@ func gather(nodes []wNode) wNode {
|
||||
new2 := (o.secondary != 0) && !resources[o.secondary]
|
||||
new3 := (o.tertiary != 0) && !resources[o.tertiary]
|
||||
arity += 1 + btoi(new2) + btoi(new3)
|
||||
if arity <= 0xFF {
|
||||
if arity <= arityBudget {
|
||||
if new2 {
|
||||
resources[o.secondary] = true
|
||||
}
|
||||
@@ -727,9 +773,19 @@ func gather(nodes []wNode) wNode {
|
||||
|
||||
func makeBranch(children []wNode, resMap map[OptResource]bool) wNode {
|
||||
dRangeSize, codec := uint64(0), Codec(0)
|
||||
for _, c := range children {
|
||||
for i, c := range children {
|
||||
dRangeSize += c.dRangeSize
|
||||
codec |= c.codec
|
||||
if i == 0 {
|
||||
codec = c.codec
|
||||
} else if codec != c.codec {
|
||||
// We construct the node tree so that a branch node's children are
|
||||
// either all branch nodes or all leaf nodes. If they are all leaf
|
||||
// nodes (with the same Codec), or all branch nodes with the same
|
||||
// Codec, the parent uses the same Codec. If they are all branch
|
||||
// nodes, with different Codecs, we set the Mix bit and since it
|
||||
// has no leaf nodes, we might as well use CodecZeroes.
|
||||
codec = codecMixBit | CodecZeroes
|
||||
}
|
||||
}
|
||||
|
||||
resList := []int(nil)
|
||||
|
||||
+62
-5
@@ -39,14 +39,71 @@ const (
|
||||
|
||||
// Codec is the compression codec for the RAC file.
|
||||
//
|
||||
// For leaf nodes, there are two categories of valid Codecs: Short and Long. A
|
||||
// Short Codec's uint64 value's high 2 bits and low 56 bits must be zero. A
|
||||
// Long Codec's uint64 value's high 8 bits must be one and then 7 zeroes.
|
||||
// Symbolically, Short and Long match:
|
||||
// - 0b00??????_00000000_00000000_00000000_00000000_00000000_00000000_00000000
|
||||
// - 0b10000000_????????_????????_????????_????????_????????_????????_????????
|
||||
//
|
||||
// In terms of the RAC file format, a Short Codec fits in a single byte: the
|
||||
// Codec Byte in the middle of a branch node. A Long Codec uses that Codec Byte
|
||||
// to locate 7 other bytes, a location which would otherwise form a "CPtr|CLen"
|
||||
// value. When converting from the 7 bytes in the wire format to this Go type
|
||||
// (a uint64 value), they are read little-endian: the "CPtr" bytes are the low
|
||||
// 6 bytes, the "CLen" is the second-highest byte and the highest byte is
|
||||
// hard-coded to 0x80.
|
||||
//
|
||||
// For example, the 7 bytes "mdo2\x00\x00\x00" would correspond to a Codec
|
||||
// value of 0x8000_0000_326F_646D.
|
||||
//
|
||||
// The Mix Bit is not part of the uint64 representation. Neither is a Long
|
||||
// Codec's 'c64' index. This package's exported API deals with leaf nodes.
|
||||
// Branch nodes' wire formats are internal implementation details.
|
||||
//
|
||||
// See the RAC specification for further discussion.
|
||||
type Codec uint8
|
||||
type Codec uint64
|
||||
|
||||
func (c Codec) isLong() bool { return int64(c) < 0 }
|
||||
func (c Codec) isShort() bool { return int64(c) >= 0 }
|
||||
|
||||
// Valid returns whether c matches the Short or Long pattern.
|
||||
func (c Codec) Valid() bool {
|
||||
if (c >> 63) == 0 {
|
||||
return ((c << 8) == 0) && ((c >> 62) == 0)
|
||||
}
|
||||
return (c >> 56) == 0x80
|
||||
}
|
||||
|
||||
func (c Codec) name() string {
|
||||
if c.isShort() && c.Valid() {
|
||||
switch c >> 56 {
|
||||
case 0:
|
||||
return "Zeroes"
|
||||
case 1:
|
||||
return "Zlib"
|
||||
case 2:
|
||||
return "Brotli"
|
||||
case 4:
|
||||
return "Zstandard"
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func parentChildCodecsValid(parent Codec, child Codec, parentHasMixBit bool) bool {
|
||||
return (parent == child) || parentHasMixBit
|
||||
}
|
||||
|
||||
const (
|
||||
CodecZeroes = Codec(0x00)
|
||||
CodecZlib = Codec(0x01)
|
||||
CodecBrotli = Codec(0x02)
|
||||
CodecZstandard = Codec(0x04)
|
||||
CodecZeroes = Codec(0x00 << 56)
|
||||
CodecZlib = Codec(0x01 << 56)
|
||||
CodecBrotli = Codec(0x02 << 56)
|
||||
CodecZstandard = Codec(0x04 << 56)
|
||||
|
||||
codecMixBit = Codec(1 << 62)
|
||||
codecLongZeroes = Codec(1 << 63)
|
||||
CodecInvalid = Codec((1 << 64) - 1)
|
||||
)
|
||||
|
||||
// IndexLocation is whether the index is at the start or end of the RAC file.
|
||||
|
||||
@@ -594,3 +594,53 @@ func TestReaderZeroes(t *testing.T) {
|
||||
t.Fatalf("got %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLongCodec(t *testing.T) {
|
||||
const codec = Codec(0x80000000326F646D) // "mdo2" backwards, with a high bit.
|
||||
buf := &bytes.Buffer{}
|
||||
w := &ChunkWriter{
|
||||
Writer: buf,
|
||||
IndexLocation: IndexLocationAtStart,
|
||||
TempFile: &bytes.Buffer{},
|
||||
}
|
||||
if err := w.AddChunk(0x66, codec, []byte{0xAA, 0xBB}, 0, 0); err != nil {
|
||||
t.Fatalf("AddChunk: %v", err)
|
||||
}
|
||||
if err := w.AddChunk(0x77, codec, []byte{0xCC}, 0, 0); err != nil {
|
||||
t.Fatalf("AddChunk: %v", err)
|
||||
}
|
||||
if err := w.Close(); err != nil {
|
||||
t.Fatalf("Close: %v", err)
|
||||
}
|
||||
|
||||
encoded := buf.Bytes()
|
||||
gotHexDump := hex.Dump(encoded)
|
||||
|
||||
const wantHexDump = "" +
|
||||
"00000000 72 c3 63 03 3d c9 00 fd 00 00 00 00 00 00 00 ff |r.c.=...........|\n" +
|
||||
"00000010 66 00 00 00 00 00 00 ff dd 00 00 00 00 00 00 80 |f...............|\n" +
|
||||
"00000020 6d 64 6f 32 00 00 00 00 40 00 00 00 00 00 01 ff |mdo2....@.......|\n" +
|
||||
"00000030 42 00 00 00 00 00 01 ff 43 00 00 00 00 00 01 03 |B.......C.......|\n" +
|
||||
"00000040 aa bb cc |...|\n"
|
||||
|
||||
if gotHexDump != wantHexDump {
|
||||
t.Fatalf("\ngot:\n%s\nwant:\n%s", gotHexDump, wantHexDump)
|
||||
}
|
||||
|
||||
r := &ChunkReader{
|
||||
ReadSeeker: bytes.NewReader(encoded),
|
||||
CompressedSize: int64(len(encoded)),
|
||||
}
|
||||
for i := 0; i < 2; i++ {
|
||||
c, err := r.NextChunk()
|
||||
if err != nil {
|
||||
t.Fatalf("i=%d: %v", i, err)
|
||||
}
|
||||
if got, want := c.Codec, codec; got != want {
|
||||
t.Fatalf("i=%d: Codec: got 0x%X, want 0x%X", i, got, want)
|
||||
}
|
||||
}
|
||||
if _, err := r.NextChunk(); err != io.EOF {
|
||||
t.Fatalf("got %v, want %v", err, io.EOF)
|
||||
}
|
||||
}
|
||||
+6
-10
@@ -368,7 +368,7 @@ func (r *Reader) nextChunk() error {
|
||||
return r.err
|
||||
}
|
||||
|
||||
if chunk.Codec == 0 {
|
||||
if (chunk.Codec == CodecZeroes) || (chunk.Codec == codecLongZeroes) {
|
||||
r.currChunk.N = 0
|
||||
r.dRange = chunk.DRange
|
||||
r.zeroes = zeroesReader(r.dRange.Size())
|
||||
@@ -384,16 +384,12 @@ func (r *Reader) nextChunk() error {
|
||||
}
|
||||
}
|
||||
if codecReader == nil {
|
||||
name := ""
|
||||
switch chunk.Codec {
|
||||
case CodecZlib:
|
||||
name = " (Zlib)"
|
||||
case CodecBrotli:
|
||||
name = " (Brotli)"
|
||||
case CodecZstandard:
|
||||
name = " (Zstandard)"
|
||||
name0, name1, name2 := "", chunk.Codec.name(), ""
|
||||
if name1 != "" {
|
||||
name0, name2 = " (", ")"
|
||||
}
|
||||
r.err = fmt.Errorf("rac: no matching CodecReader for Codec 0x%02X%s", chunk.Codec, name)
|
||||
r.err = fmt.Errorf("rac: no matching CodecReader for Codec 0x%X%s%s%s",
|
||||
chunk.Codec, name0, name1, name2)
|
||||
return r.err
|
||||
}
|
||||
|
||||
|
||||
@@ -161,7 +161,7 @@ func TestReaderConcatenation(t *testing.T) {
|
||||
buf[0x37] = 0x00 // CBiasing with COff[0], which is encLen0.
|
||||
|
||||
// Codec and Version.
|
||||
buf[0x1F] = byte(rac.CodecZlib)
|
||||
buf[0x1F] = byte(rac.CodecZlib >> 56)
|
||||
buf[0x3E] = 0x01
|
||||
|
||||
// Checksum.
|
||||
|
||||
Reference in new issue
Block a user