mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-18 00:32:11 +01:00
Update the RAC Related Work document
This commit is contained in:
1 parent
d48ca528e6
commit
e75fe256b2
1 file changed
+136
-30
+136
-30
@@ -6,58 +6,112 @@ the RAC file format, but gives further context on what RAC is and is not.
|
||||
|
||||
In general, RAC differs from most other archive or compression formats in a
|
||||
number of ways. This doesn't mean that RAC is necessarily better or worse than
|
||||
these other designs, just that different designs make different trade-offs. For
|
||||
example, supporting shared dictionaries means giving up streaming (one pass)
|
||||
decoding.
|
||||
these other designs, just that different designs make different trade-offs.
|
||||
|
||||
- RAC files must start with a magic sequence, but RAC still supports
|
||||
append-only modifications.
|
||||
For example, a key feature is that RAC supports [shared compression
|
||||
dictionaries](http://fastcompression.blogspot.com/2018/02/when-to-use-dictionary-compression.html),
|
||||
embedded within the archive file. This can claw back some of the compression
|
||||
ratio lost when splitting a source file into independently compressed chunks.
|
||||
The trade-off is that decompressing a RAC file therefore requires random access
|
||||
to the compressed file, not just sequential access. Unlike some other formats
|
||||
discussed below, RAC cannot do streaming (one pass) decoding. Decompression
|
||||
might not be able to start until the entire compressed file is available: the
|
||||
root node of the index might be at the end of the compressed file.
|
||||
|
||||
Similarly, unlike Gzip or XZ, concatenating two RAC files together does not
|
||||
produce a valid RAC file. A new root node also has to be appended afterwards,
|
||||
which is cheap and easy, but not a no-op. Nonetheless, that new root node
|
||||
allows for incremental, append-only modification of a RAC file: removing,
|
||||
replacing or inserting bytes in the middle of the decompressed form by only
|
||||
appending to the compressed form.
|
||||
|
||||
A few more differences between RAC and most if not all of the other file
|
||||
formats or algorithms discussed further below:
|
||||
|
||||
- RAC itself isn't tied to any one underlying compression codec. Obviously,
|
||||
"RAC + Zlib" is tied to Zlib, and "RAC + Brotli" is tied to Brotli, but if
|
||||
somebody invents a new "MiddleOut" compression codec, one that wasn't
|
||||
random access per se, it would be easy to have "RAC + MiddleOut".
|
||||
- RAC seek points are identified by numbers (i.e. `DOffset`s), not strings
|
||||
(i.e. file names). Some other formats' seek points are positioned only at
|
||||
source file boundaries. They cannot seek to a relative offset (in what RAC
|
||||
calls `DSpace`) within a source file.
|
||||
- RAC supports [shared compression
|
||||
dictionaries](http://fastcompression.blogspot.com/2018/02/when-to-use-dictionary-compression.html)
|
||||
embedded within an archive.
|
||||
source file boundaries, as they're primarily used to compress part of a
|
||||
file system, containing directories and (variable length) files. They
|
||||
cannot seek to a relative offset (in what RAC calls `DSpace`) within a
|
||||
source file.
|
||||
|
||||
These points apply to established archive formats like
|
||||
These differences apply to established archive formats like
|
||||
[Tar](https://en.wikipedia.org/wiki/Tar_%28computing%29) and
|
||||
[Zip](https://en.wikipedia.org/wiki/Zip_%28file_format%29), and newer archive
|
||||
formats like [Śiva](https://blog.sourced.tech/post/siva/).
|
||||
|
||||
|
||||
## Software Products
|
||||
|
||||
[XZ](https://tukaani.org/xz/format.html) supports random access and multiple
|
||||
underlying compression codecs like RAC. Out of all the formats listed here, it
|
||||
is the one most similar to RAC, but as one of XZ's explicit goals is to support
|
||||
streaming decoding, XZ does not support shared dictionaries or incremental
|
||||
modification, as discussed above. Also, unlike RAC, finding the record that
|
||||
contains any given `DOffset` requires a linear scan of all previous records,
|
||||
even if those records don't need decompressing.
|
||||
|
||||
[DictZip](http://linuxcommand.org/man_pages/dictzip1.html) supports random
|
||||
access like RAC, but is tied to the Gzip compression codec (every valid DictZip
|
||||
file is also a valid Gzip file), and due to size restrictions in the Gzip
|
||||
extension format, a DictZip file's decompressed size cannot exceed about 1.8
|
||||
GiB. [CSIO](https://github.com/hoxnox/csio/blob/master/include/csio.h) seems to
|
||||
do something similar. [Idzip](https://github.com/fidlej/idzip) avoids the 1.8
|
||||
GiB limitation on the overall size (although retaining the 64 KiB limit on
|
||||
individual chunks) by not requiring the index to fit inside a single Gzip
|
||||
extension block.
|
||||
|
||||
[BGZF](http://samtools.github.io/hts-specs/SAMv1.pdf) is very similar to
|
||||
DictZip, but also avoids the 1.8 GiB limitation. It places the uncompressed
|
||||
size of each chunk together with each chunk, instead of having a single
|
||||
contiguous index separate from chunk data. Unless the index is also [stored
|
||||
externally](https://github.com/samtools/htslib/blob/develop/bgzip.1), random
|
||||
access is achieved through following a linked list of size offsets. Some more
|
||||
discussion is on [Heng Li's
|
||||
blog](http://lh3.github.io/2014/07/05/random-access-to-zlib-compressed-files).
|
||||
|
||||
[Xflate](https://github.com/dsnet/compress/blob/master/doc/xflate-format.pdf)
|
||||
is similar to BGZF, but extends the Deflate format instead of Gzip (every valid
|
||||
Xflate file is also a valid Deflate file). Gzip extends Deflate with metadata
|
||||
(where BGZF stashes its index information) and a checksum. Deflate has no
|
||||
capacity for metadata per se, but Xflate exploits the fact that there are many
|
||||
ways to encode an empty Deflate block (empty meaning that decoding it produces
|
||||
zero bytes).
|
||||
|
||||
[Bzip2](https://sourceware.org/bzip2/) and
|
||||
[RAZip](https://sourceforge.net/projects/razip/) are other formats based around
|
||||
the same idea of partitioning the source file into independently compressed
|
||||
chunks, and for each chunk, recording both its compressed and uncompressed
|
||||
size. Once again, random access is achieved through following a linked list of
|
||||
size offsets.
|
||||
|
||||
[Riegeli](https://github.com/google/riegeli) is a sequence-of-records format. A
|
||||
RAC file arguably contains what Riegeli calls records (which are not files, as
|
||||
they don't have names) and both formats support numerical seek points, and
|
||||
multiple compression codecs, but Riegeli seeks to a point in what RAC calls
|
||||
`CSpace`, whereas RAC seeks to a point in `DSpace`.
|
||||
|
||||
[XFLATE](https://github.com/dsnet/compress/blob/master/doc/xflate-format.pdf)
|
||||
supports random access like RAC, but is tied to the DEFLATE family of
|
||||
compression codecs.
|
||||
|
||||
[XZ](https://tukaani.org/xz/format.html) supports random access like RAC, but
|
||||
as one of its explicit goals is to support streaming decoding, it does not
|
||||
support shared dictionaries. Also, unlike RAC, finding the record that contains
|
||||
any given `DOffset` requires a linear scan of all previous records, even if
|
||||
those records don't need decompressing.
|
||||
|
||||
[QCOW](https://github.com/qemu/qemu/blob/master/docs/interop/qcow2.txt)
|
||||
supports random access like RAC, but compression does not seem to be the
|
||||
foremost concern. That specification mentions compression but does not give any
|
||||
particular codec. A [separate
|
||||
document](https://people.gnome.org/~markmc/qcow-image-format.html) suggests
|
||||
that the codec is Zlib (with no other option), and that it does not support
|
||||
shared dictionaries.
|
||||
that the codec is Zlib (with no other option). Like other RAC-related work,
|
||||
QCOW does not support shared dictionaries.
|
||||
|
||||
The
|
||||
[`examples/zran.c`](https://github.com/madler/zlib/blob/master/examples/zran.c)
|
||||
program in the zlib-the-library source code repository builds an in-memory
|
||||
index, not a persistent (disk-backed) one, and building the index involves
|
||||
first decompressing the whole file. It also requires storing (in memory) the
|
||||
entire state of the decompressor, including 32 KiB of history, per seek point.
|
||||
For coarse grained seek points (e.g. once every 1 MiB), that overhead can be
|
||||
acceptable, but for fine grained seek points (e.g. once every 16 KiB), that
|
||||
overhead is prohibitive.
|
||||
program in the zlib-the-library source code repository (and e.g. the jzran Java
|
||||
language port) builds an in-memory index, not a persistent (disk-backed) one,
|
||||
and building the index involves first decompressing the whole file. It also
|
||||
requires storing (in memory) the entire state of the decompressor, including 32
|
||||
KiB of history, per seek point. For coarse grained seek points (e.g. once every
|
||||
1 MiB), that overhead can be acceptable, but for fine grained seek points (e.g.
|
||||
once every 16 KiB), that overhead is prohibitive.
|
||||
|
||||
RAC differs from the [LevelDB
|
||||
Table](https://github.com/google/leveldb/blob/master/doc/table_format.md)
|
||||
@@ -74,3 +128,55 @@ exactly one `Leaf Node` contains any given byte offset `di` in the decompressed
|
||||
file, and the RAC index's hierarchical nature places a scalable upper bound on
|
||||
the scanning cost of verifying that, based on that portion of the index tree
|
||||
visited for any given request `[di .. dj)`.
|
||||
|
||||
[zfp](https://computing.llnl.gov/projects/floating-point-compression) primarily
|
||||
concerns lossy compression of spatial floating point data.
|
||||
|
||||
|
||||
## Academic Publications
|
||||
|
||||
Kerbiriou and Chikhi's ["Parallel decompression of gzip-compressed files and
|
||||
random access to DNA sequences"](https://arxiv.org/pdf/1905.07224.pdf) does not
|
||||
modify the Gzip / Deflate file format. Seek points are located in `CSpace`, not
|
||||
`DSpace`. It uses a heuristic approach to determine block boundaries near an
|
||||
arbitrary seek point, which is susceptible to false positive matches. The
|
||||
primary use case is biometric data in the FASTQ file format, limited to ASCII.
|
||||
Reconstruction can be lossy: "The method is near-exact at low compression
|
||||
levels, but often a large fraction of sequences contains undetermined
|
||||
characters at higher compression levels".
|
||||
|
||||
Kreft and Navarro's ["LZ77-like Compression with Fast Random
|
||||
Access"](https://users.dcc.uchile.cl/~gnavarro/ps/dcc10.1.pdf) constrains the
|
||||
more general Lempel-Ziv approach so that back-reference spans always end at
|
||||
phrase boundaries. Combining with an efficient data structure for phrase
|
||||
boundaries, a bitmap with rank and select operations, allows for backwards
|
||||
reconstruction (starting from a phrase endpoint). However, in terms of
|
||||
compression ratios, "Our least-space variants take [2.5–4.0 times the space of
|
||||
p7zip](https://users.dcc.uchile.cl/~gnavarro/ps/tcs12.pdf), and 1.8–2.5 times
|
||||
the space of lz77".
|
||||
|
||||
Fredriksson and Nikitin's ["Simple Random Access
|
||||
Compression"](http://www.cs.uku.fi/~fredriks/pub/papers/fi09.pdf) provides
|
||||
random access to short substrings (as short as a single symbol) of the
|
||||
decompressed file, focusing on natural language source text (where a symbol
|
||||
corresponds to e.g. an English language word). It is unclear how well their
|
||||
technique performs on decompressing large substrings of binary data. For
|
||||
example, while their reported compression ratios appear competitive with gzip
|
||||
for an English text file, they are substantially worse for XML data.
|
||||
|
||||
Zhang and Bockelman's ["Exploring compression techniques for ROOT
|
||||
IO"](https://arxiv.org/pdf/1704.06976.pdf) has a section literally titled
|
||||
"Random Access Compression", which appears to apply to structured floating
|
||||
point data (from the High Energy Physics domain), and the reported compression
|
||||
ratios are dramatically worse (2.88x) with what they call RAC than without.
|
||||
This document discusses lossless compression of arbitrary binary data, with a
|
||||
small (e.g. 1.02x overhead).
|
||||
|
||||
Rodler's ["Compression with Fast Random
|
||||
Access"](https://www.brics.dk/DS/01/9/BRICS-DS-01-9.pdf) sounds relevant based
|
||||
on its title, but primarily concerns lossy compression of volumetric data.
|
||||
|
||||
Robert and Nadarajan's ["New Algorithms For Random Access Text
|
||||
Compression"](https://www.researchgate.net/publication/4231766_New_Algorithms_For_Random_Access_Text_Compression)
|
||||
cannot compress aribrary data, only 7-bit ASCII. The largest document they
|
||||
consider is also only 17 KiB in size.
|
||||
Reference in new issue
Block a user