From e75fe256b23f0541cb1d0a4497ec044bffb13b9f Mon Sep 17 00:00:00 2001 From: Nigel Tao Date: Sun, 25 Aug 2019 21:48:13 +1000 Subject: [PATCH] Update the RAC Related Work document --- doc/spec/rac-related-work.md | 166 ++++++++++++++++++++++++++++------- 1 file changed, 136 insertions(+), 30 deletions(-) diff --git a/doc/spec/rac-related-work.md b/doc/spec/rac-related-work.md index 8635b516..26d92b83 100644 --- a/doc/spec/rac-related-work.md +++ b/doc/spec/rac-related-work.md @@ -6,58 +6,112 @@ the RAC file format, but gives further context on what RAC is and is not. In general, RAC differs from most other archive or compression formats in a number of ways. This doesn't mean that RAC is necessarily better or worse than -these other designs, just that different designs make different trade-offs. For -example, supporting shared dictionaries means giving up streaming (one pass) -decoding. +these other designs, just that different designs make different trade-offs. - - RAC files must start with a magic sequence, but RAC still supports - append-only modifications. +For example, a key feature is that RAC supports [shared compression +dictionaries](http://fastcompression.blogspot.com/2018/02/when-to-use-dictionary-compression.html), +embedded within the archive file. This can claw back some of the compression +ratio lost when splitting a source file into independently compressed chunks. +The trade-off is that decompressing a RAC file therefore requires random access +to the compressed file, not just sequential access. Unlike some other formats +discussed below, RAC cannot do streaming (one pass) decoding. Decompression +might not be able to start until the entire compressed file is available: the +root node of the index might be at the end of the compressed file. + +Similarly, unlike Gzip or XZ, concatenating two RAC files together does not +produce a valid RAC file. A new root node also has to be appended afterwards, +which is cheap and easy, but not a no-op. Nonetheless, that new root node +allows for incremental, append-only modification of a RAC file: removing, +replacing or inserting bytes in the middle of the decompressed form by only +appending to the compressed form. + +A few more differences between RAC and most if not all of the other file +formats or algorithms discussed further below: + + - RAC itself isn't tied to any one underlying compression codec. Obviously, + "RAC + Zlib" is tied to Zlib, and "RAC + Brotli" is tied to Brotli, but if + somebody invents a new "MiddleOut" compression codec, one that wasn't + random access per se, it would be easy to have "RAC + MiddleOut". - RAC seek points are identified by numbers (i.e. `DOffset`s), not strings (i.e. file names). Some other formats' seek points are positioned only at - source file boundaries. They cannot seek to a relative offset (in what RAC - calls `DSpace`) within a source file. - - RAC supports [shared compression - dictionaries](http://fastcompression.blogspot.com/2018/02/when-to-use-dictionary-compression.html) - embedded within an archive. + source file boundaries, as they're primarily used to compress part of a + file system, containing directories and (variable length) files. They + cannot seek to a relative offset (in what RAC calls `DSpace`) within a + source file. -These points apply to established archive formats like +These differences apply to established archive formats like [Tar](https://en.wikipedia.org/wiki/Tar_%28computing%29) and [Zip](https://en.wikipedia.org/wiki/Zip_%28file_format%29), and newer archive formats like [Śiva](https://blog.sourced.tech/post/siva/). + +## Software Products + +[XZ](https://tukaani.org/xz/format.html) supports random access and multiple +underlying compression codecs like RAC. Out of all the formats listed here, it +is the one most similar to RAC, but as one of XZ's explicit goals is to support +streaming decoding, XZ does not support shared dictionaries or incremental +modification, as discussed above. Also, unlike RAC, finding the record that +contains any given `DOffset` requires a linear scan of all previous records, +even if those records don't need decompressing. + +[DictZip](http://linuxcommand.org/man_pages/dictzip1.html) supports random +access like RAC, but is tied to the Gzip compression codec (every valid DictZip +file is also a valid Gzip file), and due to size restrictions in the Gzip +extension format, a DictZip file's decompressed size cannot exceed about 1.8 +GiB. [CSIO](https://github.com/hoxnox/csio/blob/master/include/csio.h) seems to +do something similar. [Idzip](https://github.com/fidlej/idzip) avoids the 1.8 +GiB limitation on the overall size (although retaining the 64 KiB limit on +individual chunks) by not requiring the index to fit inside a single Gzip +extension block. + +[BGZF](http://samtools.github.io/hts-specs/SAMv1.pdf) is very similar to +DictZip, but also avoids the 1.8 GiB limitation. It places the uncompressed +size of each chunk together with each chunk, instead of having a single +contiguous index separate from chunk data. Unless the index is also [stored +externally](https://github.com/samtools/htslib/blob/develop/bgzip.1), random +access is achieved through following a linked list of size offsets. Some more +discussion is on [Heng Li's +blog](http://lh3.github.io/2014/07/05/random-access-to-zlib-compressed-files). + +[Xflate](https://github.com/dsnet/compress/blob/master/doc/xflate-format.pdf) +is similar to BGZF, but extends the Deflate format instead of Gzip (every valid +Xflate file is also a valid Deflate file). Gzip extends Deflate with metadata +(where BGZF stashes its index information) and a checksum. Deflate has no +capacity for metadata per se, but Xflate exploits the fact that there are many +ways to encode an empty Deflate block (empty meaning that decoding it produces +zero bytes). + +[Bzip2](https://sourceware.org/bzip2/) and +[RAZip](https://sourceforge.net/projects/razip/) are other formats based around +the same idea of partitioning the source file into independently compressed +chunks, and for each chunk, recording both its compressed and uncompressed +size. Once again, random access is achieved through following a linked list of +size offsets. + [Riegeli](https://github.com/google/riegeli) is a sequence-of-records format. A RAC file arguably contains what Riegeli calls records (which are not files, as they don't have names) and both formats support numerical seek points, and multiple compression codecs, but Riegeli seeks to a point in what RAC calls `CSpace`, whereas RAC seeks to a point in `DSpace`. -[XFLATE](https://github.com/dsnet/compress/blob/master/doc/xflate-format.pdf) -supports random access like RAC, but is tied to the DEFLATE family of -compression codecs. - -[XZ](https://tukaani.org/xz/format.html) supports random access like RAC, but -as one of its explicit goals is to support streaming decoding, it does not -support shared dictionaries. Also, unlike RAC, finding the record that contains -any given `DOffset` requires a linear scan of all previous records, even if -those records don't need decompressing. - [QCOW](https://github.com/qemu/qemu/blob/master/docs/interop/qcow2.txt) supports random access like RAC, but compression does not seem to be the foremost concern. That specification mentions compression but does not give any particular codec. A [separate document](https://people.gnome.org/~markmc/qcow-image-format.html) suggests -that the codec is Zlib (with no other option), and that it does not support -shared dictionaries. +that the codec is Zlib (with no other option). Like other RAC-related work, +QCOW does not support shared dictionaries. The [`examples/zran.c`](https://github.com/madler/zlib/blob/master/examples/zran.c) -program in the zlib-the-library source code repository builds an in-memory -index, not a persistent (disk-backed) one, and building the index involves -first decompressing the whole file. It also requires storing (in memory) the -entire state of the decompressor, including 32 KiB of history, per seek point. -For coarse grained seek points (e.g. once every 1 MiB), that overhead can be -acceptable, but for fine grained seek points (e.g. once every 16 KiB), that -overhead is prohibitive. +program in the zlib-the-library source code repository (and e.g. the jzran Java +language port) builds an in-memory index, not a persistent (disk-backed) one, +and building the index involves first decompressing the whole file. It also +requires storing (in memory) the entire state of the decompressor, including 32 +KiB of history, per seek point. For coarse grained seek points (e.g. once every +1 MiB), that overhead can be acceptable, but for fine grained seek points (e.g. +once every 16 KiB), that overhead is prohibitive. RAC differs from the [LevelDB Table](https://github.com/google/leveldb/blob/master/doc/table_format.md) @@ -74,3 +128,55 @@ exactly one `Leaf Node` contains any given byte offset `di` in the decompressed file, and the RAC index's hierarchical nature places a scalable upper bound on the scanning cost of verifying that, based on that portion of the index tree visited for any given request `[di .. dj)`. + +[zfp](https://computing.llnl.gov/projects/floating-point-compression) primarily +concerns lossy compression of spatial floating point data. + + +## Academic Publications + +Kerbiriou and Chikhi's ["Parallel decompression of gzip-compressed files and +random access to DNA sequences"](https://arxiv.org/pdf/1905.07224.pdf) does not +modify the Gzip / Deflate file format. Seek points are located in `CSpace`, not +`DSpace`. It uses a heuristic approach to determine block boundaries near an +arbitrary seek point, which is susceptible to false positive matches. The +primary use case is biometric data in the FASTQ file format, limited to ASCII. +Reconstruction can be lossy: "The method is near-exact at low compression +levels, but often a large fraction of sequences contains undetermined +characters at higher compression levels". + +Kreft and Navarro's ["LZ77-like Compression with Fast Random +Access"](https://users.dcc.uchile.cl/~gnavarro/ps/dcc10.1.pdf) constrains the +more general Lempel-Ziv approach so that back-reference spans always end at +phrase boundaries. Combining with an efficient data structure for phrase +boundaries, a bitmap with rank and select operations, allows for backwards +reconstruction (starting from a phrase endpoint). However, in terms of +compression ratios, "Our least-space variants take [2.5–4.0 times the space of +p7zip](https://users.dcc.uchile.cl/~gnavarro/ps/tcs12.pdf), and 1.8–2.5 times +the space of lz77". + +Fredriksson and Nikitin's ["Simple Random Access +Compression"](http://www.cs.uku.fi/~fredriks/pub/papers/fi09.pdf) provides +random access to short substrings (as short as a single symbol) of the +decompressed file, focusing on natural language source text (where a symbol +corresponds to e.g. an English language word). It is unclear how well their +technique performs on decompressing large substrings of binary data. For +example, while their reported compression ratios appear competitive with gzip +for an English text file, they are substantially worse for XML data. + +Zhang and Bockelman's ["Exploring compression techniques for ROOT +IO"](https://arxiv.org/pdf/1704.06976.pdf) has a section literally titled +"Random Access Compression", which appears to apply to structured floating +point data (from the High Energy Physics domain), and the reported compression +ratios are dramatically worse (2.88x) with what they call RAC than without. +This document discusses lossless compression of arbitrary binary data, with a +small (e.g. 1.02x overhead). + +Rodler's ["Compression with Fast Random +Access"](https://www.brics.dk/DS/01/9/BRICS-DS-01-9.pdf) sounds relevant based +on its title, but primarily concerns lossy compression of volumetric data. + +Robert and Nadarajan's ["New Algorithms For Random Access Text +Compression"](https://www.researchgate.net/publication/4231766_New_Algorithms_For_Random_Access_Text_Compression) +cannot compress aribrary data, only 7-bit ASCII. The largest document they +consider is also only 17 KiB in size.