ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Move seekable format content to /contrib

Browse Source
This commit is contained in:
Sean Purcell
2017-04-11 14:38:56 -07:00
parent b13da709e8
commit d048fefef7
15 changed files with 186 additions and 266 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/README.md
-3
View File
@@ -17,6 +17,3 @@ __`zstd_manual.html`__ : Documentation on the functions found in `zstd.h`.
See [http://zstd.net/zstd_manual.html](http://zstd.net/zstd_manual.html) for
the manual released with the latest official `zstd` release.
__`zstd_seekable_compression_format.md`__ : This document defines the Zstandard
format for seekable compression.
doc/zstd_manual.html
-109
View File
@@ -28,9 +28,6 @@
<li><a href="#Chapter18">Buffer-less streaming compression (synchronous mode)</a></li>
<li><a href="#Chapter19">Buffer-less streaming decompression (synchronous mode)</a></li>
<li><a href="#Chapter20">Block functions</a></li>
<li><a href="#Chapter21">Seekable Format</a></li>
<li><a href="#Chapter22">Seekable compression - HowTo</a></li>
<li><a href="#Chapter23">Seekable decompression - HowTo</a></li>
</ol>
<hr>
<a name="Chapter1"></a><h2>Introduction</h2><pre>
@@ -671,111 +668,5 @@ size_t ZSTD_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, cons
size_t ZSTD_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
size_t ZSTD_insertBlock(ZSTD_DCtx* dctx, const void* blockStart, size_t blockSize); </b>/**< insert block into `dctx` history. Useful for uncompressed blocks */<b>
</pre></b><BR>
<a name="Chapter21"></a><h2>Seekable Format</h2><pre>
The seekable format splits the compressed data into a series of "chunks",
each compressed individually so that decompression of a section in the
middle of an archive only requires zstd to decompress at most a chunk's
worth of extra data, instead of the entire archive.
<BR></pre>
<a name="Chapter22"></a><h2>Seekable compression - HowTo</h2><pre> A ZSTD_seekable_CStream object is required to tracking streaming operation.
Use ZSTD_seekable_createCStream() and ZSTD_seekable_freeCStream() to create/
release resources.
Streaming objects are reusable to avoid allocation and deallocation,
to start a new compression operation call ZSTD_seekable_initCStream() on the
compressor.
Data streamed to the seekable compressor will automatically be split into
chunks of size `maxChunkSize` (provided in ZSTD_seekable_initCStream()),
or if none is provided, will be cut off whenver ZSTD_endChunk() is called
or when the default maximum chunk size is reached (approximately 4GB).
Use ZSTD_seekable_initCStream() to initialize a ZSTD_seekable_CStream object
for a new compression operation.
`maxChunkSize` indicates the size at which to automatically start a new
seekable frame. `maxChunkSize == 0` implies the default maximum size.
`checksumFlag` indicates whether or not the seek table should include chunk
checksums on the uncompressed data for verification.
@return : a size hint for input to provide for compression, or an error code
checkable with ZSTD_isError()
Use ZSTD_seekable_compressStream() repetitively to consume input stream.
The function will automatically update both `pos` fields.
Note that it may not consume the entire input, in which case `pos < size`,
and it's up to the caller to present again remaining data.
@return : a size hint, preferred nb of bytes to use as input for next
function call or an error code, which can be tested using
ZSTD_isError().
Note 1 : it's just a hint, to help latency a little, any other
value will work fine.
Note 2 : size hint is guaranteed to be <= ZSTD_CStreamInSize()
At any time, call ZSTD_seekable_endChunk() to end the current chunk and
start a new one.
ZSTD_endStream() will end the current chunk, and then write the seek table
so that decompressors can efficiently find compressed chunks.
ZSTD_endStream() may return a number > 0 if it was unable to flush all the
necessary data to `output`. In this case, it should be called again until
all remaining data is flushed out and 0 is returned.
<BR></pre>
<h3>Seekable compressor management</h3><pre></pre><b><pre>ZSTD_seekable_CStream* ZSTD_seekable_createCStream(void);
size_t ZSTD_seekable_freeCStream(ZSTD_seekable_CStream* zcs);
</pre></b><BR>
<h3>Seekable compression functions</h3><pre></pre><b><pre>size_t ZSTD_seekable_initCStream(ZSTD_seekable_CStream* zcs, int compressionLevel, int checksumFlag, unsigned maxChunkSize);
size_t ZSTD_seekable_compressStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
size_t ZSTD_seekable_endChunk(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output);
size_t ZSTD_seekable_endStream(ZSTD_seekable_CStream* zcs, ZSTD_outBuffer* output);
</pre></b><BR>
<a name="Chapter23"></a><h2>Seekable decompression - HowTo</h2><pre> A ZSTD_seekable_DStream object is required to tracking streaming operation.
Use ZSTD_seekable_createDStream() and ZSTD_seekable_freeDStream() to create/
release resources.
Streaming objects are reusable to avoid allocation and deallocation,
to start a new compression operation call ZSTD_seekable_initDStream() on the
compressor.
Use ZSTD_seekable_loadSeekTable() to load the seek table from a file.
`src` should point to a block of data read from the end of the file,
i.e. `src + srcSize` should always be the end of the file.
@return : 0 if the table was loaded successfully, or if `srcSize` was too
small, a size hint for how much data to provide.
An error code may also be returned, checkable with ZSTD_isError()
Use ZSTD_initDStream to prepare for a new decompression operation using the
seektable loaded with ZSTD_seekable_loadSeekTable().
Data in the range [rangeStart, rangeEnd) will be decompressed.
Call ZSTD_seekable_decompressStream() repetitively to consume input stream.
@return : There are a number of possible return codes for this function
- 0, the decompression operation has completed.
- An error code checkable with ZSTD_isError
+ If this error code is ZSTD_error_needSeek, the user should seek
to the file position provided by ZSTD_seekable_getSeekOffset()
and indicate this to the stream with
ZSTD_seekable_updateOffset(), before resuming decompression
+ Otherwise, this is a regular decompression error and the input
file is likely corrupted or the API was incorrectly used.
- A size hint, the preferred nb of bytes to provide as input to the
next function call to improve latency.
ZSTD_seekable_getSeekOffset() and ZSTD_seekable_updateOffset() are helper
functions to indicate where the user should seek their file stream to, when
a different position is required to continue decompression.
Note that ZSTD_seekable_updateOffset will error if given an offset other
than the one requested from ZSTD_seekable_getSeekOffset().
<BR></pre>
<h3>Seekable decompressor management</h3><pre></pre><b><pre>ZSTD_seekable_DStream* ZSTD_seekable_createDStream(void);
size_t ZSTD_seekable_freeDStream(ZSTD_seekable_DStream* zds);
</pre></b><BR>
<h3>Seekable decompression functions</h3><pre></pre><b><pre>size_t ZSTD_seekable_loadSeekTable(ZSTD_seekable_DStream* zds, const void* src, size_t srcSize);
size_t ZSTD_seekable_initDStream(ZSTD_seekable_DStream* zds, unsigned long long rangeStart, unsigned long long rangeEnd);
size_t ZSTD_seekable_decompressStream(ZSTD_seekable_DStream* zds, ZSTD_outBuffer* output, ZSTD_inBuffer* input);
unsigned long long ZSTD_seekable_getSeekOffset(ZSTD_seekable_DStream* zds);
size_t ZSTD_seekable_updateOffset(ZSTD_seekable_DStream* zds, unsigned long long offset);
</pre></b><BR>
</html>
</body>
doc/zstd_seekable_compression_format.md
-107
View File
@@ -1,107 +0,0 @@
# Zstandard Seekable Format
### Notices
Copyright (c) 2017-present Facebook, Inc.
Permission is granted to copy and distribute this document
for any purpose and without charge,
including translations into other languages
and incorporation into compilations,
provided that the copyright notice and this notice are preserved,
and that any substantive changes or deletions from the original
are clearly marked.
Distribution of this document is unlimited.
### Version
0.1.0 (11/04/17)
## Introduction
This document defines a format for compressed data to be stored so that subranges of the data can be efficiently decompressed without requiring the entire document to be decompressed.
This is done by splitting up the input data into chunks,
each of which are compressed independently,
and so can be decompressed independently.
Decompression then takes advantage of a provided 'seek table', which allows the decompressor to immediately jump to the desired data. This is done in a way that is compatible with the original Zstandard format by placing the seek table in a Zstandard skippable frame.
### Overall conventions
In this document:
- square brackets i.e. `[` and `]` are used to indicate optional fields or parameters.
- the naming convention for identifiers is `Mixed_Case_With_Underscores`
- All numeric fields are little-endian unless specified otherwise
## Format
The format consists of a number of chunks (Zstandard compressed frames and skippable frames), followed by a final skippable frame at the end containing the seek table.
### Seek Table Format
The structure of the seek table frame is as follows:
|`Skippable_Magic_Number`|`Frame_Size`|`[Seek_Table_Entries]`|`Seek_Table_Footer`|
|------------------------|------------|----------------------|-------------------|
| 4 bytes | 4 bytes | 8-12 bytes each | 9 bytes |
__`Skippable_Magic_Number`__
Value : 0x184D2A5?, which means any value from 0x184D2A50 to 0x184D2A5F.
All 16 values are valid to identify a skippable frame.
This is for compatibility with [Zstandard skippable frames].
__`Frame_Size`__
The total size of the skippable frame, not including the `Skippable_Magic_Number` or `Frame_Size`. This is for compatibility with [Zstandard skippable frames].
[Zstandard skippable frames]: https://github.com/facebook/zstd/blob/master/doc/zstd_compression_format.md#skippable-frames
#### `Seek_Table_Footer`
The seek table footer format is as follows:
|`Number_Of_Chunks`|`Seek_Table_Descriptor`|`Seekable_Magic_Number`|
|------------------|-----------------------|-----------------------|
| 4 bytes | 1 byte | 4 bytes |
__`Number_Of_Chunks`__
The number of stored chunks in the data.
__`Seek_Table_Descriptor`__
A bitfield describing the format of the seek table.
| Bit number | Field name |
| ---------- | ---------- |
| 7 | `Checksum_Flag` |
| 6-2 | `Reserved_Bits` |
| 1-0 | `Unused_Bits` |
While only `Checksum_Flag` currently exists, there are 7 other bits in this field that can be used for future changes to the format,
for example the addition of inline dictionaries.
__`Checksum_Flag`__
If the checksum flag is set, each of the seek table entries contains a 4 byte checksum of the uncompressed data contained in its chunk.
`Reserved_Bits` are not currently used but may be used in the future for breaking changes, so a compliant decoder should ensure they are set to 0. `Unused_Bits` may be used in the future for non-breaking changes, so a compliant decoder should not interpret these bits.
#### __`Seek_Table_Entries`__
`Seek_Table_Entries` consists of `Number_Of_Chunks` (one for each chunk in the data, not including the seek table frame) entries of the following form, in sequence:
|`Compressed_Size`|`Decompressed_Size`|`[Checksum]`|
|-----------------|-------------------|------------|
| 4 bytes | 4 bytes | 4 bytes |
__`Compressed_Size`__
The compressed size of the chunk.
The cumulative sum of the `Compressed_Size` fields of chunks `0` to `i` gives the offset in the compressed file of chunk `i+1`.
__`Decompressed_Size`__
The size of the decompressed data contained in the chunk. For skippable or otherwise empty frames, this value is 0.
__`Checksum`__
Only present if `Checksum_Flag` is set in the `Seek_Table_Descriptor`. Value : the least significant 32 bits of the XXH64 digest of the uncompressed data, stored in little-endian format.
## Version Changes
- 0.1.0: initial version