The maximum value is (15*(2^38))-1 bytes, which is almost 1.875 TB.
The maximum size is (15*(2^38))-1 bytes, which is almost 1.875 TB.
`WD` byte is optional. It's not present in `single segment` mode.
In which case, the maximum back-reference distance is the content size itself, which can be any value from 1 to 2^64-1 bytes (16 EB).
To properly decode compressed data,
a decoder will need to allocate a buffer of at least `windowSize` bytes.
Note that `WD` byte is optional. It's not present in `single segment` mode.
In which case, the maximum back-reference distance is the content size itself,
which can be any value from 1 to 2^64-1 bytes (16 EB).
In order to preserve decoder from unreasonable memory requirements,
a decoder can refuse a compressed frame
which requests a memory size beyond decoder's authorized range.
For better interoperability, decoders are recommended to be compatible with window sizes up to 8 MB. Encoders are recommended to not request more than 8 MB. It's just a recommendation, decoders are free to accept or refuse larger or lower values.
For better interoperability,
decoders are recommended to be compatible with window sizes of 8 MB.
Encoders are recommended to not request more than 8 MB.
It's merely a recommendation though,
decoders are free to support larger or lower limits,
depending on local limitations.
__Frame Content Size__
@@ -235,11 +270,11 @@ Format is Little endian.
| 8 | 0 - 2^64-1 |
When field size is 1, 4 or 8 bytes, the value is read directly.
When field size is 2, an offset of 256 is added.
It's possible to represent a small size of`18` using the 8-bytes variant.
A size of `0` means `data size is unknown`.
In which case, the `WD` byte will be the only hint
to determine memory allocation.
When field size is 2, _an offset of 256 is added_.
It's allowed to represent a small size (ex:`18`) using the 8-bytes variant.
A size of `0` means `content size is unknown`.
In which case, the `WD` byte will necessarily be present,
and becomes the only hint to determine memory allocation.
In order to preserve decoder from unreasonable memory requirement,
a decoder can refuse a compressed frame
@@ -257,7 +292,8 @@ Field size depends on __Dictionary ID flag__.
2 bytes can represent an ID 0-65535.
4 bytes can represent an ID 0-(2^32-1).
It's possible to represent a small ID (for example `13`) with a large 4-bytes dictionary ID, losing some efficiency in the process.
It's allowed to represent a small ID (for example `13`)
with a large 4-bytes dictionary ID, losing some efficiency in the process.
Data Blocks
@@ -270,10 +306,10 @@ Data Blocks
__Block Header__
This field uses 3-bytes, format is big-endian.
This field uses 3-bytes, format is __big-endian__.
The 2 highest bits represent the `block type`,
while the remaining 22 bits represent the block size.
while the remaining 22 bits represent the (compressed) block size.
There are 4 block types :
@@ -281,24 +317,24 @@ There are 4 block types :
| ---------- | ---------- | --- | --- | ------- |
| Block Type | Compressed | Raw | RLE | EndMark |
- Compressed : this is a compressed block,
following Zstandard's block format specification.
The "block size" is the compressed size.
- Compressed : this is a Zstandard compressed block,
detailed in a later part of this specification.
"block size" is the compressed size.
Decompressed size is unknown,
but its maximum possible value is guaranteed (see later)
but its maximum possible value is guaranteed (see below)
- Raw : this is an uncompressed block.
"block size" is the number of bytes to read and copy.
- RLE : this is a single byte, repeated N times.
In which case, the size of the "compressed" block is always 1,
and the "block size" is the size to regenerate.
In which case, "block size" is the size to regenerate,
while the "compressed" block is just 1 byte (the byte to repeat).
- EndMark : this is not a block. Signal the end of the frame.
The rest of the field may be optionally filled by a checksum
(see frame checksum).
Block Size shall never be larger than Block Maximum Size.
Block Maximum Size is the smallest of :
- Max back-reference distance
- 128 KB
Block sizes must respect a few rules :
- In compressed mode, compressed size if always strictly `< contentSize`.
- Block decompressed size is necessarily <= maximum back-reference distance .
- Block decompressed size is necessarily <= 128 KB
__Data__
@@ -306,8 +342,8 @@ __Data__
Where the actual data to decode stands.
It might be compressed or not, depending on previous field indications.
A data block is not necessarily "full" :
an arbitrary “flush” may happen anytime. Any block can be “partially filled”.
Therefore, data can have any size, up to Block Maximum Size.
since an arbitrary “flush” may happen anytime,
block content can be any size, up to Block Maximum Size.
Block Maximum Size is the smallest of :
- Max back-reference distance
- 128 KB
@@ -329,25 +365,122 @@ over user-defined data and continue decoding.
Skippable frames defined in this specification are compatible with LZ4 ones.
__Magic Number__
__Magic Number__ :
4 Bytes, Little endian format.
Value : 0x184D2A5X, which means any value from 0x184D2A50 to 0x184D2A5F.
All 16 values are valid to identify a skippable frame.
__Frame Size__
__Frame Size__ :
This is the size, in bytes, of the following User Data
(without including the magic number nor the size field itself).
4 Bytes, Little endian format, unsigned 32-bits.
This means User Data can’t be bigger than (2^32-1) Bytes.
__User Data__
__User Data__ :
User Data can be anything. Data will just be skipped by the decoder.
Compressed block format
-----------------------
This specification details the content of a _compressed block_.
A compressed block has a size, which must be known in order to decode it.
It also has a guaranteed maximum regenerated size,
in order to properly allocate destination buffer.
See "Frame format" for more details.
A compressed block consists of 2 sections :
- Literals section
- Sequences section
### Compressed Literals
Literals are compressed using order-0 huffman compression.
During sequence phase, literals will be entangled with match copy operations.
All literals are regrouped in the first part of the block.
They can be decoded first, and then copied during sequence operations,
or they can be decoded on the flow, as needed by sequences.
Note : it's allowed to represent a short value (ex : `13`)
using a long format, accepting the reduced compacity.
__Sizes format for Compressed Block__ :
Note : also applicable to "repeat-stats" blocks.
- Value : 00 : 4 streams
Compressed and regenerated sizes use 10 bits (0-1023)
Total literal header size is 3 bytes
- Value : 01 : _Single stream_
Compressed and regenerated sizes use 10 bits (0-1023)
Total literal header size is 3 bytes
- Value : 10 : 4 streams
Compressed and regenerated sizes use 14 bits (0-16383)
Total literal header size is 4 bytes
- Value : 10 : 4 streams
Compressed and regenerated sizes use 18 bits (0-262143)
Total literal header size is 5 bytes
Version changes
---------------
0.1 : initial release
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Yann Collet