The maximum value is (15*(2^38))-1 bytes, which is almost 1.875 TB.
The maximum size is (15*(2^38))-1 bytes, which is almost 1.875 TB.
`WD` byte is optional. It's not present in `single segment` mode.
To properly decode compressed data,
In which case, the maximum back-reference distance is the content size itself, which can be any value from 1 to 2^64-1 bytes (16 EB).
a decoder will need to allocate a buffer of at least `windowSize` bytes.
Note that `WD` byte is optional. It's not present in `single segment` mode.
In which case, the maximum back-reference distance is the content size itself,
which can be any value from 1 to 2^64-1 bytes (16 EB).
In order to preserve decoder from unreasonable memory requirements,
In order to preserve decoder from unreasonable memory requirements,
a decoder can refuse a compressed frame
a decoder can refuse a compressed frame
which requests a memory size beyond decoder's authorized range.
which requests a memory size beyond decoder's authorized range.
For better interoperability, decoders are recommended to be compatible with window sizes up to 8 MB. Encoders are recommended to not request more than 8 MB. It's just a recommendation, decoders are free to accept or refuse larger or lower values.
For better interoperability,
decoders are recommended to be compatible with window sizes of 8 MB.
Encoders are recommended to not request more than 8 MB.
It's merely a recommendation though,
decoders are free to support larger or lower limits,
depending on local limitations.
__Frame Content Size__
__Frame Content Size__
@@ -235,11 +270,11 @@ Format is Little endian.
| 8 | 0 - 2^64-1 |
| 8 | 0 - 2^64-1 |
When field size is 1, 4 or 8 bytes, the value is read directly.
When field size is 1, 4 or 8 bytes, the value is read directly.
When field size is 2, an offset of 256 is added.
When field size is 2, _an offset of 256 is added_.
It's possible to represent a small size of`18` using the 8-bytes variant.
It's allowed to represent a small size (ex:`18`) using the 8-bytes variant.
A size of `0` means `data size is unknown`.
A size of `0` means `content size is unknown`.
In which case, the `WD` byte will be the only hint
In which case, the `WD` byte will necessarily be present,
to determine memory allocation.
and becomes the only hint to determine memory allocation.
In order to preserve decoder from unreasonable memory requirement,
In order to preserve decoder from unreasonable memory requirement,
a decoder can refuse a compressed frame
a decoder can refuse a compressed frame
@@ -257,7 +292,8 @@ Field size depends on __Dictionary ID flag__.
2 bytes can represent an ID 0-65535.
2 bytes can represent an ID 0-65535.
4 bytes can represent an ID 0-(2^32-1).
4 bytes can represent an ID 0-(2^32-1).
It's possible to represent a small ID (for example `13`) with a large 4-bytes dictionary ID, losing some efficiency in the process.
It's allowed to represent a small ID (for example `13`)
with a large 4-bytes dictionary ID, losing some efficiency in the process.
Data Blocks
Data Blocks
@@ -270,10 +306,10 @@ Data Blocks
__Block Header__
__Block Header__
This field uses 3-bytes, format is big-endian.
This field uses 3-bytes, format is __big-endian__.
The 2 highest bits represent the `block type`,
The 2 highest bits represent the `block type`,
while the remaining 22 bits represent the block size.
while the remaining 22 bits represent the (compressed) block size.
There are 4 block types :
There are 4 block types :
@@ -281,24 +317,24 @@ There are 4 block types :
| ---------- | ---------- | --- | --- | ------- |
| ---------- | ---------- | --- | --- | ------- |
| Block Type | Compressed | Raw | RLE | EndMark |
| Block Type | Compressed | Raw | RLE | EndMark |
- Compressed : this is a compressed block,
- Compressed : this is a Zstandard compressed block,
following Zstandard's block format specification.
detailed in a later part of this specification.
The "block size" is the compressed size.
"block size" is the compressed size.
Decompressed size is unknown,
Decompressed size is unknown,
but its maximum possible value is guaranteed (see later)
but its maximum possible value is guaranteed (see below)
- Raw : this is an uncompressed block.
- Raw : this is an uncompressed block.
"block size" is the number of bytes to read and copy.
"block size" is the number of bytes to read and copy.
- RLE : this is a single byte, repeated N times.
- RLE : this is a single byte, repeated N times.
In which case, the size of the "compressed" block is always 1,
In which case, "block size" is the size to regenerate,
and the "block size" is the size to regenerate.
while the "compressed" block is just 1 byte (the byte to repeat).
- EndMark : this is not a block. Signal the end of the frame.
- EndMark : this is not a block. Signal the end of the frame.
The rest of the field may be optionally filled by a checksum
The rest of the field may be optionally filled by a checksum
(see frame checksum).
(see frame checksum).
Block Size shall never be larger than Block Maximum Size.
Block sizes must respect a few rules :
Block Maximum Size is the smallest of :
- In compressed mode, compressed size if always strictly `< contentSize`.
- Max back-reference distance
- Block decompressed size is necessarily <= maximum back-reference distance .
- 128 KB
- Block decompressed size is necessarily <= 128 KB
__Data__
__Data__
@@ -306,8 +342,8 @@ __Data__
Where the actual data to decode stands.
Where the actual data to decode stands.
It might be compressed or not, depending on previous field indications.
It might be compressed or not, depending on previous field indications.
A data block is not necessarily "full" :
A data block is not necessarily "full" :
an arbitrary “flush” may happen anytime. Any block can be “partially filled”.
since an arbitrary “flush” may happen anytime,
Therefore, data can have any size, up to Block Maximum Size.
block content can be any size, up to Block Maximum Size.
Block Maximum Size is the smallest of :
Block Maximum Size is the smallest of :
- Max back-reference distance
- Max back-reference distance
- 128 KB
- 128 KB
@@ -329,25 +365,122 @@ over user-defined data and continue decoding.
Skippable frames defined in this specification are compatible with LZ4 ones.
Skippable frames defined in this specification are compatible with LZ4 ones.
__Magic Number__
__Magic Number__ :
4 Bytes, Little endian format.
4 Bytes, Little endian format.
Value : 0x184D2A5X, which means any value from 0x184D2A50 to 0x184D2A5F.
Value : 0x184D2A5X, which means any value from 0x184D2A50 to 0x184D2A5F.
All 16 values are valid to identify a skippable frame.
All 16 values are valid to identify a skippable frame.
__Frame Size__
__Frame Size__ :
This is the size, in bytes, of the following User Data
This is the size, in bytes, of the following User Data
(without including the magic number nor the size field itself).
(without including the magic number nor the size field itself).
4 Bytes, Little endian format, unsigned 32-bits.
4 Bytes, Little endian format, unsigned 32-bits.
This means User Data can’t be bigger than (2^32-1) Bytes.
This means User Data can’t be bigger than (2^32-1) Bytes.
__User Data__
__User Data__ :
User Data can be anything. Data will just be skipped by the decoder.
User Data can be anything. Data will just be skipped by the decoder.
Compressed block format
-----------------------
This specification details the content of a _compressed block_.
A compressed block has a size, which must be known in order to decode it.
It also has a guaranteed maximum regenerated size,
in order to properly allocate destination buffer.
See "Frame format" for more details.
A compressed block consists of 2 sections :
- Literals section
- Sequences section
### Compressed Literals
Literals are compressed using order-0 huffman compression.
During sequence phase, literals will be entangled with match copy operations.
All literals are regrouped in the first part of the block.
They can be decoded first, and then copied during sequence operations,
or they can be decoded on the flow, as needed by sequences.
Note : it's allowed to represent a short value (ex : `13`)
using a long format, accepting the reduced compacity.
__Sizes format for Compressed Block__ :
Note : also applicable to "repeat-stats" blocks.
- Value : 00 : 4 streams
Compressed and regenerated sizes use 10 bits (0-1023)
Total literal header size is 3 bytes
- Value : 01 : _Single stream_
Compressed and regenerated sizes use 10 bits (0-1023)
Total literal header size is 3 bytes
- Value : 10 : 4 streams
Compressed and regenerated sizes use 14 bits (0-16383)
Total literal header size is 4 bytes
- Value : 10 : 4 streams
Compressed and regenerated sizes use 18 bits (0-262143)
Total literal header size is 5 bytes
Version changes
Version changes
---------------
---------------
0.1 : initial release
0.1 : initial release
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Yann Collet