Dave VasilevskyandYann Collet
2d4cff69c4
Previously, parallel_compression would only handle each job's results after ALL jobs were successfully queued. This caused all src/dst buffers to remain in memory until then! It also polled to check whether a job completed, which is racy without any memory barrier. Now, we flush results as a side effect of completing a job. Completed frames are placed in an ordered linked-list, and any eligible frames are flushed. This may be zero or multiple frames, depending on the order in which jobs finish. This design also makes it simple to support streaming input, so that is now available. Just pass `-` as the filename, and stdin/stdout will be used for I/O.
Zstandard Seekable Format
The seekable format splits compressed data into a series of independent "frames", each compressed individually, so that decompression of a section in the middle of an archive only requires zstd to decompress at most a frame's worth of extra data, instead of the entire archive.
The frames are appended, so that the decompression of the entire payload still regenerates the original content, using any compliant zstd decoder.
On top of that, the seekable format generates a jump table, which makes it possible to jump directly to the position of the relevant frame when requesting only a segment of the data. The jump table is simply ignored by zstd decoders unaware of the seekable format.
The format is delivered with an API to create seekable archives and to retrieve arbitrary segments inside the archive.
Maximum Frame Size parameter
When creating a seekable archive, the main parameter is the maximum frame size.
At compression time, user can manually select the boundaries between segments, but they don't have to: long segments will be automatically split when larger than selected maximum frame size.
Small frame sizes reduce decompression cost when requesting small segments, because the decoder will nonetheless have to decompress an entire frame to recover just a single byte from it.
A good rule of thumb is to select a maximum frame size roughly equivalent to the access pattern when it's known. For example, if the application tends to request 4KB blocks, then it's a good idea to set a maximum frame size in the vicinity of 4 KB.
But small frame sizes also reduce compression ratio, and increase the cost for the jump table, so there is a balance to find.
In general, try to avoid really tiny frame sizes (<1 KB), which would have a large negative impact on compression ratio.