mirror of
https://github.com/ddidderr/wuffs.git
synced 2026-11-17 16:32:11 +01:00
And some minor clean-ups. There should be no semantic changes to the documentation in this patch.
237 lines
9.9 KiB
Markdown
237 lines
9.9 KiB
Markdown
# I/O (Input / Output)
|
|
|
|
Wuffs per se doesn't have the ability to read or write to files or network
|
|
connections. Recall that Wuffs is a programming language for writing libraries,
|
|
not applications, and that having fewer capabilities means that it's trivial to
|
|
prove that you can't misuse a capability, even when given malicious input.
|
|
|
|
Instead, the code that calls into Wuffs libraries is responsible for
|
|
interfacing with e.g. the file system or the network system. An `io_buffer` is
|
|
the mechanism for transferring data into and out of Wuffs libraries. For
|
|
example, when decompressing gzip, there are two `io_buffer`s: the caller fills
|
|
a source buffer with e.g. the compressed file's contents and the callee (the
|
|
Wuffs library) reads compressed bytes from that source buffer and writes
|
|
decompressed bytes to a destination buffer.
|
|
|
|
|
|
## I/O Buffers
|
|
|
|
An `io_buffer` is a [slice](/doc/note/slices-arrays-and-tables.md) of bytes
|
|
(the data, a `ptr` and `len`) with additional fields (the metadata): a read
|
|
index (`ri`), a write index (`wi`), a position (`pos`) and whether or not it is
|
|
`closed`.
|
|
|
|
|
|
## Read Index and Write Index
|
|
|
|
Writing to an `io_buffer`, e.g. copying from a file to a buffer, increments
|
|
`wi`. The buffer is full for writing (no more can be written) when `wi` equals
|
|
`len`. Writing does not have to fill a buffer before further processing.
|
|
|
|
Reading from an `io_buffer`, e.g. copying from a buffer to a file, increments
|
|
`ri`. The buffer is empty for reading (no more can be read) when `ri` equals
|
|
`wi`. Reading does not have to empty a buffer before further processing.
|
|
|
|
An invariant condition is that `((0 <= ri) and (ri <= wi) and (wi <= len))`.
|
|
|
|
Having separate read and write indexes simplifies connecting a sequence of
|
|
filters or processors with `io_buffer`s, similar to connecting Unix processes
|
|
with pipes. Each filter reads from the previous buffer and writes to the next
|
|
buffer. Each buffer is written to by the previous filter and is read from by
|
|
the next filter. There's no need to flip a buffer between reading and writing
|
|
modes. Nonetheless, `io_buffer`s are generally not thread-safe.
|
|
|
|
Continuing the "decompressing gzip" example, the application would write to
|
|
the source buffer by copying from e.g. `stdin`. The Wuffs library would read
|
|
from the source buffer and write to the destination buffer. The application
|
|
would read from the destination buffer by copying to e.g. `stdout`. Buffer
|
|
space can be re-used, via compaction (see below), so that neither the source
|
|
or destination data needs to be entirely in memory at any point.
|
|
|
|
For example, an `io_buffer` of length 8 could have 4 bytes available to read
|
|
and 1 byte available to write. If 1 byte was written, there would then be 5
|
|
bytes available to read. Visually:
|
|
|
|
```
|
|
[.. .. .. .. .. .. .. ..]
|
|
|<- ri ->| | |
|
|
|<------- wi ------->| |
|
|
|<-------- len -------->|
|
|
```
|
|
|
|
|
|
## Position
|
|
|
|
An `io_buffer` is a sliding window into a stream of bytes. Its position (`pos`)
|
|
is the number of bytes in the stream prior to the first element of the slice.
|
|
The total number of bytes read from and written to the stream are therefore
|
|
`(pos + ri)` and `(pos + wi)`.
|
|
|
|
While every slice element is in-memory, the stream's prior bytes do not
|
|
necessarily have to be in-memory now, or have been in-memory in the past. It is
|
|
valid to open a file, seek to the 1000'th byte and start copying from there to
|
|
an `io_buffer`, provided that `pos` was also initialized to 1000.
|
|
|
|
|
|
## Closed-ness
|
|
|
|
The `closed` field indicates that no further writes are expected to the
|
|
`io_buffer`. When copying from a file to a buffer, `closed` means that we have
|
|
reached EOF (End Of File).
|
|
|
|
For example, decoding a particular file format might, at some point, expect at
|
|
least another 4 bytes of data, but only 3 are available to read. If `closed` is
|
|
false, this isn't necessarily an error, since an `io_buffer` holds only a
|
|
partial view of the underlying data stream, and more data might be forthcoming
|
|
but not yet buffered. If `closed` is true, it is definitely an error.
|
|
|
|
|
|
## Undoing Reads and Writes
|
|
|
|
It is possible to decrement `ri` or `wi`, undoing previous reads or writes,
|
|
provided that the invariant `((0 <= ri) and (ri <= wi) and (wi <= len))` holds.
|
|
For example, it can be faster on 64 bit (8 byte) systems, if buffer space is
|
|
available, to write 8 bytes and then undo 1 byte than to write exactly 7 bytes.
|
|
|
|
The Wuffs compiler enforces that, during a Wuffs function, `ri` and `wi` will
|
|
never be decremented (by an undo operation) to be less than the initial values
|
|
at the time of the call. When considering a function as a 'black box', the two
|
|
indexes can only travel forward, and it is up to the application code (not
|
|
Wuffs library) code to rewind the indexes (e.g. by compaction).
|
|
|
|
Even though `ri` cannot drop below its initial value, Wuffs code can still
|
|
read the contents of the slice before `ri` (in sub-slice notation,
|
|
`data[0 .. ri]`) and it should still contain the `(pos + 0)`th, `(pos + 1)`th,
|
|
etc. byte of the stream.
|
|
|
|
The contents of the slice after `wi` (in sub-slice notation, `data[wi .. len]`)
|
|
are undefined, and code should not rely on its values. When passing an
|
|
`io_buffer` into a function, that function is free to modify anything in
|
|
`data[wi .. len]`, for either value of `wi` before or after the function
|
|
returns.
|
|
|
|
|
|
## Compaction
|
|
|
|
Compacting an `io_buffer` moves any written but unread bytes (those in `data[ri
|
|
.. wi]`) to the start of the buffer, and updates the metadata fields `ri`, `wi`
|
|
and `pos`. Equivalently, it moves the sliding window that is the `io_buffer` as
|
|
far forward as possible along the stream.
|
|
|
|
This generally increases `(len - wi)`, the number of bytes available for
|
|
writing, allowing for re-using the allocated buffer memory (the data slice).
|
|
|
|
Suppose that the underlying data stream's `i`th byte has value `i`, and that we
|
|
start with `ri`, `wi` and `pos` were `3`, `7` and `20`. Compaction will
|
|
subtract 3 from the first two and add 3 to the last, so that the new `ri`, `wi`
|
|
and `pos` are `0`, `4` and `23`. Note that `len`, `(pos + ri)` and `(pos + wi)`
|
|
are all unchanged.
|
|
|
|
Here are two equivalent visualizations of before and after compaction. The `xx`
|
|
means a byte whose value is undefined (as it is at or past `wi`).
|
|
|
|
The first visualization is where the slice is fixed and its contents (its view
|
|
of the stream) moves relative to the slice:
|
|
|
|
```
|
|
Before:
|
|
[20 21 22 23 24 25 26 xx]
|
|
|<- ri ->| | |
|
|
|<------- wi ------->| |
|
|
|<-------- len -------->|
|
|
|
|
After:
|
|
[23 24 25 26 xx xx xx xx]
|
|
|| | |
|
|
|<-- wi --->| |
|
|
|<-------- len -------->|
|
|
```
|
|
|
|
The second visualization is where the stream (and its contents) is fixed and
|
|
the slice (the sliding window) moves relative to the stream:
|
|
|
|
```
|
|
pos+ri pos+wi
|
|
| |
|
|
Before: [20 21 22 23 24 25 26 xx]
|
|
Stream: ... 18 19 20 21 22 23 24 25 26 27 27 28 29 30 31 ...
|
|
After: [23 24 25 26 xx xx xx xx]
|
|
| |
|
|
pos+ri pos+wi
|
|
```
|
|
|
|
|
|
## Seeking and I/O Positions
|
|
|
|
Recall that Wuffs code has limited capabilities, and cannot seek in the
|
|
underlying I/O data streams per se. When it needs to seek (e.g. when jumping
|
|
between video frames), it will typically provide an "I/O position", a
|
|
`uint64_t` value, via some package-specific API. The application (the caller of
|
|
the Wuffs code) is then responsible for configuring an `io_buffer` whose
|
|
`(pos + ri)` or `(pos + wi)` value, depending on whether we're reading or
|
|
writing, is at that "I/O position".
|
|
|
|
If the underlying file (or equivalent) isn't seekable, e.g. it's `/dev/stdin`
|
|
instead of a regular file, then the request cannot be satisfied. The
|
|
application should then decide whether that error is recoverable or fatal. This
|
|
is the application's responsibility, not the library's, as the application
|
|
usually has more context to make that decision.
|
|
|
|
If that "I/O position" is already within the sliding window, it might not be
|
|
necessary to seek in the underlying file, as it may be possible to e.g. simply
|
|
decrement `ri` to reach a target `(pos + ri)`, for the reading case. Otherwise,
|
|
the typical process is:
|
|
|
|
1. Set `ri`, `wi` and `pos` to `0`, `0` and that "I/O position". This discards
|
|
any buffered data (but does not free the buffer's memory).
|
|
2. Seek in the underlying file to that same "I/O position".
|
|
3. Copy from the underlying file to the `io_buffer`, incrementing `wi`.
|
|
|
|
Whether or not it was necessary to seek and copy from the underlying file, when
|
|
calling back into the Wuffs library, it typically checks that the `io_buffer`'s
|
|
`(pos + ri)` is now at the expected "I/O position".
|
|
|
|
|
|
## I/O Reader and I/O Writer
|
|
|
|
An `io_buffer` is the mechanism for transferring data between the application
|
|
and the Wuffs library. Application code can manipulate an `io_buffer`'s fields
|
|
as it wishes (but is responsible for maintaining the invariant condition).
|
|
Wuffs library code places a further restriction that `io_buffer`s are used
|
|
exclusively either for reading or for writing, as optimizing incremental access
|
|
to an `io_buffer`'s data, while enforcing invariants, is simpler when only one
|
|
of `ri` and `wi` can vary.
|
|
|
|
Wuffs code therefore refers to either a `base.io_reader` or `base.io_writer`,
|
|
both of which are essentially the same type (an `io_buffer`) with different
|
|
methods. Wuffs code does not reference an `io_buffer` directly.
|
|
|
|
|
|
## Binding
|
|
|
|
An `io_bind` block temporarily adapts a slice of bytes as an `io_reader` or
|
|
`io_writer`. This is typically done to call other functions that take an
|
|
`io_reader` or `io_writer` as an argument.
|
|
|
|
```
|
|
var r : base.io_reader
|
|
var s : slice base.u8
|
|
|
|
etc
|
|
|
|
// Just before the io_bind, r's state is saved.
|
|
|
|
io_bind (io: r, data: s) {
|
|
// At the top of the block, r's data slice is set to s, and r's metadata is
|
|
// set so that ri = 0, pos = 0 and closed = false.
|
|
//
|
|
// Because r is an io_reader, not an io_writer, the wi metadata field is
|
|
// set to the slice length, not 0.
|
|
//
|
|
// r must be a local variable, but s can be an expression.
|
|
etc
|
|
}
|
|
|
|
// Just after the io_bind, r's state is restored.
|
|
```
|