Add doc/note/io-input-output.md

This commit is contained in:
Nigel Tao committed 2019-07-01 14:22:57 +10:00
1 parent 944b9fb008
commit 9f76003177
4 files changed
+222 -6

No files matched your search

+212
View File
@@ -0,0 +1,212 @@
# I/O (Input / Output)
Wuffs per se doesn't have the ability to read or write to files or network
connections. Recall that Wuffs is a programming language for writing libraries,
not applications, and that having fewer capabilities means that it's trivial to
prove that you can't misuse a capability, even when given malicious input.
Instead, the code that calls into Wuffs libraries is responsible for
interfacing with e.g. the file system or the network system. An `io_buffer` is
the mechanism for transferring data into and out of Wuffs libraries. For
example, when decompressing gzip, there are two `io_buffer`'s: the caller fills
a source buffer with e.g. the compressed file's contents and the callee (the
Wuffs library) reads compressed bytes from that source buffer and writes
decompressed bytes to a destination buffer.
## I/O Buffers
An `io_buffer` is a [slice](/doc/note/slices-and-tables.md) of bytes (the data,
a `ptr` and `len`) with additional fields (the metadata): a read index (`ri`),
a write index (`wi`), a position (`pos`) and whether or not it is `closed`.
## Read Index and Write Index
Writing to an `io_buffer`, e.g. copying from a file to a buffer, increments
`wi`. The buffer is full for writing (no more can be written) when `wi` equals
`len`. Writing does not have to fill a buffer before further processing.
Reading from an `io_buffer`, e.g. copying from a buffer to a file, increments
`ri`. The buffer is empty for reading (no more can be read) when `ri` equals
`wi`. Reading does not have to empty a buffer before further processing.
An invariant condition is that `((0 <= ri) and (ri <= wi) and (wi <= len))`.
Having separate read and write indexes simplifies connecting a sequence of
filters or processors with `io_buffer`'s, similar to connecting Unix processes
with pipes. Each filter reads from the previous buffer and writes to the next
buffer. Each buffer is written to by the previous filter and is read from by
the next filter. There's no need to flip a buffer between reading and writing
modes. Nonetheless, `io_buffer`'s are generally not thread-safe.
Continuing the "decompressing gzip" example, the application would write to the
source buffer by copying from e.g. `/dev/stdin`. The Wuffs library would read
from the source buffer and write to the destination buffer. The application
would read from the destination buffer by copying to e.g. `stdout`. Buffer
space can be re-used, via compaction (see below), so that neither the source
(`/dev/stdin`) or destination (`/dev/stdout`) data needs to be entirely in
memory at any point.
For example, an `io_buffer` of length 8 could have 4 bytes available to read
and 1 byte available to write. If 1 byte was written, there would then be 5
bytes available to read. Visually:
```
[.. .. .. .. .. .. .. ..]
|<- ri ->| | |
|<------- wi ------->| |
|<-------- len -------->|
```
## Position
An `io_buffer` is a sliding window into a stream of bytes. Its position (`pos`)
is the number of bytes in the stream prior to the first element of the slice.
The total number of bytes read from and written to the stream are therefore
`(pos + ri)` and `(pos + wi)`.
While every slice element is in-memory, the stream's prior bytes do not
necessarily have to be in-memory now, or have been in-memory in the past. It is
valid to open a file, seek to the 1000'th byte and start copying from there to
an `io_buffer`, provided that `pos` was also initialized to 1000.
## Closed-ness
The `closed` field indicates that no further writes are expected to the
`io_buffer`. When copying from a file to a buffer, `closed` means that we have
reached EOF (End Of File).
For example, decoding a particular file format might, at some point, expect at
least another 4 bytes of data, but only 3 are available to read. If `closed` is
false, this isn't necessarily an error, since an `io_buffer` holds only a
partial view of the underlying data stream, and more data might be forthcoming
but not yet buffered. If `closed` is true, it is definitely an error.
## Undoing Reads and Writes
It is possible to decrement `ri` or `wi`, undoing previous reads or writes,
provided that the invariant `((0 <= ri) and (ri <= wi) and (wi <= len))` holds.
For example, it can be faster on 64 bit (8 byte) systems, if buffer space is
available, to write 8 bytes and then undo 1 byte than to write exactly 7 bytes.
The Wuffs compiler enforces that, during a Wuffs function, `ri` and `wi` will
never be decremented (by an undo operation) to be less than the initial values
at the time of the call. When considering a function as a 'black box', the two
indexes can only travel forward, and it is up to the application code (not
Wuffs library) code to rewind the indexes (e.g. by compaction).
Even though `ri` can not drop below its initial value, Wuffs code can still
read the contents of the slice before `ri` (in sub-slice notation,
`data[0:ri]`) and it should still contain the `(pos + 0)`th, `(pos + 1)`th,
etc. byte of the stream.
The contents of the slice after `wi` (in sub-slice notation, `data[wi:len]`)
are undefined, and code should not rely on its values. When passing an
`io_buffer` into a function, that function is free to modify anything in
`data[wi:len]`, for either value of `wi` before or after the function returns.
## Compaction
Compacting an `io_buffer` moves any written but unread bytes (those in
`data[ri:wi]`) to the start of the buffer, and updates the metadata fields
`ri`, `wi` and `pos`. Equivalently, it moves the sliding window that is the
`io_buffer` as far forward as possible along the stream.
This generally increases `(len - wi)`, the number of bytes available for
writing, allowing for re-using the allocated buffer memory (the data slice).
Suppose that the underlying data stream's `i`th byte has value `i`, and that we
start with `ri`, `wi` and `pos` were `3`, `7` and `20`. Compaction will
subtract 3 from the first two and add 3 to the last, so that the new `ri`, `wi`
and `pos` are `0`, `4` and `23`. Note that `len`, `(pos + ri)` and `(pos + wi)`
are all unchanged.
Here are two equivalent visualizations of before and after compaction. The `xx`
means a byte whose value is undefined (as it is at or past `wi`).
The first visualization is where the slice is fixed and its contents (its view
of the stream) moves relative to the slice:
```
Before:
[20 21 22 23 24 25 26 xx]
|<- ri ->| | |
|<------- wi ------->| |
|<-------- len -------->|
After:
[23 24 25 26 xx xx xx xx]
|| | |
|<-- wi --->| |
|<-------- len -------->|
```
The second visualization is where the stream (and its contents) is fixed and
the slice (the sliding window) moves relative to the stream:
```
pos+ri pos+wi
| |
Before: [20 21 22 23 24 25 26 xx]
Stream: ... 18 19 20 21 22 23 24 25 26 27 27 28 29 30 31 ...
After: [23 24 25 26 xx xx xx xx]
| |
pos+ri pos+wi
```
## Seeking and I/O Positions
Recall that Wuffs code has limited capabilities, and cannot seek in the
underlying I/O data streams per se. When it needs to seek (e.g. when jumping
between video frames), it will typically provide an "I/O position", a
`uint64_t` value, via some package-specific API. The application (the caller of
the Wuffs code) is then responsible for configuring an `io_buffer` whose
`(pos + ri)` or `(pos + wi)` value, depending on whether we're reading or
writing, is at that "I/O position".
If the underlying file (or equivalent) isn't seekable, e.g. it's `/dev/stdin`
instead of a regular file, then the request cannot be satisfied. The
application should then decide whether that error is recoverable or fatal. This
is the application's responsibility, not the library's, as the application
usually has more context to make that decision.
If that "I/O position" is already within the sliding window, it might not be
necessary to seek in the underlying file, as it may be possible to e.g. simply
decrement `ri` to reach a target `(pos + ri)`, for the reading case. Otherwise,
the typical process is:
1. Set `ri`, `wi` and `pos` to `0`, `0` and that "I/O position". This discards
any buffered data (but does not free the buffer's memory).
2. Seek in the underlying file to that same "I/O position".
3. Copy from the underlying file to the `io_buffer`, incrementing `wi`.
Whether or not it was necessary to seek and copy from the underlying file, when
calling back into the Wuffs library, it typically checks that the `io_buffer`'s
`(pos + ri)` is now at the expected "I/O position".
## I/O Reader and I/O Writer
An `io_buffer` is the mechanism for transferring data between the application
and the Wuffs library. Application code can manipulate an `io_buffer`'s fields
as it wishes (but is responsible for maintaining the invariant condition).
Wuffs library code places a further restriction that `io_buffer`s are used
exclusively either for reading or for writing, as optimizing incremental access
to an `io_buffer`'s data, while enforcing invariants, is simpler when only one
of `ri` and `wi` can vary.
Wuffs code therefore refers to either a `base.io_reader` or `base.io_writer`,
both of which are essentially the same type (an `io_buffer`) with different
methods. Wuffs code does not reference an `io_buffer` directly.
## Binding
TODO: discuss `io_bind`, which temporarily adapts a slice of bytes into an
`io_buffer`.
+2
View File
@@ -15,6 +15,8 @@
// limitations under the License.
// ---------------- I/O
//
// See (/doc/note/io-input-output.md).
struct wuffs_base__io_buffer__struct;
+6 -6
View File
@@ -177,12 +177,12 @@ const baseIOPrivateH = "" +
""
const baseIOPublicH = "" +
"// ---------------- I/O\n\nstruct wuffs_base__io_buffer__struct;\n\ntypedef struct {\n // Do not access the private_impl's fields directly. There is no API/ABI\n // compatibility or safety guarantee if you do so.\n struct {\n struct wuffs_base__io_buffer__struct* buf;\n // The bounds values are typically NULL, when created by the Wuffs public\n // API. NULL means that the callee substitutes the implicit bounds derived\n // from buf.\n uint8_t* mark;\n uint8_t* limit;\n } private_impl;\n} wuffs_base__io_reader;\n\ntypedef struct {\n // Do not access the private_impl's fields directly. There is no API/ABI\n // compatibility or safety guarantee if you do so.\n struct {\n struct wuffs_base__io_buffer__struct* buf;\n // The bounds values are typically NULL, when created by the Wuffs public\n // API. NULL means that the callee substitutes the implicit bounds derived\n // from buf.\n uint8_t* mark;\n uint8_t* limit;\n } private_impl;\n} wuffs_base__io_writer;\n\n// wuffs_base__io_buffer_meta is the met" +
"adata for a wuffs_base__io_buffer's\n// data.\ntypedef struct {\n size_t wi; // Write index. Invariant: wi <= len.\n size_t ri; // Read index. Invariant: ri <= wi.\n uint64_t pos; // Position of the buffer start relative to the stream start.\n bool closed; // No further writes are expected.\n} wuffs_base__io_buffer_meta;\n\n// wuffs_base__io_buffer is a 1-dimensional buffer (a pointer and length) plus\n// additional metadata.\n//\n// A value with all fields zero is a valid, empty buffer.\ntypedef struct wuffs_base__io_buffer__struct {\n wuffs_base__slice_u8 data;\n wuffs_base__io_buffer_meta meta;\n\n#ifdef __cplusplus\n inline void compact();\n inline wuffs_base__io_reader reader();\n inline wuffs_base__io_writer writer();\n inline uint64_t reader_available() const;\n inline uint64_t reader_io_position() const;\n inline uint64_t writer_available() const;\n inline uint64_t writer_io_position() const;\n#endif // __cplusplus\n\n} wuffs_base__io_buffer;\n\nstatic inline wuffs_base__io_buffer //\nwuffs_base__make_io" +
"_buffer(wuffs_base__slice_u8 data,\n wuffs_base__io_buffer_meta meta) {\n wuffs_base__io_buffer ret;\n ret.data = data;\n ret.meta = meta;\n return ret;\n}\n\nstatic inline wuffs_base__io_buffer_meta //\nwuffs_base__make_io_buffer_meta(size_t wi,\n size_t ri,\n uint64_t pos,\n bool closed) {\n wuffs_base__io_buffer_meta ret;\n ret.wi = wi;\n ret.ri = ri;\n ret.pos = pos;\n ret.closed = closed;\n return ret;\n}\n\nstatic inline wuffs_base__io_buffer //\nwuffs_base__null_io_buffer() {\n wuffs_base__io_buffer ret;\n ret.data.ptr = NULL;\n ret.data.len = 0;\n ret.meta.wi = 0;\n ret.meta.ri = 0;\n ret.meta.pos = 0;\n ret.meta.closed = false;\n return ret;\n}\n\nstatic inline wuffs_base__io_buffer_meta //\nwuffs_base__null_io_buffer_meta() {\n wuffs_base__io_buffer_meta ret;\n ret.wi = 0;\n ret.ri = 0;\n ret.pos = 0;\n ret.closed = false;\n return ret;\n}\n\nstatic inline wuffs_base__io_reader //\nwuffs_base__null" +
"_io_reader() {\n wuffs_base__io_reader ret;\n ret.private_impl.buf = NULL;\n ret.private_impl.mark = NULL;\n ret.private_impl.limit = NULL;\n return ret;\n}\n\nstatic inline wuffs_base__io_writer //\nwuffs_base__null_io_writer() {\n wuffs_base__io_writer ret;\n ret.private_impl.buf = NULL;\n ret.private_impl.mark = NULL;\n ret.private_impl.limit = NULL;\n return ret;\n}\n\n// wuffs_base__io_buffer__compact moves any written but unread bytes to the\n// start of the buffer.\nstatic inline void //\nwuffs_base__io_buffer__compact(wuffs_base__io_buffer* buf) {\n if (!buf || (buf->meta.ri == 0)) {\n return;\n }\n buf->meta.pos = wuffs_base__u64__sat_add(buf->meta.pos, buf->meta.ri);\n size_t n = buf->meta.wi - buf->meta.ri;\n if (n != 0) {\n memmove(buf->data.ptr, buf->data.ptr + buf->meta.ri, n);\n }\n buf->meta.wi = n;\n buf->meta.ri = 0;\n}\n\nstatic inline wuffs_base__io_reader //\nwuffs_base__io_buffer__reader(wuffs_base__io_buffer* buf) {\n wuffs_base__io_reader ret;\n ret.private_impl.buf = buf;\n ret.private_impl." +
"mark = NULL;\n ret.private_impl.limit = NULL;\n return ret;\n}\n\nstatic inline wuffs_base__io_writer //\nwuffs_base__io_buffer__writer(wuffs_base__io_buffer* buf) {\n wuffs_base__io_writer ret;\n ret.private_impl.buf = buf;\n ret.private_impl.mark = NULL;\n ret.private_impl.limit = NULL;\n return ret;\n}\n\nstatic inline uint64_t //\nwuffs_base__io_buffer__reader_available(const wuffs_base__io_buffer* buf) {\n return buf ? buf->meta.wi - buf->meta.ri : 0;\n}\n\nstatic inline uint64_t //\nwuffs_base__io_buffer__reader_io_position(const wuffs_base__io_buffer* buf) {\n return buf ? wuffs_base__u64__sat_add(buf->meta.pos, buf->meta.ri) : 0;\n}\n\nstatic inline uint64_t //\nwuffs_base__io_buffer__writer_available(const wuffs_base__io_buffer* buf) {\n return buf ? buf->data.len - buf->meta.wi : 0;\n}\n\nstatic inline uint64_t //\nwuffs_base__io_buffer__writer_io_position(const wuffs_base__io_buffer* buf) {\n return buf ? wuffs_base__u64__sat_add(buf->meta.pos, buf->meta.wi) : 0;\n}\n\n#ifdef __cplusplus\n\ninline void //\nwuffs_base_" +
"_io_buffer__struct::compact() {\n wuffs_base__io_buffer__compact(this);\n}\n\ninline wuffs_base__io_reader //\nwuffs_base__io_buffer__struct::reader() {\n return wuffs_base__io_buffer__reader(this);\n}\n\ninline wuffs_base__io_writer //\nwuffs_base__io_buffer__struct::writer() {\n return wuffs_base__io_buffer__writer(this);\n}\n\ninline uint64_t //\nwuffs_base__io_buffer__struct::reader_available() const {\n return wuffs_base__io_buffer__reader_available(this);\n}\n\ninline uint64_t //\nwuffs_base__io_buffer__struct::reader_io_position() const {\n return wuffs_base__io_buffer__reader_io_position(this);\n}\n\ninline uint64_t //\nwuffs_base__io_buffer__struct::writer_available() const {\n return wuffs_base__io_buffer__writer_available(this);\n}\n\ninline uint64_t //\nwuffs_base__io_buffer__struct::writer_io_position() const {\n return wuffs_base__io_buffer__writer_io_position(this);\n}\n\n#endif // __cplusplus\n" +
"// ---------------- I/O\n//\n// See (/doc/note/io-input-output.md).\n\nstruct wuffs_base__io_buffer__struct;\n\ntypedef struct {\n // Do not access the private_impl's fields directly. There is no API/ABI\n // compatibility or safety guarantee if you do so.\n struct {\n struct wuffs_base__io_buffer__struct* buf;\n // The bounds values are typically NULL, when created by the Wuffs public\n // API. NULL means that the callee substitutes the implicit bounds derived\n // from buf.\n uint8_t* mark;\n uint8_t* limit;\n } private_impl;\n} wuffs_base__io_reader;\n\ntypedef struct {\n // Do not access the private_impl's fields directly. There is no API/ABI\n // compatibility or safety guarantee if you do so.\n struct {\n struct wuffs_base__io_buffer__struct* buf;\n // The bounds values are typically NULL, when created by the Wuffs public\n // API. NULL means that the callee substitutes the implicit bounds derived\n // from buf.\n uint8_t* mark;\n uint8_t* limit;\n } private_impl;\n} wuffs_base__io_writer;" +
"\n\n// wuffs_base__io_buffer_meta is the metadata for a wuffs_base__io_buffer's\n// data.\ntypedef struct {\n size_t wi; // Write index. Invariant: wi <= len.\n size_t ri; // Read index. Invariant: ri <= wi.\n uint64_t pos; // Position of the buffer start relative to the stream start.\n bool closed; // No further writes are expected.\n} wuffs_base__io_buffer_meta;\n\n// wuffs_base__io_buffer is a 1-dimensional buffer (a pointer and length) plus\n// additional metadata.\n//\n// A value with all fields zero is a valid, empty buffer.\ntypedef struct wuffs_base__io_buffer__struct {\n wuffs_base__slice_u8 data;\n wuffs_base__io_buffer_meta meta;\n\n#ifdef __cplusplus\n inline void compact();\n inline wuffs_base__io_reader reader();\n inline wuffs_base__io_writer writer();\n inline uint64_t reader_available() const;\n inline uint64_t reader_io_position() const;\n inline uint64_t writer_available() const;\n inline uint64_t writer_io_position() const;\n#endif // __cplusplus\n\n} wuffs_base__io_buffer;\n\nstatic inline wuf" +
"fs_base__io_buffer //\nwuffs_base__make_io_buffer(wuffs_base__slice_u8 data,\n wuffs_base__io_buffer_meta meta) {\n wuffs_base__io_buffer ret;\n ret.data = data;\n ret.meta = meta;\n return ret;\n}\n\nstatic inline wuffs_base__io_buffer_meta //\nwuffs_base__make_io_buffer_meta(size_t wi,\n size_t ri,\n uint64_t pos,\n bool closed) {\n wuffs_base__io_buffer_meta ret;\n ret.wi = wi;\n ret.ri = ri;\n ret.pos = pos;\n ret.closed = closed;\n return ret;\n}\n\nstatic inline wuffs_base__io_buffer //\nwuffs_base__null_io_buffer() {\n wuffs_base__io_buffer ret;\n ret.data.ptr = NULL;\n ret.data.len = 0;\n ret.meta.wi = 0;\n ret.meta.ri = 0;\n ret.meta.pos = 0;\n ret.meta.closed = false;\n return ret;\n}\n\nstatic inline wuffs_base__io_buffer_meta //\nwuffs_base__null_io_buffer_meta() {\n wuffs_base__io_buffer_meta ret;\n ret.wi = 0;\n ret.ri = 0;\n ret.pos = 0;\n ret.closed = false;\n return ret;\n}\n\nstatic inline " +
"wuffs_base__io_reader //\nwuffs_base__null_io_reader() {\n wuffs_base__io_reader ret;\n ret.private_impl.buf = NULL;\n ret.private_impl.mark = NULL;\n ret.private_impl.limit = NULL;\n return ret;\n}\n\nstatic inline wuffs_base__io_writer //\nwuffs_base__null_io_writer() {\n wuffs_base__io_writer ret;\n ret.private_impl.buf = NULL;\n ret.private_impl.mark = NULL;\n ret.private_impl.limit = NULL;\n return ret;\n}\n\n// wuffs_base__io_buffer__compact moves any written but unread bytes to the\n// start of the buffer.\nstatic inline void //\nwuffs_base__io_buffer__compact(wuffs_base__io_buffer* buf) {\n if (!buf || (buf->meta.ri == 0)) {\n return;\n }\n buf->meta.pos = wuffs_base__u64__sat_add(buf->meta.pos, buf->meta.ri);\n size_t n = buf->meta.wi - buf->meta.ri;\n if (n != 0) {\n memmove(buf->data.ptr, buf->data.ptr + buf->meta.ri, n);\n }\n buf->meta.wi = n;\n buf->meta.ri = 0;\n}\n\nstatic inline wuffs_base__io_reader //\nwuffs_base__io_buffer__reader(wuffs_base__io_buffer* buf) {\n wuffs_base__io_reader ret;\n ret.p" +
"rivate_impl.buf = buf;\n ret.private_impl.mark = NULL;\n ret.private_impl.limit = NULL;\n return ret;\n}\n\nstatic inline wuffs_base__io_writer //\nwuffs_base__io_buffer__writer(wuffs_base__io_buffer* buf) {\n wuffs_base__io_writer ret;\n ret.private_impl.buf = buf;\n ret.private_impl.mark = NULL;\n ret.private_impl.limit = NULL;\n return ret;\n}\n\nstatic inline uint64_t //\nwuffs_base__io_buffer__reader_available(const wuffs_base__io_buffer* buf) {\n return buf ? buf->meta.wi - buf->meta.ri : 0;\n}\n\nstatic inline uint64_t //\nwuffs_base__io_buffer__reader_io_position(const wuffs_base__io_buffer* buf) {\n return buf ? wuffs_base__u64__sat_add(buf->meta.pos, buf->meta.ri) : 0;\n}\n\nstatic inline uint64_t //\nwuffs_base__io_buffer__writer_available(const wuffs_base__io_buffer* buf) {\n return buf ? buf->data.len - buf->meta.wi : 0;\n}\n\nstatic inline uint64_t //\nwuffs_base__io_buffer__writer_io_position(const wuffs_base__io_buffer* buf) {\n return buf ? wuffs_base__u64__sat_add(buf->meta.pos, buf->meta.wi) : 0;\n}\n\n#ifde" +
"f __cplusplus\n\ninline void //\nwuffs_base__io_buffer__struct::compact() {\n wuffs_base__io_buffer__compact(this);\n}\n\ninline wuffs_base__io_reader //\nwuffs_base__io_buffer__struct::reader() {\n return wuffs_base__io_buffer__reader(this);\n}\n\ninline wuffs_base__io_writer //\nwuffs_base__io_buffer__struct::writer() {\n return wuffs_base__io_buffer__writer(this);\n}\n\ninline uint64_t //\nwuffs_base__io_buffer__struct::reader_available() const {\n return wuffs_base__io_buffer__reader_available(this);\n}\n\ninline uint64_t //\nwuffs_base__io_buffer__struct::reader_io_position() const {\n return wuffs_base__io_buffer__reader_io_position(this);\n}\n\ninline uint64_t //\nwuffs_base__io_buffer__struct::writer_available() const {\n return wuffs_base__io_buffer__writer_available(this);\n}\n\ninline uint64_t //\nwuffs_base__io_buffer__struct::writer_io_position() const {\n return wuffs_base__io_buffer__writer_io_position(this);\n}\n\n#endif // __cplusplus\n" +
""
const baseRangePrivateH = "" +
+2
View File
@@ -1273,6 +1273,8 @@ wuffs_base__rect_ie_u32::height() const {
#endif // __cplusplus
// ---------------- I/O
//
// See (/doc/note/io-input-output.md).
struct wuffs_base__io_buffer__struct;