Add doc/note/tokens.md

This commit is contained in:
Nigel Tao committed 2020-04-08 23:31:18 +10:00
1 parent df3acfbb21
commit 478d1b817c
4 files changed
+173 -121

No files matched your search

+156
View File
@@ -0,0 +1,156 @@
# Tokens
Some Wuffs codecs transform between byte streams and token streams. For
example, a JSON decoder might transform `[1,true,"abc\txyz"]` as
```
Bytes [ 1 , t r u e , " a b c \ t x y z " ]
Tokens 0 1 2 3......... 4 5 6...... 7... 8...... 9 10
Chains - - - ---------- - ---------------------------- -
```
The tokens partition the bytes. Each byte belongs to exactly one token. Each
token spans zero or more bytes.
Just as [I/O buffers](/doc/note/io-input-output.md) and
[coroutines](/doc/note/coroutines.md) mean that a byte stream doesn't need to
be entirely in memory at any point, token buffers similarly allow a token
stream to be produced and consumed incrementally.
Nonetheless, conceptually, if the byte stream was a single contiguous slice
then each token would correspond to a sub-slice, one whose length was the token
length and whose position was the sum of all previous tokens' lengths.
Consecutive tokens can also form token chains, which capture higher level
concepts. For example, the JSON string `"abc\txyz"` can correspond to multiple
tokens. One reason is that the maximum token length is `65535` bytes, but JSON
strings can be longer. Another reason is that different parts of the encoded
string are treated differently when reconstructing the decoded string:
- Decoding the first or last `"` is a no-op.
- Decoding `abc` or `xyz` is a `memcpy`.
- Decoding the backslash-escaped `\t` should produce a tab character.
## Representation
A token is just a `uint64_t`. The broad divisions are:
- Bits 63 .. 18 (46 bits) is the value.
- Bits 17 .. 16 (2 bits) is `LP` and `LN` (`link_prev` and `link_next`).
- Bits 15 .. 0 (16 bits) is the length.
The `LP` and `LN` bits denote tokens that are part of a multi-token chain:
- `LP` means that this token is not the first (there is a previous token).
- `LN` means that this token is not the last (there is a next token).
A stand-alone token will have both link bits set to zero.
```
+-----+-------------+-------+-------------+-----+-----+-----------+
| 1 | 21 | 3 | 21 | 1 | 1 | 16 |
+-----+-------------+-------+-------------+-----+-----+-----------+
[..................value..................] LP LN length
[..1..|..........~value_extension.........]
[..0..|.value_major.|.....value_minor.....]
[..0..|......0......|..VBC..|.....VBD.....]
```
The value bits can be sub-divided in multiple ways. First, the high bit:
- Bits 63 .. 63 (1 bit) denote an extended (1) or simple (0) token.
### Extended Tokens
- Bits 62 .. 18 (45 bits) is the bitwise-not (~) of the `value_extension`.
Extended tokens are typically part of a multi-token chain whose first token is
a simple token that provides the semantics for each `value_extension`.
### Simple Tokens
- Bits 62 .. 42 (21 bits) is the `value_major`.
- Bits 41 .. 18 (24 bits) is the `value_minor`.
The `value_major` is a 21-bit [Base38](doc/note/base38-and-fourcc.md) number.
For example, an HTML tokenizer might produce a combination of "base" tokens
(see below) and tokens whose `value_major` is `0x109B0B`, the Base38 encoding
of `html`. The `value_major` forms a namespace that distinguishes e.g.
HTML-specific tokens from JSON-specific tokens.
If `value_major` is non-zero then `value_minor` has whatever meaning the
tokenizer's package assigns to it.
### VBCs and VBDs
A zero `value_major` is reserved for Wuffs' built-in "base" package. The
`value_minor` is further sub-divided:
- Bits 41 .. 39 (3 bits) is the `VBC` (`value_base_category`).
- Bits 38 .. 18 (21 bits) is the `VBD` (`value_base_detail`).
The high 46 bits (bits 63 .. 18) only have `VBC` and `VBD` semantics when the
high 22 bits (the `extended` and `value_major` parts) are all zero. An
equivalent test is that the high 25 bits (the notional `VBC`), as either an
unsigned integer or a sign-extended integer, is in the range `0 ..= 7`.
These `VBC`s organize tokens into broad groups, generally applicable (as
opposed to being e.g. HTML-specific or JSON-specific). For example, strings and
numbers are two `VBC`s. Structure is another, for container boundaries like the
start and end of HTML elements and JSON arrays.
Filler is yet another `VBC`. Such tokens can generally be ignored (other than
accumulating their length). Filler is most often encountered as whitespace, but
also includes JSON commas (which are [structurally
inessential](https://www.tbray.org/ongoing/When/201x/2016/08/20/Fixing-JSON#p-1))
and comments.
The `VBD` semantics depend on the `VBC`. For example, at 21 bits, the `VBD` can
hold every valid Unicode code point, up to U+10FFFF. A `\t` or `\u2603` in a
JSON string can each be represented by a single `VBC__UNICODE_CODE_POINT` token
whose `VBD` is `0x0009` or `0x2603`, meaning the Unicode code points U+0009
CHARACTER TABULATION (the ASCII tab character) or U+2603 SNOWMAN.
More details on the `VBC` and `VBD` bit assignments are in the [`source
code`](/internal/cgen/base/token-public.h).
## SAX/Pull versus DOM/Push
For file formats that conceptually decode into a node tree, such as HTML or
JSON, Wuffs typically provides a
[SAX](https://en.wikipedia.org/wiki/Simple_API_for_XML)-like pull parser, not a
[DOM](https://en.wikipedia.org/wiki/Document_Object_Model)-like push parser.
There are general reasons for [favoring pull
parsers](https://github.com/raphlinus/pulldown-cmark/blob/master/README.md#why-a-pull-parser),
but also, Wuffs code cannot [dynamically allocate
memory](/doc/note/memory-safety.md), nor does Wuffs have an `unsafe` keyword or
a foreign function interface, so a caller cannot pass arbitrary callbacks into
Wuffs code. Instead, Wuffs just outputs tokens and tokens are just `uint64_t`s.
The [example/jsonfindptrs](/example/jsonfindptrs/jsonfindptrs.cc) program
demonstrates creating a traditional DOM-like node tree from a SAX-like token
stream. The [example/jsonptr](/example/jsonptr/jsonptr.cc) program demonstrates
a different, lower-level approach that works directly on tokens, where the
entire program (not just the Wuffs library) never calls `malloc`.
## Example Token Stream
```
$ gcc script/print-json-token-debug-format.c && \
> ./a.out -all-tokens -human-readable < test/data/json-things.formatted.json
pos=0x00000000 len=0x0001 link=0b00 vbc=1:Structure........ vbd=0x004011
pos=0x00000001 len=0x0005 link=0b00 vbc=0:Filler........... vbd=0x000000
pos=0x00000006 len=0x0001 link=0b01 vbc=2:String........... vbd=0x000013
pos=0x00000007 len=0x0002 link=0b11 vbc=2:String........... vbd=0x000021
pos=0x00000009 len=0x0001 link=0b10 vbc=2:String........... vbd=0x000013
etc
pos=0x00000094 len=0x0001 link=0b10 vbc=2:String........... vbd=0x000013
pos=0x00000095 len=0x0001 link=0b00 vbc=0:Filler........... vbd=0x000000
pos=0x00000096 len=0x0001 link=0b00 vbc=1:Structure........ vbd=0x001042
```
+7 -56
View File
@@ -16,63 +16,10 @@
// ---------------- Tokens
// wuffs_base__token is an element of a byte stream's tokenization.
//
// See https://github.com/google/wuffs/blob/master/doc/note/tokens.md
typedef struct {
// The repr's 64 bits are divided as:
//
// +-----+-------------+-------+-------------+-----+-----+-----------+
// | 1 | 21 | 3 | 21 | 1 | 1 | 16 |
// +-----+-------------+-------+-------------+-----+-----+-----------+
// [..................value..................] LP LN length
// [..1..|..........~value_extension.........]
// [..0..|.value_major.|.....value_minor.....]
// [..0..|.........VBC.........|.....VBD.....]
//
// The broad divisions are:
// - Bits 63 .. 18 (46 bits) is the value.
// - Bits 17 .. 16 ( 2 bits) is LP and LN (link_prev and link_next).
// - Bits 15 .. 0 (16 bits) is the length.
//
// ----
//
// The value bits can be sub-divided in multiple ways. First, the high bit:
// - Bits 63 .. 63 ( 1 bits) is an extended (1) or simple (0) token.
//
// For extended tokens:
// - Bits 62 .. 18 (45 bits) is the bitwise-not (~) of the value_extension.
//
// For simple tokens:
// - Bits 62 .. 42 (21 bits) is the value_major.
// - Bits 41 .. 18 (24 bits) is the value_minor.
// - Bits 62 .. 39 (24 bits) is the VBC (value_base_category).
// - Bits 38 .. 18 (21 bits) is the VBD (value_base_detail).
//
// The value_major is a 21-bit [Base38](doc/note/base38-and-fourcc.md) value.
// If all of its bits are zero (special cased for Wuffs' built-in "base"
// package) then the value_minor is further sub-divided:
// - Bits 41 .. 39 ( 3 bits) is the VBC (value_base_category).
// - Bits 38 .. 18 (21 bits) is the VBD (value_base_detail).
//
// The high 46 bits (bits 63 .. 18) only have VBC and VBD semantics when the
// high 22 bits (the value_major) are all zero. An equivalent test is that
// the high 25 bits (the notional VBC) has a value in the range 0 ..= 7.
//
// At 21 bits, the VBD can hold every valid Unicode code point.
//
// If value_major is non-zero then value_minor has whatever arbitrary meaning
// the tokenizer's package assigns to it.
//
// ----
//
// Multiple consecutive tokens can form a larger conceptual unit. For
// example, an "abc\tz" string is a single higher level concept but at the
// lower level, it could consist of multiple tokens: the quotes '"', the
// ASCII texts "abc" and "z" and the backslash-escaped tab '\t'. The LP and
// LN (link_prev and link_next) bits denote tokens that are part of a
// multi-token chain:
// - LP means that this token is not the first (there is a previous token).
// - LN means that this token is not the last (there is a next token).
//
// In particular, a stand-alone token will have both link bits set to zero.
uint64_t repr;
#ifdef __cplusplus
@@ -151,6 +98,10 @@ wuffs_base__make_token(uint64_t repr) {
// CONVERT_1_DST_4_SRC_BACKSLASH_X means a source like "\\x23\\x67\\xAB", where
// 12 src bytes encode 3 dst bytes.
//
// Post-processing may further transform those D destination bytes (e.g. treat
// "\\xFF" as the Unicode code point U+00FF instead of the byte 0xFF), but that
// is out of scope of this VBD's semantics.
//
// When src is the empty string, multiple conversion algorithms are applicable
// (so these bits are not necessarily mutually exclusive), all producing the
// same empty dst string.
+3 -9
View File
@@ -382,13 +382,7 @@ const baseTokenPrivateH = "" +
""
const baseTokenPublicH = "" +
"// ---------------- Tokens\n\ntypedef struct {\n // The repr's 64 bits are divided as:\n //\n // +-----+-------------+-------+-------------+-----+-----+-----------+\n // | 1 | 21 | 3 | 21 | 1 | 1 | 16 |\n // +-----+-------------+-------+-------------+-----+-----+-----------+\n // [..................value..................] LP LN length\n // [..1..|..........~value_extension.........]\n // [..0..|.value_major.|.....value_minor.....]\n // [..0..|.........VBC.........|.....VBD.....]\n //\n // The broad divisions are:\n // - Bits 63 .. 18 (46 bits) is the value.\n // - Bits 17 .. 16 ( 2 bits) is LP and LN (link_prev and link_next).\n // - Bits 15 .. 0 (16 bits) is the length.\n //\n " +
"" +
"// ----\n //\n // The value bits can be sub-divided in multiple ways. First, the high bit:\n // - Bits 63 .. 63 ( 1 bits) is an extended (1) or simple (0) token.\n //\n // For extended tokens:\n // - Bits 62 .. 18 (45 bits) is the bitwise-not (~) of the value_extension.\n //\n // For simple tokens:\n // - Bits 62 .. 42 (21 bits) is the value_major.\n // - Bits 41 .. 18 (24 bits) is the value_minor.\n // - Bits 62 .. 39 (24 bits) is the VBC (value_base_category).\n // - Bits 38 .. 18 (21 bits) is the VBD (value_base_detail).\n //\n // The value_major is a 21-bit [Base38](doc/note/base38-and-fourcc.md) value.\n // If all of its bits are zero (special cased for Wuffs' built-in \"base\"\n // package) then the value_minor is further sub-divided:\n // - Bits 41 .. 39 ( 3 bits) is the VBC (value_base_category).\n // - Bits 38 .. 18 (21 bits) is the VBD (value_base_detail).\n //\n // The high 46 bits (bits 63 .. 18) only have VBC and VBD semantics when the\n // high 22 bits (the value_major) are all zero. An eq" +
"uivalent test is that\n // the high 25 bits (the notional VBC) has a value in the range 0 ..= 7.\n //\n // At 21 bits, the VBD can hold every valid Unicode code point.\n //\n // If value_major is non-zero then value_minor has whatever arbitrary meaning\n // the tokenizer's package assigns to it.\n //\n " +
"" +
"// ----\n //\n // Multiple consecutive tokens can form a larger conceptual unit. For\n // example, an \"abc\\tz\" string is a single higher level concept but at the\n // lower level, it could consist of multiple tokens: the quotes '\"', the\n // ASCII texts \"abc\" and \"z\" and the backslash-escaped tab '\\t'. The LP and\n // LN (link_prev and link_next) bits denote tokens that are part of a\n // multi-token chain:\n // - LP means that this token is not the first (there is a previous token).\n // - LN means that this token is not the last (there is a next token).\n //\n // In particular, a stand-alone token will have both link bits set to zero.\n uint64_t repr;\n\n#ifdef __cplusplus\n inline int64_t value() const;\n inline int64_t value_extension() const;\n inline int64_t value_major() const;\n inline int64_t value_base_category() const;\n inline uint64_t value_minor() const;\n inline uint64_t value_base_detail() const;\n inline bool link_prev() const;\n inline bool link_next() const;\n inline uint64_t length() " +
"const;\n#endif // __cplusplus\n\n} wuffs_base__token;\n\nstatic inline wuffs_base__token //\nwuffs_base__make_token(uint64_t repr) {\n wuffs_base__token ret;\n ret.repr = repr;\n return ret;\n}\n\n " +
"// ---------------- Tokens\n\n// wuffs_base__token is an element of a byte stream's tokenization.\n//\n// See https://github.com/google/wuffs/blob/master/doc/note/tokens.md\ntypedef struct {\n uint64_t repr;\n\n#ifdef __cplusplus\n inline int64_t value() const;\n inline int64_t value_extension() const;\n inline int64_t value_major() const;\n inline int64_t value_base_category() const;\n inline uint64_t value_minor() const;\n inline uint64_t value_base_detail() const;\n inline bool link_prev() const;\n inline bool link_next() const;\n inline uint64_t length() const;\n#endif // __cplusplus\n\n} wuffs_base__token;\n\nstatic inline wuffs_base__token //\nwuffs_base__make_token(uint64_t repr) {\n wuffs_base__token ret;\n ret.repr = repr;\n return ret;\n}\n\n " +
"" +
"// --------\n\n#define WUFFS_BASE__TOKEN__LENGTH__MAX_INCL 0xFFFF\n\n#define WUFFS_BASE__TOKEN__VALUE__SHIFT 18\n#define WUFFS_BASE__TOKEN__VALUE_EXTENSION__SHIFT 18\n#define WUFFS_BASE__TOKEN__VALUE_MAJOR__SHIFT 42\n#define WUFFS_BASE__TOKEN__VALUE_BASE_CATEGORY__SHIFT 39\n#define WUFFS_BASE__TOKEN__VALUE_MINOR__SHIFT 18\n#define WUFFS_BASE__TOKEN__VALUE_BASE_DETAIL__SHIFT 18\n#define WUFFS_BASE__TOKEN__LINK__SHIFT 16\n#define WUFFS_BASE__TOKEN__LENGTH__SHIFT 0\n\n#define WUFFS_BASE__TOKEN__LINK_PREV 0x20000\n#define WUFFS_BASE__TOKEN__LINK_NEXT 0x10000\n\n " +
"" +
@@ -398,8 +392,8 @@ const baseTokenPublicH = "" +
"" +
"// --------\n\n#define WUFFS_BASE__TOKEN__VBD__STRUCTURE__PUSH 0x00001\n#define WUFFS_BASE__TOKEN__VBD__STRUCTURE__POP 0x00002\n#define WUFFS_BASE__TOKEN__VBD__STRUCTURE__FROM_NONE 0x00010\n#define WUFFS_BASE__TOKEN__VBD__STRUCTURE__FROM_LIST 0x00020\n#define WUFFS_BASE__TOKEN__VBD__STRUCTURE__FROM_DICT 0x00040\n#define WUFFS_BASE__TOKEN__VBD__STRUCTURE__TO_NONE 0x01000\n#define WUFFS_BASE__TOKEN__VBD__STRUCTURE__TO_LIST 0x02000\n#define WUFFS_BASE__TOKEN__VBD__STRUCTURE__TO_DICT 0x04000\n\n" +
"" +
"// --------\n\n// \"DEFINITELY_FOO\" means that the destination bytes (and also the source\n// bytes, for 1_DST_1_SRC_COPY) are in the FOO format. Definitely means that\n// the lack of the bit is conservative: it is valid for all-ASCII strings to\n// have neither DEFINITELY_UTF_8 or DEFINITELY_ASCII bits set.\n#define WUFFS_BASE__TOKEN__VBD__STRING__DEFINITELY_UTF_8 0x00001\n#define WUFFS_BASE__TOKEN__VBD__STRING__DEFINITELY_ASCII 0x00002\n\n// \"CONVERT_D_DST_S_SRC\" means that multiples of S source bytes (possibly\n// padded) produces multiples of D destination bytes. For example,\n// CONVERT_1_DST_4_SRC_BACKSLASH_X means a source like \"\\\\x23\\\\x67\\\\xAB\", where\n// 12 src bytes encode 3 dst bytes.\n//\n// When src is the empty string, multiple conversion algorithms are applicable\n// (so these bits are not necessarily mutually exclusive), all producing the\n// same empty dst string.\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_0_DST_1_SRC_DROP 0x00010\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_1_DST_1_SRC_COPY 0x00020\n#" +
"define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_1_DST_2_SRC_HEXADECIMAL 0x00040\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_1_DST_4_SRC_BACKSLASH_X 0x00080\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_3_DST_4_SRC_BASE_64_STD 0x00100\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_3_DST_4_SRC_BASE_64_URL 0x00200\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_4_DST_5_SRC_ASCII_85 0x00400\n\n " +
"// --------\n\n// \"DEFINITELY_FOO\" means that the destination bytes (and also the source\n// bytes, for 1_DST_1_SRC_COPY) are in the FOO format. Definitely means that\n// the lack of the bit is conservative: it is valid for all-ASCII strings to\n// have neither DEFINITELY_UTF_8 or DEFINITELY_ASCII bits set.\n#define WUFFS_BASE__TOKEN__VBD__STRING__DEFINITELY_UTF_8 0x00001\n#define WUFFS_BASE__TOKEN__VBD__STRING__DEFINITELY_ASCII 0x00002\n\n// \"CONVERT_D_DST_S_SRC\" means that multiples of S source bytes (possibly\n// padded) produces multiples of D destination bytes. For example,\n// CONVERT_1_DST_4_SRC_BACKSLASH_X means a source like \"\\\\x23\\\\x67\\\\xAB\", where\n// 12 src bytes encode 3 dst bytes.\n//\n// Post-processing may further transform those D destination bytes (e.g. treat\n// \"\\\\xFF\" as the Unicode code point U+00FF instead of the byte 0xFF), but that\n// is out of scope of this VBD's semantics.\n//\n// When src is the empty string, multiple conversion algorithms are applicable\n// (so these bits are not necessarily mutual" +
"ly exclusive), all producing the\n// same empty dst string.\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_0_DST_1_SRC_DROP 0x00010\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_1_DST_1_SRC_COPY 0x00020\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_1_DST_2_SRC_HEXADECIMAL 0x00040\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_1_DST_4_SRC_BACKSLASH_X 0x00080\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_3_DST_4_SRC_BASE_64_STD 0x00100\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_3_DST_4_SRC_BASE_64_URL 0x00200\n#define WUFFS_BASE__TOKEN__VBD__STRING__CONVERT_4_DST_5_SRC_ASCII_85 0x00400\n\n " +
"" +
"// --------\n\n#define WUFFS_BASE__TOKEN__VBD__LITERAL__UNDEFINED 0x00001\n#define WUFFS_BASE__TOKEN__VBD__LITERAL__NULL 0x00002\n#define WUFFS_BASE__TOKEN__VBD__LITERAL__FALSE 0x00004\n#define WUFFS_BASE__TOKEN__VBD__LITERAL__TRUE 0x00008\n\n " +
"" +
+7 -56
View File
@@ -1875,63 +1875,10 @@ wuffs_base__io_buffer::writer_io_position() const {
// ---------------- Tokens
// wuffs_base__token is an element of a byte stream's tokenization.
//
// See https://github.com/google/wuffs/blob/master/doc/note/tokens.md
typedef struct {
// The repr's 64 bits are divided as:
//
// +-----+-------------+-------+-------------+-----+-----+-----------+
// | 1 | 21 | 3 | 21 | 1 | 1 | 16 |
// +-----+-------------+-------+-------------+-----+-----+-----------+
// [..................value..................] LP LN length
// [..1..|..........~value_extension.........]
// [..0..|.value_major.|.....value_minor.....]
// [..0..|.........VBC.........|.....VBD.....]
//
// The broad divisions are:
// - Bits 63 .. 18 (46 bits) is the value.
// - Bits 17 .. 16 ( 2 bits) is LP and LN (link_prev and link_next).
// - Bits 15 .. 0 (16 bits) is the length.
//
// ----
//
// The value bits can be sub-divided in multiple ways. First, the high bit:
// - Bits 63 .. 63 ( 1 bits) is an extended (1) or simple (0) token.
//
// For extended tokens:
// - Bits 62 .. 18 (45 bits) is the bitwise-not (~) of the value_extension.
//
// For simple tokens:
// - Bits 62 .. 42 (21 bits) is the value_major.
// - Bits 41 .. 18 (24 bits) is the value_minor.
// - Bits 62 .. 39 (24 bits) is the VBC (value_base_category).
// - Bits 38 .. 18 (21 bits) is the VBD (value_base_detail).
//
// The value_major is a 21-bit [Base38](doc/note/base38-and-fourcc.md) value.
// If all of its bits are zero (special cased for Wuffs' built-in "base"
// package) then the value_minor is further sub-divided:
// - Bits 41 .. 39 ( 3 bits) is the VBC (value_base_category).
// - Bits 38 .. 18 (21 bits) is the VBD (value_base_detail).
//
// The high 46 bits (bits 63 .. 18) only have VBC and VBD semantics when the
// high 22 bits (the value_major) are all zero. An equivalent test is that
// the high 25 bits (the notional VBC) has a value in the range 0 ..= 7.
//
// At 21 bits, the VBD can hold every valid Unicode code point.
//
// If value_major is non-zero then value_minor has whatever arbitrary meaning
// the tokenizer's package assigns to it.
//
// ----
//
// Multiple consecutive tokens can form a larger conceptual unit. For
// example, an "abc\tz" string is a single higher level concept but at the
// lower level, it could consist of multiple tokens: the quotes '"', the
// ASCII texts "abc" and "z" and the backslash-escaped tab '\t'. The LP and
// LN (link_prev and link_next) bits denote tokens that are part of a
// multi-token chain:
// - LP means that this token is not the first (there is a previous token).
// - LN means that this token is not the last (there is a next token).
//
// In particular, a stand-alone token will have both link bits set to zero.
uint64_t repr;
#ifdef __cplusplus
@@ -2010,6 +1957,10 @@ wuffs_base__make_token(uint64_t repr) {
// CONVERT_1_DST_4_SRC_BACKSLASH_X means a source like "\\x23\\x67\\xAB", where
// 12 src bytes encode 3 dst bytes.
//
// Post-processing may further transform those D destination bytes (e.g. treat
// "\\xFF" as the Unicode code point U+00FF instead of the byte 0xFF), but that
// is out of scope of this VBD's semantics.
//
// When src is the empty string, multiple conversion algorithms are applicable
// (so these bits are not necessarily mutually exclusive), all producing the
// same empty dst string.