base38: reassign alphabet and numbers

It changes from " 0-9?a-z" to ".0-9a-z~". The default or placeholder
changes from " " to "." so it's now always visible. The wildcard changes
from "?" to "~" so it's now sorted last.
This commit is contained in:
Nigel Tao committed 2024-08-26 12:30:20 +10:00
1 parent e12f10eef0
commit bd42fa7af2
18 files changed
+235 -212

No files matched your search

+29 -13
View File
@@ -31,26 +31,42 @@ Wuffs uses the big-endian interpretation, as it maintains ordering.
Base38 is a tighter encoding than FourCC, fitting four characters into 21 bits
instead of 32 bits. This is achieved by using a smaller alphabet of 38 possible
values (space, 0-9, ? or a-z), so that it cannot distinguish between e.g. an
upper case 'X' and a lower case 'x'. There's also the happy coincidence that
`38 ** 4 = 0x1FD110 = 2085136` is slightly smaller than `2 ** 21 = 0x200000 =
2097152`.
values ('.', 0-9, a-z or '~'), so that it cannot distinguish between e.g. an
upper case 'X' and a lower case 'x'. There are then two happy coincidences:
The base38 encoding of `"JPEG"` is `0x122FF6`, which is `1191926`, which is
`((21 * (38 ** 3)) + (27 * (38 ** 2)) + (16 * (38 ** 1)) + (18 * (38 ** 0)))`.
- `38 ** 4 = 0x1FD110 = 2085136` is slightly smaller than `2 ** 21 = 0x200000 =
2097152`.
- 21 bits is just large enough to hold every valid Unicode code point, so that
systems and data formats that can hold "one UCP" can be repurposed to
alternatively hold "one base38-encoded four-character string".
The base38 encoding of `"jpeg"` is `0x1153D3`, which is `1135571`, which is
`((20 * (38 ** 3)) + (26 * (38 ** 2)) + (15 * (38 ** 1)) + (17 * (38 ** 0)))`.
The minimum base38 value, 0x000000, corresponds to `"...."`. Depending on
context, this can be used as a default, built-in or placeholder value.
The maximum base38 value, 0x1FD10F = 2085135, corresponds to `"~~~~"`.
Depending on context, this can be used as a "matches everything" wildcard.
The maximum 21-bit value, 0x1FFFFF = 2097151, is not a valid base38 value but
can still be used in some contexts as a sentinel or option-not-set value.
### Base38 Namespaces
Using only 21 bits means that we can use base38 values to partition the set of
possible `uint32_t` values into file-format specific enumerations. Each package
(i.e. Wuffs implementation of a specific file format) can define up to 1024
different values in their own namespace, without conflicting with other
packages (assuming that there aren't e.g. two `"JPEG"` Wuffs packages in the
same library). The conventional `uint32_t` packing is:
local values in their own four-character namespace, without conflicting with
other packages (assuming that there aren't e.g. two `"jpeg"` Wuffs packages in
the same library). The conventional `uint32_t` packing is:
- Bit `31` (1 bit) is reserved (zero).
- Bits `10 ..= 30` (21 bits) are the base38 value, shifted by 10.
- Bits `0 ..= 9` (10 bits) are the enumeration value.
- Bits `10 ..= 30` (21 bits) are the base38-encoded namespace, shifted by 10.
- Bits `0 ..= 9` (10 bits) are the local enumeration value.
For example:
- [Quirk values](/doc/note/quirks.md) use this `((base38 << 10) | enumeration)`
- [Quirk values](/doc/note/quirks.md) use this `((namespace << 10) | local)`
scheme.
- [Tokens](/doc/note/tokens.md) assign 21 out of 64 bits for a Base38 value.
- [Tokens](/doc/note/tokens.md) assign 21 out of 64 bits for a base38 value.
+6 -5
View File
@@ -30,15 +30,16 @@ even for malformed input.
## Wuffs API
Each quirk is assigned a `uint32_t` number, packed using the [base38 namespace
Each quirk is assigned a 31-bit `uint32_t` number (the highest bit is unused),
whose high 21 bits use the [base38 namespace
convention](/doc/note/base38-and-fourcc.md). Decoders and encoders can have a
`set_quirk!(key: base.u32, value: base.u64) base.status` method whose key
argument is this `uint32_t` number.
For example, the base38 encoding of `"gif "` and `"json"` is `0x0F8586` and
`0x124265` respectively, so that the GIF-specific quirks have a `uint32_t` key
of `((0x0F8586 << 10) | g)` and the JSON-specific quirks have a `uint32_t` key
of `((0x124265 << 10) | j)`, for some small integers `g` and `j`. The high bits
For example, the base38 encoding of `"gif."` and `"json"` is `0x0EA964` and
`0x116642` respectively, so that the GIF-specific quirks have a `uint32_t` key
of `((0x0EA964 << 10) | g)` and the JSON-specific quirks have a `uint32_t` key
of `((0x116642 << 10) | j)`, for some small integers `g` and `j`. The high bits
are a namespace. The overall quirk keys are different even if `g` and `j`
re-use the same 10-bit integer.
+3 -3
View File
@@ -72,10 +72,10 @@ a simple token that provides the semantics for each `value_extension`.
- Bits `42 ..= 62` (21 bits) are the `value_major`.
- Bits `17 ..= 41` (25 bits) are the `value_minor`.
The `value_major` is a 21-bit [Base38](/doc/note/base38-and-fourcc.md) number.
The `value_major` is a 21-bit [base38](/doc/note/base38-and-fourcc.md) number.
For example, an HTML tokenizer might produce a combination of "base" tokens
(see below) and tokens whose `value_major` is `0x109B0B`, the Base38 encoding
of `html`. The `value_major` forms a namespace that distinguishes e.g.
(see below) and tokens whose `value_major` is `0x0FBEE8`, the base38 encoding
of `"html"`. The `value_major` forms a namespace that distinguishes e.g.
HTML-specific tokens from JSON-specific tokens.
If `value_major` is non-zero then `value_minor` has whatever meaning the