feat(unicode,datastruct): port Unicode tables and data structures

ghostty-unicode: Ghostty computes codepoint properties with the uucode
Zig package (v0.2.0, Unicode 17.0.0). There is no Rust equivalent, so
a trimmed UCD set is vendored in ucd/ and build.rs re-implements
uucode's derivations: cell width rules incl. Ghostty's overrides for
emoji modifiers and Prepend, grapheme break classes with the
compile-time grapheme_break state table, the is_symbol set, and emoji
presentation. Lookups are three array indexes per codepoint. Validated
against the official GraphemeBreakTest.txt.

ghostty-datastruct: split_tree (used for GTK splits; f16 ratios so
rounding and debug diagrams match Zig), circ_buf, lru, cache_table,
blocking_queue, segmented_pool, message_data, comparison,
array_list_collection. intrusive_linked_list is not needed (lru is
index based). lru now reports the evicted (old) key, Zig reported the
new key, which appears to be a bug.

Not yet covered: case_folding_full (needed by input bindings).

Test Plan:
- cargo test -p ghostty-unicode (16) -p ghostty-datastruct (88)
- cargo clippy for both crates with -D warnings
This commit is contained in:
ddidderr committed 2026-10-11 21:54:57 +02:00
1 parent b96984b020
commit 5b8c92b98c
31 files changed
+18468 -3

No files matched your search

+66
View File
@@ -0,0 +1,66 @@
# ghostty-datastruct
Port of `ZIG/src/datastruct/*`.
| Zig | Rust (`crates/ghostty-datastruct/src`) | Notes |
|---|---|---|
| `split_tree.zig` | `split_tree.rs` | Full port, all 17 tests (+4 extra). |
| `circ_buf.zig` | `circ_buf.rs` | `CircBuf<T>` takes the default value in `new`. All 21 tests (+2). |
| `lru.zig` | `lru.rs` | Slab + index linked list. All 4 tests (+1). |
| `cache_table.zig` | `cache_table.rs` | `CacheContext` trait (hash/eql/evicted). 1 test (+2). |
| `blocking_queue.zig` | `blocking_queue.rs` | `Mutex` + `Condvar`. 2 tests (+3, incl. cross-thread). |
| `segmented_pool.zig` | `segmented_pool.rs` | Index handles instead of pointers. 1 test. |
| `message_data.zig` | `message_data.rs` | enum `Small/Stable/Alloc`. 3 tests (+1). |
| `comparison.zig` | `comparison.rs` | `DeepEqual` / `ApproxEq` traits. All 24 tests. |
| `array_list_collection.zig` | `array_list_collection.rs` | 0 Zig tests (1 added). |
| `intrusive_linked_list.zig` | **not ported** | Only user is `lru.zig`; replaced by an index-based list. Its 1 test is skipped. |
| `main.zig` | `lib.rs` | `refAllDecls` test skipped (Zig only). |
Zig tests: 73 ported (all), 2 skipped (`intrusive_linked_list` basic test,
`main.zig` refAllDecls). Total Rust tests: 88.
## SplitTree design / deviations
* View type: Zig stores `*View` with `ref`/`unref`. Rust: `V: SplitTreeView`
(`Clone` + optional `split_tree_label() -> Option<String>`). `Clone` is the
ref, `Drop` is the unref (use `Rc`/`Arc`/GObject handles inside the view).
Allocation is infallible, so no allocator parameters; the arena is a `Vec`.
`SplitTree::new(&view)` replaces `init`, `empty()` replaces `.empty`,
`Clone` replaces `clone`, `Drop` replaces `deinit`.
* Only `split` can fail (`SplitError::OutOfMemory` when > 65535 nodes, as in Zig).
* `Handle` is a `u16` newtype (`Handle::ROOT`, `idx()`, `offset()`,
`from_idx()`); `nodes()`/`zoomed()` are read-only accessors.
* Ratios and spatial slots use `half::f16` exactly like Zig's `f16` (so
rounding in `{d:.2}` output, `resize`, spatial navigation match).
* `goto` no longer takes an allocator; spatial navigation allocates a `Vec`.
* Formatting: `Display` = diagram + text (like `{f}`); `format_diagram`,
`format_text` (writer based), `diagram_string()` / `text_string()` helpers
replace `std.fmt.alt(t, .formatDiagram/.formatText)`. If a view has no label,
the node index is used (digit width = `log10(nodes)+1`).
* `getGObjectType` (GTK boxed type glue) is not ported: the GTK crate should
wrap `SplitTree<V>` itself (e.g. a `glib::Boxed` newtype using `Clone`).
## Other deviations
* `lru`: `V: Default` is required (the new slot is default-initialised where
Zig leaves it undefined). `GetOrPutResult.evicted` contains the evicted
entry's **old key** and old value (Zig set the new key before copying the
entry out, which looks like a bug; only `.value` is used in tests). `get`
returns `Option<&V>` and does not affect LRU order (as in Zig).
`resize` returns the evicted values (oldest first) as `Option<Vec<V>>`.
* `cache_table`: `get` returns `Option<&V>` (moves the hit to the bucket end);
buckets are `Vec`s instead of fixed arrays; `K`/`V: Clone` for `put` because
the eviction hook and the return value both receive the evicted pair.
* `blocking_queue`: `create`/`destroy` -> `new()`/`Arc`; `drain()` returns a
guard iterator that releases the lock (and signals waiting producers) on drop.
Blocking pushes use `wait_while`, so spurious wakeups no longer fail a push.
* `segmented_pool`: `get()` returns a slot index; use `at`/`at_mut`.
* `message_data`: `Stable` holds `&'static [T]`; `Alloc` holds a `Vec`.
* `comparison`: Zig reflection (`deepEqual` over any type) is replaced by traits;
implement `DeepEqual` for custom types (rarely needed — prefer `PartialEq`).
`expect_approx_equal` panics instead of returning an error; `@Vector` and
anonymous structs map to arrays/tuples.
## TODO(integration)
None.
+84
View File
@@ -0,0 +1,84 @@
# ghostty-unicode
Port of `ZIG/src/unicode/*`.
## Module map
| Zig | Rust (`crates/ghostty-unicode`) |
|---|---|
| `main.zig` | `src/lib.rs` (re-exports `table`, `Properties`, `grapheme_break`, `BreakState`) |
| `lut.zig` | `src/lut.rs` (3-stage table generator + lookup; also included by `build.rs`) |
| `props.zig` | `src/props.rs` (`Properties`, `GraphemeBreakNoControl`) |
| `props_table.zig` | `src/props_table.rs` (`table()`; generated into `$OUT_DIR`) |
| `props_uucode.zig`, `symbols_uucode.zig` (the generators) | `build.rs` + `build_support/ucd.rs` (UCD parser and uucode-equivalent computation) |
| `symbols_table.zig` | `src/symbols_table.rs` |
| `grapheme.zig` | `src/grapheme.rs` |
| (uucode `is_emoji_presentation`, used by font `CodepointResolver`) | `src/emoji_table.rs` (extra) |
## How the data is derived (uucode)
Ghostty gets its data from the `uucode` Zig package (v0.2.0, vendored UCD =
**Unicode 17.0.0**), configured in `ZIG/src/build/uucode_config.zig`. Neither
a Rust crate nor runtime data is used; instead the exact UCD files uucode ships
(trimmed to the used properties) are vendored in `ucd/` and `build.rs`
re-implements uucode's computations, so results are identical to Ghostty:
* `ucd/extracted/DerivedGeneralCategory.txt`, `DerivedEastAsianWidth.txt`
(incl. `@missing` Wide ranges), `auxiliary/GraphemeBreakProperty.txt`,
`DerivedCoreProperties.txt` (trimmed to `Default_Ignorable_Code_Point` and
`InCB`), `emoji/emoji-data.txt`, `emoji/emoji-variation-sequences.txt`,
`Blocks.txt`, `auxiliary/GraphemeBreakTest.txt` (tests only).
(uucode uses `UnicodeData.txt` for general category; `DerivedGeneralCategory`
carries the same values and is 10x smaller.)
* **Width** (`uucode x/config_x/wcwidth.zig` + Ghostty `computeWidth`):
Cc/Cs/Zl/Zp -> 0; U+00AD -> 1; Default_Ignorable -> 0; U+2E3A -> 2;
U+2E3B -> 3 (clamped to 2 in `width`); EAW W/F -> 2; Regional Indicator -> 2;
else 1; U+20E3 standalone = 2. `width_zero_in_grapheme` = width 0, emoji
modifier, Mn, Me, GCB=V, GCB=T, or Prepend. Ghostty's final `width`:
`0` if `zero_in_grapheme && !emoji_modifier && gcb != prepend`, else
`min(2, standalone)` (so emoji modifiers and Prepend characters keep a width;
VS15/VS16/ZWJ/combining marks are 0).
* **Grapheme break** property (`grapheme_break_no_control`): emoji modifier,
emoji modifier base, extended pictographic, then InCB (linker / consonant /
extend; ZWJ and ZWNJ split out), else GCB with control/CR/LF folded into
`other` (the terminal filters controls first, as in Zig).
* **`graphemeBreak(cp1, cp2, state)`**: uucode's
`computeGraphemeBreakNoControl` (UAX #29 incl. GB9c Indic conjuncts, GB11,
GB12/13), with Ghostty/uucode's tailoring that emoji modifiers only extend
an emoji modifier base. Precomputed at compile time (const fn) into a
5 x 17 x 17 table. `grapheme_break_props(gb1, gb2, state)` takes
already-looked-up properties (hot path for the terminal).
* **`is_symbol`** (`symbols_table::table()`): general category Co or block in
{Arrows, Dingbats, Emoticons, Miscellaneous Symbols, Enclosed Alphanumerics,
Enclosed Alphanumeric Supplement, Miscellaneous Symbols and Pictographs,
Transport and Map Symbols}.
* Code points above U+10FFFF return `{width: 1, width_zero_in_grapheme: true,
Other, emoji_vs_base: false}` (as in `props_uucode.get`).
## API differences
* Code points are `u32` (Zig `u21`). `table().get(cp)` is branch-free:
three array loads. `table()` returns a tiny `Tables` view over static data.
* `Properties.width` is a `u8` in 0..=2 (Zig `u2`).
* Not ported: `uucode.get(.case_folding_full)` (used by `input/Binding.zig`;
the input crate should use `char::to_lowercase`/own folding or ask for this
later), `x.grapheme.wcwidth` (unused by Ghostty), uucode's UTF-8 iterator,
and the `main()` brute-force verification binary in `grapheme.zig` (replaced
by the `GraphemeBreakTest.txt` test below).
## Tests
Zig tests ported (4/4): `grapheme break: emoji modifier`,
`long emoji zwj sequences`, `unicode props: tables match uucode`
(`unicode_props_tables_match_uucode`, compares the generated table with a
direct computation from the UCD files for every code point),
`unicode symbols: tables match uucode`. Extras: official
`GraphemeBreakTest.txt` (with controls skipped and the emoji-modifier
tailoring, like uucode's own test), width tests for ASCII/CJK/emoji/combining/
ZWJ/VS15/VS16/RI/keycap/Hangul/Prepend, cluster tests (flags, ZWJ families,
skin tones, Indic conjunct), precompute-vs-compute equality, lut round-trip.
## Known gaps / notes
* Build time of the crate is a few seconds (generator walks 2M code points).
* Updating Unicode: replace files in `ucd/` (see uucode `bin/fetch-ucd.sh`).