0 Fork 0

Files

T

ddidderr 3323c3f525 feat(unicode,datastruct): port Unicode tables and data structures

ghostty-unicode: Ghostty computes codepoint properties with the uucode
Zig package (v0.2.0, Unicode 17.0.0). There is no Rust equivalent, so
a trimmed UCD set is vendored in ucd/ and build.rs re-implements
uucode's derivations: cell width rules incl. Ghostty's overrides for
emoji modifiers and Prepend, grapheme break classes with the
compile-time grapheme_break state table, the is_symbol set, and emoji
presentation. Lookups are three array indexes per codepoint. Validated
against the official GraphemeBreakTest.txt.

ghostty-datastruct: split_tree (used for GTK splits; f16 ratios so
rounding and debug diagrams match Zig), circ_buf, lru, cache_table,
blocking_queue, segmented_pool, message_data, comparison,
array_list_collection. intrusive_linked_list is not needed (lru is
index based). lru now reports the evicted (old) key, Zig reported the
new key, which appears to be a bug.

Not yet covered: case_folding_full (needed by input bindings).

Test Plan:
- cargo test -p ghostty-unicode (16) -p ghostty-datastruct (88)
- cargo clippy for both crates with -D warnings

2026-10-11 21:54:57 +02:00

4.6 KiB

Raw Blame History

ghostty-unicode

Port of ZIG/src/unicode/*.

Module map

Zig	Rust (`crates/ghostty-unicode`)
`main.zig`	`src/lib.rs` (re-exports `table`, `Properties`, `grapheme_break`, `BreakState`)
`lut.zig`	`src/lut.rs` (3-stage table generator + lookup; also included by `build.rs`)
`props.zig`	`src/props.rs` (`Properties`, `GraphemeBreakNoControl`)
`props_table.zig`	`src/props_table.rs` (`table()`; generated into `$OUT_DIR`)
`props_uucode.zig`, `symbols_uucode.zig` (the generators)	`build.rs` + `build_support/ucd.rs` (UCD parser and uucode-equivalent computation)
`symbols_table.zig`	`src/symbols_table.rs`
`grapheme.zig`	`src/grapheme.rs`
(uucode `is_emoji_presentation`, used by font `CodepointResolver`)	`src/emoji_table.rs` (extra)

How the data is derived (uucode)

Ghostty gets its data from the uucode Zig package (v0.2.0, vendored UCD = Unicode 17.0.0), configured in ZIG/src/build/uucode_config.zig. Neither a Rust crate nor runtime data is used; instead the exact UCD files uucode ships (trimmed to the used properties) are vendored in ucd/ and build.rs re-implements uucode's computations, so results are identical to Ghostty:

ucd/extracted/DerivedGeneralCategory.txt, DerivedEastAsianWidth.txt (incl. @missing Wide ranges), auxiliary/GraphemeBreakProperty.txt, DerivedCoreProperties.txt (trimmed to Default_Ignorable_Code_Point and InCB), emoji/emoji-data.txt, emoji/emoji-variation-sequences.txt, Blocks.txt, auxiliary/GraphemeBreakTest.txt (tests only). (uucode uses UnicodeData.txt for general category; DerivedGeneralCategory carries the same values and is 10x smaller.)
Width (uucode x/config_x/wcwidth.zig + Ghostty computeWidth): Cc/Cs/Zl/Zp -> 0; U+00AD -> 1; Default_Ignorable -> 0; U+2E3A -> 2; U+2E3B -> 3 (clamped to 2 in width); EAW W/F -> 2; Regional Indicator -> 2; else 1; U+20E3 standalone = 2. width_zero_in_grapheme = width 0, emoji modifier, Mn, Me, GCB=V, GCB=T, or Prepend. Ghostty's final width: 0 if zero_in_grapheme && !emoji_modifier && gcb != prepend, else min(2, standalone) (so emoji modifiers and Prepend characters keep a width; VS15/VS16/ZWJ/combining marks are 0).
Grapheme break property (grapheme_break_no_control): emoji modifier, emoji modifier base, extended pictographic, then InCB (linker / consonant / extend; ZWJ and ZWNJ split out), else GCB with control/CR/LF folded into other (the terminal filters controls first, as in Zig).
graphemeBreak(cp1, cp2, state): uucode's computeGraphemeBreakNoControl (UAX #29 incl. GB9c Indic conjuncts, GB11, GB12/13), with Ghostty/uucode's tailoring that emoji modifiers only extend an emoji modifier base. Precomputed at compile time (const fn) into a 5 x 17 x 17 table. grapheme_break_props(gb1, gb2, state) takes already-looked-up properties (hot path for the terminal).
is_symbol (symbols_table::table()): general category Co or block in {Arrows, Dingbats, Emoticons, Miscellaneous Symbols, Enclosed Alphanumerics, Enclosed Alphanumeric Supplement, Miscellaneous Symbols and Pictographs, Transport and Map Symbols}.
Code points above U+10FFFF return {width: 1, width_zero_in_grapheme: true, Other, emoji_vs_base: false} (as in props_uucode.get).

API differences

Code points are u32 (Zig u21). table().get(cp) is branch-free: three array loads. table() returns a tiny Tables view over static data.
Properties.width is a u8 in 0..=2 (Zig u2).
Not ported: uucode.get(.case_folding_full) (used by input/Binding.zig; the input crate should use char::to_lowercase/own folding or ask for this later), x.grapheme.wcwidth (unused by Ghostty), uucode's UTF-8 iterator, and the main() brute-force verification binary in grapheme.zig (replaced by the GraphemeBreakTest.txt test below).

Tests

Zig tests ported (4/4): grapheme break: emoji modifier, long emoji zwj sequences, unicode props: tables match uucode (unicode_props_tables_match_uucode, compares the generated table with a direct computation from the UCD files for every code point), unicode symbols: tables match uucode. Extras: official GraphemeBreakTest.txt (with controls skipped and the emoji-modifier tailoring, like uucode's own test), width tests for ASCII/CJK/emoji/combining/ ZWJ/VS15/VS16/RI/keycap/Hangul/Prepend, cluster tests (flags, ZWJ families, skin tones, Indic conjunct), precompute-vs-compute equality, lut round-trip.

Known gaps / notes

Build time of the crate is a few seconds (generator walks 2M code points).
Updating Unicode: replace files in ucd/ (see uucode bin/fetch-ucd.sh).

4.6 KiB Raw Blame History