feat(terminal): bring PageList to full parity with Ghostty

Second pass over the page memory model (done by Codex/gpt-6-astra in an
isolated worktree, reviewed and merged here):
- Resize/reflow now port Zig's incremental ReflowCursor algorithm with
  page backfill and the same capacity growth/split sequence, instead of
  rebuilding a snapshot, so page boundaries, pruning and tracked-pin
  remapping match Ghostty.
- Point<->pin conversion for the active area starts at the list tail and
  the viewport offset is cached, so hot-path operations no longer walk
  the whole scrollback from the head.
- RefCountedSet reproduces Zig's observable ID reuse and error points.
- All 202 PageList tests are ported (plus 13 extra regressions).

With this, the five Screen/Terminal tests previously ignored as blocked
on PageList behaviour pass and are re-enabled.

Test Plan:
- cargo test -p ghostty-terminal (913 passed, 0 ignored)
- cargo clippy -p ghostty-terminal --all-targets -- -D warnings
This commit is contained in:
ddidderr committed 2026-10-11 22:41:14 +02:00
1 parent e495ae72f1
commit ed7c644438
9 files changed
+4613 -1770

No files matched your search

+115 -170
View File
@@ -1,117 +1,111 @@
# ghostty-terminal page memory model — partial port
# ghostty-terminal page memory model
This is **not a completed Ghostty PageList parity port**. The core structures and
many operations are implemented, but the remaining PageList cases below must be
ported and used to drive corrections before Screen/Terminal relies on exact Zig
reflow, allocation, and viewport behavior. In particular, passing the current
crate tests does not establish equivalence of page boundaries under reflow.
The PageList parity pass is complete. All 50 previously missing PageList tests
are ported. Earlier consolidated tests now use the reference dimensions,
capacity-derived page boundaries, operations, and assertions. Shared fixture and
assertion helpers retain the complete source checks, including managed values,
page links, dirty/wrap flags, viewport offsets, and tracked locations.
No version bump or commit was made. Only `ghostty-vt` is required; no unicode,
datastruct, or config dependency was introduced. Reference files were read only.
No public signatures were removed or changed. No version bump or commit was
made. Only the assigned source files and these notes were changed. The Zig
reference remained read-only; no dependency or production module was added.
## API changes
**No breaking public API changes.** Additions to `RefCountedSet`:
- `init_with_hasher(cap, fn(&T) -> u64)` supplies deterministic probe ordering.
- `clear()` resets entries while retaining allocated storage and hash context.
The existing PageList, Page, Point, Pin, TrackedPin, Style, and Hyperlink APIs
remain available to the concurrent Screen port. New copy/reflow helpers are
private or crate-private.
## Module and test inventory
| Zig source | Rust module | Source cases represented | Extra Rust tests | Unported / skipped |
|---|---|---:|---:|---:|
| size.zig | size | 1 / 5 | 0 | 4 pointer-only tests skipped |
| point.zig | point | 0 / 0 | 0 | 0 |
| style.zig | style | 37 / 37 | 0 | 0 |
| hyperlink.zig | hyperlink | 0 / 0 | 0 | 0 |
| ref_counted_set.zig | ref_counted_set | 0 / 0 | 2 | 0 |
| page.zig | page | 46 / 46 | 0 | 0 |
| PageList.zig | page_list | 151 / 202 | 2 | 50 unported, 1 skipped |
| Zig source | Rust module | Source cases ported | Extra Rust tests | Zig-only skips | Rust total |
|---|---|---:|---:|---:|---:|
| size.zig | size | 1 / 5 | 0 | 4 | 1 |
| point.zig | point | 0 / 0 | 0 | 0 | 0 |
| style.zig | style | 37 / 37 | 0 | 0 | 37 |
| hyperlink.zig | hyperlink | 0 / 0 | 2 | 0 | 2 |
| ref_counted_set.zig | ref_counted_set | 0 / 0 | 7 | 0 | 7 |
| page.zig | page | 46 / 46 | 0 | 0 | 46 |
| PageList.zig | page_list | 201 / 202 | 4 | 1 | 205 |
Total: **239 passing Rust tests**, consisting of 235 source-named regression
cases and four added regressions. Some PageList cases use smaller equivalent
fixtures and consolidate their assertions; matching a test name is not a claim
that all incidental assertions or allocation paths in that Zig case were copied.
Style formatter fixtures retain their reference expected strings. All 17
semantic-highlight cases retain the reference grid setup and expected endpoints.
Total: **298 passing Rust tests**, including 285 source cases and 13 additional
regressions. No functional source test remains unported. Style formatter fixtures
retain the source strings; all 17 semantic-highlight tests retain their grids and
endpoints. Tests with repeated cases use loops without dropping checks.
Intentional skips:
The guide explicitly permits these Zig-only skips:
- `Offset ptr u8`, `Offset ptr structural`, `getOffset bytes`, `getOffset structs`:
raw-address/base-pointer machinery was replaced with owned storage and logical
cell indices. `Offset<T>` is a typed u32 logical offset; it does not dereference
page-relative pointers.
- `PageList init error`: Zig tripwire/allocator failure injection has no Rust
counterpart here. Fallible grid reservations return errors, but allocation
failure coverage is incomplete.
- `bitmap_allocator.zig` (21 tests) and `hash_map.zig` (22 tests) are not ported
as modules, as requested. Their logical accounting is implemented in page.rs.
raw base-address dereferencing is replaced by safe owned storage and logical
offsets. There are no page-relative Rust pointers to test.
- `PageList init error`: the Zig tripwire allocator/failure-injection harness has
no counterpart in this safe Rust ownership model. Logical capacity errors are
covered; process allocator failure injection is not ported.
## Implemented structures and behavior
`bitmap_allocator.zig` and `hash_map.zig` are not new modules. Their bounded
storage/layout responsibilities remain in page.rs as required by the scope.
- Cell and Row are transparent u64 wrappers, Copy + Clone, exactly eight bytes.
Cell preserves the Zig bit offsets: tag 0, payload 2, style ID 26, wide 42,
protected 44, hyperlink 45, semantic content 46. Row stores a cell index in its
low 32 bits and uses the same flag positions as Zig. Row cell indices are not
byte offsets; rotating rows moves metadata and preserves auxiliary-map keys.
- Page owns row/cell Vecs, a deduplicated style set, grapheme data and allocation
maps, a hyperlink map/set, and logical string/grapheme chunk allocators.
- `Page::layout` preserves the reference 64-bit Linux layout calculation,
including the unusual bitmap allocator formula `aligned_cap * chunk_size`,
bitmap rounding to 64 chunks, map headers even at zero capacity, set 13/16
load factor and reserved ID zero, and 4096-byte final rounding. A standalone
Zig ABI probe confirmed Style item size/alignment 36/4, hyperlink item 48/8,
and offset-slice 16/8. The layout does not report actual Rust heap overhead.
- Logical allocation tracks contiguous chunks, including temporary allocation
of a larger grapheme before freeing its old chunks. Grapheme and hyperlink
maps enforce separate entry limits. Dead hyperlink entries retain their
string chunks until their set slot is reclaimed.
- Page provides styles, graphemes, hyperlinks, copy/clone, partial-row copying,
moves, swaps, clearing, exact row capacity, and debug/test verification.
Unmanaged partial-row copies use a slice copy without per-cell maps or
allocation. Default styles bypass set lookup.
- PageList uses `Vec<Option<Node>>`, a free-index list, generation-bearing
NodeId, and a separate reusable standard-page pool. It accounts substandard
pages as standard pool allocations. Growing prunes whole history pages while
protecting the active rows. Reused node indices receive new generations.
- PageList has list-owned tracked handles, viewport scrolling, byte/row totals,
splitting, capacity growth, compaction, erasure, reset, history clearing,
cloning, row/column resize, reflow, prompt iteration/highlighting, and forward
or reverse page/row/cell iterators. Iterators also implement DoubleEndedIterator
where appropriate. PromptIterator follows continuation handling from Zig.
## Implementation
## Important deviations / remaining work
- Cell and Row remain transparent eight-byte u64 wrappers with the reference
field positions. Row cell references are logical indices, so rotating row
metadata preserves auxiliary keys without moving cell contents.
- Page retains the reference logical layout and allocation capacities: 64-bit
Linux, 4096-byte page rounding, bitmap chunk rounding, map headers, 13/16 set
load factor, reserved ID zero, and managed-storage item sizes. Logical byte
accounting intentionally does not measure Rust collection overhead.
- Graphemes use a page-owned codepoint arena and preallocated maps. Appending
across a chunk boundary allocates the new logical block before releasing the
old one. Copying/reflowing a grapheme allocates no per-cell Vec. Hyperlink
duplicates borrow source values, cloning owned strings only for a new
persistent destination entry. String allocations retain Zig's chunk limits
and dead-entry lifetimes, including deletion during a failed insertion.
- RefCountedSet uses a Vec-backed Robin Hood table with the reference probe
ordering, reference-count tie break, backwards-shift deletion, trailing dead-ID
trimming, encountered dead-ID reuse, preferred IDs, 32-probe ceiling, and
NeedsRehash/OutOfMemory decision order. The observable living-count behavior
of a dead preferred-ID upsert is retained and tested. Page style hashes use
Zig's packed 128-bit fold/integer hash; hyperlink hashes use its Wyhash/tag/
slice-length encoding. Wyhash block-boundary and hyperlink encoding vectors
were checked against a standalone Zig probe.
- PageList uses generation-bearing node and tracked-pin handles. Standard-sized
pages are recycled, oversized pages are released, and byte/row accounting
follows the reference allocation/pruning rules. Reset releases excess pooled
history. Split, compact, and capacity growth allocate one destination page and
clone directly into it.
- Reflow follows resizeCols/ReflowCursor: column/row resize ordering, inherited
capacities, deferred blank rows, cursor wrap-count preservation, cell-by-cell
tracked-pin remapping, wrap and continuation metadata, wide/spacer handling,
graphemes, styles, hyperlinks, and Kitty placeholder metadata. Capacity grows
in the reference sequence; an exhausted page splits off the current final row
and retries. Consumed source pages are released immediately. There are no
temporary row snapshots or per-cell temporary heap allocations in reflow.
- Widening without reflow uses the reference capacity fast path and spacer-head
fallback, backfills previous pages, remaps backfilled pins, and creates the
required remaining pages. Shrinking clears truncated managed cells and clamps
pins. Background-only trailing rows follow the reference trimming behavior.
- Active point lookup starts at the tail, Viewport lookup starts at its anchor,
and pin-to-point walks only from the requested region's origin. Pin movement
and ordering traverse adjacent nodes. Pinned viewport offsets are cached and
updated/invalidated during scrolling, pruning, erasure, and resize. Grow never
computes an uncached viewport offset. Debug integrity checks validate cached
offsets against the actual list.
- Row erasure rotates row metadata, copies only cross-page boundary rows, and
remaps pins in place. Active range erasure regrows the removed rows even when
history already supplies enough total rows. Clone preserves source capacities,
chunk boundaries, managed data, dirty flags, and optional pin remapping.
1. **Reflow and wider no-reflow resize rebuild a snapshot of rows into new pages.**
They do not yet implement Zig's incremental ReflowCursor and page backfill
algorithm. New page auxiliary capacities are sized from encountered content,
instead of following the exact sequence of capacity doubling and splitting.
Page boundaries, intermediate allocations, pruning, and maximum-size behavior
can differ. This is the most important unfinished work.
2. Snapshot reflow is linear in all cells and allocates temporary row vectors,
copies managed values, and scans tracked locations while copying. It is not
ready for the intended terminal hot-path performance requirements. Point
conversion and viewport offsets traverse the list; the Zig active/viewport
caches have not been reproduced.
3. Cursor preservation during combined row/column resize is incomplete. Basic
tracked cursor positions, blank-cell clamping, trailing blank trimming, and
several wide/spacer cases are tested, but cross-page cursors, old/new wrap
counts, viewport remapping under reflow, and all combinations need the
remaining reference cases.
4. The HashMap-backed RefCountedSet preserves zero reservation, bounded live
slots, sequential IDs, trailing dead-slot trimming, `add_with_id`, checked
reference counts, and OutOfMemory/NeedsRehash errors. It does **not** reproduce
Robin Hood probe-dependent dead-ID reuse or the 32-probe failure threshold.
Thus exact reused IDs and the point at which NeedsRehash occurs can differ.
Hash values are Rust implementation details; Zig Style/Wyhash output is not
reproduced. Style implements Hash/Eq for the Rust deduplication map.
5. Direct Page clone uses Rust Clone. Some auxiliary temporary allocations and
owned-value clones still use infallible standard collection operations.
Actual process allocator exhaustion is not completely surfaced as Result;
logical capacity exhaustion is. Allocation-failure atomicity has not been
fully audited. There is no unsafe code or todo!/unimplemented! path.
6. The row-range eraser supports arbitrary start ranges, while Zig's tested
eraseRows use is primarily prefix/history/active erasure. Exact destination
pins for arbitrary whole-page removal in the middle need further parity work.
7. `style::BoldColor` is a small local policy enum because terminal must not
depend on config. TODO(integration): convert the configuration type at the
Screen boundary. Rgb/Palette/Underline are reused from ghostty-vt.
8. Logical layout is for 64-bit Linux with a 4096-byte minimum page size. Other
ABIs are not implemented. Pool reuse reuses Pages with matching capacities;
it does not share a mmap allocator between independent PageLists.
The safe Rust representation replaces mmap/base-pointer machinery with owned
allocations. Allocation addresses are therefore not compared in the nonstandard
page reuse test; node generations and live page/pool membership verify the same
lifetime property. No unsafe code or reachable todo!/unimplemented! path was
introduced. `style::BoldColor` remains the local policy type mandated by the
terminal/config dependency boundary; Screen converts configuration at integration.
## API overview for Screen / Terminal agents
@@ -120,8 +114,13 @@ Intentional skips:
`Point::{Active,Viewport,Screen,History}(Coordinate { x: u16, y: u32 })`
selects the coordinate frame. `Point::new(Tag, Coordinate)` is useful for dynamic
frames. `pin(point)` and `point_from_pin(tag, pin)` return Option for bounds errors.
`get_top_left` and `get_bottom_right` return None for an empty region such as
history before the first scroll.
`get_top_left(History)` is the screen origin even without history, as in Zig;
`get_bottom_right(History)` returns None when history is empty. Conversion starts
at the selected frame and may continue through subsequent rows in the list, as
in Zig; it does not impose an extra frame-height bound. Active lookup traverses
backward from the tail; Viewport lookup starts at its active, top, or tracked
anchor. `viewport_row_offset(&self)` caches pinned offsets with an atomic value,
preserving the existing immutable API and Send/Sync traits.
`Pin { node: NodeId, y: u16, x: u16, garbage: bool }` is Copy and contains no
references. Pass the owning list explicitly to its access and movement methods:
@@ -170,8 +169,9 @@ new raw Cell. Direct writes must maintain row flags and dirty tracking.
`row.set_dirty(true)` / `pin.mark_dirty(&mut list)` support direct writes;
`list.clear_dirty()` resets page and row flags. `is_dirty` checks both levels.
Conservative managed-row flags may be true even after their last value is cleared.
Use the explicit `update_row_*_flag` methods when a precise flag is needed.
`clear_cells` updates managed-row flags using the reference full/partial clear
rules. Individual clear helpers may leave conservative flags set; use the
explicit `update_row_*_flag` methods when a precise flag is needed.
### Errors and capacity recovery
@@ -197,7 +197,8 @@ Always use the returned NodeId, or resolve a tracked pin again after replacement
`scrollbar()` reports row counts and hides physical history at max_size zero.
`scroll_clear()` moves the nonempty active rows into history.
`erase_rows(top, optional_bottom)` removes inclusive rows and ensures active height.
`erase_rows(top, optional_bottom)` removes inclusive rows. Active erasure regrows
every removed row, preserving history; other frames retain at least active height.
`erase_row(point)` shifts the remainder of the screen while retaining page sizes.
`erase_row_bounded(point,limit)` shifts at most limit following rows.
`erase_cells_bounded(start_pin,end_pin,left,right)` is an extra rectangular helper.
@@ -207,7 +208,9 @@ preserves the active area.
`resize(Resize { cols, rows, reflow, cursor })` takes optional new dimensions and
an optional active `Cursor { x,y }`. Reflow defaults to true. Cursor is input
context; track the actual cursor pin separately to read back its new position.
Consult the limitations above before relying on complex resize behavior.
Reflow writes incrementally with a ReflowCursor. Widening without reflow first
backfills the preceding page, then splits into pages using adjusted source
capacities. Both paths remap tracked handles as rows/cells move.
`clone_range(top,bottom)` clones inclusive **whole rows**, ignoring x, pads to the
active height, and returns `(PageList, HashMap<old_handle,new_handle>)`.
@@ -232,73 +235,15 @@ and test builds, as are assert_integrity helpers. Nestable pause methods support
batched low-level updates. Verification is explicit; raw mutable access can
break invariants until the caller completes its update.
## Remaining PageList source tests
The following 50 cases are **unported**, rather than justified skips. They include
important production behavior and should drive the next implementation pass.
- `PageList grow allows exceeding max size for active area`
- `PageList grow prune required with a single page`
- `PageList scroll delta row back without space preserves active`
- `PageList: jump zero prompts`
- `Screen: jump back one prompt`
- `Screen: jump forward prompt skips multiline continuation`
- `PageList eraseRows invalidates viewport offset cache`
- `PageList eraseRow invalidates viewport offset cache`
- `PageList eraseRowBounded invalidates viewport offset cache`
- `PageList eraseRowBounded multi-page invalidates viewport offset cache`
- `PageList eraseRowBounded full page shift invalidates viewport offset cache`
- `PageList eraseRowBounded exhausts pages invalidates viewport offset cache`
- `PageList promptIterator right_down continuation at start`
- `PageList promptIterator right_down with prompt before continuation`
- `PageList promptIterator right_down limit inclusive`
- `PageList promptIterator left_up limit inclusive`
- `PageList erase resets viewport to active if moves within active`
- `PageList erase resets viewport if inside erased page but not active`
- `PageList erase resets viewport to active if top is inside active`
- `PageList resize (no reflow) less rows cursor on bottom`
- `PageList resize (no reflow) less rows cursor in scrollback`
- `PageList resize (no reflow) more rows contains viewport`
- `PageList resize (no reflow) more cols with spacer head`
- `PageList resize (no reflow) grow cols fast path with spacer head`
- `PageList resize (no reflow) more cols forces less rows per page`
- `PageList resize more rows and cols doesn't fit in single std page`
- `PageList resize (no reflow) more cols forces smaller cap`
- `PageList resize (no reflow) more rows adds blank rows if cursor at bottom`
- `PageList resize reflow invalidates viewport offset cache`
- `PageList resize reflow more cols creates multiple pages`
- `PageList resize reflow more cols wrap across page boundary`
- `PageList resize reflow more cols wrap across page boundary cursor in second page`
- `PageList resize reflow less cols wrap across page boundary cursor in second page`
- `PageList resize reflow more cols cursor in wrapped row`
- `PageList resize reflow more cols cursor in not wrapped row`
- `PageList resize reflow more cols cursor in wrapped row that isn't unwrapped`
- `PageList resize reflow exceeds hyperlink memory forcing capacity increase`
- `PageList resize reflow exceeds grapheme memory forcing capacity increase`
- `PageList resize reflow exceeds style memory forcing capacity increase`
- `PageList resize reflow more cols unwrap wide spacer head`
- `PageList resize reflow more cols unwrap wide spacer head across two rows`
- `PageList resize reflow less cols cursor goes to scrollback`
- `PageList resize reflow less cols copy kitty placeholder`
- `PageList resize reflow more cols clears kitty placeholder`
- `PageList resize reflow wrap moves kitty placeholder`
- `PageList resize reflow grapheme map capacity exceeded`
- `PageList resize grow cols with unwrap fixes viewport pin`
- `PageList grow reuses non-standard page without leak`
- `PageList grow non-standard page prune protection`
- `PageList resize (no reflow) more cols remaps pins in backfill path`
## Quality gate
Final gate (2026-10-11):
- `cargo fmt -p ghostty-terminal`: passed; only the documented nightly-option
warnings from workspace rustfmt configuration.
- `cargo build -p ghostty-terminal`: passed.
- `cargo test -p ghostty-terminal --quiet`: 239 passed, 0 failed, 0 ignored.
- `cargo test -p ghostty-terminal`: 298 passed, 0 failed, 0 ignored; doc tests passed.
- `cargo clippy -p ghostty-terminal --all-targets -- -D warnings`: passed.
- `git diff --check -- crates/ghostty-terminal docs/rewrite/notes/ghostty-terminal-pagelist.md`:
passed for tracked changes.
- `cargo fmt -p ghostty-terminal`: passed (only workspace nightly-option warnings).
- `git diff --check`: passed.
The early missing `ghostty-vt::stream` dependency file was resolved by concurrent
workspace work; no change to that crate was needed here. The terminal Cargo.toml
already had local changes at the beginning of this task and was not edited here.
Remaining functional limitations or missing PageList cases: **none**. The five
Zig-only test skips and ownership/ABI representation choices are documented above.