feat(terminal): port page memory model (partial PageList)
First cut of the terminal core memory model from ZIG/src/terminal: size, point, style, hyperlink, ref_counted_set, page and PageList. Design (see docs/rewrite/ARCHITECTURE.md decisions 2-4): - Cell/Row are 8-byte u64 wrappers with Zig's exact bit offsets. - Page owns row/cell Vecs plus style set, grapheme and hyperlink maps; Page::layout reproduces Ghostty's byte-size formula on 64-bit Linux so scrollback-limit pruning matches. - PageList keeps nodes in a generation-checked slab with a free list and list-owned tracked pins addressed by handles. This is explicitly partial (docs/rewrite/notes/ghostty-terminal- pagelist.md): 151/202 PageList tests ported; reflow rebuilds a snapshot instead of Zig's incremental ReflowCursor; point<->pin conversion walks from the list head (no active/viewport caches); RefCountedSet doesn't reproduce probe-dependent ID reuse. A follow-up pass finishes parity. Test Plan: - cargo test -p ghostty-terminal (239 tests)
This commit is contained in:
1 parent
09cbe3bc97
commit
6cf16ed2e6
11 files changed
+8817
-3
No files matched your search
@@ -0,0 +1,304 @@
|
||||
# ghostty-terminal page memory model — partial port
|
||||
|
||||
This is **not a completed Ghostty PageList parity port**. The core structures and
|
||||
many operations are implemented, but the remaining PageList cases below must be
|
||||
ported and used to drive corrections before Screen/Terminal relies on exact Zig
|
||||
reflow, allocation, and viewport behavior. In particular, passing the current
|
||||
crate tests does not establish equivalence of page boundaries under reflow.
|
||||
|
||||
No version bump or commit was made. Only `ghostty-vt` is required; no unicode,
|
||||
datastruct, or config dependency was introduced. Reference files were read only.
|
||||
|
||||
## Module and test inventory
|
||||
|
||||
| Zig source | Rust module | Source cases represented | Extra Rust tests | Unported / skipped |
|
||||
|---|---|---:|---:|---:|
|
||||
| size.zig | size | 1 / 5 | 0 | 4 pointer-only tests skipped |
|
||||
| point.zig | point | 0 / 0 | 0 | 0 |
|
||||
| style.zig | style | 37 / 37 | 0 | 0 |
|
||||
| hyperlink.zig | hyperlink | 0 / 0 | 0 | 0 |
|
||||
| ref_counted_set.zig | ref_counted_set | 0 / 0 | 2 | 0 |
|
||||
| page.zig | page | 46 / 46 | 0 | 0 |
|
||||
| PageList.zig | page_list | 151 / 202 | 2 | 50 unported, 1 skipped |
|
||||
|
||||
Total: **239 passing Rust tests**, consisting of 235 source-named regression
|
||||
cases and four added regressions. Some PageList cases use smaller equivalent
|
||||
fixtures and consolidate their assertions; matching a test name is not a claim
|
||||
that all incidental assertions or allocation paths in that Zig case were copied.
|
||||
Style formatter fixtures retain their reference expected strings. All 17
|
||||
semantic-highlight cases retain the reference grid setup and expected endpoints.
|
||||
|
||||
Intentional skips:
|
||||
|
||||
- `Offset ptr u8`, `Offset ptr structural`, `getOffset bytes`, `getOffset structs`:
|
||||
raw-address/base-pointer machinery was replaced with owned storage and logical
|
||||
cell indices. `Offset<T>` is a typed u32 logical offset; it does not dereference
|
||||
page-relative pointers.
|
||||
- `PageList init error`: Zig tripwire/allocator failure injection has no Rust
|
||||
counterpart here. Fallible grid reservations return errors, but allocation
|
||||
failure coverage is incomplete.
|
||||
- `bitmap_allocator.zig` (21 tests) and `hash_map.zig` (22 tests) are not ported
|
||||
as modules, as requested. Their logical accounting is implemented in page.rs.
|
||||
|
||||
## Implemented structures and behavior
|
||||
|
||||
- Cell and Row are transparent u64 wrappers, Copy + Clone, exactly eight bytes.
|
||||
Cell preserves the Zig bit offsets: tag 0, payload 2, style ID 26, wide 42,
|
||||
protected 44, hyperlink 45, semantic content 46. Row stores a cell index in its
|
||||
low 32 bits and uses the same flag positions as Zig. Row cell indices are not
|
||||
byte offsets; rotating rows moves metadata and preserves auxiliary-map keys.
|
||||
- Page owns row/cell Vecs, a deduplicated style set, grapheme data and allocation
|
||||
maps, a hyperlink map/set, and logical string/grapheme chunk allocators.
|
||||
- `Page::layout` preserves the reference 64-bit Linux layout calculation,
|
||||
including the unusual bitmap allocator formula `aligned_cap * chunk_size`,
|
||||
bitmap rounding to 64 chunks, map headers even at zero capacity, set 13/16
|
||||
load factor and reserved ID zero, and 4096-byte final rounding. A standalone
|
||||
Zig ABI probe confirmed Style item size/alignment 36/4, hyperlink item 48/8,
|
||||
and offset-slice 16/8. The layout does not report actual Rust heap overhead.
|
||||
- Logical allocation tracks contiguous chunks, including temporary allocation
|
||||
of a larger grapheme before freeing its old chunks. Grapheme and hyperlink
|
||||
maps enforce separate entry limits. Dead hyperlink entries retain their
|
||||
string chunks until their set slot is reclaimed.
|
||||
- Page provides styles, graphemes, hyperlinks, copy/clone, partial-row copying,
|
||||
moves, swaps, clearing, exact row capacity, and debug/test verification.
|
||||
Unmanaged partial-row copies use a slice copy without per-cell maps or
|
||||
allocation. Default styles bypass set lookup.
|
||||
- PageList uses `Vec<Option<Node>>`, a free-index list, generation-bearing
|
||||
NodeId, and a separate reusable standard-page pool. It accounts substandard
|
||||
pages as standard pool allocations. Growing prunes whole history pages while
|
||||
protecting the active rows. Reused node indices receive new generations.
|
||||
- PageList has list-owned tracked handles, viewport scrolling, byte/row totals,
|
||||
splitting, capacity growth, compaction, erasure, reset, history clearing,
|
||||
cloning, row/column resize, reflow, prompt iteration/highlighting, and forward
|
||||
or reverse page/row/cell iterators. Iterators also implement DoubleEndedIterator
|
||||
where appropriate. PromptIterator follows continuation handling from Zig.
|
||||
|
||||
## Important deviations / remaining work
|
||||
|
||||
1. **Reflow and wider no-reflow resize rebuild a snapshot of rows into new pages.**
|
||||
They do not yet implement Zig's incremental ReflowCursor and page backfill
|
||||
algorithm. New page auxiliary capacities are sized from encountered content,
|
||||
instead of following the exact sequence of capacity doubling and splitting.
|
||||
Page boundaries, intermediate allocations, pruning, and maximum-size behavior
|
||||
can differ. This is the most important unfinished work.
|
||||
2. Snapshot reflow is linear in all cells and allocates temporary row vectors,
|
||||
copies managed values, and scans tracked locations while copying. It is not
|
||||
ready for the intended terminal hot-path performance requirements. Point
|
||||
conversion and viewport offsets traverse the list; the Zig active/viewport
|
||||
caches have not been reproduced.
|
||||
3. Cursor preservation during combined row/column resize is incomplete. Basic
|
||||
tracked cursor positions, blank-cell clamping, trailing blank trimming, and
|
||||
several wide/spacer cases are tested, but cross-page cursors, old/new wrap
|
||||
counts, viewport remapping under reflow, and all combinations need the
|
||||
remaining reference cases.
|
||||
4. The HashMap-backed RefCountedSet preserves zero reservation, bounded live
|
||||
slots, sequential IDs, trailing dead-slot trimming, `add_with_id`, checked
|
||||
reference counts, and OutOfMemory/NeedsRehash errors. It does **not** reproduce
|
||||
Robin Hood probe-dependent dead-ID reuse or the 32-probe failure threshold.
|
||||
Thus exact reused IDs and the point at which NeedsRehash occurs can differ.
|
||||
Hash values are Rust implementation details; Zig Style/Wyhash output is not
|
||||
reproduced. Style implements Hash/Eq for the Rust deduplication map.
|
||||
5. Direct Page clone uses Rust Clone. Some auxiliary temporary allocations and
|
||||
owned-value clones still use infallible standard collection operations.
|
||||
Actual process allocator exhaustion is not completely surfaced as Result;
|
||||
logical capacity exhaustion is. Allocation-failure atomicity has not been
|
||||
fully audited. There is no unsafe code or todo!/unimplemented! path.
|
||||
6. The row-range eraser supports arbitrary start ranges, while Zig's tested
|
||||
eraseRows use is primarily prefix/history/active erasure. Exact destination
|
||||
pins for arbitrary whole-page removal in the middle need further parity work.
|
||||
7. `style::BoldColor` is a small local policy enum because terminal must not
|
||||
depend on config. TODO(integration): convert the configuration type at the
|
||||
Screen boundary. Rgb/Palette/Underline are reused from ghostty-vt.
|
||||
8. Logical layout is for 64-bit Linux with a 4096-byte minimum page size. Other
|
||||
ABIs are not implemented. Pool reuse reuses Pages with matching capacities;
|
||||
it does not share a mmap allocator between independent PageLists.
|
||||
|
||||
## API overview for Screen / Terminal agents
|
||||
|
||||
### Coordinate and lifetime model
|
||||
|
||||
`Point::{Active,Viewport,Screen,History}(Coordinate { x: u16, y: u32 })`
|
||||
selects the coordinate frame. `Point::new(Tag, Coordinate)` is useful for dynamic
|
||||
frames. `pin(point)` and `point_from_pin(tag, pin)` return Option for bounds errors.
|
||||
`get_top_left` and `get_bottom_right` return None for an empty region such as
|
||||
history before the first scroll.
|
||||
|
||||
`Pin { node: NodeId, y: u16, x: u16, garbage: bool }` is Copy and contains no
|
||||
references. Pass the owning list explicitly to its access and movement methods:
|
||||
`p.row_and_cell(&list)`, `p.style(&list)`, `p.grapheme(&list)`,
|
||||
`p.cells(&list, CellSubset::All)`, `p.up(&list, n)`, `p.right_wrap(&list, n)`, etc.
|
||||
`CellSubset::Left` and `Right` both include the pin column. Vertical overflow
|
||||
methods return `Overflow::{Offset,Overflow { end, remaining }}`. `diagonal` uses
|
||||
signed deltas and clamps columns. `before`/`is_between` use row-major order.
|
||||
|
||||
Untracked pins must be reacquired after content-moving mutations. For a cursor,
|
||||
selection endpoint, or saved location, call `track_pin(pin) -> Result<TrackedPin>`.
|
||||
Read its current Copy value using `tracked_pin(handle)`; change its position with
|
||||
`set_tracked_pin(handle, pin)`. Release with `untrack_pin(handle)`. Handles are
|
||||
specific to the owning list. Recycled handles have a new generation, so releasing
|
||||
an old handle does not release its successor. Pruning/reset mark external pins
|
||||
as garbage; ordinary erasure remaps them without setting garbage. The list owns
|
||||
one internal tracked viewport handle, included in `count_tracked_pins()`.
|
||||
|
||||
### Reading and writing cells
|
||||
|
||||
`PageList::get_cell(point)` returns a Copy Cell, and `get_cell_mut(point)` returns
|
||||
a direct mutable reference. `node(id)` / `node_mut(id)` expose the page.
|
||||
`Page::get_row(y)`, `get_cells(y)`, `get_cells_mut(y)`, and `cell_index(x,y)` use
|
||||
logical row coordinates and preserve O(1) row access even after row rotation.
|
||||
Auxiliary lookup methods take a **cell index** returned by cell_index, not a
|
||||
column or pointer. The vectors include unused capacity; iteration should respect
|
||||
`page.size`, not the vector lengths.
|
||||
|
||||
Construct plain text with `Cell::init(codepoint)`. Set flags with `set_wide`,
|
||||
`set_protected`, `set_semantic_content`, etc. `set_codepoint` changes only the
|
||||
payload; use `set_content_tag` when changing between text and background colors.
|
||||
For managed cells, clear the old data with page operations before assigning a
|
||||
new raw Cell. Direct writes must maintain row flags and dirty tracking.
|
||||
|
||||
- `set_style(x,y,Style)` handles style add/release and marks the row styled.
|
||||
Default style is ID 0; `page.style(cell)` resolves it without map access.
|
||||
- `append_grapheme(x,y,cp)` and `set_graphemes(x,y,&extra)` store only codepoints
|
||||
beyond the cell's first codepoint. Set the first nonzero codepoint beforehand.
|
||||
- `insert_hyperlink(Hyperlink)` returns an already-retained ID. Attach it with
|
||||
`set_hyperlink(x,y,id)`, which consumes that reference without incrementing it.
|
||||
For another cell, call `page.hyperlink_set.r#use(id)` first. If attachment fails,
|
||||
release the retained ID. `clear_hyperlink(index)` releases the cell reference.
|
||||
- `clear_cells(y,start,end,protect)` uses half-open columns. `protect=true` is a
|
||||
Rust convenience that skips protected cells. Move and swap helpers transfer
|
||||
auxiliary ownership and avoid changing reference counts for moved values.
|
||||
|
||||
`row.set_dirty(true)` / `pin.mark_dirty(&mut list)` support direct writes;
|
||||
`list.clear_dirty()` resets page and row flags. `is_dirty` checks both levels.
|
||||
Conservative managed-row flags may be true even after their last value is cleared.
|
||||
Use the explicit `update_row_*_flag` methods when a precise flag is needed.
|
||||
|
||||
### Errors and capacity recovery
|
||||
|
||||
`page::Error` includes separate style set, grapheme allocator/map, hyperlink
|
||||
set/map, and string exhaustion variants. `ref_counted_set::AddError` distinguishes
|
||||
OutOfMemory from NeedsRehash, plus invalid IDs/counts for safe API misuse handling.
|
||||
`RefCountedSet::add_with_id(value,id)` returns None when the requested ID was used,
|
||||
otherwise Some(actual_id). `get` and `lookup` do not retain values.
|
||||
|
||||
`increase_capacity(node, Some(IncreaseCapacity::{Styles,GraphemeBytes,
|
||||
HyperlinkBytes,StringBytes}))` replaces a page with increased limits and remaps
|
||||
tracked pins; None rebuilds the same capacity to reclaim dead IDs.
|
||||
`adjust_capacity` is an alias. OutOfSpace means doubling/page-size limits prevent
|
||||
further growth; `split(pin)` can divide at a row boundary. Splitting row zero is
|
||||
otherwise a no-op, and a single-row page returns OutOfSpace. `compact(node)`
|
||||
returns Some(replacement) only when an oversized page can shrink.
|
||||
Always use the returned NodeId, or resolve a tracked pin again after replacement.
|
||||
|
||||
### List mutations and iteration
|
||||
|
||||
`grow()` adds one row, returning Some(new_node) if another page was needed.
|
||||
`scroll(Scroll::{Active,Top,Row,DeltaRow,DeltaPrompt,Pin})` only changes viewport.
|
||||
`scrollbar()` reports row counts and hides physical history at max_size zero.
|
||||
`scroll_clear()` moves the nonempty active rows into history.
|
||||
|
||||
`erase_rows(top, optional_bottom)` removes inclusive rows and ensures active height.
|
||||
`erase_row(point)` shifts the remainder of the screen while retaining page sizes.
|
||||
`erase_row_bounded(point,limit)` shifts at most limit following rows.
|
||||
`erase_cells_bounded(start_pin,end_pin,left,right)` is an extra rectangular helper.
|
||||
`reset()` clears content and invalidates tracked contents; `clear_history()`
|
||||
preserves the active area.
|
||||
|
||||
`resize(Resize { cols, rows, reflow, cursor })` takes optional new dimensions and
|
||||
an optional active `Cursor { x,y }`. Reflow defaults to true. Cursor is input
|
||||
context; track the actual cursor pin separately to read back its new position.
|
||||
Consult the limitations above before relying on complex resize behavior.
|
||||
|
||||
`clone_range(top,bottom)` clones inclusive **whole rows**, ignoring x, pads to the
|
||||
active height, and returns `(PageList, HashMap<old_handle,new_handle>)`.
|
||||
`try_clone()` clones the whole screen without retaining external tracked pins.
|
||||
Clone viewport defaults to Active; dirty information is retained by rows.
|
||||
Page itself implements Clone for an independent full copy; its inherent
|
||||
`clone_from(source,start,end)` copies a half-open row range into existing capacity.
|
||||
`clone_partial_row_from` accepts source/destination row and column plus length;
|
||||
`clone_cells_within` handles same-page copies without conflicting borrows.
|
||||
|
||||
`page_iterator`, `row_iterator`, `cell_iterator` take Direction, start Point,
|
||||
and optional inclusive end Point. End None uses the frame's bottom-right.
|
||||
Reverse direction traverses the same bounds backwards. PageIterator yields
|
||||
`Chunk { node,start,end }` with half-open page-local rows. Iterators borrow the
|
||||
list immutably; collect Copy pins/chunks or end iteration before mutating it.
|
||||
`prompt_iterator` finds semantic prompt starts; `highlight_semantic_content`
|
||||
returns an optional inclusive `(start_pin,end_pin)` for a prompt's Prompt, Input,
|
||||
or Output region.
|
||||
|
||||
`Page::verify_integrity` and `PageList::verify_integrity` are available in debug
|
||||
and test builds, as are assert_integrity helpers. Nestable pause methods support
|
||||
batched low-level updates. Verification is explicit; raw mutable access can
|
||||
break invariants until the caller completes its update.
|
||||
|
||||
## Remaining PageList source tests
|
||||
|
||||
The following 50 cases are **unported**, rather than justified skips. They include
|
||||
important production behavior and should drive the next implementation pass.
|
||||
- `PageList grow allows exceeding max size for active area`
|
||||
- `PageList grow prune required with a single page`
|
||||
- `PageList scroll delta row back without space preserves active`
|
||||
- `PageList: jump zero prompts`
|
||||
- `Screen: jump back one prompt`
|
||||
- `Screen: jump forward prompt skips multiline continuation`
|
||||
- `PageList eraseRows invalidates viewport offset cache`
|
||||
- `PageList eraseRow invalidates viewport offset cache`
|
||||
- `PageList eraseRowBounded invalidates viewport offset cache`
|
||||
- `PageList eraseRowBounded multi-page invalidates viewport offset cache`
|
||||
- `PageList eraseRowBounded full page shift invalidates viewport offset cache`
|
||||
- `PageList eraseRowBounded exhausts pages invalidates viewport offset cache`
|
||||
- `PageList promptIterator right_down continuation at start`
|
||||
- `PageList promptIterator right_down with prompt before continuation`
|
||||
- `PageList promptIterator right_down limit inclusive`
|
||||
- `PageList promptIterator left_up limit inclusive`
|
||||
- `PageList erase resets viewport to active if moves within active`
|
||||
- `PageList erase resets viewport if inside erased page but not active`
|
||||
- `PageList erase resets viewport to active if top is inside active`
|
||||
- `PageList resize (no reflow) less rows cursor on bottom`
|
||||
- `PageList resize (no reflow) less rows cursor in scrollback`
|
||||
- `PageList resize (no reflow) more rows contains viewport`
|
||||
- `PageList resize (no reflow) more cols with spacer head`
|
||||
- `PageList resize (no reflow) grow cols fast path with spacer head`
|
||||
- `PageList resize (no reflow) more cols forces less rows per page`
|
||||
- `PageList resize more rows and cols doesn't fit in single std page`
|
||||
- `PageList resize (no reflow) more cols forces smaller cap`
|
||||
- `PageList resize (no reflow) more rows adds blank rows if cursor at bottom`
|
||||
- `PageList resize reflow invalidates viewport offset cache`
|
||||
- `PageList resize reflow more cols creates multiple pages`
|
||||
- `PageList resize reflow more cols wrap across page boundary`
|
||||
- `PageList resize reflow more cols wrap across page boundary cursor in second page`
|
||||
- `PageList resize reflow less cols wrap across page boundary cursor in second page`
|
||||
- `PageList resize reflow more cols cursor in wrapped row`
|
||||
- `PageList resize reflow more cols cursor in not wrapped row`
|
||||
- `PageList resize reflow more cols cursor in wrapped row that isn't unwrapped`
|
||||
- `PageList resize reflow exceeds hyperlink memory forcing capacity increase`
|
||||
- `PageList resize reflow exceeds grapheme memory forcing capacity increase`
|
||||
- `PageList resize reflow exceeds style memory forcing capacity increase`
|
||||
- `PageList resize reflow more cols unwrap wide spacer head`
|
||||
- `PageList resize reflow more cols unwrap wide spacer head across two rows`
|
||||
- `PageList resize reflow less cols cursor goes to scrollback`
|
||||
- `PageList resize reflow less cols copy kitty placeholder`
|
||||
- `PageList resize reflow more cols clears kitty placeholder`
|
||||
- `PageList resize reflow wrap moves kitty placeholder`
|
||||
- `PageList resize reflow grapheme map capacity exceeded`
|
||||
- `PageList resize grow cols with unwrap fixes viewport pin`
|
||||
- `PageList grow reuses non-standard page without leak`
|
||||
- `PageList grow non-standard page prune protection`
|
||||
- `PageList resize (no reflow) more cols remaps pins in backfill path`
|
||||
|
||||
## Quality gate
|
||||
|
||||
Final gate (2026-10-11):
|
||||
|
||||
- `cargo fmt -p ghostty-terminal`: passed; only the documented nightly-option
|
||||
warnings from workspace rustfmt configuration.
|
||||
- `cargo build -p ghostty-terminal`: passed.
|
||||
- `cargo test -p ghostty-terminal --quiet`: 239 passed, 0 failed, 0 ignored.
|
||||
- `cargo clippy -p ghostty-terminal --all-targets -- -D warnings`: passed.
|
||||
- `git diff --check -- crates/ghostty-terminal docs/rewrite/notes/ghostty-terminal-pagelist.md`:
|
||||
passed for tracked changes.
|
||||
|
||||
The early missing `ghostty-vt::stream` dependency file was resolved by concurrent
|
||||
workspace work; no change to that crate was needed here. The terminal Cargo.toml
|
||||
already had local changes at the beginning of this task and was not edited here.
|
||||
Reference in new issue
Block a user