Files
lanspread/crates/lanspread-peer/ARCHITECTURE.md
T
ddidderr 276a919f46 fix(peer): stop QUIC promptly after application tasks drain
Cancelled handshakes can remain in transport state after their application
owners return. Waiting for client idleness before signaling endpoint stop
added a measured three-second grace timeout to shutdown.

Stop and join the owned endpoint directly after application scopes drain.
Keep supervisor joining and isolated runtime teardown, which also finish
s2n's adapter workers and release their socket clones. Use this normal cleanup
for rejected-handshake tests instead of their former special shutdown path.

Test Plan:
- A loopback silent-peer handshake reproduced a 3.001-second wait before the fix.
- New supervisor regression verifies wait_stopped completes within one second
  and releases the client UDP socket after cancelling an in-flight handshake.
- just test: all 796 workspace tests passed, including the new regression.
- just fmt and just clippy: passed.
- Actual native window-close timing was not manually measured.
2026-09-12 20:01:56 +02:00

468 lines
27 KiB
Markdown

# lanspread-peer architecture
The peer uses mDNS discovery, authenticated QUIC transport, responder-owned
metadata snapshots, and catalog-authorized file transfers. Wire protocol 8 and
ALPN `lanspread/8` are the only supported wire mode; there is no legacy decode,
fallback, or compatibility shim.
## Goals (unchanged)
- Local LAN discovery via mDNS.
- QUIC + JSON messages for control, raw streams for file data.
- UI drives operations through `PeerCommand`, peers remain headless.
- Peers can appear/disappear at any time without data loss.
## Installation identity and QUIC authentication
- One installation-local Ed25519 key signs one self-issued TLS certificate.
`PeerId` is BLAKE3 over the certificate's complete canonical DER
SubjectPublicKeyInfo, encoded as 52 lowercase unpadded RFC 4648 base32
characters. It is not a UUID, display name, IP address, or wire assertion.
- The desktop app stores a strict version-1 record at `peer-identity-v1.json` in
its Tauri app-data directory. The record binds the Ed25519 certificate,
private key, SPKI-derived `PeerId`, and exact SAN. Reads and publication do
not follow symlinks or reparse points, creation is no-clobber, corrupt bytes
are quarantined before replacement, and private material is kept at mode 0600
where the platform supports Unix modes.
- Persistence failure produces a typed `Ephemeral` outcome and a fresh identity
for that runtime; it never falls back to another peer-ID scheme. The peer CLI
can instead select an existing record with `--identity-file`; that path is
loaded strictly and is never repaired, replaced, or generated implicitly.
- Identity state assumes one normal application owner for its app-data directory
and deliberately has no cross-process identity lease. Retained no-follow
handles contain path traversal and external-target mutation, and publication
does not overwrite an existing winner. A concurrent same-user writer can still
cause availability or continuity loss during quarantine; it is outside the
supported ownership model.
- QUIC uses the rustls provider with TLS 1.3 only. Every outbound operation
carries a `PeerEndpoint { peer_id, addr }`; the expected ID is encoded in an
exact SNI name, checked against the responder certificate SPKI, and the real
TLS 1.3 CertificateVerify signature is delegated to rustls. ALPN is
`lanspread/8`. Client resumption and 0-RTT are disabled; the server issues no
tickets and sends no early or half-RTT data.
## Peer lifecycle and message flow
### 1) Startup and advertise
- Start QUIC server.
- Advertise via mDNS with TXT records:
- `peer_id` (the canonical SPKI-derived ID)
- `proto_ver`
- optional `hostname`
### 2) Discovery and handshake
When a peer is discovered:
1. Parse `peer_id`, address, and `proto_ver` into a candidate `PeerEndpoint`.
Discovery is not authentication and does not add the peer to `PeerGameDB` or
emit UI membership events. Advertised addresses must be unicast with a
nonzero port; multicast, broadcast, and unspecified targets are dropped
before any handshake packet is sent. The mDNS ingress queue and active
candidate negotiations are each capped at 64, and one observed response IP
may hold at most 8 active-or-cooling candidates. The advertised A/AAAA target
is never used as provenance, so one host cannot escape the budget by rotating
target addresses, peer IDs, or ports. The vendored mDNS daemon also caps
cached records at 1,024 globally and 128 per response IP, with at most 4,096
deduplicated timers.
2. Reserve a candidate negotiation lease before queueing or awaiting work, then
establish a TLS-pinned connection to that exact endpoint. Missing or non-v8
records are rejected, and neither an ephemeral QUIC source port nor a payload
can replace the candidate listener address.
3. Send the empty `Hello` pull request. The pinned responder returns a
`HelloSnapshot` containing a `PeerStateSnapshot` with `runtime_session_id`,
`library`, and `call_to_play`: only that responder's local library and local
Call-to-Play author slice.
4. Decode the strict bounded wire shape, then validate the library and
Call-to-Play domains independently. Commit the endpoint and each valid domain
only if the candidate lease is still current for both peer ID and address.
This domain isolation lets an invalid remote Call-to-Play slice be cleared or
preserved according to its session without discarding a valid library. After
the pinned Hello succeeds, at most eight committed identities may reside at
one stored endpoint IP, alongside the global 64-peer limit. This
authenticated residency bound is separate from the pre-authentication mDNS
candidate quota; a ninth identity is rejected without evicting an existing
address owner, and removing a peer releases its slot.
5. Assign a fresh endpoint generation to every successful authentication.
Dropped, failed, or superseded work releases its candidate claims, and a late
result cannot mutate a newer generation.
### 3) Steady state
- Successful liveness probes update `last_seen` only when both the typed
endpoint and its authenticated generation are still current. Stale probes
cannot refresh or remove a reauthenticated endpoint.
- `Ping` receives a `Pong` carrying `PeerRevisions` with the runtime-session ID
and both domain revisions. A session or revision mismatch schedules a full
pinned pull, so idle liveness heals dropped change hints.
- Local library or Call-to-Play changes send cheap revision hints. Hints are
untrusted and lossy: unknown claims are ignored, a hint is honoured only when
it arrives from the IP address at which the claimed peer was authenticated,
and a known claim can only coalesce a pull from the claimed peer's already
authenticated endpoint. Requesters are anonymous, so this source binding is
what stops a third host from choosing which peers this node pulls.
- The state-sync scheduler caps its ingress and tracked peers at 64, coalesces
work for five seconds, and runs at most eight pinned pulls and eight hint
sends concurrently. Child work is drained lexically on shutdown.
- The server admits at most 64 unauthenticated handshakes, 64 established
connection scopes, and 32 control streams per connection. One observed IP may
start at most 8 handshakes during one 3-second handshake window, before TLS
and established-connection leases exist; the source ledger itself holds at
most 256 IPs. One observed IP may retain at most 8 connections, 8 of the 16
global control tasks, 8 of the 48 global bulk tasks, and 1 of the 2 Stream
Install providers. Each control stream accepts one frame followed by request
EOF and sends at most one response. Inbound request frames are capped at 64
KiB (the QUIC receive window per server stream matches it), response frames at
8 MiB, and control I/O has ten-second deadlines. Response buffering permits at
most 8 MiB per endpoint IP within one shared 32 MiB pool. Wire collections
(library games, Call to Play author events) are deserialized through a bounded
visitor that materializes at most one element past the semantic limit and
discards the rest, so an oversize frame cannot claim more memory than a valid
one before validation rejects it.
### Call to Play replication
Call to Play is transient, responder-owned state. Each peer serves exactly one
local author snapshot:
```text
CallToPlayAuthorSnapshot { revision, display_name, events }
CallToPlayAuthorEvent { id, call_id, at, action }
```
The outer `PeerStateSnapshot` owns the runtime-session ID. A `CallId` encodes
the creator's typed `PeerId` plus a 128-bit nonce. The UI supplies only a local
intent and current display name; the peer core generates call/event nonces and
timestamps. Identity and display name are not repeated inside events. Creator
actions (`Create`, `Start`, `Cancel`, and `AddTime`) must come from the
`CallId.creator`; participant authors may contribute only participant actions.
The receiver assigns author identity from the pinned `PeerEndpoint`, validates
and normalizes the whole author slice off-lock, then commits it against that
endpoint's current generation and runtime session. For the same session, only a
higher revision replaces the slice; an equal identical snapshot is a no-op, an
equal different snapshot is a conflict, and a lower revision is stale. A new
authenticated session clears the previous slice before accepting valid new
state. Invalid same-session state preserves the last valid slice, while invalid
new-session state leaves that author absent.
Participant events are retained but hidden until the creator's direct author
slice supplies the call root. Removing a participant removes only that author's
rows; removing the creator hides the entire call. Because every snapshot is
served only by its author and attributed from the pinned responder, there is no
union merge, relay, live-event push, acknowledgement, or author claim on the
wire. The UI receives a deterministic complete replacement view.
Unresolved call history is removed as a whole after the deadline plus five
minutes; terminal history is removed as a whole after `Start` or `Cancel` plus
15 minutes. No tombstone survives pruning. Pruning a local slice advances its
revision before a snapshot or `Pong` is served, allowing peers to discover the
removal. The store caps total authors at 64, events per author at 4,096, and one
encoded author snapshot at 4 MiB, with reserved capacity for terminal actions.
### 4) Shutdown
- Cancellation stops admission, drains all lexically owned connection, stream,
state-sync, operation, and mDNS children, closes the shared endpoints, and
joins the supervisor.
- After application children drain, endpoint tasks stop and join immediately;
exit does not wait for cancelled handshakes or closed connections to exhaust
their transport timers.
- There is no `Goodbye` control message. Pinned liveness failure and stale
generation-conditional removal are authoritative for departure.
## Library sync protocol
The responder's full library slice is
`LibrarySnapshot { revision, games: Vec<GameAvailability> }`, where each sorted,
unique availability row contains only `game_id` and the catalog-derived exact
`ContentId`. A snapshot contains at most 4,096 games. It does not carry titles,
versions, paths, sizes, file lists, hashes chosen by the peer, installed state,
or another peer's availability.
The empty `Hello` request always returns the responder's current complete slice.
For the same runtime session, a higher revision atomically replaces the cached
slice; lower or equal stale work cannot roll it back. A new authenticated
runtime session resets the old domain before its candidate snapshot is
considered. `LibraryChanged(ChangeHint)` carries only an untrusted claimed
identity, runtime session, and revision. It never mutates a library directly and
has no delta payload. A pinned `Pong` revision mismatch schedules the same full
pull.
## Local library publication
- Only real local availability changes advance the revision and schedule a hint;
there is no periodic list broadcast.
- Filesystem events are gated per game ID instead of time-debounced:
- an active operation lock drops events for that game;
- a rescan already running for the ID sets a rescan-pending flag;
- the running rescan loops once more when that flag was set.
- Local library scans emit `LocalLibraryChanged` only for real library changes,
except that accepted game-directory changes can force a UI snapshot for the
new path without changing peer availability.
- Active operation mutations emit `ActiveOperationsChanged` from the mutation
path instead of riding on local library scans. Admission withdraws network
availability under the operation and library locks, then emits the active
operation before the withdrawn `LocalLibraryChanged` UI snapshot. This keeps
the game visible as busy while peers can no longer request its changing files.
- The remote UI projection is a wholesale replacement derived from all current
authenticated per-peer slices; two peers offering the same game remain
distinct exact-content sources.
## Local game scanning: fast and low cost
### Strategy
1. Maintain a persistent revisioned on-disk index of local summaries and cheap
per-game fingerprints (root-level `version.ini` contents/mtime, root-level
`.eti` name/size/mtime, `local/` presence, and recovery state).
2. Poll a bounded non-recursive metadata snapshot once per second and update
only changed games.
3. Keep a 300-second fallback scan to recover from missed events.
### Fast-path scanning
- On startup, list only top-level game directories.
- For each game, read a cheap fingerprint:
- root-level `.eti` file names, sizes, and mtimes
- root-level `version.ini` mtime
- presence of `local/` as a directory
- If the fingerprint is unchanged, reuse the cached local summary.
- Only run a recursive scan for new or changed games.
- Snapshot and recursive work has explicit entry, game, depth, byte, file-size,
and elapsed-time ceilings. Failure retains the previous complete state. Poll
failures back off from one second to five minutes per attempted root; a root
switch resets that delay once.
## Local State and Recovery
Downloaded and installed are independent predicates:
- `downloaded` is true only when `<game_root>/version.ini` exists as a regular
file. The sentinel is written last through `.version.ini.tmp` and atomic
rename. The old sentinel is parked before a pending ownership set is
published; recovery restores it only when a valid baseline proves payload
mutation never started.
- `installed` is true when `<game_root>/local/` is a directory. The contents of
`local/` are user-owned and are skipped by manifests, fingerprints, and file
serving.
- Install and update transactions unpack into staging, then overwrite the first
discovered game-provided `account_name.txt` and `language.txt` files under the
staged tree from launcher settings before promoting it to `local/`.
- Launch-settings discovery uses one bounded traversal, reads at most 1 MiB per
INI and 64 MiB in aggregate under the same ten-minute clock, and finishes all
candidate reads before its first write or marker publication.
Reserved per-game paths:
- `.version.ini.tmp` and `.version.ini.discarded` are download transaction
scratch files. Startup recovery interprets them together with the durable
ownership journal rather than sweeping them independently.
- `.local.installing/` is extraction staging.
- `.local.backup/` holds the previous install while an update or uninstall is in
flight.
- `games/<game_id>/install_intent.json` in the configured state directory is the
atomic per-game intent log. Every `games/<game_id>/` path is derived from a
game ID that satisfies the catalog's portable component rules; the public
`setup_done_path` and the launch-settings entry points validate the ID before
joining it, and legacy migration refuses a games-folder directory whose name
would not pass.
- `games/<game_id>/download_ownership/v1-<root_digest>/record.json` in that
state directory records the last committed and any pending downloader-owned
regular-file set and its exact catalog content ID. The namespace is derived
from one canonical configured games directory, and the record retains the full
lossless root identity. Same-ID state for different configured roots therefore
remains independent.
- `.lanspread_owned` inside `.local.*` directories proves Lanspread ownership
when no active intent independently proves transaction ownership.
Downloaded-file removal is not an uninstall transaction. For a catalog ID that
is a single direct child of the configured game directory, it requires a valid
ownership record and regular root-level `version.ini`, and refuses `local/`,
`.local.installing/`, or `.local.backup/`. It journals an empty pending
generation, deletes only committed downloader-owned files and the sentinel, and
keeps the game root plus every unknown file or directory.
The state layout is game-first: root namespaces live under each game ID rather
than under a global root tree. A former singleton ownership record is migrated
once into its derived namespace with copy-first durable publication. Singleton
state is not accepted as a parallel runtime format, and conflicting or split
migration evidence fails closed.
Recovery reads app-state `install_intent.json` and combines the recorded active
intent with the observed `local/`, `.local.installing/`, and `.local.backup/`
state. Intent states `Installing`, `Updating`, and `Uninstalling` prove
ownership of the corresponding reserved directories even if the marker was not
flushed before a crash. Settled state is represented by absence of the intent;
markerless `.local.*` directories are then left untouched.
Legacy `.lanspread/`, `.lanspread.json`, `.lanspread.json.tmp`,
`.softlan_game_installed`, and `local/.softlan_first_start_done` files are
handled only by the dedicated pre-start migration phase. Normal operation does
not read legacy state paths.
### Result
Most scans become O(number of game dirs), with full recursion only when needed.
## File manifests and downloads
- Tauri and the peer CLI inject one immutable `CatalogBundle`: `game.db` defines
catalog identity/version, the mandatory compact
`manifests/catalog-content-index-v1.jsonl` maps every exact catalog row to its
expected `ContentId` and Stream Install capability, and
`manifests/<game_id>.json` defines canonical paths, kinds, sizes, 128 MiB
chunks, and BLAKE3 values. The index has exact database coverage and is loaded
once; manifest bodies remain on demand and must recompute both indexed fields.
Disk body loads and full validation recheck the durable publication marker
before and after their work. Runtime peers cannot add catalog entries or
replace that byte authority.
- Authenticated remote-availability joins use the compact index only. Unknown or
mismatched `(game_id, content_id)` pairs are rejected without filesystem I/O
or manifest parsing. Outbound serving (chunk and Stream Install admission)
also gates on the compact index and in-memory local readiness first and then
reads the manifest body from the validated cache only: every publishable
game's body was primed before its library revision became visible, so an
anonymous request can neither trigger a disk read nor populate the manifest
cache. Download planning on the requesting side still loads and validates the
full body on demand.
- The local catalog artifact is converted into a `ValidatedDownloadManifest`
before any destination mutation. It contains canonical game-root-relative
paths and rejects aliases, reserved state, shape conflicts, and bounded-size
violations as one unit. Remote peers never define the storage plan.
- Source selection requires the exact catalog `(game_id, content_id)` pair from
an authenticated peer's current library slice. Every ordinary chunk request
repeats the exact `ContentId`, `CanonicalCatalogPath`, offset, and length.
There is no version-only or address-only eligibility adapter.
- A new manifest target that already exists without prior committed ownership is
rejected before the sentinel, ownership journal, or payload is mutated.
- Download mutation holds a capability handle for the direct catalog game root.
Directory components and final files are reopened relative to that handle
without following links or Windows reparse points; chunk writes and checks use
the same opened file handle.
- Downloads hash every chunk, including the buffered `version.ini`, as it is
received and require the catalog BLAKE3 value, exact length, and exact end of
stream. One absolute ten-minute application deadline covers the complete
open/request/receive/check lifecycle of each ordinary chunk.
- An integrity failure quarantines `(PeerId, content_id)` in shared memory for
the rest of that peer runtime, independent of address changes. Transport
failures remain retryable without quarantine. Each failed chunk may try at
most eight distinct authenticated identities at distinct endpoint IPs. All
retries share one nonrenewable 20-minute window created after the initial
failure; an individual attempt still has the earlier ten-minute limit.
- A no-transfer local shortcut is allowed only when settled ownership records
the exact expected committed content ID. A matching `version.ini` alone, or a
legacy/pre-content-ID record, is not catalog-content proof.
- A game is transferable only when its ID is in the catalog, no operation is
active for that ID, and the root-level `version.ini` sentinel exists.
- The sender admits only an exact catalog path and, for ranged requests, an
exact catalog chunk range before opening the retained no-follow file handle.
- `local/` paths are never served, even if a stale or malicious manifest request
asks for them.
- Cancelling or recovering a download removes only paths named by its durable
ownership journal and its validated pending manifest. Unknown user files,
`local/`, and install transaction metadata are preserved, so a cancelled
update of an installed game settles as local-only.
### Ordinary install extraction
- `install`/`update` extract every current direct regular root `.eti` archive
with the external `unrar` unpacker into the staging directory. Downloaded
archives were BLAKE3-verified, but the ordinary path deliberately accepts
later local replacements and extra archives for the S33 workflow.
- `unrar` runs with `-ol-`, so symbolic-link entries are skipped, and the
staging tree is walked without following links before promotion. Any symlink
(or reparse point on Windows) aborts the transaction and rolls staging back,
so a link can never redirect later launch-time rewrites or uninstall.
### Streamed install integrity
- Low-disk streamed installs request archive-derived file bytes from one peer
and write them directly into the install transaction staging directory.
- A sender admits Stream Install only for a manifest with verified extracted
output and only when its direct regular root `.eti` set exactly equals the
catalog archive set. Missing or extra archives stop before the extraction
provider receives authority.
- Provider egress has one nonrenewable deadline of ten minutes plus the exact
catalog streamed bytes at 1 MiB/s, so large valid games are finite without a
fixed ten-minute whole-transfer cutoff.
- The Stream Install request names the exact catalog `ContentId`. All
path-bearing frames, including archive names, use `CanonicalCatalogPath` and
the fallible checked decoder distinguishes malformed framing from an explicit
sender `Error` frame.
- The receiver independently requires the exact catalog archive set and the
complete extracted path/kind/size set. It hashes every regular file with
BLAKE3 while writing and verifies every expected entry before commit. Regular
files may occur only once across the entire archive set; repeated directory
entries are allowed because they carry no bytes. RAR CRC32 remains only an
early corruption signal, not an authority boundary.
- Integrity failures quarantine the source for that catalog content and retry a
fresh transaction from another eligible peer. Transport failures retry without
quarantine; local I/O and cancellation stop. At most four distinct endpoint
IPs are attempted. Every attempt shares a catalog-sized total deadline of ten
minutes plus two copies of the expected streamed bytes at 1 MiB/s. The
ten-minute inactivity timer refreshes only after a nonempty, catalog-checked
file chunk is written. Every failed attempt rolls back before the next source
begins.
- After complete catalog verification, account/language/persona settings are
rewritten in staging before promotion. The one-shot launch-settings marker is
written only after successful promotion; if it cannot be written, first play
safely retries the rewrite.
- A game without a verified extracted-file manifest neither offers nor accepts
Stream Install. There is no CRC32-only fallback.
## Catalog publication and packaging
- `lanspread-production-catalog` creates a production `game.db`, manifests, and
complete compact identity index in staging, validates that authority with the
application loader, and atomically replaces the generated catalog directory.
Its default selection requires every game and exact version from the
checked-in source database. Independent explicit modes may omit missing
package directories or accept package versions that differ by updating the
copied database. The generated manifests and database still form one exact
set, and the source database is never modified.
- `lanspread-catalog-publisher` incrementally regenerates selected manifests in
an existing exact indexed corpus. After it owns the durable publication
marker, it freshly validates every unselected body/index pair before deriving
and atomically publishing the complete mixed index; no pre-marker snapshot can
roll another completed publication backward.
- `lanspread-fixture-catalog` is a separate test-only generator. It derives
reduced peer-CLI `game.db` files and manifests from explicitly selected
fixtures; those outputs are development and acceptance-test authority only.
- Tauri fixture builds require both the exact committed development resource map
and `LANSPREAD_USE_FIXTURE_CATALOG=1`. Every other build mode defaults to the
production resource map, and the custom production profile cannot be
downgraded to fixtures.
- `just run` uses the existing production catalog. `just run-local GAMES_DIR`
creates a separate cached catalog from the supplied files, allowing missing
packages and package versions that differ from the metadata database. Local
builds require a distinct compile-time opt-in and resource map, are rejected
by the production profile and bundled builds, and use separate app settings.
Their startup game directory is compiled in; production embeds no override.
- Production packaging requires the exact generated production
`game.db`/manifest corpus to pass `check --all`. The external production
packages and generated authority are not present in this checkout, so that
gate remains intentionally blocked; fixture success is not production
completion.
## Fault tolerance rules
- Every authenticated peer is keyed by `PeerId`, not by IP address.
- An authenticated `PeerEndpoint` retains the candidate or explicitly supplied
listener address. Payload fields and ephemeral QUIC source ports cannot
rewrite it, and no network caller can dial an address without the expected
responder ID.
- Every successful authentication assigns a fresh endpoint generation. Refresh,
liveness update, and stale removal are conditional on the exact endpoint and
generation they observed.
- Runtime-session IDs and per-domain revisions prevent delayed snapshots from
rolling state back across restart or endpoint replacement. Change hints never
bypass the pinned pull and generation checks.
- A generation-conditional authenticated departure removes that peer's library
and Call-to-Play slices and publishes complete replacement views. Ordinary
short-lived QUIC connection closure is not roster departure.
- Protocol 8 is strict and current-only. Unknown fields, noncanonical typed IDs
or paths, duplicate/unsorted library rows, oversized domains, extra control
frames, and v7 messages are rejected instead of adapted.