Checkpoint peering identity and opt-in discovery

vision/discovery-scout.md

@@ -1,6 +1,6 @@
 # Discovery Scout — personalized manga discovery
 
-Status: **vision and implementation plan, not implemented**. Updated 2026-10-09.
+Status: **deterministic first slice implemented; later phases remain a vision**. Updated 2026-10-09. See [`docs/discovery.md`](../docs/discovery.md) for the current contract and deliberate omissions.
 
 ## 1. Outcome
 

vision/server-peering.md

@@ -0,0 +1,120 @@
+# Server peering — opt-in manga sharing
+
+Status: **v1 implemented and isolated-tested on 2026-10-09**. This document remains
+the broader vision; the exact shipped behavior, deployment and limitations are in
+[`docs/peering.md`](../docs/peering.md). VM deployment remains future work.
+
+## 1. Outcome and boundaries
+
+Two MangaCircus servers can pair, show each other a deliberately limited manga catalog, and transfer selected chapters into the receiving server's normal library. An admin on the publishing server must explicitly enable **each manga** for sharing; the default is private. A reader on the receiving server can browse and request downloads, subject to local permissions. Neither server sends user accounts, reading progress, ratings, notes, collections, session cookies, or scraper credentials.
+
+This is **server-to-server library transfer**, not synchronization. The publisher never writes to the receiver. The receiver never obtains unrestricted filesystem access. No automatic reciprocal sharing, title discovery, or mirroring follows from pairing.
+
+The publishing server will necessarily learn the receiving **server's** identity/address, requested title/chapter IDs, byte counts, and request times. “No user data exchange” means the protocol carries no reader identity or activity records; it cannot make those unavoidable transfer observations disappear. The local server mediates browser requests so the reader's browser does not contact the remote peer.
+
+Only share material the server operator is entitled to distribute. Peering must not be used to evade a source host's access controls or to export scraped credentials or source URLs.
+
+## 2. Fit with MangaCircus
+
+Preserve the current [architecture](../docs/architecture.md), [library format](../docs/library-format.md), and [scraper ownership](../docs/scraper-service.md): the reader owns accounts and admin settings, the scraper owns its queue and atomic chapter publication, and the library remains folders plus JSON rather than a database. Do not introduce a second writer to the library or scraper state.
+
+Add one optional **peer gateway** process/container on each server that chooses to publish. It can serve multiple approved peers, receives authenticated peer traffic, reads a *read-only* library mount, exposes only approved catalog fields and opaque content IDs, and enforces upload quotas. It has no accounts database, reader data directory, scraper state, Docker socket, or writable library mount. Its small local state contains only peer grants and quota counters. Admin controls live in the reader UI and are written atomically to a reader-owned peering policy file that the gateway reads; the gateway is the sole writer of usage counters. Use one peer gateway per library, **not** one gateway per peer.
+
+On the receiving side, add a **peer adapter** to the existing scraper daemon. It uses the existing single-owner queue, checkpoints, source-version policy, integrity checks, and publish-on-chapter-complete path. Treat each peer as a source lane, with one transfer worker per peer by default. This is not a separate downloader writing files beside the scraper.
+
+The optional gateway is justified by isolation: an incoming network service should not have access to private reader/scraper data. It stays a small process built from the same project image, not a new platform, database, or agent.
+
+## 3. Trust and pairing
+
+### Recommended mechanism
+
+Use **TLS 1.3 with mutual certificate authentication (mTLS)**. MangaCircus generates its own identity key and X.509 certificate **locally on each server**, through a guided admin action; no external certificate authority or command-line work is required for the initial private-network design. Generate the private key in server-side secret storage with restrictive permissions and show the admin only the public certificate and its fingerprint. Its **private key never leaves that server**. Admins exchange public pairing invitations **outside MangaCircus**—for example in a trusted messenger, on removable media, or by scanning a QR code in person—and compare the certificate/public-key fingerprints through a separate trusted channel. Each server pins the other server's public key and rejects a different key, even if DNS or the network route changes. TLS handles ephemeral session-key exchange and encrypted transport; we do not invent a crypto handshake. [TLS 1.3](https://www.rfc-editor.org/rfc/rfc8446.html) defines mutual certificate authentication.
+
+Certificate validity and access authorization are separate:
+
+- The certificate has `notBefore`/`notAfter` dates (proposal: 1 year) and is rejected outside them. Renewing a certificate with the *same* pinned key may be accepted after local validation; replacing the key requires a fresh out-of-band fingerprint check.
+- The publisher issues a **random 256-bit opaque grant token** for a specific consumer certificate fingerprint, scopes (`catalog`, `download`), `notBefore`, and `expiresAt` (proposal: 90 days). It stores only a hash of the token and checks it on every request. The grant cannot outlive either certificate. A grant can be revoked immediately without changing the identity key. The token is delivered out-of-band during pairing or across the already authenticated mTLS channel after both admins approve; it is never in a URL or browser response.
+- The token is an expiring authorization credential, **not** a second encryption key. A valid token without the matching client private key is useless. An expired/revoked grant, expired certificate, failed pin, or missing client certificate fails closed. No silent insecure fallback or automatic re-pairing.
+
+This uses standard transport cryptography plus an auditable server-side grant record. Do not implement a custom signature protocol or home-grown JWT parser. Use the runtime's verified TLS stack and a cryptographically secure random generator; compare token hashes safely. If a standard certificate-bound token format is later needed for federation, evaluate [RFC 8705](https://www.rfc-editor.org/rfc/rfc8705.html) rather than adding protocol complexity now.
+
+### Admin invitations and pairing flow
+
+1. Admin A opens **Admin → Peering**, selects **Generate identity** once, then **Create invitation** for an intended peer. MangaCircus generates a single-use, short-lived invitation (proposal: 48 hours) containing A's server name, reachable peer URL, public certificate, fingerprint, invitation ID, and one-time pairing code. The invitation can be copied as text/file or shown as a QR code. It contains **no private key and no catalog/download grant**. An admin can cancel it before use.
+2. Admin A gives that invitation to B's admin outside the platform. B imports it and verifies A's fingerprint through a separate trusted channel. B generates its own local identity if needed and returns its own public invitation/card out-of-band. The one-time code links the two pending records and limits accidental pairing with the wrong server; it does not substitute for fingerprint verification or admin approval.
+3. Each admin approves the specific peer. A test mTLS connection proves possession of both private keys and checks pins, names, certificate validity, and clock skew. An expired or already-used invitation cannot create another peer. A new invitation is required for a different server; each server can have **multiple independent peers**, but importing an already-paired fingerprint must resolve to the existing peer rather than silently creating a duplicate.
+4. Each admin separately sets **Serve to this peer** on or off (default off), selects which manga/source versions that peer may see, and sets that peer's upload limits. Serving on enables issuance of A→B's time-limited catalog/download grant; serving off revokes/blocks that direction immediately. B stores A's token in server-only secret storage. If B also serves A, B issues an independent B→A grant and chooses its own share list and limits. One direction can be enabled without the other.
+5. Each admin may also set **Allow downloads from this peer** on or off locally (default off until paired and approved). This controls whether their own readers can browse/queue that peer's catalog. Turning it off pauses existing local jobs from that peer at a checkpoint; it does not delete queued work or already downloaded chapters. It is independent of the other server's serving choice and does not grant the other server access.
+6. The UI shows **pending / paired but inactive / serving / receiving / both directions / expires soon / expired / revoked / fingerprint changed**. Revocation closes new transfers immediately; an in-flight transfer is stopped at the next chunk/checkpoint. Key rotation pauses until new out-of-band verification.
+
+Pairing is a **many-peer registry**, not a two-server global switch. Each peer has its own pinned identity, invitation history, direction switches, grant, manga allowlist, limits, queue status, and audit entries. Disabling one peer or one direction does not alter any other peer. Removing a peer invalidates both local directions and its pending invitations, but never erases already published chapters.
+
+| A's Serve to B | B's Serve to A | Result, assuming each receiver locally permits downloads |
+|---|---|---|
+| Off | Off | Paired identity only; neither catalog is readable. |
+| On | Off | B can browse/download A's approved manga; A cannot browse B's. |
+| Off | On | A can browse/download B's approved manga; B cannot browse A's. |
+| On | On | Both can browse/download, using separate grants, selections, and limits. |
+
+An admin may revoke a peer, a grant, or one manga's share independently. Removing a share withdraws its catalog and blob access immediately, not merely on the next catalog refresh. Already downloaded copies on the other server are not remotely deleted.
+
+## 4. Shared catalog and content protocol
+
+Version the protocol from day one (`/peer/v1`). Send a *projection* of library data, never raw `manga.json`, directory listings, absolute paths, or account/reader state.
+
+| Operation | Minimum payload / behavior |
+|---|---|
+| Paged catalog | Publisher-scoped opaque manga ID, title, release year/status if present, compact description, optional approved cover thumbnail, chapter/volume counts, catalog revision, next cursor. Only admin-approved manga. |
+| Manga manifest | Approved volume/chapter IDs, labels, order, page counts, sizes, content revision and per-page SHA-256 hashes. No upstream host credentials or original image URLs. |
+| Content stream | Opaque blob ID from an authorized manifest; bounded-size image bytes, MIME type, hash, ETag, and HTTP range support for restartable transfers. Never accepts a filesystem path from the client. |
+
+Sharing a manga covers its **completed, validated chapters only**. Hidden staging/partial downloads, unrelated files, and currently writing pages never enter the manifest. A completed new chapter becomes visible after the scraper's atomic publication and gateway index refresh. Revocation or share removal wins over a previously issued manifest/blob ID.
+
+Catalog snapshots use cursor/revision pagination and bounded page sizes. A revision change causes a clean refresh, not an unbounded rescan per browser click. Local titles can be matched to remote titles only as **candidates**; uncertain same-name editions require the reader/admin to choose the correct local manga. Store publisher-specific IDs and content revisions to avoid conflating two sources with the same chapter number.
+
+Input from a peer is untrusted even after pairing: cap manifests, chapter/page sizes and total transfer size; validate schemas and image MIME/content; reject traversal paths, archives/executables, and malformed names; stream rather than buffering a whole manga; verify each page hash before publication. Hashes detect corruption but do not prove editorial correctness. Never overwrite an existing chapter silently.
+
+For transport efficiency, start with original image bytes and page-level resume. A [local compression test](server-peering-compression-test.md) found an average ~6% reduction with zstd-3 but a wide 0.7–10.7% spread by chapter, so codec negotiation is an optional later optimization, not a prerequisite for peering. Measure actual chapter-ready time over a real peer link before adding chapter packing or automatic compression.
+
+## 5. Reader and queue experience
+
+Add **Peer catalogs** as a reader-facing browse surface, separate from the local dashboard. Show one tile/tab per peer that currently serves this server and is locally allowed for downloads; peers that are only receiving from us do not appear as remote catalogs. A peer tile shows the publishing server's display name and grant status; search and manga details are fetched through the local reader backend. A reader can select a whole manga, a volume, individual chapters, newest missing, or a bounded chapter range using the existing fine-grained queue model. The UI previews expected chapters/bytes and highlights what is already local. Local role permissions still govern who may submit to the shared scraper queue; browsing does not itself start a transfer.
+
+The resulting job stores the peer ID, remote manga/chapter IDs and revision, selected scope, duplicate policy, and progress checkpoint. Default policy is **skip matching existing content**; an explicit “add as another source” choice preserves the source variant when the same chapter exists from elsewhere. On receipt, the scraper publishes each complete chapter immediately through its existing atomic path, making it readable before the entire manga finishes. Interrupted transfers resume by validated ranges/checkpoints and never expose partial pages in the library.
+
+No automatic remote refresh in v1. A future admin opt-in may schedule checks for *already selected* peer manga, but must use the same queue, grant, and rate policies. No peer request may be triggered by reading a page or opening the local dashboard.
+
+## 6. Upload and download limits
+
+The **publisher sets hard ceilings** per peer and globally: concurrent streams, bytes/second (token bucket), bytes/day, catalog requests/minute, allowed time window, max response size, and idle/transfer timeout. Proposed starting defaults: one stream per peer, two streams globally, 2 MiB/s per peer, and 10 GiB/day per peer. These are product defaults, not a promise about available bandwidth; admins can lower or raise them. Quotas and current usage must be visible, with a “pause this peer” control. Persist daily counters atomically across restarts; a restart must not reset a used quota.
+
+The **consumer can only be stricter**: one active transfer per peer by default, local bandwidth cap, chapter spacing, time-of-day window, maximum queued bytes/free-space reserve, and pause/resume. The effective rate is the lower of the two sides' caps. Respect `Retry-After` on rate limits and back off on transient failures; never retry authentication failure, revocation, or fingerprint mismatch as if they were network glitches. Fair-share scheduling prevents one peer or one huge manga from starving the local source lanes. No same-peer parallel image workers by default.
+
+The gateway responds with clear, bounded errors: unauthorized/expired, revoked, not shared, quota exhausted with retry time, stale manifest, and transient unavailable. The receiver surfaces these separately in queue status and retains checkpoints. Logs and metrics identify a peer by local alias/fingerprint prefix, not a token or private key.
+
+## 7. Deployment and data ownership
+
+Add an optional `peer-gateway` Compose profile/service. It mounts `/library` read-only, a read-only policy projection, gateway-owned counter storage, and certificate/token secrets. It has no mount for `data/db.json`, the reader's account store, or `data/scraper`; the scraper still alone has writable library access. Start with a listener reachable only on the operator's intended private network. Do not expose the scraper's private API or reuse the normal reader web route for binary transfer.
+
+For the future VM, prefer a **tailnet-only raw TCP forwarder to the gateway's own mTLS listener**. Tailscale Serve's `--tcp` mode forwards raw TCP, whereas `--tls-terminated-tcp` ends TLS at Serve and is **not** the default design for end-to-end mTLS pinning. [Tailscale's Serve documentation](https://tailscale.com/docs/reference/tailscale-cli/serve) distinguishes the two. Tailnet reachability is defense in depth, not a substitute for pairing, grants, and quotas. Keep the existing Caddy reader reverse proxy separate; only consider public peering later with a dedicated mTLS endpoint, firewall policy, and an explicit threat review. Exact VM ports/hostnames await deployment configuration.
+
+Suggested NoDB files (names are design targets, not existing files): reader-owned `data/peering/policy.json` for peers, share flags, and limit settings; gateway-owned `data/peering-gateway/usage.json` for quota counters; scraper-owned existing queue/checkpoints for receiver jobs. Private keys and grant tokens live in excluded secrets storage, not policy JSON, logs, exported cards, or Git. Atomic replace plus bounded backups for JSON; one writer per file. Backups of identity keys/tokens must be encrypted and restorable only by the operator. Losing a private key means re-pairing, not copying another server's key.
+
+## 8. Implementation sequence and acceptance gates
+
+1. **Contracts and threat model:** define the exact catalog/manifest schemas, stable peer IDs, many-peer policy model, independent direction switches, duplicate semantics, grant lifecycle, filesystem limits, and forbidden fields. Add protocol fixtures and negative tests before network code.
+2. **Identity and admin controls:** generate local keys/certs, expiring single-use public invitations, verified import, pinning, grant expiry/revocation, per-peer serving/download controls, per-manga opt-in, and audit events. Test wrong pin, missing client cert, invitation reuse/expiry, clock/expiry, key rotation, token replay with a different certificate, and immediate revocation of one direction without affecting another peer.
+3. **Read-only publisher:** optional gateway, bounded approved-only catalog/manifest/blob endpoints, quota enforcement, read-only mounts. Test that unshared, partially downloaded, deleted, and revoked content cannot be listed or fetched—even by guessed IDs or stale manifests. Verify no user data fields or raw source URLs appear on the wire.
+4. **Consumer browsing:** local-server proxy and reader UI with peer status, search, details, selection preview, and queue permission checks. No browser-to-peer credentials or direct requests.
+5. **Scraper transfer lane:** chapter/page checkpoints, range resume, hashes, local throttles, source-aware deduplication, and existing atomic chapter publication. Test corruption, interrupted transfer, stale revision, out-of-space, repeated queue submission, and preservation of current jobs.
+6. **Observability and operations:** per-peer health, bytes, throttling, grant expiry warnings, queue stalls, and safe retry controls. Sentinel may report a dead gateway or stuck transfer but must not renew trust, broaden a share, or bypass a quota. Document backup/restore and key-rotation runbooks.
+7. **Local then VM pilot:** three local Compose instances with distinct data directories; pair A↔B and A↔C, exercise every direction combination, and transfer a few admin-approved test chapters. Verify disabling B does not affect C, no account/progress exchange, and quota isolation. Only then deploy the optional service on the VM's private network. Leave existing reader/scraper ports and active downloads undisturbed during the pilot.
+
+**Ship gate:** a fresh unpaired server sees nothing; a paired server with serving off or no granted manga sees an empty catalog; an expired/revoked peer cannot list or fetch; invitations cannot be reused; disabling one direction or peer leaves unrelated peers intact; quotas survive restart; a receiver can resume and publish one chapter without duplicates; and its browser never receives the grant token or any private key. The feature remains disabled by default until those gates pass.
+
+## 9. Decisions to settle before implementation
+
+- Who may queue from a peer: any authenticated reader, or only readers with a new download permission? Recommendation: explicit local `canQueuePeerDownload` permission, admin-controlled.
+- Does sharing approval apply to every source version of a manga or a chosen edition? Recommendation: opt in **per manga and source version**, with a clear “all currently complete chapters / selected volumes” scope; new chapters follow that scope only if the admin opted into future additions.
+- How long should grants last, and should renewal require out-of-band confirmation? Recommendation: 90 days with an admin-visible renewal that retains the same pinned key; any changed key requires out-of-band re-verification.
+- How should two distinct MangaCircus accounts on different servers appear to each other? Recommendation: **they do not**. The publisher grants a server, and the receiver alone decides which of its users may browse and queue.