Consolidate MangaCircus app, scraper, and Compose baseline

docs/resilience.md

@@ -0,0 +1,113 @@
+# Runtime resilience and agent operations
+
+## Ownership and failure boundaries
+
+```text
+Browser → Reader ──private API──→ Scraper → library (atomic chapter publication)
+             │                       ↑
+             └──health──→ Sentinel ───┘ bounded checkpoint recovery only
+
+Supervisor(reader)   Supervisor(scraper)   Supervisor(sentinel)
+                         ↑
+           Coding agent: inspect incidents → reproduce → patch → test → release
+```
+
+The reader is the only writer of users/progress and the catalog. The scraper is the
+only queue owner and content publisher. Sentinel owns only observations and incidents.
+Each service's supervisor owns only that child process. The coding agent is an
+**escalation/maintenance operator**, not a necessary part of normal downloading.
+The recurring Codex check is an additional backstop, not the runtime watchdog.
+
+## Automatic recovery implemented
+
+| Failure | Action | Automatic limit |
+|---|---|---|
+| Process exit or repeated failed liveness checks | Supervisor restarts only that service, with backoff | Three failures per rolling hour; then parked for operator action |
+| Network/HTTP 5xx | Preserve checkpoint; open source circuit, wait five minutes, retry job | One job-level retry; request-level retries remain bounded |
+| Title unavailable/no English chapters | Record unavailable, continue unrelated work | No blind retry |
+| Parser/local adapter failure | Quarantine job, keep partial data, continue other titles | Two different affected jobs in an hour pause source as systemic failure |
+| Storage/permission failure | Pause all downloads and report | No automatic retry |
+| HTTP 401/403/429 or access challenge | Pause affected source lane | No automatic retry/bypass |
+| Active job with no progress for 15 minutes | Sentinel requests cooperative checkpoint-safe retry | One stall recovery; next stall quarantined |
+
+The daemon remains authoritative: a recovery request includes the observed progress
+fingerprint and is rejected if the job has moved, paused, or is no longer stale.
+Sentinel also observes completed/page counts; a deliberately paused queue is not a
+stall. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds.
+Publication remains one complete chapter at a time, on the same filesystem.
+No recovery action raises pacing, creates another worker for the same source,
+changes identities or deletes partial images.
+
+The publisher retains validated pages in `.staging-mcscraper-resume-<sha256>` folders
+on pause/failure. It checks the full chapter/source/page-manifest fingerprint before
+reuse and revalidates each image; completed chapters still publish with one rename.
+A changed manifest creates separate staging rather than mixing releases. Old random
+stages remain untouched for manual review: they cannot be safely associated from
+page filenames alone. Hidden pending image files are never treated as completed pages.
+
+`retry_wait`, `unavailable` and `quarantined` are explicit queue states in schema 4.
+`ready` is a healthy inspected job awaiting operator download confirmation, not a
+stall. Inspection shares the source lane; it never downloads images or bypasses a
+source circuit. Source coverage and job completion are separate observations.
+Legacy unavailable-title errors are migrated; only pauses solely attributable to
+these unavailable titles are released. Access and unknown failures remain paused.
+Nightly discovery avoids duplicating outstanding, unavailable or quarantined jobs;
+use an explicit operator retry after fixing their cause.
+
+## Observability
+
+**Settings → Service health** shows Sentinel status, active checkpoints, failed jobs
+and incident history. **Settings → Scraper** retains queue, pacing and discovery controls.
+Service errors are classified rather than buried in a single historical `lastError`.
+`npm run check` performs read-only checks. Sentinel's private, token-protected `/metrics`
+exports reader-up, scraper-up and open-incident gauges for a future Prometheus collector.
+It is not a public endpoint.
+
+Scraper status is independent of expensive library scans. Concurrent admin/Sentinel
+probes share one background inventory refresh and receive the previous summary
+immediately. The dashboard summary refreshes at most once per minute unless a
+publication or metadata change invalidates it; duplicate-sensitive decisions still
+request a fresh inventory. `libraryStatus` reports ready/refreshing/stale/updated/error, and failed
+background refreshes do not spawn a new scan on every UI poll. Authoritative
+discovery waits for the current inventory; publication/metadata invalidation during
+a scan forces another scan before it can be used for those decisions. Cold start
+may temporarily show no library targets, but queue and rate controls remain usable.
+
+Storage:
+
+- `data/catalog.json`, `data/collection-covers/`: reader catalog and covers.
+- `data/scraper/queue.json`, `events.log`: queue, circuits, failures and incidents.
+- `data/sentinel/state.json`: Sentinel incidents and last observation.
+- `data/runtime-reader.json`, `data/scraper/runtime.json`, `data/sentinel/runtime.json`:
+  supervisor status and persisted crash budget.
+- `data/scraper/supervisor.lock`, `daemon.lock`: native process ownership; transient.
+
+Scraper/Sentinel/supervisor emit structured logs. Queue and Sentinel histories retain
+200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows
+supervisor logs remain append-only: archive them during stopped maintenance if needed.
+Never export user databases, tokens, cookies, signed query strings or unredacted environments.
+
+## What self-repair intentionally does not mean
+
+Sentinel cannot execute a shell, patch adapters, query image sources, access a Docker
+socket, or issue unrestricted restart commands. It cannot repair broken markup or
+solve source challenges. The agent handles those escalations in the repository:
+
+1. Inspect one bounded incident and relevant checkpoint; preserve the active lane.
+2. Add a fixture reproducing the failure, patch the smallest adapter/logic surface.
+3. Run `npm test`; test UI/API in isolated data if affected.
+4. Build a traceable image, deploy checkpoint-safely, verify resumed progress.
+5. Record release, backup and rollback compatibility. Stop after repeated identical failures.
+
+A VM-local monitor cannot alert on total VM/network failure. Before VM cutover choose
+an external health check and notification destination (e.g. an existing monitoring
+service and private ntfy/Gotify). No notification credentials or destination are invented.
+
+## Crash-budget operation
+
+A parked supervisor stays alive and reports `repair-required`; Docker's restart policy
+does not create an unlimited process restart storm. Its health check fails while the
+child is absent. Investigate first. Once repaired, restart the wrapper after its
+rolling one-hour budget has expired. For an intentional earlier reset, first stop the
+service, back up its `runtime*.json`, and clear only that service's `failures` array;
+never clear the queue or daemon checkpoint. Explicit deployments do not erase budgets.