MangaCircus/docs/resilience.md

# Runtime resilience and agent operations

## Ownership and failure boundaries

```text
Browser → Reader ──private API──→ Scraper → library (atomic chapter publication)
                                     └─ stall watchdog: bounded checkpoint recovery only

Supervisor(reader)   Supervisor(scraper)
                         ↑
           Coding agent: inspect incidents → reproduce → patch → test → release
```

The reader is the only writer of users/progress and the catalog. The scraper is the
only queue owner and content publisher, and it records its own failures and incidents.
Each service's supervisor owns only that child process. The coding agent is an
**escalation/maintenance operator**, not a necessary part of normal downloading.
A periodic check by an operator or coding agent is an additional backstop, not the runtime watchdog.

## Automatic recovery implemented

| Failure | Action | Automatic limit |
|---|---|---|
| Process exit or repeated failed liveness checks | Supervisor restarts only that service, with backoff | Three failures per rolling hour; then parked for operator action |
| Network/HTTP 5xx | Preserve checkpoint; open source circuit, wait five minutes, retry job | One job-level retry; request-level retries remain bounded |
| Title unavailable/no English chapters | Record unavailable, continue unrelated work | No blind retry |
| Parser/local adapter failure | Quarantine job, keep partial data, continue other titles | Two different affected jobs in an hour pause source as systemic failure |
| Storage/permission failure | Pause all downloads and report | No automatic retry |
| HTTP 401/403/429 or access challenge | Pause affected source lane | No automatic retry/bypass |
| Active job with no progress for 15 minutes | Scraper stall watchdog stops it at its next checkpoint and requeues it | One stall recovery; next stall quarantined |

The stall watchdog runs inside the scraper (`ScraperService.checkStalls`, every 30 s
from the work loop). A job is stalled when its checkpoint fingerprint (completed
chapters/pages) has not moved for 15 minutes, or for one chapter delay plus three
minutes if that is longer. Time spent paused does not count, and a deliberately
paused queue is not a stall. It only treats a job as stalled when its checkpoint can
be read: the scraper resolves source-specific work folders (including MangaPill's
`mangapill-` prefix), and a missing checkpoint is not evidence that downloads stopped.
Each stall records an incident with `failure.kind: 'stall'` and an `events.log` entry.
An operator retry resets the job's stall recovery count without deleting its partial
download. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds.
Publication remains one complete chapter at a time, on the same filesystem.
No recovery action raises pacing, creates another worker for the same source,
changes identities or deletes partial images.

The publisher retains validated pages in `.staging-mcscraper-resume-<sha256>` folders
on pause/failure. It checks the full chapter/source/page-manifest fingerprint before
reuse and revalidates each image; completed chapters still publish with one rename.
A changed manifest creates separate staging rather than mixing releases. Old random
stages remain untouched for manual review: they cannot be safely associated from
page filenames alone. Hidden pending image files are never treated as completed pages.

`retry_wait`, `unavailable` and `quarantined` are explicit queue states in schema 4.
`ready` is a healthy inspected job awaiting operator download confirmation, not a
stall. Inspection shares the source lane; it never downloads images or bypasses a
source circuit. Source coverage and job completion are separate observations.
Legacy unavailable-title errors are migrated; only pauses solely attributable to
these unavailable titles are released. Access and unknown failures remain paused.
Nightly discovery avoids duplicating outstanding, unavailable or quarantined jobs;
use an explicit operator retry after fixing their cause.

## Observability

**Settings → Service health** shows scraper reachability ("Scraper: unreachable" when
the reader cannot reach it), active checkpoints, failed jobs and the scraper's incident
history. Process crashes are handled by the supervisor and Docker restart policy, not
recorded as incidents. **Settings → Scraper** retains queue, pacing and discovery controls.
Service errors are classified rather than buried in a single historical `lastError`.
`npm run check` performs read-only checks of reader and scraper.

Scraper status is independent of expensive library scans. Concurrent admin/health
probes share one background inventory refresh and receive the previous summary
immediately. The dashboard summary refreshes at most once per minute unless a
publication or metadata change invalidates it; duplicate-sensitive decisions still
request a fresh inventory. `libraryStatus` reports ready/refreshing/stale/updated/error, and failed
background refreshes do not spawn a new scan on every UI poll. Authoritative
discovery waits for the current inventory; publication/metadata invalidation during
a scan forces another scan before it can be used for those decisions. Cold start
may temporarily show no library targets, but queue and rate controls remain usable.

Storage:

- `data/catalog.json`, `data/collection-covers/`: reader catalog and covers.
- `data/scraper/queue.json`, `events.log`: queue, circuits, failures and incidents
  (including stalls).
- `data/runtime-reader.json`, `data/scraper/runtime.json`: supervisor status and
  persisted crash budget.
- `data/sentinel/`: only on installs from before 2026-10-22; unused, safe to delete.
- `data/scraper/supervisor.lock`, `daemon.lock`: native process ownership; transient.

Scraper and supervisor emit structured logs. The queue history retains
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native supervisor logs
(`tools/run-supervisor.ps1`) remain append-only: archive them during stopped maintenance if needed.
Never export user databases, tokens, cookies, signed query strings or unredacted environments.

## What self-repair intentionally does not mean

The stall watchdog cannot execute a shell, patch adapters, access a Docker socket,
or restart processes; it only requeues or quarantines a job. Nothing at runtime
repairs broken markup or solves source challenges. The agent handles those escalations in the repository:

1. Inspect one bounded incident and relevant checkpoint; preserve the active lane.
2. Add a fixture reproducing the failure, patch the smallest adapter/logic surface.
3. Run `npm test`; test UI/API in isolated data if affected.
4. Build a traceable image, deploy checkpoint-safely, verify resumed progress.
5. Record release, backup and rollback compatibility. Stop after repeated identical failures.

A VM-local monitor cannot alert on total VM/network failure. Before VM cutover choose
an external health check and notification destination (e.g. an existing monitoring
service and private ntfy/Gotify). No notification credentials or destination are invented.

## Crash-budget operation

A parked supervisor stays alive and reports `repair-required`; Docker's restart policy
does not create an unlimited process restart storm. Its health check fails while the
child is absent. Investigate first. Once repaired, restart the wrapper after its
rolling one-hour budget has expired. For an intentional earlier reset, first stop the
service, back up its `runtime*.json`, and clear only that service's `failures` array;
never clear the queue or daemon checkpoint. Explicit deployments do not erase budgets.