MangaCircus/docs/resilience.md

Runtime resilience and agent operations

Ownership and failure boundaries

Browser → Reader ──private API──→ Scraper → library (atomic chapter publication)
                                     └─ stall watchdog: bounded checkpoint recovery only

Supervisor(reader)   Supervisor(scraper)
                         ↑
           Coding agent: inspect incidents → reproduce → patch → test → release

The reader is the only writer of users/progress and the catalog. The scraper is the only queue owner and content publisher, and it records its own failures and incidents. Each service's supervisor owns only that child process. The coding agent is an escalation/maintenance operator, not a necessary part of normal downloading. The recurring Codex check is an additional backstop, not the runtime watchdog.

Automatic recovery implemented

Failure	Action	Automatic limit
Process exit or repeated failed liveness checks	Supervisor restarts only that service, with backoff	Three failures per rolling hour; then parked for operator action
Network/HTTP 5xx	Preserve checkpoint; open source circuit, wait five minutes, retry job	One job-level retry; request-level retries remain bounded
Title unavailable/no English chapters	Record unavailable, continue unrelated work	No blind retry
Parser/local adapter failure	Quarantine job, keep partial data, continue other titles	Two different affected jobs in an hour pause source as systemic failure
Storage/permission failure	Pause all downloads and report	No automatic retry
HTTP 401/403/429 or access challenge	Pause affected source lane	No automatic retry/bypass
Active job with no progress for 15 minutes	Scraper stall watchdog stops it at its next checkpoint and requeues it	One stall recovery; next stall quarantined

The stall watchdog runs inside the scraper (ScraperService.checkStalls, every 30 s from the work loop). A job is stalled when its checkpoint fingerprint (completed chapters/pages) has not moved for 15 minutes, or for one chapter delay plus three minutes if that is longer. Time spent paused does not count, and a deliberately paused queue is not a stall. It only treats a job as stalled when its checkpoint can be read: the scraper resolves source-specific work folders (including MangaPill's mangapill- prefix), and a missing checkpoint is not evidence that downloads stopped. Each stall records an incident with failure.kind: 'stall' and an events.log entry. An operator retry resets the job's stall recovery count without deleting its partial download. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds. Publication remains one complete chapter at a time, on the same filesystem. No recovery action raises pacing, creates another worker for the same source, changes identities or deletes partial images.

The publisher retains validated pages in .staging-mcscraper-resume-<sha256> folders on pause/failure. It checks the full chapter/source/page-manifest fingerprint before reuse and revalidates each image; completed chapters still publish with one rename. A changed manifest creates separate staging rather than mixing releases. Old random stages remain untouched for manual review: they cannot be safely associated from page filenames alone. Hidden pending image files are never treated as completed pages.

retry_wait, unavailable and quarantined are explicit queue states in schema 4. ready is a healthy inspected job awaiting operator download confirmation, not a stall. Inspection shares the source lane; it never downloads images or bypasses a source circuit. Source coverage and job completion are separate observations. Legacy unavailable-title errors are migrated; only pauses solely attributable to these unavailable titles are released. Access and unknown failures remain paused. Nightly discovery avoids duplicating outstanding, unavailable or quarantined jobs; use an explicit operator retry after fixing their cause.

Observability

Settings → Service health shows scraper reachability ("Scraper: unreachable" when the reader cannot reach it), active checkpoints, failed jobs and the scraper's incident history. Process crashes are handled by the supervisor and Docker restart policy, not recorded as incidents. Settings → Scraper retains queue, pacing and discovery controls. Service errors are classified rather than buried in a single historical lastError. npm run check performs read-only checks of reader and scraper.

Scraper status is independent of expensive library scans. Concurrent admin/health probes share one background inventory refresh and receive the previous summary immediately. The dashboard summary refreshes at most once per minute unless a publication or metadata change invalidates it; duplicate-sensitive decisions still request a fresh inventory. libraryStatus reports ready/refreshing/stale/updated/error, and failed background refreshes do not spawn a new scan on every UI poll. Authoritative discovery waits for the current inventory; publication/metadata invalidation during a scan forces another scan before it can be used for those decisions. Cold start may temporarily show no library targets, but queue and rate controls remain usable.

Storage:

• data/catalog.json, data/collection-covers/: reader catalog and covers.
• data/scraper/queue.json, events.log: queue, circuits, failures and incidents (including stalls).
• data/runtime-reader.json, data/scraper/runtime.json: supervisor status and persisted crash budget.
• data/sentinel/: only on installs from before 2026-10-22; unused, safe to delete.
• data/scraper/supervisor.lock, daemon.lock: native process ownership; transient.

Scraper and supervisor emit structured logs. The queue history retains 200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows supervisor logs remain append-only: archive them during stopped maintenance if needed. Never export user databases, tokens, cookies, signed query strings or unredacted environments.

What self-repair intentionally does not mean

The stall watchdog cannot execute a shell, patch adapters, access a Docker socket, or restart processes; it only requeues or quarantines a job. Nothing at runtime repairs broken markup or solves source challenges. The agent handles those escalations in the repository:

1. Inspect one bounded incident and relevant checkpoint; preserve the active lane.
2. Add a fixture reproducing the failure, patch the smallest adapter/logic surface.
3. Run npm test; test UI/API in isolated data if affected.
4. Build a traceable image, deploy checkpoint-safely, verify resumed progress.
5. Record release, backup and rollback compatibility. Stop after repeated identical failures.

A VM-local monitor cannot alert on total VM/network failure. Before VM cutover choose an external health check and notification destination (e.g. an existing monitoring service and private ntfy/Gotify). No notification credentials or destination are invented.

Crash-budget operation

A parked supervisor stays alive and reports repair-required; Docker's restart policy does not create an unlimited process restart storm. Its health check fails while the child is absent. Investigate first. Once repaired, restart the wrapper after its rolling one-hour budget has expired. For an intentional earlier reset, first stop the service, back up its runtime*.json, and clear only that service's failures array; never clear the queue or daemon checkpoint. Explicit deployments do not erase budgets.