Runtime resilience and agent operations

Ownership and failure boundaries

Browser → Reader ──private API──→ Scraper → library (atomic chapter publication)
             │                       ↑
             └──health──→ Sentinel ───┘ bounded checkpoint recovery only

Supervisor(reader)   Supervisor(scraper)   Supervisor(sentinel)
                         ↑
           Coding agent: inspect incidents → reproduce → patch → test → release

The reader is the only writer of users/progress and the catalog. The scraper is the only queue owner and content publisher. Sentinel owns only observations and incidents. Each service's supervisor owns only that child process. The coding agent is an escalation/maintenance operator, not a necessary part of normal downloading. The recurring Codex check is an additional backstop, not the runtime watchdog.

Automatic recovery implemented

Failure	Action	Automatic limit
Process exit or repeated failed liveness checks	Supervisor restarts only that service, with backoff	Three failures per rolling hour; then parked for operator action
Network/HTTP 5xx	Preserve checkpoint; open source circuit, wait five minutes, retry job	One job-level retry; request-level retries remain bounded
Title unavailable/no English chapters	Record unavailable, continue unrelated work	No blind retry
Parser/local adapter failure	Quarantine job, keep partial data, continue other titles	Two different affected jobs in an hour pause source as systemic failure
Storage/permission failure	Pause all downloads and report	No automatic retry
HTTP 401/403/429 or access challenge	Pause affected source lane	No automatic retry/bypass
Active job with no progress for 15 minutes	Sentinel requests cooperative checkpoint-safe retry	One stall recovery; next stall quarantined

The daemon remains authoritative: a recovery request includes the observed progress fingerprint and is rejected if the job has moved, paused, or is no longer stale. Sentinel also observes completed/page counts; a deliberately paused queue is not a stall. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds. Publication remains one complete chapter at a time, on the same filesystem. No recovery action raises pacing, creates another worker for the same source, changes identities or deletes partial images.

The publisher retains validated pages in .staging-mcscraper-resume-<sha256> folders on pause/failure. It checks the full chapter/source/page-manifest fingerprint before reuse and revalidates each image; completed chapters still publish with one rename. A changed manifest creates separate staging rather than mixing releases. Old random stages remain untouched for manual review: they cannot be safely associated from page filenames alone. Hidden pending image files are never treated as completed pages.

retry_wait, unavailable and quarantined are explicit queue states in schema 4. ready is a healthy inspected job awaiting operator download confirmation, not a stall. Inspection shares the source lane; it never downloads images or bypasses a source circuit. Source coverage and job completion are separate observations. Legacy unavailable-title errors are migrated; only pauses solely attributable to these unavailable titles are released. Access and unknown failures remain paused. Nightly discovery avoids duplicating outstanding, unavailable or quarantined jobs; use an explicit operator retry after fixing their cause.

Observability

Settings → Service health shows Sentinel status, active checkpoints, failed jobs and incident history. Settings → Scraper retains queue, pacing and discovery controls. Service errors are classified rather than buried in a single historical lastError. npm run check performs read-only checks. Sentinel's private, token-protected /metrics exports reader-up, scraper-up and open-incident gauges for a future Prometheus collector. It is not a public endpoint.

Scraper status is independent of expensive library scans. Concurrent admin/Sentinel probes share one background inventory refresh and receive the previous summary immediately. The dashboard summary refreshes at most once per minute unless a publication or metadata change invalidates it; duplicate-sensitive decisions still request a fresh inventory. libraryStatus reports ready/refreshing/stale/updated/error, and failed background refreshes do not spawn a new scan on every UI poll. Authoritative discovery waits for the current inventory; publication/metadata invalidation during a scan forces another scan before it can be used for those decisions. Cold start may temporarily show no library targets, but queue and rate controls remain usable.

Storage:

data/catalog.json, data/collection-covers/: reader catalog and covers.
data/scraper/queue.json, events.log: queue, circuits, failures and incidents.
data/sentinel/state.json: Sentinel incidents and last observation.
data/runtime-reader.json, data/scraper/runtime.json, data/sentinel/runtime.json: supervisor status and persisted crash budget.
data/scraper/supervisor.lock, daemon.lock: native process ownership; transient.

Scraper/Sentinel/supervisor emit structured logs. Queue and Sentinel histories retain 200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows supervisor logs remain append-only: archive them during stopped maintenance if needed. Never export user databases, tokens, cookies, signed query strings or unredacted environments.

What self-repair intentionally does not mean

Sentinel cannot execute a shell, patch adapters, query image sources, access a Docker socket, or issue unrestricted restart commands. It cannot repair broken markup or solve source challenges. The agent handles those escalations in the repository:

Inspect one bounded incident and relevant checkpoint; preserve the active lane.
Add a fixture reproducing the failure, patch the smallest adapter/logic surface.
Run npm test; test UI/API in isolated data if affected.
Build a traceable image, deploy checkpoint-safely, verify resumed progress.
Record release, backup and rollback compatibility. Stop after repeated identical failures.

A VM-local monitor cannot alert on total VM/network failure. Before VM cutover choose an external health check and notification destination (e.g. an existing monitoring service and private ntfy/Gotify). No notification credentials or destination are invented.

Crash-budget operation

A parked supervisor stays alive and reports repair-required; Docker's restart policy does not create an unlimited process restart storm. Its health check fails while the child is absent. Investigate first. Once repaired, restart the wrapper after its rolling one-hour budget has expired. For an intentional earlier reset, first stop the service, back up its runtime*.json, and clear only that service's failures array; never clear the queue or daemon checkpoint. Explicit deployments do not erase budgets.

7.0 KiB Raw Blame History