120 lines
7.4 KiB
Markdown
120 lines
7.4 KiB
Markdown
# Runtime resilience and agent operations
|
|
|
|
## Ownership and failure boundaries
|
|
|
|
```text
|
|
Browser → Reader ──private API──→ Scraper → library (atomic chapter publication)
|
|
│ ↑
|
|
└──health──→ Sentinel ───┘ bounded checkpoint recovery only
|
|
|
|
Supervisor(reader) Supervisor(scraper) Supervisor(sentinel)
|
|
↑
|
|
Coding agent: inspect incidents → reproduce → patch → test → release
|
|
```
|
|
|
|
The reader is the only writer of users/progress and the catalog. The scraper is the
|
|
only queue owner and content publisher. Sentinel owns only observations and incidents.
|
|
Each service's supervisor owns only that child process. The coding agent is an
|
|
**escalation/maintenance operator**, not a necessary part of normal downloading.
|
|
The recurring Codex check is an additional backstop, not the runtime watchdog.
|
|
|
|
## Automatic recovery implemented
|
|
|
|
| Failure | Action | Automatic limit |
|
|
|---|---|---|
|
|
| Process exit or repeated failed liveness checks | Supervisor restarts only that service, with backoff | Three failures per rolling hour; then parked for operator action |
|
|
| Network/HTTP 5xx | Preserve checkpoint; open source circuit, wait five minutes, retry job | One job-level retry; request-level retries remain bounded |
|
|
| Title unavailable/no English chapters | Record unavailable, continue unrelated work | No blind retry |
|
|
| Parser/local adapter failure | Quarantine job, keep partial data, continue other titles | Two different affected jobs in an hour pause source as systemic failure |
|
|
| Storage/permission failure | Pause all downloads and report | No automatic retry |
|
|
| HTTP 401/403/429 or access challenge | Pause affected source lane | No automatic retry/bypass |
|
|
| Active job with no progress for 15 minutes | Sentinel requests cooperative checkpoint-safe retry | One stall recovery; next stall quarantined |
|
|
|
|
Sentinel only treats an active job as stalled when its checkpoint can be read.
|
|
The scraper resolves source-specific work folders (including MangaPill's
|
|
`mangapill-` prefix) before reporting progress; a missing checkpoint is not
|
|
evidence that downloads stopped. An operator retry resets the job's stall
|
|
recovery count without deleting its partial download.
|
|
|
|
The daemon remains authoritative: a recovery request includes the observed progress
|
|
fingerprint and is rejected if the job has moved, paused, or is no longer stale.
|
|
Sentinel also observes completed/page counts; a deliberately paused queue is not a
|
|
stall. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds.
|
|
Publication remains one complete chapter at a time, on the same filesystem.
|
|
No recovery action raises pacing, creates another worker for the same source,
|
|
changes identities or deletes partial images.
|
|
|
|
The publisher retains validated pages in `.staging-mcscraper-resume-<sha256>` folders
|
|
on pause/failure. It checks the full chapter/source/page-manifest fingerprint before
|
|
reuse and revalidates each image; completed chapters still publish with one rename.
|
|
A changed manifest creates separate staging rather than mixing releases. Old random
|
|
stages remain untouched for manual review: they cannot be safely associated from
|
|
page filenames alone. Hidden pending image files are never treated as completed pages.
|
|
|
|
`retry_wait`, `unavailable` and `quarantined` are explicit queue states in schema 4.
|
|
`ready` is a healthy inspected job awaiting operator download confirmation, not a
|
|
stall. Inspection shares the source lane; it never downloads images or bypasses a
|
|
source circuit. Source coverage and job completion are separate observations.
|
|
Legacy unavailable-title errors are migrated; only pauses solely attributable to
|
|
these unavailable titles are released. Access and unknown failures remain paused.
|
|
Nightly discovery avoids duplicating outstanding, unavailable or quarantined jobs;
|
|
use an explicit operator retry after fixing their cause.
|
|
|
|
## Observability
|
|
|
|
**Settings → Service health** shows Sentinel status, active checkpoints, failed jobs
|
|
and incident history. **Settings → Scraper** retains queue, pacing and discovery controls.
|
|
Service errors are classified rather than buried in a single historical `lastError`.
|
|
`npm run check` performs read-only checks. Sentinel's private, token-protected `/metrics`
|
|
exports reader-up, scraper-up and open-incident gauges for a future Prometheus collector.
|
|
It is not a public endpoint.
|
|
|
|
Scraper status is independent of expensive library scans. Concurrent admin/Sentinel
|
|
probes share one background inventory refresh and receive the previous summary
|
|
immediately. The dashboard summary refreshes at most once per minute unless a
|
|
publication or metadata change invalidates it; duplicate-sensitive decisions still
|
|
request a fresh inventory. `libraryStatus` reports ready/refreshing/stale/updated/error, and failed
|
|
background refreshes do not spawn a new scan on every UI poll. Authoritative
|
|
discovery waits for the current inventory; publication/metadata invalidation during
|
|
a scan forces another scan before it can be used for those decisions. Cold start
|
|
may temporarily show no library targets, but queue and rate controls remain usable.
|
|
|
|
Storage:
|
|
|
|
- `data/catalog.json`, `data/collection-covers/`: reader catalog and covers.
|
|
- `data/scraper/queue.json`, `events.log`: queue, circuits, failures and incidents.
|
|
- `data/sentinel/state.json`: Sentinel incidents and last observation.
|
|
- `data/runtime-reader.json`, `data/scraper/runtime.json`, `data/sentinel/runtime.json`:
|
|
supervisor status and persisted crash budget.
|
|
- `data/scraper/supervisor.lock`, `daemon.lock`: native process ownership; transient.
|
|
|
|
Scraper/Sentinel/supervisor emit structured logs. Queue and Sentinel histories retain
|
|
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows
|
|
supervisor logs remain append-only: archive them during stopped maintenance if needed.
|
|
Never export user databases, tokens, cookies, signed query strings or unredacted environments.
|
|
|
|
## What self-repair intentionally does not mean
|
|
|
|
Sentinel cannot execute a shell, patch adapters, query image sources, access a Docker
|
|
socket, or issue unrestricted restart commands. It cannot repair broken markup or
|
|
solve source challenges. The agent handles those escalations in the repository:
|
|
|
|
1. Inspect one bounded incident and relevant checkpoint; preserve the active lane.
|
|
2. Add a fixture reproducing the failure, patch the smallest adapter/logic surface.
|
|
3. Run `npm test`; test UI/API in isolated data if affected.
|
|
4. Build a traceable image, deploy checkpoint-safely, verify resumed progress.
|
|
5. Record release, backup and rollback compatibility. Stop after repeated identical failures.
|
|
|
|
A VM-local monitor cannot alert on total VM/network failure. Before VM cutover choose
|
|
an external health check and notification destination (e.g. an existing monitoring
|
|
service and private ntfy/Gotify). No notification credentials or destination are invented.
|
|
|
|
## Crash-budget operation
|
|
|
|
A parked supervisor stays alive and reports `repair-required`; Docker's restart policy
|
|
does not create an unlimited process restart storm. Its health check fails while the
|
|
child is absent. Investigate first. Once repaired, restart the wrapper after its
|
|
rolling one-hour budget has expired. For an intentional earlier reset, first stop the
|
|
service, back up its `runtime*.json`, and clear only that service's `failures` array;
|
|
never clear the queue or daemon checkpoint. Explicit deployments do not erase budgets.
|