Fold the sentinel into the scraper and drop the sentinel service (BL-0046)
The stall watchdog now runs inside the scraper (ScraperService.checkStalls, every 30 s from the work loop): same rule as before, one checkpoint-safe restart, quarantine on the second stall, missing checkpoint is no evidence, and paused time no longer counts. Stalls are recorded as scraper incidents (failure.kind 'stall'). The private /api/recovery endpoint is gone. Dropped with the service: reader/scraper-unreachable incidents (supervisor and Docker handle crashes; the health page shows "Scraper: unreachable"), duplicate job-failure incidents and the unused /metrics endpoint. The stack is now reader + scraper (+ optional peer-gateway). The supervisor test moved to runtime/test (npm run test:runtime stays). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
88555d4780
commit
b2bb34174c
37 files changed
+163
-311
No files matched your search
+29
-29
@@ -4,16 +4,15 @@
|
||||
|
||||
```text
|
||||
Browser → Reader ──private API──→ Scraper → library (atomic chapter publication)
|
||||
│ ↑
|
||||
└──health──→ Sentinel ───┘ bounded checkpoint recovery only
|
||||
└─ stall watchdog: bounded checkpoint recovery only
|
||||
|
||||
Supervisor(reader) Supervisor(scraper) Supervisor(sentinel)
|
||||
Supervisor(reader) Supervisor(scraper)
|
||||
↑
|
||||
Coding agent: inspect incidents → reproduce → patch → test → release
|
||||
```
|
||||
|
||||
The reader is the only writer of users/progress and the catalog. The scraper is the
|
||||
only queue owner and content publisher. Sentinel owns only observations and incidents.
|
||||
only queue owner and content publisher, and it records its own failures and incidents.
|
||||
Each service's supervisor owns only that child process. The coding agent is an
|
||||
**escalation/maintenance operator**, not a necessary part of normal downloading.
|
||||
The recurring Codex check is an additional backstop, not the runtime watchdog.
|
||||
@@ -28,18 +27,18 @@ The recurring Codex check is an additional backstop, not the runtime watchdog.
|
||||
| Parser/local adapter failure | Quarantine job, keep partial data, continue other titles | Two different affected jobs in an hour pause source as systemic failure |
|
||||
| Storage/permission failure | Pause all downloads and report | No automatic retry |
|
||||
| HTTP 401/403/429 or access challenge | Pause affected source lane | No automatic retry/bypass |
|
||||
| Active job with no progress for 15 minutes | Sentinel requests cooperative checkpoint-safe retry | One stall recovery; next stall quarantined |
|
||||
| Active job with no progress for 15 minutes | Scraper stall watchdog stops it at its next checkpoint and requeues it | One stall recovery; next stall quarantined |
|
||||
|
||||
Sentinel only treats an active job as stalled when its checkpoint can be read.
|
||||
The scraper resolves source-specific work folders (including MangaPill's
|
||||
`mangapill-` prefix) before reporting progress; a missing checkpoint is not
|
||||
evidence that downloads stopped. An operator retry resets the job's stall
|
||||
recovery count without deleting its partial download.
|
||||
|
||||
The daemon remains authoritative: a recovery request includes the observed progress
|
||||
fingerprint and is rejected if the job has moved, paused, or is no longer stale.
|
||||
Sentinel also observes completed/page counts; a deliberately paused queue is not a
|
||||
stall. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds.
|
||||
The stall watchdog runs inside the scraper (`ScraperService.checkStalls`, every 30 s
|
||||
from the work loop). A job is stalled when its checkpoint fingerprint (completed
|
||||
chapters/pages) has not moved for 15 minutes, or for one chapter delay plus three
|
||||
minutes if that is longer. Time spent paused does not count, and a deliberately
|
||||
paused queue is not a stall. It only treats a job as stalled when its checkpoint can
|
||||
be read: the scraper resolves source-specific work folders (including MangaPill's
|
||||
`mangapill-` prefix), and a missing checkpoint is not evidence that downloads stopped.
|
||||
Each stall records an incident with `failure.kind: 'stall'` and an `events.log` entry.
|
||||
An operator retry resets the job's stall recovery count without deleting its partial
|
||||
download. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds.
|
||||
Publication remains one complete chapter at a time, on the same filesystem.
|
||||
No recovery action raises pacing, creates another worker for the same source,
|
||||
changes identities or deletes partial images.
|
||||
@@ -62,14 +61,14 @@ use an explicit operator retry after fixing their cause.
|
||||
|
||||
## Observability
|
||||
|
||||
**Settings → Service health** shows Sentinel status, active checkpoints, failed jobs
|
||||
and incident history. **Settings → Scraper** retains queue, pacing and discovery controls.
|
||||
**Settings → Service health** shows scraper reachability ("Scraper: unreachable" when
|
||||
the reader cannot reach it), active checkpoints, failed jobs and the scraper's incident
|
||||
history. Process crashes are handled by the supervisor and Docker restart policy, not
|
||||
recorded as incidents. **Settings → Scraper** retains queue, pacing and discovery controls.
|
||||
Service errors are classified rather than buried in a single historical `lastError`.
|
||||
`npm run check` performs read-only checks. Sentinel's private, token-protected `/metrics`
|
||||
exports reader-up, scraper-up and open-incident gauges for a future Prometheus collector.
|
||||
It is not a public endpoint.
|
||||
`npm run check` performs read-only checks of reader and scraper.
|
||||
|
||||
Scraper status is independent of expensive library scans. Concurrent admin/Sentinel
|
||||
Scraper status is independent of expensive library scans. Concurrent admin/health
|
||||
probes share one background inventory refresh and receive the previous summary
|
||||
immediately. The dashboard summary refreshes at most once per minute unless a
|
||||
publication or metadata change invalidates it; duplicate-sensitive decisions still
|
||||
@@ -82,22 +81,23 @@ may temporarily show no library targets, but queue and rate controls remain usab
|
||||
Storage:
|
||||
|
||||
- `data/catalog.json`, `data/collection-covers/`: reader catalog and covers.
|
||||
- `data/scraper/queue.json`, `events.log`: queue, circuits, failures and incidents.
|
||||
- `data/sentinel/state.json`: Sentinel incidents and last observation.
|
||||
- `data/runtime-reader.json`, `data/scraper/runtime.json`, `data/sentinel/runtime.json`:
|
||||
supervisor status and persisted crash budget.
|
||||
- `data/scraper/queue.json`, `events.log`: queue, circuits, failures and incidents
|
||||
(including stalls).
|
||||
- `data/runtime-reader.json`, `data/scraper/runtime.json`: supervisor status and
|
||||
persisted crash budget.
|
||||
- `data/sentinel/`: only on installs from before 2026-10-22; unused, safe to delete.
|
||||
- `data/scraper/supervisor.lock`, `daemon.lock`: native process ownership; transient.
|
||||
|
||||
Scraper/Sentinel/supervisor emit structured logs. Queue and Sentinel histories retain
|
||||
Scraper and supervisor emit structured logs. The queue history retains
|
||||
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows
|
||||
supervisor logs remain append-only: archive them during stopped maintenance if needed.
|
||||
Never export user databases, tokens, cookies, signed query strings or unredacted environments.
|
||||
|
||||
## What self-repair intentionally does not mean
|
||||
|
||||
Sentinel cannot execute a shell, patch adapters, query image sources, access a Docker
|
||||
socket, or issue unrestricted restart commands. It cannot repair broken markup or
|
||||
solve source challenges. The agent handles those escalations in the repository:
|
||||
The stall watchdog cannot execute a shell, patch adapters, access a Docker socket,
|
||||
or restart processes; it only requeues or quarantines a job. Nothing at runtime
|
||||
repairs broken markup or solves source challenges. The agent handles those escalations in the repository:
|
||||
|
||||
1. Inspect one bounded incident and relevant checkpoint; preserve the active lane.
|
||||
2. Add a fixture reproducing the failure, patch the smallest adapter/logic surface.
|
||||
|
||||
Reference in new issue
Block a user