Fold the sentinel into the scraper and drop the sentinel service (BL-0046)

The stall watchdog now runs inside the scraper (ScraperService.checkStalls,
every 30 s from the work loop): same rule as before, one checkpoint-safe
restart, quarantine on the second stall, missing checkpoint is no evidence,
and paused time no longer counts. Stalls are recorded as scraper incidents
(failure.kind 'stall'). The private /api/recovery endpoint is gone.

Dropped with the service: reader/scraper-unreachable incidents (supervisor
and Docker handle crashes; the health page shows "Scraper: unreachable"),
duplicate job-failure incidents and the unused /metrics endpoint.
The stack is now reader + scraper (+ optional peer-gateway).
The supervisor test moved to runtime/test (npm run test:runtime stays).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
MaashkinoozheandClaude Opus 5.5 committed 2026-10-22 22:58:03 +02:00
1 parent 88555d4780
commit b2bb34174c
37 files changed
+163 -311

No files matched your search

+29 -29
View File
@@ -4,16 +4,15 @@
```text
Browser → Reader ──private API──→ Scraper → library (atomic chapter publication)
│ ↑
└──health──→ Sentinel ───┘ bounded checkpoint recovery only
└─ stall watchdog: bounded checkpoint recovery only
Supervisor(reader) Supervisor(scraper) Supervisor(sentinel)
Supervisor(reader) Supervisor(scraper)
↑
Coding agent: inspect incidents → reproduce → patch → test → release
```
The reader is the only writer of users/progress and the catalog. The scraper is the
only queue owner and content publisher. Sentinel owns only observations and incidents.
only queue owner and content publisher, and it records its own failures and incidents.
Each service's supervisor owns only that child process. The coding agent is an
**escalation/maintenance operator**, not a necessary part of normal downloading.
The recurring Codex check is an additional backstop, not the runtime watchdog.
@@ -28,18 +27,18 @@ The recurring Codex check is an additional backstop, not the runtime watchdog.
| Parser/local adapter failure | Quarantine job, keep partial data, continue other titles | Two different affected jobs in an hour pause source as systemic failure |
| Storage/permission failure | Pause all downloads and report | No automatic retry |
| HTTP 401/403/429 or access challenge | Pause affected source lane | No automatic retry/bypass |
| Active job with no progress for 15 minutes | Sentinel requests cooperative checkpoint-safe retry | One stall recovery; next stall quarantined |
| Active job with no progress for 15 minutes | Scraper stall watchdog stops it at its next checkpoint and requeues it | One stall recovery; next stall quarantined |
Sentinel only treats an active job as stalled when its checkpoint can be read.
The scraper resolves source-specific work folders (including MangaPill's
`mangapill-` prefix) before reporting progress; a missing checkpoint is not
evidence that downloads stopped. An operator retry resets the job's stall
recovery count without deleting its partial download.
The daemon remains authoritative: a recovery request includes the observed progress
fingerprint and is rejected if the job has moved, paused, or is no longer stale.
Sentinel also observes completed/page counts; a deliberately paused queue is not a
stall. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds.
The stall watchdog runs inside the scraper (`ScraperService.checkStalls`, every 30 s
from the work loop). A job is stalled when its checkpoint fingerprint (completed
chapters/pages) has not moved for 15 minutes, or for one chapter delay plus three
minutes if that is longer. Time spent paused does not count, and a deliberately
paused queue is not a stall. It only treats a job as stalled when its checkpoint can
be read: the scraper resolves source-specific work folders (including MangaPill's
`mangapill-` prefix), and a missing checkpoint is not evidence that downloads stopped.
Each stall records an incident with `failure.kind: 'stall'` and an `events.log` entry.
An operator retry resets the job's stall recovery count without deleting its partial
download. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds.
Publication remains one complete chapter at a time, on the same filesystem.
No recovery action raises pacing, creates another worker for the same source,
changes identities or deletes partial images.
@@ -62,14 +61,14 @@ use an explicit operator retry after fixing their cause.
## Observability
**Settings → Service health** shows Sentinel status, active checkpoints, failed jobs
and incident history. **Settings → Scraper** retains queue, pacing and discovery controls.
**Settings → Service health** shows scraper reachability ("Scraper: unreachable" when
the reader cannot reach it), active checkpoints, failed jobs and the scraper's incident
history. Process crashes are handled by the supervisor and Docker restart policy, not
recorded as incidents. **Settings → Scraper** retains queue, pacing and discovery controls.
Service errors are classified rather than buried in a single historical `lastError`.
`npm run check` performs read-only checks. Sentinel's private, token-protected `/metrics`
exports reader-up, scraper-up and open-incident gauges for a future Prometheus collector.
It is not a public endpoint.
`npm run check` performs read-only checks of reader and scraper.
Scraper status is independent of expensive library scans. Concurrent admin/Sentinel
Scraper status is independent of expensive library scans. Concurrent admin/health
probes share one background inventory refresh and receive the previous summary
immediately. The dashboard summary refreshes at most once per minute unless a
publication or metadata change invalidates it; duplicate-sensitive decisions still
@@ -82,22 +81,23 @@ may temporarily show no library targets, but queue and rate controls remain usab
Storage:
- `data/catalog.json`, `data/collection-covers/`: reader catalog and covers.
- `data/scraper/queue.json`, `events.log`: queue, circuits, failures and incidents.
- `data/sentinel/state.json`: Sentinel incidents and last observation.
- `data/runtime-reader.json`, `data/scraper/runtime.json`, `data/sentinel/runtime.json`:
supervisor status and persisted crash budget.
- `data/scraper/queue.json`, `events.log`: queue, circuits, failures and incidents
(including stalls).
- `data/runtime-reader.json`, `data/scraper/runtime.json`: supervisor status and
persisted crash budget.
- `data/sentinel/`: only on installs from before 2026-10-22; unused, safe to delete.
- `data/scraper/supervisor.lock`, `daemon.lock`: native process ownership; transient.
Scraper/Sentinel/supervisor emit structured logs. Queue and Sentinel histories retain
Scraper and supervisor emit structured logs. The queue history retains
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows
supervisor logs remain append-only: archive them during stopped maintenance if needed.
Never export user databases, tokens, cookies, signed query strings or unredacted environments.
## What self-repair intentionally does not mean
Sentinel cannot execute a shell, patch adapters, query image sources, access a Docker
socket, or issue unrestricted restart commands. It cannot repair broken markup or
solve source challenges. The agent handles those escalations in the repository:
The stall watchdog cannot execute a shell, patch adapters, access a Docker socket,
or restart processes; it only requeues or quarantines a job. Nothing at runtime
repairs broken markup or solves source challenges. The agent handles those escalations in the repository:
1. Inspect one bounded incident and relevant checkpoint; preserve the active lane.
2. Add a fixture reproducing the failure, patch the smallest adapter/logic surface.