Fold the sentinel into the scraper and drop the sentinel service (BL-0046)
The stall watchdog now runs inside the scraper (ScraperService.checkStalls, every 30 s from the work loop): same rule as before, one checkpoint-safe restart, quarantine on the second stall, missing checkpoint is no evidence, and paused time no longer counts. Stalls are recorded as scraper incidents (failure.kind 'stall'). The private /api/recovery endpoint is gone. Dropped with the service: reader/scraper-unreachable incidents (supervisor and Docker handle crashes; the health page shows "Scraper: unreachable"), duplicate job-failure incidents and the unused /metrics endpoint. The stack is now reader + scraper (+ optional peer-gateway). The supervisor test moved to runtime/test (npm run test:runtime stays). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
88555d4780
commit
b2bb34174c
37 files changed
+163
-311
No files matched your search
@@ -12,7 +12,7 @@ This is a discovery and curation feature, not a new image scraper, metadata auth
|
||||
|
||||
Current MangaCircus constraints remain: plain Node and browser JavaScript, Docker Compose, a folder-tree library, JSON state rather than a database, one reader owner of account state, one scraper owner of the queue and library publication, and source-specific slow request pacing. See [`docs/architecture.md`](../docs/architecture.md), [`docs/greatest.md`](../docs/greatest.md), and [`docs/scraper-service.md`](../docs/scraper-service.md).
|
||||
|
||||
MVP lives in the **reader service** as a bounded asynchronous job and a reader-owned JSON store. Do not add a permanently running fourth service at first. If real LLM or metadata latency later makes reader isolation necessary, move the job behind the same API to a one-off/optional worker container using the existing image; do not change the data contract or add a database merely to do that. Sentinel observes availability but does not judge recommendations, call an LLM, or repair research claims.
|
||||
MVP lives in the **reader service** as a bounded asynchronous job and a reader-owned JSON store. Do not add a permanently running third service at first. If real LLM or metadata latency later makes reader isolation necessary, move the job behind the same API to a one-off/optional worker container using the existing image; do not change the data contract or add a database merely to do that. Service health reports availability but does not judge recommendations, call an LLM, or repair research claims.
|
||||
|
||||
Explicit non-goals for the first release:
|
||||
|
||||
@@ -134,7 +134,7 @@ Queueing remains through `POST /api/admin/scraper` with `command:"enqueue"` and
|
||||
- One in-process run lock plus a persisted run ID prevents overlap across browser clicks and restarts. A stale interrupted run is marked interrupted on startup; keep the previous successful feed. Publish only after validation completes.
|
||||
- Preserve source errors and diagnostics with bounded log size. Do not store sensitive prompts or external response bodies indefinitely. Provider blocks/challenges require user attention, not automated retries or evasions.
|
||||
- Nightly scheduling comes after the manual path has passed acceptance. Make local-time schedule configurable, persist last-success time, skip duplicate same-day runs after restart, and allow a manual run without disturbing scraper nightly updates. A missed night can run once after startup, never a catch-up storm.
|
||||
- The feature is optional: reader, library, Greatest, scraper, and Sentinel continue normally if metadata or LLM access is absent.
|
||||
- The feature is optional: reader, library, Greatest, and scraper continue normally if metadata or LLM access is absent.
|
||||
|
||||
## 8. Phased implementation and acceptance
|
||||
|
||||
@@ -152,7 +152,7 @@ Add one configurable model adapter, schema validation, evidence-bound output, op
|
||||
|
||||
### Phase D — scheduling and operations
|
||||
|
||||
Add nightly schedule only after manual runs are stable. Expose last success/error and a `discovery_run_failed` incident to admin observability, but do not give Sentinel authority to rewrite recommendations or restart the scraper. Add backup/restore instructions for `data/discovery.json` and cache behavior. If runs measurably harm reader latency, move the job into an optional one-off Compose worker with one-writer handoff, not an always-on autonomous service.
|
||||
Add nightly schedule only after manual runs are stable. Expose last success/error and a `discovery_run_failed` incident to admin observability, but do not give any watchdog authority to rewrite recommendations or restart the scraper. Add backup/restore instructions for `data/discovery.json` and cache behavior. If runs measurably harm reader latency, move the job into an optional one-off Compose worker with one-writer handoff, not an always-on autonomous service.
|
||||
|
||||
### Suggested implementation map
|
||||
|
||||
|
||||
Reference in new issue
Block a user