Fold the sentinel into the scraper and drop the sentinel service (BL-0046)

The stall watchdog now runs inside the scraper (ScraperService.checkStalls,
every 30 s from the work loop): same rule as before, one checkpoint-safe
restart, quarantine on the second stall, missing checkpoint is no evidence,
and paused time no longer counts. Stalls are recorded as scraper incidents
(failure.kind 'stall'). The private /api/recovery endpoint is gone.

Dropped with the service: reader/scraper-unreachable incidents (supervisor
and Docker handle crashes; the health page shows "Scraper: unreachable"),
duplicate job-failure incidents and the unused /metrics endpoint.
The stack is now reader + scraper (+ optional peer-gateway).
The supervisor test moved to runtime/test (npm run test:runtime stays).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
MaashkinoozheandClaude Opus 5.5 committed 2026-10-22 22:58:03 +02:00
1 parent 88555d4780
commit b2bb34174c
37 files changed
+163 -311

No files matched your search

+3 -3
View File
@@ -12,7 +12,7 @@ This is a discovery and curation feature, not a new image scraper, metadata auth
Current MangaCircus constraints remain: plain Node and browser JavaScript, Docker Compose, a folder-tree library, JSON state rather than a database, one reader owner of account state, one scraper owner of the queue and library publication, and source-specific slow request pacing. See [`docs/architecture.md`](../docs/architecture.md), [`docs/greatest.md`](../docs/greatest.md), and [`docs/scraper-service.md`](../docs/scraper-service.md).
MVP lives in the **reader service** as a bounded asynchronous job and a reader-owned JSON store. Do not add a permanently running fourth service at first. If real LLM or metadata latency later makes reader isolation necessary, move the job behind the same API to a one-off/optional worker container using the existing image; do not change the data contract or add a database merely to do that. Sentinel observes availability but does not judge recommendations, call an LLM, or repair research claims.
MVP lives in the **reader service** as a bounded asynchronous job and a reader-owned JSON store. Do not add a permanently running third service at first. If real LLM or metadata latency later makes reader isolation necessary, move the job behind the same API to a one-off/optional worker container using the existing image; do not change the data contract or add a database merely to do that. Service health reports availability but does not judge recommendations, call an LLM, or repair research claims.
Explicit non-goals for the first release:
@@ -134,7 +134,7 @@ Queueing remains through `POST /api/admin/scraper` with `command:"enqueue"` and
- One in-process run lock plus a persisted run ID prevents overlap across browser clicks and restarts. A stale interrupted run is marked interrupted on startup; keep the previous successful feed. Publish only after validation completes.
- Preserve source errors and diagnostics with bounded log size. Do not store sensitive prompts or external response bodies indefinitely. Provider blocks/challenges require user attention, not automated retries or evasions.
- Nightly scheduling comes after the manual path has passed acceptance. Make local-time schedule configurable, persist last-success time, skip duplicate same-day runs after restart, and allow a manual run without disturbing scraper nightly updates. A missed night can run once after startup, never a catch-up storm.
- The feature is optional: reader, library, Greatest, scraper, and Sentinel continue normally if metadata or LLM access is absent.
- The feature is optional: reader, library, Greatest, and scraper continue normally if metadata or LLM access is absent.
## 8. Phased implementation and acceptance
@@ -152,7 +152,7 @@ Add one configurable model adapter, schema validation, evidence-bound output, op
### Phase D — scheduling and operations
Add nightly schedule only after manual runs are stable. Expose last success/error and a `discovery_run_failed` incident to admin observability, but do not give Sentinel authority to rewrite recommendations or restart the scraper. Add backup/restore instructions for `data/discovery.json` and cache behavior. If runs measurably harm reader latency, move the job into an optional one-off Compose worker with one-writer handoff, not an always-on autonomous service.
Add nightly schedule only after manual runs are stable. Expose last success/error and a `discovery_run_failed` incident to admin observability, but do not give any watchdog authority to rewrite recommendations or restart the scraper. Add backup/restore instructions for `data/discovery.json` and cache behavior. If runs measurably harm reader latency, move the job into an optional one-off Compose worker with one-writer handoff, not an always-on autonomous service.
### Suggested implementation map
+1 -1
View File
@@ -107,7 +107,7 @@ Suggested NoDB files (names are design targets, not existing files): reader-owne
3. **Read-only publisher:** optional gateway, bounded approved-only catalog/manifest/blob endpoints, quota enforcement, read-only mounts. Test that unshared, partially downloaded, deleted, and revoked content cannot be listed or fetched—even by guessed IDs or stale manifests. Verify no user data fields or raw source URLs appear on the wire.
4. **Consumer browsing:** local-server proxy and reader UI with peer status, search, details, selection preview, and queue permission checks. No browser-to-peer credentials or direct requests.
5. **Scraper transfer lane:** chapter/page checkpoints, range resume, hashes, local throttles, source-aware deduplication, and existing atomic chapter publication. Test corruption, interrupted transfer, stale revision, out-of-space, repeated queue submission, and preservation of current jobs.
6. **Observability and operations:** per-peer health, bytes, throttling, grant expiry warnings, queue stalls, and safe retry controls. Sentinel may report a dead gateway or stuck transfer but must not renew trust, broaden a share, or bypass a quota. Document backup/restore and key-rotation runbooks.
6. **Observability and operations:** per-peer health, bytes, throttling, grant expiry warnings, queue stalls, and safe retry controls. Service health may report a dead gateway or stuck transfer but must not renew trust, broaden a share, or bypass a quota. Document backup/restore and key-rotation runbooks.
7. **Local then VM pilot:** three local Compose instances with distinct data directories; pair A↔B and A↔C, exercise every direction combination, and transfer a few admin-approved test chapters. Verify disabling B does not affect C, no account/progress exchange, and quota isolation. Only then deploy the optional service on the VM's private network. Leave existing reader/scraper ports and active downloads undisturbed during the pilot.
**Ship gate:** a fresh unpaired server sees nothing; a paired server with serving off or no granted manga sees an empty catalog; an expired/revoked peer cannot list or fetch; invitations cannot be reused; disabling one direction or peer leaves unrelated peers intact; quotas survive restart; a receiver can resume and publish one chapter without duplicates; and its browser never receives the grant token or any private key. The feature remains disabled by default until those gates pass.