Plan Discovery Scout recommendation architecture

vision/discovery-scout.md

@@ -0,0 +1,187 @@
+# Discovery Scout — personalized manga discovery
+
+Status: **vision and implementation plan, not implemented**. Updated 2026-10-09.
+
+## 1. Outcome
+
+Give each MangaCircus user a small, explainable feed of manga they might genuinely want to read. It should discover beyond the existing library and the historical *Shijō Saikō no Manga* list without turning either into an automatic download list. A recommendation must answer **why this user might care**, show where its factual claims came from, and admit uncertainty. The user can express interest or dismiss it. An admin may then choose an edition and enter the existing scraper's **inspect → review → confirm download** flow.
+
+This is a discovery and curation feature, not a new image scraper, metadata authority, collection curator, or health watchdog. It must remain useful with no LLM configured.
+
+## 2. Architectural fit and non-goals
+
+Current MangaCircus constraints remain: plain Node and browser JavaScript, Docker Compose, a folder-tree library, JSON state rather than a database, one reader owner of account state, one scraper owner of the queue and library publication, and source-specific slow request pacing. See [`docs/architecture.md`](../docs/architecture.md), [`docs/greatest.md`](../docs/greatest.md), and [`docs/scraper-service.md`](../docs/scraper-service.md).
+
+MVP lives in the **reader service** as a bounded asynchronous job and a reader-owned JSON store. Do not add a permanently running fourth service at first. If real LLM or metadata latency later makes reader isolation necessary, move the job behind the same API to a one-off/optional worker container using the existing image; do not change the data contract or add a database merely to do that. Sentinel observes availability but does not judge recommendations, call an LLM, or repair research claims.
+
+Explicit non-goals for the first release:
+
+- No automatic queue edit, download, cover scraping, collection edit, or relationship approval.
+- No bulk crawl of MangaDex, WeebCentral, or other chapter/image hosters.
+- No autonomous browsing agent with unrestricted URLs or filesystem access.
+- No public/community recommendation ranking based on other users' private reading times.
+- No claim that a title search proves a particular edition or chapter availability.
+
+## 3. User experience
+
+Add a **Discover** page, reachable from the main navigation. Keep it separate from the library dashboard and the fixed Greatest survey. It displays at most 20 current suggestions per user, in a compact tile grid with cover, title, original release year, status, and one personalized sentence. A detail panel shows:
+
+1. **Why for you:** specific connections to explicit votes, owned/read titles, creators, or tags; distinguish an editorial hunch from a sourced fact.
+2. **Evidence:** linked public metadata or publisher/award sources, with observed date and uncertainty. Do not show a bare score as proof of quality.
+3. **Already here?** owned-library match, existing Greatest entry, and—admin only—scraper queue/inspection state.
+4. **Actions:** Interested, Maybe, Not for me, Hide; admin-only **Find an edition** reuses the existing source-link/inspection UI. A user can undo feedback.
+
+Explain why a suggestion disappeared: acquired, dismissed, stale source identity, or superseded by a newer run. The feed should not jump while the user is reading it; an active run publishes a complete replacement atomically. Empty state should say whether no candidate passed filtering, metadata is unavailable, or discovery has never run. Show the last successful run time and a nonintrusive error summary; retain the previous successful feed on failure.
+
+User-facing toggles: **Personalized discovery** (default off until user opts in), **Use reading activity** (separate opt-in; default off), and **Show mature titles** (default off). Disabling personalization stops new profile-based runs for that user but retains or deletes their feedback according to a clear Manage Data control. Ordinary users can view and rate their own suggestions. Admins can run discovery, configure sources/budgets, and inspect diagnostics. The first account/admin is not implicitly enrolled in personalized processing.
+
+## 4. Evidence and candidate pipeline
+
+Use a two-stage pipeline. **Code** gathers, identifies, filters, and computes reproducible signals; an optional **agent** explains or reranks a small shortlist. The agent never resolves identity by itself.
+
+### Candidate input, in priority order
+
+1. Reuse public AniList metadata/relations already cached by MangaCircus, plus the read-only Greatest catalog, for cold start and sequel/creator leads. Existing *Greatest* votes are stronger preference signals than curator rank.
+2. Add one or two explicit, public metadata feeds only after checking their API terms and stable identity fields. Cache responses with timestamps. Publisher/award lists may seed editorial candidates, but only with source links and dates.
+3. Let an admin seed a candidate by metadata URL or exact provider ID. This is the fallback when public APIs are incomplete. It is **not** a scraper URL and creates no queue item.
+
+MangaDex's metadata API may be used only for an on-demand edition lookup after a recommendation is opened, as in today's Greatest details. WeebCentral is not a discovery source. Neither image hoster is asked for chapter catalogs or image URLs during recommendation generation. On HTTP 401/403/429 or a challenge, stop that provider's run and surface the condition; never rotate identity, proxies, hosts, or user agents.
+
+### Identity and filtering
+
+Prefer stable provider IDs. Store original serialization year and type so a sequel, remake, novel, anime, or namesake does not silently inherit another manga's reputation. Exact ID beats title; normalized title plus matching year/type may be a **candidate** match, never an automatic merge. Ambiguous matches go to an admin review list. Deduplicate against the library, Greatest entries, current scraper queue, dismissed titles, and prior suggestions. A related work can still be suggested, but must be labelled as a sequel/prequel and never assume the user has read the prerequisite.
+
+No fabricated covers, rankings, awards, chapter counts, or source URLs. Missing data stays missing. The browser can lazy-load a provider's cover URL, as the Greatest page does; the Scout job does not download cover images.
+
+### Scoring before any LLM
+
+Compute an inspectable score from bounded signals, then deliberately diversify the shortlist:
+
+| Signal | Use | Caution |
+|---|---|---|
+| Explicit Interested/Maybe/Not for me and Greatest votes | Strongest positive/negative preference | A skull is a title dismissal, not a ban on the whole genre |
+| Owned and actually read manga | Moderate similarity and creator/series links | Ownership alone is weak evidence of taste |
+| Follows and completed chapters | Moderate positive signal | No assumption that a long series is loved |
+| Optional reading activity | Small, normalized signal only | Raw minutes favor long chapters and slow readers; never compare users |
+| Public recognition and editorial seed | Candidate quality prior | A prize or rank is evidence, not a personalized recommendation |
+| Novelty/diversity | Reserve roughly 20% of slots outside the strongest inferred taste cluster | Avoid 20 near-identical dark-SF picks |
+
+Save score components for debugging but show prose rather than a pseudo-scientific percentage. Seed a new account with a short taste picker and the Greatest votes; do not wait for months of reading history. Rejected or hidden titles should not reappear on the next run merely because metadata changed.
+
+## 5. Optional agent contract
+
+The first release can use deterministic templates for “why for you.” Add a provider-independent LLM adapter only after the deterministic pipeline and UI are trusted. A run sends at most the top 30 candidates, a **minimized** account-specific taste summary, and evidence IDs/URLs. Do not send usernames, passwords, tokens, raw browsing logs, notes, exact reading timestamps, or the whole library file tree. Make external-model processing opt-in and identify the provider in settings.
+
+The model's permitted work is to: (a) choose a diverse subset, (b) write concise personalized rationales grounded in supplied evidence, (c) flag identity/relation conflicts and uncertainty. It may suggest a public source to check, but it may not create a “verified” fact or direct series URL from memory. Treat model output and fetched pages as untrusted data, never as instructions. No tool access to the scraper, filesystem writes, container runtime, arbitrary HTTP destinations, or admin credentials.
+
+Accept only schema-valid output, for example:
+
+```json
+{
+  "candidateId": "anilist:12345",
+  "reason": "Because you marked BLAME! as a favorite, this explores similarly spare architectural SF.",
+  "reasonKind": "personal-fit",
+  "evidenceIds": ["taste:greatest:blame-1997", "metadata:anilist:12345"],
+  "uncertainty": "Edition and English availability not checked"
+}
+```
+
+Validate that `candidateId` and every `evidenceId` were in the input, that the reason is short and contains no unsupported objective claim, and that URLs belong to the cited evidence. Reject or fall back to the deterministic template on invalid output, timeout, budget exhaustion, or provider failure. Record model name/version and prompt schema version **without** storing the full private prompt. No recursive retries or self-initiated second research pass.
+
+## 6. Data ownership and proposed contract
+
+Keep the reader as the sole writer. Add `data/discovery.json` for global candidate records, per-user published feed snapshots, run metadata, and a schema/revision number. Write through a temporary file and atomic rename; preserve a recoverable previous version as existing JSON stores do. Keep public metadata responses in a separate bounded `data/discovery-cache/` with TTL and size cap. Store each user's explicit feedback/consent in `data/db.json` (for example `discoveryFeedback` and settings keys) so account deletion and backup rules remain consistent. Stable candidate IDs are `provider:id`; never use a normalized title as the durable key.
+
+Minimal record shape (illustrative, not a migration already made):
+
+```json
+{
+  "schema": 1,
+  "revision": 12,
+  "candidates": {
+    "anilist:12345": {
+      "title": "Example Manga", "year": 2024, "type": "manga",
+      "metadataUrl": "https://anilist.co/manga/12345",
+      "coverUrl": null, "fetchedAt": "2026-10-09T12:00:00Z",
+      "evidence": [{ "id": "metadata:anilist:12345", "kind": "metadata", "url": "https://anilist.co/manga/12345" }]
+    }
+  },
+  "feeds": {
+    "alice": { "generatedAt": "2026-10-09T12:05:00Z", "items": [
+      { "candidateId": "anilist:12345", "rank": 1, "reason": "…",
+        "reasonKind": "personal-fit", "evidenceIds": ["metadata:anilist:12345"],
+        "scoreParts": { "explicitTaste": 3, "diversity": 1 } }
+    ] }
+  },
+  "lastRun": { "status": "success", "startedAt": "…", "finishedAt": "…", "candidateCount": 40 }
+}
+```
+
+Reader API proposal:
+
+- `GET /api/discovery` — only the caller's feed, feedback, available evidence, and run freshness. Never return another user's taste/profile.
+- `POST /api/discovery/feedback` — `{candidateId, value}` where value is `interested|maybe|dismissed|hidden|null`; authenticated, validated, idempotent. `null` undoes feedback.
+- `GET /api/admin/discovery` — run health, budgets, candidate/identity-review counts, source errors, last run; no private cross-user raw reading logs.
+- `POST /api/admin/discovery/run` — starts at most one bounded run; returns a run ID or `409` if one is active. Manual MVP trigger.
+- `POST /api/admin/discovery/settings` — validated enabled flag, schedule, source budgets, optional model selection; no arbitrary URL or prompt injection fields.
+
+Queueing remains through `POST /api/admin/scraper` with `command:"enqueue"` and `inspect:true`, only after a human chooses a **series** URL. Reuse the existing duplicate/status UI and show the admin the source edition before confirmation. The Scout cannot call that route.
+
+## 7. Execution limits and failure behavior
+
+- Manual runs first. Default cap: 20 published suggestions/user, 50 candidate records examined/run, one metadata request at a time, at least 1.5 seconds between uncached requests, 10-minute wall-clock budget. These are starting limits to verify against provider terms and real response rates before enabling a schedule.
+- Cache by provider ID; avoid rerequesting unchanged metadata within a 30-day TTL unless an admin explicitly refreshes it. Store `ETag`/`Last-Modified` where supported. Never interpret a timeout as an empty catalog.
+- One in-process run lock plus a persisted run ID prevents overlap across browser clicks and restarts. A stale interrupted run is marked interrupted on startup; keep the previous successful feed. Publish only after validation completes.
+- Preserve source errors and diagnostics with bounded log size. Do not store sensitive prompts or external response bodies indefinitely. Provider blocks/challenges require user attention, not automated retries or evasions.
+- Nightly scheduling comes after the manual path has passed acceptance. Make local-time schedule configurable, persist last-success time, skip duplicate same-day runs after restart, and allow a manual run without disturbing scraper nightly updates. A missed night can run once after startup, never a catch-up storm.
+- The feature is optional: reader, library, Greatest, scraper, and Sentinel continue normally if metadata or LLM access is absent.
+
+## 8. Phased implementation and acceptance
+
+### Phase A — deterministic, manual MVP
+
+Implement reader-owned store, stable identity mapping, reuse of cached public metadata/Greatest seed, deterministic signals, explicit feedback, a Discover page, and admin **Run now**. No LLM and no new container. Acceptance: two users see different explainable feeds; a title already owned/queued is filtered or clearly labelled; 20-result and request caps hold; no job can change the scraper queue or library; malformed/corrupt store refuses to be overwritten; tests use mocked metadata and disposable data.
+
+### Phase B — source selection and feedback loop
+
+Connect recommendation details to the existing MangaDex candidate search, WeebCentral manual search, and inspect-first scraper action. Preserve selected source identity and show queue status. Acceptance: wrong-year/remake candidates are not auto-selected; duplicate queue clicks are prevented; inspection requires separate admin confirmation; ordinary users cannot queue; no chapter/image requests occur during discovery.
+
+### Phase C — optional agent-assisted rationale
+
+Add one configurable model adapter, schema validation, evidence-bound output, opt-in, strict time/token budget, and deterministic fallback. Compare at least 30 blind recommendations with and without model rationales for factual support, usefulness, variety, and hallucinated links before default-enabling it. Acceptance: model outage leaves deterministic feed usable; injected metadata text cannot issue commands; ungrounded award/record claims are rejected; no personal raw log leaves the reader.
+
+### Phase D — scheduling and operations
+
+Add nightly schedule only after manual runs are stable. Expose last success/error and a `discovery_run_failed` incident to admin observability, but do not give Sentinel authority to rewrite recommendations or restart the scraper. Add backup/restore instructions for `data/discovery.json` and cache behavior. If runs measurably harm reader latency, move the job into an optional one-off Compose worker with one-writer handoff, not an always-on autonomous service.
+
+### Suggested implementation map
+
+Keep the first code change small and reviewable:
+
+| File or area | Intended responsibility |
+|---|---|
+| `discovery/index.js` (new) | Identity keys, exclusions, scoring, diversity, bounded run orchestration; no scraper imports |
+| `discovery/store.js` (new) | Schema validation, atomic `data/discovery.json` publication, revision and previous-version recovery |
+| `discovery/providers/anilist.js` (new) | Reuse the existing public metadata cache and perform only bounded missing lookups |
+| `server.js` | Authenticated per-user feed/feedback and admin run/settings routes; keep route handlers thin |
+| `public/discover.html`, `discover.js`, `discover.css` (new) | Feed, evidence, feedback, empty/error states, admin run control |
+| `public/common.js` | One navigation link; no global discovery polling |
+| `test/discovery*.test.js`, disposable browser test | Identity, consent, isolation, budgets, action permissions and UI behavior |
+| `docs/architecture.md`, `docs/compose-deployment.md` | Record ownership, backup, local release and rollback once implemented |
+
+The exact file split is a proposal, not a requirement to create empty modules. Start with storage and identity tests, then the deterministic pipeline, then API/UI. Do not add model-provider plumbing, a new Compose service, or a nightly timer in the first implementation slice. A candidate must pass schema validation and duplicate/identity checks before it can appear in a published feed.
+
+## 9. Tests, measurement, and rollout
+
+Unit tests: identity collisions, sequel/remake/year mismatch, explicit negative feedback, diversity, cold start, source-budget accounting, deterministic ranking, JSON recovery, multi-user isolation, schema validation, and model fallback. Integration tests: authenticated APIs, admin-only run/settings, no scraper queue mutation, atomic publish and interrupted-run recovery, source block handling. Browser tests: desktop/mobile tiles, evidence panel, feedback undo, stale-feed/error state, edition-selection handoff. All network tests use fixtures by default; any live metadata smoke test is manual, bounded, and never fetches images.
+
+Measure: suggestion engagement (opened/Interested/Hidden), share of suggestions already known, successful source match rate, per-run API count/latency, cache hit rate, provider errors, reader response latency during a run, and reason rejection rate. Do not optimize a single click-through metric at the expense of variety or trust. Start with the local Docker installation; document reader-only release/rollback and verify the existing queue/scraper remain untouched. VM deployment waits for the separately planned migration.
+
+## 10. Decisions to make before implementation
+
+1. Which public metadata provider(s) and terms are acceptable for ongoing discovery? AniList reuse is the baseline; do not add multiple feeds just for volume.
+2. Should all users be able to trigger their own bounded run, or only admins initially? This plan chooses **admin trigger, per-user feeds** for MVP.
+3. Which optional model/provider may receive minimized taste summaries, and what monthly budget is acceptable? Without consent or a configured provider, stay deterministic.
+4. How long should hidden/dismissed feedback persist after account deletion or title-ID correction? Proposed: until the user undoes it; delete with the account.
+
+These questions do not block Phase A's storage/API/UI skeleton. Any later change to consent, external data transfer, or automatic queue authority requires explicit product review rather than an agent's assumption.