Discover page: refresh, harvest admin and seed reasons; docs (BL-0051)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs/discovery.md

@@ -1,23 +1,28 @@
-# Discovery Scout (deterministic first release)
+# Discovery Scout
 
-The reader now has a separate **Discover** page. This first slice deliberately uses
-the curated Greatest catalog, already-cached AniList identities/relations, library
-metadata, each user's Greatest votes, and explicit Discover feedback. It does not
-call an LLM, fetch new metadata, inspect chapter/image hosters, edit a collection,
+The reader has a separate **Discover** page. Suggestions come from the curated
+Greatest catalog, cached AniList relations of Greatest and library titles, and a
+wider pool that the reader harvests from AniList's public API for the users who
+opted in (design: [the wider-pool spec](superpowers/specs/2026-10-25-discovery-wider-pool-design.md)).
+Discovery does not call an LLM, inspect chapter/image hosters, edit a collection,
 or enqueue downloads. The reader remains the only writer of account and discovery
-state; no fourth Compose service, credential, or database is required.
+state; no further Compose service, credential, or database is required.
 
 ## Use
 
 1. Open `discover.html`. Each account independently enables **Personalized
    discovery** (off by default). Reading activity is a separate opt-in. The
-   mature-title switch only applies to candidates with a known mature tag; it is
-   not a content-safety guarantee.
-2. An admin presses **Run discovery now** on the same page. Existing suggestions
-   remain visible until the new feed is published. Runs are bounded to 50
-   candidates and 20 suggestions per opted-in user, with no more than four from
-   one period. A second concurrent run is rejected.
-3. A reader can open **Why this is here · evidence** and record Interested, Maybe,
+   mature-title switch only affects titles AniList marks as adult or with a known
+   mature tag; it is not a content-safety guarantee.
+2. **Refresh my suggestions** queues a harvest for your own account (at most once an
+   hour). The button shows the queue position, `Fetching…` while it runs, and when
+   the next refresh is possible. Your existing suggestions stay visible until the
+   new feed is published.
+3. An admin can press **Run discovery now** (re-score from local data, no network)
+   or **Harvest now** (fetch from AniList for every opted-in user, then re-score).
+   A second concurrent run or harvest is rejected. A nightly harvest runs on its
+   own; the admin panel can switch it off or change its time.
+4. A reader can open **Why this is here · evidence** and record Interested, Maybe,
    No thanks, or Hide. Feedback is private to the account. Dismissed/hidden
    titles disappear at once; other feedback influences the next run. The user
    can undo a choice by clicking its active button again.
@@ -28,17 +33,70 @@ the existing source inspection and confirmation process to download anything.
 The title/year/format matching is intentionally conservative but not a verified
 identity merge. Missing year, cover, or status remains missing.
 
+## Seeds from AniList
+
+A harvest starts from each opted-in user's **anchors**: titles they voted
+`flame`/`good` in Greatest, marked Interested/Maybe in Discover, read (only with
+reading activity opted in), or own with an AniList ID. The 15 strongest anchors
+count. It then asks AniList for four seed kinds:
+
+| Seed | What | Reason shown on the card |
+|---|---|---|
+| Community recommendations | "readers who liked X recommend Y" for each anchor | AniList readers who liked {anchor} recommend this ({rating}). |
+| Same creator | other works by the story/art authors of strong anchors | Also by {author} — you read/liked {anchor}. |
+| Top in your tags | well-rated titles in the user's three strongest tags | Top-rated in {tag}, one of your most-liked tags. |
+| Fresh | recently started, releasing and trending titles | New: started {year}, ongoing and trending. |
+
+A title found by several seeds keeps every reason and evidence item; the card
+shows the strongest, the details list all of them, each with the date it was
+fetched. A feed holds at most 20 suggestions, at most 8 from one seed kind, up to
+4 fresh titles, and some from outside your strongest tags.
+
+- **Budgets:** a nightly or admin harvest makes at most 300 requests, a
+  per-user refresh 60. Requests are sequential and limited to 25 a minute. When
+  the budget is spent the harvest stops; what was fetched is still used.
+- **Cache:** `data/discovery-cache/`, one file per response. Titles,
+  recommendations and creators stay fresh for 30 days, tag lists and the fresh
+  list for 7 days; at most 5,000 files, oldest pruned first. Scoring reads only
+  this cache and never the network, so a failed harvest means older candidates,
+  never an empty feed.
+- **Nightly time:** `03:30` by default, or `DISCOVERY_HARVEST_AT`, in the
+  container's local time zone (`TZ`). The admin panel overrides both and can turn
+  the nightly harvest off. A night missed while the reader was down runs once
+  after startup.
+- **Privacy:** for opted-in users only, the server sends AniList the IDs of
+  their anchor titles (and the names of their top tags), without login, cookies or
+  user names. Feeds and feedback stay private to this server; cache entries are
+  not tied to a user.
+
+### Stop reasons
+
+The admin panel shows the last harvest as `status · requests/budget · stop
+reason · cache files`.
+
+| Stop reason | Meaning | What happens next |
+|---|---|---|
+| `budget` | The request budget ran out | Normal; the next harvest continues where the cache is stale |
+| `rate-limited` | AniList answered 429 | Harvesting pauses; refresh buttons show "Paused by AniList" until the next nightly or admin harvest |
+| `forbidden` | AniList answered 401/403 | Same pause |
+| `challenge` | AniList answered with an HTML or challenge page | Same pause |
+| `upstream-errors` | Five timeouts or server errors in a row | The harvest ends; nothing is paused |
+| `error` | A local fault (for example a cache write) | See the reader log |
+
+Discovery never retries with a different identity, user agent or host. A
+re-scoring failure after a harvest is shown as the admin panel's error line.
+
 ## Data and API
 
 - `data/discovery.json`: atomic, reader-owned candidate and per-user feed
-  snapshots. A previous valid version is retained as `discovery.json.prev`.
-  A corrupt file makes reads/runs fail closed rather than overwrite it.
+  snapshots plus the last harvest record and the admin's harvest settings. A
+  previous valid version is retained as `discovery.json.prev`. A corrupt file
+  makes reads/runs/harvests fail closed rather than overwrite it.
+- `data/discovery-cache/`: harvested AniList responses; safe to delete.
 - `data/db.json`: consent in user settings and explicit feedback under
   `discoveryFeedback`. The account-deletion route removes that feedback.
-- `GET /api/discovery`: caller's own feed, settings, evidence, and feedback only.
-- `POST /api/discovery/feedback`: caller's candidate choice; requires opt-in.
-- `GET /api/admin/discovery` and `POST /api/admin/discovery/run`: admin-only
-  status and manual generation. A run keeps the last good feed on failure.
+- Routes: [architecture.md](architecture.md) (`/api/discovery*`,
+  `/api/admin/discovery*`) and §6.6 there for the harvester.
 
 The job asks the private scraper state API for queued titles to avoid recommending
 them. If that API is unavailable, the run still uses only cached sources; the
@@ -58,15 +116,13 @@ APP_IMAGE=mangacircus:discovery-test npm run test:discovery-ui
 
 The dry run reads `db.json`, a saved library index if present, Greatest, and
 cached AniList metadata. It writes nothing and does not contact providers. The
-Compose/browser suite uses a disposable data tree and queue, not the live one.
+Compose/browser suite uses a disposable data tree and queue, not the live one,
+and points `DISCOVERY_ANILIST_URL` at a closed local port so no harvest reaches
+AniList.
 
-## Deliberate limits and next stage
+## Limits
 
-This slice has no scheduled run, external metadata discovery, LLM, admin seed
-form, edition-finding action, or explicit purge control for old feed snapshots.
-Disabling consent stops feed display and future personalized generation, but
-existing snapshot/feedback is retained until account deletion or a later data
-management feature. Candidate quality is limited by the curated list and local
-metadata cache. Add external public metadata feeds and optional LLM only behind
-the existing bounded candidate/evidence contract after this deterministic path
-has been used and reviewed.
+There is no admin seed form, edition-finding action, LLM rationale, or explicit
+purge control for old feed snapshots. Disabling consent stops feed display,
+harvesting and future personalized generation for that account, but existing
+snapshot/feedback is retained until account deletion.