From 70a41c45311cce3f7ab1936cbd5657acf80875ba Mon Sep 17 00:00:00 2001 From: maashkinoozhe Date: Sun, 25 Oct 2026 08:36:59 +0100 Subject: [PATCH] Design: wider Discovery candidate pool from AniList (BL-0051) Co-Authored-By: Claude Opus 5.5 --- .../2026-10-25-discovery-wider-pool-design.md | 306 ++++++++++++++++++ 1 file changed, 306 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md diff --git a/docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md b/docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md new file mode 100644 index 0000000..ab104f6 --- /dev/null +++ b/docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md @@ -0,0 +1,306 @@ +# Discovery: wider candidate pool — design + +Status: approved design, not implemented. Date: 2026-10-25. Builds on the +deterministic Discovery Scout ([`docs/discovery.md`](../../discovery.md), +[`vision/discovery-scout.md`](../../../vision/discovery-scout.md)). + +## 1. Goal + +Today Discover can only suggest titles from the curated Greatest catalog plus +AniList *relations* (sequels, spin-offs) of Greatest and library titles, and only +when those are already in the local metadata cache. A run makes no network call. +Users run out of new suggestions quickly. + +This change widens the candidate pool with four seed kinds, fetched server-side +from AniList: + +1. **Community recommendations** — "readers who liked X recommend Y" for the + user's anchor titles. +2. **Same creator** — other works by the story/art authors of anchor titles. +3. **Top in your tags** — well-rated titles in the user's strongest genres/tags. +4. **Fresh** — recently started or currently releasing, trending titles. + +Success: suggestions the user did not already know, each with a traceable +reason and evidence link, without exceeding AniList's rate limits or making the +reader noticeably slower. + +### Unchanged rules from Phase A + +- Personalized discovery and reading activity stay separate opt-ins, both off + by default; feedback stays private to the account. +- Discovery never edits the scraper queue, library or collections and never + downloads anything. +- Every suggestion carries evidence; missing data stays missing. +- The reader is the only writer of discovery state; no new Compose service, no + database, no LLM. +- All fetching runs in the reader container, never in the browser. +- The repository is public: no host names, paths or personal seeds in code, + docs or fixtures. + +### Non-goals + +- No other metadata provider (AniList's API covers all four seed kinds). +- No LLM rationale or rerank (vision Phase C). +- No edition finding or queueing from a suggestion (vision Phase B). +- No worker container; it is only a fallback if section 7's latency check fails. + +## 2. Components + +| Unit | Responsibility | Depends on | +|---|---|---| +| `discovery/anilist.js` (new) | GraphQL queries for the four seed kinds and for a title by ID; one shared rate limiter; schema check and normalization of responses | `fetch`, `discovery/cache.js` | +| `discovery/cache.js` (new) | `data/discovery-cache/`: atomic writes, TTL per entry kind, size cap with oldest-first pruning | filesystem | +| `discovery/harvest.js` (new) | Anchor selection per user, request plan, budgets, single-flight run state, stop reasons | `anilist.js`, `cache.js`, `db` (read-only) | +| `discovery/engine.js` (changed) | Builds candidates from Greatest, relations **and the harvest cache**; new score parts, evidence kinds, slot mix, `isAdult` filter | `cache.js` (read-only) | +| `server.js` (changed) | Nightly scheduler, admin "Harvest now", per-user "Refresh my suggestions", admin status fields; route handlers stay thin | `harvest.js`, `engine.js`, `discovery/store.js` | +| `public/discover.*` (changed) | Refresh button states, new reason labels, fetch date in evidence, admin harvest panel | reader API | + +The existing `discovery/store.js` (atomic `discovery.json`, `.prev` recovery, +fail-closed on corruption) is reused unchanged. The reader keeps reading the +scraper's AniList cache read-only for relations; the harvester never writes into +`data/scraper/`. + +### 2.1 `discovery/anilist.js` + +Queries (all `type: MANGA`, unauthenticated, `https://graphql.anilist.co`): + +| Kind | Query | Page size | +|---|---|---| +| `title` | `Media(id)` with the fields of `scraper/metadata/anilist.js` plus `isAdult averageScore popularity trending tags { name rank isAdult } recommendations staff` | 1 | +| `recs` | `Media(id).recommendations(sort: RATING_DESC)` → `mediaRecommendation` + `rating` | 25 | +| `creator` | `Staff(id).staffMedia(type: MANGA, sort: POPULARITY_DESC)` | 25 | +| `tagTop` | `Page.media(tag_in or genre_in: [one tag], sort: SCORE_DESC, popularity_greater: 1000)` | 25 | +| `fresh` | `Page.media(status: RELEASING, startDate_greater: , sort: TRENDING_DESC)` | 50 | + +- **Rate limiter:** at most 25 requests per minute, one at a time, shared by all + harvests in the process. AniList allows more, so this stays below its limit + with headroom. +- **Normalized record:** `{ anilistId, title, altTitles, format, isAdult, + status, year, averageScore, popularity, trending, tags[{name, rank}], + authors[{staffId, name, role}], coverUrl, siteUrl, fetchedAt }`. + Each list response becomes a list of `{ anilistId, rating? }` plus one record + per entry it contains. +- **Formats:** only `MANGA` and `ONE_SHOT` are kept; novels and anime are dropped. +- **Response check:** responses are checked against a schema before they are + cached; an invalid one counts as a failed request. Titles, descriptions and + tags are treated as untrusted text: stored as data, escaped when shown. + +### 2.2 `discovery/cache.js` + +- **Location:** `data/discovery-cache//.json`. Every entry has + `{ schema: 1, kind, key, fetchedAt, data }`. +- **Lifetime:** `title` and `recs` entries live 30 days, `creator` 30 days, + `tagTop` and `fresh` 7 days. An expired entry is still used for scoring until + it is refreshed. Only the harvest checks the TTL. +- **Size cap:** 5,000 files. On write, the oldest `fetchedAt` entries are pruned + first. +- **Writes:** temporary file then rename, as in the existing stores. A corrupt + entry is ignored and deleted. + +### 2.3 `discovery/harvest.js` + +**Anchors per opted-in user.** Each anchor has a weight; the list is +deduplicated by AniList ID and capped at the 15 strongest: + +| Source | Weight | +|---|---| +| Greatest vote `flame` | 5 | +| Greatest vote `good` | 3 | +| Discover feedback `interested` | 4 | +| Discover feedback `maybe` | 2 | +| Completed chapter in a library title (only with reading activity opted in) | 3 | +| Owned library title with an AniList ID | 1 | + +**User's top tags.** The 3 tags with the highest summed anchor weight, using +the anchors' cached `title` records (tag rank ≥ 60, no adult tags unless the +user enabled mature titles). + +**Request plan, in this order:** + +1. `title` for anchors that are not cached. +2. `recs` for each anchor. +3. `creator` for the story/art staff of anchors weighted 3 or more. +4. `tagTop` for each top tag. +5. One global `fresh` list. + +Requests whose cache entry is still within its TTL are skipped. Then follow-up +`title` requests run for new candidates that are missing tags or `isAdult`, up +to the remaining budget. + +**Budgets:** nightly harvest 300 requests, per-user refresh 60 requests. When +the budget runs out, the plan stops; whatever was cached so far is still used. + +**Run state:** one harvest at a time (single-flight). + +- A queued per-user refresh waits behind a running harvest; a second refresh + for the same user while one is queued returns `409`. +- The run record is persisted in `discovery.json` `harvest`: + `{ id, kind: nightly|admin|user, user?, status, startedAt, finishedAt, + requests, budget, stopReason }`. + +## 3. Data flow + +``` +anchors ─▶ request plan ─▶ AniList (rate-limited) ─▶ discovery-cache + │ +Greatest + scraper AniList cache (relations) ─────────────┤ + ▼ + engine.generate() ─▶ discovery.json (atomic publish) +``` + +Harvest and scoring are separate steps. Scoring is local and never touches the +network, so a failed harvest means older candidates, never an empty or broken +feed. + +- **After a nightly or admin harvest:** re-score every opted-in user. +- **After a per-user refresh:** re-score only that user. The other users' feed + snapshots are carried over unchanged into the same publish. + +## 4. Scoring + +`engine.generate()` builds the candidate map from Greatest, relations, and every +record referenced by the harvest cache, limited to 2,000 candidates; the +replaced Phase A cap was 50. Exclusions stay as in Phase A: + +- titles the user already voted on in Greatest +- titles dismissed or hidden in Discover +- owned titles, matched by ID or by normalized title + year +- queued titles + +The mature filter now uses `isAdult` (title-level, or an adult tag with +rank ≥ 60). The Phase A tag regex is kept only for candidates with no `isAdult` +data. + +New score parts, all bounded: + +| Part | Value | Rule | +|---|---|---| +| `communityRec` | 0–5 | `min(5, 1 + log2(1 + rating) / 2) × anchorWeight / 5`; strongest anchor counts | +| `sameCreator` | 0–4 | `4 × anchorWeight / 5` for a shared story/art author with an anchor weighted ≥ 3 | +| `tagTop` | 0–3 | `3` for the user's top tag, `2` second, `1` third, only if `averageScore ≥ 75` | +| `fresh` | 0–2 | `2` if releasing and started within 2 years, `1` if only trending | + +Existing parts (`editorial`, `relation`, `sharedTaste`, `era`, `reading`, +`interest`) and their weights are unchanged. + +**Slot mix** for the 20 feed slots, applied greedily by descending score: + +- At most 8 items whose strongest part comes from the same seed kind. +- Up to 4 slots reserved for `fresh` candidates when such candidates exist. +- About 20% (4 slots) taken from candidates outside the user's strongest tag + cluster, as in the vision doc. +- If a rule cannot be met, the slot falls back to the next best candidate; the + feed is never padded. + +**Reasons.** A candidate found by several seeds collects every reason and +evidence item. The tile shows the reason of the strongest part; the detail +panel lists all of them. Reason templates: + +| Kind | Template | +|---|---| +| `recs` | "AniList readers who liked {anchor} recommend this ({rating})." | +| `creator` | "Also by {author} — you {read/liked} {anchor}." | +| `tagTop` | "Top-rated in {tag}, one of your most-liked tags." | +| `fresh` | "New: started {year}, ongoing and trending." | + +Evidence entries use `kind: recommendation|creator|tag-top|fresh`, link to the +AniList record, and carry `fetchedAt`. + +## 5. API and UI + +Routes: + +- `POST /api/discovery/refresh` — caller only, requires opt-in. Returns + `{ harvestId, position }`. Returns `429` with `nextAllowedAt` within 1 hour of + the caller's last refresh, and `409` if one is already queued for the caller. +- `GET /api/discovery` — adds + `refresh: { state: idle|queued|running, position?, nextAllowedAt? }` and + `fetchedAt` on evidence. +- `POST /api/admin/discovery/harvest` — admin only; starts a full harvest. + Returns `409` while one runs. +- `GET /api/admin/discovery` — adds the last harvest record, cache file count + and size, and the next nightly time. +- `POST /api/admin/discovery/settings` (new) — `{ harvestEnabled: boolean, + harvestAt: "HH:MM" }`. Defaults: `true` and `03:30`, overridable by + `DISCOVERY_HARVEST_AT`. The value is validated; nothing else is accepted. + +UI on `discover.html`: + +- **Refresh my suggestions** button with three states: idle, queued/fetching + (with queue position), and cooldown ("next refresh possible at 14:20"). +- Reason labels per section 4; evidence shows "fetched {date}". +- The settings text for **Personalized discovery** says that the server sends + AniList the IDs of titles you liked, without login, to find related work. +- Admin panel: last harvest (time, requests / budget, stop reason), cache size, + next nightly run, **Harvest now**, enable switch and time. + +## 6. Failure handling + +| Condition | Behaviour | +|---|---| +| HTTP 429, 403, or a challenge page | Stop the harvest at once with `stopReason`, shown in the admin panel. No retry until the next nightly slot or admin action; never change identity, user agent or host. | +| Timeout or 5xx | Skip that request and count it. Five in a row abort the run (`stopReason: upstream-errors`). A timeout never counts as an empty result. | +| Invalid response | Dropped, counted as a failed request, not cached. | +| Restart during a harvest | On startup the run is marked `interrupted`. If the nightly slot was missed, one harvest runs after startup; there is never a catch-up storm. A queued per-user refresh is dropped and the user can press the button again. | +| Corrupt `discovery.json` | Unchanged from Phase A: reads and runs fail closed; the file is never overwritten. | +| No network | Harvest fails with a stop reason; scoring uses whatever is cached. A fresh install behaves exactly like Phase A. | +| Account deletion | Unchanged (feedback removed). Cache entries are not user-specific and stay. | + +## 7. Performance + +- Harvesting is I/O-bound and single-flight. At 25 requests per minute, a full + nightly harvest takes about 12 minutes. +- Scoring 2,000 candidates for each user must keep the reader event loop + responsive. Each user is scored in its own `setImmediate` turn, and a test + asserts that one user's scoring stays below 200 ms on the fixture set. +- If production shows reader stalls, moving the job to a worker container is + recorded as a backlog item; it is not part of this change. + +## 8. Testing + +All network tests use recorded, anonymized AniList fixtures; no test calls the +live API. + +**Unit:** + +- query builders and response normalization +- schema rejection of malformed responses +- dropping non-manga formats +- the rate limiter, with a fake clock +- stop on 429/403/challenge, and the consecutive-error abort +- cache TTL, size pruning, and corrupt-entry handling +- anchor weights, cap and dedup; reading activity used only with opt-in +- top-tag selection +- each new score part +- slot-mix limits and fallback +- merging reasons from several seeds +- `isAdult` filter +- exclusion of owned, queued, voted and dismissed titles + +**Integration:** + +- per-user refresh: queue position, `429` cooldown, `409` duplicate +- admin harvest single-flight +- re-scoring after a per-user refresh leaves other users' feeds unchanged +- interrupted-run recovery +- one missed-night run after startup +- no harvest or score ever changes the scraper queue or library +- settings validation + +**Browser** (existing disposable Compose suite): + +- refresh button states +- reason labels for each seed kind +- evidence fetch date +- admin harvest panel +- desktop and mobile layout + +## 9. Documentation + +- `docs/discovery.md`: seed kinds, budgets, cache, nightly time, privacy note, + stop reasons. +- `docs/deployment.md`: `data/discovery-cache/` (safe to delete, rebuilt by the + next harvest; may be left out of backups) and `DISCOVERY_HARVEST_AT`. +- `docs/architecture.md`: the new routes. +- `vision/discovery-scout.md`: mark the candidate-input part of section 4 as + implemented.