Maashkinoozhe/MangaCircus
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Design: wider Discovery candidate pool from AniList (BL-0051)

Browse Source 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
MaashkinoozheandClaude Opus 5.5 committed 2026-10-25 08:36:59 +01:00
1 parent 57a571d2f0
commit 70a41c4531
1 file changed
+306
Download Patch File Download Diff File Expand all files Collapse all files
docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md
+306
View File
@@ -0,0 +1,306 @@
# Discovery: wider candidate pool — design
Status: approved design, not implemented. Date: 2026-10-25. Builds on the
deterministic Discovery Scout ([`docs/discovery.md`](../../discovery.md),
[`vision/discovery-scout.md`](../../../vision/discovery-scout.md)).
## 1. Goal
Today Discover can only suggest titles from the curated Greatest catalog plus
AniList *relations* (sequels, spin-offs) of Greatest and library titles, and only
when those are already in the local metadata cache. A run makes no network call.
Users run out of new suggestions quickly.
This change widens the candidate pool with four seed kinds, fetched server-side
from AniList:
1. **Community recommendations** — "readers who liked X recommend Y" for the
user's anchor titles.
2. **Same creator** — other works by the story/art authors of anchor titles.
3. **Top in your tags** — well-rated titles in the user's strongest genres/tags.
4. **Fresh** — recently started or currently releasing, trending titles.
Success: suggestions the user did not already know, each with a traceable
reason and evidence link, without exceeding AniList's rate limits or making the
reader noticeably slower.
### Unchanged rules from Phase A
- Personalized discovery and reading activity stay separate opt-ins, both off
by default; feedback stays private to the account.
- Discovery never edits the scraper queue, library or collections and never
downloads anything.
- Every suggestion carries evidence; missing data stays missing.
- The reader is the only writer of discovery state; no new Compose service, no
database, no LLM.
- All fetching runs in the reader container, never in the browser.
- The repository is public: no host names, paths or personal seeds in code,
docs or fixtures.
### Non-goals
- No other metadata provider (AniList's API covers all four seed kinds).
- No LLM rationale or rerank (vision Phase C).
- No edition finding or queueing from a suggestion (vision Phase B).
- No worker container; it is only a fallback if section 7's latency check fails.
## 2. Components
| Unit | Responsibility | Depends on |
|---|---|---|
| `discovery/anilist.js` (new) | GraphQL queries for the four seed kinds and for a title by ID; one shared rate limiter; schema check and normalization of responses | `fetch`, `discovery/cache.js` |
| `discovery/cache.js` (new) | `data/discovery-cache/`: atomic writes, TTL per entry kind, size cap with oldest-first pruning | filesystem |
| `discovery/harvest.js` (new) | Anchor selection per user, request plan, budgets, single-flight run state, stop reasons | `anilist.js`, `cache.js`, `db` (read-only) |
| `discovery/engine.js` (changed) | Builds candidates from Greatest, relations **and the harvest cache**; new score parts, evidence kinds, slot mix, `isAdult` filter | `cache.js` (read-only) |
| `server.js` (changed) | Nightly scheduler, admin "Harvest now", per-user "Refresh my suggestions", admin status fields; route handlers stay thin | `harvest.js`, `engine.js`, `discovery/store.js` |
| `public/discover.*` (changed) | Refresh button states, new reason labels, fetch date in evidence, admin harvest panel | reader API |
The existing `discovery/store.js` (atomic `discovery.json`, `.prev` recovery,
fail-closed on corruption) is reused unchanged. The reader keeps reading the
scraper's AniList cache read-only for relations; the harvester never writes into
`data/scraper/`.
### 2.1 `discovery/anilist.js`
Queries (all `type: MANGA`, unauthenticated, `https://graphql.anilist.co`):
| Kind | Query | Page size |
|---|---|---|
| `title` | `Media(id)` with the fields of `scraper/metadata/anilist.js` plus `isAdult averageScore popularity trending tags { name rank isAdult } recommendations staff` | 1 |
| `recs` | `Media(id).recommendations(sort: RATING_DESC)` → `mediaRecommendation` + `rating` | 25 |
| `creator` | `Staff(id).staffMedia(type: MANGA, sort: POPULARITY_DESC)` | 25 |
| `tagTop` | `Page.media(tag_in or genre_in: [one tag], sort: SCORE_DESC, popularity_greater: 1000)` | 25 |
| `fresh` | `Page.media(status: RELEASING, startDate_greater: <today − 2 years>, sort: TRENDING_DESC)` | 50 |
- **Rate limiter:** at most 25 requests per minute, one at a time, shared by all
harvests in the process. AniList allows more, so this stays below its limit
with headroom.
- **Normalized record:** `{ anilistId, title, altTitles, format, isAdult,
status, year, averageScore, popularity, trending, tags[{name, rank}],
authors[{staffId, name, role}], coverUrl, siteUrl, fetchedAt }`.
Each list response becomes a list of `{ anilistId, rating? }` plus one record
per entry it contains.
- **Formats:** only `MANGA` and `ONE_SHOT` are kept; novels and anime are dropped.
- **Response check:** responses are checked against a schema before they are
cached; an invalid one counts as a failed request. Titles, descriptions and
tags are treated as untrusted text: stored as data, escaped when shown.
### 2.2 `discovery/cache.js`
- **Location:** `data/discovery-cache/<kind>/<key>.json`. Every entry has
`{ schema: 1, kind, key, fetchedAt, data }`.
- **Lifetime:** `title` and `recs` entries live 30 days, `creator` 30 days,
`tagTop` and `fresh` 7 days. An expired entry is still used for scoring until
it is refreshed. Only the harvest checks the TTL.
- **Size cap:** 5,000 files. On write, the oldest `fetchedAt` entries are pruned
first.
- **Writes:** temporary file then rename, as in the existing stores. A corrupt
entry is ignored and deleted.
### 2.3 `discovery/harvest.js`
**Anchors per opted-in user.** Each anchor has a weight; the list is
deduplicated by AniList ID and capped at the 15 strongest:
| Source | Weight |
|---|---|
| Greatest vote `flame` | 5 |
| Greatest vote `good` | 3 |
| Discover feedback `interested` | 4 |
| Discover feedback `maybe` | 2 |
| Completed chapter in a library title (only with reading activity opted in) | 3 |
| Owned library title with an AniList ID | 1 |
**User's top tags.** The 3 tags with the highest summed anchor weight, using
the anchors' cached `title` records (tag rank ≥ 60, no adult tags unless the
user enabled mature titles).
**Request plan, in this order:**
1. `title` for anchors that are not cached.
2. `recs` for each anchor.
3. `creator` for the story/art staff of anchors weighted 3 or more.
4. `tagTop` for each top tag.
5. One global `fresh` list.
Requests whose cache entry is still within its TTL are skipped. Then follow-up
`title` requests run for new candidates that are missing tags or `isAdult`, up
to the remaining budget.
**Budgets:** nightly harvest 300 requests, per-user refresh 60 requests. When
the budget runs out, the plan stops; whatever was cached so far is still used.
**Run state:** one harvest at a time (single-flight).
- A queued per-user refresh waits behind a running harvest; a second refresh
for the same user while one is queued returns `409`.
- The run record is persisted in `discovery.json` `harvest`:
`{ id, kind: nightly|admin|user, user?, status, startedAt, finishedAt,
requests, budget, stopReason }`.
## 3. Data flow
```
anchors ─▶ request plan ─▶ AniList (rate-limited) ─▶ discovery-cache
│
Greatest + scraper AniList cache (relations) ─────────────┤
▼
engine.generate() ─▶ discovery.json (atomic publish)
```
Harvest and scoring are separate steps. Scoring is local and never touches the
network, so a failed harvest means older candidates, never an empty or broken
feed.
- **After a nightly or admin harvest:** re-score every opted-in user.
- **After a per-user refresh:** re-score only that user. The other users' feed
snapshots are carried over unchanged into the same publish.
## 4. Scoring
`engine.generate()` builds the candidate map from Greatest, relations, and every
record referenced by the harvest cache, limited to 2,000 candidates; the
replaced Phase A cap was 50. Exclusions stay as in Phase A:
- titles the user already voted on in Greatest
- titles dismissed or hidden in Discover
- owned titles, matched by ID or by normalized title + year
- queued titles
The mature filter now uses `isAdult` (title-level, or an adult tag with
rank ≥ 60). The Phase A tag regex is kept only for candidates with no `isAdult`
data.
New score parts, all bounded:
| Part | Value | Rule |
|---|---|---|
| `communityRec` | 0–5 | `min(5, 1 + log2(1 + rating) / 2) × anchorWeight / 5`; strongest anchor counts |
| `sameCreator` | 0–4 | `4 × anchorWeight / 5` for a shared story/art author with an anchor weighted ≥ 3 |
| `tagTop` | 0–3 | `3` for the user's top tag, `2` second, `1` third, only if `averageScore ≥ 75` |
| `fresh` | 0–2 | `2` if releasing and started within 2 years, `1` if only trending |
Existing parts (`editorial`, `relation`, `sharedTaste`, `era`, `reading`,
`interest`) and their weights are unchanged.
**Slot mix** for the 20 feed slots, applied greedily by descending score:
- At most 8 items whose strongest part comes from the same seed kind.
- Up to 4 slots reserved for `fresh` candidates when such candidates exist.
- About 20% (4 slots) taken from candidates outside the user's strongest tag
cluster, as in the vision doc.
- If a rule cannot be met, the slot falls back to the next best candidate; the
feed is never padded.
**Reasons.** A candidate found by several seeds collects every reason and
evidence item. The tile shows the reason of the strongest part; the detail
panel lists all of them. Reason templates:
| Kind | Template |
|---|---|
| `recs` | "AniList readers who liked {anchor} recommend this ({rating})." |
| `creator` | "Also by {author} — you {read/liked} {anchor}." |
| `tagTop` | "Top-rated in {tag}, one of your most-liked tags." |
| `fresh` | "New: started {year}, ongoing and trending." |
Evidence entries use `kind: recommendation|creator|tag-top|fresh`, link to the
AniList record, and carry `fetchedAt`.
## 5. API and UI
Routes:
- `POST /api/discovery/refresh` — caller only, requires opt-in. Returns
`{ harvestId, position }`. Returns `429` with `nextAllowedAt` within 1 hour of
the caller's last refresh, and `409` if one is already queued for the caller.
- `GET /api/discovery` — adds
`refresh: { state: idle|queued|running, position?, nextAllowedAt? }` and
`fetchedAt` on evidence.
- `POST /api/admin/discovery/harvest` — admin only; starts a full harvest.
Returns `409` while one runs.
- `GET /api/admin/discovery` — adds the last harvest record, cache file count
and size, and the next nightly time.
- `POST /api/admin/discovery/settings` (new) — `{ harvestEnabled: boolean,
harvestAt: "HH:MM" }`. Defaults: `true` and `03:30`, overridable by
`DISCOVERY_HARVEST_AT`. The value is validated; nothing else is accepted.
UI on `discover.html`:
- **Refresh my suggestions** button with three states: idle, queued/fetching
(with queue position), and cooldown ("next refresh possible at 14:20").
- Reason labels per section 4; evidence shows "fetched {date}".
- The settings text for **Personalized discovery** says that the server sends
AniList the IDs of titles you liked, without login, to find related work.
- Admin panel: last harvest (time, requests / budget, stop reason), cache size,
next nightly run, **Harvest now**, enable switch and time.
## 6. Failure handling
| Condition | Behaviour |
|---|---|
| HTTP 429, 403, or a challenge page | Stop the harvest at once with `stopReason`, shown in the admin panel. No retry until the next nightly slot or admin action; never change identity, user agent or host. |
| Timeout or 5xx | Skip that request and count it. Five in a row abort the run (`stopReason: upstream-errors`). A timeout never counts as an empty result. |
| Invalid response | Dropped, counted as a failed request, not cached. |
| Restart during a harvest | On startup the run is marked `interrupted`. If the nightly slot was missed, one harvest runs after startup; there is never a catch-up storm. A queued per-user refresh is dropped and the user can press the button again. |
| Corrupt `discovery.json` | Unchanged from Phase A: reads and runs fail closed; the file is never overwritten. |
| No network | Harvest fails with a stop reason; scoring uses whatever is cached. A fresh install behaves exactly like Phase A. |
| Account deletion | Unchanged (feedback removed). Cache entries are not user-specific and stay. |
## 7. Performance
- Harvesting is I/O-bound and single-flight. At 25 requests per minute, a full
nightly harvest takes about 12 minutes.
- Scoring 2,000 candidates for each user must keep the reader event loop
responsive. Each user is scored in its own `setImmediate` turn, and a test
asserts that one user's scoring stays below 200 ms on the fixture set.
- If production shows reader stalls, moving the job to a worker container is
recorded as a backlog item; it is not part of this change.
## 8. Testing
All network tests use recorded, anonymized AniList fixtures; no test calls the
live API.
**Unit:**
- query builders and response normalization
- schema rejection of malformed responses
- dropping non-manga formats
- the rate limiter, with a fake clock
- stop on 429/403/challenge, and the consecutive-error abort
- cache TTL, size pruning, and corrupt-entry handling
- anchor weights, cap and dedup; reading activity used only with opt-in
- top-tag selection
- each new score part
- slot-mix limits and fallback
- merging reasons from several seeds
- `isAdult` filter
- exclusion of owned, queued, voted and dismissed titles
**Integration:**
- per-user refresh: queue position, `429` cooldown, `409` duplicate
- admin harvest single-flight
- re-scoring after a per-user refresh leaves other users' feeds unchanged
- interrupted-run recovery
- one missed-night run after startup
- no harvest or score ever changes the scraper queue or library
- settings validation
**Browser** (existing disposable Compose suite):
- refresh button states
- reason labels for each seed kind
- evidence fetch date
- admin harvest panel
- desktop and mobile layout
## 9. Documentation
- `docs/discovery.md`: seed kinds, budgets, cache, nightly time, privacy note,
stop reasons.
- `docs/deployment.md`: `data/discovery-cache/` (safe to delete, rebuilt by the
next harvest; may be left out of backups) and `DISCOVERY_HARVEST_AT`.
- `docs/architecture.md`: the new routes.
- `vision/discovery-scout.md`: mark the candidate-input part of section 4 as
implemented.