MangaCircus/docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md

Discovery: wider candidate pool — design

Status: implemented on branch feature/discovery-wider-pool. Date: 2026-10-25. Builds on the deterministic Discovery Scout (docs/discovery.md, vision/discovery-scout.md).

1. Goal

Today Discover can only suggest titles from the curated Greatest catalog plus AniList relations (sequels, spin-offs) of Greatest and library titles, and only when those are already in the local metadata cache. A run makes no network call. Users run out of new suggestions quickly.

This change widens the candidate pool with four seed kinds, fetched server-side from AniList:

1. Community recommendations — "readers who liked X recommend Y" for the user's anchor titles.
2. Same creator — other works by the story/art authors of anchor titles.
3. Top in your tags — well-rated titles in the user's strongest genres/tags.
4. Fresh — recently started or currently releasing, trending titles.

Success: suggestions the user did not already know, each with a traceable reason and evidence link, without exceeding AniList's rate limits or making the reader noticeably slower.

Unchanged rules from Phase A

• Personalized discovery and reading activity stay separate opt-ins, both off by default; feedback stays private to the account.
• Discovery never edits the scraper queue, library or collections and never downloads anything.
• Every suggestion carries evidence; missing data stays missing.
• The reader is the only writer of discovery state; no new Compose service, no database, no LLM.
• All fetching runs in the reader container, never in the browser.
• The repository is public: no host names, paths or personal seeds in code, docs or fixtures.

Non-goals

• No other metadata provider (AniList's API covers all four seed kinds).
• No LLM rationale or rerank (vision Phase C).
• No edition finding or queueing from a suggestion (vision Phase B).
• No worker container; it is only a fallback if section 7's latency check fails.

2. Components

Unit	Responsibility	Depends on
discovery/anilist.js (new)	GraphQL queries for the four seed kinds and for a title by ID; one shared rate limiter; schema check and normalization of responses	fetch, discovery/cache.js
discovery/cache.js (new)	data/discovery-cache/: atomic writes, TTL per entry kind, size cap with oldest-first pruning	filesystem
discovery/harvest.js (new)	Anchor selection per user, request plan, budgets, single-flight run state, stop reasons	anilist.js, cache.js, db (read-only)
discovery/engine.js (changed)	Builds candidates from Greatest, relations and the harvest cache; new score parts, evidence kinds, slot mix, isAdult filter	cache.js (read-only)
server.js (changed)	Nightly scheduler, admin "Harvest now", per-user "Refresh my suggestions", admin status fields; route handlers stay thin	harvest.js, engine.js, discovery/store.js
public/discover.* (changed)	Refresh button states, new reason labels, fetch date in evidence, admin harvest panel	reader API

The existing discovery/store.js (atomic discovery.json, .prev recovery, fail-closed on corruption) is reused unchanged. The reader keeps reading the scraper's AniList cache read-only for relations; the harvester never writes into data/scraper/.

2.1 discovery/anilist.js

Queries (all type: MANGA, unauthenticated, https://graphql.anilist.co):

Kind	Query	Page size
title	Media(id) with the fields of scraper/metadata/anilist.js plus isAdult averageScore popularity trending tags { name rank isAdult } recommendations staff	1
recs	Media(id).recommendations(sort: RATING_DESC) → mediaRecommendation + rating	25
creator	Staff(id).staffMedia(type: MANGA, sort: POPULARITY_DESC)	25
tagTop	Page.media(tag_in or genre_in: [one tag], sort: SCORE_DESC, popularity_greater: 1000)	25
fresh	Page.media(status: RELEASING, startDate_greater: <today − 2 years>, sort: TRENDING_DESC)	50

• Rate limiter: at most 25 requests per minute, one at a time, shared by all harvests in the process. AniList allows more, so this stays below its limit with headroom.
• Normalized record: { anilistId, title, altTitles, format, isAdult, status, year, averageScore, popularity, trending, tags[{name, rank}], authors[{staffId, name, role}], coverUrl, siteUrl, fetchedAt }. Each list response becomes a list of { anilistId, rating? } plus one record per entry it contains.
• Formats: only MANGA and ONE_SHOT are kept; novels and anime are dropped.
• Response check: responses are checked against a schema before they are cached; an invalid one counts as a failed request. Titles, descriptions and tags are treated as untrusted text: stored as data, escaped when shown.

2.2 discovery/cache.js

• Location: data/discovery-cache/<kind>/<key>.json. Every entry has { schema: 1, kind, key, fetchedAt, data }.
• Lifetime: title and recs entries live 30 days, creator 30 days, tagTop and fresh 7 days. An expired entry is still used for scoring until it is refreshed. Only the harvest checks the TTL.
• Size cap: 5,000 files. On write, the oldest fetchedAt entries are pruned first.
• Writes: temporary file then rename, as in the existing stores. A corrupt entry is ignored and deleted.

2.3 discovery/harvest.js

Anchors per opted-in user. Each anchor has a weight; the list is deduplicated by AniList ID and capped at the 15 strongest:

Source	Weight
Greatest vote flame	5
Greatest vote good	3
Discover feedback interested	4
Discover feedback maybe	2
Completed chapter in a library title (only with reading activity opted in)	3
Owned library title with an AniList ID	1

User's top tags. The 3 tags with the highest summed anchor weight, using the anchors' cached title records (tag rank ≥ 60, no adult tags unless the user enabled mature titles).

Request plan, in this order:

1. title for anchors that are not cached.
2. recs for each anchor.
3. creator for the story/art staff of anchors weighted 3 or more.
4. tagTop for each top tag.
5. One global fresh list.

Requests whose cache entry is still within its TTL are skipped. Then follow-up title requests run for new candidates that are missing tags or isAdult, up to the remaining budget.

Budgets: nightly harvest 300 requests, per-user refresh 60 requests. When the budget runs out, the plan stops; whatever was cached so far is still used.

Run state: one harvest at a time (single-flight).

• A queued per-user refresh waits behind a running harvest; a second refresh for the same user while one is queued returns 409.
• The run record is persisted in discovery.json harvest: { id, kind: nightly|admin|user, user?, status, startedAt, finishedAt, requests, budget, stopReason }.

3. Data flow

anchors ─▶ request plan ─▶ AniList (rate-limited) ─▶ discovery-cache
                                                          │
Greatest + scraper AniList cache (relations) ─────────────┤
                                                          ▼
                              engine.generate() ─▶ discovery.json (atomic publish)

Harvest and scoring are separate steps. Scoring is local and never touches the network, so a failed harvest means older candidates, never an empty or broken feed.

• After a nightly or admin harvest: re-score every opted-in user.
• After a per-user refresh: re-score only that user. The other users' feed snapshots are carried over unchanged into the same publish.

4. Scoring

engine.generate() builds the candidate map from Greatest, relations, and every record referenced by the harvest cache, limited to 2,000 candidates; the replaced Phase A cap was 50. Exclusions stay as in Phase A:

• titles the user already voted on in Greatest
• titles dismissed or hidden in Discover
• owned titles, matched by ID or by normalized title + year
• queued titles

The mature filter now uses isAdult (title-level, or an adult tag with rank ≥ 60). The Phase A tag regex is kept only for candidates with no isAdult data.

New score parts, all bounded:

Part	Value	Rule
communityRec	0–5	min(5, 1 + log2(1 + rating) / 2) × anchorWeight / 5; strongest anchor counts
sameCreator	0–4	4 × anchorWeight / 5 for a shared story/art author with an anchor weighted ≥ 3
tagTop	0–3	3 for the user's top tag, 2 second, 1 third, only if averageScore ≥ 75
fresh	0–2	2 if releasing and started within 2 years, 1 if only trending

Existing parts (editorial, relation, sharedTaste, era, reading, interest) and their weights are unchanged.

Slot mix for the 20 feed slots, applied greedily by descending score:

• At most 8 items whose strongest part comes from the same seed kind.
• Up to 4 slots reserved for fresh candidates when such candidates exist.
• About 20% (4 slots) taken from candidates outside the user's strongest tag cluster, as in the vision doc.
• If a rule cannot be met, the slot falls back to the next best candidate; the feed is never padded.

Reasons. A candidate found by several seeds collects every reason and evidence item. The tile shows the reason of the strongest part; the detail panel lists all of them. Reason templates:

Kind	Template
recs	"AniList readers who liked {anchor} recommend this ({rating})."
creator	"Also by {author} — you {read/liked} {anchor}."
tagTop	"Top-rated in {tag}, one of your most-liked tags."
fresh	"New: started {year}, ongoing and trending."

Evidence entries use kind: recommendation|creator|tag-top|fresh, link to the AniList record, and carry fetchedAt.

5. API and UI

Routes:

• POST /api/discovery/refresh — caller only, requires opt-in. Returns { harvestId, position }. Returns 429 with nextAllowedAt within 1 hour of the caller's last refresh, and 409 if one is already queued for the caller.
• GET /api/discovery — adds refresh: { state: idle|queued|running, position?, nextAllowedAt? } and fetchedAt on evidence.
• POST /api/admin/discovery/harvest — admin only; starts a full harvest. Returns 409 while one runs.
• GET /api/admin/discovery — adds the last harvest record, cache file count and size, and the next nightly time.
• POST /api/admin/discovery/settings (new) — { harvestEnabled: boolean, harvestAt: "HH:MM" }. Defaults: true and 03:30, overridable by DISCOVERY_HARVEST_AT. The value is validated; nothing else is accepted.

UI on discover.html:

• Refresh my suggestions button with three states: idle, queued/fetching (with queue position), and cooldown ("next refresh possible at 14:20").
• Reason labels per section 4; evidence shows "fetched {date}".
• The settings text for Personalized discovery says that the server sends AniList the IDs of titles you liked, without login, to find related work.
• Admin panel: last harvest (time, requests / budget, stop reason), cache size, next nightly run, Harvest now, enable switch and time.

6. Failure handling

Condition	Behaviour
HTTP 429, 403, or a challenge page	Stop the harvest at once with stopReason, shown in the admin panel. No retry until the next nightly slot or admin action; never change identity, user agent or host.
Timeout or 5xx	Skip that request and count it. Five in a row abort the run (stopReason: upstream-errors). A timeout never counts as an empty result.
Invalid response	Dropped, counted as a failed request, not cached.
Restart during a harvest	On startup the run is marked interrupted. If the nightly slot was missed, one harvest runs after startup; there is never a catch-up storm. A queued per-user refresh is dropped and the user can press the button again.
Corrupt discovery.json	Unchanged from Phase A: reads and runs fail closed; the file is never overwritten.
No network	Harvest fails with a stop reason; scoring uses whatever is cached. A fresh install behaves exactly like Phase A.
Account deletion	Unchanged (feedback removed). Cache entries are not user-specific and stay.

7. Performance

• Harvesting is I/O-bound and single-flight. At 25 requests per minute, a full nightly harvest takes about 12 minutes.
• Scoring 2,000 candidates for each user must keep the reader event loop responsive. Each user is scored in its own setImmediate turn, and a test asserts that one user's scoring stays below 200 ms on the fixture set.
• If production shows reader stalls, moving the job to a worker container is recorded as a backlog item; it is not part of this change.

8. Testing

All network tests use recorded, anonymized AniList fixtures; no test calls the live API.

Unit:

• query builders and response normalization
• schema rejection of malformed responses
• dropping non-manga formats
• the rate limiter, with a fake clock
• stop on 429/403/challenge, and the consecutive-error abort
• cache TTL, size pruning, and corrupt-entry handling
• anchor weights, cap and dedup; reading activity used only with opt-in
• top-tag selection
• each new score part
• slot-mix limits and fallback
• merging reasons from several seeds
• isAdult filter
• exclusion of owned, queued, voted and dismissed titles

Integration:

• per-user refresh: queue position, 429 cooldown, 409 duplicate
• admin harvest single-flight
• re-scoring after a per-user refresh leaves other users' feeds unchanged
• interrupted-run recovery
• one missed-night run after startup
• no harvest or score ever changes the scraper queue or library
• settings validation

Browser (existing disposable Compose suite):

• refresh button states
• reason labels for each seed kind
• evidence fetch date
• admin harvest panel
• desktop and mobile layout

9. Documentation

• docs/discovery.md: seed kinds, budgets, cache, nightly time, privacy note, stop reasons.
• docs/deployment.md: data/discovery-cache/ (safe to delete, rebuilt by the next harvest; may be left out of backups) and DISCOVERY_HARVEST_AT.
• docs/architecture.md: the new routes.
• vision/discovery-scout.md: mark the candidate-input part of section 4 as implemented.