diff --git a/docs/superpowers/plans/2026-10-25-discovery-wider-pool.md b/docs/superpowers/plans/2026-10-25-discovery-wider-pool.md new file mode 100644 index 0000000..29b49fe --- /dev/null +++ b/docs/superpowers/plans/2026-10-25-discovery-wider-pool.md @@ -0,0 +1,337 @@ +# Discovery wider pool Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Widen Discover's candidate pool with four AniList-sourced seed kinds (community recommendations, same creator, top in tags, fresh), harvested server-side nightly and per user, scored locally. + +**Architecture:** New reader-side modules: a file cache, a rate-limited AniList client, anchor selection, and a single-flight harvester. The existing engine scores from Greatest + relations + harvest cache; `server.js` wires routes, the nightly scheduler and rescoring. Harvest (network) and scoring (local) stay separate, so a failed harvest never empties a feed. + +**Tech Stack:** Node.js built-ins only (`node:test`, `fetch`, `fs`), plain browser JS, Playwright for the Compose UI test. + +**Spec:** [`docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md`](../specs/2026-10-25-discovery-wider-pool-design.md) + +## Global Constraints + +- No runtime dependencies; Node built-ins only (`AGENTS.md`). +- One writer: only the reader writes `data/discovery.json` and `data/discovery-cache/`; never write under `data/scraper/`. +- All fetching runs in the reader container, never in the browser. +- AniList: at most 25 requests per minute, one at a time; stop the whole harvest on 429, 403 or a challenge; never retry with a changed identity, user agent or host. +- Budgets: nightly and admin harvests 300 requests, per-user refresh 60. Per-user cooldown 1 hour. At most 15 anchors per user. +- Cache TTL: `title`, `recs`, `creator` 30 days; `tagTop`, `fresh` 7 days. Cap 5,000 files, oldest `fetchedAt` pruned first. +- Feed: 20 items; at most 8 per seed kind, at most 4 per Greatest period/`Related works`; up to 4 slots reserved for `fresh`, up to 4 for candidates outside the user's top-3 tags. Candidate cap 2,000. +- Personalized discovery and reading activity remain separate opt-ins, both off by default. +- Discovery never edits the scraper queue, library or collections. +- Tests never call live AniList; fixtures in `test/fixtures/anilist/` are handcrafted and contain no personal data. +- Public repo: no host names, IPs, local paths or personal seeds anywhere. +- Commits end with `Co-Authored-By: Claude Opus 5.5 `; push `feature/discovery-wider-pool` after each task. + +## Review Focus + +1. **A fresh install with an empty cache and no network** must behave exactly like Phase A: the feed comes from Greatest and relations, and the harvest record shows a stop reason. Test in Task 6 (`engine: empty harvest cache equals Phase A pool`) and Task 4 (`harvest: network error stops with upstream-errors after 5`). +2. **A user with no anchors** (opted in, no votes, empty library) gets no harvest requests except the global `fresh` list, and still a feed. Test in Task 4 (`plan for anchorless user only fetches fresh`). +3. **Old `discovery.json` files without `harvest`, `settings` or `nightly` fields** must load without the fail-closed error. Test in Task 5 (`store loads a Phase A file without new fields`). +4. **A per-user rescore** must keep every other user's feed and the candidates those feeds reference. Test in Task 6 (`mergeUserFeed keeps other users' referenced candidates`). +5. **AniList text containing HTML or script** (titles, tag names) is shown as text, never as markup. Test in Task 8 (UI test renders a fixture title `` as text). + +--- + +### Task 1: Discovery cache + +**Files:** +- Create: `discovery/cache.js` +- Test: `test/discovery-cache.test.js` + +**Interfaces:** +- Produces: + - `KINDS = ['title', 'recs', 'creator', 'tagTop', 'fresh']` + - `TTL_MS = { title: 30*86400000, recs: 30*86400000, creator: 30*86400000, tagTop: 7*86400000, fresh: 7*86400000 }` + - `class DiscoveryCache { constructor(dir: string, { maxFiles = 5000, now = Date.now } = {}) }` + - `get(kind, key) -> { schema: 1, kind, key, fetchedAt: ISO string, data } | null`. A corrupt or schema-mismatched file is deleted and returns `null`. + - `isFresh(kind, key) -> boolean`: true when an entry exists and `now() - fetchedAt < TTL_MS[kind]`. + - `put(kind, key, data) -> entry`: atomic write (temp file + rename), then pruning. + - `list(kind) -> entry[]`, including expired entries. + - `stats() -> { files: number, bytes: number }` +- File path: `//.json`. An unknown kind throws `TypeError`. + +- [ ] **Step 1: Write the failing tests** in `test/discovery-cache.test.js`, each using an `fs.mkdtempSync` dir and a fake `now`: + - `cache round-trips an entry`: `put('title', 42, {a:1})`, then `get('title', 42).data` deep-equals `{a:1}` and `fetchedAt` equals `new Date(now()).toISOString()`. + - `freshness follows TTL per kind`: with `now` advanced by 8 days, `isFresh('fresh','global')` is `false` and `isFresh('title',1)` is `true`. At 31 days `isFresh('title',1)` is `false`, while `get` still returns the entry. + - `prune keeps the newest maxFiles`: with `maxFiles: 3` and 5 puts at increasing `now`, `stats().files === 3` and the first two keys return `null`. + - `corrupt entry is deleted`: write `{broken` to the entry file; `get` returns `null` and the file no longer exists. + - `odd keys are file-safe`: `put('tagTop', 'genre:Slice of Life/../x', …)` lands inside `/tagTop/`. + - `unknown kind throws`. +- [ ] **Step 2: Run** `node --test test/discovery-cache.test.js`. Expected: FAIL, `Cannot find module '../discovery/cache'`. +- [ ] **Step 3: Implement `discovery/cache.js`** per the Interfaces. Prune by reading each entry's `fetchedAt` across all kinds, sorting oldest first, and deleting until `files <= maxFiles`. Follow `discovery/store.js` style. +- [ ] **Step 4: Run** `node --test test/discovery-cache.test.js`. Expected: PASS. +- [ ] **Step 5: Commit** `Discovery cache with per-kind TTL and size cap (BL-0051)` and push. + +### Task 2: AniList client + +**Files:** +- Create: `discovery/anilist.js`, `test/fixtures/anilist/{title,recs,creator,tagtop,fresh,novel}.json` +- Test: `test/discovery-anilist.test.js` + +**Interfaces:** +- Produces: + - `createLimiter({ perMinute = 25, now = Date.now, sleep = ms => new Promise(r => setTimeout(r, ms)) }) -> { wait(): Promise }`. Calls are serialized and spaced at least `60000 / perMinute` ms apart. + - `class AniListStop extends Error { reason: 'rate-limited' | 'forbidden' | 'challenge' }` + - `class AniListSkip extends Error { reason: 'timeout' | 'upstream' | 'invalid' }` + - `normalizeMedia(media) -> Record | null`. Returns `null` unless `format` is `MANGA` or `ONE_SHOT`. + - `Record = { anilistId, title, altTitles: string[], format, isAdult: boolean, status: 'ongoing'|'completed'|'unknown', year: number|null, averageScore: number|null, popularity: number|null, trending: number|null, tags: [{ name, rank, kind: 'genre'|'tag', isAdult: boolean }], authors: [{ staffId, name, role }], coverUrl: string|null, siteUrl: string|null }`. + - Genres map to `{ rank: 100, kind: 'genre', isAdult: false }`. + - Status: `RELEASING` → `ongoing`, `FINISHED` → `completed`, anything else → `unknown`. + - `authors` keeps the staff roles matched by the regex in `scraper/metadata/anilist.js` (`original story|story(?: & art)?|art|illustration`). + - `createClient({ endpoint = process.env.DISCOVERY_ANILIST_URL || 'https://graphql.anilist.co', limiter = createLimiter(), fetchImpl = fetch, timeoutMs = 30000, today = () => new Date() })` returns: + - `title(id) -> Record` + - `recs(id) -> [{ anilistId, rating, record }]`, keeping only `rating > 0` + - `creator(staffId) -> [{ anilistId, role, record }]` + - `tagTop({ name, kind }) -> [{ anilistId, record }]` + - `fresh() -> [{ anilistId, record }]` + - Entries whose `normalizeMedia` returns `null` are dropped. +- Queries share one `MEDIA_FIELDS` fragment: `id format isAdult title{romaji english native} synonyms status startDate{year} averageScore popularity trending genres tags{name rank isAdult} staff(perPage:20){edges{role node{id name{full}}}} coverImage{extraLarge} siteUrl`. Page sizes and sort orders come from spec §2.1: + - `tagTop` uses `popularity_greater: 1000`, with `genre_in` or `tag_in` by `kind`. + - `fresh` uses `status: RELEASING, startDate_greater: <(today.year − 2) * 10000 + month * 100 + day>, sort: TRENDING_DESC, perPage: 50`. + - All list queries use the full fragment, so no follow-up `title` fetches are needed. This replaces the spec's follow-up stage on purpose. +- Error mapping, applied after `await limiter.wait()`: + - HTTP 429 → `AniListStop('rate-limited')` + - HTTP 401 or 403 → `AniListStop('forbidden')` + - non-JSON body or `content-type` text/html → `AniListStop('challenge')` + - abort or timeout → `AniListSkip('timeout')` + - HTTP 5xx → `AniListSkip('upstream')` + - missing `data.Media` / `data.Page.media` / `data.Staff`, or a media without an integer `id` → `AniListSkip('invalid')` + - Requests send `content-type: application/json` and the fixed header `user-agent: MangaCircus-Discovery/1.0`. Do not reuse `SlowHttp` identity rotation. + +- [ ] **Step 1: Write fixtures** with minimal handcrafted responses: + - `recs.json` has 3 recommendations, one with `rating: -2` and one with `format: NOVEL`. + - `title.json` has `isAdult: false`, 2 genres and 2 tags, one of them `isAdult: true`. +- [ ] **Step 2: Write the failing tests** in `test/discovery-anilist.test.js`. Use a stub `fetchImpl` returning `new Response(JSON.stringify(fixture), { status, headers })` and a fake limiter `{ wait: async () => {} }`: + - `normalizeMedia maps fields and drops novels`: asserts the `Record` shape on `title.json`; `normalizeMedia({ ...media, format: 'NOVEL' }) === null`. + - `recs keeps positive ratings and manga formats`: exactly 1 entry with its rating. + - `fresh query sends startDate two years back`: with `today = new Date('2026-10-25')`, the request body's variables contain `startDate_greater: 20241025`. + - `tagTop uses genre_in for genres and tag_in for tags`. + - `429 stops`, `403 stops`, `html body is a challenge`: each throws `AniListStop` with its reason. + - `5xx and timeout skip`, `missing data is invalid`: each throws `AniListSkip` with its reason. + - `limiter spaces calls 2400 ms apart`: the fake `sleep` records each duration and advances the fake `now` by it. Three `wait()` calls issued at the same instant record `[0, 2400, 2400]`. +- [ ] **Step 3: Run** `node --test test/discovery-anilist.test.js`. Expected: FAIL (module missing). +- [ ] **Step 4: Implement `discovery/anilist.js`** per the Interfaces. +- [ ] **Step 5: Run** the same command. Expected: PASS. +- [ ] **Step 6: Commit** `Rate-limited AniList client for discovery seeds (BL-0051)` and push. + +### Task 3: Anchors and top tags + +**Files:** +- Create: `discovery/anchors.js` +- Test: `test/discovery-anchors.test.js` + +**Interfaces:** +- Consumes: `Record` from Task 2. +- Produces: + - `ANCHOR_WEIGHTS = { flame: 5, good: 3, interested: 4, maybe: 2, read: 3, owned: 1 }` + - `MAX_ANCHORS = 15` + - `anchorsFor({ user, db, library, greatest }) -> [{ anilistId: number, weight, source: keyof ANCHOR_WEIGHTS, title }]`. Returns `[]` unless `user.settings.discoveryEnabled`. Sources: + - Greatest votes: `db.greatestRatings[user.name][entryId]` is `flame` or `good`, and `entry.metadata.id` is set. + - Feedback: `db.discoveryFeedback[user.name]['anilist:N']` is `interested` or `maybe`. + - `read`: only with `settings.discoveryReadingActivity`; a library manga with `x-anilist.id` and any chapter in `db.progress[user.name][manga.id]` that has `done`. + - `owned`: a library manga with `meta['x-anilist'].id`. + - Dedupe by id, keeping the highest weight. Sort by weight desc, then `anilistId` asc. Cap at 15. + - `topTagsFor(anchors, recordOf: id => Record|null, { mature = false }) -> [{ name, kind }]`, at most 3. Sums anchor weight per tag with `rank >= 60`. Skips `isAdult` tags unless `mature`. Ties are broken by name asc. + +- [ ] **Step 1: Write the failing tests**: + - `anchors are empty without opt-in` + - `weights, dedupe and cap`: a title voted `good` that is also owned yields one anchor with weight 3. Twenty owned titles yield 15 anchors. + - `reading anchors need the reading opt-in` + - `feedback keys parse anilist ids and ignore others`: `greatest:x` and `anilist:abc` are ignored. + - `top tags weigh anchors, skip low rank and adult tags`: an adult tag is included with `{ mature: true }`. +- [ ] **Step 2: Run** `node --test test/discovery-anchors.test.js`. Expected: FAIL. +- [ ] **Step 3: Implement `discovery/anchors.js`.** +- [ ] **Step 4: Run** the same command. Expected: PASS. +- [ ] **Step 5: Commit** `Discovery anchors and top tags (BL-0051)` and push. + +### Task 4: Harvester + +**Files:** +- Create: `discovery/harvest.js` +- Test: `test/discovery-harvest.test.js` + +**Interfaces:** +- Consumes: `DiscoveryCache` (Task 1), the client and its errors (Task 2), `anchorsFor`/`topTagsFor` (Task 3). It also consumes the store methods `load()` and `patch({ harvest })` from Task 5; tests pass a stub `{ patch(){} }`, so Task 5 can land later. +- Produces: + - `BUDGETS = { nightly: 300, admin: 300, user: 60 }` + - `REFRESH_COOLDOWN_MS = 3600000` + - `MAX_CONSECUTIVE_FAILURES = 5` + - `async runHarvest({ kind, targets: [{ user, anchors, mature }], cache, client, budget }) -> { requests, budget, status: 'success'|'stopped', stopReason: null|'budget'|'rate-limited'|'forbidden'|'challenge'|'upstream-errors' }`. Stages, in order, each skipping requests whose `cache.isFresh(kind, key)`: + 1. `title` for every anchor. + 2. `recs` per anchor. + 3. `creator` per author `staffId` of anchors with `weight >= 3`, read from the cached `title` record. + 4. `tagTop` per `topTagsFor(...)` tag, keyed `${kind}:${name}`. + 5. One `fresh`, key `global`. + + Requests are deduped across targets. Each success is `cache.put`. An `AniListSkip` increments a consecutive-failure counter that resets on success; at 5 the run returns `stopped`/`upstream-errors`. An `AniListStop` returns `stopped` with its reason immediately. Reaching `budget` returns `success`/`budget`. + - `shouldRunNightly({ now: Date, harvestAt: 'HH:MM', lastNightlyDate: 'YYYY-MM-DD'|null, enabled }) -> boolean`: true when enabled, local `now` is at or after `harvestAt`, and the local date differs from `lastNightlyDate`. + - `class Harvester`: + - `constructor({ cache, client, store, targetsFor: (kind, userName?) => targets, onDone: async (job, result) => void, now = Date.now })` + - `request({ kind: 'nightly'|'admin'|'user', user?: string }) -> { harvestId, position }`. Throws `Object.assign(new Error(msg), { status, nextAllowedAt? })` with: + - `409` for an admin or nightly harvest while one is running or queued, and for a user who already has one queued or running. + - `429` within `REFRESH_COOLDOWN_MS` of that user's last accepted request; the error carries `nextAllowedAt` as an ISO string. + - `state(userName) -> { state: 'idle'|'queued'|'running', position?: number, nextAllowedAt?: string }` + - `current() -> { id, kind, user?, startedAt } | null` + - Jobs run one at a time, FIFO. Each run calls `store.patch({ harvest: { id, kind, user, status: 'running', startedAt, requests: 0, budget } })`, then the final record (`status`, `finishedAt`, `requests`, `stopReason`), then `onDone(job, result)`. Errors thrown in `onDone` are logged and do not break the queue. + +- [ ] **Step 1: Write the failing tests** using a fake client (methods resolve from fixture maps, count calls, and can throw scripted errors) and a real `DiscoveryCache` in a temp dir: + - `plan order and dedupe`: two targets sharing an anchor. The call log starts with `title` calls, then `recs`, `creator`, `tagTop`, `fresh`; the shared anchor is fetched once. + - `fresh cache entries are skipped` + - `budget stops with reason budget`: budget 3 → `requests === 3`, `stopReason: 'budget'`. + - `429 stops at once`: the call count does not grow after the throw. + - `network error stops with upstream-errors after 5`: five consecutive `AniListSkip`; a success in between resets the counter. + - `plan for anchorless user only fetches fresh` + - `shouldRunNightly`: 03:29 → false; 03:30 → true; same date → false; disabled → false. + - `harvester single-flight, positions, 409 and 429 cooldown`: with fake `now`, a second user request inside one hour throws `429` with `nextAllowedAt`. An admin request while one is running throws `409`. A queued user's `state()` returns `queued` with `position` 1. + - `onDone failure does not stall the queue` +- [ ] **Step 2: Run** `node --test test/discovery-harvest.test.js`. Expected: FAIL. +- [ ] **Step 3: Implement `discovery/harvest.js`.** +- [ ] **Step 4: Run** the same command. Expected: PASS. +- [ ] **Step 5: Commit** `Single-flight discovery harvester with budgets and stop reasons (BL-0051)` and push. + +### Task 5: Store metadata + +**Files:** +- Modify: `discovery/store.js` +- Test: `test/discovery.test.js` (add tests) + +**Interfaces:** +- Produces: + - `load()` defaults `harvest: null`, `settings: null`, `nightly: { lastDate: null }` when the fields are absent. The validation of existing fields is unchanged. + - `publish(value)` keeps the prior `harvest`, `settings` and `nightly`. + - `patch({ harvest?, settings?, nightly? }) -> data` writes atomically through the same temp+`.prev`+rename path, leaves `revision` unchanged, and refuses on a corrupt file like `publish`. + - Extract a private `write(data)` shared by `publish` and `patch`. + +- [ ] **Step 1: Write the failing tests**: + - `store loads a Phase A file without new fields`: write a Phase A-shaped JSON; `load().nightly.lastDate === null`. + - `patch keeps revision and publish keeps harvest`: `patch({ harvest: { status: 'success' } })`, then `publish(...)`; `load().harvest.status === 'success'` and the revision went up by exactly 1. + - `patch refuses a corrupt store` +- [ ] **Step 2: Run** `node --test test/discovery.test.js`. Expected: the new tests FAIL. +- [ ] **Step 3: Implement** the store changes. +- [ ] **Step 4: Run** `npm run test:reader`. Expected: PASS. +- [ ] **Step 5: Commit** `Discovery store keeps harvest, settings and nightly state (BL-0051)` and push. + +### Task 6: Engine — wider pool, new scores, slot mix + +**Files:** +- Modify: `discovery/engine.js`, `discovery/cli.js` (pass `harvestCache` when `data/discovery-cache` exists) +- Test: `test/discovery.test.js` (update the cap assertion; add tests), `test/fixtures/anilist/` reused through a temp `DiscoveryCache` + +**Interfaces:** +- Consumes: `DiscoveryCache` (Task 1), `Record` (Task 2), `anchorsFor`/`topTagsFor` (Task 3). +- Produces: + - `MAX_CANDIDATES = 2000`, `MAX_FEED = 20` + - `SLOTS = { perKind: 8, perPeriod: 4, fresh: 4, outside: 4 }` + - `buildCandidates({ greatest, library, cacheDir, harvestCache? })`: adds one candidate per distinct `anilistId` found in `recs`, `creator`, `tagTop` and `fresh` entries. New candidate fields: `isAdult`, `averageScore`, `tagRecords: Record.tags`, `authorRecords: Record.authors`, `fetchedAt`. `tags` and `authors` stay name arrays. Greatest candidates keep their fields; when a harvested record has the same id, it fills `isAdult` and `tagRecords`. Era bucketing stays, and harvested candidates bucket by their seed kind instead of `Related works`. + - `feedForUser({ user, db, candidates, greatest, library, queuedTitles, harvestCache? })` adds these parts, with values from spec §4, rounded to 2 decimals: + - `communityRec = min(5, 1 + log2(1 + rating) / 2) * anchorWeight / 5`, using the best (anchor, rating) pair among the user's anchors whose `recs` list contains the candidate. + - `sameCreator = 4 * anchorWeight / 5`, when the candidate's `authorRecords` shares a `staffId` with an anchor of `weight >= 3`. + - `tagTop = 3 | 2 | 1` for the user's 1st, 2nd or 3rd top tag, only if `averageScore >= 75` and the candidate is in that tag's `tagTop` list. + - `fresh = 2` if `status === 'ongoing'` and `year >= today.year - 2`; `1` if only present in the `fresh` list. + - Mature filter: exclude unless `settings.discoveryMature` when `isAdult === true` or any `tagRecords` entry has `isAdult && rank >= 60`. Use the Phase A regex only when `isAdult` is `undefined`. + - Each row gets `group`. If the row's largest part is one of the four new parts, `group` is its kind (`recs`, `creator`, `tagTop`, `fresh`); otherwise it is `period || 'Related works'`. + - `reason` comes from the largest part. `reasons: string[]` lists every applicable reason in descending part order. Exact templates: + - recs: `AniList readers who liked ${anchor} recommend this (${rating}).` + - creator: `Also by ${author} — you ${read|liked} ${anchor}.` Use `read` when the anchor source is `read`, otherwise `liked`. + - tagTop: `Top-rated in ${tag}, one of your most-liked tags.` + - fresh: `New: started ${year}, ongoing and trending.` + - Evidence ids: `rec::`, `creator::`, `tagtop:::`, `fresh:`. Kinds: `recommendation|creator|tag-top|fresh`. Each carries `url: record.siteUrl` and `fetchedAt`. + - `mixSlots(rows, { topTags }) -> rows` (at most 20, ranked): + 1. Sort by score desc, then `candidateId`. + 2. Take up to 4 rows with `scoreParts.fresh > 0`. + 3. Take up to 4 rows whose `tags` intersect none of the `topTags` names. + 4. Fill greedily. + - Steps 2–4 all respect the caps: `perKind` 8 for the four kinds and `perPeriod` 4 for other groups. + - Never pad. + - The final order is by score desc, then `rank = index + 1`. + - `async generateAsync({ db, library, greatest, cacheDir, harvestCache, queuedTitles, onlyUser? }) -> same shape as generate()`. Awaits `new Promise(setImmediate)` between users; with `onlyUser`, scores only that user. `generate()` stays synchronous for `cli.js` and the existing tests. + - `mergeUserFeed(prior: { candidates, feeds }, result, userName) -> { candidates, feeds }`: replaces `feeds[userName]`, keeps every other feed, and keeps exactly the candidates referenced by any kept feed plus `result.candidates`. + +- [ ] **Step 1: Update and write the failing tests**: + - Change `assert.equal(Object.keys(result.candidates).length, 50)` to `Math.min(2000, examined)`; keep the other Phase A assertions. + - `engine: empty harvest cache equals Phase A pool`: with and without an empty `harvestCache`, the candidate ids are equal. + - `community rec, creator, tagTop and fresh parts and reasons`: fixture cache plus a user with a `flame` anchor. Asserts exact part values, for example: rating 42, weight 5 → `min(5, 1 + log2(43)/2) = 3.71`. Also asserts the exact reason strings and evidence kinds with `fetchedAt`. + - `isAdult filter`: an `isAdult` candidate is absent without the mature opt-in and present with it. + - `slot mix`: 30 `recs`-group rows → at most 8 from that kind. Fresh rows are present even when low-scored. Rows outside the top tags are present. Fewer than 20 eligible rows → no padding. + - `multiple seeds merge reasons`: one candidate found by recs and creator has both reasons and both evidence entries. + - `owned, queued, voted, dismissed still excluded` for harvested candidates. + - `mergeUserFeed keeps other users' referenced candidates` + - `scoring one user stays under 200 ms on 2000 candidates`: synthetic records; measure `feedForUser` with `performance.now()`. +- [ ] **Step 2: Run** `node --test test/discovery.test.js`. Expected: the new tests FAIL. +- [ ] **Step 3: Implement** the engine changes and the `cli.js` pass-through. +- [ ] **Step 4: Run** `npm run test:reader`. Expected: PASS. +- [ ] **Step 5: Commit** `Discovery scores AniList seeds with slot mix and isAdult filter (BL-0051)` and push. + +### Task 7: Server wiring + +**Files:** +- Modify: `server.js` (around the discovery block at lines 240–267, the routes at 490–517, `HttpError` at 81, the error response at 974, and the timers near 980–1012), `docker-compose.yml` (reader `environment`: `DISCOVERY_HARVEST_AT: ${DISCOVERY_HARVEST_AT:-03:30}`, `DISCOVERY_ANILIST_URL: ${DISCOVERY_ANILIST_URL:-}`), `docs/architecture.md` +- Test: `test/discovery-server.test.js` (spawns `server.js` on a free port with a temp `DATA`/`LIBRARY` and `DISCOVERY_ANILIST_URL` pointing to an in-test `http.createServer` serving fixtures). Copy the helper pattern from an existing server-spawning test in `test/` if one exists; otherwise write a minimal spawn/wait-for-port helper inside this file. + +**Interfaces:** +- Consumes: Tasks 1–6. +- Produces: + - `HttpError(code, msg, extra = {})`; the error response becomes `{ error, ...extra }`. + - `DISCOVERY_HARVEST_CACHE = path.join(DATA, 'discovery-cache')` + - `harvestAt` default: `process.env.DISCOVERY_HARVEST_AT` if it matches `/^([01]\d|2[0-3]):[0-5]\d$/`, else `'03:30'`. `store.settings` overrides it. + - `async rescore({ onlyUser })`: refactors the body of `startDiscoveryRun`. With `onlyUser` it calls `generateAsync({ onlyUser })` then `mergeUserFeed` before `publish`; otherwise it runs a full `generateAsync`. `startDiscoveryRun` keeps its route behavior. + - `targetsFor(kind, userName)`: all opted-in users, or only `userName` for `kind === 'user'`. Each target is `{ user, anchors: anchorsFor(...), mature: !!settings.discoveryMature }`. + - Startup: if `store.load().harvest?.status === 'running'`, `patch` it to `interrupted` with `finishedAt`. Queued jobs are not persisted. + - Scheduler: `setInterval(checkNightly, 60000).unref()`, plus one call 30 s after startup. When `shouldRunNightly` is true and no harvest is active, call `patch({ nightly: { lastDate } })` and then `harvester.request({ kind: 'nightly' })`. + - Routes: + - `POST /api/discovery/refresh`: caller only; `403` without opt-in. Returns `{ harvestId, position }`, `409`, or `429` with `{ nextAllowedAt }`. + - `GET /api/discovery`: adds `refresh: harvester.state(u.name)`. + - `POST /api/admin/discovery/harvest`: admin only; returns `{ harvestId, position }` or `409`. + - `GET /api/admin/discovery`: adds `harvest`, `currentHarvest`, `cache: harvestCache.stats()`, `settings: { harvestEnabled, harvestAt }`, `nextNightlyAt`. + - `POST /api/admin/discovery/settings`: admin only. Accepts exactly `{ harvestEnabled: boolean, harvestAt: 'HH:MM' }`; anything else → `400`. Saved via `patch({ settings })`. + +- [ ] **Step 1: Write the failing integration tests**: + - `refresh requires opt-in, queues, runs and rescores only the caller`: two opted-in users. Alice refreshes, and her feed afterwards contains a fixture rec candidate. Bob's `generatedAt` is unchanged. + - `second refresh within an hour is 429 with nextAllowedAt` + - `admin harvest is admin-only and single-flight`: `403` for a non-admin; `409` on a second call while one is running (the fixture server delays its responses by 500 ms). + - `settings validation`: `harvestAt: '25:00'` → `400`; an extra key → `400`; a valid body round-trips through `GET /api/admin/discovery`. + - `interrupted harvest is marked on startup`: pre-write a `discovery.json` with `harvest.status: 'running'`, start the server, and get `interrupted`. + - `harvest never touches scraper queue or library`: no files change under the temp `LIBRARY` or `DATA/scraper`. Snapshot the directory listings and mtimes before and after. +- [ ] **Step 2: Run** `node --test test/discovery-server.test.js`. Expected: FAIL. +- [ ] **Step 3: Implement** the server changes, the compose env lines, and the new routes in `docs/architecture.md`. +- [ ] **Step 4: Run** `npm test`. Expected: all suites PASS. +- [ ] **Step 5: Commit** `Discovery harvest routes, nightly schedule and per-user refresh (BL-0051)` and push. + +### Task 8: Discover UI, browser test and docs + +**Files:** +- Modify: `public/discover.js`, `public/discover.css` if present (else the stylesheet `discover.html` links), `tools/test-discovery-ui.js`, `tools/test-compose.js` (pass `DISCOVERY_ANILIST_URL=http://127.0.0.1:9/` for `--discovery-ui`, so a harvest fails fast without reaching AniList), `docs/discovery.md`, `docs/deployment.md`, `vision/discovery-scout.md` + +**Interfaces:** +- Consumes: Task 7 routes and response fields. +- UI, using the existing `h()`/`button()` helpers. Never use `innerHTML`; text goes in through `h()` children. + - **Refresh my suggestions** button, shown when `state.enabled`. While `refresh.state` is `queued` it reads `Queued (position N)`, while `running` it reads `Fetching…`, and both disable it. During a cooldown it shows `Next refresh possible at ` (from `nextAllowedAt`) and is disabled. It polls every 1.5 s while queued or running, like `run()`. + - Card: show `item.reason`; the details list shows all `item.reasons` and evidence, plus ` · fetched ` when `entry.fetchedAt` is set. + - Settings hint for **Personalized discovery**, exact text: `Off by default. When on, the server sends AniList the IDs of titles you liked, without login, to find related work. Your feed and feedback stay private to this server.` + - The mature hint becomes `Only affects titles AniList marks as adult or with a known mature tag.` + - Admin panel: + - adds a line `Last harvest: · / requests · · cache files`, plus the next nightly time + - adds a **Harvest now** button + - adds an **Automatic harvest** checkbox and a time `` that save through the settings route + - replaces the old "This first version…" paragraph with: `Harvests ask AniList for public metadata only. They never ask image hosters, call an LLM, or edit the scraper queue.` +- Docs: + - `docs/discovery.md`: replace the deliberate-limits section with the seed kinds, budgets, cache, nightly time, privacy note and stop reasons. + - `docs/deployment.md`: `data/discovery-cache/` is safe to delete and can be left out of backups; document `DISCOVERY_HARVEST_AT` and `DISCOVERY_ANILIST_URL` (test only). + - `vision/discovery-scout.md` §4: mark the candidate input as implemented, with a link to the spec. + +- [ ] **Step 1: Extend `tools/test-discovery-ui.js`**, after the existing Maybe click: + - Click **Refresh my suggestions**; wait until the button text matches `/Next refresh possible at \d{1,2}:\d{2}/`. + - The admin panel shows `Last harvest: stopped` and `upstream-errors`. + - Turn on **Automatic harvest**, set the time to `04:15`, reload, and see `04:15`. + - **Harvest now** while one is running shows the 409 message in the status note. + - Review Focus 5: before the stack starts, seed the disposable data tree with a `discovery.json` whose candidate title is `` and which is in the QA admin's feed. Assert that the card's `h2` `textContent` equals that string and that `h2 img` count is 0. + - The existing non-admin checks additionally assert `Harvest now` count 0 and `POST /api/admin/discovery/harvest` → 403. + - Mobile viewport (375×812): the refresh button is visible without horizontal scroll (`document.documentElement.scrollWidth <= 375`). +- [ ] **Step 2: Run** `docker build -t mangacircus:discovery-test .` then `APP_IMAGE=mangacircus:discovery-test npm run test:discovery-ui`. Expected: FAIL on the missing button. +- [ ] **Step 3: Implement** the UI and docs. +- [ ] **Step 4: Run** the same two commands, then `npm test`. Expected: PASS. +- [ ] **Step 5: Commit** `Discover page: refresh, harvest admin and seed reasons; docs (BL-0051)` and push. +- [ ] **Step 6: Update MangaOrg**: run `python D:\Projects\Manga\MangaOrg\tools\org.py` to set BL-0051 to `review`, release the `MangaCircus discovery` claim, then commit and push MangaOrg. Merging into `main` waits for the user.