MangaCircus/docs/superpowers/plans/2026-10-25-discovery-wider-pool.md

Discovery wider pool Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Widen Discover's candidate pool with four AniList-sourced seed kinds (community recommendations, same creator, top in tags, fresh), harvested server-side nightly and per user, scored locally.

Architecture: New reader-side modules: a file cache, a rate-limited AniList client, anchor selection, and a single-flight harvester. The existing engine scores from Greatest + relations + harvest cache; server.js wires routes, the nightly scheduler and rescoring. Harvest (network) and scoring (local) stay separate, so a failed harvest never empties a feed.

Tech Stack: Node.js built-ins only (node:test, fetch, fs), plain browser JS, Playwright for the Compose UI test.

Spec: docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md

Global Constraints

• No runtime dependencies; Node built-ins only (AGENTS.md).
• One writer: only the reader writes data/discovery.json and data/discovery-cache/; never write under data/scraper/.
• All fetching runs in the reader container, never in the browser.
• AniList: at most 25 requests per minute, one at a time; stop the whole harvest on 429, 403 or a challenge; never retry with a changed identity, user agent or host.
• Budgets: nightly and admin harvests 300 requests, per-user refresh 60. Per-user cooldown 1 hour. At most 15 anchors per user.
• Cache TTL: title, recs, creator 30 days; tagTop, fresh 7 days. Cap 5,000 files, oldest fetchedAt pruned first.
• Feed: 20 items; at most 8 per seed kind, at most 4 per Greatest period/Related works; up to 4 slots reserved for fresh, up to 4 for candidates outside the user's top-3 tags. Candidate cap 2,000.
• Personalized discovery and reading activity remain separate opt-ins, both off by default.
• Discovery never edits the scraper queue, library or collections.
• Tests never call live AniList; fixtures in test/fixtures/anilist/ are handcrafted and contain no personal data.
• Public repo: no host names, IPs, local paths or personal seeds anywhere.
• Commits end with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>; push feature/discovery-wider-pool after each task.

Review Focus

1. A fresh install with an empty cache and no network must behave exactly like Phase A: the feed comes from Greatest and relations, and the harvest record shows a stop reason. Test in Task 6 (engine: empty harvest cache equals Phase A pool) and Task 4 (harvest: network error stops with upstream-errors after 5).
2. A user with no anchors (opted in, no votes, empty library) gets no harvest requests except the global fresh list, and still a feed. Test in Task 4 (plan for anchorless user only fetches fresh).
3. Old discovery.json files without harvest, settings or nightly fields must load without the fail-closed error. Test in Task 5 (store loads a Phase A file without new fields).
4. A per-user rescore must keep every other user's feed and the candidates those feeds reference. Test in Task 6 (mergeUserFeed keeps other users' referenced candidates).
5. AniList text containing HTML or script (titles, tag names) is shown as text, never as markup. Test in Task 8 (UI test renders a fixture title <img src=x onerror=alert(1)> as text).

---

Task 1: Discovery cache

Files:

• Create: discovery/cache.js
• Test: test/discovery-cache.test.js

Interfaces:

• Produces:

  • KINDS = ['title', 'recs', 'creator', 'tagTop', 'fresh']
  • TTL_MS = { title: 30*86400000, recs: 30*86400000, creator: 30*86400000, tagTop: 7*86400000, fresh: 7*86400000 }
  • class DiscoveryCache { constructor(dir: string, { maxFiles = 5000, now = Date.now } = {}) }
  • get(kind, key) -> { schema: 1, kind, key, fetchedAt: ISO string, data } | null. A corrupt or schema-mismatched file is deleted and returns null.
  • isFresh(kind, key) -> boolean: true when an entry exists and now() - fetchedAt < TTL_MS[kind].
  • put(kind, key, data) -> entry: atomic write (temp file + rename), then pruning.
  • list(kind) -> entry[], including expired entries.
  • stats() -> { files: number, bytes: number }

• File path: <dir>/<kind>/<base64url(String(key))>.json. An unknown kind throws TypeError.

• Step 1: Write the failing tests in test/discovery-cache.test.js, each using an fs.mkdtempSync dir and a fake now:

  • cache round-trips an entry: put('title', 42, {a:1}), then get('title', 42).data deep-equals {a:1} and fetchedAt equals new Date(now()).toISOString().
  • freshness follows TTL per kind: with now advanced by 8 days, isFresh('fresh','global') is false and isFresh('title',1) is true. At 31 days isFresh('title',1) is false, while get still returns the entry.
  • prune keeps the newest maxFiles: with maxFiles: 3 and 5 puts at increasing now, stats().files === 3 and the first two keys return null.
  • corrupt entry is deleted: write {broken to the entry file; get returns null and the file no longer exists.
  • odd keys are file-safe: put('tagTop', 'genre:Slice of Life/../x', …) lands inside <dir>/tagTop/.
  • unknown kind throws.

• Step 2: Run node --test test/discovery-cache.test.js. Expected: FAIL, Cannot find module '../discovery/cache'.

• Step 3: Implement discovery/cache.js per the Interfaces. Prune by reading each entry's fetchedAt across all kinds, sorting oldest first, and deleting until files <= maxFiles. Follow discovery/store.js style.

• Step 4: Run node --test test/discovery-cache.test.js. Expected: PASS.

• Step 5: Commit Discovery cache with per-kind TTL and size cap (BL-0051) and push.

Task 2: AniList client

Files:

• Create: discovery/anilist.js, test/fixtures/anilist/{title,recs,creator,tagtop,fresh,novel}.json
• Test: test/discovery-anilist.test.js

Interfaces:

• Produces:

  • createLimiter({ perMinute = 25, now = Date.now, sleep = ms => new Promise(r => setTimeout(r, ms)) }) -> { wait(): Promise<void> }. Calls are serialized and spaced at least 60000 / perMinute ms apart.
  • class AniListStop extends Error { reason: 'rate-limited' | 'forbidden' | 'challenge' }
  • class AniListSkip extends Error { reason: 'timeout' | 'upstream' | 'invalid' }
  • normalizeMedia(media) -> Record | null. Returns null unless format is MANGA or ONE_SHOT.
  • Record = { anilistId, title, altTitles: string[], format, isAdult: boolean, status: 'ongoing'|'completed'|'unknown', year: number|null, averageScore: number|null, popularity: number|null, trending: number|null, tags: [{ name, rank, kind: 'genre'|'tag', isAdult: boolean }], authors: [{ staffId, name, role }], coverUrl: string|null, siteUrl: string|null }.
    • Genres map to { rank: 100, kind: 'genre', isAdult: false }.
    • Status: RELEASING → ongoing, FINISHED → completed, anything else → unknown.
    • authors keeps the staff roles matched by the regex in scraper/metadata/anilist.js (original story|story(?: & art)?|art|illustration).
  • createClient({ endpoint = process.env.DISCOVERY_ANILIST_URL || 'https://graphql.anilist.co', limiter = createLimiter(), fetchImpl = fetch, timeoutMs = 30000, today = () => new Date() }) returns:
    • title(id) -> Record
    • recs(id) -> [{ anilistId, rating, record }], keeping only rating > 0
    • creator(staffId) -> [{ anilistId, role, record }]
    • tagTop({ name, kind }) -> [{ anilistId, record }]
    • fresh() -> [{ anilistId, record }]
    • Entries whose normalizeMedia returns null are dropped.

• Queries share one MEDIA_FIELDS fragment: id format isAdult title{romaji english native} synonyms status startDate{year} averageScore popularity trending genres tags{name rank isAdult} staff(perPage:20){edges{role node{id name{full}}}} coverImage{extraLarge} siteUrl. Page sizes and sort orders come from spec §2.1:

  • tagTop uses popularity_greater: 1000, with genre_in or tag_in by kind.
  • fresh uses status: RELEASING, startDate_greater: <(today.year − 2) * 10000 + month * 100 + day>, sort: TRENDING_DESC, perPage: 50.
  • All list queries use the full fragment, so no follow-up title fetches are needed. This replaces the spec's follow-up stage on purpose.

• Error mapping, applied after await limiter.wait():

  • HTTP 429 → AniListStop('rate-limited')
  • HTTP 401 or 403 → AniListStop('forbidden')
  • non-JSON body or content-type text/html → AniListStop('challenge')
  • abort or timeout → AniListSkip('timeout')
  • HTTP 5xx → AniListSkip('upstream')
  • missing data.Media / data.Page.media / data.Staff, or a media without an integer id → AniListSkip('invalid')
  • Requests send content-type: application/json and the fixed header user-agent: MangaCircus-Discovery/1.0. Do not reuse SlowHttp identity rotation.

• Step 1: Write fixtures with minimal handcrafted responses:

  • recs.json has 3 recommendations, one with rating: -2 and one with format: NOVEL.
  • title.json has isAdult: false, 2 genres and 2 tags, one of them isAdult: true.

• Step 2: Write the failing tests in test/discovery-anilist.test.js. Use a stub fetchImpl returning new Response(JSON.stringify(fixture), { status, headers }) and a fake limiter { wait: async () => {} }:

  • normalizeMedia maps fields and drops novels: asserts the Record shape on title.json; normalizeMedia({ ...media, format: 'NOVEL' }) === null.
  • recs keeps positive ratings and manga formats: exactly 1 entry with its rating.
  • fresh query sends startDate two years back: with today = new Date('2026-10-25'), the request body's variables contain startDate_greater: 20241025.
  • tagTop uses genre_in for genres and tag_in for tags.
  • 429 stops, 403 stops, html body is a challenge: each throws AniListStop with its reason.
  • 5xx and timeout skip, missing data is invalid: each throws AniListSkip with its reason.
  • limiter spaces calls 2400 ms apart: the fake sleep records each duration and advances the fake now by it. Three wait() calls issued at the same instant record [0, 2400, 2400].

• Step 3: Run node --test test/discovery-anilist.test.js. Expected: FAIL (module missing).

• Step 4: Implement discovery/anilist.js per the Interfaces.

• Step 5: Run the same command. Expected: PASS.

• Step 6: Commit Rate-limited AniList client for discovery seeds (BL-0051) and push.

Task 3: Anchors and top tags

Files:

• Create: discovery/anchors.js
• Test: test/discovery-anchors.test.js

Interfaces:

• Consumes: Record from Task 2.

• Produces:

  • ANCHOR_WEIGHTS = { flame: 5, good: 3, interested: 4, maybe: 2, read: 3, owned: 1 }
  • MAX_ANCHORS = 15
  • anchorsFor({ user, db, library, greatest }) -> [{ anilistId: number, weight, source: keyof ANCHOR_WEIGHTS, title }]. Returns [] unless user.settings.discoveryEnabled. Sources:
    • Greatest votes: db.greatestRatings[user.name][entryId] is flame or good, and entry.metadata.id is set.
    • Feedback: db.discoveryFeedback[user.name]['anilist:N'] is interested or maybe.
    • read: only with settings.discoveryReadingActivity; a library manga with x-anilist.id and any chapter in db.progress[user.name][manga.id] that has done.
    • owned: a library manga with meta['x-anilist'].id.
    • Dedupe by id, keeping the highest weight. Sort by weight desc, then anilistId asc. Cap at 15.
  • topTagsFor(anchors, recordOf: id => Record|null, { mature = false }) -> [{ name, kind }], at most 3. Sums anchor weight per tag with rank >= 60. Skips isAdult tags unless mature. Ties are broken by name asc.

• Step 1: Write the failing tests:

  • anchors are empty without opt-in
  • weights, dedupe and cap: a title voted good that is also owned yields one anchor with weight 3. Twenty owned titles yield 15 anchors.
  • reading anchors need the reading opt-in
  • feedback keys parse anilist ids and ignore others: greatest:x and anilist:abc are ignored.
  • top tags weigh anchors, skip low rank and adult tags: an adult tag is included with { mature: true }.

• Step 2: Run node --test test/discovery-anchors.test.js. Expected: FAIL.

• Step 3: Implement discovery/anchors.js.

• Step 4: Run the same command. Expected: PASS.

• Step 5: Commit Discovery anchors and top tags (BL-0051) and push.

Task 4: Harvester

Files:

• Create: discovery/harvest.js
• Test: test/discovery-harvest.test.js

Interfaces:

• Consumes: DiscoveryCache (Task 1), the client and its errors (Task 2), anchorsFor/topTagsFor (Task 3). It also consumes the store methods load() and patch({ harvest }) from Task 5; tests pass a stub { patch(){} }, so Task 5 can land later.

• Produces:

  • BUDGETS = { nightly: 300, admin: 300, user: 60 }

  • REFRESH_COOLDOWN_MS = 3600000

  • MAX_CONSECUTIVE_FAILURES = 5

  • async runHarvest({ kind, targets: [{ user, anchors, mature }], cache, client, budget }) -> { requests, budget, status: 'success'|'stopped', stopReason: null|'budget'|'rate-limited'|'forbidden'|'challenge'|'upstream-errors' }. Stages, in order, each skipping requests whose cache.isFresh(kind, key):

    1. title for every anchor.
    2. recs per anchor.
    3. creator per author staffId of anchors with weight >= 3, read from the cached title record.
    4. tagTop per topTagsFor(...) tag, keyed ${kind}:${name}.
    5. One fresh, key global.

    Requests are deduped across targets. Each success is cache.put. An AniListSkip increments a consecutive-failure counter that resets on success; at 5 the run returns stopped/upstream-errors. An AniListStop returns stopped with its reason immediately. Reaching budget returns success/budget.

  • shouldRunNightly({ now: Date, harvestAt: 'HH:MM', lastNightlyDate: 'YYYY-MM-DD'|null, enabled }) -> boolean: true when enabled, local now is at or after harvestAt, and the local date differs from lastNightlyDate.

  • class Harvester:

    • constructor({ cache, client, store, targetsFor: (kind, userName?) => targets, onDone: async (job, result) => void, now = Date.now })
    • request({ kind: 'nightly'|'admin'|'user', user?: string }) -> { harvestId, position }. Throws Object.assign(new Error(msg), { status, nextAllowedAt? }) with:
      • 409 for an admin or nightly harvest while one is running or queued, and for a user who already has one queued or running.
      • 429 within REFRESH_COOLDOWN_MS of that user's last accepted request; the error carries nextAllowedAt as an ISO string.
    • state(userName) -> { state: 'idle'|'queued'|'running', position?: number, nextAllowedAt?: string }
    • current() -> { id, kind, user?, startedAt } | null
    • Jobs run one at a time, FIFO. Each run calls store.patch({ harvest: { id, kind, user, status: 'running', startedAt, requests: 0, budget } }), then the final record (status, finishedAt, requests, stopReason), then onDone(job, result). Errors thrown in onDone are logged and do not break the queue.

• Step 1: Write the failing tests using a fake client (methods resolve from fixture maps, count calls, and can throw scripted errors) and a real DiscoveryCache in a temp dir:

  • plan order and dedupe: two targets sharing an anchor. The call log starts with title calls, then recs, creator, tagTop, fresh; the shared anchor is fetched once.
  • fresh cache entries are skipped
  • budget stops with reason budget: budget 3 → requests === 3, stopReason: 'budget'.
  • 429 stops at once: the call count does not grow after the throw.
  • network error stops with upstream-errors after 5: five consecutive AniListSkip; a success in between resets the counter.
  • plan for anchorless user only fetches fresh
  • shouldRunNightly: 03:29 → false; 03:30 → true; same date → false; disabled → false.
  • harvester single-flight, positions, 409 and 429 cooldown: with fake now, a second user request inside one hour throws 429 with nextAllowedAt. An admin request while one is running throws 409. A queued user's state() returns queued with position 1.
  • onDone failure does not stall the queue

• Step 2: Run node --test test/discovery-harvest.test.js. Expected: FAIL.

• Step 3: Implement discovery/harvest.js.

• Step 4: Run the same command. Expected: PASS.

• Step 5: Commit Single-flight discovery harvester with budgets and stop reasons (BL-0051) and push.

Task 5: Store metadata

Files:

• Modify: discovery/store.js
• Test: test/discovery.test.js (add tests)

Interfaces:

• Produces:

  • load() defaults harvest: null, settings: null, nightly: { lastDate: null } when the fields are absent. The validation of existing fields is unchanged.
  • publish(value) keeps the prior harvest, settings and nightly.
  • patch({ harvest?, settings?, nightly? }) -> data writes atomically through the same temp+.prev+rename path, leaves revision unchanged, and refuses on a corrupt file like publish.
  • Extract a private write(data) shared by publish and patch.

• Step 1: Write the failing tests:

  • store loads a Phase A file without new fields: write a Phase A-shaped JSON; load().nightly.lastDate === null.
  • patch keeps revision and publish keeps harvest: patch({ harvest: { status: 'success' } }), then publish(...); load().harvest.status === 'success' and the revision went up by exactly 1.
  • patch refuses a corrupt store

• Step 2: Run node --test test/discovery.test.js. Expected: the new tests FAIL.

• Step 3: Implement the store changes.

• Step 4: Run npm run test:reader. Expected: PASS.

• Step 5: Commit Discovery store keeps harvest, settings and nightly state (BL-0051) and push.

Task 6: Engine — wider pool, new scores, slot mix

Files:

• Modify: discovery/engine.js, discovery/cli.js (pass harvestCache when data/discovery-cache exists)
• Test: test/discovery.test.js (update the cap assertion; add tests), test/fixtures/anilist/ reused through a temp DiscoveryCache

Interfaces:

• Consumes: DiscoveryCache (Task 1), Record (Task 2), anchorsFor/topTagsFor (Task 3).

• Produces:

  • MAX_CANDIDATES = 2000, MAX_FEED = 20
  • SLOTS = { perKind: 8, perPeriod: 4, fresh: 4, outside: 4 }
  • buildCandidates({ greatest, library, cacheDir, harvestCache? }): adds one candidate per distinct anilistId found in recs, creator, tagTop and fresh entries. New candidate fields: isAdult, averageScore, tagRecords: Record.tags, authorRecords: Record.authors, fetchedAt. tags and authors stay name arrays. Greatest candidates keep their fields; when a harvested record has the same id, it fills isAdult and tagRecords. Era bucketing stays, and harvested candidates bucket by their seed kind instead of Related works.
  • feedForUser({ user, db, candidates, greatest, library, queuedTitles, harvestCache? }) adds these parts, with values from spec §4, rounded to 2 decimals:
    • communityRec = min(5, 1 + log2(1 + rating) / 2) * anchorWeight / 5, using the best (anchor, rating) pair among the user's anchors whose recs list contains the candidate.
    • sameCreator = 4 * anchorWeight / 5, when the candidate's authorRecords shares a staffId with an anchor of weight >= 3.
    • tagTop = 3 | 2 | 1 for the user's 1st, 2nd or 3rd top tag, only if averageScore >= 75 and the candidate is in that tag's tagTop list.
    • fresh = 2 if status === 'ongoing' and year >= today.year - 2; 1 if only present in the fresh list.
  • Mature filter: exclude unless settings.discoveryMature when isAdult === true or any tagRecords entry has isAdult && rank >= 60. Use the Phase A regex only when isAdult is undefined.
  • Each row gets group. If the row's largest part is one of the four new parts, group is its kind (recs, creator, tagTop, fresh); otherwise it is period || 'Related works'.
  • reason comes from the largest part. reasons: string[] lists every applicable reason in descending part order. Exact templates:
    • recs: AniList readers who liked ${anchor} recommend this (${rating}).
    • creator: Also by ${author} — you ${read|liked} ${anchor}. Use read when the anchor source is read, otherwise liked.
    • tagTop: Top-rated in ${tag}, one of your most-liked tags.
    • fresh: New: started ${year}, ongoing and trending.
  • Evidence ids: rec:<anchorId>:<candidateId>, creator:<staffId>:<candidateId>, tagtop:<kind>:<name>:<candidateId>, fresh:<candidateId>. Kinds: recommendation|creator|tag-top|fresh. Each carries url: record.siteUrl and fetchedAt.
  • mixSlots(rows, { topTags }) -> rows (at most 20, ranked):
    1. Sort by score desc, then candidateId.
    2. Take up to 4 rows with scoreParts.fresh > 0.
    3. Take up to 4 rows whose tags intersect none of the topTags names.
    4. Fill greedily.
    • Steps 2–4 all respect the caps: perKind 8 for the four kinds and perPeriod 4 for other groups.
    • Never pad.
    • The final order is by score desc, then rank = index + 1.
  • async generateAsync({ db, library, greatest, cacheDir, harvestCache, queuedTitles, onlyUser? }) -> same shape as generate(). Awaits new Promise(setImmediate) between users; with onlyUser, scores only that user. generate() stays synchronous for cli.js and the existing tests.
  • mergeUserFeed(prior: { candidates, feeds }, result, userName) -> { candidates, feeds }: replaces feeds[userName], keeps every other feed, and keeps exactly the candidates referenced by any kept feed plus result.candidates.

• Step 1: Update and write the failing tests:

  • Change assert.equal(Object.keys(result.candidates).length, 50) to Math.min(2000, examined); keep the other Phase A assertions.
  • engine: empty harvest cache equals Phase A pool: with and without an empty harvestCache, the candidate ids are equal.
  • community rec, creator, tagTop and fresh parts and reasons: fixture cache plus a user with a flame anchor. Asserts exact part values, for example: rating 42, weight 5 → min(5, 1 + log2(43)/2) = 3.71. Also asserts the exact reason strings and evidence kinds with fetchedAt.
  • isAdult filter: an isAdult candidate is absent without the mature opt-in and present with it.
  • slot mix: 30 recs-group rows → at most 8 from that kind. Fresh rows are present even when low-scored. Rows outside the top tags are present. Fewer than 20 eligible rows → no padding.
  • multiple seeds merge reasons: one candidate found by recs and creator has both reasons and both evidence entries.
  • owned, queued, voted, dismissed still excluded for harvested candidates.
  • mergeUserFeed keeps other users' referenced candidates
  • scoring one user stays under 200 ms on 2000 candidates: synthetic records; measure feedForUser with performance.now().

• Step 2: Run node --test test/discovery.test.js. Expected: the new tests FAIL.

• Step 3: Implement the engine changes and the cli.js pass-through.

• Step 4: Run npm run test:reader. Expected: PASS.

• Step 5: Commit Discovery scores AniList seeds with slot mix and isAdult filter (BL-0051) and push.

Task 7: Server wiring

Files:

• Modify: server.js (around the discovery block at lines 240–267, the routes at 490–517, HttpError at 81, the error response at 974, and the timers near 980–1012), docker-compose.yml (reader environment: DISCOVERY_HARVEST_AT: ${DISCOVERY_HARVEST_AT:-03:30}, DISCOVERY_ANILIST_URL: ${DISCOVERY_ANILIST_URL:-}), docs/architecture.md
• Test: test/discovery-server.test.js (spawns server.js on a free port with a temp DATA/LIBRARY and DISCOVERY_ANILIST_URL pointing to an in-test http.createServer serving fixtures). Copy the helper pattern from an existing server-spawning test in test/ if one exists; otherwise write a minimal spawn/wait-for-port helper inside this file.

Interfaces:

• Consumes: Tasks 1–6.

• Produces:

  • HttpError(code, msg, extra = {}); the error response becomes { error, ...extra }.
  • DISCOVERY_HARVEST_CACHE = path.join(DATA, 'discovery-cache')
  • harvestAt default: process.env.DISCOVERY_HARVEST_AT if it matches /^([01]\d|2[0-3]):[0-5]\d$/, else '03:30'. store.settings overrides it.
  • async rescore({ onlyUser }): refactors the body of startDiscoveryRun. With onlyUser it calls generateAsync({ onlyUser }) then mergeUserFeed before publish; otherwise it runs a full generateAsync. startDiscoveryRun keeps its route behavior.
  • targetsFor(kind, userName): all opted-in users, or only userName for kind === 'user'. Each target is { user, anchors: anchorsFor(...), mature: !!settings.discoveryMature }.
  • Startup: if store.load().harvest?.status === 'running', patch it to interrupted with finishedAt. Queued jobs are not persisted.
  • Scheduler: setInterval(checkNightly, 60000).unref(), plus one call 30 s after startup. When shouldRunNightly is true and no harvest is active, call patch({ nightly: { lastDate } }) and then harvester.request({ kind: 'nightly' }).
  • Routes:
    • POST /api/discovery/refresh: caller only; 403 without opt-in. Returns { harvestId, position }, 409, or 429 with { nextAllowedAt }.
    • GET /api/discovery: adds refresh: harvester.state(u.name).
    • POST /api/admin/discovery/harvest: admin only; returns { harvestId, position } or 409.
    • GET /api/admin/discovery: adds harvest, currentHarvest, cache: harvestCache.stats(), settings: { harvestEnabled, harvestAt }, nextNightlyAt.
    • POST /api/admin/discovery/settings: admin only. Accepts exactly { harvestEnabled: boolean, harvestAt: 'HH:MM' }; anything else → 400. Saved via patch({ settings }).

• Step 1: Write the failing integration tests:

  • refresh requires opt-in, queues, runs and rescores only the caller: two opted-in users. Alice refreshes, and her feed afterwards contains a fixture rec candidate. Bob's generatedAt is unchanged.
  • second refresh within an hour is 429 with nextAllowedAt
  • admin harvest is admin-only and single-flight: 403 for a non-admin; 409 on a second call while one is running (the fixture server delays its responses by 500 ms).
  • settings validation: harvestAt: '25:00' → 400; an extra key → 400; a valid body round-trips through GET /api/admin/discovery.
  • interrupted harvest is marked on startup: pre-write a discovery.json with harvest.status: 'running', start the server, and get interrupted.
  • harvest never touches scraper queue or library: no files change under the temp LIBRARY or DATA/scraper. Snapshot the directory listings and mtimes before and after.

• Step 2: Run node --test test/discovery-server.test.js. Expected: FAIL.

• Step 3: Implement the server changes, the compose env lines, and the new routes in docs/architecture.md.

• Step 4: Run npm test. Expected: all suites PASS.

• Step 5: Commit Discovery harvest routes, nightly schedule and per-user refresh (BL-0051) and push.

Task 8: Discover UI, browser test and docs

Files:

• Modify: public/discover.js, public/discover.css if present (else the stylesheet discover.html links), tools/test-discovery-ui.js, tools/test-compose.js (pass DISCOVERY_ANILIST_URL=http://127.0.0.1:9/ for --discovery-ui, so a harvest fails fast without reaching AniList), docs/discovery.md, docs/deployment.md, vision/discovery-scout.md

Interfaces:

• Consumes: Task 7 routes and response fields.

• UI, using the existing h()/button() helpers. Never use innerHTML; text goes in through h() children.

  • Refresh my suggestions button, shown when state.enabled. While refresh.state is queued it reads Queued (position N), while running it reads Fetching…, and both disable it. During a cooldown it shows Next refresh possible at <local HH:MM> (from nextAllowedAt) and is disabled. It polls every 1.5 s while queued or running, like run().
  • Card: show item.reason; the details list shows all item.reasons and evidence, plus · fetched <local date> when entry.fetchedAt is set.
  • Settings hint for Personalized discovery, exact text: Off by default. When on, the server sends AniList the IDs of titles you liked, without login, to find related work. Your feed and feedback stay private to this server.
  • The mature hint becomes Only affects titles AniList marks as adult or with a known mature tag.
  • Admin panel:
    • adds a line Last harvest: <status> · <requests>/<budget> requests · <stopReason or —> · cache <files> files, plus the next nightly time
    • adds a Harvest now button
    • adds an Automatic harvest checkbox and a time <input type="time"> that save through the settings route
    • replaces the old "This first version…" paragraph with: Harvests ask AniList for public metadata only. They never ask image hosters, call an LLM, or edit the scraper queue.

• Docs:

  • docs/discovery.md: replace the deliberate-limits section with the seed kinds, budgets, cache, nightly time, privacy note and stop reasons.
  • docs/deployment.md: data/discovery-cache/ is safe to delete and can be left out of backups; document DISCOVERY_HARVEST_AT and DISCOVERY_ANILIST_URL (test only).
  • vision/discovery-scout.md §4: mark the candidate input as implemented, with a link to the spec.

• Step 1: Extend tools/test-discovery-ui.js, after the existing Maybe click:

  • Click Refresh my suggestions; wait until the button text matches /Next refresh possible at \d{1,2}:\d{2}/.
  • The admin panel shows Last harvest: stopped and upstream-errors.
  • Turn on Automatic harvest, set the time to 04:15, reload, and see 04:15.
  • Harvest now while one is running shows the 409 message in the status note.
  • Review Focus 5: before the stack starts, seed the disposable data tree with a discovery.json whose candidate title is <img src=x onerror=alert(1)> and which is in the QA admin's feed. Assert that the card's h2 textContent equals that string and that h2 img count is 0.
  • The existing non-admin checks additionally assert Harvest now count 0 and POST /api/admin/discovery/harvest → 403.
  • Mobile viewport (375×812): the refresh button is visible without horizontal scroll (document.documentElement.scrollWidth <= 375).

• Step 2: Run docker build -t mangacircus:discovery-test . then APP_IMAGE=mangacircus:discovery-test npm run test:discovery-ui. Expected: FAIL on the missing button.

• Step 3: Implement the UI and docs.

• Step 4: Run the same two commands, then npm test. Expected: PASS.

• Step 5: Commit Discover page: refresh, harvest admin and seed reasons; docs (BL-0051) and push.

• Step 6: Update the project's coordination backlog item and release the claim. Merging into main waits for the user.