0 Fork 0

Files

T

MaashkinoozheandClaude Opus 5.5 12c54e5ffb Plan: wider Discovery candidate pool (BL-0051)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-25 08:41:47 +01:00

31 KiB

Raw Blame History

Discovery wider pool Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Widen Discover's candidate pool with four AniList-sourced seed kinds (community recommendations, same creator, top in tags, fresh), harvested server-side nightly and per user, scored locally.

Architecture: New reader-side modules: a file cache, a rate-limited AniList client, anchor selection, and a single-flight harvester. The existing engine scores from Greatest + relations + harvest cache; server.js wires routes, the nightly scheduler and rescoring. Harvest (network) and scoring (local) stay separate, so a failed harvest never empties a feed.

Tech Stack: Node.js built-ins only (node:test, fetch, fs), plain browser JS, Playwright for the Compose UI test.

Spec: docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md

Global Constraints

No runtime dependencies; Node built-ins only (AGENTS.md).
One writer: only the reader writes data/discovery.json and data/discovery-cache/; never write under data/scraper/.
All fetching runs in the reader container, never in the browser.
AniList: at most 25 requests per minute, one at a time; stop the whole harvest on 429, 403 or a challenge; never retry with a changed identity, user agent or host.
Budgets: nightly and admin harvests 300 requests, per-user refresh 60. Per-user cooldown 1 hour. At most 15 anchors per user.
Cache TTL: title, recs, creator 30 days; tagTop, fresh 7 days. Cap 5,000 files, oldest fetchedAt pruned first.
Feed: 20 items; at most 8 per seed kind, at most 4 per Greatest period/Related works; up to 4 slots reserved for fresh, up to 4 for candidates outside the user's top-3 tags. Candidate cap 2,000.
Personalized discovery and reading activity remain separate opt-ins, both off by default.
Discovery never edits the scraper queue, library or collections.
Tests never call live AniList; fixtures in test/fixtures/anilist/ are handcrafted and contain no personal data.
Public repo: no host names, IPs, local paths or personal seeds anywhere.
Commits end with Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>; push feature/discovery-wider-pool after each task.

Review Focus

A fresh install with an empty cache and no network must behave exactly like Phase A: the feed comes from Greatest and relations, and the harvest record shows a stop reason. Test in Task 6 (engine: empty harvest cache equals Phase A pool) and Task 4 (harvest: network error stops with upstream-errors after 5).
A user with no anchors (opted in, no votes, empty library) gets no harvest requests except the global fresh list, and still a feed. Test in Task 4 (plan for anchorless user only fetches fresh).
Old discovery.json files without harvest, settings or nightly fields must load without the fail-closed error. Test in Task 5 (store loads a Phase A file without new fields).
A per-user rescore must keep every other user's feed and the candidates those feeds reference. Test in Task 6 (mergeUserFeed keeps other users' referenced candidates).
AniList text containing HTML or script (titles, tag names) is shown as text, never as markup. Test in Task 8 (UI test renders a fixture title <img src=x onerror=alert(1)> as text).

Task 1: Discovery cache

Files:

Create: discovery/cache.js
Test: test/discovery-cache.test.js

Interfaces:

Produces:
- KINDS = ['title', 'recs', 'creator', 'tagTop', 'fresh']
- TTL_MS = { title: 30*86400000, recs: 30*86400000, creator: 30*86400000, tagTop: 7*86400000, fresh: 7*86400000 }
- class DiscoveryCache { constructor(dir: string, { maxFiles = 5000, now = Date.now } = {}) }
- get(kind, key) -> { schema: 1, kind, key, fetchedAt: ISO string, data } | null. A corrupt or schema-mismatched file is deleted and returns null.
- isFresh(kind, key) -> boolean: true when an entry exists and now() - fetchedAt < TTL_MS[kind].
- put(kind, key, data) -> entry: atomic write (temp file + rename), then pruning.
- list(kind) -> entry[], including expired entries.
- stats() -> { files: number, bytes: number }
File path: <dir>/<kind>/<base64url(String(key))>.json. An unknown kind throws TypeError.
Step 1: Write the failing tests in test/discovery-cache.test.js, each using an fs.mkdtempSync dir and a fake now:
- cache round-trips an entry: put('title', 42, {a:1}), then get('title', 42).data deep-equals {a:1} and fetchedAt equals new Date(now()).toISOString().
- freshness follows TTL per kind: with now advanced by 8 days, isFresh('fresh','global') is false and isFresh('title',1) is true. At 31 days isFresh('title',1) is false, while get still returns the entry.
- prune keeps the newest maxFiles: with maxFiles: 3 and 5 puts at increasing now, stats().files === 3 and the first two keys return null.
- corrupt entry is deleted: write {broken to the entry file; get returns null and the file no longer exists.
- odd keys are file-safe: put('tagTop', 'genre:Slice of Life/../x', …) lands inside <dir>/tagTop/.
- unknown kind throws.
Step 2: Run node --test test/discovery-cache.test.js. Expected: FAIL, Cannot find module '../discovery/cache'.
Step 3: Implement discovery/cache.js per the Interfaces. Prune by reading each entry's fetchedAt across all kinds, sorting oldest first, and deleting until files <= maxFiles. Follow discovery/store.js style.
Step 4: Run node --test test/discovery-cache.test.js. Expected: PASS.
Step 5: Commit Discovery cache with per-kind TTL and size cap (BL-0051) and push.

Task 2: AniList client

Files:

Create: discovery/anilist.js, test/fixtures/anilist/{title,recs,creator,tagtop,fresh,novel}.json
Test: test/discovery-anilist.test.js

Interfaces:

Produces:
- createLimiter({ perMinute = 25, now = Date.now, sleep = ms => new Promise(r => setTimeout(r, ms)) }) -> { wait(): Promise<void> }. Calls are serialized and spaced at least 60000 / perMinute ms apart.
- class AniListStop extends Error { reason: 'rate-limited' | 'forbidden' | 'challenge' }
- class AniListSkip extends Error { reason: 'timeout' | 'upstream' | 'invalid' }
- normalizeMedia(media) -> Record | null. Returns null unless format is MANGA or ONE_SHOT.
- Record = { anilistId, title, altTitles: string[], format, isAdult: boolean, status: 'ongoing'|'completed'|'unknown', year: number|null, averageScore: number|null, popularity: number|null, trending: number|null, tags: [{ name, rank, kind: 'genre'|'tag', isAdult: boolean }], authors: [{ staffId, name, role }], coverUrl: string|null, siteUrl: string|null }.
  - Genres map to { rank: 100, kind: 'genre', isAdult: false }.
  - Status: RELEASING → ongoing, FINISHED → completed, anything else → unknown.
  - authors keeps the staff roles matched by the regex in scraper/metadata/anilist.js (original story|story(?: & art)?|art|illustration).
- createClient({ endpoint = process.env.DISCOVERY_ANILIST_URL || 'https://graphql.anilist.co', limiter = createLimiter(), fetchImpl = fetch, timeoutMs = 30000, today = () => new Date() }) returns:
  - title(id) -> Record
  - recs(id) -> [{ anilistId, rating, record }], keeping only rating > 0
  - creator(staffId) -> [{ anilistId, role, record }]
  - tagTop({ name, kind }) -> [{ anilistId, record }]
  - fresh() -> [{ anilistId, record }]
  - Entries whose normalizeMedia returns null are dropped.
Queries share one MEDIA_FIELDS fragment: id format isAdult title{romaji english native} synonyms status startDate{year} averageScore popularity trending genres tags{name rank isAdult} staff(perPage:20){edges{role node{id name{full}}}} coverImage{extraLarge} siteUrl. Page sizes and sort orders come from spec §2.1:
- tagTop uses popularity_greater: 1000, with genre_in or tag_in by kind.
- fresh uses status: RELEASING, startDate_greater: <(today.year − 2) * 10000 + month * 100 + day>, sort: TRENDING_DESC, perPage: 50.
- All list queries use the full fragment, so no follow-up title fetches are needed. This replaces the spec's follow-up stage on purpose.
Error mapping, applied after await limiter.wait():
- HTTP 429 → AniListStop('rate-limited')
- HTTP 401 or 403 → AniListStop('forbidden')
- non-JSON body or content-type text/html → AniListStop('challenge')
- abort or timeout → AniListSkip('timeout')
- HTTP 5xx → AniListSkip('upstream')
- missing data.Media / data.Page.media / data.Staff, or a media without an integer id → AniListSkip('invalid')
- Requests send content-type: application/json and the fixed header user-agent: MangaCircus-Discovery/1.0. Do not reuse SlowHttp identity rotation.
Step 1: Write fixtures with minimal handcrafted responses:
- recs.json has 3 recommendations, one with rating: -2 and one with format: NOVEL.
- title.json has isAdult: false, 2 genres and 2 tags, one of them isAdult: true.
Step 2: Write the failing tests in test/discovery-anilist.test.js. Use a stub fetchImpl returning new Response(JSON.stringify(fixture), { status, headers }) and a fake limiter { wait: async () => {} }:
- normalizeMedia maps fields and drops novels: asserts the Record shape on title.json; normalizeMedia({ ...media, format: 'NOVEL' }) === null.
- recs keeps positive ratings and manga formats: exactly 1 entry with its rating.
- fresh query sends startDate two years back: with today = new Date('2026-10-25'), the request body's variables contain startDate_greater: 20241025.
- tagTop uses genre_in for genres and tag_in for tags.
- 429 stops, 403 stops, html body is a challenge: each throws AniListStop with its reason.
- 5xx and timeout skip, missing data is invalid: each throws AniListSkip with its reason.
- limiter spaces calls 2400 ms apart: the fake sleep records each duration and advances the fake now by it. Three wait() calls issued at the same instant record [0, 2400, 2400].
Step 3: Run node --test test/discovery-anilist.test.js. Expected: FAIL (module missing).
Step 4: Implement discovery/anilist.js per the Interfaces.
Step 5: Run the same command. Expected: PASS.
Step 6: Commit Rate-limited AniList client for discovery seeds (BL-0051) and push.

Task 3: Anchors and top tags

Files:

Create: discovery/anchors.js
Test: test/discovery-anchors.test.js

Interfaces:

Consumes: Record from Task 2.
Produces:
- ANCHOR_WEIGHTS = { flame: 5, good: 3, interested: 4, maybe: 2, read: 3, owned: 1 }
- MAX_ANCHORS = 15
- anchorsFor({ user, db, library, greatest }) -> [{ anilistId: number, weight, source: keyof ANCHOR_WEIGHTS, title }]. Returns [] unless user.settings.discoveryEnabled. Sources:
  - Greatest votes: db.greatestRatings[user.name][entryId] is flame or good, and entry.metadata.id is set.
  - Feedback: db.discoveryFeedback[user.name]['anilist:N'] is interested or maybe.
  - read: only with settings.discoveryReadingActivity; a library manga with x-anilist.id and any chapter in db.progress[user.name][manga.id] that has done.
  - owned: a library manga with meta['x-anilist'].id.
  - Dedupe by id, keeping the highest weight. Sort by weight desc, then anilistId asc. Cap at 15.
- topTagsFor(anchors, recordOf: id => Record|null, { mature = false }) -> [{ name, kind }], at most 3. Sums anchor weight per tag with rank >= 60. Skips isAdult tags unless mature. Ties are broken by name asc.
Step 1: Write the failing tests:
- anchors are empty without opt-in
- weights, dedupe and cap: a title voted good that is also owned yields one anchor with weight 3. Twenty owned titles yield 15 anchors.
- reading anchors need the reading opt-in
- feedback keys parse anilist ids and ignore others: greatest:x and anilist:abc are ignored.
- top tags weigh anchors, skip low rank and adult tags: an adult tag is included with { mature: true }.
Step 2: Run node --test test/discovery-anchors.test.js. Expected: FAIL.
Step 3: Implement discovery/anchors.js.
Step 4: Run the same command. Expected: PASS.
Step 5: Commit Discovery anchors and top tags (BL-0051) and push.

Task 4: Harvester

Files:

Create: discovery/harvest.js
Test: test/discovery-harvest.test.js

Interfaces:

Consumes: DiscoveryCache (Task 1), the client and its errors (Task 2), anchorsFor/topTagsFor (Task 3). It also consumes the store methods load() and patch({ harvest }) from Task 5; tests pass a stub { patch(){} }, so Task 5 can land later.
Produces:
- BUDGETS = { nightly: 300, admin: 300, user: 60 }
- REFRESH_COOLDOWN_MS = 3600000
- MAX_CONSECUTIVE_FAILURES = 5
- async runHarvest({ kind, targets: [{ user, anchors, mature }], cache, client, budget }) -> { requests, budget, status: 'success'|'stopped', stopReason: null|'budget'|'rate-limited'|'forbidden'|'challenge'|'upstream-errors' }. Stages, in order, each skipping requests whose cache.isFresh(kind, key):
  1. title for every anchor.
  2. recs per anchor.
  3. creator per author staffId of anchors with weight >= 3, read from the cached title record.
  4. tagTop per topTagsFor(...) tag, keyed ${kind}:${name}.
  5. One fresh, key global.
  Requests are deduped across targets. Each success is cache.put. An AniListSkip increments a consecutive-failure counter that resets on success; at 5 the run returns stopped/upstream-errors. An AniListStop returns stopped with its reason immediately. Reaching budget returns success/budget.
- shouldRunNightly({ now: Date, harvestAt: 'HH:MM', lastNightlyDate: 'YYYY-MM-DD'|null, enabled }) -> boolean: true when enabled, local now is at or after harvestAt, and the local date differs from lastNightlyDate.
- class Harvester:
  - constructor({ cache, client, store, targetsFor: (kind, userName?) => targets, onDone: async (job, result) => void, now = Date.now })
  - request({ kind: 'nightly'|'admin'|'user', user?: string }) -> { harvestId, position }. Throws Object.assign(new Error(msg), { status, nextAllowedAt? }) with:
    - 409 for an admin or nightly harvest while one is running or queued, and for a user who already has one queued or running.
    - 429 within REFRESH_COOLDOWN_MS of that user's last accepted request; the error carries nextAllowedAt as an ISO string.
  - state(userName) -> { state: 'idle'|'queued'|'running', position?: number, nextAllowedAt?: string }
  - current() -> { id, kind, user?, startedAt } | null
  - Jobs run one at a time, FIFO. Each run calls store.patch({ harvest: { id, kind, user, status: 'running', startedAt, requests: 0, budget } }), then the final record (status, finishedAt, requests, stopReason), then onDone(job, result). Errors thrown in onDone are logged and do not break the queue.
Step 1: Write the failing tests using a fake client (methods resolve from fixture maps, count calls, and can throw scripted errors) and a real DiscoveryCache in a temp dir:
- plan order and dedupe: two targets sharing an anchor. The call log starts with title calls, then recs, creator, tagTop, fresh; the shared anchor is fetched once.
- fresh cache entries are skipped
- budget stops with reason budget: budget 3 → requests === 3, stopReason: 'budget'.
- 429 stops at once: the call count does not grow after the throw.
- network error stops with upstream-errors after 5: five consecutive AniListSkip; a success in between resets the counter.
- plan for anchorless user only fetches fresh
- shouldRunNightly: 03:29 → false; 03:30 → true; same date → false; disabled → false.
- harvester single-flight, positions, 409 and 429 cooldown: with fake now, a second user request inside one hour throws 429 with nextAllowedAt. An admin request while one is running throws 409. A queued user's state() returns queued with position 1.
- onDone failure does not stall the queue
Step 2: Run node --test test/discovery-harvest.test.js. Expected: FAIL.
Step 3: Implement discovery/harvest.js.
Step 4: Run the same command. Expected: PASS.
Step 5: Commit Single-flight discovery harvester with budgets and stop reasons (BL-0051) and push.

Task 5: Store metadata

Files:

Modify: discovery/store.js
Test: test/discovery.test.js (add tests)

Interfaces:

Produces:
- load() defaults harvest: null, settings: null, nightly: { lastDate: null } when the fields are absent. The validation of existing fields is unchanged.
- publish(value) keeps the prior harvest, settings and nightly.
- patch({ harvest?, settings?, nightly? }) -> data writes atomically through the same temp+.prev+rename path, leaves revision unchanged, and refuses on a corrupt file like publish.
- Extract a private write(data) shared by publish and patch.
Step 1: Write the failing tests:
- store loads a Phase A file without new fields: write a Phase A-shaped JSON; load().nightly.lastDate === null.
- patch keeps revision and publish keeps harvest: patch({ harvest: { status: 'success' } }), then publish(...); load().harvest.status === 'success' and the revision went up by exactly 1.
- patch refuses a corrupt store
Step 2: Run node --test test/discovery.test.js. Expected: the new tests FAIL.
Step 3: Implement the store changes.
Step 4: Run npm run test:reader. Expected: PASS.
Step 5: Commit Discovery store keeps harvest, settings and nightly state (BL-0051) and push.

Task 6: Engine — wider pool, new scores, slot mix

Files:

Modify: discovery/engine.js, discovery/cli.js (pass harvestCache when data/discovery-cache exists)
Test: test/discovery.test.js (update the cap assertion; add tests), test/fixtures/anilist/ reused through a temp DiscoveryCache

Interfaces:

Consumes: DiscoveryCache (Task 1), Record (Task 2), anchorsFor/topTagsFor (Task 3).
Produces:
- MAX_CANDIDATES = 2000, MAX_FEED = 20
- SLOTS = { perKind: 8, perPeriod: 4, fresh: 4, outside: 4 }
- buildCandidates({ greatest, library, cacheDir, harvestCache? }): adds one candidate per distinct anilistId found in recs, creator, tagTop and fresh entries. New candidate fields: isAdult, averageScore, tagRecords: Record.tags, authorRecords: Record.authors, fetchedAt. tags and authors stay name arrays. Greatest candidates keep their fields; when a harvested record has the same id, it fills isAdult and tagRecords. Era bucketing stays, and harvested candidates bucket by their seed kind instead of Related works.
- feedForUser({ user, db, candidates, greatest, library, queuedTitles, harvestCache? }) adds these parts, with values from spec §4, rounded to 2 decimals:
  - communityRec = min(5, 1 + log2(1 + rating) / 2) * anchorWeight / 5, using the best (anchor, rating) pair among the user's anchors whose recs list contains the candidate.
  - sameCreator = 4 * anchorWeight / 5, when the candidate's authorRecords shares a staffId with an anchor of weight >= 3.
  - tagTop = 3 | 2 | 1 for the user's 1st, 2nd or 3rd top tag, only if averageScore >= 75 and the candidate is in that tag's tagTop list.
  - fresh = 2 if status === 'ongoing' and year >= today.year - 2; 1 if only present in the fresh list.
- Mature filter: exclude unless settings.discoveryMature when isAdult === true or any tagRecords entry has isAdult && rank >= 60. Use the Phase A regex only when isAdult is undefined.
- Each row gets group. If the row's largest part is one of the four new parts, group is its kind (recs, creator, tagTop, fresh); otherwise it is period || 'Related works'.
- reason comes from the largest part. reasons: string[] lists every applicable reason in descending part order. Exact templates:
  - recs: AniList readers who liked ${anchor} recommend this (${rating}).
  - creator: Also by ${author} — you ${read|liked} ${anchor}. Use read when the anchor source is read, otherwise liked.
  - tagTop: Top-rated in ${tag}, one of your most-liked tags.
  - fresh: New: started ${year}, ongoing and trending.
- Evidence ids: rec:<anchorId>:<candidateId>, creator:<staffId>:<candidateId>, tagtop:<kind>:<name>:<candidateId>, fresh:<candidateId>. Kinds: recommendation|creator|tag-top|fresh. Each carries url: record.siteUrl and fetchedAt.
- mixSlots(rows, { topTags }) -> rows (at most 20, ranked):
  1. Sort by score desc, then candidateId.
  2. Take up to 4 rows with scoreParts.fresh > 0.
  3. Take up to 4 rows whose tags intersect none of the topTags names.
  4. Fill greedily.
  - Steps 2–4 all respect the caps: perKind 8 for the four kinds and perPeriod 4 for other groups.
  - Never pad.
  - The final order is by score desc, then rank = index + 1.
- async generateAsync({ db, library, greatest, cacheDir, harvestCache, queuedTitles, onlyUser? }) -> same shape as generate(). Awaits new Promise(setImmediate) between users; with onlyUser, scores only that user. generate() stays synchronous for cli.js and the existing tests.
- mergeUserFeed(prior: { candidates, feeds }, result, userName) -> { candidates, feeds }: replaces feeds[userName], keeps every other feed, and keeps exactly the candidates referenced by any kept feed plus result.candidates.
Step 1: Update and write the failing tests:
- Change assert.equal(Object.keys(result.candidates).length, 50) to Math.min(2000, examined); keep the other Phase A assertions.
- engine: empty harvest cache equals Phase A pool: with and without an empty harvestCache, the candidate ids are equal.
- community rec, creator, tagTop and fresh parts and reasons: fixture cache plus a user with a flame anchor. Asserts exact part values, for example: rating 42, weight 5 → min(5, 1 + log2(43)/2) = 3.71. Also asserts the exact reason strings and evidence kinds with fetchedAt.
- isAdult filter: an isAdult candidate is absent without the mature opt-in and present with it.
- slot mix: 30 recs-group rows → at most 8 from that kind. Fresh rows are present even when low-scored. Rows outside the top tags are present. Fewer than 20 eligible rows → no padding.
- multiple seeds merge reasons: one candidate found by recs and creator has both reasons and both evidence entries.
- owned, queued, voted, dismissed still excluded for harvested candidates.
- mergeUserFeed keeps other users' referenced candidates
- scoring one user stays under 200 ms on 2000 candidates: synthetic records; measure feedForUser with performance.now().
Step 2: Run node --test test/discovery.test.js. Expected: the new tests FAIL.
Step 3: Implement the engine changes and the cli.js pass-through.
Step 4: Run npm run test:reader. Expected: PASS.
Step 5: Commit Discovery scores AniList seeds with slot mix and isAdult filter (BL-0051) and push.

Task 7: Server wiring

Files:

Modify: server.js (around the discovery block at lines 240–267, the routes at 490–517, HttpError at 81, the error response at 974, and the timers near 980–1012), docker-compose.yml (reader environment: DISCOVERY_HARVEST_AT: ${DISCOVERY_HARVEST_AT:-03:30}, DISCOVERY_ANILIST_URL: ${DISCOVERY_ANILIST_URL:-}), docs/architecture.md
Test: test/discovery-server.test.js (spawns server.js on a free port with a temp DATA/LIBRARY and DISCOVERY_ANILIST_URL pointing to an in-test http.createServer serving fixtures). Copy the helper pattern from an existing server-spawning test in test/ if one exists; otherwise write a minimal spawn/wait-for-port helper inside this file.

Interfaces:

Consumes: Tasks 1–6.
Produces:
- HttpError(code, msg, extra = {}); the error response becomes { error, ...extra }.
- DISCOVERY_HARVEST_CACHE = path.join(DATA, 'discovery-cache')
- harvestAt default: process.env.DISCOVERY_HARVEST_AT if it matches /^([01]\d|2[0-3]):[0-5]\d$/, else '03:30'. store.settings overrides it.
- async rescore({ onlyUser }): refactors the body of startDiscoveryRun. With onlyUser it calls generateAsync({ onlyUser }) then mergeUserFeed before publish; otherwise it runs a full generateAsync. startDiscoveryRun keeps its route behavior.
- targetsFor(kind, userName): all opted-in users, or only userName for kind === 'user'. Each target is { user, anchors: anchorsFor(...), mature: !!settings.discoveryMature }.
- Startup: if store.load().harvest?.status === 'running', patch it to interrupted with finishedAt. Queued jobs are not persisted.
- Scheduler: setInterval(checkNightly, 60000).unref(), plus one call 30 s after startup. When shouldRunNightly is true and no harvest is active, call patch({ nightly: { lastDate } }) and then harvester.request({ kind: 'nightly' }).
- Routes:
  - POST /api/discovery/refresh: caller only; 403 without opt-in. Returns { harvestId, position }, 409, or 429 with { nextAllowedAt }.
  - GET /api/discovery: adds refresh: harvester.state(u.name).
  - POST /api/admin/discovery/harvest: admin only; returns { harvestId, position } or 409.
  - GET /api/admin/discovery: adds harvest, currentHarvest, cache: harvestCache.stats(), settings: { harvestEnabled, harvestAt }, nextNightlyAt.
  - POST /api/admin/discovery/settings: admin only. Accepts exactly { harvestEnabled: boolean, harvestAt: 'HH:MM' }; anything else → 400. Saved via patch({ settings }).
Step 1: Write the failing integration tests:
- refresh requires opt-in, queues, runs and rescores only the caller: two opted-in users. Alice refreshes, and her feed afterwards contains a fixture rec candidate. Bob's generatedAt is unchanged.
- second refresh within an hour is 429 with nextAllowedAt
- admin harvest is admin-only and single-flight: 403 for a non-admin; 409 on a second call while one is running (the fixture server delays its responses by 500 ms).
- settings validation: harvestAt: '25:00' → 400; an extra key → 400; a valid body round-trips through GET /api/admin/discovery.
- interrupted harvest is marked on startup: pre-write a discovery.json with harvest.status: 'running', start the server, and get interrupted.
- harvest never touches scraper queue or library: no files change under the temp LIBRARY or DATA/scraper. Snapshot the directory listings and mtimes before and after.
Step 2: Run node --test test/discovery-server.test.js. Expected: FAIL.
Step 3: Implement the server changes, the compose env lines, and the new routes in docs/architecture.md.
Step 4: Run npm test. Expected: all suites PASS.
Step 5: Commit Discovery harvest routes, nightly schedule and per-user refresh (BL-0051) and push.

Task 8: Discover UI, browser test and docs

Files:

Modify: public/discover.js, public/discover.css if present (else the stylesheet discover.html links), tools/test-discovery-ui.js, tools/test-compose.js (pass DISCOVERY_ANILIST_URL=http://127.0.0.1:9/ for --discovery-ui, so a harvest fails fast without reaching AniList), docs/discovery.md, docs/deployment.md, vision/discovery-scout.md

Interfaces:

Consumes: Task 7 routes and response fields.
UI, using the existing h()/button() helpers. Never use innerHTML; text goes in through h() children.
- Refresh my suggestions button, shown when state.enabled. While refresh.state is queued it reads Queued (position N), while running it reads Fetching…, and both disable it. During a cooldown it shows Next refresh possible at <local HH:MM> (from nextAllowedAt) and is disabled. It polls every 1.5 s while queued or running, like run().
- Card: show item.reason; the details list shows all item.reasons and evidence, plus · fetched <local date> when entry.fetchedAt is set.
- Settings hint for Personalized discovery, exact text: Off by default. When on, the server sends AniList the IDs of titles you liked, without login, to find related work. Your feed and feedback stay private to this server.
- The mature hint becomes Only affects titles AniList marks as adult or with a known mature tag.
- Admin panel:
  - adds a line Last harvest: <status> · <requests>/<budget> requests · <stopReason or —> · cache <files> files, plus the next nightly time
  - adds a Harvest now button
  - adds an Automatic harvest checkbox and a time <input type="time"> that save through the settings route
  - replaces the old "This first version…" paragraph with: Harvests ask AniList for public metadata only. They never ask image hosters, call an LLM, or edit the scraper queue.
Docs:
- docs/discovery.md: replace the deliberate-limits section with the seed kinds, budgets, cache, nightly time, privacy note and stop reasons.
- docs/deployment.md: data/discovery-cache/ is safe to delete and can be left out of backups; document DISCOVERY_HARVEST_AT and DISCOVERY_ANILIST_URL (test only).
- vision/discovery-scout.md §4: mark the candidate input as implemented, with a link to the spec.
Step 1: Extend tools/test-discovery-ui.js, after the existing Maybe click:
- Click Refresh my suggestions; wait until the button text matches /Next refresh possible at \d{1,2}:\d{2}/.
- The admin panel shows Last harvest: stopped and upstream-errors.
- Turn on Automatic harvest, set the time to 04:15, reload, and see 04:15.
- Harvest now while one is running shows the 409 message in the status note.
- Review Focus 5: before the stack starts, seed the disposable data tree with a discovery.json whose candidate title is <img src=x onerror=alert(1)> and which is in the QA admin's feed. Assert that the card's h2 textContent equals that string and that h2 img count is 0.
- The existing non-admin checks additionally assert Harvest now count 0 and POST /api/admin/discovery/harvest → 403.
- Mobile viewport (375×812): the refresh button is visible without horizontal scroll (document.documentElement.scrollWidth <= 375).
Step 2: Run docker build -t mangacircus:discovery-test . then APP_IMAGE=mangacircus:discovery-test npm run test:discovery-ui. Expected: FAIL on the missing button.
Step 3: Implement the UI and docs.
Step 4: Run the same two commands, then npm test. Expected: PASS.
Step 5: Commit Discover page: refresh, harvest admin and seed reasons; docs (BL-0051) and push.
Step 6: Update MangaOrg: run python D:\Projects\Manga\MangaOrg\tools\org.py to set BL-0051 to review, release the MangaCircus discovery claim, then commit and push MangaOrg. Merging into main waits for the user.

31 KiB Raw Blame History Unescape Escape

Discovery wider pool Implementation Plan

Global Constraints

Review Focus

Task 1: Discovery cache

Task 2: AniList client

Task 3: Anchors and top tags

Task 4: Harvester

Task 5: Store metadata

Task 6: Engine — wider pool, new scores, slot mix

Task 7: Server wiring

Task 8: Discover UI, browser test and docs

31 KiB

Raw Blame History