MaashkinoozheandClaude Opus 5.5
cd6a30a76b
- Cache prune counts directory entries first and only reads files when over the cap; tests for cross-kind prune, schema-mismatch and corrupt entries. - Scoring tolerates malformed manga.json and cached records (non-list tags, object or null relations, non-object items); a tag without a rank no longer passes the top-tag filter; authors without an integer AniList id are dropped. - discovery.json publishes only candidates some feed shows; examined/clipped still describe the pool. Tests that asserted the published count equals the pool size now check the pool via candidatePool() and the published set against the feeds. - The post-harvest rescore failure test now breaks discovery.json right before the publish (via a scraper stand-in) instead of relying on tags: 5, and a following successful harvest clears the stale lastError. - Consent text names everything sent to AniList; paused label mentions admin harvests; the UI test asserts the XSS seed instead of skipping it. - Slot-mix test proves the fresh/outside reservations matter. - Docs: privacy note, fresh fallback reason, per-period cap, dry-run wording, published-candidate rule; plan/spec hygiene and spec status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
133 lines
7.4 KiB
Markdown
133 lines
7.4 KiB
Markdown
# Discovery Scout
|
|
|
|
The reader has a separate **Discover** page. Suggestions come from the curated
|
|
Greatest catalog, cached AniList relations of Greatest and library titles, and a
|
|
wider pool that the reader harvests from AniList's public API for the users who
|
|
opted in (design: [the wider-pool spec](superpowers/specs/2026-10-25-discovery-wider-pool-design.md)).
|
|
Discovery does not call an LLM, inspect chapter/image hosters, edit a collection,
|
|
or enqueue downloads. The reader remains the only writer of account and discovery
|
|
state; no further Compose service, credential, or database is required.
|
|
|
|
## Use
|
|
|
|
1. Open `discover.html`. Each account independently enables **Personalized
|
|
discovery** (off by default). Reading activity is a separate opt-in. The
|
|
mature-title switch only affects titles AniList marks as adult or with a known
|
|
mature tag; it is not a content-safety guarantee.
|
|
2. **Refresh my suggestions** queues a harvest for your own account (at most once an
|
|
hour). The button shows the queue position, `Fetching…` while it runs, and when
|
|
the next refresh is possible. Your existing suggestions stay visible until the
|
|
new feed is published.
|
|
3. An admin can press **Run discovery now** (re-score from local data, no network)
|
|
or **Harvest now** (fetch from AniList for every opted-in user, then re-score).
|
|
A second concurrent run or harvest is rejected. A nightly harvest runs on its
|
|
own; the admin panel can switch it off or change its time.
|
|
4. A reader can open **Why this is here · evidence** and record Interested, Maybe,
|
|
No thanks, or Hide. Feedback is private to the account. Dismissed/hidden
|
|
titles disappear at once; other feedback influences the next run. The user
|
|
can undo a choice by clicking its active button again.
|
|
|
|
The recommendation is an editorial lead, **not** an availability or edition
|
|
check. The public metadata link opens the provider record; an admin still uses
|
|
the existing source inspection and confirmation process to download anything.
|
|
The title/year/format matching is intentionally conservative but not a verified
|
|
identity merge. Missing year, cover, or status remains missing.
|
|
|
|
## Seeds from AniList
|
|
|
|
A harvest starts from each opted-in user's **anchors**: titles they voted
|
|
`flame`/`good` in Greatest, marked Interested/Maybe in Discover, read (only with
|
|
reading activity opted in), or own with an AniList ID. The 15 strongest anchors
|
|
count. It then asks AniList for four seed kinds:
|
|
|
|
| Seed | What | Reason shown on the card |
|
|
|---|---|---|
|
|
| Community recommendations | "readers who liked X recommend Y" for each anchor | AniList readers who liked {anchor} recommend this ({rating}). |
|
|
| Same creator | other works by the story/art authors of strong anchors | Also by {author} — you read/liked {anchor}. |
|
|
| Top in your tags | well-rated titles in the user's three strongest tags | Top-rated in {tag}, one of your most-liked tags. |
|
|
| Fresh | recently started, releasing and trending titles | New: started {year}, ongoing and trending. — or, when the cached record does not show an ongoing title started in the last two years, Trending on AniList right now. |
|
|
|
|
A title found by several seeds keeps every reason and evidence item; the card
|
|
shows the strongest, the details list all of them, each with the date it was
|
|
fetched. A feed holds at most 20 suggestions, at most 8 from one seed kind, at most
|
|
4 from one Greatest period (or from related works), up to 4 fresh titles, and up to 4
|
|
from outside your strongest tags.
|
|
|
|
- **Budgets:** a nightly or admin harvest makes at most 300 requests, a
|
|
per-user refresh 60. Requests are sequential and limited to 25 a minute. When
|
|
the budget is spent the harvest stops; what was fetched is still used.
|
|
- **Cache:** `data/discovery-cache/`, one file per response. Titles,
|
|
recommendations and creators stay fresh for 30 days, tag lists and the fresh
|
|
list for 7 days; at most 5,000 files, oldest pruned first. Scoring reads only
|
|
this cache and never the network, so a failed harvest means older candidates,
|
|
never an empty feed.
|
|
- **Nightly time:** `03:30` by default, or `DISCOVERY_HARVEST_AT`, in the
|
|
container's local time zone (`TZ`). The admin panel overrides both and can turn
|
|
the nightly harvest off. A night missed while the reader was down runs once
|
|
after startup.
|
|
- **Privacy:** for opted-in users only, the server sends AniList the IDs of
|
|
their anchor titles (liked, read or owned), the AniList staff IDs of those titles'
|
|
authors and the names of their top tags, without login, cookies or user names. Feeds and feedback stay private to this server; cache entries are
|
|
not tied to a user.
|
|
|
|
### Stop reasons
|
|
|
|
The admin panel shows the last harvest as `status · requests/budget · stop
|
|
reason · cache files`.
|
|
|
|
| Stop reason | Meaning | What happens next |
|
|
|---|---|---|
|
|
| `budget` | The request budget ran out | Normal; the next harvest continues where the cache is stale |
|
|
| `rate-limited` | AniList answered 429 | Harvesting pauses; refresh buttons show "Paused by AniList" until the next nightly or admin harvest |
|
|
| `forbidden` | AniList answered 401/403 | Same pause |
|
|
| `challenge` | AniList answered with an HTML or challenge page | Same pause |
|
|
| `upstream-errors` | Five timeouts or server errors in a row | The harvest ends; nothing is paused |
|
|
| `error` | A local fault (for example a cache write) | See the reader log |
|
|
|
|
Discovery never retries with a different identity, user agent or host. A
|
|
re-scoring failure after a harvest is shown as the admin panel's error line.
|
|
|
|
## Data and API
|
|
|
|
- `data/discovery.json`: atomic, reader-owned per-user feed snapshots and the
|
|
candidates those feeds show (the rest of the pool is only counted:
|
|
`examined`, `clipped`), plus the last harvest record and the admin's harvest settings. A
|
|
previous valid version is retained as `discovery.json.prev`. A corrupt file
|
|
makes reads/runs/harvests fail closed rather than overwrite it.
|
|
- `data/discovery-cache/`: harvested AniList responses; safe to delete.
|
|
- `data/db.json`: consent in user settings and explicit feedback under
|
|
`discoveryFeedback`. The account-deletion route removes that feedback.
|
|
- Routes: [architecture.md](architecture.md) (`/api/discovery*`,
|
|
`/api/admin/discovery*`) and §6.6 there for the harvester.
|
|
|
|
The job asks the private scraper state API for queued titles to avoid recommending
|
|
them. If that API is unavailable, the run still uses only cached sources; the
|
|
admin result records `queueAvailable: false` and queue exclusion is incomplete.
|
|
No scraper queue edit occurs.
|
|
|
|
## Read-only dry run and verification
|
|
|
|
From the project directory:
|
|
|
|
```bash
|
|
node discovery/cli.js run --dry-run --data data --show
|
|
npm run test:reader
|
|
docker build -t mangacircus:discovery-test .
|
|
APP_IMAGE=mangacircus:discovery-test npm run test:discovery-ui
|
|
```
|
|
|
|
The dry run reads `db.json`, a saved library index if present, Greatest, and
|
|
cached AniList metadata. It does not contact AniList or any other provider and does
|
|
not publish `discovery.json`; reading the harvest cache may still delete a corrupt
|
|
cache file, as the reader would. The
|
|
Compose/browser suite uses a disposable data tree and queue, not the live one,
|
|
and points `DISCOVERY_ANILIST_URL` at a closed local port so no harvest reaches
|
|
AniList.
|
|
|
|
## Limits
|
|
|
|
There is no admin seed form, edition-finding action, LLM rationale, or explicit
|
|
purge control for old feed snapshots. Disabling consent stops feed display,
|
|
harvesting and future personalized generation for that account, but existing
|
|
snapshot/feedback is retained until account deletion.
|