Discovery final review fixes: lean publish, robust scoring, cheap prune (BL-0051)
- Cache prune counts directory entries first and only reads files when over the cap; tests for cross-kind prune, schema-mismatch and corrupt entries. - Scoring tolerates malformed manga.json and cached records (non-list tags, object or null relations, non-object items); a tag without a rank no longer passes the top-tag filter; authors without an integer AniList id are dropped. - discovery.json publishes only candidates some feed shows; examined/clipped still describe the pool. Tests that asserted the published count equals the pool size now check the pool via candidatePool() and the published set against the feeds. - The post-harvest rescore failure test now breaks discovery.json right before the publish (via a scraper stand-in) instead of relying on tags: 5, and a following successful harvest clears the stale lastError. - Consent text names everything sent to AniList; paused label mentions admin harvests; the UI test asserts the XSS seed instead of skipping it. - Slot-mix test proves the fresh/outside reservations matter. - Docs: privacy note, fresh fallback reason, per-period cap, dry-run wording, published-candidate rule; plan/spec hygiene and spec status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
613db96355
commit
cd6a30a76b
18 files changed
+250
-65
No files matched your search
@@ -251,6 +251,8 @@ The reader alone talks to AniList for discovery ([discovery.md](discovery.md)).
|
||||
startedAt, finishedAt, requests, budget, stopReason }`. `stopReason` is `budget`, `rate-limited`, `forbidden`,
|
||||
`challenge`, `upstream-errors` or `error` (a local fault). A run cut off by a restart is marked `interrupted`
|
||||
at startup; queued refreshes are not persisted.
|
||||
- **Published candidates:** only those at least one feed shows. `lastRun.candidateCount` (the admin
|
||||
candidate count) counts that set; `lastRun.examined` and `clipped` still describe the whole pool.
|
||||
- **Published fields:** candidates found through AniList carry `tagRecords`, `authorRecords`, `seed`
|
||||
(`recs|creator|tagTop|fresh`) and `fetchedAt`; feed items carry `group` (seed kind or period) and `reasons`
|
||||
(all reasons, strongest first) next to `reason`; seed evidence carries `fetchedAt`.
|
||||
|
||||
+12
-8
@@ -45,12 +45,13 @@ count. It then asks AniList for four seed kinds:
|
||||
| Community recommendations | "readers who liked X recommend Y" for each anchor | AniList readers who liked {anchor} recommend this ({rating}). |
|
||||
| Same creator | other works by the story/art authors of strong anchors | Also by {author} — you read/liked {anchor}. |
|
||||
| Top in your tags | well-rated titles in the user's three strongest tags | Top-rated in {tag}, one of your most-liked tags. |
|
||||
| Fresh | recently started, releasing and trending titles | New: started {year}, ongoing and trending. |
|
||||
| Fresh | recently started, releasing and trending titles | New: started {year}, ongoing and trending. — or, when the cached record does not show an ongoing title started in the last two years, Trending on AniList right now. |
|
||||
|
||||
A title found by several seeds keeps every reason and evidence item; the card
|
||||
shows the strongest, the details list all of them, each with the date it was
|
||||
fetched. A feed holds at most 20 suggestions, at most 8 from one seed kind, up to
|
||||
4 fresh titles, and some from outside your strongest tags.
|
||||
fetched. A feed holds at most 20 suggestions, at most 8 from one seed kind, at most
|
||||
4 from one Greatest period (or from related works), up to 4 fresh titles, and up to 4
|
||||
from outside your strongest tags.
|
||||
|
||||
- **Budgets:** a nightly or admin harvest makes at most 300 requests, a
|
||||
per-user refresh 60. Requests are sequential and limited to 25 a minute. When
|
||||
@@ -65,8 +66,8 @@ fetched. A feed holds at most 20 suggestions, at most 8 from one seed kind, up t
|
||||
the nightly harvest off. A night missed while the reader was down runs once
|
||||
after startup.
|
||||
- **Privacy:** for opted-in users only, the server sends AniList the IDs of
|
||||
their anchor titles (and the names of their top tags), without login, cookies or
|
||||
user names. Feeds and feedback stay private to this server; cache entries are
|
||||
their anchor titles (liked, read or owned), the AniList staff IDs of those titles'
|
||||
authors and the names of their top tags, without login, cookies or user names. Feeds and feedback stay private to this server; cache entries are
|
||||
not tied to a user.
|
||||
|
||||
### Stop reasons
|
||||
@@ -88,8 +89,9 @@ re-scoring failure after a harvest is shown as the admin panel's error line.
|
||||
|
||||
## Data and API
|
||||
|
||||
- `data/discovery.json`: atomic, reader-owned candidate and per-user feed
|
||||
snapshots plus the last harvest record and the admin's harvest settings. A
|
||||
- `data/discovery.json`: atomic, reader-owned per-user feed snapshots and the
|
||||
candidates those feeds show (the rest of the pool is only counted:
|
||||
`examined`, `clipped`), plus the last harvest record and the admin's harvest settings. A
|
||||
previous valid version is retained as `discovery.json.prev`. A corrupt file
|
||||
makes reads/runs/harvests fail closed rather than overwrite it.
|
||||
- `data/discovery-cache/`: harvested AniList responses; safe to delete.
|
||||
@@ -115,7 +117,9 @@ APP_IMAGE=mangacircus:discovery-test npm run test:discovery-ui
|
||||
```
|
||||
|
||||
The dry run reads `db.json`, a saved library index if present, Greatest, and
|
||||
cached AniList metadata. It writes nothing and does not contact providers. The
|
||||
cached AniList metadata. It does not contact AniList or any other provider and does
|
||||
not publish `discovery.json`; reading the harvest cache may still delete a corrupt
|
||||
cache file, as the reader would. The
|
||||
Compose/browser suite uses a disposable data tree and queue, not the live one,
|
||||
and points `DISCOVERY_ANILIST_URL` at a closed local port so no harvest reaches
|
||||
AniList.
|
||||
|
||||
@@ -334,4 +334,4 @@
|
||||
- [ ] **Step 3: Implement** the UI and docs.
|
||||
- [ ] **Step 4: Run** the same two commands, then `npm test`. Expected: PASS.
|
||||
- [ ] **Step 5: Commit** `Discover page: refresh, harvest admin and seed reasons; docs (BL-0051)` and push.
|
||||
- [ ] **Step 6: Update MangaOrg**: run `python D:\Projects\Manga\MangaOrg\tools\org.py` to set BL-0051 to `review`, release the `MangaCircus discovery` claim, then commit and push MangaOrg. Merging into `main` waits for the user.
|
||||
- [ ] **Step 6: Update the project's coordination backlog item and release the claim.** Merging into `main` waits for the user.
|
||||
@@ -1,6 +1,6 @@
|
||||
# Discovery: wider candidate pool — design
|
||||
|
||||
Status: approved design, not implemented. Date: 2026-10-25. Builds on the
|
||||
Status: implemented on branch feature/discovery-wider-pool. Date: 2026-10-25. Builds on the
|
||||
deterministic Discovery Scout ([`docs/discovery.md`](../../discovery.md),
|
||||
[`vision/discovery-scout.md`](../../../vision/discovery-scout.md)).
|
||||
|
||||
|
||||
Reference in new issue
Block a user