Checkpoint peering identity and opt-in discovery

This commit is contained in:
Codex committed 2026-10-13 22:00:54 +02:00
1 parent eba9b50e30
commit 47de081721
44 files changed
+1557 -55

No files matched your search

+2
View File
@@ -15,6 +15,8 @@ Related documents:
- [resilience.md](resilience.md): supervision, recovery limits and incidents.
- [compose-deployment.md](compose-deployment.md): local/VM deployment and release workflow.
- [settings-navigation.md](settings-navigation.md): User/Admin workspace, scraper tabs and deep links.
- [peering.md](peering.md): optional private server-to-server catalog and transfer.
- [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
## 1. Principles
+72
View File
@@ -0,0 +1,72 @@
# Discovery Scout (deterministic first release)
The reader now has a separate **Discover** page. This first slice deliberately uses
the curated Greatest catalog, already-cached AniList identities/relations, library
metadata, each user's Greatest votes, and explicit Discover feedback. It does not
call an LLM, fetch new metadata, inspect chapter/image hosters, edit a collection,
or enqueue downloads. The reader remains the only writer of account and discovery
state; no fourth Compose service, credential, or database is required.
## Use
1. Open `discover.html`. Each account independently enables **Personalized
discovery** (off by default). Reading activity is a separate opt-in. The
mature-title switch only applies to candidates with a known mature tag; it is
not a content-safety guarantee.
2. An admin presses **Run discovery now** on the same page. Existing suggestions
remain visible until the new feed is published. Runs are bounded to 50
candidates and 20 suggestions per opted-in user, with no more than four from
one period. A second concurrent run is rejected.
3. A reader can open **Why this is here · evidence** and record Interested, Maybe,
No thanks, or Hide. Feedback is private to the account. Dismissed/hidden
titles disappear at once; other feedback influences the next run. The user
can undo a choice by clicking its active button again.
The recommendation is an editorial lead, **not** an availability or edition
check. The public metadata link opens the provider record; an admin still uses
the existing source inspection and confirmation process to download anything.
The title/year/format matching is intentionally conservative but not a verified
identity merge. Missing year, cover, or status remains missing.
## Data and API
- `data/discovery.json`: atomic, reader-owned candidate and per-user feed
snapshots. A previous valid version is retained as `discovery.json.prev`.
A corrupt file makes reads/runs fail closed rather than overwrite it.
- `data/db.json`: consent in user settings and explicit feedback under
`discoveryFeedback`. The account-deletion route removes that feedback.
- `GET /api/discovery`: caller's own feed, settings, evidence, and feedback only.
- `POST /api/discovery/feedback`: caller's candidate choice; requires opt-in.
- `GET /api/admin/discovery` and `POST /api/admin/discovery/run`: admin-only
status and manual generation. A run keeps the last good feed on failure.
The job asks the private scraper state API for queued titles to avoid recommending
them. If that API is unavailable, the run still uses only cached sources; the
admin result records `queueAvailable: false` and queue exclusion is incomplete.
No scraper queue edit occurs.
## Read-only dry run and verification
From the project directory:
```powershell
node discovery/cli.js run --dry-run --data data --show
npm run test:reader
docker build -t mangacircus:discovery-test .
$env:APP_IMAGE='mangacircus:discovery-test'; npm run test:discovery-ui
```
The dry run reads `db.json`, a saved library index if present, Greatest, and
cached AniList metadata. It writes nothing and does not contact providers. The
Compose/browser suite uses a disposable data tree and queue, not the live one.
## Deliberate limits and next stage
This slice has no scheduled run, external metadata discovery, LLM, admin seed
form, edition-finding action, or explicit purge control for old feed snapshots.
Disabling consent stops feed display and future personalized generation, but
existing snapshot/feedback is retained until account deletion or a later data
management feature. Candidate quality is limited by the curated list and local
metadata cache. Add external public metadata feeds and optional LLM only behind
the existing bounded candidate/evidence contract after this deterministic path
has been used and reviewed.
+9 -3
View File
@@ -81,7 +81,13 @@ require administrator-supervised re-pairing; there is no automatic key rotation
direction needs its own approvals and grant. The server stores only a hash of a
grant it issued; the receiver stores its imported token server-side.
4. An admin browses the permitted catalog, selects missing/newest/last-five or an exact
chapter, and queues it through the normal scraper. The peer adapter downloads up
chapter, and queues it through the normal scraper. The queue carries a validated
stable work identity: AniList first, then MangaDex/WeebCentral IDs, then another
known hoster ID. If none exists, the owning server supplies an opaque origin ID
that follows the work across peers without exposing its folder name. Imports retain
that identity, so a round trip resolves to the original library title; `sync missing`
skips chapters already present and fetches only additions. Titles alone are never
used to merge works. The peer adapter downloads up
to the configured pages in flight, validates size and SHA-256, resumes a partial
page with HTTP Range, checkpoints between batches, and atomically publishes each
complete chapter into the ordinary library. Existing-chapter policy is applied
@@ -102,9 +108,9 @@ an admin can distinguish pairing from permission.
| --- | --- |
| `GET /peer/v1/hello` with one-use invitation headers | Protocol version and server identity fingerprint; no catalog |
| `GET /peer/v1/access` | Pairing state, certificate expiry and current grant status/scopes; no catalog |
| `GET /peer/v1/catalog?cursor=N` | Up to 50 approved opaque title IDs, safe metadata and cover availability |
| `GET /peer/v1/catalog?cursor=N` | Up to 50 approved opaque title IDs, safe metadata, a stable work identity and cover availability |
| `GET /peer/v1/covers/{mangaId}` | Approved cover image only; bounded to 5 MiB and charged to peer byte quota |
| `GET /peer/v1/manga/{id}?cursor=N` | Up to 100 complete chapter IDs and public metadata |
| `GET /peer/v1/manga/{id}?cursor=N` | Up to 100 complete chapter IDs, public metadata and the same stable work identity |
| `GET /peer/v1/manga/{id}/chapters/{id}` | Ordered page IDs, MIME, byte size, SHA-256 and ETag |
| `GET /peer/v1/blobs/{id}` | One image, optional `Range: bytes=N-` and `If-Match`; 206 on resume |
+11
View File
@@ -59,6 +59,17 @@ written atomically. Individual
archives retain their existing resumable checkpoints under
`library/.mcscraper-work-<source-id>/job.json`. `events.log` is append-only.
The same state file also holds a `sourceLedger` keyed by adapter and source-series
ID. It keeps source leads after completed queue rows are cleared. Existing queue
URLs and recognized URLs in library metadata are backfilled without contacting a
host; a successful catalog inspection records the selected chapter count, source
edition, coverage signal, and observation time. Errors remain visible beside the
last observation until a later successful inspection. **Catalog found**, **adapter
compatible**, and **download rights** are separate states: a URL alone verifies
none of them, and download rights remain unverified by scraper inspection. The
admin-only **Settings → Scraper → Sources** tab shows this log. It is evidence for
review, not an automatic source switch or a whole-series completeness claim.
On its first start, the daemon imports the old `.mcscraper-queue/queue.json` wishlist
when present. Completed chapter ids in each archive checkpoint remain authoritative,
so migration does not redownload published chapters.