Checkpoint peering identity and opt-in discovery
This commit is contained in:
1 parent
eba9b50e30
commit
47de081721
44 files changed
+1557
-55
No files matched your search
@@ -15,6 +15,8 @@ Related documents:
|
||||
- [resilience.md](resilience.md): supervision, recovery limits and incidents.
|
||||
- [compose-deployment.md](compose-deployment.md): local/VM deployment and release workflow.
|
||||
- [settings-navigation.md](settings-navigation.md): User/Admin workspace, scraper tabs and deep links.
|
||||
- [peering.md](peering.md): optional private server-to-server catalog and transfer.
|
||||
- [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
|
||||
|
||||
## 1. Principles
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# Discovery Scout (deterministic first release)
|
||||
|
||||
The reader now has a separate **Discover** page. This first slice deliberately uses
|
||||
the curated Greatest catalog, already-cached AniList identities/relations, library
|
||||
metadata, each user's Greatest votes, and explicit Discover feedback. It does not
|
||||
call an LLM, fetch new metadata, inspect chapter/image hosters, edit a collection,
|
||||
or enqueue downloads. The reader remains the only writer of account and discovery
|
||||
state; no fourth Compose service, credential, or database is required.
|
||||
|
||||
## Use
|
||||
|
||||
1. Open `discover.html`. Each account independently enables **Personalized
|
||||
discovery** (off by default). Reading activity is a separate opt-in. The
|
||||
mature-title switch only applies to candidates with a known mature tag; it is
|
||||
not a content-safety guarantee.
|
||||
2. An admin presses **Run discovery now** on the same page. Existing suggestions
|
||||
remain visible until the new feed is published. Runs are bounded to 50
|
||||
candidates and 20 suggestions per opted-in user, with no more than four from
|
||||
one period. A second concurrent run is rejected.
|
||||
3. A reader can open **Why this is here · evidence** and record Interested, Maybe,
|
||||
No thanks, or Hide. Feedback is private to the account. Dismissed/hidden
|
||||
titles disappear at once; other feedback influences the next run. The user
|
||||
can undo a choice by clicking its active button again.
|
||||
|
||||
The recommendation is an editorial lead, **not** an availability or edition
|
||||
check. The public metadata link opens the provider record; an admin still uses
|
||||
the existing source inspection and confirmation process to download anything.
|
||||
The title/year/format matching is intentionally conservative but not a verified
|
||||
identity merge. Missing year, cover, or status remains missing.
|
||||
|
||||
## Data and API
|
||||
|
||||
- `data/discovery.json`: atomic, reader-owned candidate and per-user feed
|
||||
snapshots. A previous valid version is retained as `discovery.json.prev`.
|
||||
A corrupt file makes reads/runs fail closed rather than overwrite it.
|
||||
- `data/db.json`: consent in user settings and explicit feedback under
|
||||
`discoveryFeedback`. The account-deletion route removes that feedback.
|
||||
- `GET /api/discovery`: caller's own feed, settings, evidence, and feedback only.
|
||||
- `POST /api/discovery/feedback`: caller's candidate choice; requires opt-in.
|
||||
- `GET /api/admin/discovery` and `POST /api/admin/discovery/run`: admin-only
|
||||
status and manual generation. A run keeps the last good feed on failure.
|
||||
|
||||
The job asks the private scraper state API for queued titles to avoid recommending
|
||||
them. If that API is unavailable, the run still uses only cached sources; the
|
||||
admin result records `queueAvailable: false` and queue exclusion is incomplete.
|
||||
No scraper queue edit occurs.
|
||||
|
||||
## Read-only dry run and verification
|
||||
|
||||
From the project directory:
|
||||
|
||||
```powershell
|
||||
node discovery/cli.js run --dry-run --data data --show
|
||||
npm run test:reader
|
||||
docker build -t mangacircus:discovery-test .
|
||||
$env:APP_IMAGE='mangacircus:discovery-test'; npm run test:discovery-ui
|
||||
```
|
||||
|
||||
The dry run reads `db.json`, a saved library index if present, Greatest, and
|
||||
cached AniList metadata. It writes nothing and does not contact providers. The
|
||||
Compose/browser suite uses a disposable data tree and queue, not the live one.
|
||||
|
||||
## Deliberate limits and next stage
|
||||
|
||||
This slice has no scheduled run, external metadata discovery, LLM, admin seed
|
||||
form, edition-finding action, or explicit purge control for old feed snapshots.
|
||||
Disabling consent stops feed display and future personalized generation, but
|
||||
existing snapshot/feedback is retained until account deletion or a later data
|
||||
management feature. Candidate quality is limited by the curated list and local
|
||||
metadata cache. Add external public metadata feeds and optional LLM only behind
|
||||
the existing bounded candidate/evidence contract after this deterministic path
|
||||
has been used and reviewed.
|
||||
+9
-3
@@ -81,7 +81,13 @@ require administrator-supervised re-pairing; there is no automatic key rotation
|
||||
direction needs its own approvals and grant. The server stores only a hash of a
|
||||
grant it issued; the receiver stores its imported token server-side.
|
||||
4. An admin browses the permitted catalog, selects missing/newest/last-five or an exact
|
||||
chapter, and queues it through the normal scraper. The peer adapter downloads up
|
||||
chapter, and queues it through the normal scraper. The queue carries a validated
|
||||
stable work identity: AniList first, then MangaDex/WeebCentral IDs, then another
|
||||
known hoster ID. If none exists, the owning server supplies an opaque origin ID
|
||||
that follows the work across peers without exposing its folder name. Imports retain
|
||||
that identity, so a round trip resolves to the original library title; `sync missing`
|
||||
skips chapters already present and fetches only additions. Titles alone are never
|
||||
used to merge works. The peer adapter downloads up
|
||||
to the configured pages in flight, validates size and SHA-256, resumes a partial
|
||||
page with HTTP Range, checkpoints between batches, and atomically publishes each
|
||||
complete chapter into the ordinary library. Existing-chapter policy is applied
|
||||
@@ -102,9 +108,9 @@ an admin can distinguish pairing from permission.
|
||||
| --- | --- |
|
||||
| `GET /peer/v1/hello` with one-use invitation headers | Protocol version and server identity fingerprint; no catalog |
|
||||
| `GET /peer/v1/access` | Pairing state, certificate expiry and current grant status/scopes; no catalog |
|
||||
| `GET /peer/v1/catalog?cursor=N` | Up to 50 approved opaque title IDs, safe metadata and cover availability |
|
||||
| `GET /peer/v1/catalog?cursor=N` | Up to 50 approved opaque title IDs, safe metadata, a stable work identity and cover availability |
|
||||
| `GET /peer/v1/covers/{mangaId}` | Approved cover image only; bounded to 5 MiB and charged to peer byte quota |
|
||||
| `GET /peer/v1/manga/{id}?cursor=N` | Up to 100 complete chapter IDs and public metadata |
|
||||
| `GET /peer/v1/manga/{id}?cursor=N` | Up to 100 complete chapter IDs, public metadata and the same stable work identity |
|
||||
| `GET /peer/v1/manga/{id}/chapters/{id}` | Ordered page IDs, MIME, byte size, SHA-256 and ETag |
|
||||
| `GET /peer/v1/blobs/{id}` | One image, optional `Range: bytes=N-` and `If-Match`; 206 on resume |
|
||||
|
||||
|
||||
@@ -59,6 +59,17 @@ written atomically. Individual
|
||||
archives retain their existing resumable checkpoints under
|
||||
`library/.mcscraper-work-<source-id>/job.json`. `events.log` is append-only.
|
||||
|
||||
The same state file also holds a `sourceLedger` keyed by adapter and source-series
|
||||
ID. It keeps source leads after completed queue rows are cleared. Existing queue
|
||||
URLs and recognized URLs in library metadata are backfilled without contacting a
|
||||
host; a successful catalog inspection records the selected chapter count, source
|
||||
edition, coverage signal, and observation time. Errors remain visible beside the
|
||||
last observation until a later successful inspection. **Catalog found**, **adapter
|
||||
compatible**, and **download rights** are separate states: a URL alone verifies
|
||||
none of them, and download rights remain unverified by scraper inspection. The
|
||||
admin-only **Settings → Scraper → Sources** tab shows this log. It is evidence for
|
||||
review, not an automatic source switch or a whole-series completeness claim.
|
||||
|
||||
On its first start, the daemon imports the old `.mcscraper-queue/queue.json` wishlist
|
||||
when present. Completed chapter ids in each archive checkpoint remain authoritative,
|
||||
so migration does not redownload published chapters.
|
||||
|
||||
Reference in new issue
Block a user