Compare commits

..
Author SHA1 Message Date
Codex fcc50d576c Record main reading-hours settings release 2026-10-14 22:44:39 +02:00
Codex ddd9c62c40 Show read-only server reading hours in settings 2026-10-14 22:42:54 +02:00
Codex de8e7ed988 Record main reading limiter cutover 2026-10-14 22:24:20 +02:00
Codex e944a87b42 Allow reader-only release selection in Compose 2026-10-14 22:22:56 +02:00
Codex 77553bbece Style reading quiz feedback by result 2026-10-14 21:48:31 +02:00
Codex f607994a5d Reset stale reading challenges after navigation 2026-10-14 15:36:43 +02:00
Codex d8eba9169f Add deployment reading windows and progress-aware quiz unlock 2026-10-14 15:34:55 +02:00
Codex edbc9c6056 Handle large WeebCentral catalogs and correct MangaPill checkpoint monitoring 2026-10-14 14:01:05 +02:00
Codex 0d51f214d5 Test paused-source confirmation through reader UI 2026-10-14 09:44:01 +02:00
Codex b96ab42add Allow queued confirmation during source pause 2026-10-14 09:42:21 +02:00
Codex a22913129a Preserve per-source pacing across queue jobs and add source pause 2026-10-14 09:23:04 +02:00
Codex a08e847932 Add first-party MangaPill source and distinct volume releases 2026-10-14 09:05:34 +02:00
Codex a8df57f165 Recognize additional WeebCentral chapter labels and record source pilot 2026-10-13 23:58:33 +02:00
Codex 6d713e0a8c Keep source plugins in single scraper process 2026-10-13 22:42:20 +02:00
Codex 0fc1a7b52a Mark sparse MangaDex hosted catalogs as partial 2026-10-13 22:01:57 +02:00
Codex 47de081721 Checkpoint peering identity and opt-in discovery 2026-10-13 22:00:54 +02:00
Codex eba9b50e30 Record source pilot boundaries and upstream decision 2026-10-13 21:58:58 +02:00
Codex 1c61686c9b Checkpoint opt-in MangaDex parser worker 2026-10-13 21:57:59 +02:00
Codex 93f07e838e Fix pinned lookup for dual-stack HTTPS requests 2026-10-13 21:53:59 +02:00
Codex cc2fd05062 Document and package opt-in source worker 2026-10-13 21:52:22 +02:00
Codex f4123b32e4 Checkpoint guarded source request broker 2026-10-13 21:51:03 +02:00
Codex 6599d0a244 Validate first-party parser boundaries 2026-10-13 21:47:18 +02:00
Codex c5a92282a9 Checkpoint first-party source parser pilot 2026-10-13 21:45:27 +02:00
Codex 605fa5b9d8 Add per-user page tone modes to reader 2026-10-12 23:49:02 +02:00
Codex 26d6cd59fc Repair metadata title on queue retry 2026-10-11 09:09:37 +02:00
Codex a500ea7c3f Use inspected manga title for metadata lookup 2026-10-11 09:07:42 +02:00
Codex da40d73f02 Redesign admin peering settings and catalogs 2026-10-11 01:06:03 +02:00
Codex 492477f86a Add scoped peer access and reversible serving controls 2026-10-11 00:25:14 +02:00
Codex 6c994113c6 Plan Discovery Scout recommendation architecture 2026-10-09 19:23:19 +02:00
Codex d0275223a7 Expose Greatest source links and safe scraper inspection 2026-10-09 14:53:52 +02:00
Codex f493afcc18 Explain why each Greatest manga earned its place 2026-10-09 11:26:57 +02:00
Codex bedd0bc3b3 Show Greatest inclusion reasons on catalog tiles 2026-10-09 10:25:18 +02:00
Codex f713d0c0a9 Complete Greatest covers and use Japanese navigation label 2026-10-09 10:22:01 +02:00
Codex 5df94db9a3 Add Greatest of All Time manga dashboard 2026-10-09 10:10:34 +02:00
Codex c8dd46d0fb Avoid redundant full scan after warm restart 2026-10-08 21:58:28 +02:00
Codex 8ac27e1f3d Cache library index and refresh changed manga 2026-10-08 21:53:08 +02:00
Maashkinoozhe eef4ffa07c Consolidate MangaCircus app, scraper, and Compose baseline 2026-10-07 23:28:20 +02:00
171 changed files with 18230 additions and 458 deletions

No files matched your search

+36 -5
View File
@@ -1,5 +1,36 @@
data
library
.claude
.git
node_modules
# Explicit allowlist; runtime state and credentials never enter image layers.
**
!package.json
!server.js
!library.js
!reading-direction.js
!chapter-identity.js
!catalog.js
!reading-analytics.js
!reading-access.js
!public/
!public/**
!greatest/
!greatest/**
!discovery/
!discovery/**
!runtime/
!runtime/**
!sentinel/
!sentinel/**
!scraper/
!scraper/**
!peering/
!peering/**
!tools/
!tools/**
scraper/.cache/**
scraper/logs/**
scraper/*.manifest.json
**/*.log
**/*.tmp
**/test/**
**/node_modules/**
**/.env*
**/*.pem
**/*.key
+21
View File
@@ -0,0 +1,21 @@
APP_IMAGE=mangacircus:local
# Optional reader-only release. Leave empty to use APP_IMAGE.
# READER_IMAGE=mangacircus:local
VCS_REF=development
READER_BIND=127.0.0.1
READER_PORT=8080
LIBRARY_PATH=./library
DATA_PATH=./data
SCRAPER_DATA_PATH=./data/scraper
SENTINEL_DATA_PATH=./data/sentinel
SCRAPER_TOKEN_PATH=./.secrets/scraper_token
TZ=Europe/Berlin
COOKIE_SECURE=0
# Optional server-enforced reading windows. Day keys: 0=Sunday through 6=Saturday.
# Ranges may cross midnight. Multiple ranges per day are supported.
# READING_WINDOWS_JSON='{"0":[["08:00","01:00"]],"1":[["07:00","24:00"]],"2":[["07:00","24:00"]],"3":[["07:00","24:00"]],"4":[["07:00","24:00"]],"5":[["07:00","24:00"]],"6":[["08:00","01:00"]]}'
READING_TIMEZONE=Europe/Berlin
READING_QUIZ_UNLOCK_MINUTES=60
# Optional deployment-managed JSON file under DATA_PATH, e.g. /data/reading-quiz.json.
# READING_QUIZ_CATALOG_PATH=/data/reading-quiz.json
PROXY_NETWORK=mangacircus-edge
+6 -4
View File
@@ -17,6 +17,9 @@ pnpm-debug.log*
.env
.env.*
!.env.example
!.env.*.example
.secrets/
/ops/proxmox/environment.json
*.pem
*.key
@@ -25,6 +28,9 @@ pnpm-debug.log*
*.log
.staging-*/
.old-*/
/scraper/.cache/
/scraper/*.manifest.json
/scraper/logs/
# --- Claude Code: keep the shared launch config, drop personal settings ---
.claude/settings.local.json
@@ -44,7 +50,3 @@ $RECYCLE.BIN/
*.swp
*.swo
*~
# --- personal agent coordination (synced by a private tool; never published) ---
CLAUDE.local.md
.claude/skills/manga-org/
-8
View File
@@ -1,8 +0,0 @@
# Manga Circus — instructions for agents
Self-hosted manga/comic/webtoon reader. Read `README.md` and `docs/architecture.md`, `docs/library-format.md`, `docs/scraper-guide.md` before changing anything.
## Rules
- **Principles (architecture §1):** zero npm dependencies, no framework, no build step, one process and one data file, the file system is the source of truth, the app never writes to the library.
- **UI:** build DOM only with `h()` (no `innerHTML`), relative URLs only, `mc.`-prefixed localStorage keys, `http(s)` links only.
- **Senpai integration** is on branch `senpai-integration` (`/api/senpai/*` proxy, "Who's here" side sheet), optional and invisible without `SENPAI_URL`. It must never show anything beyond the reader's horizon (spoiler rules live in Senpai, `docs/decisions.md`).
+14 -6
View File
@@ -1,12 +1,20 @@
FROM node:24-alpine
RUN apk add --no-cache tzdata openssl
WORKDIR /app
COPY package.json server.js library.js ./
ARG VCS_REF=development
LABEL org.opencontainers.image.title="MangaCircus" org.opencontainers.image.revision=$VCS_REF
COPY package.json server.js library.js reading-direction.js chapter-identity.js reading-analytics.js reading-access.js catalog.js ./
COPY public ./public
COPY greatest ./greatest
COPY discovery ./discovery
COPY tools ./tools
RUN mkdir -p /library /data && chown node:node /data
ENV PORT=8080 LIBRARY=/library DATA=/data
VOLUME ["/data"]
COPY scraper ./scraper
COPY peering ./peering
COPY sentinel ./sentinel
COPY runtime ./runtime
RUN mkdir -p /library /data && chown node:node /data /library
ENV PORT=8080 LIBRARY=/library DATA=/data TZ=Europe/Berlin
EXPOSE 8080
USER node
HEALTHCHECK --interval=1m --timeout=5s CMD wget -qO- http://127.0.0.1:8080/api/auth >/dev/null || exit 1
CMD ["node", "server.js"]
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s CMD node tools/healthcheck.js reader
CMD ["node", "runtime/launch.js", "reader"]
+58 -18
View File
@@ -4,8 +4,9 @@ A no-frills, self-hosted manga / comic / webtoon reader.
- **Zero dependencies.** A Node.js server (`server.js`, `library.js`) and plain HTML/CSS/JS pages. No build step, no framework, no database server.
- Reads the manga straight from a folder tree. The same folders stay readable in any normal picture viewer.
- Accounts with per-user reading progress, view settings and "new chapter" notifications.
- Accounts with per-user reading progress, view settings, an opt-in Active Reading Clock, a private Reading Story and "new chapter" notifications.
- Reader with three view modes (**Flow**, **Page**, **Book**), a page zoom that leaves the controls alone, and tap zones for mobile.
- A separate **Greatest of All Time** manga-history dashboard, with five-year periods, per-user wish ratings, community ranking, and local-library badges. Its curated titles do not enter the download queue.
## Run it
@@ -19,13 +20,44 @@ Open http://localhost:8080. The first account you create becomes the admin. Try
node tools/make-dummy.js library
```
To run the reader, managed scraper and Sentinel together, use `npm run start:all`. Admins can
then manage the persistent download queue, progress, pauses, retries, and rate limits
under **Settings → Admin → Scraper**. The reader continues working if the scraper is stopped.
**Settings → Admin → Collections & metadata** manages grouped tiles, reading order and relationships;
**Settings → Admin → Service health** shows recovery incidents. See [collections](docs/collections.md)
and [resilience](docs/resilience.md) for the ownership and agent-maintenance contracts.
The current Windows installation uses Docker Compose; its old `MangaCircus Supervisor`
Scheduled Task is disabled. Do not run a native duplicate writer. See
[docs/local-deployment.md](docs/local-deployment.md) for its observed layout, health
checks, restart procedure, and recovery point.
Deployment-controlled reading hours and the optional progress-aware quiz are
documented in [reading limiter](docs/reading-limiter.md). They are disabled
unless configured on the reader container.
See [Greatest of All Time](docs/greatest.md) for catalog ownership, voting, and metadata refresh.
### Docker
```bash
node tools/prepare-compose.js
docker compose config -q
docker compose up -d --build
```
`docker-compose.yml` mounts `./library` read-only at `/library` and keeps accounts/progress in the `mangacircus-data` volume (`/data`).
Stop existing native writers before using their data in containers. `docker-compose.yml`
runs reader, scraper and Sentinel with explicit persistent bind mounts (`library/`,
`data/`), file-backed secrets and only a loopback reader port. No named data volume is
used. See [Compose deployment](docs/compose-deployment.md) for local development, VM
recommendations, cutover, image releases and backup/rollback. The local Docker stack
is verified; VM deployment is still pending.
### Settings workspace
Settings has **User** and **Admin** menu groups. User sections cover Reading,
Reading time & privacy, Notifications and Account. Admin sections cover Scraper,
Server & library, Users & access, Collections & metadata and Service health.
Scraper has local Queue, Automatic updates and Hoster settings tabs. Only admins
see administration; personal settings do not poll scraper services. See
[settings navigation](docs/settings-navigation.md) for routes and verification.
### Settings (environment variables)
@@ -33,12 +65,15 @@ docker compose up -d --build
|-----------------|-------------|---------|
| `PORT` | `8080` | HTTP port |
| `LIBRARY` | `./library` | Library folder (read-only access is enough) |
| `DATA` | `./data` | Where `db.json` (users, progress, votes, notifications) and `scan-cache.json` live |
| `DATA` | `./data` | Where `db.json` (users, progress, votes, notifications) and disposable library-index caches live |
| `SIGNUP` | `closed` | `open` lets anyone create an account. Otherwise the admin adds users under Settings |
| `SCAN_INTERVAL` | `10` | Minutes between automatic library rescans (`0` = only at start and on "Rescan") |
| `SCAN_INTERVAL` | `10` | Minutes between shallow library change checks (`0` disables this timer) |
| `FULL_SCAN_INTERVAL` | `1440` | Minutes between full library reconciliations (`0` disables this timer) |
| `COOKIE_SECURE` | – | `1` marks the login cookie `Secure` (use it when you serve over HTTPS) |
| `SENPAI_URL` | – | Optional: base URL of a [Senpai](#whos-here-senpai-optional) story companion, e.g. `http://senpai-api:8780`. Unset = feature off |
| `SENPAI_TOKEN` | – | Service token for `SENPAI_URL` |
| `SCRAPER_URL` | `http://127.0.0.1:8091` | Private scraper daemon URL |
| `SCRAPER_TOKEN` | empty | Shared daemon token; required when it listens beyond loopback |
| `SCRAPER_TOKEN_FILE` | empty | Alternative secret file (used in Compose) |
| `SENTINEL_URL` | `http://127.0.0.1:8092` | Private observation service URL |
Serve it behind a reverse proxy with HTTPS if it is reachable from the internet. All pages use relative URLs, so a sub-path like `/manga/` works.
@@ -65,12 +100,22 @@ library/
## Documentation
Unnumbered MangaDex pilots/one-shots use stable `special-mangadex-<chapter-uuid>`
chapter keys and human labels, not fabricated numbers. Existing numeric IDs remain
unchanged. In Settings → Admin → Scraper, **Inspect & queue** shows the source edition
before **Confirm download**. “Job finished” refers to its selected scope, not verified
whole-series completeness. See the scraper service and library format contracts below.
| Document | For |
|----------|-----|
| [docs/library-format.md](docs/library-format.md) | The folder format: naming rules, page tags, metadata fields, versions, extras, what survives a rename |
| [docs/import.md](docs/import.md) | How the library is scanned and indexed, caching, change detection and notifications, troubleshooting |
| [docs/architecture.md](docs/architecture.md) | Components, data model, API, security, the reader's layout engine, deployment, design decisions |
| [docs/local-deployment.md](docs/local-deployment.md) | Current local Compose deployment, Tailscale access, persistent state, checks, restart, and recovery |
| [docs/settings-navigation.md](docs/settings-navigation.md) | User/Admin settings sections, scraper tabs, routes and UI checks |
| [docs/reading-insights-concept.md](docs/reading-insights-concept.md) | Active-time presentation, private summaries, deletion/export controls, and the future journal design |
| [docs/scraper-guide.md](docs/scraper-guide.md) | Rule book (agent-readable) for scrapers and other tools that write into the library |
| [scraper/AGENT_RUNBOOK.md](scraper/AGENT_RUNBOOK.md) | Conservative scraper workflow and host-specific procedures for automated agents |
---
@@ -83,7 +128,11 @@ library/
| **Book** | Two facing pages, in reading order (right page first for `rtl`). Covers and double pages stand alone. Pairing restarts after each of them, so one odd page never shifts the whole chapter. If pairs still look wrong, **⇄ Shift** moves the pairs of the current section by one page (remembered per chapter) |
- **Which mode opens:** Settings → *Open chapters in* = a fixed mode or *the mode I used last*. With *Use the fitting mode…* on, content that isn't made for your choice opens in the mode that fits (webtoons always open in Flow). That automatic switch never changes your remembered mode. Only clicking a mode in the reader does.
- **Page zoom:** 100 % means the page fits the screen (in Flow: a normal single page fits the screen; for webtoons, the strip is as wide as the screen but never wider than its own width). Pages may end up larger than the screen, exactly fitting, or smaller. Zoom with Ctrl + mouse wheel, touchpad pinch or two-finger pinch on the pages, or with `+` / `-` / `0`. This zooms only the pages; zooming over the toolbar zooms the whole page as usual. When a page is larger than the screen, drag it with the mouse (hand cursor) or pan with a finger. The zoom is remembered per mode for your account. Click the `100%` button to fit again.
- **Page zoom:** 100 % means the page fits the screen (in Flow: a normal single page fits the screen; for webtoons, the strip is as wide as the screen but never wider than its own width). Pages may end up larger than the screen, exactly fitting, or smaller. The toolbar −/+ buttons and keyboard `-`/`+` move in 5-percentage-point steps from 20 % through 600 %; `0` or the percentage button returns to 100 %. Ctrl + mouse wheel, touchpad pinch and two-finger pinch remain continuous for still finer control. This zooms only the pages; zooming over the toolbar zooms the whole page as usual. When a page is larger than the screen, drag it with the mouse (hand cursor) or pan with a finger. Flow locks horizontal movement while the content fits within the viewport, then enables it automatically when zoom makes the content wider. The zoom is remembered per mode for your account.
- **Flow page gap:** Settings → Reading lets each user keep the normal separator between Flow pages or remove it for continuous webtoon artwork and text that crosses image edges.
- **Page tone:** the reader's **Tone** button cycles Original → Inverted → Sepia → Dim → Extra dim in Flow, Page and Book. It changes only the artwork, never the controls or site theme. The choice is saved per user; it can also be selected in Settings → User → Reading.
- **Active Reading Clock:** opt in under Settings → User → Reading time & privacy. No scrolling-speed limit; a visit counts at 60% viewed-page coverage and saves on exit. Pauses in the background or after 30 seconds without reader activity. Tap the clock for this visit's time, coverage, qualification and loading wait.
- **Your Reading Story:** private active-time summaries appear on the library and manga pages when history exists. Chapter rows show the last timed read (60%-coverage visit), rereads remain visible as lifetime time, and Settings lets each user hide, export, or delete their own history. No leaderboards or speed ranking are used.
- **Tap / click zones:**
- Tapping **beside the pages** always shows or hides the controls.
- On a page, its left and right part turn the page, following the reading direction. For `rtl`, the left part goes forward.
@@ -96,17 +145,8 @@ library/
- **Loading:** the next pages (next 3 in Page mode, next 2 spreads in Book mode) are downloaded and decoded in advance, so page turns don't flash. A page that still isn't there shows a spinner. On phones, "Add to home screen" opens the app without browser bars, and the ⛶ button (where supported) goes fullscreen.
- **Progress** is saved per user and chapter version: the last page, whether the chapter is done, and book-mode shifts. The manga page offers *Continue*. Tap a chapter's status to mark it read/unread.
- **Notifications:** press *Notify me* on a manga. When a rescan finds new chapter numbers, they show up under the bell. Optionally, the server also POSTs a short text to a webhook URL of your choice (e.g. an ntfy.sh or Gotify topic, set in Settings).
## Who's here (Senpai, optional)
With `SENPAI_URL` set, the reader's top bar gets a **Who's here** button (key `x`). It opens a side sheet with
*This page* / *This chapter* / *So far*: the characters on the visible page(s) (hover or tap outlines them on the
page), the cast of the chapter and of the series with their prominence, and a character card with the names known so
far, first appearance and a timeline whose links jump to the page. Everything is limited to what you have read: the
server forwards `GET /api/senpai/v1/…` to Senpai with your reading horizon in this manga (the furthest page of the
furthest chapter you have opened; paging back to re-read does not lower it). Options in the sheet: *Only up to this
page* (for re-reading) and *Show unconfirmed* (machine findings). Without `SENPAI_URL` nothing changes.
Running both together: Senpai's `docker-compose.circus.yml` and `docs/runbook.md`.
- **Finding manga:** the library can search titles, alternative titles, and authors; filter by publication status; combine tags with AND matching; and sort by title, activity, or first release date. Tag links on a manga open the library with that tag selected.
- **Manga overview:** metadata is shown in a structured panel (with the complete raw `manga.json` available for inspection). Chapters are newest-first, with multi-select volume filters; selecting no volume means all volumes.
## Files
+115
View File
@@ -0,0 +1,115 @@
'use strict';
// Reader-owned catalog. Membership never changes filesystem or reading IDs.
const fs = require('fs'), path = require('path'), crypto = require('crypto');
const TYPES = ['prequel', 'sequel', 'side-story', 'spin-off', 'adaptation', 'alternate-version', 'alternate-timeline', 'compilation', 'remake', 'shared-universe', 'other'];
const ROLES = ['main', ...TYPES];
const STATES = ['suggested', 'confirmed', 'rejected', 'conflicted'];
class CatalogError extends Error { constructor(code, message) { super(message); this.code = code; } }
const fail = message => { throw new CatalogError(400, message); };
const text = (value, max = 200) => String(value || '').trim().slice(0, max);
function normalize(input, mangaIds, prior = {}) {
if (!input || !Array.isArray(input.collections) || !Array.isArray(input.relations)) fail('Expected collections and relations arrays');
if (input.collections.length > 2000 || input.relations.length > 10000) fail('Catalog is too large');
const seen = new Set(), membership = new Set();
const collections = input.collections.map(c => {
const id = text(c.id || crypto.randomUUID(), 80);
if (!/^[a-zA-Z0-9_-]+$/.test(id) || seen.has(id)) fail('Collection IDs must be unique letters, numbers, dash or underscore');
seen.add(id);
if (!text(c.title)) fail('Collection title is required');
if (!Array.isArray(c.members) || !c.members.length) fail('A collection needs members');
const local = new Set();
const members = c.members.map((member, index) => {
const mangaId = text(member.mangaId, 500);
// Keep absent members on subsequent edits: an temporarily unavailable mount must not erase them.
const wasMember = (prior.collections || []).some(old => old.id === id && old.members.some(m => m.mangaId === mangaId));
if (!mangaIds.has(mangaId) && !wasMember) fail(`Unknown manga: ${mangaId}`);
if (local.has(mangaId) || membership.has(mangaId)) fail(`Manga already belongs to a collection: ${mangaId}`);
local.add(mangaId); membership.add(mangaId);
const role = text(member.role || 'main');
if (!ROLES.includes(role)) fail('Unknown member role');
const readingOrder = Number(member.readingOrder ?? index + 1);
if (!Number.isFinite(readingOrder) || readingOrder < 0) fail('Reading order must be a nonnegative number');
return { mangaId, role, readingOrder };
}).sort((a, b) => a.readingOrder - b.readingOrder);
const coverMode = c.coverMode || 'selected';
if (!['selected', 'mosaic', 'custom'].includes(coverMode)) fail('Unknown cover mode');
const coverMangaId = text(c.coverMangaId || members[0].mangaId, 500);
if (!local.has(coverMangaId)) fail('Selected cover must belong to this collection');
const customCover = text(c.customCover, 100);
if (customCover && !/^[a-f0-9]{64}\.(jpg|png|webp|gif|avif|bmp)$/.test(customCover)) fail('Invalid custom cover');
return { id, title: text(c.title), description: text(c.description, 5000), members, coverMode, coverMangaId,
...(customCover ? { customCover } : {}), grouped: c.grouped !== false };
});
const relationKeys = new Set();
const relations = input.relations.map(r => {
const from = text(r.from, 500), to = text(r.to, 500), type = text(r.type), state = r.state || 'suggested';
if (from === to || !from || !to || !TYPES.includes(type) || !STATES.includes(state)) fail('Invalid relation');
for (const id of [from, to]) if (!mangaIds.has(id) && !(prior.relations || []).some(old => old.from === id || old.to === id)) fail(`Unknown manga: ${id}`);
const key = JSON.stringify([from, to, type]);
if (relationKeys.has(key)) fail('Duplicate relationship');
relationKeys.add(key);
const evidence = (Array.isArray(r.evidence) ? r.evidence : []).slice(0, 12).map(e => {
const url = text(e.url, 1000);
if (url && !/^https?:\/\/[^\s]+$/i.test(url)) fail('Evidence must be an HTTP URL');
return { provider: text(e.provider), url, note: text(e.note, 1500) };
});
return { from, to, type, state, evidence, explanation: text(r.explanation, 3000), method: text(r.method || 'manual'),
confidence: Math.max(0, Math.min(1, Number(r.confidence) || 0)) };
});
return { version: 1, collections, relations };
}
class Catalog {
constructor(dataRoot) {
this.file = path.join(dataRoot, 'catalog.json');
this.state = { version: 1, revision: 0, collections: [], relations: [], audit: [] };
if (fs.existsSync(this.file)) {
const saved = JSON.parse(fs.readFileSync(this.file, 'utf8'));
if (saved.version !== 1 || !Array.isArray(saved.collections) || !Array.isArray(saved.relations)) throw new Error('Unsupported catalog file');
this.state = saved;
}
}
update(input, mangaIds, actor) {
if (input.revision !== this.state.revision) throw new CatalogError(409, 'Catalog changed; reload before saving');
const value = normalize(input, mangaIds, this.state);
const next = { ...value, revision: this.state.revision + 1,
audit: [...(this.state.audit || []), { at: new Date().toISOString(), actor, revision: this.state.revision + 1 }].slice(-200) };
fs.writeFileSync(this.file + '.tmp', JSON.stringify(next, null, 2) + '\n');
fs.renameSync(this.file + '.tmp', this.file);
this.state = next;
return next;
}
suggest(mangas) {
const byAniList = new Map(mangas.filter(m => m.meta?.['x-anilist']?.id).map(m => [String(m.meta['x-anilist'].id), m]));
const relations = [...this.state.relations];
const mapping = { PREQUEL: 'prequel', SEQUEL: 'sequel', SIDE_STORY: 'side-story', SPIN_OFF: 'spin-off', ADAPTATION: 'adaptation', ALTERNATIVE: 'alternate-version', COMPILATION: 'compilation', OTHER: 'other', SOURCE: 'adaptation', PARENT: 'other' };
for (const m of mangas) for (const r of m.meta?.['x-anilist']?.relations || []) {
const target = byAniList.get(String(r.id)), type = mapping[r.type] || 'other';
if (!target || target.id === m.id || relations.some(e => e.from === m.id && e.to === target.id && e.type === type)) continue;
relations.push({ from: m.id, to: target.id, type, state: 'suggested', method: 'anilist', confidence: 1,
explanation: `AniList identifies the target as ${r.type}. Review before grouping.`,
evidence: [{ provider: 'AniList', url: m.meta['x-anilist'].url || `https://anilist.co/manga/${m.meta['x-anilist'].id}`, note: r.type }] });
}
return relations;
}
}
function collectionViews(catalog, mangas) {
const byId = new Map(mangas.map(m => [m.id, m]));
return catalog.collections.map(c => {
const members = c.members.filter(member => byId.has(member.mangaId)).map(member => ({ ...byId.get(member.mangaId), role: member.role, readingOrder: member.readingOrder }));
const chosen = byId.get(c.coverMangaId) || members[0];
const sum = key => members.reduce((n, m) => n + (+m[key] || 0), 0);
const dates = members.map(m => m.firstReleaseAt).filter(Boolean);
return { ...c, members, cover: c.coverMode === 'custom' && c.customCover ? `api/collection-cover?id=${encodeURIComponent(c.id)}&v=${c.customCover}` : chosen?.cover,
covers: members.map(m => m.cover).filter(Boolean).slice(0, 4), chapters: sum('chapters'), read: sum('read'), unread: sum('unread'),
tags: [...new Set(members.flatMap(m => m.tags || []))], authors: [...new Set(members.flatMap(m => m.authors || []))],
altTitles: members.flatMap(m => [m.title, ...(m.altTitles || [])]),
status: members.some(m => m.status === 'ongoing') ? 'ongoing' : members.every(m => m.status === 'completed') ? 'completed' : 'mixed',
updated: Math.max(0, ...members.map(m => +m.updated || 0)), lastRead: Math.max(0, ...members.map(m => +m.lastRead || 0)),
firstReleaseAt: dates.length ? Math.min(...dates) : 0, firstRelease: dates.length ? String(new Date(Math.min(...dates)).getUTCFullYear()) : '',
reading: { activeMs: members.reduce((n, m) => n + (m.reading?.activeMs || 0), 0), updatedAt: Math.max(0, ...members.map(m => m.reading?.updatedAt || 0)) } };
}).filter(c => c.members.length);
}
module.exports = { Catalog, CatalogError, collectionViews, normalize, TYPES, ROLES };
+38
View File
@@ -0,0 +1,38 @@
'use strict';
// An unnumbered release is not chapter zero. Keep its upstream identity intact.
const UUID = '[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}';
const SPECIAL = new RegExp(`^special-mangadex-(${UUID})$`, 'i');
const VOLUME = /^special-weebcentral-volume-([a-z0-9]{26})$/i;
const SPECIAL_FOLDER = new RegExp(`^(special-mangadex-${UUID}|special-weebcentral-volume-[a-z0-9]{26})(?: (.*))?$`, 'i');
const NUMBERED_FOLDER = /^([a-z]*) ?(\d+(?:\.\d+)?)(?=$|[\s_\-[(])[\s_-]*(.*)$/i;
const isSpecialKey = value => SPECIAL.test(String(value)) || VOLUME.test(String(value));
function specialKey(id) {
const key = `special-mangadex-${String(id).toLowerCase()}`;
if (!isSpecialKey(key)) throw new Error('Unnumbered MangaDex release needs a stable chapter UUID');
return key;
}
function volumeKey(id) {
const key = `special-weebcentral-volume-${String(id).toLowerCase()}`;
if (!VOLUME.test(key)) throw new Error('WeebCentral volume needs a stable 26-character release ID');
return key;
}
function parseSpecialFolder(name) {
const match = SPECIAL_FOLDER.exec(name);
if (!match) return null;
let title = (match[2] || '').trim(), src = '';
const source = /\s*\[([^\]]*)\]$/.exec(title);
if (source) { src = source[1].trim(); title = title.slice(0, source.index).trim(); }
return { key: match[1].toLowerCase(), special: true, releaseKind: VOLUME.test(match[1]) ? 'volume' : 'special', x: false, num: null, title, src };
}
// Reader and scraper must assign the same identity to every numbered folder.
function parseKey(name) {
const match = NUMBERED_FOLDER.exec(name);
if (!match) return null;
const x = match[1].toLowerCase() === 'x', num = parseFloat(match[2]);
let title = match[3].trim(), src = '';
const source = /\s*\[([^\]]*)\]$/.exec(title);
if (source) { src = source[1].trim(); title = title.slice(0, source.index).trim(); }
return { key: (x ? 'x' : '') + num, x, num, title: title.replace(/^[-:.]\s*/, ''), src };
}
module.exports = { isSpecialKey, specialKey, volumeKey, parseSpecialFolder, parseKey };
+16
View File
@@ -0,0 +1,16 @@
# Controlled service reloads preserve checkpoints; public assets update immediately.
services:
reader:
volumes:
- ./public:/app/public:ro
- ./server.js:/app/server.js:ro
- ./catalog.js:/app/catalog.js:ro
- ./reading-analytics.js:/app/reading-analytics.js:ro
- ./reading-access.js:/app/reading-access.js:ro
- ./library.js:/app/library.js:ro
scraper:
volumes:
- ./scraper:/app/scraper:ro
sentinel:
volumes:
- ./sentinel:/app/sentinel:ro
+7
View File
@@ -0,0 +1,7 @@
services:
reader:
ports:
- target: 8080
published: "${READER_PORT:-8080}"
host_ip: "${READER_LAN_BIND:?Set READER_LAN_BIND to the host LAN IPv4 address}"
protocol: tcp
+12
View File
@@ -0,0 +1,12 @@
# Compose >= 2.24.4. Infrastructure owns the edge network.
services:
reader:
ports: !reset []
environment:
COOKIE_SECURE: "1"
networks: [backend, egress, edge]
networks:
edge:
external: true
name: ${PROXY_NETWORK:-mangacircus-edge}
+14
View File
@@ -0,0 +1,14 @@
{
"version": 1,
"manga": {
"Exact library manga folder name": [
{
"id": "unique-question-id",
"prompt": "Which character is revealed to be the captain in chapter 12?",
"answers": ["Full Character Name", "Commonly Used Name"],
"requires": ["1/12"],
"evidence": "Reviewed source URL or chapter citation; never sent to the browser"
}
]
}
}
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env node
'use strict';
// Read-only CLI. Never writes the reader's discovery.json or scraper queue.
const fs = require('node:fs');
const path = require('node:path');
const { loadCatalog } = require('../greatest');
const { generate } = require('./engine');
if (!process.argv.includes('--dry-run')) {
console.error('Usage: node discovery/cli.js run --dry-run [--data DIR] [--catalog FILE] [--show]');
process.exit(2);
}
const arg = (name, fallback) => { const i = process.argv.indexOf(name); return i < 0 ? fallback : process.argv[i + 1]; };
const data = path.resolve(arg('--data', process.env.DATA || 'data'));
const db = JSON.parse(fs.readFileSync(path.join(data, 'db.json'), 'utf8'));
let library = [];
try { library = JSON.parse(fs.readFileSync(path.join(data, 'library-index.json'), 'utf8')).mangas || []; }
catch (error) { if (error.code !== 'ENOENT') throw error; }
const greatest = loadCatalog(arg('--catalog', path.join(__dirname, '..', 'greatest', 'catalog.json')));
const result = generate({ db, library, greatest, cacheDir: path.join(data, 'scraper', 'metadata', 'anilist') });
const summary = { examined: result.examined, candidates: Object.keys(result.candidates).length,
optedInUsers: Object.values(db.users || {}).filter(u => u.settings?.discoveryEnabled).length,
feedSizes: Object.values(result.feeds).filter(f => !f.disabled).map(f => f.items.length) };
if (process.argv.includes('--show')) summary.titles = Object.values(result.feeds).find(f => !f.disabled)?.items
.map(item => result.candidates[item.candidateId]?.title) || [];
console.log(JSON.stringify(summary, null, 2));
+174
View File
@@ -0,0 +1,174 @@
'use strict';
const fs = require('node:fs');
const path = require('node:path');
const { normalize } = require('../greatest');
const MAX_CANDIDATES = 50;
const MAX_FEED = 20;
const VALID_FORMATS = new Set(['MANGA', 'ONE_SHOT']);
const safeUrl = value => /^https:\/\/[^\s]+$/i.test(String(value || '')) ? String(value) : null;
const unique = values => [...new Set(values.filter(Boolean))];
function cachedMetadata(dir, id) {
if (!Number.isSafeInteger(Number(id)) || Number(id) < 1) return null;
try {
const value = JSON.parse(fs.readFileSync(path.join(dir, `id-${Number(id)}.json`), 'utf8'));
return value._cacheVersion === 4 && value.metadata?.['x-anilist']?.id === Number(id) ? value.metadata : null;
} catch { return null; }
}
function candidateFromGreatest(entry, cacheDir) {
const id = Number(entry.metadata?.id);
const cached = id ? cachedMetadata(cacheDir, id) : null;
return {
id: id ? `anilist:${id}` : `greatest:${entry.id}`,
title: cached?.title || entry.title, year: cached?.year || entry.year || null,
type: cached?.['x-anilist']?.format || 'MANGA', status: cached?.status || entry.status || 'unknown',
cover: safeUrl(cached?.['x-anilist']?.coverUrl) || safeUrl(entry.cover),
metadataUrl: safeUrl(cached?.['x-anilist']?.url) || safeUrl(entry.metadata?.url) || null,
tags: unique(cached?.tags || []), authors: unique(cached?.authors || []),
period: entry.period, editorialRank: entry.rank, greatestId: entry.id,
evidence: [{ id: `greatest:${entry.id}`, kind: 'curation', label: 'Greatest of All Time selection',
note: String(entry.note || '').slice(0, 600), url: safeUrl(entry.source?.url) || safeUrl(entry.metadata?.url) }],
relations: cached?.['x-anilist']?.relations || [],
};
}
function addRelations(map, anchor, cacheDir) {
for (const relation of anchor.relations || []) {
if (!VALID_FORMATS.has(relation.format) || !Number.isSafeInteger(Number(relation.id))) continue;
const id = `anilist:${Number(relation.id)}`;
if (map.has(id)) continue;
const cached = cachedMetadata(cacheDir, relation.id);
map.set(id, {
id, title: cached?.title || relation.title || `AniList ${relation.id}`,
year: cached?.year || null, type: relation.format, status: cached?.status || 'unknown',
cover: safeUrl(cached?.['x-anilist']?.coverUrl),
metadataUrl: safeUrl(cached?.['x-anilist']?.url) || safeUrl(relation.url) || `https://anilist.co/manga/${relation.id}`,
tags: unique(cached?.tags || []), authors: unique(cached?.authors || []), period: null,
editorialRank: null, greatestId: null,
evidence: [{ id: `relation:${anchor.id}:${id}`, kind: 'relation', label: `${relation.type || 'RELATED'} to ${anchor.title}`,
note: '', url: safeUrl(relation.url) || `https://anilist.co/manga/${relation.id}` }],
relatedTo: { id: anchor.id, title: anchor.title, type: relation.type || 'RELATED' },
relations: cached?.['x-anilist']?.relations || [],
});
}
}
function buildCandidates({ greatest, library, cacheDir }) {
const map = new Map();
for (const entry of greatest.entries) {
const candidate = candidateFromGreatest(entry, cacheDir);
if (candidate.type === 'MANGA' || candidate.type === 'ONE_SHOT') map.set(candidate.id, candidate);
}
for (const manga of library) {
const identity = manga.meta?.['x-anilist'];
if (!identity?.id) continue;
addRelations(map, { id: `anilist:${identity.id}`, title: manga.title, relations: identity.relations }, cacheDir);
}
for (const candidate of [...map.values()]) if (candidate.greatestId) addRelations(map, candidate, cacheDir);
return [...map.values()];
}
function ownedKeySet(library) {
const ids = new Set(), names = new Set();
for (const manga of library) {
const id = Number(manga.meta?.['x-anilist']?.id);
if (id) ids.add(`anilist:${id}`);
names.add(`${normalize(manga.title)}:${Number(manga.meta?.year) || '?'}`);
}
return { ids, names };
}
function labelForVote(greatest, id) { return greatest.entries.find(entry => entry.id === id)?.title || null; }
function feedForUser({ user, db, candidates, greatest, library, queuedTitles = [] }) {
const settings = user.settings || {};
if (!settings.discoveryEnabled) return { generatedAt: new Date().toISOString(), items: [], disabled: true };
const votes = db.greatestRatings?.[user.name] || {};
const feedback = db.discoveryFeedback?.[user.name] || {};
const owned = ownedKeySet(library);
const queued = new Set(queuedTitles.map(normalize));
const positive = Object.entries(votes).filter(([, vote]) => vote === 'flame' || vote === 'good')
.map(([id, vote]) => ({ id, vote, title: labelForVote(greatest, id), entry: greatest.entries.find(e => e.id === id) })).filter(x => x.entry);
const likedTags = new Map(), likedPeriods = new Map();
for (const anchor of positive) {
likedPeriods.set(anchor.entry.period, anchor);
const candidate = candidates.find(c => c.greatestId === anchor.id);
for (const tag of candidate?.tags || []) likedTags.set(tag.toLowerCase(), anchor);
}
const ownedTags = new Map();
for (const manga of library) for (const tag of manga.meta?.tags || []) if (typeof tag === 'string') ownedTags.set(tag.toLowerCase(), manga.title);
const readTags = new Map();
if (settings.discoveryReadingActivity) for (const manga of library) {
const progress = db.progress?.[user.name]?.[manga.id] || {};
if (!Object.values(progress).some(chapter => chapter?.done)) continue;
for (const tag of manga.meta?.tags || []) if (typeof tag === 'string') readTags.set(tag.toLowerCase(), manga.title);
}
const rows = [];
for (const candidate of candidates) {
if (candidate.greatestId && votes[candidate.greatestId]) continue;
if (['dismissed', 'hidden'].includes(feedback[candidate.id])) continue;
if (owned.ids.has(candidate.id) || owned.names.has(`${normalize(candidate.title)}:${candidate.year || '?'}`)) continue;
if (queued.has(normalize(candidate.title))) continue;
if (!settings.discoveryMature && candidate.tags.some(tag => /adult|hentai|erotica/i.test(tag))) continue;
const matchingLikedTag = candidate.tags.find(tag => likedTags.has(tag.toLowerCase()));
const matchingOwnedTag = candidate.tags.find(tag => ownedTags.has(tag.toLowerCase()));
const matchingReadTag = candidate.tags.find(tag => readTags.has(tag.toLowerCase()));
const era = likedPeriods.get(candidate.period);
const relation = candidate.relatedTo;
const scoreParts = { editorial: candidate.greatestId ? Math.max(0, 6 - Math.min(6, Number(candidate.editorialRank) || 6)) : 0,
relation: relation ? 3 : 0, sharedTaste: matchingLikedTag ? 4 : 0, sharedLibrary: matchingOwnedTag ? 1 : 0,
era: era ? 2 : 0, reading: matchingReadTag ? 2 : 0,
interest: feedback[candidate.id] === 'interested' ? 5 : feedback[candidate.id] === 'maybe' ? 2 : 0 };
let reason, personalEvidence = null;
if (relation) reason = `${candidate.title} is listed as ${relation.type.toLowerCase().replaceAll('_', ' ')} to ${relation.title}. Check the edition and reading order first.`;
else if (matchingLikedTag) {
const anchor = likedTags.get(matchingLikedTag.toLowerCase());
reason = `You liked ${anchor.title}; this also carries the ${matchingLikedTag} tag.`;
personalEvidence = { id: `taste:greatest:${anchor.id}`, kind: 'personal', label: `Your Greatest vote for ${anchor.title}`, note: `Shared tag: ${matchingLikedTag}`, url: null };
} else if (era) {
reason = `You liked ${era.title}; this is another curated work from ${candidate.period}.`;
personalEvidence = { id: `taste:greatest:${era.id}`, kind: 'personal', label: `Your Greatest vote for ${era.title}`, note: `Shared period: ${candidate.period}`, url: null };
} else if (matchingReadTag) {
const title = readTags.get(matchingReadTag.toLowerCase());
reason = `You have read ${title}; this also carries the ${matchingReadTag} tag.`;
personalEvidence = { id: `taste:read:${normalize(title)}`, kind: 'personal', label: `Your completed reading in ${title}`, note: `Shared tag: ${matchingReadTag}`, url: null };
} else if (matchingOwnedTag) {
const title = ownedTags.get(matchingOwnedTag.toLowerCase());
reason = `This shares the ${matchingOwnedTag} tag with ${title} in your library.`;
personalEvidence = { id: `taste:library:${normalize(title)}`, kind: 'personal', label: `${title} in your library`, note: `Shared tag: ${matchingOwnedTag}`, url: null };
}
else reason = candidate.evidence[0]?.note || 'A curated manga worth a closer look.';
reason = reason.slice(0, 280);
const evidence = [...candidate.evidence, ...(personalEvidence ? [personalEvidence] : [])];
rows.push({ candidateId: candidate.id, reason, evidence, evidenceIds: evidence.map(e => e.id), scoreParts,
score: Object.values(scoreParts).reduce((a, b) => a + b, 0), period: candidate.period || 'Related works' });
}
rows.sort((a, b) => b.score - a.score || a.candidateId.localeCompare(b.candidateId));
const picked = [], periodCounts = new Map();
for (const row of rows) {
if (picked.length >= MAX_FEED) break;
if ((periodCounts.get(row.period) || 0) >= 4) continue;
picked.push(row); periodCounts.set(row.period, (periodCounts.get(row.period) || 0) + 1);
}
return { generatedAt: new Date().toISOString(), items: picked.map((row, index) => ({ ...row, rank: index + 1 })) };
}
function generate({ db, library, greatest, cacheDir, queuedTitles = [] }) {
const all = buildCandidates({ greatest, library, cacheDir });
// Sample across eras before the work cap; insertion order in the curated catalog
// must not make recent manga invisible.
const buckets = new Map();
for (const candidate of all) {
const key = candidate.period || 'Related works';
if (!buckets.has(key)) buckets.set(key, []);
buckets.get(key).push(candidate);
}
const candidates = [];
while (candidates.length < MAX_CANDIDATES && [...buckets.values()].some(values => values.length)) {
for (const values of buckets.values()) {
if (values.length && candidates.length < MAX_CANDIDATES) candidates.push(values.shift());
}
}
const feeds = {};
for (const user of Object.values(db.users || {})) feeds[user.name] = feedForUser({ user, db, candidates, greatest, library, queuedTitles });
return { candidates: Object.fromEntries(candidates.map(c => [c.id, c])), feeds,
examined: all.length, clipped: Math.max(0, all.length - candidates.length) };
}
module.exports = { generate, buildCandidates, feedForUser, cachedMetadata, MAX_CANDIDATES, MAX_FEED };
+35
View File
@@ -0,0 +1,35 @@
'use strict';
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');
class DiscoveryStore {
constructor(dataDir) { this.file = path.join(path.resolve(dataDir), 'discovery.json'); }
load() {
try {
const data = JSON.parse(fs.readFileSync(this.file, 'utf8'));
if (data.schema !== 1 || !Number.isSafeInteger(data.revision) || !data.candidates || !data.feeds || !data.lastRun)
throw new Error('Invalid discovery store schema');
return data;
} catch (error) {
if (error.code === 'ENOENT') return { schema: 1, revision: 0, candidates: {}, feeds: {}, lastRun: { status: 'never' } };
throw new Error(`Discovery store unavailable; refusing to overwrite it: ${error.message}`);
}
}
publish(value) {
const prior = this.load();
const data = { schema: 1, revision: prior.revision + 1, candidates: value.candidates, feeds: value.feeds,
lastRun: { status: 'success', startedAt: value.startedAt, finishedAt: new Date().toISOString(),
candidateCount: Object.keys(value.candidates).length, examined: value.examined, clipped: value.clipped } };
fs.mkdirSync(path.dirname(this.file), { recursive: true });
const temp = `${this.file}.${crypto.randomBytes(6).toString('hex')}.tmp`;
try {
fs.writeFileSync(temp, JSON.stringify(data, null, 2) + '\n', { mode: 0o600 });
if (fs.existsSync(this.file)) fs.copyFileSync(this.file, `${this.file}.prev`);
fs.renameSync(temp, this.file);
} finally { try { fs.rmSync(temp, { force: true }); } catch {} }
return data;
}
}
module.exports = { DiscoveryStore };
+160 -14
View File
@@ -1,15 +1,161 @@
services:
mangacircus:
build: .
ports:
- "8080:8080"
volumes:
- ./library:/library:ro # your manga folders (read-only is fine)
- mangacircus-data:/data # users, progress, votes, scan cache
environment:
SIGNUP: closed # "open" lets anyone create an account
SCAN_INTERVAL: 10 # minutes between automatic rescans (0 = off)
restart: unless-stopped
name: mangacircus
volumes:
mangacircus-data:
x-app: &app
image: ${APP_IMAGE:-mangacircus:local}
build:
context: .
args:
VCS_REF: ${VCS_REF:-development}
init: true
restart: unless-stopped
stop_grace_period: 90s
read_only: true
tmpfs: [/tmp]
security_opt: [no-new-privileges:true]
cap_drop: [ALL]
secrets: [scraper_token]
logging:
driver: local
options:
max-size: "10m"
max-file: "3"
services:
reader:
<<: *app
image: ${READER_IMAGE:-${APP_IMAGE:-mangacircus:local}}
command: [node, runtime/launch.js, reader]
ports: ["${READER_BIND:-127.0.0.1}:${READER_PORT:-8080}:8080"]
volumes:
- type: bind
source: ${LIBRARY_PATH:-./library}
target: /library
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${DATA_PATH:-./data}
target: /data
bind: {create_host_path: false}
environment:
PORT: "8080"
DATA: /data
LIBRARY: /library
TZ: ${TZ:-Europe/Berlin}
SIGNUP: closed
SCAN_INTERVAL: "10"
COOKIE_SECURE: ${COOKIE_SECURE:-0}
READING_WINDOWS_JSON: ${READING_WINDOWS_JSON:-}
READING_TIMEZONE: ${READING_TIMEZONE:-Europe/Berlin}
READING_QUIZ_UNLOCK_MINUTES: ${READING_QUIZ_UNLOCK_MINUTES:-60}
READING_QUIZ_CATALOG_PATH: ${READING_QUIZ_CATALOG_PATH:-}
SCRAPER_URL: http://scraper:8091
SENTINEL_URL: http://sentinel:8092
SCRAPER_TOKEN_FILE: /run/secrets/scraper_token
networks: [backend, egress]
healthcheck:
test: [CMD, node, tools/healthcheck.js, reader]
interval: 30s
timeout: 5s
start_period: 60s
retries: 3
scraper:
<<: *app
command: [node, runtime/launch.js, scraper]
volumes:
- type: bind
source: ${LIBRARY_PATH:-./library}
target: /library
bind: {create_host_path: false}
- type: bind
source: ${SCRAPER_DATA_PATH:-./data/scraper}
target: /data/scraper
bind: {create_host_path: false}
- type: bind
source: ${PEER_DATA_PATH:-./data/peering}
target: /data/peering
read_only: true
environment:
DATA: /data
LIBRARY: /library
SCRAPER_DATA: /data/scraper
METADATA_CACHE: /data/scraper/metadata/anilist
TZ: ${TZ:-Europe/Berlin}
SCRAPER_HOST: 0.0.0.0
SCRAPER_PORT: "8091"
SCRAPER_TOKEN_FILE: /run/secrets/scraper_token
PEER_DATA: /data/peering
networks: [backend, egress]
healthcheck:
test: [CMD, node, tools/healthcheck.js, scraper]
interval: 30s
timeout: 5s
start_period: 60s
retries: 3
sentinel:
<<: *app
command: [node, runtime/launch.js, sentinel]
volumes:
- type: bind
source: ${SENTINEL_DATA_PATH:-./data/sentinel}
target: /data/sentinel
bind: {create_host_path: false}
environment:
DATA: /data
SENTINEL_DATA: /data/sentinel
SENTINEL_HOST: 0.0.0.0
SENTINEL_PORT: "8092"
READER_URL: http://reader:8080
SCRAPER_URL: http://scraper:8091
SCRAPER_TOKEN_FILE: /run/secrets/scraper_token
TZ: ${TZ:-Europe/Berlin}
networks: [backend]
healthcheck:
test: [CMD, node, tools/healthcheck.js, sentinel]
interval: 30s
timeout: 5s
start_period: 60s
retries: 3
peer-gateway:
<<: *app
profiles: [peering]
command: [node, peering/gateway.js]
ports: ["${PEER_BIND:-127.0.0.1}:${PEER_PORT:-8093}:8093"]
volumes:
- type: bind
source: ${LIBRARY_PATH:-./library}
target: /library
read_only: true
bind: {create_host_path: false}
- type: bind
source: ${PEER_DATA_PATH:-./data/peering}
target: /data/peering
read_only: true
- type: bind
source: ${PEER_USAGE_PATH:-./data/peering-usage}
target: /data/peering-usage
environment:
LIBRARY: /library
PEER_DATA: /data/peering
PEER_USAGE_FILE: /data/peering-usage/usage.json
PEER_HOST: 0.0.0.0
PEER_PORT: "8093"
# Docker Desktop does not publish host ports from an internal-only network.
networks: [backend, egress]
healthcheck:
test: [CMD, node, tools/healthcheck-peer.js]
interval: 30s
timeout: 5s
start_period: 30s
retries: 3
networks:
backend:
internal: true
egress: {}
secrets:
scraper_token:
file: ${SCRAPER_TOKEN_PATH:-./.secrets/scraper_token}
+94 -22
View File
@@ -1,11 +1,23 @@
# MangaCircus architecture
MangaCircus is a self-hosted reader for manga, comics and webtoons. One Node.js process serves a handful of static pages and a small JSON API. The content is a folder tree on disk, and everything user-related lives in one JSON file.
MangaCircus is a self-hosted reader for manga, comics and webtoons. The reader owns
accounts/progress and a separate collection catalog. The scraper owns the persistent
queue and chapter publication. Sentinel observes health/progress and requests bounded
checkpoint recovery. Separate supervisors isolate service failures. All content remains
a plain folder tree; there is no database server or embedded autonomous LLM.
Related documents:
- [library-format.md](library-format.md): the folder format.
- [import.md](import.md): how the folder tree is indexed.
- [scraper-guide.md](scraper-guide.md): rules for tools that write the library.
- [scraper-service.md](scraper-service.md): managed queue, private API, and adapter contract.
- [collections.md](collections.md): relations, groups, curator UI and agent research.
- [resilience.md](resilience.md): supervision, recovery limits and incidents.
- [compose-deployment.md](compose-deployment.md): local/VM deployment and release workflow.
- [settings-navigation.md](settings-navigation.md): User/Admin workspace, scraper tabs and deep links.
- [peering.md](peering.md): optional private server-to-server catalog and transfer.
- [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
- [source-modules.md](source-modules.md): adding and maintaining first-party hosters inside the single scraper process.
## 1. Principles
@@ -13,7 +25,7 @@ Related documents:
|-----------|-------------|
| **The file system is the source of truth for content** | No import step, no copies, no content database. The library can be mounted read-only. Anything that writes files can feed it |
| **As dumb as possible** | No dependencies (`package.json` has none), no framework, no build step, no transpiler. Plain ES2019+ JavaScript in the browser, plain Node on the server |
| **One process, one data file** | In-memory library index, user data in `data/db.json`. Back up one file, run one container |
| **Small independent services** | The reader stays dependency-free; the optional scraper daemon can stop or restart without taking reading offline |
| **Readable without the app** | The folder format works in any picture viewer. The app adds progress, layout and convenience, not lock-in |
## 2. System context
@@ -35,6 +47,22 @@ Related documents:
## 3. Components
Runtime topology (2026-10-05): native `tools/start-all.js` launches reader, scraper and
Sentinel independently. In Compose, each container runs `runtime/launch.js <service>`
and exactly one service child. A failure does not stop unrelated services. Exactly one
reader and queue-owning scraper may write each shared state tree; do not scale replicas.
The catalog and relation store is reader-owned `data/catalog.json`. Sentinel writes only
`data/sentinel/`; it has no Docker socket or arbitrary code-repair authority.
The admin browser never talks to the scraper daemon directly. Admin-only reader
routes proxy commands to the daemon's private API, so its shared token stays on the
server. The daemon publishes complete folders atomically and increments a durable
revision; the reader polls that revision and refreshes changed manga after publication.
The reader restores a disposable full-index snapshot at startup so browsing need not
wait for a filesystem walk. Shallow directory stamps identify changed manga; a
daily full reconciliation catches in-place manual edits. The last full-scan time
survives restarts, so restarting does not itself force another complete walk.
| Path | Lines | Responsibility |
|------|------:|----------------|
| `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks |
@@ -43,7 +71,7 @@ Related documents:
| `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) |
| `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle |
| `public/read.html` + `reader.js` | ~470 | The reader (§7.2) |
| `public/settings.html` + `settings.js` | ~115 | Reading preferences, webhook, password, admin (rescan, users) |
| `public/settings.html` + `settings.js` + `settings-nav.js` | — | User/Admin navigation, personal preferences, lazy admin sections and scraper tabs |
| `public/login.html` + `login.js` | ~50 | Sign-in, first-run admin creation, optional sign-up |
| `public/app.css` | ~150 | All styles (dark theme, responsive) |
| `tools/png.js`, `tools/make-dummy.js` | ~310 | Demo library generator: a tiny PNG encoder with a bitmap font, no image libraries |
@@ -51,14 +79,16 @@ Related documents:
## 4. Runtime model
- **One process, event loop only.** All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests.
- **Reader process:** one event loop. All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests.
- **Optional scraper process:** one queue worker, one remote request at a time, durable checkpoints, and a private control API.
- **In-memory state:**
- `lib` is the current library index, replaced as a whole after each scan.
- `db` is all user data, loaded from `db.json` at start.
- **Timers:**
- a library scan at start and every `SCAN_INTERVAL` minutes,
- a saved-index restore and shallow check at start (full scan when no snapshot),
- a shallow check every `SCAN_INTERVAL` minutes, full reconciliation every `FULL_SCAN_INTERVAL` minutes,
- a debounced `db.json` write, 500 ms after the last change.
- **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json` synchronously, then exit. That matters for `docker stop`.
- **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json`; the daemon cooperatively stops at a page boundary and keeps its archive checkpoint.
- **Durability:** every file write goes to `*.tmp` first and is then renamed over the target, so a file is never half-written. A hard kill loses at most the changes of the last ~0.5 s. A corrupt `db.json` makes the server **refuse to start** rather than start empty and overwrite it.
## 5. Data model
@@ -93,6 +123,9 @@ This keeps ids human-readable and derivable without a database. The price: renam
"lastMode": "book", // the user's intended mode; changed only by clicking a mode
"adapt": true, // use a fitting mode for incompatible content (webtoon → flow)
"skipExtras": false, "skipCovers": false,
"flowPageGap": true, // show the 8 px separator between pages in Flow
"readingClock": false, // opt in to credible per-chapter active-time tracking
"readingStory": true, // show private retrospective summaries when history exists
"zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 }, // page zoom per mode
"webhook": ""
}
@@ -104,8 +137,13 @@ This keeps ids human-readable and derivable without a database. The price: renam
"p": 4, "n": 14, // page index in that version (all pages), page count
"t": 1790796800000, // last activity
"done": false, // sticky once true (reset via "mark unread")
"shift": [], // book mode: run starts shown alone
"top": 4 // furthest page read in this version (Senpai horizon)
"shift": [] // book mode: run starts shown alone
} } } },
"reading": { "admin": { "Moon Garden": { "1/2": {
"schema": 1, "totalActiveMs": 412000, "loadingMs": 7400,
"sessions": 3, "completedPasses": 2,
"inputMs": { "mouse": 12000, "touch": 380000, "keyboard": 20000 },
"currentPass": { "version": "c0002 Roots [Official]", "seenRanges": [[0, 13]], "activeMs": 151000 }
} } } },
"follows": { "admin": { "Moon Garden": true } },
"notes": { "admin": [ { "id": "a1b2c3d4e5f6", "t": 1790797300000, "m": "Moon Garden", "c": "2/6",
@@ -116,6 +154,9 @@ This keeps ids human-readable and derivable without a database. The price: renam
```
- `data/scan-cache.json` holds `{ "<Manga>/<Book>/<Chapter>": { m: <folder mtimeMs>, p: [[file, w, h], …] } }`. It is disposable.
- `data/library-index.json` holds a disposable, versioned index snapshot and shallow
directory stamps. The library folders remain authoritative; a missing or invalid
snapshot triggers a full scan. Rollback-compatible `scan-cache.json` is retained.
- **Backups:** copy `data/db.json`. The atomic writes make a copy at any time consistent.
## 6. Server
@@ -143,10 +184,13 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
| `GET /api/me` | user | Name, admin flag, settings, unread count |
| `POST /api/settings` `{…partial}` | user | Validated update of the settings keys above |
| `POST /api/password` `{old, new}` | user | Change password |
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read |
| `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions and the user's progress, `continue` target |
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read, and optional personal Reading Story aggregates |
| `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions, progress, optional chapter-time summaries, and `continue` target |
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, optional Active Reading Clock summary, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
| `POST /api/progress` `{m, c, v, p, n, done, shift}` | user | Save the reading position |
| `POST /api/reading/pulse` `{m, c, v, session, seq, …deltas}` | user | When opted in, idempotently merge a clamped active-time pulse, verified page coverage and input totals under a single-reader lease |
| `GET /api/reading/export` | user | Export the user's private derived reading-time summary without raw input or page-exposure diagnostics |
| `POST /api/reading/delete` `{m}` or `{all:true,password}` | user | Delete one manga's reading history, or all history after password confirmation, without changing normal reading progress |
| `POST /api/unread` `{m, c}` | user | Forget progress for a chapter |
| `POST /api/vote` `{m, c, v, value: 1\|-1\|0}` | user | Vote a version; returns the re-ranked versions |
| `POST /api/follow` `{m, on}` | user | Follow a manga for notifications |
@@ -155,7 +199,6 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
| `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result |
| `POST /api/admin/user` `{name, password, admin}` | admin | Create a user or reset a password |
| `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data |
| `GET /api/senpai/v1/{locate,xray,characters/<ref>[/timeline],portraits/<ref>}?m=…` | user | Optional proxy to Senpai (`SENPAI_URL`): adds `Authorization: Bearer SENPAI_TOKEN`, `X-Senpai-User` and `X-Senpai-Horizon` (furthest chapter with progress in this manga, its version and `max(p, top)`, percent-encoded). 404 when `SENPAI_URL` is unset; other paths are not forwarded |
### 6.3 Authentication and security
@@ -239,9 +282,9 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
| Click or tap on a page, top/bottom 35 % of the screen (flow) | Scroll to the previous/next page top, at most 90 % of a screen (85 % steps for webtoons). At the very bottom → next chapter |
| Click or tap on a page, middle band | Toggle the bars |
| Horizontal swipe ≥ 50 px within 800 ms (paged, not zoomed in) | Previous/next |
| Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Page zoom 20–600 %, snapping to 100 % within ±3 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages |
| Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Continuous page zoom 20–600 %, snapping to 100 % within ±1 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages |
| Mouse drag (moved ≥ 6 px) | Pan; the following click is swallowed. Hand cursor when the content is larger than the screen |
| Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` `-` `0` zoom; `m`/Esc bars; `f` fullscreen; `x` Who's here (Esc closes it) |
| Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` and `-` zoom in 5-percentage-point steps, `0` fits at 100 %; `m`/Esc bars; `f` fullscreen |
**Bars:**
- Shown on open.
@@ -262,23 +305,50 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
- Stores the page index in the full page list, so skipping extras doesn't corrupt it.
- On open, the saved page is used for the same version, and mapped proportionally for another version. A finished chapter reopens at the start.
**Active Reading Clock (opt-in):**
- Runs only while the reader is visible and focused, a sufficiently exposed page is decoded, a server lease is held, and accepted reader input occurred in the last 50 seconds.
- Backgrounding or blurring pauses immediately and requires fresh input to resume. A delayed callback can never add time beyond the 50-second input deadline.
- A page is verified after at least 35% meaningful exposure for 200 ms cumulatively. Exposure is closed and evaluated when the page leaves the viewport, so a fast but plausible webtoon scroll is not lost between samples. A five-pages-per-second token bucket rejects implausible coverage without discarding otherwise credible active time.
- `BroadcastChannel` coordinates tabs on one browser. The authenticated API adds a 15-second per-user lease for other devices, and releases it when the tab pauses or leaves.
- Pulses are sent at most every 10 seconds and on pause/page exit. The server validates the manga/chapter/version and page indices, de-duplicates `(session, seq)`, clamps time to server-observed intervals and stores compact page ranges separately from ordinary progress.
- The bottom toolbar chip shows active time and the current state; its panel adds pass time, verified pages, predominant input and loading wait.
**Zoom persistence:** per mode key (`flow`, `page`, `book`, `strip`), saved 800 ms after the last change.
## 8. Deployment
### Current Windows deployment
The observed local installation uses the Windows Scheduled Task
`\MangaCircus Supervisor`. At logon it runs `tools/run-supervisor.ps1`, which appends
logs under `data/scraper/` and starts one `tools/start-all.js` supervisor. That parent
owns exactly one reader (`server.js`, port 8080) and one scraper daemon
(`scraper/daemon.js`, private port 8091). Starting `npm run start:all` manually at the
same time would create conflicting writers and is not supported.
The scraper's authoritative `data/scraper/queue.json` is schema version 3. It supports
legacy archive checkpoints and paused conversion of completed legacy work while adding
source-aware lanes, scoped queue jobs, incremental chapter publication, and nightly catalog discovery. See
[local-deployment.md](local-deployment.md) for operational checks and restart/recovery,
and [scraper-service.md](scraper-service.md) for the queue/API contract.
### Docker or manual deployment
```bash
docker compose up -d --build
```
- **Image:** `node:24-alpine` plus the app files. It runs as user `node` (uid 1000) and has a health check against `/api/auth`.
- **Images/services:** the same `node:24-alpine` image runs a reader service and a scraper service. The reader health check targets `/api/auth`.
- **Volumes:**
- `/library`: read-only is enough. Files must be readable by uid 1000.
- `/data`: must be writable.
- **Configuration:** only environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `COOKIE_SECURE`, optional `SENPAI_URL` + `SENPAI_TOKEN`).
- **With Senpai:** the Senpai repository's `docker-compose.circus.yml` builds this branch next to Senpai and sets `SENPAI_URL=http://senpai-api:8780`; Senpai is reached only through the proxy, never by the browser.
- **Without Docker:** `node server.js` with Node ≥ 18.
- `/library`: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000.
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
- **Upgrades:** replace the code and restart. There are no migrations: unknown fields in `db.json` are kept, and missing settings fall back to defaults.
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
unknown fields in `db.json` are kept. The scraper normalizes old queue entries into
schema version 3 on startup and preserves legacy job checkpoint behavior. Take a
queue/checkpoint recovery copy before upgrading a live scraper.
## 9. Capacity and limits
@@ -311,5 +381,7 @@ docker compose up -d --build
- **Notifications:** in-app plus webhook only. Web Push would need VAPID keys and a service worker.
- **No login rate limiting and no CSP header.** Both are small additions in `server.js`.
- **In-place image overwrites** are only detected by a full rescan.
- **No automated test suite.** Verification so far was done with the demo library (`tools/make-dummy.js`), API smoke scripts and manual browser testing on desktop and mobile viewports. `parseKey` and `sizeFromBuffer` are pure functions, and `scanLibrary` only needs a folder. Together with the book pairing they are the natural first targets for unit tests.
- **Reader layout remains mostly smoke/manual.** Reading analytics has focused Node tests
for its persisted merge rules, while DOM layout, authentication and mobile browser
lifecycle behavior still need browser-level/manual verification.
- **Adding content by API** (instead of files) would be a new route plus the staging and renaming rules from [import.md §7](import.md#7-consistency-writing-into-a-live-library).
+141
View File
@@ -0,0 +1,141 @@
# Collections and relationships
Collections are a presentation layer, not a filesystem migration. Creating, reordering,
or removing one never renames manga/chapter folders or changes reading IDs.
## Using the UI
Open **Settings → Admin → Collections & metadata** (or **Manage collections** on the dashboard).
Create a collection, add library manga, choose each member's role and reading order,
and save. Choose a member cover, a mosaic of up to four covers, or upload a custom
image under 600 KB. Dashboard display can be grouped, expanded, or individual; the
choice is remembered per browser. A collection page shows aggregate progress and
continues the first unfinished member in the chosen order.
**New collection** starts an unsaved draft using an ungrouped library manga. Rename
it, add members and choose **Save collection** (or **Save all changes**). Each manga
can belong to one collection. If all manga are grouped, remove a member from another
collection first; a collection's final member is retained until you delete the
collection. Deletion never removes manga or reading progress.
Save feedback and errors appear above the editor. Controls are locked during a save;
failed saves retain drafts. If another admin has changed the catalog, the stale write
is rejected. **Reload catalog** fetches the latest revision, with confirmation before
discarding edits; copy any wanted changes first. Unsaved JSON survives collection
selection and must use **Validate & save JSON**, not an ordinary collection save.
Metadata imports/research are blocked while drafts exist so they cannot silently
replace them. Relationship edits save immediately when there is no collection draft.
Creation works on HTTPS, localhost and ordinary HTTP LAN/Tailscale addresses. The
browser uses secure random bytes when its secure-context-only `randomUUID` API is
unavailable. This does not change networking or the server's revision/validation rules.
Tag combinations match a single member, not tags scattered across unrelated members.
Reading-time totals only include the current account's visible Reading Story. Collection
publication status is ongoing if any member is ongoing, completed if all are completed,
otherwise mixed. Collections do not control scraping or merge editions.
## Three distinct concepts
| Concept | Meaning | Owner |
|---|---|---|
| Relationship | Directed fact: target is a prequel/sequel/adaptation/etc. of source | Reader's Catalog Curator |
| Collection | Which library manga share a dashboard tile | Admin decision |
| Reading order | Recommended order within that collection | Admin decision, independent of chronology |
One manga belongs to at most one collection, preventing duplicate dashboard tiles and
double-counted progress. Relationships can connect any number of works across groups.
Only confirmed relationships appear on reader manga pages. Suggestions, rejections
and conflicts remain visible to the admin. Removing a collection leaves its manga,
reading history and relationships intact.
## Metadata and agent responsibilities
The metadata provider, **not the image scraper**, supplies relationships. AniList
metadata now retains manga relationship edges in `manga.json` → `x-anilist.relations`.
Existing metadata will not magically gain these fields: use **Research relationships**
for a manga with a verified AniList ID, or import already-cached suggestions.
Research uses one public metadata lookup with a 30-day cache, never image-host traffic.
Only relationships whose two exact AniList identities already exist in the library
become suggestions. This does not auto-enqueue missing manga or guess title matches.
An agent is a **catalog researcher/curator assistant**. It can read the admin JSON export,
research public evidence, identify conflicting claims and propose relationships or
membership edits. It should preserve rejected decisions and report uncertainty.
It must not turn a shared-universe link into an automatic collection or an unquestioned
reading order. No embedded LLM, API credentials, automatic research loop, or autonomous
catalog approval is enabled in this release. That optional capability belongs behind
this same evidence/review boundary, not inside adapters or Sentinel.
## Persistence and API contract
`data/catalog.json` is reader-owned, version 1, atomically replaced on save. Its
revision supports optimistic concurrency; stale writes return 409. Unknown members,
duplicate membership, invalid relationships and cover paths are rejected. Existing
members absent from a temporary mount are retained rather than erased.
Custom covers are content-addressed in `data/collection-covers/`. Include both in backups.
Authenticated reader APIs:
- `GET /api/collections`, `GET /api/collection?id=…` — account-specific public views.
- `GET /api/admin/catalog` — admin export plus available manga, roles and relation types.
- `POST /api/admin/catalog` — validated full replacement, with current `revision`.
- `POST /api/admin/catalog/suggest` — import cached edges, with current revision.
- `POST /api/admin/catalog/research` — `{m: libraryMangaId, revision}`; verified metadata lookup.
- `POST /api/admin/collection-cover` — `{id, revision, base64}`; validated image upload.
Example proposal (use real IDs and the latest revision from the export):
```json
{
"version": 1,
"revision": 0,
"collections": [{
"id": "moon-garden",
"title": "Moon Garden Universe",
"description": "Main story, then the prequel.",
"grouped": true,
"coverMode": "mosaic",
"coverMangaId": "Moon Garden",
"members": [
{"mangaId": "Moon Garden", "role": "main", "readingOrder": 1},
{"mangaId": "Before the Rain", "role": "prequel", "readingOrder": 2}
]
}],
"relations": [{
"from": "Moon Garden", "to": "Before the Rain", "type": "prequel",
"state": "suggested", "method": "agent-research", "confidence": 0.8,
"explanation": "Verify the publisher's chronology before accepting.",
"evidence": [{"provider": "Publisher", "url": "https://example.com/work", "note": "Chronology"}]
}]
}
```
Provider contract: [AniList relationship connections](https://docs.anilist.co/guide/graphql/connections).
## Repeatable functional acceptance
Run `npm.cmd test`, build `docker compose build reader`, then:
```powershell
# Playwright is optional developer tooling; not a deployed application dependency.
# Set NODE_PATH to its installed package root if it is not on normal resolution paths.
node tools/test-compose.js --collections-ui --settings-ui
```
The runner creates an isolated Compose project on loopback 8180 with three synthetic
manga, disposable accounts, paused scraper and disabled discovery. It never mounts
the real library, DB or queue. Headless Edge (override `QA_BROWSER_CHANNEL` if needed)
proxies a synthetic `http://manga-qa.test:8180` origin to that fixture, reproducing the
browser restrictions of a non-localhost HTTP address without using a real tailnet
account. Screenshots: `data/qa/collections/desktop-editor.png` and `mobile-editor.png`.
Coverage: creation, validation, persistence after reload, member add/remove/role/order,
selected/mosaic/custom covers and upload limits, JSON round-trip/draft protection,
relationship creation/rejection/duplicates/removal, cached import, missing verified
metadata identity, desktop/mobile edits, dashboard grouping and collection reading
order, delete/cancel, overlapping saves, transient save failure/retry, concurrent-admin
revision conflicts, and ordinary-reader read/write denial. No external manga or
metadata research is performed; provider lookup success remains a separate test.
The earlier settings test checked collections navigation only; this suite exercises
the actual collection workflow and the insecure-origin failure it missed.
+72
View File
@@ -0,0 +1,72 @@
# Discovery Scout (deterministic first release)
The reader now has a separate **Discover** page. This first slice deliberately uses
the curated Greatest catalog, already-cached AniList identities/relations, library
metadata, each user's Greatest votes, and explicit Discover feedback. It does not
call an LLM, fetch new metadata, inspect chapter/image hosters, edit a collection,
or enqueue downloads. The reader remains the only writer of account and discovery
state; no fourth Compose service, credential, or database is required.
## Use
1. Open `discover.html`. Each account independently enables **Personalized
discovery** (off by default). Reading activity is a separate opt-in. The
mature-title switch only applies to candidates with a known mature tag; it is
not a content-safety guarantee.
2. An admin presses **Run discovery now** on the same page. Existing suggestions
remain visible until the new feed is published. Runs are bounded to 50
candidates and 20 suggestions per opted-in user, with no more than four from
one period. A second concurrent run is rejected.
3. A reader can open **Why this is here · evidence** and record Interested, Maybe,
No thanks, or Hide. Feedback is private to the account. Dismissed/hidden
titles disappear at once; other feedback influences the next run. The user
can undo a choice by clicking its active button again.
The recommendation is an editorial lead, **not** an availability or edition
check. The public metadata link opens the provider record; an admin still uses
the existing source inspection and confirmation process to download anything.
The title/year/format matching is intentionally conservative but not a verified
identity merge. Missing year, cover, or status remains missing.
## Data and API
- `data/discovery.json`: atomic, reader-owned candidate and per-user feed
snapshots. A previous valid version is retained as `discovery.json.prev`.
A corrupt file makes reads/runs fail closed rather than overwrite it.
- `data/db.json`: consent in user settings and explicit feedback under
`discoveryFeedback`. The account-deletion route removes that feedback.
- `GET /api/discovery`: caller's own feed, settings, evidence, and feedback only.
- `POST /api/discovery/feedback`: caller's candidate choice; requires opt-in.
- `GET /api/admin/discovery` and `POST /api/admin/discovery/run`: admin-only
status and manual generation. A run keeps the last good feed on failure.
The job asks the private scraper state API for queued titles to avoid recommending
them. If that API is unavailable, the run still uses only cached sources; the
admin result records `queueAvailable: false` and queue exclusion is incomplete.
No scraper queue edit occurs.
## Read-only dry run and verification
From the project directory:
```powershell
node discovery/cli.js run --dry-run --data data --show
npm run test:reader
docker build -t mangacircus:discovery-test .
$env:APP_IMAGE='mangacircus:discovery-test'; npm run test:discovery-ui
```
The dry run reads `db.json`, a saved library index if present, Greatest, and
cached AniList metadata. It writes nothing and does not contact providers. The
Compose/browser suite uses a disposable data tree and queue, not the live one.
## Deliberate limits and next stage
This slice has no scheduled run, external metadata discovery, LLM, admin seed
form, edition-finding action, or explicit purge control for old feed snapshots.
Disabling consent stops feed display and future personalized generation, but
existing snapshot/feedback is retained until account deletion or a later data
management feature. Candidate quality is limited by the curated list and local
metadata cache. Add external public metadata feeds and optional LLM only behind
the existing bounded candidate/evidence contract after this deterministic path
has been used and reviewed.
+34
View File
@@ -0,0 +1,34 @@
# Shijō Saikō no Manga — Greatest of All Time
`greatest.html` is a second, single-page dashboard. The Japanese title is **史上最高の漫画**, romanized **Shijō Saikō no Manga** (“greatest manga of all time”). It displays the [research reading map](manga-reading-map-2026-10-09.md) as ranked five-year cohorts from before 1970 through 2025–2026. Each period keeps its own separator and tile grid.
Each tile shows a four-line **Why it matters** preview from the editorial rationale; opening the tile shows the complete text under the same heading. The note is a reason for inclusion, not a plot synopsis or a machine-generated rating.
The inclusion rationale should answer **why this title belongs in this particular survey**: a documented award or record, a demonstrable genre/cultural legacy, exceptional artistic craft, or a clearly labelled editorial fit for this library. A synopsis alone is not sufficient. Do not turn the curator's cohort rank into a claim about an external top-10 poll. The 2025–2026 cohort is deliberately labelled as *future-watch* or *provisional*; a recent reader vote is evidence of early enthusiasm, not proof of all-time importance. Specific “first” and record claims need a primary source before publication.
Primary-source checks for selected concrete claims in the seed include [Tezuka's Princess Knight archive](https://tezukaosamu.net/en/manga/559.html), [Guinness on Golgo 13's volume record](https://www.guinnessworldrecords.com/world-records/405899-most-volumes-published-for-a-single-manga-series), [Kodansha's manga-prize history](https://www.kodansha.co.jp/awards/comics/histories), [the 2025 Harvey Award winners](https://www.harveyawards.com/en-us/about/news/harvey-awards-reveal-2025-winners.html), and the official [2026 Next Manga Awards print](https://tsugimanga.jp/winners_announcement/comics) and [web](https://tsugimanga.jp/winners_announcement/web) results. Keep award year/category precise when editing the notes. Statements of artistic merit or personal-library fit are editorial judgments, not measurements.
## Separation and data ownership
- `greatest/catalog.json` is a versioned, read-only editorial seed in the application image. Entries have stable IDs, debut year, cohort, curator rank, rationale, official source lead, and optional AniList cover/chapter/status metadata. It is **not** part of `library/`, scraper queue, or `data/catalog.json` collections.
- `greatest/index.js` validates the seed and performs only exact title/alias matching against the existing library index. A match adds the local cover, chapter count, status and **In library** badge; it never moves or downloads a title.
- `data/db.json` holds `greatestRatings[userName][entryId]`. Each account can select `flame`, `good`, `neutral`, or `skip`, or click its current choice to clear it. Existing chapter-version votes are separate. Deleting a user removes their ratings. Include `data/db.json` in backups as before.
- The rating write route is authenticated `POST /api/greatest/rating` with `{id,rating}` (`null` clears). Authenticated `GET /api/greatest` returns the catalog, per-user choice, valid-user community counts, score, local-library match, and supported series links from that manga's library metadata. Unknown IDs/ratings are rejected. There is no admin-only gate for voting.
- The details popup shows MangaDex and WeebCentral **series** links from the matched library manga's `manga.json`. Admin-only `GET /api/greatest/sources?id=…` checks live scraper queue state by source series ID and conservative title/alias match. Queue-only title matches are visibly marked as needing edition verification; no search result is silently treated as a confirmed match. If the scraper is unavailable, library links remain visible while queue actions are disabled.
- Admins can use `GET /api/greatest/mangadex-search?id=…` for a single on-demand, metadata-only title lookup (10 candidates maximum, two-second minimum between uncached lookups, one-hour result cache). The UI shows candidate title and year so an admin can choose the right edition. The WeebCentral search-page link and a pasted series URL cover sources without a safe automatic title lookup. Neither source is bulk-searched when the dashboard opens.
- **Add to scraper** uses the existing admin `enqueue` command with `inspect:true`, `scope:{mode:"missing"}`, `existingPolicy:"skip-any"`, and the owned library ID when available. It adds an inspection job; an admin must confirm any download later in Admin → Scraper. Existing jobs are marked with their queue status and cannot be added again from a known link. The daemon remains responsible for URL validation, duplicate prevention, and source protections. Ordinary users can view known links but cannot inspect queue state or add jobs.
Within each cohort, **Rank by my vote** sorts flame → good → neutral → skull → unrated; **Rank by everyone** uses `3×flames + thumbs-up − 2×skulls`, with curator rank as the tie-breaker. These are preference scores, not independent critical ratings. The quick chips filter by *your* ratings (or in-library status), even when community sorting is active. The popup shows the full rationale, metadata, official source link, community counts, and a library link where available.
## Covers and metadata
The one-time seed is curated from the research document. All 106 catalog entries now have a cover URL. Most were matched to public AniList manga metadata; the five initial gaps were checked individually. Phoenix (1954) and Nausicaä use matching AniList records; Kitarō uses Drawn & Quarterly's Kitaro edition, Someone Hertz uses Shonen Jump's first-volume cover, and Hitoner uses Shueisha's 2026 first-volume cover. Hitoner's AniList 2024 entry represents an earlier work, so it was **not** used for the 2026 series. The five added URLs returned image content on 2026-10-09, but remote hosts can change them; the UI still shows a title-letter fallback if an image fails.
`node tools/enrich-greatest-catalog.js` is an **optional manual** public-AniList metadata refresh, paced at one request every 1.5 seconds. It only accepts matching manga title and **original serialization year**, leaving uncertain results untouched rather than using a sequel or remake cover. The dashboard makes **no metadata calls when loading**; only an admin pressing **Search MangaDex** triggers a bounded lookup. Browser covers are lazy-loaded from their listed source; local library covers take precedence.
Public catalog chapter counts are often absent; the UI says “Chapters unknown” rather than inventing one. For owned titles, the local library count wins. Status comes from AniList's releasing/finished flag or local library metadata, and may be unknown. Official source links may be publisher editions, chapter services, or an award/info lead; they do not imply all chapters are available in every country. No unverified hoster coverage is claimed.
## Deployment and QA
The Docker image now copies `greatest/` as app code; no new Compose service, volume, port, secret, or migration is needed. Rebuild/recreate **reader only** to pick up catalog/UI changes. The scraper and sentinel do not need restarting. Existing `data/db.json` accepts the new `greatestRatings` object without migration.
Run `npm run test:reader` for catalog/API tests and `node tools/test-greatest-ui.js` for disposable headless-browser QA. The UI fixture blocks remote covers and never contacts the scraper. Manual catalog edits should preserve stable IDs once anyone has voted; changing a title's ID strands its votes.
+23 -8
View File
@@ -4,11 +4,12 @@ MangaCircus does not copy or convert anything. "Import" means **indexing the lib
```
┌───────────── triggers ─────────────┐
│ start-up · every SCAN_INTERVAL min │
│ start-up · shallow interval · daily full │
│ admin "Rescan" · admin "Full rescan"│
└────────────────┬───────────────────┘
▼
library/ ──read──▶ scanLibrary(root, oldCache) library.js
data/library-index.json ──▶ immediate in-memory index at start (if valid)
library/ ──read──▶ shallow changed-manga check or full scan library.js
├─ scanManga (per manga folder)
│ ├─ book folders → parseKey, book.json, cover
│ └─ chapter folders → scanVersion
@@ -20,19 +21,28 @@ MangaCircus does not copy or convert anything. "Import" means **indexing the lib
▼
lib = index (atomic swap, requests see old or new, never half)
▼
data/scan-cache.json (written atomically)
data/scan-cache.json + data/library-index.json (atomic)
```
## 1. Triggers
| Trigger | When | Cache |
|---------|------|-------|
| Start-up | Right after the HTTP server starts listening. Requests are served meanwhile; the library page shows "Scanning…" until the first scan is done | used |
| Interval | Every `SCAN_INTERVAL` minutes (default 10, `0` = off) | used |
| Start-up | Restore the saved index immediately if its library mount is present, then check changed manga. Without a valid snapshot, scan the whole library; the page shows "Scanning…" until done | used |
| Interval | Shallow change check every `SCAN_INTERVAL` minutes (default 10, `0` = off); full reconciliation every `FULL_SCAN_INTERVAL` minutes (default 1440, `0` = off) | used |
| **Rescan** (Settings → Admin, or `POST /api/admin/rescan {}`) | On demand; the request returns when the scan is done | used |
| **Full rescan** (`POST /api/admin/rescan {"full":true}`) | On demand | ignored: every image header is read again |
Scans are **single-flight**. A trigger that fires while a scan is running joins that scan instead of starting a second one.
The saved index records when its last full reconciliation finished. A warm restart
does a shallow check but does not reset the daily full-scan clock; if that scan is
overdue, the reader schedules it within a minute. Set `FULL_SCAN_INTERVAL=0` to
disable automatic full reconciliations, while retaining admin Rescan.
Scans are **single-flight**. A trigger during a scan queues one coalesced follow-up,
with a full reconciliation taking precedence over a shallow check. Publication
revisions trigger a shallow check; if no changed directory is visible, the reader
falls back to a full scan. In-place edits to `chapter.json` are found by the full
reconciliation or an admin Rescan, not by shallow parent-directory stamps.
## 2. Walking the tree
@@ -91,7 +101,7 @@ Each chapter folder is one **version**:
| `id` | folder name |
| `source`, `sourceUrl` | `chapter.json.source` (`{name,url}` or a string), else the `[…]` part of the folder name |
| `added` | `Date.parse(chapter.json.added)`, else the folder mtime |
| `direction` | first valid of chapter.json → book.json → manga.json → `rtl`. Aliases: `vertical`, `webtoon` → `ttb` |
| `direction` | manual chapter → book → manga; else coherent tall-strip dimensions → `ttb`; else prior generated vertical series direction; else fallback `rtl`. Aliases: `vertical`, `webtoon` → `ttb`. Legacy plain values remain manual |
6. A version without images is dropped silently.
@@ -155,6 +165,11 @@ The reader lays pages out before they load, so it needs every image's size. Inst
## 4. The scan cache
- Stored in `DATA/scan-cache.json`. It is rewritten atomically after every successful scan and holds only the folders seen in that scan, so deleted chapters drop out.
- `DATA/library-index.json` also stores the complete rebuildable index and shallow
manga/book directory stamps. It lets a restarted reader serve the dashboard and
manga pages before the background filesystem check completes. Both files can be
deleted safely; without the index the next startup walks the library, and without
the size cache it also rereads image headers.
- Invalidation is **per chapter folder, by its mtime**. Adding, removing or renaming a file changes the folder mtime, so that folder is re-read on the next scan.
- **Blind spot:** overwriting an image *in place* (same name) usually doesn't change the folder's mtime, so the old size stays cached. Run a **Full rescan** after such edits, or write new files and rename them (§7).
- Deleting the cache file is harmless. The next scan is simply slower.
@@ -196,7 +211,7 @@ The index is read by the API at request time. Nothing is precomputed per user.
| Endpoint | Built from |
|----------|------------|
| `GET /api/library` | manga list + the user's progress, follows and unread notifications |
| `GET /api/library` | manga list + searchable metadata (`altTitles`, `authors`, `tags`, `status`, normalized first release), the user's progress, follows and unread notifications |
| `GET /api/manga?m=` | manga incl. `meta` (the raw `manga.json`), books, chapters with ranked versions and the user's progress, "Continue" target |
| `GET /api/chapter?m=&c=&v=` | pages of the chosen version (`v`, else the best ranked), direction, prev/next chapter (skipping extras if the user wants that), the user's progress |
| `GET /lib/<path>` | the image file itself. Logged-in users only, image extensions only, path must stay inside `LIBRARY`. ETag + `Cache-Control: private, max-age=86400` |
+36 -1
View File
@@ -72,6 +72,24 @@ Ignored, with a warning in the log:
### The key
The additive unnumbered MangaDex chapter form is
`special-mangadex-<chapter-uuid> Pilot [MangaDex · Group]`. It is accepted only for
chapters, never books. Its logical key is the complete lower-case prefix/UUID;
`chapter.json` supplies `special:true`, `label` (Pilot/One-shot/Special), `sourceId`
and nullable `sourceNumber`. The reader shows that label with no fabricated number.
Such chapters sort after numbered entries and are not inherently skippable extras.
Different source UUIDs remain distinct releases; no content equivalence is guessed.
Numeric IDs and source-version folders are unchanged. Older reader releases ignore
these new folders but retain them on disk; upgrade again to restore visibility.
Whole-volume WeebCentral releases use the analogous
`special-weebcentral-volume-<26-character-release-id>` chapter key. They live in
the matching volume book and carry `releaseKind:"volume"`, `volume`, `sourceId`,
and a `Volume N` label in `chapter.json`. This keeps an entire volume distinct
from a numbered chapter in the same book. The default `skip-any` download policy
also skips a whole-volume release if that book already contains numbered chapters;
`add-source` allows the extra source edition explicitly.
The number is normalised: leading zeros and trailing decimal zeros are dropped (`0010.50` → `10.5`), and an `x` prefix is kept. The result is the **key**:
| Folder | Key |
@@ -165,10 +183,17 @@ UTF-8 JSON (a BOM is tolerated). All fields are optional, and unknown fields are
| `tags` | string[] | Genres / tags |
| `description` | string | Synopsis (line breaks are kept) |
| `direction` | `"rtl"` \| `"ltr"` \| `"ttb"` | How the pages are read. `rtl` (the default): manga, right to left. `ltr`: comics, printed manhwa. `ttb`: webtoon, one vertical strip cut into slices (aliases `vertical`, `webtoon`) |
| `directionOverride` | same values | Explicit manual choice. Use this even when choosing `rtl`, which is also the fallback. May be set at manga, book, or chapter level |
| `x-reading-direction` | object | Generated provenance: `{ "version": 1, "value": "ttb", "source": "detected" }` or `source: "fallback"`, with optional numeric evidence. The scraper owns this field; it is not a confidence score |
| `sources` | `{name, url}[]` | Upstream sources the manga was scraped from. Several are allowed |
| `x-anilist` | object | AniList work ID and linked metadata. The ID identifies a work, not necessarily a particular pilot or edition |
| `x-metadata.imported` | object | Last imported values for title, alternative titles, authors, status, year, tags and description. The scraper updates a displayed field only when it is empty or still equals its previous imported value; edits in legacy files without this snapshot are preserved |
| `x-scraper.autoUpdate` | boolean | Optional explicit admin choice. When absent, `status: "ongoing"` enables nightly checks and other statuses disable them |
The whole file is sent to logged-in users as part of the manga page. **Don't put secrets in it** (scraper tokens, cookies).
An AniList refresh with a different ID is stopped for review before chapter publication. Existing files that already contain `x-scraper.autoUpdate` keep that choice; the scraper does not guess whether an older value was automatic or manual. New files leave the choice absent so the default can follow status changes.
### `book.json`
| Field | Type | Meaning |
@@ -188,7 +213,17 @@ The whole file is sent to logged-in users as part of the manga page. **Don't put
| `priority` | number | Tie-breaker between versions with equal votes. Higher wins (default `0`) |
| `direction` | as above | Overrides book/manga direction, e.g. a colour webtoon special inside a printed series |
`direction` is inherited **chapter → book → manga → `rtl`**, and each version can carry its own.
Manual directions have priority at **chapter → book → manga**. A `directionOverride`
or a plain `direction` without matching generated provenance is manual. Otherwise,
the scanner checks the version's existing image dimensions for coherent tall strips
and uses `ttb` when they are present. A prior generated vertical series direction
applies to chapters with insufficient shape evidence. The final fallback is `rtl`;
this does **not** classify a work as Japanese or prove its reading order. A direct
edit to a generated `direction` becomes manual when it differs from its provenance.
Existing plain directions are treated as manual because old scraper defaults and
human edits cannot be distinguished safely. Set `directionOverride` to make a
lasting choice, or explicitly review and migrate a legacy entry. Image files and
chapter IDs do not change when direction metadata changes.
## Picture-viewer compatibility
+163
View File
@@ -0,0 +1,163 @@
# Server peering (v1)
Status: implemented, tested in isolated local fixtures, and enabled for a local
two-instance pilot on 2026-10-10. It has **not** been deployed to the VM. See the broader design and
future ideas in [`vision/server-peering.md`](../vision/server-peering.md).
Peering is an opt-in, pull-only transfer between MangaCircus servers. It does not
exchange accounts, reading history, ratings, notes, or scraper credentials. The
publishing gateway reads a library mount, and only the existing scraper publishes
received chapters. No database server or second library writer is introduced.
## Roles and direction
The interface is administrator-only and split by task: **Settings → Admin → Peering**
manages server identity, peers, grants and transfer limits; **Peer catalogs** manages
per-peer title approvals and remote catalog browsing. Ordinary users cannot see either
page or call their APIs. Each server may have many peers.
- **Grant scopes**: Catalog permits catalog, covers and metadata. Download includes
Catalog and permits chapter transfers. Each direction has its own grant.
- **Serving state**: Active or Paused. Pausing preserves the grant/token; explicit
Revoke invalidates it. Scope changes take effect without exchanging a new token.
- **Approved titles**: the publisher chooses titles on Peer catalogs, not in the
connection settings. Pairing or a grant alone does not share a title.
- **Local receiving** is a separate switch. Browsing works with Catalog permission
even when local receiving is off; downloads require an active Download grant and
local receiving enabled.
- Transfer limits cover upload streams, bytes/s and bytes/day, plus receiving pages
in flight and bytes/s. Defaults are two in-flight pages and 2 MiB/s each direction
per peer. Publisher caps remain authoritative; a gateway-wide four-stream ceiling
applies.
The required local server display name is 1–25 ASCII letters/digits. It is a friendly
label, not the cryptographic identity; changing it does not rotate the certificate.
An invitation expires after 48 hours, the certificate currently lasts one year, and
a grant lasts up to 90 days bounded by certificate validity. The UI labels these as
separate lifetimes.
## Deployment
The optional `peer-gateway` Compose profile publishes port 8093 on **127.0.0.1** by
default. It uses the same image as reader/scraper, a read-only library mount, a
read-only peering policy/identity mount, and its own writable usage-counter mount.
The gateway needs an external-facing Docker network for its host port to work on
Docker Desktop; it also joins the private backend. It has no reader account database,
scraper queue, Docker socket, or writable library mount. It must never be scaled
past one instance for the same usage-counter directory.
1. Build the image and start the normal stack as documented in
[`compose-deployment.md`](compose-deployment.md). Set the local server name and
generate its certificate in **Settings → Admin → Peering**. Keep
`/data/peering/identity.key` private and backed up.
2. Start the optional service with `docker compose --profile peering up -d
peer-gateway`. Verify `docker compose --profile peering ps` shows healthy and
`https://127.0.0.1:8093/health` responds. `/health` is deliberately unauthenticated
and contains only `{ "ok": true }`.
3. Give the other server a reachable HTTPS endpoint for port 8093. A private
Tailscale TCP forward is recommended for the pilot; the default loopback binding
is not remotely reachable. Do not expose port 8093 on the public internet merely
to make pairing work. Use a stable tailnet DNS name, not a changing IP. Keep
scraper (8091) and Sentinel (8092) private.
The `PEER_BIND`, `PEER_PORT`, `PEER_DATA_PATH`, and `PEER_USAGE_PATH` Compose variables
control the bind and persistent paths. The gateway's internal port remains 8093.
OpenSSL is bundled in the app image for one-time P-256 self-signed certificate
generation. Each certificate is valid for one year. Expired/replaced identities
require administrator-supervised re-pairing; there is no automatic key rotation yet.
## Pairing and transfer
1. Each admin generates an identity, creates a 48-hour invitation, and exchanges the
JSON outside the app. The JSON contains a public certificate, SHA-256 fingerprint,
URL, invitation ID and one-use code—**not** the private key. Compare the
fingerprints through a separate trusted channel.
2. Both admins import the other's invitation and press **Test & verify pairing**.
The HTTPS handshake presents client and server certificates, checks their exact
pins and dates, and consumes the one-use invitation code. Both sides must probe.
3. A publisher approves titles for that peer on **Peer catalogs**, chooses Catalog
or Download grant scopes, and sends the one-time token privately. The receiving
admin imports it and independently enables local receiving for downloads. Reverse
direction needs its own approvals and grant. The server stores only a hash of a
grant it issued; the receiver stores its imported token server-side.
4. An admin browses the permitted catalog, selects missing/newest/last-five or an exact
chapter, and queues it through the normal scraper. The queue carries a validated
stable work identity: AniList first, then MangaDex/WeebCentral IDs, then another
known hoster ID. If none exists, the owning server supplies an opaque origin ID
that follows the work across peers without exposing its folder name. Imports retain
that identity, so a round trip resolves to the original library title; `sync missing`
skips chapters already present and fetches only additions. Titles alone are never
used to merge works. The peer adapter downloads up
to the configured pages in flight, validates size and SHA-256, resumes a partial
page with HTTP Range, checkpoints between batches, and atomically publishes each
complete chapter into the ordinary library. Existing-chapter policy is applied
before download. Partial transfers remain in the hidden scraper work folder.
The publisher sees the receiving **server** certificate and transfer requests, not
the human reader. It can still observe requested titles, times and byte counts.
## Peer v1 HTTP interface
Catalog, cover, manifest and blob endpoints require TLS 1.3 client certificate
pinning, paired identity, an active grant with the matching scope, and no serving
pause. There is no insecure HTTP fallback or path-based file browsing. `/access`
requires the pinned client certificate and reports status without a bearer grant so
an admin can distinguish pairing from permission.
| Request | Response |
| --- | --- |
| `GET /peer/v1/hello` with one-use invitation headers | Protocol version and server identity fingerprint; no catalog |
| `GET /peer/v1/access` | Pairing state, certificate expiry and current grant status/scopes; no catalog |
| `GET /peer/v1/catalog?cursor=N` | Up to 50 approved opaque title IDs, safe metadata, a stable work identity and cover availability |
| `GET /peer/v1/covers/{mangaId}` | Approved cover image only; bounded to 5 MiB and charged to peer byte quota |
| `GET /peer/v1/manga/{id}?cursor=N` | Up to 100 complete chapter IDs, public metadata and the same stable work identity |
| `GET /peer/v1/manga/{id}/chapters/{id}` | Ordered page IDs, MIME, byte size, SHA-256 and ETag |
| `GET /peer/v1/blobs/{id}` | One image, optional `Range: bytes=N-` and `If-Match`; 206 on resume |
Unknown/unshared IDs return 404. Revoked serving returns 403; bad/missing grants
return 401; caps return 429 and `Retry-After`. No source-site URL, local filesystem
path, or user record is sent in protocol responses. Individual complete chapter
manifests are discovered from final `chapter.json` folders; an unfinished folder is
not shared. Catalog and chapter lists are short-lived in-memory caches. Turning off
serving blocks subsequent requests immediately, though an already streaming image
may finish its current chunk/response.
The initial implementation sends original image bytes; it does not transcode them.
See [`vision/server-peering-compression-test.md`](../vision/server-peering-compression-test.md)
for the measured reason. The transport accepts a small amount of controlled page
parallelism, not unbounded connections.
## Tests and safe operation
`npm run test:peer` creates disposable A/B identities, gateways and libraries and
tests mTLS pairing, no-cert/bad-grant rejection, private catalog, per-title sharing,
catalog-only cover access, manifest hashes, a resumed range response, scraper
publication, duplicate skip, pause and revocation. `npm run test:peer-ab` additionally
builds two independent Compose projects from synthetic fixtures, exercises real
two-way pairing, browsing, cover retrieval, permission updates and sequential queueing,
checks the queue remains paused, and tears both stacks down. The UI suite checks the
guided setup, approvals, and non-admin boundary. An isolated single-stack Compose
acceptance is:
```powershell
$env:APP_IMAGE = 'mangacircus:local' # replace with the image just built
$env:QA_PORT = '8188'
$env:QA_PEER_PORT = '8193'
node tools/test-compose.js --peer-gateway --peer-ui
```
It makes disposable library/data, four containers, a certificate and a browser
session, then tears them down. It never mounts the live library/queue.
For live rollout, wait for a scraper checkpoint, back up the library and data
(including the peering identity, policy, outbound grants, gateway usage and hidden
work folders), and deploy reader/scraper/gateway from one immutable image digest.
There must remain exactly one queue-owning scraper and one reader database writer.
Peering is optional: stop the `peer-gateway` profile to close incoming service
without affecting reading or existing local downloads. Do not delete the identity
to recover from a network error; that would break every peer pin.
Current limits: no automatic certificate renewal, invitation cancellation UI,
multi-peer performance pilot, VM deployment, or public internet exposure review.
The gateway has no automated hoster-interaction logic;
source scraping remains subject to the existing source-safety policy.
+244
View File
@@ -0,0 +1,244 @@
# Reading Story concept
Status: first release implemented on 2026-10-04. The Active Reading Clock records the
underlying per-user, per-chapter measurements and MangaCircus now returns the compact
library, manga and chapter summaries described below. The bounded daily Reading Journal
remains a later phase.
Policy update 2026-10-06: a timed read now means a visit with at least 60% viewed
page coverage, not exhaustive 100% verification. No speed ceiling applies. Below-
threshold lookup visits do not enter these summaries; existing history is retained.
First/last durations continue growing until departure. See `reading-time-concept.md`.
## Product idea
Call the retrospective feature **Your Reading Story**. The Active Reading Clock is the
instrument visible while reading; Reading Story is the quiet, personal record it builds.
The feature should answer four pleasant questions:
1. How much time have I spent with this manga?
2. What did this chapter take me to read?
3. Where am I in the story, including partially timed chapters?
4. What does my reading habit look like over time?
It is not a productivity score. Do not rank users, award points for longer sessions, or
describe a reader as fast or slow. Those patterns would reward timer inflation and turn a
private leisure activity into a competition.
## Display language
Use **active time**, **reading time**, and **timed**. Never call it screen time or claim
that MangaCircus proved comprehension.
Durations are friendly rather than stopwatch-precise outside the live reader:
| Stored measurement | Display |
|---|---|
| no reading record | `Not timed` |
| 1–59 seconds | `Under a minute` |
| 60 seconds–59 minutes | rounded to the nearest minute, for example `7 min` |
| 1 hour or more | hours and minutes, for example `3 h 42 min` |
| incomplete current pass | `4 min active · in progress` |
| one completed pass | `6 min first read` |
| completed rereads | `5 min last read · 18 min total · 3 passes` |
Zero must not mean “read instantly.” A chapter with no record is **Not timed**, because
the setting may have been disabled or the chapter may have been marked read manually.
Loading wait and unqualified visit drafts remain diagnostic details; they are not mixed
into the headline reading duration.
## 1. Reader: the live clock
Keep the implemented bottom-toolbar clock as the immediate view:
```text
[ ⏱ 06:14 · Reading ]
```
Tapping it opens the detail panel. At 60% coverage show a gentle confirmation:
**Reading time is now counted · 60% of pages viewed**. Time continues accumulating;
this does not claim the entire chapter has been read or alter ordinary progress.
For subsequent qualifying visits say **Reread time is now counted**.
“Together” makes the lifetime number feel connected to the story rather than to work.
## 2. Manga page: Your Reading Story card
Place one card after the hero and before metadata. Hide it entirely when the user has no
timing records for this manga; do not show a large empty dashboard.
```text
┌ Your Reading Story ─────────────────────────────────────────┐
│ 3 h 42 min together 31 chapters timed 4 rereads │
│ Typical chapter 7 min Last read Ch. 45 · 18 min ago │
│ │
│ Reading rhythm │
│ 01 ▂▂▃▂▅▃▂ 02 ▃▄▂▂▃ 03 ▂▂▆▃▂ │
│ Each bar is one completed chapter; height is active time. │
└────────────────────────────────────────────────────────────┘
```
The headline values are:
- **time together:** sum of `totalActiveMs` for every chapter in this manga;
- **chapters timed:** chapters with any credible active time;
- **completed passes:** sum of completed passes, phrased as rereads only after subtracting
the first completed pass per chapter;
- **typical chapter:** median `lastCompleteMs` across completed chapters, never the mean;
- **last read:** chapter with the newest `updatedAt`.
The **Reading rhythm** is a compact chapter sequence, grouped by the existing volume
boundaries. It visualises only completed-pass duration. A gap means “not timed,” not zero.
Bars use a log-like capped scale so one very long chapter does not flatten the rest.
Hovering or tapping a bar reveals chapter number, title, last completed duration, lifetime
time and pass count. Selecting a volume with the existing filter also filters this rhythm.
On narrow screens, show the first four headline values in a two-column grid and make the
rhythm horizontally scrollable. The card is collapsible and remembers its state per user.
## 3. Chapter list: useful, quiet timing
Add the duration between the page count and chapter title, using a fixed-width column so
the current chapter-row alignment stays clean:
```text
Ch. 45 35 p. 7 min The Twin Swords ✓ read
Ch. 44 31 p. 4 min The Visitor ✓ read
Ch. 43 29 p. Not timed A Quiet Morning ✓ read
```
The compact column shows the last completed pass. An incomplete chapter shows its current
active time with a small hollow clock, such as `◷ 3 min`. A chapter with multiple passes
gets a subtle repeat marker: `7 min ↻2`. Its accessible label and tooltip expand this to
`Last read 7 minutes; 2 completed passes; 19 minutes total`.
Do not put lifetime time directly into every row; that makes reread chapters visually
noisy. Lifetime detail belongs in the tooltip/tap sheet and Reading Story card.
The chapter timing column is hidden when the clock setting is disabled. Existing history
is preserved and becomes visible again if the user re-enables the setting.
## 4. Library dashboard: one personal summary strip
Above the current filters and cover grid, show a compact strip only when at least one
chapter has been timed:
```text
Your Reading Story 18 h 26 min together · 146 chapters timed
Most time with Berserk · Continue your last chapter →
```
Add two optional library sorts:
- **Recently timed** — newest reading `updatedAt` first;
- **Time together** — manga with the largest personal active-time total first.
Do not put duration badges on every cover by default. They compete with new-chapter badges
and reading progress. A future `Show reading time on covers` preference may opt into a
small `3 h 42 min` line below the existing read count.
The first release should not claim “today,” “this week,” streaks, or a calendar. The
current compact database has totals and latest timestamps, but no honest time-series
history from which those values can be reconstructed.
## 5. Later: Reading Journal
If readers enjoy Reading Story, add bounded daily aggregates rather than raw activity
events:
```jsonc
"daily": {
"2026-10-04": { "activeMs": 2840000, "chapters": 6, "completedPasses": 5 }
}
```
Keep at most 400 daily buckets per user, folding older values into yearly totals. This
enables a private activity calendar, `this week`, `this year`, and “you returned to this
story after 3 months” without storing pointer movements, individual page timestamps or a
surveillance-style event log.
Recommended journal messages are descriptive, not judgemental:
- **A long evening with Berserk · 1 h 18 min**
- **You returned to Gantz after 42 days**
- **Volume 4 was your longest volume · 2 h 06 min**
- **12 chapters reread this year**
Avoid streak pressure, “goal missed,” speed comparisons, and global leaderboards.
## API and derived-data design
Keep the raw `reading` records private to their owner. Add derived summaries at the
existing authenticated boundaries:
### `GET /api/library`
Each manga may include:
```jsonc
"reading": {
"activeMs": 13320000,
"timedChapters": 31,
"completedChapters": 29,
"completedPasses": 35,
"rereads": 6,
"typicalChapterMs": 421000,
"lastChapterId": "3/45",
"updatedAt": 1791130200000
}
```
Also return a top-level personal total derived in the same pass. This avoids an extra
request and lets library sorting work locally like the existing filters.
### `GET /api/manga`
Return the manga aggregate plus a small public summary on each chapter:
```jsonc
"reading": {
"totalActiveMs": 1140000,
"lastCompleteMs": 421000,
"completedPasses": 2,
"currentPassMs": 0,
"currentPassComplete": true,
"updatedAt": 1791130200000
}
```
Do not send input-mode totals, rejected views or page ranges to the manga page. They are
reader diagnostics and would make the overview payload and interface unnecessarily noisy.
Derive all aggregates on the server from the compact records during these requests. The
library is small and JSON-backed; a second denormalised cache would create invalidation
work without a current need. If measurement proves this slow, cache by a monotonic
reading-data revision rather than writing duplicate totals into `db.json`.
## Controls, privacy and corrections
Under **Settings → Reading**, explain: “Reading Story is private to your account.” Add:
- **Show Reading Story** — controls retrospective UI separately from collecting the clock;
- **Forget one manga’s reading time** — available on that manga's card menu;
- **Delete all reading-time history** — destructive, confirmed with the account password;
- **Export my reading-time summary** — compact JSON, with no raw input events.
Turning off the Active Reading Clock stops collection but does not silently erase history.
The retrospective display setting may hide existing history without changing it. An admin
must not see another user’s Reading Story through the normal admin interface.
## Delivery status
Implemented:
1. Server-side summary helpers and tests for missing, partial, complete and reread records.
2. Manga-page card, volume-aware rhythm and aligned chapter-duration column.
3. Library summary strip and the two personal-time sorts.
4. User controls for hide, per-manga deletion, password-confirmed complete deletion and export.
5. A gentle reader completion message for first reads and rereads.
Deferred: observe real use before adding bounded daily aggregates and the Reading Journal.
This order immediately makes the existing measurements useful without changing the clock
protocol or committing MangaCircus to a larger analytics history.
+83
View File
@@ -0,0 +1,83 @@
# Deployment-controlled reading hours and quiz
The reader enforces reading hours on both `GET /api/chapter` and chapter page-image
requests under `/lib/` (HTTP 423 outside hours). The dashboard, manga metadata,
covers, progress, reading-time summaries, scraper and administrator pages remain
available. A reader tab already open when a window ends checks access every 15
seconds and returns to the gate. Chapter images use `Cache-Control: no-store` when
the limiter is configured. Already downloaded images in a browser or files available
directly on disk cannot be recalled; this is a self-imposed server access rule, not
DRM. There is no schedule-changing API or UI. Signed-in readers and admins can
view the effective weekly schedule and timezone under **Settings → Reading
hours**; that page is read-only.
## Docker configuration
Set these deployment variables in the Compose `.env` file next to
`docker-compose.yml`, then recreate **only the reader**. The `reader` service
passes them through to the container. `READER_IMAGE` selects a reader-only
release while `APP_IMAGE` continues to select the scraper and Sentinel image.
Leaving `READING_WINDOWS_JSON` empty disables the limiter. Day keys
are `0` Sunday through `6` Saturday. Each day has any number of `[start,end]`
intervals; an end earlier than its start carries into the next day. `24:00` is
valid only as an end. Intervals are start-inclusive and end-exclusive, resolved
using the IANA timezone, including daylight-saving changes.
```dotenv
READING_WINDOWS_JSON='{"0":[["08:00","01:00"]],"1":[["07:00","12:00"],["13:00","24:00"]],"2":[["07:00","24:00"]],"3":[["07:00","24:00"]],"4":[["07:00","24:00"]],"5":[["07:00","24:00"]],"6":[["08:00","01:00"]]}'
READING_TIMEZONE=Europe/Berlin
READING_QUIZ_UNLOCK_MINUTES=60
READING_QUIZ_CATALOG_PATH=/data/reading-quiz.json
```
The Monday split above illustrates multiple on/off periods; replace it with
`[["07:00","24:00"]]` for an uninterrupted weekday. The Saturday and Sunday
windows extend to 01:00 on the following mornings. A malformed timezone,
schedule, unlock length or catalog stops the reader at startup rather than
silently removing the restriction. `READING_QUIZ_UNLOCK_MINUTES` accepts 1–1440.
The scraper is independent and must not be restarted for this configuration.
Until a reviewed catalog is installed at `READING_QUIZ_CATALOG_PATH`, the
outside-hours gate cannot offer a quiz. It does not relax the schedule.
## Question bank and progress watermark
The bank is a deployment-managed JSON file, separate from `manga.json` and from
the scraper. See [`config/reading-quiz.example.json`](../config/reading-quiz.example.json).
Its `manga` keys are **exact library folder IDs**. Each question has a stable ID,
prompt, accepted answer variants, an evidence citation, and at least one
`requires` chapter ID. The chapter IDs are the reader's IDs (for example `1/12`,
not the on-disk `v01/c0012`). Every required chapter must be marked completed
for that user before the question enters their pool. This supports a single
watermark or a set of specific sections. Five eligible questions are needed to
offer a quiz; an empty bank simply leaves outside-hours reading locked. A
question's answers and evidence never go to the browser.
The challenge randomly samples five progress-eligible questions across *all*
library manga, not only the one being opened. It asks them sequentially, accepts
unlimited retries, and has no answer timer. Accepted variants are normalized
for case, punctuation and accents; one typo is tolerated for answers of five
or more characters, two for answers of eleven or more. Starting a new visit
invalidates the previous challenge, and leaving the page abandons it. Passing
all five grants that user reading access across manga for the configured number
of minutes, persisted in `db.json` across reader restarts. Questions are not
shown before their watermark; this depends on the reader's existing
self-reported completion records.
Validate a proposed file with
`node tools/validate-reading-quiz.js <catalog.json>` before pointing the reader
at it. For an eventual research agent: write **drafts** with source citations,
aliases and proposed chapter watermarks, then have a human verify identity,
answer and spoiler boundary. Only a reviewed catalog should be copied to the
configured path and loaded by restarting the reader. No wiki or transcript
fetching is performed by the reader itself.
## Testing and rollback
`node tools/test-compose.js --reading-limit-ui` creates a disposable Compose
fixture with two progress-qualified sample manga, tests chapter and image
denial, quiz flow and the one-hour unlock, then tears it down. Add `--keep`
to leave an isolated playable instance locked for manual review; its printed
URL and fixture path identify it. Main data and the running main scraper are
not mounted. To roll back a future main deployment, restore the previous
reader image and configuration, recreate only the reader, and preserve current
`db.json` and library. The additional `readingAccess` field is backward-safe.
+84
View File
@@ -0,0 +1,84 @@
# Active Reading Clock
Current policy: relaxed personal estimates, implemented 2026-10-06. This is not an
anti-cheat system, proof of comprehension, or a basis for competitive rewards.
Enable it in User → Reading time. Disabled means no clock requests.
## What counts
Each chapter visit measures foreground, loaded-page viewing time. After viewing
**at least 60% of eligible pages**, rounded up, the whole visit's active time counts,
including time before reaching that threshold. It continues counting through the
rest of the visit and is saved periodically and when leaving the reader.
- There is **no pages-per-second ceiling** and no skimming penalty.
- A loaded image needs **100 ms cumulative visible exposure**. Flow exposure uses
10% of the smaller of page/viewport area and samples scroll positions plus a
50 ms timer. This detects brief views; it does not assess reading speed.
- Only actually observed pages count. A scrollbar/slider jump does not credit
all pages between the starting position and destination.
- Hidden covers/extras are excluded from the denominator.
- Source versions have separate visit coverage; qualifying time combines into
existing chapter totals without changing reading identities.
- Below-60% visits remain bounded diagnostic drafts, excluded from Reading Story,
first/last-read duration, reread counts and chapter headline totals.
- New navigation starts a fresh visit. Old coverage does not qualify later picture
lookups. Separate sub-60% visits are not combined. Back/forward-cache restoration
continues the same visit.
Example: view 12 of 20 pages, then leave → all active time from that visit counts.
Jump to page 19, dwell, leave → no new reading total. Historical times are retained;
this update neither deletes old measurements nor invents times for missed reads.
Intent cannot be inferred perfectly: slowly searching most of a chapter can look
like reading. This rule filters quick-search-and-one-picture visits without
penalizing honest fast readers. Ordinary read/unread progress is independent.
## Pauses and display
Pause when hidden/unfocused; resume on reader input after returning. Pause after
**30 seconds** without activity (updated at the user's request on 2026-10-06). The first loaded page can start
without an extra click. Mouse/pen motion, wheel, touch movement, momentum scrolling
and keyboard controls—including held Up/Down arrows—refresh reader activity.
Loading wait stays separate. An unloaded neighbour no longer pauses the clock if
another visible image is readable. Failed/unloaded images do not earn coverage.
An end card pauses time. Other reader tabs/devices coordinate to avoid accidental
duplicate clocks; this is not an anti-cheat score.
The toolbar shows **this visit's** time. Its panel shows pages viewed, the 60%
requirement, whether the visit counts, saved chapter total, loading wait and
pause/save-pending state. “Verified”, “rejects” and “skimming” labels are removed.
## Reliable saves and compatibility
New clients POST cumulative snapshots to authenticated `/api/reading/visit` every
10 seconds, on attention changes, and using `sendBeacon` (keepalive fallback) on
`pagehide`. Final snapshots dispatch even with an ordinary request in flight.
Visit ID plus increasing sequence makes retries/out-of-order packets idempotent.
Delayed saves avoid the legacy 12-second packet truncation. Final saves don't
reclaim the next chapter's lease, nor do stale pulses reclaim an exited lease.
First/last duration keeps growing after qualification until departure. Late finals
update their own receipt, not a newer visit. The latest eight visit receipts are
retained per chapter; extremely old delayed requests after eviction are outside
this bounded retry window. There is no durable offline queue: closing an offline
tab or a browser/OS crash can still lose undelivered time. “Save pending” is explicit.
Legacy `/api/reading/pulse` remains for already-open old readers. Existing records
and totals are retained. Reload open readers for this policy. On code rollback the
separate new endpoint returns 404 instead of interpreting cumulative totals as
legacy deltas. Refresh open readers after rollback as well.
## Privacy and verification
Only chapter/version IDs, visit/sequence IDs, cumulative active/loading/input-type
times, viewed page indexes and exit flags are sent. No coordinates, key contents,
recordings, screenshots or raw event stream are saved. Exposure geometry stays
local. History is private to the user and existing export/reset controls remain.
`test/reading-visits.test.js` covers thresholds, historical preservation, fast
slices, lookup exclusion, attention/loading, late/duplicate snapshots and delayed
delivery. `node tools/test-compose.js --reading-ui` adds actual headless Edge
mobile-layout QA with disposable users/generated images/isolated storage. It does
not observe production readers or query image hosters.
+119
View File
@@ -0,0 +1,119 @@
# Runtime resilience and agent operations
## Ownership and failure boundaries
```text
Browser → Reader ──private API──→ Scraper → library (atomic chapter publication)
│ ↑
└──health──→ Sentinel ───┘ bounded checkpoint recovery only
Supervisor(reader) Supervisor(scraper) Supervisor(sentinel)
↑
Coding agent: inspect incidents → reproduce → patch → test → release
```
The reader is the only writer of users/progress and the catalog. The scraper is the
only queue owner and content publisher. Sentinel owns only observations and incidents.
Each service's supervisor owns only that child process. The coding agent is an
**escalation/maintenance operator**, not a necessary part of normal downloading.
The recurring Codex check is an additional backstop, not the runtime watchdog.
## Automatic recovery implemented
| Failure | Action | Automatic limit |
|---|---|---|
| Process exit or repeated failed liveness checks | Supervisor restarts only that service, with backoff | Three failures per rolling hour; then parked for operator action |
| Network/HTTP 5xx | Preserve checkpoint; open source circuit, wait five minutes, retry job | One job-level retry; request-level retries remain bounded |
| Title unavailable/no English chapters | Record unavailable, continue unrelated work | No blind retry |
| Parser/local adapter failure | Quarantine job, keep partial data, continue other titles | Two different affected jobs in an hour pause source as systemic failure |
| Storage/permission failure | Pause all downloads and report | No automatic retry |
| HTTP 401/403/429 or access challenge | Pause affected source lane | No automatic retry/bypass |
| Active job with no progress for 15 minutes | Sentinel requests cooperative checkpoint-safe retry | One stall recovery; next stall quarantined |
Sentinel only treats an active job as stalled when its checkpoint can be read.
The scraper resolves source-specific work folders (including MangaPill's
`mangapill-` prefix) before reporting progress; a missing checkpoint is not
evidence that downloads stopped. An operator retry resets the job's stall
recovery count without deleting its partial download.
The daemon remains authoritative: a recovery request includes the observed progress
fingerprint and is rejected if the job has moved, paused, or is no longer stale.
Sentinel also observes completed/page counts; a deliberately paused queue is not a
stall. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds.
Publication remains one complete chapter at a time, on the same filesystem.
No recovery action raises pacing, creates another worker for the same source,
changes identities or deletes partial images.
The publisher retains validated pages in `.staging-mcscraper-resume-<sha256>` folders
on pause/failure. It checks the full chapter/source/page-manifest fingerprint before
reuse and revalidates each image; completed chapters still publish with one rename.
A changed manifest creates separate staging rather than mixing releases. Old random
stages remain untouched for manual review: they cannot be safely associated from
page filenames alone. Hidden pending image files are never treated as completed pages.
`retry_wait`, `unavailable` and `quarantined` are explicit queue states in schema 4.
`ready` is a healthy inspected job awaiting operator download confirmation, not a
stall. Inspection shares the source lane; it never downloads images or bypasses a
source circuit. Source coverage and job completion are separate observations.
Legacy unavailable-title errors are migrated; only pauses solely attributable to
these unavailable titles are released. Access and unknown failures remain paused.
Nightly discovery avoids duplicating outstanding, unavailable or quarantined jobs;
use an explicit operator retry after fixing their cause.
## Observability
**Settings → Service health** shows Sentinel status, active checkpoints, failed jobs
and incident history. **Settings → Scraper** retains queue, pacing and discovery controls.
Service errors are classified rather than buried in a single historical `lastError`.
`npm run check` performs read-only checks. Sentinel's private, token-protected `/metrics`
exports reader-up, scraper-up and open-incident gauges for a future Prometheus collector.
It is not a public endpoint.
Scraper status is independent of expensive library scans. Concurrent admin/Sentinel
probes share one background inventory refresh and receive the previous summary
immediately. The dashboard summary refreshes at most once per minute unless a
publication or metadata change invalidates it; duplicate-sensitive decisions still
request a fresh inventory. `libraryStatus` reports ready/refreshing/stale/updated/error, and failed
background refreshes do not spawn a new scan on every UI poll. Authoritative
discovery waits for the current inventory; publication/metadata invalidation during
a scan forces another scan before it can be used for those decisions. Cold start
may temporarily show no library targets, but queue and rate controls remain usable.
Storage:
- `data/catalog.json`, `data/collection-covers/`: reader catalog and covers.
- `data/scraper/queue.json`, `events.log`: queue, circuits, failures and incidents.
- `data/sentinel/state.json`: Sentinel incidents and last observation.
- `data/runtime-reader.json`, `data/scraper/runtime.json`, `data/sentinel/runtime.json`:
supervisor status and persisted crash budget.
- `data/scraper/supervisor.lock`, `daemon.lock`: native process ownership; transient.
Scraper/Sentinel/supervisor emit structured logs. Queue and Sentinel histories retain
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows
supervisor logs remain append-only: archive them during stopped maintenance if needed.
Never export user databases, tokens, cookies, signed query strings or unredacted environments.
## What self-repair intentionally does not mean
Sentinel cannot execute a shell, patch adapters, query image sources, access a Docker
socket, or issue unrestricted restart commands. It cannot repair broken markup or
solve source challenges. The agent handles those escalations in the repository:
1. Inspect one bounded incident and relevant checkpoint; preserve the active lane.
2. Add a fixture reproducing the failure, patch the smallest adapter/logic surface.
3. Run `npm test`; test UI/API in isolated data if affected.
4. Build a traceable image, deploy checkpoint-safely, verify resumed progress.
5. Record release, backup and rollback compatibility. Stop after repeated identical failures.
A VM-local monitor cannot alert on total VM/network failure. Before VM cutover choose
an external health check and notification destination (e.g. an existing monitoring
service and private ntfy/Gotify). No notification credentials or destination are invented.
## Crash-budget operation
A parked supervisor stays alive and reports `repair-required`; Docker's restart policy
does not create an unlimited process restart storm. Its health check fails while the
child is absent. Investigate first. Once repaired, restart the wrapper after its
rolling one-hour budget has expired. For an intentional earlier reset, first stop the
service, back up its `runtime*.json`, and clear only that service's `failures` array;
never clear the queue or daemon checkpoint. Explicit deployments do not erase budgets.
+37 -2
View File
@@ -83,9 +83,13 @@
```
c<NNNN>[.<d>][ <title>][ [<Source>]] regular chapter
x<NN>[ <title>][ [<Source>]] extra chapter outside the numbering
special-mangadex-<chapter-uuid>[ <title>][ [<Source>]] unnumbered source release
```
- A **space** MUST follow the number when anything follows. `c0012_Title` sorts wrongly in file browsers; `c0012a` and `c0012Title` are not recognised at all.
- Recommended check: `^(c\d{4}(\.\d*[1-9])?|x\d{2})( [^\[\]]+?)?( \[[^\[\]]+\])?$`
- For the additive `special-mangadex-` form, validate the complete UUID using
`chapter-identity.js`. Do not apply the numeric-only check above. The key is the
lower-case source UUID prefix, not an invented numeric chapter. Books remain numeric.
- **R-C2** Pad the chapter number's integer part to 4 digits. Never change the padding of an existing manga: chapter folder names are version ids.
- **R-C3** Map upstream numbering like this:
@@ -102,6 +106,14 @@
| Volume extras (afterword, sketches) | `x<NN>` in that volume's book | |
| One-shot | its own manga (`v01/c0001`), or `x<NN>` inside an `x01 Oneshots` book of the related series | |
| Negative or non-numeric only | `x<NN>` | Never invent a number that collides with a real one |
| Unnumbered MangaDex release | `special-mangadex-<chapter-uuid>` | Stable upstream release identity; label Pilot, One-shot or Special; do not assign chapter 0/1 |
Unnumbered source-ID releases sort after numeric chapters. Their `chapter.json`
stores `special:true`, `label`, `sourceId` and the original `sourceNumber` (usually
null). They are not automatically skippable extras: an entire pilot may be story
content. Keep different release UUIDs separate rather than guessing that they are
the same logical chapter. Reimporting the same key is duplicate-safe. Existing
numeric and `x` folder conventions and progress IDs MUST NOT be migrated or renamed.
- **R-C4** The title part of the folder name is for humans only.
- The authoritative title is `chapter.json` → `title`, without the "Chapter 12:" prefix.
@@ -174,7 +186,7 @@ All are UTF-8 JSON **without** BOM, pretty-printing optional.
}
```
- `direction`:
- `direction` when an operator or reliable source explicitly provides the reading order:
| Upstream type | Value |
|---------------|-------|
@@ -186,6 +198,24 @@ All are UTF-8 JSON **without** BOM, pretty-printing optional.
- Unknown fields are kept and ignored by the app. Keep your own state in one namespaced field such as `"x-scraper"`.
- The whole file is shown to logged-in users. It MUST NOT contain secrets (tokens, cookies, credentials).
- **Read-modify-write:** keep fields you don't own, because people may have edited them. Write atomically (R-W5).
- **R-J4** The current adapters do not infer printed reading order from host,
language, country, or catalog labels. They emit `rtl` with
`"x-reading-direction":{"version":1,"value":"rtl","source":"fallback"}`.
The publisher checks already downloaded image dimensions: at least three pages
with height/width ≥ 4, at least 75% tall among known pages, consistent strip
widths within 25%, and dimensions known for at least 75% of pages yield
`direction: "ttb"` with `source: "detected"` and numeric evidence. A credits
card can coexist with the strips. This detects layout, not content or language.
Plain legacy directions and explicit `directionOverride` at any level are
preserved across later imports. Do not bulk-convert old `rtl` values: a past
scraper default and a human choice are indistinguishable. Use the reviewed
migration tool on a named series after checking representative chapters.
`node tools/review-reading-direction.js <absolute-library-manga-folder>
<weebcentral-series-id> --chapter=c0056` is a read-only review. To apply a
supported single-series correction, first pause and stop every writer and
create a stopped-writer snapshot, then add `--apply --writers-stopped`.
The tool verifies source identity, unchanged legacy RTL metadata, several
chapter samples, and the requested chapter before an atomic metadata write.
- **R-J2** `chapter.json`, one per version folder:
@@ -240,7 +270,12 @@ MangaCircus can scan at any moment. Everything it can see must always be complet
4. delete `.old-<random>`.
The chapter is briefly absent, which is harmless: it won't be re-announced, and progress is kept.
- **R-W5** Updating a JSON file: write `.<name>.tmp` in the same folder, then `rename` it over the target.
- **R-W6** At start-up, delete your own leftover `.staging-*` / `.old-*` folders older than a few hours (from crashed runs).
- **R-W6** Preserve incomplete staging folders: validated images are recovery assets,
not disposable garbage. The managed publisher resumes a hidden folder only when
its complete chapter/source/page-manifest fingerprint matches, and revalidates
stored images. Legacy random stages lack this identity and require manual review;
do not infer ownership or delete them merely because they are old. Completed
replacement `.old-*` backups may be cleaned after successful publication.
- **R-W7** On Windows, `rename` can fail with `EPERM`/`EBUSY` while another process reads the folder. Retry a few times with a short delay.
- **R-W8** Files MUST be readable by the MangaCircus process. In Docker that is uid 1000, so use mode `0644` for files and `0755` for folders.
- **R-W9** You MUST NOT write anything into MangaCircus's `DATA` folder (`db.json`, `scan-cache.json`).
+267
View File
@@ -0,0 +1,267 @@
# Managed scraper service
The scraper is an optional Node.js daemon that runs beside the MangaCircus reader.
It owns the durable queue and is the only process allowed to run a queued archive.
The browser never contacts it directly: authenticated admin routes in `server.js`
proxy commands to its private HTTP API.
## In-process source modules
There is one scraper daemon/container and one durable queue. WeebCentral and
MangaDex source modules are registered explicitly in `scraper/adapters/index.js`.
Their descriptors live in `scraper/sources/`; they call source-specific catalog
and archive code in the same process. Pure parsing functions in
`scraper/plugins/` handle source page/API shapes. No source parser has a separate
container, HTTP endpoint, runtime download, or independent queue. Only reviewed
first-party modules ship in the image; Uchiyomi packages are not executed.
The scraper's paced HTTP client fetches source content. Parsing is an ordinary
function call. The registry checks source URLs and validates parsed output before
the scraper follows links. WeebCentral catalog requests use a pinned-DNS HTTPS
transport that validates redirects and stops on access errors. Image hosts are
not restricted to a reviewed CDN allowlist yet. Existing queue state, source
delays, chapter delays, and partial-download checkpoints are unchanged by this
code organization. See [adding a source](source-modules.md) for the full path.
## Start and storage
```bash
npm run start:all # reader + daemon under one local supervisor
npm start # reader only
npm run start:scraper # daemon only
```
These commands are for native development only. The current Windows installation
runs the three-service Docker Compose stack; the old Scheduled Task is disabled.
Never start a native supervisor or daemon against its mounted state. Its observed
ports, logs, persisted settings, restart procedure, and recovery snapshot are recorded in
[`local-deployment.md`](local-deployment.md).
| Variable | Default | Purpose |
|---|---|---|
| `SCRAPER_HOST` | `127.0.0.1` | Daemon bind address |
| `SCRAPER_PORT` | `8091` | Daemon port |
| `SCRAPER_TOKEN` | empty | Bearer token; mandatory when the bind address is not loopback |
| `SCRAPER_DATA` | `<DATA>/scraper` | Durable queue, lock, and event log |
| `LIBRARY` | `./library` | Shared writable library for the daemon |
| `SCRAPER_URL` | `http://127.0.0.1:8091` | Daemon URL used by the reader |
The authoritative queue is `data/scraper/queue.json` (currently schema version 4),
written atomically. Individual
archives retain their existing resumable checkpoints under
`library/.mcscraper-work-<source-id>/job.json`. `events.log` is append-only.
The same state file also holds a `sourceLedger` keyed by adapter and source-series
ID. It keeps source leads after completed queue rows are cleared. Existing queue
URLs and recognized URLs in library metadata are backfilled without contacting a
host; a successful catalog inspection records the selected chapter count, source
edition, coverage signal, and observation time. Errors remain visible beside the
last observation until a later successful inspection. **Catalog found**, **adapter
compatible**, and **download rights** are separate states: a URL alone verifies
none of them, and download rights remain unverified by scraper inspection. The
admin-only **Settings → Scraper → Sources** tab shows this log. It is evidence for
review, not an automatic source switch or a whole-series completeness claim.
On its first start, the daemon imports the old `.mcscraper-queue/queue.json` wishlist
when present. Completed chapter ids in each archive checkpoint remain authoritative,
so migration does not redownload published chapters.
## Lifecycle
The daemon owns one durable queue and runs at most one sequential worker per source. Different sources may run side by side, but two jobs from the same source never overlap:
1. Select the first `queued` item for every idle, unpaused source.
2. Resolve its URL through the adapter registry in `scraper/adapters/index.js`.
3. Run one adapter job per source with that source's request and chapter delays.
4. For legacy in-progress checkpoints, continue publishing into their private work directory and expose the complete manga at the end.
5. For every new job, stage one chapter beside its final book and expose that complete chapter immediately with one rename.
6. Increment `publishedRevision` after every newly visible chapter and start the next item for that source when the selected scope is complete.
Legacy work can be converted without redownloading. While the queue is paused and no
worker is active, `publishLegacyReady` validates every completed chapter, quarantines
abandoned hidden partial stages, atomically exposes the accumulated manga folder, and
changes its checkpoint to incremental mode. The resumed job then derives its remaining
work from the visible library and publishes every later chapter immediately.
A source failure pauses only that source, allowing unrelated source workers to continue. Manual pause remains global. Request and chapter delays are configured independently for every installed host in **Settings → Scraper**. The current adapters default to the fast profile: 1–4 seconds between sequential requests and 5–15 seconds between chapters. The in-process request gate is shared across successive jobs on the same source, so changing titles does not reset the request gap.
The reader polls `publishedRevision` every 15 seconds and rescans after it increases.
A source error pauses only that source lane for human review. Pause and shutdown requests
are checked between requests; validated partial pages are retained in hidden,
fingerprinted staging and reused only when the complete manifest still matches.
## Inspect, confirm, and interpret completion
The admin form uses **Inspect & queue** (`inspect:true`). This queues a catalog-only
`kind:"inspection"` item on the normal sequential source lane, at the configured
delays. No image or cover requests occur during inspection. Successful inspection
sets `status:"ready"`, displays the canonical `sourceTitle` and source availability,
and waits for **Confirm download**. Ready items survive daemon restarts and suppress
duplicate nightly checks. Other jobs may proceed while confirmation is pending.
Confirmation can save an inspected edition even while its source lane is paused;
the job then remains queued and makes no source requests until that lane is
reviewed and explicitly resumed. Confirmation does not resume a deliberately
paused queue or clear a source block. **Inspect again** remains unavailable
while the source lane is paused.
The download rediscovers the catalog; an edition title change requires re-inspection
and confirmation. Direct API clients can still enqueue authorized downloads without
inspection by omitting the flag. Existing download/discovery jobs are unchanged.
**Inspect again** operates on an inactive ready, completed, unavailable or quarantined
item. It never overrides a source access block and does not fetch external-link pages.
Old MangaDex unavailable messages are marked as needing re-inspection, not silently
retried. No remote diagnosis is inferred from that old numeric-only adapter result.
If the item was a completed download, inspection compares its selected chapters with
the shared library inventory. An already-present selection remains **Complete** with
its original completion and publication counts; it does not become an awaiting-download
item. Newly available chapters instead remain **Ready** for explicit confirmation.
WeebCentral catalog inspection accepts numbered Chapter, Episode, Rating, Night,
Log and Act labels. If chapter links exist but none can be numbered, the parser
reports a label-format problem; an actually empty catalog is reported separately.
`complete` means the selected **job** finished, not that a series is fully loaded.
`publishedCount` counts chapters newly exposed by that run; `selectedCount` is its
remaining selected scope. Resume may reduce those counts because the inventory skips
chapters already published. MangaDex availability records hosted release/selection
counts, unnumbered release counts, metadata's last chapter and the highest hosted
number. A lower hosted maximum proves partial coverage; an equal maximum does **not**
prove completeness. Other cases say completeness is unverified. The last chapter
number is never used as a chapter count.
Empty English hosted catalogs distinguish no English releases listed, English
external-link-only releases, and no hosted English releases with pages. External
availability is checked through a separate, paced `includeExternalUrl=1` feed request
only when hosted selections are empty; downloads use `includeExternalUrl=0`.
## Queue scopes and duplicate policy
Every new queue item has its own UUID plus the stable upstream `seriesId`. This allows
several jobs for one manga (for example volume 2 followed by volume 3) without sharing
queue identity. Its `scope` is one of:
| Mode | Meaning |
|---|---|
| `missing` | Every eligible chapter absent from the library |
| `latest` | The newest eligible missing chapter |
| `last` / `first` | The last/first `count` eligible missing chapters |
| `chapter` | One numeric chapter |
| `after` | Every numeric chapter after the supplied chapter |
| `volume` | Chapters whose source catalog explicitly reports that volume |
| `ids` | Internal exact source chapter ids produced by nightly discovery |
Unnumbered MangaDex releases use `special-mangadex-<chapter-uuid>` as their logical
key, never an invented chapter zero or one. Numbered release deduplication is unchanged;
unnumbered releases stay distinct unless their source UUID is identical. Two pilot
translations therefore remain two labeled, group-credited releases, not a guessed
numbered chapter or silently merged translation. Numeric `chapter`/`after` scopes
exclude them; missing, exact-ID and applicable volume scopes include them. Catalog
ordering puts numbered chapters first and source-ID-ordered specials afterward;
first/last/latest operate on that ordering, not on publication timestamps.
WeebCentral whole-volume releases use `special-weebcentral-volume-<release-id>`
and a `Volume N` label rather than an invented chapter number. MangaPill is a
first-party module in the same scraper process; its catalog and chapter HTML
must remain on `mangapill.com`, and its page URLs must match the reviewed image
host. It uses the ordinary source lane, pacing, checkpoint and publication rules.
`existingPolicy:"skip-any"` skips a chapter when any version already exists.
`existingPolicy:"add-source"` permits another host's version but still skips a version
already present from the selected host. Existing version folders are never overwritten.
Before downloading, the scraper reads a lightweight library inventory. Work identity is
matched by exact source id, exact source URL, explicit library manga id, or verified
AniList id—never by fuzzy title. The filesystem library remains authoritative: a stale
checkpoint cannot make a missing published chapter look complete.
## Nightly discovery
`manga.json` stores the admin override at `x-scraper.autoUpdate`. When it is absent,
`status:"ongoing"` defaults to enabled and other statuses default to disabled. At the
configured local `autoUpdateTime`, the daemon queues one catalog-only discovery item for
the preferred supported source of every enabled manga. Discovery shares the normal
sequential source lane and pacing. If strictly newer eligible chapters are found, that
same item is promoted to a download job containing their source ids. A successful
no-change check records `lastCheckedAt` and its result in the per-manga/source
update summary, appends history, and retires the discovery queue row without page
requests. `lastDownloadedAt` is recorded when a chapter is published or a download
job completes. These summaries persist in `queue.json` as `updateChecks`; on load,
legacy completed no-change discovery rows migrate into the summary and are retired.
Failed checks remain visible for attention. Duplicate pending checks for the same
source series are suppressed.
Only numbered chapters strictly newer than the library's highest regular chapter,
plus unseen source-ID specials, are considered updates. Historical numbered gaps
are never backfilled by the nightly task; use an explicit queue
scope when a gap is intentional work.
## Private daemon API
All responses are JSON. When `SCRAPER_TOKEN` is set, send
`Authorization: Bearer <token>`.
| Method | Path | Body / result |
|---|---|---|
| `GET` | `/health` | Basic liveness |
| `GET` | `/api/state` | Queue, all active source checkpoints, settings, errors, history, adapters |
| `POST` | `/api/queue` | `{url,title?,metadataSearch?,scope?,existingPolicy?,libraryMangaId?,inspect?}` |
| `POST` | `/api/queue/remove` | `{id}`; active downloads cannot be removed |
| `POST` | `/api/queue/clear-completed` | `{source?}`; remove inactive successful download job rows only; preserve library, history, checkpoints and other statuses |
| `POST` | `/api/queue/reorder` | `{ids}` containing every queue id exactly once |
| `POST` | `/api/control` | `{action:"pause"|"resume"|"retry"|"inspect"|"confirm"|"skip",id?}`; `pause-source`/`resume-source` use `id` as the source name and do not change the global pause |
| `POST` | `/api/settings` | `sourceRates.{adapter}.{minMs,maxMs,chapterMinMs,chapterMaxMs}` plus `autoUpdateEnabled` and local `autoUpdateTime` (`HH:MM`) |
| `POST` | `/api/discovery/run` | Queue an immediate update check for every enabled manga |
| `POST` | `/api/library/auto-update` | `{id,enabled}` updates `manga.json` atomically |
| `POST` | `/api/legacy/publish-ready` | Paused-only validation/publication of completed legacy work; converts its checkpoint to incremental mode |
Legacy top-level rate fields remain a fallback for adapters without a per-source entry. Settings also accept `autoUpdateEnabled` and a local `autoUpdateTime` in `HH:MM` form.
The reader exposes only `GET/POST /api/admin/scraper`, protected by the existing
admin session. It maps a small command allowlist to the private API and never returns
the shared token.
## Adapter contract
Each first-party source module is registered in `scraper/adapters/index.js` with:
```js
{
id: 'host-id',
label: 'Human name',
match(url) { /* return {source,id,title,url} or null */ },
async discover(item, { settings, shouldStop, onEvent }) { /* catalog only */ },
async run(item, {
libraryRoot, settings, shouldStop, onPublished, onCatalog
}) { /* publish selected chapters and return counts/final path */ }
}
```
Adapters discover host-specific catalogs and pages, but reuse the central pacing,
checkpoint, metadata, and atomic publisher modules. They must stop on HTTP 401, 403,
429, CAPTCHA, challenge, or access-denied responses. Never add protection bypasses.
`onCatalog(catalog)` persists source title and optional availability before any page
acquisition. The MangaDex adapter accepts only explicit title UUID URLs, verifies
AniList cross-references, selects one English release per numeric chapter, preserves
unnumbered source identities, credits groups and downloads sequential full-resolution
MangaDex@Home pages. Special-only editions do not use fuzzy AniList lookup or AniList
fallback library matching; their canonical source title names new folders. Explicit
library targets remain operator choices; existing folders are never renamed.
## Admin UI
**Settings → Admin → Scraper** polls the authenticated proxy every three seconds
only while that section is active. Activity, Queue, Updates and Sources are
separate local tabs with a shared source filter. The section shows:
- every active source, manga, book/season, chapter, page, and overall progress;
- source warnings and fatal errors;
- queue ordering and removal controls;
- add-by-URL, pause, resume, and retry controls;
- queue scope, exact chapter/count/volume, target manga, and alternate-source controls;
- per-manga auto-update checkboxes, nightly time, and a manual **Check now** action;
- last checked and last downloaded times per manga/source, with old check events
collapsed into recent history instead of occupying queue space;
- a collapsed completed-jobs section with confirmation before bulk cleanup;
- persistent image and chapter delay ranges.
Rate changes apply on the next manga. To apply them to the active manga, pause and
resume it; the current chapter restarts safely if it was incomplete.
+81
View File
@@ -0,0 +1,81 @@
# Settings workspace
The settings page has two menu levels: ownership (User/Admin), then a focused
section. Only one section is shown at a time. Scraper tabs are local controls,
not a third global menu level.
```text
Settings
User
Reading
Reading time & privacy
Notifications
Account
Admin
Scraper
[Activity | Queue | Updates | Sources]
Server & library
Users & access
Collections & metadata
Service health
```
Desktop uses a sticky sidebar; mobile uses a wrapping menu above the content.
Active sections have a visual marker and `aria-current`. Navigation uses real
links, supports keyboard use and browser back/forward, and focuses the section
heading when switching within Settings.
## Routes and ownership
Personal routes: `settings.html#user/reading`, `#user/reading-time`,
`#user/notifications`, `#user/account`.
Admin routes: `settings.html#admin/scraper`, `#admin/library`, `#admin/users`.
Scraper deep links append `/activity`, `/queue`, `/updates` or `/sources`.
The old `/hosters` link opens Sources. The scraper default is Activity.
Existing `collections.html` and `health.html` retain their URLs but share the
same menu. `#admin/collections` and `#admin/health` redirect to those pages.
Unknown routes default to Reading. Non-admin accounts never render the Admin menu
or load admin data from Settings; admin hash routes normalize to Reading.
Server-side authorization remains authoritative and still rejects non-admin API calls.
`public/settings-nav.js` defines the shared menu and route model. `settings.js`
preserves personal form nodes across section changes and loads admin sections lazily.
Scraper polling runs only while its section is active. Draft inputs survive polling
and tab switches; successful commands reload state. Command failures are visible
without deleting the user's draft. Source chips filter the four scraper views
without changing queue ownership or per-source pacing.
The Add manga, Completed downloads and Recent check activity disclosure boxes
also retain their open/closed state across polling and command redraws; a page
reload still starts them collapsed.
Activity shows active jobs, source attention, and the next queued items. Queue
keeps reorder/inspect/confirm/remove controls, scopes, and an Add manga form;
completed jobs are collapsed by default. **Clear completed downloads** removes
only inactive, successful download job rows, optionally limited to the selected
source. It never deletes library content, checkpoints, history or other statuses.
Updates shows the nightly schedule, per-manga switches, the latest check and
download times by manga/source, and a collapsed recent-check log. Successful
no-change nightly checks update this summary and leave the queue. Sources owns
the per-hoster request and chapter delay settings.
Reading-time collection controls, export and deletion live under Reading time &
privacy; reader layout controls stay under Reading. Rescan controls and user
administration are separate. Server & library describes existing maintenance,
not an editor for Docker environment variables or an unrestricted restart console.
## Verification
`npm test` covers route defaults, deep links, ownership and existing service/API
behavior. `node tools/test-compose.js --settings-ui` adds optional Playwright
headless browser QA against the disposable Compose fixture, never the real user
database or queue. Make Playwright available on the development machine; Windows
uses installed headless Edge by default, overridable with `QA_BROWSER_CHANNEL`.
The application itself remains dependency-free.
The UI check verifies desktop and 390×844 mobile navigation, form saves, draft
retention, tab visibility, back/forward, stopping hidden scraper polling, shared
collections/health navigation, non-admin denial and horizontal overflow.
`--queue-ui` additionally checks the source filter, edition confirmation, add form,
completed cleanup and mobile layout. Screenshots are generated under ignored
`data/qa/settings/` and `data/qa/queue/`. No image sources are contacted.
+42
View File
@@ -0,0 +1,42 @@
# Adding a source to the single scraper
MangaCircus uses **one scraper process**. A source module is reviewed first-party
code bundled into that process, not a container or a downloaded extension.
`scraper/adapters/index.js` is the explicit registration point. The library,
queue, pacing settings, checkpointing, source-error policy, and publication
remain shared infrastructure. Do not add another queue writer for a new hoster.
## Source boundary
| Concern | Location | New-source responsibility |
| --- | --- | --- |
| URL recognition and adapter contract | `scraper/sources/<hoster>.js` | `id`, `label`, `match`, `discover`, `run` |
| Catalog and chapter discovery | `scraper/adapters/<hoster>.js` | Map source responses to stable chapter IDs, metadata and page URLs |
| Source HTML/API parsing | `scraper/plugins/<hoster>.js` | Pure functions; no network or filesystem writes |
| Parser operation/URL validation | `scraper/plugins/manifest.js`, `registry.js`, `validation.js` | Allow only known operations and expected hosts/links |
| Resumable archive workflow | `scraper/jobs/<hoster>-archive.js` | Use the common publisher, inventory, pacing and checkpoints |
| Registration | `scraper/adapters/index.js` | Add one explicit `require` to `adapters` |
The module contract is shown in [scraper-service.md](scraper-service.md#adapter-contract).
Start from the closest existing hoster. `match(url)` returns `null` for unrelated
URLs and `{ source, id, title, url }` for an exact supported series URL. `discover`
must be read-only and return a catalog. `run` must honor scope, existing-content
policy, `shouldStop`, source-specific delays, and publication callbacks. Stable
source series/chapter IDs are essential for retries and duplicate detection.
## Before enabling a hoster
1. Document its permitted access method, terms and source-specific pacing. Do
not work around 401, 403, 429, CAPTCHA, challenges or source protections.
2. Add fixture-backed tests for URL matching, catalog, empty/partial catalogs,
chapter pages, unsafe links, and duplicate release handling. No test should
contact the live hoster by default.
3. Test pause/resume from a partial chapter and an existing library target with
the common publisher; verify one source lane and no parallel image workers.
4. Run `npm run test:scraper` and an isolated Compose smoke test. Review and deploy
the single scraper image with a checkpoint-safe procedure; never start another
daemon against the live queue.
Removing a hoster from registration stops new matches; it does **not** delete its
library chapters or queued/checkpointed work. Inspect affected queue items and
preserve checkpoints before disabling a source in a release.
+2202
View File
@@ -0,0 +1,2202 @@
{
"schema": 1,
"updatedAt": "2026-10-09",
"title": "Shijō Saikō no Manga",
"periods": [
"Before 1970",
"1970–1974",
"1975–1979",
"1980–1984",
"1985–1989",
"1990–1994",
"1995–1999",
"2000–2004",
"2005–2009",
"2010–2014",
"2015–2019",
"2020–2024",
"2025–2026"
],
"entries": [
{
"id": "astro-boy-1952",
"title": "Astro Boy",
"year": 1952,
"period": "Before 1970",
"periodOrder": 0,
"rank": 1,
"note": "A foundational postwar story manga and one of Tezuka's defining works. It belongs here because its humane robot science fiction helped establish the medium's long-form adventure vocabulary.",
"source": {
"kind": "edition",
"url": "https://www.darkhorse.com/Books/25-825/Astro-Boy-Omnibus-Volume-1-TPB"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30728-deU0SZKHwSp5.png",
"chapters": 233,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30728,
"url": "https://anilist.co/manga/30728"
}
},
{
"id": "phoenix-1954",
"title": "Phoenix",
"year": 1954,
"period": "Before 1970",
"periodOrder": 0,
"rank": 2,
"note": "Tezuka's decades-spanning life work is here for its ambition: it treats manga as a place for a single creator to wrestle with life, death, and history on an epic scale.",
"source": {
"kind": "info",
"url": "https://tezukaosamu.net/en/manga/656.html"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/b109836-NwYxXNYLkjxQ.jpg",
"chapters": null,
"status": "unknown"
},
{
"id": "ashita-no-joe-1968",
"title": "Ashita no Joe",
"year": 1968,
"period": "Before 1970",
"periodOrder": 0,
"rank": 3,
"note": "A defining sports-manga cultural touchstone in Japan. Its tragic intensity helped prove that a popular boxing serial could carry class conflict and a lasting national memory.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/ashita-no-joe-fighting-for-tomorrow/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31303-l21HJn9AXl7w.jpg",
"chapters": 171,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31303,
"url": "https://anilist.co/manga/31303"
}
},
{
"id": "princess-knight-1953",
"title": "Princess Knight",
"year": 1953,
"period": "Before 1970",
"periodOrder": 0,
"rank": 4,
"note": "Tezuka's official archive calls this the first Japanese story manga for girls. It earns a place as a foundational shōjo landmark, even though its gender ideas reflect its era.",
"source": {
"kind": "info",
"url": "https://tezukaosamu.net/en/manga/559.html"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/b31676-ReKRBqwbdml1.jpg",
"chapters": 27,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31676,
"url": "https://anilist.co/manga/31676"
}
},
{
"id": "golgo-13-1968",
"title": "Golgo 13",
"year": 1968,
"period": "Before 1970",
"periodOrder": 0,
"rank": 5,
"note": "A singular feat of serialization: Guinness recorded 201 volumes in 2021. Its longevity makes the hard-boiled assassin series a changing political and pop-cultural time capsule.",
"source": {
"kind": "read",
"url": "https://www.viz.com/vizmanga/chapters/golgo-13"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx31298-Cik2Ff9Ip3MD.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 31298,
"url": "https://anilist.co/manga/31298"
}
},
{
"id": "dororo-1967",
"title": "Dororo",
"year": 1967,
"period": "Before 1970",
"periodOrder": 0,
"rank": 6,
"note": "An early dark-fantasy milestone from Tezuka. It is included as a historical ancestor of the body-horror quests and morally bruised heroes you already favor.",
"source": {
"kind": "info",
"url": "https://tezukaosamu.net/en/manga/310.html"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31658-xZFgMcIzhbB4.png",
"chapters": 19,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31658,
"url": "https://anilist.co/manga/31658"
}
},
{
"id": "the-legend-of-kamui-kamui-den-1964",
"title": "The Legend of Kamui / Kamui Den",
"year": 1964,
"period": "Before 1970",
"periodOrder": 0,
"rank": 7,
"note": "A politically engaged historical-manga landmark. It broadens this list beyond ninja spectacle by making caste and rebellion central to the form's adult ambitions.",
"source": {
"kind": "edition",
"url": "https://drawnandquarterly.com/books/the-legend-of-kamui/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx47839-3sf0J3dOnN2O.png",
"chapters": 74,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 47839,
"url": "https://anilist.co/manga/47839"
}
},
{
"id": "kitaro-gegege-no-kitaro-1960",
"title": "Kitarō / GeGeGe no Kitarō",
"year": 1960,
"period": "Before 1970",
"periodOrder": 0,
"rank": 8,
"note": "Mizuki's Kitarō made Japanese yōkai folklore a durable pop-manga world. It belongs here for the cultural lineage that later supernatural and horror manga keep drawing on.",
"source": {
"kind": "edition",
"url": "https://drawnandquarterly.com/books/kitaros-yokai-battles/"
},
"cover": "https://drawnandquarterly.com/wp-content/uploads/biblioshare_attachments/9781770463196.png",
"chapters": null,
"status": "unknown"
},
{
"id": "lone-wolf-and-cub-1970",
"title": "Lone Wolf and Cub",
"year": 1970,
"period": "1970–1974",
"periodOrder": 1,
"rank": 1,
"note": "A benchmark for cinematic samurai storytelling in manga. Its austere page rhythm and long-form father–son journey make it essential to the adult-action lineage.",
"source": {
"kind": "edition",
"url": "https://www.darkhorse.com/Books/40-092/Lone-Wolf-and-Cub-Vol-1-The-Assassins-Road-TPB"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30904-KRN1WE3ZU3vl.jpg",
"chapters": 142,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30904,
"url": "https://anilist.co/manga/30904"
}
},
{
"id": "the-rose-of-versailles-1972",
"title": "The Rose of Versailles",
"year": 1972,
"period": "1970–1974",
"periodOrder": 1,
"rank": 2,
"note": "A shōjo landmark that made historical politics, gender roles, and grand tragedy feel central to a mass-audience girls' comic, expanding the genre's possibilities.",
"source": {
"kind": "edition",
"url": "https://store.udonentertainment.com/collections/the-rose-of-versailles"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30660-hBFkOt2Ab2Ti.jpg",
"chapters": 85,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30660,
"url": "https://anilist.co/manga/30660"
}
},
{
"id": "devilman-1972",
"title": "Devilman",
"year": 1972,
"period": "1970–1974",
"periodOrder": 1,
"rank": 3,
"note": "A hugely influential apocalyptic-horror turning point. Its moral collapse and uncompromising ending make it a key ancestor of the darker fantasy and action manga on your shelf.",
"source": {
"kind": "edition",
"url": "https://sevenseasentertainment.com/series/devilman-the-classic-collection/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31474-R4x7qkjUfBw6.jpg",
"chapters": 53,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31474,
"url": "https://anilist.co/manga/31474"
}
},
{
"id": "buddha-1972",
"title": "Buddha",
"year": 1972,
"period": "1970–1974",
"periodOrder": 1,
"rank": 4,
"note": "A prestige-scale Tezuka epic that shows how far popular manga could stretch toward biography and philosophy without losing its emotional directness.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/buddha/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31111-ldPYMNd3PvHa.png",
"chapters": 69,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31111,
"url": "https://anilist.co/manga/31111"
}
},
{
"id": "black-jack-1973",
"title": "Black Jack",
"year": 1973,
"period": "1970–1974",
"periodOrder": 1,
"rank": 5,
"note": "Winner of an inaugural 1977 Kodansha Manga Award. Its enduring case-of-the-week moral dilemmas show why Tezuka's medical drama became a mainstream classic.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/black-jack/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx31600-CCA5TUKJBQJh.jpg",
"chapters": 243,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31600,
"url": "https://anilist.co/manga/31600"
}
},
{
"id": "barefoot-gen-1973",
"title": "Barefoot Gen",
"year": 1973,
"period": "1970–1974",
"periodOrder": 1,
"rank": 6,
"note": "An essential act of Hiroshima survivor testimony in comics. Its place is about historical memory and manga's power as witness, not just entertainment value.",
"source": {
"kind": "edition",
"url": "https://lastgasp.com/collections/barefoot-gen"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx33281-2NXB8uHRxpI3.png",
"chapters": 54,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 33281,
"url": "https://anilist.co/manga/33281"
}
},
{
"id": "the-poe-clan-1972",
"title": "The Poe Clan",
"year": 1972,
"period": "1970–1974",
"periodOrder": 1,
"rank": 7,
"note": "A Moto Hagio shōjo milestone that pushed the form toward psychological, nonlinear, and gothic storytelling, widening what later creators could attempt.",
"source": {
"kind": "edition",
"url": "https://www.fantagraphics.com/products/the-complete-poe-clan-vols-1-2"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx34305-ul28wvP0lhjv.jpg",
"chapters": 26,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 34305,
"url": "https://anilist.co/manga/34305"
}
},
{
"id": "the-heart-of-thomas-1974",
"title": "The Heart of Thomas",
"year": 1974,
"period": "1970–1974",
"periodOrder": 1,
"rank": 8,
"note": "A formative work in the genealogy of boys'-love and psychological shōjo manga. It belongs for the emotional and formal territory it opened, not just its tragic premise.",
"source": {
"kind": "edition",
"url": "https://www.fantagraphics.com/products/heart-of-thomas"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx39152-QcXrecYJwsEQ.jpg",
"chapters": 40,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 39152,
"url": "https://anilist.co/manga/39152"
}
},
{
"id": "the-glass-mask-1975",
"title": "The Glass Mask",
"year": 1975,
"period": "1975–1979",
"periodOrder": 2,
"rank": 1,
"note": "A long-running shōjo institution and the acting-manga benchmark. It shows how a single rivalry can sustain character growth and reader investment across decades.",
"source": null,
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31469-xYQ8kWQynM7n.png",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 31469,
"url": "https://anilist.co/manga/31469"
}
},
{
"id": "galaxy-express-999-1977",
"title": "Galaxy Express 999",
"year": 1977,
"period": "1975–1979",
"periodOrder": 2,
"rank": 2,
"note": "One of Leiji Matsumoto's signature space-opera worlds. It earns its slot for making melancholy, mortality, and travel as defining to manga science fiction as machinery and battles.",
"source": null,
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx39705-uxZAOISKjKgB.jpg",
"chapters": 91,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 39705,
"url": "https://anilist.co/manga/39705"
}
},
{
"id": "urusei-yatsura-1978",
"title": "Urusei Yatsura",
"year": 1978,
"period": "1975–1979",
"periodOrder": 2,
"rank": 3,
"note": "Rumiko Takahashi's breakthrough serial became a template for chaotic ensemble rom-coms. Its influence reaches far beyond its original alien-invader gag.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/urusei-yatsura"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30304-jAQt0fJkMgUd.jpg",
"chapters": 366,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30304,
"url": "https://anilist.co/manga/30304"
}
},
{
"id": "captain-harlock-1977",
"title": "Captain Harlock",
"year": 1977,
"period": "1975–1979",
"periodOrder": 2,
"rank": 4,
"note": "A lasting icon of late-1970s manga and anime science fiction. Harlock's silhouette and romantic rebel ethos give this era a very different kind of hero.",
"source": {
"kind": "edition",
"url": "https://sevenseasentertainment.com/series/captain-harlock-the-classic-collection/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx35115-fZzfIJXnDYds.jpg",
"chapters": 55,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 35115,
"url": "https://anilist.co/manga/35115"
}
},
{
"id": "candy-candy-1975",
"title": "Candy Candy",
"year": 1975,
"period": "1975–1979",
"periodOrder": 2,
"rank": 5,
"note": "A 1977 Kodansha Manga Award winner and cross-generational shōjo phenomenon. It represents the popular emotional reach of 1970s girls' manga.",
"source": null,
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx31388-JDIwg74gvgt3.jpg",
"chapters": 44,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31388,
"url": "https://anilist.co/manga/31388"
}
},
{
"id": "the-poem-of-wind-and-trees-1976",
"title": "The Poem of Wind and Trees",
"year": 1976,
"period": "1975–1979",
"periodOrder": 2,
"rank": 6,
"note": "A seminal early boys'-love work whose emotional severity helped widen the psychological and thematic range of shōjo manga.",
"source": null,
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31365-ucezaL74sIgK.jpg",
"chapters": 139,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31365,
"url": "https://anilist.co/manga/31365"
}
},
{
"id": "kochikame-1976",
"title": "Kochikame",
"year": 1976,
"period": "1975–1979",
"periodOrder": 2,
"rank": 7,
"note": "A four-decade comedy institution. Its extraordinary run turns everyday police-station jokes into a record of changing Japanese life and publishing culture.",
"source": null,
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx30733-QmyPwBhjbgyX.jpg",
"chapters": 1968,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30733,
"url": "https://anilist.co/manga/30733"
}
},
{
"id": "akira-1982",
"title": "Akira",
"year": 1982,
"period": "1980–1984",
"periodOrder": 3,
"rank": 1,
"note": "A 1984 Kodansha Manga Award winner and an international cyberpunk landmark. Ōtomo's urban scale and page craft make it a direct precursor to the dystopian SF you collect.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/akira/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30664-JJq9tosQvlpW.jpg",
"chapters": 120,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30664,
"url": "https://anilist.co/manga/30664"
}
},
{
"id": "nausicaa-of-the-valley-of-the-wind-1982",
"title": "Nausicaä of the Valley of the Wind",
"year": 1982,
"period": "1980–1984",
"periodOrder": 3,
"rank": 2,
"note": "Miyazaki's manga continued far beyond the film, making it a major auteur science-fiction work in its own right. Its ecological scale earns it a place apart from the adaptation.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/manga-books/manga/nausicaa-of-the-valley-of-the-wind/all"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30651-xLufeLnDPqpn.jpg",
"chapters": null,
"status": "unknown"
},
{
"id": "dragon-ball-1984",
"title": "Dragon Ball",
"year": 1984,
"period": "1980–1984",
"periodOrder": 3,
"rank": 3,
"note": "The template that generations of battle shōnen followed. Toriyama's readable motion, training arcs, and escalating rivals changed the genre's global vocabulary.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100011"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30042-4SetGiEbGc9x.jpg",
"chapters": 520,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30042,
"url": "https://anilist.co/manga/30042"
}
},
{
"id": "fist-of-the-north-star-1983",
"title": "Fist of the North Star",
"year": 1983,
"period": "1980–1984",
"periodOrder": 3,
"rank": 4,
"note": "An action-manga icon whose post-apocalyptic martial arts and larger-than-life visual language became reference points for later fighters and dark-fantasy heroes.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/fist-of-the-north-star"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31149-TOwqxOlY1Zs0.jpg",
"chapters": 245,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31149,
"url": "https://anilist.co/manga/31149"
}
},
{
"id": "maison-ikkoku-1980",
"title": "Maison Ikkoku",
"year": 1980,
"period": "1980–1984",
"periodOrder": 3,
"rank": 5,
"note": "A Rumiko Takahashi long-form romance benchmark. It belongs here as proof that patient adult character change could sustain a popular comedy serial.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/maison-ikkoku"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30688-vhFGM9jUcGrU.png",
"chapters": 162,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30688,
"url": "https://anilist.co/manga/30688"
}
},
{
"id": "touch-1981",
"title": "Touch",
"year": 1981,
"period": "1980–1984",
"periodOrder": 3,
"rank": 6,
"note": "A Mitsuru Adachi sports-romance touchstone. Its restraint and emotional timing offer an influential alternative to louder victory-driven sports manga.",
"source": null,
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31013-Suolsv3rhJnA.jpg",
"chapters": 257,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31013,
"url": "https://anilist.co/manga/31013"
}
},
{
"id": "captain-tsubasa-1981",
"title": "Captain Tsubasa",
"year": 1981,
"period": "1980–1984",
"periodOrder": 3,
"rank": 7,
"note": "A foundational football manga with worldwide reach. It helped make the sport's ambitions and signature on-field spectacle a lasting part of manga culture.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/200014"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31789-qiNlHJDMJncz.png",
"chapters": 356,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31789,
"url": "https://anilist.co/manga/31789"
}
},
{
"id": "dr-slump-1980",
"title": "Dr. Slump",
"year": 1980,
"period": "1980–1984",
"periodOrder": 3,
"rank": 8,
"note": "Toriyama's breakout hit belongs beside Dragon Ball: it established his elastic comic timing and visual invention before he redefined battle manga.",
"source": {
"kind": "read",
"url": "https://www.viz.com/shonenjump/chapters/dr-slump"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30796-PFF4n7Y2spz9.png",
"chapters": 236,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30796,
"url": "https://anilist.co/manga/30796"
}
},
{
"id": "berserk-1989",
"title": "Berserk",
"year": 1989,
"period": "1985–1989",
"periodOrder": 4,
"rank": 1,
"note": "A modern dark-fantasy benchmark for detailed art, sustained ambition, and emotional brutality. It is also the clearest anchor for your collection's darker tastes.",
"source": {
"kind": "edition",
"url": "https://www.darkhorse.com/books/12-705/berserk-volume-1-tpb/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30002-Cul4OeN7bYtn.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 30002,
"url": "https://anilist.co/manga/30002"
}
},
{
"id": "jojo-s-bizarre-adventure-1987",
"title": "JoJo's Bizarre Adventure",
"year": 1987,
"period": "1985–1989",
"periodOrder": 4,
"rank": 2,
"note": "Few long-running series reinvent themselves so radically. Its shifting casts, powers, fashion, and poses made JoJo a cultural and visual reference point across decades.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/jojo-s-bizarre-adventure"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31630-Fg8JERVKuWB5.jpg",
"chapters": 69,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31630,
"url": "https://anilist.co/manga/31630"
}
},
{
"id": "parasyte-1989",
"title": "Parasyte",
"year": 1989,
"period": "1985–1989",
"periodOrder": 4,
"rank": 3,
"note": "An enduring body-horror classic whose ecological and ethical questions keep it more unsettling than a creature-feature premise. It is already a strong match to your shelf.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/parasyte/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30401-siY6wmwnHFi6.png",
"chapters": 64,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30401,
"url": "https://anilist.co/manga/30401"
}
},
{
"id": "the-ghost-in-the-shell-1989",
"title": "The Ghost in the Shell",
"year": 1989,
"period": "1985–1989",
"periodOrder": 4,
"rank": 4,
"note": "A cyberpunk touchstone whose questions about networks, identity, and the body traveled far beyond manga. It is a natural historical companion to BLAME!.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/the-ghost-in-the-shell/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31023-iozeApEMjmPP.jpg",
"chapters": 11,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31023,
"url": "https://anilist.co/manga/31023"
}
},
{
"id": "banana-fish-1985",
"title": "Banana Fish",
"year": 1985,
"period": "1985–1989",
"periodOrder": 4,
"rank": 5,
"note": "A landmark shōjo crime epic that refused a narrow idea of what girls' manga could contain. Its intense relationship drama crossed genre and demographic expectations.",
"source": {
"kind": "read",
"url": "https://www.viz.com/vizmanga/chapters/banana-fish"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30756-rlxjQWBAizHX.jpg",
"chapters": 110,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30756,
"url": "https://anilist.co/manga/30756"
}
},
{
"id": "ranma-1-2-1987",
"title": "Ranma ½",
"year": 1987,
"period": "1985–1989",
"periodOrder": 4,
"rank": 6,
"note": "A Rumiko Takahashi export-era classic. Its gender-bending martial-arts farce helped make fast, ensemble-driven manga comedy recognizable worldwide.",
"source": {
"kind": "read",
"url": "https://www.viz.com/vizmanga/chapters/ranma"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30023-NWoD28ZKC73c.jpg",
"chapters": 407,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30023,
"url": "https://anilist.co/manga/30023"
}
},
{
"id": "city-hunter-1985",
"title": "City Hunter",
"year": 1985,
"period": "1985–1989",
"periodOrder": 4,
"rank": 7,
"note": "A late-1980s urban action-comedy emblem. Its mix of hard-boiled city style and broad slapstick gives this period a pop-culture pillar beyond fantasy battles.",
"source": null,
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30792-62RxEnU7tlDg.png",
"chapters": 336,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30792,
"url": "https://anilist.co/manga/30792"
}
},
{
"id": "saint-seiya-1985",
"title": "Saint Seiya",
"year": 1985,
"period": "1985–1989",
"periodOrder": 4,
"rank": 8,
"note": "Its mythic armor and team-battle designs became internationally recognizable shōnen iconography, especially through manga-and-anime fandom outside Japan.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/knights-of-the-zodiac-saint-seiya"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31045-YUcWBMk7RpeK.png",
"chapters": 249,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 31045,
"url": "https://anilist.co/manga/31045"
}
},
{
"id": "slam-dunk-1990",
"title": "Slam Dunk",
"year": 1990,
"period": "1990–1994",
"periodOrder": 5,
"rank": 1,
"note": "A basketball-manga and sports-storytelling benchmark. Inoue's readable games and earned character growth helped make the series a durable cultural reference point.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/slam-dunk"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30051-5KJyPlO7z5F4.png",
"chapters": 276,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30051,
"url": "https://anilist.co/manga/30051"
}
},
{
"id": "monster-1994",
"title": "Monster",
"year": 1994,
"period": "1990–1994",
"periodOrder": 5,
"rank": 2,
"note": "A benchmark for long-form manga suspense. Urasawa's humane, intricate plotting is a top-priority bridge from your existing mysteries to the wider adult-manga canon.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/monster"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30001-Knby7l1jevE7.jpg",
"chapters": 162,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30001,
"url": "https://anilist.co/manga/30001"
}
},
{
"id": "sailor-moon-1991",
"title": "Sailor Moon",
"year": 1991,
"period": "1990–1994",
"periodOrder": 5,
"rank": 3,
"note": "A global magical-girl touchstone. Its team transformations, style, and romantic stakes helped define how later generations imagine the genre.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/pretty-guardian-sailor-moon/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30092-tjzz7ymlFfQu.png",
"chapters": 61,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30092,
"url": "https://anilist.co/manga/30092"
}
},
{
"id": "yu-yu-hakusho-1990",
"title": "Yu Yu Hakusho",
"year": 1990,
"period": "1990–1994",
"periodOrder": 5,
"rank": 4,
"note": "A 1990s supernatural-battle touchstone and the clearest path into Togashi's later Hunter × Hunter. Its lively cast kept tournament manga character-led.",
"source": {
"kind": "read",
"url": "https://www.viz.com/shonenjump/chapters/yuyu-hakusho"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30053-wCR6xyGzeUYo.png",
"chapters": 176,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30053,
"url": "https://anilist.co/manga/30053"
}
},
{
"id": "blade-of-the-immortal-1993",
"title": "Blade of the Immortal",
"year": 1993,
"period": "1990–1994",
"periodOrder": 5,
"rank": 5,
"note": "A prestige samurai-manga pick for Samura's unmistakable ink work and morally corroded revenge story, extending the adult-action canon beyond lone-hero spectacle.",
"source": {
"kind": "publisher catalog",
"url": "https://images.darkhorse.com/libraries/downloads/LIBCON25_ESSENTIALS.pdf"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/b30658-4KzCEiCDN7Yq.jpg",
"chapters": 219,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30658,
"url": "https://anilist.co/manga/30658"
}
},
{
"id": "detective-conan-case-closed-1994",
"title": "Detective Conan / Case Closed",
"year": 1994,
"period": "1990–1994",
"periodOrder": 5,
"rank": 6,
"note": "A rare decades-long mystery institution. It deserves a place for the sheer durability of its puzzle format, even if sampling arcs is wiser than chasing completion.",
"source": {
"kind": "read",
"url": "https://www.viz.com/vizmanga/chapters/case-closed"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31061-SJQb7931MG7m.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 31061,
"url": "https://anilist.co/manga/31061"
}
},
{
"id": "basara-1990",
"title": "Basara",
"year": 1990,
"period": "1990–1994",
"periodOrder": 5,
"rank": 7,
"note": "A shōjo epic that crosses revolutionary politics, romance, and post-apocalyptic adventure. It is here to keep the historical action canon from becoming male-only.",
"source": {
"kind": "read",
"url": "https://www.viz.com/vizmanga/chapters/basara"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30974-Ck0pncZ7LEnW.png",
"chapters": 107,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30974,
"url": "https://anilist.co/manga/30974"
}
},
{
"id": "rurouni-kenshin-1994",
"title": "Rurouni Kenshin",
"year": 1994,
"period": "1990–1994",
"periodOrder": 5,
"rank": 8,
"note": "A major 1990s shōnen sword-action touchstone. Its Meiji setting and violence-versus-repentance question give this era a distinct historical-action strand.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100023"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30022-jxTlViun1o10.jpg",
"chapters": 259,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30022,
"url": "https://anilist.co/manga/30022"
}
},
{
"id": "one-piece-1997",
"title": "One Piece",
"year": 1997,
"period": "1995–1999",
"periodOrder": 6,
"rank": 1,
"note": "A record-setting global adventure phenomenon whose world-building and long emotional payoffs shaped how readers think about serial manga. It is already central to your library.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100020"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30013-BeslEMqiPhlk.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 30013,
"url": "https://anilist.co/manga/30013"
}
},
{
"id": "vagabond-1998",
"title": "Vagabond",
"year": 1998,
"period": "1995–1999",
"periodOrder": 6,
"rank": 2,
"note": "An art-and-craft benchmark for adult historical manga. Inoue's brushwork and inward-looking swordsmanship make it a strong Berserk-adjacent priority.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/vagabond"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30656-9mW113O7rDnA.png",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 30656,
"url": "https://anilist.co/manga/30656"
}
},
{
"id": "hunter-hunter-1998",
"title": "Hunter × Hunter",
"year": 1998,
"period": "1995–1999",
"periodOrder": 6,
"rank": 3,
"note": "A battle-shōnen benchmark that keeps changing its own rules. Its strategic fights and moral uncertainty reward readers who want more than power escalation.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100015"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30026-uCvXMudMzmwI.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 30026,
"url": "https://anilist.co/manga/30026"
}
},
{
"id": "20th-century-boys-1999",
"title": "20th Century Boys",
"year": 1999,
"period": "1995–1999",
"periodOrder": 6,
"rank": 4,
"note": "Urasawa's intergenerational conspiracy is a second pillar of modern long-form manga suspense. It belongs beside Monster as a test of his range.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/naoki-urasawa-s-20th-century-boys"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30003-E84fwIh22LAQ.jpg",
"chapters": 249,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30003,
"url": "https://anilist.co/manga/30003"
}
},
{
"id": "blame-1997",
"title": "BLAME!",
"year": 1997,
"period": "1995–1999",
"periodOrder": 6,
"rank": 5,
"note": "A cult cyberpunk touchstone whose vast architecture and near-wordless storytelling show what manga can do without explanatory dialogue. You already collect it.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/blame/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30149-YetcNrfacNVj.jpg",
"chapters": 66,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30149,
"url": "https://anilist.co/manga/30149"
}
},
{
"id": "uzumaki-1998",
"title": "Uzumaki",
"year": 1998,
"period": "1995–1999",
"periodOrder": 6,
"rank": 6,
"note": "One of Junji Ito's signature horror works. Its page-level visual invention makes it a strong example of dread that depends on the manga form itself.",
"source": {
"kind": "read",
"url": "https://www.viz.com/vizmanga/chapters/uzumaki"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30436-Opk9ZubMPuDU.jpg",
"chapters": 20,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30436,
"url": "https://anilist.co/manga/30436"
}
},
{
"id": "fruits-basket-1998",
"title": "Fruits Basket",
"year": 1998,
"period": "1995–1999",
"periodOrder": 6,
"rank": 7,
"note": "A modern shōjo touchstone whose long trauma-and-healing arc shows the genre's emotional range and sustained character payoff.",
"source": {
"kind": "edition",
"url": "https://yenpress.com/series/fruits-basket-collector-s-edition"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30102-MJuQ0e0k2CgU.png",
"chapters": 137,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30102,
"url": "https://anilist.co/manga/30102"
}
},
{
"id": "naruto-1999",
"title": "Naruto",
"year": 1999,
"period": "1995–1999",
"periodOrder": 6,
"rank": 8,
"note": "One of the global gateway manga of the 2000s. Its ninja world and found-family rivalries made long-running shōnen a formative reading experience for many fans.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100018"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx30011-9yUF1dXWgDOx.jpg",
"chapters": 700,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30011,
"url": "https://anilist.co/manga/30011"
}
},
{
"id": "gto-great-teacher-onizuka-1997",
"title": "GTO: Great Teacher Onizuka",
"year": 1997,
"period": "1995–1999",
"periodOrder": 6,
"rank": 9,
"note": "A school-manga favorite that turns delinquent comedy into a critique of adults who fail students, giving the 1990s list a different kind of outsider hero.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/gto-great-teacher-onizuka/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30336-KWTPMwsNTeSa.png",
"chapters": 208,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30336,
"url": "https://anilist.co/manga/30336"
}
},
{
"id": "fullmetal-alchemist-2001",
"title": "Fullmetal Alchemist",
"year": 2001,
"period": "2000–2004",
"periodOrder": 7,
"rank": 1,
"note": "A rare long-running battle fantasy with a widely admired complete ending. It earns its place as a model of tightly planned stakes and payoff rather than endless escalation.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/fullmetal-alchemist"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30025-mpPVpCKFTowt.png",
"chapters": 116,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30025,
"url": "https://anilist.co/manga/30025"
}
},
{
"id": "death-note-2003",
"title": "Death Note",
"year": 2003,
"period": "2000–2004",
"periodOrder": 7,
"rank": 2,
"note": "A mainstream shōnen phenomenon built on strategy instead of physical combat. Its high-concept duel widened the range of stories a blockbuster manga could carry.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100008"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30021-FE6kmrfpuKyb.jpg",
"chapters": 108,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30021,
"url": "https://anilist.co/manga/30021"
}
},
{
"id": "nana-2000",
"title": "Nana",
"year": 2000,
"period": "2000–2004",
"periodOrder": 7,
"rank": 3,
"note": "A defining adult-relationship manga whose friendships, music, and compromises remain a benchmark for emotionally unsparing josei storytelling.",
"source": {
"kind": "read",
"url": "https://www.viz.com/vizmanga/chapters/nana"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30028-VJqBC1ar6AxE.png",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 30028,
"url": "https://anilist.co/manga/30028"
}
},
{
"id": "pluto-2003",
"title": "Pluto",
"year": 2003,
"period": "2000–2004",
"periodOrder": 7,
"rank": 4,
"note": "Urasawa's acclaimed reworking of Tezuka's Astro Boy shows the manga canon talking to itself: a children's robot tale becomes a mature story about war and memory.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/pluto-urasawa-x-tezuka"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30745-kueBpwriQkvZ.jpg",
"chapters": 65,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30745,
"url": "https://anilist.co/manga/30745"
}
},
{
"id": "yotsuba-2003",
"title": "Yotsuba&!",
"year": 2003,
"period": "2000–2004",
"periodOrder": 7,
"rank": 5,
"note": "A modern slice-of-life benchmark. Its precision at finding wonder in ordinary moments earns it a place beside louder classics as a masterclass in quiet comic timing.",
"source": {
"kind": "edition",
"url": "https://yenpress.com/series/yotsuba"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30104-sUVzNlTWZ5cu.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 30104,
"url": "https://anilist.co/manga/30104"
}
},
{
"id": "bleach-2001",
"title": "Bleach",
"year": 2001,
"period": "2000–2004",
"periodOrder": 7,
"rank": 6,
"note": "One of the globally defining 2000s shōnen hits. Its graphic design and sword-fighting style are influential enough to earn a place even beyond plot preferences; you own it already.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100004"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30012-1epmVfTSv2rr.png",
"chapters": 706,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30012,
"url": "https://anilist.co/manga/30012"
}
},
{
"id": "gantz-2000",
"title": "Gantz",
"year": 2000,
"period": "2000–2004",
"periodOrder": 7,
"rank": 7,
"note": "A cult adult survival-SF staple of the 2000s. Its extreme visual spectacle and ruthless pacing make it one of the clearest matches to your Gantz/BLAME! side of the shelf.",
"source": {
"kind": "edition",
"url": "https://www.darkhorse.com/Books/30-524/Gantz-Omnibus-Volume-1-TPB"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30564-RbyFbLzbnKea.jpg",
"chapters": 383,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30564,
"url": "https://anilist.co/manga/30564"
}
},
{
"id": "nodame-cantabile-2001",
"title": "Nodame Cantabile",
"year": 2001,
"period": "2000–2004",
"periodOrder": 7,
"rank": 8,
"note": "A standout music-and-romance manga that treats practice and artistic growth seriously, broadening the list's idea of what an ambitious long serial can be.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/nodame-cantabile/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30419-lT7p72ItXPNK.jpg",
"chapters": 152,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30419,
"url": "https://anilist.co/manga/30419"
}
},
{
"id": "vinland-saga-2005",
"title": "Vinland Saga",
"year": 2005,
"period": "2005–2009",
"periodOrder": 8,
"rank": 1,
"note": "A prestige historical-manga epic whose turn from revenge toward nonviolence makes it a distinctive counterpoint to the samurai and dark-fantasy works you favor.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/vinland-saga/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx30642-0mjRDkf4THpo.jpg",
"chapters": 224,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 30642,
"url": "https://anilist.co/manga/30642"
}
},
{
"id": "kingdom-2006",
"title": "Kingdom",
"year": 2006,
"period": "2005–2009",
"periodOrder": 8,
"rank": 2,
"note": "A giant-scale historical-war serial and long-running readership favorite. Its tactical campaigns and career progression fit the epic-world side of your library.",
"source": {
"kind": "read",
"url": "https://www.viz.com/shonenjump/chapters/kingdom"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx46765-KPXir4sRqJBW.png",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 46765,
"url": "https://anilist.co/manga/46765"
}
},
{
"id": "goodnight-punpun-2007",
"title": "Goodnight Punpun",
"year": 2007,
"period": "2005–2009",
"periodOrder": 8,
"rank": 3,
"note": "A modern adult-manga critical favorite for its visual risk and unflinching psychology. It deserves a place for formal ambition, with a serious heavy-content warning.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/goodnight-punpun"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx34632-5xMDkx3pXsEh.png",
"chapters": 147,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 34632,
"url": "https://anilist.co/manga/34632"
}
},
{
"id": "chihayafuru-2007",
"title": "Chihayafuru",
"year": 2007,
"period": "2005–2009",
"periodOrder": 8,
"rank": 4,
"note": "A landmark for making a niche competitive art feel as gripping as any sports battle, while keeping its relationships and female perspective central.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/chihayafuru/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx43245-Ied8R99LH3u6.jpg",
"chapters": 248,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 43245,
"url": "https://anilist.co/manga/43245"
}
},
{
"id": "a-bride-s-story-2008",
"title": "A Bride's Story",
"year": 2008,
"period": "2005–2009",
"periodOrder": 8,
"rank": 5,
"note": "Kaoru Mori's meticulous linework and unusually researched Central Asian setting make this a craft benchmark for historical slice-of-life manga.",
"source": {
"kind": "edition",
"url": "https://yenpress.com/series/a-bride-s-story"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx41514-IHeXFwqmWivk.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 41514,
"url": "https://anilist.co/manga/41514"
}
},
{
"id": "attack-on-titan-2009",
"title": "Attack on Titan",
"year": 2009,
"period": "2005–2009",
"periodOrder": 8,
"rank": 6,
"note": "A worldwide 2010s manga phenomenon. Its escalating mystery and repeated shifts in perspective made it a defining conversation piece far beyond action fandom.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/attack-on-titan/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx53390-1RsuABC34P9D.jpg",
"chapters": 141,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 53390,
"url": "https://anilist.co/manga/53390"
}
},
{
"id": "march-comes-in-like-a-lion-2007",
"title": "March Comes in Like a Lion",
"year": 2007,
"period": "2005–2009",
"periodOrder": 8,
"rank": 7,
"note": "A prestige character-drama pick that shows competitive manga can be as much about isolation and care as winning. It keeps the list's emotional range honest.",
"source": {
"kind": "edition",
"url": "https://www.denpa.pub/book"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx31224-dxugU08jahRT.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 31224,
"url": "https://anilist.co/manga/31224"
}
},
{
"id": "bakuman-2008",
"title": "Bakuman",
"year": 2008,
"period": "2005–2009",
"periodOrder": 8,
"rank": 8,
"note": "A rare blockbuster manga about the manga industry itself. Its idealized but revealing account of serialization belongs on a list about how the medium works.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100002"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx39711-tjPWXT1AW321.jpg",
"chapters": 176,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 39711,
"url": "https://anilist.co/manga/39711"
}
},
{
"id": "golden-kamuy-2014",
"title": "Golden Kamuy",
"year": 2014,
"period": "2010–2014",
"periodOrder": 9,
"rank": 1,
"note": "A 2016 Manga Taisho winner and an unusually ambitious genre hybrid. Its attention to Ainu culture makes it more than another historical treasure hunt.",
"source": {
"kind": "read",
"url": "https://www.viz.com/shonenjump/chapters/golden-kamuy"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx86559-Y4xDmoz4ud43.jpg",
"chapters": 314,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 86559,
"url": "https://anilist.co/manga/86559"
}
},
{
"id": "land-of-the-lustrous-2012",
"title": "Land of the Lustrous",
"year": 2012,
"period": "2010–2014",
"periodOrder": 9,
"rank": 2,
"note": "A distinctive visual and philosophical experiment in speculative manga. It earns its slot for treating identity and transformation with a formal daring few fantasy serials attempt.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/land-of-the-lustrous/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx74489-5HtDCFfut8Be.jpg",
"chapters": 120,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 74489,
"url": "https://anilist.co/manga/74489"
}
},
{
"id": "delicious-in-dungeon-2014",
"title": "Delicious in Dungeon",
"year": 2014,
"period": "2010–2014",
"periodOrder": 9,
"rank": 3,
"note": "A modern fantasy-worldbuilding benchmark: it makes ecology, food, and consequences part of the dungeon's logic rather than decorative lore.",
"source": {
"kind": "edition",
"url": "https://yenpress.com/series/delicious-in-dungeon"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx86082-MXizJxzbijdd.jpg",
"chapters": 111,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 86082,
"url": "https://anilist.co/manga/86082"
}
},
{
"id": "one-punch-man-2012",
"title": "One-Punch Man",
"year": 2012,
"period": "2010–2014",
"periodOrder": 9,
"rank": 4,
"note": "A webcomic-to-manga phenomenon and a showcase for Yusuke Murata's action art. Its superhero parody made a familiar power fantasy feel new; it is already on your shelf.",
"source": {
"kind": "read",
"url": "https://www.viz.com/shonenjump/chapters/one-punch-man"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx74347-sZpmNJ5xLwRK.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 74347,
"url": "https://anilist.co/manga/74347"
}
},
{
"id": "haikyu-2012",
"title": "Haikyu!!",
"year": 2012,
"period": "2010–2014",
"periodOrder": 9,
"rank": 5,
"note": "A modern sports-manga benchmark for ensemble writing and extraordinarily legible motion. It earns its place even for readers with no prior interest in volleyball.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100014"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx65243-mR4MnJFmfaOF.png",
"chapters": 404,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 65243,
"url": "https://anilist.co/manga/65243"
}
},
{
"id": "a-silent-voice-2013",
"title": "A Silent Voice",
"year": 2013,
"period": "2010–2014",
"periodOrder": 9,
"rank": 6,
"note": "A key 2010s conversation about bullying, disability, and accountability. Its willingness to make its lead answer for harm sets it apart from easy redemption stories.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/a-silent-voice/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx85135-11OOnyaqV71k.png",
"chapters": 64,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 85135,
"url": "https://anilist.co/manga/85135"
}
},
{
"id": "ajin-demi-human-2012",
"title": "Ajin: Demi-Human",
"year": 2012,
"period": "2010–2014",
"periodOrder": 9,
"rank": 7,
"note": "A cult tactical-action pick that makes immortality a system to exploit rather than a simple superpower. It is one of the closest 2010s matches to your action-SF taste.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/ajin-demi-human/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/b79865-rwKmISrhJJyP.jpg",
"chapters": 99,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 79865,
"url": "https://anilist.co/manga/79865"
}
},
{
"id": "blue-giant-2013",
"title": "Blue Giant",
"year": 2013,
"period": "2010–2014",
"periodOrder": 9,
"rank": 8,
"note": "An unusual music-manga craft pick: it turns practice and jazz performance into kinetic page storytelling, proving that silent panels can make readers feel sound.",
"source": {
"kind": "edition",
"url": "https://sevenseasentertainment.com/series/blue-giant-omnibus/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx86890-hYxIiVM6rBBG.jpg",
"chapters": 80,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 86890,
"url": "https://anilist.co/manga/86890"
}
},
{
"id": "witch-hat-atelier-2016",
"title": "Witch Hat Atelier",
"year": 2016,
"period": "2015–2019",
"periodOrder": 10,
"rank": 1,
"note": "Earns its place for Kamome Shirahama's exceptional page design and international critical recognition: it won the 2025 Harvey Award for Best Manga.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/witch-hat-atelier/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx98263-kIeaTXrAbGkj.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 98263,
"url": "https://anilist.co/manga/98263"
}
},
{
"id": "chainsaw-man-2018",
"title": "Chainsaw Man",
"year": 2018,
"period": "2015–2019",
"periodOrder": 10,
"rank": 2,
"note": "A defining recent Jump-era breakout: Fujimoto made horror-comedy, abrupt tonal shifts, and cinematic visual storytelling feel like mainstream shonen possibilities.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100037"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx105778-euxXZEIfDY2u.png",
"chapters": 232,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 105778,
"url": "https://anilist.co/manga/105778"
}
},
{
"id": "blue-period-2017",
"title": "Blue Period",
"year": 2017,
"period": "2015–2019",
"periodOrder": 10,
"rank": 3,
"note": "A rare major manga that treats learning to make art as seriously as a sports contest; its acclaim includes Manga Taisho 2020 and a Kodansha Manga Award.",
"source": {
"kind": "award history",
"url": "https://www.kodansha.co.jp/awards/comics/histories"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx107237-tyQXghsKmE0t.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 107237,
"url": "https://anilist.co/manga/107237"
}
},
{
"id": "beastars-2016",
"title": "Beastars",
"year": 2016,
"period": "2015–2019",
"periodOrder": 10,
"rank": 4,
"note": "Paru Itagaki's distinctive animal-society drama broadened what a breakout shonen hit could look like; its major manga-prize recognition backs the pick.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/beastars"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx98587-useiMvZ0nfya.png",
"chapters": 196,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 98587,
"url": "https://anilist.co/manga/98587"
}
},
{
"id": "the-apothecary-diaries-2017",
"title": "The Apothecary Diaries",
"year": 2017,
"period": "2015–2019",
"periodOrder": 10,
"rank": 5,
"note": "Included as a standout modern mystery adaptation: Maomao's forensic reasoning and court politics brought a different kind of heroine to a large manga audience.",
"source": {
"kind": "read",
"url": "https://global.manga-up.com/manga/164"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/nx99022-Hh2WdyNgR8HM.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 99022,
"url": "https://anilist.co/manga/99022"
}
},
{
"id": "heavenly-delusion-2018",
"title": "Heavenly Delusion",
"year": 2018,
"period": "2015–2019",
"periodOrder": 10,
"rank": 6,
"note": "An editorial pick for your darker SF shelf: its layered post-apocalyptic mystery rewards the same long-form clue reading that makes your current favorites compelling.",
"source": {
"kind": "read",
"url": "https://kmanga.kodansha.com/title/10511/episode/345745"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx101942-yjslDpT6XcpK.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 101942,
"url": "https://anilist.co/manga/101942"
}
},
{
"id": "demon-slayer-kimetsu-no-yaiba-2016",
"title": "Demon Slayer: Kimetsu no Yaiba",
"year": 2016,
"period": "2015–2019",
"periodOrder": 10,
"rank": 7,
"note": "A cultural phenomenon whose manga-to-anime success brought an enormous new audience to the medium; the compact completed run also makes it a useful modern shonen benchmark.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/demon-slayer-kimetsu-no-yaiba"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx87216-c9bSNVD10UuD.png",
"chapters": 207,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 87216,
"url": "https://anilist.co/manga/87216"
}
},
{
"id": "jujutsu-kaisen-2018",
"title": "Jujutsu Kaisen",
"year": 2018,
"period": "2015–2019",
"periodOrder": 10,
"rank": 8,
"note": "One of the era's most visible dark-battle shonen series, worth including for how its rule-heavy fights and harsher tone shaped the current mainstream conversation.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100034"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx101517-H3TdM3g5ZUe9.jpg",
"chapters": 272,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 101517,
"url": "https://anilist.co/manga/101517"
}
},
{
"id": "spy-family-2019",
"title": "Spy × Family",
"year": 2019,
"period": "2015–2019",
"periodOrder": 10,
"rank": 9,
"note": "A leading Manga Plus-era crossover hit: its broad family-comedy appeal shows how a digital-first Jump title can reach far beyond battle-manga readers.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100056"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx108556-NHjkz0BNJhLx.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 108556,
"url": "https://anilist.co/manga/108556"
}
},
{
"id": "kaguya-sama-love-is-war-2015",
"title": "Kaguya-sama: Love Is War",
"year": 2015,
"period": "2015–2019",
"periodOrder": 10,
"rank": 10,
"note": "A modern rom-com reference point: its mind-game structure, long character arcs, and popularity justify a place beside the period's action-heavy giants.",
"source": {
"kind": "edition",
"url": "https://www.viz.com/kaguya-sama-love-is-war"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx86635-EdaLQmsn86Fy.png",
"chapters": 281,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 86635,
"url": "https://anilist.co/manga/86635"
}
},
{
"id": "frieren-beyond-journey-s-end-2020",
"title": "Frieren: Beyond Journey's End",
"year": 2020,
"period": "2020–2024",
"periodOrder": 11,
"rank": 1,
"note": "The 2021 Manga Taisho winner stands out for making aftermath and passing time the engine of an adventure manga; it also fits your fantasy-heavy library.",
"source": {
"kind": "read",
"url": "https://www.viz.com/vizmanga/chapters/frieren-the-journeys-end"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx118586-CXKgWikBFQgS.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 118586,
"url": "https://anilist.co/manga/118586"
}
},
{
"id": "dandadan-2021",
"title": "Dandadan",
"year": 2021,
"period": "2020–2024",
"periodOrder": 11,
"rank": 2,
"note": "Included for Yukinobu Tatsu's unusually kinetic page craft and genre-mixing: it is a strong example of how bold digital-era shonen can look.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100171"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx132029-prGF4gePdSKv.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 132029,
"url": "https://anilist.co/manga/132029"
}
},
{
"id": "the-summer-hikaru-died-2021",
"title": "The Summer Hikaru Died",
"year": 2021,
"period": "2020–2024",
"periodOrder": 11,
"rank": 3,
"note": "A recent horror standout whose unsettling visual restraint and queer emotional core make it a distinctive counterpoint to the list's louder action hits.",
"source": {
"kind": "edition",
"url": "https://yenpress.com/series/the-summer-hikaru-died"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx138603-j0gcshf7PVkN.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 138603,
"url": "https://anilist.co/manga/138603"
}
},
{
"id": "medalist-2020",
"title": "Medalist",
"year": 2020,
"period": "2020–2024",
"periodOrder": 11,
"rank": 4,
"note": "A Kodansha Manga Award winner that belongs here for turning an unusually technical sport into an emotionally forceful, widely acclaimed manga.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/medalist/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx118371-3xk9VUbWnnRR.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 118371,
"url": "https://anilist.co/manga/118371"
}
},
{
"id": "orb-on-the-movements-of-the-earth-2020",
"title": "Orb: On the Movements of the Earth",
"year": 2020,
"period": "2020–2024",
"periodOrder": 11,
"rank": 5,
"note": "An unusually ambitious short historical manga: its shifting cast and argument about knowledge make it a formal outlier worth preserving in a greatest-works survey.",
"source": {
"kind": "edition",
"url": "https://sevenseasentertainment.com/series/orb-on-the-movements-of-the-earth-omnibus/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx124139-OOL1bW0HecJT.jpg",
"chapters": 62,
"status": "completed",
"metadata": {
"provider": "AniList",
"id": 124139,
"url": "https://anilist.co/manga/124139"
}
},
{
"id": "the-darwin-incident-2020",
"title": "The Darwin Incident",
"year": 2020,
"period": "2020–2024",
"periodOrder": 11,
"rank": 6,
"note": "Manga Taisho 2022 and a 2026 Kodansha Manga Award mark this as more than a provocative premise; its moral and political questions earn it a place here.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/the-darwin-incident/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx120638-cWwWsAGL3t9d.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 120638,
"url": "https://anilist.co/manga/120638"
}
},
{
"id": "kagurabachi-2023",
"title": "Kagurabachi",
"year": 2023,
"period": "2020–2024",
"periodOrder": 11,
"rank": 7,
"note": "A provisional new-generation pick: its rapid rise from internet joke to sincerely followed Jump action series makes it a useful marker of the 2020s fandom cycle.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100274"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx169355-5pqzh1Wb4NOQ.png",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 169355,
"url": "https://anilist.co/manga/169355"
}
},
{
"id": "gachiakuta-2022",
"title": "Gachiakuta",
"year": 2022,
"period": "2020–2024",
"periodOrder": 11,
"rank": 8,
"note": "Its graffiti-inflected visual identity stands apart in recent action manga, and the 2026 Kodansha Manga Award gives this newer pick a concrete acclaim signal.",
"source": {
"kind": "edition",
"url": "https://kodansha.us/series/gachiakuta/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx144946-cscic3n2SwdY.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 144946,
"url": "https://anilist.co/manga/144946"
}
},
{
"id": "sakamoto-days-2020",
"title": "Sakamoto Days",
"year": 2020,
"period": "2020–2024",
"periodOrder": 11,
"rank": 9,
"note": "Selected chiefly for its inventive, legible action choreography: a contemporary benchmark for how much visual storytelling can carry a comedy-action serial.",
"source": {
"kind": "read",
"url": "https://mangaplus.shueisha.co.jp/titles/100127"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx125828-p78Z8SflkfmO.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 125828,
"url": "https://anilist.co/manga/125828"
}
},
{
"id": "ichi-the-witch-2024",
"title": "Ichi the Witch",
"year": 2024,
"period": "2020–2024",
"periodOrder": 11,
"rank": 10,
"note": "A promising rather than proven inclusion: its playful approach to fantasy battles makes it worth watching among the latest Jump launches.",
"source": {
"kind": "read",
"url": "https://www.viz.com/shonenjump/chapters/ichi-the-witch"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx180752-13CVO310XiEs.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 180752,
"url": "https://anilist.co/manga/180752"
}
},
{
"id": "home-at-the-horizon-2025",
"title": "Home at the Horizon",
"year": 2025,
"period": "2025–2026",
"periodOrder": 12,
"rank": 1,
"note": "A future-watch pick, not yet an all-time classic: it placed #1 in the 2026 Next Manga Awards web category, a clear early reader-enthusiasm signal.",
"source": {
"kind": "official award and reading lead",
"url": "https://mangaplus.shueisha.co.jp/web_pages/2441/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx188733-YMOdYQ8llVnm.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 188733,
"url": "https://anilist.co/manga/188733"
}
},
{
"id": "someone-hertz-2025",
"title": "Someone Hertz",
"year": 2025,
"period": "2025–2026",
"periodOrder": 12,
"rank": 2,
"note": "A future-watch pick backed by readers: #1 in the 2026 Next Manga Awards print category. Its radio-romance premise also broadens this action-heavy list.",
"source": {
"kind": "read",
"url": "https://www.viz.com/shonenjump/chapters/someone-hertz"
},
"cover": "https://www.shonenjump.com/j/comics/_comicimage/someone001.jpg",
"chapters": null,
"status": "unknown"
},
{
"id": "futari-bus-2025",
"title": "Futari Bus",
"year": 2025,
"period": "2025–2026",
"periodOrder": 12,
"rank": 3,
"note": "A promising newcomer rather than a settled classic: #2 in the 2026 Next Manga Awards print category, showing strong early support for a quieter romance.",
"source": {
"kind": "award/info",
"url": "https://tsugimanga.jp/winners_announcement/comics"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx201855-7dEgpKSZhWsH.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 201855,
"url": "https://anilist.co/manga/201855"
}
},
{
"id": "war-of-the-adults-2025",
"title": "War of the Adults",
"year": 2025,
"period": "2025–2026",
"periodOrder": 12,
"rank": 4,
"note": "A future-watch pick with a concrete vote signal: #4 in the 2026 Next Manga Awards web category, plus an editorial fit for your darker shelf.",
"source": {
"kind": "official award and reading lead",
"url": "https://mangaplus.shueisha.co.jp/web_pages/2441/"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx189691-TYkHsBOxbQW8.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 189691,
"url": "https://anilist.co/manga/189691"
}
},
{
"id": "the-marshal-king-2025",
"title": "The Marshal King",
"year": 2025,
"period": "2025–2026",
"periodOrder": 12,
"rank": 5,
"note": "An exploratory pick for Boichi's striking Western-inflected art, not a legacy claim; its 2025 launch makes long-term cultural importance impossible to judge yet.",
"source": {
"kind": "read",
"url": "https://www.viz.com/shonenjump/chapters/the-marshal-king"
},
"cover": "https://s4.anilist.co/file/anilistcdn/media/manga/cover/medium/bx186517-AP09vY1BVb1E.jpg",
"chapters": null,
"status": "ongoing",
"metadata": {
"provider": "AniList",
"id": 186517,
"url": "https://anilist.co/manga/186517"
}
},
{
"id": "hitoner-2026",
"title": "Hitoner",
"year": 2026,
"period": "2025–2026",
"periodOrder": 12,
"rank": 6,
"note": "A deliberate wildcard, not a proven classic: #2 in the 2026 Next Manga Awards web category gives it a strong early reader-vote signal.",
"source": {
"kind": "official award and reading lead",
"url": "https://mangaplus.shueisha.co.jp/web_pages/2441/"
},
"cover": "https://dosbg3xlm0x1t.cloudfront.net/images/items/9784088852317/500/9784088852317.jpg",
"chapters": null,
"status": "unknown"
}
]
}
+43
View File
@@ -0,0 +1,43 @@
'use strict';
const normalize = value => String(value || '').normalize('NFKD').toLowerCase().replace(/[^a-z0-9]+/g, '');
function seriesLink(value) {
let url;
try { url = new URL(String(value || '')); } catch { return null; }
if (url.protocol !== 'https:') return null;
const weeb = /^\/series\/([A-Z0-9]{26})(?:\/[^/?#]+)?\/?$/i.exec(url.pathname);
if (url.hostname === 'weebcentral.com' && weeb) return {
source: 'weebcentral', label: 'WeebCentral', seriesId: weeb[1].toUpperCase(),
url: `https://weebcentral.com${url.pathname.replace(/\/$/, '')}`,
};
const dex = /^\/title\/([0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12})(?:\/[^/?#]+)?\/?$/i.exec(url.pathname);
if (['mangadex.org', 'www.mangadex.org'].includes(url.hostname) && dex) return {
source: 'mangadex', label: 'MangaDex', seriesId: dex[1].toLowerCase(),
url: `https://mangadex.org/title/${dex[1].toLowerCase()}`,
};
return null;
}
const keyOf = link => `${link.source}:${link.seriesId}`;
function hosterLinks(entry, owned, queue = []) {
const names = new Set([entry.title, ...(entry.aliases || [])].map(normalize));
const known = new Map();
for (const source of owned?.meta?.sources || []) {
const link = seriesLink(source.url);
if (link) known.set(keyOf(link), { ...link, origin: 'library' });
}
const jobs = [];
for (const item of queue) {
const link = seriesLink(item.url);
if (!link) continue;
if (item.libraryMangaId && owned && item.libraryMangaId !== owned.id) continue;
if (!(owned && item.libraryMangaId === owned.id) && !names.has(normalize(item.title)) && !known.has(keyOf(link))) continue;
if (!known.has(keyOf(link))) known.set(keyOf(link), { ...link, origin: 'queue title match' });
jobs.push({ id: item.id, source: link.source, seriesId: link.seriesId, kind: item.kind, status: item.status });
}
return { links: [...known.values()], jobs };
}
module.exports = { seriesLink, hosterLinks };
+44
View File
@@ -0,0 +1,44 @@
'use strict';
const fs = require('fs');
const path = require('path');
const RATINGS = Object.freeze(['flame', 'good', 'neutral', 'skip']);
const WEIGHT = Object.freeze({ flame: 3, good: 1, neutral: 0, skip: -2 });
const normalize = value => String(value || '').normalize('NFKD').toLowerCase().replace(/[^a-z0-9]+/g, '');
function loadCatalog(file = path.join(__dirname, 'catalog.json')) {
const parsed = JSON.parse(fs.readFileSync(file, 'utf8'));
if (parsed.schema !== 1 || !Array.isArray(parsed.entries)) throw new Error('Invalid Greatest of All Time catalog');
const ids = new Set();
for (const entry of parsed.entries) {
if (!/^[a-z0-9-]+$/.test(entry.id) || ids.has(entry.id) || !entry.title || !entry.period || !Number.isInteger(entry.rank))
throw new Error(`Invalid or duplicate Greatest of All Time entry: ${entry.id}`);
if (entry.cover && !/^https:\/\//.test(entry.cover)) throw new Error(`Invalid cover URL for ${entry.id}`);
if (entry.source?.url && !/^https:\/\//.test(entry.source.url)) throw new Error(`Invalid source URL for ${entry.id}`);
ids.add(entry.id);
}
return parsed;
}
function findOwned(entry, mangas) {
const names = new Set([entry.title, ...(entry.aliases || [])].map(normalize));
return mangas.find(m => {
const alt = m.meta?.altTitles;
return [m.title, ...(Array.isArray(alt) ? alt : alt ? [alt] : [])].some(name => names.has(normalize(name)));
}) || null;
}
function countsFor(entryId, votes, users) {
const counts = { flame: 0, good: 0, neutral: 0, skip: 0 };
for (const name of Object.keys(users)) {
const rating = votes[name]?.[entryId];
if (RATINGS.includes(rating)) counts[rating]++;
}
return counts;
}
function scoreFor(counts) {
return RATINGS.reduce((sum, rating) => sum + counts[rating] * WEIGHT[rating], 0);
}
module.exports = { RATINGS, WEIGHT, normalize, loadCatalog, findOwned, countsFor, scoreFor };
+77 -24
View File
@@ -7,30 +7,14 @@
* Image sizes are cached per chapter folder and only re-read when the folder's mtime changes.
*/
const fsp = require('fs').promises, path = require('path');
const { parseSpecialFolder, parseKey } = require('./chapter-identity');
const { resolveReadingDirection } = require('./reading-direction');
const IMG = /\.(jpe?g|png|webp|gif|avif|bmp)$/i;
const COVER = /^cover\.(jpe?g|png|webp|gif|avif|bmp)$/i;
const KEY = /^([a-z]*) ?(\d+(?:\.\d+)?)(?=$|[\s_\-[(])[\s_-]*(.*)$/i;
const DIRECTIONS = { rtl: 'rtl', ltr: 'ltr', ttb: 'ttb', vertical: 'ttb', webtoon: 'ttb' };
const cmp = new Intl.Collator('en', { numeric: true, sensitivity: 'base' }).compare;
const str = x => (typeof x === 'string' ? x.trim() : '');
const order = (a, b) => a.x - b.x || a.num - b.num;
// "c0002.5 Tea Time [MangaDex]" -> { key: '2.5', x: false, num: 2.5, title: 'Tea Time', src: 'MangaDex' }
function parseKey(name) {
const m = KEY.exec(name);
if (!m) return null;
const x = m[1].toLowerCase() === 'x', num = parseFloat(m[2]);
let title = m[3].trim(), src = '';
const b = /\s*\[([^\]]*)\]$/.exec(title);
if (b) { src = b[1].trim(); title = title.slice(0, b.index).trim(); }
return { key: (x ? 'x' : '') + num, x, num, title: title.replace(/^[-:.]\s*/, ''), src };
}
function direction(...metas) {
for (const m of metas) if (DIRECTIONS[str(m.direction).toLowerCase()]) return DIRECTIONS[str(m.direction).toLowerCase()];
return 'rtl';
}
const order = (a, b) => Number(!!a.special) - Number(!!b.special) || a.x - b.x || a.num - b.num || cmp(a.key, b.key);
async function readJson(file) {
try { return JSON.parse((await fsp.readFile(file, 'utf8')).replace(/^/, '')); }
@@ -123,7 +107,7 @@ const coverIn = (files, rel) => { const f = files.find(f => COVER.test(f)); retu
async function scanManga(root, name, oldCache, cache) {
const { dirs, files } = await list(path.join(root, name));
const meta = await readJson(path.join(root, name, 'manga.json'));
const m = { id: name, title: str(meta.title) || name, meta, direction: direction(meta), cover: coverIn(files, name), books: [], chapters: [], chapterById: new Map(), updated: 0 };
const m = { id: name, title: str(meta.title) || name, meta, direction: resolveReadingDirection([meta]).value, cover: coverIn(files, name), books: [], chapters: [], chapterById: new Map(), updated: 0 };
const bookKeys = new Set();
for (const bName of dirs) {
const bk = parseKey(bName);
@@ -138,14 +122,15 @@ async function scanManga(root, name, oldCache, cache) {
};
const byKey = new Map();
for (const cName of bl.dirs) {
const ck = parseKey(cName);
const ck = parseSpecialFolder(cName) || parseKey(cName);
if (!ck) { console.warn(`[scan] ignoring "${bRel}/${cName}": chapter folders must be named <letters><number>[ title], e.g. "c0001 The Seed"`); continue; }
const v = await scanVersion(root, bRel + '/' + cName, ck, oldCache, cache);
if (!v.pages.length) continue;
v.direction = direction(v.meta, bMeta, meta);
v.directionInfo = resolveReadingDirection([v.meta, bMeta, meta], v.pages);
v.direction = v.directionInfo.value;
let ch = byKey.get(ck.key);
if (!ch) {
ch = { id: book.key + '/' + ck.key, book: book.key, key: ck.key, num: ck.num, x: ck.x, extra: book.extra || ck.x, label: (ck.x ? 'Extra ' : 'Ch. ') + ck.num, title: '', versions: [] };
ch = { id: book.key + '/' + ck.key, book: book.key, key: ck.key, num: ck.num, x: ck.x, special: !!ck.special, extra: book.extra || ck.x, label: ck.special ? str(v.meta.label) || 'Special' : (ck.x ? 'Extra ' : 'Ch. ') + ck.num, title: '', versions: [] };
byKey.set(ck.key, ch);
}
if (v.meta.extra === true) ch.extra = true;
@@ -179,4 +164,72 @@ async function scanLibrary(root, oldCache = {}) {
return { index: { mangas, byId: new Map(mangas.map(m => [m.id, m])) }, cache };
}
module.exports = { scanLibrary, parseKey, sizeFromBuffer };
// Book-directory renames are the publisher's atomic commit point. A shallow
// stamp pass costs one stat per book, rather than one stat/JSON read per chapter.
// In-place chapter.json edits are picked up by the scheduled deep reconciliation.
async function fileStamp(file) {
try { const s = await fsp.stat(file); return `${s.mtimeMs}:${s.size}`; }
catch (e) { if (e.code === 'ENOENT') return ''; throw e; }
}
async function libraryStamps(root) {
const stamps = {};
for (const name of (await list(root)).dirs) {
const mangaDir = path.join(root, name), books = {};
const listed = await list(mangaDir);
for (const bookName of listed.dirs.filter(parseKey)) {
const bookDir = path.join(mangaDir, bookName);
books[bookName] = [await fileStamp(bookDir), await fileStamp(path.join(bookDir, 'book.json'))];
}
stamps[name] = [await fileStamp(mangaDir), await fileStamp(path.join(mangaDir, 'manga.json')), books];
}
return stamps;
}
async function scanChangedLibrary(root, prior, oldCache, oldStamps, stamps) {
stamps = stamps || await libraryStamps(root);
const names = new Set([...Object.keys(oldStamps || {}), ...Object.keys(stamps)]);
const changed = [...names].filter(name => JSON.stringify(oldStamps?.[name]) !== JSON.stringify(stamps[name]));
if (!changed.length) return { index: prior, cache: oldCache, stamps, changed };
const byId = new Map(prior.mangas.map(m => [m.id, m]));
const cache = { ...oldCache };
for (const name of changed) {
byId.delete(name);
for (const key of Object.keys(cache)) if (key.startsWith(name + '/')) delete cache[key];
if (!Object.hasOwn(stamps, name)) continue;
const m = await scanManga(root, name, oldCache, cache);
if (m.chapters.length) byId.set(name, m);
}
const mangas = [...byId.values()].sort((a, b) => cmp(a.title, b.title));
return { index: { mangas, byId: new Map(mangas.map(m => [m.id, m])) }, cache, stamps, changed };
}
// Store each chapter only once. Maps and book chapter references are rebuilt
// from the snapshot, which is a disposable cache rather than library authority.
function snapshotIndex(index) {
return index.mangas.map(m => ({ ...m, chapterById: undefined,
books: m.books.map(b => ({ ...b, chapters: b.chapters.map(ch => ch.id) })) }));
}
function restoreIndex(mangas) {
if (!Array.isArray(mangas)) throw new Error('Invalid library index snapshot');
const byId = new Map();
for (const m of mangas) {
if (!m || typeof m.id !== 'string' || !Array.isArray(m.books) || !Array.isArray(m.chapters) || byId.has(m.id))
throw new Error('Invalid manga in library index snapshot');
m.chapterById = new Map();
for (const ch of m.chapters) {
if (!ch || typeof ch.id !== 'string' || !Array.isArray(ch.versions) || ch.versions.some(v => !Array.isArray(v.pages)))
throw new Error('Invalid chapter in library index snapshot');
m.chapterById.set(ch.id, ch);
}
for (const book of m.books) {
if (!Array.isArray(book.chapters) || book.chapters.some(id => !m.chapterById.has(id)))
throw new Error('Invalid book in library index snapshot');
book.chapters = book.chapters.map(id => m.chapterById.get(id));
}
byId.set(m.id, m);
}
return { mangas, byId };
}
module.exports = { scanLibrary, scanChangedLibrary, libraryStamps, snapshotIndex, restoreIndex,
parseKey, sizeFromBuffer, imageSize };
+65
View File
@@ -0,0 +1,65 @@
{
"name": "mangacircus",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "mangacircus",
"version": "1.0.0",
"devDependencies": {
"playwright": "1.62.1"
},
"engines": {
"node": ">=18"
}
},
"node_modules/fsevents": {
"version": "2.3.2",
"resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.3.2.tgz",
"integrity": "sha512-xiqMQR4xAeHTuB9uWm+fFRcIOgKBMiOBP+eXiyT7jsgVCq1bkVygt00oASowB7EdtpOHaaPgKt812P9ab+DDKA==",
"dev": true,
"hasInstallScript": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": "^8.16.0 || ^10.6.0 || >=11.0.0"
}
},
"node_modules/playwright": {
"version": "1.62.1",
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.62.1.tgz",
"integrity": "sha512-0M+L3LAD8/nm554LOla9Ayx0j0tmFZ0FBcoQ7F1VuVHpM/XpiC8RcDzBQB8W5+hA8L22THxELzeF+2WcUzvcLg==",
"dev": true,
"license": "Apache-2.0",
"dependencies": {
"playwright-core": "1.62.1"
},
"bin": {
"playwright": "cli.js"
},
"engines": {
"node": ">=20"
},
"optionalDependencies": {
"fsevents": "2.3.2"
}
},
"node_modules/playwright-core": {
"version": "1.62.1",
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.62.1.tgz",
"integrity": "sha512-wPYSwEBJY9GHraISXqyqtx0na0LpO3XEX7jNDhntbex7tzUS7kLnZsOlFruFJB4Hi/rhDMjXGqHewDZ68nYZVw==",
"dev": true,
"license": "Apache-2.0",
"bin": {
"playwright-core": "cli.js"
},
"engines": {
"node": ">=20"
}
}
}
}
+22 -2
View File
@@ -2,13 +2,33 @@
"name": "mangacircus",
"version": "1.0.0",
"private": true,
"description": "No-frills self-hosted manga reader. No dependencies.",
"description": "No-frills self-hosted manga reader with no runtime dependencies.",
"main": "server.js",
"scripts": {
"start": "node server.js",
"demo": "node tools/make-dummy.js library"
"start:scraper": "node scraper/daemon.js",
"start:peer": "node peering/gateway.js",
"start:all": "node tools/start-all.js",
"start:sentinel": "node sentinel/daemon.js",
"check": "node tools/check.js",
"check:compose": "node tools/compose-control.js check",
"test:runtime": "node --test sentinel/test/*.test.js",
"test:compose": "node tools/test-compose.js",
"test:ui": "node tools/test-compose.js --settings-ui --queue-ui --collections-ui --reading-ui",
"test:direction-ui": "node tools/test-compose.js --direction-ui",
"test:discovery-ui": "node tools/test-compose.js --discovery-ui",
"demo": "node tools/make-dummy.js library",
"scrape": "node scraper/cli.js",
"test": "npm run test:reader && npm run test:scraper && npm run test:runtime && npm run test:peer",
"test:peer": "node --test peering/test/*.test.js",
"test:peer-ab": "node tools/test-peer-ab.js",
"test:reader": "node --test test/*.test.js",
"test:scraper": "node --test scraper/test/*.test.js"
},
"engines": {
"node": ">=18"
},
"devDependencies": {
"playwright": "1.62.1"
}
}
+160
View File
@@ -0,0 +1,160 @@
'use strict';
const fs = require('node:fs').promises;
const path = require('node:path');
const crypto = require('node:crypto');
const { PeeringState, requirePeer } = require('./state');
const client = require('./client');
const { ArchiveControlError } = require('../scraper/lib/control');
const { selectByScope } = require('../scraper/lib/scope');
const { scanLibrary, findManga, shouldDownload } = require('../scraper/lib/library-inventory');
const { publishManifest } = require('../scraper/lib/publisher');
const { writeJsonAtomic } = require('../scraper/lib/json-file');
const { sleep } = require('../scraper/lib/delay');
const { normalizeWorkIdentity, sameWorkIdentity, resolveWorkIdentity } = require('../scraper/lib/work-identity');
const dataRoot = () => path.resolve(process.env.PEER_DATA || path.join(process.env.DATA || 'data', 'peering'));
const store = () => new PeeringState(dataRoot());
const infoFor = value => {
try {
const u = new URL(String(value));
if (u.protocol !== 'peer:' || u.username || u.password || u.search || u.hash || !/^p[0-9a-f]{24}$/.test(u.hostname)
|| !/^\/m[0-9a-f]{24}$/.test(u.pathname)) return null;
const peer = requirePeer(store().load(), u.hostname);
return { source: `peer-${peer.id}`, id: `${peer.id}-${u.pathname.slice(1)}`, peerId: peer.id,
shareId: u.pathname.slice(1), title: peer.alias, url: `peer://${peer.id}${u.pathname}` };
} catch { return null; }
};
function refsOf(result) {
return result.chapters.map(ref => ({ id: ref.id, number: ref.number, volume: ref.volume,
book: { number: Number(ref.volume) || 1 }, title: ref.title, pages: ref.pageCount }));
}
function sameTitle(left, right) {
return String(left || '').normalize('NFKC').trim().replace(/\s+/g, ' ').toLowerCase()
=== String(right || '').normalize('NFKC').trim().replace(/\s+/g, ' ').toLowerCase();
}
async function pageHashes(folder) {
const names = (await fs.readdir(folder, { withFileTypes: true }))
.filter(entry => entry.isFile() && /\.(?:jpe?g|png|webp|gif|avif|bmp)$/i.test(entry.name))
.map(entry => entry.name).sort();
if (!names.length) return [];
return Promise.all(names.map(async name => crypto.createHash('sha256')
.update(await fs.readFile(path.join(folder, name))).digest('hex')));
}
async function matchLegacyPeerCopy(inventory, remote, peerStore, info) {
// Pre-work-ID peer imports can be adopted only with byte-for-byte evidence;
// a matching title alone is never sufficient to merge library works.
const candidates = inventory.mangas.filter(manga => sameTitle(manga.title, remote.title));
if (candidates.length !== 1) return null;
const manga = candidates[0];
for (const ref of remote.chapters) {
const book = String(Number(ref.volume) || Number(ref.book?.number) || 1);
const local = manga.chapters.filter(chapter => Number(chapter.book) === Number(book)
&& String(chapter.number) === String(ref.number));
if (!local.length) continue;
const manifest = await client.chapter(peerStore, info.peerId, info.shareId, ref.id);
const expected = manifest.pages.map(page => page.sha256);
for (const chapter of local) {
const actual = await pageHashes(path.join(manga.path, chapter.bookFolder, chapter.version));
if (actual.length === expected.length && actual.every((hash, index) => hash === expected[index])) return manga;
}
}
return null;
}
async function discover(info) {
const result = await client.manga(store(), info.peerId, info.shareId);
return { title: result.title, workIdentity: normalizeWorkIdentity(result.workIdentity), chapters: refsOf(result), availability: { selectedChapters: result.chapters.length, coverage: 'shared-selection' } };
}
async function run(info, item, options) {
const stateStore = store(), peer = requirePeer(stateStore.load(), info.peerId);
if (!peer.paired || !peer.receive) throw new Error('Downloads from this peer are disabled');
const stop = () => { const reason = options.shouldStop?.(); if (reason) throw new ArchiveControlError(reason === true ? 'paused' : String(reason)); };
stop();
const remote = await client.manga(stateStore, info.peerId, info.shareId);
const workIdentity = normalizeWorkIdentity(remote.workIdentity);
const queuedIdentity = normalizeWorkIdentity(item.workIdentity);
if (queuedIdentity && !sameWorkIdentity(queuedIdentity, workIdentity)) throw new Error('Shared manga identity changed after it was queued; review the title before syncing');
const catalog = { title: remote.title, workIdentity, chapters: refsOf(remote), availability: { selectedChapters: remote.chapters.length, coverage: 'shared-selection' } };
if (options.onCatalog) await options.onCatalog(catalog);
const sourceUrl = info.url, source = info.source;
const workRoot = path.join(options.libraryRoot, `.mcscraper-work-${info.id}`);
const stateFile = path.join(workRoot, 'job.json');
await fs.mkdir(workRoot, { recursive: true });
let job = { version: 1, publicationMode: 'incremental', source, seriesUrl: sourceUrl, completed: [], jobCompleted: 0,
status: 'running', started: new Date().toISOString(), updated: new Date().toISOString(), total: 0, current: null };
try { const prior = JSON.parse(await fs.readFile(stateFile, 'utf8')); if (prior.seriesUrl === sourceUrl) job = { ...prior, status: 'running' }; }
catch (e) { if (e.code !== 'ENOENT') throw e; }
const record = async () => { job.updated = new Date().toISOString(); await writeJsonAtomic(stateFile, job); };
const inventory = await scanLibrary(options.libraryRoot);
let manga = findManga(inventory, { source, seriesId: info.id, sourceUrl, workIdentity,
localServerId: stateStore.identity()?.id, libraryMangaId: item.libraryMangaId });
if (!manga && (!workIdentity || workIdentity.kind === 'origin')) {
manga = await matchLegacyPeerCopy(inventory, remote, stateStore, info);
}
if (!manga && inventory.mangas.some(candidate => sameTitle(candidate.title, remote.title))) {
throw new Error(`A manga titled “${remote.title}” already exists, but neither its canonical identity nor shared chapter page hashes verify that it is the same work. No files were downloaded or changed; review the identity before syncing.`);
}
if (item.libraryMangaId && !manga) throw new Error(`Library manga target not found: ${item.libraryMangaId}`);
const localIdentity = manga ? resolveWorkIdentity(manga.meta, manga.id, stateStore.identity()?.id) : null;
if (manga && localIdentity && !normalizeWorkIdentity(manga.meta['x-work-identity'])) {
manga.meta = { ...manga.meta, 'x-work-identity': localIdentity };
await writeJsonAtomic(path.join(manga.path, 'manga.json'), manga.meta);
}
const selected = selectByScope(catalog.chapters.filter(ref => shouldDownload(manga, ref, source, item.existingPolicy)), item.scope);
job.total = selected.length; job.jobCompleted = 0; job.current = null; await record();
let publishedCount = 0;
const identityToPersist = localIdentity || workIdentity;
const series = { title: remote.title, direction: ['rtl', 'ltr', 'ttb'].includes(remote.direction) ? remote.direction : 'rtl',
sources: [{ name: `Peer ${peer.alias}`, url: sourceUrl }],
metadata: { title: remote.title, ...(identityToPersist ? { 'x-work-identity': identityToPersist } : {}) },
scraper: { ids: { [source]: info.id } } };
for (let index = 0; index < selected.length; index++) {
stop();
const ref = selected[index];
if (!shouldDownload(manga, ref, source, item.existingPolicy)) { job.jobCompleted++; await record(); continue; }
const chapter = await client.chapter(stateStore, info.peerId, info.shareId, ref.id);
if (String(chapter.number) !== String(ref.number) || String(chapter.volume) !== String(ref.volume)) throw new Error('Peer chapter identity changed');
const digest = crypto.createHash('sha256').update(JSON.stringify(chapter.pages.map(p => [p.id, p.sha256]))).digest('hex').slice(0, 20);
const stage = path.join(workRoot, 'pages', `${ref.id}-${digest}`);
await fs.mkdir(stage, { recursive: true });
job.current = { number: ref.number, sourceId: ref.id, position: index + 1, total: selected.length,
downloadedPages: 0, expectedPages: chapter.pages.length };
await record();
const pages = chapter.pages.map((page, i) => ({ ...page, path: path.join(stage, `${String(i + 1).padStart(4, '0')}.${page.mime.split('/')[1] === 'jpeg' ? 'jpg' : page.mime.split('/')[1]}`) }));
const concurrency = Math.min(4, Math.max(1, Number(requirePeer(stateStore.load(), info.peerId).download?.concurrent || 2)));
for (let offset = 0; offset < pages.length; offset += concurrency) {
stop();
const batch = pages.slice(offset, offset + concurrency);
await Promise.all(batch.map(page => client.downloadPage(stateStore, info.peerId, page, page.path)));
job.current.downloadedPages = Math.min(pages.length, offset + batch.length);
await record();
stop();
}
const publication = await publishManifest({ version: 1, series, chapters: [{
number: ref.number, title: ref.title, book: { number: Number(ref.volume) || 1 },
source: { name: `Peer ${peer.alias} ${peer.id.slice(1, 7)}`, url: sourceUrl },
pages: pages.map(page => ({ path: page.path })),
}] }, { libraryRoot: options.libraryRoot, ...(manga ? { mangaDir: manga.path } : {}) });
if (publication.results[0].status === 'published') {
publishedCount++;
if (!manga) manga = { id: path.basename(publication.mangaDir), path: publication.mangaDir, chapters: [], meta: {} };
manga.chapters.push({ book: String(Number(ref.volume) || 1), number: String(ref.number), sourceKey: source });
if (options.onPublished) await options.onPublished({ mangaDir: publication.mangaDir, chapter: ref.number });
}
job.jobCompleted++; job.lastChapter = ref.number; job.current = null; await record();
// Only a fully verified and published chapter loses its resumable page cache.
await fs.rm(stage, { recursive: true, force: true });
if (index < selected.length - 1) { stop(); await sleep(Math.max(500, Number(options.settings.chapterMinMs || 500))); }
}
job.status = 'complete'; job.current = null; await record();
return { finalPath: manga?.path || null, publishedCount, selectedCount: selected.length };
}
function adapterForPeer(value) {
const info = infoFor(value);
if (!info) return null;
const adapter = { id: info.source, label: `Peer ${info.peerId.slice(1, 7)}`,
async discover() { return discover(info); },
async run(item, options) { return run(info, item, options); } };
return { adapter, info };
}
module.exports = { adapterForPeer, infoFor, discover, run };
+164
View File
@@ -0,0 +1,164 @@
'use strict';
const https = require('node:https');
const fs = require('node:fs');
const fsp = fs.promises;
const crypto = require('node:crypto');
const { normalizeWorkIdentity } = require('../scraper/lib/work-identity');
const { pipeline } = require('node:stream/promises');
const { Transform } = require('node:stream');
const { PeeringState, fp, requirePeer, grantStatus } = require('./state');
const MAX_JSON = 8 * 1024 * 1024;
const MAX_PAGE = 30 * 1024 * 1024;
const agents = new Map();
const downloadSlots = new Map();
function remoteError(code, message, retryAfter) {
const error = new Error(`Peer HTTP ${code}: ${String(message).slice(0, 300)}`);
error.status = code;
error.retryAfter = retryAfter;
return error;
}
function connection(store, id, needGrant = true, requireReceive = false) {
const state = store.load(), peer = requirePeer(state, id), identity = store.identity();
if (!identity) throw new Error('Generate local peering identity first');
if (needGrant && !peer.paired) throw new Error('Verify peer identity first');
if (requireReceive && !peer.receive) throw new Error('Downloads from this peer are disabled');
const token = store.outbound()[id];
if (needGrant && !token) throw new Error('Import the peer grant first');
const cacheKey = `${id}:${fp(peer.cert)}:${identity.fingerprint}`;
let agent = agents.get(cacheKey);
if (!agent) {
agent = new https.Agent({ keepAlive: true, maxSockets: Math.max(1, peer.download?.concurrent || 2), minVersion: 'TLSv1.3',
cert: identity.cert, key: identity.key, ca: peer.cert, rejectUnauthorized: true,
checkServerIdentity(_host, cert) {
if (!cert.raw || fp(cert.raw) !== peer.fingerprint) return new Error('Peer certificate fingerprint changed');
return undefined; // The out-of-band pin replaces a hostname CA hierarchy.
} });
agents.set(cacheKey, agent);
}
return { peer, agent, token };
}
function request(store, id, endpoint, options = {}) {
const { peer, agent, token } = connection(store, id, !options.pairing && !options.access, !!options.requireReceive);
const base = new URL(peer.url), target = new URL(endpoint, base);
if (target.origin !== base.origin || !target.pathname.startsWith('/peer/v1/')) throw new Error('Invalid peer endpoint');
return new Promise((resolve, reject) => {
const headers = { accept: options.binary ? 'image/*' : 'application/json',
...(options.pairing ? { 'x-invitation-id': peer.invitationId || '', 'x-invitation-code': peer.invitationCode || '' }
: options.access ? {} : { authorization: `Bearer ${token}` }),
...options.headers };
const req = https.request(target, { method: 'GET', agent, headers, timeout: 30000 }, res => resolve(res));
req.on('timeout', () => req.destroy(new Error('Peer request timed out')));
req.on('error', reject); req.end();
});
}
async function readResponse(res, limit) {
const chunks = []; let size = 0;
for await (const chunk of res) { size += chunk.length; if (size > limit) throw new Error('Peer response exceeds limit'); chunks.push(chunk); }
return Buffer.concat(chunks);
}
async function jsonRequest(store, id, endpoint, options = {}) {
const res = await request(store, id, endpoint, options);
const data = await readResponse(res, MAX_JSON);
let value;
try { value = JSON.parse(data.toString('utf8')); } catch { throw new Error('Peer returned invalid JSON'); }
if (res.statusCode !== 200) throw remoteError(res.statusCode, value.error || 'Request failed', res.headers['retry-after']);
return value;
}
async function probe(store, id) {
const peer = requirePeer(store.load(), id);
if (Date.parse(peer.invitationExpiresAt) <= Date.now() && !peer.paired) throw new Error('Peer invitation expired');
const hello = await jsonRequest(store, id, '/peer/v1/hello', { pairing: !peer.paired });
if (hello.version !== 1 || hello.fingerprint !== peer.fingerprint) throw new Error('Peer hello identity mismatch');
const state = store.load(), paired = requirePeer(state, id); paired.paired = true;
delete paired.invitationId; delete paired.invitationCode; delete paired.invitationExpiresAt;
const file = store.stateFile(), temp = `${file}.${crypto.randomBytes(6).toString('hex')}.tmp`;
fs.writeFileSync(temp, JSON.stringify(state, null, 2) + '\n', { mode: 0o600 }); fs.renameSync(temp, file);
return { ok: true, id: hello.id };
}
async function access(store, id) {
const result = await jsonRequest(store, id, '/peer/v1/access', { access: true });
if (result.version !== 1 || typeof result.paired !== 'boolean' || !['none', 'active', 'paused', 'expired'].includes(result.grant?.status)
|| !Array.isArray(result.grant?.scopes)) throw new Error('Invalid peer access status');
return result;
}
async function cover(store, id, shareId) {
if (!/^m[0-9a-f]{24}$/.test(shareId)) throw new Error('Invalid peer manga ID');
const res = await request(store, id, `/peer/v1/covers/${shareId}`, { binary: true });
if (res.statusCode !== 200) {
const body = await readResponse(res, 4096).catch(() => Buffer.alloc(0));
let message; try { message = JSON.parse(body.toString()).error; } catch { message = 'Cover unavailable'; }
throw remoteError(res.statusCode, message, res.headers['retry-after']);
}
const mime = String(res.headers['content-type'] || '').split(';')[0];
if (!/^image\/(?:jpeg|png|webp|gif|avif|bmp)$/.test(mime)) throw new Error('Peer returned an unsupported cover type');
const buffer = await readResponse(res, 5 * 1024 * 1024);
if (!buffer.length) throw new Error('Peer returned an empty cover');
return { buffer, mime, etag: res.headers.etag || null };
}
async function catalog(store, id) {
const items = []; let cursor = null;
for (let page = 0; page < 100; page++) {
const result = await jsonRequest(store, id, `/peer/v1/catalog${cursor ? `?cursor=${encodeURIComponent(cursor)}` : ''}`);
if (!Array.isArray(result.items) || result.items.length > 50) throw new Error('Invalid peer catalog');
for (const item of result.items) if (item.workIdentity != null && !normalizeWorkIdentity(item.workIdentity)) throw new Error('Invalid work identity in peer catalog');
items.push(...result.items); cursor = result.nextCursor;
if (!cursor) return items;
}
throw new Error('Peer catalog exceeds page limit');
}
async function manga(store, id, shareId) {
if (!/^m[0-9a-f]{24}$/.test(shareId)) throw new Error('Invalid manga ID');
const chapters = []; let cursor = null, first;
for (let page = 0; page < 100; page++) {
const result = await jsonRequest(store, id, `/peer/v1/manga/${shareId}${cursor ? `?cursor=${encodeURIComponent(cursor)}` : ''}`);
if (result.id !== shareId || !Array.isArray(result.chapters) || result.chapters.length > 100) throw new Error('Invalid peer manga manifest');
if (result.workIdentity != null && !normalizeWorkIdentity(result.workIdentity)) throw new Error('Invalid work identity in peer manga manifest');
first ||= result; chapters.push(...result.chapters); cursor = result.nextCursor;
if (!cursor) return { id: shareId, title: first.title, ...first, chapters };
}
throw new Error('Peer manga exceeds chapter limit');
}
async function chapter(store, id, shareId, chapterId) {
if (!/^m[0-9a-f]{24}$/.test(shareId) || !/^c[0-9a-f]{32}$/.test(chapterId)) throw new Error('Invalid chapter ID');
const result = await jsonRequest(store, id, `/peer/v1/manga/${shareId}/chapters/${chapterId}`);
if (result.id !== chapterId || !Array.isArray(result.pages) || !result.pages.length || result.pages.length > 500) throw new Error('Invalid peer chapter manifest');
for (const page of result.pages) if (!/^b[0-9a-f]{32}$/.test(page.id) || !Number.isSafeInteger(page.bytes) || page.bytes < 1 || page.bytes > MAX_PAGE
|| !/^[0-9a-f]{64}$/.test(page.sha256) || !/^image\/(?:jpeg|png|webp|gif|avif|bmp)$/.test(page.mime)) throw new Error('Invalid peer page');
return result;
}
async function downloadPage(store, id, page, file, shouldStop) {
const verify = async () => {
const digest = crypto.createHash('sha256');
for await (const chunk of fs.createReadStream(file)) digest.update(chunk);
return digest.digest('hex') === page.sha256;
};
let stat = await fsp.stat(file).catch(() => null);
if (stat?.size === page.bytes && await verify()) return { reused: true };
if (stat?.size === page.bytes) { await fsp.rm(file, { force: true }); stat = null; }
if (stat && stat.size > page.bytes) { await fsp.rm(file, { force: true }); stat = null; }
const start = stat?.size || 0;
if (shouldStop?.()) throw new Error('Peer transfer paused');
const res = await request(store, id, `/peer/v1/blobs/${page.id}`, { binary: true, requireReceive: true,
headers: start ? { range: `bytes=${start}-`, ...(page.etag ? { 'if-match': page.etag } : {}) } : {} });
if (res.statusCode !== (start ? 206 : 200)) {
const body = await readResponse(res, 4096).catch(() => Buffer.alloc(0));
let message; try { message = JSON.parse(body.toString()).error; } catch { message = 'Transfer failed'; }
throw remoteError(res.statusCode, message, res.headers['retry-after']);
}
if (start && !String(res.headers['content-range'] || '').startsWith(`bytes ${start}-`)) throw new Error('Peer returned an invalid range');
if (Number(res.headers['content-length']) !== page.bytes - start) throw new Error('Peer returned an unexpected page size');
const rate = requirePeer(store.load(), id).download?.bytesPerSecond || 2 * 1024 * 1024;
const throttle = new Transform({ transform(chunk, _encoding, callback) {
const now = Date.now(), ready = Math.max(now, downloadSlots.get(id) || now) + chunk.length * 1000 / rate;
downloadSlots.set(id, ready);
setTimeout(() => callback(null, chunk), Math.max(0, ready - now));
} });
await pipeline(res, throttle, fs.createWriteStream(file, { flags: start ? 'a' : 'w', mode: 0o600 }));
stat = await fsp.stat(file);
if (stat.size !== page.bytes || !await verify()) { await fsp.rm(file, { force: true }); throw new Error('Peer page checksum mismatch'); }
return { reused: false };
}
module.exports = { connection, jsonRequest, probe, access, catalog, manga, cover, chapter, downloadPage };
+268
View File
@@ -0,0 +1,268 @@
#!/usr/bin/env node
'use strict';
const https = require('node:https');
const fs = require('node:fs');
const fsp = fs.promises;
const path = require('node:path');
const crypto = require('node:crypto');
const { once } = require('node:events');
const { PeeringState, fp, hash, requirePeer, grantStatus } = require('./state');
const { parseKey, parseSpecialFolder } = require('../chapter-identity');
const { resolveWorkIdentity } = require('../scraper/lib/work-identity');
const DATA = path.resolve(process.env.PEER_DATA || path.join(process.env.DATA || 'data', 'peering'));
const LIBRARY = path.resolve(process.env.LIBRARY || 'library');
const HOST = process.env.PEER_HOST || '127.0.0.1';
const PORT = Number(process.env.PEER_PORT || 8093);
const peering = new PeeringState(DATA);
const identity = peering.identity();
if (!identity) throw new Error('Generate peering identity in the admin UI before starting the gateway');
const MAX_PAGE = 30 * 1024 * 1024;
const MAX_COVER = 5 * 1024 * 1024;
const MIME = { '.jpg': 'image/jpeg', '.jpeg': 'image/jpeg', '.png': 'image/png', '.webp': 'image/webp', '.gif': 'image/gif', '.avif': 'image/avif', '.bmp': 'image/bmp' };
const usageFile = path.resolve(process.env.PEER_USAGE_FILE || path.join(DATA, 'gateway-usage.json'));
fs.mkdirSync(path.dirname(usageFile), { recursive: true, mode: 0o700 });
const usage = (() => { try { return JSON.parse(fs.readFileSync(usageFile, 'utf8')); } catch (e) { if (e.code === 'ENOENT') return { day: '', peers: {}, usedInvites: [] }; throw e; } })();
const active = new Map();
const uploadSlots = new Map();
const GLOBAL_STREAMS = Math.max(1, Math.min(32, Number(process.env.PEER_GLOBAL_STREAMS || 4)));
const indexCache = new Map();
const blobCache = new Map();
const sha = s => crypto.createHash('sha256').update(s).digest('hex');
const opaque = s => sha(s).slice(0, 32);
function saveUsage() { const tmp = `${usageFile}.tmp`; fs.writeFileSync(tmp, JSON.stringify(usage), { mode: 0o600 }); fs.renameSync(tmp, usageFile); }
function fail(code, message, retryAfter) { const e = new Error(message); e.code = code; e.retryAfter = retryAfter; throw e; }
function json(res, code, value, extra = {}) {
res.writeHead(code, { 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-store', ...extra });
res.end(JSON.stringify(value));
}
function peerOf(req) {
const cert = req.socket.getPeerCertificate(true);
if (!cert?.raw || Date.parse(cert.valid_to) <= Date.now() || Date.parse(cert.valid_from) > Date.now()) fail(401, 'Valid client certificate required');
const fingerprint = fp(cert.raw);
const state = peering.load();
const peer = state.peers.find(p => p.fingerprint === fingerprint);
if (!peer || fp(peer.cert) !== fingerprint) fail(401, 'Unrecognized client certificate');
return { state, peer };
}
function grant(req, peer, scope = 'catalog') {
const status = grantStatus(peer);
if (!peer.paired || status.status !== 'active') fail(403, `Peering grant ${status.status}`);
const match = /^Bearer ([A-Za-z0-9_-]{40,60})$/.exec(String(req.headers.authorization || ''));
if (!match) fail(401, 'Grant required');
const got = Buffer.from(hash(match[1]), 'hex'), want = Buffer.from(peer.grantHash, 'hex');
if (got.length !== want.length || !crypto.timingSafeEqual(got, want)) fail(401, 'Invalid grant');
if (!status.scopes.includes(scope)) fail(403, `Grant does not allow ${scope}`);
}
function quota(peer, bytes = 0, catalog = false) {
const now = new Date(), day = now.toISOString().slice(0, 10), minute = Math.floor(Date.now() / 60000);
if (usage.day !== day) { usage.day = day; usage.peers = {}; saveUsage(); }
const entry = usage.peers[peer.id] ||= { bytes: 0, minute, catalogs: 0 };
if (entry.minute !== minute) { entry.minute = minute; entry.catalogs = 0; }
if (catalog && entry.catalogs >= peer.limits.catalogPerMinute) fail(429, 'Catalog request limit reached', 60);
if (entry.bytes + bytes > peer.limits.bytesPerDay) fail(429, 'Daily upload limit reached', 3600);
if (catalog) entry.catalogs++;
if (bytes) entry.bytes += bytes; // Reserve before streaming; an interrupted stream never resets quota.
saveUsage();
}
function safeDir(parent, name) {
if (typeof name !== 'string' || !name || name.startsWith('.') || name.includes('/') || name.includes('\\') || name === '..') return null;
const file = path.join(parent, name);
if (!file.startsWith(parent + path.sep)) return null;
try { if (!fs.lstatSync(file).isDirectory()) return null; } catch { return null; }
return file;
}
async function dirs(root) {
try { return (await fsp.readdir(root, { withFileTypes: true })).filter(e => e.isDirectory() && !e.isSymbolicLink() && !e.name.startsWith('.')).map(e => e.name).sort(); }
catch { return []; }
}
async function jsonFile(file) { try { return JSON.parse(await fsp.readFile(file, 'utf8')); } catch { return {}; } }
function publicMetadata(meta = {}) {
const tags = value => Array.isArray(value) ? value.map(item => String(typeof item === 'string' ? item : item?.name || '')).filter(Boolean).slice(0, 40) : [];
return { year: Number(meta.year || meta.startDate?.year) || null, status: String(meta.status || 'unknown').slice(0, 40),
description: String(meta.description || meta.synopsis || '').slice(0, 3000), tags: tags(meta.tags), genres: tags(meta.genres),
direction: ['rtl', 'ltr', 'ttb'].includes(meta.direction) ? meta.direction : null };
}
function coverFile(mangaDir) {
try {
const entry = fs.readdirSync(mangaDir, { withFileTypes: true }).find(item => item.isFile() && !item.isSymbolicLink()
&& /^cover\.(jpe?g|png|webp|gif|avif|bmp)$/i.test(item.name));
if (!entry) return null;
const file = path.join(mangaDir, entry.name), stat = fs.lstatSync(file);
return stat.isFile() && stat.size > 0 && stat.size <= MAX_COVER ? { file, size: stat.size, mime: MIME[path.extname(file).toLowerCase()], mtimeMs: stat.mtimeMs } : null;
} catch { return null; }
}
async function mangaIndex(peer, share) {
const key = `${peer.id}:${share.id}`;
const old = indexCache.get(key);
if (old && Date.now() - old.at < 30000) return old;
const mangaDir = safeDir(LIBRARY, share.mangaId);
if (!mangaDir) fail(404, 'Manga not found');
const meta = await jsonFile(path.join(mangaDir, 'manga.json'));
const chapters = [], byId = new Map();
for (const bookName of await dirs(mangaDir)) {
const book = parseKey(bookName);
if (!book) continue;
const bookDir = safeDir(mangaDir, bookName);
for (const chapterName of await dirs(bookDir)) {
const parsed = parseSpecialFolder(chapterName) || parseKey(chapterName);
if (!parsed) continue;
const chapterDir = safeDir(bookDir, chapterName);
// A chapter without final metadata may still be a manually written partial.
if (!fs.existsSync(path.join(chapterDir, 'chapter.json'))) continue;
const entries = await fsp.readdir(chapterDir, { withFileTypes: true });
const pages = entries.filter(e => e.isFile() && MIME[path.extname(e.name).toLowerCase()]).map(e => e.name).sort();
if (!pages.length) continue;
const id = `c${opaque(`${share.id}/${bookName}/${chapterName}`)}`;
const chapterMeta = await jsonFile(path.join(chapterDir, 'chapter.json'));
const item = { id, number: parsed.key, volume: book.key, title: String(chapterMeta.title || parsed.title || '').slice(0, 200),
pageCount: pages.length, chapterDir, pages };
chapters.push(item); byId.set(id, item);
if (chapters.length > 10000) fail(413, 'Shared manga chapter limit exceeded');
}
}
const result = { at: Date.now(), mangaDir, meta, chapters, byId };
indexCache.set(key, result);
return result;
}
function permittedShare(peer, id) {
const share = (peer.shares || []).find(s => s.id === id);
if (!share) fail(404, 'Manga not found');
return share;
}
async function chapterManifest(peer, share, chapterId) {
const index = await mangaIndex(peer, share), chapter = index.byId.get(chapterId);
if (!chapter) fail(404, 'Chapter not found');
const pages = [];
for (const name of chapter.pages) {
const file = path.join(chapter.chapterDir, name), stat = await fsp.lstat(file);
if (!stat.isFile() || stat.size < 1 || stat.size > MAX_PAGE) fail(413, 'Page size outside transfer limits');
const digest = crypto.createHash('sha256');
for await (const chunk of fs.createReadStream(file)) digest.update(chunk);
const id = `b${opaque(`${share.id}/${chapter.id}/${name}`)}`;
const etag = `"${id}-${stat.size}-${Math.round(stat.mtimeMs)}"`;
pages.push({ id, bytes: stat.size, sha256: digest.digest('hex'), mime: MIME[path.extname(name).toLowerCase()], etag });
blobCache.set(`${peer.id}:${id}`, { file, chapterDir: chapter.chapterDir, shareId: share.id, chapterId,
size: stat.size, mtimeMs: stat.mtimeMs, at: Date.now() });
if (pages.length > 500) fail(413, 'Chapter page limit exceeded');
}
return { id: chapter.id, number: chapter.number, volume: chapter.volume, title: chapter.title, pages };
}
async function streamBlob(req, res, peer, id) {
const item = blobCache.get(`${peer.id}:${id}`);
if (!item || Date.now() - item.at > 30 * 60 * 1000) fail(404, 'Refresh chapter manifest');
permittedShare(peer, item.shareId);
if (!fs.existsSync(path.join(item.chapterDir, 'chapter.json'))) fail(404, 'Chapter no longer complete');
const stat = await fsp.lstat(item.file).catch(() => null);
if (!stat?.isFile() || stat.size !== item.size || stat.mtimeMs !== item.mtimeMs) fail(412, 'Page changed; refresh manifest');
const etag = `"${id}-${stat.size}-${Math.round(stat.mtimeMs)}"`;
if (req.headers['if-match'] && req.headers['if-match'] !== etag) fail(412, 'Page changed; refresh manifest');
const match = /^bytes=(\d+)-$/.exec(String(req.headers.range || ''));
if (req.headers.range && !match) fail(416, 'Unsupported range');
const start = match ? Number(match[1]) : 0;
if (!Number.isSafeInteger(start) || start >= stat.size) fail(416, 'Range outside page');
const count = stat.size - start;
if ((active.get(peer.id) || 0) >= peer.limits.concurrent) fail(429, 'Too many transfers', 2);
if ([...active.values()].reduce((a, b) => a + b, 0) >= GLOBAL_STREAMS) fail(429, 'Server transfer limit reached', 2);
quota(peer, count);
active.set(peer.id, (active.get(peer.id) || 0) + 1);
try {
res.writeHead(start ? 206 : 200, {
'content-type': MIME[path.extname(item.file).toLowerCase()], 'content-length': count,
'accept-ranges': 'bytes', etag,
...(start ? { 'content-range': `bytes ${start}-${stat.size - 1}/${stat.size}` } : {}),
'cache-control': 'no-store',
});
const stream = fs.createReadStream(item.file, { start, highWaterMark: 64 * 1024 });
for await (const chunk of stream) {
if (req.destroyed || res.destroyed) break;
// Reserve each chunk on a shared per-peer timeline, including simultaneous streams.
const now = Date.now(), ready = Math.max(now, uploadSlots.get(peer.id) || now) + chunk.length * 1000 / peer.limits.bytesPerSecond;
uploadSlots.set(peer.id, ready);
if (ready > now) await new Promise(resolve => setTimeout(resolve, ready - now));
if (!res.write(chunk)) await once(res, 'drain');
}
res.end();
} finally { active.set(peer.id, Math.max(0, (active.get(peer.id) || 1) - 1)); }
}
async function streamCover(req, res, peer, shareId) {
const share = permittedShare(peer, shareId), mangaDir = safeDir(LIBRARY, share.mangaId);
if (!mangaDir) fail(404, 'Manga not found');
const cover = coverFile(mangaDir);
if (!cover) fail(404, 'Cover not available');
const etag = `\"${opaque(`${share.id}/${cover.size}/${cover.mtimeMs}`)}\"`;
if (req.headers['if-none-match'] === etag) { res.writeHead(304, { etag, 'cache-control': 'private, max-age=3600' }); return res.end(); }
quota(peer, cover.size);
res.writeHead(200, { 'content-type': cover.mime, 'content-length': cover.size, etag, 'cache-control': 'private, max-age=3600', 'x-content-type-options': 'nosniff' });
if (req.method === 'HEAD') return res.end();
await new Promise((resolve, reject) => fs.createReadStream(cover.file).on('error', reject).on('end', resolve).pipe(res));
}
const server = https.createServer({ key: identity.key, cert: identity.cert, requestCert: true, rejectUnauthorized: false, minVersion: 'TLSv1.3' }, async (req, res) => {
try {
const url = new URL(req.url, 'https://peer');
if (req.method !== 'GET') fail(405, 'GET only');
if (url.pathname === '/health') return json(res, 200, { ok: true });
const { state, peer } = peerOf(req);
if (url.pathname === '/peer/v1/hello') {
if (!peer.paired) {
const invitation = state.invitations.find(i => i.id === req.headers['x-invitation-id']);
const code = String(req.headers['x-invitation-code'] || '');
if (!invitation || Date.parse(invitation.expiresAt) <= Date.now() || invitation.codeHash !== hash(code)
|| usage.usedInvites.includes(invitation.id)) fail(403, 'Valid unused invitation required');
usage.usedInvites.push(invitation.id); usage.usedInvites = usage.usedInvites.slice(-1000); saveUsage();
}
return json(res, 200, { version: 1, id: identity.id, fingerprint: identity.fingerprint });
}
if (url.pathname === '/peer/v1/access') {
return json(res, 200, { version: 1, paired: !!peer.paired, name: peering.load().name,
certificateExpiresAt: new crypto.X509Certificate(identity.cert).validTo,
grant: grantStatus(peer) });
}
const requiredScope = url.pathname.startsWith('/peer/v1/blobs/') ? 'download' : 'catalog';
grant(req, peer, requiredScope);
if (url.pathname === '/peer/v1/catalog') {
quota(peer, 0, true);
const cursor = Number(url.searchParams.get('cursor') || 0);
if (!Number.isSafeInteger(cursor) || cursor < 0) fail(400, 'Invalid cursor');
const shares = peer.shares || [], items = [];
for (const share of shares.slice(cursor, cursor + 50)) {
const mangaDir = safeDir(LIBRARY, share.mangaId);
if (!mangaDir) continue;
const meta = await jsonFile(path.join(mangaDir, 'manga.json'));
items.push({ id: share.id, title: String(meta.title || share.mangaId).slice(0, 200),
...publicMetadata(meta), workIdentity: resolveWorkIdentity(meta, share.mangaId, identity.id),
coverAvailable: !!coverFile(mangaDir) });
}
return json(res, 200, { version: 1, items, nextCursor: cursor + 50 < shares.length ? String(cursor + 50) : null });
}
const coverMatch = /^\/peer\/v1\/covers\/(m[0-9a-f]{24})$/.exec(url.pathname);
if (coverMatch) return await streamCover(req, res, peer, coverMatch[1]);
const mangaMatch = /^\/peer\/v1\/manga\/(m[0-9a-f]{24})$/.exec(url.pathname);
if (mangaMatch) {
quota(peer, 0, true);
const share = permittedShare(peer, mangaMatch[1]), index = await mangaIndex(peer, share);
const cursor = Number(url.searchParams.get('cursor') || 0);
if (!Number.isSafeInteger(cursor) || cursor < 0) fail(400, 'Invalid cursor');
return json(res, 200, { id: share.id, title: String(index.meta.title || share.mangaId).slice(0, 200),
workIdentity: resolveWorkIdentity(index.meta, share.mangaId, identity.id),
...publicMetadata(index.meta), coverAvailable: !!coverFile(index.mangaDir),
chapters: index.chapters.slice(cursor, cursor + 100).map(({ id, number, volume, title, pageCount }) => ({ id, number, volume, title, pageCount })),
nextCursor: cursor + 100 < index.chapters.length ? String(cursor + 100) : null });
}
const chapterMatch = /^\/peer\/v1\/manga\/(m[0-9a-f]{24})\/chapters\/(c[0-9a-f]{32})$/.exec(url.pathname);
if (chapterMatch) {
quota(peer, 0, true);
return json(res, 200, await chapterManifest(peer, permittedShare(peer, chapterMatch[1]), chapterMatch[2]));
}
const blobMatch = /^\/peer\/v1\/blobs\/(b[0-9a-f]{32})$/.exec(url.pathname);
if (blobMatch) return await streamBlob(req, res, peer, blobMatch[1]);
fail(404, 'Not found');
} catch (error) {
if (!res.headersSent) json(res, Number(error.code) || 500, { error: Number(error.code) ? error.message : 'Internal error' }, error.retryAfter ? { 'retry-after': String(error.retryAfter) } : {});
else res.destroy();
if (!Number(error.code)) console.error('[peer-gateway]', error.message);
}
});
server.listen(PORT, HOST, () => console.log(`[peer-gateway] listening on ${HOST}:${PORT}`));
+248
View File
@@ -0,0 +1,248 @@
'use strict';
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');
const { execFileSync } = require('node:child_process');
const MAX_PEERS = 64;
const INVITE_MS = 48 * 60 * 60 * 1000;
const GRANT_MS = 90 * 24 * 60 * 60 * 1000;
const NAME_RE = /^[A-Za-z0-9]{1,25}$/;
const GRANT_SCOPES = new Set(['catalog', 'download']);
const fp = pem => new crypto.X509Certificate(pem).fingerprint256.replaceAll(':', '').toLowerCase();
const hash = value => crypto.createHash('sha256').update(value).digest('hex');
const peerId = pem => `p${fp(pem).slice(0, 24)}`;
const own = (root, name) => path.join(root, name);
function readJson(file, fallback) {
try { return JSON.parse(fs.readFileSync(file, 'utf8')); }
catch (e) { if (e.code === 'ENOENT') return fallback; throw e; }
}
function atomic(file, value) {
const temp = `${file}.${crypto.randomBytes(6).toString('hex')}.tmp`;
fs.writeFileSync(temp, JSON.stringify(value, null, 2) + '\n', { mode: 0o600 });
fs.renameSync(temp, file);
}
function requirePeer(state, id) {
const peer = state.peers.find(p => p.id === id);
if (!peer) throw new Error('Peer not found');
return peer;
}
function validUrl(value) {
const url = new URL(String(value));
if (url.protocol !== 'https:' || url.username || url.password || url.search || url.hash || url.pathname !== '/') throw new Error('Peer URL must be an HTTPS origin');
if (!url.hostname || url.hostname.length > 253) throw new Error('Invalid peer hostname');
return url.origin;
}
function publicPeer(peer, outbound) {
const grantValid = !!peer.grantHash && Date.parse(peer.grantExpiresAt) > Date.now();
return {
id: peer.id, alias: peer.alias, url: peer.url, fingerprint: peer.fingerprint,
paired: !!peer.paired, serve: grantValid && !peer.servingPaused, receive: !!peer.receive,
grantStatus: !peer.grantHash ? 'none' : !grantValid ? 'expired' : peer.servingPaused ? 'paused' : 'active',
grantScopes: peer.grantHash ? scopesFor(peer) : [], grantIssuedAt: peer.grantIssuedAt || null,
readerQueue: !!peer.readerQueue, shares: peer.shares || [],
limits: peer.limits, download: peer.download,
grantExpiresAt: peer.grantExpiresAt || null,
hasOutboundGrant: !!outbound?.[peer.id],
invitationExpiresAt: peer.invitationExpiresAt || null,
};
}
function serverName(value) {
const name = String(value ?? '').trim();
if (!NAME_RE.test(name)) throw new Error('Server name must contain 1–25 letters or numbers');
return name;
}
function normalizeScopes(value) {
const scopes = new Set((Array.isArray(value) ? value : []).map(String).filter(scope => GRANT_SCOPES.has(scope)));
if (scopes.has('download')) scopes.add('catalog');
return [...GRANT_SCOPES].filter(scope => scopes.has(scope));
}
function scopesFor(peer) {
return Array.isArray(peer.grantScopes) ? normalizeScopes(peer.grantScopes) : ['catalog', 'download'];
}
function grantStatus(peer) {
if (!peer.grantHash) return { status: 'none', scopes: [], issuedAt: null, expiresAt: null };
const scopes = scopesFor(peer), expiresAt = peer.grantExpiresAt || null;
const status = !expiresAt || Date.parse(expiresAt) <= Date.now() ? 'expired' : peer.servingPaused ? 'paused' : 'active';
return { status, scopes: status === 'expired' ? [] : scopes, issuedAt: peer.grantIssuedAt || null, expiresAt };
}
function normalizeLimits(input = {}, previous = {}) {
const defaults = { concurrent: 2, bytesPerSecond: 2 * 1024 * 1024, bytesPerDay: 10 * 1024 ** 3, catalogPerMinute: 30 };
const result = { ...defaults, ...previous };
for (const [key, min, max] of [['concurrent', 1, 8], ['bytesPerSecond', 65536, 100 * 1024 * 1024],
['bytesPerDay', 1024 * 1024, 10 * 1024 ** 4], ['catalogPerMinute', 1, 600]]) {
if (input[key] === undefined) continue;
const value = Number(input[key]);
if (!Number.isSafeInteger(value) || value < min || value > max) throw new Error(`Invalid ${key}`);
result[key] = value;
}
return result;
}
function normalizeDownload(input = {}, previous = {}) {
const result = { concurrent: 2, bytesPerSecond: 2 * 1024 * 1024, ...previous };
for (const [key, min, max] of [['concurrent', 1, 4], ['bytesPerSecond', 65536, 100 * 1024 * 1024]]) {
if (input[key] === undefined) continue;
const value = Number(input[key]);
if (!Number.isSafeInteger(value) || value < min || value > max) throw new Error(`Invalid ${key}`);
result[key] = value;
}
return result;
}
class PeeringState {
constructor(root) { this.root = path.resolve(root); }
stateFile() { return own(this.root, 'state.json'); }
outboundFile() { return own(this.root, 'outbound.json'); }
certFile() { return own(this.root, 'identity.crt'); }
keyFile() { return own(this.root, 'identity.key'); }
load() { return readJson(this.stateFile(), { version: 1, name: '', peers: [], invitations: [] }); }
outbound() { return readJson(this.outboundFile(), {}); }
identity() {
if (!fs.existsSync(this.certFile()) || !fs.existsSync(this.keyFile())) return null;
const cert = fs.readFileSync(this.certFile(), 'utf8');
return { cert, key: fs.readFileSync(this.keyFile(), 'utf8'), fingerprint: fp(cert), id: peerId(cert) };
}
setName(value) {
const state = this.load(); state.name = serverName(value); atomic(this.stateFile(), state);
return state.name;
}
generateIdentity(name) {
fs.mkdirSync(this.root, { recursive: true, mode: 0o700 });
if (this.identity()) throw new Error('Identity already exists; rotate it through a separate recovery procedure');
const state = this.load();
const cleanName = serverName(name || state.name);
if (state.name && state.name !== cleanName) throw new Error('Save this server name before generating its certificate');
const openssl = process.env.OPENSSL_BIN || 'openssl';
const key = this.keyFile(), cert = this.certFile();
try {
execFileSync(openssl, ['req', '-x509', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:P-256', '-sha256', '-nodes',
'-keyout', key, '-out', cert, '-days', '365', '-subj', `/CN=${cleanName}`], { stdio: 'pipe', timeout: 15000 });
fs.chmodSync(key, 0o600); fs.chmodSync(cert, 0o600);
state.name = cleanName; atomic(this.stateFile(), state);
return { id: peerId(fs.readFileSync(cert, 'utf8')), fingerprint: fp(fs.readFileSync(cert, 'utf8')), name: cleanName };
} catch (error) {
for (const file of [key, cert]) { try { fs.rmSync(file, { force: true }); } catch {} }
throw new Error(`Could not generate peering identity: ${error.message}`);
}
}
createInvitation(url) {
const identity = this.identity();
if (!identity) throw new Error('Generate a server identity first');
const origin = validUrl(url);
const state = this.load();
const id = crypto.randomUUID(), code = crypto.randomBytes(32).toString('base64url');
const expiresAt = new Date(Date.now() + INVITE_MS).toISOString();
state.invitations = (state.invitations || []).filter(item => Date.parse(item.expiresAt) > Date.now()).slice(-50);
state.invitations.push({ id, codeHash: hash(code), expiresAt });
atomic(this.stateFile(), state);
return { version: 1, name: state.name, url: origin, cert: identity.cert,
fingerprint: identity.fingerprint, invitationId: id, code, expiresAt };
}
importInvitation(card) {
if (!card || card.version !== 1 || typeof card.cert !== 'string' || card.cert.length > 10000) throw new Error('Invalid invitation');
if (Date.parse(card.expiresAt) <= Date.now() || Date.parse(card.expiresAt) > Date.now() + INVITE_MS + 60000) throw new Error('Invitation expired or invalid');
if (!crypto.randomUUID || !/^[0-9a-f-]{36}$/i.test(String(card.invitationId))) throw new Error('Invalid invitation ID');
if (!/^[A-Za-z0-9_-]{40,60}$/.test(String(card.code))) throw new Error('Invalid invitation code');
const cert = new crypto.X509Certificate(card.cert);
const name = serverName(card.name);
if (Date.parse(cert.validTo) <= Date.now() || Date.parse(cert.validFrom) > Date.now()) throw new Error('Peer certificate is not valid now');
const fingerprint = fp(card.cert);
if (fingerprint !== String(card.fingerprint).replaceAll(':', '').toLowerCase()) throw new Error('Invitation fingerprint mismatch');
if (this.identity()?.fingerprint === fingerprint) throw new Error('Cannot pair with this server itself');
const state = this.load();
if (state.peers.length >= MAX_PEERS && !state.peers.some(p => p.fingerprint === fingerprint)) throw new Error('Peer limit reached');
let peer = state.peers.find(p => p.fingerprint === fingerprint);
if (!peer) {
peer = { id: peerId(card.cert), alias: name, url: validUrl(card.url),
cert: card.cert, fingerprint, paired: false, servingPaused: false, receive: false, readerQueue: false,
shares: [], limits: normalizeLimits(), download: normalizeDownload() };
state.peers.push(peer);
} else if (peer.url !== validUrl(card.url)) {
// A changed endpoint needs a new admin-reviewed invitation; never silently redirect traffic.
peer.url = validUrl(card.url); peer.paired = false; peer.servingPaused = false; peer.receive = false;
delete peer.grantHash; delete peer.grantScopes; delete peer.grantExpiresAt; delete peer.grantIssuedAt;
}
peer.invitationId = card.invitationId; peer.invitationCode = card.code; peer.invitationExpiresAt = card.expiresAt;
atomic(this.stateFile(), state);
return publicPeer(peer, this.outbound());
}
updatePeer(id, patch) {
const state = this.load(), peer = requirePeer(state, id);
if (patch.alias !== undefined) peer.alias = serverName(patch.alias);
if (patch.limits !== undefined) peer.limits = normalizeLimits(patch.limits, peer.limits);
if (patch.download !== undefined) peer.download = normalizeDownload(patch.download, peer.download);
if (patch.readerQueue !== undefined) peer.readerQueue = !!patch.readerQueue;
if (patch.receive !== undefined) peer.receive = !!patch.receive && !!peer.paired;
if (patch.servingPaused !== undefined) peer.servingPaused = !!patch.servingPaused;
// Accept the legacy field without revoking credentials: false now means pause.
if (patch.serve !== undefined) {
if (patch.serve && !peer.paired) throw new Error('Verify pairing before enabling serving');
peer.servingPaused = !patch.serve;
}
if (patch.grantScopes !== undefined) {
if (!peer.grantHash) throw new Error('Issue a grant before changing its permissions');
const scopes = normalizeScopes(patch.grantScopes);
if (!scopes.length) throw new Error('Select at least one grant permission or revoke the grant');
peer.grantScopes = scopes;
}
atomic(this.stateFile(), state);
return publicPeer(peer, this.outbound());
}
setShare(id, mangaId, enabled) {
const peer = requirePeer(this.load(), id), ids = new Set((peer.shares || []).map(s => s.mangaId));
if (enabled) ids.add(mangaId); else ids.delete(mangaId);
return this.setShares(id, [...ids]);
}
setShares(id, mangaIds) {
const state = this.load(), peer = requirePeer(state, id);
if (!Array.isArray(mangaIds) || mangaIds.length > 5000) throw new Error('Choose no more than 5,000 manga');
const unique = [...new Set(mangaIds)];
if (unique.some(mangaId => typeof mangaId !== 'string' || !mangaId || mangaId.startsWith('.') || mangaId.includes('/') || mangaId.includes('\\')))
throw new Error('Invalid manga ID');
const existing = new Map((peer.shares || []).map(share => [share.mangaId, share]));
peer.shares = unique.map(mangaId => existing.get(mangaId) || { mangaId, id: `m${crypto.randomBytes(12).toString('hex')}` });
atomic(this.stateFile(), state);
return publicPeer(peer, this.outbound());
}
issueGrant(id, scopes = ['catalog', 'download']) {
const state = this.load(), peer = requirePeer(state, id);
if (!peer.paired) throw new Error('Verify pairing before issuing a grant');
const selected = normalizeScopes(scopes);
if (!selected.length) throw new Error('Select at least one grant permission');
const token = crypto.randomBytes(32).toString('base64url');
peer.grantHash = hash(token);
peer.grantScopes = selected; peer.grantIssuedAt = new Date().toISOString(); peer.servingPaused = false;
peer.grantExpiresAt = new Date(Math.min(Date.now() + GRANT_MS, Date.parse(new crypto.X509Certificate(peer.cert).validTo), Date.parse(new crypto.X509Certificate(this.identity().cert).validTo))).toISOString();
atomic(this.stateFile(), state);
return { peerId: peer.id, token, expiresAt: peer.grantExpiresAt };
}
revokeGrant(id) {
const state = this.load(), peer = requirePeer(state, id);
for (const key of ['grantHash', 'grantScopes', 'grantIssuedAt', 'grantExpiresAt']) delete peer[key];
peer.servingPaused = false; atomic(this.stateFile(), state);
return publicPeer(peer, this.outbound());
}
acceptGrant(id, token) {
const state = this.load(), peer = requirePeer(state, id);
if (!/^[A-Za-z0-9_-]{40,60}$/.test(String(token))) throw new Error('Invalid grant token');
const outbound = this.outbound(); outbound[id] = String(token); atomic(this.outboundFile(), outbound);
return publicPeer(peer, outbound);
}
removePeer(id) {
const state = this.load(); requirePeer(state, id);
state.peers = state.peers.filter(p => p.id !== id); atomic(this.stateFile(), state);
const outbound = this.outbound(); delete outbound[id]; atomic(this.outboundFile(), outbound);
}
publicState() {
const state = this.load(), identity = this.identity(), outbound = this.outbound();
return { identity: identity ? { id: identity.id, fingerprint: identity.fingerprint, name: state.name,
expiresAt: new crypto.X509Certificate(identity.cert).validTo } : null,
peers: state.peers.map(p => publicPeer(p, outbound)),
invitations: (state.invitations || []).filter(i => Date.parse(i.expiresAt) > Date.now()).map(i => ({ expiresAt: i.expiresAt })),
certificateExpiresAt: identity ? new crypto.X509Certificate(identity.cert).validTo : null };
}
}
module.exports = { PeeringState, fp, hash, peerId, validUrl, requirePeer, normalizeLimits, normalizeDownload,
serverName, normalizeScopes, scopesFor, grantStatus };
+214
View File
@@ -0,0 +1,214 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const fsp = fs.promises;
const os = require('node:os');
const path = require('node:path');
const net = require('node:net');
const https = require('node:https');
const { spawn } = require('node:child_process');
const { PeeringState } = require('../state');
const client = require('../client');
const ROOT = path.resolve(__dirname, '../..');
const OPENSSL = process.env.OPENSSL_BIN || (process.platform === 'win32' ? 'C:\\Program Files\\Git\\usr\\bin\\openssl.exe' : 'openssl');
const PNG = Buffer.from('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mP8/x8AAusB9Y9Zr5sAAAAASUVORK5CYII=', 'base64');
async function port() {
const server = net.createServer(); await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
const value = server.address().port; await new Promise(resolve => server.close(resolve)); return value;
}
async function health(value) {
return new Promise((resolve, reject) => {
https.get({ hostname: '127.0.0.1', port: value, path: '/health', rejectUnauthorized: false }, res => {
res.resume(); res.on('end', () => resolve(res.statusCode));
}).on('error', reject);
});
}
async function gateway(data, library, value, usage) {
const child = spawn(process.execPath, ['peering/gateway.js'], { cwd: ROOT, env: { ...process.env,
PEER_DATA: data, LIBRARY: library, PEER_PORT: String(value), PEER_HOST: '127.0.0.1', PEER_USAGE_FILE: usage }, stdio: ['ignore', 'pipe', 'pipe'] });
let error = ''; child.stderr.on('data', chunk => { error += chunk.toString(); });
for (let i = 0; i < 50; i++) {
if (child.exitCode !== null) throw new Error(`Gateway exited: ${error}`);
try { if (await health(value) === 200) return child; } catch {}
await new Promise(resolve => setTimeout(resolve, 100));
}
child.kill(); throw new Error(`Gateway failed to start: ${error}`);
}
async function stop(child) {
if (child.exitCode !== null) return;
child.kill('SIGTERM'); await new Promise(resolve => { child.once('exit', resolve); setTimeout(resolve, 3000).unref(); });
}
test('mTLS pairing, opt-in catalog, peer queue publication, and revocation', { timeout: 30000 }, async t => {
const root = await fsp.mkdtemp(path.join(os.tmpdir(), 'mangacircus-peering-'));
const dataA = path.join(root, 'a-data'), dataB = path.join(root, 'b-data');
const libA = path.join(root, 'a-library'), libB = path.join(root, 'b-library');
await Promise.all([dataA, dataB, libA, libB].map(dir => fsp.mkdir(dir, { recursive: true })));
t.after(async () => { await fsp.rm(root, { recursive: true, force: true }); });
const a = new PeeringState(dataA), b = new PeeringState(dataB);
process.env.OPENSSL_BIN = OPENSSL;
a.generateIdentity('ServerA'); b.generateIdentity('ServerB');
const beforeInvalidInvite = a.load().invitations.length;
assert.throws(() => a.createInvitation('https://localhost:8080/path'), /HTTPS origin/);
assert.throws(() => a.createInvitation('localhost:8080'), /HTTPS origin/);
assert.equal(a.load().invitations.length, beforeInvalidInvite, 'Invalid URLs must not record unusable invitations');
const portA = await port(), portB = await port();
const inviteA = a.createInvitation(`https://127.0.0.1:${portA}/`);
const inviteB = b.createInvitation(`https://127.0.0.1:${portB}/`);
const inA = a.importInvitation(inviteB), inB = b.importInvitation(inviteA);
assert.equal(inA.paired, false);
assert.equal(a.importInvitation(inviteB).id, inA.id, 'Importing one certificate must not duplicate a peer');
assert.throws(() => a.importInvitation({ ...inviteB, fingerprint: '0'.repeat(64) }), /fingerprint mismatch/);
assert.throws(() => a.importInvitation({ ...inviteB, expiresAt: new Date(Date.now() - 1000).toISOString() }), /expired/);
await assert.rejects(() => client.catalog(b, inB.id), /Verify peer identity first/);
const mangaDir = path.join(libA, 'Sample Manga');
const chapterDir = path.join(mangaDir, 'v01', 'c0001 [Local]');
await fsp.mkdir(chapterDir, { recursive: true });
await fsp.writeFile(path.join(mangaDir, 'manga.json'), JSON.stringify({ title: 'Sample Manga', direction: 'ltr' }));
await fsp.writeFile(path.join(mangaDir, 'cover.png'), PNG);
await fsp.writeFile(path.join(chapterDir, 'chapter.json'), JSON.stringify({ title: 'The first chapter', source: { name: 'Local' } }));
await fsp.writeFile(path.join(chapterDir, '0001.png'), PNG);
const ga = await gateway(dataA, libA, portA, path.join(root, 'usage-a.json'));
let gb = await gateway(dataB, libB, portB, path.join(root, 'usage-b.json'));
t.after(async () => { await stop(ga); await stop(gb); });
await client.probe(b, inB.id);
await client.probe(a, inA.id);
assert.equal(a.publicState().peers[0].paired, true);
a.setShare(inA.id, 'Sample Manga', true);
const c = new PeeringState(path.join(root, 'c-data'));
c.generateIdentity('ServerC');
const inC = a.importInvitation(c.createInvitation('https://third-peer.example:8093/'));
assert.equal(a.publicState().peers.length, 2);
assert.deepEqual(a.publicState().peers.find(p => p.id === inC.id).shares, []);
assert.equal(a.publicState().peers.find(p => p.id === inA.id).shares.length, 1);
a.updatePeer(inA.id, { serve: true });
const grant = a.issueGrant(inA.id, ['catalog']);
b.acceptGrant(inB.id, grant.token);
const listing = await client.catalog(b, inB.id);
assert.equal(listing.length, 1);
assert.equal(listing[0].title, 'Sample Manga');
assert.equal(listing[0].coverAvailable, true);
assert.deepEqual(listing[0].workIdentity, require('../../scraper/lib/work-identity').resolveWorkIdentity({}, 'Sample Manga', a.identity().id));
assert.deepEqual(listing[0].tags, []);
assert.equal(JSON.stringify(listing).includes('Sample Manga\\v01'), false);
const remote = await client.manga(b, inB.id, listing[0].id);
assert.equal(remote.chapters.length, 1);
const cover = await client.cover(b, inB.id, listing[0].id);
assert.equal(cover.mime, 'image/png');
assert.deepEqual(cover.buffer, PNG, 'Catalog-only grants may browse approved covers');
const remoteAccess = await client.access(b, inB.id);
assert.equal(remoteAccess.grant.status, 'active');
assert.deepEqual(remoteAccess.grant.scopes, ['catalog']);
assert.throws(() => client.connection(b, inB.id, true, true), /Downloads from this peer are disabled/);
b.updatePeer(inB.id, { receive: true });
const manifest = await client.chapter(b, inB.id, listing[0].id, remote.chapters[0].id);
assert.equal(manifest.pages.length, 1);
assert.equal(manifest.pages[0].sha256, require('node:crypto').createHash('sha256').update(PNG).digest('hex'));
await assert.rejects(() => client.downloadPage(b, inB.id, manifest.pages[0], path.join(root, 'denied.png')),
/Peer HTTP 403: Grant does not allow download/);
a.updatePeer(inA.id, { grantScopes: ['catalog', 'download'] });
assert.deepEqual((await client.access(b, inB.id)).grant.scopes, ['catalog', 'download'], 'Permission edits apply without exchanging a new token');
a.updatePeer(inA.id, { grantScopes: ['download'] });
assert.deepEqual((await client.access(b, inB.id)).grant.scopes, ['catalog', 'download'], 'Download always implies catalog in canonical order');
await assert.rejects(() => client.jsonRequest(b, inB.id, '/peer/v1/manga/m000000000000000000000000'), /Peer HTTP 404/);
await assert.rejects(() => client.jsonRequest(b, inB.id, '/peer/v1/catalog', { headers: { authorization: 'Bearer ' + 'x'.repeat(43) } }), /Peer HTTP 401/);
const noCert = await new Promise((resolve, reject) => https.get({ hostname: '127.0.0.1', port: portA,
path: '/peer/v1/catalog', rejectUnauthorized: false }, res => { res.resume(); res.on('end', () => resolve(res.statusCode)); }).on('error', reject));
assert.equal(noCert, 401);
const page = manifest.pages[0], { agent } = client.connection(b, inB.id);
const partial = await new Promise((resolve, reject) => https.get(`https://127.0.0.1:${portA}/peer/v1/blobs/${page.id}`,
{ agent, headers: { authorization: `Bearer ${b.outbound()[inB.id]}`, range: 'bytes=4-', 'if-match': page.etag } }, res => {
const chunks = []; res.on('data', chunk => chunks.push(chunk)); res.on('end', () => resolve({ status: res.statusCode, body: Buffer.concat(chunks), range: res.headers['content-range'] }));
}).on('error', reject));
assert.equal(partial.status, 206);
assert.equal(partial.range, `bytes 4-${PNG.length - 1}/${PNG.length}`);
assert.deepEqual(partial.body, PNG.subarray(4));
const { infoFor, run } = require('../adapter');
process.env.PEER_DATA = dataB;
const uri = `peer://${inB.id}/${listing[0].id}`;
const info = infoFor(uri);
const result = await run(info, { scope: { mode: 'missing' }, existingPolicy: 'skip-any' },
{ libraryRoot: libB, settings: { chapterMinMs: 500 }, shouldStop: () => false });
assert.equal(result.publishedCount, 1);
assert.deepEqual(JSON.parse(await fsp.readFile(path.join(libB, 'Sample Manga', 'manga.json'), 'utf8'))['x-work-identity'], remote.workIdentity,
'The stable work identity is stored with the imported peer edition');
assert.equal(fs.existsSync(path.join(libB, 'Sample Manga', 'v01', 'c0001 The first chapter [Peer ServerA ' + inB.id.slice(1, 7) + ']', '0001.png')), true);
const second = await run(info, { scope: { mode: 'missing' }, existingPolicy: 'skip-any' },
{ libraryRoot: libB, settings: { chapterMinMs: 500 }, shouldStop: () => false });
assert.equal(second.publishedCount, 0);
// Simulate B adding one chapter to its imported edition, then sharing it back.
// A must identify its original folder by the carried origin identity and fetch
// only the new chapter, rather than creating a title-conflict quarantine.
const chapterTwo = path.join(libB, 'Sample Manga', 'v01', 'c0002 Chapter from B');
await fsp.mkdir(chapterTwo, { recursive: true });
await fsp.writeFile(path.join(chapterTwo, 'chapter.json'), JSON.stringify({ title: 'Chapter from B', source: { name: 'Local' } }));
await fsp.writeFile(path.join(chapterTwo, '0001.png'), PNG);
b.setShare(inB.id, 'Sample Manga', true);
b.updatePeer(inB.id, { serve: true });
const reverseGrant = b.issueGrant(inB.id, ['download']);
a.acceptGrant(inA.id, reverseGrant.token);
a.updatePeer(inA.id, { receive: true });
const reverseCatalog = await client.catalog(a, inA.id);
assert.equal(reverseCatalog.length, 1);
const reverseManga = await client.manga(a, inA.id, reverseCatalog[0].id);
assert.deepEqual(reverseManga.workIdentity, remote.workIdentity, 'The original work identity survives a peer round trip');
process.env.PEER_DATA = dataA;
const reverseInfo = require('../adapter').infoFor(`peer://${inA.id}/${reverseCatalog[0].id}`);
const returned = await require('../adapter').run(reverseInfo, { scope: { mode: 'missing' }, existingPolicy: 'skip-any', workIdentity: reverseManga.workIdentity },
{ libraryRoot: libA, settings: { chapterMinMs: 500 }, shouldStop: () => false });
assert.equal(returned.finalPath, path.join(libA, 'Sample Manga'), 'The round-trip publication targets the original library folder');
assert.equal(returned.selectedCount, 1, 'Only the chapter added on B is selected for sync-missing');
assert.equal(returned.publishedCount, 1);
assert.equal(fs.existsSync(path.join(libA, 'Sample Manga', 'v01', 'c0002 Chapter from B [Peer ServerB ' + inA.id.slice(1, 7) + ']', '0001.png')), true);
const noChanges = await require('../adapter').run(reverseInfo, { scope: { mode: 'missing' }, existingPolicy: 'skip-any', workIdentity: reverseManga.workIdentity },
{ libraryRoot: libA, settings: { chapterMinMs: 500 }, shouldStop: () => false });
assert.equal(noChanges.selectedCount, 0, 'A repeated round trip queues no duplicate chapters');
// Simulate an older B copy which predates persisted identities. Exact shared
// page hashes may bridge that legacy gap, but title similarity alone may not.
const legacyMetadataPath = path.join(libB, 'Sample Manga', 'manga.json');
const legacyMetadata = JSON.parse(await fsp.readFile(legacyMetadataPath, 'utf8'));
delete legacyMetadata['x-work-identity'];
await fsp.writeFile(legacyMetadataPath, JSON.stringify(legacyMetadata));
const chapterThree = path.join(libB, 'Sample Manga', 'v01', 'c0003 Chapter from B');
await fsp.mkdir(chapterThree, { recursive: true });
await fsp.writeFile(path.join(chapterThree, 'chapter.json'), JSON.stringify({ title: 'Chapter from B', source: { name: 'Local' } }));
await fsp.writeFile(path.join(chapterThree, '0001.png'), PNG);
await stop(gb); // Clear the gateway's short-lived manifest cache after this legacy-data migration.
gb = await gateway(dataB, libB, portB, path.join(root, 'usage-b.json'));
const legacyManga = await client.manga(a, inA.id, reverseCatalog[0].id);
assert.notDeepEqual(legacyManga.workIdentity, remote.workIdentity, 'Legacy copy has no propagated identity and gets a peer-local fallback');
const legacySync = await require('../adapter').run(reverseInfo, { scope: { mode: 'missing' }, existingPolicy: 'skip-any', workIdentity: legacyManga.workIdentity },
{ libraryRoot: libA, settings: { chapterMinMs: 500 }, shouldStop: () => false });
assert.equal(legacySync.finalPath, path.join(libA, 'Sample Manga'), 'Content evidence safely reuses the original folder for legacy peer copies');
assert.equal(legacySync.selectedCount, 1, 'Legacy matching still selects only the chapter absent on A');
assert.equal(legacySync.publishedCount, 1);
assert.equal(fs.existsSync(path.join(libA, 'Sample Manga', 'v01', 'c0003 Chapter from B [Peer ServerB ' + inA.id.slice(1, 7) + ']', '0001.png')), true);
for (const book of await fsp.readdir(path.join(libB, 'Sample Manga'))) {
const bookPath = path.join(libB, 'Sample Manga', book);
if (!(await fsp.stat(bookPath)).isDirectory()) continue;
for (const chapter of await fsp.readdir(bookPath)) {
const page = path.join(bookPath, chapter, '0001.png');
if (fs.existsSync(page)) await fsp.writeFile(page, Buffer.from(`different-page-${chapter}`));
}
}
await stop(gb);
gb = await gateway(dataB, libB, portB, path.join(root, 'usage-b.json'));
const unverifiedLegacy = await client.manga(a, inA.id, reverseCatalog[0].id);
await assert.rejects(() => require('../adapter').run(reverseInfo,
{ scope: { mode: 'missing' }, existingPolicy: 'skip-any', workIdentity: unverifiedLegacy.workIdentity },
{ libraryRoot: libA, settings: { chapterMinMs: 500 }, shouldStop: () => false }),
/neither its canonical identity nor shared chapter page hashes verify.*No files were downloaded or changed/);
assert.equal((await require('../../scraper/lib/library-inventory').scanLibrary(libA)).mangas[0].chapters.length, 3,
'A title collision with differing chapter bytes leaves the local library untouched');
a.updatePeer(inA.id, { servingPaused: true });
assert.equal((await client.access(b, inB.id)).grant.status, 'paused');
await assert.rejects(() => client.catalog(b, inB.id), /Peer HTTP 403/);
assert.equal(a.publicState().peers.find(p => p.id === inA.id).grantStatus, 'paused', 'Pausing must retain the grant');
a.revokeGrant(inA.id);
assert.equal(a.publicState().peers.find(p => p.id === inA.id).grantStatus, 'none');
await assert.rejects(() => client.chapter(b, inB.id, listing[0].id, remote.chapters[0].id), /Peer HTTP 403/);
});
+312 -36
View File
@@ -20,6 +20,7 @@ h1 { font-size: 26px; line-height: 1.2; margin: 0 0 6px; }
}
.btn:hover { background: var(--card2); }
.btn.primary { background: var(--accent); border-color: var(--accent); color: #fff; }
.btn.danger { border-color: #8c3f48; color: #ff9aa7; }
.btn.on { border-color: var(--accent); color: #ffb3bd; }
.btn.small { min-height: 28px; padding: 2px 9px; font-size: 13px; }
.btn.icon { padding: 6px 9px; }
@@ -33,6 +34,7 @@ label { display: block; }
padding: 8px 16px; padding-top: max(8px, env(safe-area-inset-top)); background: rgba(18, 18, 20, .94); border-bottom: 1px solid var(--line);
}
.brand { font-weight: 800; text-decoration: none; font-size: 17px; letter-spacing: .2px; }
.nav-greatest { border-color: #5b3c46; color: #f0b9a0; }
.bell-wrap { position: relative; }
.bell .count { background: var(--accent); color: #fff; border-radius: 9px; font-size: 12px; font-weight: 700; padding: 0 6px; }
.panel {
@@ -48,8 +50,39 @@ label { display: block; }
main { max-width: 1200px; margin: 0 auto; padding: 16px; }
/* library */
.toolbar { display: flex; gap: 8px; margin-bottom: 16px; }
.toolbar input { flex: 1; min-width: 0; }
.reading-summary {
display: grid; grid-template-columns: minmax(150px, .7fr) minmax(220px, 1fr) auto; align-items: center; gap: 12px 22px;
padding: 14px 16px; margin-bottom: 16px; border: 1px solid color-mix(in srgb, var(--accent) 45%, var(--line));
border-radius: 12px; background: linear-gradient(120deg, color-mix(in srgb, var(--accent) 12%, var(--card)), var(--card));
}
.collection-mosaic { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); }
.collection-mosaic img { width: 100%; height: 100%; object-fit: cover; min-height: 0; }
.collection-count { display: inline-block; background: var(--card2); padding: 2px 6px; border-radius: 5px; font-size: 12px; }
.collection-heading { grid-column: 1 / -1; margin: 12px 0 0; }
.collection-member { display: flex; gap: 16px; align-items: center; padding: 10px; border-bottom: 1px solid var(--line); text-decoration: none; }
.collection-member img { width: 55px; height: 80px; object-fit: cover; }
.collection-editor > div { padding: 10px 0; border-bottom: 1px solid var(--line); }
.collection-editor input[type=number] { width: 75px; }
.catalog-json { width: 100%; min-height: 350px; font: 13px/1.4 monospace; }
.card > textarea { display: block; width: 100%; margin: 10px 0; }
.reading-summary-title { display: flex; gap: 8px; align-items: center; }
.reading-summary-stats { display: flex; flex-wrap: wrap; align-items: baseline; gap: 4px 7px; }
.reading-summary-stats strong { font-size: 18px; }
.reading-summary-links { display: flex; flex-wrap: wrap; justify-content: flex-end; gap: 5px 14px; font-size: 13px; }
.story-pill { padding: 1px 7px; border-radius: 10px; background: var(--card2); color: var(--muted); font-size: 12px; }
.library-controls { margin-bottom: 18px; }
.toolbar { display: flex; gap: 8px; margin-bottom: 10px; }
.toolbar input { flex: 1; min-width: 180px; }
.filter-group { border-top: 1px solid var(--line); padding-top: 10px; }
.filter-heading { display: flex; align-items: baseline; gap: 10px; margin-bottom: 7px; }
.tag-filter, .volume-filter { display: flex; gap: 7px; overflow-x: auto; padding: 1px 1px 7px; scrollbar-width: thin; }
.filter-chip {
flex: none; border: 1px solid var(--line); border-radius: 14px; padding: 3px 10px; background: var(--card); color: var(--fg);
font: inherit; font-size: 13px; cursor: pointer; white-space: nowrap;
}
.filter-chip:hover { background: var(--card2); }
.filter-chip.on { border-color: var(--accent); background: color-mix(in srgb, var(--accent) 22%, var(--card)); color: #ffd8de; }
.result-count { margin-top: 6px; }
.grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(160px, 1fr)); gap: 22px 16px; }
.tile { position: relative; display: block; text-decoration: none; }
.tile .cv { aspect-ratio: 2 / 3; border-radius: 8px; overflow: hidden; background: var(--card); box-shadow: 0 3px 14px rgba(0, 0, 0, .45); }
@@ -63,8 +96,19 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
.empty { text-align: center; padding: 60px 10px; }
@media (max-width: 560px) {
main { padding: 12px; }
.toolbar { flex-wrap: wrap; }
.toolbar input { flex-basis: 100%; }
.toolbar select { flex: 1; }
.grid { grid-template-columns: repeat(2, 1fr); gap: 16px 12px; }
.nav { padding-left: 12px; padding-right: 12px; }
.nav { gap: 5px; flex-wrap: wrap; }
.nav .grow { display: none; }
.nav .brand { font-size: 14px; }
.nav .btn { padding-left: 7px; padding-right: 7px; font-size: 12px; }
.nav-greatest { font-size: 0 !important; }
.nav-greatest::after { content: 'Saikō'; font-size: 12px; }
.reading-summary { grid-template-columns: 1fr; gap: 5px; }
.reading-summary-links { justify-content: flex-start; }
}
/* manga page */
@@ -73,26 +117,81 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
.hero .info { min-width: 0; }
.chips { display: flex; flex-wrap: wrap; gap: 6px; margin: 8px 0; }
.chip { background: var(--card2); border-radius: 12px; padding: 1px 9px; font-size: 13px; }
.chip-link { text-decoration: none; border: 1px solid transparent; }
.chip-link:hover { border-color: var(--accent); color: #ffd8de; }
.desc { white-space: pre-line; margin: 10px 0; }
.coverage-indicators { display: flex; flex-wrap: wrap; align-items: center; gap: 6px 9px; margin: 6px 0; }
.coverage-badge { display: inline-block; border: 1px solid; border-radius: 999px; padding: 3px 9px; font-size: 12px; font-weight: 700; line-height: 1.3; }
.coverage-complete { color: #a7f3c7; border-color: #39875d; background: #153323; }
.coverage-partial { color: #ffd09b; border-color: #9a6630; background: #332417; }
.coverage-unknown { color: var(--fg); border-color: var(--line); background: var(--card2); }
.actions { display: flex; flex-wrap: wrap; gap: 8px; margin-top: 12px; }
.sources a { margin-right: 10px; }
.metadata.card { max-width: none; }
.reading-story.card { max-width: none; padding: 0; overflow: hidden; }
.reading-story-summary {
display: flex; justify-content: space-between; align-items: center; gap: 14px; padding: 14px 16px; cursor: pointer;
background: linear-gradient(120deg, color-mix(in srgb, var(--accent) 12%, var(--card)), var(--card));
}
.reading-story[open] > .reading-story-summary { border-bottom: 1px solid var(--line); }
.story-stats { display: grid; grid-template-columns: repeat(4, minmax(110px, 1fr)); gap: 10px; padding: 14px 16px 8px; }
.story-stat { min-width: 0; }
.story-stat strong, .story-stat span { display: block; }
.story-stat strong { font-size: 19px; font-variant-numeric: tabular-nums; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.story-last { margin: 4px 16px 16px; }
.story-rhythm-head { display: flex; align-items: baseline; gap: 9px; margin: 0 16px 7px; }
.story-rhythm {
display: flex; align-items: flex-end; gap: 3px; min-height: 68px; margin: 0 16px; padding: 6px 3px 3px;
overflow-x: auto; border-bottom: 1px solid var(--line); scrollbar-width: thin;
}
.rhythm-bar { flex: 0 0 7px; min-height: 3px; border-radius: 3px 3px 1px 1px; background: var(--accent); opacity: .86; }
.rhythm-bar:hover, .rhythm-bar:focus { opacity: 1; outline: 2px solid #fff; outline-offset: 1px; }
.rhythm-bar.empty { height: 3px; background: var(--line); opacity: 1; }
.story-actions { display: flex; justify-content: space-between; align-items: center; gap: 12px; padding: 12px 16px 14px; }
.metadata-summary { display: flex; align-items: center; justify-content: space-between; gap: 12px; cursor: pointer; list-style-position: outside; }
.metadata.card .metadata-summary h2 { display: inline; margin: 0; }
.metadata-summary::after { content: 'Show'; color: var(--muted); font-size: 13px; font-weight: 400; }
.metadata[open] > .metadata-summary { margin-bottom: 12px; }
.metadata[open] > .metadata-summary::after { content: 'Hide'; }
.meta-grid { display: grid; grid-template-columns: minmax(130px, max-content) minmax(0, 1fr); gap: 8px 18px; margin: 0; }
.meta-grid dt { color: var(--muted); }
.meta-grid dd { min-width: 0; margin: 0; overflow-wrap: anywhere; }
.meta-tags, .meta-links { display: flex; flex-wrap: wrap; align-items: center; gap: 6px 10px; }
.raw-meta { margin-top: 16px; }
.raw-meta summary { cursor: pointer; color: var(--muted); }
.raw-meta pre { max-height: 440px; overflow: auto; padding: 12px; border-radius: 8px; background: var(--bg); color: var(--muted); font-size: 12px; white-space: pre-wrap; overflow-wrap: anywhere; }
@media (max-width: 640px) {
.hero { flex-direction: column; align-items: center; text-align: center; }
.hero > img { width: 180px; }
.chips, .actions { justify-content: center; }
.meta-grid { grid-template-columns: 1fr; gap: 2px; }
.meta-grid dd { margin-bottom: 9px; }
.story-stats { grid-template-columns: repeat(2, minmax(0, 1fr)); }
.story-actions { align-items: flex-start; flex-direction: column; }
}
.chapters { border-top: 1px solid var(--line); }
.chapter-section { margin-top: 24px; }
.chapter-heading { display: flex; align-items: baseline; gap: 10px; }
.chapter-heading h2 { margin: 0; }
.volume-filter { margin: 9px 0 6px; max-width: 100%; }
.chapter-list-wrap { max-width: 100%; overflow-x: auto; }
.chapters { display: inline-block; width: max-content; min-width: min(440px, 100%); max-width: 100%; border-top: 1px solid var(--line); }
.book-sep { color: var(--muted); font-size: 12px; text-transform: uppercase; letter-spacing: .08em; padding: 16px 4px 6px; }
.ch { display: flex; align-items: center; gap: 8px; border-bottom: 1px solid var(--line); }
.ch-main { flex: 1; min-width: 0; display: flex; gap: 10px; align-items: baseline; padding: 12px 4px; text-decoration: none; }
.chapter-entry { width: 100%; }
.ch { display: flex; width: 100%; max-width: 100%; align-items: center; gap: 6px; border-bottom: 1px solid var(--line); white-space: nowrap; }
.ch-main { flex: 1 1 auto; min-width: 0; display: grid; grid-template-columns: 76px 52px minmax(70px, 1fr) max-content; gap: 8px; align-items: baseline; padding: 6px 4px; text-decoration: none; }
.ch-main.has-time { grid-template-columns: 76px 52px 72px minmax(70px, 1fr) max-content; }
.ch-main:hover .ch-title { text-decoration: underline; }
.ch-label { font-weight: 650; flex: none; }
.ch-label { min-width: 0; overflow: hidden; text-overflow: ellipsis; font-weight: 650; font-variant-numeric: tabular-nums; }
.ch-pages { min-width: 0; overflow: hidden; color: var(--muted); font-size: 13px; font-variant-numeric: tabular-nums; text-align: left; white-space: nowrap; }
.ch-time { min-width: 0; overflow: hidden; color: #c9a7ad; font-size: 13px; font-variant-numeric: tabular-nums; text-overflow: ellipsis; white-space: nowrap; }
.ch-title { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
.ch.read .ch-main { color: var(--muted); }
.tag { font-size: 11px; border: 1px solid var(--line); border-radius: 6px; padding: 0 5px; color: var(--muted); flex: none; }
.state { color: var(--muted); font-size: 13px; background: none; border: 0; cursor: pointer; padding: 6px; font: inherit; font-size: 13px; }
.state.done { color: #7fd3a0; }
.versions { background: var(--card); border-radius: 8px; margin: 4px 0 10px; padding: 4px 10px; }
.ch > .btn.small { min-height: 24px; padding: 1px 7px; }
.ch > .state { padding: 3px 4px; }
.versions { width: 100%; background: var(--card); border-radius: 8px; margin: 4px 0 10px; padding: 4px 10px; }
.ver { display: flex; align-items: center; gap: 10px; padding: 7px 0; flex-wrap: wrap; }
.ver + .ver { border-top: 1px solid var(--line); }
.ver .src { flex: 1; min-width: 140px; }
@@ -100,8 +199,158 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
.vote button { background: var(--card2); border: 1px solid var(--line); color: var(--fg); border-radius: 6px; width: 30px; height: 28px; cursor: pointer; }
.vote button.on { background: var(--accent); border-color: var(--accent); color: #fff; }
.vote b { min-width: 22px; text-align: center; }
.chapter-empty { margin: 16px 0; }
@media (max-width: 640px) {
.chapters { display: block; width: 100%; min-width: 0; }
.ch { flex-wrap: nowrap; gap: 4px; overflow: hidden; }
.ch-main { grid-template-columns: 64px 44px minmax(0, 1fr) max-content; gap: 6px; }
.ch-main.has-time { grid-template-columns: 64px 44px 62px minmax(0, 1fr) max-content; }
.ch > .btn.small { min-width: 0; padding-left: 5px; padding-right: 5px; }
.ch > .state { flex: none; }
}
/* settings & login */
.settings-layout { display: grid; grid-template-columns: 225px minmax(0, 1fr); gap: 28px; align-items: start; }
.settings-menu { position: sticky; top: 76px; border: 1px solid var(--line); border-radius: 12px; padding: 12px; background: var(--card); }
.settings-menu-group + .settings-menu-group { margin-top: 18px; border-top: 1px solid var(--line); padding-top: 14px; }
.settings-menu-group h2 { margin: 0 8px 7px; font-size: 12px; text-transform: uppercase; letter-spacing: .08em; color: var(--muted); }
.settings-menu a { display: block; padding: 8px; border-radius: 7px; text-decoration: none; }
.settings-menu a:hover { background: var(--card2); }
.settings-menu a[aria-current=page] { background: var(--card2); color: #ffb3bd; box-shadow: inset 3px 0 var(--accent); }
.settings-workspace, .settings-content { min-width: 0; }
.settings-breadcrumb { margin: 0 0 5px; }
.settings-title-row { display: flex; align-items: center; gap: 12px; }
.settings-hint { margin: 5px 0 20px; }
.settings-content .card { max-width: none; }
.reading-hours-list { border: 1px solid var(--line); border-radius: 9px; overflow: hidden; }
.reading-hours-day { display: grid; grid-template-columns: minmax(95px, 8em) minmax(0, 1fr); gap: 12px; padding: 10px 12px; }
.reading-hours-day + .reading-hours-day { border-top: 1px solid var(--line); }
.reading-hours-day span { overflow-wrap: anywhere; }
.peer-card { min-width: 0; overflow-wrap: anywhere; }
.peer-card textarea, .peer-card input { min-width: 0; max-width: 100%; }
.peer-card .row > .field { min-width: 0; }
.peer-name { color: #9ac8ff; font-weight: 650; white-space: nowrap; }
.peer-name.local { color: #ffc78a; }
.peer-page-tabs, .peer-section-head, .peer-catalog-heading, .peer-catalog-toolbar, .peer-catalog-actions, .peer-card-actions { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; }
.peer-page-tabs { margin: 0 0 16px; }
.peer-section-head, .peer-catalog-heading { justify-content: space-between; }
.peer-catalog-heading { margin-bottom: 16px; }
.peer-catalog-heading h1, .peer-section-head h2 { margin: 0; }
.peer-catalog-toolbar { padding: 12px; border: 1px solid var(--line); border-radius: 10px; background: var(--card); margin-bottom: 14px; }
.peer-catalog-peer { min-width: min(220px, 100%); }
.peer-catalog-actions { padding: 12px; margin: 12px 0; border: 1px solid var(--line); border-radius: 10px; background: var(--card); }
.peer-search { flex: 1 1 220px; max-width: 420px; }
.peer-catalog-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(160px, 1fr)); gap: 14px; align-items: start; }
.peer-catalog-card { min-width: 0; overflow: hidden; border: 1px solid var(--line); border-radius: 10px; background: var(--card); }
.peer-cover-wrap { position: relative; overflow: hidden; aspect-ratio: 2 / 3; background: #17181b; }
.peer-cover { display: block; width: 100%; height: 100%; object-fit: cover; }
.peer-cover-placeholder { display: grid; place-items: center; width: 100%; height: 100%; color: var(--muted); }
.peer-approval-badge { position: absolute; top: 8px; left: 8px; display: grid; place-items: center; width: 28px; height: 28px; border-radius: 50%; background: #1f9d61; color: white; font-weight: 800; }
.peer-select-overlay { position: absolute; top: 8px; right: 8px; width: 32px; height: 32px; display: grid; place-items: center; border-radius: 7px; background: #101114d9; }
.peer-select-overlay input { width: 19px; height: 19px; margin: 0; accent-color: #4aa675; }
.peer-catalog-card-info { display: grid; gap: 7px; padding: 10px; }
.peer-catalog-card-info .small { min-height: 2.5em; }
.peer-title { padding: 0; border: 0; background: transparent; color: var(--fg); font: inherit; font-weight: 650; text-align: left; cursor: pointer; }
.peer-selected { border-color: #278256; color: #8fe0b2; }
.peer-catalog-message { margin: 10px 0; }
.peer-table { overflow-x: auto; }
.peer-table-row { display: grid; grid-template-columns: minmax(140px, 1.3fr) minmax(150px, 1.1fr) minmax(130px, 1fr) minmax(130px, 1fr) auto; align-items: center; gap: 12px; min-width: 720px; padding: 12px; border-bottom: 1px solid var(--line); }
.peer-table-head { color: var(--muted); font-size: 12px; text-transform: uppercase; }
.peer-row-button { width: 100%; color: var(--fg); background: transparent; border: 0; border-bottom: 1px solid var(--line); text-align: left; cursor: pointer; font: inherit; }
.peer-row-button:hover { background: #ffffff08; }
.peer-status-ok { color: #a7f3c7; font-weight: 700; }
.peer-status-pending { color: #ffe08a; font-weight: 650; }
.peer-state { display: grid; gap: 3px; margin: 10px 0 14px; padding: 11px 13px; border: 1px solid var(--line); border-left-width: 4px; border-radius: 8px; background: var(--card2); }
.peer-state strong { font-size: 14px; font-weight: 750; }
.peer-state span { font-size: 13px; line-height: 1.4; }
.peer-state-ok { color: #a7f3c7; border-color: #39875d; background: #153323; }
.peer-state-pending { color: #ffe08a; border-color: #987631; background: #332b16; }
.peer-state-warn { color: #ffd09b; border-color: #9a6630; background: #332417; }
.peer-state-error { color: #ffb4b4; border-color: #9a3e48; background: #351b20; }
.peer-state-neutral { color: var(--fg); border-color: var(--line); background: var(--card2); }
.peer-row-action { white-space: nowrap; color: var(--accent); }
.peer-wizard-grid, .peer-limits-grid { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 12px; }
.peer-step { min-width: 0; padding: 14px; border: 1px solid var(--line); border-radius: 10px; background: var(--card); }
.peer-step h3, .peer-step h4 { margin-top: 0; }
.peer-fingerprint { overflow-wrap: anywhere; }
.peer-danger-zone { padding: 16px 0; border-top: 1px solid var(--line); }
.peer-manga-dialog { width: min(860px, calc(100vw - 24px)); max-height: min(90vh, 900px); overflow: auto; color: var(--fg); background: var(--bg); border: 1px solid var(--line); border-radius: 12px; padding: 22px; }
.peer-manga-dialog::backdrop { background: #000b; }
.peer-dialog-close { float: right; font-size: 20px; }
.peer-dialog-layout { display: grid; grid-template-columns: minmax(120px, 220px) minmax(0, 1fr); gap: 20px; }
.peer-dialog-cover { width: 100%; aspect-ratio: 2 / 3; object-fit: cover; border-radius: 8px; }
.peer-chapter-list { max-height: 40vh; overflow: auto; padding-left: 24px; }
.peer-chapter-list li { padding: 4px 0; }
.peer-detail-queue { display: grid; gap: 10px; margin-top: 14px; }
.peer-detail-actions { display: flex; gap: 8px; flex-wrap: wrap; }
.peer-chapter-picker { margin-top: 12px; border-top: 1px solid var(--line); padding-top: 10px; }
.peer-chapter-picker summary { cursor: pointer; color: var(--muted); }
.peer-chapter-picker .peer-chapter-list { display: grid; gap: 4px; margin-top: 10px; padding: 8px; border: 1px solid var(--line); border-radius: 8px; }
.peer-chapter-option { display: flex; align-items: flex-start; gap: 8px; padding: 5px; }
.peer-chapter-option input { min-height: 0; margin-top: 4px; accent-color: #4aa675; }
.peer-tags { display: flex; gap: 6px; flex-wrap: wrap; }
.peer-tags .tag { padding: 3px 8px; border-radius: 999px; background: var(--card); color: var(--muted); }
@media (max-width: 620px) {
.peer-wizard-grid, .peer-limits-grid, .peer-dialog-layout { grid-template-columns: 1fr; }
.peer-catalog-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 9px; }
.peer-catalog-toolbar > .seg { width: 100%; }
.peer-catalog-toolbar > .seg .btn { flex: 1; }
.peer-dialog-cover { max-width: 180px; }
}
.discovery-settings, .discovery-admin { max-width: none; }
.discovery-hint { display: block; font-weight: 400; margin-top: 3px; }
.discovery-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(min(100%, 330px), 1fr)); gap: 14px; }
.discovery-card { display: grid; grid-template-columns: 90px minmax(0, 1fr); gap: 12px; margin: 0; max-width: none; min-width: 0; }
.discovery-card > img, .discovery-no-cover { width: 90px; height: 130px; object-fit: cover; border-radius: 5px; background: var(--card2); }
.discovery-no-cover { display: grid; place-items: center; text-align: center; color: var(--muted); font-size: 12px; }
.discovery-card-body { min-width: 0; overflow-wrap: anywhere; }
.discovery-card-body h2 { font-size: 17px; margin-bottom: 4px; }
.discovery-card-body p { margin: 6px 0 10px; }
.discovery-card-body details { margin: 10px 0; }
.discovery-card-body ul { padding-left: 18px; }
.discovery-actions { margin: 10px 0; }
.catalog-workspace { min-width: 0; margin: 0; padding: 0; border: 0; }
.catalog-workspace > .row { margin-bottom: 16px; }
.catalog-workspace input, .catalog-workspace select, .catalog-workspace textarea { min-width: 0; max-width: 100%; }
.catalog-workspace .card > input, .catalog-workspace .card > textarea { display: block; width: 100%; margin-bottom: 12px; }
.catalog-workspace .collection-editor .row { padding: 8px 0; border-bottom: 1px solid var(--line); }
.catalog-workspace .collection-editor input[type=number] { width: 80px; }
.catalog-workspace .catalog-json { width: 100%; min-height: 280px; font-family: monospace; }
@media (max-width: 760px) {
.catalog-workspace [data-member] > span { flex-basis: 100%; overflow-wrap: anywhere; }
}
.settings-tabs { display: flex; flex-wrap: wrap; gap: 8px; margin: 0 0 16px; }
.settings-tabs a[aria-current=page] { border-color: var(--accent); color: #ffb3bd; }
.scraper-filters { margin: 0 0 18px; }
.scraper-filters button[aria-pressed=true] { border-color: var(--accent); color: #ffb3bd; background: color-mix(in srgb, var(--accent) 12%, var(--card)); }
.scraper-overview { display: flex; flex-wrap: wrap; gap: 8px; margin: 0 0 18px; }
.scraper-overview span { border: 1px solid var(--line); border-radius: 999px; padding: 5px 10px; font-size: 13px; }
.scraper-add, .scraper-completed, .scraper-history { border: 1px solid var(--line); border-radius: 9px; padding: 11px 13px; margin: 14px 0; }
.scraper-add > summary, .scraper-completed > summary, .scraper-history > summary { cursor: pointer; font-weight: 600; }
.scraper-add[open] > summary, .scraper-completed[open] > summary, .scraper-history[open] > summary { margin-bottom: 12px; }
.scraper-update-list { margin: 12px 0; border-top: 1px solid var(--line); }
.scraper-update-row { display: grid; grid-template-columns: minmax(180px, 1.5fr) minmax(100px, .7fr) repeat(2, minmax(130px, 1fr)); gap: 5px 12px; align-items: center; padding: 11px 0; border-bottom: 1px solid var(--line); font-size: 13px; }
.scraper-update-row label { min-width: 0; overflow-wrap: anywhere; }
.scraper-update-row .saved { grid-column: 3 / -1; }
.scraper-source-card { border: 1px solid var(--line); border-radius: 10px; padding: 12px; margin: 0 0 14px; }
.scraper-source-card h3 { margin: 0 0 5px; }
.scraper-source-card .source-rate { border-top: 0; }
@media (max-width: 760px) {
.scraper-update-row { grid-template-columns: minmax(0, 1fr) auto; }
.scraper-update-row label { grid-column: 1 / -1; }
.scraper-update-row .saved { grid-column: 1 / -1; }
}
.settings-content .scraper-item .row { flex-shrink: 0; }
.settings-menu a:focus-visible, .settings-tabs a:focus-visible { outline: 2px solid var(--accent); outline-offset: 2px; }
@media (max-width: 760px) {
.settings-layout { grid-template-columns: minmax(0, 1fr); gap: 18px; }
.settings-menu { position: static; padding: 10px; }
.settings-menu-group { display: flex; flex-wrap: wrap; gap: 3px; align-items: center; }
.settings-menu-group h2 { flex: 0 0 100%; margin: 0 5px 4px; }
.settings-menu-group + .settings-menu-group { margin-top: 9px; padding-top: 9px; }
.settings-menu a { font-size: 13px; padding: 7px 9px; }
.settings-content .scraper-item { flex-wrap: wrap; }
}
.card { background: var(--card); border: 1px solid var(--line); border-radius: 12px; padding: 16px; margin-bottom: 16px; max-width: 720px; }
.card h2 { margin: 0 0 12px; font-size: 18px; }
.field { margin-bottom: 14px; }
@@ -113,6 +362,24 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
.login { max-width: 360px; margin: 12vh auto 0; }
.login input { width: 100%; }
.saved { color: #7fd3a0; font-size: 13px; }
.reading-data { margin-top: 16px; padding-top: 12px; border-top: 1px solid var(--line); }
.reading-data > summary { cursor: pointer; font-weight: 600; }
.reading-danger { margin-top: 14px; padding-top: 12px; border-top: 1px solid var(--line); }
.reading-danger p { margin: 3px 0 8px; }
.scraper-card { max-width: 980px; }
.scraper-progress { height: 7px; max-width: 620px; margin: 8px 0 4px; overflow: hidden; border-radius: 5px; background: var(--line); }
.scraper-progress i { display: block; height: 100%; background: var(--accent); transition: width .25s; }
.scraper-queue { border-top: 1px solid var(--line); }
.scraper-item { display: flex; align-items: center; gap: 7px; padding: 8px 0; border-bottom: 1px solid var(--line); }
.scraper-item.active { border-left: 3px solid var(--accent); padding-left: 8px; }
@media (max-width: 600px) {
.scraper-item { flex-wrap: wrap; }
.scraper-item > .grow { flex-basis: 100%; width: 100%; }
}
.source-rate { padding: 9px 0; border-top: 1px solid var(--line); }
.source-rate > b { display: block; margin-bottom: 6px; }
.rate-row label { display: grid; gap: 3px; color: var(--muted); }
.rate-row input { width: 115px; }
/* ---------------- reader ---------------- */
body.reader { position: fixed; top: 0; left: 0; right: 0; bottom: 0; overflow: hidden; background: #0b0b0c; overscroll-behavior: none; }
@@ -122,11 +389,28 @@ body.reader { position: fixed; top: 0; left: 0; right: 0; bottom: 0; overflow: h
-webkit-user-select: none; user-select: none; -webkit-tap-highlight-color: transparent;
overscroll-behavior: contain; scrollbar-width: thin; scrollbar-color: #3a3a40 transparent;
}
#stage.lock-x { overflow-x: hidden; touch-action: pan-y; overscroll-behavior-x: none; }
#stage.pan { cursor: grab; }
#stage.drag { cursor: grabbing; }
#canvas { position: relative; }
.reading-gate { box-sizing: border-box; max-width: 620px; margin: min(13vh, 100px) auto 100px; padding: 24px;
border: 1px solid var(--line); border-radius: 14px; background: var(--card); color: var(--fg); line-height: 1.5; }
.reading-gate h1 { margin-top: 0; }
.reading-gate-question { margin: 20px 0; }
.reading-gate-question input { display: block; box-sizing: border-box; width: 100%; margin: 14px 0; padding: 10px; }
.reading-gate-actions { display: flex; flex-wrap: wrap; gap: 10px; }
.reading-gate-message { margin: 18px 0 0; padding: 11px 13px; border: 1px solid #46505b;
border-radius: 9px; background: #222a33; color: #d8e6f4; }
.reading-gate-message[data-tone="success"] { border-color: #39765a; background: #193528; color: #b8f2d0; }
.reading-gate-message[data-tone="retry"] { border-color: #9b7233; background: #382b1a; color: #ffdaa0; }
.reading-gate-message[data-tone="error"] { border-color: #944c55; background: #3c2025; color: #ffc0c6; }
@media (max-width: 660px) { .reading-gate { margin: 28px 12px 100px; padding: 18px; } }
.pg { position: absolute; overflow: hidden; background: #19191c; }
.pg img { display: block; width: 100%; height: 100%; -webkit-user-drag: none; }
#canvas[data-page-tone="inverted"] .pg img { filter: invert(1); }
#canvas[data-page-tone="sepia"] .pg img { filter: sepia(.8) brightness(.9); }
#canvas[data-page-tone="dim"] .pg img { filter: brightness(.7); }
#canvas[data-page-tone="extra-dim"] .pg img { filter: brightness(.45); }
.pg.rot img { position: absolute; left: 50%; top: 50%; max-width: none; }
/* still loading: a spinner that only appears after 150 ms, so fast loads never flash it */
.pg.loading::after {
@@ -166,39 +450,31 @@ body.nobars #bot { transform: translateY(100%); opacity: 0; pointer-events: none
.seg button + button { border-left: 1px solid var(--line); }
.seg button.on { background: var(--accent); color: #fff; }
.seg button.meh:not(.on) { color: var(--muted); }
.zoom-controls button { min-width: 34px; padding-left: 9px; padding-right: 9px; }
.zoom-controls #zoom { min-width: 58px; font-variant-numeric: tabular-nums; }
.page-tone { white-space: nowrap; }
.page-tone:not([data-page-tone="original"]) { border-color: var(--accent); }
.clock-chip { font-variant-numeric: tabular-nums; white-space: nowrap; }
.clock-chip.clock-running { border-color: #3d8b62; background: #183426; }
.clock-chip.clock-warning { border-color: #c58a27; background: #3b2c14; }
.clock-panel {
position: fixed; right: max(8px, env(safe-area-inset-right)); bottom: calc(112px + env(safe-area-inset-bottom)); z-index: 25;
width: min(360px, calc(100vw - 16px)); padding: 14px; border: 1px solid var(--line); border-radius: 10px;
background: rgba(20, 20, 23, .98); color: var(--fg); box-shadow: 0 12px 40px rgba(0, 0, 0, .55);
}
.clock-head { display: flex; justify-content: space-between; align-items: center; gap: 12px; }
.clock-state { color: var(--muted); font-size: 13px; text-align: right; }
.clock-details { display: grid; grid-template-columns: auto 1fr; gap: 5px 16px; margin: 12px 0; }
.clock-details dt { color: var(--muted); }
.clock-details dd { margin: 0; text-align: right; font-variant-numeric: tabular-nums; }
#ver { max-width: 170px; }
#toast {
position: fixed; left: 50%; top: 50%; transform: translate(-50%, -50%); z-index: 30; pointer-events: none;
background: rgba(0, 0, 0, .75); color: #fff; padding: 9px 16px; border-radius: 10px; opacity: 0; transition: opacity .25s; text-align: center;
}
#toast.show { opacity: 1; }
@media (max-width: 560px) { .seg button { padding: 7px 10px; } #bot .btn { padding: 6px 10px; } .wide-only { display: none; } }
/* Who's here (Senpai): a side sheet in the .panel style */
.panel.sheet {
position: fixed; top: 0; right: 0; bottom: 0; width: min(380px, 100vw); max-height: none; border-radius: 0; border-width: 0 0 0 1px;
padding: max(6px, env(safe-area-inset-top)) 0 0; display: flex; flex-direction: column; z-index: 25;
@media (max-width: 700px) { .clock-label { display: none; } }
@media (max-width: 560px) {
.seg button { padding: 7px 10px; } #bot .btn { padding: 6px 10px; } .wide-only { display: none; }
.clock-panel { bottom: calc(124px + env(safe-area-inset-bottom)); }
}
.sheet-head { display: flex; align-items: flex-start; gap: 8px; padding: 6px 12px; }
.sheet .seg { display: flex; margin: 2px 12px 8px; }
.sheet .seg button { flex: 1; padding: 6px 4px; }
.sheet-body { flex: 1; min-height: 0; overflow: auto; padding: 0 8px 10px; }
.sheet-foot { border-top: 1px solid var(--line); padding: 8px 12px max(8px, env(safe-area-inset-bottom)); }
.sheet-foot .check { margin: 4px 0; }
.who { display: flex; align-items: center; gap: 10px; width: 100%; padding: 6px; border: 0; border-radius: 8px; background: none; color: inherit; font: inherit; text-align: left; cursor: pointer; }
.who:hover { background: var(--card2); }
.who .pic { width: 44px; height: 44px; flex: none; border-radius: 50%; object-fit: cover; object-position: top; background: var(--card2); }
.pic.none, .big.none { display: flex; align-items: center; justify-content: center; color: var(--muted); font-weight: 700; }
.who .nm { font-weight: 600; overflow: hidden; text-overflow: ellipsis; }
.who .nm.unnamed { color: var(--muted); font-style: italic; font-weight: 400; }
.sp-chip { flex: none; font-size: 11px; border: 1px solid var(--line); border-radius: 6px; padding: 0 5px; color: var(--muted); }
.sp-more { margin: 6px; }
.tier { font-size: 11px; text-transform: uppercase; letter-spacing: .08em; color: var(--muted); padding: 12px 6px 4px; }
.sp-card { display: flex; gap: 12px; align-items: flex-start; margin: 10px 4px; }
.sp-card h2 { margin: 0 0 6px; font-size: 18px; overflow-wrap: anywhere; }
.sp-card .big { width: 96px; height: 120px; flex: none; border-radius: 10px; object-fit: cover; object-position: top; background: var(--card2); }
.sp-names { margin: 0; padding-left: 22px; }
.sp-time a { display: block; padding: 6px; border-radius: 8px; text-decoration: none; }
.sp-time a:hover { background: var(--card2); }
.sp-box { position: absolute; border: 3px solid var(--accent); border-radius: 6px; box-shadow: 0 0 0 2px rgba(0, 0, 0, .5); pointer-events: none; }
#who.on { border-color: var(--accent); color: #ffb3bd; }
+2
View File
@@ -0,0 +1,2 @@
<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>Collection · MangaCircus</title><link rel="stylesheet" href="app.css"><script src="common.js" defer></script><script src="collection.js" defer></script></head><body><main><p>Loading…</p></main></body></html>
+20
View File
@@ -0,0 +1,20 @@
'use strict';
(async () => {
try {
const [me, c] = await Promise.all([api('api/me'), api('api/collection?' + qsa({ id: new URLSearchParams(location.search).get('id') }))]);
topNav(me); document.title = c.title + ' · MangaCircus';
const main = $('main'); main.textContent = '';
main.append(h('section', { class: 'hero' }, c.cover ? h('img', { src: c.cover, alt: '' }) : null,
h('div', { class: 'info' }, h('h1', {}, c.title), h('p', { class: 'muted' }, `${c.members.length} manga · ${c.read}/${c.chapters} chapters read · ${c.unread} new`),
c.description ? h('p', {}, c.description) : null,
c.reading.activeMs ? h('p', {}, `${readingDuration(c.reading.activeMs)} together`) : null,
h('div', { class: 'actions' }, h('a', { class: 'btn primary', href: readUrl(c.continue.m, c.continue.c, c.continue.v) }, c.continue.text),
me.admin ? h('a', { class: 'btn', href: 'collections.html?' + qsa({ id: c.id }) }, 'Edit collection') : null))));
main.append(h('h2', {}, 'Reading order'), h('div', { class: 'collection-members' }, c.members.map(m =>
h('a', { class: 'collection-member', href: mangaUrl(m.id) }, m.cover ? h('img', { src: m.cover, alt: '', loading: 'lazy' }) : null,
h('div', {}, h('strong', {}, `${m.readingOrder}. ${m.title}`), h('div', { class: 'muted' }, `${m.role.replaceAll('-', ' ')} · ${m.status} · ${m.read}/${m.chapters} read`))))));
if (c.relations.length) main.append(h('details', { class: 'card' }, h('summary', {}, 'Relationships & evidence'), c.relations.map(r =>
h('p', {}, `${c.members.find(m => m.id === r.from)?.title || r.from} → ${r.type} → ${c.members.find(m => m.id === r.to)?.title || r.to}`,
h('span', { class: 'muted' }, ' · ' + r.explanation)))));
} catch (e) { showError(e); }
})();
+2
View File
@@ -0,0 +1,2 @@
<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>Catalog Curator · MangaCircus</title><link rel="stylesheet" href="app.css"><script src="common.js" defer></script><script src="settings-nav.js" defer></script><script src="collections.js" defer></script></head><body><main><p>Loading…</p></main></body></html>
+126
View File
@@ -0,0 +1,126 @@
'use strict';
// randomUUID is secure-context-only; tailnet HTTP and LAN IPs need this fallback.
function collectionId() {
if (typeof crypto.randomUUID === 'function') return crypto.randomUUID();
const bytes = crypto.getRandomValues(new Uint8Array(16));
bytes[6] = (bytes[6] & 15) | 64; bytes[8] = (bytes[8] & 63) | 128;
const hex = Array.from(bytes, byte => byte.toString(16).padStart(2, '0')).join('');
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`;
}
(async () => {
let me, data;
try { [me, data] = await Promise.all([api('api/me'), api('api/admin/catalog')]); } catch (e) { return showError(e); }
topNav(me);
let selected = new URLSearchParams(location.search).get('id');
const main = settingsShell(me, 'admin/collections').content;
const message = h('p', { role: 'status', 'aria-live': 'polite' });
let dirty = false, busy = false, workspace, jsonDraft = null;
const notice = (text, error = false) => {
message.textContent = text; message.className = error ? 'error' : '';
if (error) message.scrollIntoView({ block: 'nearest' });
};
const changed = () => { dirty = true; notice('Unsaved changes — save the collection to apply them.'); };
addEventListener('beforeunload', event => { if (dirty || jsonDraft !== null) { event.preventDefault(); event.returnValue = ''; } });
async function request(endpoint, body, success = 'Saved', replaceJson = false) {
if (busy) return false;
if (jsonDraft !== null && !replaceJson) { notice('Validate & save JSON first, or Reload catalog to discard the JSON draft.', true); return false; }
busy = true; workspace.disabled = true; main.setAttribute('aria-busy', 'true'); notice('Saving…');
try {
data = { ...data, ...await api(endpoint, body) }; dirty = false; jsonDraft = null; busy = false; draw(); notice(success); return true;
} catch (e) { notice(e.message + (dirty ? ' Your unsaved changes are still here.' : ''), true); return false; }
finally { busy = false; workspace.disabled = false; main.removeAttribute('aria-busy'); }
}
const persist = () => request('api/admin/catalog', data);
const cachedRequest = endpoint => dirty || jsonDraft !== null
? notice('Save your changes before importing or researching metadata.', true)
: request(endpoint, { m: researchManga, revision: data.revision });
let researchManga;
const select = (items, value, change, label = 'Catalog selection') => {
const input = h('select', { 'aria-label': label, onchange: e => change(e.target.value) }, items.map(item => h('option', { value: typeof item === 'object' ? item.id : item }, typeof item === 'object' ? item.title : item)));
input.value = value; return input;
};
function draw() {
main.textContent = '';
workspace = h('fieldset', { class: 'catalog-workspace', disabled: busy });
main.append(h('p', { class: 'muted' }, 'Collections organize the library. Relationships describe works. Reading order is your choice.'), message, workspace);
workspace.append(h('div', { class: 'row' }, select([{ id: '', title: 'Choose collection' }, ...data.collections], selected || '', value => { selected = value; draw(); }, 'Choose collection'),
h('button', { class: 'btn', onclick: () => {
const unused = data.mangas.find(m => !data.collections.some(c => c.members.some(member => member.mangaId === m.id)));
if (!unused) { notice(data.mangas.length ? 'All manga are already grouped. Remove a manga from its current collection before creating another.' : 'Add manga to the library before creating a collection.', true); return; }
try {
const c = { id: collectionId(), title: 'New collection', description: '', members: [{ mangaId: unused.id, role: 'main', readingOrder: 1 }], coverMode: 'selected', coverMangaId: unused.id, grouped: true };
data.collections.push(c); selected = c.id; changed(); draw(); $('input[aria-label="Collection title"]', main).focus();
} catch (e) { notice(e.message, true); }
} }, 'New collection'),
h('button', { class: 'btn', onclick: async () => {
if ((dirty || jsonDraft !== null) && !confirm('Discard unsaved catalog changes and reload?')) return;
await request('api/admin/catalog', undefined, 'Catalog reloaded', true);
} }, 'Reload catalog'), h('button', { class: 'btn', onclick: persist }, 'Save all changes')));
const c = data.collections.find(c => c.id === selected);
if (c) {
const title = h('input', { value: c.title, 'aria-label': 'Collection title', oninput: e => { c.title = e.target.value; changed(); } });
const description = h('textarea', { 'aria-label': 'Collection description', oninput: e => { c.description = e.target.value; changed(); } }, c.description);
const file = h('input', { type: 'file', accept: 'image/jpeg,image/png,image/webp,image/gif,image/avif,image/bmp', 'aria-label': 'Custom cover', onchange: async e => {
const chosen = e.target.files[0]; if (!chosen) return;
if (chosen.size > 600000) { notice('Choose an image under 600 KB.', true); return; }
// Read first, then serialize save + upload so no other edit can race its revision.
let base64;
try { base64 = await new Promise((resolve, reject) => {
const reader = new FileReader(); reader.onerror = () => reject(new Error('Could not read this image.'));
reader.onload = () => resolve(reader.result.split(',')[1]); reader.readAsDataURL(chosen);
}); } catch (err) { notice(err.message, true); return; }
if (!await persist()) return;
await request('api/admin/collection-cover', { id: c.id, revision: data.revision, base64 }, 'Cover saved');
} });
const members = h('div', { class: 'collection-editor' });
for (const member of c.members) members.append(h('div', { class: 'row', 'data-member': member.mangaId },
h('span', { class: 'grow' }, data.mangas.find(m => m.id === member.mangaId)?.title || member.mangaId),
select(data.roles, member.role, value => { member.role = value; changed(); }, 'Member role'),
h('input', { type: 'number', min: 0, value: member.readingOrder, 'aria-label': 'Reading order', oninput: e => { member.readingOrder = +e.target.value; changed(); } }),
h('button', { class: 'btn small', disabled: c.members.length === 1, title: c.members.length === 1 ? 'A collection needs a member. Delete the collection to ungroup its last manga.' : 'Remove from this collection', onclick: () => { c.members = c.members.filter(m => m !== member); if (c.coverMangaId === member.mangaId) c.coverMangaId = c.members[0]?.mangaId; changed(); draw(); } }, 'Remove')));
const available = data.mangas.filter(m => !data.collections.some(group => group.members.some(member => member.mangaId === m.id)));
const add = select([{ id: '', title: available.length ? 'Add a manga…' : 'No ungrouped manga available' }, ...available], '', value => { if (value) { c.members.push({ mangaId: value, role: 'side-story', readingOrder: Math.max(0, ...c.members.map(m => m.readingOrder)) + 1 }); changed(); draw(); } }, 'Add manga to collection');
add.disabled = !available.length;
workspace.append(h('section', { class: 'card' }, h('h2', {}, 'Collection'), title, description,
h('label', { class: 'check' }, h('input', { type: 'checkbox', checked: c.grouped, onchange: e => { c.grouped = e.target.checked; changed(); } }), 'Group on the dashboard'),
h('div', { class: 'row' }, h('span', {}, 'Cover'), select(['selected', 'mosaic', ...(c.customCover ? ['custom'] : [])], c.coverMode, value => { c.coverMode = value; changed(); }, 'Cover mode'),
select(c.members.map(member => ({ id: member.mangaId, title: data.mangas.find(m => m.id === member.mangaId)?.title || member.mangaId })), c.coverMangaId, value => { c.coverMangaId = value; changed(); }, 'Cover manga'), file),
h('p', { class: 'muted small' }, 'Uploading a custom cover saves your changes first. A collection must contain at least one manga.'),
h('h3', {}, 'Members & reading order'), members, add,
h('div', { class: 'row' }, h('button', { class: 'btn primary', onclick: persist }, 'Save collection'),
h('button', { class: 'btn danger', onclick: async () => { if (!confirm('Remove this collection? Its manga and reading progress remain.')) return; data.collections = data.collections.filter(group => group !== c); selected = ''; changed(); if (!await persist()) draw(); } }, 'Delete collection'))));
}
const from = select(data.mangas, data.mangas[0]?.id, () => {}, 'Relationship source'), to = select(data.mangas, data.mangas[1]?.id, () => {}, 'Relationship target'), type = select(data.types, 'sequel', () => {}, 'Relationship type');
workspace.append(h('section', { class: 'card' }, h('h2', {}, 'Relationships'),
h('p', { class: 'muted' }, 'Direction: the target is a sequel/prequel/etc. of the source. Suggestions need review; accepting a relationship does not create a collection.'),
h('div', { class: 'row' }, from, h('span', {}, '→'), type, h('span', {}, '→'), to,
h('button', { class: 'btn', onclick: () => {
if (dirty || jsonDraft !== null) return notice('Save your collection changes before editing relationships.', true);
if (from.value === to.value) return notice('Choose two different manga for a relationship.', true);
if (data.relations.some(r => r.from === from.value && r.to === to.value && r.type === type.value)) return notice('That relationship already exists.', true);
data.relations.push({ from: from.value, to: to.value, type: type.value, state: 'confirmed', method: 'manual', evidence: [], explanation: '' }); changed(); persist();
} }, 'Add relationship')),
h('button', { class: 'btn', onclick: () => cachedRequest('api/admin/catalog/suggest') }, 'Import cached metadata suggestions'),
h('div', { class: 'collection-editor' }, data.relations.map(r => h('div', { 'data-relation': JSON.stringify([r.from, r.to, r.type]) },
h('p', {}, `${data.mangas.find(m => m.id === r.from)?.title || r.from} → ${r.type} → ${data.mangas.find(m => m.id === r.to)?.title || r.to}`),
h('p', { class: 'muted small' }, r.explanation),
r.evidence.map(e => h('a', { href: safeHref(e.url), target: '_blank', rel: 'noopener', class: 'btn small' }, e.provider || 'Evidence')),
select(['suggested', 'confirmed', 'rejected', 'conflicted'], r.state, value => {
if (dirty || jsonDraft !== null) { notice('Save your collection changes before editing relationships.', true); draw(); return; }
r.state = value; changed(); persist();
}, 'Relationship state'),
h('button', { class: 'btn small', onclick: () => {
if (dirty || jsonDraft !== null) return notice('Save your collection changes before editing relationships.', true);
data.relations = data.relations.filter(item => item !== r); changed(); persist();
} }, 'Remove'))))));
researchManga = researchManga || data.mangas[0]?.id;
const research = select(data.mangas, researchManga, value => researchManga = value, 'Research manga');
workspace.append(h('section', { class: 'card' }, h('h2', {}, 'Public metadata research'),
h('p', { class: 'muted' }, 'Uses the verified AniList ID and a 30-day cache. Results remain suggestions. An agent can use the JSON export to research conflicts and propose edits.'),
h('div', { class: 'row' }, research, h('button', { class: 'btn', onclick: () => cachedRequest('api/admin/catalog/research') }, 'Research relationships'))));
const json = h('textarea', { class: 'catalog-json', 'aria-label': 'Catalog JSON', oninput: e => { jsonDraft = e.target.value; notice('Unsaved JSON — use Validate & save JSON to apply it.'); } }, jsonDraft ?? JSON.stringify({ version: 1, revision: data.revision, collections: data.collections, relations: data.relations }, null, 2));
workspace.append(h('details', { class: 'card' }, h('summary', {}, 'Advanced · JSON import/export & agent research packet'), json,
h('button', { class: 'btn', onclick: async () => { try { const parsed = JSON.parse(json.value); await request('api/admin/catalog', parsed, 'Saved', true); } catch (e) { notice(e.message, true); } } }, 'Validate & save JSON')));
}
draw();
})();
+28 -3
View File
@@ -21,7 +21,6 @@ function h(tag, props, ...kids) {
function icon(svg) { const t = document.createElement('template'); t.innerHTML = svg.trim(); return t.content.firstChild; }
const ICONS = {
bell: '<svg viewBox="0 0 24 24" width="20" height="20" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M6 8a6 6 0 0 1 12 0c0 7 3 9 3 9H3s3-2 3-9"/><path d="M10.3 21a1.94 1.94 0 0 0 3.4 0"/></svg>',
people: '<svg viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="9" cy="8" r="3.5"/><path d="M2.5 20a6.5 6.5 0 0 1 13 0"/><circle cx="17" cy="9" r="2.5"/><path d="M16 14.2a5 5 0 0 1 6 4.8"/></svg>',
};
async function api(path, body) {
@@ -50,9 +49,25 @@ function ago(t) {
return new Date(t).toLocaleDateString();
}
function readingDuration(ms, compact = false) {
const seconds = Math.max(0, Math.round((+ms || 0) / 1000));
if (!seconds) return 'Not timed';
if (seconds < 60) return compact ? '<1 min' : 'Under a minute';
const minutes = Math.max(1, Math.round(seconds / 60));
if (minutes < 60) return `${minutes} min`;
const hours = Math.floor(minutes / 60), rest = minutes % 60;
return rest ? `${hours} h ${rest} min` : `${hours} h`;
}
const qsa = obj => { const p = new URLSearchParams(); for (const k in obj) if (obj[k] != null && obj[k] !== '') p.set(k, obj[k]); return p.toString(); };
const readUrl = (m, c, v, p) => 'read.html?' + qsa({ m, c, v, p });
const readUrl = (m, c, v) => 'read.html?' + qsa({ m, c, v });
const mangaUrl = m => 'manga.html?' + qsa({ m });
const libraryUrlForTags = tags => {
const p = new URLSearchParams();
for (const tag of tags || []) if (tag) p.append('tag', tag);
const query = p.toString();
return './' + (query ? '?' + query : '');
};
const safeHref = u => (/^https?:\/\//i.test(u || '') ? u : null);
// Top navigation with the notification bell.
@@ -79,13 +94,23 @@ function topNav(me) {
const bell = h('button', { class: 'btn icon bell', title: 'Notifications', 'aria-label': 'Notifications', onclick: e => { e.stopPropagation(); open(); } }, icon(ICONS.bell), count);
panel.addEventListener('click', e => e.stopPropagation());
document.addEventListener('click', () => { panel.hidden = true; });
const settingsLink = h('a', { class: 'btn', href: 'settings.html' }, 'Settings');
const nav = h('header', { class: 'nav' },
h('a', { class: 'brand', href: './' }, 'MangaCircus'),
h('a', { class: 'btn nav-greatest', href: 'greatest.html', title: 'Shijō Saikō no Manga · Greatest of All Time', 'aria-label': 'Shijō Saikō no Manga' }, 'Shijō Saikō no Manga'),
h('a', { class: 'btn', href: 'discover.html' }, 'Discover'),
h('div', { class: 'grow' }),
h('div', { class: 'bell-wrap' }, bell, panel),
h('a', { class: 'btn', href: 'settings.html' }, 'Settings'),
settingsLink,
h('button', { class: 'btn', onclick: async () => { await api('api/logout', {}); location.href = 'login.html'; } }, 'Log out'));
document.body.prepend(nav);
if (me.admin) {
api('api/admin/peering').then(state => {
if (state.peers?.length && nav.isConnected) {
nav.insertBefore(h('a', { class: 'btn', href: 'peer-catalog.html' }, 'Peer catalogs'), settingsLink);
}
}).catch(() => {}).finally(() => { nav.dataset.peerCatalogResolved = 'true'; });
} else nav.dataset.peerCatalogResolved = 'true';
}
function showError(e) {
+10
View File
@@ -0,0 +1,10 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<title>Discover · MangaCircus</title>
<link rel="stylesheet" href="app.css"><link rel="icon" href="icon.svg">
<script src="common.js" defer></script><script src="discover.js" defer></script>
</head>
<body><main><p class="loading">Loading…</p></main></body>
</html>
+84
View File
@@ -0,0 +1,84 @@
'use strict';
(async () => {
let me, state, adminState, active = null;
try { me = await api('api/me'); state = await api('api/discovery'); } catch (error) { return showError(error); }
topNav(me);
const main = $('main'), note = h('p', { role: 'status', 'aria-live': 'polite', class: 'muted small' });
const button = (label, action, className = '') => h('button', { type: 'button', class: `btn ${className}`, onclick: action }, label);
async function refresh() {
try { state = await api('api/discovery'); if (me.admin) adminState = await api('api/admin/discovery'); render(); }
catch (error) { note.textContent = error.message; note.className = 'error'; }
}
async function preference(key, checked) {
try { await api('api/settings', { [key]: checked }); await refresh(); }
catch (error) { note.textContent = error.message; note.className = 'error'; }
}
async function feedback(candidateId, value) {
try {
await api('api/discovery/feedback', { candidateId, value });
state.feedback[candidateId] = value;
note.textContent = value ? 'Preference saved. The next run will use it.' : 'Preference removed.';
render();
} catch (error) { note.textContent = error.message; note.className = 'error'; }
}
async function run() {
try {
await api('api/admin/discovery/run', {});
note.textContent = 'Discovery running. Existing suggestions remain visible until it finishes.';
await refresh();
if (active) clearInterval(active);
active = setInterval(async () => {
await refresh();
if (adminState?.currentRun?.status !== 'running') { clearInterval(active); active = null; }
}, 1500);
} catch (error) { note.textContent = error.message; note.className = 'error'; }
}
function toggle(label, key, hint) {
return h('label', { class: 'check' }, h('input', { type: 'checkbox', checked: !!state.settings[key],
onchange: event => preference(key, event.target.checked) }),
h('span', {}, label, h('small', { class: 'muted discovery-hint' }, hint)));
}
function preferences() {
return h('section', { class: 'card discovery-settings' }, h('h2', {}, 'Your discovery preferences'),
toggle('Personalized discovery', 'discoveryEnabled', 'Off by default. Your feed and feedback stay private to this server.'),
toggle('Use reading activity', 'discoveryReadingActivity', 'Separately opt in to using completed chapters as a small taste signal.'),
toggle('Include mature titles', 'discoveryMature', 'Only affects titles with a known mature tag.'));
}
function adminPanel() {
if (!me.admin) return null;
const runState = adminState?.currentRun?.status || adminState?.lastRun?.status || 'never';
return h('section', { class: 'card discovery-admin' }, h('h2', {}, 'Discovery job'),
h('p', { class: 'muted small' }, `Status: ${runState} · ${adminState?.candidateCount || 0} candidates · ${adminState?.optedInUsers || 0} opted-in users`),
adminState?.lastError ? h('p', { class: 'error' }, adminState.lastError) : null,
button('Run discovery now', run, 'primary'),
h('p', { class: 'muted small' }, 'This first version uses cached public metadata and the curated catalog. It does not ask image hosters, call an LLM, or edit the scraper queue.'));
}
function card(item) {
const candidate = item.candidate, vote = state.feedback[candidate.id] || null;
const evidence = item.evidence || candidate.evidence.filter(entry => item.evidenceIds.includes(entry.id));
const votes = [['interested', '🔥 Interested'], ['maybe', '👍 Maybe'], ['dismissed', 'No thanks'], ['hidden', 'Hide']];
return h('article', { class: 'card discovery-card' },
candidate.cover ? h('img', { src: candidate.cover, alt: '', loading: 'lazy', referrerpolicy: 'no-referrer' }) : h('div', { class: 'discovery-no-cover' }, 'No cover'),
h('div', { class: 'discovery-card-body' },
h('h2', {}, candidate.title),
h('p', { class: 'muted small' }, [candidate.year || 'Year unknown', candidate.status || 'Status unknown', candidate.period].filter(Boolean).join(' · ')),
h('p', {}, item.reason),
h('details', {}, h('summary', {}, 'Why this is here · evidence'),
h('ul', {}, evidence.map(entry => h('li', {}, entry.url ? h('a', { href: entry.url, target: '_blank', rel: 'noopener noreferrer' }, entry.label) : entry.label,
entry.note ? h('span', { class: 'muted' }, ` — ${entry.note}`) : null))),
h('p', { class: 'muted small' }, 'A recommendation does not verify a particular edition or English chapter availability.')),
h('div', { class: 'row discovery-actions' }, votes.map(([value, label]) => button(label, () => feedback(candidate.id, vote === value ? null : value), vote === value ? 'on' : ''))),
candidate.metadataUrl ? h('a', { class: 'small', href: candidate.metadataUrl, target: '_blank', rel: 'noopener noreferrer' }, 'Public metadata ↗') : null));
}
function render() {
main.textContent = '';
main.append(h('h1', {}, 'Discover'), h('p', { class: 'muted' }, 'A small, evidence-backed shelf of manga you might like. Suggestions are not downloads.'), note,
preferences(), adminPanel());
if (!state.enabled) return main.append(h('p', { class: 'muted' }, 'Turn on personalized discovery to see your suggestions.'));
if (!state.items.length) return main.append(h('p', { class: 'muted' }, state.lastRun?.status === 'never'
? 'No discovery run yet. An admin can run one above.' : 'No current suggestions passed the filters. Your previous feedback is retained.'));
main.append(h('p', { class: 'muted small' }, `Last built ${new Date(state.generatedAt).toLocaleString()} · ${state.items.length} suggestions`),
h('div', { class: 'discovery-grid' }, state.items.filter(item => !['hidden', 'dismissed'].includes(state.feedback[item.candidateId])).map(card)));
}
await refresh();
})();
+51
View File
@@ -0,0 +1,51 @@
.greatest-intro { padding: 18px 0 14px; max-width: 750px; }
.greatest-intro h1 { font-size: clamp(28px, 5vw, 42px); letter-spacing: -.03em; }
.greatest-intro p { margin: 7px 0; }
.greatest-eyebrow { color: #e9a47d; text-transform: uppercase; letter-spacing: .15em; font-size: 12px; font-weight: 800; }
.greatest-toolbar { display: flex; align-items: center; gap: 10px; margin: 12px 0 10px; }
.greatest-toolbar > input { flex: 1; min-width: 0; }
.greatest-sort, .greatest-filters { display: flex; flex-wrap: wrap; gap: 7px; }
.greatest-filters { margin: 10px 0 12px; }
.greatest-periods { margin-top: 12px; }
.greatest-period { padding: 20px 0 28px; border-top: 2px solid #48404a; }
.greatest-period-heading { display: flex; align-items: baseline; justify-content: space-between; gap: 12px; margin-bottom: 16px; }
.greatest-period-heading h2 { margin: 0; font-size: 21px; }
.greatest-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(170px, 1fr)); gap: 22px 16px; }
.greatest-tile { min-width: 0; display: flex; flex-direction: column; }
.greatest-open { position: relative; display: block; flex: 1; width: 100%; padding: 0; border: 0; text-align: left; color: inherit; background: none; cursor: pointer; font: inherit; }
.greatest-open:hover .greatest-cover img { transform: scale(1.035); }
.greatest-open:focus-visible, .greatest-vote:focus-visible { outline: 2px solid #fff; outline-offset: 3px; }
.greatest-cover { position: relative; display: block; width: 100%; aspect-ratio: 2 / 3; overflow: hidden; border-radius: 9px; background: linear-gradient(150deg, #49343c, #282c38 60%, #18181d); box-shadow: 0 4px 17px #0007; }
.greatest-cover img { position: absolute; inset: 0; width: 100%; height: 100%; object-fit: cover; transition: transform .2s; }
.greatest-cover-fallback { display: grid; place-items: center; width: 100%; height: 100%; color: #dbc8bd; font: 700 80px/1 Georgia, serif; opacity: .6; }
.greatest-owned { position: absolute; top: 7px; right: 7px; padding: 2px 7px; border-radius: 9px; background: #183d34; color: #a7f0d1; font-size: 11px; font-weight: 700; box-shadow: 0 1px 8px #0008; }
.greatest-name { display: block; margin-top: 8px; font-weight: 700; line-height: 1.25; }
.greatest-meta { display: block; margin-top: 2px; color: var(--muted); font-size: 12px; }
.greatest-reason-label { display: block; margin-top: 10px; color: #e9a47d; font-size: 11px; font-weight: 800; letter-spacing: .08em; text-transform: uppercase; }
.greatest-reason { display: -webkit-box; overflow: hidden; -webkit-box-orient: vertical; -webkit-line-clamp: 4; line-clamp: 4; margin-top: 3px; color: var(--fg); font-size: 12px; line-height: 1.4; }
.greatest-votes { display: flex; gap: 8px; margin-top: 12px; }
.greatest-votes.compact { gap: 3px; margin-top: 6px; }
.greatest-vote { display: inline-grid; place-items: center; flex: 1; min-width: 0; min-height: 36px; border: 1px solid var(--line); border-radius: 8px; background: var(--card); color: var(--fg); cursor: pointer; font: inherit; }
.greatest-vote:hover { background: var(--card2); }
.greatest-vote.selected { border-color: var(--accent); background: color-mix(in srgb, var(--accent) 25%, var(--card)); }
.greatest-vote:disabled { opacity: .5; cursor: wait; }
.greatest-dialog { width: min(780px, calc(100vw - 24px)); max-height: min(850px, calc(100vh - 30px)); overflow: auto; padding: 18px; border: 1px solid var(--line); border-radius: 14px; background: var(--card); color: var(--fg); box-shadow: 0 18px 70px #000a; }
.greatest-dialog::backdrop { background: #000b; }
.greatest-dialog-head { display: flex; align-items: center; justify-content: space-between; gap: 10px; margin-bottom: 14px; }
.greatest-dialog-body { display: grid; grid-template-columns: minmax(130px, 230px) minmax(0, 1fr); gap: 22px; }
.greatest-dialog-body h2 { margin: 0 0 8px; font-size: 25px; }
.greatest-reason-heading { margin: 18px 0 3px; font-size: 15px; color: #e9a47d; }
.greatest-dialog-body p { margin: 9px 0; }
.greatest-links { display: flex; flex-wrap: wrap; gap: 8px; margin: 17px 0; }
.greatest-hoster-panel { margin-top: 15px; border-top: 1px solid var(--line); padding-top: 4px; }
.greatest-hoster-row, .greatest-hoster-tools { display: flex; align-items: center; flex-wrap: wrap; gap: 8px; margin: 8px 0; }
.greatest-hoster-row > a { overflow-wrap: anywhere; }
.greatest-hoster-row > .btn { margin-left: auto; }
.greatest-hoster-tools > input { flex: 1 1 220px; min-width: 0; }
.greatest-hoster-results { max-height: 240px; overflow: auto; padding: 0 8px; border: 1px solid var(--line); border-radius: 8px; }
@media (max-width: 680px) {
.greatest-toolbar { align-items: stretch; flex-direction: column; }
.greatest-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 18px 12px; }
.greatest-dialog-body { grid-template-columns: 1fr; }
.greatest-dialog-body > .greatest-cover { width: min(160px, 42vw); }
}
+17
View File
@@ -0,0 +1,17 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<title>Shijō Saikō no Manga · MangaCircus</title>
<link rel="stylesheet" href="app.css">
<link rel="stylesheet" href="greatest.css">
<link rel="icon" href="icon.svg">
<meta name="theme-color" content="#121214">
<script src="common.js" defer></script>
<script src="greatest.js" defer></script>
</head>
<body>
<main><p class="loading">Loading the collection…</p></main>
</body>
</html>
+240
View File
@@ -0,0 +1,240 @@
'use strict';
/* Curated catalog UI. Ratings are personal; scraper inspection is admin-only. */
(async function () {
let me, data;
try { [me, data] = await Promise.all([api('api/me'), api('api/greatest')]); } catch (error) { return showError(error); }
topNav(me);
const main = $('main'); main.textContent = '';
let sortMode = store.get('greatest-sort') === 'all' ? 'all' : 'mine';
let filterMode = 'all';
const query = h('input', { type: 'search', placeholder: 'Search the collection…', 'aria-label': 'Search Greatest of All Time' });
const count = h('span', { class: 'muted small', 'aria-live': 'polite' });
const mine = h('button', { type: 'button', class: 'btn small' }, 'Rank by my vote');
const everyone = h('button', { type: 'button', class: 'btn small' }, 'Rank by everyone');
const filterBar = h('div', { class: 'greatest-filters', 'aria-label': 'Filter by your rating' });
const filters = [
['all', 'All'], ['rated', 'Rated'], ['flame', '🔥 Flame'], ['good', '👍 Looks good'],
['neutral', '● Neutral'], ['skip', '💀 Not for me'], ['unrated', 'Unrated'], ['owned', '📚 In library'],
];
const filterButtons = new Map();
for (const [value, label] of filters) {
const button = h('button', { type: 'button', class: 'filter-chip' }, label);
button.onclick = () => { filterMode = value; render(); };
filterButtons.set(value, button); filterBar.append(button);
}
const results = h('div', { class: 'greatest-periods' });
const dialog = h('dialog', { class: 'greatest-dialog', 'aria-label': 'Manga details' });
const sourceCache = new Map();
let detailId = null;
document.body.append(dialog);
mine.onclick = () => { sortMode = 'mine'; store.set('greatest-sort', sortMode); render(); };
everyone.onclick = () => { sortMode = 'all'; store.set('greatest-sort', sortMode); render(); };
query.oninput = render;
main.append(
h('div', { class: 'greatest-intro' },
h('p', { class: 'greatest-eyebrow' }, 'Greatest of All Time'),
h('h1', {}, data.title || 'Shijō Saikō no Manga'),
h('p', { class: 'muted' }, '史上最高の漫画 · A curated journey through manga history. Your votes are private to your account; the community ranking combines everyone’s votes.'),
h('p', { class: 'muted small' }, '🔥 Definitely want · 👍 Looks good · ● Neutral · 💀 Not for me · 📚 Already in your library')),
h('div', { class: 'greatest-toolbar' }, query, h('div', { class: 'greatest-sort' }, mine, everyone)),
filterBar, count, results);
function ratingButtons(item, compact = false) {
const bar = h('div', { class: 'greatest-votes' + (compact ? ' compact' : ''), 'aria-label': `Rate ${item.title}` });
for (const [value, symbol, label] of [
['flame', '🔥', 'Definitely want'], ['good', '👍', 'Looks good'],
['neutral', '●', 'Neutral'], ['skip', '💀', 'Not for me'],
]) {
const selected = item.myRating === value;
const button = h('button', { type: 'button', class: 'greatest-vote' + (selected ? ' selected' : ''),
'aria-label': `${label}: ${item.title}`, 'aria-pressed': String(selected), title: label }, symbol);
button.onclick = async event => {
event.stopPropagation(); button.disabled = true;
try {
const next = selected ? null : value;
const saved = await api('api/greatest/rating', { id: item.id, rating: next });
Object.assign(item, saved);
render();
if (dialog.open) openDetails(item);
} catch (error) { alert(error.message || String(error)); button.disabled = false; }
};
bar.append(button);
}
return bar;
}
function cover(item) {
const box = h('span', { class: 'greatest-cover' }, h('span', { class: 'greatest-cover-fallback', 'aria-hidden': 'true' }, item.title.slice(0, 1)));
if (item.cover && safeHref(item.cover)) {
const image = h('img', { src: item.cover, alt: '', loading: 'lazy' });
image.onerror = () => image.remove();
box.append(image);
}
return box;
}
function tile(item) {
const open = h('button', { type: 'button', class: 'greatest-open', 'aria-label': `Details for ${item.title}` },
cover(item),
item.library ? h('span', { class: 'greatest-owned', title: 'Already in your library' }, '📚 In library') : null,
h('span', { class: 'greatest-name' }, item.title),
h('span', { class: 'greatest-meta' }, `${item.year} · ${item.chapters == null ? 'Chapters unknown' : `${item.chapters} chapters`}`),
h('span', { class: 'greatest-meta' }, statusLabel(item.status)),
h('span', { class: 'greatest-reason-label' }, 'Why it matters'),
h('span', { class: 'greatest-reason' }, plainNote(item.note)));
open.onclick = () => openDetails(item);
return h('article', { class: 'greatest-tile' }, open, ratingButtons(item, true));
}
function statusLabel(status) {
return status === 'ongoing' ? 'Ongoing' : status === 'completed' ? 'Completed' : 'Status unknown';
}
function plainNote(note) { return String(note || '').replace(/\*/g, ''); }
function openDetails(item) {
detailId = item.id;
dialog.textContent = '';
const close = h('button', { type: 'button', class: 'btn small', onclick: () => dialog.close() }, 'Close');
const links = h('div', { class: 'greatest-links' });
if (item.library) links.append(h('a', { class: 'btn primary', href: mangaUrl(item.library.id) }, 'Open in library'));
if (safeHref(item.source?.url)) links.append(h('a', { class: 'btn', href: item.source.url, target: '_blank', rel: 'noopener noreferrer' },
`Official ${item.source.kind || 'source'} ↗`));
if (safeHref(item.metadata?.url)) links.append(h('a', { class: 'btn', href: item.metadata.url, target: '_blank', rel: 'noopener noreferrer' }, 'AniList metadata ↗'));
const hosterPanel = h('section', { class: 'greatest-hoster-panel', 'aria-label': 'Scraper sources' });
const votes = item.votes || {};
dialog.append(
h('div', { class: 'greatest-dialog-head' }, h('span', { class: 'muted small' }, `${item.period} · curator #${item.rank}`), close),
h('div', { class: 'greatest-dialog-body' }, cover(item), h('div', {},
h('h2', {}, item.title),
h('p', { class: 'muted' }, `${item.year} · ${item.chapters == null ? 'Chapter count unknown' : `${item.chapters} chapters`} · ${statusLabel(item.status)}`),
h('h3', { class: 'greatest-reason-heading' }, 'Why it matters'),
h('p', {}, plainNote(item.note)),
h('p', { class: 'muted small' }, `Community votes: 🔥 ${votes.flame || 0} · 👍 ${votes.good || 0} · ● ${votes.neutral || 0} · 💀 ${votes.skip || 0}`),
ratingButtons(item), links, hosterPanel,
h('p', { class: 'muted small' }, 'Chapter counts and status come from public catalog metadata, or from your local library when present. Availability may vary by region.'))));
if (!dialog.open) dialog.showModal();
renderHosters(item, hosterPanel, sourceCache.get(item.id));
if (me.admin && !sourceCache.has(item.id)) refreshHosters(item, hosterPanel);
}
async function refreshHosters(item, panel) {
try {
const state = await api(`api/greatest/sources?id=${encodeURIComponent(item.id)}`);
sourceCache.set(item.id, state);
if (detailId === item.id && dialog.open) renderHosters(item, panel, state);
} catch (error) {
if (detailId === item.id && dialog.open) renderHosters(item, panel, { available: false, error: error.message, links: item.hosters || [] });
}
}
function renderHosters(item, panel, state) {
panel.textContent = '';
panel.append(h('h3', { class: 'greatest-reason-heading' }, 'MangaDex / WeebCentral'));
if (me.admin && !state) {
panel.append(h('p', { class: 'muted small' }, 'Checking scraper queue…'));
return;
}
const links = state?.links || item.hosters || [];
const jobs = state?.jobs || [];
if (!links.length) panel.append(h('p', { class: 'muted small' }, 'No confirmed series link yet. A title search may find a different edition; check it before adding.'));
for (const link of links) {
const job = jobs.find(job => job.source === link.source && job.seriesId === link.seriesId);
panel.append(h('div', { class: 'greatest-hoster-row' },
h('a', { href: link.url, target: '_blank', rel: 'noopener noreferrer' }, `${link.label} ↗`),
link.origin === 'queue title match' ? h('span', { class: 'muted small' }, 'Queue title match · verify edition') : null,
me.admin ? h('span', { class: 'muted small' }, job ? `In scraper · ${job.status.replaceAll('_', ' ')}` : state?.available === false ? 'Queue status unavailable' : 'Not in scraper queue') : null,
me.admin && state?.available !== false ? queueButton(item, link.url, job) : null));
}
if (me.admin) {
if (state?.available === false) panel.append(h('p', { class: 'small error' }, `Scraper status unavailable: ${state.error || 'unknown error'}. Queue actions are disabled.`));
const message = h('span', { class: 'small', role: 'status' });
const search = h('button', { class: 'btn small', type: 'button', onclick: async () => {
search.disabled = true; message.textContent = 'Looking up MangaDex titles…';
try {
const found = await api(`api/greatest/mangadex-search?id=${encodeURIComponent(item.id)}`);
message.textContent = '';
const results = h('div', { class: 'greatest-hoster-results' });
for (const candidate of found.results || []) {
const existing = jobs.find(job => job.source === 'mangadex' && job.seriesId === candidate.url.split('/').at(-1));
results.append(h('div', { class: 'greatest-hoster-row' },
h('a', { href: candidate.url, target: '_blank', rel: 'noopener noreferrer' }, `${candidate.title}${candidate.year ? ` (${candidate.year})` : ''} ↗`),
existing ? h('span', { class: 'muted small' }, `In scraper · ${existing.status}`) : null,
state?.available !== false ? queueButton(item, candidate.url, existing) : null));
}
if (!found.results?.length) results.append(h('p', { class: 'muted small' }, 'No MangaDex candidates found.'));
panel.append(h('p', { class: 'muted small' }, 'Search candidates are not verified matches. Check title, year, and edition before adding.'), results);
} catch (error) { message.textContent = error.message || String(error); }
finally { search.disabled = false; }
} }, 'Search MangaDex');
const refresh = h('button', { class: 'btn small', type: 'button', onclick: () => refreshHosters(item, panel) }, 'Refresh queue status');
panel.append(h('div', { class: 'greatest-hoster-tools' }, search,
h('a', { class: 'btn small', href: 'https://weebcentral.com/search', target: '_blank', rel: 'noopener noreferrer' }, 'Search WeebCentral ↗'),
refresh, message));
const manual = h('input', { type: 'url', placeholder: 'Paste a MangaDex or WeebCentral series URL', 'aria-label': 'Series URL' });
const add = h('button', { class: 'btn small', type: 'button', disabled: state?.available === false, onclick: () => inspectUrl(item, manual.value, add, message, panel) }, 'Add URL to scraper');
panel.append(h('div', { class: 'greatest-hoster-tools' }, manual, add));
}
}
function queueButton(item, url, job) {
return h('button', { class: 'btn small', type: 'button', disabled: !!job,
title: job ? `Already in scraper: ${job.status}` : 'Queue an inspection; confirm in Admin → Scraper before downloading',
onclick: async event => {
const panel = event.currentTarget.closest('.greatest-hoster-panel');
await inspectUrl(item, url, event.currentTarget, null, panel);
} }, job ? 'Already added' : 'Add to scraper');
}
async function inspectUrl(item, url, button, message, panel) {
button.disabled = true;
try {
await api('api/admin/scraper', { command: 'enqueue', url: String(url || '').trim(), title: item.title,
metadataSearch: item.title, libraryMangaId: item.library?.id, scope: { mode: 'missing' },
existingPolicy: 'skip-any', inspect: true });
if (message) message.textContent = 'Added for inspection. Confirm any download in Admin → Scraper.';
sourceCache.delete(item.id);
await refreshHosters(item, panel);
} catch (error) {
if (message) message.textContent = error.message || String(error);
else alert(error.message || String(error));
button.disabled = false;
}
}
function matches(item, q) {
if (q && !`${item.title} ${item.note || ''}`.toLowerCase().includes(q)) return false;
if (filterMode === 'rated') return !!item.myRating;
if (filterMode === 'unrated') return !item.myRating;
if (filterMode === 'owned') return !!item.library;
if (['flame', 'good', 'neutral', 'skip'].includes(filterMode)) return item.myRating === filterMode;
return true;
}
function render() {
mine.classList.toggle('on', sortMode === 'mine'); everyone.classList.toggle('on', sortMode === 'all');
mine.setAttribute('aria-pressed', String(sortMode === 'mine'));
everyone.setAttribute('aria-pressed', String(sortMode === 'all'));
for (const [value, button] of filterButtons) {
button.classList.toggle('on', filterMode === value);
button.setAttribute('aria-pressed', String(filterMode === value));
}
results.textContent = '';
const q = query.value.trim().toLowerCase();
let visible = 0;
for (const period of data.periods) {
const items = data.entries.filter(item => item.period === period && matches(item, q));
if (!items.length) continue;
items.sort((a, b) => sortMode === 'all'
? b.score - a.score || a.rank - b.rank
: (ratingScore(b.myRating) - ratingScore(a.myRating)) || a.rank - b.rank);
visible += items.length;
results.append(h('section', { class: 'greatest-period' },
h('div', { class: 'greatest-period-heading' }, h('h2', {}, period), h('span', { class: 'muted small' }, `${items.length} manga`)),
h('div', { class: 'greatest-grid' }, items.map(tile))));
}
count.textContent = `${visible} of ${data.entries.length} manga · ${sortMode === 'all' ? 'community score' : 'your vote'} within each period`;
if (!visible) results.append(h('p', { class: 'empty muted' }, 'No manga match these filters.'));
}
function ratingScore(value) { return { flame: 3, good: 1, neutral: 0, skip: -2 }[value] ?? -3; }
render();
})();
+2
View File
@@ -0,0 +1,2 @@
<!doctype html>
<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1"><title>Service health · MangaCircus</title><link rel="stylesheet" href="app.css"><script src="common.js" defer></script><script src="settings-nav.js" defer></script><script src="health.js" defer></script></head><body><main><p>Loading…</p></main></body></html>
+34
View File
@@ -0,0 +1,34 @@
'use strict';
(async () => {
let me;
try { me = await api('api/me'); } catch (e) { return showError(e); }
topNav(me);
if (!me.admin) return showError(new Error('Administrator access required.'));
const main = settingsShell(me, 'admin/health').content;
async function load() {
try {
const [health, scraper] = await Promise.all([api('api/admin/health'), api('api/admin/scraper')]);
main.textContent = '';
main.append(h('section', { class: 'card' },
h('p', {}, `Reader: ${health.reader.ok ? 'healthy' : 'scan error'} · Sentinel: ${health.sentinel.status} · Scraper: ${scraper.status || 'unavailable'}`),
h('p', { class: 'muted small' }, health.sentinel.checkedAt ? `Last independent check: ${new Date(health.sentinel.checkedAt).toLocaleString()}` : 'Sentinel has not reported yet.'),
h('a', { class: 'btn', href: 'settings.html#admin/scraper' }, 'Queue & hoster settings')));
for (const entry of scraper.active || []) main.append(h('section', { class: 'card' }, h('h2', {}, entry.source),
h('p', {}, `${entry.current.title} · ${entry.job?.completed || 0}/${entry.job?.total || '?'} chapters`),
h('p', { class: 'muted' }, `Checkpoint: ${entry.job?.updated || 'catalog discovery'}`)));
const failures = (scraper.queue || []).filter(item => ['error', 'quarantined', 'unavailable', 'retry_wait'].includes(item.status));
main.append(h('section', { class: 'card' }, h('h2', {}, 'Jobs needing attention'), failures.length ? failures.map(item => h('div', { class: 'collection-editor' },
h('p', {}, `${item.title} · ${item.status} · ${item.failure?.kind || 'unknown'}`), h('p', { class: 'muted' }, item.error || ''),
item.nextAttemptAt ? h('p', {}, `Retry scheduled: ${new Date(item.nextAttemptAt).toLocaleString()}`) : null,
h('div', { class: 'row' }, h('button', { class: 'btn', onclick: async () => {
if (item.failure?.kind === 'access' && !confirm('Has the source access problem been resolved? This manually authorizes one retry.')) return;
await api('api/admin/scraper', { command: 'control', action: item.status === 'unavailable' ? 'inspect' : 'retry', id: item.id }); load();
} }, item.status === 'unavailable' ? 'Inspect source' : 'Retry'), h('button', { class: 'btn', onclick: async () => { await api('api/admin/scraper', { command: 'control', action: 'skip', id: item.id }); load(); } }, 'Skip job')))) : h('p', { class: 'muted' }, 'No failed jobs.')));
const incidents = [...(health.sentinel.incidents || []), ...(scraper.incidents || [])];
main.append(h('details', { class: 'card', open: incidents.some(i => i.state === 'open') }, h('summary', {}, 'Incident & recovery history'), incidents.slice(-50).map(i =>
h('p', {}, `${i.at} · ${i.state} · ${i.message || i.failure?.message || ''}${i.resolution ? ' · ' + i.resolution : ''}`))));
} catch (e) { main.textContent = ''; main.append(h('p', { class: 'error', role: 'alert' }, e.message)); }
setTimeout(load, document.hidden ? 30000 : 10000);
}
load();
})();
+129 -14
View File
@@ -1,5 +1,5 @@
'use strict';
/* Library overview: a grid of covers. */
/* Library overview: searchable covers with combinable metadata filters. */
(async function () {
let me, data;
try { [me, data] = await Promise.all([api('api/me'), api('api/library')]); } catch (e) { return showError(e); }
@@ -14,32 +14,147 @@
if (data.scanning) setTimeout(() => location.reload(), 2000);
return;
}
const storyAvailable = !!(data.reading && data.reading.activeMs > 0);
const filter = h('input', { type: 'search', placeholder: 'Filter…', 'aria-label': 'Filter', value: store.get('filter') || '' });
const sort = h('select', { 'aria-label': 'Sort' },
h('option', { value: 'title' }, 'A–Z'), h('option', { value: 'updated' }, 'Recently updated'), h('option', { value: 'read' }, 'Recently read'));
sort.value = store.get('sort') || 'title';
const allTags = [...new Set(data.mangas.flatMap(m => m.tags || []).filter(Boolean))].sort((a, b) => a.localeCompare(b));
const allStatuses = [...new Set(data.mangas.map(m => m.status).filter(Boolean))].sort((a, b) => a.localeCompare(b));
const requestedTags = new URLSearchParams(location.search).getAll('tag').filter(tag => allTags.includes(tag));
let rememberedTags = [];
try { rememberedTags = JSON.parse(store.get('tags') || '[]'); } catch (_) { /* ignore old invalid state */ }
const selectedTags = new Set((requestedTags.length ? requestedTags : rememberedTags).filter(tag => allTags.includes(tag)));
const filter = h('input', { type: 'search', placeholder: 'Search titles or authors…', 'aria-label': 'Search library', value: requestedTags.length ? '' : (store.get('filter') || '') });
const status = h('select', { 'aria-label': 'Filter by publication status' }, h('option', { value: '' }, 'All statuses'),
allStatuses.map(value => h('option', { value }, value)));
status.value = !requestedTags.length && allStatuses.includes(store.get('status')) ? store.get('status') : '';
const sort = h('select', { 'aria-label': 'Sort library' },
h('option', { value: 'title' }, 'Title A–Z'),
h('option', { value: 'title-desc' }, 'Title Z–A'),
h('option', { value: 'release-desc' }, 'First release · newest'),
h('option', { value: 'release-asc' }, 'First release · oldest'),
h('option', { value: 'updated' }, 'Recently updated'),
h('option', { value: 'read' }, 'Recently read'),
storyAvailable ? h('option', { value: 'timed' }, 'Recently timed') : null,
storyAvailable ? h('option', { value: 'time' }, 'Time together') : null);
const allowedSorts = ['title', 'title-desc', 'release-desc', 'release-asc', 'updated', 'read', ...(storyAvailable ? ['timed', 'time'] : [])];
sort.value = allowedSorts.includes(store.get('sort')) ? store.get('sort') : 'title';
const resultCount = h('div', { class: 'muted small result-count', 'aria-live': 'polite' });
const tagButtons = new Map();
const tagBar = h('div', { class: 'tag-filter', 'aria-label': 'Filter by tags' });
const clearTags = h('button', { class: 'filter-chip', type: 'button' }, 'All tags');
clearTags.onclick = () => { selectedTags.clear(); syncTags(); render(); };
tagBar.append(clearTags);
for (const tag of allTags) {
const button = h('button', { class: 'filter-chip', type: 'button' }, tag);
button.onclick = () => { selectedTags.has(tag) ? selectedTags.delete(tag) : selectedTags.add(tag); syncTags(); render(); };
tagButtons.set(tag, button);
tagBar.append(button);
}
const grid = h('div', { class: 'grid' });
main.append(h('div', { class: 'toolbar' }, filter, sort), grid);
const grouping = h('select', { 'aria-label': 'Collection display' },
h('option', { value: 'grouped' }, 'Group collections'), h('option', { value: 'expanded' }, 'Expand collections'), h('option', { value: 'individual' }, 'Individual manga'));
grouping.value = ['grouped', 'expanded', 'individual'].includes(store.get('grouping')) ? store.get('grouping') : 'grouped';
grouping.onchange = render;
if (storyAvailable) {
const most = data.mangas.find(manga => manga.id === data.reading.mostTimeMangaId);
const latest = data.mangas.find(manga => manga.id === data.reading.lastMangaId);
main.append(h('section', { class: 'reading-summary' },
h('div', { class: 'reading-summary-title' }, h('span', { 'aria-hidden': 'true' }, '⏱'), h('b', {}, 'Your Reading Story')),
h('div', { class: 'reading-summary-stats' },
h('strong', {}, readingDuration(data.reading.activeMs)),
h('span', { class: 'muted' }, `together · ${data.reading.timedChapters} chapter${data.reading.timedChapters === 1 ? '' : 's'} timed`),
data.reading.rereads ? h('span', { class: 'story-pill' }, `${data.reading.rereads} reread${data.reading.rereads === 1 ? '' : 's'}`) : null),
h('div', { class: 'reading-summary-links' },
most ? h('a', { href: mangaUrl(most.id) }, `Most time with ${most.title}`) : null,
latest && data.reading.lastChapterId ? h('a', { href: readUrl(latest.id, data.reading.lastChapterId) }, `Continue ${latest.title} →`) : null)));
}
main.append(
h('div', { class: 'library-controls' },
h('div', { class: 'toolbar' }, filter, status, sort, grouping,
me.admin ? h('a', { class: 'btn', href: 'collections.html' }, 'Manage collections') : null),
allTags.length ? h('div', { class: 'filter-group' },
h('div', { class: 'filter-heading' }, h('b', {}, 'Tags'), h('span', { class: 'muted small' }, 'Manga must match every selected tag')),
tagBar) : null,
resultCount),
grid);
function syncTags() {
store.set('tags', JSON.stringify([...selectedTags]));
const params = new URLSearchParams(location.search);
params.delete('tag');
for (const tag of selectedTags) params.append('tag', tag);
const query = params.toString();
history.replaceState(null, '', location.pathname + (query ? '?' + query : ''));
}
function releaseSort(a, b, direction) {
const at = +a.firstReleaseAt || 0, bt = +b.firstReleaseAt || 0;
if (!at && !bt) return a.title.localeCompare(b.title);
if (!at) return 1;
if (!bt) return -1;
return direction * (at - bt) || a.title.localeCompare(b.title);
}
function render() {
store.set('filter', filter.value); store.set('sort', sort.value);
store.set('filter', filter.value); store.set('sort', sort.value); store.set('status', status.value); store.set('grouping', grouping.value);
const q = filter.value.trim().toLowerCase();
const list = data.mangas.filter(m => !q || m.title.toLowerCase().includes(q));
const groups = (data.collections || []).filter(c => c.grouped);
const membership = new Set(groups.flatMap(c => c.members.map(m => m.id)));
const candidates = grouping.value === 'grouped'
? [...data.mangas.filter(m => !membership.has(m.id)), ...groups.map(c => ({ ...c, isCollection: true }))] : [...data.mangas];
const matches = m => {
const haystack = [m.title, ...(m.altTitles || []), ...(m.authors || [])].join(' ').toLowerCase();
return (!q || haystack.includes(q)) && (!status.value || m.status === status.value) &&
[...selectedTags].every(tag => (m.tags || []).includes(tag));
};
// Match complete tag combinations on a single member, not unrelated tags spread across a group.
const list = candidates.filter(m => m.isCollection ? m.members.some(matches) || (q && m.title.toLowerCase().includes(q) && !status.value && !selectedTags.size) : matches(m));
if (sort.value === 'title') list.sort((a, b) => a.title.localeCompare(b.title));
if (sort.value === 'title-desc') list.sort((a, b) => b.title.localeCompare(a.title));
if (sort.value === 'release-asc') list.sort((a, b) => releaseSort(a, b, 1));
if (sort.value === 'release-desc') list.sort((a, b) => releaseSort(a, b, -1));
if (sort.value === 'updated') list.sort((a, b) => b.updated - a.updated);
if (sort.value === 'read') list.sort((a, b) => b.lastRead - a.lastRead);
if (sort.value === 'timed') list.sort((a, b) => (b.reading?.updatedAt || 0) - (a.reading?.updatedAt || 0));
if (sort.value === 'time') list.sort((a, b) => (b.reading?.activeMs || 0) - (a.reading?.activeMs || 0));
clearTags.classList.toggle('on', !selectedTags.size);
clearTags.setAttribute('aria-pressed', String(!selectedTags.size));
for (const [tag, button] of tagButtons) {
button.classList.toggle('on', selectedTags.has(tag));
button.setAttribute('aria-pressed', String(selectedTags.has(tag)));
}
resultCount.textContent = `${list.length} results · ${data.mangas.length} manga in the library`;
grid.textContent = '';
for (const m of list) {
grid.append(h('a', { class: 'tile', href: mangaUrl(m.id) },
h('div', { class: 'cv' }, m.cover ? h('img', { loading: 'lazy', src: m.cover, alt: '' }) : null),
const tile = m => {
const cover = m.isCollection && m.coverMode === 'mosaic' && m.covers.length
? h('div', { class: 'cv collection-mosaic' }, m.covers.map(src => h('img', { loading: 'lazy', src, alt: '' })))
: h('div', { class: 'cv' }, m.cover ? h('img', { loading: 'lazy', src: m.cover, alt: '' }) : null);
return h('a', { class: 'tile', href: m.isCollection ? 'collection.html?' + qsa({ id: m.id }) : mangaUrl(m.id) },
cover, m.isCollection ? h('span', { class: 'collection-count' }, `${m.members.length} manga`) : null,
m.unread ? h('span', { class: 'badge' }, m.unread + ' new') : null,
h('div', { class: 't' }, m.title),
h('div', { class: 's' }, m.read ? `${m.read} / ${m.chapters} read` : `${m.chapters} chapter${m.chapters === 1 ? '' : 's'}`),
m.read ? h('div', { class: 'prog' }, h('i', { style: `width:${Math.round((100 * m.read) / m.chapters)}%` })) : null));
h('div', { class: 's' }, [m.firstRelease, m.status, m.read ? `${m.read} / ${m.chapters} read` : `${m.chapters} chapters`].filter(Boolean).join(' · ')),
m.read ? h('div', { class: 'prog' }, h('i', { style: `width:${Math.round((100 * m.read) / Math.max(1, m.chapters))}%` })) : null);
};
if (grouping.value === 'expanded') {
const visible = new Set(list.map(m => m.id));
for (const c of groups) {
const members = c.members.filter(m => visible.has(m.id));
if (!members.length) continue;
grid.append(h('h2', { class: 'collection-heading' }, h('a', { href: 'collection.html?' + qsa({ id: c.id }) }, c.title)));
members.forEach(m => { grid.append(tile(m)); visible.delete(m.id); });
}
if (!list.length) grid.append(h('p', { class: 'muted' }, 'Nothing matches.'));
list.filter(m => visible.has(m.id)).forEach(m => grid.append(tile(m)));
return;
}
for (const m of list) {
grid.append(tile(m));
}
if (!list.length) grid.append(h('p', { class: 'muted' }, 'Nothing matches these filters.'));
}
filter.addEventListener('input', render);
status.addEventListener('change', render);
sort.addEventListener('change', render);
syncTags();
render();
})();
+216 -30
View File
@@ -1,5 +1,5 @@
'use strict';
/* Manga page: info, follow toggle and the chapter list (books are just separators). */
/* Manga overview: complete metadata, volume filters and newest-first chapters. */
(async function () {
const id = new URLSearchParams(location.search).get('m') || '';
let me, M;
@@ -11,63 +11,231 @@
const TYPE = { rtl: 'Manga · right to left', ltr: 'Comic · left to right', ttb: 'Webtoon · vertical strip' };
const arr = x => (Array.isArray(x) ? x : x ? [x] : []);
const text = value => {
if (value == null || value === '') return '';
if (Array.isArray(value)) return value.map(text).filter(Boolean).join(', ');
if (typeof value === 'object') return JSON.stringify(value);
return String(value);
};
const releaseDate = () => {
const raw = meta.firstReleaseDate || meta.releaseDate || meta.startDate;
if (!raw) return text(meta.year);
if (typeof raw !== 'object') return text(raw);
return [raw.year, raw.month && String(raw.month).padStart(2, '0'), raw.day && String(raw.day).padStart(2, '0')].filter(Boolean).join('-');
};
const follow = h('button', { class: 'btn' }, '');
const drawFollow = () => { follow.textContent = M.following ? '🔔 Notifying you' : 'Notify me about new chapters'; follow.classList.toggle('on', M.following); };
follow.onclick = async () => { M.following = (await api('api/follow', { m: M.id, on: !M.following })).following; drawFollow(); };
drawFollow();
const sources = arr(meta.sources).filter(s => s && s.name);
const sources = arr(meta.sources).filter(source => source && (source.name || source.url));
const tags = arr(meta.tags).filter(Boolean);
const numberedChapters = M.chapters.map(chapter => /^Ch\. (\d+(?:\.\d+)?)/.exec(chapter.label || ''))
.filter(Boolean).map(match => Number(match[1]));
const firstNumber = numberedChapters.length ? Math.min(...numberedChapters) : null;
const books = {};
for (const book of M.books) books[book.key] = book;
const selectedBooks = new Set();
main.append(h('section', { class: 'hero' },
M.cover ? h('img', { src: M.cover, alt: '' }) : null,
h('div', { class: 'info' },
h('h1', {}, M.title),
arr(meta.altTitles).length ? h('div', { class: 'muted' }, arr(meta.altTitles).join(' · ')) : null,
h('div', { class: 'muted' }, [arr(meta.authors).join(', '), meta.status, meta.year, TYPE[M.direction]].filter(Boolean).join(' · ')),
h('div', { class: 'chips' }, arr(meta.tags).map(t => h('span', { class: 'chip' }, t))),
h('div', { class: 'muted' }, [arr(meta.authors).join(', '), meta.status, releaseDate(), TYPE[M.direction]].filter(Boolean).join(' · ')),
firstNumber > 1 ? h('div', { class: 'coverage-indicators', role: 'status' },
h('span', { class: 'coverage-badge coverage-partial' }, 'Series coverage partial'),
h('span', { class: 'small' }, `This library starts at chapter ${firstNumber}; earlier numbered chapters are absent.`)) : null,
h('div', { class: 'chips' }, tags.map(tag => h('a', { class: 'chip chip-link', href: libraryUrlForTags([tag]), title: `Show manga tagged ${tag}` }, tag))),
meta.description ? h('p', { class: 'desc' }, meta.description) : null,
sources.length ? h('div', { class: 'sources small' }, h('span', { class: 'muted' }, 'Sources: '),
sources.map(s => (safeHref(s.url) ? h('a', { href: safeHref(s.url), target: '_blank', rel: 'noopener noreferrer' }, s.name) : h('span', {}, s.name + ' ')))) : null,
h('div', { class: 'actions' },
M.continue ? h('a', { class: 'btn primary', href: readUrl(M.id, M.continue.c, M.continue.v) }, M.continue.text) : null,
follow))));
const books = {};
for (const b of M.books) books[b.key] = b;
let storyRhythm = null;
if (M.collections?.length || M.relations?.length) main.append(h('section', { class: 'card' },
h('h2', {}, 'Collections & related manga'),
h('div', { class: 'chips' }, (M.collections || []).map(c => h('a', { class: 'btn', href: 'collection.html?' + qsa({ id: c.id }) }, c.title))),
(M.relations || []).map(r => h('p', {}, h('a', { href: mangaUrl(r.otherId) }, r.otherTitle),
` · ${r.from === M.id ? 'target is ' : 'this manga is '}${r.type.replaceAll('-', ' ')}`,
h('details', {}, h('summary', {}, 'Evidence'), h('p', { class: 'muted' }, r.explanation),
r.evidence.map(e => h('a', { class: 'btn small', href: safeHref(e.url), target: '_blank', rel: 'noopener' }, e.provider || 'Source')))))));
if (M.reading && M.reading.activeMs > 0) {
const lastChapter = M.chapters.find(chapter => chapter.id === M.reading.lastChapterId);
storyRhythm = h('div', { class: 'story-rhythm', role: 'list', 'aria-label': 'Completed chapter reading times' });
const openKey = `reading-story-open.${me.name}`;
const story = h('details', { class: 'reading-story card', open: store.get(openKey) !== '0' },
h('summary', { class: 'reading-story-summary' },
h('span', {}, h('span', { 'aria-hidden': 'true' }, '⏱ '), h('b', {}, 'Your Reading Story')),
h('span', { class: 'muted small' }, readingDuration(M.reading.activeMs) + ' together')),
h('div', { class: 'story-stats' },
storyStat(readingDuration(M.reading.activeMs), 'time together'),
storyStat(M.reading.timedChapters, `chapter${M.reading.timedChapters === 1 ? '' : 's'} timed`),
storyStat(M.reading.typicalChapterMs ? readingDuration(M.reading.typicalChapterMs) : '—', 'typical chapter'),
storyStat(M.reading.rereads, `reread${M.reading.rereads === 1 ? '' : 's'}`)),
lastChapter ? h('p', { class: 'story-last muted' }, 'Last timed ',
h('a', { href: readUrl(M.id, lastChapter.id) }, `${lastChapter.label}${lastChapter.title ? ' · ' + lastChapter.title : ''}`),
` · ${ago(M.reading.updatedAt)}`) : null,
h('div', { class: 'story-rhythm-head' }, h('b', {}, 'Reading rhythm'), h('span', { class: 'muted small' }, 'Completed chapters · active time')),
storyRhythm,
h('div', { class: 'story-actions' },
h('span', { class: 'muted small' }, 'Only you can see these measurements.'),
h('button', { class: 'btn small', onclick: async () => {
if (!confirm(`Forget all reading time recorded for ${M.title}? Reading progress is kept.`)) return;
await api('api/reading/delete', { m: M.id });
location.reload();
} }, 'Forget this manga’s time')));
story.addEventListener('toggle', () => store.set(openKey, story.open ? '1' : '0'));
main.append(story);
drawStoryRhythm();
}
function storyStat(value, label) {
return h('div', { class: 'story-stat' }, h('strong', {}, value), h('span', { class: 'muted small' }, label));
}
function drawStoryRhythm() {
if (!storyRhythm) return;
const chapters = M.chapters.filter(chapter => !selectedBooks.size || selectedBooks.has(chapter.book));
const completed = chapters.map(chapter => chapter.reading?.lastCompleteMs || 0).filter(Boolean);
const scale = Math.max(1, ...completed.map(value => Math.log1p(value)));
storyRhythm.textContent = '';
for (const chapter of chapters) {
const value = chapter.reading?.lastCompleteMs || 0;
const label = `${chapter.label}${chapter.title ? ' · ' + chapter.title : ''}`;
if (!value) {
storyRhythm.append(h('span', { class: 'rhythm-bar empty', role: 'listitem', title: `${label} · Not timed`, 'aria-label': `${label}, not timed` }));
continue;
}
const height = Math.max(12, Math.round(12 + 48 * Math.log1p(value) / scale));
const passes = chapter.reading.completedPasses;
const title = `${label} · ${readingDuration(value)}${passes > 1 ? ` · ${passes} timed reads · ${readingDuration(chapter.reading.totalActiveMs)} total` : ''}`;
storyRhythm.append(h('a', {
class: 'rhythm-bar', role: 'listitem', href: readUrl(M.id, chapter.id), title,
'aria-label': title, style: `height:${height}px`,
}));
}
if (!chapters.length) storyRhythm.append(h('span', { class: 'muted small' }, 'No chapters in the selected volumes.'));
}
const metadata = h('dl', { class: 'meta-grid' });
function addMeta(label, value) {
if (value == null || value === '' || (Array.isArray(value) && !value.length)) return;
metadata.append(h('dt', {}, label), h('dd', {}, value instanceof Node ? value : text(value)));
}
addMeta('Title', meta.title || M.title);
addMeta('Alternative titles', arr(meta.altTitles));
addMeta('Authors', arr(meta.authors));
addMeta('Status', meta.status);
addMeta('First release', releaseDate());
addMeta('Format', meta['x-anilist'] && meta['x-anilist'].format);
addMeta('Reading style', TYPE[M.direction] || M.direction);
addMeta('Volumes', M.books.length);
addMeta('Chapters', M.chapters.length);
addMeta('Library ID', M.id);
addMeta('Tags', h('span', { class: 'meta-tags' }, tags.map(tag => h('a', { class: 'chip chip-link', href: libraryUrlForTags([tag]) }, tag))));
if (sources.length) addMeta('Content sources', h('span', { class: 'meta-links' }, sources.map(source => {
const label = source.name || source.url;
return safeHref(source.url) ? h('a', { href: safeHref(source.url), target: '_blank', rel: 'noopener noreferrer' }, label) : h('span', {}, label);
})));
const anilist = meta['x-anilist'] || {};
if (anilist.url || anilist.id) addMeta('AniList', safeHref(anilist.url)
? h('a', { href: safeHref(anilist.url), target: '_blank', rel: 'noopener noreferrer' }, anilist.id ? `Title ${anilist.id}` : 'Open title')
: anilist.id);
if (anilist.idMal) addMeta('MyAnimeList', h('a', { href: `https://myanimelist.net/manga/${encodeURIComponent(anilist.idMal)}`, target: '_blank', rel: 'noopener noreferrer' }, `Title ${anilist.idMal}`));
if (anilist.coverUrl) addMeta('Metadata cover', safeHref(anilist.coverUrl)
? h('a', { href: safeHref(anilist.coverUrl), target: '_blank', rel: 'noopener noreferrer' }, 'Open source image') : anilist.coverUrl);
const cover = meta['x-cover'] || {};
addMeta('Cover provider', [cover.source, cover.label].filter(Boolean).join(' · '));
if (cover.url) addMeta('Cover source', safeHref(cover.url)
? h('a', { href: safeHref(cover.url), target: '_blank', rel: 'noopener noreferrer' }, cover.url) : cover.url);
const scraper = meta['x-scraper'] || {};
addMeta('Auto update', typeof scraper.autoUpdate === 'boolean' ? (scraper.autoUpdate ? 'Enabled' : 'Disabled') : '');
addMeta('Scraper strategy', scraper.strategy);
addMeta('Source identifiers', scraper.ids);
const handled = new Set(['title', 'altTitles', 'authors', 'status', 'year', 'firstReleaseDate', 'releaseDate', 'startDate', 'tags', 'description', 'direction', 'sources', 'x-anilist', 'x-cover', 'x-scraper']);
for (const [key, value] of Object.entries(meta)) if (!handled.has(key)) addMeta(key, value);
const rawMeta = h('details', { class: 'raw-meta' },
h('summary', {}, 'Raw metadata'),
h('pre', {}, JSON.stringify(meta, null, 2)));
const metadataOpen = !window.matchMedia('(max-width: 640px)').matches;
main.append(h('details', { class: 'metadata card', open: metadataOpen },
h('summary', { class: 'metadata-summary' }, h('h2', {}, 'Metadata')),
metadata, rawMeta));
const volumeButtons = new Map();
const volumeBar = h('div', { class: 'volume-filter', 'aria-label': 'Filter chapters by volume' });
const allVolumes = h('button', { class: 'filter-chip', type: 'button' }, 'All volumes');
allVolumes.onclick = () => { selectedBooks.clear(); drawVolumeFilter(); drawChapters(); drawStoryRhythm(); };
volumeBar.append(allVolumes);
for (const book of M.books) {
const label = book.label + (book.title ? ' · ' + book.title : '');
const button = h('button', { class: 'filter-chip', type: 'button', title: label }, label);
button.onclick = () => { selectedBooks.has(book.key) ? selectedBooks.delete(book.key) : selectedBooks.add(book.key); drawVolumeFilter(); drawChapters(); drawStoryRhythm(); };
volumeButtons.set(book.key, button);
volumeBar.append(button);
}
const list = h('section', { class: 'chapters' });
main.append(list);
const empty = h('p', { class: 'muted chapter-empty', hidden: true }, 'No chapters match these volumes.');
main.append(h('section', { class: 'chapter-section' },
h('div', { class: 'chapter-heading' }, h('h2', {}, 'Chapters'), h('span', { class: 'muted small' }, 'Newest first')),
volumeBar,
h('div', { class: 'chapter-list-wrap' }, list), empty));
function drawVolumeFilter() {
allVolumes.classList.toggle('on', !selectedBooks.size);
allVolumes.setAttribute('aria-pressed', String(!selectedBooks.size));
for (const [key, button] of volumeButtons) {
button.classList.toggle('on', selectedBooks.has(key));
button.setAttribute('aria-pressed', String(selectedBooks.has(key)));
}
}
function drawChapters() {
list.textContent = '';
const chapters = [...M.chapters].reverse().filter(chapter => !selectedBooks.size || selectedBooks.has(chapter.book));
empty.hidden = !!chapters.length;
let lastBook = null;
for (const ch of M.chapters) {
if (ch.book !== lastBook && M.books.length > 1) {
const b = books[ch.book];
list.append(h('div', { class: 'book-sep' }, b.label + (b.title ? ' · ' + b.title : '')));
for (const chapter of chapters) {
if (chapter.book !== lastBook && M.books.length > 1) {
const book = books[chapter.book];
list.append(h('div', { class: 'book-sep' }, book.label + (book.title ? ' · ' + book.title : '')));
}
lastBook = chapter.book;
list.append(chapterRow(chapter));
}
lastBook = ch.book;
list.append(chapterRow(ch));
}
function chapterRow(ch) {
const wrap = h('div', {});
const wrap = h('div', { class: 'chapter-entry' });
const panel = h('div', { class: 'versions', hidden: true });
function draw() {
const p = ch.progress, done = p && p.done;
const state = h('button', { class: 'state' + (done ? ' done' : ''), title: done ? 'Mark as unread' : 'Mark as read' },
done ? '✓ read' : p ? `page ${p.p + 1} / ${p.n}` : `${ch.versions[0].pages} p.`);
done ? '✓ read' : p ? `page ${p.p + 1} / ${p.n}` : 'unread');
state.onclick = async () => {
if (done) { await api('api/unread', { m: M.id, c: ch.id }); ch.progress = null; }
else {
const v = ch.versions[0];
await api('api/progress', { m: M.id, c: ch.id, v: v.id, p: v.pages - 1, n: v.pages, done: true });
ch.progress = { v: v.id, p: v.pages - 1, n: v.pages, done: true };
const version = ch.versions[0];
await api('api/progress', { m: M.id, c: ch.id, v: version.id, p: version.pages - 1, n: version.pages, done: true });
ch.progress = { v: version.id, p: version.pages - 1, n: version.pages, done: true };
}
draw();
};
const verBtn = ch.versions.length > 1
? h('button', { class: 'btn small', onclick: () => { panel.hidden = !panel.hidden; } }, ch.versions.length + ' versions')
: null;
const pages = ch.versions[0] ? ch.versions[0].pages : 0;
const timing = chapterTiming(ch);
wrap.textContent = '';
wrap.append(h('div', { class: 'ch' + (done ? ' read' : '') },
h('a', { class: 'ch-main', href: readUrl(M.id, ch.id) },
h('span', { class: 'ch-label' }, ch.label), h('span', { class: 'ch-title' }, ch.title || ''),
h('a', { class: 'ch-main' + (timing ? ' has-time' : ''), href: readUrl(M.id, ch.id) },
h('span', { class: 'ch-label' }, ch.label),
h('span', { class: 'ch-pages' }, `${pages} p.`),
timing ? h('span', { class: 'ch-time', title: timing.title, 'aria-label': timing.title }, timing.text) : null,
h('span', { class: 'ch-title', title: ch.title || '' }, ch.title || 'Untitled'),
ch.extra ? h('span', { class: 'tag' }, 'extra') : null),
verBtn, state), panel);
drawVersions();
@@ -75,21 +243,39 @@
function drawVersions() {
panel.textContent = '';
panel.append(h('div', { class: 'muted small', style: 'padding-top:6px' }, 'The best rated version opens by default. Vote to change that for everyone.'));
ch.versions.forEach((v, i) => {
const vote = async value => { ch.versions = (await api('api/vote', { m: M.id, c: ch.id, v: v.id, value: v.my === value ? 0 : value })).versions; drawVersions(); };
ch.versions.forEach((version, i) => {
const vote = async value => { ch.versions = (await api('api/vote', { m: M.id, c: ch.id, v: version.id, value: version.my === value ? 0 : value })).versions; drawVersions(); };
panel.append(h('div', { class: 'ver' },
h('div', { class: 'src' },
safeHref(v.sourceUrl) ? h('a', { href: safeHref(v.sourceUrl), target: '_blank', rel: 'noopener noreferrer' }, v.source || v.id) : h('span', {}, v.source || v.id),
safeHref(version.sourceUrl) ? h('a', { href: safeHref(version.sourceUrl), target: '_blank', rel: 'noopener noreferrer' }, version.source || version.id) : h('span', {}, version.source || version.id),
i === 0 ? h('span', { class: 'tag', style: 'margin-left:8px' }, 'default') : null,
h('div', { class: 'muted small' }, `${v.pages} pages · added ${new Date(v.added).toLocaleDateString()}`)),
h('div', { class: 'muted small' }, `${version.pages} pages · added ${new Date(version.added).toLocaleDateString()}`)),
h('span', { class: 'vote' },
h('button', { class: v.my > 0 ? 'on' : '', title: 'Better', onclick: () => vote(1) }, '▲'),
h('b', {}, v.score),
h('button', { class: v.my < 0 ? 'on' : '', title: 'Worse', onclick: () => vote(-1) }, '▼')),
h('a', { class: 'btn small', href: readUrl(M.id, ch.id, v.id) }, 'Read')));
h('button', { class: version.my > 0 ? 'on' : '', title: 'Better', onclick: () => vote(1) }, '▲'),
h('b', {}, version.score),
h('button', { class: version.my < 0 ? 'on' : '', title: 'Worse', onclick: () => vote(-1) }, '▼')),
h('a', { class: 'btn small', href: readUrl(M.id, ch.id, version.id) }, 'Read')));
});
}
draw();
return wrap;
}
function chapterTiming(chapter) {
if (!M.reading) return null;
const value = chapter.reading;
if (!value) return { text: 'Not timed', title: 'No reading time recorded' };
if (value.completedPasses) {
const text = readingDuration(value.lastCompleteMs, true) + (value.completedPasses > 1 ? ` ↻${value.completedPasses}` : '');
return {
text,
title: `Last timed read ${readingDuration(value.lastCompleteMs)}; ${value.completedPasses} timed read${value.completedPasses === 1 ? '' : 's'}; ${readingDuration(value.totalActiveMs)} total`,
};
}
const active = value.currentPassMs || value.totalActiveMs;
return { text: `◷ ${readingDuration(active, true)}`, title: `${readingDuration(active)} active; current pass in progress` };
}
drawVolumeFilter();
drawChapters();
})();
+10
View File
@@ -0,0 +1,10 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<title>Peer catalogs · MangaCircus</title>
<link rel="stylesheet" href="app.css"><link rel="icon" href="icon.svg">
<script src="common.js" defer></script><script src="peer-catalog.js" defer></script>
</head>
<body><main><p class="loading">Loading peer catalogs…</p></main><dialog id="peer-manga-dialog" class="peer-manga-dialog"></dialog></body>
</html>
+185
View File
@@ -0,0 +1,185 @@
'use strict';
(async function () {
const me = await api('api/me');
if (!me.admin) return showError(new Error('Peer catalogs are available to administrators only.'));
topNav(me);
const main = $('main'); main.textContent = '';
const state = await api('api/admin/peering');
const peersResult = await api('api/peering');
let peers = peersResult.peers || [], peerId = new URLSearchParams(location.search).get('peer') || peers[0]?.id || '';
const peer = () => peers.find(item => item.id === peerId);
let mode = location.hash === '#browse' ? 'browse' : 'share', localSearch = '', remoteSearch = '',
localSelected = new Set((peer()?.shares || []).map(item => item.mangaId)), remoteSelected = new Set();
let access = null, remoteItems = [], noteText = '', busy = false;
const shareDirty = () => { const saved = new Set((peer()?.shares || []).map(item => item.mangaId)); return saved.size !== localSelected.size || [...saved].some(id => !localSelected.has(id)); };
const button = (label, click, cls = '', disabled = false) => h('button', { type: 'button', class: `btn ${cls}`, onclick: click, disabled }, label);
function coverUrl(item) { return `api/peering/cover?${qsa({ peer: peerId, manga: item.id })}`; }
function tabs() {
const select = h('select', { 'aria-label': 'Select peer', onchange: async event => { peerId = event.target.value; history.replaceState(null, '', `?peer=${encodeURIComponent(peerId)}${mode === 'browse' ? '#browse' : ''}`); localSelected = new Set((peer()?.shares || []).map(item => item.mangaId)); remoteSelected.clear(); await loadRemote(); draw(); } },
peers.map(item => h('option', { value: item.id, selected: item.id === peerId }, item.alias)));
const bar = h('div', { class: 'peer-catalog-toolbar' }, h('label', { class: 'field peer-catalog-peer' }, h('span', {}, 'Peer'), select),
h('div', { class: 'seg' }, button('Share my library', () => setMode('share', true), mode === 'share' ? 'on' : ''), button('Browse peer catalog', () => setMode('browse', true), mode === 'browse' ? 'on' : '')),
button('Peer settings', () => { location.href = `peers.html#admin/peering/${encodeURIComponent(peerId)}`; }));
return bar;
}
function setMode(value, reload = false) { mode = value; history.replaceState(null, '', `?peer=${encodeURIComponent(peerId)}${value === 'browse' ? '#browse' : ''}`); if (reload) { loadRemote().then(draw); } else draw(); }
async function loadRemote() {
access = null; remoteItems = [];
if (!peer()?.paired) return;
try {
access = await api(`api/peering/access?peer=${encodeURIComponent(peerId)}`);
if (access.grant?.status === 'active' && access.grant.scopes?.includes('catalog')) {
const result = await api(`api/peering/catalog?peer=${encodeURIComponent(peerId)}`); remoteItems = result.items || [];
}
} catch (error) { access = { error: error.message }; }
}
function statusBanner() {
if (!peers.length) return h('section', { class: 'card' }, h('h2', {}, 'Add a peer first'), h('p', { class: 'muted' }, 'Connect another MangaCircus server and set up a catalog grant before sharing or browsing.'), button('Open peering settings', () => { location.href = 'peers.html#admin'; }, 'primary'));
const p = peer();
if (mode === 'browse' && !p?.paired) return h('section', { class: 'card peer-catalog-message' }, h('b', {}, `Setup is incomplete for <${p?.alias || 'peer'}>.`), h('p', { class: 'muted' }, 'Finish the two-way identity exchange in Peering settings.'), button('Open peer settings', () => { location.href = `peers.html#admin/peering/${encodeURIComponent(peerId)}`; }));
if (mode === 'browse') {
if (access?.error) return h('section', { class: 'card error' }, `Could not read ${p.alias}’s current permissions: ${access.error}`);
if (!access) return h('section', { class: 'card muted' }, 'Checking the peer’s current access grant…');
if (access.grant?.status !== 'active' || !access.grant.scopes?.includes('catalog')) return h('section', { class: 'card peer-catalog-message' },
h('b', {}, `<${p.alias}> has not granted catalog browsing.`), h('p', { class: 'muted' }, 'Ask that server’s admin to issue an active Catalog grant. Download permission is not needed to browse.'));
return null;
}
return h('section', { class: 'card peer-catalog-message' }, h('b', {}, `Choose which of your titles <${p.alias}> may see.`),
h('p', { class: 'muted' }, 'Only approved manga titles and their available chapter manifests are shared. User accounts and reading history are never included.'),
!p.paired ? h('p', { class: 'muted small' }, 'You can prepare this list now; sharing begins after identity pairing and an active Catalog grant.') : null);
}
function tile(item, kind) {
const isLocal = kind === 'local', p = peer();
const selected = isLocal ? localSelected.has(item.id) : remoteSelected.has(item.id);
const approved = isLocal && (p?.shares || []).some(share => share.mangaId === item.id);
const quickAllowed = !isLocal && p?.receive && access?.grant?.status === 'active' && access.grant.scopes?.includes('download');
const image = h('div', { class: 'peer-cover-wrap' }, item.coverAvailable && !isLocal ? h('img', { class: 'peer-cover', src: coverUrl(item), alt: '', loading: 'lazy' })
: isLocal && item.cover ? h('img', { class: 'peer-cover', src: item.cover, alt: '', loading: 'lazy' }) : h('div', { class: 'peer-cover-placeholder' }, 'No cover'),
approved ? h('span', { class: 'peer-approval-badge', title: 'Shared with this peer', 'aria-label': 'Shared with this peer' }, '✓') : null,
!isLocal ? h('label', { class: 'peer-select-overlay', title: `Select ${item.title}` }, h('input', { type: 'checkbox', checked: selected, 'aria-label': `Select ${item.title}`, onchange: event => { event.stopPropagation(); selected ? remoteSelected.delete(item.id) : remoteSelected.add(item.id); draw(); } })) : null);
const selectLocal = button(selected ? 'Will share ✓' : 'Will not share', () => {
selected ? localSelected.delete(item.id) : localSelected.add(item.id); draw();
}, selected ? 'peer-selected' : '', false);
const open = button(item.title, () => isLocal ? null : showDetails(item), 'peer-title');
return h('article', { class: 'peer-catalog-card' }, image,
h('div', { class: 'peer-catalog-card-info' }, open,
h('div', { class: 'muted small' }, [item.year, item.status, item.chapters ? `${item.chapters} chapters` : ''].filter(Boolean).join(' · ') || 'Metadata not supplied'),
isLocal ? selectLocal : h('div', { class: 'peer-card-actions' }, quickAllowed ? button('Queue missing', () => queueOne(item), 'small') : null,
button('Details', () => showDetails(item), 'small'))));
}
function libraryView() {
const p = peer();
const search = h('input', { type: 'search', value: localSearch, placeholder: 'Search your library', oninput: event => { localSearch = event.target.value; draw(); } });
const items = (state.library || []).filter(item => item.title.toLowerCase().includes(localSearch.toLowerCase()));
const selectVisible = button('Select visible', () => { for (const item of items) localSelected.add(item.id); draw(); });
const clearVisible = button('Clear visible', () => { for (const item of items) localSelected.delete(item.id); draw(); });
const save = button('Save approvals', async () => {
await runAction('Saving title approvals…', async () => { const result = await api('api/admin/peering', { command: 'share-set', id: peerId, mangaIds: [...localSelected] });
peers = result.peers || peers; localSelected = new Set((peer()?.shares || []).map(item => item.mangaId)); noteText = 'Approved titles updated.'; });
}, 'primary', !p || !shareDirty() || busy);
const toolbar = h('div', { class: 'peer-catalog-actions' }, h('label', { class: 'field peer-search' }, h('span', {}, 'Search'), search), selectVisible, clearVisible, save);
const changedCount = [...new Set([...(p?.shares || []).map(item => item.mangaId), ...localSelected])].filter(id => localSelected.has(id) !== (p?.shares || []).some(item => item.mangaId === id)).length;
const summary = h('p', { class: 'muted small' }, `${p?.shares?.length || 0} title${p?.shares?.length === 1 ? '' : 's'} approved for <${p?.alias || 'peer'}> · ${changedCount} changes`);
return h('section', {}, toolbar, summary, h('div', { class: 'peer-catalog-grid' }, items.map(item => tile(item, 'local'))));
}
function remoteView() {
const p = peer();
const search = h('input', { type: 'search', value: remoteSearch, placeholder: 'Search peer catalog', oninput: event => { remoteSearch = event.target.value; draw(); } });
const eligible = !!(p?.receive && access?.grant?.status === 'active' && access.grant.scopes?.includes('download'));
const items = remoteItems.filter(item => item.title.toLowerCase().includes(remoteSearch.toLowerCase()));
const actions = h('div', { class: 'peer-catalog-actions' }, h('label', { class: 'field peer-search' }, h('span', {}, 'Search'), search),
button('Select visible', () => { for (const item of items) remoteSelected.add(item.id); draw(); }),
button('Clear selection', () => { remoteSelected.clear(); draw(); }),
button(`Queue ${remoteSelected.size} missing`, queueSelected, 'primary', !eligible || !remoteSelected.size || busy));
const banner = h('p', { class: 'muted small' }, eligible ? 'Download permission is active. Queueing is sequential and skips existing chapters.' : 'Browsing is separate from downloads. Queue actions require an active Download grant and local receiving enabled.');
const cards = h('div', { class: 'peer-catalog-grid' }, items.map(item => tile(item, 'remote')));
return h('section', {}, actions, banner, cards);
}
async function queueOne(item, scope = { mode: 'missing' }, label = 'missing chapters') {
await runAction(`Queueing ${label} from ${item.title}…`, async () => {
const result = await api('api/peering/queue', { peer: peerId, manga: item.id, scope, existingPolicy: 'skip-any' });
noteText = `Queued ${label} from ${item.title}. ${result?.added ? `${result.added} added.` : ''}`;
});
}
async function queueSelected() {
if (!remoteSelected.size) return;
const titles = new Map(remoteItems.map(item => [item.id, item.title]));
await runAction('Queueing selected manga sequentially…', async () => {
const ids = [...remoteSelected], failed = []; let queued = 0;
for (let offset = 0; offset < ids.length; offset += 50) {
const result = await api('api/peering/queue-bulk', { peer: peerId, mangas: ids.slice(offset, offset + 50) });
queued += result.queued || 0; failed.push(...(result.results || []).filter(item => item.status !== 'queued'));
}
noteText = `${queued} manga queued; ${failed.length} unavailable or failed.`;
remoteSelected.clear();
if (failed.length) noteText += ` Check results for: ${failed.map(item => titles.get(item.id) || item.id).join(', ')}.`;
});
}
async function runAction(label, action) {
if (busy) return;
busy = true; noteText = label; draw();
try { await action(); }
catch (error) { noteText = error.message; }
finally { busy = false; await reloadCurrent(); draw(); }
}
async function reloadCurrent() { try { peers = (await api('api/peering')).peers || peers; if (mode === 'browse') await loadRemote(); } catch (error) { noteText = error.message; } }
function showDetails(item) {
const dialog = $('#peer-manga-dialog'); dialog.textContent = 'Loading manga details…';
dialog.showModal();
api(`api/peering/manga?peer=${encodeURIComponent(peerId)}&manga=${encodeURIComponent(item.id)}`).then(detail => {
dialog.textContent = '';
dialog.append(h('button', { class: 'btn peer-dialog-close', type: 'button', onclick: () => dialog.close(), 'aria-label': 'Close details' }, '×'),
h('div', { class: 'peer-dialog-layout' }, item.coverAvailable ? h('img', { class: 'peer-dialog-cover', src: coverUrl(item), alt: '' }) : null,
h('div', {}, h('h2', {}, detail.title || item.title), h('p', { class: 'muted' }, [detail.year, detail.status, detail.direction].filter(Boolean).join(' · ')),
detail.description ? h('p', {}, detail.description) : null,
[...(detail.tags || []), ...(detail.genres || [])].length ? h('p', { class: 'peer-tags' }, [...new Set([...(detail.tags || []), ...(detail.genres || [])])].map(tag => h('span', { class: 'tag' }, tag))) : null,
h('h3', {}, `${detail.chapters.length} complete chapters`),
(() => {
const scope = h('select', { 'aria-label': 'Queue scope' },
h('option', { value: 'missing' }, 'All missing chapters'), h('option', { value: 'latest' }, 'Newest chapter only'),
h('option', { value: 'last-5' }, 'Last five chapters'), h('option', { value: 'first-10' }, 'First ten chapters'),
h('option', { value: 'after' }, 'After chapter number…'),
[...new Set(detail.chapters.map(chapter => chapter.volume).filter(value => value !== null && value !== undefined && value !== ''))]
.map(volume => h('option', { value: `volume:${volume}` }, `Volume ${volume}`)));
const after = h('input', { type: 'number', min: '0', step: '0.1', value: '0', 'aria-label': 'Chapter threshold' });
const afterField = h('label', { class: 'field peer-scope-number', hidden: true }, h('span', {}, 'Chapter threshold'), after);
scope.onchange = () => { afterField.hidden = scope.value !== 'after'; };
const checks = detail.chapters.map(chapter => h('label', { class: 'peer-chapter-option' },
h('input', { type: 'checkbox', value: chapter.id, 'aria-label': `Select chapter ${chapter.number}` }),
h('span', {}, `Vol. ${chapter.volume} · Ch. ${chapter.number}${chapter.title ? ` · ${chapter.title}` : ''} · ${chapter.pageCount} pages`)));
const canQueue = !!(peer()?.receive && access?.grant?.status === 'active' && access.grant.scopes?.includes('download'));
const applyRule = () => {
const value = scope.value;
const selection = value === 'latest' ? { mode: 'latest' } : value === 'last-5' ? { mode: 'last', count: 5 }
: value === 'first-10' ? { mode: 'first', count: 10 } : value === 'after' ? { mode: 'after', chapter: after.value }
: value.startsWith('volume:') ? { mode: 'volume', volume: value.slice(7) } : { mode: 'missing' };
dialog.close(); queueOne(item, selection, scope.selectedOptions[0]?.textContent.toLowerCase() || 'selected chapters');
};
const selectedChapters = () => [...dialog.querySelectorAll('.peer-chapter-option input:checked')].map(input => input.value);
const exactButton = button('Queue checked chapters', () => { const ids = selectedChapters(); if (!ids.length) return; dialog.close(); queueOne(item, { mode: 'ids', ids }, `${ids.length} selected chapters`); }, '', true);
const refreshExactButton = () => { exactButton.disabled = !selectedChapters().length; };
for (const option of checks) option.querySelector('input').addEventListener('change', refreshExactButton);
return h('div', { class: 'peer-detail-queue' },
h('label', { class: 'field' }, h('span', {}, 'Queue scope'), scope),
afterField,
canQueue ? h('div', { class: 'peer-detail-actions' }, button('Queue with selected rule', applyRule, 'primary'), exactButton)
: h('p', { class: 'muted small' }, 'Queueing requires active Download permission and local receiving enabled.'),
h('details', { class: 'peer-chapter-picker' }, h('summary', {}, 'Choose exact chapters'), h('div', { class: 'peer-chapter-list' }, checks)));
})())));
}).catch(error => { dialog.textContent = ''; dialog.append(h('p', { class: 'error' }, error.message), button('Close', () => dialog.close())); });
}
function draw() {
main.textContent = '';
main.append(h('div', { class: 'peer-catalog-heading' }, h('div', {}, h('p', { class: 'muted small' }, 'Admin / Peering'), h('h1', {}, 'Peer catalogs'), h('p', { class: 'muted' }, 'Approve the titles you serve, or browse catalogs granted to you. These actions do not change grants.')),
button('Peering settings', () => { location.href = 'peers.html#admin'; })));
main.append(tabs());
if (noteText) main.append(h('p', { role: 'status', class: 'muted small peer-catalog-status' }, noteText));
const banner = statusBanner();
if (banner) { main.append(banner); if (mode === 'browse' || !peers.length) return; }
if (mode === 'share') main.append(libraryView()); else main.append(remoteView());
}
const libraryResponse = await api('api/admin/peering');
state.library = libraryResponse.library || [];
async function init() { await loadRemote(); draw(); }
await init();
})().catch(showError);
+10
View File
@@ -0,0 +1,10 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<title>Peering settings · MangaCircus</title>
<link rel="stylesheet" href="app.css"><link rel="icon" href="icon.svg">
<script src="common.js" defer></script><script src="settings-nav.js" defer></script><script src="peers.js" defer></script>
</head>
<body><main><p class="loading">Loading…</p></main></body>
</html>
+299
View File
@@ -0,0 +1,299 @@
'use strict';
(async function () {
let me;
try { me = await api('api/me'); } catch (error) { return showError(error); }
if (!me.admin) return showError(new Error('Peering settings are available to administrators only.'));
topNav(me);
const shell = settingsShell(me, 'admin/peering'), main = shell.content;
let state = null, peers = [], access = new Map(), route = '', busy = false;
let pendingInvitation = '', pendingGrant = null, messageText = '', messageError = false;
const note = (text, error = false) => { messageText = text; messageError = error; draw(); };
const field = (label, control, hint) => h('label', { class: 'field' }, h('span', {}, label), control,
hint ? h('small', { class: 'muted' }, hint) : null);
const button = (text, click, cls = '', disabled = false) => h('button', { class: `btn ${cls}`, type: 'button', onclick: click, disabled }, text);
const peerTag = (name, local = false) => h('span', { class: `peer-name${local ? ' local' : ''}` }, `<${name || 'Unnamed'}>${local ? ' · this server' : ''}`);
const date = value => value ? new Date(value).toLocaleString() : '—';
async function command(payload, success = 'Saved') {
if (busy) return null;
busy = true; messageText = 'Working…'; messageError = false; draw();
try {
const result = await api('api/admin/peering', payload);
if (result.identity || result.peers) state = { ...state, ...result };
messageText = success; messageError = false;
await reload(false); return result;
} catch (error) { messageText = error.message; messageError = true; draw(); return null; }
finally { busy = false; draw(); }
}
async function reload(redraw = true) {
try {
[state, peers] = await Promise.all([api('api/admin/peering'), api('api/peering').then(value => value.peers)]);
await Promise.all(peers.filter(peer => peer.paired).map(async peer => {
try { access.set(peer.id, await api(`api/peering/access?peer=${encodeURIComponent(peer.id)}`)); }
catch (error) { access.set(peer.id, { error: error.message }); }
}));
if (redraw) draw();
} catch (error) { messageText = error.message; messageError = true; if (redraw) draw(); }
}
async function refreshAccess(id) {
access.delete(id);
try { access.set(id, await api(`api/peering/access?peer=${encodeURIComponent(id)}`)); }
catch (error) { access.set(id, { error: error.message }); }
draw();
}
function setRoute(value) { location.hash = value; }
function statusLabel(peer) {
if (!peer.paired) return 'Identity not verified here';
const remote = access.get(peer.id);
if (remote?.paired) return 'Verified on both servers';
if (remote?.error) return 'Waiting for the other server';
return 'Verified here · checking other side';
}
function serverIdentityPanel() {
const identity = state?.identity, currentName = state?.name || identity?.name || '';
const panel = h('section', { class: 'card peer-card' }, h('h2', {}, 'This server'),
h('p', { class: 'muted' }, 'Choose the short name other admins will see. It is a display name; changing it does not replace this server’s certificate.'),
peerTag(currentName || 'Set a name', true));
const name = h('input', { value: currentName, maxlength: '25', pattern: '[A-Za-z0-9]{1,25}', autocomplete: 'off', placeholder: 'MangaCircus' });
panel.append(field('Server name · letters and numbers, up to 25 characters', name),
button('Save server name', () => command({ command: 'set-name', name: name.value }, 'Server name saved'), '', busy));
if (!identity) {
panel.append(h('h3', {}, 'Create this server’s certificate'),
h('p', { class: 'muted small' }, 'The certificate is this server’s durable cryptographic identity. It currently lasts one year. It is separate from the 48-hour invitation code and is not recreated when you edit the display name.'),
button('Create certificate', () => command({ command: 'generate', name: currentName }, 'Certificate created'), 'primary', !currentName || busy));
return panel;
}
panel.append(h('h3', {}, 'Server certificate'),
h('p', { class: 'muted small' }, `Valid until ${date(identity.expiresAt)}. This is the certificate lifetime; an invitation expires after 48 hours, and a grant lasts up to 90 days.`),
h('details', {}, h('summary', {}, 'Certificate fingerprint'), h('code', { class: 'peer-fingerprint' }, identity.fingerprint)));
if (state.invitations?.length) panel.append(h('p', { class: 'muted small' }, `Unexpired invitations: ${state.invitations.map(item => date(item.expiresAt)).join(' · ')}`));
return panel;
}
function peerTable() {
const panel = h('section', { class: 'card peer-card' },
h('div', { class: 'peer-section-head' }, h('div', {}, h('h2', {}, 'Known peers'), h('p', { class: 'muted small' }, 'Each row shows identity setup and permissions in both directions.')),
button('Add a peer', () => setRoute('#admin/peering/add'), 'primary')));
if (!peers.length) return panel.append(h('p', { class: 'muted' }, 'No peers yet. Add a peer to start the two-way identity exchange.')), panel;
const table = h('div', { class: 'peer-table', role: 'table', 'aria-label': 'Known peers' });
table.append(h('div', { class: 'peer-table-row peer-table-head', role: 'row' },
...['Peer', 'Connection', 'I grant them', 'They grant me', ''].map(label => h('span', { role: 'columnheader' }, label))));
for (const peer of peers) {
const remote = access.get(peer.id), mine = peer.grantStatus === 'active' ? peer.grantScopes : [];
const theirs = remote?.grant?.status === 'active' ? remote.grant.scopes : [];
const summary = scopes => scopes.includes('download') ? 'Browse + download' : scopes.includes('catalog') ? 'Browse catalog' : '—';
table.append(h('button', { class: 'peer-table-row peer-row-button', type: 'button', role: 'row', onclick: () => setRoute(`#admin/peering/${encodeURIComponent(peer.id)}`) },
h('span', { role: 'cell' }, peerTag(peer.alias)),
h('span', { role: 'cell', class: peer.paired && remote?.paired ? 'peer-status-ok' : 'peer-status-pending' }, peer.paired && remote?.paired ? '✓ Fully paired' : '○ Setup incomplete'),
h('span', { role: 'cell' }, peer.grantStatus === 'paused' ? 'Paused' : summary(mine)),
h('span', { role: 'cell' }, remote?.error ? 'Check status' : summary(theirs)),
h('span', { role: 'cell', class: 'peer-row-action' }, 'Manage →')));
}
panel.append(table); return panel;
}
function invitationWizard() {
const identity = state?.identity;
const panel = h('section', { class: 'card peer-card' }, h('div', { class: 'peer-section-head' },
h('div', {}, h('h2', {}, 'Add a peer'), h('p', { class: 'muted small' }, 'Both admins exchange one invitation, then verify the other server. Pairing sets up identity in both directions; sharing manga remains a separate choice.')),
button('Back to peers', () => setRoute('#admin/peering'))));
if (!identity) return panel.append(h('p', { class: 'error' }, 'Set this server name and create its certificate before inviting a peer.'));
const url = h('input', { type: 'url', placeholder: 'https://your-tailnet-name:8093', value: pendingUrl, class: 'grow' });
const outgoing = h('textarea', { rows: '7', readonly: true, placeholder: 'Create your invitation first. It will appear here with its 48-hour setup expiry.', 'aria-label': 'Invitation to send' }, pendingInvitation);
const incoming = h('textarea', { rows: '7', placeholder: 'Paste the invitation JSON from the other admin.', 'aria-label': 'Invitation received' });
const send = h('section', { class: 'peer-step' }, h('h3', {}, '1 · Create and send your invitation'),
h('p', { class: 'muted small' }, `Tell the other admin to use this gateway URL and send their invitation back. Your server: `, peerTag(state.name, true)),
field('This server’s reachable peer gateway URL', url, 'Use its HTTPS gateway address on port 8093. This is not the reader URL.'),
button('Create 48-hour invitation', async () => {
const parsed = (() => { try { return new URL(url.value); } catch { return null; } })();
if (!parsed || parsed.protocol !== 'https:' || parsed.username || parsed.password || parsed.search || parsed.hash || parsed.pathname !== '/') {
url.setCustomValidity('Enter an HTTPS peer gateway origin, without a path.'); url.reportValidity(); return;
}
const result = await command({ command: 'invite', url: url.value }, 'Invitation created. Send it privately to the other admin.');
if (result?.invitation) { pendingInvitation = JSON.stringify(result.invitation, null, 2); pendingUrl = url.value; draw(); }
}, '', busy), field('Invitation to send', outgoing),
button('Copy invitation', async () => { try { await navigator.clipboard.writeText(outgoing.value); note('Invitation copied. It contains a one-use setup code; share it privately.'); } catch { note('Copy failed; select and copy the invitation text.', true); } }, '', !pendingInvitation));
const receive = h('section', { class: 'peer-step' }, h('h3', {}, '2 · Receive and import their invitation'),
h('p', { class: 'muted small' }, 'This tells your server which certificate and gateway belong to the other server. Compare the displayed fingerprint with the other admin through a separate channel.'),
field('Invitation received', incoming), button('Import invitation', async () => {
let invitation; try { invitation = JSON.parse(incoming.value); } catch { note('That is not valid invitation JSON.', true); return; }
const result = await command({ command: 'import', invitation }, 'Invitation imported. Next, verify the other server’s identity.');
if (result?.peers) {
const match = result.peers.find(p => p.fingerprint === invitation.fingerprint);
if (match) setRoute(`#admin/peering/${encodeURIComponent(match.id)}`);
}
}, 'primary', busy));
panel.append(h('div', { class: 'peer-wizard-grid' }, send, receive),
h('p', { class: 'muted small' }, 'An invitation expires after 48 hours. It does not expire an established peering connection or the server certificate.'));
return panel;
}
let pendingUrl = '';
function connectionPanel(peer) {
const remote = access.get(peer.id);
const identityStatus = !peer.paired
? peerState('pending', 'Identity not verified yet', 'Verify the other server after importing its invitation.')
: remote?.error
? peerState('error', 'Other server status unavailable', `This server is verified; refresh status to check the other side. ${remote.error}`)
: remote?.paired
? peerState('ok', 'Peering confirmed on both servers', 'Both servers have verified each other’s certificate.')
: peerState('pending', 'Identity verified on this server', 'Waiting for the other server to verify this one.');
const panel = h('section', { class: 'peer-step' }, h('h3', {}, 'Connection and identity'),
identityStatus,
h('p', { class: 'muted small' }, `Our fingerprint: ${state.identity?.fingerprint || 'certificate not created'}`),
h('p', { class: 'muted small' }, `Their fingerprint: ${peer.fingerprint}`),
peer.paired ? null : button('Verify the other server', () => command({ command: 'probe', id: peer.id }, 'Identity verified here. Ask the other admin to verify back.'), 'primary', busy),
button('Refresh other-side status', () => refreshAccess(peer.id), '', busy));
if (!peer.paired || !remote?.paired) {
const url = h('input', { type: 'url', placeholder: 'https://this-server:8093', value: pendingUrl, class: 'grow' });
const invitation = h('textarea', { rows: '6', readonly: true, placeholder: 'Create your invitation to send to the other admin.', 'aria-label': 'Invitation to send' }, pendingInvitation);
const received = h('textarea', { rows: '5', placeholder: 'Paste the other server’s invitation JSON.', 'aria-label': 'Invitation received' });
panel.append(h('details', { class: 'peer-setup-again', open: !peer.paired }, h('summary', {}, 'Continue the two-way invitation exchange'),
h('div', { class: 'peer-wizard-grid' },
h('section', { class: 'peer-step' }, h('h4', {}, 'Send from this server'), field('Gateway URL', url),
button('Create invitation', async () => {
const parsed = (() => { try { return new URL(url.value); } catch { return null; } })();
if (!parsed || parsed.protocol !== 'https:' || parsed.pathname !== '/') return note('Enter this server’s HTTPS gateway origin.', true);
const result = await command({ command: 'invite', url: url.value });
if (result?.invitation) { pendingInvitation = JSON.stringify(result.invitation, null, 2); pendingUrl = url.value; draw(); }
}), field('JSON to send', invitation), button('Copy invitation', async () => { try { await navigator.clipboard.writeText(invitation.value); note('Invitation copied privately.'); } catch { note('Copy failed; select and copy it.', true); } }, '', !pendingInvitation)),
h('section', { class: 'peer-step' }, h('h4', {}, 'Receive from this server'), field('Their invitation', received),
button('Import invitation', async () => { let value; try { value = JSON.parse(received.value); } catch { return note('Invalid invitation JSON.', true); }
const result = await command({ command: 'import', invitation: value }, 'Invitation imported. Verify this server next.');
if (result?.peers) setRoute(`#admin/peering/${encodeURIComponent(peer.id)}`);
})))));
}
return panel;
}
function grantPanel(peer) {
const panel = h('section', { class: 'peer-step' }, h('h3', {}, 'Access between servers'),
h('div', { class: 'peer-wizard-grid' }));
const grid = panel.lastChild;
const catalog = h('input', { type: 'checkbox', checked: (peer.grantScopes || []).includes('catalog') || (peer.grantScopes || []).includes('download') });
const download = h('input', { type: 'checkbox', checked: (peer.grantScopes || []).includes('download') });
const grantHint = h('p', { class: 'muted small', role: 'status' });
let grantAction = null;
const updateGrantHint = () => {
grantHint.textContent = !peer.paired
? 'Granting is locked until identity pairing is complete. Finish the invitation exchange and verify this server above.'
: (!catalog.checked && !download.checked ? 'Choose a permission above to enable Issue grant.' : '');
grantHint.hidden = !grantHint.textContent;
if (grantAction) grantAction.disabled = busy || !peer.paired || (!catalog.checked && !download.checked);
};
catalog.onchange = () => { if (!catalog.checked) download.checked = false; updateGrantHint(); };
download.onchange = () => { if (download.checked) catalog.checked = true; updateGrantHint(); };
updateGrantHint();
const token = h('textarea', { rows: '3', readonly: true, placeholder: 'Issue a grant to create a one-time token.', 'aria-label': 'Grant token' }, pendingGrant?.id === peer.id ? pendingGrant.token : '');
grantAction = peer.grantStatus === 'none' || peer.grantStatus === 'expired'
? button('Issue grant', async () => { const result = await command({ command: 'grant', id: peer.id,
scopes: [...(catalog.checked ? ['catalog'] : []), ...(download.checked ? ['download'] : [])] }, 'Grant issued; send the token privately.');
if (result?.grant) { pendingGrant = { id: peer.id, token: result.grant.token }; draw(); } }, 'primary', !peer.paired || (!catalog.checked && !download.checked) || busy)
: button('Save grant permissions', () => command({ command: 'set', id: peer.id, patch: { grantScopes: [...(catalog.checked ? ['catalog'] : []), ...(download.checked ? ['download'] : [])] } }, 'Grant permissions updated'), '', !peer.paired || (!catalog.checked && !download.checked) || busy);
updateGrantHint();
const outbound = h('section', { class: 'peer-step' }, h('h4', {}, 'We grant access to ', peerTag(peer.alias)),
localGrantStatus(peer),
h('label', { class: 'check' }, catalog, h('span', {}, 'Browse approved catalog')),
h('label', { class: 'check' }, download, h('span', {}, 'Download chapters (includes catalog access)')),
grantHint,
grantAction,
peer.grantStatus === 'active' ? button('Pause access', () => command({ command: 'set', id: peer.id, patch: { servingPaused: true } }, 'Access paused; grant retained'))
: peer.grantStatus === 'paused' ? button('Resume access', () => command({ command: 'set', id: peer.id, patch: { servingPaused: false } }, 'Access resumed')) : null,
field('Grant token · copy once and send privately', token),
button('Copy grant token', async () => { try { await navigator.clipboard.writeText(token.value); note('Grant token copied privately.'); } catch { note('Copy failed; select and copy it.', true); } }, '', !token.value),
peer.grantStatus !== 'none' ? button('Revoke grant', async () => {
if (!confirm(`Revoke access for ${peer.alias}? Their saved token will stop working. You can issue a new grant later.`)) return;
pendingGrant = null; await command({ command: 'revoke-grant', id: peer.id }, 'Grant revoked.');
}, 'danger small') : null);
const inbound = h('section', { class: 'peer-step' }, h('h4', {}, peerTag(peer.alias), ' grants access to us'),
remoteGrantStatus(access.get(peer.id)),
h('label', { class: 'check' }, h('input', { type: 'checkbox', checked: !!peer.receive, disabled: !peer.hasGrant,
onchange: event => command({ command: 'set', id: peer.id, patch: { receive: event.target.checked } }, 'Local download permission saved') }),
h('span', {}, 'Allow this server to download from them')),
field('Grant token received', h('input', { type: 'password', autocomplete: 'off', placeholder: 'Paste their grant token', 'aria-label': 'Grant token received' })),
button('Import their grant', async () => {
const input = panel.querySelector('[aria-label="Grant token received"]');
if (await command({ command: 'accept-grant', id: peer.id, token: input.value }, 'Their grant imported')) {
input.value = ''; access.delete(peer.id); await refreshAccess(peer.id);
}
}, 'primary', busy),
button('Refresh their grant status', () => refreshAccess(peer.id), '', busy),
h('p', { class: 'muted small' }, `Allow ordinary readers to request queue items: ${peer.readerQueue ? 'yes' : 'no'}.`),
h('label', { class: 'check' }, h('input', { type: 'checkbox', checked: !!peer.readerQueue,
onchange: event => command({ command: 'set', id: peer.id, patch: { readerQueue: event.target.checked } }, 'Reader queue permission saved') }),
h('span', {}, 'Let ordinary readers queue downloads')));
grid.append(outbound, inbound); return panel;
}
function peerState(tone, title, detail) {
return h('div', { class: `peer-state peer-state-${tone}`, role: 'status', 'aria-live': 'polite' },
h('strong', {}, title), detail ? h('span', {}, detail) : null);
}
function localGrantStatus(peer) {
const scopes = peer.grantScopes || [];
const access = scopes.includes('download') ? 'Browse and download chapters' : scopes.includes('catalog') ? 'Browse approved catalog' : '';
if (peer.grantStatus === 'active') return peerState('ok', 'Grant active', `${access} · expires ${date(peer.grantExpiresAt)}.`);
if (peer.grantStatus === 'paused') return peerState('pending', 'Grant paused', 'The grant is retained. Resume access to make it usable again.');
if (peer.grantStatus === 'expired') return peerState('warn', 'Grant expired', 'Issue a new grant to restore access.');
return peerState('neutral', 'No grant issued', 'Choose the access level below to create a grant.');
}
function remoteGrantStatus(value) {
if (!value) return peerState('pending', 'Status not checked', 'Refresh to ask the other server for its current access status.');
if (value.error) return peerState('error', 'Could not confirm access', value.error);
const grant = value.grant || {};
const scopes = grant.scopes || [];
if (!value.paired) return peerState('pending', 'Identity not verified by peer', 'Ask the other admin to verify this server.');
if (grant.status === 'active') {
const role = scopes.includes('download') ? 'Browse and download chapters' : 'Browse approved catalog';
return peerState('ok', 'Active grant from peer', `${role} · expires ${date(grant.expiresAt)}.`);
}
if (grant.status === 'paused') return peerState('pending', 'Peer grant paused', 'The other admin can resume it without exchanging a new token.');
if (grant.status === 'expired') return peerState('warn', 'Peer grant expired', 'Ask the other admin to issue a new grant.');
return peerState('neutral', 'No active grant from peer', 'The other server has not granted catalog or download access.');
}
function limitPanel(peer) {
const limits = peer.limits || {}, download = peer.download || {};
const concurrent = h('input', { type: 'number', min: '1', max: '8', value: limits.concurrent || 2 });
const upload = h('input', { type: 'number', min: '1', max: '100', value: Math.round((limits.bytesPerSecond || 2097152) / 1048576) });
const daily = h('input', { type: 'number', min: '1', value: Math.round((limits.bytesPerDay || 10737418240) / 1073741824) });
const inFlight = h('input', { type: 'number', min: '1', max: '4', value: download.concurrent || 2 });
const inbound = h('input', { type: 'number', min: '1', max: '100', value: Math.round((download.bytesPerSecond || 2097152) / 1048576) });
return h('details', { class: 'card peer-card peer-details' }, h('summary', {}, 'Transfer limits'),
h('p', { class: 'muted small' }, `These are local caps for ${peer.alias}; the other server’s upload cap still applies.`),
h('div', { class: 'peer-limits-grid' }, field('Upload streams to them', concurrent), field('Upload MiB/s', upload), field('Upload GiB/day', daily),
field('Download pages in flight', inFlight), field('Download MiB/s', inbound)),
button('Save limits', () => command({ command: 'set', id: peer.id, patch: {
limits: { concurrent: +concurrent.value, bytesPerSecond: Math.round(+upload.value * 1048576), bytesPerDay: Math.round(+daily.value * 1073741824) },
download: { concurrent: +inFlight.value, bytesPerSecond: Math.round(+inbound.value * 1048576) },
} }, 'Transfer limits saved')));
}
function managePeer(peer) {
const remote = access.get(peer.id), name = h('input', { value: peer.alias, maxlength: '25', pattern: '[A-Za-z0-9]{1,25}' });
return h('div', { class: 'peer-workspace' },
h('div', { class: 'peer-section-head' }, h('div', {}, h('p', { class: 'muted small' }, 'Settings / Peering / Peer'), h('h2', {}, peerTag(peer.alias))),
button('Back to peers', () => setRoute('#admin/peering'))),
h('section', { class: 'card peer-card' }, h('h3', {}, 'Peer name and address'),
h('div', { class: 'peer-manage-name' }, field('Local name for this peer', name), button('Save name', () => command({ command: 'set', id: peer.id, patch: { alias: name.value } }, 'Peer name saved'))),
h('p', { class: 'muted small' }, `${peer.url} · certificate ${peer.fingerprint}`)),
connectionPanel(peer), grantPanel(peer), limitPanel(peer),
h('section', { class: 'peer-danger-zone' }, button('Remove peer', async () => {
if (!confirm(`Remove ${peer.alias}, delete its grants and pause future transfers? Already downloaded chapters remain.`)) return;
await command({ command: 'remove', id: peer.id }, 'Peer removed'); setRoute('#admin/peering');
}, 'danger small')));
}
function draw() {
const open = new Set([...main.querySelectorAll('details[data-peer-section][open]')].map(item => item.dataset.peerSection));
main.textContent = '';
const parts = location.hash.split('/');
route = parts[1] === 'peering' ? decodeURIComponent(parts[2] || '') : '';
const status = h('p', { class: messageError ? 'error' : 'muted small', role: 'status', 'aria-live': 'polite' }, messageText);
const tabs = h('div', { class: 'peer-page-tabs' }, h('a', { class: 'btn', href: 'peers.html#admin' }, 'Peering settings'),
h('a', { class: 'btn', href: 'peer-catalog.html' }, 'Peer catalogs'));
main.append(tabs, status);
if (route === 'add') main.append(invitationWizard());
else if (route) {
const peer = peers.find(item => item.id === decodeURIComponent(route));
if (!peer) main.append(button('Back to peers', () => setRoute('#admin/peering')), h('p', { class: 'muted' }, 'Peer not found.'));
else main.append(managePeer(peer));
} else main.append(serverIdentityPanel(), peerTable());
for (const details of main.querySelectorAll('details[data-peer-section]')) details.open = open.has(details.dataset.peerSection);
}
window.addEventListener('hashchange', () => { draw(); const peer = peers.find(item => item.id === decodeURIComponent(route)); if (peer) refreshAccess(peer.id); });
await reload(); draw();
const first = peers.find(peer => peer.id === decodeURIComponent(location.hash.split('/')[2] || ''));
if (first) refreshAccess(first.id);
})().catch(showError);
+10 -1
View File
@@ -12,6 +12,8 @@
<link rel="manifest" href="manifest.webmanifest">
<meta name="theme-color" content="#0b0b0c">
<script src="common.js" defer></script>
<script src="reading-clock-core.js" defer></script>
<script src="reading-clock.js" defer></script>
<script src="reader.js" defer></script>
</head>
<body class="reader">
@@ -36,12 +38,19 @@
<button data-mode="page" title="One page at a time">Page</button>
<button data-mode="book" title="Two pages side by side, like an open book">Book</button>
</div>
<button id="page-tone" class="btn page-tone" type="button" title="Change page colors; controls and theme stay the same" aria-label="Page tone: Original. Change page colors">Tone: Original</button>
<button id="shift" class="btn" title="Page pairs look wrong? Shift them by one page">⇄<span class="wide-only"> Shift</span></button>
<button id="zoom" class="btn" title="Page zoom (Ctrl + scroll or pinch). Click to fit the screen again.">100%</button>
<div class="seg zoom-controls" role="group" aria-label="Page zoom">
<button id="zoomout" title="Zoom out 5%" aria-label="Zoom out 5%">−</button>
<button id="zoom" title="Current page zoom. Click to fit the screen again.">100%</button>
<button id="zoomin" title="Zoom in 5%" aria-label="Zoom in 5%">+</button>
</div>
<button id="readclock" class="btn clock-chip" hidden aria-label="Active Reading Clock"><span class="clock-icon">⏸</span> <span class="clock-time">00:00</span><span class="clock-label"></span></button>
<select id="ver" title="Version of this chapter" aria-label="Version"></select>
<a id="nextch" class="btn" title="Next chapter"><span class="wide-only">Next </span>›</a>
</div>
</div>
<div id="clockpanel" class="clock-panel" hidden></div>
<div id="toast"></div>
</body>
</html>
+182 -170
View File
@@ -12,11 +12,78 @@
const P = new URLSearchParams(location.search);
const stage = $('#stage'), canvas = $('#canvas'), slider = $('#slider'), ver = $('#ver');
let me, D;
try { [me, D] = await Promise.all([api('api/me'), api('api/chapter?' + qsa({ m: P.get('m'), c: P.get('c'), v: P.get('v') }))]); }
try {
me = await api('api/me');
const response = await fetch('api/chapter?' + qsa({ m: P.get('m'), c: P.get('c'), v: P.get('v') }));
if (response.status === 423) {
const access = await api('api/reading-access?' + qsa({ m: P.get('m') }));
canvas.textContent = '';
const card = h('section', { class: 'reading-gate' },
h('h1', {}, 'Reading is closed right now'),
h('p', {}, `The reading hours are set by this server's deployment (${access.timezone}). The library and your progress remain available.`),
h('p', {}, `Answer five questions drawn from manga you have already read to unlock ${access.unlockMinutes} minutes of reading. There is no answer timer or retry limit.`));
const message = h('p', { class: 'reading-gate-message', role: 'status', hidden: true });
const setMessage = (text, tone = 'info') => {
message.textContent = text;
message.dataset.tone = tone;
message.hidden = !text;
};
const question = h('div', { class: 'reading-gate-question' });
const start = h('button', { class: 'btn', type: 'button' }, 'Start five-question challenge');
const back = h('a', { class: 'btn', href: mangaUrl(P.get('m')) }, 'Back to manga');
card.append(message, question, h('div', { class: 'reading-gate-actions' }, start, back));
canvas.append(card);
let challenge = null;
const showQuestion = data => {
challenge = data.challenge;
question.textContent = '';
const input = h('input', { type: 'text', autocomplete: 'off', 'aria-label': 'Your answer', maxlength: '120', required: true });
const form = h('form', {}, h('p', {}, `Question ${data.number} of ${data.total}`),
h('p', { class: 'muted' }, data.mangaTitle), h('h2', {}, data.prompt), input, h('button', { class: 'btn', type: 'submit' }, 'Check answer'));
form.addEventListener('submit', async event => {
event.preventDefault();
try {
const result = await api('api/reading-quiz/answer', { m: P.get('m'), challenge, answer: input.value });
if (!result.correct) { setMessage('Not quite. Try another spelling or answer—there is no attempt limit.', 'retry'); input.select(); return; }
if (result.complete) { setMessage('Unlocked. Opening your chapter…', 'success'); location.reload(); return; }
setMessage('Correct.', 'success');
showQuestion(result);
} catch (error) {
setMessage(error.message, 'error');
if (/challenge expired/i.test(error.message)) { challenge = null; question.textContent = ''; start.hidden = false; }
}
});
question.append(form); input.focus();
};
start.addEventListener('click', async () => {
try { showQuestion(await api('api/reading-quiz/start', { m: P.get('m') })); start.hidden = true; setMessage(''); }
catch (error) { setMessage(error.message, 'error'); }
});
if (!access.quizAvailable) { start.hidden = true; setMessage('Fewer than five questions match your completed chapters. Please return during reading hours.', 'info'); }
window.addEventListener('pagehide', () => {
if (challenge) navigator.sendBeacon('api/reading-quiz/abandon', new Blob([JSON.stringify({ challenge })], { type: 'application/json' }));
});
window.addEventListener('pageshow', event => { if (event.persisted) location.reload(); });
setInterval(async () => {
try { if ((await api('api/reading-access?' + qsa({ m: P.get('m') }))).allowed) location.reload(); } catch { /* keep the gate visible */ }
}, 15000);
return;
}
const payload = await response.json();
if (!response.ok) throw new Error(payload.error || response.statusText);
D = payload;
}
catch (e) { canvas.textContent = ''; canvas.append(h('p', { class: 'loading' }, e.message + ' · ', h('a', { href: './' }, 'Library'))); return; }
// An already-open chapter also closes when a window ends or an unlock expires.
const checkReadingAccess = async () => {
try { if (!(await api('api/reading-access?' + qsa({ m: P.get('m') }))).allowed) location.reload(); } catch { /* transient check failure */ }
};
setInterval(checkReadingAccess, 15000);
document.addEventListener('visibilitychange', () => { if (!document.hidden) checkReadingAccess(); });
const S = me.settings, rtl = D.direction === 'rtl', ttb = D.direction === 'ttb';
const GAP = ttb ? 0 : 8, END_H = 280;
const GAP = ttb || S.flowPageGap === false ? 0 : 8, END_H = 280;
document.title = `${D.chapter.label} · ${D.manga.title}`;
// ---------- pages ----------
@@ -41,9 +108,40 @@
let cur = 0, atEnd = false, done = !!(D.progress && D.progress.done), bars = true, barsAt = 0;
let spreads = [], spreadIdx = 0, shown = [], els = [], boxes = [], endEl = null, lastW = 0, lastH = 0;
let lockFlowX = false, readingClock = null;
const shift = new Set(D.progress && D.progress.v === D.version ? D.progress.shift : []);
const zkey = () => (mode === 'flow' && ttb ? 'strip' : mode);
const zoom = () => S.zoom[zkey()] || 1;
const tones = [
{ id: 'original', label: 'Original' },
{ id: 'inverted', label: 'Inverted' },
{ id: 'sepia', label: 'Sepia' },
{ id: 'dim', label: 'Dim' },
{ id: 'extra-dim', label: 'Extra dim' },
];
const toneButton = $('#page-tone');
const toneName = id => tones.find(tone => tone.id === id)?.label || 'Original';
function applyTone(id) {
S.pageTone = tones.some(tone => tone.id === id) ? id : 'original';
canvas.dataset.pageTone = S.pageTone;
toneButton.dataset.pageTone = S.pageTone;
toneButton.textContent = `Tone: ${toneName(S.pageTone)}`;
toneButton.title = `Page images: ${toneName(S.pageTone)}. Click for the next tone; controls and theme stay unchanged.`;
toneButton.setAttribute('aria-label', toneButton.title);
}
applyTone(S.pageTone);
let savedTone = S.pageTone, toneSave = Promise.resolve(), toneChange = 0;
function cycleTone() {
const next = tones[(tones.findIndex(tone => tone.id === S.pageTone) + 1) % tones.length];
applyTone(next.id);
const change = ++toneChange;
toneSave = toneSave.catch(() => {}).then(async () => {
await api('api/settings', { pageTone: next.id });
savedTone = next.id;
}).catch(() => {
if (change === toneChange) { applyTone(savedTone); toast('Could not save page tone'); }
});
}
const pr = D.progress;
if (pr && !(pr.done && pr.p >= pr.n - 1)) {
@@ -51,11 +149,6 @@
const k = list.findIndex(x => x.i >= p);
cur = k < 0 ? list.length - 1 : k;
}
// read.html?...&p=<page index> opens that page (jump links from Who's here).
if (P.get('p') !== null && /^\d+$/.test(P.get('p'))) {
const k = list.findIndex(x => x.i >= +P.get('p'));
if (k >= 0) cur = k;
}
// Book mode: pages pair up inside each run of single pages. Runs restart after every double page / cover,
// so one missing pair never shifts the rest of the chapter. "Shift" makes a run start with a lone page.
@@ -117,8 +210,9 @@
Object.assign(p, { unknown: false, w, a: w / hh, wide: p.wide || w > hh });
relayout();
}
if (readingClock) readingClock.viewChanged();
};
img.onerror = state;
img.onerror = () => { state(); if (readingClock) readingClock.viewChanged(); };
state();
}
function setSrc(el, url) { el.firstChild.src = url; watch(el); }
@@ -167,6 +261,7 @@
}))).then(() => { if (token === buildToken) preloadAround(); });
}
layout();
if (readingClock) readingClock.viewChanged();
}
let buildToken = 0;
@@ -203,7 +298,11 @@
canvas.style.width = cw + 'px';
canvas.style.height = ch + 'px';
els.forEach((el, j) => place(el, boxes[j]));
stage.classList.toggle('pan', cw > vw + 1 || (mode === 'flow' ? z > 1.01 : ch > vh + 1));
const overflowsX = cw > vw + 1;
lockFlowX = mode === 'flow' && !overflowsX;
stage.classList.toggle('pan', overflowsX || (mode !== 'flow' && ch > vh + 1));
stage.classList.toggle('lock-x', lockFlowX);
if (lockFlowX) stage.scrollLeft = 0;
lastW = vw; lastH = vh;
}
@@ -235,17 +334,18 @@
function restore(a, cx, cy) {
if (!a || !boxes.length) return;
const b = a.k < 0 ? union() : boxes[a.k];
stage.scrollLeft = b.x + a.fx * b.w - cx;
stage.scrollLeft = lockFlowX ? 0 : b.x + a.fx * b.w - cx;
stage.scrollTop = b.y + a.fy * b.h - cy;
barsAt = stage.scrollTop;
}
function relayout() { const a = anchor(lastW / 2, lastH / 2); layout(); restore(a, lastW / 2, lastH / 2); }
// ---------- zoom ----------
const MIN_ZOOM = 0.2, MAX_ZOOM = 6, ZOOM_STEP = 0.05;
let zoomTimer = 0;
function setZoom(z, cx = stage.clientWidth / 2, cy = stage.clientHeight / 2) {
z = Math.min(6, Math.max(0.2, z));
if (Math.abs(z - 1) < 0.03) z = 1; // easy to get back to "fit"
z = Math.min(MAX_ZOOM, Math.max(MIN_ZOOM, z));
if (Math.abs(z - 1) < 0.01) z = 1; // easy to get back to "fit" without swallowing fine adjustments
if (Math.abs(z - zoom()) < 0.001) return;
const a = anchor(cx, cy);
S.zoom[zkey()] = z;
@@ -257,11 +357,39 @@
zoomTimer = setTimeout(() => api('api/settings', { zoom: S.zoom }).catch(() => {}), 800);
}
// Pages exposed enough to count as something the reader could actually inspect.
// The clock handles decoding, duration and plausibility; this keeps layout knowledge here.
function readingView() {
const viewport = stage.getBoundingClientRect(), viewportArea = viewport.width * viewport.height, pages = [];
for (let j = 0; j < els.length; j++) {
const el = els[j], rect = el.getBoundingClientRect();
const w = Math.max(0, Math.min(rect.right, viewport.right) - Math.max(rect.left, viewport.left));
const hh = Math.max(0, Math.min(rect.bottom, viewport.bottom) - Math.max(rect.top, viewport.top));
const visibleArea = w * hh, pageArea = Math.max(1, rect.width * rect.height);
if (visibleArea < 0.10 * Math.min(pageArea, viewportArea)) continue;
const img = el.firstChild, page = mode === 'flow' ? list[j] : list[shown[j]];
pages.push({
index: page.i,
loaded: !!(img.complete && img.naturalWidth),
failed: !!(img.complete && img.getAttribute('src') && !img.naturalWidth),
});
}
const last = boxes[boxes.length - 1];
const flowEnd = mode === 'flow' && last && !pages.length && stage.scrollTop + stage.clientHeight >= last.y + last.h + 32;
return { pages, atEnd: mode === 'flow' ? !!flowEnd : atEnd };
}
function stepZoom(direction) {
const units = zoom() / ZOOM_STEP;
const stepped = (direction > 0 ? Math.floor(units + 1e-6) + 1 : Math.ceil(units - 1e-6) - 1) * ZOOM_STEP;
setZoom(Math.round(stepped * 100) / 100);
}
// Ctrl + wheel (and touchpad pinch, which browsers report the same way) zooms the pages, not the page controls.
stage.addEventListener('wheel', e => {
if (!e.ctrlKey) return;
e.preventDefault();
setZoom(zoom() * Math.exp(-e.deltaY * (e.deltaMode === 1 ? 0.05 : 0.0025)), e.clientX, e.clientY);
setZoom(zoom() * Math.exp(-e.deltaY * (e.deltaMode === 1 ? 0.015 : 0.0005)), e.clientX, e.clientY);
}, { passive: false });
// Touch: two fingers zoom, a quick horizontal swipe turns the page (page/book mode, when not zoomed in).
@@ -271,10 +399,18 @@
stage.addEventListener('touchstart', e => {
const t = e.touches;
if (t.length === 2) { pinch = { d: dist(t), z: zoom() }; swipe = null; }
else if (t.length === 1) swipe = { x: t[0].clientX, y: t[0].clientY, t: Date.now() };
else if (t.length === 1) swipe = { x: t[0].clientX, y: t[0].clientY, top: stage.scrollTop, t: Date.now() };
}, { passive: true });
stage.addEventListener('touchmove', e => {
if (!pinch || e.touches.length !== 2) return;
if (!pinch || e.touches.length !== 2) {
const t = e.touches[0], s = swipe;
if (t && s && lockFlowX && Math.abs(t.clientX - s.x) > Math.abs(t.clientY - s.y)) {
e.preventDefault();
stage.scrollLeft = 0;
stage.scrollTop = s.top - (t.clientY - s.y);
}
return;
}
e.preventDefault();
const t = e.touches;
pinchNext = [pinch.z * dist(t) / pinch.d, (t[0].clientX + t[1].clientX) / 2, (t[0].clientY + t[1].clientY) / 2];
@@ -298,7 +434,14 @@
// Mouse: drag to move a zoomed page around; a click without dragging uses the tap zones.
let drag = null, dragged = false;
stage.addEventListener('pointerdown', e => {
if (e.pointerType === 'mouse' && e.button === 0) drag = { x: e.clientX, y: e.clientY, l: stage.scrollLeft, t: stage.scrollTop, moved: false };
if (e.pointerType === 'mouse' && e.button === 0) drag = {
x: e.clientX,
y: e.clientY,
l: stage.scrollLeft,
t: stage.scrollTop,
panX: !lockFlowX && stage.scrollWidth > stage.clientWidth + 1,
moved: false
};
});
addEventListener('pointermove', e => {
if (!drag) return;
@@ -306,7 +449,7 @@
if (!drag.moved && Math.abs(dx) + Math.abs(dy) < 6) return;
drag.moved = true;
stage.classList.add('drag');
stage.scrollLeft = drag.l - dx;
if (drag.panX) stage.scrollLeft = drag.l - dx;
stage.scrollTop = drag.t - dy;
});
addEventListener('pointerup', () => {
@@ -343,17 +486,16 @@
if (k === 'ArrowLeft' || k === 'ArrowRight') { e.preventDefault(); (k === (rtl ? 'ArrowLeft' : 'ArrowRight') ? next : prev)(); }
else if ((k === ' ' || k === 'PageDown') && mode !== 'flow') { e.preventDefault(); (e.shiftKey ? prev : next)(); }
else if (k === 'PageUp' && mode !== 'flow') { e.preventDefault(); prev(); }
else if (k === '+' || k === '=') setZoom(zoom() * 1.15);
else if (k === '-') setZoom(zoom() / 1.15);
else if (k === '+' || k === '=') stepZoom(1);
else if (k === '-') stepZoom(-1);
else if (k === '0') setZoom(1);
else if (k === 'Escape' && senpai && senpai.close()) return;
else if (k === 'm' || k === 'Escape') toggleBars();
else if (k === 'f') toggleFullscreen();
else if (k === 'x' && senpai) senpai.toggle();
});
let scrollRaf = 0;
stage.addEventListener('scroll', () => {
if (lockFlowX && stage.scrollLeft !== 0) stage.scrollLeft = 0;
if (mode !== 'flow' || scrollRaf) return;
scrollRaf = requestAnimationFrame(() => {
scrollRaf = 0;
@@ -485,21 +627,21 @@
$('#zoom').textContent = Math.round(zoom() * 100) + '%';
}
let saveTimer = 0, senpai = null;
let saveTimer = 0;
function moved() {
if (mode === 'flow') {
const b = boxes[boxes.length - 1];
if (b && stage.scrollTop + stage.clientHeight >= b.y + b.h - 4) { done = true; cur = list.length - 1; }
} else if (atEnd || (mode === 'page' ? cur === list.length - 1 : spreadIdx === spreads.length - 1)) done = true;
updateUI();
if (readingClock) readingClock.viewChanged();
clearTimeout(saveTimer);
saveTimer = setTimeout(saveProgress, 700);
if (senpai) senpai.moved();
}
function saveProgress(keepalive) {
clearTimeout(saveTimer);
const body = JSON.stringify({ m: D.manga.id, c: D.chapter.id, v: D.version, p: list[cur].i, n: all.length, done, shift: [...shift] });
return fetch('api/progress', { method: 'POST', keepalive: !!keepalive, headers: { 'Content-Type': 'application/json' }, body }).catch(() => {});
fetch('api/progress', { method: 'POST', keepalive: !!keepalive, headers: { 'Content-Type': 'application/json' }, body }).catch(() => {});
}
addEventListener('pagehide', () => saveProgress(true));
@@ -526,8 +668,11 @@
ver.hidden = D.versions.length < 2;
ver.onchange = () => { saveProgress(true); location.href = readUrl(D.manga.id, D.chapter.id, ver.value); };
for (const b of modeBtns) b.onclick = () => { if (b.dataset.mode !== mode) setMode(b.dataset.mode, true); };
toneButton.onclick = cycleTone;
$('#shift').onclick = toggleShift;
$('#zoomout').onclick = () => stepZoom(-1);
$('#zoom').onclick = () => setZoom(1);
$('#zoomin').onclick = () => stepZoom(1);
slider.dir = rtl ? 'rtl' : 'ltr';
slider.addEventListener('input', () => goTo(+slider.value - 1));
function toggleFullscreen() {
@@ -540,153 +685,20 @@
// Keep keyboard focus on the pages so Space/arrows never "press" the last clicked button again.
for (const bar of [$('#top'), $('#bot')]) bar.addEventListener('click', e => { if (e.target.closest('button')) stage.focus({ preventScroll: true }); });
// ---------- Who's here (Senpai, optional: only when the server has SENPAI_URL) ----------
// A side sheet in the .panel style with three tabs and a character card. Everything comes from
// api/senpai/v1/..., which the server answers only up to this user's reading horizon.
function whosHere() {
const sheet = h('aside', { class: 'panel sheet', hidden: true, 'aria-label': "Who's here" });
const body = h('div', { class: 'sheet-body' }), sub = h('div', { class: 'muted small' });
const tabs = [['page', 'This page'], ['chapter', 'This chapter'], ['so_far', 'So far']];
const tabBtns = tabs.map(([id, label]) => h('button', { 'data-tab': id, onclick: () => { tab = id; store.set('senpai.tab', id); card = null; unmark(); render(); } }, label));
const option = (key, label, on) => h('label', { class: 'check small' },
h('input', { type: 'checkbox', checked: store.get(key) === on, onchange: e => { store.set(key, e.target.checked ? on : ''); card = null; refresh(); } }), h('span', {}, label));
sheet.append(
h('div', { class: 'sheet-head' }, h('div', { class: 'grow' }, h('b', {}, "Who's here"), sub),
h('button', { class: 'btn icon', title: 'Close (Esc)', 'aria-label': 'Close', onclick: () => close() }, '×')),
h('div', { class: 'seg' }, tabBtns), body,
h('div', { class: 'sheet-foot' }, option('senpai.asof', 'Only up to this page (for re-reading)', 'page'), option('senpai.proposed', 'Show unconfirmed (machine) findings', '1')));
document.body.append(sheet);
let tab = ['page', 'chapter', 'so_far'].includes(store.get('senpai.tab')) ? store.get('senpai.tab') : 'page';
let data = null, card = null, loadToken = 0, movedTimer = 0, marks = [];
const opts = extra => qsa({ m: D.manga.id, asof: store.get('senpai.asof') === 'page' ? 'page' : null, proposed: store.get('senpai.proposed') === '1' ? '1' : null, ...extra });
const pagesShown = () => (mode === 'flow' || atEnd || !shown.length ? [list[cur]] : shown.map(k => list[k])).filter(Boolean).map(p => p.i);
const sp = path => 'api/senpai/' + path;
const plural = (n, word) => `${n} ${word}${n === 1 ? '' : 's'}`;
function unmark() { for (const m of marks) m.remove(); marks = []; }
// Outline a character's figures on the visible page(s); boxes are in page pixels.
function mark(item) {
unmark();
if (!item.boxes || !item.page_size || !item.page_size[0]) return;
const [w, hh] = item.page_size;
for (const b of item.boxes) {
for (const el of els) {
if (el._p.i !== b.page || el._p.r) continue;
const [x1, y1, x2, y2] = b.box;
const box = h('div', { class: 'sp-box' });
box.style.cssText = `left:${(x1 / w) * 100}%;top:${(y1 / hh) * 100}%;width:${((x2 - x1) / w) * 100}%;height:${((y2 - y1) / hh) * 100}%`;
el.append(box);
marks.push(box);
}
}
}
function pic(item, cls) {
const none = () => h('div', { class: (cls || 'pic') + ' none' }, item.name ? item.name.slice(0, 1).toUpperCase() : '?');
return item.portrait_url ? h('img', { class: cls || 'pic', src: sp(item.portrait_url), alt: '', loading: 'lazy', onerror: e => e.target.replaceWith(none()) }) : none();
}
function who(item, detail) {
return h('button', { class: 'who', onclick: () => { mark(item); openCard(item); }, onmouseenter: () => mark(item), onmouseleave: () => { if (!card) unmark(); } },
pic(item),
h('div', { class: 'grow' }, h('div', { class: item.name ? 'nm' : 'nm unnamed' }, item.name || item.descriptor || 'unknown'), h('div', { class: 'muted small' }, detail)),
item.prominence ? h('span', { class: 'sp-chip' }, item.prominence) : null,
item.confirmed === false ? h('span', { class: 'sp-chip' }, 'unconfirmed') : null);
}
// Jump links stay inside the reader: same chapter and version -> go to the page, otherwise open it.
function jumpLink(j, label, text) {
const same = j.c === D.chapter.id && j.v === D.version;
return h('a', { href: readUrl(j.m, j.c, j.v, j.p), onclick: e => {
if (!same) { saveProgress(true); return; }
e.preventDefault();
const k = list.findIndex(x => x.i >= j.p);
if (k >= 0) goTo(k);
} }, h('div', { class: 'muted small' }, label), text ? h('div', {}, text) : null);
}
function render() {
for (const b of tabBtns) b.classList.toggle('on', b.dataset.tab === tab && !card);
body.textContent = '';
if (card) return renderCard();
if (!data) return body.append(h('p', { class: 'muted' }, 'Loading…'));
if (data.error) return body.append(h('p', { class: 'error' }, data.error));
sub.textContent = data.horizon && data.horizon.label ? 'What you know up to ' + data.horizon.label : 'Nothing read yet';
if (tab === 'page') {
const named = data.page.filter(x => x.kind === 'character'), figures = data.page.filter(x => x.kind !== 'character');
if (data.beyond_horizon) body.append(h('p', { class: 'muted' }, "You haven't read this far yet."));
else if (!data.page.length) body.append(h('p', { class: 'muted' }, data.coverage.page_has_evidence ? 'Nobody on this page.' : 'Senpai has no notes for this page yet.'));
for (const x of named) body.append(who(x, x.voice_only ? 'heard, not seen' : x.lines ? plural(x.lines, 'line') : 'seen'));
if (figures.length) {
const more = h('div', { hidden: true }, figures.map(x => who(x, x.lines ? plural(x.lines, 'line') : 'seen')));
body.append(h('button', { class: 'btn small sp-more', onclick: () => { more.hidden = !more.hidden; } }, `${plural(figures.length, 'unknown figure')} on this page`), more);
}
if (data.crowd_on_page) body.append(h('p', { class: 'muted small' }, 'and a crowd'));
} else if (tab === 'chapter') {
if (!data.chapter.length) body.append(h('p', { class: 'muted' }, 'Senpai has no notes for this chapter up to here.'));
for (const x of data.chapter) body.append(who(x, `${plural(x.pages, 'page')} · ${plural(x.lines, 'line')} in this chapter`));
if (data.chapter_unnamed_figures) body.append(h('p', { class: 'muted small' }, `and ${plural(data.chapter_unnamed_figures, 'unnamed figure')}`));
} else {
let any = false;
for (const [key, label] of [['lead', 'Lead'], ['frequent', 'Frequent'], ['occasional', 'Occasional'], ['minor', 'Minor']]) {
if (!data.so_far[key].length) continue;
any = true;
body.append(h('div', { class: 'tier' }, label));
for (const x of data.so_far[key]) body.append(who(x, `${plural(x.pages, 'page')} · ${plural(x.lines, 'line')} so far`));
}
if (!any) body.append(h('p', { class: 'muted' }, 'Senpai has no notes up to here.'));
body.append(h('p', { class: 'muted small' }, `Based on ${plural(data.coverage.pages_so_far, 'page')} with notes, up to where you are.`));
}
}
async function openCard(item) {
const mine = (card = { item, info: null, time: null, error: null });
render();
try {
const [info, time] = await Promise.all([api(sp(`v1/characters/${item.ref}?` + opts())), api(sp(`v1/characters/${item.ref}/timeline?` + opts()))]);
Object.assign(mine, { info, time });
} catch (e) { mine.error = e.message; }
if (card === mine) render();
}
function renderCard() {
const { item, info, time, error } = card, c = info || item;
body.append(h('button', { class: 'btn small', onclick: () => { card = null; unmark(); render(); } }, '‹ Back'),
h('div', { class: 'sp-card' }, pic(c, 'big'), h('div', { class: 'grow' }, h('h2', {}, c.name || c.descriptor || 'unknown'),
c.prominence ? h('div', {}, h('span', { class: 'sp-chip' }, c.prominence)) : null,
info ? h('div', { class: 'muted small' }, `${plural(info.counts.pages, 'page')} · ${plural(info.counts.lines, 'line')} so far`) : null)));
if (error) return body.append(h('p', { class: 'error' }, error));
if (!info) return body.append(h('p', { class: 'muted' }, 'Loading…'));
if (info.names_known.length > 1) {
body.append(h('div', { class: 'tier' }, 'Names so far'), h('ul', { class: 'sp-names' }, info.names_known.map(n => h('li', {}, n.name, h('span', { class: 'muted small' }, ' · since ' + n.since)))));
}
if (info.first_seen) body.append(h('div', { class: 'tier' }, 'First seen'), h('div', { class: 'sp-time' }, jumpLink(info.first_seen.jump, info.first_seen.label)));
if (time && time.items.length) body.append(h('div', { class: 'tier' }, 'Timeline'), h('div', { class: 'sp-time' }, time.items.map(t => jumpLink(t.jump, t.label, t.text))));
body.append(h('p', { class: 'muted small' }, 'Only what you have read, up to ' + info.horizon.label + '.'));
}
async function refresh() {
const token = ++loadToken;
await saveProgress(); // the server's horizon must include the page on screen
let x;
try { x = await api(sp('v1/xray?' + opts({ c: D.chapter.id, v: D.version, p: pagesShown().join(',') }))); }
catch (e) { x = { error: e.message }; }
if (token === loadToken) { data = x; render(); }
}
const btn = h('button', { id: 'who', class: 'btn', title: "Who's here (x)", 'aria-label': "Who's here", onclick: () => toggle() },
icon(ICONS.people), h('span', { class: 'wide-only' }, ' Who’s here'));
function show() { sheet.hidden = false; btn.classList.add('on'); card = null; data = null; render(); refresh(); }
function close() { if (sheet.hidden) return false; sheet.hidden = true; btn.classList.remove('on'); unmark(); stage.focus({ preventScroll: true }); return true; }
function toggle() { if (sheet.hidden) show(); else close(); }
$('#top').insertBefore(btn, $('#fs'));
return {
toggle, close,
moved() {
if (sheet.hidden) return;
clearTimeout(movedTimer);
unmark();
movedTimer = setTimeout(() => { if (!card) refresh(); }, 500);
},
};
}
if (me.senpai) senpai = whosHere();
setMode(mode, false);
if (S.readingClock) readingClock = createReadingClock({
button: $('#readclock'),
panel: $('#clockpanel'),
identity: { m: D.manga.id, c: D.chapter.id, v: D.version },
initial: D.reading,
eligibleCount: list.length,
eligiblePages: list.map(page => page.i),
surface: stage,
view: readingView,
onComplete: result => toast(result.completedPasses > 1
? 'Reread time is now counted · 60% of pages viewed'
: 'Reading time is now counted · 60% of pages viewed', 3500),
});
stage.focus({ preventScroll: true });
setBars(true);
})();
+32
View File
@@ -0,0 +1,32 @@
'use strict';
// Shared pure policy. No speed limit; credit only loaded pages actually observed,
// not pages skipped between two scroll positions. Coordinates never leave reader.
(function (root) {
const IDLE_MS = 30000, EXPOSURE_MS = 100;
function exposureTracker() {
const seen = new Set(), exposures = new Map();
function update(pages, now, active) {
const visible = new Set(active ? pages.filter(page => page.loaded).map(page => page.index) : []);
for (const [page, entry] of exposures) {
if (entry.visible) entry.ms += Math.max(0, now - entry.at);
entry.at = now; entry.visible = visible.has(page);
if (entry.ms >= EXPOSURE_MS) { seen.add(page); exposures.delete(page); }
}
for (const page of visible) if (!seen.has(page) && !exposures.has(page)) exposures.set(page, { ms: 0, at: now, visible: true });
return seen;
}
return { seen, update };
}
function stateFor({ visible, focused, lastInputAt, now, owned, view }) {
if (!visible) return 'hidden';
if (!focused) return 'unfocused';
if (now - lastInputAt >= IDLE_MS) return 'idle';
if (!owned) return 'other-tab';
if (view.atEnd) return 'finished';
if (view.pages.some(page => page.loaded)) return 'reading';
return view.pages.some(page => !page.failed) || !view.pages.length ? 'loading' : 'page-error';
}
const policy = { IDLE_MS, EXPOSURE_MS, exposureTracker, stateFor };
if (typeof module !== 'undefined' && module.exports) module.exports = policy;
else root.ReadingClockPolicy = policy;
})(typeof window === 'undefined' ? globalThis : window);
+177
View File
@@ -0,0 +1,177 @@
'use strict';
/* Opt-in, personal reading estimates. The server counts visits with 60% coverage. */
(function () {
const { IDLE_MS, exposureTracker, stateFor } = window.ReadingClockPolicy;
const INPUTS = ['mouse', 'touch', 'keyboard'], PULSE_MS = 10000;
const labels = {
reading: ['⏱', 'Reading'], loading: ['⌛', 'Loading page'], idle: ['⏸', 'Idle'],
hidden: ['⏸', 'Tab hidden'], unfocused: ['⏸', 'Window inactive'],
'other-tab': ['⏸', 'Another reader is active'], 'page-error': ['⏸', 'Page unavailable'],
finished: ['⏸', 'Chapter finished'],
};
const blank = () => ({ activeMs: 0, loadingMs: 0, inputMs: { mouse: 0, touch: 0, keyboard: 0 } });
const fmt = ms => {
const seconds = Math.max(0, Math.floor(ms / 1000)), hours = Math.floor(seconds / 3600);
return (hours ? hours + ':' : '') + String(Math.floor(seconds / 60) % 60).padStart(2, '0') + ':' + String(seconds % 60).padStart(2, '0');
};
window.createReadingClock = function (options) {
const button = options.button, panel = options.panel;
const session = (crypto.randomUUID ? crypto.randomUUID() : Math.random().toString(36).slice(2) + Date.now().toString(36)).replace(/[^a-zA-Z0-9_-]/g, '');
const started = performance.now(), eligible = new Set(options.eligiblePages);
const required = Math.ceil(eligible.size * 0.6);
let visit = blank(), acknowledged = blank(), tracker = exposureTracker(), acknowledgedSeen = [];
let reading = options.initial || {}, state = 'idle', lastSample = started, lastInputAt = started, lastType = 'mouse';
let focused = document.hasFocus(), needsInput = false, owned = true, claimAt = 0, channel;
let sequence = 0, lastSent = started, sending = false, urgent = false, releasePending = false;
let retryTimer = 0, saveError = false, ended = false, destroyed = false, notified = false;
const listeners = [];
function listen(target, name, fn, opts) {
target.addEventListener(name, fn, opts); listeners.push(() => target.removeEventListener(name, fn, opts));
}
function accrue(now) {
const delta = Math.max(0, Math.min(now, lastInputAt + IDLE_MS) - lastSample);
if (state === 'reading') { visit.activeMs += delta; visit.inputMs[lastType] += delta; }
else if (state === 'loading') visit.loadingMs += delta;
lastSample = now;
}
function covered() { return [...tracker.seen].filter(page => eligible.has(page)); }
function sample() {
if (destroyed || ended) return;
const now = performance.now(); accrue(now);
const view = options.view();
const previous = state;
state = stateFor({ visible: !document.hidden, focused, now, lastInputAt: needsInput ? -Infinity : lastInputAt, owned, view });
tracker.update(view.pages, now, state === 'reading');
if (previous !== state && ['idle', 'finished'].includes(state) && ['reading', 'loading'].includes(previous)) requestSave(true);
if (now - lastSent >= PULSE_MS && owned && ['reading', 'loading'].includes(state)) requestSave();
render();
}
function snapshot(final = false, release = false) {
lastSent = performance.now();
return { ...options.identity, session, seq: ++sequence, elapsedMs: Math.round(lastSent - started),
activeMs: Math.round(visit.activeMs), loadingMs: Math.round(visit.loadingMs),
inputMs: Object.fromEntries(INPUTS.map(key => [key, Math.round(visit.inputMs[key])])),
seen: covered(), final, release: final || release };
}
function requestSave(release = false) {
urgent = true; releasePending ||= release;
pump();
}
async function pump() {
if (sending || destroyed || ended || !urgent) return;
sending = true; urgent = false;
const body = snapshot(false, releasePending); releasePending = false;
clearTimeout(retryTimer);
try {
const response = await fetch('api/reading/visit', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify(body) });
const data = await response.json();
if (!response.ok) throw new Error(data.error || response.statusText);
saveError = false;
if (data.accepted) {
acknowledged = body; acknowledgedSeen = body.seen; reading = data.reading || reading;
if (reading.currentPass?.complete && !notified) {
notified = true;
options.onComplete?.({ activeMs: visit.activeMs, totalActiveMs: reading.totalActiveMs, completedPasses: reading.completedPasses });
}
} else {
accrue(performance.now()); owned = false; state = 'other-tab';
visit = { ...acknowledged, inputMs: { ...acknowledged.inputMs } };
tracker = exposureTracker(); acknowledgedSeen.forEach(page => tracker.seen.add(page));
retryTimer = setTimeout(() => { if (!needsInput && !ended) { owned = true; requestSave(); sample(); } }, Math.max(500, data.retryAfterMs || 2500));
}
} catch {
// Network delays aren't fake reading. Retain cumulative time and retry.
saveError = true;
retryTimer = setTimeout(() => requestSave(), 2500);
} finally {
sending = false; sample();
// New milliseconds accrued during a response must NOT trigger another
// request. Only an explicitly requested save can run the pump again.
if (urgent) pump();
}
}
function flush() {
if (destroyed || ended) return;
sample(); ended = true;
const body = JSON.stringify(snapshot(true, true));
// Send even with a pulse in flight. Cumulative sequence handling makes
// either arrival order safe, including the final beacon arriving first.
let queued = false;
try { queued = navigator.sendBeacon?.('api/reading/visit', new Blob([body], { type: 'application/json' })); } catch {}
if (!queued) fetch('api/reading/visit', { method: 'POST', keepalive: true, headers: { 'Content-Type': 'application/json' }, body }).catch(() => {});
}
function input(type) {
if (ended || destroyed || document.hidden) return;
const now = performance.now(); accrue(now);
lastType = type; lastInputAt = now; needsInput = false; focused = document.hasFocus();
if (!owned) { owned = true; requestSave(); }
const at = Date.now();
if (at - claimAt >= 1000) { claimAt = at; channel?.postMessage({ session, at }); }
sample();
}
function render() {
const [icon, label] = labels[state] || labels.idle, coverage = covered().length;
const qualified = coverage >= required && visit.activeMs > 0;
const total = (+reading.totalActiveMs || 0) + (qualified ? Math.max(0, visit.activeMs - (reading.currentPass?.complete ? acknowledged.activeMs : 0)) : 0);
button.querySelector('.clock-icon').textContent = icon;
button.querySelector('.clock-time').textContent = fmt(visit.activeMs);
button.querySelector('.clock-label').textContent = ` · ${label}${saveError ? ' · Save pending' : ''}`;
button.classList.toggle('clock-running', state === 'reading');
button.classList.remove('clock-warning');
button.title = `Active Reading Clock · ${label} · ${coverage}/${eligible.size} pages`;
if (!panel.hidden) {
panel.textContent = '';
panel.append(h('div', { class: 'clock-head' }, h('b', {}, 'Active Reading Clock'), h('span', { class: 'clock-state' }, `${icon} ${label}`)),
h('dl', { class: 'clock-details' },
h('dt', {}, 'This visit'), h('dd', {}, fmt(visit.activeMs)),
h('dt', {}, 'Pages viewed'), h('dd', {}, `${coverage} / ${eligible.size}`),
h('dt', {}, 'Counted as a read'), h('dd', {}, qualified ? 'Yes · time continues counting' : `Not yet · ${required} pages needed (60%)`),
h('dt', {}, 'Saved chapter total'), h('dd', {}, fmt(total)),
h('dt', {}, 'Loading wait'), h('dd', {}, fmt(visit.loadingMs))),
h('p', { class: 'muted small' }, saveError ? 'Save pending. Your time is retained in this tab and will be retried.' :
'No scrolling-speed limit. Loaded pages count after 100 ms of visible exposure. At 60% coverage, all active time from this visit counts and is saved when you leave. Short picture lookups stay out of your totals. Pauses in the background and after 30 seconds without reader activity.'));
}
}
button.hidden = false;
listen(button, 'click', e => { e.stopPropagation(); panel.hidden = !panel.hidden; render(); });
listen(panel, 'click', e => e.stopPropagation());
listen(document, 'click', () => { panel.hidden = true; });
try {
channel = new BroadcastChannel('mangacircus-reading-clock');
channel.onmessage = event => {
const msg = event.data || {};
if (msg.session === session || !msg.at || msg.at < claimAt || (msg.at === claimAt && msg.session < session)) return;
accrue(performance.now()); owned = false; state = 'other-tab'; requestSave(true); render();
};
} catch { channel = null; }
let pointer = null;
listen(document, 'pointerdown', e => { if (e.isTrusted) input(e.pointerType === 'touch' ? 'touch' : 'mouse'); }, { passive: true });
listen(document, 'pointermove', e => {
if (!e.isTrusted) return;
const now = performance.now();
if (!pointer || (now - pointer.at >= 150 && Math.hypot(e.clientX - pointer.x, e.clientY - pointer.y) >= 3)) {
pointer = { x: e.clientX, y: e.clientY, at: now }; input(e.pointerType === 'touch' ? 'touch' : 'mouse');
}
}, { passive: true });
listen(document, 'wheel', e => { if (e.isTrusted) input('mouse'); }, { passive: true });
listen(document, 'keydown', e => {
if (e.isTrusted && ['ArrowUp', 'ArrowDown', 'ArrowLeft', 'ArrowRight', ' ', 'PageDown', 'PageUp', 'Home', 'End', '+', '=', '-', '0', 'm', 'Escape', 'f'].includes(e.key)) input('keyboard');
});
// Momentum scrolling remains activity after the finger lifts.
if (options.surface) listen(options.surface, 'scroll', e => { if (e.isTrusted && !needsInput) input(lastType); }, { passive: true });
listen(window, 'blur', () => { accrue(performance.now()); focused = false; needsInput = true; sample(); requestSave(true); });
listen(window, 'focus', () => { focused = true; sample(); });
listen(document, 'visibilitychange', () => {
if (document.hidden) { accrue(performance.now()); needsInput = true; sample(); requestSave(true); } else sample();
});
listen(window, 'pagehide', flush);
listen(document, 'freeze', () => { accrue(performance.now()); needsInput = true; sample(); requestSave(true); });
listen(window, 'pageshow', e => { if (e.persisted) { ended = false; lastSample = performance.now(); input(lastType); requestSave(); } });
const timer = setInterval(sample, 50);
sample(); requestSave();
return { viewChanged: sample, flush, destroy() {
flush(); destroyed = true; clearInterval(timer); clearTimeout(retryTimer);
listeners.forEach(remove => remove()); channel?.close();
} };
};
})();
+72
View File
@@ -0,0 +1,72 @@
'use strict';
// Two navigation levels; scraper tabs stay within their section, not another menu.
const SETTINGS_SECTIONS = [
{ id: 'user/reading', group: 'User', title: 'Reading', hint: 'Reader modes, page spacing and zoom.' },
{ id: 'user/reading-hours', group: 'User', title: 'Reading hours', hint: 'When this server allows chapter reading.' },
{ id: 'user/reading-time', group: 'User', title: 'Reading time & privacy', hint: 'Your clock, Reading Story and personal data.' },
{ id: 'user/notifications', group: 'User', title: 'Notifications', hint: 'Chapter alerts and your optional webhook.' },
{ id: 'user/account', group: 'User', title: 'Account', hint: 'Your account and password.' },
{ id: 'admin/scraper', group: 'Admin', title: 'Scraper', hint: 'Live activity, downloads, update checks and sources.' },
{ id: 'admin/peering', group: 'Admin', title: 'Peering', hint: 'Pair servers and manage their permissions.', href: 'peers.html#admin' },
{ id: 'admin/peer-catalog', group: 'Admin', title: 'Peer catalogs', hint: 'Approve titles to share and browse peer libraries.', href: 'peer-catalog.html' },
{ id: 'admin/library', group: 'Admin', title: 'Server & library', hint: 'Library location, scan status and maintenance.' },
{ id: 'admin/users', group: 'Admin', title: 'Users & access', hint: 'Manage accounts, passwords and administrators.' },
{ id: 'admin/collections', group: 'Admin', title: 'Collections & metadata', hint: 'Group manga and review their relationships.', href: 'collections.html' },
{ id: 'admin/health', group: 'Admin', title: 'Service health', hint: 'Service status, incidents and recovery.', href: 'health.html' },
];
const SCRAPER_TABS = ['activity', 'queue', 'updates', 'sources'];
const READING_DAYS = ['Sunday', 'Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday'];
function readingDayText(schedule, index) {
const previous = (index + 6) % 7;
const carried = schedule.days[previous].filter(([start, end]) => end < start)
.map(([, end]) => `00:00–${end} (from ${READING_DAYS[previous]})`);
const starting = schedule.days[index].map(([start, end]) => `${start}–${end}${end < start ? ' (next day)' : ''}`);
return [...carried, ...starting].join(' · ') || 'Closed all day';
}
function settingsRoute(hash, admin) {
const parts = String(hash || '').replace(/^#/, '').split('/');
const candidate = parts.slice(0, 2).join('/'), tab = parts[2];
const section = SETTINGS_SECTIONS.find(item => item.id === candidate && (admin || item.group === 'User'));
return { id: section?.id || 'user/reading', tab: SCRAPER_TABS.includes(tab) ? tab : tab === 'hosters' ? 'sources' : 'activity' };
}
function settingsShell(me, active = 'user/reading') {
const main = $('main'); main.textContent = '';
const nav = h('nav', { class: 'settings-menu', 'aria-label': 'Settings sections' });
const links = new Map();
for (const group of me.admin ? ['User', 'Admin'] : ['User']) {
nav.append(h('div', { class: 'settings-menu-group' }, h('h2', {}, group),
SETTINGS_SECTIONS.filter(section => section.group === group).map(section => {
const link = h('a', { href: section.href || 'settings.html#' + section.id }, section.title);
links.set(section.id, link);
return link;
})));
}
const crumb = h('p', { class: 'settings-breadcrumb muted small' });
const title = h('h1', { tabindex: '-1' });
const titleRow = h('div', { class: 'settings-title-row' }, title);
const hint = h('p', { class: 'muted settings-hint' });
const content = h('div', { class: 'settings-content' });
main.append(h('div', { class: 'settings-layout' }, nav,
h('div', { class: 'settings-workspace' }, crumb, titleRow, hint, content)));
function select(id, focus = false) {
const section = SETTINGS_SECTIONS.find(item => item.id === id);
if (!section || (section.group === 'Admin' && !me.admin)) return;
for (const [key, link] of links) {
if (key === id) link.setAttribute('aria-current', 'page');
else link.removeAttribute('aria-current');
}
crumb.textContent = 'Settings / ' + section.group;
title.textContent = section.title;
hint.textContent = section.hint;
document.title = section.title + ' · Settings · MangaCircus';
if (focus) title.focus({ preventScroll: true });
}
select(active);
return { content, titleRow, select };
}
if (typeof module !== 'undefined') module.exports = { SETTINGS_SECTIONS, SCRAPER_TABS, settingsRoute, readingDayText };
+1
View File
@@ -9,6 +9,7 @@
<link rel="manifest" href="manifest.webmanifest">
<meta name="theme-color" content="#121214">
<script src="common.js" defer></script>
<script src="settings-nav.js" defer></script>
<script src="settings.js" defer></script>
</head>
<body>
+427 -13
View File
@@ -1,13 +1,19 @@
'use strict';
/* Settings: reading preferences, notifications, password and (for admins) library + users. */
/* Settings workspace: User/Admin sections, with local scraper tabs. */
(async function () {
let me;
try { me = await api('api/me'); } catch (e) { return showError(e); }
topNav(me);
const S = me.settings, main = $('main');
main.textContent = '';
const S = me.settings, shell = settingsShell(me);
const sections = new Map(), controllers = new Map();
const section = id => {
if (!sections.has(id)) sections.set(id, h('div', { 'data-settings-section': id }));
return sections.get(id);
};
let route = settingsRoute(location.hash, me.admin);
const saved = h('span', { class: 'saved', hidden: true }, 'Saved');
shell.titleRow.append(saved);
let savedTimer = 0;
async function save(patch) {
Object.assign(S, (await api('api/settings', patch)).settings);
@@ -25,25 +31,87 @@
h('option', { value: 'page' }, 'Always Page (one page at a time)'),
h('option', { value: 'book' }, 'Always Book (two pages side by side)'));
startMode.value = S.startMode;
const tone = h('select', { onchange: e => save({ pageTone: e.target.value }) },
h('option', { value: 'original' }, 'Original'),
h('option', { value: 'inverted' }, 'Inverted'),
h('option', { value: 'sepia' }, 'Sepia'),
h('option', { value: 'dim' }, 'Dim (70% brightness)'),
h('option', { value: 'extra-dim' }, 'Extra dim (45% brightness)'));
tone.value = S.pageTone;
const LABEL = { flow: 'Flow', page: 'Page', book: 'Book', strip: 'Webtoon flow' };
const zooms = () => Object.keys(LABEL).map(k => `${LABEL[k]} ${Math.round((S.zoom[k] || 1) * 100)}%`).join(' · ');
const zoomText = h('span', { class: 'muted' }, zooms());
const readingPassword = h('input', { type: 'password', autocomplete: 'current-password', placeholder: 'Current password' });
const readingDataMessage = h('span', { class: 'small' });
const readingData = h('details', { class: 'reading-data' },
h('summary', {}, 'Manage my reading-time data'),
h('p', { class: 'muted small' }, 'Reading Story is private to your account. Turning the clock off stops collection but keeps existing history.'),
h('div', { class: 'row' },
h('button', { class: 'btn small', onclick: async () => {
try {
const data = await api('api/reading/export');
const blob = new Blob([JSON.stringify(data, null, 2) + '\n'], { type: 'application/json' });
const link = h('a', { href: URL.createObjectURL(blob), download: `mangacircus-reading-${me.name}.json` });
document.body.append(link); link.click(); link.remove(); setTimeout(() => URL.revokeObjectURL(link.href), 0);
readingDataMessage.textContent = 'Export created'; readingDataMessage.className = 'saved';
} catch (error) { readingDataMessage.textContent = error.message; readingDataMessage.className = 'small error'; }
} }, 'Export my summary')),
h('div', { class: 'reading-danger' },
h('b', {}, 'Delete all reading-time history'),
h('p', { class: 'muted small' }, 'Reading progress and accounts are not affected. This cannot be undone.'),
h('div', { class: 'row' }, readingPassword,
h('button', { class: 'btn small danger', onclick: async () => {
if (!readingPassword.value || !confirm('Permanently delete all of your reading-time history?')) return;
try {
await api('api/reading/delete', { all: true, password: readingPassword.value });
readingPassword.value = '';
readingDataMessage.textContent = 'Reading-time history deleted'; readingDataMessage.className = 'saved';
} catch (error) { readingDataMessage.textContent = error.message; readingDataMessage.className = 'small error'; }
} }, 'Delete history'), readingDataMessage)));
main.append(h('section', { class: 'card' },
h('h2', {}, 'Reading ', saved),
section('user/reading').append(h('section', { class: 'card' },
h('h2', {}, 'Reader preferences'),
h('label', { class: 'field' }, h('span', {}, 'Open chapters in'), startMode,
h('div', { class: 'muted small' }, `Last used: ${LABEL[S.lastMode]}. Only clicking a mode in the reader changes it.`)),
check('adapt', 'Use the fitting mode when the content is not made for my choice',
'Example: webtoons are one long strip, so they always open in Flow. This never changes the mode you picked.'),
check('skipExtras', 'Skip extra pages and extra chapters', 'Pages or chapters marked as extra (omake, notes, specials) are left out while reading.'),
check('skipCovers', 'Skip chapter cover pages'),
check('flowPageGap', 'Show a gap between pages in Flow mode',
'Turn this off for continuous webtoon artwork or text that crosses page edges.'),
h('label', { class: 'field' }, h('span', {}, 'Page tone'), tone,
h('div', { class: 'muted small' }, 'The reader button cycles through these choices. Only page artwork changes; the site theme and controls stay as they are.')),
h('div', { class: 'field' }, h('span', {}, 'Page zoom'), zoomText, ' ',
h('button', { class: 'btn small', onclick: async () => { await save({ zoom: {} }); zoomText.textContent = zooms(); } }, 'Reset to fit'),
h('div', { class: 'muted small' }, '100% = the page fits the screen. Change it in the reader with Ctrl + mouse wheel, a pinch, or the + / - keys.'))));
const readingHoursBody = h('div', {}, h('p', { class: 'muted' }, 'Loading server reading hours…'));
section('user/reading-hours').append(h('section', { class: 'card' },
h('h2', {}, 'Server reading hours'), readingHoursBody,
h('p', { class: 'muted small' }, 'These hours are set in the server deployment and cannot be changed here. Overnight windows continue into the next day.')));
api('api/reading-schedule').then(schedule => {
readingHoursBody.textContent = '';
if (!schedule.enabled) {
readingHoursBody.append(h('p', {}, 'Reading is available at any time. No schedule is configured.'));
return;
}
readingHoursBody.append(h('p', { class: 'muted' }, `Server timezone: ${schedule.timezone}`),
h('div', { class: 'reading-hours-list' }, READING_DAYS.map((day, index) =>
h('div', { class: 'reading-hours-day' }, h('strong', {}, day),
h('span', {}, readingDayText(schedule, index))))));
}).catch(error => { readingHoursBody.replaceChildren(h('p', { class: 'error', role: 'alert' }, `Could not load reading hours: ${error.message}`)); });
section('user/reading-time').append(h('section', { class: 'card' },
h('h2', {}, 'Clock & Reading Story'),
check('readingClock', 'Track and show the Active Reading Clock',
'Counts a visit after you view 60% of its pages, with no scrolling-speed limit. Pauses in the background or after 30 seconds without reader activity. Disabled by default.'),
check('readingStory', 'Show Your Reading Story',
'Shows your private chapter times and personal summaries on manga and library pages. Hiding it does not delete existing history.'),
readingData));
const hook = h('input', { type: 'url', placeholder: 'https://ntfy.sh/my-manga-topic', value: S.webhook || '', style: 'width:100%' });
const hookMsg = h('div', { class: 'small' });
main.append(h('section', { class: 'card' },
section('user/notifications').append(h('section', { class: 'card' },
h('h2', {}, 'Notifications'),
h('p', { class: 'muted' }, 'Press "Notify me" on a manga page. New chapters then show up under the bell at the top.'),
h('label', { class: 'field' }, h('span', {}, 'Also send them to a webhook (optional)'), hook,
@@ -55,18 +123,332 @@
const oldPw = h('input', { type: 'password', autocomplete: 'current-password', placeholder: 'Current password' });
const newPw = h('input', { type: 'password', autocomplete: 'new-password', placeholder: 'New password' });
const pwMsg = h('span', { class: 'small' });
main.append(h('section', { class: 'card' },
section('user/account').append(h('section', { class: 'card' },
h('h2', {}, 'Account · ' + me.name),
h('div', { class: 'row' }, oldPw, newPw, h('button', { class: 'btn', onclick: async () => {
try { await api('api/password', { old: oldPw.value, new: newPw.value }); pwMsg.textContent = 'Password changed'; pwMsg.className = 'saved'; oldPw.value = newPw.value = ''; }
catch (e) { pwMsg.textContent = e.message; pwMsg.className = 'small error'; }
} }, 'Change password'), pwMsg)));
if (me.admin) adminCard();
function scraperCard() {
const card = h('section', { class: 'card scraper-card' });
section('admin/scraper').append(card);
let timer = 0, active = false, lastState = null, dirty = false, sourceFilter = '', onlyEnabled = false;
const disclosureOpen = { add: false, completed: false, history: false };
const commandError = h('p', { class: 'error small', role: 'alert', hidden: true });
card.addEventListener('input', () => { dirty = true; });
// A poll can resolve between the summary click and the browser's deferred
// details toggle event. Record the intended state synchronously, rather
// than sampling the old DOM immediately before replacing it.
card.addEventListener('click', event => {
const summary = event.target.closest('summary');
const details = summary?.parentElement;
const key = details?.dataset.scraperDisclosure;
if (key && Object.hasOwn(disclosureOpen, key)) {
const intended = !details.open;
disclosureOpen[key] = intended;
// A filter click can redraw the panel while this summary click is
// bubbling. Apply the click to the replacement node as well; the
// browser's native toggle will only affect the detached original.
if (!details.isConnected) {
const replacement = card.querySelector(`[data-scraper-disclosure="${key}"]`);
if (replacement) replacement.open = intended;
}
}
});
card.addEventListener('toggle', event => {
const details = event.target;
const key = details.dataset?.scraperDisclosure;
if (details.isConnected && key && Object.hasOwn(disclosureOpen, key)) disclosureOpen[key] = details.open;
}, true);
async function adminCard() {
const command = async payload => {
try {
await api('api/admin/scraper', payload);
dirty = false; commandError.hidden = true;
await load(true);
} catch (e) { commandError.textContent = e.message; commandError.hidden = false; }
};
const statusText = entry => {
const job = entry.job;
const bits = [entry.current.title];
if (job?.current?.book?.title) bits.push(job.current.book.title);
if (job?.current?.number) bits.push(job.current.label || 'chapter ' + job.current.number);
if (job?.current?.expectedPages) bits.push(`page ${job.current.downloadedPages || 0}/${job.current.expectedPages}`);
return bits.join(' · ');
};
const draw = state => {
lastState = state;
card.textContent = '';
const tabs = h('nav', { class: 'settings-tabs', 'aria-label': 'Scraper sections' },
[['activity', 'Activity'], ['queue', 'Queue'], ['updates', 'Updates'], ['sources', 'Sources']].map(([id, label]) =>
h('a', { class: 'btn', href: '#admin/scraper/' + id, 'aria-current': route.tab === id ? 'page' : null }, label)));
card.append(tabs, commandError);
if (!state.available) {
card.append(h('h2', {}, 'Scraper'),
h('p', { class: 'error' }, state.error || 'Scraper service unavailable'),
h('p', { class: 'muted small' }, 'Check the managed services and their logs. Do not start a second scraper against the same library.'),
h('a', { class: 'btn small', href: 'health.html' }, 'Service health & incidents'));
return;
}
const activeEntries = state.active?.length ? state.active : state.current ? [{ source: state.current.source, current: state.current, job: state.job }] : [];
const url = h('input', { type: 'url', placeholder: 'WeebCentral or MangaDex title URL', class: 'grow' });
const title = h('input', { placeholder: 'Title override (optional)' });
const scope = h('select', {},
h('option', { value: 'missing' }, 'All missing chapters'),
h('option', { value: 'latest' }, 'Newest chapter only'),
h('option', { value: 'last' }, 'Last N chapters'),
h('option', { value: 'first' }, 'First N chapters'),
h('option', { value: 'chapter' }, 'One chapter'),
h('option', { value: 'after' }, 'After chapter'),
h('option', { value: 'volume' }, 'One volume'));
const scopeValue = h('input', { type: 'number', min: '0', step: '0.1', placeholder: 'N / chapter / volume', disabled: true });
scope.onchange = () => { scopeValue.disabled = ['missing', 'latest'].includes(scope.value); scopeValue.placeholder = ['last', 'first'].includes(scope.value) ? 'Count' : scope.value === 'volume' ? 'Volume' : 'Chapter'; };
const addSource = h('input', { type: 'checkbox' });
const libraryTarget = h('select', {}, h('option', { value: '' }, 'Auto-detect library manga'),
...(state.library || []).map(manga => h('option', { value: manga.id }, manga.title)));
const addMessage = h('span', { class: 'small' });
const rate = state.settings || {};
const sourceRateEditors = (state.adapters || []).map(adapter => {
const sourceRate = { ...rate, ...((rate.sourceRates || {})[adapter.id] || {}) };
const min = h('input', { type: 'number', min: '.5', step: '.5', value: (sourceRate.minMs ?? 1000) / 1000, title: 'Minimum seconds between image and API requests' });
const max = h('input', { type: 'number', min: '.5', step: '.5', value: (sourceRate.maxMs ?? 4000) / 1000, title: 'Maximum seconds between image and API requests' });
const chapterMin = h('input', { type: 'number', min: '.5', step: '1', value: (sourceRate.chapterMinMs ?? 5000) / 1000, title: 'Minimum seconds between chapters' });
const chapterMax = h('input', { type: 'number', min: '.5', step: '1', value: (sourceRate.chapterMaxMs ?? 15000) / 1000, title: 'Maximum seconds between chapters' });
return h('div', { class: 'source-rate' },
h('b', {}, adapter.label),
h('div', { class: 'row rate-row' },
h('label', {}, h('small', {}, 'Requests min (s)'), min), h('label', {}, h('small', {}, 'Requests max (s)'), max),
h('label', {}, h('small', {}, 'Chapters min (s)'), chapterMin), h('label', {}, h('small', {}, 'Chapters max (s)'), chapterMax),
h('button', { class: 'btn', onclick: () => command({ command: 'settings', settings: { sourceRates: { [adapter.id]: {
minMs: +min.value * 1000, maxMs: +max.value * 1000,
chapterMinMs: +chapterMin.value * 1000, chapterMaxMs: +chapterMax.value * 1000,
} } } }) }, `Save ${adapter.label}`)));
});
const autoEnabled = h('input', { type: 'checkbox', checked: rate.autoUpdateEnabled !== false });
const autoTime = h('input', { type: 'time', value: rate.autoUpdateTime || '03:00' });
const visible = items => items.filter(item => !sourceFilter || item.source === sourceFilter);
const pending = visible(state.queue.filter(item => ['queued', 'retry_wait'].includes(item.status)));
const attention = visible(state.queue.filter(item => ['ready', 'error', 'quarantined', 'unavailable', 'skipped'].includes(item.status)));
const completed = visible(state.queue.filter(item => item.status === 'complete' && item.kind === 'download'));
const when = value => value && !Number.isNaN(Date.parse(value)) ? new Date(value).toLocaleString() : 'Never';
const chips = h('div', { class: 'row scraper-filters', role: 'group', 'aria-label': 'Filter by source' },
[{ id: '', label: 'All sources' }, ...(state.adapters || []).map(adapter => ({ id: adapter.id, label: adapter.label }))].map(option =>
h('button', { class: 'btn small', type: 'button', 'aria-pressed': sourceFilter === option.id ? 'true' : 'false',
onclick: () => { sourceFilter = option.id; draw(lastState); } }, option.label)));
card.append(chips);
const move = async (item, delta) => {
const position = pending.findIndex(entry => entry.id === item.id), other = pending[position + delta];
if (!other) return;
const ids = state.queue.map(entry => entry.id), a = ids.indexOf(item.id), b = ids.indexOf(other.id);
[ids[a], ids[b]] = [ids[b], ids[a]];
await command({ command: 'reorder', ids });
};
const jobRow = (item, movable = false) => {
const running = activeEntries.some(entry => entry.current.id === item.id);
const availability = item.availability;
const expected = availability?.expectedLastChapter;
const sparse = /^\d+$/.test(String(expected || '')) && Number(availability?.numberedChapters) < Number(expected);
const partialCoverage = availability?.coverage === 'partial' || sparse;
const sourceIsPaused = !!state.sourcePaused?.[item.source];
const outcome = item.status === 'complete'
? `${item.publishedCount ?? 0} chapter${item.publishedCount === 1 ? '' : 's'} published · finished ${when(item.completed)}`
: item.status === 'ready' ? 'Awaiting edition confirmation'
: item.status === 'queued' && sourceIsPaused ? 'Confirmed · waiting for source access review'
: item.status.replaceAll('_', ' ');
return h('div', { class: 'scraper-item' + (running ? ' active' : ''), 'data-source': item.source },
h('div', { class: 'grow' }, h('b', {}, item.title),
item.status === 'complete' || availability ? h('div', { class: 'coverage-indicators' },
item.status === 'complete' ? h('span', { class: 'coverage-badge coverage-complete' }, 'Hosted selection complete') : null,
availability ? h('span', { class: 'coverage-badge ' + (partialCoverage ? 'coverage-partial' : 'coverage-unknown') },
partialCoverage ? 'Series coverage partial' : 'Series coverage unverified') : null,
item.status === 'complete' && !availability ? h('span', { class: 'coverage-badge coverage-unknown' }, 'Series coverage unverified') : null) : null,
h('div', { class: 'muted small' }, `${item.source} · ${item.kind === 'discovery' ? 'Checking for updates' : item.scopeLabel || 'all missing chapters'} · ${outcome}`
+ (item.existingPolicy === 'add-source' ? ' · alternate source' : '')),
item.error ? h('div', { class: 'small error' }, item.error) : null,
item.sourceTitle ? h('div', { class: 'small' }, `Source edition: ${item.sourceTitle}`) : null,
item.status === 'ready' && sourceIsPaused ? h('div', { class: 'small error' },
'This source is paused after an access response. Confirming saves your choice, but downloading will not start until access is reviewed and the source is resumed.') : null,
availability ? h('div', { class: 'muted small' }, `${availability.selectedChapters} hosted selections${availability.specialReleases ? ` · ${availability.specialReleases} unnumbered releases` : ''}${availability.expectedLastChapter ? ` · metadata last chapter: ${availability.expectedLastChapter}` : ''}`) : null),
movable ? h('button', { class: 'btn small', disabled: running || pending[0]?.id === item.id, title: 'Move up', onclick: () => move(item, -1) }, '↑') : null,
movable ? h('button', { class: 'btn small', disabled: running || pending.at(-1)?.id === item.id, title: 'Move down', onclick: () => move(item, 1) }, '↓') : null,
item.status === 'error' ? h('button', { class: 'btn small primary', onclick: () => command({ command: 'control', action: 'retry', id: item.id }) }, 'Retry') : null,
['unavailable', 'quarantined', 'complete', 'ready'].includes(item.status) ? h('button', {
class: 'btn small', disabled: sourceIsPaused,
title: sourceIsPaused ? 'Source access must be reviewed before another inspection' : '',
onclick: () => command({ command: 'control', action: 'inspect', id: item.id }),
}, 'Inspect again') : null,
item.status === 'ready' ? h('button', { class: 'btn small primary', onclick: () => command({ command: 'control', action: 'confirm', id: item.id }) }, 'Confirm download') : null,
h('button', { class: 'btn small', disabled: running, onclick: async () => {
if (confirm(`Remove ${item.title} from the queue? Its library chapters will stay.`)) await command({ command: 'remove', id: item.id });
} }, 'Remove'));
};
const activeDownloads = h('div', { class: 'scraper-queue' });
const activeVisible = activeEntries.filter(entry => !sourceFilter || entry.source === sourceFilter || entry.current?.source === sourceFilter);
if (!activeVisible.length) activeDownloads.append(h('p', { class: 'muted small' }, state.paused ? 'Queue paused' : 'No active download on this source'));
activeVisible.forEach(entry => {
const job = entry.job, percent = job?.percent || 0;
const problem = job?.lastError || job?.lastProblem;
activeDownloads.append(h('div', { class: 'scraper-item active' }, h('div', { class: 'grow' },
h('b', {}, entry.source || entry.current.source),
h('div', {}, statusText(entry)),
job?.total ? h('div', { class: 'scraper-progress', title: `${job.completed}/${job.total}` }, h('i', { style: `width:${percent}%` })) : null,
job?.total ? h('div', { class: 'muted small' }, `${job.completed}/${job.total} chapters · ${percent}%`) : null,
problem ? h('div', { class: problem.level === 'warning' ? 'small' : 'small error' }, problem.message || String(problem)) : null)));
});
const activityPanel = h('div', { 'data-scraper-tab': 'activity', hidden: route.tab !== 'activity' },
h('h2', {}, 'Activity ', h('span', { class: 'tag' }, state.status)),
h('div', { class: 'scraper-overview' },
h('span', {}, `${activeVisible.length} active`), h('span', {}, `${pending.length} pending`),
h('span', {}, `${attention.length} need attention`)),
h('div', { class: 'field' }, h('span', {}, 'Downloading now'), activeDownloads,
h('div', { class: 'row', style: 'margin-top:8px' }, state.paused
? h('button', { class: 'btn primary', onclick: () => command({ command: 'control', action: 'resume' }) }, 'Resume queue')
: h('button', { class: 'btn', onclick: () => command({ command: 'control', action: 'pause' }) }, 'Pause queue'))),
Object.entries(state.sourcePaused || {}).filter(([, paused]) => paused).map(([source]) =>
h('p', { class: 'small error', role: 'status' }, `${source} is paused after a source-access problem. Confirmed jobs wait; confirmation does not resume this source.`)),
h('div', { class: 'field' }, h('span', {}, `Needs attention · ${attention.length}`),
attention.length ? h('div', { class: 'scraper-queue' }, attention.map(item => jobRow(item)))
: h('p', { class: 'muted small' }, 'Nothing needs action on this source.'),
h('a', { class: 'btn small', href: 'health.html' }, 'Service health & incidents')),
h('div', { class: 'field' }, h('span', {}, 'Up next'), pending.filter(item => item.status !== 'ready').length
? h('div', { class: 'scraper-queue' }, pending.filter(item => item.status !== 'ready').slice(0, 5).map(item => jobRow(item)))
: h('p', { class: 'muted small' }, 'No downloads waiting.')));
const addPanel = h('details', { class: 'scraper-add', 'data-scraper-disclosure': 'add', open: disclosureOpen.add }, h('summary', {}, 'Add manga to queue'),
h('div', { class: 'field' },
h('div', { class: 'row' }, url, title),
h('div', { class: 'row', style: 'margin-top:8px' }, scope, scopeValue, libraryTarget,
h('label', { class: 'row small' }, addSource, 'Add as another source version'),
h('button', { class: 'btn primary', onclick: async () => {
try {
const selectedScope = { mode: scope.value };
if (['last', 'first'].includes(scope.value)) selectedScope.count = +scopeValue.value;
if (['chapter', 'after'].includes(scope.value)) selectedScope.chapter = scopeValue.value;
if (scope.value === 'volume') selectedScope.volume = scopeValue.value;
await command({
command: 'enqueue', url: url.value, title: title.value, scope: selectedScope,
existingPolicy: addSource.checked ? 'add-source' : 'skip-any',
libraryMangaId: libraryTarget.value || undefined,
inspect: true,
});
}
catch (e) { addMessage.textContent = e.message; addMessage.className = 'small error'; }
} }, 'Inspect & queue'), addMessage),
h('div', { class: 'muted small' }, 'Inspection uses the normal source lane and request delays. Confirm the source edition before images are downloaded.'),
h('div', { class: 'muted small' }, 'Existing chapters are never overwritten. Enable alternate versions only when you want the same chapter from this additional host.')));
const queuePanel = h('div', { 'data-scraper-tab': 'queue', hidden: route.tab !== 'queue' },
h('h2', {}, 'Queue'), addPanel,
h('div', { class: 'field' }, h('span', {}, `Pending · ${pending.length}`),
pending.length ? h('div', { class: 'scraper-queue' }, pending.map(item => jobRow(item, true)))
: h('p', { class: 'muted small' }, 'No pending downloads on this source.')),
h('details', { class: 'scraper-completed', 'data-scraper-disclosure': 'completed', open: disclosureOpen.completed }, h('summary', {}, `Completed downloads · ${completed.length}`),
h('p', { class: 'muted small' }, 'Clearing removes finished queue rows only. Library chapters and activity records stay.'),
h('button', { class: 'btn small', disabled: !completed.length, onclick: async () => {
if (confirm(`Clear ${completed.length} completed download job${completed.length === 1 ? '' : 's'}${sourceFilter ? ` from ${sourceFilter}` : ''} from the queue? Library chapters will stay.`)) {
await command({ command: 'clear-completed', source: sourceFilter });
}
} }, `Clear completed downloads (${completed.length})`),
h('div', { class: 'scraper-queue' }, completed.slice().reverse().map(item => jobRow(item)))));
const checkFor = (manga, source) => (state.updateChecks || []).find(item => item.mangaId === manga.id && item.source === source);
const updateRows = (state.library || []).filter(manga => !onlyEnabled || manga.autoUpdate)
.filter(manga => !sourceFilter || manga.sources?.some(source => source.adapter === sourceFilter))
.sort((a, b) => a.title.localeCompare(b.title));
const updatesPanel = h('div', { 'data-scraper-tab': 'updates', hidden: route.tab !== 'updates' },
h('h2', {}, 'Updates'),
h('div', { class: 'field' },
state.libraryStatus?.error ? h('div', { class: 'small error' }, `Library index refresh failed: ${state.libraryStatus.error}`) : null,
state.libraryStatus?.refreshing ? h('div', { class: 'muted small', role: 'status' },
state.libraryStatus.ready ? 'Refreshing library index; showing the previous snapshot.' : 'Loading library index… Queue and rate controls are ready.') : null,
h('div', { class: 'row' },
h('label', { class: 'row' }, autoEnabled, 'Nightly discovery'),
h('label', {}, h('small', {}, 'Local time'), autoTime),
h('button', { class: 'btn', onclick: () => command({ command: 'settings', settings: {
autoUpdateEnabled: autoEnabled.checked, autoUpdateTime: autoTime.value,
} }) }, 'Save schedule'),
h('button', { class: 'btn', onclick: () => command({ command: 'discover' }) }, 'Check now')),
h('div', { class: 'muted small' }, state.nextDiscoveryAt ? `Next check: ${new Date(state.nextDiscoveryAt).toLocaleString()}` : 'Nightly discovery is disabled.'),
h('label', { class: 'row small' }, h('input', { type: 'checkbox', checked: onlyEnabled,
onchange: e => { onlyEnabled = e.target.checked; draw(lastState); } }), 'Only auto-updated manga'),
h('div', { class: 'scraper-update-list' }, updateRows.map(manga => {
const checks = (manga.sources || []).filter(source => source.adapter && (!sourceFilter || source.adapter === sourceFilter))
.map(source => checkFor(manga, source.adapter)).filter(Boolean);
const newest = key => checks.map(check => check[key]).filter(Boolean).sort().at(-1);
const found = newest('lastNewContentAt'), downloaded = newest('lastDownloadedAt');
return h('div', { class: 'scraper-update-row' },
h('label', { class: 'row' }, h('input', { type: 'checkbox', checked: !!manga.autoUpdate,
onchange: e => command({ command: 'auto-update', id: manga.id, enabled: e.target.checked }) }), h('b', {}, manga.title)),
h('span', { class: 'muted small' }, `${manga.status} · ${manga.chapters} chapters`),
h('span', {}, `Last checked: ${when(newest('lastCheckedAt'))}`),
h('span', {}, `Last downloaded: ${when(downloaded)}`),
found && (!downloaded || found > downloaded) ? h('span', { class: 'small saved' }, `New content found ${when(found)}; download pending`) : null);
})),
!updateRows.length ? h('p', { class: 'muted small' }, 'No manga match this filter.') : null,
h('div', { class: 'muted small' }, 'Ongoing manga default to enabled. Completed manga default to disabled. A successful check updates “Last checked”; a published chapter updates “Last downloaded”.')),
h('details', { class: 'scraper-history', 'data-scraper-disclosure': 'history', open: disclosureOpen.history }, h('summary', {}, 'Recent check activity'),
h('div', { class: 'scraper-queue' }, (state.history || []).filter(item => item.status === 'checked' && (!sourceFilter || item.source === sourceFilter)).slice(0, 20).map(item =>
h('div', { class: 'scraper-item small' }, `${item.title} · ${item.source} · checked ${when(item.time)} · no new chapters`)))));
const sourcesPanel = h('div', { 'data-scraper-tab': 'sources', hidden: route.tab !== 'sources' },
h('h2', {}, 'Sources'),
(state.adapters || []).filter(adapter => !sourceFilter || adapter.id === sourceFilter).map(adapter => {
const running = activeEntries.find(entry => entry.source === adapter.id);
const waiting = state.queue.filter(item => item.source === adapter.id && ['queued', 'retry_wait', 'ready'].includes(item.status)).length;
const problems = state.queue.filter(item => item.source === adapter.id && ['error', 'quarantined', 'unavailable'].includes(item.status)).length;
return h('section', { class: 'scraper-source-card' }, h('h3', {}, adapter.label),
h('p', { class: 'muted small' }, `${running ? 'Active' : state.sourcePaused?.[adapter.id] ? 'Paused by source protection' : 'Idle'} · ${waiting} waiting · ${problems} need attention`),
sourceRateEditors.find(editor => editor.querySelector('b')?.textContent === adapter.label));
}), h('p', { class: 'muted small' }, 'Each source runs sequentially with its own limits. New limits apply when that source starts or resumes a manga.'),
h('h2', {}, 'Available-source log'),
h('p', { class: 'muted small' }, 'A queued URL is only a lead. Catalog found, adapter compatibility and download rights are separate facts. This log does not approve or start a download.'),
h('div', { class: 'scraper-queue' }, (state.sourceLedger || []).filter(entry => !sourceFilter || entry.source === sourceFilter).map(entry =>
h('div', { class: 'scraper-item' }, h('div', { class: 'grow' },
h('b', {}, entry.title || entry.sourceTitle || entry.seriesId || 'Unknown title'),
h('div', { class: 'muted small' }, `${entry.source} · catalog ${entry.catalog} · adapter ${entry.adapter} · rights ${entry.rights}`),
h('div', { class: 'muted small' }, `${entry.hostedCount == null ? 'Chapter count unknown' : `${entry.hostedCount} selected chapters`} · series coverage ${entry.coverage || 'unknown'} · checked ${when(entry.lastObservedAt)}`),
entry.sourceTitle && entry.sourceTitle !== entry.title ? h('div', { class: 'small' }, `Source edition: ${entry.sourceTitle}`) : null,
entry.reason ? h('div', { class: 'small' }, `Availability: ${entry.reason}`) : null,
entry.lastFailure ? h('div', { class: 'small error' }, `${entry.lastFailure.kind}: ${entry.lastFailure.message}`) : null),
entry.url && /^https?:\/\//.test(entry.url) ? h('a', { class: 'btn small', href: entry.url, target: '_blank', rel: 'noopener noreferrer' }, 'Source ↗') : null))),
!(state.sourceLedger || []).some(entry => !sourceFilter || entry.source === sourceFilter)
? h('p', { class: 'muted small' }, 'No source observations yet for this filter.') : null);
card.append(activityPanel, queuePanel, updatesPanel, sourcesPanel);
};
async function load(force = false) {
if (!active) return;
try {
const state = await api('api/admin/scraper');
lastState = state;
const focused = card.contains(document.activeElement) && /^(INPUT|SELECT|TEXTAREA)$/.test(document.activeElement.tagName);
// Replacing a live <details> during polling is disruptive on mobile:
// the browser can deliver its toggle after the replacement. Leave the
// disclosure's DOM alone until the user closes it or takes an action.
const readingDisclosure = [...card.querySelectorAll('details[data-scraper-disclosure]')]
.some(details => details.open);
if (force || (!focused && !dirty && !readingDisclosure)) draw(state);
}
catch (error) { draw({ available: false, error: error.message }); }
clearTimeout(timer);
if (active) timer = setTimeout(load, document.hidden ? 15000 : 3000);
}
return {
activate() { active = true; if (lastState && !dirty) draw(lastState); load(); },
deactivate() { active = false; clearTimeout(timer); },
tab() {
card.querySelectorAll('[data-scraper-tab]').forEach(panel => { panel.hidden = panel.getAttribute('data-scraper-tab') !== route.tab; });
card.querySelectorAll('.settings-tabs a').forEach(link => {
if (link.getAttribute('href') === '#admin/scraper/' + route.tab) link.setAttribute('aria-current', 'page');
else link.removeAttribute('aria-current');
});
},
};
}
function adminCard(id) {
const card = h('section', { class: 'card' });
main.append(card);
section(id).append(card);
const draw = data => {
const s = data.scan;
const scanText = s.running ? 'Scanning…' : s.error ? 'Last scan failed: ' + s.error
@@ -80,11 +462,14 @@
const isAdmin = h('input', { type: 'checkbox' });
const msg = h('span', { class: 'small' });
card.textContent = '';
card.append(h('h2', {}, 'Admin'),
if (id === 'admin/library') card.append(h('h2', {}, 'Library maintenance'),
h('div', { class: 'field' }, h('span', {}, 'Library'), h('div', { class: 'muted small' }, s.library), h('div', {}, scanText),
h('div', { class: 'row', style: 'margin-top:8px' },
h('button', { class: 'btn', onclick: rescan(false) }, 'Rescan'),
h('button', { class: 'btn', onclick: rescan(true), title: 'Also re-read all image sizes' }, 'Full rescan'))),
h('p', { class: 'muted small' }, 'Deployment, mounts and service startup are managed by Docker Compose. Library scans do not change download settings.'),
h('a', { class: 'btn', href: 'health.html' }, 'Service health & incidents'));
if (id === 'admin/users') card.append(h('h2', {}, 'User administration'),
h('div', { class: 'field' }, h('span', {}, 'Users'),
h('div', { class: 'users' }, data.users.map(u => h('div', {},
h('span', {}, u.name, u.admin ? h('span', { class: 'tag', style: 'margin-left:8px' }, 'admin') : null),
@@ -99,7 +484,36 @@
catch (e) { msg.textContent = e.message; msg.className = 'small error'; }
} }, 'Save user'), msg)));
};
async function load() { draw(await api('api/admin')); }
load();
async function load() {
try { draw(await api('api/admin')); }
catch (e) { card.textContent = ''; card.append(h('p', { class: 'error', role: 'alert' }, e.message), h('button', { class: 'btn', onclick: load }, 'Try again')); }
}
return { activate: load, deactivate() {} };
}
function navigate(focus = false) {
const next = settingsRoute(location.hash, me.admin);
const entry = SETTINGS_SECTIONS.find(item => item.id === next.id);
if (entry.href) { location.href = entry.href; return; }
if (route.id !== next.id) controllers.get(route.id)?.deactivate();
const changed = route.id !== next.id;
route = next;
shell.select(route.id, focus);
if (!controllers.has(route.id) && route.id.startsWith('admin/')) {
controllers.set(route.id, route.id === 'admin/scraper' ? scraperCard() : adminCard(route.id));
}
shell.content.replaceChildren(section(route.id));
const controller = controllers.get(route.id);
if (changed || !controller?.started) {
if (controller) { controller.started = true; controller.activate(); }
} else controller?.tab?.();
// Normalize unknown/unauthorized routes without creating an extra history entry.
if (location.hash !== '#' + route.id && !location.hash.startsWith('#' + route.id + '/')) {
history.replaceState(null, '', '#' + route.id);
}
}
window.addEventListener('hashchange', () => navigate(true));
window.addEventListener('pagehide', () => controllers.get(route.id)?.deactivate());
window.addEventListener('pageshow', e => { if (e.persisted) controllers.get(route.id)?.activate(); });
navigate();
})();
+144
View File
@@ -0,0 +1,144 @@
'use strict';
const fs = require('fs');
const crypto = require('crypto');
function minute(value, end = false) {
if (end && value === '24:00') return 1440;
if (!/^(?:[01]\d|2[0-3]):[0-5]\d$/.test(value)) throw new Error(`Invalid reading-window time: ${value}`);
return Number(value.slice(0, 2)) * 60 + Number(value.slice(3));
}
function parseSchedule(json, timezone) {
if (!json) return null;
// Validate the IANA zone at startup; a bad deployment must not silently use UTC.
new Intl.DateTimeFormat('en-US', { timeZone: timezone }).format();
const raw = JSON.parse(json);
if (!raw || Array.isArray(raw) || typeof raw !== 'object' ||
Object.keys(raw).some(key => !/^[0-6]$/.test(key))) throw new Error('READING_WINDOWS_JSON must map weekdays 0–6 to time ranges');
const days = Array.from({ length: 7 }, (_, day) => {
const ranges = raw[day] || [];
if (!Array.isArray(ranges)) throw new Error(`Reading windows for day ${day} must be an array`);
return ranges.map(range => {
if (!Array.isArray(range) || range.length !== 2) throw new Error(`Invalid reading window on day ${day}`);
const start = minute(range[0]), end = minute(range[1], true);
if (start === end) throw new Error(`Zero-length reading window on day ${day}`);
return { start, end, label: range };
});
});
return { days, timezone };
}
function localParts(date, timezone) {
const parts = new Intl.DateTimeFormat('en-US', { timeZone: timezone, weekday: 'short', hour: '2-digit', minute: '2-digit', hourCycle: 'h23' })
.formatToParts(date);
const get = type => parts.find(part => part.type === type).value;
return { day: ['Sun', 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat'].indexOf(get('weekday')),
minute: Number(get('hour')) * 60 + Number(get('minute')) };
}
function scheduled(schedule, date = new Date()) {
if (!schedule) return true;
const { day, minute: now } = localParts(date, schedule.timezone);
return schedule.days[day].some(({ start, end }) => end > start && now >= start && now < end || end < start && now >= start) ||
schedule.days[(day + 6) % 7].some(({ start, end }) => end < start && now < end);
}
function publicSchedule(schedule) {
return { enabled: !!schedule, timezone: schedule?.timezone || null,
days: schedule ? schedule.days.map(ranges => ranges.map(({ label }) => [...label])) : [] };
}
function normalizeAnswer(value) {
return String(value || '').normalize('NFKD').replace(/[\u0300-\u036f]/g, '').toLowerCase()
.replace(/[^a-z0-9]+/g, ' ').trim().replace(/\s+/g, ' ');
}
function distance(a, b, cap) {
if (Math.abs(a.length - b.length) > cap) return cap + 1;
let prev = Array.from({ length: b.length + 1 }, (_, i) => i);
for (let i = 1; i <= a.length; i++) {
const row = [i];
for (let j = 1; j <= b.length; j++) row[j] = Math.min(row[j - 1] + 1, prev[j] + 1, prev[j - 1] + (a[i - 1] === b[j - 1] ? 0 : 1));
if (Math.min(...row) > cap) return cap + 1;
prev = row;
}
return prev[b.length];
}
function answerMatches(input, answers) {
const given = normalizeAnswer(input);
if (!given || given.length > 120) return false;
return answers.some(answer => {
const target = normalizeAnswer(answer), tolerance = target.length >= 11 ? 2 : target.length >= 5 ? 1 : 0;
return target && distance(given, target, tolerance) <= tolerance;
});
}
function loadQuestionBank(file) {
if (!file) return {};
const raw = JSON.parse(fs.readFileSync(file, 'utf8'));
if (raw.version !== 1 || !raw.manga || Array.isArray(raw.manga) || typeof raw.manga !== 'object')
throw new Error('Reading quiz catalog must have version 1 and a manga object');
for (const [id, questions] of Object.entries(raw.manga)) {
if (!id || !Array.isArray(questions)) throw new Error('Invalid reading quiz manga entry');
const ids = new Set();
for (const q of questions) {
if (!q || typeof q.id !== 'string' || !q.id || ids.has(q.id) || typeof q.prompt !== 'string' || !q.prompt.trim() ||
!Array.isArray(q.requires) || !q.requires.length || q.requires.some(chapter => typeof chapter !== 'string' || !chapter) ||
typeof q.evidence !== 'string' || !q.evidence.trim() ||
!Array.isArray(q.answers) || !q.answers.length || q.answers.some(a => typeof a !== 'string' || !normalizeAnswer(a)))
throw new Error(`Invalid reading quiz question for ${id}`);
ids.add(q.id);
}
}
return raw.manga;
}
function createReadingAccess({ schedule, bank = {}, unlockMinutes = 60, grants, save, getProgress = () => ({}), getMangaTitle = id => id, now = () => Date.now() }) {
if (!Number.isInteger(unlockMinutes) || unlockMinutes < 1 || unlockMinutes > 1440) throw new Error('READING_QUIZ_UNLOCK_MINUTES must be 1–1440');
const challenges = new Map();
const eligibleQuestions = user => Object.entries(bank).flatMap(([mangaId, questions]) => {
const title = getMangaTitle(mangaId), progress = getProgress(user, mangaId);
if (!title) return [];
return questions.filter(q => q.requires.every(chapter => progress?.[chapter]?.done === true))
.map(q => ({ ...q, mangaTitle: title }));
});
const status = (user, withQuiz = false) => {
const scheduledNow = scheduled(schedule, new Date(now()));
const unlockUntil = Number(grants[user] || 0);
return { allowed: scheduledNow || unlockUntil > now(), scheduled: scheduledNow,
unlockUntil: unlockUntil > now() ? unlockUntil : null,
quizAvailable: withQuiz ? eligibleQuestions(user).length >= 5 : undefined,
unlockMinutes, timezone: schedule?.timezone || null };
};
const publicQuestion = (challenge) => ({ challenge: challenge.id, number: challenge.index + 1, total: 5,
prompt: challenge.questions[challenge.index].prompt, mangaTitle: challenge.questions[challenge.index].mangaTitle });
return {
status,
start(user) {
if (status(user).allowed) throw new Error('Reading is already available');
const pool = eligibleQuestions(user);
if (pool.length < 5) throw new Error('Five progress-eligible questions are not available yet');
const shuffled = [...pool];
for (let i = shuffled.length - 1; i > 0; i--) { const j = crypto.randomInt(i + 1); [shuffled[i], shuffled[j]] = [shuffled[j], shuffled[i]]; }
const challenge = { id: crypto.randomBytes(16).toString('hex'), questions: shuffled.slice(0, 5), index: 0 };
challenges.set(user, challenge); // A new visit starts at question one; older tokens cannot resume.
return publicQuestion(challenge);
},
answer(user, id, answer) {
const challenge = challenges.get(user);
if (!challenge || challenge.id !== id) throw new Error('Challenge expired; start again at question one');
if (!answerMatches(answer, challenge.questions[challenge.index].answers)) return { correct: false, ...publicQuestion(challenge) };
challenge.index++;
if (challenge.index < 5) return { correct: true, ...publicQuestion(challenge) };
challenges.delete(user);
grants[user] = now() + unlockMinutes * 60000;
save();
return { correct: true, complete: true, unlockUntil: grants[user] };
},
abandon(user, id) { if (challenges.get(user)?.id === id) challenges.delete(user); },
};
}
module.exports = { parseSchedule, scheduled, publicSchedule, normalizeAnswer, answerMatches, loadQuestionBank, createReadingAccess };
+300
View File
@@ -0,0 +1,300 @@
'use strict';
const INPUT_TYPES = ['mouse', 'touch', 'keyboard'];
const MAX_RECENT_SESSIONS = 8;
const whole = (value, max = Number.MAX_SAFE_INTEGER) => Math.min(max, Math.max(0, Math.round(Number(value) || 0)));
function rangeSet(ranges, pageCount = Number.MAX_SAFE_INTEGER) {
const out = new Set();
for (const pair of Array.isArray(ranges) ? ranges : []) {
if (!Array.isArray(pair) || pair.length < 2) continue;
const start = whole(pair[0], pageCount - 1), end = whole(pair[1], pageCount - 1);
for (let i = Math.min(start, end); i <= Math.max(start, end); i++) out.add(i);
}
return out;
}
function compactRanges(values) {
const sorted = [...new Set(values)].filter(Number.isInteger).sort((a, b) => a - b);
const ranges = [];
for (const value of sorted) {
const last = ranges[ranges.length - 1];
if (last && value <= last[1] + 1) last[1] = Math.max(last[1], value);
else ranges.push([value, value]);
}
return ranges;
}
function eligiblePages(version, settings) {
const eligible = [];
for (let i = 0; i < version.pages.length; i++) {
const tags = version.pages[i].t || [];
if (settings.skipExtras && tags.includes('extra')) continue;
if (settings.skipCovers && tags.includes('cover')) continue;
eligible.push(i);
}
return eligible.length ? eligible : version.pages.map((_, i) => i);
}
function emptyRecord() {
return {
schema: 1,
totalActiveMs: 0,
firstCompleteMs: null,
lastCompleteMs: null,
loadingMs: 0,
sessions: 0,
completedPasses: 0,
inputMs: { mouse: 0, touch: 0, keyboard: 0 },
fastBursts: 0,
rejectedPageViews: 0,
maxObservedPps: 0,
currentPass: null,
recentSessions: [],
updatedAt: 0,
};
}
function normalizeRecord(record) {
const base = emptyRecord(), out = record && typeof record === 'object' ? record : {};
for (const [key, value] of Object.entries(base)) if (out[key] === undefined) out[key] = value;
out.inputMs = { ...base.inputMs, ...(out.inputMs || {}) };
out.recentSessions = Array.isArray(out.recentSessions) ? out.recentSessions : [];
return out;
}
function publicSummary(record, version, eligible) {
const r = normalizeRecord(record || emptyRecord());
const pass = r.currentPass && r.currentPass.version === version ? r.currentPass : null;
const allowed = new Set(eligible);
const seen = pass ? [...rangeSet(pass.seenRanges)].filter(x => allowed.has(x)) : [];
const predominantInput = INPUT_TYPES.reduce((best, key) => r.inputMs[key] > r.inputMs[best] ? key : best, 'mouse');
const hasInput = INPUT_TYPES.some(key => r.inputMs[key] > 0);
return {
schema: 1,
totalActiveMs: whole(r.totalActiveMs),
firstCompleteMs: r.firstCompleteMs == null ? null : whole(r.firstCompleteMs),
lastCompleteMs: r.lastCompleteMs == null ? null : whole(r.lastCompleteMs),
loadingMs: whole(r.loadingMs),
sessions: whole(r.sessions),
completedPasses: whole(r.completedPasses),
inputMs: Object.fromEntries(INPUT_TYPES.map(key => [key, whole(r.inputMs[key])])),
predominantInput: hasInput ? predominantInput : null,
fastBursts: whole(r.fastBursts),
rejectedPageViews: whole(r.rejectedPageViews),
maxObservedPps: whole(r.maxObservedPps),
eligiblePages: eligible.length,
currentPass: {
version,
activeMs: whole(pass && pass.activeMs),
seenRanges: compactRanges(seen),
seenPages: seen.length,
complete: !!(pass && pass.completedAt),
startedAt: pass ? whole(pass.startedAt) : null,
protocol: pass?.protocol || 1,
requiredPages: Math.ceil(eligible.length * (pass?.protocol === 2 ? 0.6 : 1)),
},
updatedAt: whole(r.updatedAt),
};
}
function chapterInsight(record) {
if (!record || typeof record !== 'object') return null;
const r = normalizeRecord(record);
const totalActiveMs = whole(r.totalActiveMs);
if (!totalActiveMs && !whole(r.completedPasses) && (!whole(r.updatedAt) || r.currentPass?.protocol === 2)) return null;
return {
totalActiveMs,
firstCompleteMs: r.firstCompleteMs == null ? null : whole(r.firstCompleteMs),
lastCompleteMs: r.lastCompleteMs == null ? null : whole(r.lastCompleteMs),
completedPasses: whole(r.completedPasses),
currentPassMs: whole(r.currentPass && r.currentPass.activeMs),
currentPassComplete: !!(r.currentPass && r.currentPass.completedAt),
updatedAt: whole(r.qualifiedUpdatedAt || r.updatedAt),
};
}
function median(values) {
const sorted = values.filter(value => Number.isFinite(value) && value > 0).sort((a, b) => a - b);
if (!sorted.length) return null;
const middle = Math.floor(sorted.length / 2);
return sorted.length % 2 ? whole(sorted[middle]) : whole((sorted[middle - 1] + sorted[middle]) / 2);
}
function mangaInsight(records, chapterIds) {
const source = records && typeof records === 'object' ? records : {};
const ids = Array.isArray(chapterIds) ? chapterIds : Object.keys(source);
const chapters = ids.map(id => ({ id, reading: chapterInsight(source[id]) })).filter(item => item.reading);
const latest = chapters.reduce((best, item) => !best || item.reading.updatedAt > best.reading.updatedAt ? item : best, null);
const completed = chapters.filter(item => item.reading.completedPasses > 0);
return {
activeMs: chapters.reduce((sum, item) => sum + item.reading.totalActiveMs, 0),
timedChapters: chapters.filter(item => item.reading.totalActiveMs > 0).length,
completedChapters: completed.length,
completedPasses: completed.reduce((sum, item) => sum + item.reading.completedPasses, 0),
rereads: completed.reduce((sum, item) => sum + Math.max(0, item.reading.completedPasses - 1), 0),
typicalChapterMs: median(completed.map(item => item.reading.lastCompleteMs)),
lastChapterId: latest ? latest.id : null,
updatedAt: latest ? latest.reading.updatedAt : 0,
};
}
function libraryInsight(mangas) {
const tracked = (Array.isArray(mangas) ? mangas : []).filter(item => item && item.reading && item.reading.activeMs > 0);
const most = tracked.reduce((best, item) => !best || item.reading.activeMs > best.reading.activeMs ? item : best, null);
const latest = tracked.reduce((best, item) => !best || item.reading.updatedAt > best.reading.updatedAt ? item : best, null);
return {
activeMs: tracked.reduce((sum, item) => sum + item.reading.activeMs, 0),
timedChapters: tracked.reduce((sum, item) => sum + item.reading.timedChapters, 0),
completedPasses: tracked.reduce((sum, item) => sum + item.reading.completedPasses, 0),
rereads: tracked.reduce((sum, item) => sum + item.reading.rereads, 0),
timedMangas: tracked.length,
mostTimeMangaId: most ? most.id : null,
lastMangaId: latest ? latest.id : null,
lastChapterId: latest ? latest.reading.lastChapterId : null,
updatedAt: latest ? latest.reading.updatedAt : 0,
};
}
// Cumulative visits tolerate retry and an exit beacon overtaking a regular pulse.
// Coverage belongs to THIS visit, so an old read cannot qualify a picture lookup.
// Legacy totals are preserved; unqualified visit time is a draft, not a statistic.
function applyVisit(record, pulse, { version, pageCount, eligible, now = Date.now() } = {}) {
const r = normalizeRecord(record), session = String(pulse.session || ''), seq = whole(pulse.seq, 1e9);
let receipt = r.recentSessions.find(item => item.id === session);
if (receipt && seq <= whole(receipt.seq)) return { duplicate: true, record: r };
if (!receipt) {
if (!r.qualifiedUpdatedAt && r.totalActiveMs) r.qualifiedUpdatedAt = r.updatedAt;
receipt = { id: session, seq: 0, at: now, pass: {
protocol: 2, visit: session, version, pageCount, seenRanges: [], activeMs: 0,
loadingMs: 0, inputMs: { mouse: 0, touch: 0, keyboard: 0 }, creditedMs: 0,
startedAt: now, completedAt: null,
} };
r.recentSessions.push(receipt);
r.currentPass = receipt.pass;
}
const pass = receipt.pass;
if (!pass || pass.version !== version) return { duplicate: true, record: r };
// Wall-time sanity, not a speed test or the legacy twelve-second packet cap.
const budget = whole(pulse.elapsedMs, 24 * 60 * 60 * 1000) + 1500;
const requestedActive = whole(pulse.activeMs, budget), requestedLoading = whole(pulse.loadingMs, budget);
const scale = requestedActive + requestedLoading > budget ? budget / (requestedActive + requestedLoading) : 1;
const active = Math.max(pass.activeMs, Math.round(requestedActive * scale));
const loading = Math.max(pass.loadingMs, Math.round(requestedLoading * scale));
r.loadingMs += loading - pass.loadingMs;
pass.activeMs = active; pass.loadingMs = loading;
const input = Object.fromEntries(INPUT_TYPES.map(key => [key, whole(pulse.inputMs?.[key], active)]));
const inputTotal = INPUT_TYPES.reduce((sum, key) => sum + input[key], 0);
const inputScale = inputTotal > active ? active / inputTotal : 1;
for (const key of INPUT_TYPES) pass.inputMs[key] = Math.max(pass.inputMs[key], Math.round(input[key] * inputScale));
const allowed = new Set(eligible), seen = rangeSet(pass.seenRanges, pageCount);
for (const page of Array.isArray(pulse.seen) ? pulse.seen.slice(0, pageCount) : []) {
if (Number.isInteger(page) && allowed.has(page)) seen.add(page);
}
pass.seenRanges = compactRanges(seen);
const covered = eligible.filter(page => seen.has(page)).length;
if (!pass.completedAt && eligible.length && covered >= Math.ceil(eligible.length * 0.6) && active > 0) {
pass.completedAt = now;
pass.ordinal = ++r.completedPasses;
r.sessions++;
}
if (pass.completedAt) {
r.totalActiveMs += active - pass.creditedMs;
pass.creditedMs = active;
pass.creditedInput = pass.creditedInput || { mouse: 0, touch: 0, keyboard: 0 };
for (const key of INPUT_TYPES) {
r.inputMs[key] += pass.inputMs[key] - pass.creditedInput[key];
pass.creditedInput[key] = pass.inputMs[key];
}
// Duration keeps growing through the last page and departure, not just 60%.
if (pass.ordinal === 1) r.firstCompleteMs = active;
if (pass.ordinal === r.completedPasses) r.lastCompleteMs = active;
r.qualifiedUpdatedAt = now;
}
pass.closed = !!pulse.final;
receipt.seq = seq; receipt.at = now;
if (r.currentPass?.visit === session) r.currentPass = pass;
r.recentSessions.sort((a, b) => whole(b.at) - whole(a.at));
r.recentSessions.splice(MAX_RECENT_SESSIONS);
r.updatedAt = now;
return { duplicate: false, record: r };
}
function applyPulse(record, pulse, { version, pageCount, eligible, now = Date.now(), maxDeltaMs = 12000 } = {}) {
const r = normalizeRecord(record);
const session = String(pulse.session || '');
const seq = whole(pulse.seq, 1e9);
let receipt = r.recentSessions.find(item => item.id === session);
if (receipt && seq <= whole(receipt.seq, 1e9)) return { duplicate: true, record: r };
const newSession = !receipt;
if (!receipt) {
receipt = { id: session, seq: 0, at: now, counted: false };
r.recentSessions.push(receipt);
}
const elapsed = newSession ? maxDeltaMs : Math.max(0, now - whole(receipt.at));
const allowedDelta = Math.min(maxDeltaMs, elapsed + 1500);
const requestedActive = whole(pulse.activeMs, maxDeltaMs * 2);
const requestedLoading = whole(pulse.loadingMs, maxDeltaMs * 2);
const requestedTotal = requestedActive + requestedLoading;
const scale = requestedTotal > allowedDelta && requestedTotal ? allowedDelta / requestedTotal : 1;
const activeMs = Math.round(requestedActive * scale);
const loadingMs = Math.round(requestedLoading * scale);
receipt.seq = seq;
receipt.at = now;
if (!receipt.counted && activeMs > 0) { receipt.counted = true; r.sessions++; }
r.recentSessions.sort((a, b) => whole(b.at) - whole(a.at));
r.recentSessions.splice(MAX_RECENT_SESSIONS);
const prior = r.currentPass;
if (!prior || prior.version !== version || (prior.completedAt && newSession)) {
r.currentPass = {
version,
pageCount,
seenRanges: [],
activeMs: 0,
startedAt: now,
completedAt: null,
};
}
const pass = r.currentPass;
pass.pageCount = pageCount;
pass.activeMs = whole(pass.activeMs) + activeMs;
r.totalActiveMs = whole(r.totalActiveMs) + activeMs;
r.loadingMs = whole(r.loadingMs) + loadingMs;
const rawInput = pulse.inputMs && typeof pulse.inputMs === 'object' ? pulse.inputMs : {};
const requestedInput = Object.fromEntries(INPUT_TYPES.map(key => [key, whole(rawInput[key], maxDeltaMs * 2)]));
const inputTotal = INPUT_TYPES.reduce((sum, key) => sum + requestedInput[key], 0);
const inputScale = inputTotal > activeMs && inputTotal ? activeMs / inputTotal : 1;
for (const key of INPUT_TYPES) r.inputMs[key] = whole(r.inputMs[key]) + Math.round(requestedInput[key] * inputScale);
const allowedPages = new Set(eligible);
const seen = rangeSet(pass.seenRanges, pageCount);
for (const raw of Array.isArray(pulse.seen) ? pulse.seen.slice(0, pageCount) : []) {
const page = Number(raw);
if (Number.isInteger(page) && allowedPages.has(page)) seen.add(page);
}
pass.seenRanges = compactRanges([...seen]);
r.fastBursts = whole(r.fastBursts) + whole(pulse.fastBursts, 100);
r.rejectedPageViews = whole(r.rejectedPageViews) + whole(pulse.rejectedPageViews, pageCount * 2);
r.maxObservedPps = Math.max(whole(r.maxObservedPps), whole(pulse.maxPps, 1000));
if (!pass.completedAt && eligible.length && eligible.every(page => seen.has(page))) {
pass.completedAt = now;
r.completedPasses = whole(r.completedPasses) + 1;
r.lastCompleteMs = pass.activeMs;
if (r.firstCompleteMs == null) r.firstCompleteMs = pass.activeMs;
}
r.updatedAt = now;
return { duplicate: false, record: r, activeMs, loadingMs };
}
module.exports = {
INPUT_TYPES, applyPulse, applyVisit, chapterInsight, compactRanges, eligiblePages, emptyRecord,
libraryInsight, mangaInsight, publicSummary, rangeSet,
};
+71
View File
@@ -0,0 +1,71 @@
'use strict';
// Shared by the scanner and publisher. Dimensions identify strip layout, not
// language or the reading order of a printed page. RTL is only a default.
const KEY = 'x-reading-direction';
const aliases = { rtl: 'rtl', ltr: 'ltr', ttb: 'ttb', vertical: 'ttb', webtoon: 'ttb' };
const normalizeDirection = value => aliases[String(value || '').trim().toLowerCase()] || null;
function manualDirection(meta = {}) {
const override = normalizeDirection(meta.directionOverride);
if (override) return override;
const value = normalizeDirection(meta.direction);
if (!value) return null;
const record = meta[KEY];
// Legacy plain directions are ambiguous: never assume a human didn't set one.
// Editing a generated value without its provenance is also a manual override.
if (!record || record.version !== 1 || record.value !== value ||
!['detected', 'fallback'].includes(record.source)) return value;
return null;
}
function detectVerticalStrip(pages = []) {
const dimensions = pages.map(p => {
const rotated = p.r === 90 || p.r === 270;
return { w: Number(rotated ? p.h : p.w), h: Number(rotated ? p.w : p.h) };
}).filter(p => Number.isFinite(p.w) && Number.isFinite(p.h) && p.w > 0 && p.h > 0);
const tall = dimensions.filter(p => p.h / p.w >= 4);
const widths = tall.map(p => p.w).sort((a, b) => a - b);
const middle = widths.length ? widths[Math.floor(widths.length / 2)] : 0;
const consistent = tall.filter(p => Math.abs(p.w - middle) <= middle * .25).length;
const evidence = { pages: pages.length, knownPages: dimensions.length, tallPages: tall.length,
consistentTallPages: consistent, medianStripWidth: middle };
// At least three coherent strips, >=75% of known pages, and >=75% known
// dimensions. A landscape credits card may coexist with the vertical body.
const detected = tall.length >= 3 && dimensions.length >= pages.length * .75 &&
tall.length >= dimensions.length * .75 && consistent >= tall.length * .75;
return detected ? { value: 'ttb', source: 'detected', evidence } : null;
}
function fallbackDirection() {
return { direction: 'rtl', [KEY]: { version: 1, value: 'rtl', source: 'fallback' } };
}
// Scopes are chapter -> book -> manga (or existing -> incoming when publishing).
// Any explicit choice beats inference; a more specific explicit choice wins.
function resolveReadingDirection(scopes = [], pages = []) {
for (let i = 0; i < scopes.length; i++) {
const value = manualDirection(scopes[i]);
if (value) return { value, source: 'manual', scope: i };
}
const detected = detectVerticalStrip(pages);
if (detected) return detected;
for (let i = 0; i < scopes.length; i++) {
const meta = scopes[i] || {}, record = meta[KEY];
if (record?.version === 1 && record.source === 'detected' &&
record.value === normalizeDirection(meta.direction) && record.value === 'ttb') {
return { value: 'ttb', source: 'detected', scope: i, evidence: record.evidence };
}
}
return { value: 'rtl', source: 'fallback' };
}
function directionFields(scopes, pages) {
const { value, source, evidence } = resolveReadingDirection(scopes, pages);
return { direction: value,
...(source === 'manual' && scopes.some(m => normalizeDirection(m?.directionOverride)) ? { directionOverride: value } : {}),
[KEY]: { version: 1, value, source, ...(evidence ? { evidence } : {}) } };
}
module.exports = { KEY, normalizeDirection, manualDirection, detectVerticalStrip,
fallbackDirection, resolveReadingDirection, directionFields };
+7
View File
@@ -0,0 +1,7 @@
'use strict';
const fs = require('fs');
function secret(name) {
if (process.env[`${name}_FILE`]) return fs.readFileSync(process.env[`${name}_FILE`], 'utf8').trim();
return process.env[name] || '';
}
module.exports = { secret };
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env node
'use strict';
const { Supervisor } = require('./supervisor');
const supervisor = new Supervisor(process.argv[2] || 'reader');
supervisor.start();
const keepalive = setInterval(() => {}, 60000);
let closing = false;
for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, async () => {
if (closing) return; closing = true; clearInterval(keepalive); await supervisor.close(); process.exit(0);
});
+64
View File
@@ -0,0 +1,64 @@
'use strict';
const fs = require('fs'), path = require('path'), { spawn } = require('child_process');
const SPECS = {
reader: { script: 'server.js', port: () => process.env.PORT || 8080, file: () => path.join(process.env.DATA || 'data', 'runtime-reader.json') },
scraper: { script: 'scraper/daemon.js', port: () => process.env.SCRAPER_PORT || 8091, file: () => path.join(process.env.SCRAPER_DATA || path.join(process.env.DATA || 'data', 'scraper'), 'runtime.json') },
sentinel: { script: 'sentinel/daemon.js', port: () => process.env.SENTINEL_PORT || 8092, file: () => path.join(process.env.SENTINEL_DATA || path.join(process.env.DATA || 'data', 'sentinel'), 'runtime.json') },
};
class Supervisor {
constructor(name, options = {}) {
this.name = name; this.spec = SPECS[name]; if (!this.spec) throw new Error('Unknown service');
this.root = options.root || path.resolve(__dirname, '..'); this.child = null; this.stopped = false; this.timer = null;
this.healthFailures = 0; this.restarting = false; this.startedAt = 0;
this.spawn = options.spawn || spawn; this.retryBaseMs = options.retryBaseMs ?? 5000;
this.file = options.file || path.resolve(this.root, this.spec.file()); fs.mkdirSync(path.dirname(this.file), { recursive: true });
this.state = { failures: [], status: 'starting' };
if (fs.existsSync(this.file)) this.state = JSON.parse(fs.readFileSync(this.file, 'utf8'));
}
save() { fs.writeFileSync(this.file + '.tmp', JSON.stringify(this.state, null, 2) + '\n'); fs.renameSync(this.file + '.tmp', this.file); }
log(event, extra = {}) { console.log(JSON.stringify({ service: 'supervisor', target: this.name, event, ...extra })); }
start() {
if (this.stopped) return;
this.state.failures = this.state.failures.filter(at => Date.now() - at < 3600000);
if (this.state.failures.length >= 3) { this.state.status = 'parked'; this.save(); this.log('repair-required', { reason: 'Three failures within one hour' }); return; }
this.child = this.spawn(process.execPath, [this.spec.script], { cwd: this.root, env: process.env, stdio: 'inherit', windowsHide: true });
this.state.status = 'running'; this.state.pid = this.child.pid; this.startedAt = Date.now(); this.healthFailures = 0; this.restarting = false; this.save();
this.log('started', { pid: this.child.pid });
const child = this.child;
child.on('error', e => this.log('spawn-error', { message: e.message }));
child.once('close', (code, signal) => {
if (this.child !== child) return;
this.child = null; clearTimeout(this.killTimer);
if (this.stopped) return;
clearTimeout(this.timer);
this.state.failures.push(Date.now()); this.state.status = 'retry_wait'; this.state.lastExit = { at: new Date().toISOString(), code, signal }; this.save();
this.log('exited', { code, signal });
this.timer = setTimeout(() => this.start(), Math.min(60000, this.retryBaseMs * 2 ** (this.state.failures.length - 1)));
});
this.timer = setTimeout(() => this.check(), 30000);
}
async check() {
if (this.stopped || !this.child) return;
const observedChild = this.child;
const healthy = await fetch(`http://127.0.0.1:${this.spec.port()}/health`, { signal: AbortSignal.timeout(5000) }).then(async r => r.ok && (await r.json()).ok !== false).catch(() => false);
if (this.stopped || this.child !== observedChild) return;
this.healthFailures = healthy ? 0 : this.healthFailures + 1;
if (this.healthFailures >= 3 && Date.now() - this.startedAt > 90000 && !this.restarting) {
this.restarting = true; this.log('liveness-recovery');
const child = this.child; child.kill('SIGTERM');
this.killTimer = setTimeout(() => { if (this.child === child) child.kill('SIGKILL'); }, 65000);
}
if (!this.stopped) this.timer = setTimeout(() => this.check(), 30000);
}
async close() {
this.stopped = true; clearTimeout(this.timer); clearTimeout(this.killTimer);
if (!this.child) return;
const child = this.child;
await new Promise(resolve => {
const timer = setTimeout(() => child.kill('SIGKILL'), 65000);
child.once('exit', () => { clearTimeout(timer); resolve(); }); child.kill('SIGTERM');
});
this.state.status = 'stopped'; this.save();
}
}
module.exports = { Supervisor, SPECS };
+155
View File
@@ -0,0 +1,155 @@
# MangaCircus scraper agent runbook
This document is normative for automated agents operating the scraper. Read it together with `../docs/scraper-guide.md`.
## 1. Non-negotiable safety policy
1. Scrape only works the operator is authorised to copy.
2. Never bypass a CAPTCHA, Cloudflare challenge, login, paywall, geographic restriction, signed-URL check, or other access control.
3. If a normal browser session displays a challenge, stop. A human may complete it normally; the agent may resume afterward. Repeated challenges end the run.
4. Never request pages concurrently. Use the persisted per-host settings. The current operator-selected fast profile is 1–4 seconds between requests/images and 5–15 seconds between chapters. A host-specific published limit overrides these settings when it is slower.
5. Stop immediately on HTTP 401, 403, or 429. On 5xx, retry at most twice after 30 and 60 seconds. Do not rotate IP addresses, identities, user agents, or mirrors to evade a block.
6. Inspect metadata through AniList and its local 30-day cache. Do not revisit a reader/image host merely for authors, genres, descriptions, status, year, or cover metadata.
7. Download only missing chapters. Never probe every historical chapter merely to discover the newest one when a chapter list is available.
8. Publish only through the shared publisher, either `scraper/cli.js publish` or the
managed daemon. Both validate images and use MangaCircus-safe staging and rename
operations. Legacy conversion must be paused, validate every completed chapter,
quarantine incomplete stages, and use an atomic manga-folder rename.
## 2. Pipeline
Runtime recovery is documented in `../docs/resilience.md`. Request-level 5xx retries
remain limited to 30/60 seconds; after those are exhausted the daemon can resume the
same checkpoint once after a five-minute source cooldown. No automatic access-error
retry is permitted. Unavailable titles and parser failures are job-scoped; repeated
independent parser failures stop the source lane. Sentinel never changes adapters.
```text
public metadata -> cached series metadata
chapter host -> newest chapter IDs only
reader/browser -> ordered page manifest or local browser asset bundle
manifest validation -> slow sequential acquisition -> hidden staging folder
complete chapter -> one atomic rename -> optional one MangaCircus rescan per batch
```
The interchange format is `manifest.example.json`. Each page has exactly one of:
- `path`: an already downloaded browser asset; preferred for protected/browser-only hosts.
- `url`: a public image URL fetched sequentially by the publisher.
Do not store cookies, tokens, signed URLs, account identifiers, or other secrets in `manga.json`. Temporary manifests containing expiring URLs must stay outside the library and should be deleted after a successful import.
## 3. Metadata procedure
Run one AniList lookup per new series:
```bash
node scraper/cli.js metadata-anilist "series title"
```
Verify the result using title, native title, author and year. Copy the returned fields into `series.metadata`. Keep the chapter host URLs only in `series.sources`. If the lookup is ambiguous, stop for human selection; never silently attach metadata from a similarly named work.
For the selected Heavy Knight manga, the verified AniList identity is `150193` / MAL `147799`. AniList `150194` is the light novel and MUST NOT be used for manga metadata or cover art. For One Piece it is AniList `30013` / MAL `13`.
## 4. WeebCentral adapter
Observed 2026-10-01:
- Series: `https://weebcentral.com/series/<id>`
- Full list: `/series/<id>/full-chapter-list`
- Chapter: `/chapters/<id>`
- Page fragment: `/chapters/<id>/images?is_prev=False&reading_style=long_strip&current_page=1`
The fragment contains the ordered image elements. Fetch the series page once, the full list once, and only the selected chapter/page fragments. Current chapter 177 exposed 15 PNG pages. Do not fetch cover art from this host when AniList metadata is available.
```bash
node scraper/cli.js discover-weebcentral "SERIES_URL" --limit 3 --out manifest.json
node scraper/cli.js publish manifest.json --library library
```
If the HTML no longer matches, stop and use a normal browser to inspect the visible reader. Do not guess endpoint grids.
## 5. ReiManga adapter
Observed 2026-10-01: anonymous scripted requests receive a managed Cloudflare challenge. The indexed series page lists chapters 177 downward, but that index is not sufficient to acquire pages.
Required approach:
1. Open the supplied series page in a normal browser.
2. If a challenge appears, stop for the human. Do not automate it.
3. After the user reaches the normal series page, open only the selected chapter.
4. Read the visible chapter number/title/date and inventory the image assets already rendered by the page.
5. Export only assets belonging to that chapter into a local temporary folder.
6. Create a manifest with local `path` entries and source name `ReiManga`.
7. Publish locally. Wait 30–60 seconds before opening another chapter.
If the site sends the chapter to another domain, record the ReiManga chapter URL as the source URL, but accept assets only from domains actually loaded by that reader page. Any login, paywall, error, or renewed challenge ends the run.
## 5A. MangaDex adapter
Observed 2026-10-03:
- Title: `https://mangadex.org/title/<uuid>`
- API: `https://api.mangadex.org`
- English feed: `/manga/<uuid>/feed`
- Page manifest: `/at-home/server/<chapter-uuid>`
- Images: the HTTPS MangaDex@Home host returned by that manifest
Only explicit MangaDex title UUIDs may enter the managed queue. Never automatically replace a queued source from fuzzy title search: verify the MangaDex AniList/MAL link against the selected work first. Select one stable release per numeric English chapter, credit MangaDex and the scanlation group in the chapter source, and ignore external-link or zero-page releases. Honor group removal requests. MangaDex-backed content must not be used to support advertising or paid access.
Unnumbered hosted releases are supported with `special-mangadex-<chapter-uuid>` keys
and human labels, never guessed chapter numbers. Keep distinct source UUIDs separate.
Use the queued catalog-only inspection and confirm the canonical edition before
downloading a new title. Empty hosted-English catalogs distinguish external-only,
no-English-listed and no-hosted-pages; none is an access error or a reason to probe
external hosts. A finished job does not prove whole-series coverage. See
`../docs/scraper-service.md` for inspection, scopes and availability semantics.
Use full-resolution `data` pages, validate that the manifest page count still matches the feed, and accept only HTTPS `uploads.mangadex.org` or `*.mangadex.network` image hosts. Keep one sequential MangaDex worker using its configured per-host rate (currently 1–4 seconds between requests and 5–15 seconds between chapters). Stop on 401, 403, or 429 exactly as for other sources; never add authentication or rotate hosts to evade a response.
## 6. MANGA Plus adapter
Observed 2026-10-01 for title `100020` (One Piece): the title page states that only the latest three chapters remain viewable. They are:
| Chapter | Viewer id | Published | Title |
|---|---:|---|---|
| 1192 | 1030379 | 2026-09-06 | We Won't Stand For It!!! |
| 1193 | 1030380 | 2026-09-13 | Still Practicing |
| 1194 | 1030381 | 2026-09-27 | All Things Are In Flux |
The normal web app calls `title_detailV4` and `manga_viewer_v3`, then renders signed `jumpg-assets*.tokyo-cdn.com` payloads. Direct non-browser API calls currently return an account-ban response. A browser asset export on 2026-10-01 produced payloads labelled `image/jpeg`, but their bytes were obfuscated and failed JPEG signature validation. Therefore:
1. Use only the normal MANGA Plus browser UI.
2. Read the title page and select the last three chapters explicitly marked viewable.
3. Open one viewer at a time, wait for its page count to stabilise, and inventory the rendered assets.
4. Export only URLs matching `/secure/title/100020/chapter/<viewer-id>/manga_page/` from that page's observed assets.
5. Validate the first exported payload by file signature before exporting or processing the rest. A `.jpg` name or HTTP content type is not proof of a JPEG.
6. If valid images are available, preserve numeric order and use local `path` entries in the import manifest. Set source name `MANGA Plus`, source URL `https://mangaplus.shueisha.co.jp/viewer/<viewer-id>`, priority `1`, direction `rtl`.
7. If payloads are obfuscated, abort the source. Do not decrypt them, extract pixels from the rendering canvas, or reconstruct chapters with automated screenshots.
8. Wait at least 30 seconds between chapters. If the UI says a chapter is unavailable, do not attempt alternate APIs or URLs.
Do not implement API protobuf decoding, image deobfuscation, signed-URL generation, device-id synthesis, canvas extraction, screenshot reconstruction, or retry identity rotation. The browser-exported file boundary plus ordinary image validation is the allowed boundary. As of the observation date, chapters 1192–1194 are identified but **not importable by this compliant adapter**.
## 7. MangaCircus publication rules
- Use the flat book strategy: `v01`.
- Chapter folders include stable source suffixes, for example `c1194 All Things Are In Flux [MANGA Plus]`.
- Existing chapters are skipped by default. `--replace` replaces the entire version folder atomically; never overwrite image files in place.
- New manga are built wholly under `.staging-mcscraper-*` and published with one rename.
- The publisher sniffs image bytes; filename extensions are never trusted.
- Page numbers are contiguous from `0001`.
- Add `cover`, `double`, `extra`, or rotation tags only when certain. Do not infer tags from aspect ratio alone.
- Trigger at most one MangaCircus rescan after the whole batch.
## 8. Abort checklist
Abort without workaround when any of these occurs:
- CAPTCHA, challenge, access-denied, rate-limit, account-ban, or login/paywall prompt.
- Page count changes during acquisition.
- Duplicate/non-numeric page order, unsupported image bytes, or an HTML error page disguised as an image.
- Metadata match is ambiguous.
- The target manga folder exists but belongs to another source identity.
- A chapter would need to move books or rename an existing version folder.
Log the URL, status, and phase, but never log cookies, query signatures, passwords, or browser storage.
+48
View File
@@ -0,0 +1,48 @@
# MangaCircus scraper
The scraper uses small host adapters to create a common manifest, followed by one conservative publisher that writes the MangaCircus library atomically.
Start with [AGENT_RUNBOOK.md](AGENT_RUNBOOK.md). It contains the host-specific procedures, safety limits, and stop conditions. `manifest.example.json` documents the interchange format.
The historical human-readable wishlist is [WISHLIST.md](WISHLIST.md). Once the
managed daemon has started, `data/scraper/queue.json` is the authoritative queue and
is edited through the admin UI.
## Managed service
Run MangaCircus and the scraper daemon together:
```bash
npm run start:all
```
Or run them as two independently supervised processes:
```bash
npm start
npm run start:scraper
```
The daemon binds to `127.0.0.1:8091` by default, keeps its durable queue in
`data/scraper/queue.json`, and resumes incomplete archive checkpoints. An admin can
control it from **Settings → Scraper**. Set `SCRAPER_TOKEN` on both processes when
the daemon is not restricted to loopback.
Queue schema version 3 supports scoped jobs, multiple jobs for the same upstream
series, one sequential worker per source, incremental chapter publication, library
duplicate/version checks, and configurable nightly update discovery. See
[`docs/scraper-service.md`](../docs/scraper-service.md) for the service contract and
[`docs/local-deployment.md`](../docs/local-deployment.md) for the currently running
Windows Scheduled Task deployment. On that machine the scheduled task owns the single
supervisor; do not run `npm run start:all` beside it.
```bash
npm run test:scraper
node scraper/cli.js metadata-anilist "One Piece"
node scraper/cli.js discover-weebcentral "https://weebcentral.com/series/..." --limit 3 --out manifest.json
node scraper/cli.js publish manifest.json --library library --metadata-search "Series title"
node scraper/cli.js archive-weebcentral "https://weebcentral.com/series/..." --library library --metadata-search "Series title"
node scraper/cli.js status-weebcentral "https://weebcentral.com/series/..." --library library
```
The implementation uses only Node.js built-ins. Browser-only sources must first export observed page assets to local files; the publisher consumes those files and does not automate access-control challenges.
+24
View File
@@ -0,0 +1,24 @@
'use strict';
// First-party registration only; never load executable source code from a
// remote catalog or scan a writable directory for modules.
const adapters = Object.freeze([
require('../sources/weebcentral'),
require('../sources/mangadex'),
require('../sources/mangapill'),
]);
const { adapterForPeer } = require('../../peering/adapter');
function adapterFor(url) {
const peer = adapterForPeer(url);
if (peer) return peer;
for (const adapter of adapters) {
const info = adapter.match(url);
if (info) return { adapter, info };
}
return null;
}
module.exports = { adapters, adapterFor,
weebCentralInfo: require('./weebcentral').weebCentralInfo,
mangaDexInfo: require('./mangadex').mangaDexInfo };
+85
View File
@@ -0,0 +1,85 @@
'use strict';
const API = 'https://api.mangadex.org';
const { parse } = require('../plugins/registry');
const { localizedTitle, chapterNumber, selectChapterReleases, availabilitySummary, pageUrls } = require('../plugins/mangadex');
function mangaDexInfo(value) {
let url;
try { url = new URL(String(value)); } catch { return null; }
if (url.protocol !== 'https:' || !['mangadex.org', 'www.mangadex.org'].includes(url.hostname)) return null;
const match = /^\/title\/([0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12})(?:\/([^/?#]+))?\/?$/i.exec(url.pathname);
if (!match) return null;
const id = match[1].toLowerCase();
const title = decodeURIComponent(match[2] || id).replace(/[-_]+/g, ' ').trim();
return { source: 'mangadex', id, title, url: `https://mangadex.org/title/${id}${match[2] ? `/${match[2]}` : ''}` };
}
async function jsonGet(http, url) {
const response = await http.get(url, { accept: 'application/json' });
const body = await response.json();
if (body.result && body.result !== 'ok') throw new Error(`MangaDex API result: ${body.result}`);
return body;
}
async function discoverMangaDexCatalog(seriesUrl, http) {
const info = mangaDexInfo(seriesUrl);
if (!info) throw new Error('Invalid MangaDex title URL');
const manga = (await jsonGet(http, `${API}/manga/${info.id}`)).data;
if (!manga || manga.type !== 'manga') throw new Error('MangaDex title was not found');
const metadata = parse('mangadex', 'metadata', { data: manga, url: `${API}/manga/${info.id}` });
const releases = [];
let offset = 0;
for (;;) {
const params = new URLSearchParams({ limit: '100', offset: String(offset), includeFutureUpdates: '0', includeExternalUrl: '0' });
params.append('translatedLanguage[]', 'en');
params.append('includes[]', 'scanlation_group');
params.append('order[chapter]', 'asc');
const page = await jsonGet(http, `${API}/manga/${info.id}/feed?${params}`);
releases.push(...parse('mangadex', 'feed', { data: page.data || [], url: `${API}/manga/${info.id}/feed` }).releases);
offset += page.data?.length || 0;
if (!page.data?.length || offset >= Number(page.total || 0)) break;
}
const { title, specialLabel } = metadata;
const parseCatalog = externalReleases => parse('mangadex', 'catalog', {
data: { attributes: metadata.availabilityAttributes, releases, specialLabel, externalReleases },
url: `${API}/manga/${info.id}`,
});
let parsed = parseCatalog(null);
const chapters = parsed.chapters;
// External-only is a separate feed mode, not a combined hosted+external catalog.
let externalReleases = null;
if (!chapters.length) {
const params = new URLSearchParams({ limit: '1', includeFutureUpdates: '0', includeExternalUrl: '1' });
params.append('translatedLanguage[]', 'en');
externalReleases = Number((await jsonGet(http, `${API}/manga/${info.id}/feed?${params}`)).total || 0);
parsed = parseCatalog(externalReleases);
}
return {
id: info.id,
title,
originalLanguage: metadata.originalLanguage,
links: metadata.links,
availability: parsed.availability,
chapters,
};
}
async function discoverMangaDexChapter(ref, http) {
const atHome = await jsonGet(http, `${API}/at-home/server/${ref.id}?forcePort443=true`);
const groupName = ref.group?.name ? ` · ${ref.group.name}` : '';
const parsed = parse('mangadex', 'atHome', { data: { response: atHome, chapterUrl: ref.url, expectedPages: ref.pages }, url: `${API}/at-home/server/${ref.id}` });
return {
number: ref.number, title: ref.title || '', added: ref.added,
...(ref.special ? { special: true, label: ref.label, sourceId: ref.id, sourceNumber: ref.sourceNumber ?? null } : {}),
source: { name: `MangaDex${groupName}`, url: ref.url },
pages: parsed.pages,
};
}
module.exports = {
API, mangaDexInfo, localizedTitle, chapterNumber, selectChapterReleases, availabilitySummary,
discoverMangaDexCatalog, discoverMangaDexChapter, pageUrls,
};
+35
View File
@@ -0,0 +1,35 @@
'use strict';
const { parse } = require('../plugins/registry');
function mangaPillInfo(value) {
let url;
try { url = new URL(String(value)); } catch { return null; }
if (url.protocol !== 'https:' || url.hostname !== 'mangapill.com' || url.username || url.password || url.port) return null;
const match = /^\/manga\/(\d+)\/([a-z0-9-]+)\/?$/i.exec(url.pathname);
if (!match) return null;
return { source: 'mangapill', id: match[1], title: match[2].replace(/-/g, ' '),
url: `https://mangapill.com/manga/${match[1]}/${match[2]}` };
}
async function discoverMangaPillCatalog(seriesUrl, http) {
const html = await (await http.get(seriesUrl, { accept: 'text/html' })).text();
const parsed = parse('mangapill', 'catalog', { html, url: seriesUrl });
const chapters = parsed.chapters.sort((a, b) => Number(a.number) - Number(b.number));
return { ...parsed, chapters,
...(parsed.chapterLinks && !chapters.length ? { availability: { reason: 'unrecognized-labels',
message: `MangaPill lists ${parsed.chapterLinks} chapter links, but their numbering labels are not recognized.` } } : {}),
};
}
async function discoverMangaPillChapter(ref, http) {
const html = await (await http.get(ref.url, { accept: 'text/html' })).text();
const parsed = parse('mangapill', 'pages', { html, url: ref.url });
if (!parsed.pages.length) throw new Error(`No pages found for MangaPill chapter ${ref.number}`);
return { number: ref.number, title: ref.title || '',
source: { name: 'MangaPill', url: ref.url },
pages: parsed.pages.map(url => ({ url, headers: { referer: ref.url } })),
};
}
module.exports = { mangaPillInfo, discoverMangaPillCatalog, discoverMangaPillChapter };
+72
View File
@@ -0,0 +1,72 @@
'use strict';
const { SlowHttp } = require('../lib/http');
const { fallbackDirection } = require('../../reading-direction');
const parser = require('../plugins/weebcentral');
const { parse } = require('../plugins/registry');
function weebCentralInfo(value) {
let url;
try { url = new URL(String(value)); } catch { return null; }
if (url.protocol !== 'https:' || url.hostname !== 'weebcentral.com') return null;
const match = /^\/series\/([A-Z0-9]+)(?:\/([^/?#]+))?\/?$/i.exec(url.pathname);
if (!match) return null;
const title = decodeURIComponent(match[2] || match[1]).replace(/[-_]+/g, ' ').trim();
return { source: 'weebcentral', id: match[1].toUpperCase(), title, url: `https://weebcentral.com/series/${match[1]}${match[2] ? `/${match[2]}` : ''}` };
}
const parseChapters = (html, baseUrl) => parser.catalog({ html, url: baseUrl }).chapters;
const parsePageUrls = (html, baseUrl) => parser.pages({ html, url: baseUrl }).pages;
async function discoverWeebCatalog(seriesUrl, http) {
const seriesHtml = await (await http.get(seriesUrl, { accept: 'text/html' })).text();
const { title, listUrl } = parse('weebcentral', 'series', { html: seriesHtml, url: seriesUrl });
const listHtml = listUrl ? await (await http.get(listUrl, { accept: 'text/html', catalog: true })).text() : seriesHtml;
const parsed = parse('weebcentral', 'catalog', { html: listHtml, url: seriesUrl });
const chapterLinks = parsed.chapterLinks;
const chapters = parsed.chapters.sort((a, b) =>
Number(a.book?.number || 1) - Number(b.book?.number || 1)
|| Number(a.releaseKind === 'volume') - Number(b.releaseKind === 'volume')
|| Number(a.number) - Number(b.number));
return { title, chapters, chapterLinks,
...(chapterLinks && !chapters.length ? { availability: { reason: 'unrecognized-labels',
message: `WeebCentral lists ${chapterLinks} chapter links, but their numbering labels are not recognized by the parser.` } } : {}),
};
}
async function discoverWeebChapter(chapter, http) {
const pageHtml = await (await http.get(chapter.url, { accept: 'text/html' })).text();
const { manifestUrl, added } = parse('weebcentral', 'chapter', { html: pageHtml, url: chapter.url });
const fragment = await (await http.get(`${manifestUrl}&reading_style=long_strip&current_page=1`, { accept: 'text/html' })).text();
const parsed = parse('weebcentral', 'pages', { html: fragment, url: manifestUrl });
const pages = parsed.pages.map(url => ({ url, headers: { referer: chapter.url } }));
if (!pages.length) throw new Error(`No pages found for chapter ${chapter.number}`);
return {
number: chapter.number, title: chapter.title || '',
...(chapter.special ? { special: true, label: chapter.label, sourceId: chapter.id,
sourceNumber: chapter.volume, releaseKind: chapter.releaseKind, volume: chapter.volume } : {}),
...(chapter.book ? { book: chapter.book } : {}),
added,
source: { name: 'WeebCentral', url: chapter.url }, pages,
};
}
async function discoverWeebCentral(seriesUrl, options = {}) {
const http = new SlowHttp(options.http || {});
const catalog = await discoverWeebCatalog(seriesUrl, http);
const chapters = catalog.chapters
.sort((a, b) => Number(b.number) - Number(a.number))
.slice(0, options.limit || 3)
.sort((a, b) => Number(a.number) - Number(b.number));
const output = [];
for (const chapter of chapters) {
output.push(await discoverWeebChapter(chapter, http));
}
return {
version: 1,
series: { title: catalog.title, ...fallbackDirection(), sources: [{ name: 'WeebCentral', url: seriesUrl }], scraper: { ids: { weebcentral: seriesUrl.split('/series/')[1]?.split('/')[0] } } },
http: options.http || {}, chapters: output,
};
}
module.exports = { weebCentralInfo, discoverWeebCentral, discoverWeebCatalog, discoverWeebChapter, parseChapters, parsePageUrls };
+115
View File
@@ -0,0 +1,115 @@
#!/usr/bin/env node
'use strict';
const fs = require('fs').promises;
const path = require('path');
const { publishManifest } = require('./lib/publisher');
const { discoverWeebCentral } = require('./adapters/weebcentral');
const { lookupAniList } = require('./metadata/anilist');
const { runArchive } = require('./jobs/weebcentral-archive');
const { runQueue } = require('./jobs/weebcentral-queue');
function flag(args, name, fallback) {
const index = args.indexOf(name);
return index < 0 ? fallback : args[index + 1];
}
async function main() {
const args = process.argv.slice(2);
const command = args.shift();
if (command === 'publish') {
const file = args[0];
if (!file) throw new Error('Usage: publish <manifest.json> [--library path] [--metadata-search title] [--replace]');
const manifest = JSON.parse(await fs.readFile(path.resolve(file), 'utf8'));
const metadataSearch = flag(args, '--metadata-search');
if (metadataSearch) {
const metadata = await lookupAniList(metadataSearch);
manifest.series.metadata = { ...metadata, title: manifest.series.title };
manifest.series.scraper = { ...(manifest.series.scraper || {}), ids: { ...(manifest.series.scraper?.ids || {}), anilist: metadata['x-anilist']?.id } };
}
const result = await publishManifest(manifest, { libraryRoot: flag(args, '--library'), replace: args.includes('--replace') });
console.log(JSON.stringify(result, null, 2));
return;
}
if (command === 'discover-weebcentral') {
const url = args[0];
if (!url) throw new Error('Usage: discover-weebcentral <series-url> --out manifest.json [--limit 3]');
const manifest = await discoverWeebCentral(url, { limit: Number(flag(args, '--limit', 3)) });
const out = flag(args, '--out');
if (out) await fs.writeFile(path.resolve(out), JSON.stringify(manifest, null, 2) + '\n');
else console.log(JSON.stringify(manifest, null, 2));
return;
}
if (command === 'metadata-anilist') {
const search = args.join(' ');
if (!search) throw new Error('Usage: metadata-anilist <title>');
console.log(JSON.stringify(await lookupAniList(search), null, 2));
return;
}
if (command === 'archive-weebcentral') {
const url = args[0];
if (!url) throw new Error('Usage: archive-weebcentral <series-url> [--library path] [--metadata-search title]');
await runArchive({
seriesUrl: url,
libraryRoot: flag(args, '--library', 'library'),
metadataSearch: flag(args, '--metadata-search'),
minMs: Number(flag(args, '--min-ms', 8000)),
maxMs: Number(flag(args, '--max-ms', 15000)),
chapterMinMs: Number(flag(args, '--chapter-min-ms', 30000)),
chapterMaxMs: Number(flag(args, '--chapter-max-ms', 60000)),
});
return;
}
if (command === 'status-weebcentral') {
const url = args[0];
if (!url) throw new Error('Usage: status-weebcentral <series-url> [--library path]');
const sourceId = url.split('/series/')[1]?.split('/')[0];
if (!sourceId) throw new Error('Could not read WeebCentral series id');
const workRoot = path.resolve(flag(args, '--library', 'library'), `.mcscraper-work-${sourceId}`);
const state = JSON.parse(await fs.readFile(path.join(workRoot, 'job.json'), 'utf8'));
console.log(JSON.stringify({
status: state.status || 'unknown',
completed: state.completed?.length || 0,
total: state.total || null,
percent: state.total ? Math.round((state.completed?.length || 0) * 1000 / state.total) / 10 : null,
lastChapter: state.lastChapter || null,
current: state.current || null,
lastProblem: state.lastProblem || null,
lastError: state.lastError || null,
problemCount: state.problemCount || 0,
rate: state.rate || null,
updated: state.updated || null,
log: path.join(workRoot, 'job.log'),
}, null, 2));
return;
}
if (command === 'run-wishlist') {
const entries = [
['The Gamer', '01J76XY9DYH4MH2HZG2PPF74A6', 'The-Gamer'],
['Kanojo Okarishimasu', '01J76XYC7KP5M48JAY8S4ZNDX5', 'Kanojo-Okarishimasu'],
['Toradora', '01J76XY7J4Z4G6E42GED6QEQXZ', 'Toradora'],
['Solo Leveling', '01J76XYCPSY3C4BNPBRY8JMCBE', 'Solo-Leveling'],
['Gantz', '01J76XY7EREQ4B1Q5Z5RJ749V6', 'Gantz'],
['One Piece', '01J76XY7E9FNDZ1DBBM6PBJPFK', 'One-Piece'],
].map(([title, id, slug]) => ({ id, title, url: `https://weebcentral.com/series/${id}/${slug}` }));
await runQueue({
entries,
waitForFolder: flag(args, '--wait-for', 'Berserk'),
libraryRoot: flag(args, '--library', 'library'),
minMs: Number(flag(args, '--min-ms', 3000)),
maxMs: Number(flag(args, '--max-ms', 7000)),
chapterMinMs: Number(flag(args, '--chapter-min-ms', 12000)),
chapterMaxMs: Number(flag(args, '--chapter-max-ms', 25000)),
});
return;
}
if (command === 'status-queue') {
const queueRoot = path.resolve(flag(args, '--library', 'library'), '.mcscraper-queue');
const state = JSON.parse(await fs.readFile(path.join(queueRoot, 'queue.json'), 'utf8'));
console.log(JSON.stringify({ ...state, log: path.join(queueRoot, 'queue.log') }, null, 2));
return;
}
console.log('Commands:\n publish <manifest.json> [--library path] [--metadata-search title] [--replace]\n discover-weebcentral <url> [--limit 3] [--out file]\n archive-weebcentral <url> [--library path] [--metadata-search title]\n status-weebcentral <url> [--library path]\n run-wishlist [--library path] [--wait-for Berserk]\n status-queue [--library path]\n metadata-anilist <title>');
}
main().catch(error => { console.error(`[scraper] ${error.message}`); process.exitCode = 1; });
+65
View File
@@ -0,0 +1,65 @@
#!/usr/bin/env node
'use strict';
const http = require('http');
const path = require('path');
const { ScraperService } = require('./service');
const { secret } = require('../runtime/config');
const HOST = process.env.SCRAPER_HOST || '127.0.0.1';
const PORT = Number(process.env.SCRAPER_PORT || 8091);
const TOKEN = secret('SCRAPER_TOKEN');
const LIBRARY = path.resolve(process.env.LIBRARY || 'library');
const DATA = path.resolve(process.env.SCRAPER_DATA || path.join(process.env.DATA || 'data', 'scraper'));
if (!['127.0.0.1', 'localhost', '::1'].includes(HOST) && !TOKEN) {
throw new Error('SCRAPER_TOKEN is required when SCRAPER_HOST is not loopback');
}
function json(res, code, value) {
res.writeHead(code, { 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-store' });
res.end(JSON.stringify(value));
}
async function body(req) {
let text = '';
for await (const chunk of req) { text += chunk; if (text.length > 1000000) throw Object.assign(new Error('Request too large'), { code: 413 }); }
try { return text ? JSON.parse(text) : {}; } catch { throw Object.assign(new Error('Invalid JSON'), { code: 400 }); }
}
(async () => {
const service = await new ScraperService({ libraryRoot: LIBRARY, dataRoot: DATA }).init();
const server = http.createServer(async (req, res) => {
try {
const url = new URL(req.url, 'http://localhost');
if (url.pathname === '/health') return json(res, 200, { ok: true, status: service.state.status });
if (TOKEN && req.headers.authorization !== `Bearer ${TOKEN}`) return json(res, 401, { error: 'Unauthorized' });
if (req.method === 'GET' && url.pathname === '/api/state') return json(res, 200, await service.publicState());
if (req.method !== 'POST') return json(res, 404, { error: 'Not found' });
const input = await body(req);
if (url.pathname === '/api/queue') return json(res, 200, { item: await service.enqueue(input), state: await service.publicState() });
if (url.pathname === '/api/queue/remove') { await service.remove(String(input.id || '')); return json(res, 200, await service.publicState()); }
if (url.pathname === '/api/queue/clear-completed') { const result = await service.clearCompleted(String(input.source || '')); return json(res, 200, { ...result, state: await service.publicState() }); }
if (url.pathname === '/api/queue/reorder') { await service.reorder(input.ids); return json(res, 200, await service.publicState()); }
if (url.pathname === '/api/control') { await service.control(String(input.action || ''), input.id); return json(res, 200, await service.publicState()); }
if (url.pathname === '/api/recovery') return json(res, 200, await service.recover(input));
if (url.pathname === '/api/settings') { await service.updateSettings(input); return json(res, 200, await service.publicState()); }
if (url.pathname === '/api/discovery/run') { await service.enqueueDiscoveryChecks('manual'); return json(res, 200, await service.publicState()); }
if (url.pathname === '/api/library/auto-update') { await service.setMangaAutoUpdate(String(input.id || ''), !!input.enabled); return json(res, 200, await service.publicState()); }
if (url.pathname === '/api/legacy/publish-ready') return json(res, 200, { result: await service.publishLegacyReady(), state: await service.publicState() });
return json(res, 404, { error: 'Not found' });
} catch (error) {
console.error('[scraper-daemon]', error.message);
json(res, error.code || (/not found/i.test(error.message) ? 404 : 400), { error: error.message });
}
});
server.listen(PORT, HOST, () => console.log(`MangaCircus scraper on http://${HOST}:${PORT} (library: ${LIBRARY}, data: ${DATA})`));
let closing = false;
const close = async () => {
if (closing) return; closing = true;
server.close(); await service.close(); process.exit(0);
};
process.on('SIGINT', close);
process.on('SIGTERM', close);
})().catch(error => { console.error(`[scraper-daemon] ${error.message}`); process.exitCode = 1; });
+215
View File
@@ -0,0 +1,215 @@
'use strict';
const fs = require('fs').promises;
const path = require('path');
const { SlowHttp } = require('../lib/http');
const { sleep } = require('../lib/delay');
const { ArchiveControlError } = require('../lib/control');
const { cleanPart } = require('../lib/names');
const { publishManifest } = require('../lib/publisher');
const { selectByScope } = require('../lib/scope');
const { scanLibrary, findManga, shouldDownload, sourceKey } = require('../lib/library-inventory');
const { lookupAniList, lookupAniListById } = require('../metadata/anilist');
const { mangaDexInfo, discoverMangaDexCatalog, discoverMangaDexChapter } = require('../adapters/mangadex');
const randomBetween = (min, max) => min + Math.floor(Math.random() * (max - min + 1));
const exists = async file => { try { await fs.access(file); return true; } catch { return false; } };
async function writeJsonAtomic(file, value) {
const temp = `${file}.tmp`;
await fs.writeFile(temp, JSON.stringify(value, null, 2) + '\n');
await fs.rename(temp, file).catch(async error => {
if (!['EPERM', 'EEXIST'].includes(error.code)) throw error;
await fs.rm(file, { force: true });
await fs.rename(temp, file);
});
}
async function renameRetry(from, to) {
for (let attempt = 0; ; attempt++) {
try { await fs.rename(from, to); return; } catch (error) {
if (attempt >= 5 || !['EPERM', 'EBUSY', 'EACCES'].includes(error.code)) throw error;
await sleep(200 * (attempt + 1));
}
}
}
async function runMangaDexArchive(options) {
const info = mangaDexInfo(options.seriesUrl);
if (!info) throw new Error('Could not read MangaDex title id');
const checkControl = () => {
const reason = options.shouldStop && options.shouldStop();
if (reason) throw new ArchiveControlError(reason === true ? 'paused' : String(reason));
};
const libraryRoot = path.resolve(options.libraryRoot || 'library');
const workRoot = path.join(libraryRoot, `.mcscraper-work-${info.id}`);
const stateFile = path.join(workRoot, 'job.json');
const logFile = path.join(workRoot, 'job.log');
await fs.mkdir(workRoot, { recursive: true });
const resumed = await exists(stateFile);
let state = { version: 2, seriesUrl: options.seriesUrl, source: 'mangadex', publicationMode: 'incremental', completed: [], started: new Date().toISOString() };
if (resumed) state = JSON.parse(await fs.readFile(stateFile, 'utf8'));
const incremental = state.publicationMode === 'incremental';
const completed = new Set(state.completed || []);
const event = async (level, message, details = {}) => {
const entry = { time: new Date().toISOString(), level, message, ...details };
await fs.appendFile(logFile, JSON.stringify(entry) + '\n');
state.lastEvent = entry;
if (level === 'warning' || level === 'error') {
state.lastProblem = entry;
state.problemCount = Number(state.problemCount || 0) + 1;
}
state.updated = entry.time;
await writeJsonAtomic(stateFile, state);
};
const minMs = Number(options.minMs);
const maxMs = Number(options.maxMs);
const chapterMinMs = Number(options.chapterMinMs);
const chapterMaxMs = Number(options.chapterMaxMs);
const http = new SlowHttp({
gateKey: 'mangadex', minMs, maxMs, maxRetries: 2,
userAgent: 'MangaCircus-Scraper/1.0 (personal archival; sequential MangaDex requests)',
onEvent: item => event(item.level, item.message, { status: item.status }),
});
checkControl();
console.log(`[mangadex] discovering catalog ${info.id}`);
const catalog = await discoverMangaDexCatalog(options.seriesUrl, http);
if (options.onCatalog) await options.onCatalog(catalog);
if (!catalog.chapters.length) throw Object.assign(new Error(catalog.availability.message), { failureKind: 'unavailable' });
const specialOnly = catalog.chapters.every(ref => ref.special);
const folderTitle = String(specialOnly ? catalog.title : options.title || catalog.title).trim() || catalog.title;
console.log(`[mangadex] ${folderTitle}: ${catalog.chapters.length} unique English chapters`);
const expectedAniList = catalog.links.al ? String(catalog.links.al) : null;
const metadata = expectedAniList
? await lookupAniListById(expectedAniList)
: options.metadataSearch && !specialOnly ? await lookupAniList(options.metadataSearch) : {};
const actualAniList = metadata['x-anilist']?.id ? String(metadata['x-anilist'].id) : null;
if (expectedAniList && actualAniList && expectedAniList !== actualAniList) {
throw new Error(`AniList identity mismatch: MangaDex ${expectedAniList}, lookup ${actualAniList}`);
}
const series = {
title: folderTitle, ...require('../../reading-direction').fallbackDirection(),
sources: [{ name: 'MangaDex', url: options.seriesUrl }],
metadata: { ...metadata, title: specialOnly ? catalog.title : metadata.title || folderTitle },
coverUrl: metadata['x-anilist']?.coverUrl,
scraper: { ids: { mangadex: info.id, ...(expectedAniList ? { anilist: expectedAniList } : {}) } },
};
let inventory = await scanLibrary(libraryRoot);
let manga = findManga(inventory, {
source: 'mangadex', seriesId: info.id, sourceUrl: options.seriesUrl,
anilist: specialOnly ? null : expectedAniList || actualAniList, libraryMangaId: options.libraryMangaId,
});
if (options.libraryMangaId && !manga) throw new Error(`Library manga target not found: ${options.libraryMangaId}`);
const eligible = incremental
? catalog.chapters.filter(ref => shouldDownload(manga, ref, 'mangadex', options.existingPolicy))
: catalog.chapters;
const refs = incremental ? selectByScope(eligible, options.scope) : eligible;
state.status = 'running';
state.catalogTotal = catalog.chapters.length;
state.total = incremental ? refs.length : catalog.chapters.length;
state.jobCompleted = 0;
state.scope = options.scope || { mode: 'missing' };
state.existingPolicy = options.existingPolicy || 'skip-any';
state.rate = { minMs, maxMs, chapterMinMs, chapterMaxMs };
state.lastError = null;
await event('info', `Catalog ready: ${catalog.chapters.length} chapters; ${refs.length} selected`);
if (incremental && !refs.length) {
state.status = 'complete'; state.current = null;
await event('info', 'No eligible chapters to download');
return { finalPath: manga?.path || null, publishedCount: 0, selectedCount: 0 };
}
let publishedCount = 0;
for (let index = 0; index < refs.length; index++) {
checkControl();
const ref = refs[index];
if (!incremental && completed.has(ref.id)) {
if (!incremental) console.log(`[mangadex] skip completed ${ref.number} (${index + 1}/${refs.length})`);
continue;
}
if (incremental && !shouldDownload(manga, ref, 'mangadex', options.existingPolicy)) {
state.jobCompleted++;
await event('info', `Skipped existing chapter ${ref.number}`, { chapter: ref.number });
continue;
}
let chapter, result;
try {
state.status = 'inspecting';
state.current = { number: ref.number, label: ref.special ? ref.label : null, ...(ref.volume ? { volume: ref.volume } : {}), sourceId: ref.id, position: index + 1, total: refs.length, downloadedPages: 0, expectedPages: ref.pages || null };
await event('info', `Inspecting chapter ${ref.number}`, { chapter: ref.number });
chapter = await discoverMangaDexChapter(ref, http);
state.status = 'downloading';
state.current.expectedPages = chapter.pages.length;
await event('info', `Downloading chapter ${ref.number}: ${chapter.pages.length} pages`, { chapter: ref.number });
console.log(`[mangadex] download ${ref.number}: ${chapter.pages.length} pages`);
result = await publishManifest({ version: 1, series, chapters: [chapter] }, {
libraryRoot: incremental ? libraryRoot : workRoot,
...(incremental && manga ? { mangaDir: manga.path } : {}),
httpClient: http,
onPage: async progress => {
state.current.downloadedPages = progress.completed;
state.updated = new Date().toISOString();
await writeJsonAtomic(stateFile, state);
checkControl();
},
});
} catch (error) {
if (error instanceof ArchiveControlError) {
state.status = error.reason;
state.current = null;
await event('info', `Archive ${error.reason}`, { chapter: ref.number });
throw error;
}
state.status = 'error';
state.lastError = { time: new Date().toISOString(), chapter: ref.number, message: error.message };
await event('error', error.message, { chapter: ref.number });
throw error;
}
completed.add(ref.id);
state.completed = [...completed];
state.jobCompleted = Number(state.jobCompleted || 0) + 1;
state.lastChapter = ref.number;
state.status = 'running';
state.current = null;
await writeJsonAtomic(stateFile, state);
const published = result.results[0].status === 'published';
if (published && incremental) {
publishedCount++;
if (!manga) manga = { id: path.basename(result.mangaDir), path: result.mangaDir, meta: { 'x-scraper': { strategy: 'flat' } }, chapters: [] };
manga.chapters.push({ book: '1', number: String(ref.number), sourceKey: sourceKey(chapter.source.url, chapter.source.name) });
if (options.onPublished) await options.onPublished({ mangaDir: result.mangaDir, chapter: ref.number });
}
await event('info', `Completed chapter ${ref.number} (${state.jobCompleted}/${state.total})`, { chapter: ref.number });
console.log(`[mangadex] complete ${ref.number}: ${result.results[0].status} (${state.jobCompleted}/${state.total})`);
if (index < refs.length - 1) {
checkControl();
await sleep(randomBetween(chapterMinMs, chapterMaxMs));
checkControl();
}
}
if (incremental) {
state.status = 'complete'; state.current = null;
await event('info', `Published ${publishedCount} chapter${publishedCount === 1 ? '' : 's'} incrementally`);
return { finalPath: manga?.path || null, publishedCount, selectedCount: refs.length };
}
const stagedManga = path.join(workRoot, cleanPart(folderTitle));
const finalManga = path.join(libraryRoot, cleanPart(folderTitle));
if (await exists(finalManga)) throw new Error(`Final manga folder already exists: ${finalManga}`);
await renameRetry(stagedManga, finalManga);
state.status = 'complete';
state.current = null;
await event('info', `Published ${finalManga}`);
await fs.rm(workRoot, { recursive: true, force: true });
if (options.onPublished) await options.onPublished({ mangaDir: finalManga, completeManga: true });
console.log(`[mangadex] PUBLISHED ${finalManga}`);
return { finalPath: finalManga, publishedCount: 1, selectedCount: catalog.chapters.length };
}
module.exports = { runMangaDexArchive };
+229
View File
@@ -0,0 +1,229 @@
'use strict';
const fs = require('fs').promises;
const path = require('path');
const { createWeebHttp } = require('../plugins/broker');
const { sleep } = require('../lib/delay');
const { ArchiveControlError } = require('../lib/control');
const { cleanPart } = require('../lib/names');
const { publishManifest } = require('../lib/publisher');
const { selectByScope } = require('../lib/scope');
const { scanLibrary, findManga, shouldDownload, sourceKey } = require('../lib/library-inventory');
const { lookupAniList } = require('../metadata/anilist');
const { discoverWeebCatalog, discoverWeebChapter } = require('../adapters/weebcentral');
function randomBetween(min, max) {
return min + Math.floor(Math.random() * (max - min + 1));
}
async function exists(file) {
try { await fs.access(file); return true; } catch { return false; }
}
async function writeJsonAtomic(file, value) {
const temp = `${file}.tmp`;
await fs.writeFile(temp, JSON.stringify(value, null, 2) + '\n');
await fs.rename(temp, file).catch(async error => {
if (!['EPERM', 'EEXIST'].includes(error.code)) throw error;
await fs.rm(file, { force: true });
await fs.rename(temp, file);
});
}
async function renameRetry(from, to) {
for (let attempt = 0; ; attempt++) {
try { await fs.rename(from, to); return; } catch (error) {
if (attempt >= 5 || !['EPERM', 'EBUSY', 'EACCES'].includes(error.code)) throw error;
await sleep(200 * (attempt + 1));
}
}
}
async function runArchive(options) {
const source = options.source || 'weebcentral';
const sourceLabel = options.sourceLabel || 'WeebCentral';
const discoverCatalog = options.discoverCatalog || discoverWeebCatalog;
const discoverChapter = options.discoverChapter || discoverWeebChapter;
const checkControl = () => {
const reason = options.shouldStop && options.shouldStop();
if (reason) throw new ArchiveControlError(reason === true ? 'paused' : String(reason));
};
const libraryRoot = path.resolve(options.libraryRoot || 'library');
const sourceId = options.sourceId || options.seriesUrl.split('/series/')[1]?.split('/')[0];
if (!sourceId) throw new Error(`Could not read ${sourceLabel} series id`);
const workRoot = path.join(libraryRoot, source === 'weebcentral' ? `.mcscraper-work-${sourceId}` : `.mcscraper-work-${source}-${sourceId}`);
const stateFile = path.join(workRoot, 'job.json');
const logFile = path.join(workRoot, 'job.log');
await fs.mkdir(workRoot, { recursive: true });
const resumed = await exists(stateFile);
let state = { version: 2, seriesUrl: options.seriesUrl, publicationMode: 'incremental', completed: [], started: new Date().toISOString() };
if (resumed) state = JSON.parse(await fs.readFile(stateFile, 'utf8'));
const incremental = state.publicationMode === 'incremental';
const completed = new Set(state.completed || []);
const event = async (level, message, details = {}) => {
const entry = { time: new Date().toISOString(), level, message, ...details };
await fs.appendFile(logFile, JSON.stringify(entry) + '\n');
state.lastEvent = entry;
if (level === 'warning' || level === 'error') {
state.lastProblem = entry;
state.problemCount = Number(state.problemCount || 0) + 1;
}
state.updated = entry.time;
await writeJsonAtomic(stateFile, state);
};
const httpOptions = {
minMs: options.minMs, maxMs: options.maxMs, maxRetries: 2,
onEvent: item => event(item.level, item.message, { url: item.url, ...(item.status ? { status: item.status } : {}) }),
};
const http = options.createHttp ? options.createHttp(httpOptions) : createWeebHttp(httpOptions);
checkControl();
console.log(`[archive] discovering catalog ${options.seriesUrl}`);
const catalog = await discoverCatalog(options.seriesUrl, http);
if (options.onCatalog) await options.onCatalog(catalog);
if (!catalog.chapters.length) throw Object.assign(new Error(catalog.availability?.message || `${sourceLabel} lists no chapter links for this series.`), {
failureKind: catalog.chapterLinks ? 'adapter' : 'unavailable',
});
const keys = new Map();
for (const chapter of catalog.chapters) {
const key = `${chapter.book?.number || 1}:${chapter.number}`;
const prior = keys.get(key);
if (prior) throw new Error(`Ambiguous mapped chapter ${key}: ${prior.url} and ${chapter.url}`);
keys.set(key, chapter);
}
console.log(`[archive] ${catalog.title}: ${catalog.chapters.length} source chapters`);
const metadata = options.metadataSearch ? await lookupAniList(options.metadataSearch) : {};
const series = {
title: catalog.title, ...require('../../reading-direction').fallbackDirection(),
sources: [{ name: sourceLabel, url: options.seriesUrl }],
metadata: { ...metadata, title: catalog.title },
coverUrl: metadata['x-anilist']?.coverUrl,
scraper: { ids: { [source]: sourceId, ...(metadata['x-anilist']?.id ? { anilist: metadata['x-anilist'].id } : {}) } },
};
let inventory = await scanLibrary(libraryRoot);
let manga = findManga(inventory, {
source, seriesId: sourceId, sourceUrl: options.seriesUrl,
anilist: metadata['x-anilist']?.id, libraryMangaId: options.libraryMangaId,
});
if (options.libraryMangaId && !manga) throw new Error(`Library manga target not found: ${options.libraryMangaId}`);
const eligible = incremental
? catalog.chapters.filter(ref => shouldDownload(manga, ref, source, options.existingPolicy))
: catalog.chapters;
const refs = incremental ? selectByScope(eligible, options.scope) : eligible;
state.status = 'running';
state.catalogTotal = catalog.chapters.length;
state.total = incremental ? refs.length : catalog.chapters.length;
state.jobCompleted = 0;
state.scope = options.scope || { mode: 'missing' };
state.existingPolicy = options.existingPolicy || 'skip-any';
state.rate = { minMs: options.minMs, maxMs: options.maxMs, chapterMinMs: options.chapterMinMs, chapterMaxMs: options.chapterMaxMs };
state.lastError = null;
await event('info', `Catalog ready: ${catalog.chapters.length} chapters; ${refs.length} selected`);
if (incremental && !refs.length) {
state.status = 'complete'; state.current = null;
await event('info', 'No eligible chapters to download');
return { finalPath: manga?.path || null, publishedCount: 0, selectedCount: 0 };
}
let publishedCount = 0;
for (let index = 0; index < refs.length; index++) {
checkControl();
const ref = refs[index];
const display = ref.label || ref.number;
if (!incremental && completed.has(ref.id)) {
if (!incremental) console.log(`[archive] skip completed ${ref.number} (${index + 1}/${refs.length})`);
continue;
}
if (incremental && !shouldDownload(manga, ref, source, options.existingPolicy)) {
state.jobCompleted++;
await event('info', `Skipped existing ${display}`, { chapter: ref.number });
continue;
}
console.log(`[archive] inspect ${display} (${index + 1}/${refs.length})`);
let chapter;
let result;
try {
state.status = 'inspecting';
state.current = { number: ref.number, ...(ref.label ? { label: ref.label } : {}),
...(ref.book ? { book: ref.book } : {}), sourceId: ref.id, position: index + 1,
total: refs.length, downloadedPages: 0, expectedPages: null };
await event('info', `Inspecting ${display}`, { chapter: ref.number });
chapter = await discoverChapter(ref, http);
state.status = 'downloading';
state.current.expectedPages = chapter.pages.length;
await event('info', `Downloading ${display}: ${chapter.pages.length} pages`, { chapter: ref.number });
console.log(`[archive] download ${display}: ${chapter.pages.length} pages`);
result = await publishManifest({ version: 1, series, chapters: [chapter] }, {
libraryRoot: incremental ? libraryRoot : workRoot,
...(incremental && manga ? { mangaDir: manga.path } : {}),
httpClient: http,
onPage: async progress => {
state.current.downloadedPages = progress.completed;
state.updated = new Date().toISOString();
await writeJsonAtomic(stateFile, state);
checkControl();
},
});
} catch (error) {
if (error instanceof ArchiveControlError) {
state.status = error.reason;
state.current = null;
await event('info', `Archive ${error.reason}`, { chapter: ref.number });
throw error;
}
state.status = 'error';
state.lastError = { time: new Date().toISOString(), chapter: ref.number, message: error.message };
await event('error', error.message, { chapter: ref.number });
throw error;
}
completed.add(ref.id);
state.completed = [...completed];
state.jobCompleted = Number(state.jobCompleted || 0) + 1;
state.lastChapter = ref.number;
state.updated = new Date().toISOString();
state.total = incremental ? refs.length : catalog.chapters.length;
state.status = 'running';
state.current = null;
await writeJsonAtomic(stateFile, state);
const published = result.results[0].status === 'published';
if (published && incremental) {
publishedCount++;
if (!manga) manga = { id: path.basename(result.mangaDir), path: result.mangaDir, meta: { 'x-scraper': { strategy: 'flat' } }, chapters: [] };
manga.chapters.push({ book: String(ref.book?.number || 1), number: String(ref.number),
releaseKind: ref.releaseKind || null, sourceKey: sourceKey(chapter.source.url, chapter.source.name) });
if (options.onPublished) await options.onPublished({ mangaDir: result.mangaDir, chapter: ref.number });
}
await event('info', `Completed ${display} (${state.jobCompleted}/${state.total})`, { chapter: ref.number });
console.log(`[archive] complete ${display}: ${result.results[0].status} (${state.jobCompleted}/${state.total})`);
if (index < refs.length - 1) {
checkControl();
const pause = randomBetween(options.chapterMinMs, options.chapterMaxMs);
console.log(`[archive] chapter pause ${Math.round(pause / 1000)}s`);
await sleep(pause);
checkControl();
}
}
if (incremental) {
state.status = 'complete'; state.current = null;
await event('info', `Published ${publishedCount} chapter${publishedCount === 1 ? '' : 's'} incrementally`);
return { finalPath: manga?.path || null, publishedCount, selectedCount: refs.length };
}
const stagedManga = path.join(workRoot, cleanPart(catalog.title));
const finalManga = path.join(libraryRoot, cleanPart(catalog.title));
if (await exists(finalManga)) throw new Error(`Final manga folder already exists: ${finalManga}`);
await renameRetry(stagedManga, finalManga);
state.status = 'complete';
state.current = null;
await event('info', `Published ${finalManga}`);
await fs.rm(workRoot, { recursive: true, force: true });
if (options.onPublished) await options.onPublished({ mangaDir: finalManga, completeManga: true });
console.log(`[archive] PUBLISHED ${finalManga}`);
return { finalPath: finalManga, publishedCount: 1, selectedCount: catalog.chapters.length };
}
module.exports = { runArchive };
+124
View File
@@ -0,0 +1,124 @@
'use strict';
const fs = require('fs').promises;
const path = require('path');
const { runArchive } = require('./weebcentral-archive');
const { sleep } = require('../lib/delay');
async function exists(file) {
try { await fs.access(file); return true; } catch { return false; }
}
async function writeJsonAtomic(file, value) {
const temp = `${file}.tmp`;
await fs.writeFile(temp, JSON.stringify(value, null, 2) + '\n');
await fs.rename(temp, file).catch(async error => {
if (!['EPERM', 'EEXIST'].includes(error.code)) throw error;
await fs.rm(file, { force: true });
await fs.rename(temp, file);
});
}
async function acquireLock(lockFile) {
try {
const handle = await fs.open(lockFile, 'wx');
await handle.writeFile(String(process.pid));
await handle.close();
return;
} catch (error) {
if (error.code !== 'EEXIST') throw error;
}
const priorPid = Number((await fs.readFile(lockFile, 'utf8').catch(() => '')).trim());
try {
if (priorPid) process.kill(priorPid, 0);
throw new Error(`Queue runner is already active (PID ${priorPid || 'unknown'})`);
} catch (error) {
if (error.message.startsWith('Queue runner is already active')) throw error;
await fs.rm(lockFile, { force: true });
const handle = await fs.open(lockFile, 'wx');
await handle.writeFile(String(process.pid));
await handle.close();
}
}
async function runQueue(options) {
const libraryRoot = path.resolve(options.libraryRoot || 'library');
const queueRoot = path.join(libraryRoot, '.mcscraper-queue');
const stateFile = path.join(queueRoot, 'queue.json');
const logFile = path.join(queueRoot, 'queue.log');
const lockFile = path.join(queueRoot, 'queue.lock');
await fs.mkdir(queueRoot, { recursive: true });
await acquireLock(lockFile);
let state = {
version: 1,
status: 'waiting',
waitForFolder: options.waitForFolder || null,
entries: options.entries,
completed: [],
created: new Date().toISOString(),
};
if (await exists(stateFile)) {
const saved = JSON.parse(await fs.readFile(stateFile, 'utf8'));
state = { ...state, ...saved, entries: saved.entries?.length ? saved.entries : options.entries };
}
state.pid = process.pid;
const event = async (level, message, details = {}) => {
const entry = { time: new Date().toISOString(), level, message, ...details };
await fs.appendFile(logFile, JSON.stringify(entry) + '\n');
state.lastEvent = entry;
state.updated = entry.time;
if (level === 'error') state.lastError = entry;
await writeJsonAtomic(stateFile, state);
};
const cleanup = async () => { await fs.rm(lockFile, { force: true }).catch(() => {}); };
process.once('exit', () => {});
try {
await event('info', `Queue runner started (PID ${process.pid})`);
if (state.waitForFolder) {
const prerequisite = path.join(libraryRoot, state.waitForFolder);
state.status = 'waiting';
await event('info', `Waiting for ${state.waitForFolder} to finish`);
while (!(await exists(prerequisite))) await sleep(60000);
await event('info', `${state.waitForFolder} is complete; starting queue`);
}
const completed = new Set(state.completed || []);
for (let index = 0; index < state.entries.length; index++) {
const item = state.entries[index];
if (completed.has(item.id)) continue;
state.status = 'downloading';
state.current = { ...item, position: index + 1, total: state.entries.length };
state.lastError = null;
await event('info', `Starting ${item.title}`, { title: item.title });
try {
await runArchive({
seriesUrl: item.url,
libraryRoot,
metadataSearch: item.metadataSearch || item.title,
minMs: options.minMs,
maxMs: options.maxMs,
chapterMinMs: options.chapterMinMs,
chapterMaxMs: options.chapterMaxMs,
});
} catch (error) {
state.status = 'error';
await event('error', error.message, { title: item.title });
throw error;
}
completed.add(item.id);
state.completed = [...completed];
state.current = null;
await event('info', `Completed ${item.title}`, { title: item.title });
}
state.status = 'complete';
state.current = null;
await event('info', 'Queue complete');
} finally {
await cleanup();
}
}
module.exports = { runQueue };
+11
View File
@@ -0,0 +1,11 @@
'use strict';
class ArchiveControlError extends Error {
constructor(reason = 'paused') {
super(`Archive ${reason}`);
this.name = 'ArchiveControlError';
this.reason = reason;
}
}
module.exports = { ArchiveControlError };
+27
View File
@@ -0,0 +1,27 @@
'use strict';
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
class PoliteGate {
constructor({ minMs = 8000, maxMs = 15000 } = {}) {
if (!Number.isFinite(minMs) || !Number.isFinite(maxMs) || minMs < 0 || maxMs < minMs) {
throw new Error('Invalid delay range');
}
this.minMs = minMs;
this.maxMs = maxMs;
this.lastRequestAt = 0;
}
async wait() {
if (!this.lastRequestAt) return;
const target = this.minMs + Math.floor(Math.random() * (this.maxMs - this.minMs + 1));
const remaining = target - (Date.now() - this.lastRequestAt);
if (remaining > 0) await sleep(remaining);
}
mark() {
this.lastRequestAt = Date.now();
}
}
module.exports = { PoliteGate, sleep };
+23
View File
@@ -0,0 +1,23 @@
'use strict';
function safeMessage(error) {
return String(error?.message || error || 'Unknown error').replace(/https?:\/\/[^\s]+/gi, raw => {
try { const u = new URL(raw); return u.origin + u.pathname; } catch { return '[source URL]'; }
}).slice(0, 1500);
}
function classifyFailure(error) {
const message = safeMessage(error);
const status = Number(error?.status || /(?:HTTP|returned)\s+(\d{3})/i.exec(message)?.[1]);
const code = error?.code || error?.cause?.code;
const failure = (kind, scope, retryable = false) => ({ kind, scope, retryable, message, ...(status ? { httpStatus: status } : {}) });
if ([401, 403, 429].includes(status) || /captcha|cloudflare|challenge|access.denied|account.ban|paywall|login required/i.test(message)) return failure('access', 'source');
if (['ENOSPC', 'EROFS', 'EACCES', 'EPERM', 'EIO'].includes(code)) return failure('storage', 'global');
if (error?.failureKind === 'unavailable' || /No downloadable English|title was not found|HTTP 404/i.test(message)) return failure('unavailable', 'job');
if (error?.failureKind === 'transient' || status >= 500 || /network|fetch failed|timed? ?out|timeout|socket|connection reset/i.test(message)
|| ['ETIMEDOUT', 'ECONNRESET', 'ENOTFOUND', 'EAI_AGAIN', 'UND_ERR_CONNECT_TIMEOUT'].includes(code)) return failure('transient', 'source', true);
return failure('adapter', 'job');
}
function progressKey(job) {
return JSON.stringify([job?.completed || 0, job?.total || 0, job?.current?.sourceId || '', job?.current?.downloadedPages || 0, job?.lastChapter || '']);
}
module.exports = { classifyFailure, safeMessage, progressKey };
+83
View File
@@ -0,0 +1,83 @@
'use strict';
const { PoliteGate, sleep } = require('./delay');
class StopScrapeError extends Error {}
const sourceGates = new Map();
function gateFor(options) {
const next = new PoliteGate(options);
if (!options.gateKey) return next;
const key = String(options.gateKey);
const existing = sourceGates.get(key);
if (!existing) { sourceGates.set(key, next); return next; }
// A new job may use updated host settings, but it must not forget when the
// previous job last requested this same source.
existing.minMs = next.minMs;
existing.maxMs = next.maxMs;
return existing;
}
class SlowHttp {
constructor(options = {}) {
this.gate = gateFor(options);
this.userAgent = options.userAgent || 'MangaCircus-Scraper/1.0 (personal archival; sequential requests)';
this.maxRetries = options.maxRetries ?? 2;
this.onEvent = options.onEvent;
this.timeoutMs = options.timeoutMs || 60000;
this.transport = options.transport || fetch;
this.validateUrl = options.validateUrl || null;
}
async get(url, options = {}) {
let redirects = 0;
let currentUrl = url;
for (let attempt = 0; ; attempt++) {
if (this.validateUrl) this.validateUrl(currentUrl, options);
await this.gate.wait();
let response;
try {
response = await this.transport(currentUrl, {
redirect: this.validateUrl ? 'manual' : 'follow',
signal: AbortSignal.timeout(this.timeoutMs),
catalog: options.catalog === true,
headers: { 'user-agent': this.userAgent, accept: options.accept || '*/*', ...(options.headers || {}) },
});
this.gate.mark();
} catch (error) {
this.gate.mark();
if (error.nonRetryable) throw error;
if (attempt >= this.maxRetries) throw new Error(`Network request failed after ${attempt + 1} attempts: ${currentUrl}`, { cause: error });
const backoff = 30000 * (2 ** attempt);
const message = `Network failure; retrying in ${backoff / 1000}s (${attempt + 1}/${this.maxRetries})`;
console.warn(`[http] ${message}`);
if (this.onEvent) await this.onEvent({ level: 'warning', message, url: currentUrl });
await sleep(backoff);
continue;
}
if (this.validateUrl && response.status >= 300 && response.status < 400) {
const location = response.headers.get('location');
if (!location || ++redirects > 5) throw new Error('Source redirect limit or missing location');
currentUrl = new URL(location, currentUrl).href;
this.validateUrl(currentUrl, options);
await response.body?.cancel?.();
attempt = -1;
continue;
}
if ([401, 403, 429].includes(response.status)) {
if (this.onEvent) await this.onEvent({ level: 'error', message: `Host returned ${response.status}; stopping`, url: currentUrl, status: response.status });
throw Object.assign(new StopScrapeError(`Host returned ${response.status}; stop and require human review: ${currentUrl}`), { status: response.status });
}
if (response.status >= 500 && attempt < this.maxRetries) {
await sleep(30000 * (2 ** attempt));
continue;
}
if (!response.ok) throw Object.assign(new Error(`HTTP ${response.status}: ${currentUrl}`), { status: response.status });
return response;
}
}
}
module.exports = { SlowHttp, StopScrapeError };
+16
View File
@@ -0,0 +1,16 @@
'use strict';
const TYPES = [
{ ext: 'jpg', test: b => b.length >= 3 && b[0] === 0xff && b[1] === 0xd8 && b[2] === 0xff },
{ ext: 'png', test: b => b.length >= 8 && b.subarray(0, 8).equals(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10])) },
{ ext: 'gif', test: b => b.length >= 6 && /^GIF8[79]a$/.test(b.subarray(0, 6).toString('ascii')) },
{ ext: 'webp', test: b => b.length >= 12 && b.subarray(0, 4).toString('ascii') === 'RIFF' && b.subarray(8, 12).toString('ascii') === 'WEBP' },
{ ext: 'bmp', test: b => b.length >= 2 && b.subarray(0, 2).toString('ascii') === 'BM' },
{ ext: 'avif', test: b => b.length >= 12 && b.subarray(4, 12).toString('ascii').includes('ftyp') && /avif|avis/.test(b.subarray(8, 32).toString('ascii')) },
];
function imageExtension(buffer) {
return TYPES.find(type => type.test(buffer))?.ext || null;
}
module.exports = { imageExtension };
+30
View File
@@ -0,0 +1,30 @@
'use strict';
const fs = require('fs').promises;
const path = require('path');
const crypto = require('crypto');
async function writeJsonAtomic(file, value, io = fs) {
const folder = path.dirname(file), name = path.basename(file);
const temp = path.join(folder, `.${name}.${crypto.randomUUID()}.tmp`);
await io.writeFile(temp, JSON.stringify(value, null, 2) + '\n');
try {
try { await io.rename(temp, file); return; }
catch (error) {
if (!['EPERM', 'EACCES', 'EBUSY', 'EEXIST'].includes(error.code)) throw error;
}
// Windows may refuse rename-over-existing. Keep the old JSON recoverable
// until the replacement is safely in place; never delete it first.
const backup = path.join(folder, `.old-${name}-${crypto.randomUUID()}`);
await io.rename(file, backup);
try { await io.rename(temp, file); }
catch (error) {
try { await io.rename(backup, file); }
catch (restoreError) { error.message += `; previous JSON remains at ${backup}: ${restoreError.message}`; }
throw error;
}
await io.rm(backup, { force: true });
} finally { await io.rm(temp, { force: true }); }
}
module.exports = { writeJsonAtomic };
+179
View File
@@ -0,0 +1,179 @@
'use strict';
const fs = require('fs').promises;
const path = require('path');
const { imageExtension } = require('./images');
const { assertInside } = require('./names');
const PAGE = /^(\d{4,5})(?:_[a-z0-9]+)*\.(jpe?g|png|webp|gif|avif|bmp)$/i;
const BOOK = /^(?:v|s|x)\d{2,3}(?: |$)/i;
const CHAPTER = /^(?:c\d{4}(?:\.\d*[1-9])?|x\d{2})(?: |$)/i;
const OWN_STAGE = /^\.staging-mcscraper-[a-f0-9]+$/;
async function exists(target) {
try { await fs.access(target); return true; } catch { return false; }
}
async function renameRetry(from, to, tries = 5) {
for (let attempt = 0; ; attempt++) {
try { await fs.rename(from, to); return; } catch (error) {
if (attempt >= tries || !['EPERM', 'EBUSY', 'EACCES'].includes(error.code)) throw error;
await new Promise(resolve => setTimeout(resolve, 200 * (attempt + 1)));
}
}
}
async function writeJsonAtomic(file, value) {
const temp = `${file}.migration.tmp`;
const old = `${file}.migration.old`;
await fs.writeFile(temp, JSON.stringify(value, null, 2) + '\n', { mode: 0o644 });
await fs.rm(old, { force: true });
await renameRetry(file, old);
try {
await renameRetry(temp, file);
} catch (error) {
await renameRetry(old, file).catch(() => {});
await fs.rm(temp, { force: true });
throw error;
}
await fs.rm(old, { force: true }).catch(() => {});
}
async function directories(folder) {
return (await fs.readdir(folder, { withFileTypes: true })).filter(entry => entry.isDirectory());
}
async function validatePage(file, expected) {
const handle = await fs.open(file, 'r');
try {
const prefix = Buffer.alloc(16);
const { bytesRead } = await handle.read(prefix, 0, prefix.length, 0);
const actual = imageExtension(prefix.subarray(0, bytesRead));
if (!actual) throw new Error(`Unsupported image bytes: ${file}`);
const normalized = expected.toLowerCase() === 'jpeg' ? 'jpg' : expected.toLowerCase();
if (actual !== normalized) throw new Error(`Image extension does not match bytes: ${file}`);
} finally {
await handle.close();
}
}
async function validateChapter(folder) {
const entries = await fs.readdir(folder, { withFileTypes: true });
if (entries.some(entry => entry.isDirectory())) throw new Error(`Chapter contains a subdirectory: ${folder}`);
const chapterFile = path.join(folder, 'chapter.json');
const chapter = JSON.parse(await fs.readFile(chapterFile, 'utf8'));
if (!chapter.source?.name || !chapter.source?.url) throw new Error(`Invalid chapter source: ${chapterFile}`);
const pages = entries.filter(entry => entry.isFile() && entry.name !== 'chapter.json').map(entry => {
const match = PAGE.exec(entry.name);
if (!match) throw new Error(`Invalid chapter file: ${path.join(folder, entry.name)}`);
return { name: entry.name, number: Number(match[1]), extension: match[2] };
}).sort((a, b) => a.number - b.number);
if (!pages.length) throw new Error(`Chapter has no pages: ${folder}`);
for (let index = 0; index < pages.length; index++) {
if (pages[index].number !== index + 1) throw new Error(`Non-contiguous pages in ${folder}`);
await validatePage(path.join(folder, pages[index].name), pages[index].extension);
}
return pages.length;
}
async function inspectLegacyWork({ libraryRoot, seriesId }) {
const id = String(seriesId || '');
if (!/^[A-Za-z0-9-]+$/.test(id)) throw new Error(`Unsafe legacy series id: ${id}`);
const root = path.resolve(libraryRoot);
const workRoot = assertInside(root, path.join(root, `.mcscraper-work-${id}`));
const stateFile = path.join(workRoot, 'job.json');
if (!(await exists(stateFile))) return null;
const state = JSON.parse(await fs.readFile(stateFile, 'utf8'));
if (state.publicationMode === 'incremental') return null;
const mangaDirs = (await directories(workRoot)).filter(entry => !entry.name.startsWith('.'));
if (!mangaDirs.length) return null;
if (mangaDirs.length !== 1) throw new Error(`Legacy work ${id} must contain exactly one manga folder`);
const mangaDir = path.join(workRoot, mangaDirs[0].name);
const target = assertInside(root, path.join(root, mangaDirs[0].name));
if (await exists(target)) throw new Error(`Cannot publish legacy work; target already exists: ${target}`);
const meta = JSON.parse(await fs.readFile(path.join(mangaDir, 'manga.json'), 'utf8'));
if (!meta.title || !Array.isArray(meta.sources) || !meta.sources.length) throw new Error(`Invalid legacy manga metadata: ${mangaDir}`);
let chapterCount = 0, pageCount = 0;
const incompleteStages = [];
for (const book of await directories(mangaDir)) {
if (book.name.startsWith('.')) throw new Error(`Unexpected hidden folder remains in ${mangaDir}`);
if (!BOOK.test(book.name)) throw new Error(`Invalid legacy book folder: ${book.name}`);
const bookDir = path.join(mangaDir, book.name);
for (const chapter of await directories(bookDir)) {
if (chapter.name.startsWith('.')) {
if (!OWN_STAGE.test(chapter.name)) throw new Error(`Unexpected hidden folder remains in ${bookDir}: ${chapter.name}`);
incompleteStages.push(path.join(bookDir, chapter.name));
continue;
}
if (!CHAPTER.test(chapter.name)) throw new Error(`Invalid legacy chapter folder: ${chapter.name}`);
pageCount += await validateChapter(path.join(bookDir, chapter.name));
chapterCount++;
}
}
const completedCount = new Set(state.completed || []).size;
if (!chapterCount || chapterCount !== completedCount) {
throw new Error(`Legacy work ${id} has ${chapterCount} complete folders but ${completedCount} completed checkpoint entries`);
}
return {
id, root, workRoot, stateFile, state, mangaDir, target,
mangaId: path.basename(target), chapterCount, pageCount, incompleteStages,
};
}
async function promoteLegacyWork(inspection) {
const { stateFile, mangaDir, target, state, workRoot } = inspection;
const quarantineRoot = path.join(workRoot, '.legacy-incomplete');
const quarantined = [];
if (inspection.incompleteStages.length) await fs.mkdir(quarantineRoot, { recursive: true, mode: 0o755 });
for (const stage of inspection.incompleteStages) {
const relative = path.relative(mangaDir, stage).split(path.sep).join('__');
const quarantine = path.join(quarantineRoot, relative);
await renameRetry(stage, quarantine);
quarantined.push({ stage, quarantine });
}
try {
await renameRetry(mangaDir, target);
} catch (error) {
for (const item of quarantined.reverse()) await renameRetry(item.quarantine, item.stage).catch(() => {});
throw error;
}
const migrated = {
...state,
publicationMode: 'incremental',
status: 'paused',
current: null,
jobCompleted: 0,
total: 0,
migratedToIncremental: {
time: new Date().toISOString(),
mangaId: inspection.mangaId,
chaptersPublished: inspection.chapterCount,
pagesPublished: inspection.pageCount,
incompleteStagesQuarantined: quarantined.length,
},
};
try {
await writeJsonAtomic(stateFile, migrated);
} catch (error) {
await renameRetry(target, mangaDir).catch(() => {});
for (const item of quarantined.reverse()) await renameRetry(item.quarantine, item.stage).catch(() => {});
throw error;
}
await fs.appendFile(path.join(workRoot, 'job.log'), JSON.stringify({
time: migrated.migratedToIncremental.time,
level: 'info',
message: `Published ${inspection.chapterCount} completed legacy chapters and switched to incremental mode`,
mangaId: inspection.mangaId,
}) + '\n').catch(() => {});
return {
seriesId: inspection.id,
mangaId: inspection.mangaId,
finalPath: target,
chaptersPublished: inspection.chapterCount,
pagesPublished: inspection.pageCount,
incompleteStagesQuarantined: quarantined.length,
};
}
module.exports = { inspectLegacyWork, promoteLegacyWork };
+175
View File
@@ -0,0 +1,175 @@
'use strict';
const fs = require('fs').promises;
const path = require('path');
const { assertInside } = require('./names');
const { chapterNumber } = require('./scope');
const { isSpecialKey, parseSpecialFolder, parseKey } = require('../../chapter-identity');
const { writeJsonAtomic } = require('./json-file');
const { resolveWorkIdentity, sameWorkIdentity } = require('./work-identity');
async function readJson(file) {
try { return JSON.parse(await fs.readFile(file, 'utf8')); } catch { return {}; }
}
async function directories(folder) {
try { return (await fs.readdir(folder, { withFileTypes: true })).filter(entry => entry.isDirectory() && !entry.name.startsWith('.')); }
catch { return []; }
}
function keyFromName(name) { return parseKey(name)?.key || null; }
function sourceKey(url, name = '') {
try {
const parsed = new URL(String(url));
const host = parsed.hostname.replace(/^www\./, '').toLowerCase();
if (parsed.protocol === 'peer:' && /^p[0-9a-f]{24}$/.test(host)) return `peer-${host}`;
if (host === 'weebcentral.com') return 'weebcentral';
if (host === 'mangadex.org') return 'mangadex';
if (host === 'mangapill.com') return 'mangapill';
return host;
} catch {}
const label = String(name).toLowerCase();
if (label.startsWith('mangadex')) return 'mangadex';
if (label.startsWith('weebcentral')) return 'weebcentral';
return label.replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '');
}
function autoUpdateFor(meta) {
const configured = meta?.['x-scraper']?.autoUpdate;
return configured === undefined ? String(meta?.status || '').toLowerCase() === 'ongoing' : !!configured;
}
async function scanLibrary(libraryRoot) {
const root = path.resolve(libraryRoot);
const mangas = [];
for (const mangaEntry of await directories(root)) {
const mangaPath = path.join(root, mangaEntry.name);
const meta = await readJson(path.join(mangaPath, 'manga.json'));
const chapters = [];
for (const bookEntry of await directories(mangaPath)) {
const book = keyFromName(bookEntry.name);
if (!book) continue;
const bookPath = path.join(mangaPath, bookEntry.name);
for (const versionEntry of await directories(bookPath)) {
const number = parseSpecialFolder(versionEntry.name)?.key || keyFromName(versionEntry.name);
if (!number) continue;
const chapterMeta = await readJson(path.join(bookPath, versionEntry.name, 'chapter.json'));
const source = typeof chapterMeta.source === 'string' ? { name: chapterMeta.source, url: '' } : (chapterMeta.source || {});
chapters.push({
book, bookFolder: bookEntry.name, number, version: versionEntry.name,
sourceName: String(source.name || ''), sourceUrl: String(source.url || ''),
sourceKey: sourceKey(source.url, source.name),
releaseKind: chapterMeta.releaseKind || null,
});
}
}
mangas.push({
id: mangaEntry.name, path: mangaPath, title: String(meta.title || mangaEntry.name), meta,
status: String(meta.status || 'unknown').toLowerCase(), autoUpdate: autoUpdateFor(meta),
ids: { ...(meta['x-scraper']?.ids || {}) }, sources: Array.isArray(meta.sources) ? meta.sources : [], chapters,
});
}
return { root, mangas };
}
function findManga(inventory, identity = {}) {
const source = String(identity.source || '');
const seriesId = identity.seriesId == null ? '' : String(identity.seriesId);
const sourceUrl = String(identity.sourceUrl || '');
const anilist = identity.anilist == null ? '' : String(identity.anilist);
const explicit = identity.libraryMangaId == null ? '' : String(identity.libraryMangaId);
if (explicit) return inventory.mangas.find(manga => manga.id === explicit) || null;
if (identity.workIdentity) {
const matches = inventory.mangas.filter(manga => sameWorkIdentity(
resolveWorkIdentity(manga.meta, manga.id, identity.localServerId), identity.workIdentity));
if (matches.length !== 1) return null;
if (source && seriesId && matches[0].ids[source] != null && String(matches[0].ids[source]) !== seriesId) return null;
return matches[0];
}
if (source && seriesId) {
const bySourceId = inventory.mangas.find(manga => String(manga.ids[source] ?? '') === seriesId);
if (bySourceId) return bySourceId;
}
if (sourceUrl) {
const byUrl = inventory.mangas.find(manga => manga.sources.some(item => item.url === sourceUrl));
if (byUrl) return byUrl;
}
if (anilist) {
const matches = inventory.mangas.filter(manga => String(manga.ids.anilist ?? manga.meta?.['x-anilist']?.id ?? '') === anilist);
// AniList identifies the work, not every pilot/edition on a source. Never
// silently merge a different listing from that same host into its sibling.
if (matches.length !== 1) return null;
if (source && seriesId && matches[0].ids[source] != null && String(matches[0].ids[source]) !== seriesId) return null;
return matches[0];
}
return null;
}
function chapterBook(ref, manga) {
if (manga?.meta?.['x-scraper']?.strategy === 'volumes' && ref.volume != null) return chapterNumber(ref.volume, 'volume');
return chapterNumber(ref.book?.number || 1, 'book');
}
function existingVersions(manga, ref) {
if (!manga) return [];
const book = chapterBook(ref, manga);
const number = isSpecialKey(ref.number) ? String(ref.number).toLowerCase() : chapterNumber(ref.number);
if (ref.releaseKind === 'volume') return manga.chapters.filter(item => item.book === book && item.releaseKind === 'volume');
return manga.chapters.filter(item => item.book === book && item.number === number);
}
function shouldDownload(manga, ref, source, existingPolicy = 'skip-any') {
const versions = existingVersions(manga, ref);
// A whole-volume archive can overlap every numbered chapter in that book.
// Require an explicit add-source choice before storing both representations.
if (ref.releaseKind === 'volume' && existingPolicy === 'skip-any' && manga?.chapters.some(item =>
item.book === chapterBook(ref, manga) && item.releaseKind !== 'volume')) return false;
if (!versions.length) return true;
if (existingPolicy === 'add-source') return !versions.some(item => item.sourceKey === source);
return false;
}
function isNewerThanLibrary(manga, ref) {
if (isSpecialKey(ref.number)) return !existingVersions(manga, ref).length;
if (!manga?.chapters?.length) return true;
const regular = manga.chapters.filter(item => !String(item.book).startsWith('x') && /^\d+(?:\.\d+)?$/.test(String(item.number)));
if (!regular.length) return true;
const latest = regular.reduce((best, item) => {
if (!best) return item;
return Number(item.book) > Number(best.book)
|| (Number(item.book) === Number(best.book) && Number(item.number) > Number(best.number)) ? item : best;
}, null);
const candidate = { book: chapterBook(ref, manga), number: chapterNumber(ref.number) };
return Number(candidate.book) > Number(latest.book)
|| (Number(candidate.book) === Number(latest.book) && Number(candidate.number) > Number(latest.number));
}
function summarizeInventory(inventory, supported) {
return inventory.mangas.map(manga => ({
id: manga.id, title: manga.title, status: manga.status, autoUpdate: manga.autoUpdate,
chapters: new Set(manga.chapters.map(item => `${item.book}/${item.number}`)).size,
versions: manga.chapters.length,
sources: manga.sources.map(source => {
const found = supported(source.url);
return { name: source.name, url: source.url, adapter: found?.adapter.id || null };
}),
}));
}
async function setAutoUpdate(libraryRoot, mangaId, enabled) {
const root = path.resolve(libraryRoot);
const folder = assertInside(root, path.join(root, String(mangaId)));
const stat = await fs.stat(folder).catch(() => null);
if (!stat?.isDirectory() || path.basename(folder).startsWith('.')) throw new Error('Library manga not found');
const file = path.join(folder, 'manga.json');
const meta = await readJson(file);
meta['x-scraper'] = { ...(meta['x-scraper'] || {}), autoUpdate: !!enabled };
await writeJsonAtomic(file, meta);
return { id: mangaId, autoUpdate: !!enabled };
}
module.exports = {
scanLibrary, findManga, existingVersions, shouldDownload, summarizeInventory,
isNewerThanLibrary, setAutoUpdate, sourceKey, autoUpdateFor, keyFromName,
};
+41
View File
@@ -0,0 +1,41 @@
'use strict';
const IMPORTED_FIELDS = ['title', 'altTitles', 'authors', 'status', 'year', 'tags', 'description'];
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
const { normalizeWorkIdentity, sameWorkIdentity } = require('./work-identity');
// The effective fields remain at the top of manga.json. A snapshot of the last
// imported values lets later imports refresh unchanged fields without replacing
// values somebody edited in the library. Legacy files have no snapshot, so their
// existing values are conservatively treated as intentional.
function mergeImportedMetadata(prior, imported, fallbackTitle) {
const previous = prior['x-metadata']?.imported || {};
const result = { ...prior };
for (const field of IMPORTED_FIELDS) {
if (!Object.hasOwn(imported, field)) continue;
if (result[field] === undefined || result[field] === null || result[field] === ''
|| (Object.hasOwn(previous, field) && same(result[field], previous[field]))) {
result[field] = imported[field];
}
}
if (!result.title) result.title = fallbackTitle;
const oldId = prior['x-anilist']?.id;
const newId = imported['x-anilist']?.id;
if (oldId != null && newId != null && String(oldId) !== String(newId)) {
throw new Error(`AniList identity conflict: existing ${oldId}, imported ${newId}; review the manga match`);
}
if (imported['x-anilist']) result['x-anilist'] = { ...(prior['x-anilist'] || {}), ...imported['x-anilist'] };
const oldWorkIdentity = normalizeWorkIdentity(prior['x-work-identity']);
const newWorkIdentity = normalizeWorkIdentity(imported['x-work-identity']);
if (oldWorkIdentity && newWorkIdentity && !sameWorkIdentity(oldWorkIdentity, newWorkIdentity)) {
throw new Error(`MangaCircus work identity conflict: existing ${oldWorkIdentity.kind} identity differs from imported ${newWorkIdentity.kind} identity`);
}
if (newWorkIdentity) result['x-work-identity'] = newWorkIdentity;
const snapshot = { ...previous };
for (const field of IMPORTED_FIELDS) if (Object.hasOwn(imported, field)) snapshot[field] = imported[field];
if (Object.keys(snapshot).length) result['x-metadata'] = { ...(prior['x-metadata'] || {}), imported: snapshot };
return result;
}
module.exports = { mergeImportedMetadata, IMPORTED_FIELDS };
+50
View File
@@ -0,0 +1,50 @@
'use strict';
const path = require('path');
const { isSpecialKey } = require('../../chapter-identity');
const RESERVED = /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i;
function cleanPart(value, max = 120) {
let result = String(value || '')
.normalize('NFC')
.replace(/[<>:"/\\|?*\x00-\x1f]/g, '')
.replace(/[\[\]]/g, '')
.replace(/\s+/g, ' ')
.trim()
.replace(/[ .]+$/g, '');
if (!result) result = 'Untitled';
if (RESERVED.test(result)) result += '-manga';
return result.slice(0, max).replace(/[ .]+$/g, '');
}
function chapterKey(value) {
const raw = String(value).trim();
if (isSpecialKey(raw)) return raw.toLowerCase();
if (/^\d+(?:\.\d+)?$/.test(raw)) {
const [integer, decimal = ''] = raw.split('.');
const normalized = decimal.replace(/0+$/g, '');
return `c${integer.padStart(4, '0')}${normalized ? `.${normalized}` : ''}`;
}
const extra = raw.match(/^x(\d+)$/i);
if (extra) return `x${extra[1].padStart(2, '0')}`;
throw new Error(`Unsupported chapter number: ${value}`);
}
function chapterFolder(chapter) {
const parts = [chapterKey(chapter.number)];
if (chapter.title) parts.push(cleanPart(chapter.title));
parts.push(`[${cleanPart(chapter.source.name, 50)}]`);
return parts.join(' ');
}
function assertInside(parent, child) {
const root = path.resolve(parent);
const target = path.resolve(child);
if (target !== root && !target.startsWith(root + path.sep)) {
throw new Error(`Path escapes library root: ${target}`);
}
return target;
}
module.exports = { cleanPart, chapterKey, chapterFolder, assertInside };
+301
View File
@@ -0,0 +1,301 @@
'use strict';
const fs = require('fs');
const fsp = fs.promises;
const path = require('path');
const crypto = require('crypto');
const { SlowHttp } = require('./http');
const { imageExtension } = require('./images');
const { cleanPart, chapterFolder, assertInside } = require('./names');
const { sizeFromBuffer, imageSize } = require('../../library');
const { directionFields } = require('../../reading-direction');
const { mergeImportedMetadata } = require('./metadata-merge');
const { normalizeWorkIdentity } = require('./work-identity');
const { parseKey } = require('../../chapter-identity');
const token = () => crypto.randomBytes(5).toString('hex');
// Staging is a recovery asset, not disposable garbage. Legacy stages without an
// identity must be reviewed manually; never infer a chapter from page filenames.
const ownTemp = name => /^\.old-mcscraper-[a-f0-9]+$/.test(name);
const digest = value => crypto.createHash('sha256').update(JSON.stringify(value)).digest('hex');
const coverFile = name => /^cover\.(?:jpe?g|png|webp|gif|avif|bmp)$/i.test(name);
async function exists(file) {
try { await fsp.access(file); return true; } catch { return false; }
}
async function readMetadata(file) {
try { return JSON.parse((await fsp.readFile(file, 'utf8')).replace(/^\uFEFF/, '')); }
catch (error) { if (error.code === 'ENOENT') return {}; throw error; }
}
function pageDimensions(size, tags = []) {
const rotation = tags.find(tag => /^rot(90|180|270)$/.test(tag));
return { w: size?.w || 0, h: size?.h || 0, r: rotation ? Number(rotation.slice(3)) : 0 };
}
async function publishedDimensions(folder) {
const pages = [];
for (const name of (await fsp.readdir(folder)).sort()) {
if (!/\.(?:jpe?g|png|webp|gif|avif|bmp)$/i.test(name)) continue;
const tags = name.slice(0, name.lastIndexOf('.')).split('_').slice(1);
pages.push(pageDimensions(await imageSize(path.join(folder, name)).catch(() => null), tags));
}
return pages;
}
async function renameRetry(from, to, tries = 5) {
for (let attempt = 0; ; attempt++) {
try { await fsp.rename(from, to); return; } catch (error) {
if (attempt >= tries || !['EPERM', 'EBUSY', 'EACCES'].includes(error.code)) throw error;
await new Promise(resolve => setTimeout(resolve, 200 * (attempt + 1)));
}
}
}
async function writeJsonAtomic(file, value) {
const temp = path.join(path.dirname(file), `.${path.basename(file)}.tmp`);
await fsp.writeFile(temp, JSON.stringify(value, null, 2) + '\n', { mode: 0o644 });
if (!(await exists(file))) return renameRetry(temp, file);
try {
await renameRetry(temp, file);
} catch (error) {
if (!['EPERM', 'EBUSY', 'EACCES', 'EEXIST'].includes(error.code)) throw error;
const old = path.join(path.dirname(file), `.old-mcscraper-${token()}`);
await renameRetry(file, old);
try { await renameRetry(temp, file); } catch (moveError) { await renameRetry(old, file); throw moveError; }
await fsp.rm(old, { force: true });
}
}
async function cleanupOwnTemps(folder, maxAgeMs = 6 * 60 * 60 * 1000) {
if (!(await exists(folder))) return;
const now = Date.now();
for (const entry of await fsp.readdir(folder, { withFileTypes: true })) {
if (!entry.isDirectory() || !ownTemp(entry.name)) continue;
const target = path.join(folder, entry.name);
const stat = await fsp.stat(target);
if (now - stat.mtimeMs > maxAgeMs) await fsp.rm(target, { recursive: true, force: true });
}
}
async function findManga(libraryRoot, sourceUrls) {
if (!(await exists(libraryRoot))) return null;
for (const entry of await fsp.readdir(libraryRoot, { withFileTypes: true })) {
if (!entry.isDirectory() || entry.name.startsWith('.')) continue;
const file = path.join(libraryRoot, entry.name, 'manga.json');
try {
const json = JSON.parse(await fsp.readFile(file, 'utf8'));
if ((json.sources || []).some(source => sourceUrls.has(source.url))) return path.join(libraryRoot, entry.name);
} catch {}
}
return null;
}
async function bookDirectory(mangaDir, bookNumber, bookTitle) {
const names = (await fsp.readdir(mangaDir, { withFileTypes: true }))
.filter(entry => entry.isDirectory() && parseKey(entry.name)?.key === String(bookNumber))
.map(entry => entry.name);
if (names.length > 1) throw new Error(`Ambiguous library book ${bookNumber}: ${names.join(', ')}`);
const defaultName = `v${String(bookNumber).padStart(2, '0')}`;
return path.join(mangaDir, names[0] || (bookTitle ? `${defaultName} ${cleanPart(bookTitle)}` : defaultName));
}
function validateManifest(manifest) {
if (!manifest || manifest.version !== 1) throw new Error('Manifest version must be 1');
if (!manifest.series?.title) throw new Error('series.title is required');
if (!['rtl', 'ltr', 'ttb'].includes(manifest.series.direction)) throw new Error('series.direction must be rtl, ltr, or ttb');
if (!Array.isArray(manifest.series.sources) || !manifest.series.sources.length) throw new Error('series.sources is required');
if (!Array.isArray(manifest.chapters) || !manifest.chapters.length) throw new Error('At least one chapter is required');
for (const chapter of manifest.chapters) {
if (chapter.number == null || chapter.number === '' || !chapter.source?.name || !chapter.source?.url) throw new Error('Every chapter needs number and source {name,url}');
if (!Array.isArray(chapter.pages) || !chapter.pages.length) throw new Error(`Chapter ${chapter.number} has no pages`);
for (const page of chapter.pages) {
if (!!page.url === !!page.path) throw new Error('Every page must have exactly one of url or path');
for (const tag of page.tags || []) if (!/^[a-z0-9]+$/.test(tag)) throw new Error(`Invalid page tag: ${tag}`);
}
}
}
async function pageBytes(page, http) {
if (page.path) return fsp.readFile(path.resolve(page.path));
const response = await http.get(page.url, { headers: page.headers || {} });
return Buffer.from(await response.arrayBuffer());
}
function validateCoverUrl(value) {
const url = new URL(value);
if (url.protocol !== 'https:' || !(url.hostname === 'anilist.co' || url.hostname.endsWith('.anilist.co'))) {
throw new Error('Series cover must use the AniList HTTPS image host');
}
return url.href;
}
async function installSeriesCover(mangaDir, coverUrl, http) {
const files = await fsp.readdir(mangaDir);
if (files.some(coverFile)) return { status: 'existing' };
const response = await http.get(validateCoverUrl(coverUrl));
const data = Buffer.from(await response.arrayBuffer());
if (data.length > 10 * 1024 * 1024) throw new Error('Series cover exceeds 10 MiB');
const ext = imageExtension(data);
if (!ext) throw new Error('Series cover is not a supported image');
const temp = path.join(mangaDir, `.cover-mcscraper-${token()}.${ext}`);
const target = path.join(mangaDir, `cover.${ext}`);
await fsp.writeFile(temp, data, { mode: 0o644 });
try { await renameRetry(temp, target); } catch (error) { await fsp.rm(temp, { force: true }); throw error; }
return { status: 'installed', file: path.basename(target) };
}
async function stageChapter(bookDir, chapter, http, onPage, priorMeta = {}) {
// Full manifest fingerprint prevents reusing a page from a changed release.
// Store only the hash, never image URLs/headers (which may contain signatures).
const identity = digest([chapterFolder(chapter), chapter.source, chapter.pages.map(p => [p.url || path.resolve(p.path), p.tags || []])]);
const stage = path.join(bookDir, `.staging-mcscraper-resume-${identity}`);
await fsp.mkdir(stage, { recursive: true, mode: 0o755 });
const dimensions = [];
{
for (let index = 0; index < chapter.pages.length; index++) {
const tags = (chapter.pages[index].tags || []).map(tag => `_${tag}`).join('');
const prefix = `${String(index + 1).padStart(4, '0')}${tags}`;
const prior = (await fsp.readdir(stage)).filter(name => name.startsWith(prefix + '.') && !name.endsWith('.tmp'));
if (prior.length === 1) {
const bytes = await fsp.readFile(path.join(stage, prior[0]));
if (imageExtension(bytes) === path.extname(prior[0]).slice(1)) {
dimensions.push(pageDimensions(sizeFromBuffer(bytes), chapter.pages[index].tags));
if (onPage) await onPage({ completed: index + 1, total: chapter.pages.length });
continue;
}
}
if (prior.length) throw new Error(`Partial page ${index + 1} requires review: invalid or ambiguous image`);
const data = await pageBytes(chapter.pages[index], http);
const ext = imageExtension(data);
if (!ext) throw new Error(`Page ${index + 1} is not a supported image`);
dimensions.push(pageDimensions(sizeFromBuffer(data), chapter.pages[index].tags));
const target = path.join(stage, `${prefix}.${ext}`);
const pending = path.join(stage, `.${prefix}.tmp`);
await fsp.writeFile(pending, data, { mode: 0o644 });
await renameRetry(pending, target);
if (onPage) await onPage({ completed: index + 1, total: chapter.pages.length });
}
await fsp.writeFile(path.join(stage, 'chapter.json'), JSON.stringify({
...priorMeta,
title: chapter.title || '', source: chapter.source,
added: chapter.added || new Date().toISOString(),
priority: Number(chapter.priority || 0), extra: !!chapter.extra,
...(chapter.special ? { special: true, label: chapter.label || 'Special', sourceId: chapter.sourceId,
sourceNumber: chapter.sourceNumber ?? null,
...(chapter.releaseKind ? { releaseKind: chapter.releaseKind } : {}),
...(chapter.volume ? { volume: chapter.volume } : {}) } : {}),
...directionFields([priorMeta, chapter], dimensions),
}, null, 2) + '\n', { mode: 0o644 });
// A process interruption during a write can leave a hidden pending file.
for (const name of await fsp.readdir(stage)) {
if (/^\.\d{4,5}(?:_[a-z0-9]+)*\.tmp$/.test(name)) await fsp.rm(path.join(stage, name), { force: true });
}
return { stage, dimensions };
}
}
async function publishStaged(stage, target, replace) {
if (!(await exists(target))) return renameRetry(stage, target);
if (!replace) { await fsp.rm(stage, { recursive: true, force: true }); return false; }
const old = path.join(path.dirname(target), `.old-mcscraper-${token()}`);
await renameRetry(target, old);
try { await renameRetry(stage, target); } catch (error) { await renameRetry(old, target); throw error; }
await fsp.rm(old, { recursive: true, force: true });
return true;
}
async function publishManifest(manifest, options = {}) {
validateManifest(manifest);
const libraryRoot = path.resolve(options.libraryRoot || manifest.libraryRoot || 'library');
await fsp.mkdir(libraryRoot, { recursive: true, mode: 0o755 });
await cleanupOwnTemps(libraryRoot);
const sourceUrls = new Set(manifest.series.sources.map(source => source.url));
let mangaDir = options.mangaDir ? assertInside(libraryRoot, options.mangaDir) : await findManga(libraryRoot, sourceUrls);
if (mangaDir && !(await exists(mangaDir))) throw new Error(`Target manga folder does not exist: ${mangaDir}`);
if (mangaDir && path.basename(mangaDir).startsWith('.')) throw new Error('Target manga folder must be published');
const isNew = !mangaDir;
const finalName = cleanPart(manifest.series.folder || manifest.series.title);
const finalDir = assertInside(libraryRoot, path.join(libraryRoot, finalName));
if (!mangaDir && await exists(finalDir)) {
const existing = await readMetadata(path.join(finalDir, 'manga.json'));
const incomingIdentity = normalizeWorkIdentity(manifest.series.metadata?.['x-work-identity']);
const existingIdentity = normalizeWorkIdentity(existing['x-work-identity']);
if (incomingIdentity && existingIdentity && incomingIdentity.kind === existingIdentity.kind
&& incomingIdentity.value === existingIdentity.value) {
throw new Error(`Work identity matches the existing “${existing.title || finalName}” folder, but its source association was not resolved. Review the existing manga target; no files were overwritten.`);
}
throw new Error(`A library folder named “${finalName}” already exists, but this download could not be safely matched to it. No files were overwritten. Choose the existing manga as the target or review its work identity before syncing.`);
}
mangaDir = mangaDir || path.join(libraryRoot, `.staging-mcscraper-resume-${digest([finalName, [...sourceUrls].sort()])}`);
await fsp.mkdir(mangaDir, { recursive: true, mode: 0o755 });
const http = options.httpClient || new SlowHttp(options.http || manifest.http || {});
const results = [];
const dimensions = [];
let cover = null;
const prior = await readMetadata(path.join(mangaDir, 'manga.json'));
const owned = manifest.series.metadata || {};
// Validate identity before fetching or exposing any chapter pages.
const merged = mergeImportedMetadata(prior, owned, manifest.series.title);
try {
for (const chapter of manifest.chapters) {
const bookNumber = Number(chapter.book?.number || 1);
if (!Number.isInteger(bookNumber) || bookNumber < 1) throw new Error(`Invalid book number for chapter ${chapter.number}`);
const bookDir = await bookDirectory(mangaDir, bookNumber, chapter.book?.title);
await fsp.mkdir(bookDir, { recursive: true, mode: 0o755 });
await cleanupOwnTemps(bookDir);
if (chapter.book?.title) {
const file = path.join(bookDir, 'book.json'), priorBook = await readMetadata(file);
await writeJsonAtomic(file, { ...priorBook, title: chapter.book.title,
...(chapter.book.direction || chapter.book.directionOverride
? directionFields([priorBook, chapter.book], []) : {}) });
}
const folder = chapterFolder(chapter);
const target = path.join(bookDir, folder);
if (await exists(target) && !options.replace) {
dimensions.push(...await publishedDimensions(target));
results.push({ chapter: chapter.number, status: 'skipped' }); continue;
}
const priorChapter = await readMetadata(path.join(target, 'chapter.json'));
const staged = await stageChapter(bookDir, chapter, http, options.onPage, priorChapter);
dimensions.push(...staged.dimensions);
await publishStaged(staged.stage, target, !!options.replace);
results.push({ chapter: chapter.number, status: 'published', folder });
}
const sources = [...(prior.sources || [])];
for (const source of manifest.series.sources) {
const index = sources.findIndex(item => item.url === source.url);
if (index < 0) sources.push(source); else sources[index] = { ...sources[index], ...source };
}
const priorScraper = prior['x-scraper'] || {};
const nextScraper = { ...priorScraper, strategy: priorScraper.strategy || 'flat', ...(manifest.series.scraper || {}) };
await writeJsonAtomic(path.join(mangaDir, 'manga.json'), {
...merged,
...directionFields([prior, merged, manifest.series], dimensions),
sources,
'x-scraper': nextScraper,
});
if (manifest.series.coverUrl && !manifest.series.coverAttempted) {
manifest.series.coverAttempted = true;
try {
cover = await installSeriesCover(mangaDir, manifest.series.coverUrl, http);
} catch (error) {
cover = { status: 'error', message: error.message };
console.warn(`[publisher] cover unavailable for ${manifest.series.title}: ${error.message}`);
}
}
if (isNew) await renameRetry(mangaDir, finalDir);
return { mangaDir: isNew ? finalDir : mangaDir, results, cover };
} catch (error) {
// Preserve hidden validated pages on pause, transient failure or shutdown.
// Publication still occurs only after the whole chapter validates.
throw error;
}
}
module.exports = { publishManifest, validateManifest, cleanupOwnTemps, installSeriesCover, validateCoverUrl, writeJsonAtomic };
+74
View File
@@ -0,0 +1,74 @@
'use strict';
const PUBLIC_MODES = new Set(['missing', 'latest', 'last', 'first', 'chapter', 'after', 'volume']);
function chapterNumber(value, field = 'chapter') {
const raw = String(value ?? '').trim();
if (!/^\d+(?:\.\d+)?$/.test(raw)) throw new Error(`${field} must be a numeric chapter`);
const [whole, decimal = ''] = raw.split('.');
const normalizedWhole = whole.replace(/^0+(?=\d)/, '') || '0';
const normalizedDecimal = decimal.replace(/0+$/, '');
return normalizedDecimal ? `${normalizedWhole}.${normalizedDecimal}` : normalizedWhole;
}
function positiveCount(value) {
const count = Number(value);
if (!Number.isInteger(count) || count < 1 || count > 10000) throw new Error('count must be an integer from 1 to 10000');
return count;
}
function normalizeScope(input = {}) {
const raw = input && typeof input === 'object' ? input : {};
const mode = String(raw.mode || 'missing').toLowerCase();
if (mode === 'ids') {
if (!Array.isArray(raw.ids) || !raw.ids.length || raw.ids.length > 10000) throw new Error('ids scope requires 1 to 10000 source chapter ids');
return { mode, ids: [...new Set(raw.ids.map(value => String(value).trim()).filter(Boolean))] };
}
if (!PUBLIC_MODES.has(mode)) throw new Error(`Unsupported queue scope: ${mode}`);
if (mode === 'last' || mode === 'first') return { mode, count: positiveCount(raw.count) };
if (mode === 'chapter' || mode === 'after') return { mode, chapter: chapterNumber(raw.chapter) };
if (mode === 'volume') {
const volume = String(raw.volume ?? '').trim();
if (!/^\d+(?:\.\d+)?$/.test(volume)) throw new Error('volume must be numeric');
return { mode, volume: chapterNumber(volume, 'volume') };
}
return { mode };
}
function volumeOf(ref) {
const value = ref.volume ?? ref.book?.number;
if (value === undefined || value === null || value === '') return null;
try { return chapterNumber(value, 'volume'); } catch { return null; }
}
function selectByScope(chapters, input) {
const scope = normalizeScope(input);
const list = [...chapters];
if (scope.mode === 'missing') return list;
if (scope.mode === 'latest') return list.length ? [list[list.length - 1]] : [];
if (scope.mode === 'last') return list.slice(-scope.count);
if (scope.mode === 'first') return list.slice(0, scope.count);
const numbered = list.filter(ref => /^\d+(?:\.\d+)?$/.test(String(ref.number)));
if (scope.mode === 'chapter') return numbered.filter(ref => chapterNumber(ref.number) === scope.chapter);
if (scope.mode === 'after') return numbered.filter(ref => Number(chapterNumber(ref.number)) > Number(scope.chapter));
if (scope.mode === 'volume') return list.filter(ref => volumeOf(ref) === scope.volume);
if (scope.mode === 'ids') {
const ids = new Set(scope.ids);
return list.filter(ref => ids.has(String(ref.id)));
}
return [];
}
function scopeLabel(input) {
const scope = normalizeScope(input);
if (scope.mode === 'missing') return 'all missing chapters';
if (scope.mode === 'latest') return 'newest chapter';
if (scope.mode === 'last') return `last ${scope.count} chapters`;
if (scope.mode === 'first') return `first ${scope.count} chapters`;
if (scope.mode === 'chapter') return `chapter ${scope.chapter}`;
if (scope.mode === 'after') return `chapters after ${scope.chapter}`;
if (scope.mode === 'volume') return `volume ${scope.volume}`;
return `${scope.ids.length} discovered chapters`;
}
module.exports = { normalizeScope, selectByScope, scopeLabel, chapterNumber, volumeOf };
+54
View File
@@ -0,0 +1,54 @@
'use strict';
const { normalizeWorkIdentity } = require('./work-identity');
function workKey(item) {
const identity = normalizeWorkIdentity(item.workIdentity);
if (identity) return `${identity.scheme}:${identity.kind}:${identity.value}`;
if (item.libraryMangaId) return `library:${item.libraryMangaId}`;
return `source:${item.source}:${item.seriesId || item.url}`;
}
function sourceKey(item) {
return `${item.source}:${item.seriesId || item.url}`;
}
function recordSource(ledger, item, observation = {}) {
const key = sourceKey(item);
const prior = ledger[key] || {};
const availability = observation.availability || item.availability;
const count = observation.catalogTotal ?? item.catalogTotal;
const observedAt = observation.observedAt || (count != null ? item.checked || item.completed : null) || null;
const catalog = count != null ? 'found' : prior.catalog || 'unverified';
const adapter = count != null
? Number(count) > 0 ? 'compatible' : availability?.reason === 'unrecognized-labels' ? 'unsupported-labels' : 'no-selected-pages'
: prior.adapter || 'unverified';
ledger[key] = {
...prior, workKey: workKey(item), source: item.source, seriesId: item.seriesId || null,
url: item.url || prior.url || null, title: item.title || prior.title || null,
sourceTitle: observation.sourceTitle || item.sourceTitle || prior.sourceTitle || null,
language: availability?.language || prior.language || null,
catalog, adapter, rights: prior.rights || 'unverified',
hostedCount: count == null ? prior.hostedCount ?? null : Number(count),
externalCount: availability?.externalReleases ?? prior.externalCount ?? null,
coverage: availability?.coverage || prior.coverage || 'unknown',
reason: observation.availability ? availability.reason || null : availability?.reason || prior.reason || null,
lastObservedAt: observedAt || prior.lastObservedAt || null,
evidence: count != null && observedAt ? { kind: 'scraper-catalog', jobId: item.id || null } : prior.evidence || { kind: 'queue-url', jobId: item.id || null },
};
if (observation.catalogTotal != null && Number(count) > 0) delete ledger[key].lastFailure;
return ledger[key];
}
function recordFailure(ledger, item, failure, at = new Date().toISOString()) {
const row = recordSource(ledger, item);
row.lastFailure = { kind: failure.kind, message: String(failure.message || '').slice(0, 300), at };
return row;
}
function ledgerRows(ledger) {
return Object.values(ledger || {}).sort((a, b) => String(a.title || '').localeCompare(String(b.title || ''))
|| String(a.source || '').localeCompare(String(b.source || '')));
}
module.exports = { workKey, sourceKey, recordSource, recordFailure, ledgerRows };
+49
View File
@@ -0,0 +1,49 @@
'use strict';
const crypto = require('node:crypto');
const SCHEME = 'mangacircus-work-v1';
const KINDS = new Set(['anilist', 'mangadex', 'weebcentral', 'hoster', 'origin']);
function normalizeWorkIdentity(value) {
if (!value || typeof value !== 'object' || value.scheme !== SCHEME
|| !KINDS.has(value.kind) || typeof value.value !== 'string'
|| !value.value.trim() || value.value.length > 300) return null;
return { scheme: SCHEME, kind: value.kind, value: value.value.trim() };
}
function resolveWorkIdentity(meta = {}, mangaId, serverId) {
const carried = normalizeWorkIdentity(meta['x-work-identity']);
if (carried) return carried;
const anilist = meta['x-anilist']?.id ?? meta['x-scraper']?.ids?.anilist;
if (anilist != null && String(anilist).trim()) return { scheme: SCHEME, kind: 'anilist', value: String(anilist).trim() };
const ids = meta['x-scraper']?.ids || {};
for (const source of ['mangadex', 'weebcentral']) {
if (ids[source] != null && String(ids[source]).trim()) {
return { scheme: SCHEME, kind: source, value: String(ids[source]).trim() };
}
}
for (const source of Object.keys(ids).sort()) {
if (!['anilist', 'mangadex', 'weebcentral'].includes(source) && !source.startsWith('peer-')
&& ids[source] != null && String(ids[source]).trim()) {
return { scheme: SCHEME, kind: 'hoster', value: `${source}:${String(ids[source]).trim()}` };
}
}
// A private, stable lineage ID lets a title safely make a round trip even if
// neither server has external metadata. It reveals neither the folder nor title.
if (typeof mangaId === 'string' && mangaId && typeof serverId === 'string' && serverId) {
const value = crypto.createHash('sha256').update(`${serverId}\0${mangaId}`).digest('hex');
return { scheme: SCHEME, kind: 'origin', value };
}
return null;
}
function sameWorkIdentity(left, right) {
const a = normalizeWorkIdentity(left), b = normalizeWorkIdentity(right);
return !!a && !!b && a.scheme === b.scheme && a.kind === b.kind && a.value === b.value;
}
module.exports = { SCHEME, normalizeWorkIdentity, resolveWorkIdentity, sameWorkIdentity };
+22
View File
@@ -0,0 +1,22 @@
{
"version": 1,
"libraryRoot": "library",
"series": {
"title": "Example Manga",
"direction": "rtl",
"sources": [{ "name": "Example", "url": "https://example.invalid/series/1" }],
"metadata": { "authors": ["Example Author"], "status": "ongoing" },
"scraper": { "ids": { "example": "1" } }
},
"http": { "minMs": 8000, "maxMs": 15000, "maxRetries": 2 },
"chapters": [{
"number": "1",
"title": "Opening",
"added": "2026-01-01T00:00:00Z",
"source": { "name": "Example", "url": "https://example.invalid/chapter/1" },
"pages": [
{ "path": "C:/absolute/path/0001.jpg", "tags": ["cover"] },
{ "url": "https://example.invalid/images/2.jpg", "headers": { "referer": "https://example.invalid/chapter/1" } }
]
}]
}
+87
View File
@@ -0,0 +1,87 @@
'use strict';
const fs = require('fs').promises;
const path = require('path');
const crypto = require('crypto');
const { SlowHttp } = require('../lib/http');
const QUERY = `query ($search: String) {
Media(search: $search, type: MANGA, format: MANGA) {
id idMal format title { romaji english native } synonyms
staff(perPage: 20) { edges { role node { name { full } } } }
status startDate { year } genres description(asHtml: false)
countryOfOrigin siteUrl coverImage { extraLarge }
relations { edges { relationType node { id type format title { romaji english native } siteUrl } } }
}
}`;
const QUERY_BY_ID = `query ($id: Int) {
Media(id: $id, type: MANGA, format: MANGA) {
id idMal format title { romaji english native } synonyms
staff(perPage: 20) { edges { role node { name { full } } } }
status startDate { year } genres description(asHtml: false)
countryOfOrigin siteUrl coverImage { extraLarge }
relations { edges { relationType node { id type format title { romaji english native } siteUrl } } }
}
}`;
function stripHtml(value) {
return String(value || '').replace(/<br\s*\/?>/gi, '\n').replace(/<[^>]+>/g, '').trim();
}
function metadataFromMedia(media) {
return {
title: media.title.english || media.title.romaji,
altTitles: [...new Set([media.title.romaji, media.title.native, ...(media.synonyms || [])].filter(Boolean))],
authors: [...new Set((media.staff?.edges || [])
.filter(edge => /^(?:original story|story(?: & art)?|art|illustration)$/i.test(edge.role || ''))
.map(edge => edge.node?.name?.full).filter(Boolean))],
status: media.status === 'FINISHED' ? 'completed' : media.status === 'RELEASING' ? 'ongoing' : 'unknown',
year: media.startDate?.year || undefined,
tags: media.genres || [], description: stripHtml(media.description),
'x-anilist': { id: media.id, idMal: media.idMal, format: media.format, url: media.siteUrl, coverUrl: media.coverImage?.extraLarge,
relations: (media.relations?.edges || []).filter(edge => edge.node?.type === 'MANGA').map(edge => ({ id: edge.node.id, type: edge.relationType,
format: edge.node.format, title: edge.node.title?.english || edge.node.title?.romaji || edge.node.title?.native, url: edge.node.siteUrl })) },
};
}
async function lookup(query, variables, cacheKey, options = {}) {
const cacheDir = path.resolve(options.cacheDir || process.env.METADATA_CACHE || path.join(__dirname, '..', '.cache', 'anilist'));
await fs.mkdir(cacheDir, { recursive: true });
const cacheFile = path.join(cacheDir, `${cacheKey}.json`);
try {
const stat = await fs.stat(cacheFile);
const cached = JSON.parse(await fs.readFile(cacheFile, 'utf8'));
if (cached._cacheVersion === 4 && Date.now() - stat.mtimeMs < (options.maxAgeMs || 30 * 86400000)) return cached.metadata;
} catch {}
const http = new SlowHttp({ minMs: 0, maxMs: 0, maxRetries: 1 });
const response = await fetch('https://graphql.anilist.co', {
method: 'POST', headers: { 'content-type': 'application/json', 'user-agent': http.userAgent },
body: JSON.stringify({ query, variables }),
signal: AbortSignal.timeout(30000),
});
if (!response.ok) throw Object.assign(new Error(`AniList HTTP ${response.status}`), { status: response.status });
const body = await response.json();
if (body.errors?.length) throw new Error(body.errors[0].message);
const metadata = metadataFromMedia(body.data.Media);
const temporary = `${cacheFile}.${crypto.randomUUID()}.tmp`;
try {
await fs.writeFile(temporary, JSON.stringify({ _cacheVersion: 4, metadata }, null, 2) + '\n');
await fs.rename(temporary, cacheFile);
} finally { await fs.rm(temporary, { force: true }); }
return metadata;
}
async function lookupAniList(search, options = {}) {
const key = Buffer.from(search).toString('base64url');
return lookup(QUERY, { search }, key, options);
}
async function lookupAniListById(id, options = {}) {
const value = Number(id);
if (!Number.isInteger(value) || value < 1) throw new Error('AniList id must be a positive integer');
return lookup(QUERY_BY_ID, { id: value }, `id-${value}`, options);
}
module.exports = { lookupAniList, lookupAniListById, metadataFromMedia };
+107
View File
@@ -0,0 +1,107 @@
'use strict';
const dns = require('node:dns').promises;
const https = require('node:https');
const { BlockList, isIP } = require('node:net');
const { SlowHttp } = require('../lib/http');
const blocked = new BlockList();
for (const [address, bits] of [
['0.0.0.0', 8], ['10.0.0.0', 8], ['100.64.0.0', 10], ['127.0.0.0', 8],
['169.254.0.0', 16], ['172.16.0.0', 12], ['192.0.0.0', 24],
['192.0.2.0', 24], ['192.168.0.0', 16], ['198.18.0.0', 15],
['198.51.100.0', 24], ['203.0.113.0', 24], ['224.0.0.0', 4],
['240.0.0.0', 4], ['::', 128], ['::1', 128], ['fc00::', 7],
['fe80::', 10], ['ff00::', 8], ['2001:db8::', 32],
]) blocked.addSubnet(address, bits, isIP(address) === 6 ? 'ipv6' : 'ipv4');
function publicAddress(address) {
const family = isIP(address);
return !!family && !address.toLowerCase().startsWith('::ffff:')
&& !blocked.check(address, family === 4 ? 'ipv4' : 'ipv6');
}
function validateWeebRequest(value, options = {}) {
return validateHostRequest(value, 'weebcentral.com', options);
}
function validateHostRequest(value, sourceHost, options = {}) {
const url = new URL(value);
const host = url.hostname.toLowerCase();
if (url.protocol !== 'https:' || url.username || url.password || url.port
|| isIP(host) || host === 'localhost' || host.endsWith('.localhost')
|| host.endsWith('.local') || !host.includes('.')) {
throw new Error('Source request URL is not public HTTPS');
}
if (options.accept === 'text/html' && host !== sourceHost) {
throw new Error('Source catalog request left its source host');
}
return url;
}
function pinnedLookup(selected) {
return (_hostname, lookupOptions, callback) => lookupOptions.all
? callback(null, [{ address: selected.address, family: selected.family }])
: callback(null, selected.address, selected.family);
}
async function pinnedFetch(value, options = {}) {
const url = new URL(value);
if (url.protocol !== 'https:') throw new Error('Pinned transport requires HTTPS');
const answers = await dns.lookup(url.hostname, { all: true });
if (!answers.length || answers.some(answer => !publicAddress(answer.address))) {
throw Object.assign(new Error('Source hostname resolved to a non-public address'), { nonRetryable: true });
}
const selected = answers[0];
const maxBytes = responseByteLimit(options);
return new Promise((resolve, reject) => {
const request = https.request(url, {
method: 'GET', headers: options.headers, signal: options.signal,
agent: false,
lookup: pinnedLookup(selected),
}, response => {
let bytes = 0;
const chunks = [];
response.on('data', chunk => {
bytes += chunk.length;
if (bytes > maxBytes) {
response.destroy(Object.assign(new Error('Source response exceeds size limit'), { nonRetryable: true }));
return;
}
chunks.push(chunk);
});
response.on('error', reject);
response.on('end', () => {
const body = Buffer.concat(chunks);
const status = response.statusCode || 0;
resolve({
status, ok: status >= 200 && status < 300,
headers: { get: name => response.headers[String(name).toLowerCase()] || null },
text: async () => body.toString('utf8'),
json: async () => JSON.parse(body.toString('utf8')),
arrayBuffer: async () => body,
});
});
});
request.on('error', reject);
request.end();
});
}
function responseByteLimit(options = {}) {
// Chapter lists can be much larger than series/chapter HTML. Only an
// explicit first-party catalog request gets this bounded allowance.
if (options.headers?.accept === 'text/html') return (options.catalog ? 16 : 2) * 1024 * 1024;
return 50 * 1024 * 1024;
}
function createWeebHttp(options = {}) {
return new SlowHttp({ ...options, gateKey: 'weebcentral', transport: pinnedFetch, validateUrl: validateWeebRequest });
}
function createHostHttp(sourceHost, options = {}) {
return new SlowHttp({ ...options, gateKey: sourceHost, transport: pinnedFetch,
validateUrl: (value, requestOptions) => validateHostRequest(value, sourceHost, requestOptions) });
}
module.exports = { publicAddress, validateWeebRequest, validateHostRequest, pinnedLookup, pinnedFetch, responseByteLimit, createWeebHttp, createHostHttp };
+130
View File
@@ -0,0 +1,130 @@
'use strict';
const { specialKey } = require('../../chapter-identity');
function localizedTitle(attributes = {}) {
const titles = attributes.title || {};
return titles.en || titles['ja-ro'] || titles.ja || Object.values(titles).find(Boolean) || 'Untitled';
}
function chapterNumber(value) {
const raw = String(value ?? '').trim();
if (!/^\d+(?:\.\d+)?$/.test(raw)) return null;
const [whole, decimal = ''] = raw.split('.');
const normalizedWhole = whole.replace(/^0+(?=\d)/, '') || '0';
const normalizedDecimal = decimal.replace(/0+$/, '');
return normalizedDecimal ? `${normalizedWhole}.${normalizedDecimal}` : normalizedWhole;
}
function groupFor(resource) {
const relation = (resource.relationships || []).find(item => item.type === 'scanlation_group');
return relation ? { id: relation.id, name: relation.attributes?.name || 'Unknown group' } : null;
}
function releaseRank(resource) {
const group = groupFor(resource)?.name || '';
const credited = /official|license/i.test(group) ? 0 : group ? 1 : 2;
const date = Date.parse(resource.attributes?.publishAt || resource.attributes?.readableAt || resource.attributes?.createdAt || '') || Number.MAX_SAFE_INTEGER;
return [credited, date, resource.id];
}
function compareRank(left, right) {
const a = releaseRank(left), b = releaseRank(right);
return a[0] - b[0] || a[1] - b[1] || String(a[2]).localeCompare(String(b[2]));
}
function selectChapterReleases(resources = [], specialLabel = 'Special') {
const selected = new Map();
for (const resource of resources) {
if (resource.attributes?.externalUrl || Number(resource.attributes?.pages || 0) < 1) continue;
const number = chapterNumber(resource.attributes?.chapter) || specialKey(resource.id);
const prior = selected.get(number);
if (!prior || compareRank(resource, prior) < 0) selected.set(number, resource);
}
return [...selected.entries()].map(([number, resource]) => {
const group = groupFor(resource);
return {
id: resource.id, number, title: resource.attributes?.title || '',
...(!chapterNumber(resource.attributes?.chapter) ? { special: true, label: specialLabel, sourceId: resource.id, sourceNumber: resource.attributes?.chapter ?? null } : {}),
...(resource.attributes?.volume ? { volume: chapterNumber(resource.attributes.volume) || String(resource.attributes.volume) } : {}),
pages: Number(resource.attributes?.pages || 0),
added: resource.attributes?.publishAt || resource.attributes?.readableAt || resource.attributes?.createdAt,
group,
url: `https://mangadex.org/chapter/${resource.id}`,
};
}).sort((a, b) => Number(!!a.special) - Number(!!b.special) || (a.special ? a.number.localeCompare(b.number) : Number(a.number) - Number(b.number) || a.number.localeCompare(b.number)));
}
function availabilitySummary(attributes, releases, chapters, externalReleases = null) {
const numbered = chapters.filter(ref => !ref.special);
const expected = chapterNumber(attributes.lastChapter);
const maximum = numbered.length ? Math.max(...numbered.map(ref => Number(ref.number))) : null;
const expectedCount = expected && /^\d+$/.test(expected) ? Number(expected) : null;
const sparse = expectedCount && numbered.length < expectedCount;
const reason = chapters.length ? null : externalReleases > 0 ? 'external-only'
: Array.isArray(attributes.availableTranslatedLanguages) && !attributes.availableTranslatedLanguages.includes('en') ? 'no-english' : 'no-hosted-pages';
const message = reason === 'external-only' ? `MangaDex lists ${externalReleases} English external-link releases, but no hosted English pages. External links are not downloaded.`
: reason === 'no-english' ? 'This MangaDex edition has no English releases listed.'
: reason ? 'This MangaDex edition has no hosted English releases with pages.' : null;
return { language: 'en', hostedReleases: releases.filter(r => !r.attributes?.externalUrl && Number(r.attributes?.pages) > 0).length,
selectedChapters: chapters.length, numberedChapters: numbered.length, specialReleases: chapters.length - numbered.length,
externalReleases, expectedLastChapter: expected, highestHostedChapter: maximum,
coverage: expected && (maximum === null || maximum < Number(expected) || sparse) ? 'partial' : 'unknown', reason, message };
}
function pageUrls(atHome, chapterUrl, expectedPages) {
const base = new URL(atHome.baseUrl);
if (base.protocol !== 'https:' || !(base.hostname === 'uploads.mangadex.org' || base.hostname.endsWith('.mangadex.network'))) {
throw new Error(`Unexpected MangaDex@Home host: ${base.hostname}`);
}
const hash = String(atHome.chapter?.hash || '');
const files = atHome.chapter?.data;
if (!hash || !Array.isArray(files) || !files.length) throw new Error('MangaDex@Home returned no pages');
if (expectedPages && files.length !== expectedPages) throw new Error(`MangaDex page count changed: expected ${expectedPages}, received ${files.length}`);
return files.map(file => ({
url: new URL(`/data/${encodeURIComponent(hash)}/${encodeURIComponent(file)}`, base).href,
headers: { referer: chapterUrl },
}));
}
function metadata({ data: manga }) {
if (!manga || manga.type !== 'manga') throw new Error('MangaDex title was not found');
const attributes = manga.attributes || {};
const title = localizedTitle(attributes);
const specialLabel = /pre.serialization|pilot/i.test(title) ? 'Pilot'
: (attributes.tags || []).some(tag => tag.attributes?.name?.en === 'Oneshot') ? 'One-shot' : 'Special';
return { title, originalLanguage: attributes.originalLanguage || null, links: attributes.links || {},
specialLabel, availabilityAttributes: {
lastChapter: attributes.lastChapter ?? null,
availableTranslatedLanguages: attributes.availableTranslatedLanguages ?? null,
} };
}
function feed({ data: releases }) {
if (!Array.isArray(releases)) throw new Error('Invalid MangaDex feed');
return { releases: releases.map(resource => ({
id: resource.id,
attributes: {
chapter: resource.attributes?.chapter, volume: resource.attributes?.volume,
title: resource.attributes?.title, pages: resource.attributes?.pages,
publishAt: resource.attributes?.publishAt, readableAt: resource.attributes?.readableAt,
createdAt: resource.attributes?.createdAt, externalUrl: resource.attributes?.externalUrl,
},
relationships: (resource.relationships || []).filter(relation => relation.type === 'scanlation_group'),
})) };
}
function catalog({ data }) {
const { attributes, releases, specialLabel, externalReleases } = data;
if (!attributes || !Array.isArray(releases)) throw new Error('Invalid MangaDex catalog input');
const chapters = selectChapterReleases(releases, specialLabel);
return { chapters, availability: availabilitySummary(attributes, releases, chapters, externalReleases) };
}
function atHome({ data }) {
const { response, chapterUrl, expectedPages } = data;
return { pages: pageUrls(response, chapterUrl, expectedPages) };
}
module.exports = { localizedTitle, chapterNumber, selectChapterReleases, availabilitySummary,
pageUrls, metadata, feed, catalog, atHome };
+32
View File
@@ -0,0 +1,32 @@
'use strict';
// First-party parser; inputs are fetched by the paced source broker, not here.
const decode = value => String(value).replace(/&amp;/g, '&').replace(/&#39;|&#x27;/gi, "'")
.replace(/&quot;/g, '"').replace(/<[^>]+>/g, '').trim();
function catalog({ html, url }) {
const sourceId = new URL(url).pathname.split('/')[2];
const title = decode(html.match(/<h1\b[^>]*>([\s\S]*?)<\/h1>/i)?.[1] || 'Untitled');
const chapters = [];
const re = /<a\b[^>]*href="(\/chapters\/(\d+-\d+)\/[^"?#]+)"[^>]*>([\s\S]*?)<\/a>/gi;
for (const match of html.matchAll(re)) {
if (match[2].split('-')[0] !== sourceId) continue;
const label = decode(match[3]);
const number = label.match(/^Chapter\s+([0-9]+(?:\.[0-9]+)?)(?:\b|$)/i)?.[1];
if (!number) continue;
chapters.push({ id: match[2], number, title: '', url: new URL(match[1], url).href, raw: label });
}
return { title, chapters: [...new Map(chapters.map(ref => [ref.id, ref])).values()],
chapterLinks: [...html.matchAll(/href="\/chapters\/\d+-\d+\//gi)].length };
}
function pages({ html }) {
const images = [];
for (const match of html.matchAll(/<img\b[^>]*class="[^"]*\bjs-page\b[^"]*"[^>]*>/gi)) {
const url = match[0].match(/\bdata-src="([^"]+)"/i)?.[1];
if (url) images.push(url.replace(/&amp;/g, '&'));
}
return { pages: [...new Set(images)] };
}
module.exports = { catalog, pages };
+28
View File
@@ -0,0 +1,28 @@
'use strict';
// This list is shipped with MangaCircus. Neither the admin UI nor a remote
// source may add executable modules at runtime.
const manifests = Object.freeze({
weebcentral: Object.freeze({
id: 'weebcentral', version: 1, label: 'WeebCentral', status: 'in-process',
sourceHosts: Object.freeze(['weebcentral.com']),
implementation: 'first-party',
provenance: 'MangaCircus existing WeebCentral adapter; no Uchiyomi code copied',
operations: Object.freeze(['series', 'catalog', 'chapter', 'pages']),
}),
mangadex: Object.freeze({
id: 'mangadex', version: 1, label: 'MangaDex', status: 'in-process',
sourceHosts: Object.freeze(['api.mangadex.org']),
implementation: 'first-party',
provenance: 'MangaCircus existing MangaDex adapter; no Uchiyomi code copied',
operations: Object.freeze(['metadata', 'feed', 'catalog', 'atHome']),
}),
mangapill: Object.freeze({
id: 'mangapill', version: 1, label: 'MangaPill', status: 'in-process',
sourceHosts: Object.freeze(['mangapill.com']), implementation: 'first-party',
provenance: 'MangaCircus implementation; upstream extension reviewed, no code copied',
operations: Object.freeze(['catalog', 'pages']),
}),
});
module.exports = { manifests };
+26
View File
@@ -0,0 +1,26 @@
'use strict';
// Reviewed first-party modules only. No filesystem scanning or runtime installs.
const { manifests } = require('./manifest');
const { validate } = require('./validation');
const plugins = Object.freeze({
weebcentral: require('./weebcentral'),
mangadex: require('./mangadex'),
mangapill: require('./mangapill'),
});
function parse(plugin, operation, input) {
const manifest = manifests[plugin];
const handler = manifest?.operations.includes(operation) ? plugins[plugin]?.[operation] : null;
if (typeof handler !== 'function') throw new Error('Unknown source plugin or operation');
if (!input || typeof input !== 'object' || typeof input.url !== 'string'
|| (typeof input.html !== 'string' && !(plugin === 'mangadex' && input.data && typeof input.data === 'object'))) {
throw new Error('Invalid parser input');
}
const url = new URL(input.url);
if (!['https:', 'http:'].includes(url.protocol)) throw new Error('Invalid source URL');
if (!manifest.sourceHosts.includes(url.hostname)) throw new Error('Source URL does not belong to plugin');
return validate(plugin, operation, handler({ html: input.html, data: input.data, url: url.href }));
}
module.exports = { parse, manifests };
+100
View File
@@ -0,0 +1,100 @@
'use strict';
const net = require('node:net');
function sourceUrl(value, prefix) {
const url = new URL(value);
if (url.protocol !== 'https:' || url.hostname !== 'weebcentral.com' || url.username || url.password || !url.pathname.startsWith(prefix)) {
throw new Error('Source plugin returned an unexpected WeebCentral URL');
}
return url.href;
}
function imageUrl(value) {
const url = new URL(value);
const hostname = url.hostname.toLowerCase();
if (url.protocol !== 'https:' || url.username || url.password || net.isIP(hostname)
|| hostname === 'localhost' || hostname.endsWith('.localhost') || hostname.endsWith('.local')
|| !hostname.includes('.')) {
throw new Error('Source plugin returned an unsafe image URL');
}
return url.href;
}
function mangaDexUrl(value, kind) {
const url = new URL(value);
if (url.protocol !== 'https:' || url.username || url.password || url.port) throw new Error('Invalid MangaDex plugin URL');
if (kind === 'chapter' && (url.hostname !== 'mangadex.org' || !url.pathname.startsWith('/chapter/'))) {
throw new Error('Unexpected MangaDex chapter URL');
}
if (kind === 'page' && !(url.hostname === 'uploads.mangadex.org' || url.hostname.endsWith('.mangadex.network'))) {
throw new Error('Unexpected MangaDex page host');
}
return url.href;
}
function mangaPillUrl(value, kind) {
const url = new URL(value);
if (url.protocol !== 'https:' || url.username || url.password || url.port) throw new Error('Invalid MangaPill plugin URL');
if (kind === 'chapter' && (url.hostname !== 'mangapill.com' || !/^\/chapters\/\d+-\d+\//.test(url.pathname))) {
throw new Error('Unexpected MangaPill chapter URL');
}
if (kind === 'page' && (url.hostname !== 'cdn.readdetectiveconan.com' || !url.pathname.startsWith('/file/mangap/'))) {
throw new Error('Unexpected MangaPill page host');
}
return url.href;
}
function validate(plugin, operation, output) {
if (!['weebcentral', 'mangadex', 'mangapill'].includes(plugin) || !output || typeof output !== 'object' || Array.isArray(output)) {
throw new Error('Invalid source plugin output');
}
if (plugin === 'mangadex') {
if (operation === 'metadata') {
if (typeof output.title !== 'string' || !output.availabilityAttributes || typeof output.specialLabel !== 'string') throw new Error('Invalid MangaDex metadata');
} else if (operation === 'feed') {
if (!Array.isArray(output.releases) || output.releases.some(ref => typeof ref?.id !== 'string' || !ref.attributes)) throw new Error('Invalid MangaDex feed');
} else if (operation === 'catalog') {
if (!Array.isArray(output.chapters) || !output.availability) throw new Error('Invalid MangaDex catalog');
for (const chapter of output.chapters) mangaDexUrl(chapter.url, 'chapter');
} else if (operation === 'atHome') {
if (!Array.isArray(output.pages) || !output.pages.length) throw new Error('Invalid MangaDex pages');
for (const page of output.pages) mangaDexUrl(page.url, 'page');
} else throw new Error('Unknown source plugin operation');
return output;
}
if (plugin === 'mangapill') {
if (operation === 'catalog') {
if (typeof output.title !== 'string' || !Array.isArray(output.chapters)
|| !Number.isInteger(output.chapterLinks) || output.chapterLinks < 0) throw new Error('Invalid MangaPill catalog');
for (const chapter of output.chapters) {
if (typeof chapter?.id !== 'string' || !/^\d+-\d+$/.test(chapter.id)
|| !/^\d+(?:\.\d+)?$/.test(chapter.number)) throw new Error('Invalid MangaPill chapter');
mangaPillUrl(chapter.url, 'chapter');
}
} else if (operation === 'pages') {
if (!Array.isArray(output.pages)) throw new Error('Invalid MangaPill pages');
for (const page of output.pages) mangaPillUrl(page, 'page');
} else throw new Error('Unknown MangaPill operation');
return output;
}
if (operation === 'series') {
if (typeof output.title !== 'string' || typeof output.listUrl !== 'string' && output.listUrl !== null) throw new Error('Invalid series parse');
if (output.listUrl !== null) sourceUrl(output.listUrl, '/series/');
} else if (operation === 'catalog') {
if (!Array.isArray(output.chapters) || !Number.isInteger(output.chapterLinks) || output.chapterLinks < 0) throw new Error('Invalid catalog parse');
for (const chapter of output.chapters) {
if (!chapter || typeof chapter.id !== 'string' || typeof chapter.number !== 'string') throw new Error('Invalid chapter reference');
sourceUrl(chapter.url, '/chapters/');
}
} else if (operation === 'chapter') {
sourceUrl(output.manifestUrl, '/chapters/');
if (!new URL(output.manifestUrl).pathname.endsWith('/images')) throw new Error('Invalid chapter image manifest');
} else if (operation === 'pages') {
if (!Array.isArray(output.pages)) throw new Error('Invalid page parse');
for (const page of output.pages) imageUrl(page);
} else throw new Error('Unknown source plugin operation');
return output;
}
module.exports = { validate };
+58
View File
@@ -0,0 +1,58 @@
'use strict';
const { volumeKey } = require('../../chapter-identity');
// First-party, pure parsing functions. This module never fetches remote content.
const decode = value => value.replace(/&amp;/g, '&').replace(/&#39;/g, "'").replace(/&quot;/g, '"').replace(/<[^>]+>/g, '').trim();
const absolute = (base, value) => new URL(value, base).href;
function series({ html, url }) {
const title = decode(html.match(/<h1\b[^>]*>([\s\S]*?)<\/h1>/i)?.[1] || 'Untitled');
const listUrl = html.match(/hx-get="([^"]+\/full-chapter-list)"/i)?.[1];
return { title, listUrl: listUrl ? absolute(url, listUrl) : null };
}
function catalog({ html, url }) {
const found = [];
const re = /<a\b[^>]*href="([^"]*\/chapters\/([A-Z0-9]+)[^"]*)"[^>]*>([\s\S]*?)<\/a>/gi;
for (const match of html.matchAll(re)) {
const text = decode(match[3]).replace(/\s+/g, ' ');
const season = Number(text.match(/^S(\d+)\s*-/i)?.[1] || 0) || null;
const regular = text.match(/\b(?:Chapter|Rating|Episode|Night|Log|Act|Scene|Kapitel|File|Battle)\s*[.:]?\s*([0-9]+(?:\.[0-9]+)?)/i)?.[1];
const volume = !regular ? text.match(/^Volume\s+([0-9]+(?:\.[0-9]+)?)(?!\d)/i)?.[1] : null;
const prologue = text.match(/Prologue(?:\s+([0-9]+))?/i)?.[1];
const number = regular || (prologue ? `0.${prologue.padStart(2, '0')}` : /Prologue/i.test(text) ? '0' : null);
if (number) found.push({
id: match[2], number, title: regular ? '' : `Prologue ${prologue || ''}`.trim(),
...(season ? { book: { number: season, title: `Season ${season}` } } : {}),
url: absolute(url, match[1]), raw: text,
});
else if (volume) found.push({
id: match[2], number: volumeKey(match[2]), title: '', special: true,
label: `Volume ${volume}`, releaseKind: 'volume', volume,
book: { number: Number(volume), title: `Volume ${volume}` },
url: absolute(url, match[1]), raw: text,
});
}
const chapters = [...new Map(found.map(chapter => [chapter.id, chapter])).values()];
const chapterLinks = [...html.matchAll(/<a\b[^>]*href="[^"]*\/chapters\/[A-Z0-9]+[^"]*"/gi)].length;
return { chapters, chapterLinks };
}
function chapter({ html, url }) {
return {
manifestUrl: html.match(/['"](https?:\/\/weebcentral\.com\/chapters\/[^'"]+\/images\?is_prev=[^'"]+)['"]/i)?.[1]
|| `${url}/images?is_prev=False`,
added: html.match(/datetime="([^"]+)"/i)?.[1],
};
}
function pages({ html, url }) {
const urls = [];
for (const match of html.matchAll(/<img\b[^>]*(?:src|data-src)="([^"]+)"/gi)) {
const pageUrl = absolute(url, match[1].replace(/&amp;/g, '&'));
if (!pageUrl.startsWith('data:')) urls.push(pageUrl);
}
return { pages: [...new Set(urls)] };
}
module.exports = { series, catalog, chapter, pages };
+899
View File
@@ -0,0 +1,899 @@
'use strict';
const fs = require('fs').promises;
const path = require('path');
const crypto = require('crypto');
const { adapterFor, adapters } = require('./adapters');
const { ArchiveControlError } = require('./lib/control');
const { sleep } = require('./lib/delay');
const { normalizeScope, selectByScope, scopeLabel } = require('./lib/scope');
const { normalizeWorkIdentity, resolveWorkIdentity } = require('./lib/work-identity');
const { recordSource, recordFailure, sourceKey, ledgerRows } = require('./lib/source-ledger');
const { scanLibrary, findManga, shouldDownload, isNewerThanLibrary, summarizeInventory, setAutoUpdate } = require('./lib/library-inventory');
const { inspectLegacyWork, promoteLegacyWork } = require('./lib/legacy-publication');
const { classifyFailure, safeMessage, progressKey } = require('./lib/failures');
const { writeJsonAtomic } = require('./lib/json-file');
const RATE_KEYS = ['minMs', 'maxMs', 'chapterMinMs', 'chapterMaxMs'];
const FAST_RATE_SETTINGS = {
minMs: 1000, maxMs: 4000, chapterMinMs: 5000, chapterMaxMs: 15000,
};
const DEFAULT_SOURCE_RATES = {
weebcentral: { ...FAST_RATE_SETTINGS },
mangadex: { ...FAST_RATE_SETTINGS },
mangapill: { ...FAST_RATE_SETTINGS },
};
const DEFAULT_SETTINGS = {
...FAST_RATE_SETTINGS,
sourceRates: Object.fromEntries(Object.entries(DEFAULT_SOURCE_RATES).map(([source, rate]) => [source, { ...rate }])),
autoUpdateEnabled: true, autoUpdateTime: '03:00',
};
async function exists(file) {
try { await fs.access(file); return true; } catch { return false; }
}
function validateRateSettings(input, prior, prefix = '') {
const next = { ...prior };
for (const key of RATE_KEYS) {
if (input[key] === undefined) continue;
const value = Number(input[key]);
if (!Number.isFinite(value) || value < 500 || value > 3600000) throw new Error(`${prefix}${key} must be between 500 and 3600000 ms`);
next[key] = Math.round(value);
}
if (next.maxMs < next.minMs) throw new Error(`${prefix}maxMs must be at least ${prefix}minMs`);
if (next.chapterMaxMs < next.chapterMinMs) throw new Error(`${prefix}chapterMaxMs must be at least ${prefix}chapterMinMs`);
return next;
}
function validateSettings(input = {}, prior = DEFAULT_SETTINGS) {
const priorBase = { ...FAST_RATE_SETTINGS, ...prior };
const next = validateRateSettings(input, priorBase);
if (input.sourceRates !== undefined && (!input.sourceRates || typeof input.sourceRates !== 'object' || Array.isArray(input.sourceRates))) {
throw new Error('sourceRates must be an object');
}
const requestedRates = input.sourceRates || {};
const sourceIds = new Set([
...Object.keys(DEFAULT_SOURCE_RATES),
...Object.keys(prior.sourceRates || {}),
...Object.keys(requestedRates),
]);
next.sourceRates = {};
for (const source of sourceIds) {
if (!/^[a-z0-9][a-z0-9-]*$/.test(source)) throw new Error(`Invalid source id: ${source}`);
const patch = requestedRates[source];
if (patch !== undefined && (!patch || typeof patch !== 'object' || Array.isArray(patch))) {
throw new Error(`sourceRates.${source} must be an object`);
}
const base = {
...FAST_RATE_SETTINGS,
...(DEFAULT_SOURCE_RATES[source] || {}),
...((prior.sourceRates || {})[source] || {}),
};
next.sourceRates[source] = validateRateSettings(patch || {}, base, `sourceRates.${source}.`);
}
if (input.autoUpdateEnabled !== undefined) next.autoUpdateEnabled = !!input.autoUpdateEnabled;
if (input.autoUpdateTime !== undefined) {
const value = String(input.autoUpdateTime).trim();
if (!/^(?:[01]\d|2[0-3]):[0-5]\d$/.test(value)) throw new Error('autoUpdateTime must be HH:MM');
next.autoUpdateTime = value;
}
return next;
}
function settingsForSource(settings, source) {
return { ...settings, ...((settings.sourceRates || {})[source] || {}) };
}
function existingPolicy(value) {
const policy = String(value || 'skip-any');
if (!['skip-any', 'add-source'].includes(policy)) throw new Error('existingPolicy must be skip-any or add-source');
return policy;
}
function nextSchedule(time, from = new Date()) {
const [hours, minutes] = String(time).split(':').map(Number);
const next = new Date(from);
next.setHours(hours, minutes, 0, 0);
if (next <= from) next.setDate(next.getDate() + 1);
return next;
}
class ScraperService {
constructor(options = {}) {
this.libraryRoot = path.resolve(options.libraryRoot || 'library');
this.dataRoot = path.resolve(options.dataRoot || path.join('data', 'scraper'));
this.stateFile = path.join(this.dataRoot, 'queue.json');
this.logFile = path.join(this.dataRoot, 'events.log');
this.lockFile = path.join(this.dataRoot, 'daemon.lock');
this.legacyStateFile = path.join(this.libraryRoot, '.mcscraper-queue', 'queue.json');
this.state = null;
this.stopping = false;
this.loopPromise = null;
this.active = new Map();
this.writeChain = Promise.resolve();
this.discoveryTimer = null;
this.inventoryCache = null;
this.inventoryCachedAt = 0;
this.inventoryGeneration = 0;
this.inventoryCacheGeneration = -1;
this.inventoryRefresh = null;
this.inventoryAttemptedAt = 0;
this.inventoryError = null;
this.scanInventory = options.scanLibrary || scanLibrary;
this.resolveAdapter = options.adapterFor || adapterFor;
this.adapterList = options.adapters || adapters;
this.retryDelayMs = options.retryDelayMs ?? 300000;
this.runtimeHeartbeat = Date.now();
}
async init() {
await fs.mkdir(this.dataRoot, { recursive: true });
await fs.mkdir(this.libraryRoot, { recursive: true });
await this.acquireLock();
this.state = await this.loadState();
this.state.incidents = this.state.incidents || [];
this.state.circuits = this.state.circuits || {};
this.state.updateChecks = this.state.updateChecks || {};
this.state.sourceLedger = this.state.sourceLedger || {};
// Older releases retained successful no-change checks in the work queue.
// Keep their latest result before retiring those finished rows.
for (const item of this.state.queue) {
if (item.kind !== 'discovery' || item.status !== 'complete' || !item.checked || !item.libraryMangaId) continue;
const key = JSON.stringify([item.libraryMangaId, item.source]);
const prior = this.state.updateChecks[key];
if (!prior?.lastCheckedAt || Date.parse(item.checked) > Date.parse(prior.lastCheckedAt)) {
this.state.updateChecks[key] = { ...prior, mangaId: item.libraryMangaId, source: item.source,
lastCheckedAt: item.checked, lastResult: 'no-updates' };
}
}
this.state.queue = this.state.queue.filter(item => item.kind !== 'discovery' || item.status !== 'complete' || !item.libraryMangaId);
this.state.queue.forEach((item, index) => { item.order = index; });
for (const item of this.state.queue) {
delete item.stopReason;
if (item.status === 'downloading' || item.status === 'paused' || item.status === 'discovering') item.status = 'queued';
const found = this.resolveAdapter(item.url);
item.source = found?.adapter.id || item.source || 'unknown';
item.seriesId = item.seriesId || found?.info.id || item.id;
item.kind = ['discovery', 'inspection'].includes(item.kind) ? item.kind : 'download';
item.scope = normalizeScope(item.scope || {});
item.scopeLabel = item.scopeLabel || scopeLabel(item.scope);
item.existingPolicy = existingPolicy(item.existingPolicy);
if (item.status === 'error') {
item.failure = item.failure || classifyFailure(new Error(item.error));
if (item.failure.kind === 'unavailable') item.status = 'unavailable';
}
if (item.source === 'mangadex' && item.status === 'unavailable' && !item.availability
&& item.error === 'No downloadable English MangaDex chapters found') {
item.error = 'The previous adapter selected no numbered hosted English chapters. Inspect this source again to verify its edition and availability, including pilots and specials.';
if (item.failure) item.failure = { ...item.failure, message: item.error };
}
// Existing queue rows are leads. Only a recorded catalog observation may
// promote one to an adapter-compatible source.
if (!this.state.sourceLedger[sourceKey(item)]) recordSource(this.state.sourceLedger, item);
}
this.state.version = 4;
this.state.currentId = null;
this.state.currentIds = {};
this.state.sourcePaused = this.state.sourcePaused || {};
// A legacy title-unavailable pause can be released; access and unknown failures remain paused.
for (const source of Object.keys(this.state.sourcePaused)) {
const failed = this.state.queue.filter(item => item.source === source && ['error', 'unavailable', 'quarantined'].includes(item.status));
if (failed.length && failed.every(item => item.failure?.kind === 'unavailable')) delete this.state.sourcePaused[source];
}
this.state.status = this.state.paused ? 'paused' : 'idle';
this.state.pid = process.pid;
await this.event('info', 'Scraper daemon started', { pid: process.pid });
this.scheduleDiscovery();
this.loopPromise = this.workLoop();
return this;
}
async acquireLock() {
try {
const handle = await fs.open(this.lockFile, 'wx');
await handle.writeFile(String(process.pid));
return await handle.close();
} catch (error) {
if (error.code !== 'EEXIST') throw error;
}
const oldPid = Number((await fs.readFile(this.lockFile, 'utf8').catch(() => '')).trim());
try {
if (oldPid) process.kill(oldPid, 0);
throw new Error(`Scraper daemon is already running (PID ${oldPid || 'unknown'})`);
} catch (error) {
if (error.code !== 'ESRCH') throw error;
await fs.rm(this.lockFile, { force: true });
const handle = await fs.open(this.lockFile, 'wx');
await handle.writeFile(String(process.pid));
await handle.close();
}
}
async loadState() {
if (await exists(this.stateFile)) {
const saved = JSON.parse(await fs.readFile(this.stateFile, 'utf8'));
return {
version: 3, paused: false, status: 'idle', queue: [], history: [], publishedRevision: 0,
currentIds: {}, sourcePaused: {},
settings: DEFAULT_SETTINGS, ...saved,
settings: validateSettings(saved.settings || {}, DEFAULT_SETTINGS),
};
}
const state = {
version: 3, paused: false, status: 'idle', currentId: null, currentIds: {}, sourcePaused: {},
settings: { ...DEFAULT_SETTINGS }, queue: [], history: [], publishedRevision: 0,
created: new Date().toISOString(), lastError: null,
};
if (await exists(this.legacyStateFile)) {
const legacy = JSON.parse(await fs.readFile(this.legacyStateFile, 'utf8'));
const completed = new Set(legacy.completed || []);
state.queue = (legacy.entries || []).map((item, index) => ({
id: item.id, seriesId: item.id, source: 'weebcentral', title: item.title, url: item.url,
metadataSearch: item.metadataSearch || item.title,
kind: 'download', scope: { mode: 'missing' }, existingPolicy: 'skip-any',
status: completed.has(item.id) ? 'complete' : 'queued', order: index,
added: legacy.created || new Date().toISOString(),
}));
state.settings = validateSettings(legacy.settings || {}, DEFAULT_SETTINGS);
state.migratedFrom = this.legacyStateFile;
}
return state;
}
async persist() {
const write = async () => {
this.state.updated = new Date().toISOString();
await writeJsonAtomic(this.stateFile, this.state);
};
const pending = this.writeChain.then(write, write);
this.writeChain = pending.catch(() => {});
return pending;
}
async event(level, message, details = {}) {
const entry = { time: new Date().toISOString(), level, message: safeMessage(message), ...details };
if (entry.url) entry.url = safeMessage(entry.url);
await fs.appendFile(this.logFile, JSON.stringify(entry) + '\n');
console.log(JSON.stringify({ service: 'scraper', ...entry }));
this.state.lastEvent = entry;
if (level === 'error') this.state.lastError = entry;
await this.persist();
}
async recentEvents(limit = 30) {
const handle = await fs.open(this.logFile, 'r').catch(error => {
if (error.code === 'ENOENT') return null;
throw error;
});
if (!handle) return [];
let text;
try {
const { size } = await handle.stat();
const length = Math.min(size, 512 * 1024), start = size - length;
const buffer = Buffer.alloc(length);
const { bytesRead } = await handle.read(buffer, 0, length, start);
text = buffer.subarray(0, bytesRead).toString('utf8');
if (start > 0) {
const boundary = text.indexOf('\n');
text = boundary < 0 ? '' : text.slice(boundary + 1);
}
} finally { await handle.close(); }
return text.trim().split(/\r?\n/).filter(Boolean).slice(-Math.max(1, Math.min(200, limit)))
.map(line => { try { return JSON.parse(line); } catch { return { level: 'warning', message: line }; } });
}
invalidateInventory() {
// Keep the last summary visible while rebuilding. Download/discovery decisions
// still await an authoritative scan through libraryInventory(), never this stale view.
this.inventoryGeneration++;
}
refreshInventory() {
if (this.inventoryRefresh) return this.inventoryRefresh;
const generation = this.inventoryGeneration;
this.inventoryAttemptedAt = Date.now();
this.inventoryRefresh = Promise.resolve().then(() => this.scanInventory(this.libraryRoot))
.then(async inventory => {
// A chapter or metadata edit during the scan invalidates its result.
if (generation === this.inventoryGeneration) {
this.inventoryCache = inventory;
this.inventoryCachedAt = Date.now();
this.inventoryCacheGeneration = generation;
}
// Backfill source leads from existing library metadata without probing
// hosts or asserting that their current catalog is complete.
let added = 0;
if (generation === this.inventoryGeneration) {
for (const manga of inventory.mangas || []) {
for (const source of manga.sources || []) {
const found = this.resolveAdapter(source.url);
if (!found) continue;
const lead = { source: found.adapter.id, seriesId: found.info.id, url: source.url,
title: manga.title, libraryMangaId: manga.id,
workIdentity: resolveWorkIdentity(manga.meta, manga.id) };
const existing = this.state.sourceLedger[sourceKey(lead)];
if (existing && !existing.workKey?.startsWith('source:')) continue;
recordSource(this.state.sourceLedger, lead);
added++;
}
}
}
if (added) await this.persist();
this.inventoryError = null;
return inventory;
}).catch(error => {
this.inventoryError = safeMessage(error.message);
throw error;
}).finally(() => { this.inventoryRefresh = null; });
return this.inventoryRefresh;
}
async libraryInventory(force = false) {
if (!force && this.inventoryCache && this.inventoryCacheGeneration === this.inventoryGeneration
&& Date.now() - this.inventoryCachedAt < 60000) return this.inventoryCache;
do {
await this.refreshInventory();
} while (this.inventoryCacheGeneration !== this.inventoryGeneration);
return this.inventoryCache;
}
scheduleDiscovery() {
if (this.discoveryTimer) clearTimeout(this.discoveryTimer);
this.discoveryTimer = null;
if (!this.state?.settings?.autoUpdateEnabled || this.stopping) {
if (this.state) this.state.nextDiscoveryAt = null;
return;
}
const next = nextSchedule(this.state.settings.autoUpdateTime);
this.state.nextDiscoveryAt = next.toISOString();
const delay = Math.max(1000, next.getTime() - Date.now());
this.discoveryTimer = setTimeout(() => {
this.discoveryTimer = null;
this.enqueueDiscoveryChecks('schedule').catch(error => this.event('error', `Nightly discovery failed: ${error.message}`))
.finally(() => { if (!this.stopping) this.scheduleDiscovery(); });
}, delay);
this.discoveryTimer.unref();
}
pendingSeries(source, seriesId) {
return this.state.queue.some(item => item.source === source && String(item.seriesId) === String(seriesId)
&& ['queued', 'downloading', 'discovering', 'ready', 'error', 'retry_wait', 'quarantined', 'unavailable'].includes(item.status));
}
recordUpdateCheck(item, patch) {
if (!item.libraryMangaId || !item.source) return;
const key = JSON.stringify([item.libraryMangaId, item.source]);
this.state.updateChecks[key] = { ...(this.state.updateChecks[key] || {}),
mangaId: item.libraryMangaId, source: item.source, ...patch };
}
async enqueueDiscoveryChecks(reason = 'manual') {
const inventory = await this.libraryInventory(true);
let queued = 0, skipped = 0;
for (const manga of inventory.mangas) {
if (!manga.autoUpdate) continue;
const supported = manga.sources.map(source => ({ source, found: this.resolveAdapter(source.url) })).filter(item => item.found);
if (!supported.length) { skipped++; continue; }
const preferred = manga.meta?.['x-scraper']?.preferredSource;
const selected = supported.find(item => item.found.adapter.id === preferred) || supported[0];
const source = selected.found.adapter.id, seriesId = selected.found.info.id;
if (this.pendingSeries(source, seriesId)) { skipped++; continue; }
const item = {
id: crypto.randomUUID(), seriesId, source, kind: 'discovery',
title: manga.title, url: selected.found.info.url, metadataSearch: manga.title,
libraryMangaId: manga.id, scope: { mode: 'missing' }, existingPolicy: 'skip-any',
status: 'queued', order: this.state.queue.length, added: new Date().toISOString(), reason,
};
this.state.queue.push(item); queued++;
}
this.state.lastDiscovery = { time: new Date().toISOString(), reason, queued, skipped };
await this.event('info', `Queued ${queued} update check${queued === 1 ? '' : 's'}`, { reason, queued, skipped });
return { queued, skipped };
}
async setMangaAutoUpdate(id, enabled) {
const result = await setAutoUpdate(this.libraryRoot, id, enabled);
this.invalidateInventory();
await this.event('info', `${enabled ? 'Enabled' : 'Disabled'} auto update for ${id}`, { mangaId: id, enabled: !!enabled });
return result;
}
async publishLegacyReady() {
if (!this.state.paused || this.active.size || Object.keys(this.state.currentIds || {}).length) {
throw new Error('Pause the queue and wait for active workers to stop before publishing legacy work');
}
const candidates = [];
const seen = new Set();
for (const item of this.state.queue) {
const seriesId = String(item.seriesId || item.id || '');
if (!seriesId || seen.has(seriesId) || item.status === 'complete') continue;
seen.add(seriesId);
const inspection = await inspectLegacyWork({ libraryRoot: this.libraryRoot, seriesId });
if (inspection) candidates.push({ item, inspection });
}
const published = [];
for (const candidate of candidates) {
const result = await promoteLegacyWork(candidate.inspection);
candidate.item.libraryMangaId = result.mangaId;
candidate.item.legacyPromotedAt = new Date().toISOString();
published.push(result);
this.state.publishedRevision = Number(this.state.publishedRevision || 0) + 1;
this.invalidateInventory();
await this.event('info', `Published ${result.chaptersPublished} completed chapters for ${candidate.item.title}`, {
id: candidate.item.id, seriesId: candidate.item.seriesId, title: candidate.item.title,
mangaId: result.mangaId, chaptersPublished: result.chaptersPublished,
});
}
return { published };
}
async runDiscovery(item, found, source) {
if (this.stopping || this.state.paused) throw new ArchiveControlError(this.stopping ? 'stopped' : 'paused');
item.status = 'discovering';
const catalog = await found.adapter.discover(item, {
settings: settingsForSource(this.state.settings, source),
shouldStop: () => this.stopping ? 'stopped' : this.state.paused ? 'paused' : item.stopReason || false,
onEvent: event => this.event(event.level, event.message, { id: item.id, title: item.title, source, ...(event.status ? { status: event.status } : {}) }),
});
if (this.stopping || this.state.paused || item.stopReason) throw new ArchiveControlError(item.stopReason || (this.stopping ? 'stopped' : 'paused'));
await this.recordCatalog(item, catalog);
if (item.kind === 'inspection') {
if (!catalog.chapters?.length) throw Object.assign(new Error(catalog.availability?.message || 'This source edition has no downloadable chapters'), {
failureKind: catalog.availability?.reason === 'unrecognized-labels' ? 'adapter' : 'unavailable',
});
if (item.inspectionFromComplete) {
const inventory = await this.libraryInventory(true);
const manga = findManga(inventory, {
source, seriesId: item.seriesId, sourceUrl: item.url,
anilist: catalog.links?.al, libraryMangaId: item.libraryMangaId,
});
const missing = selectByScope(catalog.chapters, item.scope)
.filter(ref => shouldDownload(manga, ref, source, item.existingPolicy));
if (!missing.length) {
item.kind = 'download'; item.status = 'complete'; item.checked = new Date().toISOString();
delete item.inspectionFromComplete;
for (const incident of this.state.incidents) if (incident.jobId === item.id && incident.state === 'open') {
incident.state = 'resolved'; incident.resolution = 'inspected-complete';
}
await this.event('info', `Source inspected: ${item.sourceTitle}; completed download is already in the library`, { id: item.id, source });
return;
}
}
delete item.inspectionFromComplete;
item.status = 'ready'; item.checked = new Date().toISOString();
for (const incident of this.state.incidents) if (incident.jobId === item.id && incident.state === 'open') {
incident.state = 'resolved'; incident.resolution = 'inspected-ready';
}
await this.event('info', `Source inspected: ${item.sourceTitle}; awaiting download confirmation`, { id: item.id, source });
return;
}
const inventory = await this.libraryInventory(true);
if (this.stopping || this.state.paused) throw new ArchiveControlError(this.stopping ? 'stopped' : 'paused');
const manga = findManga(inventory, {
source, seriesId: item.seriesId, sourceUrl: item.url,
anilist: catalog.links?.al, libraryMangaId: item.libraryMangaId,
});
if (item.libraryMangaId && !manga) throw new Error(`Library manga target not found: ${item.libraryMangaId}`);
const candidates = selectByScope(catalog.chapters || [], item.scope)
.filter(ref => shouldDownload(manga, ref, source, item.existingPolicy))
.filter(ref => isNewerThanLibrary(manga, ref));
item.checked = new Date().toISOString();
item.catalogTotal = catalog.chapters?.length || 0;
this.recordUpdateCheck(item, { lastCheckedAt: item.checked,
...(candidates.length ? { lastNewContentAt: item.checked, lastResult: 'new-content' } : { lastResult: 'no-updates' }) });
if (!candidates.length) {
item.status = 'complete'; item.completed = item.checked; item.discovered = 0;
this.state.history.push({ id: item.id, title: item.title, source, status: 'checked', time: item.completed, discovered: 0 });
this.state.queue = this.state.queue.filter(entry => entry.id !== item.id);
this.state.queue.forEach((entry, index) => { entry.order = index; });
await this.event('info', `No updates found for ${item.title}`, { id: item.id, title: item.title, source });
return;
}
item.kind = 'download'; item.status = 'queued'; item.discovered = candidates.length;
item.scope = normalizeScope({ mode: 'ids', ids: candidates.map(ref => ref.id) });
item.scopeLabel = scopeLabel(item.scope);
await this.event('info', `Found ${candidates.length} new chapter${candidates.length === 1 ? '' : 's'} for ${item.title}`, {
id: item.id, title: item.title, source, discovered: candidates.length,
});
}
async recordCatalog(item, catalog) {
const title = String(catalog.title || item.title).slice(0, 200);
const changedEdition = item.confirmedSourceTitle && item.confirmedSourceTitle !== title;
item.sourceTitle = title;
item.catalogTotal = catalog.chapters?.length || 0;
item.availability = catalog.availability || { selectedChapters: item.catalogTotal, coverage: 'unknown' };
recordSource(this.state.sourceLedger ||= {}, item, {
catalogTotal: item.catalogTotal, availability: item.availability,
sourceTitle: title, observedAt: new Date().toISOString(),
});
await this.persist();
if (changedEdition) throw Object.assign(new Error('The source edition title changed since inspection. Inspect and confirm it again before downloading.'), { failureKind: 'unavailable' });
}
async jobState(item) {
if (!item) return null;
if (item.kind === 'inspection') return { status: 'discovering', updated: item.started, completed: 0, total: null, current: null };
const sourceId = item.seriesId || item.id;
const workName = item.source === 'mangapill'
? `.mcscraper-work-mangapill-${sourceId}`
: `.mcscraper-work-${sourceId}`;
const file = path.join(this.libraryRoot, workName, 'job.json');
try {
const job = JSON.parse(await fs.readFile(file, 'utf8'));
return {
status: job.status || 'unknown', completed: job.publicationMode === 'incremental' ? Number(job.jobCompleted || 0) : job.completed?.length || 0, total: job.total || null,
percent: job.total ? Math.round((job.publicationMode === 'incremental' ? Number(job.jobCompleted || 0) : job.completed?.length || 0) * 1000 / job.total) / 10 : null,
lastChapter: job.lastChapter || null, current: job.current || null,
lastProblem: job.lastProblem || null, lastError: job.lastError || null,
problemCount: job.problemCount || 0, rate: job.rate || null, updated: job.updated || null,
};
} catch { return null; }
}
async publicState() {
const activeItems = this.state.queue.filter(item => Object.values(this.state.currentIds || {}).includes(item.id));
const active = await Promise.all(activeItems.map(async current => ({
source: this.resolveAdapter(current.url)?.adapter.id || current.source,
current, job: await this.jobState(current),
})));
const current = active[0]?.current || null;
const job = active[0]?.job || null;
// Status and watchdog probes must not wait for thousands of bind-mount reads.
// Share one background scan, serve the previous summary, and surface its age/error.
if (!this.inventoryRefresh && (this.inventoryCacheGeneration !== this.inventoryGeneration || Date.now() - this.inventoryCachedAt >= 60000)
&& Date.now() - this.inventoryAttemptedAt >= 10000) {
this.refreshInventory().catch(() => {});
}
const inventory = this.inventoryCache;
return {
version: this.state.version, status: this.state.status, paused: this.state.paused,
pid: this.state.pid, settings: this.state.settings, publishedRevision: this.state.publishedRevision || 0,
current, job, active, sourcePaused: this.state.sourcePaused || {}, queue: this.state.queue,
history: (this.state.history || []).slice(-20).reverse(), lastError: this.state.lastError || null,
lastEvent: this.state.lastEvent || null, updated: this.state.updated || null,
nextDiscoveryAt: this.state.nextDiscoveryAt || null, lastDiscovery: this.state.lastDiscovery || null,
library: inventory ? summarizeInventory(inventory, value => this.resolveAdapter(value)) : [],
updateChecks: Object.values(this.state.updateChecks || {}),
libraryStatus: {
ready: !!inventory, refreshing: !!this.inventoryRefresh,
stale: this.inventoryCacheGeneration !== this.inventoryGeneration || Date.now() - this.inventoryCachedAt >= 60000,
updated: this.inventoryCachedAt ? new Date(this.inventoryCachedAt).toISOString() : null,
error: this.inventoryError,
},
adapters: this.adapterList.map(item => ({ id: item.id, label: item.label })),
sourceLedger: ledgerRows(this.state.sourceLedger),
events: await this.recentEvents(25),
incidents: (this.state.incidents || []).slice(-50).reverse(), circuits: this.state.circuits || {},
runtimeHeartbeat: new Date(this.runtimeHeartbeat).toISOString(),
};
}
async enqueue(input) {
const found = this.resolveAdapter(input.url);
if (!found) throw new Error('No installed scraper adapter accepts this URL');
const workIdentity = normalizeWorkIdentity(input.workIdentity);
if (input.workIdentity != null && !workIdentity) throw new Error('Invalid manga work identity');
const scope = normalizeScope(input.scope || {});
const policy = existingPolicy(input.existingPolicy);
const signature = JSON.stringify([found.adapter.id, found.info.id, scope, policy, input.libraryMangaId || null]);
if (this.state.queue.some(item => item.status !== 'complete'
&& JSON.stringify([item.source, item.seriesId, item.scope, item.existingPolicy, item.libraryMangaId || null]) === signature)) {
throw new Error('That exact manga scope is already queued');
}
const item = {
...found.info, id: crypto.randomUUID(), seriesId: found.info.id, kind: input.inspect === true ? 'inspection' : 'download',
title: String(input.title || found.info.title).trim().slice(0, 200),
metadataSearch: String(input.metadataSearch || input.title || found.info.title).trim().slice(0, 200),
metadataSearchExplicit: !!String(input.metadataSearch || '').trim(),
scope, scopeLabel: scopeLabel(scope), existingPolicy: policy,
...(workIdentity ? { workIdentity } : {}),
...(input.libraryMangaId ? { libraryMangaId: String(input.libraryMangaId) } : {}),
status: 'queued', order: this.state.queue.length, added: new Date().toISOString(),
};
this.state.queue.push(item);
recordSource(this.state.sourceLedger ||= {}, item);
await this.event('info', `Queued ${item.title}: ${item.scopeLabel}`, { id: item.id, seriesId: item.seriesId, title: item.title });
return item;
}
async remove(id) {
const item = this.state.queue.find(entry => entry.id === id);
if (!item) throw new Error('Queue item not found');
if (Object.values(this.state.currentIds || {}).includes(item.id)) throw new Error('Pause the active download before removing it');
this.state.queue = this.state.queue.filter(entry => entry.id !== id);
this.state.queue.forEach((entry, index) => { entry.order = index; });
await this.event('info', `Removed ${item.title} from queue`, { id: item.id, title: item.title });
}
async clearCompleted(source = '') {
if (source && !this.adapterList.some(adapter => adapter.id === source)) throw new Error('Unknown source');
const before = this.state.queue.length;
const activeIds = new Set(Object.values(this.state.currentIds || {}));
this.state.queue = this.state.queue.filter(item => !(item.status === 'complete' && item.kind === 'download'
&& !activeIds.has(item.id) && (!source || item.source === source)));
this.state.queue.forEach((item, index) => { item.order = index; });
const cleared = before - this.state.queue.length;
if (cleared) await this.event('info', `Cleared ${cleared} completed download job${cleared === 1 ? '' : 's'} from queue`, { source: source || 'all', cleared });
return { cleared };
}
async reorder(ids) {
if (!Array.isArray(ids) || ids.length !== this.state.queue.length || new Set(ids).size !== ids.length) throw new Error('Order must contain every queue id exactly once');
const byId = new Map(this.state.queue.map(item => [item.id, item]));
if (ids.some(id => !byId.has(id))) throw new Error('Order contains an unknown queue id');
for (const current of Object.values(this.state.currentIds || {})) {
if (ids.indexOf(current) !== this.state.queue.findIndex(item => item.id === current)) throw new Error('Active items cannot be moved while downloading');
}
this.state.queue = ids.map((id, index) => ({ ...byId.get(id), order: index }));
await this.event('info', 'Queue reordered');
}
async updateSettings(patch) {
this.state.settings = validateSettings(patch, this.state.settings);
this.scheduleDiscovery();
await this.event('info', 'Scraper settings updated', { settings: this.state.settings, nextDiscoveryAt: this.state.nextDiscoveryAt });
return this.state.settings;
}
async control(action, id) {
if (action === 'pause-source' || action === 'resume-source') {
const source = String(id || '');
if (!this.adapterList.some(adapter => adapter.id === source)) throw new Error('Unknown source');
if (action === 'resume-source' && this.state.queue.some(item => item.source === source
&& item.status === 'error' && item.failure?.kind === 'access')) {
throw new Error('Review the source access block before resuming');
}
if (action === 'pause-source') {
this.state.sourcePaused[source] = true;
const activeId = this.state.currentIds[source];
const activeItem = this.state.queue.find(item => item.id === activeId);
if (activeItem) activeItem.stopReason = 'paused';
} else delete this.state.sourcePaused[source];
this.recomputeStatus();
await this.event('info', `${source} source ${action === 'pause-source' ? 'paused' : 'resumed'} by operator`, { source });
return;
}
if (action === 'inspect' || action === 'confirm') {
const item = this.state.queue.find(entry => entry.id === id);
if (!item || Object.values(this.state.currentIds || {}).includes(item.id)) throw new Error('Job not found or already active');
// Confirmation only changes the persisted queue. The work loop still
// refuses to start jobs for a paused source, so an admin can approve an
// already inspected edition without accidentally retrying blocked access.
if (item.failure?.kind === 'access' || (action === 'inspect' && this.state.sourcePaused[item.source])) {
throw new Error('Review the source access block before inspecting');
}
if (action === 'confirm') {
if (item.status !== 'ready' || item.kind !== 'inspection') throw new Error('Inspect the source first');
item.confirmedSourceTitle = item.sourceTitle;
// Prefer the inspected canonical title when metadata lookup was derived
// from the URL/title fallback. Keep an explicitly supplied search term.
const canonicalTitle = String(item.sourceTitle || '').trim();
const search = String(item.metadataSearch || '').trim();
const seriesId = String(item.seriesId || '').trim();
const fallbackSearch = !item.metadataSearchExplicit
&& (!search || search === seriesId || search === String(item.title || '').trim());
if (canonicalTitle && fallbackSearch) {
item.metadataSearch = canonicalTitle;
item.metadataSearchExplicit = true;
}
if (canonicalTitle && String(item.title || '') === String(item.seriesId || '')) item.title = canonicalTitle;
item.kind = 'download';
} else {
if (!['unavailable', 'quarantined', 'complete', 'ready'].includes(item.status)) throw new Error('Only inactive finished, unavailable or quarantined jobs can be inspected again');
item.inspectionFromComplete = !!item.completed && Number(item.publishedCount || 0) > 0;
item.kind = 'inspection'; delete item.confirmedSourceTitle;
}
item.status = 'queued'; delete item.error; delete item.failure; delete item.nextAttemptAt;
await this.event('info', action === 'confirm' ? `Download confirmed: ${item.sourceTitle}` : `Source inspection queued: ${item.title}`, { id: item.id, source: item.source });
return;
}
if (action === 'pause') {
this.state.paused = true;
this.state.status = 'pausing';
await this.event('info', 'Pause requested');
return;
}
if (action === 'resume') {
this.state.paused = false;
this.recomputeStatus();
await this.event('info', 'Queue resumed');
return;
}
if (action === 'retry') {
const item = this.state.queue.find(entry => entry.id === id) || this.state.queue.find(entry => entry.status === 'error');
if (!item) throw new Error('No failed queue item to retry');
if (Object.values(this.state.currentIds || {}).includes(item.id)) throw new Error('That item is already active');
const source = this.resolveAdapter(item.url)?.adapter.id || item.source;
const otherAccessError = this.state.queue.some(entry => entry.id !== item.id && entry.source === source && entry.status === 'error' && entry.failure?.kind === 'access');
if (otherAccessError) throw new Error('Review the access-blocked job for this source first');
// Older queued records may still carry the URL slug/ID as an automatic
// metadata query. A retry must repair that fallback too (not just confirm).
const canonicalTitle = String(item.sourceTitle || '').trim();
const search = String(item.metadataSearch || '').trim();
const seriesId = String(item.seriesId || '').trim();
if (canonicalTitle && !item.metadataSearchExplicit
&& (!search || search === seriesId || search === String(item.title || '').trim())) {
item.metadataSearch = canonicalTitle;
if (String(item.title || '') === seriesId) item.title = canonicalTitle;
}
item.status = 'queued'; delete item.error; delete item.failure; delete item.nextAttemptAt;
item.recoveryAttempts = 0;
item.stallRecoveries = 0;
delete this.state.sourcePaused[source]; delete this.state.circuits[source];
for (const incident of this.state.incidents) if (incident.jobId === item.id && incident.state === 'open') { incident.state = 'resolved'; incident.resolution = 'operator-retry'; }
this.state.lastError = null; this.state.paused = false; this.recomputeStatus();
await this.event('info', `Retrying ${item.title}`, { id: item.id, title: item.title });
return;
}
if (action === 'skip') {
const item = this.state.queue.find(entry => entry.id === id);
if (!item || this.state.currentIds[item.source] === item.id) throw new Error('Job not found or it is active');
item.status = 'skipped';
if (item.failure?.scope === 'job') delete this.state.sourcePaused[item.source];
await this.event('info', `Skipped ${item.title}`, { id: item.id, source: item.source });
return;
}
throw new Error('Unknown control action');
}
async recover(input) {
const item = this.state.queue.find(entry => entry.id === input.id);
if (!item) throw new Error('Job not found');
// Compare observations so a stale watcher cannot disturb a progressing worker.
if (Object.values(this.state.currentIds).includes(item.id)) {
const job = await this.jobState(item);
if (!job) return { changed: false };
if (progressKey(job) !== input.progressKey || this.state.paused || item.stopReason) return { changed: false };
if (Date.now() - Date.parse(job?.updated || item.started) < 900000) return { changed: false };
if ((item.stallRecoveries || 0) >= 1) {
item.stopReason = 'quarantined';
} else { item.stallRecoveries = (item.stallRecoveries || 0) + 1; item.stopReason = 'recovering'; }
await this.event('warning', `Sentinel requested recovery at the next checkpoint for ${item.title}`, { id: item.id, source: item.source });
return { changed: true };
}
return { changed: false };
}
recomputeStatus() {
const activeIds = Object.values(this.state.currentIds || {});
this.state.currentId = activeIds[0] || null;
if (this.stopping) this.state.status = 'stopping';
else if (this.state.paused) this.state.status = activeIds.length ? 'pausing' : 'paused';
else if (activeIds.length) this.state.status = 'downloading';
else if (this.state.queue.some(item => item.status === 'retry_wait')) this.state.status = 'retry_wait';
else if (this.state.queue.some(item => ['error', 'quarantined'].includes(item.status))) this.state.status = 'attention';
else this.state.status = 'idle';
}
startItem(item, found) {
const source = found.adapter.id;
item.status = 'downloading'; item.started = item.kind === 'inspection' ? new Date().toISOString() : item.started || new Date().toISOString(); delete item.error;
this.state.currentIds[source] = item.id;
this.state.lastError = null;
this.recomputeStatus();
const entry = { id: item.id, source, promise: null };
this.active.set(source, entry);
entry.promise = this.runItem(item, found, source).finally(async () => {
this.active.delete(source);
if (this.state.currentIds[source] === item.id) delete this.state.currentIds[source];
this.recomputeStatus();
await this.persist();
}).catch(error => { console.error('[scraper] persistence failed:', safeMessage(error)); this.state.paused = true; });
}
async runItem(item, found, source) {
await this.event('info', `${item.kind === 'discovery' ? 'Checking' : 'Starting'} ${item.title}`, { id: item.id, title: item.title, source, kind: item.kind });
try {
if (item.kind === 'discovery' || item.kind === 'inspection') {
await this.runDiscovery(item, found, source);
return;
}
let revisionNotifications = 0;
const result = await found.adapter.run(item, {
libraryRoot: this.libraryRoot, settings: settingsForSource(this.state.settings, source),
shouldStop: () => this.stopping ? 'stopped' : this.state.paused ? 'paused' : item.stopReason || false,
onCatalog: catalog => this.recordCatalog(item, catalog),
onPublished: async details => {
revisionNotifications++;
this.state.publishedRevision = Number(this.state.publishedRevision || 0) + 1;
const mangaId = item.libraryMangaId || (details.mangaDir && path.basename(details.mangaDir));
if (mangaId) this.recordUpdateCheck({ ...item, libraryMangaId: mangaId }, { lastDownloadedAt: new Date().toISOString() });
this.invalidateInventory();
await this.event('info', `Published ${item.title}${details.chapter ? ` chapter ${details.chapter}` : ''}`, {
id: item.id, title: item.title, source, ...(details.chapter ? { chapter: details.chapter } : {}),
});
},
});
const finalPath = typeof result === 'string' ? result : result?.finalPath || null;
if (!revisionNotifications && (typeof result === 'string' || Number(result?.publishedCount || 0) > 0)) {
this.state.publishedRevision = Number(this.state.publishedRevision || 0) + 1;
this.invalidateInventory();
}
item.status = 'complete'; item.completed = new Date().toISOString(); item.finalPath = finalPath;
delete this.state.circuits[source];
for (const incident of this.state.incidents) if (incident.jobId === item.id && incident.state === 'open') { incident.state = 'resolved'; incident.resolution = 'recovered'; }
item.publishedCount = Number(result?.publishedCount ?? 0);
if (item.publishedCount > 0 && !revisionNotifications) {
const mangaId = item.libraryMangaId || (finalPath && path.basename(finalPath));
if (mangaId) this.recordUpdateCheck({ ...item, libraryMangaId: mangaId }, { lastDownloadedAt: item.completed });
}
item.selectedCount = Number(result?.selectedCount ?? 0);
this.state.history.push({ id: item.id, title: item.title, sourceTitle: item.sourceTitle, availability: item.availability, source, status: 'complete', time: item.completed, finalPath, publishedCount: item.publishedCount, selectedCount: item.selectedCount });
await this.event('info', `Completed ${item.title}`, { id: item.id, title: item.title, source, finalPath, publishedCount: item.publishedCount });
} catch (error) {
if (error instanceof ArchiveControlError) {
item.status = error.reason === 'quarantined' ? 'quarantined' : 'queued';
delete item.stopReason;
await this.event('info', `${item.title} ${error.reason}`, { id: item.id, title: item.title, source });
return;
}
const failure = classifyFailure(error);
item.failure = failure; item.error = failure.message;
recordFailure(this.state.sourceLedger ||= {}, item, failure);
const incident = { id: crypto.randomUUID(), at: new Date().toISOString(), jobId: item.id, source, title: item.title, failure, state: 'open' };
this.state.incidents.push(incident); this.state.incidents = this.state.incidents.slice(-200);
if (failure.kind === 'unavailable') { item.status = 'unavailable'; incident.state = 'resolved'; incident.resolution = 'skipped-unavailable'; }
else if (failure.retryable && (item.recoveryAttempts || 0) < 1) {
item.recoveryAttempts = (item.recoveryAttempts || 0) + 1;
item.status = 'retry_wait'; item.nextAttemptAt = new Date(Date.now() + this.retryDelayMs).toISOString();
this.state.circuits[source] = { state: 'open', until: item.nextAttemptAt, jobId: item.id, reason: 'transient' };
} else if (failure.scope === 'source') { item.status = 'error'; this.state.sourcePaused[source] = true; }
else if (failure.scope === 'global') { item.status = 'error'; this.state.paused = true; }
else {
item.status = 'quarantined';
const similar = this.state.incidents.filter(i => i.source === source && i.failure.kind === 'adapter' && Date.now() - Date.parse(i.at) < 3600000);
if (new Set(similar.map(i => i.jobId)).size >= 2) this.state.sourcePaused[source] = true;
}
await this.event(failure.kind === 'unavailable' ? 'warning' : 'error', failure.message, { id: item.id, title: item.title, source, failure, nextAttemptAt: item.nextAttemptAt || null });
}
}
async workLoop() {
while (!this.stopping) {
this.runtimeHeartbeat = Date.now();
if (!this.state.paused) {
for (const item of this.state.queue) {
if (item.status === 'retry_wait' && Date.parse(item.nextAttemptAt) <= Date.now()) { item.status = 'queued'; delete this.state.circuits[item.source]; }
if (item.status !== 'queued') continue;
const found = this.resolveAdapter(item.url);
if (!found) {
item.status = 'quarantined'; item.error = 'Adapter no longer available';
await this.event('error', item.error, { id: item.id, title: item.title, source: item.source });
continue;
}
const source = found.adapter.id;
if (this.active.has(source) || this.state.sourcePaused[source]) continue;
if (this.state.circuits[source]?.state === 'open') continue;
if (item.libraryMangaId && [...this.active.values()].some(entry => {
const activeItem = this.state.queue.find(candidate => candidate.id === entry.id);
return activeItem?.libraryMangaId === item.libraryMangaId;
})) continue;
this.startItem(item, found);
}
}
const prior = this.state.status;
this.recomputeStatus();
if (prior !== this.state.status) await this.persist();
await sleep(500);
}
}
async close() {
if (!this.state) return;
this.stopping = true;
if (this.discoveryTimer) clearTimeout(this.discoveryTimer);
this.discoveryTimer = null;
this.recomputeStatus();
await this.persist();
const timeout = new Promise(resolve => { const timer = setTimeout(resolve, 30000); timer.unref(); });
const workers = [...this.active.values()].map(entry => entry.promise).filter(Boolean);
const drained = await Promise.race([Promise.allSettled([this.loopPromise || Promise.resolve(), ...workers]).then(() => true), timeout.then(() => false)]);
await this.writeChain;
// Keep the lock until the process exits if an in-flight worker did not drain.
if (drained) await fs.rm(this.lockFile, { force: true });
}
}
module.exports = { ScraperService, DEFAULT_SETTINGS, validateSettings, settingsForSource };
+12
View File
@@ -0,0 +1,12 @@
'use strict';
const { ArchiveControlError } = require('../lib/control');
function controlledHttp(http, shouldStop) {
return { async get(...args) {
const check = () => { const reason = shouldStop?.(); if (reason) throw new ArchiveControlError(reason === true ? 'paused' : String(reason)); };
check(); const response = await http.get(...args); check(); return response;
} };
}
module.exports = { controlledHttp };
+26
View File
@@ -0,0 +1,26 @@
'use strict';
const { SlowHttp } = require('../lib/http');
const { mangaDexInfo, discoverMangaDexCatalog } = require('../adapters/mangadex');
const { runMangaDexArchive } = require('../jobs/mangadex-archive');
const { controlledHttp } = require('./common');
module.exports = {
id: 'mangadex', label: 'MangaDex', match: mangaDexInfo,
async discover(item, options) {
const http = new SlowHttp({ gateKey: 'mangadex', minMs: options.settings.minMs, maxMs: options.settings.maxMs,
maxRetries: 2, userAgent: 'MangaCircus-Scraper/1.0 (personal archival; sequential MangaDex requests)',
onEvent: options.onEvent });
return discoverMangaDexCatalog(item.url, controlledHttp(http, options.shouldStop));
},
run(item, options) {
return runMangaDexArchive({
seriesUrl: item.url, title: item.title, libraryRoot: options.libraryRoot,
metadataSearch: item.metadataSearch || item.title,
minMs: options.settings.minMs, maxMs: options.settings.maxMs,
chapterMinMs: options.settings.chapterMinMs, chapterMaxMs: options.settings.chapterMaxMs,
shouldStop: options.shouldStop, scope: item.scope, existingPolicy: item.existingPolicy,
libraryMangaId: item.libraryMangaId, onPublished: options.onPublished, onCatalog: options.onCatalog,
});
},
};
+29
View File
@@ -0,0 +1,29 @@
'use strict';
const { createHostHttp } = require('../plugins/broker');
const { mangaPillInfo, discoverMangaPillCatalog, discoverMangaPillChapter } = require('../adapters/mangapill');
const { runArchive } = require('../jobs/weebcentral-archive');
const { controlledHttp } = require('./common');
const createHttp = options => createHostHttp('mangapill.com', options);
module.exports = {
id: 'mangapill', label: 'MangaPill', match: mangaPillInfo,
async discover(item, options) {
const http = createHttp({ minMs: options.settings.minMs, maxMs: options.settings.maxMs,
maxRetries: 2, onEvent: options.onEvent });
return discoverMangaPillCatalog(item.url, controlledHttp(http, options.shouldStop));
},
run(item, options) {
return runArchive({
source: 'mangapill', sourceLabel: 'MangaPill', sourceId: mangaPillInfo(item.url)?.id,
discoverCatalog: discoverMangaPillCatalog, discoverChapter: discoverMangaPillChapter, createHttp,
seriesUrl: item.url, libraryRoot: options.libraryRoot,
metadataSearch: item.metadataSearch || item.title,
minMs: options.settings.minMs, maxMs: options.settings.maxMs,
chapterMinMs: options.settings.chapterMinMs, chapterMaxMs: options.settings.chapterMaxMs,
shouldStop: options.shouldStop, scope: item.scope, existingPolicy: item.existingPolicy,
libraryMangaId: item.libraryMangaId, onPublished: options.onPublished, onCatalog: options.onCatalog,
});
},
};
+25
View File
@@ -0,0 +1,25 @@
'use strict';
const { createWeebHttp } = require('../plugins/broker');
const { discoverWeebCatalog, weebCentralInfo } = require('../adapters/weebcentral');
const { runArchive } = require('../jobs/weebcentral-archive');
const { controlledHttp } = require('./common');
module.exports = {
id: 'weebcentral', label: 'WeebCentral', match: weebCentralInfo,
async discover(item, options) {
const http = createWeebHttp({ minMs: options.settings.minMs, maxMs: options.settings.maxMs,
maxRetries: 2, onEvent: options.onEvent });
return discoverWeebCatalog(item.url, controlledHttp(http, options.shouldStop));
},
run(item, options) {
return runArchive({
seriesUrl: item.url, libraryRoot: options.libraryRoot,
metadataSearch: item.metadataSearch || item.title,
minMs: options.settings.minMs, maxMs: options.settings.maxMs,
chapterMinMs: options.settings.chapterMinMs, chapterMaxMs: options.settings.chapterMaxMs,
shouldStop: options.shouldStop, scope: item.scope, existingPolicy: item.existingPolicy,
libraryMangaId: item.libraryMangaId, onPublished: options.onPublished, onCatalog: options.onCatalog,
});
},
};
+40
View File
@@ -0,0 +1,40 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs/promises');
const os = require('node:os');
const path = require('node:path');
const { ScraperService } = require('../service');
test('status and watchdog read the checkpoint for each source layout', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-checkpoint-location-'));
t.after(() => fs.rm(root, { recursive: true, force: true }));
for (const [source, id, folder] of [
['weebcentral', 'W1', '.mcscraper-work-W1'],
['mangadex', 'D1', '.mcscraper-work-D1'],
['mangapill', 'P1', '.mcscraper-work-mangapill-P1'],
]) {
const dir = path.join(root, folder);
await fs.mkdir(dir);
await fs.writeFile(path.join(dir, 'job.json'), JSON.stringify({
status: 'downloading', publicationMode: 'incremental', jobCompleted: 3,
total: 10, updated: '2026-10-14T12:00:00Z', current: { downloadedPages: 4 },
}));
const service = Object.create(ScraperService.prototype);
service.libraryRoot = root;
const job = await service.jobState({ source, seriesId: id, kind: 'download' });
assert.equal(job.completed, 3, source);
assert.equal(job.current.downloadedPages, 4, source);
}
});
test('daemon refuses recovery when the checkpoint cannot be read', async () => {
const service = Object.create(ScraperService.prototype);
service.state = { paused: false, currentIds: { mangapill: 'job' },
queue: [{ id: 'job', source: 'mangapill', status: 'downloading' }] };
service.jobState = async () => null;
const result = await service.recover({ id: 'job', progressKey: '[0,0,"",0,""]' });
assert.deepEqual(result, { changed: false });
assert.equal(service.state.queue[0].stopReason, undefined);
});
+71
View File
@@ -0,0 +1,71 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs').promises;
const os = require('node:os');
const path = require('node:path');
const { ScraperService } = require('../service');
const adapter = { id: 'alpha', label: 'Alpha',
match(value) { return value.startsWith('https://alpha.test/') ? { id: '42', source: 'alpha', title: 'Test', url: value } : null; },
async discover() { return { title: 'Test', chapters: [{ id: 'one', number: '1' }] }; },
async run() { throw new Error('A no-change check must not download'); },
};
const resolve = value => { const info = adapter.match(value); return info ? { adapter, info } : null; };
const waitFor = async predicate => {
const deadline = Date.now() + 3000;
while (!predicate()) {
if (Date.now() > deadline) throw new Error('Timed out waiting for local discovery');
await new Promise(resolveWait => setTimeout(resolveWait, 25));
}
};
test('finished-job cleanup preserves unfinished jobs, legacy check summaries, history and library files', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-scraper-dashboard-'));
const dataRoot = path.join(root, 'data'), libraryRoot = path.join(root, 'library');
await fs.mkdir(dataRoot); await fs.mkdir(path.join(libraryRoot, 'Test'), { recursive: true });
const chapterFile = path.join(libraryRoot, 'Test', 'keep.jpg');
await fs.writeFile(chapterFile, 'library-bytes');
const queue = [
{ id: 'done-a', kind: 'download', source: 'alpha', url: 'https://alpha.test/42', status: 'complete' },
{ id: 'checked', kind: 'discovery', source: 'alpha', url: 'https://alpha.test/42', status: 'complete', libraryMangaId: 'Test', checked: '2026-10-06T12:00:00Z' },
{ id: 'ready', kind: 'inspection', source: 'alpha', url: 'https://alpha.test/42', status: 'ready' },
{ id: 'failed', kind: 'download', source: 'alpha', url: 'https://alpha.test/42', status: 'quarantined' },
{ id: 'queued', kind: 'download', source: 'alpha', url: 'https://alpha.test/42', status: 'queued' },
{ id: 'done-b', kind: 'download', source: 'beta', url: 'https://beta.test/42', status: 'complete' },
];
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({ paused: true, queue, history: [{ id: 'done-a', status: 'complete' }] }));
const service = await new ScraperService({ libraryRoot, dataRoot, adapterFor: resolve, adapters: [adapter, { id: 'beta', label: 'Beta' }] }).init();
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
assert.deepEqual(service.state.queue.map(item => item.id), ['done-a', 'ready', 'failed', 'queued', 'done-b']);
assert.equal((await service.publicState()).updateChecks[0].lastCheckedAt, '2026-10-06T12:00:00Z');
assert.equal((await service.clearCompleted('alpha')).cleared, 1);
assert.deepEqual(service.state.queue.map(item => item.id), ['ready', 'failed', 'queued', 'done-b']);
assert.equal((await service.clearCompleted()).cleared, 1);
assert.equal((await service.clearCompleted()).cleared, 0);
assert.deepEqual(service.state.queue.map(item => item.id), ['ready', 'failed', 'queued']);
assert.deepEqual(service.state.queue.map(item => item.order), [0, 1, 2]);
assert.equal((await fs.readFile(chapterFile, 'utf8')), 'library-bytes');
assert.equal(service.state.history[0].id, 'done-a');
assert.equal(JSON.parse(await fs.readFile(path.join(dataRoot, 'queue.json'), 'utf8')).updateChecks['["Test","alpha"]'].lastCheckedAt, '2026-10-06T12:00:00Z');
});
test('a nightly no-change check updates the latest timestamp and retires its queue row', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-scraper-check-summary-'));
const dataRoot = path.join(root, 'data'), libraryRoot = path.join(root, 'library');
const chapter = path.join(libraryRoot, 'Test', 'v01', 'c0001 [Alpha]');
await fs.mkdir(dataRoot); await fs.mkdir(chapter, { recursive: true });
await fs.writeFile(path.join(libraryRoot, 'Test', 'manga.json'), JSON.stringify({ title: 'Test', status: 'ongoing', sources: [{ name: 'Alpha', url: 'https://alpha.test/title/42' }] }));
await fs.writeFile(path.join(chapter, 'chapter.json'), JSON.stringify({ source: { name: 'Alpha', url: 'https://alpha.test/chapter/one' } }));
await fs.writeFile(path.join(chapter, '0001.png'), Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
const service = await new ScraperService({ libraryRoot, dataRoot, adapterFor: resolve, adapters: [adapter] }).init();
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
await service.enqueueDiscoveryChecks('schedule');
await waitFor(() => service.state.updateChecks['["Test","alpha"]']?.lastCheckedAt);
assert.equal(service.state.queue.length, 0);
const summary = (await service.publicState()).updateChecks[0];
assert.equal(summary.lastResult, 'no-updates');
assert.equal(summary.lastNewContentAt, undefined);
assert.equal(summary.lastDownloadedAt, undefined);
assert.ok(service.state.history.some(item => item.status === 'checked'));
});
+92
View File
@@ -0,0 +1,92 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const { ScraperService, validateSettings } = require('../service');
function fixture(scanLibrary) {
const service = new ScraperService({ scanLibrary, adapters: [] });
service.state = { version: 4, paused: true, status: 'paused', queue: [], settings: validateSettings({}), currentIds: {} };
service.recentEvents = async () => [];
return service;
}
const inventory = title => ({ mangas: title ? [{ id: title, title, status: 'ongoing', autoUpdate: true, chapters: [], sources: [] }] : [] });
test('cold status returns controls immediately and simultaneous probes share one slow scan', { timeout: 1000 }, async () => {
let finish, scans = 0;
const service = fixture(() => { scans++; return new Promise(resolve => { finish = resolve; }); });
const states = await Promise.all(Array.from({ length: 8 }, () => service.publicState()));
assert.equal(scans, 1);
for (const state of states) {
assert.equal(state.paused, true);
assert.equal(state.settings.sourceRates.mangadex.minMs, 1000);
assert.deepEqual(state.library, []);
assert.equal(state.libraryStatus.ready, false);
assert.equal(state.libraryStatus.refreshing, true);
}
finish(inventory('Demo'));
await service.inventoryRefresh;
const ready = await service.publicState();
assert.equal(ready.library[0].title, 'Demo');
assert.equal(ready.libraryStatus.ready, true);
assert.equal(ready.libraryStatus.stale, false);
assert.equal(scans, 1);
});
test('status keeps previous summary but authoritative callers await scans and reject invalidated results', { timeout: 1000 }, async () => {
const completions = [];
const service = fixture(() => new Promise(resolve => completions.push(resolve)));
const first = service.libraryInventory(true);
await Promise.resolve();
completions[0](inventory('Old'));
await first;
service.invalidateInventory();
const fresh = service.libraryInventory(true);
await Promise.resolve();
let finished = false;
fresh.then(() => { finished = true; });
let state = await service.publicState();
assert.equal(state.library[0].title, 'Old');
assert.equal(state.libraryStatus.stale, true);
assert.equal(state.libraryStatus.refreshing, true);
assert.equal(finished, false);
// Publication during an in-flight scan requires another authoritative scan.
service.invalidateInventory();
const scan = service.inventoryRefresh;
completions[1](inventory('Obsolete'));
await scan;
await Promise.resolve();
assert.equal(completions.length, 3);
assert.equal(finished, false);
completions[2](inventory('Published'));
assert.equal((await fresh).mangas[0].title, 'Published');
state = await service.publicState();
assert.equal(state.library[0].title, 'Published');
assert.equal(state.libraryStatus.stale, false);
});
test('background inventory failures are visible without breaking controls or launching a poll storm', { timeout: 1000 }, async () => {
let scans = 0;
const service = fixture(async () => { scans++; throw new Error('Scan failed https://test.example/a?token=private'); });
await service.publicState();
await assert.rejects(service.inventoryRefresh, /Scan failed/);
for (let n = 0; n < 3; n++) {
const state = await service.publicState();
assert.equal(state.paused, true);
assert.match(state.libraryStatus.error, /Scan failed/);
assert.ok(!state.libraryStatus.error.includes('private'));
}
assert.equal(scans, 1);
await assert.rejects(service.libraryInventory(true), /Scan failed/);
assert.equal(scans, 2);
});
test('dashboard polling reuses a recent snapshot while decisions still force a fresh scan', async () => {
let scans = 0;
const service = fixture(async () => { scans++; return inventory('Demo'); });
await service.libraryInventory(true);
service.inventoryCachedAt = Date.now() - 30000;
for (let n = 0; n < 5; n++) assert.equal((await service.publicState()).library[0].title, 'Demo');
assert.equal(scans, 1);
await service.libraryInventory(true);
assert.equal(scans, 2);
});
+41
View File
@@ -0,0 +1,41 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs/promises');
const os = require('node:os');
const path = require('node:path');
const { writeJsonAtomic } = require('../lib/json-file');
test('Windows rename fallback replaces JSON without deleting the old state first', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-json-'));
t.after(() => fs.rm(root, { recursive: true, force: true }));
const file = path.join(root, 'queue.json');
await fs.writeFile(file, '{"old":true}\n');
let failedOnce = false;
const io = { ...fs, async rename(from, to) {
if (to === file && !failedOnce) {
failedOnce = true;
const error = new Error('destination busy'); error.code = 'EEXIST'; throw error;
}
return fs.rename(from, to);
} };
await writeJsonAtomic(file, { next: true }, io);
assert.deepEqual(JSON.parse(await fs.readFile(file, 'utf8')), { next: true });
assert.deepEqual(await fs.readdir(root), ['queue.json']);
});
test('failed replacement restores the old JSON', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-json-'));
t.after(() => fs.rm(root, { recursive: true, force: true }));
const file = path.join(root, 'queue.json');
await fs.writeFile(file, '{"old":true}\n');
const io = { ...fs, async rename(from, to) {
if (to === file && from.endsWith('.tmp')) {
const error = new Error('replacement blocked'); error.code = 'EEXIST'; throw error;
}
return fs.rename(from, to);
} };
await assert.rejects(writeJsonAtomic(file, { next: true }, io), /replacement blocked/);
assert.deepEqual(JSON.parse(await fs.readFile(file, 'utf8')), { old: true });
assert.deepEqual(await fs.readdir(root), ['queue.json']);
});
+138
View File
@@ -0,0 +1,138 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('node:fs/promises'), os = require('node:os'), path = require('node:path');
const { specialKey, parseSpecialFolder } = require('../../chapter-identity');
const { selectChapterReleases, availabilitySummary, discoverMangaDexCatalog, discoverMangaDexChapter } = require('../adapters/mangadex');
const { selectByScope } = require('../lib/scope');
const inventory = require('../lib/library-inventory');
const { publishManifest } = require('../lib/publisher');
const reader = require('../../library');
const { ScraperService } = require('../service');
const ids = ['37d6949a-c7ce-4855-b838-4fd572e87629', '27dd738c-eca2-488d-a791-53ab4b24313b'];
const titleId = '06a37094-b3ba-4f25-bf2b-54c9751e0e91';
const release = (id, chapter = null) => ({ id, attributes: { chapter, pages: 46, title: 'Pilot', externalUrl: null }, relationships: [] });
const pause = ms => new Promise(resolve => setTimeout(resolve, ms));
async function until(predicate) {
const end = Date.now() + 5000;
while (!predicate()) { if (Date.now() > end) throw new Error('Fixture timed out'); await pause(20); }
}
test('unnumbered releases retain source IDs, do not invent chapter zero, and remain distinct', async () => {
const selected = selectChapterReleases(ids.map(id => release(id)), 'Pilot');
assert.equal(selected.length, 2);
assert.deepEqual(new Set(selected.map(r => r.number)), new Set(ids.map(specialKey)));
assert.ok(selected.every(r => r.special && r.label === 'Pilot' && r.sourceNumber === null));
assert.equal(selectByScope(selected, { mode: 'chapter', chapter: '0' }).length, 0);
assert.equal(selectByScope(selected, { mode: 'after', chapter: '1' }).length, 0);
assert.equal(selectByScope(selected, { mode: 'missing' }).length, 2);
assert.equal(selectByScope(selected, { mode: 'ids', ids: [ids[0]] })[0].id, ids[0]);
assert.throws(() => specialKey('../escape'));
assert.equal(parseSpecialFolder(specialKey(ids[0]) + ' Pilot [MangaDex]').src, 'MangaDex');
const chapter = await discoverMangaDexChapter(selected[0], { async get() { return { async json() { return { result: 'ok', baseUrl: 'https://uploads.mangadex.org', chapter: { hash: 'fixture', data: Array.from({ length: 46 }, (_, n) => `${n}.png`) } }; } }; } });
assert.equal(chapter.label, 'Pilot'); assert.equal(chapter.sourceId, selected[0].id);
});
test('availability distinguishes no English, external-only, and partial hosted catalogs', () => {
assert.equal(availabilitySummary({ availableTranslatedLanguages: ['ar'] }, [], [], 0).reason, 'no-english');
const external = availabilitySummary({}, [], [], 8);
assert.equal(external.reason, 'external-only'); assert.match(external.message, /8 English external-link/);
const chapters = selectChapterReleases([release('regular-fixture', '336')]);
const summary = availabilitySummary({ lastChapter: '686' }, [release('regular-fixture', '336')], chapters);
assert.equal(summary.coverage, 'partial'); assert.equal(summary.selectedChapters, 1);
assert.equal(availabilitySummary({ lastChapter: '336' }, [], chapters).coverage, 'partial', 'One late chapter does not cover the earlier 335');
const full = selectChapterReleases(['1', '2', '3'].map((number, index) => release(`full-${index}`, number)));
assert.equal(availabilitySummary({ lastChapter: '3' }, [], full).coverage, 'unknown', 'A plausible count is still not proof of whole-series completeness');
});
test('catalog uses separate external-only feed only when hosted selections are empty', async () => {
const modes = [];
const http = { async get(url) {
const parsed = new URL(url), mode = parsed.searchParams.get('includeExternalUrl');
if (!mode) return { async json() { return { data: { type: 'manga', attributes: { title: { en: 'CLAYMORE' }, lastChapter: '155' } } }; } };
modes.push(mode);
return { async json() { return { result: 'ok', data: [], total: mode === '1' ? 8 : 0 }; } };
} };
const catalog = await discoverMangaDexCatalog(`https://mangadex.org/title/${titleId}`, http);
assert.deepEqual(modes, ['0', '1']); assert.equal(catalog.availability.reason, 'external-only');
modes.length = 0;
http.get = async url => {
const mode = new URL(url).searchParams.get('includeExternalUrl');
if (!mode) return { async json() { return { data: { type: 'manga', attributes: { title: { en: 'Bleach (Pre-Serialization)' } } } }; } };
modes.push(mode); return { async json() { return { result: 'ok', data: ids.map(id => release(id)), total: 2 }; } };
};
const pilot = await discoverMangaDexCatalog(`https://mangadex.org/title/${titleId}`, http);
assert.deepEqual(modes, ['0']); assert.equal(pilot.availability.specialReleases, 2); assert.equal(pilot.chapters[0].label, 'Pilot');
});
test('special publication is atomic, readable, stable across rescans and duplicate-safe', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-special-'));
t.after(() => fs.rm(root, { recursive: true, force: true }));
// Real fixture PNG; no upstream image requests.
const { Canvas } = require('../../tools/png');
const page = path.join(root, '.fixture.png'); await fs.writeFile(page, new Canvas(10, 20, [1, 2, 3]).png());
const manifest = { version: 1, series: { title: 'Pilot edition', direction: 'rtl', sources: [{ name: 'MangaDex', url: `https://mangadex.org/title/${titleId}` }] },
chapters: selectChapterReleases(ids.map(id => release(id)), 'Pilot').map(ref => ({ ...ref, source: { name: 'MangaDex', url: ref.url }, pages: [{ path: page }] })) };
const first = await publishManifest(manifest, { libraryRoot: root });
assert.ok(first.results.every(result => result.status === 'published'));
const scanned = (await reader.scanLibrary(root)).index.mangas[0];
assert.equal(scanned.chapters.length, 2); assert.ok(scanned.chapters.every(c => c.label === 'Pilot' && c.num === null && !c.extra));
const stableIds = scanned.chapters.map(c => c.id);
assert.deepEqual((await publishManifest(manifest, { libraryRoot: root })).results.map(r => r.status), ['skipped', 'skipped']);
assert.deepEqual((await reader.scanLibrary(root)).index.mangas[0].chapters.map(c => c.id), stableIds);
const manga = (await inventory.scanLibrary(root)).mangas[0];
assert.equal(inventory.shouldDownload(manga, manifest.chapters[0], 'mangadex'), false);
assert.equal(inventory.isNewerThanLibrary(manga, manifest.chapters[0]), false);
assert.equal(inventory.isNewerThanLibrary(manga, { number: '1' }), true);
assert.equal(reader.parseKey('c0012.5 Existing [Host]').key, '12.5');
assert.equal(reader.parseKey('v01').key, '1');
const regular = { ...manifest, chapters: [{ number: '12.5', title: 'Existing chapter', source: { name: 'Host', url: 'https://host.test/12' }, pages: [{ path: page }] }] };
await publishManifest(regular, { libraryRoot: root });
assert.ok((await reader.scanLibrary(root)).index.mangas[0].chapterById.has('1/12.5'));
});
test('legacy unavailable messages become unverified without retrying or downloading', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-unavailable-')), dataRoot = path.join(root, 'data');
await fs.mkdir(dataRoot); await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({ paused: true,
queue: [{ id: 'old-job', source: 'mangadex', url: `https://mangadex.org/title/${titleId}`, status: 'unavailable', error: 'No downloadable English MangaDex chapters found', failure: { kind: 'unavailable', scope: 'job' } }],
settings: { autoUpdateEnabled: false } }));
const service = await new ScraperService({ libraryRoot: path.join(root, 'library'), dataRoot }).init();
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
assert.equal(service.state.queue[0].status, 'unavailable'); assert.equal(service.active.size, 0);
assert.match(service.state.queue[0].error, /Inspect this source again/);
assert.equal(service.state.queue[0].failure.kind, 'unavailable');
});
test('inspection waits for confirmation, survives restart, and uses one normal source lane', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-inspect-')), dataRoot = path.join(root, 'data');
await fs.mkdir(dataRoot); await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({ paused: true, queue: [], settings: { autoUpdateEnabled: false } }));
let running = 0, maxRunning = 0, downloads = 0;
const catalog = { title: 'Canonical Pilot', chapters: ids.map(id => ({ id, number: specialKey(id), special: true })), availability: { selectedChapters: 2, specialReleases: 2, coverage: 'unknown' } };
const adapter = { id: 'fixture', label: 'Fixture', match: url => ({ source: 'fixture', id: url.split('/').pop(), url, title: 'Misleading slug' }),
async discover() { maxRunning = Math.max(maxRunning, ++running); await pause(50); running--; return catalog; },
async run(item, options) { downloads++; await options.onCatalog(catalog); return { publishedCount: 1, selectedCount: 1 }; } };
const options = { libraryRoot: path.join(root, 'library'), dataRoot, adapters: [adapter], adapterFor: url => ({ adapter, info: adapter.match(url) }) };
let service = await new ScraperService(options).init();
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
const item = await service.enqueue({ url: 'https://fixture.test/one', inspect: true });
const other = await service.enqueue({ url: 'https://fixture.test/two', inspect: true });
await assert.rejects(service.control('confirm', item.id), /Inspect the source first/);
await service.control('resume'); await until(() => item.status === 'ready' && other.status === 'ready');
assert.equal(downloads, 0); assert.equal(maxRunning, 1); assert.equal(item.sourceTitle, 'Canonical Pilot');
await service.close(); service = await new ScraperService(options).init();
const restored = service.state.queue.find(q => q.id === item.id);
assert.equal(restored.kind, 'inspection'); assert.equal(restored.status, 'ready');
await service.control('confirm', restored.id); await until(() => restored.status === 'complete');
await until(() => service.active.size === 0);
assert.equal(downloads, 1); assert.equal(restored.selectedCount, 1);
assert.equal(service.state.history.at(-1).availability.specialReleases, 2);
service.state.sourcePaused.fixture = true;
await assert.rejects(service.control('inspect', restored.id), /access block/);
});
test('edition change after confirmation cannot silently download another edition', async () => {
const service = Object.create(ScraperService.prototype);
service.persist = async () => {};
service.state = { sourceLedger: {} };
const item = { title: 'slug', confirmedSourceTitle: 'Pilot edition' };
await assert.rejects(service.recordCatalog(item, { title: 'Main edition', chapters: [] }), /edition title changed/);
});
+61
View File
@@ -0,0 +1,61 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs/promises');
const os = require('node:os');
const path = require('node:path');
const { adapterFor } = require('../adapters');
const { parse } = require('../plugins/registry');
const { discoverMangaPillCatalog, discoverMangaPillChapter } = require('../adapters/mangapill');
const { validateHostRequest } = require('../plugins/broker');
const { runArchive } = require('../jobs/weebcentral-archive');
const { Canvas } = require('../../tools/png');
const { scanLibrary } = require('../lib/library-inventory');
const seriesUrl = 'https://mangapill.com/manga/552/bleach';
const chapterUrl = 'https://mangapill.com/chapters/552-10001000/bleach-chapter-1';
const imageUrl = 'https://cdn.readdetectiveconan.com/file/mangap/552/10001000/1.jpg';
const catalogHtml = `<h1 class="font-bold">Bleach</h1><div id="chapters"><a href="/chapters/552-10002000/bleach-chapter-2">Chapter 2</a><a href="/chapters/552-10001000/bleach-chapter-1">Chapter 1</a></div>`;
const chapterHtml = `<picture><img class="js-page" data-src="${imageUrl}"/></picture>`;
test('MangaPill discovery stays in its host and keeps ordered source chapter IDs', async () => {
assert.equal(adapterFor(seriesUrl).adapter.id, 'mangapill');
assert.equal(adapterFor('http://mangapill.com/manga/552/bleach'), null);
const http = { async get(url) {
assert.ok([seriesUrl, chapterUrl].includes(url));
return { async text() { return url === seriesUrl ? catalogHtml : chapterHtml; } };
} };
const catalog = await discoverMangaPillCatalog(seriesUrl, http);
assert.deepEqual(catalog.chapters.map(chapter => [chapter.id, chapter.number]),
[['552-10001000', '1'], ['552-10002000', '2']]);
const chapter = await discoverMangaPillChapter(catalog.chapters[0], http);
assert.deepEqual(chapter.pages, [{ url: imageUrl, headers: { referer: chapterUrl } }]);
assert.throws(() => parse('mangapill', 'pages', {
html: '<img class="js-page" data-src="https://other.example/file/mangap/1.jpg">', url: chapterUrl,
}), /page host/);
assert.throws(() => validateHostRequest('https://other.example/manga/552', 'mangapill.com', { accept: 'text/html' }), /left its source host/);
});
test('MangaPill uses the shared checkpoint-safe publisher and skips an existing chapter', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-mangapill-'));
t.after(() => fs.rm(root, { recursive: true, force: true }));
const png = new Canvas(10, 20, [1, 2, 3]).png();
const createHttp = () => ({ async get(url) {
if (url === seriesUrl) return { async text() { return catalogHtml.replace(/<a[^>]*>Chapter 2<\/a>/, ''); } };
if (url === chapterUrl) return { async text() { return chapterHtml; } };
if (url === imageUrl) return { async arrayBuffer() { return png; } };
throw new Error(`Unexpected source request: ${url}`);
} });
const options = { source: 'mangapill', sourceLabel: 'MangaPill', sourceId: '552',
discoverCatalog: discoverMangaPillCatalog, discoverChapter: discoverMangaPillChapter, createHttp,
seriesUrl, libraryRoot: root, scope: { mode: 'missing' },
minMs: 1000, maxMs: 1000, chapterMinMs: 5000, chapterMaxMs: 5000 };
const first = await runArchive(options);
assert.equal(first.publishedCount, 1);
const manga = (await scanLibrary(root)).mangas[0];
assert.equal(manga.ids.mangapill, '552');
assert.equal(manga.chapters[0].sourceKey, 'mangapill');
const second = await runArchive(options);
assert.equal(second.publishedCount, 0);
assert.equal(second.selectedCount, 0);
});
+60
View File
@@ -0,0 +1,60 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const { ScraperService } = require('../service');
async function confirm(item) {
const service = Object.create(ScraperService.prototype);
service.state = { queue: [item], currentIds: {}, sourcePaused: {} };
service.event = async () => {};
await service.control('confirm', item.id);
return item;
}
async function retry(item) {
const service = Object.create(ScraperService.prototype);
service.state = { queue: [item], currentIds: {}, sourcePaused: {}, circuits: {}, incidents: [], paused: true };
service.event = async () => {};
service.recomputeStatus = () => {};
service.resolveAdapter = () => ({ adapter: { id: item.source } });
await service.control('retry', item.id);
return item;
}
test('confirming an inspection uses the canonical title for automatic metadata lookup', async () => {
const item = await confirm({ id: 'job', seriesId: '01J76XYFHVMSQVDJPQ42TN2XYN',
title: '01J76XYFHVMSQVDJPQ42TN2XYN', metadataSearch: '01J76XYFHVMSQVDJPQ42TN2XYN',
sourceTitle: 'The Exiled Heavy Knight Knows How to Game the System', kind: 'inspection', status: 'ready' });
assert.equal(item.title, item.sourceTitle);
assert.equal(item.metadataSearch, item.sourceTitle);
assert.equal(item.metadataSearchExplicit, true);
assert.equal(item.kind, 'download');
});
test('confirming an inspection preserves an explicitly supplied metadata search', async () => {
const item = await confirm({ id: 'job', seriesId: 'series-id', title: 'Heavy Knight',
metadataSearch: 'Heavy Knight (Romanized Title)', metadataSearchExplicit: true,
sourceTitle: 'The Exiled Heavy Knight Knows How to Game the System', kind: 'inspection', status: 'ready' });
assert.equal(item.metadataSearch, 'Heavy Knight (Romanized Title)');
assert.equal(item.title, 'Heavy Knight');
});
test('legacy automatic title searches are replaced with the inspected canonical title', async () => {
const item = await confirm({ id: 'job', seriesId: 'series-id', title: 'heavy-knight-slug',
metadataSearch: 'heavy-knight-slug', sourceTitle: 'The Exiled Heavy Knight Knows How to Game the System',
kind: 'inspection', status: 'ready' });
assert.equal(item.metadataSearch, item.sourceTitle);
});
test('retry repairs a legacy URL-ID lookup before requeueing it', async () => {
const item = await retry({ id: 'job', seriesId: '01J76XYFHVMSQVDJPQ42TN2XYN',
title: '01J76XYFHVMSQVDJPQ42TN2XYN', metadataSearch: '01J76XYFHVMSQVDJPQ42TN2XYN',
sourceTitle: 'The Exiled Heavy Knight Knows How to Game the System', source: 'weebcentral',
status: 'unavailable', error: 'AniList HTTP 404', stallRecoveries: 1 });
assert.equal(item.metadataSearch, item.sourceTitle);
assert.equal(item.title, item.sourceTitle);
assert.equal(item.status, 'queued');
assert.equal(item.error, undefined);
assert.equal(item.stallRecoveries, 0);
});
+111
View File
@@ -0,0 +1,111 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs').promises;
const os = require('node:os');
const path = require('node:path');
const { detectVerticalStrip, resolveReadingDirection, fallbackDirection } = require('../../reading-direction');
const { publishManifest } = require('../lib/publisher');
const { scanLibrary } = require('../../library');
const strip = (w = 800) => ({ w, h: 12000 });
test('strip detection tolerates a landscape title card and rejects weak evidence', () => {
const pages = [{ w: 1200, h: 800 }, strip(), strip(), strip()];
assert.equal(detectVerticalStrip(pages).value, 'ttb');
assert.equal(detectVerticalStrip(pages).evidence.tallPages, 3);
assert.equal(detectVerticalStrip([strip()]), null);
assert.equal(detectVerticalStrip([strip(), strip(), { w: 800, h: 1200 }, { w: 800, h: 1200 }]), null);
assert.equal(detectVerticalStrip([strip(), strip(), strip(1400), { w: 1200, h: 800 }]), null);
assert.equal(detectVerticalStrip([{ w: 800, h: 12000, r: 90 }, strip(), strip()]), null);
assert.equal(detectVerticalStrip([{ w: 12000, h: 800, r: 90 }, strip(), strip()]).value, 'ttb');
});
test('manual choices at chapter, book or manga level beat detection and fallback', () => {
const pages = [strip(), strip(), strip()];
const automatic = fallbackDirection();
assert.equal(resolveReadingDirection([automatic], pages).value, 'ttb');
assert.equal(resolveReadingDirection([automatic]).source, 'fallback');
assert.equal(resolveReadingDirection([{ direction: 'ltr' }, automatic], pages).value, 'ltr');
assert.equal(resolveReadingDirection([{}, { direction: 'rtl' }, automatic], pages).value, 'rtl');
assert.equal(resolveReadingDirection([{ directionOverride: 'rtl', ...automatic }], pages).value, 'rtl');
assert.equal(resolveReadingDirection([{ ...automatic, direction: 'ltr' }], pages).value, 'ltr');
assert.equal(resolveReadingDirection([{ direction: 'rtl' }], pages).source, 'manual');
assert.equal(resolveReadingDirection([], pages).value, 'ttb');
});
function png(w, h) {
const b = Buffer.alloc(24);
Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]).copy(b);
b.writeUInt32BE(w, 16); b.writeUInt32BE(h, 20);
return b;
}
async function json(file) { return JSON.parse(await fs.readFile(file, 'utf8')); }
test('publisher detects new strips, preserves updates and manual metadata, and scanner agrees', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-direction-'));
t.after(() => fs.rm(temp, { recursive: true, force: true }));
const libraryRoot = path.join(temp, 'library');
const sizes = [[1200, 800], [800, 12000], [800, 12500], [800, 14210]];
const files = [];
for (const [i, size] of sizes.entries()) {
const file = path.join(temp, `page${i}.png`);
await fs.writeFile(file, png(...size)); files.push(file);
}
const series = { title: 'Strip Work', ...fallbackDirection(),
sources: [{ name: 'Test', url: 'https://example.test/series' }] };
const chapter = number => ({ number, source: { name: 'Test', url: `https://example.test/${number}` },
book: { number: 1, title: 'Season 1' }, pages: files.map(file => ({ path: file })) });
const first = await publishManifest({ version: 1, series, chapters: [chapter('1')] }, { libraryRoot });
const mangaDir = first.mangaDir, bookDir = path.join(mangaDir, 'v01 Season 1');
const mangaFile = path.join(mangaDir, 'manga.json');
assert.equal((await json(mangaFile)).direction, 'ttb');
assert.equal((await json(mangaFile))['x-reading-direction'].source, 'detected');
const scan = (await scanLibrary(libraryRoot)).index.mangas[0];
assert.equal(scan.chapters[0].versions[0].direction, 'ttb');
assert.equal(scan.chapters[0].versions[0].pages[0].w, 1200);
const same = await publishManifest({ version: 1, series, chapters: [chapter('2')] }, { libraryRoot });
assert.equal(same.mangaDir, mangaDir);
assert.equal((await json(mangaFile)).direction, 'ttb');
const bookFile = path.join(bookDir, 'book.json');
await fs.writeFile(bookFile, JSON.stringify({ title: 'Season 1', direction: 'ltr', note: 'human' }));
const chapterDir = path.join(bookDir, first.results[0].folder);
const chapterFile = path.join(chapterDir, 'chapter.json');
await fs.writeFile(chapterFile, JSON.stringify({ ...await json(chapterFile), directionOverride: 'rtl', note: 'human' }));
await publishManifest({ version: 1, series, chapters: [chapter('1')] }, { libraryRoot, replace: true });
assert.equal((await json(bookFile)).direction, 'ltr');
assert.equal((await json(bookFile)).note, 'human');
assert.equal((await json(chapterFile)).direction, 'rtl');
assert.equal((await json(chapterFile)).directionOverride, 'rtl');
assert.equal((await json(chapterFile)).note, 'human');
const rescanned = (await scanLibrary(libraryRoot)).index.mangas[0];
assert.equal(rescanned.chapters[0].versions[0].direction, 'rtl');
assert.equal(rescanned.chapters[1].versions[0].direction, 'ltr');
await fs.writeFile(mangaFile, JSON.stringify({ ...await json(mangaFile), direction: 'rtl' }));
await publishManifest({ version: 1, series, chapters: [chapter('3')] }, { libraryRoot });
assert.equal((await json(mangaFile)).direction, 'rtl');
assert.equal((await json(mangaFile))['x-reading-direction'].source, 'manual');
assert.equal((await scanLibrary(libraryRoot)).index.mangas[0].chapters[2].versions[0].direction, 'ltr');
});
test('legacy plain RTL stays protected across automatic imports', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-legacy-direction-'));
t.after(() => fs.rm(temp, { recursive: true, force: true }));
const libraryRoot = path.join(temp, 'library');
const page = path.join(temp, 'tall.png'); await fs.writeFile(page, png(800, 12000));
const series = { title: 'Legacy', ...fallbackDirection(), sources: [{ name: 'Test', url: 'https://example.test/legacy' }] };
const chapter = { number: '1', source: { name: 'Test', url: 'https://example.test/legacy/1' },
pages: [1, 2, 3].map(() => ({ path: page })) };
const first = await publishManifest({ version: 1, series, chapters: [chapter] }, { libraryRoot });
const file = path.join(first.mangaDir, 'manga.json');
const meta = await json(file); delete meta['x-reading-direction']; meta.direction = 'rtl';
await fs.writeFile(file, JSON.stringify(meta));
await publishManifest({ version: 1, series, chapters: [{ ...chapter, number: '2', source: { name: 'Test', url: 'https://example.test/legacy/2' } }] }, { libraryRoot });
assert.equal((await json(file)).direction, 'rtl');
assert.equal((await json(file))['x-reading-direction'].source, 'manual');
});
+48
View File
@@ -0,0 +1,48 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict'), fs = require('fs').promises, os = require('os'), path = require('path');
const { ScraperService, validateSettings } = require('../service');
const { classifyFailure, safeMessage } = require('../lib/failures');
test('failure classification protects access challenges and distinguishes job failures', () => {
for (const status of [401, 403, 429]) assert.equal(classifyFailure(Object.assign(new Error('Host response'), { status })).kind, 'access');
assert.equal(classifyFailure(new Error('No downloadable English MangaDex chapters found')).scope, 'job');
assert.equal(classifyFailure(Object.assign(new Error('Disk full'), { code: 'ENOSPC' })).scope, 'global');
assert.equal(classifyFailure(new Error('HTTP 503')).retryable, true);
assert.equal(safeMessage('HTTP 403 https://test.example/a?token=secret'), 'HTTP 403 https://test.example/a');
});
async function fixture(t, errors, options = {}) {
const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-recovery-'));
const dataRoot = path.join(dir, 'data'); await fs.mkdir(dataRoot);
const counts = {};
const adapter = { id: 'test', label: 'Test', run: async item => {
counts[item.id] = (counts[item.id] || 0) + 1;
if (errors[item.id]) throw errors[item.id]; return { publishedCount: 0 };
} };
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({ version: 3, paused: true, settings: validateSettings({ autoUpdateEnabled: false }),
queue: Object.keys(errors).concat('healthy').map(id => ({ id, seriesId: id, source: 'test', title: id, url: 'https://test.example/' + id, status: 'queued' })), history: [] }));
const service = await new ScraperService({ libraryRoot: path.join(dir, 'library'), dataRoot, adapterFor: url => ({ adapter, info: { id: url.split('/').pop() } }), adapters: [adapter], ...options }).init();
t.after(async () => { await service.close(); await fs.rm(dir, { recursive: true, force: true }); });
await service.control('resume');
return { service, counts };
}
async function wait(predicate) {
const end = Date.now() + 6000;
while (!predicate()) { if (Date.now() > end) throw new Error('Timeout'); await new Promise(r => setTimeout(r, 25)); }
}
test('unavailable titles and one parser failure do not stop healthy jobs', async t => {
const { service } = await fixture(t, { absent: new Error('No downloadable English MangaDex chapters found'), parser: new Error('Catalog markup changed') });
await wait(() => service.state.queue.find(i => i.id === 'healthy').status === 'complete');
assert.equal(service.state.queue.find(i => i.id === 'absent').status, 'unavailable');
assert.equal(service.state.queue.find(i => i.id === 'parser').status, 'quarantined');
assert.equal(service.state.sourcePaused.test, undefined);
});
test('transient recovery is bounded and access errors are never retried automatically', async t => {
const { service, counts } = await fixture(t, { broken: new Error('HTTP 503') }, { retryDelayMs: 10 });
await wait(() => service.state.sourcePaused.test);
assert.equal(counts.broken, 2); assert.equal(service.state.queue[0].status, 'error');
});
test('protection failures pause only the affected lane and keep checkpoints', async t => {
const { service, counts } = await fixture(t, { blocked: new Error('HTTP 403') });
await wait(() => service.state.sourcePaused.test);
assert.equal(counts.blocked, 1); assert.equal(counts.healthy, undefined);
assert.equal(service.state.incidents[0].failure.kind, 'access');
});
+580
View File
@@ -0,0 +1,580 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs/promises');
const os = require('node:os');
const path = require('node:path');
const { cleanPart, chapterKey, chapterFolder, assertInside } = require('../lib/names');
const { imageExtension } = require('../lib/images');
const { publishManifest } = require('../lib/publisher');
const { mergeImportedMetadata } = require('../lib/metadata-merge');
const { metadataFromMedia } = require('../metadata/anilist');
const { normalizeScope, selectByScope, scopeLabel } = require('../lib/scope');
const { scanLibrary, findManga, shouldDownload, isNewerThanLibrary, setAutoUpdate } = require('../lib/library-inventory');
const { inspectLegacyWork } = require('../lib/legacy-publication');
const { discoverWeebCatalog, parseChapters, parsePageUrls } = require('../adapters/weebcentral');
const { chapterNumber, selectChapterReleases, pageUrls } = require('../adapters/mangadex');
const { adapterFor } = require('../adapters');
const { ScraperService, validateSettings, settingsForSource } = require('../service');
test('MangaCircus chapter keys are normalized', () => {
assert.equal(chapterKey('12'), 'c0012');
assert.equal(chapterKey('12.50'), 'c0012.5');
assert.equal(chapterKey('x2'), 'x02');
assert.throws(() => chapterKey('special'));
});
test('folder names are portable and versioned', () => {
assert.equal(cleanPart('Bad:<Title>. '), 'BadTitle');
assert.equal(cleanPart('CON'), 'CON-manga');
assert.equal(chapterFolder({ number: '7', title: 'A [Test]', source: { name: 'Host' } }), 'c0007 A Test [Host]');
assert.throws(() => assertInside('C:/library', 'C:/outside'));
});
test('image signatures are recognized', () => {
assert.equal(imageExtension(Buffer.from([0xff, 0xd8, 0xff, 0x00])), 'jpg');
assert.equal(imageExtension(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10])), 'png');
assert.equal(imageExtension(Buffer.from('not an image')), null);
});
test('per-source rate settings default fast and update independently', () => {
const defaults = validateSettings({});
const fast = { minMs: 1000, maxMs: 4000, chapterMinMs: 5000, chapterMaxMs: 15000 };
assert.deepEqual(Object.fromEntries(Object.entries(settingsForSource(defaults, 'weebcentral')).filter(([key]) => key.endsWith('Ms'))), fast);
assert.deepEqual(Object.fromEntries(Object.entries(settingsForSource(defaults, 'mangadex')).filter(([key]) => key.endsWith('Ms'))), fast);
const changed = validateSettings({ sourceRates: { mangadex: {
minMs: 1500, maxMs: 3500, chapterMinMs: 6000, chapterMaxMs: 12000,
} } }, defaults);
assert.deepEqual(Object.fromEntries(Object.entries(settingsForSource(changed, 'mangadex')).filter(([key]) => key.endsWith('Ms'))), {
minMs: 1500, maxMs: 3500, chapterMinMs: 6000, chapterMaxMs: 12000,
});
assert.deepEqual(Object.fromEntries(Object.entries(settingsForSource(changed, 'weebcentral')).filter(([key]) => key.endsWith('Ms'))), fast);
assert.throws(() => validateSettings({ sourceRates: { mangadex: { minMs: 5000, maxMs: 1000 } } }, defaults));
});
test('AniList metadata keeps manga format and original story credits', () => {
const metadata = metadataFromMedia({
id: 150193, idMal: null, format: 'MANGA',
title: { english: 'Heavy Knight', romaji: 'Juukishi', native: '重騎士' }, synonyms: [],
staff: { edges: [
{ role: 'Original Story', node: { name: { full: 'Nekoko' } } },
{ role: 'Story & Art', node: { name: { full: 'Lee Brocco' } } },
{ role: 'Character Design', node: { name: { full: 'Jaian' } } },
] },
status: 'RELEASING', startDate: { year: 2022 }, genres: ['Fantasy'], description: '',
siteUrl: 'https://anilist.co/manga/150193', coverImage: { extraLarge: 'https://s4.anilist.co/cover.jpg' },
});
assert.deepEqual(metadata.authors, ['Nekoko', 'Lee Brocco']);
assert.equal(metadata['x-anilist'].format, 'MANGA');
});
test('metadata refresh preserves legacy and manual fields while updating untouched imports', () => {
const first = mergeImportedMetadata({ title: 'My preferred title', status: 'ongoing' }, {
title: 'Provider title', status: 'completed', tags: ['Fantasy'], 'x-anilist': { id: 42 },
}, 'Fallback');
assert.equal(first.title, 'My preferred title');
assert.equal(first.status, 'ongoing');
assert.deepEqual(first.tags, ['Fantasy']);
assert.equal(first['x-metadata'].imported.title, 'Provider title');
const second = mergeImportedMetadata({ ...first, tags: ['Fantasy', 'Favorite'] }, {
title: 'New provider title', status: 'ongoing', tags: ['Action'], 'x-anilist': { id: 42 },
}, 'Fallback');
assert.equal(second.title, 'My preferred title');
assert.equal(second.status, 'ongoing');
assert.deepEqual(second.tags, ['Fantasy', 'Favorite']);
const third = mergeImportedMetadata(second, { description: 'First', 'x-anilist': { id: 42 } }, 'Fallback');
assert.equal(mergeImportedMetadata(third, { description: 'Revised', 'x-anilist': { id: 42 } }, 'Fallback').description, 'Revised');
assert.throws(() => mergeImportedMetadata(first, { 'x-anilist': { id: 99 } }, 'Fallback'), /identity conflict/);
assert.throws(() => mergeImportedMetadata({ 'x-work-identity': { scheme: 'mangacircus-work-v1', kind: 'origin', value: 'a'.repeat(64) } },
{ 'x-work-identity': { scheme: 'mangacircus-work-v1', kind: 'origin', value: 'b'.repeat(64) } }, 'Fallback'), /work identity conflict/);
});
test('publisher rejects a conflicting work ID before fetching any pages', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-identity-'));
t.after(() => fs.rm(root, { recursive: true, force: true }));
const manga = path.join(root, 'Existing');
await fs.mkdir(manga);
await fs.writeFile(path.join(manga, 'manga.json'), JSON.stringify({ title: 'Existing',
sources: [{ name: 'Host', url: 'https://example.test/series' }], 'x-anilist': { id: 42 } }));
let requested = 0;
await assert.rejects(publishManifest({ version: 1,
series: { title: 'Existing', direction: 'rtl', metadata: { 'x-anilist': { id: 99 } },
sources: [{ name: 'Host', url: 'https://example.test/series' }] },
chapters: [{ number: '1', source: { name: 'Host', url: 'https://example.test/chapter' },
pages: [{ url: 'https://example.test/page.png' }] }],
}, { libraryRoot: root, httpClient: { async get() { requested++; throw new Error('unexpected request'); } } }), /identity conflict/);
assert.equal(requested, 0);
assert.equal((await fs.readdir(manga)).includes('v01'), false);
});
test('publisher explains an unresolved same-title folder collision without touching its contents', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-title-collision-'));
t.after(() => fs.rm(root, { recursive: true, force: true }));
const manga = path.join(root, 'Same Title');
await fs.mkdir(manga);
await fs.writeFile(path.join(manga, 'manga.json'), JSON.stringify({ title: 'Same Title' }));
await fs.writeFile(path.join(manga, 'operator-note.txt'), 'keep');
let requested = 0;
await assert.rejects(publishManifest({ version: 1,
series: { title: 'Same Title', direction: 'rtl', sources: [{ name: 'Peer', url: 'peer://p123/m123' }] },
chapters: [{ number: '1', source: { name: 'Peer', url: 'peer://p123/m123' }, pages: [{ url: 'https://example.test/page.png' }] }],
}, { libraryRoot: root, httpClient: { async get() { requested++; throw new Error('unexpected request'); } } }),
/could not be safely matched.*No files were overwritten.*Choose the existing manga/s);
assert.equal(requested, 0);
assert.equal(await fs.readFile(path.join(manga, 'operator-note.txt'), 'utf8'), 'keep');
});
test('queue scopes select exact catalog slices', () => {
const chapters = [
{ id: 'a', number: '1', volume: '1' }, { id: 'b', number: '2', volume: '1' },
{ id: 'c', number: '3', volume: '2' }, { id: 'd', number: '4', volume: '2' },
];
assert.deepEqual(selectByScope(chapters, { mode: 'latest' }).map(x => x.id), ['d']);
assert.deepEqual(selectByScope(chapters, { mode: 'last', count: 2 }).map(x => x.id), ['c', 'd']);
assert.deepEqual(selectByScope(chapters, { mode: 'first', count: 2 }).map(x => x.id), ['a', 'b']);
assert.deepEqual(selectByScope(chapters, { mode: 'chapter', chapter: '003.0' }).map(x => x.id), ['c']);
assert.deepEqual(selectByScope(chapters, { mode: 'after', chapter: '2' }).map(x => x.id), ['c', 'd']);
assert.deepEqual(selectByScope(chapters, { mode: 'volume', volume: '2' }).map(x => x.id), ['c', 'd']);
assert.deepEqual(selectByScope(chapters, { mode: 'ids', ids: ['b', 'd'] }).map(x => x.id), ['b', 'd']);
assert.equal(scopeLabel(normalizeScope({ mode: 'last', count: 5 })), 'last 5 chapters');
});
test('library inventory distinguishes missing chapters from alternate source versions', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-inventory-'));
t.after(() => fs.rm(temp, { recursive: true, force: true }));
const manga = path.join(temp, 'Library Manga'), book = path.join(manga, 'v01');
await fs.mkdir(path.join(book, 'c0001 [WeebCentral]'), { recursive: true });
await fs.mkdir(path.join(book, 'c0002 [MangaDex]'), { recursive: true });
await fs.writeFile(path.join(manga, 'manga.json'), JSON.stringify({
title: 'Library Manga', status: 'ongoing',
sources: [{ name: 'WeebCentral', url: 'https://weebcentral.com/series/SERIES/Test' }],
'x-scraper': { ids: { weebcentral: 'SERIES', anilist: 42 } },
}));
await fs.writeFile(path.join(book, 'c0001 [WeebCentral]', 'chapter.json'), JSON.stringify({ source: { name: 'WeebCentral', url: 'https://weebcentral.com/chapters/A' } }));
await fs.writeFile(path.join(book, 'c0002 [MangaDex]', 'chapter.json'), JSON.stringify({ source: { name: 'MangaDex · Group', url: 'https://mangadex.org/chapter/B' } }));
const inventory = await scanLibrary(temp);
const found = findManga(inventory, { source: 'weebcentral', seriesId: 'SERIES' });
assert.equal(found.id, 'Library Manga');
assert.equal(findManga(inventory, { source: 'mangadex', seriesId: 'other', anilist: 42 }).id, 'Library Manga');
assert.equal(found.autoUpdate, true);
assert.equal(shouldDownload(found, { number: '1' }, 'mangadex', 'skip-any'), false);
assert.equal(shouldDownload(found, { number: '1' }, 'mangadex', 'add-source'), true);
assert.equal(shouldDownload(found, { number: '1' }, 'weebcentral', 'add-source'), false);
assert.equal(shouldDownload(found, { number: '3' }, 'weebcentral', 'skip-any'), true);
assert.equal(isNewerThanLibrary(found, { number: '1' }), false);
assert.equal(isNewerThanLibrary(found, { number: '3' }), true);
await setAutoUpdate(temp, 'Library Manga', false);
assert.equal((await scanLibrary(temp)).mangas[0].autoUpdate, false);
});
test('a shared work ID cannot merge distinct source listings or ambiguous editions', () => {
const original = { id: 'Main', ids: { weebcentral: 'MAIN', anilist: 42 }, sources: [] };
assert.equal(findManga({ mangas: [original] }, { source: 'weebcentral', seriesId: 'PILOT', anilist: 42 }), null);
assert.equal(findManga({ mangas: [original] }, { source: 'mangadex', seriesId: 'OTHER', anilist: 42 }), original);
assert.equal(findManga({ mangas: [original, { id: 'Pilot', ids: { anilist: 42 }, sources: [] }] },
{ source: 'mangadex', seriesId: 'OTHER', anilist: 42 }), null);
});
test('recent events read the bounded log tail without losing the newest incident', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-events-'));
t.after(() => fs.rm(root, { recursive: true, force: true }));
const service = new ScraperService({ dataRoot: root });
await fs.writeFile(service.logFile, `${'x'.repeat(600 * 1024)}\n${JSON.stringify({ level: 'error', message: 'recent' })}\n`);
assert.deepEqual(await service.recentEvents(1), [{ level: 'error', message: 'recent' }]);
assert.equal((await service.recentEvents(10)).at(-1).message, 'recent');
});
test('paused legacy work is validated, published, and converted to incremental mode', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-legacy-publish-'));
const dataRoot = path.join(temp, 'data'), libraryRoot = path.join(temp, 'library');
const workRoot = path.join(libraryRoot, '.mcscraper-work-SERIES42');
const stagedManga = path.join(workRoot, 'Legacy Manga'), book = path.join(stagedManga, 'v01');
await fs.mkdir(dataRoot, { recursive: true });
await fs.mkdir(book, { recursive: true });
await fs.writeFile(path.join(stagedManga, 'manga.json'), JSON.stringify({
title: 'Legacy Manga', status: 'ongoing',
sources: [{ name: 'Alpha', url: 'https://alpha.test/title/SERIES42' }],
'x-scraper': { ids: { alpha: 'SERIES42' } },
}));
for (const number of [1, 2]) {
const chapter = path.join(book, `c000${number} [Alpha]`);
await fs.mkdir(chapter, { recursive: true });
await fs.writeFile(path.join(chapter, 'chapter.json'), JSON.stringify({ source: { name: 'Alpha', url: `https://alpha.test/chapter/${number}` } }));
await fs.writeFile(path.join(chapter, '0001.png'), Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
}
const incomplete = path.join(book, '.staging-mcscraper-abc123');
await fs.mkdir(incomplete);
await fs.writeFile(path.join(incomplete, '0001.png'), Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
await fs.writeFile(path.join(workRoot, 'job.json'), JSON.stringify({
version: 2, status: 'paused', completed: ['one', 'two'], total: 4,
}));
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
version: 3, paused: true, status: 'paused', settings: validateSettings({}), history: [], publishedRevision: 0,
queue: [{ id: 'old-id', seriesId: 'SERIES42', source: 'alpha', title: 'Legacy Manga', url: 'https://alpha.test/title/SERIES42', status: 'queued', order: 0 }],
}));
const alpha = {
id: 'alpha', label: 'Alpha',
match(value) { return String(value).includes('alpha.test/title/') ? { source: 'alpha', id: 'SERIES42', title: 'Legacy Manga', url: value } : null; },
async discover() { return { chapters: [] }; }, async run() { return null; },
};
const service = await new ScraperService({
libraryRoot, dataRoot, adapters: [alpha],
adapterFor: value => { const info = alpha.match(value); return info ? { adapter: alpha, info } : null; },
}).init();
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
const inspection = await inspectLegacyWork({ libraryRoot, seriesId: 'SERIES42' });
assert.equal(inspection.chapterCount, 2);
assert.equal(inspection.pageCount, 2);
assert.equal(inspection.incompleteStages.length, 1);
const result = await service.publishLegacyReady();
assert.equal(result.published.length, 1);
assert.equal(result.published[0].chaptersPublished, 2);
assert.equal(result.published[0].incompleteStagesQuarantined, 1);
assert.equal(await fs.stat(path.join(libraryRoot, 'Legacy Manga', 'v01', 'c0002 [Alpha]')).then(stat => stat.isDirectory()), true);
await assert.rejects(fs.stat(stagedManga));
const migrated = JSON.parse(await fs.readFile(path.join(workRoot, 'job.json'), 'utf8'));
assert.equal(migrated.publicationMode, 'incremental');
assert.equal(migrated.migratedToIncremental.chaptersPublished, 2);
assert.equal(migrated.migratedToIncremental.incompleteStagesQuarantined, 1);
assert.equal(await fs.stat(path.join(workRoot, '.legacy-incomplete', 'v01__.staging-mcscraper-abc123')).then(stat => stat.isDirectory()), true);
assert.equal(service.state.queue[0].libraryMangaId, 'Legacy Manga');
assert.equal(service.state.publishedRevision, 1);
});
test('WeebCentral fragments preserve chapter and page order', () => {
const chapters = parseChapters('<a href="/chapters/P1">Prologue 1</a><a href="/chapters/ABC">Chapter 12</a><a href="/chapters/DEF">Chapter 12.5</a><a href="/chapters/S2">S2 - Chapter 1</a><a href="/chapters/R442"><span>Rating 442</span><span>Last Read</span></a><a href="/chapters/E128"><span>Episode 128</span><span>Last Read</span></a><a href="/chapters/E78"><span>Episode: 78</span><span>Last Read</span></a><a href="/chapters/N182"><span>Night 182</span><span>Last Read</span></a><a href="/chapters/L65">Log. 65</a><a href="/chapters/A606">Act 60.6</a><a href="/chapters/C155">Scene 155</a><a href="/chapters/K162">Kapitel 162</a><a href="/chapters/F83">File 83</a><a href="/chapters/B281">Battle 281</a>', 'https://weebcentral.com/s');
assert.deepEqual(chapters.map(item => item.number), ['0.01', '12', '12.5', '1', '442', '128', '78', '182', '65', '60.6', '155', '162', '83', '281']);
assert.deepEqual(chapters[3].book, { number: 2, title: 'Season 2' });
const pages = parsePageUrls('<img src="/001.png"><img data-src="https://img.example/002.webp">', 'https://weebcentral.com/c');
assert.deepEqual(pages, ['https://weebcentral.com/001.png', 'https://img.example/002.webp']);
});
test('WeebCentral reports unrecognized chapter labels separately from an empty catalog', async () => {
const series = '<h1>Example</h1><div hx-get="/series/X/full-chapter-list"></div>';
const requests = [];
const httpFor = list => ({ get: async (url, options) => { requests.push({ url, options }); return {
text: async () => url.endsWith('full-chapter-list') ? list : series,
}; } });
const unknown = await discoverWeebCatalog('https://weebcentral.com/series/X/Example', httpFor('<a href="/chapters/ABC">Part One</a>'));
assert.equal(requests[0].options.catalog, undefined);
assert.equal(requests[1].options.catalog, true);
assert.equal(unknown.chapterLinks, 1);
assert.equal(unknown.chapters.length, 0);
assert.equal(unknown.availability.reason, 'unrecognized-labels');
assert.match(unknown.availability.message, /1 chapter links/);
const empty = await discoverWeebCatalog('https://weebcentral.com/series/X/Example', httpFor('<div>No chapters</div>'));
assert.equal(empty.chapterLinks, 0);
assert.equal(empty.availability, undefined);
});
test('completed download reinspection preserves complete state when all scoped chapters are present', async () => {
const service = new ScraperService({ libraryRoot: 'unused', dataRoot: 'unused' });
const item = { id: 'done', title: 'Done', source: 'weebcentral', url: 'https://weebcentral.com/series/EXAMPLE/Done',
seriesId: 'EXAMPLE', kind: 'inspection', status: 'queued', completed: '2026-10-01T00:00:00Z',
publishedCount: 2, selectedCount: 2, scope: { mode: 'missing' }, inspectionFromComplete: true };
service.state = { paused: false, settings: validateSettings({}), queue: [item], incidents: [], sourcePaused: {}, currentIds: {} };
service.recordCatalog = async (entry, catalog) => { entry.sourceTitle = catalog.title; entry.catalogTotal = catalog.chapters.length; };
service.libraryInventory = async () => ({ mangas: [{ id: 'Done', ids: { weebcentral: 'EXAMPLE' }, sources: [],
chapters: [{ book: '1', number: '1' }, { book: '1', number: '2' }] }] });
service.event = async () => {};
const adapter = { discover: async () => ({ title: 'Done', chapters: [{ id: 'one', number: '1' }, { id: 'two', number: '2' }] }) };
await service.runDiscovery(item, { adapter }, 'weebcentral');
assert.equal(item.status, 'complete');
assert.equal(item.kind, 'download');
assert.equal(item.completed, '2026-10-01T00:00:00Z');
assert.equal(item.publishedCount, 2);
assert.equal(item.inspectionFromComplete, undefined);
});
test('publisher keeps season chapter numbers in separate books', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-books-'));
t.after(() => fs.rm(temp, { recursive: true, force: true }));
const page = path.join(temp, 'page.png');
await fs.writeFile(page, Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
const base = { title: '', source: { name: 'Host', url: 'https://example.test/chapter' }, pages: [{ path: page }] };
await publishManifest({
version: 1,
series: { title: 'Season Test', direction: 'ttb', sources: [{ name: 'Host', url: 'https://example.test/series' }] },
chapters: [
{ ...base, number: '1', book: { number: 1, title: 'Season 1' } },
{ ...base, number: '1', book: { number: 2, title: 'Season 2' } },
],
}, { libraryRoot: path.join(temp, 'library') });
assert.equal(await fs.readFile(path.join(temp, 'library', 'Season Test', 'v01 Season 1', 'book.json'), 'utf8').then(JSON.parse).then(x => x.title), 'Season 1');
assert.equal(await fs.readFile(path.join(temp, 'library', 'Season Test', 'v02 Season 2', 'book.json'), 'utf8').then(JSON.parse).then(x => x.title), 'Season 2');
});
test('publisher stages and atomically exposes a complete new manga', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-test-'));
t.after(() => fs.rm(temp, { recursive: true, force: true }));
const page = path.join(temp, 'page.png');
await fs.writeFile(page, Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
const result = await publishManifest({
version: 1,
series: { title: 'Test Manga', direction: 'rtl', metadata: { status: 'ongoing' },
sources: [{ name: 'Test', url: 'https://example.test/manga' }] },
chapters: [{ number: '1', title: 'Start', source: { name: 'Test', url: 'https://example.test/chapter/1' }, pages: [{ path: page }] }],
}, { libraryRoot: path.join(temp, 'library') });
assert.equal(result.results[0].status, 'published');
const chapter = path.join(temp, 'library', 'Test Manga', 'v01', 'c0001 Start [Test]');
assert.deepEqual((await fs.readdir(chapter)).sort(), ['0001.png', 'chapter.json']);
const meta = JSON.parse(await fs.readFile(path.join(temp, 'library', 'Test Manga', 'manga.json'), 'utf8'));
assert.equal(meta['x-scraper'].autoUpdate, undefined, 'ongoing is a derived default, not a frozen override');
assert.equal((await scanLibrary(path.join(temp, 'library'))).mangas[0].autoUpdate, true);
assert.equal((await fs.readdir(path.join(temp, 'library'))).some(name => name.startsWith('.staging-')), false);
});
test('publisher installs a validated AniList series cover', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-cover-'));
t.after(() => fs.rm(temp, { recursive: true, force: true }));
const page = path.join(temp, 'page.png');
const png = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]);
await fs.writeFile(page, png);
let requests = 0;
const httpClient = {
async get(url) {
requests++;
assert.equal(url, 'https://s4.anilist.co/example.png');
return { arrayBuffer: async () => png };
},
};
const series = {
title: 'Covered Manga', direction: 'rtl',
sources: [{ name: 'Test', url: 'https://example.test/covered' }],
coverUrl: 'https://s4.anilist.co/example.png',
};
const result = await publishManifest({
version: 1, series,
chapters: [{ number: '1', source: { name: 'Test', url: 'https://example.test/chapter/1' }, pages: [{ path: page }] }],
}, { libraryRoot: path.join(temp, 'library'), httpClient });
assert.equal(result.cover.status, 'installed');
assert.deepEqual(await fs.readFile(path.join(temp, 'library', 'Covered Manga', 'cover.png')), png);
assert.equal(requests, 1);
});
test('publisher retains validated partial pages and resumes without duplicate requests', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-resume-'));
t.after(() => fs.rm(temp, { recursive: true, force: true }));
const png = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]);
const manifest = {
version: 1, series: { title: 'Resumable', direction: 'rtl', sources: [{ name: 'Test', url: 'https://example.test/series' }] },
chapters: [{ number: '1', source: { name: 'Test', url: 'https://example.test/chapter/1' },
pages: [1, 2].map(n => ({ url: `https://example.test/${n}.png` })) }],
};
const requests = [];
const httpClient = { async get(url) { requests.push(url); return { arrayBuffer: async () => png }; } };
const libraryRoot = path.join(temp, 'library');
await assert.rejects(publishManifest(manifest, { libraryRoot, httpClient,
onPage() { throw new Error('cooperative pause'); } }), /cooperative pause/);
await assert.rejects(fs.stat(path.join(libraryRoot, 'Resumable')));
const hidden = (await fs.readdir(libraryRoot))[0];
assert.match(hidden, /^\.staging-mcscraper-resume-/);
const { cleanupOwnTemps } = require('../lib/publisher');
await cleanupOwnTemps(libraryRoot, -1);
assert.ok(await fs.stat(path.join(libraryRoot, hidden)));
await publishManifest(manifest, { libraryRoot, httpClient });
assert.deepEqual(requests, ['https://example.test/1.png', 'https://example.test/2.png']);
const chapter = path.join(libraryRoot, 'Resumable', 'v01', 'c0001 [Test]');
assert.deepEqual((await fs.readdir(chapter)).sort(), ['0001.png', '0002.png', 'chapter.json']);
});
test('adapter registry recognizes supported series URLs', () => {
const found = adapterFor('https://weebcentral.com/series/01ABCDEF/My-Series');
assert.equal(found.adapter.id, 'weebcentral');
assert.equal(found.info.id, '01ABCDEF');
assert.equal(found.info.title, 'My Series');
const dex = adapterFor('https://mangadex.org/title/5bdb093f-9707-4a30-b02e-c8b66554ecf0/biomega');
assert.equal(dex.adapter.id, 'mangadex');
assert.equal(dex.info.id, '5bdb093f-9707-4a30-b02e-c8b66554ecf0');
assert.equal(adapterFor('https://example.test/series/1'), null);
});
test('MangaDex releases are normalized, deduplicated, credited, and host-validated', () => {
assert.equal(chapterNumber('0012.50'), '12.5');
assert.equal(chapterNumber('Special'), null);
const release = (id, chapter, group, publishAt) => ({
id, attributes: { chapter, title: '', pages: 2, publishAt, externalUrl: null },
relationships: group ? [{ id: `group-${id}`, type: 'scanlation_group', attributes: { name: group } }] : [],
});
const selected = selectChapterReleases([
release('fan', '1', 'Fan Group', '2020-01-01T00:00:00Z'),
release('official', '1.0', 'Publisher Official', '2021-01-01T00:00:00Z'),
release('two', '2', 'Fan Group', '2020-01-02T00:00:00Z'),
]);
assert.deepEqual(selected.map(item => item.id), ['official', 'two']);
assert.equal(selected[0].group.name, 'Publisher Official');
const pages = pageUrls({
baseUrl: 'https://node.mangadex.network',
chapter: { hash: 'abc', data: ['1.jpg', '2.png'] },
}, 'https://mangadex.org/chapter/test', 2);
assert.deepEqual(pages.map(item => item.url), [
'https://node.mangadex.network/data/abc/1.jpg',
'https://node.mangadex.network/data/abc/2.png',
]);
assert.throws(() => pageUrls({ baseUrl: 'https://evil.example', chapter: { hash: 'abc', data: ['1.jpg'] } }, 'https://mangadex.org/chapter/test', 1));
});
test('daemon queue state is durable while paused', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-service-'));
const dataRoot = path.join(temp, 'data');
await fs.mkdir(dataRoot, { recursive: true });
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
version: 1, paused: true, status: 'paused', settings: validateSettings({}), queue: [], history: [], publishedRevision: 0,
}));
const service = await new ScraperService({ libraryRoot: path.join(temp, 'library'), dataRoot }).init();
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
const item = await service.enqueue({ url: 'https://weebcentral.com/series/01ABCDEF/My-Series' });
assert.equal(item.status, 'queued');
const scoped = await service.enqueue({ url: 'https://weebcentral.com/series/01ABCDEF/My-Series', scope: { mode: 'latest' } });
assert.notEqual(item.id, scoped.id);
assert.equal(item.seriesId, scoped.seriesId);
assert.equal((await service.publicState()).queue.length, 2);
await service.updateSettings({ minMs: 1500, maxMs: 4500 });
assert.equal(JSON.parse(await fs.readFile(path.join(dataRoot, 'queue.json'), 'utf8')).settings.minMs, 1500);
await service.remove(item.id);
await service.remove(scoped.id);
assert.equal((await service.publicState()).queue.length, 0);
});
test('retry recovers stale downloading entries before work resumes', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-retry-'));
const dataRoot = path.join(temp, 'data');
await fs.mkdir(dataRoot, { recursive: true });
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
version: 1, paused: true, status: 'error', currentId: 'FAILED', settings: validateSettings({}),
queue: [
{ id: 'STALE', title: 'Stale', url: 'https://weebcentral.com/series/STALE/Stale', status: 'downloading', order: 0 },
{ id: 'FAILED', title: 'Failed', url: 'https://weebcentral.com/series/FAILED/Failed', status: 'error', order: 1 },
], history: [], publishedRevision: 0,
}));
const service = await new ScraperService({ libraryRoot: path.join(temp, 'library'), dataRoot }).init();
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
await service.control('retry', 'FAILED');
assert.deepEqual(service.state.queue.map(item => item.status), ['queued', 'queued']);
assert.equal(service.state.currentId, null);
assert.equal(service.state.paused, false);
});
test('nightly discovery queues one preferred supported source for ongoing manga', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-discovery-'));
const dataRoot = path.join(temp, 'data'), libraryRoot = path.join(temp, 'library');
const manga = path.join(libraryRoot, 'Ongoing Manga');
await fs.mkdir(manga, { recursive: true });
await fs.mkdir(dataRoot, { recursive: true });
await fs.writeFile(path.join(manga, 'manga.json'), JSON.stringify({
title: 'Ongoing Manga', status: 'ongoing',
sources: [{ name: 'Alpha', url: 'https://alpha.test/title/42' }],
'x-scraper': { ids: { alpha: '42' } },
}));
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
version: 3, paused: true, status: 'paused', settings: validateSettings({}), queue: [], history: [], publishedRevision: 0,
}));
const alpha = {
id: 'alpha', label: 'Alpha',
match(value) { const url = new URL(value); return url.hostname === 'alpha.test' ? { source: 'alpha', id: url.pathname.split('/').pop(), title: 'Alpha', url: value } : null; },
async discover() { return { chapters: [] }; }, async run() { return null; },
};
const resolve = value => { const info = alpha.match(value); return info ? { adapter: alpha, info } : null; };
const service = await new ScraperService({ libraryRoot, dataRoot, adapterFor: resolve, adapters: [alpha] }).init();
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
const first = await service.enqueueDiscoveryChecks('test');
const second = await service.enqueueDiscoveryChecks('test');
assert.deepEqual(first, { queued: 1, skipped: 0 });
assert.deepEqual(second, { queued: 0, skipped: 1 });
assert.equal(service.state.queue[0].kind, 'discovery');
assert.equal(service.state.queue[0].libraryMangaId, 'Ongoing Manga');
});
test('discovery promotes only chapters newer than the library frontier', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-discovery-run-'));
const dataRoot = path.join(temp, 'data'), libraryRoot = path.join(temp, 'library');
const chapter = path.join(libraryRoot, 'Ongoing Manga', 'v01', 'c0001 [Alpha]');
const laterChapter = path.join(libraryRoot, 'Ongoing Manga', 'v01', 'c0003 [Alpha]');
await fs.mkdir(chapter, { recursive: true });
await fs.mkdir(laterChapter, { recursive: true });
await fs.mkdir(dataRoot, { recursive: true });
await fs.writeFile(path.join(libraryRoot, 'Ongoing Manga', 'manga.json'), JSON.stringify({
title: 'Ongoing Manga', status: 'ongoing', sources: [{ name: 'Alpha', url: 'https://alpha.test/title/42' }],
'x-scraper': { ids: { alpha: '42' } },
}));
await fs.writeFile(path.join(chapter, 'chapter.json'), JSON.stringify({ source: { name: 'Alpha', url: 'https://alpha.test/chapter/one' } }));
await fs.writeFile(path.join(laterChapter, 'chapter.json'), JSON.stringify({ source: { name: 'Alpha', url: 'https://alpha.test/chapter/three' } }));
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
version: 3, paused: true, status: 'paused', settings: validateSettings({}), queue: [], history: [], publishedRevision: 0,
}));
let downloadedScope = null;
const alpha = {
id: 'alpha', label: 'Alpha',
match(value) { const url = new URL(value); return url.hostname === 'alpha.test' ? { source: 'alpha', id: '42', title: 'Alpha', url: value } : null; },
async discover() { return { chapters: [{ id: 'one', number: '1' }, { id: 'two', number: '2' }, { id: 'three', number: '3' }, { id: 'four', number: '4' }] }; },
async run(item, options) { downloadedScope = item.scope; await options.onPublished({ chapter: '4' }); return { finalPath: path.join(libraryRoot, 'Ongoing Manga'), publishedCount: 1, selectedCount: 1 }; },
};
const resolve = value => { const info = alpha.match(value); return info ? { adapter: alpha, info } : null; };
const service = await new ScraperService({ libraryRoot, dataRoot, adapterFor: resolve, adapters: [alpha] }).init();
const waitFor = async predicate => {
const deadline = Date.now() + 3000;
while (!predicate()) { if (Date.now() > deadline) throw new Error('Timed out waiting for discovery'); await new Promise(resolveWait => setTimeout(resolveWait, 20)); }
};
try {
await service.enqueueDiscoveryChecks('test');
await service.control('resume');
await waitFor(() => service.state.queue[0]?.status === 'complete');
assert.deepEqual(downloadedScope, { mode: 'ids', ids: ['four'] });
assert.equal(service.state.queue[0].publishedCount, 1);
const summary = (await service.publicState()).updateChecks.find(item => item.mangaId === 'Ongoing Manga' && item.source === 'alpha');
assert.ok(summary.lastCheckedAt && summary.lastNewContentAt && summary.lastDownloadedAt);
assert.equal(summary.lastResult, 'new-content');
} finally {
await service.close();
await fs.rm(temp, { recursive: true, force: true });
}
});
test('source-aware scheduler runs one item per source in parallel', async t => {
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-sources-'));
const dataRoot = path.join(temp, 'data');
await fs.mkdir(dataRoot, { recursive: true });
const queue = [
{ id: 'A1', source: 'alpha', title: 'Alpha One', url: 'https://alpha.test/1', status: 'queued', order: 0 },
{ id: 'A2', source: 'alpha', title: 'Alpha Two', url: 'https://alpha.test/2', status: 'queued', order: 1 },
{ id: 'B1', source: 'beta', title: 'Beta One', url: 'https://beta.test/1', status: 'queued', order: 2 },
];
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
version: 2, paused: true, status: 'paused', settings: validateSettings({}), queue, history: [], publishedRevision: 0,
}));
const resolvers = new Map(), started = [];
const fakeAdapter = id => ({
id, label: id,
match(value) { const url = new URL(value); return url.hostname === `${id}.test` ? { source: id, id: url.pathname.slice(1), title: id, url: value } : null; },
run(item) { started.push(item.id); return new Promise(resolve => resolvers.set(item.id, () => resolve(path.join(temp, item.id)))); },
});
const fakeAdapters = [fakeAdapter('alpha'), fakeAdapter('beta')];
const resolveAdapter = value => {
for (const adapter of fakeAdapters) { const info = adapter.match(value); if (info) return { adapter, info }; }
return null;
};
const service = await new ScraperService({ libraryRoot: path.join(temp, 'library'), dataRoot, adapterFor: resolveAdapter, adapters: fakeAdapters }).init();
const waitFor = async predicate => {
const deadline = Date.now() + 3000;
while (!predicate()) { if (Date.now() > deadline) throw new Error('Timed out waiting for scheduler'); await new Promise(resolve => setTimeout(resolve, 20)); }
};
try {
await service.control('resume');
await waitFor(() => resolvers.has('A1') && resolvers.has('B1'));
assert.deepEqual(new Set(started), new Set(['A1', 'B1']));
assert.equal(resolvers.has('A2'), false);
resolvers.get('A1')();
await waitFor(() => resolvers.has('A2'));
assert.equal(service.state.queue.find(item => item.id === 'A2').status, 'downloading');
resolvers.get('A2')(); resolvers.get('B1')();
await waitFor(() => service.state.queue.every(item => item.status === 'complete'));
assert.equal(service.state.publishedRevision, 3);
} finally {
for (const resolve of resolvers.values()) resolve();
await service.close();
await fs.rm(temp, { recursive: true, force: true });
}
});
+82
View File
@@ -0,0 +1,82 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const { SlowHttp, StopScrapeError } = require('../lib/http');
const { publicAddress, validateWeebRequest, pinnedLookup, responseByteLimit } = require('../plugins/broker');
test('only explicit HTML catalogs get the larger bounded response allowance', async () => {
const html = { headers: { accept: 'text/html' } };
assert.equal(responseByteLimit(html), 2 * 1024 * 1024);
assert.equal(responseByteLimit({ ...html, catalog: true }), 16 * 1024 * 1024);
assert.equal(responseByteLimit({ headers: { accept: 'image/jpeg' }, catalog: true }), 50 * 1024 * 1024);
const calls = [];
const http = new SlowHttp({ minMs: 0, maxMs: 0, transport: async (_url, options) => {
calls.push(options.catalog); return { status: 200, ok: true };
} });
await http.get('https://weebcentral.com/series/X', { accept: 'text/html' });
await http.get('https://weebcentral.com/series/X/full-chapter-list', { accept: 'text/html', catalog: true });
assert.deepEqual(calls, [false, true]);
});
test('plugin broker rejects local, private, non-TLS, and cross-source URLs', () => {
for (const address of ['127.0.0.1', '10.0.0.5', '192.168.1.1', '169.254.169.254', '::1', '::ffff:127.0.0.1']) {
assert.equal(publicAddress(address), false, address);
}
assert.equal(publicAddress('1.1.1.1'), true);
assert.equal(publicAddress('2606:4700:4700::1111'), true);
assert.throws(() => validateWeebRequest('http://weebcentral.com/series/X', { accept: 'text/html' }), /public HTTPS/);
assert.throws(() => validateWeebRequest('https://localhost/secret'), /public HTTPS/);
assert.throws(() => validateWeebRequest('https://other.example/series/X', { accept: 'text/html' }), /source host/);
assert.doesNotThrow(() => validateWeebRequest('https://img.example/page.jpg'));
});
test('pinned DNS lookup supports both Node single-address and all-address requests', () => {
const lookup = pinnedLookup({ address: '1.1.1.1', family: 4 });
lookup('example.com', { all: false }, (error, address, family) => {
assert.equal(error, null);
assert.equal(address, '1.1.1.1');
assert.equal(family, 4);
});
lookup('example.com', { all: true }, (error, addresses) => {
assert.equal(error, null);
assert.deepEqual(addresses, [{ address: '1.1.1.1', family: 4 }]);
});
});
test('redirect hops are validated and paced before another request', async () => {
const calls = [];
const http = new SlowHttp({
minMs: 0, maxMs: 0, maxRetries: 0, validateUrl: validateWeebRequest,
transport: async url => {
calls.push(url);
return calls.length === 1
? { status: 302, headers: { get: () => 'https://localhost/private' } }
: { status: 200, ok: true };
},
});
await assert.rejects(http.get('https://weebcentral.com/series/X', { accept: 'text/html' }), /public HTTPS/);
assert.deepEqual(calls, ['https://weebcentral.com/series/X']);
});
test('a guarded 403 stops immediately without redirect or retry', async () => {
let calls = 0;
const http = new SlowHttp({
minMs: 0, maxMs: 0, maxRetries: 2, validateUrl: validateWeebRequest,
transport: async () => { calls++; return { status: 403 }; },
});
await assert.rejects(http.get('https://weebcentral.com/series/X', { accept: 'text/html' }), StopScrapeError);
assert.equal(calls, 1);
});
test('separate jobs on one source retain the configured request gap', async () => {
const gateKey = `fixture-${Date.now()}-${Math.random()}`;
const transport = async () => ({ status: 200, ok: true });
const first = new SlowHttp({ gateKey, minMs: 40, maxMs: 40, transport });
await first.get('https://example.org/one');
const second = new SlowHttp({ gateKey, minMs: 40, maxMs: 40, transport });
assert.equal(second.gate, first.gate);
const started = Date.now();
await second.get('https://example.org/two');
assert.ok(Date.now() - started >= 30, 'a new job must not reset its source delay');
});
+50
View File
@@ -0,0 +1,50 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs/promises');
const os = require('node:os');
const path = require('node:path');
const { recordSource, recordFailure, ledgerRows } = require('../lib/source-ledger');
const { ScraperService } = require('../service');
test('a URL lead, a parsed catalog, and rights remain separate states', () => {
const ledger = {};
const item = { id: 'job', source: 'weebcentral', seriesId: 'series', url: 'https://weebcentral.com/series/series', title: 'Example' };
assert.equal(recordSource(ledger, item).catalog, 'unverified');
const row = recordSource(ledger, item, { catalogTotal: 0, availability: { reason: 'unrecognized-labels', coverage: 'unknown' },
observedAt: '2026-10-13T12:00:00Z' });
assert.equal(row.catalog, 'found');
assert.equal(row.adapter, 'unsupported-labels');
assert.equal(row.rights, 'unverified');
assert.equal(row.hostedCount, 0);
recordFailure(ledger, item, { kind: 'adapter', message: 'Parser could not read labels' });
assert.equal(ledger['weebcentral:series'].lastFailure.kind, 'adapter');
assert.equal(ledgerRows(ledger).length, 1);
});
test('a catalog observation survives completed queue cleanup and service restart', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-source-ledger-'));
const dataRoot = path.join(root, 'data'), libraryRoot = path.join(root, 'library');
await fs.mkdir(dataRoot);
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({ paused: true, queue: [],
settings: { autoUpdateEnabled: false } }));
const adapter = { id: 'fixture', label: 'Fixture', match: url => ({ source: 'fixture', id: url.split('/').at(-1), url, title: 'Example' }) };
const options = { dataRoot, libraryRoot, adapters: [adapter], adapterFor: url => ({ adapter, info: adapter.match(url) }) };
let service = await new ScraperService(options).init();
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
const item = await service.enqueue({ url: 'https://example.test/series', inspect: true });
assert.equal(service.state.sourceLedger['fixture:series'].catalog, 'unverified');
await service.recordCatalog(item, { title: 'Canonical Example', chapters: [{ id: 'one' }],
availability: { language: 'en', coverage: 'unknown' } });
item.kind = 'download'; item.status = 'complete';
await service.clearCompleted();
await service.close();
service = await new ScraperService(options).init();
const [row] = (await service.publicState()).sourceLedger;
assert.equal(service.state.queue.length, 0);
assert.equal(row.sourceTitle, 'Canonical Example');
assert.equal(row.adapter, 'compatible');
assert.equal(row.hostedCount, 1);
assert.equal(row.rights, 'unverified');
});
@@ -0,0 +1,40 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const { discoverMangaDexCatalog, discoverMangaDexChapter } = require('../adapters/mangadex');
const { parse } = require('../plugins/registry');
const mangaId = '123e4567-e89b-42d3-a456-426614174000';
const chapterIds = [
'123e4567-e89b-42d3-a456-426614174001',
'123e4567-e89b-42d3-a456-426614174002',
];
const seriesUrl = `https://mangadex.org/title/${mangaId}/Fixture`;
const manga = { data: { type: 'manga', attributes: {
title: { en: 'Fixture' }, originalLanguage: 'ja', lastChapter: '2',
availableTranslatedLanguages: ['en'], links: {}, tags: [],
} } };
const feed = { total: 2, data: chapterIds.map((id, index) => ({
id, attributes: { chapter: String(index + 1), title: `Part ${index + 1}`, pages: 2,
publishAt: `2026-01-0${index + 1}T00:00:00Z` }, relationships: [],
})) };
function fakeHttp() {
return { async get(url) {
const body = url.includes('/at-home/server/') ? {
baseUrl: 'https://uploads.mangadex.org', chapter: { hash: 'HASH', data: ['1.jpg', '2.jpg'] },
} : url.includes('/feed?') ? feed : manga;
return { json: async () => body };
} };
}
test('MangaDex parses a catalog and pages in the scraper process', async () => {
const catalog = await discoverMangaDexCatalog(seriesUrl, fakeHttp());
const chapter = await discoverMangaDexChapter(catalog.chapters[0], fakeHttp());
assert.equal(catalog.title, 'Fixture');
assert.deepEqual(catalog.chapters.map(ref => ref.number), ['1', '2']);
assert.equal(chapter.pages.length, 2);
assert.match(chapter.pages[0].url, /^https:\/\/uploads\.mangadex\.org\/data\//);
assert.throws(() => parse('mangadex', 'metadata', { data: manga.data, url: 'https://other.example/manga/x' }), /does not belong/);
});
+42
View File
@@ -0,0 +1,42 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs/promises');
const os = require('node:os');
const path = require('node:path');
const { ScraperService } = require('../service');
test('an operator can pause one source while other lanes continue', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-source-pause-'));
const service = await new ScraperService({ libraryRoot: path.join(root, 'library'),
dataRoot: path.join(root, 'data') }).init();
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
await assert.rejects(service.control('pause-source', 'unknown'), /Unknown source/);
await service.control('pause-source', 'weebcentral');
assert.equal(service.state.sourcePaused.weebcentral, true);
await service.control('resume');
assert.equal(service.state.paused, false);
assert.equal(service.state.sourcePaused.weebcentral, true);
service.state.queue.push({ source: 'weebcentral', status: 'error', failure: { kind: 'access' } });
await assert.rejects(service.control('resume-source', 'weebcentral'), /access block/);
service.state.queue.pop();
await service.control('resume-source', 'weebcentral');
assert.equal(service.state.sourcePaused.weebcentral, undefined);
});
test('confirmation can queue an inspected job without unpausing its source', async t => {
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-source-confirm-'));
const service = await new ScraperService({ libraryRoot: path.join(root, 'library'),
dataRoot: path.join(root, 'data') }).init();
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
const item = { id: 'inspected', source: 'weebcentral', title: 'Sample', sourceTitle: 'Sample',
kind: 'inspection', status: 'ready' };
service.state.queue.push(item);
await service.control('pause-source', 'weebcentral');
await service.control('confirm', item.id);
assert.equal(item.kind, 'download');
assert.equal(item.status, 'queued');
assert.equal(service.state.sourcePaused.weebcentral, true);
await assert.rejects(service.control('inspect', item.id), /source access block/);
});
+61
View File
@@ -0,0 +1,61 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const { parse } = require('../plugins/registry');
const { validate } = require('../plugins/validation');
const { discoverWeebCatalog, discoverWeebChapter } = require('../adapters/weebcentral');
const { adapters, adapterFor } = require('../adapters');
const seriesUrl = 'https://weebcentral.com/series/ABC/title';
const chapterUrl = 'https://weebcentral.com/chapters/XYZ';
const seriesHtml = '<h1>Test Manga</h1><div hx-get="/series/ABC/full-chapter-list"></div>';
const listHtml = '<a href="/chapters/XYZ">Chapter 1</a><a href="/chapters/LMN">Log. 2</a>';
const chapterHtml = '<time datetime="2026-01-01"></time>';
const imagesHtml = '<img src="https://img.example/1.jpg"><img data-src="https://img.example/2.jpg">';
test('first-party registry rejects unknown plugins and malformed input', () => {
assert.throws(() => parse('remote-module', 'catalog', { html: listHtml, url: seriesUrl }), /Unknown/);
assert.throws(() => parse('weebcentral', 'catalog', { html: listHtml, url: 'file:///tmp/x' }), /Invalid source URL/);
assert.throws(() => parse('weebcentral', 'catalog', { html: listHtml, url: 'https://other.example/series/a' }), /does not belong/);
assert.equal(parse('weebcentral', 'catalog', { html: listHtml, url: seriesUrl }).chapters.length, 2);
});
test('parser output cannot turn catalog and image links into local or unrelated requests', () => {
assert.throws(() => validate('weebcentral', 'series', { title: 'X', listUrl: 'http://127.0.0.1:8091/api/state' }), /unexpected/);
assert.throws(() => validate('weebcentral', 'catalog', { chapterLinks: 1, chapters: [{ id: 'X', number: '1', url: 'https://other.example/chapters/X' }] }), /unexpected/);
assert.throws(() => validate('weebcentral', 'chapter', { manifestUrl: 'https://weebcentral.com/chapters/X/not-images' }), /manifest/);
assert.throws(() => validate('weebcentral', 'pages', { pages: ['http://127.0.0.1:8080/private'] }), /unsafe/);
assert.throws(() => validate('weebcentral', 'pages', { pages: ['https://localhost/private'] }), /unsafe/);
assert.doesNotThrow(() => validate('weebcentral', 'pages', { pages: ['https://img.example/1.jpg'] }));
});
test('in-process parser preserves WeebCentral catalog and page discovery', async () => {
const requested = [];
const http = { async get(url) {
requested.push(url);
const body = url === seriesUrl ? seriesHtml
: url.endsWith('/full-chapter-list') ? listHtml
: url === chapterUrl ? chapterHtml
: imagesHtml;
return { text: async () => body };
} };
const catalog = await discoverWeebCatalog(seriesUrl, http);
assert.equal(catalog.title, 'Test Manga');
assert.deepEqual(catalog.chapters.map(chapter => chapter.number), ['1', '2']);
const chapter = await discoverWeebChapter(catalog.chapters[0], http);
assert.equal(chapter.pages.length, 2);
assert.equal(chapter.added, '2026-01-01');
assert.equal(requested.length, 4);
});
test('a new source is an explicit adapter, not another container or remote module', () => {
assert.deepEqual(adapters.map(adapter => adapter.id), ['weebcentral', 'mangadex', 'mangapill']);
for (const adapter of adapters) {
assert.equal(typeof adapter.match, 'function');
assert.equal(typeof adapter.discover, 'function');
assert.equal(typeof adapter.run, 'function');
}
assert.equal(adapterFor(seriesUrl)?.adapter.id, 'weebcentral');
assert.equal(adapterFor('https://unknown.example/series/ABC'), null);
});
+69
View File
@@ -0,0 +1,69 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs/promises');
const os = require('node:os');
const path = require('node:path');
const { parseChapters, discoverWeebCatalog, discoverWeebChapter } = require('../adapters/weebcentral');
const { parseSpecialFolder } = require('../../chapter-identity');
const { selectByScope } = require('../lib/scope');
const { publishManifest } = require('../lib/publisher');
const { scanLibrary, shouldDownload } = require('../lib/library-inventory');
const reader = require('../../library');
const { Canvas } = require('../../tools/png');
const ids = ['01JHVPNS8PNMSNF4H18ERX3SMK', '01JHVPNS8PNSDS66BEPVHH536F'];
const seriesUrl = 'https://weebcentral.com/series/01J76XY8CF6MA6W5VJWZDNWESZ/Akira';
test('whole-volume releases keep their source IDs and do not masquerade as chapters', async t => {
const html = `<a href="/chapters/${ids[0]}">Volume 1</a><a href="/chapters/${ids[1]}">Volume 2</a>`;
const refs = parseChapters(html, seriesUrl);
assert.equal(refs.length, 2);
assert.deepEqual(refs.map(ref => ref.volume), ['1', '2']);
assert.ok(refs.every(ref => ref.special && ref.releaseKind === 'volume'));
assert.equal(parseSpecialFolder(refs[0].number + ' [WeebCentral]').releaseKind, 'volume');
assert.equal(selectByScope(refs, { mode: 'volume', volume: '2' })[0].id, ids[1]);
assert.equal(selectByScope(refs, { mode: 'chapter', chapter: '1' }).length, 0);
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-volume-'));
t.after(() => fs.rm(root, { recursive: true, force: true }));
const page = path.join(root, '.fixture.png');
await fs.writeFile(page, new Canvas(10, 20, [1, 2, 3]).png());
const manifest = { version: 1, series: { title: 'Volume fixture', direction: 'rtl', sources: [{ name: 'WeebCentral', url: seriesUrl }] },
chapters: [refs[0], { number: '1', source: { name: 'Other', url: 'https://example.org/1' } }].map(ref => ({
...ref, source: ref.source || { name: 'WeebCentral', url: ref.url }, pages: [{ path: page }],
})) };
await publishManifest(manifest, { libraryRoot: root });
const scanned = (await reader.scanLibrary(root)).index.mangas[0];
assert.equal(scanned.chapters.length, 2);
assert.ok(scanned.chapters.some(ch => ch.label === 'Volume 1'));
assert.ok(scanned.chapters.some(ch => ch.label === 'Ch. 1'));
const manga = (await scanLibrary(root)).mangas[0];
assert.equal(shouldDownload(manga, refs[0], 'weebcentral'), false);
assert.equal(shouldDownload(manga, refs[1], 'weebcentral'), true);
assert.equal(shouldDownload(manga, refs[0], 'weebcentral', 'add-source'), false);
const numberedOnly = { ...manga, chapters: manga.chapters.filter(chapter => chapter.releaseKind !== 'volume') };
assert.equal(shouldDownload(numberedOnly, refs[0], 'weebcentral'), false);
assert.equal(shouldDownload(numberedOnly, refs[0], 'weebcentral', 'add-source'), true);
});
test('catalog ordering and chapter discovery preserve whole-volume metadata', async () => {
const html = `<h1>Volume fixture</h1><a href="/chapters/${ids[0]}">Volume 1</a>`
+ `<a href="/chapters/${ids[1]}">Chapter 1</a>`;
const http = { async get(url) {
const body = url === seriesUrl ? html
: url === `https://weebcentral.com/chapters/${ids[0]}` ? '<time datetime="2026-01-01"></time>'
: '<img src="https://images.example/1.jpg">';
return { async text() { return body; } };
} };
const catalog = await discoverWeebCatalog(seriesUrl, http);
assert.equal(catalog.chapters.length, 2);
assert.equal(catalog.chapters[0].number, '1');
assert.equal(catalog.chapters[1].label, 'Volume 1');
const chapter = await discoverWeebChapter(catalog.chapters[1], http);
assert.equal(chapter.releaseKind, 'volume');
assert.equal(chapter.sourceId, ids[0]);
assert.equal(chapter.sourceNumber, '1');
assert.equal(chapter.book.number, 1);
assert.equal(chapter.pages.length, 1);
});
+33
View File
@@ -0,0 +1,33 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const { normalizeWorkIdentity, resolveWorkIdentity, sameWorkIdentity } = require('../lib/work-identity');
const { findManga } = require('../lib/library-inventory');
test('work identity follows canonical source priority and preserves an imported lineage', () => {
const anilist = { scheme: 'mangacircus-work-v1', kind: 'anilist', value: '123' };
const meta = { 'x-anilist': { id: 123 }, 'x-scraper': { ids: { mangadex: 'md-1', weebcentral: 'wc-1' } } };
assert.deepEqual(resolveWorkIdentity(meta, 'folder', 'server'), anilist);
const imported = { scheme: 'mangacircus-work-v1', kind: 'origin', value: 'a'.repeat(64) };
assert.deepEqual(resolveWorkIdentity({ ...meta, 'x-work-identity': imported }, 'other', 'other-server'), imported);
assert.equal(normalizeWorkIdentity({ ...imported, value: '' }), null);
assert.equal(sameWorkIdentity(imported, { ...imported }), true);
assert.equal(sameWorkIdentity(imported, { ...imported, value: 'b'.repeat(64) }), false);
assert.equal(resolveWorkIdentity({ 'x-scraper': { ids: { 'peer-p123': 'edition-only' } } }, 'folder', 'server').kind,
'origin', 'A peer edition ID is not a canonical work identity');
});
test('library matching uses a peer-carried stable identity before title or peer-edition IDs', () => {
const localServerId = 'p-local-server';
const inventory = { mangas: [
{ id: 'Peer A Test Manga', meta: { title: 'Peer A Test Manga' }, ids: {}, sources: [], chapters: [] },
{ id: 'Unrelated edition', meta: { title: 'Peer A Test Manga' }, ids: { 'peer-p-other': 'p-other-manga' }, sources: [], chapters: [] },
] };
const identity = resolveWorkIdentity(inventory.mangas[0].meta, inventory.mangas[0].id, localServerId);
assert.equal(findManga(inventory, { source: 'peer-p-returning', seriesId: 'p-returning-manga',
workIdentity: identity, localServerId }).id, 'Peer A Test Manga');
assert.equal(findManga(inventory, { source: 'peer-p-returning', seriesId: 'p-returning-manga',
workIdentity: { ...identity, value: 'f'.repeat(64) }, localServerId }), null,
'A matching title alone must not merge unrelated work');
});
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env node
'use strict';
const fs = require('fs'), path = require('path'), http = require('http');
const { secret } = require('../runtime/config');
const { Monitor } = require('./monitor');
const root = path.resolve(process.env.SENTINEL_DATA || path.join(process.env.DATA || 'data', 'sentinel'));
fs.mkdirSync(root, { recursive: true });
const token = secret('SCRAPER_TOKEN');
const host = process.env.SENTINEL_HOST || '127.0.0.1', port = +(process.env.SENTINEL_PORT || 8092);
if (!['127.0.0.1', 'localhost', '::1'].includes(host) && !token) throw new Error('SCRAPER_TOKEN is required for non-loopback Sentinel');
const reader = process.env.READER_URL || 'http://127.0.0.1:8080';
const scraper = process.env.SCRAPER_URL || 'http://127.0.0.1:8091';
const file = path.join(root, 'state.json');
let saved = {};
if (fs.existsSync(file)) saved = JSON.parse(fs.readFileSync(file, 'utf8'));
const monitor = new Monitor({ incidents: saved.incidents || [] });
const request = async (base, endpoint, body) => {
const response = await fetch(base + endpoint, { headers: { ...(token ? { authorization: `Bearer ${token}` } : {}), 'content-type': 'application/json' },
signal: AbortSignal.timeout(8000), ...(body ? { method: 'POST', body: JSON.stringify(body) } : {}) });
if (!response.ok) throw new Error(`Service HTTP ${response.status}`);
return response.json();
};
let stopped = false, timer, lastTick = Date.now();
async function tick() {
try {
const results = await Promise.allSettled([request(reader, '/health'), request(scraper, '/api/state')]);
const state = await monitor.observe(results[0].status === 'fulfilled' ? results[0].value : null,
results[1].status === 'fulfilled' ? results[1].value : null, input => request(scraper, '/api/recovery', input));
fs.writeFileSync(file + '.tmp', JSON.stringify(state, null, 2) + '\n'); fs.renameSync(file + '.tmp', file);
lastTick = Date.now();
for (const incident of state.incidents.filter(i => i.state === 'open' && !i.logged)) {
console.log(JSON.stringify({ ...incident, targetService: incident.service, service: 'sentinel', level: 'warning' })); incident.logged = true;
}
} catch (error) { console.error('[sentinel]', error.message); }
if (!stopped) timer = setTimeout(tick, 30000);
}
const server = http.createServer((req, res) => {
const endpoint = new URL(req.url, 'http://localhost').pathname;
let code = 200, value;
if (endpoint === '/health') { code = Date.now() - lastTick < 120000 ? 200 : 503; value = { ok: code === 200, service: 'sentinel', checkedAt: monitor.state.checkedAt }; }
else if (token && req.headers.authorization !== `Bearer ${token}`) { code = 401; value = { error: 'Unauthorized' }; }
else if (endpoint === '/api/state' && req.method === 'GET') value = monitor.state;
else if (endpoint === '/metrics' && req.method === 'GET') {
res.writeHead(200, { 'content-type': 'text/plain; version=0.0.4' });
return res.end(`mangacircus_reader_up ${Number(!!monitor.state.services.reader)}\nmangacircus_scraper_up ${Number(!!monitor.state.services.scraper)}\nmangacircus_open_incidents ${monitor.incidents.filter(i => i.state === 'open').length}\n`);
} else { code = 404; value = { error: 'Not found' }; }
res.writeHead(code, { 'content-type': 'application/json', 'cache-control': 'no-store' }); res.end(JSON.stringify(value));
});
server.listen(port, host, () => console.log(JSON.stringify({ service: 'sentinel', event: 'started', port })));
tick();
for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, () => { stopped = true; clearTimeout(timer); server.close(() => process.exit(0)); });
+65
View File
@@ -0,0 +1,65 @@
'use strict';
const { progressKey } = require('../scraper/lib/failures');
// No source requests, filesystem checkpoint edits, shell execution or Docker socket.
class Monitor {
constructor(options = {}) {
this.now = options.now || Date.now;
this.stallMs = options.stallMs || 900000;
this.observations = new Map();
this.incidents = options.incidents || [];
this.state = { status: 'starting', services: {}, incidents: this.incidents };
}
incident(key, message, extra = {}) {
let incident = this.incidents.find(i => i.key === key && i.state === 'open');
if (!incident) {
incident = { key, message, state: 'open', at: new Date(this.now()).toISOString(), ...extra };
this.incidents.push(incident);
if (this.incidents.length > 200) this.incidents.splice(0, this.incidents.length - 200);
}
return incident;
}
resolve(key) {
for (const i of this.incidents) if (i.key === key && i.state === 'open') { i.state = 'resolved'; i.resolvedAt = new Date(this.now()).toISOString(); }
}
async observe(reader, scraper, recover) {
const now = this.now();
this.state.checkedAt = new Date(now).toISOString();
this.state.services = { reader: !!reader?.ok, scraper: !!scraper };
if (reader?.ready === false) this.incident('reader:scan', 'Reader is alive but its library scan failed; inspect storage and scan diagnostics', { service: 'reader' });
else if (reader?.ok) this.resolve('reader:scan');
for (const [service, healthy] of Object.entries(this.state.services)) {
if (healthy) this.resolve(`service:${service}`);
else this.incident(`service:${service}`, `${service} is unreachable or unhealthy; inspect its supervisor`, { service });
}
if (!scraper) { this.state.status = 'degraded'; return this.state; }
const ids = new Set();
for (const entry of scraper.active || []) {
const id = entry.current.id; ids.add(id);
// Without a checkpoint, a null progress key stays constant forever and
// cannot prove a stall. Do not quarantine a live worker on that basis.
if (!entry.job) { this.observations.delete(id); this.resolve(`stall:${id}`); continue; }
const key = progressKey(entry.job), prior = this.observations.get(id);
if (!prior || prior.key !== key) { this.observations.set(id, { key, at: now }); this.resolve(`stall:${id}`); continue; }
if (scraper.paused || scraper.sourcePaused?.[entry.source]) continue;
const allowance = Math.max(this.stallMs, Number(entry.job?.rate?.chapterMaxMs || 0) + 180000);
if (now - prior.at < allowance) continue;
const incident = this.incident(`stall:${id}`, 'Job progress has stopped; recovery is limited to one checkpoint-safe attempt', { jobId: id, source: entry.source, title: entry.current.title });
if (!incident.attempted && recover) {
// Persist attempted in the daemon via stopReason/stallRecoveries; monitor records audit too.
incident.attempted = true;
try { incident.recovery = await recover({ id, progressKey: key }); }
catch { incident.recovery = { changed: false, error: 'Recovery API unavailable' }; }
}
}
for (const id of this.observations.keys()) if (!ids.has(id)) { this.observations.delete(id); this.resolve(`stall:${id}`); }
const failures = (scraper.queue || []).filter(item => ['error', 'quarantined'].includes(item.status));
for (const item of failures) this.incident(`job:${item.id}`, item.failure?.message || item.error || 'Job needs attention', { jobId: item.id, source: item.source, kind: item.failure?.kind || 'unknown' });
for (const i of this.incidents) if (i.key.startsWith('job:') && !failures.some(item => i.key === `job:${item.id}`)) this.resolve(i.key);
this.state.sourcePaused = scraper.sourcePaused || {};
this.state.active = (scraper.active || []).map(e => ({ source: e.source, id: e.current.id, title: e.current.title, completed: e.job?.completed, total: e.job?.total, updated: e.job?.updated }));
this.state.status = this.incidents.some(i => i.state === 'open') ? 'attention' : scraper.paused ? 'paused' : 'healthy';
return this.state;
}
}
module.exports = { Monitor };
+38
View File
@@ -0,0 +1,38 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict');
const { Monitor } = require('../monitor');
test('Sentinel does not disturb progressing or deliberately paused jobs', async () => {
let now = 0, recoveries = 0;
const monitor = new Monitor({ now: () => now, stallMs: 100 });
const state = { paused: false, active: [{ source: 'a', current: { id: 'job', title: 'Manga' }, job: { completed: 0, current: { downloadedPages: 1 } } }], queue: [] };
const recover = async () => { recoveries++; return { changed: true }; };
await monitor.observe({ ok: true }, state, recover);
now = 500000; state.active[0].job.current.downloadedPages = 2;
await monitor.observe({ ok: true }, state, recover);
now = 1000000; state.paused = true;
await monitor.observe({ ok: true }, state, recover);
assert.equal(recoveries, 0);
state.paused = false; await monitor.observe({ ok: true }, state, recover); await monitor.observe({ ok: true }, state, recover);
assert.equal(recoveries, 1);
state.active[0].job.completed = 1; await monitor.observe({ ok: true }, state, recover);
assert.equal(monitor.incidents.find(i => i.key === 'stall:job').state, 'resolved');
});
test('Sentinel reports service failures without restart loops and resolves recovery', async () => {
const monitor = new Monitor();
await monitor.observe(null, null); await monitor.observe(null, null);
assert.equal(monitor.incidents.length, 2);
await monitor.observe({ ok: true }, { active: [], queue: [], paused: false });
assert.equal(monitor.state.status, 'healthy'); assert.ok(monitor.incidents.every(i => i.state === 'resolved'));
});
test('missing checkpoint does not trigger a false stall recovery', async () => {
let now = 0, recoveries = 0;
const monitor = new Monitor({ now: () => now, stallMs: 100 });
const state = { paused: false, active: [{ source: 'mangapill', current: { id: 'job', title: 'Manga' }, job: null }], queue: [] };
for (const advance of [0, 1000000, 2000000]) {
now = advance;
await monitor.observe({ ok: true }, state, async () => { recoveries++; return { changed: true }; });
}
assert.equal(recoveries, 0);
assert.equal(monitor.incidents.some(item => item.key === 'stall:job'), false);
});
+28
View File
@@ -0,0 +1,28 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('fs'), os = require('os'), path = require('path'), { EventEmitter } = require('events');
const { Supervisor } = require('../../runtime/supervisor');
const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
test('supervisor isolates a child crash and persists its bounded restart budget', async () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-supervisor-')), file = path.join(dir, 'runtime.json');
let starts = 0;
const spawn = () => { const child = new EventEmitter(); child.pid = ++starts; child.kill = () => { child.emit('exit', 0); child.emit('close', 0); }; return child; };
const supervisor = new Supervisor('reader', { file, spawn, retryBaseMs: 1 });
try {
supervisor.start();
for (let n = 0; n < 3; n++) {
supervisor.child.emit('close', 1, null);
for (let i = 0; i < 50 && supervisor.state.status === 'retry_wait'; i++) await delay(2);
}
assert.equal(starts, 3); assert.equal(supervisor.state.status, 'parked'); assert.equal(supervisor.child, null);
const restored = new Supervisor('reader', { file, spawn, retryBaseMs: 1 }); restored.start();
assert.equal(starts, 3); assert.equal(restored.state.status, 'parked'); await restored.close();
} finally { await supervisor.close(); fs.rmSync(dir, { recursive: true, force: true }); }
});
test('intentional supervisor shutdown does not consume the crash budget', async () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-supervisor-stop-'));
const child = new EventEmitter(); child.pid = 1; child.kill = () => { child.emit('exit', 0); child.emit('close', 0); };
const supervisor = new Supervisor('scraper', { file: path.join(dir, 'runtime.json'), spawn: () => child });
try { supervisor.start(); await supervisor.close(); assert.equal(supervisor.state.status, 'stopped'); assert.equal(supervisor.state.failures.length, 0); }
finally { await supervisor.close(); fs.rmSync(dir, { recursive: true, force: true }); }
});
+620 -77
View File
@@ -5,22 +5,43 @@
* Environment:
* PORT=8080 LIBRARY=./library DATA=./data
* SIGNUP=closed|open (the very first account can always be created and becomes admin)
* SCAN_INTERVAL=10 minutes between automatic library rescans, 0 = off
* SCAN_INTERVAL=10 minutes between shallow change checks, 0 = off
* FULL_SCAN_INTERVAL=1440 minutes between full reconciliations, 0 = off
* COOKIE_SECURE=1 mark the session cookie Secure (when served over HTTPS)
* SENPAI_URL=http://senpai-api:8780 optional story companion ("Who's here"); off when unset
* SENPAI_TOKEN=... service token for SENPAI_URL
* SCRAPER_URL=http://127.0.0.1:8091 scraper daemon API
* SCRAPER_TOKEN=... shared daemon token (optional on loopback)
*/
const http = require('http'), https = require('https'), fs = require('fs'), path = require('path'), crypto = require('crypto');
const { scanLibrary } = require('./library');
const http = require('http'), fs = require('fs'), path = require('path'), crypto = require('crypto');
const { scanLibrary, scanChangedLibrary, libraryStamps, snapshotIndex, restoreIndex } = require('./library');
const { Catalog, CatalogError, collectionViews, TYPES, ROLES } = require('./catalog');
const { RATINGS: GREATEST_RATINGS, loadCatalog: loadGreatestCatalog, findOwned: findGreatestOwned,
countsFor: greatestCounts, scoreFor: greatestScore } = require('./greatest');
const { hosterLinks } = require('./greatest/hosters');
const { secret } = require('./runtime/config');
const { PeeringState, requirePeer } = require('./peering/state');
const peerClient = require('./peering/client');
const { DiscoveryStore } = require('./discovery/store');
const { generate: generateDiscovery } = require('./discovery/engine');
const {
applyPulse, applyVisit, chapterInsight, eligiblePages, emptyRecord, libraryInsight, mangaInsight, publicSummary,
} = require('./reading-analytics');
const { parseSchedule, publicSchedule, loadQuestionBank, createReadingAccess } = require('./reading-access');
const PORT = +process.env.PORT || 8080;
const LIB = path.resolve(process.env.LIBRARY || 'library');
const DATA = path.resolve(process.env.DATA || 'data');
const OPEN_SIGNUP = process.env.SIGNUP === 'open';
const SCAN_MINUTES = +(process.env.SCAN_INTERVAL || 10);
const FULL_SCAN_MINUTES = +(process.env.FULL_SCAN_INTERVAL || 1440);
const SECURE_COOKIE = process.env.COOKIE_SECURE === '1';
const SENPAI = process.env.SENPAI_URL ? process.env.SENPAI_URL.replace(/\/?$/, '/') : '';
const SENPAI_TOKEN = process.env.SENPAI_TOKEN || '';
const SCRAPER_URL = String(process.env.SCRAPER_URL || 'http://127.0.0.1:8091').replace(/\/$/, '');
const SCRAPER_TOKEN = secret('SCRAPER_TOKEN');
const SENTINEL_URL = String(process.env.SENTINEL_URL || 'http://127.0.0.1:8092').replace(/\/$/, '');
const PEER_DATA = path.resolve(process.env.PEER_DATA || path.join(DATA, 'peering'));
const peerStore = new PeeringState(PEER_DATA);
const discoveryStore = new DiscoveryStore(DATA);
const DISCOVERY_CACHE = path.join(DATA, 'scraper', 'metadata', 'anilist');
let discoveryRun = null, discoveryError = null;
const PUB = path.join(__dirname, 'public');
const IMG = /\.(jpe?g|png|webp|gif|avif|bmp)$/i;
const MIME = {
@@ -31,8 +52,11 @@ const MIME = {
// ---------- storage: one JSON file, written atomically ----------
fs.mkdirSync(DATA, { recursive: true });
const DB_FILE = path.join(DATA, 'db.json'), CACHE_FILE = path.join(DATA, 'scan-cache.json');
const db = { users: {}, sessions: {}, progress: {}, follows: {}, notes: {}, votes: {}, known: {} };
const catalog = new Catalog(DATA);
const greatestCatalog = loadGreatestCatalog();
const greatestById = new Map(greatestCatalog.entries.map(entry => [entry.id, entry]));
const DB_FILE = path.join(DATA, 'db.json'), CACHE_FILE = path.join(DATA, 'scan-cache.json'), INDEX_FILE = path.join(DATA, 'library-index.json');
const db = { users: {}, sessions: {}, progress: {}, reading: {}, readingAccess: {}, follows: {}, notes: {}, votes: {}, greatestRatings: {}, discoveryFeedback: {}, known: {} };
// A corrupt db.json stops the server instead of silently starting empty and overwriting it.
if (fs.existsSync(DB_FILE)) Object.assign(db, JSON.parse(fs.readFileSync(DB_FILE, 'utf8')));
function writeAtomic(file, text) { fs.writeFileSync(file + '.tmp', text); fs.renameSync(file + '.tmp', file); }
@@ -40,9 +64,24 @@ let saveTimer = null;
function save() { saveTimer = saveTimer || setTimeout(flush, 500); }
function flush() { clearTimeout(saveTimer); saveTimer = null; writeAtomic(DB_FILE, JSON.stringify(db)); }
for (const sig of ['SIGINT', 'SIGTERM']) process.on(sig, () => { flush(); process.exit(0); });
db.readingAccess ||= {};
const readingSchedule = parseSchedule(process.env.READING_WINDOWS_JSON || '', process.env.READING_TIMEZONE || process.env.TZ || 'Europe/Berlin');
const readingAccess = createReadingAccess({ schedule: readingSchedule,
bank: loadQuestionBank(process.env.READING_QUIZ_CATALOG_PATH || ''),
unlockMinutes: Number(process.env.READING_QUIZ_UNLOCK_MINUTES || 60), grants: db.readingAccess, save,
getProgress: (user, mangaId) => db.progress[user]?.[mangaId] || {},
getMangaTitle: mangaId => lib.byId.get(mangaId)?.title || null });
class HttpError extends Error { constructor(code, msg) { super(msg); this.code = code; } }
function bad(code, msg) { throw new HttpError(code, msg); }
function peerForRoute(id) {
try { return requirePeer(peerStore.load(), id); }
catch { bad(404, 'Peer not found'); }
}
async function peerRemote(task) {
try { return await task(); }
catch (error) { bad(502, `Peer service: ${String(error.message || error).slice(0, 300)}`); }
}
// ---------- users & sessions ----------
const sha = s => crypto.createHash('sha256').update(s).digest('hex');
@@ -52,7 +91,8 @@ function checkPw(pw, stored) {
return want.length === got.length && crypto.timingSafeEqual(want, got);
}
const DUMMY_HASH = hashPw('x');
const DEFAULTS = { startMode: 'last', lastMode: 'page', adapt: true, skipExtras: false, skipCovers: false, zoom: {}, webhook: '' };
const DEFAULTS = { startMode: 'last', lastMode: 'page', adapt: true, skipExtras: false, skipCovers: false, flowPageGap: true, pageTone: 'original', readingClock: false, readingStory: true,
discoveryEnabled: false, discoveryReadingActivity: false, discoveryMature: false, zoom: {}, webhook: '' };
const settingsOf = u => ({ ...DEFAULTS, ...u.settings, zoom: { ...(u.settings && u.settings.zoom) } });
function createUser(name, pw, admin, replace) {
@@ -76,28 +116,67 @@ function userOf(req) { const t = tokenOf(req), s = t && db.sessions[sha(t)]; ret
for (const [k, s] of Object.entries(db.sessions)) if (Date.now() - s.t > 365 * 864e5 || !db.users[s.u]) delete db.sessions[k];
// ---------- library index ----------
let lib = { mangas: [], byId: new Map() }, scanning = null, scanState = {}, scanCache = {};
let lib = { mangas: [], byId: new Map() }, scanning = null, pendingScan = null, scanState = {}, scanCache = {}, scanStamps = {};
let lastFullAt = 0, lastFullAttemptAt = 0;
try { scanCache = JSON.parse(fs.readFileSync(CACHE_FILE, 'utf8')); } catch { /* first run */ }
try {
const saved = JSON.parse(fs.readFileSync(INDEX_FILE, 'utf8'));
if (saved.version !== 1 || saved.root !== LIB || !saved.cache || !saved.stamps) throw new Error('Invalid library index snapshot');
const restored = restoreIndex(saved.mangas);
// A disconnected/replaced library mount must not appear populated from cache.
const present = new Set(fs.readdirSync(LIB, { withFileTypes: true }).filter(e => e.isDirectory()).map(e => e.name));
if (restored.mangas.length && !restored.mangas.some(m => present.has(m.id))) throw new Error('Library mount differs from snapshot');
lib = restored; scanCache = saved.cache; scanStamps = saved.stamps;
lastFullAt = Number(saved.fullAt) || Number(saved.at) || 0;
scanState = { at: saved.at, mode: 'snapshot', stale: true, mangas: lib.mangas.length, error: null };
console.log(`[scan] restored ${lib.mangas.length} manga from saved index; verifying in background`);
} catch (e) { if (e.code !== 'ENOENT') console.warn(`[scan] ignoring saved index: ${e.message}`); }
function rescan(full) {
scanning = scanning || (async () => {
async function performScan(mode) {
const t0 = Date.now();
try {
const { index, cache } = await scanLibrary(LIB, full ? {} : scanCache);
// Capture before the scan: a publication during the walk remains visible
// as a changed stamp on the next check, rather than being silently missed.
const stamps = await libraryStamps(LIB);
const result = mode === 'changed' && Object.keys(scanStamps).length
? await scanChangedLibrary(LIB, lib, scanCache, scanStamps, stamps)
: { ...(await scanLibrary(LIB, mode === 'deep' ? {} : scanCache)), stamps, changed: Object.keys(stamps) };
const { index, cache, changed } = result;
if (mode !== 'changed' || changed.length) {
announce(index);
lib = index; scanCache = cache;
lib = index; scanCache = cache; scanStamps = stamps;
if (mode !== 'changed') lastFullAt = Date.now();
writeAtomic(CACHE_FILE, JSON.stringify(cache));
writeAtomic(INDEX_FILE, JSON.stringify({ version: 1, root: LIB, at: Date.now(), fullAt: lastFullAt,
mangas: snapshotIndex(index), cache, stamps }));
}
let chapters = 0, pages = 0;
for (const m of index.mangas) for (const ch of m.chapters) { chapters++; for (const v of ch.versions) pages += v.pages.length; }
scanState = { at: Date.now(), ms: Date.now() - t0, mangas: index.mangas.length, chapters, pages, error: null };
console.log(`[scan] ${index.mangas.length} manga, ${chapters} chapters, ${pages} pages in ${scanState.ms} ms`);
for (const m of lib.mangas) for (const ch of m.chapters) { chapters++; for (const v of ch.versions) pages += v.pages.length; }
scanState = { at: Date.now(), ms: Date.now() - t0, mode, changedMangas: changed.length,
mangas: lib.mangas.length, chapters, pages, stale: false, error: null };
console.log(`[scan] ${mode}: ${changed.length} changed manga, ${lib.mangas.length} manga, ${chapters} chapters, ${pages} pages in ${scanState.ms} ms`);
return scanState;
} catch (e) {
scanState = { ...scanState, at: Date.now(), error: e.message };
console.error('[scan] failed:', e.message);
} finally { scanning = null; }
})();
return scanState;
}
}
function queueScan(mode) {
if (scanning) {
const priority = { changed: 1, full: 2, deep: 3 };
if (!pendingScan || priority[mode] > priority[pendingScan]) pendingScan = mode;
return scanning;
}
scanning = (async () => {
let next = mode, result;
do { result = await performScan(next); next = pendingScan; pendingScan = null; } while (next);
return result;
})().finally(() => { scanning = null; });
return scanning;
}
function rescan(full) { return queueScan(full ? 'deep' : 'full'); }
function refreshChanged() { return queueScan('changed'); }
// New chapter ids (compared with every id ever seen for that manga) notify the followers.
function announce(index) {
@@ -130,8 +209,78 @@ const url = rel => (rel ? 'lib/' + rel.split('/').map(encodeURIComponent).join('
const mangaOf = id => lib.byId.get(String(id)) || bad(404, 'Manga not found');
const chapterOf = (m, id) => m.chapterById.get(String(id)) || bad(404, 'Chapter not found');
const progressOf = (u, m) => (db.progress[u.name] && db.progress[u.name][m.id]) || {};
const mangaReadingOf = (u, m) => (db.reading[u.name] && db.reading[u.name][m.id]) || {};
const readingOf = (u, m, ch) => (db.reading[u.name] && db.reading[u.name][m.id] && db.reading[u.name][m.id][ch.id]) || null;
const unreadOf = u => (db.notes[u.name] || []).filter(n => !n.read).length;
const admin = u => u.admin || bad(403, 'Admins only');
const READING_LEASE_MS = 15000;
const readingLeases = new Map();
async function scraperRequest(endpoint, body) {
const headers = { accept: 'application/json' };
if (SCRAPER_TOKEN) headers.authorization = `Bearer ${SCRAPER_TOKEN}`;
const options = { headers, signal: AbortSignal.timeout(7000) };
if (body !== undefined) {
options.method = 'POST'; headers['content-type'] = 'application/json'; options.body = JSON.stringify(body);
}
let response;
try { response = await fetch(SCRAPER_URL + endpoint, options); }
catch (error) { throw new HttpError(503, `Scraper service unavailable: ${error.message}`); }
const data = await response.json().catch(() => ({}));
if (!response.ok) throw new HttpError(response.status >= 500 ? 502 : response.status, data.error || `Scraper HTTP ${response.status}`);
return data;
}
function readDiscovery() {
try { return discoveryStore.load(); }
catch (error) { bad(503, error.message); }
}
function startDiscoveryRun() {
if (discoveryRun?.status === 'running') bad(409, 'A discovery run is already active');
readDiscovery(); // Refuse a new run if the on-disk store is corrupt.
const id = crypto.randomUUID(), startedAt = new Date().toISOString();
discoveryRun = { id, status: 'running', startedAt };
discoveryError = null;
setImmediate(async () => {
try {
let queuedTitles = [], queueAvailable = true;
try { queuedTitles = (await scraperRequest('/api/state')).queue?.filter(item => !['complete', 'removed'].includes(item.status)).map(item => item.title) || []; }
catch { queueAvailable = false; }
const result = generateDiscovery({ db, library: lib.mangas, greatest: greatestCatalog,
cacheDir: DISCOVERY_CACHE, queuedTitles });
const published = discoveryStore.publish({ ...result, startedAt });
discoveryRun = { id, status: 'success', startedAt, finishedAt: published.lastRun.finishedAt,
candidates: published.lastRun.candidateCount, queueAvailable };
} catch (error) {
discoveryError = String(error.message || error).slice(0, 300);
discoveryRun = { id, status: 'failed', startedAt, finishedAt: new Date().toISOString() };
console.error('[discovery]', discoveryError);
}
});
return discoveryRun;
}
const mangaDexSearchCache = new Map();
let lastMangaDexSearch = 0;
async function searchGreatestMangaDex(entry) {
const cached = mangaDexSearchCache.get(entry.id);
if (cached && Date.now() - cached.time < 60 * 60 * 1000) return cached.results;
if (Date.now() - lastMangaDexSearch < 2000) bad(429, 'Wait two seconds before another MangaDex lookup');
lastMangaDexSearch = Date.now();
let response;
try {
response = await fetch(`https://api.mangadex.org/manga?title=${encodeURIComponent(entry.title)}&limit=10`,
{ headers: { accept: 'application/json' }, signal: AbortSignal.timeout(7000) });
} catch (error) { throw new HttpError(503, `MangaDex lookup unavailable: ${error.message}`); }
if (!response.ok) throw new HttpError(response.status === 429 ? 503 : 502, `MangaDex lookup returned HTTP ${response.status}`);
const body = await response.json();
const results = (body.data || []).filter(row => /^[0-9a-f-]{36}$/i.test(row.id)).map(row => ({
title: row.attributes?.title?.en || Object.values(row.attributes?.title || {})[0] || 'Untitled',
year: row.attributes?.year || null, url: `https://mangadex.org/title/${row.id}`,
}));
mangaDexSearchCache.set(entry.id, { time: Date.now(), results });
return results;
}
// Versions sorted by the sum of all user votes; ties go to the higher "priority" in chapter.json, then folder order.
function ranked(m, ch) {
@@ -146,6 +295,28 @@ function versionsOf(m, ch, u) {
id: v.id, source: v.source, sourceUrl: v.sourceUrl, pages: v.pages.length, added: v.added, score, my: (votes[v.id] && votes[v.id][u.name]) || 0,
}));
}
// Normalize the release-date shapes accepted by manga.json for library sorting.
function firstReleaseOf(meta) {
meta = meta || {};
const raw = meta.firstReleaseDate || meta.releaseDate || meta.startDate;
let year, month, day, label, time;
if (raw && typeof raw === 'object') {
year = +raw.year; month = +raw.month || 1; day = +raw.day || 1;
if (year > 0) {
label = [String(year).padStart(4, '0'), raw.month ? String(month).padStart(2, '0') : null, raw.day ? String(day).padStart(2, '0') : null].filter(Boolean).join('-');
time = Date.UTC(year, month - 1, day);
}
} else if (typeof raw === 'string' && raw.trim()) {
const parsed = Date.parse(raw);
if (Number.isFinite(parsed)) { label = raw.trim(); time = parsed; }
}
if (!Number.isFinite(time)) {
year = +meta.year;
if (year > 0) { label = String(year); time = Date.UTC(year, 0, 1); }
}
return { label: label || '', time: Number.isFinite(time) ? time : 0 };
}
function continueOf(m, prog, s) {
const ok = c => !(s.skipExtras && c.extra);
let last = null;
@@ -166,6 +337,48 @@ function neighbour(m, ch, step, s) {
// ---------- API ----------
const PUBLIC = new Set(['/api/auth', '/api/login', '/api/signup']);
const routes = {
'GET /api/health': () => ({ ok: !scanState.error, service: 'reader', uptime: Math.round(process.uptime()), scan: scanState }),
'GET /api/collections': ({ u }) => ({ collections: routes['GET /api/library']({ u }).collections }),
'GET /api/collection': ({ u, q }) => {
const view = routes['GET /api/library']({ u }).collections.find(c => c.id === q.get('id')) || bad(404, 'Collection not found');
const next = view.members.find(m => m.read < m.chapters) || view.members[0];
const manga = mangaOf(next.id);
return { ...view, continue: { m: next.id, ...continueOf(manga, progressOf(u, manga), settingsOf(u)) },
relations: catalog.state.relations.filter(r => r.state === 'confirmed' && view.members.some(m => m.id === r.from || m.id === r.to)) };
},
'GET /api/admin/catalog': ({ u }) => { admin(u); return { ...catalog.state, types: TYPES, roles: ROLES, mangas: lib.mangas.map(m => ({ id: m.id, title: m.title })) }; },
'POST /api/admin/catalog': ({ u, b }) => { admin(u); return catalog.update(b, new Set(lib.byId.keys()), u.name); },
'POST /api/admin/catalog/suggest': ({ u, b }) => {
admin(u);
return catalog.update({ ...catalog.state, revision: b.revision, relations: catalog.suggest(lib.mangas) }, new Set(lib.byId.keys()), u.name);
},
'POST /api/admin/catalog/research': async ({ u, b }) => {
admin(u);
const m = mangaOf(b.m), id = m.meta?.['x-anilist']?.id;
if (!id) bad(400, 'Assign a verified AniList identity first');
const { lookupAniListById } = require('./scraper/metadata/anilist');
const meta = await lookupAniListById(id, { cacheDir: path.join(DATA, 'metadata', 'anilist') });
const enriched = lib.mangas.map(item => item.id === m.id ? { ...item, meta } : item);
return catalog.update({ ...catalog.state, revision: b.revision, relations: catalog.suggest(enriched) }, new Set(lib.byId.keys()), u.name);
},
'POST /api/admin/collection-cover': ({ u, b }) => {
admin(u);
const c = catalog.state.collections.find(c => c.id === b.id) || bad(404, 'Collection not found');
if (b.revision !== catalog.state.revision) bad(409, 'Catalog changed; reload before saving');
const image = Buffer.from(String(b.base64 || ''), 'base64');
const ext = require('./scraper/lib/images').imageExtension(image);
if (!ext || image.length > 600000) bad(400, 'Use a JPG, PNG, WebP, GIF, AVIF or BMP under 600 KB');
const name = sha(image) + '.' + ext;
const dir = path.join(DATA, 'collection-covers'); fs.mkdirSync(dir, { recursive: true });
if (!fs.existsSync(path.join(dir, name))) fs.writeFileSync(path.join(dir, name), image, { flag: 'wx' });
return catalog.update({ ...catalog.state, collections: catalog.state.collections.map(item => item.id === c.id ? { ...item, coverMode: 'custom', customCover: name } : item) }, new Set(lib.byId.keys()), u.name);
},
'GET /api/admin/health': async ({ u }) => {
admin(u);
const sentinel = await fetch(SENTINEL_URL + '/api/state', { headers: SCRAPER_TOKEN ? { authorization: `Bearer ${SCRAPER_TOKEN}` } : {}, signal: AbortSignal.timeout(4000) })
.then(async r => { if (!r.ok) throw new Error(); return r.json(); }).catch(() => ({ status: 'unreachable' }));
return { reader: { ok: !scanState.error, uptime: Math.round(process.uptime()), scan: scanState }, sentinel };
},
'GET /api/auth': () => ({ firstRun: !Object.keys(db.users).length, signup: OPEN_SIGNUP }),
'POST /api/signup': ({ b, res }) => {
@@ -186,14 +399,31 @@ const routes = {
res.setHeader('Set-Cookie', 'sid=; Path=/; HttpOnly; SameSite=Lax; Max-Age=0');
return { ok: true };
},
'GET /api/me': ({ u }) => ({ name: u.name, admin: !!u.admin, settings: settingsOf(u), unread: unreadOf(u), senpai: !!SENPAI }),
'GET /api/me': ({ u }) => ({ name: u.name, admin: !!u.admin, settings: settingsOf(u), unread: unreadOf(u) }),
'GET /api/reading-schedule': () => publicSchedule(readingSchedule),
'GET /api/reading-access': ({ u, q }) => { mangaOf(q.get('m')); return readingAccess.status(u.name, true); },
'POST /api/reading-quiz/start': ({ u, b }) => {
try { return readingAccess.start(u.name); }
catch (e) { bad(409, e.message); }
},
'POST /api/reading-quiz/answer': ({ u, b }) => {
try { return readingAccess.answer(u.name, String(b.challenge || ''), String(b.answer || '')); }
catch (e) { bad(409, e.message); }
},
'POST /api/reading-quiz/abandon': ({ u, b }) => {
readingAccess.abandon(u.name, String(b.challenge || ''));
return { ok: true };
},
'POST /api/settings': ({ u, b }) => {
if (typeof b.webhook === 'string' && b.webhook.trim() && !/^https?:\/\/\S+$/i.test(b.webhook.trim())) bad(400, 'The webhook must be an http(s) URL.');
const s = (u.settings = u.settings || {});
if (['last', 'flow', 'page', 'book'].includes(b.startMode)) s.startMode = b.startMode;
if (['flow', 'page', 'book'].includes(b.lastMode)) s.lastMode = b.lastMode;
for (const k of ['adapt', 'skipExtras', 'skipCovers']) if (typeof b[k] === 'boolean') s[k] = b[k];
if (['original', 'inverted', 'sepia', 'dim', 'extra-dim'].includes(b.pageTone)) s.pageTone = b.pageTone;
for (const k of ['adapt', 'skipExtras', 'skipCovers', 'flowPageGap', 'readingClock', 'readingStory',
'discoveryEnabled', 'discoveryReadingActivity', 'discoveryMature']) if (typeof b[k] === 'boolean') s[k] = b[k];
if (b.zoom && typeof b.zoom === 'object') {
s.zoom = {};
for (const k of ['flow', 'page', 'book', 'strip']) if (+b.zoom[k] > 0) s.zoom[k] = Math.min(8, Math.max(0.1, +b.zoom[k]));
@@ -210,35 +440,166 @@ const routes = {
'GET /api/library': ({ u }) => {
const prog = db.progress[u.name] || {}, fol = db.follows[u.name] || {}, notes = (db.notes[u.name] || []).filter(n => !n.read);
return {
scanning: !!scanning && !lib.mangas.length,
mangas: lib.mangas.map(m => {
const p = prog[m.id] || {};
const showStory = settingsOf(u).readingStory;
const mangas = lib.mangas.map(m => {
const p = prog[m.id] || {}, meta = m.meta || {}, release = firstReleaseOf(meta);
return {
id: m.id, title: m.title, cover: url(m.cover), direction: m.direction, chapters: m.chapters.length, updated: m.updated,
altTitles: Array.isArray(meta.altTitles) ? meta.altTitles : (meta.altTitles ? [String(meta.altTitles)] : []),
authors: Array.isArray(meta.authors) ? meta.authors : (meta.authors ? [String(meta.authors)] : []),
tags: Array.isArray(meta.tags) ? meta.tags : (meta.tags ? [String(meta.tags)] : []), status: meta.status || '', year: meta.year || '',
firstRelease: release.label, firstReleaseAt: release.time,
read: m.chapters.filter(c => p[c.id] && p[c.id].done).length,
lastRead: Object.values(p).reduce((t, x) => Math.max(t, x.t || 0), 0),
following: !!fol[m.id], unread: notes.filter(n => n.m === m.id).length,
reading: showStory ? mangaInsight(mangaReadingOf(u, m), m.chapters.map(ch => ch.id)) : null,
};
}),
});
return {
scanning: !!scanning && !lib.mangas.length,
reading: showStory ? libraryInsight(mangas) : null,
mangas,
collections: collectionViews(catalog.state, mangas),
};
},
'GET /api/discovery': ({ u }) => {
const state = readDiscovery(), enabled = !!settingsOf(u).discoveryEnabled;
const feed = enabled ? state.feeds[u.name] || null : null;
return { enabled, settings: { discoveryEnabled: enabled,
discoveryReadingActivity: !!settingsOf(u).discoveryReadingActivity,
discoveryMature: !!settingsOf(u).discoveryMature },
generatedAt: feed?.generatedAt || null, revision: state.revision, lastRun: state.lastRun,
feedback: db.discoveryFeedback?.[u.name] || {},
items: (feed?.items || []).map(item => ({ ...item, candidate: state.candidates[item.candidateId] || null })).filter(item => item.candidate) };
},
'POST /api/discovery/feedback': ({ u, b }) => {
if (!settingsOf(u).discoveryEnabled) bad(403, 'Enable personalized discovery first');
const id = String(b.candidateId || ''), value = b.value === null ? null : String(b.value || '');
if (!readDiscovery().candidates[id]) bad(404, 'Discovery candidate not found');
if (value !== null && !['interested', 'maybe', 'dismissed', 'hidden'].includes(value)) bad(400, 'Unknown feedback');
const mine = (db.discoveryFeedback ||= {})[u.name] ||= {};
if (value === null) delete mine[id]; else mine[id] = value;
save();
return { candidateId: id, value };
},
'GET /api/admin/discovery': ({ u }) => {
admin(u);
const state = readDiscovery();
return { revision: state.revision, lastRun: state.lastRun, currentRun: discoveryRun,
lastError: discoveryError, optedInUsers: Object.values(db.users).filter(user => user.settings?.discoveryEnabled).length,
candidateCount: Object.keys(state.candidates).length };
},
'POST /api/admin/discovery/run': ({ u }) => { admin(u); return startDiscoveryRun(); },
'GET /api/greatest': ({ u }) => ({
title: greatestCatalog.title, updatedAt: greatestCatalog.updatedAt, periods: greatestCatalog.periods,
ratings: GREATEST_RATINGS,
entries: greatestCatalog.entries.map(entry => {
const owned = findGreatestOwned(entry, lib.mangas);
const counts = greatestCounts(entry.id, db.greatestRatings || {}, db.users);
return {
...entry,
cover: owned?.cover ? url(owned.cover) : entry.cover,
chapters: owned ? owned.chapters.length : entry.chapters,
status: owned?.meta?.status || entry.status,
library: owned ? { id: owned.id, chapters: owned.chapters.length } : null,
hosters: hosterLinks(entry, owned).links,
myRating: db.greatestRatings?.[u.name]?.[entry.id] || null,
votes: counts, score: greatestScore(counts),
};
}),
}),
'POST /api/greatest/rating': ({ u, b }) => {
const id = String(b.id || '');
if (!greatestById.has(id)) bad(404, 'Catalog entry not found');
const rating = b.rating === null || b.rating === '' ? null : String(b.rating || '');
if (rating !== null && !GREATEST_RATINGS.includes(rating)) bad(400, 'Unknown rating');
const all = (db.greatestRatings = db.greatestRatings || {});
const mine = (all[u.name] = all[u.name] || {});
if (rating === null) delete mine[id]; else mine[id] = rating;
save();
const votes = greatestCounts(id, all, db.users);
return { id, myRating: rating, votes, score: greatestScore(votes) };
},
'GET /api/greatest/sources': async ({ u, q }) => {
admin(u);
const entry = greatestById.get(String(q.get('id') || ''));
if (!entry) bad(404, 'Catalog entry not found');
const owned = findGreatestOwned(entry, lib.mangas);
try {
const state = await scraperRequest('/api/state');
return { available: true, ...hosterLinks(entry, owned, state.queue || []) };
} catch (error) {
return { available: false, error: error.message, ...hosterLinks(entry, owned) };
}
},
'GET /api/greatest/mangadex-search': async ({ u, q }) => {
admin(u);
const entry = greatestById.get(String(q.get('id') || ''));
if (!entry) bad(404, 'Catalog entry not found');
return { results: await searchGreatestMangaDex(entry) };
},
'GET /api/manga': ({ u, q }) => {
const m = mangaOf(q.get('m')), prog = progressOf(u, m);
const m = mangaOf(q.get('m')), prog = progressOf(u, m), showStory = settingsOf(u).readingStory;
const records = mangaReadingOf(u, m);
return {
id: m.id, title: m.title, meta: m.meta, cover: url(m.cover), direction: m.direction,
collections: catalog.state.collections.filter(c => c.members.some(member => member.mangaId === m.id)).map(c => ({ id: c.id, title: c.title })),
relations: catalog.state.relations.filter(r => r.state === 'confirmed' && (r.from === m.id || r.to === m.id)).map(r => ({ ...r,
otherId: r.from === m.id ? r.to : r.from, otherTitle: lib.byId.get(r.from === m.id ? r.to : r.from)?.title || (r.from === m.id ? r.to : r.from) })),
following: !!(db.follows[u.name] && db.follows[u.name][m.id]),
books: m.books.map(b => ({ key: b.key, label: b.label, title: b.title, extra: b.extra })),
chapters: m.chapters.map(ch => ({
id: ch.id, book: ch.book, label: ch.label, title: ch.title, extra: ch.extra, progress: prog[ch.id] || null, versions: versionsOf(m, ch, u),
reading: showStory ? chapterInsight(records[ch.id]) : null,
})),
reading: showStory ? mangaInsight(records, m.chapters.map(ch => ch.id)) : null,
continue: continueOf(m, prog, settingsOf(u)),
};
},
'GET /api/reading/export': ({ u }) => {
const byManga = db.reading[u.name] || {};
const mangas = Object.entries(byManga).map(([id, records]) => {
const m = lib.byId.get(id), chapterIds = m ? m.chapters.map(ch => ch.id) : Object.keys(records || {});
const chapterMap = m ? m.chapterById : new Map();
return {
id, title: m ? m.title : id,
reading: mangaInsight(records, chapterIds),
chapters: chapterIds.map(chapterId => {
const chapter = chapterMap.get(chapterId), reading = chapterInsight(records && records[chapterId]);
return reading ? { id: chapterId, label: chapter ? chapter.label : chapterId, title: chapter ? chapter.title : '', reading } : null;
}).filter(Boolean),
};
}).filter(item => item.reading.timedChapters || item.reading.completedPasses);
return {
schema: 1,
exportedAt: new Date().toISOString(),
user: u.name,
reading: libraryInsight(mangas),
mangas,
};
},
'POST /api/reading/delete': ({ u, b }) => {
if (b.all) {
if (!checkPw(String(b.password || ''), u.hash)) bad(403, 'The current password is wrong.');
delete db.reading[u.name];
readingLeases.delete(u.name);
save();
return { ok: true, deleted: 'all' };
}
const m = mangaOf(b.m), all = db.reading[u.name];
if (all) delete all[m.id];
save();
return { ok: true, deleted: m.id };
},
'GET /api/chapter': ({ u, q }) => {
const m = mangaOf(q.get('m')), ch = chapterOf(m, q.get('c')), s = settingsOf(u), prog = progressOf(u, m);
if (!readingAccess.status(u.name).allowed) bad(423, 'Reading is outside the deployment schedule. Complete the five-question challenge to unlock it.');
const v = ch.versions.find(x => x.id === q.get('v')) || ranked(m, ch)[0].v;
// Opening a chapter clears its "new chapter" notification.
let seen = false;
@@ -251,6 +612,7 @@ const routes = {
pages: v.pages.map(p => ({ u: url(v.rel + '/' + p.f), w: p.w, h: p.h, t: p.t, r: p.r })),
prev: neighbour(m, ch, -1, s), next: neighbour(m, ch, 1, s),
progress: prog[ch.id] || null,
reading: s.readingClock ? publicSummary(readingOf(u, m, ch), v.id, eligiblePages(v, s)) : null,
// for the chapter picker: the whole manga across all books
books: m.books.map(b => ({ key: b.key, label: b.label, title: b.title })),
chapters: m.chapters.map(c => ({ id: c.id, book: c.book, label: c.label, title: c.title, extra: c.extra, done: !!(prog[c.id] && prog[c.id].done) })),
@@ -260,16 +622,58 @@ const routes = {
'POST /api/progress': ({ u, b }) => {
const m = mangaOf(b.m), ch = chapterOf(m, b.c);
const all = (db.progress[u.name] = db.progress[u.name] || {}), mp = (all[m.id] = all[m.id] || {}), old = mp[ch.id] || {};
const v = String(b.v || ''), p = Math.max(0, b.p | 0);
mp[ch.id] = {
v, p, n: Math.max(0, b.n | 0), t: Date.now(), done: !!(old.done || b.done),
v: String(b.v || ''), p: Math.max(0, b.p | 0), n: Math.max(0, b.n | 0), t: Date.now(), done: !!(old.done || b.done),
shift: Array.isArray(b.shift) ? b.shift.filter(Number.isInteger).slice(0, 1000) : old.shift || [],
// furthest page read in this version (paging back to re-read does not lower the reading horizon)
top: Math.max(p, old.v === v ? old.top | 0 || old.p | 0 : 0),
};
save();
return { ok: true };
},
// Separate endpoint: an older server on rollback cannot misread cumulative
// visit totals as legacy deltas. Old open tabs retain their original protocol.
'POST /api/reading/visit': context => routes['POST /api/reading/pulse']({ ...context, cumulative: true }),
'POST /api/reading/pulse': ({ u, b, cumulative = false }) => {
const settings = settingsOf(u);
if (!settings.readingClock) bad(403, 'The Active Reading Clock is disabled.');
const m = mangaOf(b.m), ch = chapterOf(m, b.c);
const v = ch.versions.find(x => x.id === b.v) || bad(404, 'Version not found');
const session = String(b.session || '');
const seq = Number(b.seq);
if (!/^[a-zA-Z0-9_-]{8,80}$/.test(session)) bad(400, 'Invalid reading session.');
if (!Number.isInteger(seq) || seq < 1 || seq > 1e9) bad(400, 'Invalid reading pulse sequence.');
const now = Date.now(), eligible = eligiblePages(v, settings), lease = readingLeases.get(u.name);
const previous = readingOf(u, m, ch)?.recentSessions?.find(item => item.id === session);
// A stale request must not reclaim a lease after a newer exit save. Final
// snapshots can also arrive before the initial claim: do not lose that visit.
if (cumulative && previous && seq <= previous.seq) return {
accepted: true, duplicate: true, leaseMs: 0,
reading: publicSummary(readingOf(u, m, ch), v.id, eligible),
};
const lateFinal = cumulative && b.final;
if (!lateFinal && lease && lease.session !== session && now - lease.at < READING_LEASE_MS) {
return {
accepted: false,
reason: 'other-tab',
retryAfterMs: READING_LEASE_MS - (now - lease.at),
reading: publicSummary(readingOf(u, m, ch), v.id, eligible),
};
}
if (!lateFinal) readingLeases.set(u.name, { session, at: now });
const byUser = (db.reading[u.name] = db.reading[u.name] || {});
const byManga = (byUser[m.id] = byUser[m.id] || {});
const record = (byManga[ch.id] = readingOf(u, m, ch) || emptyRecord());
const result = (cumulative ? applyVisit : applyPulse)(record, b, { version: v.id, pageCount: v.pages.length, eligible, now });
if (b.release && readingLeases.get(u.name)?.session === session) readingLeases.delete(u.name);
if (!result.duplicate) save();
return {
accepted: true,
duplicate: result.duplicate,
leaseMs: b.release ? 0 : READING_LEASE_MS,
reading: publicSummary(record, v.id, eligible),
};
},
'POST /api/unread': ({ u, b }) => {
const m = mangaOf(b.m), mp = db.progress[u.name] && db.progress[u.name][m.id];
if (mp) delete mp[chapterOf(m, b.c).id];
@@ -318,49 +722,133 @@ const routes = {
admin(u);
const name = String(b.name || '');
if (name === u.name) bad(400, "You can't delete your own account.");
delete db.users[name]; delete db.progress[name]; delete db.follows[name]; delete db.notes[name];
delete db.users[name]; delete db.progress[name]; delete db.reading[name]; delete db.follows[name]; delete db.notes[name];
if (db.discoveryFeedback) delete db.discoveryFeedback[name];
if (db.greatestRatings) delete db.greatestRatings[name];
for (const [k, s] of Object.entries(db.sessions)) if (s.u === name) delete db.sessions[k];
save();
return { ok: true };
},
'GET /api/admin/peering': ({ u }) => {
admin(u);
return { ...peerStore.publicState(), library: lib.mangas.map(m => ({ id: m.id, title: m.title, cover: url(m.cover),
chapters: m.chapters.length, year: Number(m.meta.year || m.meta.startDate?.year) || null,
status: String(m.meta.status || 'unknown'), description: String(m.meta.description || m.meta.synopsis || '').slice(0, 3000),
tags: Array.isArray(m.meta.tags) ? m.meta.tags.map(tag => typeof tag === 'string' ? tag : tag.name).filter(Boolean).slice(0, 20) : [] })) };
},
'POST /api/admin/peering': async ({ u, b }) => {
admin(u);
const command = String(b.command || '');
if (command === 'set-name') peerStore.setName(b.name);
else if (command === 'generate') peerStore.generateIdentity(b.name);
else if (command === 'invite') return { ...peerStore.publicState(), invitation: peerStore.createInvitation(b.url) };
else if (command === 'import') peerStore.importInvitation(b.invitation);
else if (command === 'probe') await peerClient.probe(peerStore, String(b.id || ''));
else if (command === 'set') peerStore.updatePeer(String(b.id || ''), b.patch || {});
else if (command === 'revoke-grant') peerStore.revokeGrant(String(b.id || ''));
else if (command === 'share') {
if (!lib.byId.has(String(b.mangaId || ''))) bad(404, 'Library manga not found');
peerStore.setShare(String(b.id || ''), String(b.mangaId), !!b.enabled);
}
else if (command === 'share-set') {
const ids = Array.isArray(b.mangaIds) ? [...new Set(b.mangaIds.map(String))] : null;
if (!ids) bad(400, 'Manga selection is required');
if (ids.some(id => !lib.byId.has(id))) bad(404, 'One or more library manga are no longer available');
peerStore.setShares(String(b.id || ''), ids);
}
else if (command === 'grant') return { ...peerStore.publicState(), grant: peerStore.issueGrant(String(b.id || ''), b.scopes) };
else if (command === 'accept-grant') peerStore.acceptGrant(String(b.id || ''), b.token);
else if (command === 'remove') peerStore.removePeer(String(b.id || ''));
else bad(400, 'Unknown peering command');
return { ...peerStore.publicState(), library: lib.mangas.map(m => ({ id: m.id, title: m.title })) };
},
'GET /api/peering': ({ u }) => { admin(u); return { peers: peerStore.publicState().peers
.map(p => ({ id: p.id, alias: p.alias, receive: !!p.receive, paired: !!p.paired, fingerprint: p.fingerprint,
url: p.url, grantStatus: p.grantStatus, grantScopes: p.grantScopes,
readerQueue: !!p.readerQueue, shares: p.shares, hasGrant: p.hasOutboundGrant })) }; },
'GET /api/peering/access': async ({ u, q }) => {
admin(u);
const id = String(q.get('peer') || ''), peer = peerForRoute(id);
if (!peer.paired) return { paired: false, grant: { status: 'none', scopes: [] } };
return peerRemote(() => peerClient.access(peerStore, id));
},
'GET /api/peering/catalog': async ({ u, q }) => {
admin(u);
const id = String(q.get('peer') || '');
const peer = peerForRoute(id);
if (!peer.paired) bad(403, 'Verify peer identity first');
const access = await peerRemote(() => peerClient.access(peerStore, id));
if (access.grant.status !== 'active' || !access.grant.scopes.includes('catalog')) bad(403, 'This peer has not granted active catalog access');
return { peer: { id, alias: peer.alias }, items: await peerRemote(() => peerClient.catalog(peerStore, id)) };
},
'GET /api/peering/manga': async ({ u, q }) => {
admin(u);
const id = String(q.get('peer') || ''), shareId = String(q.get('manga') || '');
const peer = peerForRoute(id);
if (!peer.paired) bad(403, 'Verify peer identity first');
return peerRemote(() => peerClient.manga(peerStore, id, shareId));
},
'POST /api/peering/queue': async ({ u, b }) => {
admin(u);
const id = String(b.peer || ''), shareId = String(b.manga || '');
const peer = peerForRoute(id);
if (!peer.paired || !peer.receive || (!u.admin && !peer.readerQueue)) bad(403, 'Peer downloads are disabled for this account');
const access = await peerRemote(() => peerClient.access(peerStore, id));
if (!access.grant.scopes.includes('download')) bad(403, 'This peer has granted catalog browsing only');
if (!/^m[0-9a-f]{24}$/.test(shareId)) bad(400, 'Invalid peer manga ID');
const remote = await peerRemote(() => peerClient.manga(peerStore, id, shareId));
const present = await peerRemote(() => peerClient.catalog(peerStore, id));
if (!present.some(entry => entry.id === shareId)) bad(404, 'Peer manga not available');
return scraperRequest('/api/queue', { url: `peer://${id}/${shareId}`, title: remote.title, workIdentity: remote.workIdentity,
scope: b.scope || { mode: 'missing' }, existingPolicy: b.existingPolicy || 'skip-any',
...(b.libraryMangaId ? { libraryMangaId: String(b.libraryMangaId) } : {}) });
},
'POST /api/peering/queue-bulk': async ({ u, b }) => {
admin(u);
const id = String(b.peer || ''), peer = peerForRoute(id);
const ids = Array.isArray(b.mangas) ? [...new Set(b.mangas.map(String))] : [];
if (!ids.length || ids.length > 50 || ids.some(value => !/^m[0-9a-f]{24}$/.test(value))) bad(400, 'Select 1–50 peer titles');
if (!peer.paired || !peer.receive) bad(403, 'Enable downloads from this peer first');
const access = await peerRemote(() => peerClient.access(peerStore, id));
if (access.grant.status !== 'active' || !access.grant.scopes.includes('download')) bad(403, 'This peer has not granted active download access');
const available = new Set(await peerRemote(() => peerClient.catalog(peerStore, id)).then(items => items.map(item => item.id)));
const results = [];
for (const mangaId of ids) {
if (!available.has(mangaId)) { results.push({ id: mangaId, status: 'unavailable' }); continue; }
try {
const remote = await peerRemote(() => peerClient.manga(peerStore, id, mangaId));
const result = await scraperRequest('/api/queue', { url: `peer://${id}/${mangaId}`, title: remote.title, workIdentity: remote.workIdentity,
scope: { mode: 'missing' }, existingPolicy: 'skip-any' });
results.push({ id: mangaId, title: remote.title, status: 'queued', result });
} catch (error) { results.push({ id: mangaId, status: 'failed', error: String(error.message || error).slice(0, 240) }); }
}
return { results, queued: results.filter(item => item.status === 'queued').length };
},
'GET /api/admin/scraper': async ({ u }) => {
admin(u);
try { return { available: true, ...(await scraperRequest('/api/state')) }; }
catch (error) { return { available: false, error: error.message, url: SCRAPER_URL }; }
},
'POST /api/admin/scraper': async ({ u, b }) => {
admin(u);
const command = String(b.command || '');
if (command === 'enqueue') return scraperRequest('/api/queue', {
url: b.url, title: b.title, metadataSearch: b.metadataSearch,
scope: b.scope, existingPolicy: b.existingPolicy, libraryMangaId: b.libraryMangaId,
inspect: b.inspect === true,
});
if (command === 'remove') return scraperRequest('/api/queue/remove', { id: b.id });
if (command === 'clear-completed') return scraperRequest('/api/queue/clear-completed', { source: b.source });
if (command === 'reorder') return scraperRequest('/api/queue/reorder', { ids: b.ids });
if (command === 'control') return scraperRequest('/api/control', { action: b.action, id: b.id });
if (command === 'settings') return scraperRequest('/api/settings', b.settings || {});
if (command === 'discover') return scraperRequest('/api/discovery/run', {});
if (command === 'auto-update') return scraperRequest('/api/library/auto-update', { id: b.id, enabled: !!b.enabled });
if (command === 'publish-legacy-ready') return scraperRequest('/api/legacy/publish-ready', {});
bad(400, 'Unknown scraper command');
},
};
// ---------- Senpai proxy (optional) ----------
// GET /api/senpai/v1/... is forwarded to SENPAI_URL with the service token, the user name and the user's reading
// horizon in this manga: the furthest chapter (in reading order) with saved progress, its version and page.
// Senpai never sees Circus sessions, and the browser can't raise its own horizon.
const SENPAI_PATHS = /^v1\/(locate|xray|characters\/[A-Za-z0-9-]{1,80}(\/timeline)?|portraits\/[A-Za-z0-9-]{1,80})$/;
function horizonOf(u, m) {
const prog = progressOf(u, m);
let last = null;
for (const ch of m.chapters) if (prog[ch.id]) last = { ch, p: prog[ch.id] };
if (!last) return '';
const enc = encodeURIComponent;
const page = Math.max(last.p.p | 0, last.p.top | 0);
return `m=${enc(m.id)};c=${enc(last.ch.id)};v=${enc(last.p.v || '')};p=${page};done=${last.p.done ? 1 : 0}`;
}
function senpaiProxy(req, res, u, rest, q) {
if (!SENPAI) bad(404, 'Senpai is not enabled');
if (req.method !== 'GET') bad(405, 'Method not allowed');
if (!SENPAI_PATHS.test(rest)) bad(404, 'Unknown endpoint');
const m = mangaOf(q.get('m'));
const target = new URL(rest + '?' + q.toString(), SENPAI);
const headers = { Authorization: 'Bearer ' + SENPAI_TOKEN, 'X-Senpai-User': u.name, 'X-Senpai-Horizon': horizonOf(u, m), Accept: req.headers.accept || '*/*' };
const up = (target.protocol === 'https:' ? https : http).request(target, { method: 'GET', headers, timeout: 15000 }, r => {
const out = { 'Content-Type': r.headers['content-type'] || 'application/octet-stream', 'Cache-Control': r.headers['cache-control'] || 'no-store' };
if (r.headers['content-length']) out['Content-Length'] = r.headers['content-length'];
res.writeHead(r.statusCode, out);
r.pipe(res);
});
up.on('timeout', () => up.destroy(new Error('timeout')));
up.on('error', e => {
console.warn(`[senpai] ${e.message}`);
if (!res.headersSent) sendJson(res, 502, { error: 'Senpai is not reachable' });
else res.destroy();
});
up.end();
}
// ---------- HTTP ----------
function sendJson(res, code, obj) {
res.writeHead(code, { 'Content-Type': 'application/json; charset=utf-8', 'Cache-Control': 'no-store' });
@@ -398,9 +886,24 @@ http.createServer(async (req, res) => {
try {
const { pathname, searchParams } = new URL(req.url, 'http://localhost');
const u = userOf(req);
if (pathname.startsWith('/api/senpai/')) {
if (pathname === '/health' && req.method === 'GET') return sendJson(res, 200, { ...routes['GET /api/health'](), ok: true, ready: !scanState.error });
if (pathname === '/api/collection-cover' && req.method === 'GET') {
if (!u) bad(401, 'Please log in');
return senpaiProxy(req, res, u, pathname.slice('/api/senpai/'.length), searchParams);
const c = catalog.state.collections.find(c => c.id === searchParams.get('id'));
if (!c?.customCover) bad(404, 'Cover not found');
return serveFile(req, res, path.join(DATA, 'collection-covers'), c.customCover, 'private, no-cache', IMG);
}
if (pathname === '/api/peering/cover' && req.method === 'GET') {
if (!u) bad(401, 'Please log in');
admin(u);
const peerId = String(searchParams.get('peer') || ''), mangaId = String(searchParams.get('manga') || ''), peer = peerForRoute(peerId);
if (!peer.paired) bad(403, 'Verify peer identity first');
const access = await peerRemote(() => peerClient.access(peerStore, peerId));
if (access.grant.status !== 'active' || !access.grant.scopes.includes('catalog')) bad(403, 'This peer has not granted active catalog access');
const image = await peerRemote(() => peerClient.cover(peerStore, peerId, mangaId));
res.writeHead(200, { 'Content-Type': image.mime, 'Content-Length': image.buffer.length,
...(image.etag ? { ETag: image.etag } : {}), 'Cache-Control': 'private, max-age=3600', 'X-Content-Type-Options': 'nosniff' });
return res.end(image.buffer);
}
if (pathname.startsWith('/api/')) {
const route = routes[req.method + ' ' + pathname];
@@ -411,14 +914,54 @@ http.createServer(async (req, res) => {
if (req.method !== 'GET' && req.method !== 'HEAD') bad(405, 'Method not allowed');
if (pathname.startsWith('/lib/')) {
if (!u) bad(401, 'Please log in');
return serveFile(req, res, LIB, pathname.slice(5), 'private, max-age=86400', IMG);
let imagePath;
try { imagePath = path.resolve(LIB, decodeURIComponent(pathname.slice(5))); } catch { bad(400, 'Bad image path'); }
const relative = path.relative(LIB, imagePath);
const parts = relative.split(path.sep);
if (parts.length >= 4 && !readingAccess.status(u.name).allowed)
bad(423, 'Reading is outside the deployment schedule.');
return serveFile(req, res, LIB, pathname.slice(5), readingSchedule && parts.length >= 4 ? 'private, no-store' : 'private, max-age=86400', IMG);
}
serveFile(req, res, PUB, pathname === '/' ? 'index.html' : pathname.slice(1), 'no-cache');
} catch (e) {
if (!(e instanceof HttpError)) console.error(e);
if (!res.headersSent) sendJson(res, e.code || 500, { error: e instanceof HttpError ? e.message : 'Internal error' });
const expected = e instanceof HttpError || e instanceof CatalogError;
if (!expected) console.error(e);
if (!res.headersSent) sendJson(res, expected ? e.code : 500, { error: expected ? e.message : 'Internal error' });
}
}).listen(PORT, () => console.log(`MangaCircus on http://localhost:${PORT} (library: ${LIB}, data: ${DATA})${SENPAI ? ' senpai: ' + SENPAI : ''}`));
}).listen(PORT, () => console.log(`MangaCircus on http://localhost:${PORT} (library: ${LIB}, data: ${DATA})`));
rescan();
if (SCAN_MINUTES > 0) setInterval(() => rescan(), SCAN_MINUTES * 60000).unref();
if (lib.mangas.length) refreshChanged();
else rescan();
if (SCAN_MINUTES > 0) setInterval(() => refreshChanged(), SCAN_MINUTES * 60000).unref();
// Persist the last reconciliation time. Restarting a healthy reader should not
// trigger another full tree walk; overdue reconciliations run within a minute.
if (FULL_SCAN_MINUTES > 0) setInterval(() => {
const now = Date.now();
if (now - Math.max(lastFullAt, lastFullAttemptAt) >= FULL_SCAN_MINUTES * 60000) {
lastFullAttemptAt = now;
rescan();
}
}, 60000).unref();
// A completed scraper publish increments a durable revision. Polling it here keeps the
// reader index fresh without exposing an unauthenticated rescan endpoint to the daemon.
let scraperRevision = null;
async function pollScraper() {
try {
const state = await scraperRequest('/api/state');
const revision = Number(state.publishedRevision || 0);
if (scraperRevision === null || revision > scraperRevision) {
const checked = await refreshChanged();
if (checked.error) throw new Error(checked.error);
// A publication without a visible directory change should not disappear
// because of coarse filesystem timestamps or a missed scan race.
if (scraperRevision !== null && revision > scraperRevision && !checked.changedMangas) {
const reconciled = await rescan();
if (reconciled.error) throw new Error(reconciled.error);
}
}
scraperRevision = revision;
} catch { /* the reader also works when the optional scraper daemon is offline */ }
}
pollScraper();
setInterval(pollScraper, 15000).unref();
+32
View File
@@ -0,0 +1,32 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict'), fs = require('fs'), os = require('os'), path = require('path');
const { Catalog, normalize, collectionViews } = require('../catalog');
const input = () => ({ collections: [{ id: 'franchise', title: 'Franchise', members: [{ mangaId: 'a', role: 'main', readingOrder: 1 }, { mangaId: 'b', role: 'sequel', readingOrder: 2 }] }], relations: [{ from: 'a', to: 'b', type: 'sequel', state: 'confirmed', evidence: [] }] });
test('collections persist without changing manga identities and reject stale edits', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-catalog-'));
try {
const c = new Catalog(dir); c.update({ ...input(), revision: 0 }, new Set(['a', 'b']), 'admin');
assert.equal(new Catalog(dir).state.revision, 1);
assert.throws(() => c.update({ ...input(), revision: 0 }, new Set(['a', 'b']), 'admin'), { code: 409 });
assert.deepEqual(c.state.collections[0].members.map(m => m.mangaId), ['a', 'b']);
assert.throws(() => normalize({ ...input(), collections: [...input().collections, { id: 'other', title: 'Other', members: [{ mangaId: 'a' }] }] }, new Set(['a', 'b'])), /already belongs/);
assert.throws(() => normalize({ ...input(), relations: [{ from: 'a', to: 'a', type: 'sequel' }] }, new Set(['a', 'b'])), /Invalid relation/);
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
});
test('collections aggregate progress once and preserve absent members', () => {
const normalized = normalize(input(), new Set(['a', 'b']));
const views = collectionViews(normalized, [{ id: 'a', title: 'A', chapters: 3, read: 2, unread: 1, tags: ['Fantasy'], status: 'completed', reading: { activeMs: 30 } }, { id: 'b', title: 'B', chapters: 2, read: 1, unread: 1, tags: ['Fantasy'], status: 'ongoing', reading: { activeMs: 20 } }]);
assert.equal(views[0].chapters, 5); assert.equal(views[0].read, 3); assert.equal(views[0].reading.activeMs, 50);
assert.equal(views[0].status, 'ongoing'); assert.deepEqual(views[0].tags, ['Fantasy']);
assert.equal(normalize(normalized, new Set(['a']), normalized).collections[0].members.length, 2);
});
test('metadata suggestions retain manual and rejected decisions', () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-suggestions-'));
try {
const c = new Catalog(dir);
const mangas = [{ id: 'a', meta: { 'x-anilist': { id: 10, relations: [{ id: 11, type: 'SEQUEL' }] } } }, { id: 'b', meta: { 'x-anilist': { id: 11 } } }];
const relations = c.suggest(mangas); assert.equal(relations.length, 1); assert.equal(relations[0].state, 'suggested');
c.update({ revision: 0, collections: [], relations: [{ ...relations[0], state: 'rejected' }] }, new Set(['a', 'b']), 'admin');
assert.equal(c.suggest(mangas).length, 1); assert.equal(c.suggest(mangas)[0].state, 'rejected');
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
});
+17
View File
@@ -0,0 +1,17 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('node:fs'), path = require('node:path'), vm = require('node:vm');
const crypto = require('node:crypto');
const source = fs.readFileSync(path.join(__dirname, '../public/collections.js'), 'utf8').split('(async () => {')[0];
test('Collection IDs use the native secure-context API when available', () => {
const context = { crypto: { randomUUID: () => 'native-uuid' } };
vm.runInNewContext(source, context);
assert.equal(context.collectionId(), 'native-uuid');
});
test('Collection IDs work on HTTP without randomUUID and remain unique valid UUIDs', () => {
const context = { crypto: { getRandomValues: values => crypto.randomFillSync(values) } };
vm.runInNewContext(source, context);
const ids = Array.from({ length: 200 }, () => context.collectionId());
assert.equal(new Set(ids).size, ids.length);
for (const id of ids) assert.match(id, /^[a-f0-9]{8}-[a-f0-9]{4}-4[a-f0-9]{3}-[89ab][a-f0-9]{3}-[a-f0-9]{12}$/);
});
+48
View File
@@ -0,0 +1,48 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { generate } = require('../discovery/engine');
const { DiscoveryStore } = require('../discovery/store');
test('discovery is bounded, diversified, opt-in, evidence-linked and user-specific', () => {
const { loadCatalog } = require('../greatest');
const greatest = loadCatalog();
const users = { alice: { name: 'alice', settings: { discoveryEnabled: true } },
bob: { name: 'bob', settings: { discoveryEnabled: true } },
off: { name: 'off', settings: {} } };
const db = { users, greatestRatings: { alice: { [greatest.entries[0].id]: 'flame' },
bob: { [greatest.entries[20].id]: 'flame' } }, discoveryFeedback: { alice: {} }, progress: {} };
const result = generate({ db, library: [], greatest, cacheDir: path.join(os.tmpdir(), 'nonexistent-discovery-cache') });
assert.equal(Object.keys(result.candidates).length, 50);
assert.ok(result.feeds.alice.items.length > 0 && result.feeds.alice.items.length <= 20);
assert.ok(result.feeds.bob.items.length > 0);
assert.equal(result.feeds.off.disabled, true);
assert.ok(new Set(Object.values(result.candidates).map(c => c.period)).size >= 10, 'An early catalog page must not crowd out later eras');
assert.ok(result.feeds.alice.items.every(item => item.evidenceIds.every(id => item.evidence.some(e => e.id === id))));
assert.ok(result.feeds.alice.items.some(item => item.evidence.some(e => e.kind === 'personal')));
assert.ok(!result.feeds.bob.items.some(item => item.evidence.some(e => e.label?.includes(greatest.entries[0].title) && e.kind === 'personal')));
assert.notDeepEqual(result.feeds.alice.items.map(x => x.candidateId), result.feeds.bob.items.map(x => x.candidateId));
const first = result.feeds.alice.items[0].candidateId;
db.discoveryFeedback.alice[first] = 'dismissed';
const rerun = generate({ db, library: [], greatest, cacheDir: path.join(os.tmpdir(), 'nonexistent-discovery-cache') });
assert.ok(!rerun.feeds.alice.items.some(item => item.candidateId === first));
assert.equal(rerun.feeds.bob.items.length, result.feeds.bob.items.length);
});
test('discovery store publishes atomically and refuses to overwrite corrupt data', t => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-discovery-store-'));
t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
const store = new DiscoveryStore(dir);
assert.equal(store.load().revision, 0);
const value = store.publish({ candidates: {}, feeds: {}, startedAt: new Date().toISOString(), examined: 0, clipped: 0 });
assert.equal(value.revision, 1);
store.publish({ candidates: {}, feeds: {}, startedAt: new Date().toISOString(), examined: 0, clipped: 0 });
assert.equal(store.load().revision, 2);
assert.ok(fs.existsSync(store.file + '.prev'));
fs.writeFileSync(store.file, '{broken');
assert.throws(() => store.publish({ candidates: {}, feeds: {} }), /refusing to overwrite/);
assert.equal(fs.readFileSync(store.file, 'utf8'), '{broken');
});
+51
View File
@@ -0,0 +1,51 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const { loadCatalog, findOwned, countsFor, scoreFor } = require('../greatest');
const { seriesLink, hosterLinks } = require('../greatest/hosters');
test('curated manga catalog has unique identities and ordered five-year periods', () => {
const catalog = loadCatalog();
assert.equal(catalog.periods.length, 13);
assert.equal(catalog.entries.length, 106);
assert.ok(catalog.entries.every(entry => /^https:\/\//.test(entry.cover)), 'Every catalog title has cover artwork');
assert.ok(catalog.entries.every(entry => entry.note?.length >= 45 && /[.!?]$/.test(entry.note)), 'Every catalog title has a complete reason for inclusion');
for (const [index, period] of catalog.periods.entries()) {
const entries = catalog.entries.filter(entry => entry.period === period);
assert.ok(entries.length >= 5 && entries.length <= 10);
assert.deepEqual(entries.map(entry => entry.rank), entries.map((_, rank) => rank + 1));
assert.ok(entries.every(entry => entry.periodOrder === index));
}
assert.equal(catalog.entries.find(entry => entry.title === 'Phoenix').year, 1954);
assert.equal(catalog.entries.find(entry => entry.title === 'JoJo\'s Bizarre Adventure').year, 1987);
});
test('library matching and vote aggregation stay independent of scraper data', () => {
const catalog = loadCatalog();
const berserk = catalog.entries.find(entry => entry.title === 'Berserk');
const manga = { id: 'local-Berserk', title: 'Berserk', meta: {} };
assert.equal(findOwned(berserk, [manga]), manga);
assert.equal(findOwned(berserk, [{ id: 'different', title: 'Berserker', meta: {} }]), null);
const votes = { ada: { [berserk.id]: 'flame' }, bob: { [berserk.id]: 'good' }, gone: { [berserk.id]: 'skip' } };
const counts = countsFor(berserk.id, votes, { ada: {}, bob: {} });
assert.deepEqual(counts, { flame: 1, good: 1, neutral: 0, skip: 0 });
assert.equal(scoreFor(counts), 4);
});
test('hoster links use confirmed series URLs and do not mistake unrelated queue entries for matches', () => {
const entry = { title: 'BLAME!', aliases: ['Blame'] };
const weeb = 'https://weebcentral.com/series/01J76XY8CJ2RJGWMHM69ZS0J3H';
const dex = 'https://mangadex.org/title/06a37094-b3ba-4f25-bf2b-54c9751e0e91/bleach';
const owned = { id: 'BLAME!', meta: { sources: [{ name: 'WeebCentral', url: weeb }] } };
const result = hosterLinks(entry, owned, [
{ id: 'match', title: 'blame', url: weeb, status: 'complete', kind: 'download' },
{ id: 'other', title: 'Bleach', url: dex, status: 'queued', kind: 'download' },
]);
assert.equal(result.links.length, 1);
assert.equal(result.links[0].source, 'weebcentral');
assert.equal(result.jobs.length, 1);
assert.equal(result.jobs[0].status, 'complete');
assert.equal(seriesLink('https://weebcentral.com/chapters/01J76XY8CJ2RJGWMHM69ZS0J3H'), null);
assert.equal(seriesLink('http://weebcentral.com/series/01J76XY8CJ2RJGWMHM69ZS0J3H'), null);
assert.equal(seriesLink(dex).seriesId, '06a37094-b3ba-4f25-bf2b-54c9751e0e91');
});
+102
View File
@@ -0,0 +1,102 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('fs'), os = require('os'), path = require('path'), net = require('net');
const { spawn } = require('child_process');
const { Canvas } = require('../tools/png');
const { scanLibrary, scanChangedLibrary, libraryStamps, snapshotIndex, restoreIndex } = require('../library');
const image = new Canvas(20, 40, [20, 40, 80]).png();
function chapter(root, manga, number) {
const dir = path.join(root, manga, 'v01', `c${String(number).padStart(4, '0')}`);
fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(path.join(dir, '0001.png'), image);
fs.writeFileSync(path.join(dir, 'chapter.json'), JSON.stringify({ title: `Part ${number}` }));
}
test('saved index restores chapter maps and book references without rereading images', async () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-index-'));
try {
chapter(root, 'Alpha', 1); chapter(root, 'Beta', 1);
const first = await scanLibrary(root);
const stamps = await libraryStamps(root);
const restored = restoreIndex(JSON.parse(JSON.stringify(snapshotIndex(first.index))));
assert.equal(restored.byId.get('Alpha').chapterById.get('1/1').title, 'Part 1');
assert.strictEqual(restored.byId.get('Alpha').books[0].chapters[0], restored.byId.get('Alpha').chapters[0]);
const same = await scanChangedLibrary(root, restored, first.cache, stamps);
assert.deepEqual(same.changed, []);
assert.strictEqual(same.index, restored);
assert.throws(() => restoreIndex([{ id: 'Broken', books: [], chapters: [{ id: 'x', versions: [{}] }] }]), /Invalid chapter/);
} finally { fs.rmSync(root, { recursive: true, force: true }); }
});
test('shallow refresh replaces only a published manga and removes deleted cache entries', async () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-changed-'));
try {
chapter(root, 'Alpha', 1); chapter(root, 'Beta', 1);
const first = await scanLibrary(root), stamps = await libraryStamps(root);
const beta = first.index.byId.get('Beta');
chapter(root, 'Alpha', 2);
const next = await scanChangedLibrary(root, first.index, first.cache, stamps);
assert.deepEqual(next.changed, ['Alpha']);
assert.equal(next.index.byId.get('Alpha').chapters.length, 2);
assert.strictEqual(next.index.byId.get('Beta'), beta);
assert.equal(Object.keys(next.cache).length, 3);
fs.rmSync(path.join(root, 'Alpha'), { recursive: true });
const removed = await scanChangedLibrary(root, next.index, next.cache, next.stamps);
assert.deepEqual(removed.changed, ['Alpha']);
assert.equal(removed.index.byId.has('Alpha'), false);
assert.equal(Object.keys(removed.cache).length, 1);
} finally { fs.rmSync(root, { recursive: true, force: true }); }
});
test('in-place chapter metadata edits require a full reconciliation', async () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-deep-'));
try {
chapter(root, 'Alpha', 1);
const first = await scanLibrary(root), stamps = await libraryStamps(root);
fs.writeFileSync(path.join(root, 'Alpha', 'v01', 'c0001', 'chapter.json'), JSON.stringify({ title: 'Revised Part 1' }));
const shallow = await scanChangedLibrary(root, first.index, first.cache, stamps);
assert.deepEqual(shallow.changed, []);
assert.equal((await scanLibrary(root, first.cache)).index.byId.get('Alpha').chapters[0].title, 'Revised Part 1');
} finally { fs.rmSync(root, { recursive: true, force: true }); }
});
test('reader serves a saved index immediately after restart and verifies it in background', { timeout: 15000 }, async () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-warm-start-'));
let child;
try {
const library = path.join(root, 'library'), data = path.join(root, 'data');
fs.mkdirSync(data);
chapter(library, 'Alpha', 1);
const scanned = await scanLibrary(library), stamps = await libraryStamps(library);
fs.writeFileSync(path.join(data, 'library-index.json'), JSON.stringify({ version: 1, root: library,
at: Date.now(), mangas: snapshotIndex(scanned.index), cache: scanned.cache, stamps }));
const listener = net.createServer();
await new Promise(resolve => listener.listen(0, '127.0.0.1', resolve));
const port = listener.address().port;
await new Promise(resolve => listener.close(resolve));
const output = [];
child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '' } });
child.stdout.on('data', data => output.push(String(data)));
child.stderr.on('data', data => output.push(String(data)));
const base = `http://127.0.0.1:${port}`;
let ready = false;
for (let n = 0; n < 100 && !ready; n++) {
ready = await fetch(base + '/health').then(r => r.ok).catch(() => false);
if (!ready) await new Promise(resolve => setTimeout(resolve, 50));
}
assert.ok(ready, output.join(''));
const signup = await fetch(base + '/api/signup', { method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: 'fixture', password: 'fixture-password' }) });
assert.equal(signup.status, 200);
const response = await fetch(base + '/api/library', { headers: { cookie: signup.headers.get('set-cookie').split(';')[0] } });
assert.equal(response.status, 200);
assert.equal((await response.json()).mangas[0].title, 'Alpha');
assert.match(output.join(''), /restored 1 manga from saved index/);
} finally {
if (child) await new Promise(resolve => { if (child.exitCode !== null) return resolve(); child.once('exit', resolve); child.kill(); });
fs.rmSync(root, { recursive: true, force: true });
}
});
+69
View File
@@ -0,0 +1,69 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const { parseSchedule, scheduled, publicSchedule, answerMatches, createReadingAccess } = require('../reading-access');
const windows = parseSchedule(JSON.stringify({
0: [['08:00', '01:00']], 1: [['07:00', '12:00'], ['13:00', '24:00']],
2: [['07:00', '24:00']], 3: [['07:00', '24:00']], 4: [['07:00', '24:00']],
5: [['07:00', '24:00']], 6: [['08:00', '01:00']],
}), 'Europe/Berlin');
const at = iso => new Date(iso);
test('multiple daily windows and weekend overnight carry in local time', () => {
assert.equal(scheduled(windows, at('2026-10-12T08:00:00Z')), true); // Monday 10:00
assert.equal(scheduled(windows, at('2026-10-12T10:30:00Z')), false); // lunch gap 12:30
assert.equal(scheduled(windows, at('2026-10-12T11:30:00Z')), true);
assert.equal(scheduled(windows, at('2026-10-10T22:30:00Z')), true); // Sunday 00:30 from Saturday
assert.equal(scheduled(windows, at('2026-10-11T22:30:00Z')), true); // Monday 00:30 from Sunday
assert.equal(scheduled(windows, at('2026-10-11T23:30:00Z')), false); // Monday 01:30
});
test('schedule config rejects malformed and zero-length windows', () => {
assert.throws(() => parseSchedule('{"1":[["12:00","12:00"]]}', 'Europe/Berlin'));
assert.throws(() => parseSchedule('{"7":[]}', 'Europe/Berlin'));
assert.throws(() => parseSchedule('{"1":[["25:00","26:00"]]}', 'Europe/Berlin'));
});
test('public schedule exposes only the configured read-only windows', () => {
const view = publicSchedule(windows);
assert.equal(view.enabled, true);
assert.equal(view.timezone, 'Europe/Berlin');
assert.deepEqual(view.days[1], [['07:00', '12:00'], ['13:00', '24:00']]);
assert.deepEqual(view.days[6], [['08:00', '01:00']]);
assert.deepEqual(publicSchedule(null), { enabled: false, timezone: null, days: [] });
});
test('answers accept aliases and small typos but not unrelated words', () => {
assert.equal(answerMatches(' luffy ', ['Monkey D. Luffy', 'Luffy']), true);
assert.equal(answerMatches('Lufy', ['Luffy']), true);
assert.equal(answerMatches('pirate', ['Luffy']), false);
});
test('five ordered answers grant time, allow retries and reset on a new visit', () => {
let clock = Date.parse('2026-10-14T01:00:00Z'), saves = 0;
const grants = {}, bank = { moon: Array.from({ length: 5 }, (_, i) => ({ id: `q${i}`, prompt: `Question ${i}`, answers: [`answer${i}`], requires: ['v01/c0001'] })) };
let finished = false;
const gate = createReadingAccess({ schedule: parseSchedule('{"0":[],"1":[],"2":[],"3":[],"4":[],"5":[],"6":[]}', 'UTC'),
bank, grants, save: () => saves++, now: () => clock,
getProgress: () => ({ 'v01/c0001': { done: finished } }), getMangaTitle: () => 'Moon Garden' });
assert.equal(gate.status('reader', true).quizAvailable, false);
assert.throws(() => gate.start('reader'));
finished = true;
assert.equal(gate.status('reader', true).quizAvailable, true);
const first = gate.start('reader');
assert.equal(gate.answer('reader', first.challenge, 'wrong').correct, false);
assert.equal(gate.start('reader').number, 1);
assert.throws(() => gate.answer('reader', first.challenge, 'answer0'));
let current = gate.start('reader');
for (let i = 0; i < 5; i++) {
const answer = bank.moon.find(q => q.prompt === current.prompt).answers[0];
current = gate.answer('reader', current.challenge, answer);
}
assert.equal(current.complete, true);
assert.equal(grants.reader, clock + 3600000);
assert.equal(saves, 1);
assert.equal(gate.status('reader').allowed, true);
clock += 3600000;
assert.equal(gate.status('reader').allowed, false);
});
+102
View File
@@ -0,0 +1,102 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const {
applyPulse, chapterInsight, compactRanges, eligiblePages, emptyRecord, libraryInsight, mangaInsight, publicSummary,
} = require('../reading-analytics');
const pulse = (session, seq, extra = {}) => ({
session, seq, activeMs: 0, loadingMs: 0,
inputMs: { mouse: 0, touch: 0, keyboard: 0 }, seen: [],
fastBursts: 0, rejectedPageViews: 0, maxPps: 0,
...extra,
});
test('page ranges remain compact and deterministic', () => {
assert.deepEqual(compactRanges([8, 2, 3, 3, 4, 9, 12]), [[2, 4], [8, 9], [12, 12]]);
});
test('reading pulses are idempotent and server-clamped', () => {
const record = emptyRecord(), options = { version: 'source-a', pageCount: 4, eligible: [0, 1, 2, 3], now: 10000 };
applyPulse(record, pulse('session-one', 1, { activeMs: 9800, inputMs: { touch: 9800 } }), options);
assert.equal(record.totalActiveMs, 9800);
assert.equal(record.sessions, 1);
assert.equal(record.inputMs.touch, 9800);
const duplicate = applyPulse(record, pulse('session-one', 1, { activeMs: 9800 }), { ...options, now: 20000 });
assert.equal(duplicate.duplicate, true);
assert.equal(record.totalActiveMs, 9800);
applyPulse(record, pulse('session-one', 2, { activeMs: 50000, loadingMs: 50000, inputMs: { mouse: 50000 } }), { ...options, now: 21000 });
assert.equal(record.totalActiveMs + record.loadingMs, 9800 + 12000);
assert.ok(record.inputMs.mouse <= 6000);
});
test('eligible coverage completes a pass and a reread starts a fresh pass', () => {
const version = { pages: [{ t: ['cover'] }, { t: [] }, { t: ['extra'] }, { t: [] }] };
const eligible = eligiblePages(version, { skipCovers: true, skipExtras: true });
assert.deepEqual(eligible, [1, 3]);
const record = emptyRecord();
applyPulse(record, pulse('first-pass', 1, { activeMs: 1000, inputMs: { keyboard: 1000 }, seen: [0, 1, 2, 3] }), {
version: 'source-a', pageCount: 4, eligible, now: 1000,
});
assert.equal(record.completedPasses, 1);
assert.equal(record.firstCompleteMs, 1000);
assert.deepEqual(record.currentPass.seenRanges, [[1, 1], [3, 3]]);
applyPulse(record, pulse('second-pass', 1), { version: 'source-a', pageCount: 4, eligible, now: 2000 });
const summary = publicSummary(record, 'source-a', eligible);
assert.equal(summary.currentPass.complete, false);
assert.equal(summary.currentPass.seenPages, 0);
assert.equal(summary.totalActiveMs, 1000);
});
test('switching source versions starts separate page coverage without losing lifetime time', () => {
const record = emptyRecord();
applyPulse(record, pulse('version-test', 1, { activeMs: 800, seen: [0] }), {
version: 'source-a', pageCount: 2, eligible: [0, 1], now: 1000,
});
applyPulse(record, pulse('version-test', 2, { activeMs: 500, seen: [2] }), {
version: 'source-b', pageCount: 3, eligible: [0, 1, 2], now: 2000,
});
const summary = publicSummary(record, 'source-b', [0, 1, 2]);
assert.equal(summary.totalActiveMs, 1300);
assert.deepEqual(summary.currentPass.seenRanges, [[2, 2]]);
});
test('chapter and manga insights distinguish untimed, partial, complete and reread records', () => {
assert.equal(chapterInsight(null), null);
const records = {
first: { ...emptyRecord(), totalActiveMs: 60000, updatedAt: 100, currentPass: { activeMs: 60000 } },
second: { ...emptyRecord(), totalActiveMs: 540000, lastCompleteMs: 180000, firstCompleteMs: 210000, completedPasses: 2, updatedAt: 300, currentPass: { activeMs: 180000, completedAt: 300 } },
third: { ...emptyRecord(), totalActiveMs: 300000, lastCompleteMs: 300000, firstCompleteMs: 300000, completedPasses: 1, updatedAt: 200, currentPass: { activeMs: 300000, completedAt: 200 } },
};
assert.deepEqual(chapterInsight(records.first), {
totalActiveMs: 60000, firstCompleteMs: null, lastCompleteMs: null, completedPasses: 0,
currentPassMs: 60000, currentPassComplete: false, updatedAt: 100,
});
assert.deepEqual(mangaInsight(records, ['first', 'second', 'third', 'missing']), {
activeMs: 900000,
timedChapters: 3,
completedChapters: 2,
completedPasses: 3,
rereads: 1,
typicalChapterMs: 240000,
lastChapterId: 'second',
updatedAt: 300,
});
});
test('library insights select the most-read and most-recent manga', () => {
const summary = libraryInsight([
{ id: 'a', reading: { activeMs: 1000, timedChapters: 1, completedPasses: 1, rereads: 0, updatedAt: 100, lastChapterId: '1/1' } },
{ id: 'b', reading: { activeMs: 3000, timedChapters: 2, completedPasses: 4, rereads: 2, updatedAt: 90, lastChapterId: '1/2' } },
{ id: 'empty', reading: { activeMs: 0, timedChapters: 0, completedPasses: 0, rereads: 0, updatedAt: 0 } },
]);
assert.deepEqual(summary, {
activeMs: 4000, timedChapters: 3, completedPasses: 5, rereads: 2, timedMangas: 2,
mostTimeMangaId: 'b', lastMangaId: 'a', lastChapterId: '1/1', updatedAt: 100,
});
});
+83
View File
@@ -0,0 +1,83 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict');
const { applyVisit, emptyRecord, chapterInsight, publicSummary } = require('../reading-analytics');
const { exposureTracker, stateFor, IDLE_MS } = require('../public/reading-clock-core');
const pages = n => Array.from({ length: n }, (_, i) => i);
const config = { version: 'webtoon', pageCount: 20, eligible: pages(20), now: 10000 };
const visit = (extra = {}) => ({ session: 'reading-visit', seq: 1, elapsedMs: 10000, activeMs: 9000, seen: [], ...extra });
test('59% is a draft; 60% credits the entire visit and final duration keeps growing', () => {
const r = emptyRecord(), cfg = { ...config, pageCount: 100, eligible: pages(100) };
applyVisit(r, visit({ seen: pages(59) }), cfg);
assert.equal(r.totalActiveMs, 0); assert.equal(chapterInsight(r), null);
applyVisit(r, visit({ seq: 2, elapsedMs: 12000, activeMs: 11000, seen: pages(60) }), cfg);
assert.equal(r.totalActiveMs, 11000); assert.equal(r.completedPasses, 1);
applyVisit(r, visit({ seq: 3, elapsedMs: 18000, activeMs: 16000, seen: pages(65), final: true }), cfg);
assert.equal(r.totalActiveMs, 16000); assert.equal(r.firstCompleteMs, 16000); assert.equal(r.lastCompleteMs, 16000);
assert.equal(r.currentPass.closed, true);
});
test('an exit snapshot overtaking a pulse counts once and survives a JSON reload', () => {
const r = emptyRecord();
applyVisit(r, visit({ seq: 3, seen: pages(12), final: true }), config);
const restored = JSON.parse(JSON.stringify(r));
assert.equal(applyVisit(restored, visit({ seq: 1, activeMs: 0 }), config).duplicate, true);
assert.equal(applyVisit(restored, visit({ seq: 3, seen: pages(12), final: true }), config).duplicate, true);
applyVisit(restored, visit({ seq: 4, seen: pages(13), activeMs: 10000, elapsedMs: 11000 }), config);
assert.equal(restored.totalActiveMs, 10000); assert.equal(restored.completedPasses, 1);
});
test('delayed delivery does not truncate an honest minute to twelve seconds', () => {
const r = emptyRecord();
applyVisit(r, visit({ elapsedMs: 65000, activeMs: 60000, inputMs: { touch: 60000 }, seen: pages(12) }), config);
assert.equal(r.totalActiveMs, 60000); assert.equal(r.inputMs.touch, 60000);
});
test('one-picture lookup does not borrow prior coverage or change saved statistics', () => {
const r = emptyRecord();
applyVisit(r, visit({ seen: pages(12), final: true }), config);
applyVisit(r, visit({ session: 'lookup-visit', elapsedMs: 60000, activeMs: 59000, seen: [15], final: true }), { ...config, now: 70000 });
assert.equal(r.totalActiveMs, 9000); assert.equal(r.completedPasses, 1);
assert.equal(chapterInsight(r).updatedAt, 10000);
});
test('late finals update their own visit without replacing a newer visit', () => {
const r = emptyRecord();
applyVisit(r, visit({ seen: pages(12) }), config);
applyVisit(r, visit({ session: 'new-visit', seen: pages(12), activeMs: 4000 }), { ...config, now: 20000 });
applyVisit(r, visit({ seq: 2, seen: pages(14), elapsedMs: 15000, activeMs: 13000, final: true }), { ...config, now: 21000 });
assert.equal(r.currentPass.visit, 'new-visit'); assert.equal(r.firstCompleteMs, 13000);
assert.equal(r.lastCompleteMs, 4000); assert.equal(r.totalActiveMs, 17000);
});
test('legacy time and source IDs are retained; covers/extras use the eligible denominator', () => {
const r = { ...emptyRecord(), totalActiveMs: 5000, completedPasses: 1, firstCompleteMs: 5000, lastCompleteMs: 5000, updatedAt: 500 };
const cfg = { ...config, eligible: [1, 3, 5, 7, 9] };
applyVisit(r, visit({ seen: [0, 1, 3, 5, 19], activeMs: 2000 }), cfg);
assert.equal(r.totalActiveMs, 7000); assert.equal(r.firstCompleteMs, 5000);
assert.equal(publicSummary(r, 'webtoon', cfg.eligible).currentPass.requiredPages, 3);
});
test('fast webtoon slices above five pages per second are credited without a ceiling', () => {
const t = exposureTracker();
for (let i = 0; i < 10; i++) t.update([{ index: i, loaded: true }], i * 120, true);
t.update([], 1200, true);
assert.deepEqual([...t.seen], pages(10));
});
test('jumping over pages to dwell on one image never invents skipped coverage', () => {
const t = exposureTracker();
t.update([{ index: 0, loaded: true }], 0, true);
t.update([{ index: 19, loaded: true }], 50, true);
t.update([{ index: 19, loaded: true }], 60000, true);
assert.deepEqual([...t.seen], [19]);
});
test('unloaded and hidden pages cannot qualify; mixed loading does not pause readable pages', () => {
assert.equal(IDLE_MS, 30000, 'The user-selected inactivity allowance is 30 seconds');
const t = exposureTracker();
t.update([{ index: 0, loaded: false }], 0, true);
t.update([{ index: 0, loaded: false }], 1000, true);
assert.equal(t.seen.size, 0);
const view = { pages: [{ index: 0, loaded: true }, { index: 1, loaded: false }] };
const s = { visible: true, focused: true, lastInputAt: 0, now: 5000, owned: true, view };
assert.equal(stateFor(s), 'reading');
assert.equal(stateFor({ ...s, visible: false }), 'hidden');
assert.equal(stateFor({ ...s, focused: false }), 'unfocused');
assert.equal(stateFor({ ...s, now: IDLE_MS }), 'idle');
assert.equal(stateFor({ ...s, now: IDLE_MS - 1 }), 'reading');
t.update(view.pages, 1000, false); t.update(view.pages, 100000, false);
assert.equal(t.seen.size, 0);
});
+160
View File
@@ -0,0 +1,160 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('fs'), os = require('os'), path = require('path'), net = require('net');
const { spawn } = require('child_process');
const { Canvas } = require('../tools/png');
const root = path.resolve(__dirname, '..');
const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
async function port() {
const server = net.createServer();
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
const result = server.address().port; await new Promise(resolve => server.close(resolve)); return result;
}
test('three services integrate collections, auth, custom covers and private health without source traffic', { timeout: 30000 }, async () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-services-'));
const children = [], output = [];
try {
const data = path.join(dir, 'data'), library = path.join(dir, 'library');
fs.mkdirSync(path.join(data, 'scraper'), { recursive: true });
fs.writeFileSync(path.join(data, 'scraper', 'queue.json'), JSON.stringify({ paused: true, queue: [], settings: { autoUpdateEnabled: false } }));
const png = new Canvas(32, 48, [10, 90, 130]).png();
for (const name of ['Main Work', 'Sequel']) {
const folder = path.join(library, name, 'v01', 'c001'); fs.mkdirSync(folder, { recursive: true });
fs.writeFileSync(path.join(folder, '001.png'), png);
fs.writeFileSync(path.join(library, name, 'cover.png'), png);
fs.writeFileSync(path.join(library, name, 'manga.json'), JSON.stringify({ title: name, tags: ['Fantasy'], status: 'ongoing' }));
}
const [rp, sp, wp] = await Promise.all([port(), port(), port()]);
const base = `http://127.0.0.1:${rp}`, scraper = `http://127.0.0.1:${sp}`, sentinel = `http://127.0.0.1:${wp}`;
const token = 'integration-test-only';
const env = { ...process.env, DATA: data, LIBRARY: library, SCRAPER_DATA: path.join(data, 'scraper'), SENTINEL_DATA: path.join(data, 'sentinel'),
PORT: String(rp), SCRAPER_PORT: String(sp), SENTINEL_PORT: String(wp), SCRAPER_HOST: '127.0.0.1', SENTINEL_HOST: '127.0.0.1',
READER_URL: base, SCRAPER_URL: scraper, SENTINEL_URL: sentinel, SCRAPER_TOKEN: token, SCRAPER_TOKEN_FILE: '', SCAN_INTERVAL: '0', COOKIE_SECURE: '0' };
for (const script of ['server.js', 'scraper/daemon.js', 'sentinel/daemon.js']) {
const child = spawn(process.execPath, [script], { cwd: root, env, windowsHide: true }); children.push(child);
for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => output.push(chunk.toString()));
}
for (const url of [base, scraper, sentinel]) {
let ready = false;
for (let n = 0; n < 80 && !ready; n++) { ready = await fetch(url + '/health').then(r => r.ok).catch(() => false); if (!ready) await delay(100); }
assert.ok(ready, output.join(''));
}
assert.equal((await fetch(scraper + '/api/state')).status, 401);
assert.equal((await fetch(sentinel + '/api/state')).status, 401);
const signup = await fetch(base + '/api/signup', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ name: 'tester', password: 'local-test-password' }) });
assert.equal(signup.status, 200); const cookie = signup.headers.get('set-cookie').split(';')[0];
const request = (endpoint, body) => fetch(base + endpoint, { headers: { cookie, 'content-type': 'application/json' }, ...(body ? { method: 'POST', body: JSON.stringify(body) } : {}) });
let inventory;
for (let n = 0; n < 50; n++) { inventory = await (await request('/api/library')).json(); if (inventory.mangas.length === 2) break; await delay(100); }
assert.equal(inventory.mangas.length, 2);
assert.equal((await fetch(base + '/api/greatest')).status, 401);
const greatest = await (await request('/api/greatest')).json();
assert.equal(greatest.entries.length, 106);
const firstPick = greatest.entries[0];
assert.equal(firstPick.myRating, null);
assert.equal((await request('/api/greatest/rating', { id: firstPick.id, rating: 'flame' })).status, 200);
assert.equal((await request('/api/greatest/rating', { id: firstPick.id, rating: 'invalid' })).status, 400);
assert.equal((await request('/api/greatest/rating', { id: 'missing', rating: 'good' })).status, 404);
const afterRating = (await (await request('/api/greatest')).json()).entries[0];
assert.equal(afterRating.myRating, 'flame');
assert.equal(afterRating.votes.flame, 1);
assert.equal(afterRating.score, 3);
assert.equal((await request('/api/admin/user', { name: 'another', password: 'local-test-password' })).status, 200);
const anotherLogin = await fetch(base + '/api/login', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ name: 'another', password: 'local-test-password' }) });
assert.equal(anotherLogin.status, 200);
const anotherCookie = anotherLogin.headers.get('set-cookie').split(';')[0];
const anotherRequest = (endpoint, body) => fetch(base + endpoint, { headers: { cookie: anotherCookie, 'content-type': 'application/json' }, ...(body ? { method: 'POST', body: JSON.stringify(body) } : {}) });
assert.equal((await (await request('/api/me')).json()).settings.pageTone, 'original');
assert.equal((await (await request('/api/settings', { pageTone: 'sepia' })).json()).settings.pageTone, 'sepia');
assert.equal((await (await request('/api/settings', { pageTone: 'invalid' })).json()).settings.pageTone, 'sepia');
assert.equal((await (await anotherRequest('/api/me')).json()).settings.pageTone, 'original');
assert.equal((await fetch(base + '/api/discovery')).status, 401);
assert.equal((await anotherRequest('/api/admin/discovery/run', {})).status, 403);
assert.equal((await request('/api/settings', { discoveryEnabled: true })).status, 200);
assert.equal((await anotherRequest('/api/settings', { discoveryEnabled: true })).status, 200);
const beforeDiscoveryQueue = fs.readFileSync(path.join(data, 'scraper', 'queue.json'), 'utf8');
const firstRun = await request('/api/admin/discovery/run', {});
assert.equal(firstRun.status, 200);
const firstRunId = (await firstRun.json()).id;
let discoveryAdmin;
for (let n = 0; n < 50; n++) {
discoveryAdmin = await (await request('/api/admin/discovery')).json();
if (discoveryAdmin.currentRun?.id === firstRunId && discoveryAdmin.currentRun?.status === 'success') break;
await delay(100);
}
assert.equal(discoveryAdmin.currentRun?.status, 'success');
const discoveryMine = await (await request('/api/discovery')).json();
const discoveryOther = await (await anotherRequest('/api/discovery')).json();
assert.ok(discoveryMine.items.length > 0 && discoveryMine.items.length <= 20);
assert.ok(discoveryOther.items.length > 0 && discoveryOther.items.length <= 20);
assert.equal(discoveryMine.items[0].candidate.id, discoveryMine.items[0].candidateId);
const dismissed = discoveryMine.items[0].candidateId;
assert.equal((await request('/api/discovery/feedback', { candidateId: dismissed, value: 'dismissed' })).status, 200);
assert.equal((await anotherRequest('/api/discovery')).status, 200);
assert.equal((await anotherRequest('/api/discovery/feedback', { candidateId: 'nonexistent', value: 'interested' })).status, 404);
const secondRun = await request('/api/admin/discovery/run', {});
assert.equal(secondRun.status, 200);
const secondRunId = (await secondRun.json()).id;
for (let n = 0; n < 50; n++) {
discoveryAdmin = await (await request('/api/admin/discovery')).json();
if (discoveryAdmin.currentRun?.id === secondRunId && discoveryAdmin.currentRun?.status === 'success') break;
await delay(100);
}
assert.ok(!(await (await request('/api/discovery')).json()).items.some(item => item.candidateId === dismissed));
assert.equal(fs.readFileSync(path.join(data, 'scraper', 'queue.json'), 'utf8'), beforeDiscoveryQueue);
assert.equal((await (await anotherRequest('/api/greatest')).json()).entries[0].myRating, null);
assert.equal((await anotherRequest('/api/greatest/rating', { id: firstPick.id, rating: 'good' })).status, 200);
const communityPick = (await (await request('/api/greatest')).json()).entries[0];
assert.equal(communityPick.myRating, 'flame');
assert.deepEqual(communityPick.votes, { flame: 1, good: 1, neutral: 0, skip: 0 });
assert.equal((await request('/api/greatest/rating', { id: firstPick.id, rating: null })).status, 200);
assert.equal((await (await request('/api/greatest')).json()).entries[0].votes.flame, 0);
assert.equal((await request('/api/admin/user/delete', { name: 'another' })).status, 200);
assert.equal((await (await request('/api/greatest')).json()).entries[0].votes.good, 0);
const ids = inventory.mangas.map(m => m.id);
const catalog = { revision: 0, collections: [{ id: 'demo', title: 'Demo collection', members: ids.map((mangaId, index) => ({ mangaId, readingOrder: index + 1, role: index ? 'sequel' : 'main' })), coverMode: 'mosaic' }], relations: [{ from: ids[0], to: ids[1], type: 'sequel', state: 'confirmed' }] };
let response = await request('/api/admin/catalog', catalog); assert.equal(response.status, 200); const saved = await response.json();
response = await request('/api/admin/catalog', catalog); assert.equal(response.status, 409); assert.match((await response.json()).error, /reload/);
const view = await (await request('/api/collection?id=demo')).json(); assert.equal(view.members.length, 2); assert.equal(view.chapters, 2); assert.equal(view.relations.length, 1);
let revision = saved.revision;
for (let n = 0; n < 2; n++) {
const cover = await (await request('/api/admin/collection-cover', { id: 'demo', revision, base64: png.toString('base64') })).json(); revision = cover.revision;
const image = await request('/api/collection-cover?id=demo'); assert.equal(image.status, 200); assert.deepEqual(Buffer.from(await image.arrayBuffer()), png);
}
const health = await (await request('/api/admin/health')).json(); assert.ok(health.reader); assert.ok(health.sentinel);
assert.equal((await request('/api/admin/scraper')).status, 200);
// Exercise the actual authenticated visit route and cross-chapter lease races.
assert.equal((await request('/api/settings', { readingClock: true })).status, 200);
const readerChapter = async m => {
const detail = await (await request('/api/manga?m=' + encodeURIComponent(m))).json();
return (await request('/api/chapter?m=' + encodeURIComponent(m) + '&c=' + encodeURIComponent(detail.chapters[0].id))).json();
};
const first = await readerChapter(ids[0]), second = await readerChapter(ids[1]);
const visit = (d, session, seq, extra = {}) => ({ m: d.manga.id, c: d.chapter.id, v: d.version, session, seq, elapsedMs: 2000, activeMs: 0, seen: [], ...extra });
const send = async b => (await request('/api/reading/visit', b)).json();
await send(visit(first, 'first-visit', 1));
let result = await send(visit(first, 'first-visit', 3, { activeMs: 1000, seen: [0], final: true, release: true }));
assert.equal(result.reading.totalActiveMs, 1000);
assert.equal((await send(visit(second, 'second-visit', 1))).accepted, true);
// The stale in-flight pulse arrives after exit and after the next reader claims.
assert.equal((await send(visit(first, 'first-visit', 2))).duplicate, true);
result = await send(visit(first, 'first-visit', 4, { activeMs: 1500, seen: [0], final: true, release: true }));
assert.equal(result.accepted, true); assert.equal(result.reading.totalActiveMs, 1500);
// A beacon arriving before its first claim is also retained, without stealing
// the current reader's lease.
assert.equal((await send(visit(first, 'beacon-first', 2, { activeMs: 500, seen: [0], final: true, release: true }))).accepted, true);
assert.equal((await send(visit(second, 'third-visit', 1))).accepted, false);
await send(visit(second, 'second-visit', 2, { release: true }));
assert.equal((await fetch(base + '/api/reading/visit', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(visit(first, 'unauth-visit', 1)) })).status, 401);
await request('/api/settings', { readingClock: false });
assert.equal((await request('/api/reading/visit', visit(first, 'disabled-visit', 1))).status, 403);
const unauth = await fetch(base + '/api/admin/catalog'); assert.equal(unauth.status, 401);
await request('/api/signup', { name: 'reader', password: 'local-test-password' });
const login = await fetch(base + '/api/login', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ name: 'reader', password: 'local-test-password' }) });
// Sign-up is closed after first-run; the admin catalog remains protected.
assert.equal(login.status, 403);
} finally {
await Promise.all(children.map(child => new Promise(resolve => { if (child.exitCode !== null) return resolve(); child.once('exit', resolve); child.kill(); })));
fs.rmSync(dir, { recursive: true, force: true });
}
});
+32
View File
@@ -0,0 +1,32 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const { SETTINGS_SECTIONS, SCRAPER_TABS, settingsRoute, readingDayText } = require('../public/settings-nav');
test('settings organize unique routes into User and Admin, with scraper-local tabs', () => {
assert.equal(new Set(SETTINGS_SECTIONS.map(s => s.id)).size, SETTINGS_SECTIONS.length);
assert.deepEqual([...new Set(SETTINGS_SECTIONS.map(s => s.group))], ['User', 'Admin']);
assert.ok(SETTINGS_SECTIONS.every(s => s.id.split('/').length === 2));
assert.deepEqual(SCRAPER_TABS, ['activity', 'queue', 'updates', 'sources']);
for (const section of SETTINGS_SECTIONS) assert.equal(settingsRoute('#' + section.id, true).id, section.id);
});
test('settings deep links default safely and never route ordinary readers into admin', () => {
assert.deepEqual(settingsRoute('', false), { id: 'user/reading', tab: 'activity' });
assert.equal(settingsRoute('#not/a/section', true).id, 'user/reading');
assert.equal(settingsRoute('#admin/users', false).id, 'user/reading');
assert.equal(settingsRoute('#admin/scraper/hosters', false).id, 'user/reading');
assert.equal(settingsRoute('#user/notifications', false).id, 'user/notifications');
assert.deepEqual(settingsRoute('#admin/scraper/hosters', true), { id: 'admin/scraper', tab: 'sources' });
assert.deepEqual(settingsRoute('#admin/scraper/sources', true), { id: 'admin/scraper', tab: 'sources' });
assert.equal(settingsRoute('#admin/scraper/unknown', true).tab, 'activity');
});
test('reading-hours rows include overnight carry from the previous day', () => {
const schedule = { days: [
[['08:00', '01:00']], [['07:00', '24:00']], [], [], [], [], [['08:00', '01:00']],
] };
assert.equal(readingDayText(schedule, 0), '00:00–01:00 (from Saturday) · 08:00–01:00 (next day)');
assert.equal(readingDayText(schedule, 1), '00:00–01:00 (from Sunday) · 07:00–24:00');
assert.equal(readingDayText(schedule, 2), 'Closed all day');
});
+20
View File
@@ -0,0 +1,20 @@
'use strict';
// Isolated browser QA. Never reads production user data or contacts image sources.
const fs = require('fs'), os = require('os'), path = require('path'), { spawn } = require('child_process');
const { Canvas } = require('../tools/png');
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-ui-')), data = path.join(dir, 'data'), library = path.join(dir, 'library');
fs.mkdirSync(path.join(data, 'scraper'), { recursive: true });
fs.writeFileSync(path.join(data, 'scraper', 'queue.json'), JSON.stringify({ paused: true, queue: [], settings: { autoUpdateEnabled: false } }));
for (const [index, name] of ['Moon Garden', 'Moon Garden Before the Rain', 'Iron Harbor'].entries()) {
const folder = path.join(library, name, 'v01', 'c001 The Beginning'); fs.mkdirSync(folder, { recursive: true });
const png = new Canvas(180, 270, [50 + index * 40, 60, 120]).png();
fs.writeFileSync(path.join(folder, '001.png'), png); fs.writeFileSync(path.join(library, name, 'cover.png'), png);
fs.writeFileSync(path.join(library, name, 'manga.json'), JSON.stringify({ title: name, status: index ? 'completed' : 'ongoing', tags: ['Fantasy', 'Demo'] }));
}
const env = { ...process.env, DATA: data, LIBRARY: library, PORT: '8180', SCRAPER_PORT: '8191', SENTINEL_PORT: '8192',
READER_URL: 'http://127.0.0.1:8180', SCRAPER_URL: 'http://127.0.0.1:8191', SENTINEL_URL: 'http://127.0.0.1:8192',
SCRAPER_DATA: path.join(data, 'scraper'), SENTINEL_DATA: path.join(data, 'sentinel'), COOKIE_SECURE: '0', SCAN_INTERVAL: '0', SCRAPER_TOKEN: 'fixture', SCRAPER_TOKEN_FILE: '' };
const child = spawn(process.execPath, ['tools/start-all.js'], { cwd: path.resolve(__dirname, '..'), env, stdio: 'inherit', windowsHide: true });
console.log('Isolated UI fixture at http://127.0.0.1:8180 · first account is test admin');
for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, () => child.kill('SIGTERM'));
child.once('exit', code => { fs.rmSync(dir, { recursive: true, force: true }); process.exit(code || 0); });
+120
View File
@@ -0,0 +1,120 @@
// Local-only research benchmark. Reads completed library files; writes no library state.
const fs = require('node:fs');
const path = require('node:path');
const zlib = require('node:zlib');
const crypto = require('node:crypto');
const library = path.resolve(__dirname, '..', 'library');
const series = [
'Solo Leveling',
'Gosick',
'The Martial God Who Regressed Back to Level 2',
'The Exiled Heavy Knight Knows How to Game the System',
];
const extensions = new Set(['.jpg', '.jpeg', '.png', '.webp', '.avif']);
const codecs = [
{ name: 'gzip-6', encode: (b) => zlib.gzipSync(b, { level: 6 }), decode: zlib.gunzipSync },
{ name: 'brotli-4', encode: (b) => zlib.brotliCompressSync(b, { params: { [zlib.constants.BROTLI_PARAM_QUALITY]: 4 } }), decode: zlib.brotliDecompressSync },
{ name: 'zstd-3', encode: (b) => zlib.zstdCompressSync(b, { params: { [zlib.constants.ZSTD_c_compressionLevel]: 3 } }), decode: zlib.zstdDecompressSync },
{ name: 'zstd-10', encode: (b) => zlib.zstdCompressSync(b, { params: { [zlib.constants.ZSTD_c_compressionLevel]: 10 } }), decode: zlib.zstdDecompressSync },
];
function firstCompleteChapter(manga) {
const root = path.join(library, manga);
for (const volume of fs.readdirSync(root, { withFileTypes: true }).filter((x) => x.isDirectory()).sort((a, b) => a.name.localeCompare(b.name))) {
const volumePath = path.join(root, volume.name);
for (const chapter of fs.readdirSync(volumePath, { withFileTypes: true }).filter((x) => x.isDirectory()).sort((a, b) => a.name.localeCompare(b.name))) {
const chapterPath = path.join(volumePath, chapter.name);
if (!fs.existsSync(path.join(chapterPath, 'chapter.json'))) continue;
const images = fs.readdirSync(chapterPath).filter((f) => extensions.has(path.extname(f).toLowerCase())).sort();
if (images.length) return { chapterPath, images };
}
}
throw new Error(`No complete chapter for ${manga}`);
}
function measure(fn) {
const begin = process.hrtime.bigint();
const result = fn();
return { result, ms: Number(process.hrtime.bigint() - begin) / 1e6 };
}
const chapters = series.map((manga) => ({ manga, ...firstCompleteChapter(manga) }));
const totals = Object.fromEntries(codecs.map((codec) => [codec.name, { bytes: 0, encodeMs: 0, decodeMs: 0 }]));
const byChapter = [];
const seenHashes = new Set();
let rawBytes = 0;
let imageCount = 0;
let duplicateBytes = 0;
let duplicateCount = 0;
const formats = {};
let metadataRaw = 0;
let metadataZstd = 0;
const adaptive = { sampleBytes: 32768, threshold: 0.03, transmittedBytes: 0, sampleMs: 0, selectedEncodeMs: 0, selectedDecodeMs: 0, compressedImages: 0, missedUsefulImages: 0, wastedImages: 0 };
for (const chapter of chapters) {
const row = { manga: chapter.manga, chapter: path.relative(library, chapter.chapterPath), images: chapter.images.length, rawBytes: 0, formats: {}, codecs: {} };
const buffers = [];
for (const file of chapter.images) {
const buffer = fs.readFileSync(path.join(chapter.chapterPath, file));
buffers.push(buffer);
imageCount++;
rawBytes += buffer.length;
row.rawBytes += buffer.length;
const ext = path.extname(file).toLowerCase();
formats[ext] = (formats[ext] || 0) + buffer.length;
row.formats[ext] = (row.formats[ext] || 0) + 1;
const hash = crypto.createHash('sha256').update(buffer).digest('hex');
if (seenHashes.has(hash)) { duplicateCount++; duplicateBytes += buffer.length; }
else seenHashes.add(hash);
const sample = buffer.subarray(0, Math.min(buffer.length, adaptive.sampleBytes));
const sampleEncoded = measure(() => codecs[2].encode(sample));
adaptive.sampleMs += sampleEncoded.ms;
const shouldCompress = sampleEncoded.result.length <= sample.length * (1 - adaptive.threshold);
for (const codec of codecs) {
const encoded = measure(() => codec.encode(buffer));
const decoded = measure(() => codec.decode(encoded.result));
if (!decoded.result.equals(buffer)) throw new Error(`Round trip failed: ${codec.name} ${file}`);
const total = totals[codec.name];
total.bytes += encoded.result.length;
total.encodeMs += encoded.ms;
total.decodeMs += decoded.ms;
const cell = row.codecs[codec.name] ||= { bytes: 0, encodeMs: 0, decodeMs: 0 };
cell.bytes += encoded.result.length;
cell.encodeMs += encoded.ms;
cell.decodeMs += decoded.ms;
if (codec.name === 'zstd-3') {
if (shouldCompress) {
adaptive.compressedImages++;
adaptive.transmittedBytes += encoded.result.length;
adaptive.selectedEncodeMs += encoded.ms;
adaptive.selectedDecodeMs += decoded.ms;
if (encoded.result.length >= buffer.length) adaptive.wastedImages++;
} else {
adaptive.transmittedBytes += buffer.length;
if (encoded.result.length <= buffer.length * 0.97) adaptive.missedUsefulImages++;
}
}
}
}
// Packing a chapter lets zstd exploit any repetition across page boundaries.
const packed = Buffer.concat(buffers);
const chapterEncoded = measure(() => codecs[2].encode(packed));
const chapterDecoded = measure(() => codecs[2].decode(chapterEncoded.result));
if (!chapterDecoded.result.equals(packed)) throw new Error(`Packed round trip failed: ${chapter.manga}`);
row.packedZstd3 = { bytes: chapterEncoded.result.length, encodeMs: chapterEncoded.ms, decodeMs: chapterDecoded.ms };
const metadata = fs.readFileSync(path.join(chapter.chapterPath, 'chapter.json'));
metadataRaw += metadata.length;
metadataZstd += codecs[2].encode(metadata).length;
byChapter.push(row);
}
console.log(JSON.stringify({
runtime: process.version,
platform: `${process.platform}/${process.arch}`,
sample: { imageCount, rawBytes, formats, duplicateCount, duplicateBytes, metadataRaw, metadataZstd },
codecs: totals,
adaptive,
chapters: byChapter,
method: 'One cold file read per image; encode/decode timed separately in-process using synchronous Node zlib. No network, TLS, disk-write, or browser decode time. Sizes omit protocol headers. Exact-byte round-trip verified.',
}, null, 2));
+19
View File
@@ -0,0 +1,19 @@
#!/usr/bin/env node
'use strict';
const { secret } = require('../runtime/config');
const token = secret('SCRAPER_TOKEN');
const root = process.env.READER_URL || 'http://127.0.0.1:8080';
const scraper = process.env.SCRAPER_URL || 'http://127.0.0.1:8091';
const sentinel = process.env.SENTINEL_URL || 'http://127.0.0.1:8092';
(async () => {
for (const [name, base, endpoint] of [['reader', root, '/health'], ['scraper', scraper, '/api/state'], ['sentinel', sentinel, '/api/state']]) {
try {
const response = await fetch(base + endpoint, { headers: token ? { authorization: `Bearer ${token}` } : {}, signal: AbortSignal.timeout(10000) });
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const state = await response.json();
console.log(JSON.stringify({ service: name, reachable: true, status: state.status || (state.ok ? 'healthy' : 'unhealthy'),
active: state.active?.map(e => ({ source: e.source, title: e.current?.title || e.title, completed: e.job?.completed ?? e.completed, total: e.job?.total ?? e.total })),
openIncidents: state.incidents?.filter(i => i.state === 'open').length, sourcePaused: state.sourcePaused }));
} catch (error) { console.log(JSON.stringify({ service: name, reachable: false, error: error.message })); process.exitCode = 1; }
}
})();
+37
View File
@@ -0,0 +1,37 @@
#!/usr/bin/env node
'use strict';
// Run on the host. Authentication stays inside the reader container; no token
// appears in command arguments, terminal output or a published daemon port.
const { spawnSync } = require('node:child_process');
const path = require('node:path');
const action = process.argv[2] || 'check';
const id = process.argv[3];
if (!['check', 'state', 'pause', 'resume', 'retry'].includes(action) ||
(action === 'retry' && !id) || (id && !/^[a-zA-Z0-9-]{1,80}$/.test(id))) {
console.error('Usage: node tools/compose-control.js check|state|pause|resume|retry <job-id>');
process.exit(2);
}
const script = action === 'check' ? "require('./tools/check.js')" : `
const {secret}=require('./runtime/config');
(async()=>{
const action=${JSON.stringify(action)},id=${JSON.stringify(id || null)};
const response=await fetch(process.env.SCRAPER_URL+(action==='state'?'/api/state':'/api/control'),{
method:action==='state'?'GET':'POST',signal:AbortSignal.timeout(20000),
headers:{authorization:'Bearer '+secret('SCRAPER_TOKEN'),'content-type':'application/json'},
...(action==='state'?{}:{body:JSON.stringify({action,id})})
});
if(!response.ok)throw new Error('Scraper returned HTTP '+response.status);
const s=await response.json();
const job=j=>j?{updated:j.updated,completed:j.completed,total:j.total,current:j.current,lastChapter:j.lastChapter}:null;
console.log(JSON.stringify({version:s.version,status:s.status,paused:s.paused,settings:s.settings,
sourcePaused:s.sourcePaused,publishedRevision:s.publishedRevision,
active:s.active?.map(a=>({source:a.source,id:a.current?.id,title:a.current?.title,job:job(a.job)})),
queue:s.queue?.map(q=>({id:q.id,title:q.title,source:q.source,status:q.status,failure:q.failure, nextAttemptAt:q.nextAttemptAt})),
openIncidents:s.incidents?.filter(i=>i.state==='open').map(i=>({kind:i.kind,title:i.title,message:i.message})),
runtimeHeartbeat:s.runtimeHeartbeat,nextDiscoveryAt:s.nextDiscoveryAt,libraryStatus:s.libraryStatus},null,2));
})().catch(e=>{console.error(e.message);process.exitCode=1;});`;
const result = spawnSync('docker', ['compose', 'exec', '-T', 'reader', 'node', '-e', script], {
cwd: path.resolve(__dirname, '..'), encoding: 'utf8', stdio: 'inherit', windowsHide: true,
});
if (result.error) console.error(result.error.message);
process.exit(result.status ?? 1);
+70
View File
@@ -0,0 +1,70 @@
'use strict';
// Optional, manually run metadata pass. The reader never contacts AniList at runtime.
// This intentionally only enriches an exact first-serialization-year match; a sequel,
// remake, or similarly named work is worse than a missing cover.
const fs = require('fs');
const path = require('path');
const { normalize } = require('../greatest');
const file = path.join(__dirname, '..', 'greatest', 'catalog.json');
const catalog = JSON.parse(fs.readFileSync(file, 'utf8'));
const wait = ms => new Promise(resolve => setTimeout(resolve, ms));
const query = `query ($search: String) {
Page(perPage: 10) {
media(search: $search, type: MANGA, format: MANGA, sort: POPULARITY_DESC) {
id title { english romaji native } startDate { year } chapters status siteUrl
coverImage { large }
}
}
}`;
const aliases = {
'kitaro-gegege-no-kitaro-1960': 'Hakaba Kitarou',
'the-legend-of-kamui-kamui-den-1964': 'Kamui Den',
'the-poem-of-wind-and-trees-1976': 'Kaze to Ki no Uta',
'the-heart-of-thomas-1974': 'Thomas no Shinzou',
'gto-great-teacher-onizuka-1997': 'Great Teacher Onizuka',
'someone-hertz-2025': 'Samawan Hertz',
'home-at-the-horizon-2025': 'Okaeri Suiheisen',
'war-of-the-adults-2025': 'Otona Taisen',
'frieren-beyond-journey-s-end-2020': 'Sousou no Frieren',
};
async function search(title) {
for (let attempt = 0; attempt < 3; attempt++) {
const response = await fetch('https://graphql.anilist.co', {
method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ query, variables: { search: title } }), signal: AbortSignal.timeout(12000),
});
if (response.status === 429 || response.status >= 500) { await wait(10000 * (attempt + 1)); continue; }
const body = await response.json();
return body.data?.Page?.media || [];
}
return [];
}
(async () => {
let matched = 0;
for (const [index, entry] of catalog.entries.entries()) {
const title = aliases[entry.id] || entry.title.split(' / ')[0].split(': ')[0];
try {
const results = await search(title);
const target = normalize(title);
const match = results.find(item => item.startDate?.year === entry.year &&
[item.title?.english, item.title?.romaji, item.title?.native].some(name => normalize(name) === target)) ||
results.find(item => item.startDate?.year === entry.year &&
[item.title?.english, item.title?.romaji].some(name => normalize(name).includes(target) || target.includes(normalize(name))));
if (match) {
entry.cover = /^https:\/\//.test(match.coverImage?.large || '') ? match.coverImage.large : null;
entry.chapters = Number.isInteger(match.chapters) && match.chapters > 0 ? match.chapters : null;
entry.status = match.status === 'RELEASING' ? 'ongoing' : match.status === 'FINISHED' ? 'completed' : 'unknown';
entry.metadata = { provider: 'AniList', id: match.id, url: match.siteUrl };
matched++;
} else process.stderr.write(`No exact year/title match: ${entry.title} (${entry.year})\n`);
} catch (error) { process.stderr.write(`Metadata unavailable: ${entry.title}: ${error.message}\n`); }
if ((index + 1) % 10 === 0) process.stderr.write(`Checked ${index + 1}/${catalog.entries.length}; matched ${matched}\n`);
await wait(1500);
}
fs.writeFileSync(file + '.tmp', JSON.stringify(catalog, null, 2) + '\n');
fs.renameSync(file + '.tmp', file);
process.stderr.write(`Saved ${matched}/${catalog.entries.length} exact matches\n`);
})().catch(error => { console.error(error); process.exitCode = 1; });
+8
View File
@@ -0,0 +1,8 @@
'use strict';
const https = require('node:https');
const req = https.get({ host: '127.0.0.1', port: Number(process.env.PEER_PORT || 8093), path: '/health', rejectUnauthorized: false,
timeout: 4000 }, res => {
let text = ''; res.on('data', chunk => { text += chunk; if (text.length > 1000) req.destroy(); });
res.on('end', () => { try { if (res.statusCode !== 200 || !JSON.parse(text).ok) process.exitCode = 1; } catch { process.exitCode = 1; } });
});
req.on('timeout', () => req.destroy()); req.on('error', () => { process.exitCode = 1; });
+5
View File
@@ -0,0 +1,5 @@
'use strict';
const ports = { reader: process.env.PORT || 8080, scraper: process.env.SCRAPER_PORT || 8091, sentinel: process.env.SENTINEL_PORT || 8092 };
const service = process.argv[2] || 'reader';
fetch(`http://127.0.0.1:${ports[service]}/health`, { signal: AbortSignal.timeout(4000) })
.then(async r => { const body = await r.json(); if (!r.ok || !body.ok) process.exitCode = 1; }).catch(() => { process.exitCode = 1; });
+9
View File
@@ -0,0 +1,9 @@
#!/usr/bin/env node
'use strict';
const fs = require('fs'), path = require('path'), crypto = require('crypto');
const root = path.resolve(__dirname, '..');
for (const relative of ['data', 'data/scraper', 'data/sentinel', 'library', '.secrets']) fs.mkdirSync(path.join(root, relative), { recursive: true });
const file = path.join(root, '.secrets', 'scraper_token');
if (!fs.existsSync(file)) fs.writeFileSync(file, crypto.randomBytes(32).toString('hex') + '\n', { flag: 'wx', mode: 0o600 });
if (!fs.existsSync(path.join(root, '.env'))) fs.copyFileSync(path.join(root, '.env.example'), path.join(root, '.env'));
console.log('Compose directories and private token are ready. Stop existing writers before using these paths in containers.');
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env node
'use strict';
// Reviewed, one-series migration for ambiguous pre-provenance metadata. Dry run
// by default. Apply only during a stopped-writer cutover with a state snapshot.
const fs = require('node:fs').promises;
const path = require('node:path');
const { imageSize } = require('../library');
const { detectVerticalStrip, KEY } = require('../reading-direction');
const { writeJsonAtomic } = require('../scraper/lib/publisher');
const [folder, expectedId, ...flags] = process.argv.slice(2);
const apply = flags.includes('--apply');
const mustInclude = flags.find(flag => flag.startsWith('--chapter='))?.slice('--chapter='.length);
if (!folder || !/^[A-Z0-9]{20,40}$/.test(expectedId || '') ||
(mustInclude && !/^c\d{1,6}$/i.test(mustInclude)) ||
flags.some(flag => !['--apply', '--writers-stopped'].includes(flag) && !/^--chapter=c\d{1,6}$/i.test(flag)) ||
(apply && !flags.includes('--writers-stopped'))) {
console.error('Usage: node tools/review-reading-direction.js <library-manga-folder> <weebcentral-id> [--chapter=c0056] [--apply --writers-stopped]');
process.exit(2);
}
async function main() {
const libraryRoot = path.resolve(process.env.LIBRARY || path.join(__dirname, '..', 'library'));
const mangaDir = path.resolve(folder);
if (path.dirname(mangaDir) !== libraryRoot) throw new Error('Manga folder must be directly inside the configured library');
const file = path.join(mangaDir, 'manga.json');
const original = await fs.readFile(file, 'utf8');
const meta = JSON.parse(original);
if (meta['x-scraper']?.ids?.weebcentral !== expectedId) throw new Error('Source identity did not match');
if (meta.direction !== 'rtl' || meta.directionOverride || meta[KEY]) throw new Error('Only an unmarked legacy RTL value can be reviewed');
const chapters = [];
for (const book of await fs.readdir(mangaDir, { withFileTypes: true })) {
if (!book.isDirectory() || !/^v\d/i.test(book.name)) continue;
for (const chapter of await fs.readdir(path.join(mangaDir, book.name), { withFileTypes: true })) {
if (chapter.isDirectory() && /^c\d/i.test(chapter.name)) chapters.push(path.join(mangaDir, book.name, chapter.name));
}
}
chapters.sort((a, b) => a.localeCompare(b, undefined, { numeric: true }));
if (chapters.length < 3) throw new Error('Too few chapters for reviewed migration');
const requiredChapter = mustInclude && chapters.find(item => path.basename(item).toLowerCase().startsWith(mustInclude.toLowerCase() + ' '));
if (mustInclude && !requiredChapter) throw new Error('Required chapter was not found');
const selected = [...new Set([chapters[0], chapters[Math.floor(chapters.length / 2)], chapters.at(-1),
requiredChapter].filter(Boolean))];
const results = [];
for (const dir of selected) {
const pages = [];
for (const entry of (await fs.readdir(dir)).sort()) {
if (!/\.(?:jpe?g|png|webp|gif|avif|bmp)$/i.test(entry)) continue;
const size = await imageSize(path.join(dir, entry)).catch(() => null);
const rotation = /_rot(90|180|270)(?:_|\.)/i.exec(entry);
pages.push({ w: size?.w || 0, h: size?.h || 0, r: rotation ? Number(rotation[1]) : 0 });
}
const detected = detectVerticalStrip(pages);
results.push({ chapter: path.basename(dir), detected: detected?.value || null,
evidence: detected?.evidence || { pages: pages.length } });
}
const supporting = results.filter(item => item.detected === 'ttb').length;
const reviewed = supporting >= Math.max(3, Math.ceil(results.length * .75)) &&
(!requiredChapter || results.some(item => item.chapter === path.basename(requiredChapter) && item.detected === 'ttb'));
if (apply) {
if (!reviewed) throw new Error('Insufficient consistent strip evidence; metadata unchanged');
if (await fs.readFile(file, 'utf8') !== original) throw new Error('Metadata changed during review');
await writeJsonAtomic(file, { ...meta, direction: 'ttb',
[KEY]: { version: 1, value: 'ttb', source: 'detected',
evidence: { reviewedChapters: results.map(item => item.chapter), supportingChapters: supporting } } });
}
console.log(JSON.stringify({ manga: path.basename(mangaDir), reviewed, applied: apply, samples: results }, null, 2));
}
main().catch(error => { console.error(error.message); process.exitCode = 1; });
+10
View File
@@ -0,0 +1,10 @@
$ErrorActionPreference = 'Stop'
$projectRoot = (Resolve-Path -LiteralPath (Join-Path $PSScriptRoot '..')).Path
$nodePath = (Get-Command node -ErrorAction Stop).Source
$outputLog = Join-Path $projectRoot 'data\scraper\supervisor.out.log'
$errorLog = Join-Path $projectRoot 'data\scraper\supervisor.err.log'
Set-Location -LiteralPath $projectRoot
& $nodePath 'tools/start-all.js' 1>> $outputLog 2>> $errorLog
exit $LASTEXITCODE
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env node
'use strict';
// Targeted recovery snapshot, not a library backup. Caller must prove all writers
// are stopped first. Never run this concurrently with the reader or scraper.
const fs = require('node:fs'), path = require('node:path'), crypto = require('node:crypto');
if (process.argv[2] !== '--writers-stopped') throw new Error('First verify that every writer is stopped, then pass --writers-stopped');
const root = path.resolve(__dirname, '..'), library = path.join(root, 'library'), data = path.join(root, 'data');
const stamp = new Date().toISOString().replace(/[-:]/g, '').replace(/\.\d+Z$/, 'Z');
const backup = path.join(data, 'scraper', 'backups', stamp + '-compose-cutover');
fs.mkdirSync(backup, { recursive: true });
const files = [];
function copy(source, relative) {
if (!fs.existsSync(source)) return;
if (fs.statSync(source).isDirectory()) {
for (const name of fs.readdirSync(source)) copy(path.join(source, name), path.join(relative, name));
return;
}
const target = path.join(backup, relative); fs.mkdirSync(path.dirname(target), { recursive: true });
fs.copyFileSync(source, target);
files.push({ path: relative, bytes: fs.statSync(source).size, sha256: crypto.createHash('sha256').update(fs.readFileSync(source)).digest('hex') });
}
for (const name of ['db.json', 'scan-cache.json', 'catalog.json', 'collection-covers', 'runtime-reader.json']) copy(path.join(data, name), path.join('data', name));
for (const name of ['queue.json', 'runtime.json', 'daemon.lock', 'supervisor.lock']) copy(path.join(data, 'scraper', name), path.join('data', 'scraper', name));
for (const name of ['state.json', 'runtime.json']) copy(path.join(data, 'sentinel', name), path.join('data', 'sentinel', name));
for (const name of fs.readdirSync(library)) {
copy(path.join(library, name, name.startsWith('.mcscraper-work-') ? 'job.json' : 'manga.json'), path.join('library', name, name.startsWith('.mcscraper-work-') ? 'job.json' : 'manga.json'));
}
fs.writeFileSync(path.join(backup, 'snapshot.json'), JSON.stringify({ created: new Date().toISOString(), kind: 'targeted-state-not-image-backup', files }, null, 2));
console.log(JSON.stringify({ backup, files: files.length, imageFilesCopied: false }));
+23
View File
@@ -0,0 +1,23 @@
#!/usr/bin/env node
'use strict';
const fs = require('fs'), path = require('path');
const { Supervisor } = require('../runtime/supervisor');
const root = path.resolve(__dirname, '..');
const lock = path.resolve(process.env.SCRAPER_DATA || path.join(process.env.DATA || path.join(root, 'data'), 'scraper'), 'supervisor.lock');
fs.mkdirSync(path.dirname(lock), { recursive: true });
if (fs.existsSync(lock)) {
const pid = Number(fs.readFileSync(lock, 'utf8'));
try { process.kill(pid, 0); throw new Error(`Supervisor already alive (PID ${pid})`); }
catch (e) { if (e.code !== 'ESRCH') throw e; fs.unlinkSync(lock); }
}
fs.writeFileSync(lock, String(process.pid), { flag: 'wx' });
const supervisors = ['reader', 'scraper', 'sentinel'].map(name => new Supervisor(name, { root }));
supervisors.forEach(service => service.start());
const keepalive = setInterval(() => {}, 60000);
let closing = false;
for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, async () => {
if (closing) return; closing = true; clearInterval(keepalive);
await Promise.all(supervisors.map(service => service.close()));
fs.unlinkSync(lock); process.exit(0);
});
+243
View File
@@ -0,0 +1,243 @@
'use strict';
// Real browser, disposable Compose storage; never uses a production account/catalog.
const assert = require('node:assert/strict');
const fs = require('node:fs'), path = require('node:path');
module.exports = async function testCollectionsUI(url, cookie) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
const origin = 'http://manga-qa.test:' + new URL(url).port;
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
const [name, ...value] = cookie.split('=');
await context.addCookies([{ name, value: value.join('='), url: origin }]);
let delaySave = false, failSave = false, saveCount = 0;
// Proxy only this synthetic origin to the local disposable fixture. The browser
// still sees ordinary non-loopback HTTP, as it does on a Tailscale IP address.
await context.route(origin + '/**', async route => {
if (new URL(route.request().url()).pathname === '/api/admin/catalog' && route.request().method() === 'POST') {
saveCount++;
if (failSave) { failSave = false; return route.fulfill({ status: 503, contentType: 'application/json', body: JSON.stringify({ error: 'Fixture temporary save failure' }) }); }
if (delaySave) await new Promise(resolve => setTimeout(resolve, 600));
}
const response = await route.fetch({ url: url + route.request().url().slice(origin.length) });
await route.fulfill({ response });
});
const page = await context.newPage(), errors = [];
page.on('pageerror', e => errors.push(e.message));
const shots = path.resolve(__dirname, '..', 'data', 'qa', 'collections');
fs.mkdirSync(shots, { recursive: true });
const catalog = () => page.evaluate(() => fetch('/api/admin/catalog').then(r => r.json()));
const status = page.getByRole('status');
const save = async (button = 'Save collection') => {
await page.getByRole('button', { name: button, exact: true }).click();
await page.waitForFunction(() => document.querySelector('[role=status]')?.textContent === 'Saved');
};
const noOverflow = async () => assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth), 'Collections page has horizontal overflow');
try {
await page.goto(origin + '/collections.html');
await page.getByRole('button', { name: 'New collection', exact: true }).waitFor();
assert.equal(await page.evaluate(() => isSecureContext), false, 'QA must reproduce an insecure browser origin');
assert.equal(await page.evaluate(() => typeof crypto.randomUUID), 'undefined');
await page.getByRole('button', { name: 'New collection', exact: true }).click();
if (process.argv.includes('--expect-collections-bug')) {
await page.waitForTimeout(200);
assert.ok(errors.some(e => e.includes('randomUUID')), 'Expected to reproduce the creation failure');
assert.equal(await page.getByLabel('Collection title', { exact: true }).count(), 0);
console.log('REPRODUCED: New collection throws crypto.randomUUID is not a function on non-loopback HTTP');
return;
}
await page.getByLabel('Collection title', { exact: true }).waitFor();
await page.getByLabel('Collection title', { exact: true }).fill('Harbor Stories');
await page.getByLabel('Collection description', { exact: true }).fill('A QA reading collection');
assert.match(await status.textContent(), /Unsaved/);
// An unrelated metadata action must not silently wipe a collection draft.
await page.getByRole('button', { name: 'Import cached metadata suggestions', exact: true }).click();
assert.match(await status.textContent(), /Save your changes/);
assert.equal(await page.getByLabel('Collection title', { exact: true }).inputValue(), 'Harbor Stories');
delaySave = true;
const beforeSave = saveCount;
await page.getByRole('button', { name: 'Save collection', exact: true }).click();
assert.equal(await page.getByRole('button', { name: 'Save collection', exact: true }).isDisabled(), true);
await page.evaluate(() => document.querySelector('fieldset button.btn.primary').click());
await page.waitForFunction(() => document.querySelector('[role=status]')?.textContent === 'Saved');
assert.equal(saveCount, beforeSave + 1, 'Duplicate save issued while one was in flight');
delaySave = false;
let state = await catalog();
const created = state.collections.find(c => c.title === 'Harbor Stories');
assert.ok(created); assert.match(created.id, /^[a-f0-9-]{36}$/);
const harborId = state.mangas.find(m => m.title === 'Iron Harbor').id;
const rainId = state.mangas.find(m => m.title === 'Before the Rain').id;
const moonId = state.mangas.find(m => m.title === 'Moon Garden').id;
assert.equal(created.members[0].mangaId, harborId);
await page.reload();
await page.getByLabel('Choose collection', { exact: true }).selectOption(created.id);
assert.equal(await page.getByLabel('Collection title', { exact: true }).inputValue(), 'Harbor Stories');
assert.equal(await page.getByLabel('Collection description', { exact: true }).inputValue(), 'A QA reading collection');
await page.getByRole('button', { name: 'New collection', exact: true }).click();
assert.match(await status.textContent(), /All manga are already grouped/);
assert.equal(await page.locator('[data-member] button').isDisabled(), true, 'Last member must not be removable');
// Server validation must be visible, recoverable, and retain the draft.
await page.getByLabel('Collection title', { exact: true }).fill('');
await page.getByRole('button', { name: 'Save collection', exact: true }).click();
await page.getByText('Collection title is required', { exact: false }).waitFor();
assert.equal(await page.getByRole('button', { name: 'Save collection', exact: true }).isEnabled(), true);
await page.getByLabel('Collection title', { exact: true }).fill('Harbor Stories');
await save();
// Edit/reorder and remove a member from the original fixture collection.
await page.getByLabel('Choose collection', { exact: true }).selectOption('moon-garden');
const member = page.locator('[data-member]').filter({ hasText: 'Before the Rain' });
await member.getByLabel('Member role').selectOption('side-story');
await member.getByLabel('Reading order').fill('0');
await page.getByLabel('Cover mode', { exact: true }).selectOption('selected');
await page.getByLabel('Cover manga', { exact: true }).selectOption(rainId);
await save();
state = await catalog();
assert.equal(state.collections.find(c => c.id === 'moon-garden').members[0].mangaId, rainId);
assert.equal(state.collections.find(c => c.id === 'moon-garden').coverMangaId, rainId);
await member.getByRole('button', { name: 'Remove', exact: true }).click();
await save();
state = await catalog();
assert.equal(state.collections.find(c => c.id === 'moon-garden').coverMangaId, moonId, 'Removing the cover member must repair the cover selection');
await page.getByLabel('Choose collection', { exact: true }).selectOption(created.id);
await page.getByLabel('Add manga to collection', { exact: true }).selectOption(rainId);
await page.getByLabel('Cover mode', { exact: true }).selectOption('mosaic');
await save();
state = await catalog();
assert.equal(state.collections.find(c => c.id === created.id).members.length, 2);
assert.equal(new Set(state.collections.flatMap(c => c.members.map(m => m.mangaId))).size, 3);
// Simulate a second admin's revision. A stale write must not overwrite it.
const other = await fetch(url + '/api/admin/catalog', { method: 'POST', headers: { cookie, 'content-type': 'application/json' },
body: JSON.stringify({ ...state, collections: state.collections.map(c => c.id === created.id ? { ...c, description: 'Changed by another editor' } : c) }) });
assert.equal(other.status, 200);
await page.getByLabel('Collection title', { exact: true }).fill('My unsaved title');
await page.getByRole('button', { name: 'Save collection', exact: true }).click();
await page.getByText('Catalog changed; reload before saving', { exact: false }).waitFor();
assert.equal(await page.getByLabel('Collection title', { exact: true }).inputValue(), 'My unsaved title');
page.once('dialog', d => d.dismiss());
await page.getByRole('button', { name: 'Reload catalog', exact: true }).click();
assert.equal(await page.getByLabel('Collection title', { exact: true }).inputValue(), 'My unsaved title');
page.once('dialog', d => d.accept());
await page.getByRole('button', { name: 'Reload catalog', exact: true }).click();
await page.getByText('Catalog reloaded', { exact: true }).waitFor();
assert.equal(await page.getByLabel('Collection description', { exact: true }).inputValue(), 'Changed by another editor');
// Network/server failure is visible and retryable without losing edits.
await page.getByLabel('Collection description', { exact: true }).fill('Retry kept my draft');
failSave = true;
await page.getByRole('button', { name: 'Save collection', exact: true }).click();
await page.getByText('Fixture temporary save failure', { exact: false }).waitFor();
assert.equal(await page.getByLabel('Collection description', { exact: true }).inputValue(), 'Retry kept my draft');
await save('Save all changes');
// Custom cover: real image encoding, backend validation, persisted selection.
const png = new (require('./png').Canvas)(80, 120, [140, 60, 90]).png();
await page.getByLabel('Custom cover', { exact: true }).setInputFiles({ name: 'qa.png', mimeType: 'image/png', buffer: png });
await page.getByText('Cover saved', { exact: true }).waitFor();
state = await catalog();
const cover = state.collections.find(c => c.id === created.id).customCover;
assert.ok(cover); assert.equal(state.collections.find(c => c.id === created.id).coverMode, 'custom');
assert.equal(await page.getByLabel('Cover mode', { exact: true }).inputValue(), 'custom');
assert.equal(await page.evaluate(id => fetch('/api/collection-cover?id=' + id).then(r => r.status), created.id), 200);
await page.getByLabel('Custom cover', { exact: true }).setInputFiles({ name: 'large.png', mimeType: 'image/png', buffer: Buffer.alloc(600001) });
await page.getByText('Choose an image under 600 KB.', { exact: true }).waitFor();
assert.equal((await catalog()).collections.find(c => c.id === created.id).customCover, cover);
// Relationships: create, reject, duplicate prevention, remove, cached import.
await page.getByLabel('Relationship source', { exact: true }).selectOption(harborId);
await page.getByLabel('Relationship target', { exact: true }).selectOption(moonId);
await page.getByLabel('Relationship type', { exact: true }).selectOption('spin-off');
await save('Add relationship');
const relation = page.locator('[data-relation]').filter({ hasText: 'spin-off' });
await relation.getByLabel('Relationship state', { exact: true }).selectOption('rejected');
await page.waitForFunction(() => document.querySelector('[role=status]')?.textContent === 'Saved');
assert.equal((await catalog()).relations.find(r => r.type === 'spin-off').state, 'rejected');
await page.getByLabel('Relationship source', { exact: true }).selectOption(harborId);
await page.getByLabel('Relationship target', { exact: true }).selectOption(moonId);
await page.getByLabel('Relationship type', { exact: true }).selectOption('spin-off');
const beforeDuplicate = saveCount;
await page.getByRole('button', { name: 'Add relationship', exact: true }).click();
assert.match(await status.textContent(), /already exists/); assert.equal(saveCount, beforeDuplicate);
await relation.getByRole('button', { name: 'Remove', exact: true }).click();
await page.waitForFunction(() => document.querySelector('[role=status]')?.textContent === 'Saved');
assert.ok(!(await catalog()).relations.some(r => r.type === 'spin-off'));
await save('Import cached metadata suggestions');
await page.getByRole('button', { name: 'Research relationships', exact: true }).click();
await page.getByText('Assign a verified AniList identity first', { exact: true }).waitFor();
await page.waitForFunction(() => !document.querySelector('fieldset').disabled);
assert.equal(await page.getByRole('button', { name: 'Research relationships', exact: true }).isEnabled(), true);
// Advanced JSON validation + round trip.
await page.getByText('Advanced · JSON import/export & agent research packet', { exact: true }).click();
await page.getByLabel('Catalog JSON', { exact: true }).fill('{');
await page.getByRole('button', { name: 'Validate & save JSON', exact: true }).click();
assert.equal(await page.getByLabel('Catalog JSON', { exact: true }).inputValue(), '{');
await page.getByLabel('Choose collection', { exact: true }).selectOption('moon-garden');
await page.getByLabel('Choose collection', { exact: true }).selectOption(created.id);
await page.getByText('Advanced · JSON import/export & agent research packet', { exact: true }).click();
assert.equal(await page.getByLabel('Catalog JSON', { exact: true }).inputValue(), '{', 'Changing collection discarded the JSON draft');
const beforeJSON = saveCount;
await page.getByRole('button', { name: 'Save all changes', exact: true }).click();
assert.match(await status.textContent(), /Validate & save JSON first/);
assert.equal(saveCount, beforeJSON, 'Ordinary save silently overwrote the JSON draft');
await page.getByLabel('Catalog JSON', { exact: true }).fill(JSON.stringify(await catalog()));
await save('Validate & save JSON');
await noOverflow();
await page.evaluate(() => scrollTo(0, 0));
await page.screenshot({ path: path.join(shots, 'desktop-editor.png'), fullPage: true });
// Mobile can edit + save too, not merely display the editor.
await page.setViewportSize({ width: 390, height: 844 });
await page.getByLabel('Collection description', { exact: true }).fill('Saved from mobile');
await save(); await noOverflow();
for (const row of await page.locator('[data-member]').all()) {
const text = await row.locator('span').first().boundingBox();
assert.ok(text.width > 150, 'Mobile member title is squeezed between controls');
}
await page.evaluate(() => scrollTo(0, 0));
await page.screenshot({ path: path.join(shots, 'mobile-editor.png'), fullPage: true });
// Reader presentation: grouped tiles, custom cover, order, expand/individual.
await page.goto(origin + '/');
await page.getByRole('link', { name: 'Harbor Stories', exact: false }).waitFor();
assert.equal(await page.locator('.tile').count(), 2);
await page.getByRole('link', { name: 'Harbor Stories', exact: false }).click();
await page.getByRole('heading', { name: 'Harbor Stories', exact: true }).waitFor();
assert.equal(await page.locator('.collection-member').count(), 2);
assert.match(await page.locator('.collection-member').first().textContent(), /Iron Harbor/);
assert.equal(await page.locator('.hero img').evaluate(img => img.complete && img.naturalWidth > 0), true);
await page.goto(origin + '/');
await page.getByLabel('Collection display', { exact: true }).selectOption('individual');
assert.equal(await page.locator('.tile').count(), 3);
await page.getByLabel('Collection display', { exact: true }).selectOption('expanded');
assert.equal(await page.locator('.tile').count(), 3);
assert.equal(await page.locator('.collection-heading').count(), 2);
// Group flag and deletion never delete manga or reading content.
await page.goto(origin + '/collections.html?id=' + created.id);
await page.getByLabel('Group on the dashboard', { exact: true }).uncheck(); await save();
assert.equal((await catalog()).collections.find(c => c.id === created.id).grouped, false);
page.once('dialog', d => d.dismiss());
await page.getByRole('button', { name: 'Delete collection', exact: true }).click();
assert.ok((await catalog()).collections.some(c => c.id === created.id));
page.once('dialog', d => d.accept());
await save('Delete collection');
state = await catalog(); assert.ok(!state.collections.some(c => c.id === created.id));
assert.equal(state.mangas.length, 3); assert.equal(state.relations.length, 1);
await page.reload();
await page.getByLabel('Choose collection', { exact: true }).waitFor();
assert.equal(await page.getByLabel('Choose collection', { exact: true }).locator('option').count(), 2);
// A reader can see groups, but cannot retrieve/change the admin catalog.
const result = await page.evaluate(async () => {
const post = (p, b) => fetch(p, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(b) });
const created = await post('/api/admin/user', { name: 'collections-reader-qa', password: 'disposable-reader-test', admin: false });
if (!created.ok) return created.status;
await post('/api/logout', {});
return (await post('/api/login', { name: 'collections-reader-qa', password: 'disposable-reader-test' })).status;
});
assert.equal(result, 200);
assert.equal(await page.evaluate(() => fetch('/api/admin/catalog').then(r => r.status)), 403);
assert.equal(await page.evaluate(() => fetch('/api/admin/catalog', { method: 'POST', headers: { 'content-type': 'application/json' }, body: '{}' }).then(r => r.status)), 403);
await page.goto(origin + '/collections.html?qa=reader');
await page.getByRole('link', { name: 'Back to the library', exact: true }).waitFor();
assert.equal(await page.getByRole('button', { name: 'New collection', exact: true }).count(), 0);
assert.equal(await page.evaluate(() => fetch('/api/collections').then(r => r.status)), 200);
assert.deepEqual(errors, []);
console.log('PASS: collections create/edit/save/reload, members/order/covers, relationships, JSON, failure/conflict recovery, mobile, dashboard and deletion on non-loopback HTTP');
console.log('Screenshots: ' + shots);
} catch (e) {
await page.screenshot({ path: path.join(shots, 'failure.png'), fullPage: true }).catch(() => {});
throw e;
} finally { await browser.close(); }
};
+254
View File
@@ -0,0 +1,254 @@
#!/usr/bin/env node
'use strict';
// Isolated, repeatable container acceptance. Never mounts the real library/db/queue.
const fs = require('fs'), os = require('os'), path = require('path'), crypto = require('crypto');
const assert = require('assert/strict'), { spawn } = require('child_process');
const { Canvas } = require('./png');
const root = path.resolve(__dirname, '..'), dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-compose-'));
const project = 'mc-qa-' + crypto.randomBytes(3).toString('hex'), port = process.env.QA_PORT || '8180';
const library = path.join(dir, 'library'), data = path.join(dir, 'data'), tokenFile = path.join(dir, 'token');
const directionQa = process.argv.includes('--direction-ui');
const peerGatewayQa = process.argv.includes('--peer-gateway');
fs.mkdirSync(path.join(data, 'scraper'), { recursive: true }); fs.mkdirSync(path.join(data, 'sentinel'));
if (peerGatewayQa) {
const peerData = path.join(data, 'peering');
const peerUsage = path.join(data, 'peering-usage');
fs.mkdirSync(peerUsage, { recursive: true });
const { PeeringState } = require('../peering/state');
if (process.platform === 'win32' && !process.env.OPENSSL_BIN) process.env.OPENSSL_BIN = 'C:\\Program Files\\Git\\usr\\bin\\openssl.exe';
new PeeringState(peerData).generateIdentity('ComposeQA');
}
const pausedConfirmQa = process.argv.includes('--paused-confirm');
const readingGateQa = process.argv.includes('--reading-limit-ui');
if (readingGateQa) fs.writeFileSync(path.join(data, 'reading-quiz.json'), JSON.stringify({ version: 1, manga: {
'Moon Garden': [0, 1, 2].map(i => ({ id: `moon-${i}`, prompt: `Moon Garden question ${i}?`, answers: [`moonanswer${i}`], requires: ['1/1'], evidence: 'Disposable Compose fixture' })),
'Before the Rain': [0, 1, 2].map(i => ({ id: `rain-${i}`, prompt: `Before the Rain question ${i}?`, answers: [`rainanswer${i}`], requires: ['1/1'], evidence: 'Disposable Compose fixture' })),
} }));
fs.writeFileSync(path.join(data, 'scraper', 'queue.json'), JSON.stringify({ paused: true,
sourcePaused: pausedConfirmQa ? { weebcentral: true } : {},
queue: pausedConfirmQa ? [{ id: 'ready-fixture', source: 'weebcentral',
url: 'https://weebcentral.com/series/01J76XYAWYDTD62TB8GEBFTARZ/Fixture',
seriesId: '01J76XYAWYDTD62TB8GEBFTARZ', title: 'Fixture', sourceTitle: 'Fixture',
kind: 'inspection', status: 'ready' }] : [], settings: { autoUpdateEnabled: false } }));
fs.writeFileSync(tokenFile, crypto.randomBytes(32).toString('hex'), { mode: 0o600 });
const png = new Canvas(180, 270, [60, 90, 140]).png();
for (const [index, name] of ['Moon Garden', 'Before the Rain', 'Iron Harbor'].entries()) {
const folder = path.join(library, name, 'v01', 'c0001 The Beginning'); fs.mkdirSync(folder, { recursive: true });
fs.writeFileSync(path.join(folder, '0001.png'), png); fs.writeFileSync(path.join(library, name, 'cover.png'), png);
if (process.argv.includes('--reading-ui') && index === 0) {
for (let page = 2; page <= 20; page++) fs.writeFileSync(path.join(folder, String(page).padStart(4, '0') + '.png'), png);
}
if (directionQa && index === 0) {
fs.writeFileSync(path.join(folder, '0001.png'), new Canvas(270, 180, [60, 90, 140]).png());
for (let page = 2; page <= 4; page++) fs.writeFileSync(path.join(folder, String(page).padStart(4, '0') + '.png'), new Canvas(180, 1000, [60, 90, 140]).png());
}
fs.writeFileSync(path.join(library, name, 'manga.json'), JSON.stringify({ title: name,
direction: directionQa && index === 1 ? 'ltr' : process.argv.includes('--reading-ui') && index === 0 ? 'ttb' : 'rtl',
...(directionQa && index === 0 ? { 'x-reading-direction': { version: 1, value: 'rtl', source: 'fallback' } } : {}),
status: index ? 'completed' : 'ongoing', tags: ['Fantasy', 'Demo'] }));
}
const env = { ...process.env, LIBRARY_PATH: library, DATA_PATH: data, SCRAPER_DATA_PATH: path.join(data, 'scraper'), SENTINEL_DATA_PATH: path.join(data, 'sentinel'),
SCRAPER_TOKEN_PATH: tokenFile, READER_PORT: port, READER_BIND: '127.0.0.1', COOKIE_SECURE: '0', APP_IMAGE: process.env.APP_IMAGE || 'mangacircus:local',
READER_IMAGE: process.env.APP_IMAGE || 'mangacircus:local',
...(readingGateQa ? { READING_WINDOWS_JSON: '{"0":[],"1":[],"2":[],"3":[],"4":[],"5":[],"6":[]}',
READING_TIMEZONE: 'Europe/Berlin', READING_QUIZ_UNLOCK_MINUTES: '60', READING_QUIZ_CATALOG_PATH: '/data/reading-quiz.json' }
: { READING_WINDOWS_JSON: '', READING_QUIZ_CATALOG_PATH: '' }),
...(peerGatewayQa ? { PEER_DATA_PATH: path.join(data, 'peering'), PEER_USAGE_PATH: path.join(data, 'peering-usage'), PEER_PORT: process.env.QA_PEER_PORT || '8193' } : {}) };
const args = ['compose', '-p', project, '-f', path.join(root, 'docker-compose.yml'), ...(peerGatewayQa ? ['--profile', 'peering'] : [])];
const docker = (...command) => new Promise((resolve, reject) => {
const child = spawn('docker', [...args, ...command], { cwd: root, env, windowsHide: true, stdio: ['ignore', 'pipe', 'pipe'] });
let result = ''; child.stdout.on('data', chunk => result += chunk); child.stderr.on('data', chunk => process.stderr.write(chunk));
child.on('error', reject); child.once('exit', code => code ? reject(new Error(`Docker ${command[0]} failed (${code})`)) : resolve(result));
});
const pause = ms => new Promise(resolve => setTimeout(resolve, ms));
async function until(check, message, attempts = 100) {
for (let n = 0; n < attempts; n++) { try { if (await check()) return; } catch {} await pause(300); }
throw new Error(message);
}
let keep = false;
(async () => {
try {
await docker('config', '-q'); await docker('up', '-d', '--no-build');
const url = `http://127.0.0.1:${port}`;
await until(() => fetch(url + '/health').then(r => r.ok), 'Reader did not start');
if (peerGatewayQa) {
const https = require('node:https');
try {
await until(() => new Promise(resolve => https.get({ hostname: '127.0.0.1', port: Number(env.PEER_PORT), path: '/health', rejectUnauthorized: false },
response => { response.resume(); resolve(response.statusCode === 200); }).on('error', () => resolve(false))), 'Peer gateway did not start');
} catch (error) { console.error(await docker('logs', '--tail', '30', 'peer-gateway').catch(() => 'Gateway logs unavailable')); throw error; }
const gateway = await docker('ps', '--format', 'json');
assert.ok(gateway.includes('peer-gateway'), 'Peer gateway missing from isolated Compose');
}
const signup = await fetch(url + '/api/signup', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ name: 'qa', password: 'disposable-local-test' }) });
assert.equal(signup.status, 200); const cookie = signup.headers.get('set-cookie').split(';')[0];
const api = (endpoint, body) => fetch(url + endpoint, { headers: { cookie, 'content-type': 'application/json' }, ...(body ? { method: 'POST', body: JSON.stringify(body) } : {}) });
assert.equal((await fetch(url + '/api/reading-schedule')).status, 401);
const schedule = await (await api('/api/reading-schedule')).json();
assert.equal(schedule.enabled, readingGateQa);
if (readingGateQa) {
assert.equal(schedule.timezone, 'Europe/Berlin');
assert.deepEqual(schedule.days, Array.from({ length: 7 }, () => []));
}
let inventory;
await until(async () => { inventory = await (await api('/api/library')).json(); return inventory.mangas?.length === 3; }, 'Fixture inventory missing');
const byTitle = title => inventory.mangas.find(m => m.title === title).id;
if (readingGateQa) {
const manga = byTitle('Moon Garden'), other = byTitle('Before the Rain');
const chapter = (await (await api('/api/manga?m=' + encodeURIComponent(manga))).json()).chapters[0].id;
const chapterUrl = '/api/chapter?m=' + encodeURIComponent(manga) + '&c=' + encodeURIComponent(chapter);
const pageUrl = '/lib/' + encodeURIComponent(manga) + '/v01/c0001%20The%20Beginning/0001.png';
assert.equal((await api(chapterUrl)).status, 423, 'chapter JSON must be gated');
assert.equal((await api(pageUrl)).status, 423, 'chapter image must be gated');
assert.equal((await api('/lib/' + encodeURIComponent(manga) + '/cover.png')).status, 200);
assert.equal((await (await api('/api/reading-access?m=' + encodeURIComponent(manga))).json()).quizAvailable, false);
for (const title of [manga, other]) {
const id = (await (await api('/api/manga?m=' + encodeURIComponent(title))).json()).chapters[0].id;
assert.equal((await api('/api/progress', { m: title, c: id, p: 0, n: 1, done: true })).status, 200);
}
assert.equal((await (await api('/api/reading-access?m=' + encodeURIComponent(manga))).json()).quizAvailable, true);
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: 'msedge', headless: true });
try {
const context = await browser.newContext();
const [name, ...values] = cookie.split('=');
await context.addCookies([{ name, value: values.join('='), url }]);
const page = await context.newPage();
await page.goto(url + '/read.html?m=' + encodeURIComponent(manga) + '&c=' + encodeURIComponent(chapter));
await page.getByRole('heading', { name: 'Reading is closed right now' }).waitFor();
await page.getByRole('button', { name: 'Start five-question challenge' }).click();
await page.getByText('Question 1 of 5').waitFor();
const firstPrompt = await page.locator('.reading-gate-question h2').textContent();
const firstTitle = await page.locator('.reading-gate-question .muted').textContent();
const firstNumber = /question (\d)/.exec(firstPrompt)[1];
const answerBox = page.getByRole('textbox', { name: 'Your answer' });
await answerBox.fill('wrong');
await page.getByRole('button', { name: 'Check answer' }).click();
const retryNotice = page.locator('.reading-gate-message[data-tone="retry"]');
await retryNotice.waitFor();
const retryStyle = await retryNotice.evaluate(el => ({ color: getComputedStyle(el).color,
border: getComputedStyle(el).borderTopWidth, background: getComputedStyle(el).backgroundColor }));
assert.notEqual(retryStyle.border, '0px');
await answerBox.fill((firstTitle === 'Moon Garden' ? 'moonanswer' : 'rainanswer') + firstNumber);
await page.getByRole('button', { name: 'Check answer' }).click();
const successNotice = page.locator('.reading-gate-message[data-tone="success"]');
await successNotice.waitFor();
const successStyle = await successNotice.evaluate(el => ({ color: getComputedStyle(el).color,
background: getComputedStyle(el).backgroundColor }));
assert.notEqual(successStyle.color, retryStyle.color);
assert.notEqual(successStyle.background, retryStyle.background);
await page.getByText('Question 2 of 5').waitFor();
await page.goto(url + '/manga.html?m=' + encodeURIComponent(manga));
await page.goto(url + '/read.html?m=' + encodeURIComponent(manga) + '&c=' + encodeURIComponent(chapter));
await page.getByRole('button', { name: 'Start five-question challenge' }).waitFor();
} finally { await browser.close(); }
if (!process.argv.includes('--keep')) {
let question = await (await api('/api/reading-quiz/start', {})).json();
const wrong = await api('/api/reading-quiz/answer', { challenge: question.challenge, answer: 'incorrect' });
assert.equal((await wrong.json()).correct, false);
for (let i = 0; i < 5; i++) {
const match = /(?:Moon Garden|Before the Rain) question (\d)/.exec(question.prompt);
const answer = (question.mangaTitle === 'Moon Garden' ? 'moonanswer' : 'rainanswer') + match[1];
question = await (await api('/api/reading-quiz/answer', { challenge: question.challenge, answer })).json();
}
assert.equal(question.complete, true);
assert.equal((await api(chapterUrl)).status, 200);
const image = await api(pageUrl);
assert.equal(image.status, 200);
assert.equal(image.headers.get('cache-control'), 'private, no-store');
console.log('PASS: schedule blocks chapter and page bytes, progress-gated mixed-manga quiz unlocks reading');
} else {
console.log('PASS: isolated playable gate stays locked with progress-qualified sample questions');
}
}
const catalog = { revision: 0, collections: [{ id: 'moon-garden', title: 'Moon Garden Universe', coverMode: 'mosaic', members: [
{ mangaId: byTitle('Moon Garden'), role: 'main', readingOrder: 1 }, { mangaId: byTitle('Before the Rain'), role: 'prequel', readingOrder: 2 }] }],
relations: [{ from: byTitle('Moon Garden'), to: byTitle('Before the Rain'), type: 'prequel', state: 'confirmed' }] };
assert.equal((await api('/api/admin/catalog', catalog)).status, 200);
assert.equal((await api('/api/collection?id=moon-garden')).status, 200);
let state = await (await api('/api/admin/scraper')).json(); assert.equal(state.available, true); assert.equal(state.paused, true); assert.equal(state.version, 4);
const cleanup = await api('/api/admin/scraper', { command: 'clear-completed' });
assert.equal(cleanup.status, 200);
assert.equal((await cleanup.json()).cleared, 0);
assert.ok(state.libraryStatus);
await until(async () => { state = await (await api('/api/admin/scraper')).json(); return state.libraryStatus?.ready && state.library.length === 3; }, 'Background scraper inventory missing');
if (pausedConfirmQa) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: 'msedge', headless: true });
try {
const context = await browser.newContext();
const [name, ...values] = cookie.split('=');
await context.addCookies([{ name, value: values.join('='), url }]);
const page = await context.newPage();
await page.goto(url + '/settings.html#admin/scraper');
const row = page.locator('[data-scraper-tab=activity] .scraper-item').filter({ hasText: 'Fixture' });
await row.getByRole('button', { name: 'Confirm download' }).click();
await row.getByText('Confirmed · waiting for source access review').waitFor();
state = await (await api('/api/admin/scraper')).json();
assert.equal(state.sourcePaused.weebcentral, true);
assert.equal(state.queue.find(item => item.id === 'ready-fixture').status, 'queued');
assert.equal(state.queue.find(item => item.id === 'ready-fixture').kind, 'download');
console.log('PASS: paused-source confirmation returns success and queues without source traffic');
} finally { await browser.close(); }
}
const settingsJs = await (await fetch(url + '/settings.js')).text();
assert.ok(!settingsJs.includes('Start it with: npm run start:scraper'));
await until(async () => { const h = await (await api('/api/admin/health')).json(); return h.sentinel.services?.reader && h.sentinel.services?.scraper; }, 'Internal Sentinel networking failed');
const mounts = JSON.parse(await docker('config', '--format', 'json'));
assert.ok(mounts.services.reader.volumes.find(v => v.target === '/library').read_only);
assert.ok(!mounts.services.scraper.ports?.length && !mounts.services.sentinel.ports?.length);
const denied = JSON.parse(await docker('exec', '-T', 'reader', 'node', '-e', "Promise.all(['http://scraper:8091/api/state','http://sentinel:8092/api/state'].map(u=>fetch(u).then(r=>r.status))).then(v=>console.log(JSON.stringify(v)))"));
assert.deepEqual(denied, [401, 401]);
// Deliberate fixture-only child crash: its supervisor must recover just the reader.
const before = JSON.parse(fs.readFileSync(path.join(data, 'runtime-reader.json'))).pid;
await docker('exec', '-T', 'reader', 'node', '-e', "const fs=require('fs');process.kill(JSON.parse(fs.readFileSync('/data/runtime-reader.json')).pid,'SIGKILL')");
await until(async () => { const runtime = JSON.parse(fs.readFileSync(path.join(data, 'runtime-reader.json'))); return runtime.pid !== before && runtime.status === 'running' && (await fetch(url + '/health')).ok; }, 'Reader child did not recover');
assert.equal(JSON.parse(fs.readFileSync(path.join(data, 'scraper', 'runtime.json'))).failures.length, 0);
const image = await api('/lib/' + encodeURIComponent('Moon Garden') + '/cover.png'); assert.equal(image.status, 200);
state = await (await api('/api/admin/scraper')).json(); assert.equal(state.paused, true);
if (process.argv.includes('--settings-ui')) await require('./test-settings-ui')(url, cookie);
if (process.argv.includes('--discovery-ui')) {
const login = await api('/api/login', { name: 'qa', password: 'disposable-local-test' });
assert.equal(login.status, 200);
await require('./test-discovery-ui')(url, login.headers.get('set-cookie').split(';')[0]);
}
if (process.argv.includes('--peer-ui')) {
const login = await api('/api/login', { name: 'qa', password: 'disposable-local-test' });
assert.equal(login.status, 200);
await require('./test-peer-ui')(url, login.headers.get('set-cookie').split(';')[0]);
}
if (process.argv.includes('--collections-ui')) {
// Each UI suite tests logout/ordinary-reader boundaries. Give the next suite
// a fresh disposable admin session rather than reusing a logged-out cookie.
const login = await api('/api/login', { name: 'qa', password: 'disposable-local-test' });
assert.equal(login.status, 200);
await require('./test-collections-ui')(url, login.headers.get('set-cookie').split(';')[0]);
}
if (process.argv.includes('--queue-ui')) {
const login = await api('/api/login', { name: 'qa', password: 'disposable-local-test' });
assert.equal(login.status, 200);
await require('./test-queue-ui')(url, login.headers.get('set-cookie').split(';')[0]);
}
if (process.argv.includes('--reading-ui')) {
const login = await api('/api/login', { name: 'qa', password: 'disposable-local-test' });
assert.equal(login.status, 200);
await require('./test-reading-ui')(url, login.headers.get('set-cookie').split(';')[0]);
}
if (directionQa) {
const login = await api('/api/login', { name: 'qa', password: 'disposable-local-test' });
assert.equal(login.status, 200);
await require('./test-direction-ui')(url, login.headers.get('set-cookie').split(';')[0]);
}
console.log('PASS: isolated Compose auth, collections, read-only reader, private APIs, Sentinel networking and child crash recovery');
keep = process.argv.includes('--keep');
if (keep) {
const envFile = path.join(dir, 'qa.env');
fs.writeFileSync(envFile, ['LIBRARY_PATH', 'DATA_PATH', 'SCRAPER_DATA_PATH', 'SENTINEL_DATA_PATH', 'SCRAPER_TOKEN_PATH', 'READER_PORT', 'READER_BIND', 'COOKIE_SECURE', 'APP_IMAGE',
...(readingGateQa ? ['READING_WINDOWS_JSON', 'READING_TIMEZONE', 'READING_QUIZ_UNLOCK_MINUTES', 'READING_QUIZ_CATALOG_PATH'] : [])]
.map(k => `${k}=${k === 'READING_WINDOWS_JSON' ? `'${env[k]}'` : env[k].replaceAll('\\', '/')}`).join('\n') + '\n');
console.log(JSON.stringify({ project, url, envFile, fixture: dir }));
}
} finally {
if (!keep) { await docker('down').catch(() => {}); fs.rmSync(dir, { recursive: true, force: true }); }
}
})().catch(error => { console.error(error.message); process.exitCode = 1; });
+34
View File
@@ -0,0 +1,34 @@
'use strict';
const assert = require('node:assert/strict');
module.exports = async function (url, cookie) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
const context = await browser.newContext({ viewport: { width: 390, height: 844 }, isMobile: true, hasTouch: true });
const [name, ...value] = cookie.split('=');
await context.addCookies([{ name, value: value.join('='), url }]);
try {
const api = async path => {
const response = await context.request.get(url + path);
assert.equal(response.ok(), true, await response.text()); return response.json();
};
const inventory = await api('/api/library');
const pick = async title => {
const manga = inventory.mangas.find(item => item.title === title);
const detail = await api('/api/manga?m=' + encodeURIComponent(manga.id));
const chapter = detail.chapters[0];
return { manga, chapter, data: await api('/api/chapter?m=' + encodeURIComponent(manga.id) + '&c=' + encodeURIComponent(chapter.id)) };
};
const strip = await pick('Moon Garden');
const printed = await pick('Before the Rain');
assert.equal(strip.data.direction, 'ttb');
assert.equal(printed.data.direction, 'ltr');
const page = await context.newPage();
await page.goto(url + '/read.html?m=' + encodeURIComponent(strip.manga.id) + '&c=' + encodeURIComponent(strip.chapter.id));
await page.waitForFunction(() => document.querySelectorAll('#canvas img').length === 4);
const widths = await page.locator('#canvas img').evaluateAll(images => images.map(img => Math.round(img.getBoundingClientRect().width)));
assert.equal(widths.length, 4);
assert.ok(widths.every(width => Math.abs(width - widths[1]) <= 1), `Flow widths differ: ${widths.join(', ')}`);
console.log('PASS: mobile flow gives the landscape credits card the same width as detected webtoon strips; printed manual LTR remains LTR');
} finally { await context.close(); await browser.close(); }
};
+42
View File
@@ -0,0 +1,42 @@
'use strict';
const assert = require('node:assert/strict');
module.exports = async function testDiscoveryUI(url, cookie) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
const [name, ...value] = cookie.split('=');
await context.addCookies([{ name, value: value.join('='), url }]);
const page = await context.newPage(), errors = [];
page.on('pageerror', error => errors.push(error.message));
try {
await page.goto(url + '/discover.html');
await page.getByRole('heading', { name: 'Discover', exact: true }).waitFor();
await page.getByText('Turn on personalized discovery').waitFor();
await page.getByLabel('Personalized discovery').check();
await page.getByRole('button', { name: 'Run discovery now' }).click();
await page.getByText(/suggestions/, { exact: false }).first().waitFor();
await page.waitForFunction(() => document.querySelectorAll('.discovery-card').length > 0);
await page.locator('.discovery-card').first().getByText('Why this is here · evidence').click();
assert.ok(await page.locator('.discovery-card').first().locator('details li').count() > 0);
await page.locator('.discovery-card').first().getByRole('button', { name: '👍 Maybe' }).click();
await page.getByText('Preference saved').waitFor();
assert.equal(errors.length, 0, `Browser errors: ${errors.join('; ')}`);
await page.setViewportSize({ width: 390, height: 844 });
const overflow = await page.evaluate(() => ({ width: innerWidth, scroll: document.documentElement.scrollWidth,
elements: [...document.querySelectorAll('body *')].filter(el => el.getBoundingClientRect().right > innerWidth + 1)
.slice(0, 8).map(el => `${el.tagName}.${el.className}: ${Math.round(el.getBoundingClientRect().right)}`) }));
assert.ok(overflow.scroll <= overflow.width, `Discover page overflows on mobile: ${JSON.stringify(overflow)}`);
const create = await page.evaluate(() => fetch('/api/admin/user', { method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: 'discovery-reader-qa', password: 'disposable-reader-test', admin: false }) }).then(response => response.status));
assert.equal(create, 200);
await page.evaluate(() => fetch('/api/logout', { method: 'POST', headers: { 'content-type': 'application/json' }, body: '{}' }));
await page.evaluate(() => fetch('/api/login', { method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: 'discovery-reader-qa', password: 'disposable-reader-test' }) }));
await page.reload();
await page.getByRole('heading', { name: 'Discover', exact: true }).waitFor();
assert.equal(await page.getByRole('button', { name: 'Run discovery now' }).count(), 0);
assert.equal(await page.evaluate(() => fetch('/api/admin/discovery').then(response => response.status)), 403);
assert.equal(errors.length, 0, `Browser errors: ${errors.join('; ')}`);
} finally { await browser.close(); }
};
+107
View File
@@ -0,0 +1,107 @@
'use strict';
// Disposable local reader + browser; no production accounts or scraper traffic.
const assert = require('node:assert/strict');
const fs = require('node:fs'), os = require('node:os'), path = require('node:path'), net = require('node:net');
const { spawn } = require('node:child_process');
const http = require('node:http');
const { chromium } = require('playwright');
const { loadCatalog } = require('../greatest');
const root = path.resolve(__dirname, '..');
const delay = ms => new Promise(resolve => setTimeout(resolve, ms));
async function freePort() {
const server = net.createServer();
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
const port = server.address().port;
await new Promise(resolve => server.close(resolve));
return port;
}
(async () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-greatest-ui-'));
const library = path.join(dir, 'library'), data = path.join(dir, 'data');
fs.mkdirSync(library); fs.mkdirSync(data);
const port = await freePort(), base = `http://127.0.0.1:${port}`;
const queue = [{ id: 'existing', title: 'Astro Boy', url: 'https://mangadex.org/title/ca4c84bb-7272-45aa-a22d-dc1282b52372',
source: 'mangadex', seriesId: 'ca4c84bb-7272-45aa-a22d-dc1282b52372', kind: 'download', status: 'queued' }];
const fakeScraper = http.createServer(async (req, res) => {
let result;
if (req.method === 'GET' && req.url === '/api/state') result = { queue };
else if (req.method === 'POST' && req.url === '/api/queue') {
let body = ''; for await (const chunk of req) body += chunk;
const input = JSON.parse(body);
queue.push({ id: 'added', title: input.title, url: input.url, source: 'weebcentral',
seriesId: String(input.url || '').split('/series/')[1]?.split('/')[0] || '',
kind: input.inspect ? 'inspection' : 'download', status: 'queued' });
result = { item: queue.at(-1) };
} else { res.writeHead(404); return res.end(); }
res.writeHead(200, { 'content-type': 'application/json' }); res.end(JSON.stringify(result));
});
await new Promise(resolve => fakeScraper.listen(0, '127.0.0.1', resolve));
const scraperPort = fakeScraper.address().port;
const child = spawn(process.execPath, ['server.js'], { cwd: root, windowsHide: true,
env: { ...process.env, PORT: String(port), LIBRARY: library, DATA: data, SCAN_INTERVAL: '0',
FULL_SCAN_INTERVAL: '0', SCRAPER_URL: `http://127.0.0.1:${scraperPort}`, COOKIE_SECURE: '0' }, stdio: 'ignore' });
let browser;
try {
for (let i = 0; i < 80; i++) {
if (await fetch(base + '/health').then(r => r.ok).catch(() => false)) break;
await delay(100);
}
assert.equal((await fetch(base + '/health')).status, 200);
const signup = await fetch(base + '/api/signup', { method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: 'curator', password: 'local-test-password' }) });
assert.equal(signup.status, 200);
const cookie = signup.headers.get('set-cookie').split(';')[0];
browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
const [name, value] = cookie.split('=');
await context.addCookies([{ name, value, url: base }]);
await context.route(/^https:\/\//, route => route.abort()); // Never load remote cover art in fixture.
const page = await context.newPage(), errors = [];
page.on('pageerror', error => errors.push(error.message));
await page.goto(base + '/greatest.html');
await page.getByRole('heading', { name: 'Shijō Saikō no Manga' }).waitFor();
assert.equal(await page.getByRole('link', { name: 'Shijō Saikō no Manga' }).textContent(), 'Shijō Saikō no Manga');
assert.equal(await page.locator('.greatest-period').count(), 13);
assert.equal(await page.locator('.greatest-tile').count(), 106);
const astro = page.locator('.greatest-tile').filter({ has: page.getByRole('button', { name: 'Details for Astro Boy' }) });
assert.equal(await astro.locator('.greatest-reason').isVisible(), true);
const astroReason = loadCatalog().entries.find(entry => entry.title === 'Astro Boy').note;
assert.match(await astro.locator('.greatest-reason').textContent(), /foundational postwar story manga/);
assert.equal(await astro.locator('.greatest-reason-label').textContent(), 'Why it matters');
await astro.getByRole('button', { name: 'Definitely want: Astro Boy' }).click();
await page.getByRole('button', { name: '🔥 Flame' }).click();
assert.equal(await page.locator('.greatest-tile').count(), 1);
await page.getByRole('button', { name: 'All', exact: true }).click();
await page.getByRole('button', { name: 'Rank by everyone' }).click();
await page.getByRole('button', { name: 'Details for Astro Boy' }).click();
await page.getByRole('dialog').getByRole('heading', { name: 'Why it matters' }).waitFor();
assert.ok((await page.getByRole('dialog').textContent()).includes(astroReason));
await page.getByRole('dialog').getByText('In scraper · queued').waitFor();
assert.equal(await page.getByRole('dialog').getByRole('button', { name: 'Already added' }).isDisabled(), true);
await page.getByRole('dialog').getByText('Community votes: 🔥 1').waitFor();
await page.getByRole('dialog').getByRole('button', { name: 'Close' }).click();
await page.getByRole('button', { name: 'Details for Princess Knight' }).click();
await page.getByRole('dialog').getByRole('textbox', { name: 'Series URL' })
.fill('https://weebcentral.com/series/01J76XY7E9FNDZ1DBBM6PBJPF0');
await page.getByRole('dialog').getByRole('button', { name: 'Add URL to scraper' }).click();
await page.getByRole('dialog').getByText('In scraper · queued').waitFor();
assert.equal(queue.at(-1).kind, 'inspection');
assert.equal(queue.at(-1).title, 'Princess Knight');
await page.getByRole('dialog').getByRole('button', { name: 'Close' }).click();
await page.setViewportSize({ width: 390, height: 844 });
assert.equal(await astro.locator('.greatest-reason').isVisible(), true);
assert.equal(await page.locator('.nav-greatest').evaluate(el => getComputedStyle(el, '::after').content), '"Saikō"');
assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth), 'Mobile horizontal overflow');
await page.getByRole('button', { name: '📚 In library' }).click();
assert.equal(await page.locator('.greatest-tile').count(), 0);
assert.deepEqual(errors, []);
await context.close();
console.log('Greatest of All Time browser QA passed: periods, ratings, filters, source status, modal, mobile layout.');
} finally {
if (browser) await browser.close();
await new Promise(resolve => { if (child.exitCode !== null) return resolve(); child.once('exit', resolve); child.kill(); });
await new Promise(resolve => fakeScraper.close(resolve));
fs.rmSync(dir, { recursive: true, force: true });
}
})().catch(error => { console.error(error); process.exitCode = 1; });
+260
View File
@@ -0,0 +1,260 @@
#!/usr/bin/env node
'use strict';
// Disposable, two-project Compose acceptance. No live library, db or queue is mounted.
const assert = require('node:assert/strict');
const fs = require('node:fs');
const fsp = fs.promises;
const os = require('node:os');
const path = require('node:path');
const crypto = require('node:crypto');
const net = require('node:net');
const { spawn } = require('node:child_process');
const { PeeringState } = require('../peering/state');
const { Canvas } = require('./png');
const ROOT = path.resolve(__dirname, '..');
const APP_IMAGE = process.env.PEER_AB_IMAGE || `mangacircus:peer-ab-${crypto.randomBytes(3).toString('hex')}`;
const TEST_ROOT = fs.mkdtempSync(path.join(os.tmpdir(), 'mangacircus-peer-ab-'));
const projects = ['mc-peer-a-' + crypto.randomBytes(3).toString('hex'), 'mc-peer-b-' + crypto.randomBytes(3).toString('hex')];
const OPENSSL_BIN = process.env.OPENSSL_BIN || (process.platform === 'win32' ? 'C:\\Program Files\\Git\\usr\\bin\\openssl.exe' : 'openssl');
process.env.OPENSSL_BIN = OPENSSL_BIN;
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
async function freePort() {
const server = net.createServer();
await new Promise((resolve, reject) => server.once('error', reject).listen(0, '127.0.0.1', resolve));
const port = server.address().port;
await new Promise(resolve => server.close(resolve));
return port;
}
function command(bin, args, options = {}) {
return new Promise((resolve, reject) => {
const child = spawn(bin, args, { cwd: ROOT, windowsHide: true, env: options.env || process.env, stdio: ['ignore', 'pipe', 'pipe'] });
let stdout = '', stderr = '';
child.stdout.on('data', data => stdout += data);
child.stderr.on('data', data => stderr += data);
child.on('error', reject);
child.once('exit', code => code === 0 ? resolve(stdout.trim()) : reject(new Error(`${bin} ${args[0]} failed (${code}): ${stderr.slice(-4000)}`)));
});
}
function composeArgs(project) { return ['compose', '-p', project, '-f', path.join(ROOT, 'docker-compose.yml'), '--profile', 'peering']; }
async function docker(project, env, ...args) { return command('docker', [...composeArgs(project), ...args], { env }); }
async function until(check, message, attempts = 120) {
let last;
for (let i = 0; i < attempts; i++) {
try { if (await check()) return; } catch (error) { last = error; }
await sleep(500);
}
throw new Error(`${message}${last ? `: ${last.message}` : ''}`);
}
async function setup(label, readerPort, peerPort) {
const root = path.join(TEST_ROOT, label), library = path.join(root, 'library'), data = path.join(root, 'data');
const scraper = path.join(data, 'scraper'), sentinel = path.join(data, 'sentinel'), peerData = path.join(data, 'peering');
const usage = path.join(data, 'peer-usage'), token = path.join(root, 'scraper-token');
for (const dir of [library, scraper, sentinel, peerData, usage]) await fsp.mkdir(dir, { recursive: true });
await fsp.writeFile(path.join(scraper, 'queue.json'), JSON.stringify({ paused: true, queue: [], settings: { autoUpdateEnabled: false } }));
await fsp.writeFile(token, crypto.randomBytes(32).toString('hex'), { mode: 0o600 });
const png = new Canvas(180, 270, [55, 105, 150]).png();
const manga = `${label === 'a' ? 'Peer A' : 'Peer B'} Test Manga`;
await fsp.writeFile(path.join(library, '.fixture'), '');
const mangaDir = path.join(library, manga);
await fsp.mkdir(mangaDir);
await fsp.writeFile(path.join(mangaDir, 'cover.png'), png);
await fsp.writeFile(path.join(mangaDir, 'manga.json'), JSON.stringify({ title: manga, year: 2024,
status: 'ongoing', description: 'A disposable peer test title.', tags: ['Fixture', 'Peering'], genres: ['Test'], direction: 'ttb' }));
for (let n = 1; n <= 2; n++) {
const chapter = path.join(mangaDir, 'v01', `c${String(n).padStart(4, '0')} Chapter ${n}`);
await fsp.mkdir(chapter, { recursive: true });
await fsp.writeFile(path.join(chapter, 'chapter.json'), JSON.stringify({ title: `Chapter ${n}`, source: { name: 'LocalFixture' } }));
await fsp.writeFile(path.join(chapter, '0001.png'), png);
}
new PeeringState(peerData).generateIdentity(label === 'a' ? 'PeerA' : 'PeerB');
const env = { ...process.env, APP_IMAGE, VCS_REF: 'peer-redesign-test', LIBRARY_PATH: library, DATA_PATH: data,
SCRAPER_DATA_PATH: scraper, SENTINEL_DATA_PATH: sentinel, PEER_DATA_PATH: peerData, PEER_USAGE_PATH: usage,
SCRAPER_TOKEN_PATH: token, READER_PORT: String(readerPort), READER_BIND: '127.0.0.1',
PEER_PORT: String(peerPort), PEER_BIND: '127.0.0.1', COOKIE_SECURE: '0', TZ: 'UTC' };
return { label, root, library, data, peerData, manga, readerPort, peerPort, env, url: `http://127.0.0.1:${readerPort}` };
}
async function json(url, cookie, body) {
const response = await fetch(url, { headers: { ...(cookie ? { cookie } : {}), 'content-type': 'application/json' },
...(body === undefined ? {} : { method: 'POST', body: JSON.stringify(body) }) });
const data = response.headers.get('content-type')?.includes('application/json') ? await response.json() : await response.arrayBuffer();
return { response, data };
}
async function login(instance) {
const { response } = await json(`${instance.url}/api/signup`, null, { name: 'qa', password: 'DisposablePeerTest42' });
assert.equal(response.status, 200, `${instance.label}: first admin signup`);
return response.headers.get('set-cookie').split(';')[0];
}
async function browserCheck(instance, cookie, peerId, shareId) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
try {
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
const [name, ...value] = cookie.split('='); await context.addCookies([{ name, value: value.join('='), url: instance.url }]);
const page = await context.newPage(), errors = []; page.on('pageerror', error => errors.push(error.message));
await page.goto(`${instance.url}/peer-catalog.html?peer=${encodeURIComponent(peerId)}#browse`);
await page.getByRole('heading', { name: 'Peer catalogs', exact: true }).waitFor();
await page.locator('.peer-catalog-card').filter({ hasText: 'Peer A Test Manga' }).waitFor();
assert.equal(await page.locator('.peer-cover').count(), 1, 'Remote cover must render in the catalog');
await page.getByRole('button', { name: 'Details', exact: true }).click();
const dialog = page.getByRole('dialog');
await dialog.getByText('A disposable peer test title.').waitFor();
assert.equal(await dialog.locator('.peer-chapter-option').count(), 2);
await dialog.locator('.peer-chapter-picker summary').click();
await dialog.getByLabel('Queue scope').selectOption('after');
await dialog.getByLabel('Chapter threshold').waitFor({ state: 'visible' });
const exactQueue = dialog.getByRole('button', { name: 'Queue checked chapters' });
assert.equal(await exactQueue.isDisabled(), true, 'Exact chapter queue starts disabled');
await dialog.getByLabel('Select chapter 1').check();
assert.equal(await exactQueue.isDisabled(), false);
await dialog.getByLabel('Select chapter 1').uncheck();
await dialog.getByRole('button', { name: 'Close details' }).click();
await page.getByRole('checkbox', { name: 'Select Peer A Test Manga' }).check();
await page.getByRole('button', { name: 'Queue 1 missing' }).click();
await page.getByText(/1 manga queued; 0 unavailable or failed/).waitFor();
assert.equal(errors.length, 0, `Peer catalog browser errors: ${errors.join('; ')}`);
} finally { await browser.close(); }
}
async function peerSettingsUiCheck(instance, cookie, options = {}) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
try {
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
const [name, ...value] = cookie.split('='); await context.addCookies([{ name, value: value.join('='), url: instance.url }]);
const page = await context.newPage(), errors = []; page.on('pageerror', error => errors.push(error.message));
await page.goto(instance.url);
await page.locator('.nav[data-peer-catalog-resolved="true"]').waitFor();
assert.equal(await page.locator('.nav a[href="peer-catalog.html"]').count(), options.hasPeer ? 1 : 0,
'Peer catalogs nav appears only after a peer is configured');
if (options.peerId) {
await page.goto(`${instance.url}/peers.html#admin/peering/${encodeURIComponent(options.peerId)}`);
await page.getByRole('heading', { name: 'Access between servers' }).waitFor();
const issue = page.getByRole('button', { name: options.outboundGrant ? 'Save grant permissions' : 'Issue grant', exact: true });
if (options.paired) {
await page.getByText('Peering confirmed on both servers', { exact: true }).waitFor();
if (!options.outboundGrant) await page.getByLabel('Browse approved catalog').check();
assert.equal(await issue.isDisabled(), false, 'A selected grant permission enables the grant action after pairing');
} else {
await page.getByText('Identity not verified yet', { exact: true }).waitFor();
await page.getByLabel('Browse approved catalog').check();
await page.getByLabel('Download chapters (includes catalog access)').check();
assert.equal(await issue.isDisabled(), true, 'Grant issue remains locked before identity verification');
await page.getByText(/Granting is locked until identity pairing is complete/).waitFor();
}
if (options.outboundGrant) await page.getByText('Grant active', { exact: true }).waitFor();
if (options.inboundGrant) await page.getByText('Active grant from peer', { exact: true }).waitFor();
if (options.paired) {
const stateCard = page.locator('.peer-state-ok').first();
const visual = await stateCard.evaluate(element => {
const style = getComputedStyle(element);
return { weight: getComputedStyle(element.querySelector('strong')).fontWeight,
border: style.borderLeftWidth, background: style.backgroundColor };
});
assert.ok(Number(visual.weight) >= 700 && visual.border === '4px' && visual.background !== 'rgba(0, 0, 0, 0)',
`Confirmed state must be visually prominent: ${JSON.stringify(visual)}`);
}
}
assert.equal(errors.length, 0, `Peer settings browser errors: ${errors.join('; ')}`);
} finally { await browser.close(); }
}
async function main() {
let started = [], succeeded = false;
try {
const ports = [];
while (ports.length < 4) { const value = await freePort(); if (!ports.includes(value)) ports.push(value); }
const [portA, portB, gateA, gateB] = ports;
const A = await setup('a', portA, gateA), B = await setup('b', portB, gateB);
const args = ['build', '-t', APP_IMAGE, '--build-arg', 'VCS_REF=peer-redesign-test', '.'];
await command('docker', args);
for (const [index, instance] of [A, B].entries()) {
const project = projects[index];
await docker(project, instance.env, 'config', '-q');
await docker(project, instance.env, 'up', '-d'); started.push({ project, env: instance.env });
await until(async () => (await fetch(`${instance.url}/health`)).ok, `${instance.label}: reader health`);
await until(async () => new Promise(resolve => {
const https = require('node:https');
https.get({ hostname: '127.0.0.1', port: instance.peerPort, path: '/health', rejectUnauthorized: false }, res => { res.resume(); resolve(res.statusCode === 200); }).on('error', () => resolve(false));
}), `${instance.label}: peer gateway health`);
}
const cookieA = await login(A), cookieB = await login(B);
await peerSettingsUiCheck(A, cookieA, { hasPeer: false });
await peerSettingsUiCheck(B, cookieB, { hasPeer: false });
const stateA = (await json(`${A.url}/api/admin/peering`, cookieA)).data;
const stateB = (await json(`${B.url}/api/admin/peering`, cookieB)).data;
const hostUrlA = `https://host.docker.internal:${gateA}/`, hostUrlB = `https://host.docker.internal:${gateB}/`;
const inviteA = (await json(`${A.url}/api/admin/peering`, cookieA, { command: 'invite', url: hostUrlA })).data.invitation;
const inviteB = (await json(`${B.url}/api/admin/peering`, cookieB, { command: 'invite', url: hostUrlB })).data.invitation;
let importedA = await json(`${A.url}/api/admin/peering`, cookieA, { command: 'import', invitation: inviteB });
let importedB = await json(`${B.url}/api/admin/peering`, cookieB, { command: 'import', invitation: inviteA });
assert.equal(importedA.response.status, 200); assert.equal(importedB.response.status, 200);
const peerA = importedA.data.peers.find(peer => peer.fingerprint === inviteB.fingerprint);
const peerB = importedB.data.peers.find(peer => peer.fingerprint === inviteA.fingerprint);
assert.ok(peerA && peerB);
await peerSettingsUiCheck(B, cookieB, { hasPeer: true, peerId: peerB.id, paired: false });
const probeA = await json(`${A.url}/api/admin/peering`, cookieA, { command: 'probe', id: peerA.id });
const probeB = await json(`${B.url}/api/admin/peering`, cookieB, { command: 'probe', id: peerB.id });
if (probeA.response.status !== 200 || probeB.response.status !== 200) {
console.error('Peer probe responses:', JSON.stringify({ a: probeA.data, b: probeB.data }));
for (const [index, instance] of [A, B].entries()) {
console.error(`${instance.label} gateway logs:\n${await docker(projects[index], instance.env, 'logs', '--tail', '60', 'peer-gateway').catch(() => '(unavailable)')}`);
console.error(`${instance.label} reader logs:\n${await docker(projects[index], instance.env, 'logs', '--tail', '60', 'reader').catch(() => '(unavailable)')}`);
}
}
assert.equal(probeA.response.status, 200); assert.equal(probeB.response.status, 200);
assert.equal(probeA.data.peers.find(peer => peer.id === peerA.id).paired, true);
assert.equal(probeB.data.peers.find(peer => peer.id === peerB.id).paired, true);
await peerSettingsUiCheck(A, cookieA, { hasPeer: true, peerId: peerA.id, paired: true });
assert.ok(stateA.identity?.fingerprint && stateB.identity?.fingerprint);
let libraryA = [];
await until(async () => { libraryA = (await json(`${A.url}/api/admin/peering`, cookieA)).data.library || []; return libraryA.some(item => item.title === A.manga); }, 'A fixture manga was not scanned');
const latestLibraryA = (await json(`${A.url}/api/admin/peering`, cookieA)).data.library || libraryA;
const mangaA = latestLibraryA.find(item => item.title === A.manga);
assert.ok(mangaA, 'A fixture manga is indexed');
let saved = await json(`${A.url}/api/admin/peering`, cookieA, { command: 'share-set', id: peerA.id, mangaIds: [mangaA.id] });
assert.equal(saved.response.status, 200); assert.equal(saved.data.peers.find(peer => peer.id === peerA.id).shares.length, 1);
const grant = await json(`${A.url}/api/admin/peering`, cookieA, { command: 'grant', id: peerA.id, scopes: ['catalog'] });
assert.equal(grant.response.status, 200);
assert.equal((await json(`${B.url}/api/admin/peering`, cookieB, { command: 'accept-grant', id: peerB.id, token: grant.data.grant.token })).response.status, 200);
await peerSettingsUiCheck(A, cookieA, { hasPeer: true, peerId: peerA.id, paired: true, outboundGrant: true });
await peerSettingsUiCheck(B, cookieB, { hasPeer: true, peerId: peerB.id, paired: true, inboundGrant: true });
const catalog = await json(`${B.url}/api/peering/catalog?peer=${encodeURIComponent(peerB.id)}`, cookieB);
assert.equal(catalog.response.status, 200); assert.equal(catalog.data.items.length, 1);
assert.equal(catalog.data.items[0].workIdentity?.scheme, 'mangacircus-work-v1', 'Peer catalogs expose a stable, opaque work identity');
const shareId = catalog.data.items[0].id;
const cover = await fetch(`${B.url}/api/peering/cover?peer=${encodeURIComponent(peerB.id)}&manga=${shareId}`, { headers: { cookie: cookieB } });
assert.equal(cover.status, 200); assert.match(cover.headers.get('content-type'), /^image\/png/);
const denied = await json(`${B.url}/api/peering/queue`, cookieB, { peer: peerB.id, manga: shareId, scope: { mode: 'missing' } });
assert.equal(denied.response.status, 403, 'Catalog grant alone must not permit chapter queueing');
const updateGrant = await json(`${A.url}/api/admin/peering`, cookieA, { command: 'set', id: peerA.id, patch: { grantScopes: ['download'] } });
assert.equal(updateGrant.response.status, 200);
const accessB = await json(`${B.url}/api/peering/access?peer=${encodeURIComponent(peerB.id)}`, cookieB);
assert.deepEqual(accessB.data.grant.scopes, ['catalog', 'download'], 'Grant scope updates must not need token reimport');
const receiving = await json(`${B.url}/api/admin/peering`, cookieB, { command: 'set', id: peerB.id, patch: { receive: true } });
assert.equal(receiving.response.status, 200);
await browserCheck(B, cookieB, peerB.id, shareId);
const queue = (await json(`${B.url}/api/admin/scraper`, cookieB)).data;
assert.equal(queue.paused, true, 'Peer testing must leave the scraper queue paused');
const peerQueueItems = queue.queue.filter(item => item.url === `peer://${peerB.id}/${shareId}`);
assert.equal(peerQueueItems.length, 1, 'Exactly one peer queue item is added');
assert.deepEqual(peerQueueItems[0].workIdentity, catalog.data.items[0].workIdentity,
'The resolved work identity is persisted on the queued peer transfer');
const volumes = await Promise.all(started.map(({ project, env }) => docker(project, env, 'ps', '--format', '{{.Service}}')));
for (const services of volumes) for (const name of ['reader', 'scraper', 'sentinel', 'peer-gateway']) assert.ok(services.includes(name), `${name} is running in A and B`);
console.log('PASS: isolated Compose A↔B pairing, catalog-only access, cover proxy, grant update, browser details/bulk queue, one paused queue writer, and admin-only API/UI boundary.');
console.log(`Test Compose projects: ${projects.join(', ')}; disposable roots: ${TEST_ROOT}`);
succeeded = true;
} finally {
if (!succeeded && process.argv.includes('--keep-on-failure')) console.error(`Failure fixtures retained for cleanup: ${TEST_ROOT}; projects: ${projects.join(', ')}`);
else {
for (const { project, env } of [...started].reverse()) await docker(project, env, 'down').catch(error => console.error(`Cleanup warning for ${project}: ${error.message}`));
await fsp.rm(TEST_ROOT, { recursive: true, force: true });
if (!process.env.PEER_AB_IMAGE) await command('docker', ['image', 'rm', APP_IMAGE]).catch(() => {});
}
}
}
main().catch(error => { console.error(error.stack || error); process.exitCode = 1; });
+85
View File
@@ -0,0 +1,85 @@
'use strict';
const assert = require('node:assert/strict');
const fs = require('node:fs'), os = require('node:os'), path = require('node:path');
const { PeeringState } = require('../peering/state');
module.exports = async function testPeerUI(url, cookie) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
const [name, ...value] = cookie.split('=');
await context.addCookies([{ name, value: value.join('='), url }]);
const page = await context.newPage(), errors = [];
page.on('pageerror', error => errors.push(error.message));
try {
await page.goto(url + '/peers.html#admin');
await page.getByRole('heading', { name: 'Known peers' }).waitFor();
assert.ok(await page.getByRole('link', { name: 'Peer catalogs' }).count() >= 1);
const state = await page.evaluate(() => fetch('/api/admin/peering').then(response => response.json()));
assert.equal(state.identity?.name, 'ComposeQA');
await page.getByRole('button', { name: 'Add a peer' }).click();
await page.getByRole('heading', { name: 'Add a peer' }).waitFor();
await page.getByLabel('This server’s reachable peer gateway URL').fill('http://localhost:8080');
await page.getByRole('button', { name: 'Create 48-hour invitation' }).click();
assert.match(await page.getByLabel('This server’s reachable peer gateway URL').evaluate(el => el.validationMessage), /HTTPS peer gateway origin/);
await page.getByLabel('This server’s reachable peer gateway URL').fill('https://peer-test.example:8093');
await page.getByRole('button', { name: 'Create 48-hour invitation' }).click();
const outgoing = page.getByLabel('Invitation to send');
await page.waitForFunction(() => document.querySelector('textarea[aria-label="Invitation to send"]')?.value.includes('fingerprint'));
const invitation = JSON.parse(await outgoing.inputValue());
assert.equal(invitation.url, 'https://peer-test.example:8093');
assert.match(invitation.name, /^[A-Za-z0-9]{1,25}$/);
assert.ok(Date.parse(invitation.expiresAt) > Date.now());
await page.getByLabel('Invitation received').fill('{bad json');
await page.getByRole('button', { name: 'Import invitation' }).click();
await page.getByText('That is not valid invitation JSON.').waitFor();
const peerDir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-peer-ui-'));
let peerId;
try {
if (process.platform === 'win32' && !process.env.OPENSSL_BIN) process.env.OPENSSL_BIN = 'C:\\Program Files\\Git\\usr\\bin\\openssl.exe';
const peer = new PeeringState(peerDir);
peer.generateIdentity('UIFixturePeer');
const fixtureInvitation = peer.createInvitation('https://fixture-peer.example:8093');
await page.getByLabel('Invitation received').fill(JSON.stringify(fixtureInvitation));
await page.getByRole('button', { name: 'Import invitation' }).click();
await page.getByRole('heading', { name: '<UIFixturePeer>', exact: true }).waitFor();
const apiState = await page.evaluate(() => fetch('/api/peering').then(response => response.json()));
peerId = apiState.peers.find(item => item.alias === 'UIFixturePeer').id;
await page.locator('.peer-page-tabs').getByRole('link', { name: 'Peer catalogs' }).click();
await page.getByRole('heading', { name: 'Peer catalogs' }).waitFor();
if (!await page.getByRole('button', { name: 'Select visible' }).count()) {
console.error('Peer catalog page content:', await page.locator('main').innerText());
console.error('Peer catalog JS errors:', errors);
}
await page.getByLabel('Select peer').selectOption(peerId);
assert.equal(await page.getByText('Setup is incomplete', { exact: false }).count(), 0, 'Sharing can be prepared before pairing');
await page.getByRole('button', { name: 'Select visible' }).click();
await page.getByRole('button', { name: 'Save approvals' }).click();
await page.getByText('Approved titles updated.').waitFor();
const saved = await page.evaluate(id => fetch('/api/admin/peering').then(r => r.json()).then(s => s.peers.find(p => p.id === id).shares.length), peerId);
assert.equal(saved, 3, 'Bulk approval saves the selected local catalog');
assert.equal(await page.getByRole('button', { name: 'Browse peer catalog' }).count(), 1);
await page.getByRole('button', { name: 'Browse peer catalog' }).click();
await page.getByText('Setup is incomplete', { exact: false }).waitFor();
await page.setViewportSize({ width: 390, height: 844 });
assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth), 'Peer catalog should fit mobile width');
} finally { fs.rmSync(peerDir, { recursive: true, force: true }); }
assert.equal(errors.length, 0, `Browser errors: ${errors.join('; ')}`);
const created = await page.evaluate(async () => (await fetch('/api/admin/user', { method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: 'peer-reader-qa', password: 'disposable-reader-test', admin: false }) })).status);
assert.equal(created, 200);
await page.evaluate(() => fetch('/api/logout', { method: 'POST', headers: { 'content-type': 'application/json' }, body: '{}' }));
await page.evaluate(() => fetch('/api/login', { method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: 'peer-reader-qa', password: 'disposable-reader-test' }) }));
await page.goto(url + '/peer-catalog.html');
await page.getByText('Peer catalogs are available to administrators only.').waitFor();
const denied = await page.evaluate(async () => Promise.all(['/api/admin/peering', '/api/peering', '/api/peering/catalog?peer=x', '/api/peering/cover?peer=x&manga=x']
.map(async endpoint => (await fetch(endpoint)).status)));
assert.deepEqual(denied, [403, 403, 403, 403], 'Every peering API remains admin-only');
await page.goto(url + '/');
await page.getByRole('link', { name: 'Peer catalogs' }).waitFor({ state: 'detached' });
assert.equal(errors.length, 0, `Browser errors: ${errors.join('; ')}`);
} finally { await browser.close(); }
};
+118
View File
@@ -0,0 +1,118 @@
'use strict';
// Fixture-only state interception: no source traffic or production accounts.
const assert = require('node:assert/strict'), fs = require('node:fs'), path = require('node:path');
module.exports = async function testQueueUI(url, cookie) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: 'msedge', headless: true });
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
const [name, ...values] = cookie.split('=');
await context.addCookies([{ name, value: values.join('='), url }]);
const page = await context.newPage(), errors = [], commands = [], upstream = [];
page.on('pageerror', error => errors.push(error.message));
page.on('request', request => { if (new URL(request.url()).origin !== new URL(url).origin) upstream.push(request.url()); });
const state = await (await fetch(url + '/api/admin/scraper', { headers: { cookie } })).json();
state.library[0].sources = [{ name: 'MangaDex', adapter: 'mangadex', url: 'https://mangadex.org/title/fixture' }];
state.updateChecks = [{ mangaId: state.library[0].id, source: 'mangadex', lastCheckedAt: '2026-10-07T08:00:00Z', lastDownloadedAt: '2026-10-07T07:00:00Z', lastResult: 'no-updates' }];
state.history = [
{ title: 'Moon Garden', source: 'mangadex', status: 'checked', time: '2026-10-07T08:00:00Z' },
{ title: 'Another manga', source: 'weebcentral', status: 'checked', time: '2026-10-07T07:00:00Z' },
];
state.queue = [
{ id: 'pilot-fixture', title: 'bleach', source: 'mangadex', kind: 'inspection', status: 'ready', sourceTitle: 'Bleach (Pre-Serialization)', availability: { selectedChapters: 2, specialReleases: 2, coverage: 'unknown' } },
{ id: 'main-fixture', title: 'Bleach', source: 'mangadex', kind: 'download', status: 'complete', publishedCount: 1, sourceTitle: 'Bleach', availability: { selectedChapters: 1, numberedChapters: 1, expectedLastChapter: '686', highestHostedChapter: 686.1, coverage: 'unknown' } },
{ id: 'external-fixture', title: 'Claymore', source: 'mangadex', kind: 'download', status: 'unavailable', error: 'MangaDex lists 8 English external-link releases, but no hosted English pages. External links are not downloaded.' },
{ id: 'other-source-fixture', title: 'Another manga', source: 'weebcentral', kind: 'download', status: 'queued' },
];
await page.route('**/api/admin/scraper', async route => {
if (route.request().method() === 'POST') {
const command = route.request().postDataJSON(); commands.push(command);
if (command.action === 'confirm') state.queue.find(item => item.id === command.id).status = 'queued';
if (command.action === 'inspect') state.queue.find(item => item.id === command.id).status = 'queued';
if (command.command === 'clear-completed') state.queue = state.queue.filter(item => item.status !== 'complete' || item.kind !== 'download');
}
await route.fulfill({ contentType: 'application/json', body: JSON.stringify(state) });
});
const shots = path.resolve(__dirname, '..', 'data', 'qa', 'queue'); fs.mkdirSync(shots, { recursive: true });
try {
await page.goto(url + '/settings.html#admin/scraper');
const activity = page.locator('[data-scraper-tab=activity]'), queuePanel = page.locator('[data-scraper-tab=queue]');
await activity.getByText('Source edition: Bleach (Pre-Serialization)', { exact: true }).waitFor();
await activity.getByText(/no hosted English pages/).waitFor();
assert.equal(commands.length, 0, 'Inspection status must not start a download');
await page.screenshot({ path: path.join(shots, 'desktop.png'), fullPage: true });
await activity.getByRole('button', { name: 'Confirm download', exact: true }).click();
await page.waitForFunction(() => ![...document.querySelectorAll('button')].some(b => b.textContent === 'Confirm download'));
assert.deepEqual(commands.at(-1), { command: 'control', action: 'confirm', id: 'pilot-fixture' });
await activity.locator('.scraper-item').filter({ hasText: 'Claymore' }).getByRole('button', { name: 'Inspect again', exact: true }).click();
assert.deepEqual(commands.at(-1), { command: 'control', action: 'inspect', id: 'external-fixture' });
await page.getByRole('link', { name: 'Queue', exact: true }).click();
await queuePanel.getByText('Completed downloads · 1').click();
await page.waitForTimeout(3500);
assert.equal(await queuePanel.locator('.scraper-completed').evaluate(el => el.open), true, 'Completed section collapsed on polling');
// Rebuild the panel during the summary click, before the native toggle.
// A state snapshot taken only at redraw time misses this click and recloses it.
await queuePanel.getByText('Completed downloads · 1').click();
await page.evaluate(() => {
const summary = document.querySelector('.scraper-completed > summary');
summary.addEventListener('click', () => {
const buttons = document.querySelectorAll('.scraper-filters button');
buttons[1].click(); buttons[0].click();
}, { capture: true, once: true });
});
await queuePanel.getByText('Completed downloads · 1').click();
await page.waitForTimeout(3500);
assert.equal(await queuePanel.locator('.scraper-completed').evaluate(el => el.open), true, 'Completed section lost a click during redraw');
await queuePanel.getByText(/1 chapter published · finished/).waitFor();
await queuePanel.getByText('Hosted selection complete', { exact: true }).waitFor();
await queuePanel.getByText('Series coverage partial', { exact: true }).waitFor();
page.once('dialog', dialog => dialog.accept());
await page.getByRole('button', { name: 'Clear completed downloads (1)' }).click();
assert.deepEqual(commands.at(-1), { command: 'clear-completed', source: '' });
assert.equal(await queuePanel.locator('.scraper-completed').evaluate(el => el.open), true, 'Completed section collapsed after cleanup');
assert.equal(state.queue.some(item => item.id === 'main-fixture'), false);
assert.ok(state.queue.some(item => item.id === 'pilot-fixture'));
await page.getByRole('button', { name: 'MangaDex', exact: true }).click();
assert.equal(await page.getByText('Another manga').count(), 0);
await page.getByRole('button', { name: 'All sources', exact: true }).click();
await queuePanel.getByText('Another manga').waitFor();
await page.getByRole('link', { name: 'Updates', exact: true }).click();
await page.getByText(/Last checked:.*2026/).waitFor();
await page.getByText(/Last downloaded:.*2026/).waitFor();
await page.getByRole('button', { name: 'MangaDex', exact: true }).click();
await page.getByText('Recent check activity').click();
await page.waitForTimeout(3500);
assert.equal(await page.locator('.scraper-history').evaluate(el => el.open), true, 'Check history collapsed on polling');
assert.equal(await page.locator('[data-scraper-tab=updates] .scraper-history').getByText('Another manga', { exact: false }).count(), 0);
await page.getByRole('button', { name: 'All sources', exact: true }).click();
await page.getByRole('link', { name: 'Queue', exact: true }).click();
await page.getByText('Add manga to queue').click();
await page.waitForTimeout(3500);
assert.equal(await queuePanel.locator('.scraper-add').evaluate(el => el.open), true, 'Add form collapsed on polling');
await page.locator('input[type=url]').fill('https://mangadex.org/title/06a37094-b3ba-4f25-bf2b-54c9751e0e91');
await page.getByRole('button', { name: 'Inspect & queue', exact: true }).click();
assert.equal(commands.at(-1).inspect, true); assert.equal(commands.at(-1).command, 'enqueue');
await page.setViewportSize({ width: 390, height: 844 });
assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth));
await page.screenshot({ path: path.join(shots, 'mobile.png'), fullPage: true });
const detailWidth = await page.locator('[data-scraper-tab=queue] .scraper-queue .scraper-item > .grow').first().evaluate(el => el.getBoundingClientRect().width);
assert.ok(detailWidth > 280, `Queue details squeezed by action buttons (${detailWidth}px)`);
await page.route('**/api/manga?*', async route => {
const title = new URL(route.request().url()).searchParams.get('m');
const first = title === 'Bleach' ? 336 : 1;
await route.fulfill({ contentType: 'application/json', body: JSON.stringify({
id: title, title, meta: { title, sources: [{ name: 'MangaDex', url: 'https://mangadex.org/' }] },
cover: null, direction: 'rtl', books: [{ key: 'v01', label: 'Vol. 1' }],
chapters: [{ id: `v01/c${first}`, book: 'v01', label: `Ch. ${first}`, title: '',
versions: [{ id: 'fixture', pages: 1, added: '2026-10-01', source: 'MangaDex', score: 0, my: 0 }] }],
collections: [], relations: [], following: false, reading: null, continue: null,
}) });
});
await page.goto(url + '/manga.html?m=Bleach');
await page.getByText('Series coverage partial', { exact: true }).waitFor();
await page.getByText(/starts at chapter 336/).waitFor();
await page.goto(url + '/manga.html?m=Complete%20Fixture');
assert.equal(await page.getByText('Series coverage partial', { exact: true }).count(), 0);
assert.deepEqual(errors, []); assert.deepEqual(upstream, []);
console.log('PASS: scraper activity, queue coverage, manga gap warning, cleanup, source filters, edition controls and mobile layout; no source traffic');
} finally { await browser.close(); }
};
+116
View File
@@ -0,0 +1,116 @@
'use strict';
// Browser QA uses ONLY test-compose's disposable user/library, never live history.
const assert = require('node:assert/strict');
module.exports = async function (url, cookie) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
const context = await browser.newContext({ viewport: { width: 390, height: 844 }, isMobile: true, hasTouch: true });
const [name, ...value] = cookie.split('=');
await context.addCookies([{ name, value: value.join('='), url }]);
const page = await context.newPage(), errors = [], packets = [];
page.on('pageerror', e => errors.push(e.message));
let delayed = false;
context.on('request', request => {
if (!request.url().endsWith('/api/reading/visit') || request.method() !== 'POST') return;
try { packets.push(request.postDataJSON()); } catch {}
});
await context.route(url + '/api/reading/visit', async route => {
const body = route.request().postDataJSON();
if (delayed && !body.final && body.activeMs > 0) { delayed = false; await new Promise(resolve => setTimeout(resolve, 1000)); }
const response = await route.fetch(); await route.fulfill({ response });
});
const api = async (path, body) => {
const response = await context.request.fetch(url + path, { method: body ? 'POST' : 'GET', data: body });
assert.equal(response.ok(), true, await response.text()); return response.json();
};
try {
await api('/api/settings', { readingClock: true, startMode: 'flow', flowPageGap: false });
const inventory = await api('/api/library'), manga = inventory.mangas.find(m => m.title === 'Moon Garden');
const detail = await api('/api/manga?m=' + encodeURIComponent(manga.id));
const chapter = detail.chapters[0];
const reader = url + '/read.html?m=' + encodeURIComponent(manga.id) + '&c=' + encodeURIComponent(chapter.id);
const summary = async () => (await api('/api/chapter?m=' + encodeURIComponent(manga.id) + '&c=' + encodeURIComponent(chapter.id))).reading;
const load = async () => {
await page.goto(reader); await page.locator('#readclock:not([hidden])').waitFor();
await page.waitForFunction(() => [...document.querySelectorAll('#canvas img')].some(img => img.complete && img.naturalWidth));
await page.locator('#stage').focus();
// Loaded pages, fast actual observed positions; no fictional intermediate views.
await page.waitForTimeout(130);
};
await load();
const images = page.locator('#canvas img');
for (let i = 0; i < 12; i++) {
await images.nth(i).evaluate(img => { const stage = document.querySelector('#stage'); stage.scrollTop += img.getBoundingClientRect().top - stage.getBoundingClientRect().top; });
await page.waitForFunction(index => { const img = document.querySelectorAll('#canvas img')[index]; return img?.complete && img.naturalWidth; }, i);
await page.waitForTimeout(125);
}
// Flow auto-hides the bars while scrolling; inspect the clock panel without
// introducing another page-navigation gesture into the coverage test.
await page.locator('#readclock').dispatchEvent('click');
await page.getByText('Yes · time continues counting', { exact: true }).waitFor();
// Force a normal in-flight save, then exercise pagehide explicitly before
// Playwright's programmatic navigation tears down the old document.
// The final beacon must not wait for the delayed ordinary save.
delayed = true;
await page.evaluate(() => window.dispatchEvent(new Event('blur')));
await page.evaluate(() => window.dispatchEvent(new Event('pagehide')));
await page.waitForTimeout(100);
await page.goto(url + '/manga.html?m=' + encodeURIComponent(manga.id));
await page.waitForTimeout(1400);
const saved = await summary();
assert.equal(saved.completedPasses, 1); assert.ok(saved.totalActiveMs >= 1000);
assert.ok(packets.some(p => p.final && p.seen.length >= 12),
`Final coverage packet missing (closed=${saved.currentPass?.closed}): ${JSON.stringify(packets.map(p => ({ seq: p.seq, final: p.final, seen: p.seen?.length, activeMs: p.activeMs })))}`);
const before = saved.totalActiveMs;
await load();
// Jump to one picture and dwell. Prior read must not lend this visit coverage.
await images.nth(18).evaluate(img => { const stage = document.querySelector('#stage'); stage.scrollTop += img.getBoundingClientRect().top - stage.getBoundingClientRect().top; });
await page.waitForTimeout(1500);
await page.goto(url + '/manga.html?m=' + encodeURIComponent(manga.id));
await page.waitForTimeout(400);
const lookup = await summary();
assert.equal(lookup.totalActiveMs, before); assert.equal(lookup.completedPasses, 1);
assert.ok(lookup.currentPass.seenPages < 12);
// A stationary reader must pulse periodically, not continuously after replies.
await load(); const count = packets.length;
await page.waitForTimeout(11500);
assert.ok(packets.length - count <= 2, 'Response-driven request storm');
await page.waitForFunction(() => document.querySelector('#readclock .clock-label')?.textContent.includes('Idle'), null, { timeout: 23000 });
const pausedTime = await page.locator('#readclock .clock-time').textContent();
await page.waitForTimeout(1100);
assert.equal(await page.locator('#readclock .clock-time').textContent(), pausedTime, 'Idle time still accrued');
await page.mouse.move(20, 200);
await page.waitForFunction(() => document.querySelector('#readclock .clock-label')?.textContent.includes('Reading'));
await load();
const background = await page.locator('body').evaluate(body => getComputedStyle(body).backgroundColor);
const toneButton = page.locator('#page-tone');
await toneButton.click();
for (const viewMode of ['flow', 'page', 'book']) {
await page.locator(`#modes button[data-mode="${viewMode}"]`).click();
const appearance = await page.evaluate(() => ({
tone: document.querySelector('#canvas').dataset.pageTone,
image: getComputedStyle(document.querySelector('#canvas .pg img')).filter,
controls: getComputedStyle(document.querySelector('#bot')).filter,
background: getComputedStyle(document.body).backgroundColor,
}));
assert.equal(appearance.tone, 'inverted');
assert.match(appearance.image, /invert\(1\)/);
assert.equal(appearance.controls, 'none');
assert.equal(appearance.background, background);
}
for (const [tone, filter] of [['sepia', /sepia/], ['dim', /brightness\(0\.7\)/], ['extra-dim', /brightness\(0\.45\)/]]) {
await toneButton.click();
const appearance = await page.evaluate(() => ({ tone: document.querySelector('#canvas').dataset.pageTone,
filter: getComputedStyle(document.querySelector('#canvas .pg img')).filter }));
assert.equal(appearance.tone, tone);
assert.match(appearance.filter, filter);
}
await page.waitForFunction(async () => (await (await fetch('api/me')).json()).settings.pageTone === 'extra-dim');
await load();
assert.equal(await page.locator('#canvas').getAttribute('data-page-tone'), 'extra-dim');
await toneButton.click();
await page.waitForFunction(async () => (await (await fetch('api/me')).json()).settings.pageTone === 'original');
assert.equal(errors.length, 0, errors.join('\n'));
console.log('PASS: mobile reading clock and five page tones across Flow, Page and Book without changing the theme');
} finally { await context.close(); await browser.close(); }
};
+128
View File
@@ -0,0 +1,128 @@
'use strict';
// Optional headless browser QA against the disposable Compose fixture only.
// Playwright is a development tool, not an application dependency.
const assert = require('node:assert/strict');
const fs = require('node:fs'), path = require('node:path');
module.exports = async function testSettingsUI(url, cookie) {
const { chromium } = require('playwright');
const browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
const [name, ...value] = cookie.split('=');
await context.addCookies([{ name, value: value.join('='), url }]);
const page = await context.newPage(), errors = [], requests = [];
page.on('pageerror', e => errors.push(e.message));
page.on('request', r => requests.push(new URL(r.url()).pathname));
const shots = path.resolve(__dirname, '..', 'data', 'qa', 'settings');
fs.mkdirSync(shots, { recursive: true });
const clickSection = async (id, title) => {
await page.locator(`.settings-menu a[href="settings.html#${id}"]`).click();
await page.getByRole('heading', { name: title, exact: true, level: 1 }).waitFor();
};
const noOverflow = async () => assert.ok(await page.evaluate(() => document.documentElement.scrollWidth <= innerWidth), 'Horizontal page overflow');
try {
await page.goto(url + '/settings.html');
await page.getByRole('heading', { name: 'Reading', exact: true, level: 1 }).waitFor();
assert.equal(await page.locator('.settings-menu-group').count(), 2);
assert.equal(requests.filter(p => p.startsWith('/api/admin') && p !== '/api/admin/peering').length, 0,
'Personal settings fetched admin data beyond the top navigation peer indicator');
await clickSection('user/reading-hours', 'Reading hours');
await page.getByRole('heading', { name: 'Server reading hours' }).waitFor();
const schedule = await page.evaluate(() => fetch('/api/reading-schedule').then(r => r.json()));
if (schedule.enabled) {
await page.getByText(`Server timezone: ${schedule.timezone}`).waitFor();
assert.equal(await page.locator('.reading-hours-day').count(), 7);
} else await page.getByText('Reading is available at any time. No schedule is configured.').waitFor();
assert.equal(await page.locator('[data-settings-section="user/reading-hours"] input').count(), 0);
await page.screenshot({ path: path.join(shots, 'desktop-reading-hours.png'), fullPage: true });
await page.setViewportSize({ width: 390, height: 844 });
await noOverflow();
await page.screenshot({ path: path.join(shots, 'mobile-reading-hours.png'), fullPage: true });
await page.setViewportSize({ width: 1280, height: 900 });
await clickSection('user/reading', 'Reading');
await page.getByLabel('Skip chapter cover pages', { exact: true }).check();
await page.waitForFunction(async () => (await (await fetch('/api/me')).json()).settings.skipCovers === true);
await page.screenshot({ path: path.join(shots, 'desktop-reading.png'), fullPage: true });
await clickSection('user/notifications', 'Notifications');
const webhook = page.locator('input[type=url]');
await webhook.fill('https://example.test/unsaved-draft');
await clickSection('admin/scraper', 'Scraper');
await page.getByRole('heading', { name: 'Activity', exact: false, level: 2 }).waitFor();
assert.equal(await page.locator('[data-scraper-tab]:visible').getAttribute('data-scraper-tab'), 'activity');
await page.getByRole('link', { name: 'Sources', exact: true }).click();
const min = page.locator('[data-scraper-tab=sources] input[type=number]').first();
await min.fill('1.5');
await page.getByRole('link', { name: 'Updates', exact: true }).click();
await page.getByRole('link', { name: 'Sources', exact: true }).click();
assert.equal(await min.inputValue(), '1.5', 'Tab switch discarded a draft');
await page.waitForTimeout(3300);
assert.equal(await min.inputValue(), '1.5', 'Polling discarded a draft');
await page.getByRole('button', { name: 'Save WeebCentral', exact: true }).click();
await page.waitForFunction(() => document.querySelector('[data-scraper-tab=sources] input[type=number]')?.value === '1.5');
const state = await page.evaluate(() => fetch('/api/admin/scraper').then(r => r.json()));
assert.equal(state.settings.sourceRates.weebcentral.minMs, 1500);
await noOverflow();
await page.screenshot({ path: path.join(shots, 'desktop-hosters.png'), fullPage: true });
await clickSection('user/notifications', 'Notifications');
assert.equal(await page.locator('input[type=url]').inputValue(), 'https://example.test/unsaved-draft');
await page.waitForTimeout(300);
const before = requests.filter(p => p === '/api/admin/scraper').length;
await page.waitForTimeout(3300);
assert.equal(requests.filter(p => p === '/api/admin/scraper').length, before, 'Hidden scraper keeps polling');
await page.goBack();
await page.locator('[data-scraper-tab=sources]:visible').waitFor();
await page.goForward();
await page.getByRole('heading', { name: 'Notifications', exact: true, level: 1 }).waitFor();
await clickSection('user/reading-time', 'Reading time & privacy');
await page.getByText('Track and show the Active Reading Clock').waitFor();
await clickSection('admin/library', 'Server & library');
await page.getByRole('button', { name: 'Full rescan', exact: true }).waitFor();
assert.equal(await page.getByRole('button', { name: 'Save user', exact: true }).count(), 0);
await clickSection('admin/users', 'Users & access');
await page.getByRole('button', { name: 'Save user', exact: true }).waitFor();
assert.equal(await page.getByRole('button', { name: 'Full rescan', exact: true }).count(), 0);
await page.locator('.settings-menu a[href="collections.html"]').click();
await page.getByRole('heading', { name: 'Collections & metadata', exact: true, level: 1 }).waitFor();
assert.equal(await page.locator('.settings-menu a[aria-current=page]').getAttribute('href'), 'collections.html');
await page.locator('.settings-menu a[href="health.html"]').click();
await page.getByRole('heading', { name: 'Service health', exact: true, level: 1 }).waitFor();
await page.getByText('No failed jobs.', { exact: true }).waitFor();
await page.setViewportSize({ width: 390, height: 844 });
await noOverflow();
await clickSection('admin/scraper', 'Scraper');
await page.getByRole('heading', { name: 'Activity', exact: false, level: 2 }).waitFor();
await noOverflow();
await page.screenshot({ path: path.join(shots, 'mobile-queue.png'), fullPage: true });
await page.getByRole('link', { name: 'Sources', exact: true }).click();
await noOverflow();
await page.screenshot({ path: path.join(shots, 'mobile-hosters.png'), fullPage: true });
// Create/login a fixture-only ordinary reader. No production accounts are touched.
const result = await page.evaluate(async () => {
const post = (p, b) => fetch(p, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(b) });
const created = await post('/api/admin/user', { name: 'ordinary-qa', password: 'disposable-reader-test', admin: false });
if (!created.ok) return created.status;
await post('/api/logout', {});
return (await post('/api/login', { name: 'ordinary-qa', password: 'disposable-reader-test' })).status;
});
assert.equal(result, 200);
const readerRequestStart = requests.length;
// Change the document URL, not just its hash, after switching fixture accounts.
await page.goto(url + '/settings.html?qa=ordinary#admin/users');
await page.getByRole('heading', { name: 'Reading', exact: true, level: 1 }).waitFor();
assert.equal(await page.locator('.settings-menu-group').count(), 1);
assert.equal(await page.locator('.settings-menu a[href*="admin"]').count(), 0);
await clickSection('user/reading-hours', 'Reading hours');
await page.getByRole('heading', { name: 'Server reading hours' }).waitFor();
assert.equal(await page.evaluate(() => fetch('/api/reading-schedule').then(r => r.status)), 200);
await noOverflow();
assert.equal(requests.slice(readerRequestStart).filter(p => p.startsWith('/api/admin')).length, 0, 'Ordinary reader fetched admin data');
assert.equal(await page.evaluate(() => fetch('/api/admin').then(r => r.status)), 403);
await noOverflow();
assert.deepEqual(errors, []);
console.log('PASS: settings desktop/mobile navigation, deep links, role boundaries, draft retention, saves and polling lifecycle');
console.log('Screenshots: ' + shots);
} catch (e) {
await page.screenshot({ path: path.join(shots, 'failure.png'), fullPage: true }).catch(() => {});
throw e;
} finally { await browser.close(); }
};
+11
View File
@@ -0,0 +1,11 @@
#!/usr/bin/env node
'use strict';
const path = require('path');
const { loadQuestionBank } = require('../reading-access');
const file = process.argv[2];
if (!file) { console.error('Usage: node tools/validate-reading-quiz.js <catalog.json>'); process.exit(2); }
try {
const bank = loadQuestionBank(path.resolve(file));
console.log(JSON.stringify({ works: Object.keys(bank).length,
questions: Object.values(bank).reduce((count, questions) => count + questions.length, 0) }));
} catch (error) { console.error(error.message); process.exitCode = 1; }
+187
View File
@@ -0,0 +1,187 @@
# Discovery Scout — personalized manga discovery
Status: **deterministic first slice implemented; later phases remain a vision**. Updated 2026-10-09. See [`docs/discovery.md`](../docs/discovery.md) for the current contract and deliberate omissions.
## 1. Outcome
Give each MangaCircus user a small, explainable feed of manga they might genuinely want to read. It should discover beyond the existing library and the historical *Shijō Saikō no Manga* list without turning either into an automatic download list. A recommendation must answer **why this user might care**, show where its factual claims came from, and admit uncertainty. The user can express interest or dismiss it. An admin may then choose an edition and enter the existing scraper's **inspect → review → confirm download** flow.
This is a discovery and curation feature, not a new image scraper, metadata authority, collection curator, or health watchdog. It must remain useful with no LLM configured.
## 2. Architectural fit and non-goals
Current MangaCircus constraints remain: plain Node and browser JavaScript, Docker Compose, a folder-tree library, JSON state rather than a database, one reader owner of account state, one scraper owner of the queue and library publication, and source-specific slow request pacing. See [`docs/architecture.md`](../docs/architecture.md), [`docs/greatest.md`](../docs/greatest.md), and [`docs/scraper-service.md`](../docs/scraper-service.md).
MVP lives in the **reader service** as a bounded asynchronous job and a reader-owned JSON store. Do not add a permanently running fourth service at first. If real LLM or metadata latency later makes reader isolation necessary, move the job behind the same API to a one-off/optional worker container using the existing image; do not change the data contract or add a database merely to do that. Sentinel observes availability but does not judge recommendations, call an LLM, or repair research claims.
Explicit non-goals for the first release:
- No automatic queue edit, download, cover scraping, collection edit, or relationship approval.
- No bulk crawl of MangaDex, WeebCentral, or other chapter/image hosters.
- No autonomous browsing agent with unrestricted URLs or filesystem access.
- No public/community recommendation ranking based on other users' private reading times.
- No claim that a title search proves a particular edition or chapter availability.
## 3. User experience
Add a **Discover** page, reachable from the main navigation. Keep it separate from the library dashboard and the fixed Greatest survey. It displays at most 20 current suggestions per user, in a compact tile grid with cover, title, original release year, status, and one personalized sentence. A detail panel shows:
1. **Why for you:** specific connections to explicit votes, owned/read titles, creators, or tags; distinguish an editorial hunch from a sourced fact.
2. **Evidence:** linked public metadata or publisher/award sources, with observed date and uncertainty. Do not show a bare score as proof of quality.
3. **Already here?** owned-library match, existing Greatest entry, and—admin only—scraper queue/inspection state.
4. **Actions:** Interested, Maybe, Not for me, Hide; admin-only **Find an edition** reuses the existing source-link/inspection UI. A user can undo feedback.
Explain why a suggestion disappeared: acquired, dismissed, stale source identity, or superseded by a newer run. The feed should not jump while the user is reading it; an active run publishes a complete replacement atomically. Empty state should say whether no candidate passed filtering, metadata is unavailable, or discovery has never run. Show the last successful run time and a nonintrusive error summary; retain the previous successful feed on failure.
User-facing toggles: **Personalized discovery** (default off until user opts in), **Use reading activity** (separate opt-in; default off), and **Show mature titles** (default off). Disabling personalization stops new profile-based runs for that user but retains or deletes their feedback according to a clear Manage Data control. Ordinary users can view and rate their own suggestions. Admins can run discovery, configure sources/budgets, and inspect diagnostics. The first account/admin is not implicitly enrolled in personalized processing.
## 4. Evidence and candidate pipeline
Use a two-stage pipeline. **Code** gathers, identifies, filters, and computes reproducible signals; an optional **agent** explains or reranks a small shortlist. The agent never resolves identity by itself.
### Candidate input, in priority order
1. Reuse public AniList metadata/relations already cached by MangaCircus, plus the read-only Greatest catalog, for cold start and sequel/creator leads. Existing *Greatest* votes are stronger preference signals than curator rank.
2. Add one or two explicit, public metadata feeds only after checking their API terms and stable identity fields. Cache responses with timestamps. Publisher/award lists may seed editorial candidates, but only with source links and dates.
3. Let an admin seed a candidate by metadata URL or exact provider ID. This is the fallback when public APIs are incomplete. It is **not** a scraper URL and creates no queue item.
MangaDex's metadata API may be used only for an on-demand edition lookup after a recommendation is opened, as in today's Greatest details. WeebCentral is not a discovery source. Neither image hoster is asked for chapter catalogs or image URLs during recommendation generation. On HTTP 401/403/429 or a challenge, stop that provider's run and surface the condition; never rotate identity, proxies, hosts, or user agents.
### Identity and filtering
Prefer stable provider IDs. Store original serialization year and type so a sequel, remake, novel, anime, or namesake does not silently inherit another manga's reputation. Exact ID beats title; normalized title plus matching year/type may be a **candidate** match, never an automatic merge. Ambiguous matches go to an admin review list. Deduplicate against the library, Greatest entries, current scraper queue, dismissed titles, and prior suggestions. A related work can still be suggested, but must be labelled as a sequel/prequel and never assume the user has read the prerequisite.
No fabricated covers, rankings, awards, chapter counts, or source URLs. Missing data stays missing. The browser can lazy-load a provider's cover URL, as the Greatest page does; the Scout job does not download cover images.
### Scoring before any LLM
Compute an inspectable score from bounded signals, then deliberately diversify the shortlist:
| Signal | Use | Caution |
|---|---|---|
| Explicit Interested/Maybe/Not for me and Greatest votes | Strongest positive/negative preference | A skull is a title dismissal, not a ban on the whole genre |
| Owned and actually read manga | Moderate similarity and creator/series links | Ownership alone is weak evidence of taste |
| Follows and completed chapters | Moderate positive signal | No assumption that a long series is loved |
| Optional reading activity | Small, normalized signal only | Raw minutes favor long chapters and slow readers; never compare users |
| Public recognition and editorial seed | Candidate quality prior | A prize or rank is evidence, not a personalized recommendation |
| Novelty/diversity | Reserve roughly 20% of slots outside the strongest inferred taste cluster | Avoid 20 near-identical dark-SF picks |
Save score components for debugging but show prose rather than a pseudo-scientific percentage. Seed a new account with a short taste picker and the Greatest votes; do not wait for months of reading history. Rejected or hidden titles should not reappear on the next run merely because metadata changed.
## 5. Optional agent contract
The first release can use deterministic templates for “why for you.” Add a provider-independent LLM adapter only after the deterministic pipeline and UI are trusted. A run sends at most the top 30 candidates, a **minimized** account-specific taste summary, and evidence IDs/URLs. Do not send usernames, passwords, tokens, raw browsing logs, notes, exact reading timestamps, or the whole library file tree. Make external-model processing opt-in and identify the provider in settings.
The model's permitted work is to: (a) choose a diverse subset, (b) write concise personalized rationales grounded in supplied evidence, (c) flag identity/relation conflicts and uncertainty. It may suggest a public source to check, but it may not create a “verified” fact or direct series URL from memory. Treat model output and fetched pages as untrusted data, never as instructions. No tool access to the scraper, filesystem writes, container runtime, arbitrary HTTP destinations, or admin credentials.
Accept only schema-valid output, for example:
```json
{
"candidateId": "anilist:12345",
"reason": "Because you marked BLAME! as a favorite, this explores similarly spare architectural SF.",
"reasonKind": "personal-fit",
"evidenceIds": ["taste:greatest:blame-1997", "metadata:anilist:12345"],
"uncertainty": "Edition and English availability not checked"
}
```
Validate that `candidateId` and every `evidenceId` were in the input, that the reason is short and contains no unsupported objective claim, and that URLs belong to the cited evidence. Reject or fall back to the deterministic template on invalid output, timeout, budget exhaustion, or provider failure. Record model name/version and prompt schema version **without** storing the full private prompt. No recursive retries or self-initiated second research pass.
## 6. Data ownership and proposed contract
Keep the reader as the sole writer. Add `data/discovery.json` for global candidate records, per-user published feed snapshots, run metadata, and a schema/revision number. Write through a temporary file and atomic rename; preserve a recoverable previous version as existing JSON stores do. Keep public metadata responses in a separate bounded `data/discovery-cache/` with TTL and size cap. Store each user's explicit feedback/consent in `data/db.json` (for example `discoveryFeedback` and settings keys) so account deletion and backup rules remain consistent. Stable candidate IDs are `provider:id`; never use a normalized title as the durable key.
Minimal record shape (illustrative, not a migration already made):
```json
{
"schema": 1,
"revision": 12,
"candidates": {
"anilist:12345": {
"title": "Example Manga", "year": 2024, "type": "manga",
"metadataUrl": "https://anilist.co/manga/12345",
"coverUrl": null, "fetchedAt": "2026-10-09T12:00:00Z",
"evidence": [{ "id": "metadata:anilist:12345", "kind": "metadata", "url": "https://anilist.co/manga/12345" }]
}
},
"feeds": {
"alice": { "generatedAt": "2026-10-09T12:05:00Z", "items": [
{ "candidateId": "anilist:12345", "rank": 1, "reason": "…",
"reasonKind": "personal-fit", "evidenceIds": ["metadata:anilist:12345"],
"scoreParts": { "explicitTaste": 3, "diversity": 1 } }
] }
},
"lastRun": { "status": "success", "startedAt": "…", "finishedAt": "…", "candidateCount": 40 }
}
```
Reader API proposal:
- `GET /api/discovery` — only the caller's feed, feedback, available evidence, and run freshness. Never return another user's taste/profile.
- `POST /api/discovery/feedback` — `{candidateId, value}` where value is `interested|maybe|dismissed|hidden|null`; authenticated, validated, idempotent. `null` undoes feedback.
- `GET /api/admin/discovery` — run health, budgets, candidate/identity-review counts, source errors, last run; no private cross-user raw reading logs.
- `POST /api/admin/discovery/run` — starts at most one bounded run; returns a run ID or `409` if one is active. Manual MVP trigger.
- `POST /api/admin/discovery/settings` — validated enabled flag, schedule, source budgets, optional model selection; no arbitrary URL or prompt injection fields.
Queueing remains through `POST /api/admin/scraper` with `command:"enqueue"` and `inspect:true`, only after a human chooses a **series** URL. Reuse the existing duplicate/status UI and show the admin the source edition before confirmation. The Scout cannot call that route.
## 7. Execution limits and failure behavior
- Manual runs first. Default cap: 20 published suggestions/user, 50 candidate records examined/run, one metadata request at a time, at least 1.5 seconds between uncached requests, 10-minute wall-clock budget. These are starting limits to verify against provider terms and real response rates before enabling a schedule.
- Cache by provider ID; avoid rerequesting unchanged metadata within a 30-day TTL unless an admin explicitly refreshes it. Store `ETag`/`Last-Modified` where supported. Never interpret a timeout as an empty catalog.
- One in-process run lock plus a persisted run ID prevents overlap across browser clicks and restarts. A stale interrupted run is marked interrupted on startup; keep the previous successful feed. Publish only after validation completes.
- Preserve source errors and diagnostics with bounded log size. Do not store sensitive prompts or external response bodies indefinitely. Provider blocks/challenges require user attention, not automated retries or evasions.
- Nightly scheduling comes after the manual path has passed acceptance. Make local-time schedule configurable, persist last-success time, skip duplicate same-day runs after restart, and allow a manual run without disturbing scraper nightly updates. A missed night can run once after startup, never a catch-up storm.
- The feature is optional: reader, library, Greatest, scraper, and Sentinel continue normally if metadata or LLM access is absent.
## 8. Phased implementation and acceptance
### Phase A — deterministic, manual MVP
Implement reader-owned store, stable identity mapping, reuse of cached public metadata/Greatest seed, deterministic signals, explicit feedback, a Discover page, and admin **Run now**. No LLM and no new container. Acceptance: two users see different explainable feeds; a title already owned/queued is filtered or clearly labelled; 20-result and request caps hold; no job can change the scraper queue or library; malformed/corrupt store refuses to be overwritten; tests use mocked metadata and disposable data.
### Phase B — source selection and feedback loop
Connect recommendation details to the existing MangaDex candidate search, WeebCentral manual search, and inspect-first scraper action. Preserve selected source identity and show queue status. Acceptance: wrong-year/remake candidates are not auto-selected; duplicate queue clicks are prevented; inspection requires separate admin confirmation; ordinary users cannot queue; no chapter/image requests occur during discovery.
### Phase C — optional agent-assisted rationale
Add one configurable model adapter, schema validation, evidence-bound output, opt-in, strict time/token budget, and deterministic fallback. Compare at least 30 blind recommendations with and without model rationales for factual support, usefulness, variety, and hallucinated links before default-enabling it. Acceptance: model outage leaves deterministic feed usable; injected metadata text cannot issue commands; ungrounded award/record claims are rejected; no personal raw log leaves the reader.
### Phase D — scheduling and operations
Add nightly schedule only after manual runs are stable. Expose last success/error and a `discovery_run_failed` incident to admin observability, but do not give Sentinel authority to rewrite recommendations or restart the scraper. Add backup/restore instructions for `data/discovery.json` and cache behavior. If runs measurably harm reader latency, move the job into an optional one-off Compose worker with one-writer handoff, not an always-on autonomous service.
### Suggested implementation map
Keep the first code change small and reviewable:
| File or area | Intended responsibility |
|---|---|
| `discovery/index.js` (new) | Identity keys, exclusions, scoring, diversity, bounded run orchestration; no scraper imports |
| `discovery/store.js` (new) | Schema validation, atomic `data/discovery.json` publication, revision and previous-version recovery |
| `discovery/providers/anilist.js` (new) | Reuse the existing public metadata cache and perform only bounded missing lookups |
| `server.js` | Authenticated per-user feed/feedback and admin run/settings routes; keep route handlers thin |
| `public/discover.html`, `discover.js`, `discover.css` (new) | Feed, evidence, feedback, empty/error states, admin run control |
| `public/common.js` | One navigation link; no global discovery polling |
| `test/discovery*.test.js`, disposable browser test | Identity, consent, isolation, budgets, action permissions and UI behavior |
| `docs/architecture.md`, `docs/compose-deployment.md` | Record ownership, backup, local release and rollback once implemented |
The exact file split is a proposal, not a requirement to create empty modules. Start with storage and identity tests, then the deterministic pipeline, then API/UI. Do not add model-provider plumbing, a new Compose service, or a nightly timer in the first implementation slice. A candidate must pass schema validation and duplicate/identity checks before it can appear in a published feed.
## 9. Tests, measurement, and rollout
Unit tests: identity collisions, sequel/remake/year mismatch, explicit negative feedback, diversity, cold start, source-budget accounting, deterministic ranking, JSON recovery, multi-user isolation, schema validation, and model fallback. Integration tests: authenticated APIs, admin-only run/settings, no scraper queue mutation, atomic publish and interrupted-run recovery, source block handling. Browser tests: desktop/mobile tiles, evidence panel, feedback undo, stale-feed/error state, edition-selection handoff. All network tests use fixtures by default; any live metadata smoke test is manual, bounded, and never fetches images.
Measure: suggestion engagement (opened/Interested/Hidden), share of suggestions already known, successful source match rate, per-run API count/latency, cache hit rate, provider errors, reader response latency during a run, and reason rejection rate. Do not optimize a single click-through metric at the expense of variety or trust. Start with the local Docker installation; document reader-only release/rollback and verify the existing queue/scraper remain untouched. VM deployment waits for the separately planned migration.
## 10. Decisions to make before implementation
1. Which public metadata provider(s) and terms are acceptable for ongoing discovery? AniList reuse is the baseline; do not add multiple feeds just for volume.
2. Should all users be able to trigger their own bounded run, or only admins initially? This plan chooses **admin trigger, per-user feeds** for MVP.
3. Which optional model/provider may receive minimized taste summaries, and what monthly budget is acceptable? Without consent or a configured provider, stay deterministic.
4. How long should hidden/dismissed feedback persist after account deletion or title-ID correction? Proposed: until the user undoes it; delete with the account.
These questions do not block Phase A's storage/API/UI skeleton. Any later change to consent, external data transfer, or automatic queue authority requires explicit product review rather than an agent's assumption.
+94
View File
@@ -0,0 +1,94 @@
# First-party source plugins for MangaCircus
Prepared 2026-10-13. The first implementation slice extracts the WeebCentral
parser into a static first-party registry and adds an opt-in, network-isolated
Compose worker. No external plugin was installed, no host was probed, and no
live queue, delay, or container was changed. The broker, generic runner, new
hoster pilot, admin controls, and full deployment validation below remain work
to do; the diagram describes the target rather than the current implementation.
Parser-result URL validation and bounded worker messages were added next. The
opt-in WeebCentral path now validates redirect hops and pins public DNS answers;
source-specific image CDN allowlists and generic broker integration remain pending.
MangaDex metadata, feed selection, catalog availability, and at-home page parsing
now have a second opt-in worker path with fixture parity; its transport still
uses the existing API client and runner.
The MangaDex pilot deliberately retains the original in-process parser as a
rollback path, so source-specific pure parsing is temporarily duplicated. Remove
that duplication only after broader catalog fixture parity and a checkpoint-safe
rollout. No Uchiyomi implementation has been copied: the reviewed Madara engine
requires a challenge-solving fetch layer, and no third host has yet passed the
access/permission and stable-interface gate for an archive adapter.
## Decision
Keep MangaCircus's no-database library, source-aware queue, checkpoints, source ledger, and chapter publisher. Extract source-specific discovery/parsing into versioned, first-party plugins. Review and build every plugin from **our repository**; never install an arbitrary repository, APK, JavaScript package, or remotely supplied parser at runtime. An admin may enable/disable a reviewed plugin, not supply executable code. A catalog match remains distinct from adapter compatibility, actual chapter coverage, and download rights.
The upstream model is Uchiyomi v0.61.1, commit [`726bd9fa6fe0090fd4b5de1cdb2b6926e5382127`](https://github.com/AngeloSha/uchiyomi/tree/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127). Uchiyomi registers a common `search/getSeries/listChapters/getPageUrls` interface across built-in MangaDex, generic site engines, mounted JS source packs, and Suwayomi-backed Mihon/Tachiyomi extensions. We want the interface idea and selected reviewed parsers, **not** its executable-code loader or Suwayomi/FlareSolverr runtime.
## Existing seam and problem
- `scraper/adapters/index.js` already dispatches `match/discover/run`; `scraper/service.js` schedules one job at a time per source, applies scopes and source settings, records evidence, and owns the durable queue.
- `scraper/adapters/{weebcentral,mangadex}.js` parse source catalogs and pages. `scraper/jobs/{weebcentral,mangadex}-archive.js` repeat transport, checkpoint, metadata, and publication steps. `scraper/lib/http.js` stops on 401/403/429. `scraper/lib/publisher.js` makes chapters visible atomically.
- Adding another hoster today means copying a whole archive runner. The plugin work must eliminate that repetition **without** allowing plugin code to write the library or bypass request controls.
## Boundary
```text
Admin/UI -> queue + source ledger -> scraper scheduler -> first-party plugin worker
| ^
v | request/response IPC
HTTP broker ----------+
|
v
host / approved image CDN
scraper scheduler -> library inventory + scope/dedupe -> common downloader
-> existing checkpoint/publisher -> library
```
The worker describes requests and parses replies. It has no library or state mount and no internet-facing network. The scraper-owned broker makes **every** source request, including image requests. It enforces one source lane, configured request/chapter delays, declared hostname allowlists, redirect and resolved-address checks, response size/time limits, and stop-on-401/403/429/challenge behavior. HTTP 5xx and network retry remain centrally bounded. No proxy/identity/user-agent rotation and no challenge solver. A source that requires a protection bypass is reported as unsupported/blocked, not silently routed elsewhere.
For production Compose, run one small `plugin-worker` instance per enabled plugin from the same pinned first-party image. Give each a dedicated internal-only network shared with the scraper (not the general backend or `egress` network); mount no library/data/secrets; use a read-only filesystem, dropped capabilities, no-new-privileges, and memory/PID limits. Do **not** inherit Compose's existing `x-app` anchor, which mounts the scraper token. Only the scraper has egress and library write access. On Windows local development, use the same Compose worker: an ordinary Node child process is crash isolation but **not** a network sandbox. No exposed plugin port or browser-to-worker route.
### Proposed v1 IPC contract
Transport: versioned JSON messages over a private internal HTTP endpoint or stdio; bounded payloads and request IDs. The exact transport is an implementation choice, not exposed to users. A worker is pinned to one `pluginId` and code revision. The host validates every returned object.
```json
{ "v": 1, "id": "req-1", "op": "catalog", "pluginId": "example", "seriesUrl": "https://allowed.example/series/123" }
{ "v": 1, "id": "req-2", "op": "fetch", "url": "https://allowed.example/series/123", "accept": "text/html" }
{ "v": 1, "id": "req-2", "ok": true, "status": 200, "contentType": "text/html", "body": "...bounded..." }
{ "v": 1, "id": "req-1", "ok": true, "title": "...", "chapters": [{ "id": "source-chapter-id", "number": "1", "volume": "1", "url": "https://allowed.example/chapter/1" }], "coverage": "unknown" }
```
Operations: `match` from a declarative manifest, `catalog`, `chapterPages`, optional `search`/`seriesMetadata` later. `chapterPages` returns ordered image descriptors; **only** the common downloader fetches bytes. Plugin output may report `unrecognized-labels`, `external-only`, and confidence/evidence, but may not mark rights verified or override the library's work identity. Plugin failures are typed (`parser`, `catalog-empty`, `network`, `access-block`, `rate-limit`) and visible in the source ledger. Version/schema mismatch disables that plugin lane without losing checkpoints.
Manifest, reviewed with code: stable plugin ID/version, exact series URL matcher, primary hostnames, separately reviewed image/CDN hostnames, optional language, parser fixture set, upstream provenance, license, and support state (`experimental`/`supported`/`disabled`). Hostnames are **not** administrator-provided at runtime in v1. Redirects and DNS answers must be checked again by the broker; declared hostnames alone do not protect against SSRF.
## Uchiyomi code triage
| Upstream implementation | Use in MangaCircus | Rationale |
| --- | --- | --- |
| [`sources/types.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/types.ts) | Adapt **concept**, write our own contract | Its narrow catalog/page operations are valuable; its Cloudflare/image-header capabilities and database-specific assumptions are not our policy. |
| [`sources/engines/madara.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/engines/madara.ts) | Candidate for pure parser extraction | Broad WordPress/Madara markup; separate parsing from `cfGet/cfPost`, test with fixtures, then trial only on a specifically reviewed accessible host. Not a blanket license to scrape every Madara site. |
| [`sources/engines/mangathemesia.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/engines/mangathemesia.ts) | Later candidate | Similar, but source-specific page and image hosts need review; currently solver-coupled. |
| [`sources/engines/manganato.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/engines/manganato.ts) | Later candidate, not first pilot | More variant/fallback logic and an explicit challenge dependency; higher maintenance and source-policy risk. |
| [`sources/mangadex.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/mangadex.ts) | Compare behavior/tests only | MangaCircus already has a MangaDex adapter. Evaluate its language and rate-limit edge cases without replacing working code wholesale. |
| [`sources/suwayomi/*`](https://github.com/AngeloSha/uchiyomi/tree/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/suwayomi), [`loader.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/loader.ts) | Do not port | Designed to run operator-supplied APKs/JS. Conflicts with the first-party-only requirement. |
| [`downloader.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/downloader.ts) | Learn from tests/failure cases, do not transplant | CBZ/database assumptions differ from MangaCircus's folder/checkpoint publisher; its challenge session handling does not fit our policy. |
The upstream `sources/` directory contains no bundled plugin implementations beyond a placeholder; the generic engines above live in its main server. Do not mistake its extension catalog for code licensed by Uchiyomi: separately authored APKs have separately determined provenance and licenses.
### Code provenance and license gate
Uchiyomi is [MPL-2.0](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/LICENSE). **No verbatim or adapted upstream code has been copied into MangaCircus in this preparation.** Before copying any implementation, identify the exact upstream file/revision and third-party notices, preserve attribution/notices, add the MPL license and a `THIRD_PARTY_SOURCES.md` ledger, and keep copied/modified MPL-covered code in clearly marked files. If distributing a Docker image with those files, provide the corresponding source for the covered files as the license requires. Mozilla describes MPL as file-level copyleft; it does not require unrelated MangaCircus files to become MPL. This is an implementation checklist, not legal advice; review the final distribution arrangement before release. Reimplementing an idea from documented behavior is distinct from copying its expressive code, but provenance should still be documented.
## Incremental implementation passes
1. **Contract and broker, no new host.** Add manifest validation and IPC schema, central fetch broker, disabled-by-default worker, fixture-driven contract tests, worker crash/timeouts, SSRF/redirect tests, 401/403/429 stop tests, and checkpoint-preservation tests. Existing adapters continue unchanged.
2. **First-party compatibility plugins.** Migrate MangaDex and WeebCentral catalog/page parsing one at a time behind the contract, with old-vs-new fixture parity. Keep existing job IDs, source IDs, queue order, delays, and work directory paths. The common runner calls the existing inventory/scope/publisher. Rollback is a plugin enable switch, not a queue rewrite.
3. **One reviewed generic parser pilot.** Start with a pure Madara-family parser based on upstream lessons. Choose a single source only after checking accessibility, terms, stable hostnames, and parser fixtures. Inspection/catalog-only first; no auto-queue. Record source evidence, compare with library identity, then test a small chapter in isolated A/B fixtures. Add further families only if they solve a concrete catalog gap.
4. **Admin UI.** Under Scraper → Sources, show installed first-party plugins, version/provenance, enabled state, supported hosts, last probe stage, observed catalog/coverage, rights state, and blocked reason. Enabling requires admin confirmation; no repository URL/APK upload/install button. Source candidates stay leads until reviewed and explicitly queued.
5. **Deployment.** Add the worker to local and VM Compose with internal-only networking and resource limits; run compose checks and end-to-end fixture tests, then stage on a test instance. Leave the live stack untouched until jobs are checkpoint-safe and rollback verified. Document plugin upgrades and recovery.
Acceptance: no plugin has direct internet or library-write capability; one worker per plugin may crash without taking down the reader/scraper; no source request can evade host limits or 401/403/429 handling; existing jobs resume unchanged across a rollback; the library still needs no database; source provenance and MPL obligations are traceable; no dynamic third-party code execution is introduced.
+120
View File
@@ -0,0 +1,120 @@
# Server peering — opt-in manga sharing
Status: **v1 implemented and isolated-tested on 2026-10-09**. This document remains
the broader vision; the exact shipped behavior, deployment and limitations are in
[`docs/peering.md`](../docs/peering.md). VM deployment remains future work.
## 1. Outcome and boundaries
Two MangaCircus servers can pair, show each other a deliberately limited manga catalog, and transfer selected chapters into the receiving server's normal library. An admin on the publishing server must explicitly enable **each manga** for sharing; the default is private. A reader on the receiving server can browse and request downloads, subject to local permissions. Neither server sends user accounts, reading progress, ratings, notes, collections, session cookies, or scraper credentials.
This is **server-to-server library transfer**, not synchronization. The publisher never writes to the receiver. The receiver never obtains unrestricted filesystem access. No automatic reciprocal sharing, title discovery, or mirroring follows from pairing.
The publishing server will necessarily learn the receiving **server's** identity/address, requested title/chapter IDs, byte counts, and request times. “No user data exchange” means the protocol carries no reader identity or activity records; it cannot make those unavoidable transfer observations disappear. The local server mediates browser requests so the reader's browser does not contact the remote peer.
Only share material the server operator is entitled to distribute. Peering must not be used to evade a source host's access controls or to export scraped credentials or source URLs.
## 2. Fit with MangaCircus
Preserve the current [architecture](../docs/architecture.md), [library format](../docs/library-format.md), and [scraper ownership](../docs/scraper-service.md): the reader owns accounts and admin settings, the scraper owns its queue and atomic chapter publication, and the library remains folders plus JSON rather than a database. Do not introduce a second writer to the library or scraper state.
Add one optional **peer gateway** process/container on each server that chooses to publish. It can serve multiple approved peers, receives authenticated peer traffic, reads a *read-only* library mount, exposes only approved catalog fields and opaque content IDs, and enforces upload quotas. It has no accounts database, reader data directory, scraper state, Docker socket, or writable library mount. Its small local state contains only peer grants and quota counters. Admin controls live in the reader UI and are written atomically to a reader-owned peering policy file that the gateway reads; the gateway is the sole writer of usage counters. Use one peer gateway per library, **not** one gateway per peer.
On the receiving side, add a **peer adapter** to the existing scraper daemon. It uses the existing single-owner queue, checkpoints, source-version policy, integrity checks, and publish-on-chapter-complete path. Treat each peer as a source lane, with one transfer worker per peer by default. This is not a separate downloader writing files beside the scraper.
The optional gateway is justified by isolation: an incoming network service should not have access to private reader/scraper data. It stays a small process built from the same project image, not a new platform, database, or agent.
## 3. Trust and pairing
### Recommended mechanism
Use **TLS 1.3 with mutual certificate authentication (mTLS)**. MangaCircus generates its own identity key and X.509 certificate **locally on each server**, through a guided admin action; no external certificate authority or command-line work is required for the initial private-network design. Generate the private key in server-side secret storage with restrictive permissions and show the admin only the public certificate and its fingerprint. Its **private key never leaves that server**. Admins exchange public pairing invitations **outside MangaCircus**—for example in a trusted messenger, on removable media, or by scanning a QR code in person—and compare the certificate/public-key fingerprints through a separate trusted channel. Each server pins the other server's public key and rejects a different key, even if DNS or the network route changes. TLS handles ephemeral session-key exchange and encrypted transport; we do not invent a crypto handshake. [TLS 1.3](https://www.rfc-editor.org/rfc/rfc8446.html) defines mutual certificate authentication.
Certificate validity and access authorization are separate:
- The certificate has `notBefore`/`notAfter` dates (proposal: 1 year) and is rejected outside them. Renewing a certificate with the *same* pinned key may be accepted after local validation; replacing the key requires a fresh out-of-band fingerprint check.
- The publisher issues a **random 256-bit opaque grant token** for a specific consumer certificate fingerprint, scopes (`catalog`, `download`), `notBefore`, and `expiresAt` (proposal: 90 days). It stores only a hash of the token and checks it on every request. The grant cannot outlive either certificate. A grant can be revoked immediately without changing the identity key. The token is delivered out-of-band during pairing or across the already authenticated mTLS channel after both admins approve; it is never in a URL or browser response.
- The token is an expiring authorization credential, **not** a second encryption key. A valid token without the matching client private key is useless. An expired/revoked grant, expired certificate, failed pin, or missing client certificate fails closed. No silent insecure fallback or automatic re-pairing.
This uses standard transport cryptography plus an auditable server-side grant record. Do not implement a custom signature protocol or home-grown JWT parser. Use the runtime's verified TLS stack and a cryptographically secure random generator; compare token hashes safely. If a standard certificate-bound token format is later needed for federation, evaluate [RFC 8705](https://www.rfc-editor.org/rfc/rfc8705.html) rather than adding protocol complexity now.
### Admin invitations and pairing flow
1. Admin A opens **Admin → Peering**, selects **Generate identity** once, then **Create invitation** for an intended peer. MangaCircus generates a single-use, short-lived invitation (proposal: 48 hours) containing A's server name, reachable peer URL, public certificate, fingerprint, invitation ID, and one-time pairing code. The invitation can be copied as text/file or shown as a QR code. It contains **no private key and no catalog/download grant**. An admin can cancel it before use.
2. Admin A gives that invitation to B's admin outside the platform. B imports it and verifies A's fingerprint through a separate trusted channel. B generates its own local identity if needed and returns its own public invitation/card out-of-band. The one-time code links the two pending records and limits accidental pairing with the wrong server; it does not substitute for fingerprint verification or admin approval.
3. Each admin approves the specific peer. A test mTLS connection proves possession of both private keys and checks pins, names, certificate validity, and clock skew. An expired or already-used invitation cannot create another peer. A new invitation is required for a different server; each server can have **multiple independent peers**, but importing an already-paired fingerprint must resolve to the existing peer rather than silently creating a duplicate.
4. Each admin separately sets **Serve to this peer** on or off (default off), selects which manga/source versions that peer may see, and sets that peer's upload limits. Serving on enables issuance of A→B's time-limited catalog/download grant; serving off revokes/blocks that direction immediately. B stores A's token in server-only secret storage. If B also serves A, B issues an independent B→A grant and chooses its own share list and limits. One direction can be enabled without the other.
5. Each admin may also set **Allow downloads from this peer** on or off locally (default off until paired and approved). This controls whether their own readers can browse/queue that peer's catalog. Turning it off pauses existing local jobs from that peer at a checkpoint; it does not delete queued work or already downloaded chapters. It is independent of the other server's serving choice and does not grant the other server access.
6. The UI shows **pending / paired but inactive / serving / receiving / both directions / expires soon / expired / revoked / fingerprint changed**. Revocation closes new transfers immediately; an in-flight transfer is stopped at the next chunk/checkpoint. Key rotation pauses until new out-of-band verification.
Pairing is a **many-peer registry**, not a two-server global switch. Each peer has its own pinned identity, invitation history, direction switches, grant, manga allowlist, limits, queue status, and audit entries. Disabling one peer or one direction does not alter any other peer. Removing a peer invalidates both local directions and its pending invitations, but never erases already published chapters.
| A's Serve to B | B's Serve to A | Result, assuming each receiver locally permits downloads |
|---|---|---|
| Off | Off | Paired identity only; neither catalog is readable. |
| On | Off | B can browse/download A's approved manga; A cannot browse B's. |
| Off | On | A can browse/download B's approved manga; B cannot browse A's. |
| On | On | Both can browse/download, using separate grants, selections, and limits. |
An admin may revoke a peer, a grant, or one manga's share independently. Removing a share withdraws its catalog and blob access immediately, not merely on the next catalog refresh. Already downloaded copies on the other server are not remotely deleted.
## 4. Shared catalog and content protocol
Version the protocol from day one (`/peer/v1`). Send a *projection* of library data, never raw `manga.json`, directory listings, absolute paths, or account/reader state.
| Operation | Minimum payload / behavior |
|---|---|
| Paged catalog | Publisher-scoped opaque manga ID, title, release year/status if present, compact description, optional approved cover thumbnail, chapter/volume counts, catalog revision, next cursor. Only admin-approved manga. |
| Manga manifest | Approved volume/chapter IDs, labels, order, page counts, sizes, content revision and per-page SHA-256 hashes. No upstream host credentials or original image URLs. |
| Content stream | Opaque blob ID from an authorized manifest; bounded-size image bytes, MIME type, hash, ETag, and HTTP range support for restartable transfers. Never accepts a filesystem path from the client. |
Sharing a manga covers its **completed, validated chapters only**. Hidden staging/partial downloads, unrelated files, and currently writing pages never enter the manifest. A completed new chapter becomes visible after the scraper's atomic publication and gateway index refresh. Revocation or share removal wins over a previously issued manifest/blob ID.
Catalog snapshots use cursor/revision pagination and bounded page sizes. A revision change causes a clean refresh, not an unbounded rescan per browser click. Local titles can be matched to remote titles only as **candidates**; uncertain same-name editions require the reader/admin to choose the correct local manga. Store publisher-specific IDs and content revisions to avoid conflating two sources with the same chapter number.
Input from a peer is untrusted even after pairing: cap manifests, chapter/page sizes and total transfer size; validate schemas and image MIME/content; reject traversal paths, archives/executables, and malformed names; stream rather than buffering a whole manga; verify each page hash before publication. Hashes detect corruption but do not prove editorial correctness. Never overwrite an existing chapter silently.
For transport efficiency, start with original image bytes and page-level resume. A [local compression test](server-peering-compression-test.md) found an average ~6% reduction with zstd-3 but a wide 0.7–10.7% spread by chapter, so codec negotiation is an optional later optimization, not a prerequisite for peering. Measure actual chapter-ready time over a real peer link before adding chapter packing or automatic compression.
## 5. Reader and queue experience
Add **Peer catalogs** as a reader-facing browse surface, separate from the local dashboard. Show one tile/tab per peer that currently serves this server and is locally allowed for downloads; peers that are only receiving from us do not appear as remote catalogs. A peer tile shows the publishing server's display name and grant status; search and manga details are fetched through the local reader backend. A reader can select a whole manga, a volume, individual chapters, newest missing, or a bounded chapter range using the existing fine-grained queue model. The UI previews expected chapters/bytes and highlights what is already local. Local role permissions still govern who may submit to the shared scraper queue; browsing does not itself start a transfer.
The resulting job stores the peer ID, remote manga/chapter IDs and revision, selected scope, duplicate policy, and progress checkpoint. Default policy is **skip matching existing content**; an explicit “add as another source” choice preserves the source variant when the same chapter exists from elsewhere. On receipt, the scraper publishes each complete chapter immediately through its existing atomic path, making it readable before the entire manga finishes. Interrupted transfers resume by validated ranges/checkpoints and never expose partial pages in the library.
No automatic remote refresh in v1. A future admin opt-in may schedule checks for *already selected* peer manga, but must use the same queue, grant, and rate policies. No peer request may be triggered by reading a page or opening the local dashboard.
## 6. Upload and download limits
The **publisher sets hard ceilings** per peer and globally: concurrent streams, bytes/second (token bucket), bytes/day, catalog requests/minute, allowed time window, max response size, and idle/transfer timeout. Proposed starting defaults: one stream per peer, two streams globally, 2 MiB/s per peer, and 10 GiB/day per peer. These are product defaults, not a promise about available bandwidth; admins can lower or raise them. Quotas and current usage must be visible, with a “pause this peer” control. Persist daily counters atomically across restarts; a restart must not reset a used quota.
The **consumer can only be stricter**: one active transfer per peer by default, local bandwidth cap, chapter spacing, time-of-day window, maximum queued bytes/free-space reserve, and pause/resume. The effective rate is the lower of the two sides' caps. Respect `Retry-After` on rate limits and back off on transient failures; never retry authentication failure, revocation, or fingerprint mismatch as if they were network glitches. Fair-share scheduling prevents one peer or one huge manga from starving the local source lanes. No same-peer parallel image workers by default.
The gateway responds with clear, bounded errors: unauthorized/expired, revoked, not shared, quota exhausted with retry time, stale manifest, and transient unavailable. The receiver surfaces these separately in queue status and retains checkpoints. Logs and metrics identify a peer by local alias/fingerprint prefix, not a token or private key.
## 7. Deployment and data ownership
Add an optional `peer-gateway` Compose profile/service. It mounts `/library` read-only, a read-only policy projection, gateway-owned counter storage, and certificate/token secrets. It has no mount for `data/db.json`, the reader's account store, or `data/scraper`; the scraper still alone has writable library access. Start with a listener reachable only on the operator's intended private network. Do not expose the scraper's private API or reuse the normal reader web route for binary transfer.
For the future VM, prefer a **tailnet-only raw TCP forwarder to the gateway's own mTLS listener**. Tailscale Serve's `--tcp` mode forwards raw TCP, whereas `--tls-terminated-tcp` ends TLS at Serve and is **not** the default design for end-to-end mTLS pinning. [Tailscale's Serve documentation](https://tailscale.com/docs/reference/tailscale-cli/serve) distinguishes the two. Tailnet reachability is defense in depth, not a substitute for pairing, grants, and quotas. Keep the existing Caddy reader reverse proxy separate; only consider public peering later with a dedicated mTLS endpoint, firewall policy, and an explicit threat review. Exact VM ports/hostnames await deployment configuration.
Suggested NoDB files (names are design targets, not existing files): reader-owned `data/peering/policy.json` for peers, share flags, and limit settings; gateway-owned `data/peering-gateway/usage.json` for quota counters; scraper-owned existing queue/checkpoints for receiver jobs. Private keys and grant tokens live in excluded secrets storage, not policy JSON, logs, exported cards, or Git. Atomic replace plus bounded backups for JSON; one writer per file. Backups of identity keys/tokens must be encrypted and restorable only by the operator. Losing a private key means re-pairing, not copying another server's key.
## 8. Implementation sequence and acceptance gates
1. **Contracts and threat model:** define the exact catalog/manifest schemas, stable peer IDs, many-peer policy model, independent direction switches, duplicate semantics, grant lifecycle, filesystem limits, and forbidden fields. Add protocol fixtures and negative tests before network code.
2. **Identity and admin controls:** generate local keys/certs, expiring single-use public invitations, verified import, pinning, grant expiry/revocation, per-peer serving/download controls, per-manga opt-in, and audit events. Test wrong pin, missing client cert, invitation reuse/expiry, clock/expiry, key rotation, token replay with a different certificate, and immediate revocation of one direction without affecting another peer.
3. **Read-only publisher:** optional gateway, bounded approved-only catalog/manifest/blob endpoints, quota enforcement, read-only mounts. Test that unshared, partially downloaded, deleted, and revoked content cannot be listed or fetched—even by guessed IDs or stale manifests. Verify no user data fields or raw source URLs appear on the wire.
4. **Consumer browsing:** local-server proxy and reader UI with peer status, search, details, selection preview, and queue permission checks. No browser-to-peer credentials or direct requests.
5. **Scraper transfer lane:** chapter/page checkpoints, range resume, hashes, local throttles, source-aware deduplication, and existing atomic chapter publication. Test corruption, interrupted transfer, stale revision, out-of-space, repeated queue submission, and preservation of current jobs.
6. **Observability and operations:** per-peer health, bytes, throttling, grant expiry warnings, queue stalls, and safe retry controls. Sentinel may report a dead gateway or stuck transfer but must not renew trust, broaden a share, or bypass a quota. Document backup/restore and key-rotation runbooks.
7. **Local then VM pilot:** three local Compose instances with distinct data directories; pair A↔B and A↔C, exercise every direction combination, and transfer a few admin-approved test chapters. Verify disabling B does not affect C, no account/progress exchange, and quota isolation. Only then deploy the optional service on the VM's private network. Leave existing reader/scraper ports and active downloads undisturbed during the pilot.
**Ship gate:** a fresh unpaired server sees nothing; a paired server with serving off or no granted manga sees an empty catalog; an expired/revoked peer cannot list or fetch; invitations cannot be reused; disabling one direction or peer leaves unrelated peers intact; quotas survive restart; a receiver can resume and publish one chapter without duplicates; and its browser never receives the grant token or any private key. The feature remains disabled by default until those gates pass.
## 9. Decisions to settle before implementation
- Who may queue from a peer: any authenticated reader, or only readers with a new download permission? Recommendation: explicit local `canQueuePeerDownload` permission, admin-controlled.
- Does sharing approval apply to every source version of a manga or a chosen edition? Recommendation: opt in **per manga and source version**, with a clear “all currently complete chapters / selected volumes” scope; new chapters follow that scope only if the admin opted into future additions.
- How long should grants last, and should renewal require out-of-band confirmation? Recommendation: 90 days with an admin-visible renewal that retains the same pinned key; any changed key requires out-of-band re-verification.
- How should two distinct MangaCircus accounts on different servers appear to each other? Recommendation: **they do not**. The publisher grants a server, and the receiver alone decides which of its users may browse and queue.