Consolidate MangaCircus app, scraper, and Compose baseline
This commit is contained in:
Maashkinoozhe committed
1 parent
cfb2d79103
commit
eef4ffa07c
103 files changed
+9316
-172
No files matched your search
+81
-17
@@ -1,11 +1,20 @@
|
||||
# MangaCircus architecture
|
||||
|
||||
MangaCircus is a self-hosted reader for manga, comics and webtoons. One Node.js process serves a handful of static pages and a small JSON API. The content is a folder tree on disk, and everything user-related lives in one JSON file.
|
||||
MangaCircus is a self-hosted reader for manga, comics and webtoons. The reader owns
|
||||
accounts/progress and a separate collection catalog. The scraper owns the persistent
|
||||
queue and chapter publication. Sentinel observes health/progress and requests bounded
|
||||
checkpoint recovery. Separate supervisors isolate service failures. All content remains
|
||||
a plain folder tree; there is no database server or embedded autonomous LLM.
|
||||
|
||||
Related documents:
|
||||
- [library-format.md](library-format.md): the folder format.
|
||||
- [import.md](import.md): how the folder tree is indexed.
|
||||
- [scraper-guide.md](scraper-guide.md): rules for tools that write the library.
|
||||
- [scraper-service.md](scraper-service.md): managed queue, private API, and adapter contract.
|
||||
- [collections.md](collections.md): relations, groups, curator UI and agent research.
|
||||
- [resilience.md](resilience.md): supervision, recovery limits and incidents.
|
||||
- [compose-deployment.md](compose-deployment.md): local/VM deployment and release workflow.
|
||||
- [settings-navigation.md](settings-navigation.md): User/Admin workspace, scraper tabs and deep links.
|
||||
|
||||
## 1. Principles
|
||||
|
||||
@@ -13,7 +22,7 @@ Related documents:
|
||||
|-----------|-------------|
|
||||
| **The file system is the source of truth for content** | No import step, no copies, no content database. The library can be mounted read-only. Anything that writes files can feed it |
|
||||
| **As dumb as possible** | No dependencies (`package.json` has none), no framework, no build step, no transpiler. Plain ES2019+ JavaScript in the browser, plain Node on the server |
|
||||
| **One process, one data file** | In-memory library index, user data in `data/db.json`. Back up one file, run one container |
|
||||
| **Small independent services** | The reader stays dependency-free; the optional scraper daemon can stop or restart without taking reading offline |
|
||||
| **Readable without the app** | The folder format works in any picture viewer. The app adds progress, layout and convenience, not lock-in |
|
||||
|
||||
## 2. System context
|
||||
@@ -35,6 +44,18 @@ Related documents:
|
||||
|
||||
## 3. Components
|
||||
|
||||
Runtime topology (2026-10-05): native `tools/start-all.js` launches reader, scraper and
|
||||
Sentinel independently. In Compose, each container runs `runtime/launch.js <service>`
|
||||
and exactly one service child. A failure does not stop unrelated services. Exactly one
|
||||
reader and queue-owning scraper may write each shared state tree; do not scale replicas.
|
||||
The catalog and relation store is reader-owned `data/catalog.json`. Sentinel writes only
|
||||
`data/sentinel/`; it has no Docker socket or arbitrary code-repair authority.
|
||||
|
||||
The admin browser never talks to the scraper daemon directly. Admin-only reader
|
||||
routes proxy commands to the daemon's private API, so its shared token stays on the
|
||||
server. The daemon publishes complete folders atomically and increments a durable
|
||||
revision; the reader polls that revision and rescans the library after publication.
|
||||
|
||||
| Path | Lines | Responsibility |
|
||||
|------|------:|----------------|
|
||||
| `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks |
|
||||
@@ -43,7 +64,7 @@ Related documents:
|
||||
| `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) |
|
||||
| `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle |
|
||||
| `public/read.html` + `reader.js` | ~470 | The reader (§7.2) |
|
||||
| `public/settings.html` + `settings.js` | ~115 | Reading preferences, webhook, password, admin (rescan, users) |
|
||||
| `public/settings.html` + `settings.js` + `settings-nav.js` | — | User/Admin navigation, personal preferences, lazy admin sections and scraper tabs |
|
||||
| `public/login.html` + `login.js` | ~50 | Sign-in, first-run admin creation, optional sign-up |
|
||||
| `public/app.css` | ~150 | All styles (dark theme, responsive) |
|
||||
| `tools/png.js`, `tools/make-dummy.js` | ~310 | Demo library generator: a tiny PNG encoder with a bitmap font, no image libraries |
|
||||
@@ -51,14 +72,15 @@ Related documents:
|
||||
|
||||
## 4. Runtime model
|
||||
|
||||
- **One process, event loop only.** All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests.
|
||||
- **Reader process:** one event loop. All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests.
|
||||
- **Optional scraper process:** one queue worker, one remote request at a time, durable checkpoints, and a private control API.
|
||||
- **In-memory state:**
|
||||
- `lib` is the current library index, replaced as a whole after each scan.
|
||||
- `db` is all user data, loaded from `db.json` at start.
|
||||
- **Timers:**
|
||||
- a library scan at start and every `SCAN_INTERVAL` minutes,
|
||||
- a debounced `db.json` write, 500 ms after the last change.
|
||||
- **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json` synchronously, then exit. That matters for `docker stop`.
|
||||
- **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json`; the daemon cooperatively stops at a page boundary and keeps its archive checkpoint.
|
||||
- **Durability:** every file write goes to `*.tmp` first and is then renamed over the target, so a file is never half-written. A hard kill loses at most the changes of the last ~0.5 s. A corrupt `db.json` makes the server **refuse to start** rather than start empty and overwrite it.
|
||||
|
||||
## 5. Data model
|
||||
@@ -93,6 +115,9 @@ This keeps ids human-readable and derivable without a database. The price: renam
|
||||
"lastMode": "book", // the user's intended mode; changed only by clicking a mode
|
||||
"adapt": true, // use a fitting mode for incompatible content (webtoon → flow)
|
||||
"skipExtras": false, "skipCovers": false,
|
||||
"flowPageGap": true, // show the 8 px separator between pages in Flow
|
||||
"readingClock": false, // opt in to credible per-chapter active-time tracking
|
||||
"readingStory": true, // show private retrospective summaries when history exists
|
||||
"zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 }, // page zoom per mode
|
||||
"webhook": ""
|
||||
}
|
||||
@@ -106,6 +131,12 @@ This keeps ids human-readable and derivable without a database. The price: renam
|
||||
"done": false, // sticky once true (reset via "mark unread")
|
||||
"shift": [] // book mode: run starts shown alone
|
||||
} } } },
|
||||
"reading": { "admin": { "Moon Garden": { "1/2": {
|
||||
"schema": 1, "totalActiveMs": 412000, "loadingMs": 7400,
|
||||
"sessions": 3, "completedPasses": 2,
|
||||
"inputMs": { "mouse": 12000, "touch": 380000, "keyboard": 20000 },
|
||||
"currentPass": { "version": "c0002 Roots [Official]", "seenRanges": [[0, 13]], "activeMs": 151000 }
|
||||
} } } },
|
||||
"follows": { "admin": { "Moon Garden": true } },
|
||||
"notes": { "admin": [ { "id": "a1b2c3d4e5f6", "t": 1790797300000, "m": "Moon Garden", "c": "2/6",
|
||||
"text": "Moon Garden · Ch. 6: Frost", "read": false } ] },
|
||||
@@ -142,10 +173,13 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
|
||||
| `GET /api/me` | user | Name, admin flag, settings, unread count |
|
||||
| `POST /api/settings` `{…partial}` | user | Validated update of the settings keys above |
|
||||
| `POST /api/password` `{old, new}` | user | Change password |
|
||||
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read |
|
||||
| `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions and the user's progress, `continue` target |
|
||||
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
|
||||
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read, and optional personal Reading Story aggregates |
|
||||
| `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions, progress, optional chapter-time summaries, and `continue` target |
|
||||
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, optional Active Reading Clock summary, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
|
||||
| `POST /api/progress` `{m, c, v, p, n, done, shift}` | user | Save the reading position |
|
||||
| `POST /api/reading/pulse` `{m, c, v, session, seq, …deltas}` | user | When opted in, idempotently merge a clamped active-time pulse, verified page coverage and input totals under a single-reader lease |
|
||||
| `GET /api/reading/export` | user | Export the user's private derived reading-time summary without raw input or page-exposure diagnostics |
|
||||
| `POST /api/reading/delete` `{m}` or `{all:true,password}` | user | Delete one manga's reading history, or all history after password confirmation, without changing normal reading progress |
|
||||
| `POST /api/unread` `{m, c}` | user | Forget progress for a chapter |
|
||||
| `POST /api/vote` `{m, c, v, value: 1\|-1\|0}` | user | Vote a version; returns the re-ranked versions |
|
||||
| `POST /api/follow` `{m, on}` | user | Follow a manga for notifications |
|
||||
@@ -237,9 +271,9 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
|
||||
| Click or tap on a page, top/bottom 35 % of the screen (flow) | Scroll to the previous/next page top, at most 90 % of a screen (85 % steps for webtoons). At the very bottom → next chapter |
|
||||
| Click or tap on a page, middle band | Toggle the bars |
|
||||
| Horizontal swipe ≥ 50 px within 800 ms (paged, not zoomed in) | Previous/next |
|
||||
| Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Page zoom 20–600 %, snapping to 100 % within ±3 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages |
|
||||
| Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Continuous page zoom 20–600 %, snapping to 100 % within ±1 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages |
|
||||
| Mouse drag (moved ≥ 6 px) | Pan; the following click is swallowed. Hand cursor when the content is larger than the screen |
|
||||
| Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` `-` `0` zoom; `m`/Esc bars; `f` fullscreen |
|
||||
| Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` and `-` zoom in 5-percentage-point steps, `0` fits at 100 %; `m`/Esc bars; `f` fullscreen |
|
||||
|
||||
**Bars:**
|
||||
- Shown on open.
|
||||
@@ -260,22 +294,50 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
|
||||
- Stores the page index in the full page list, so skipping extras doesn't corrupt it.
|
||||
- On open, the saved page is used for the same version, and mapped proportionally for another version. A finished chapter reopens at the start.
|
||||
|
||||
**Active Reading Clock (opt-in):**
|
||||
- Runs only while the reader is visible and focused, a sufficiently exposed page is decoded, a server lease is held, and accepted reader input occurred in the last 50 seconds.
|
||||
- Backgrounding or blurring pauses immediately and requires fresh input to resume. A delayed callback can never add time beyond the 50-second input deadline.
|
||||
- A page is verified after at least 35% meaningful exposure for 200 ms cumulatively. Exposure is closed and evaluated when the page leaves the viewport, so a fast but plausible webtoon scroll is not lost between samples. A five-pages-per-second token bucket rejects implausible coverage without discarding otherwise credible active time.
|
||||
- `BroadcastChannel` coordinates tabs on one browser. The authenticated API adds a 15-second per-user lease for other devices, and releases it when the tab pauses or leaves.
|
||||
- Pulses are sent at most every 10 seconds and on pause/page exit. The server validates the manga/chapter/version and page indices, de-duplicates `(session, seq)`, clamps time to server-observed intervals and stores compact page ranges separately from ordinary progress.
|
||||
- The bottom toolbar chip shows active time and the current state; its panel adds pass time, verified pages, predominant input and loading wait.
|
||||
|
||||
**Zoom persistence:** per mode key (`flow`, `page`, `book`, `strip`), saved 800 ms after the last change.
|
||||
|
||||
## 8. Deployment
|
||||
|
||||
### Current Windows deployment
|
||||
|
||||
The observed local installation uses the Windows Scheduled Task
|
||||
`\MangaCircus Supervisor`. At logon it runs `tools/run-supervisor.ps1`, which appends
|
||||
logs under `data/scraper/` and starts one `tools/start-all.js` supervisor. That parent
|
||||
owns exactly one reader (`server.js`, port 8080) and one scraper daemon
|
||||
(`scraper/daemon.js`, private port 8091). Starting `npm run start:all` manually at the
|
||||
same time would create conflicting writers and is not supported.
|
||||
|
||||
The scraper's authoritative `data/scraper/queue.json` is schema version 3. It supports
|
||||
legacy archive checkpoints and paused conversion of completed legacy work while adding
|
||||
source-aware lanes, scoped queue jobs, incremental chapter publication, and nightly catalog discovery. See
|
||||
[local-deployment.md](local-deployment.md) for operational checks and restart/recovery,
|
||||
and [scraper-service.md](scraper-service.md) for the queue/API contract.
|
||||
|
||||
### Docker or manual deployment
|
||||
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
- **Image:** `node:24-alpine` plus the app files. It runs as user `node` (uid 1000) and has a health check against `/api/auth`.
|
||||
- **Images/services:** the same `node:24-alpine` image runs a reader service and a scraper service. The reader health check targets `/api/auth`.
|
||||
- **Volumes:**
|
||||
- `/library`: read-only is enough. Files must be readable by uid 1000.
|
||||
- `/data`: must be writable.
|
||||
- **Configuration:** only environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `COOKIE_SECURE`).
|
||||
- **Without Docker:** `node server.js` with Node ≥ 18.
|
||||
- `/library`: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000.
|
||||
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
|
||||
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
|
||||
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
|
||||
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
|
||||
- **Upgrades:** replace the code and restart. There are no migrations: unknown fields in `db.json` are kept, and missing settings fall back to defaults.
|
||||
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
|
||||
unknown fields in `db.json` are kept. The scraper normalizes old queue entries into
|
||||
schema version 3 on startup and preserves legacy job checkpoint behavior. Take a
|
||||
queue/checkpoint recovery copy before upgrading a live scraper.
|
||||
|
||||
## 9. Capacity and limits
|
||||
|
||||
@@ -308,5 +370,7 @@ docker compose up -d --build
|
||||
- **Notifications:** in-app plus webhook only. Web Push would need VAPID keys and a service worker.
|
||||
- **No login rate limiting and no CSP header.** Both are small additions in `server.js`.
|
||||
- **In-place image overwrites** are only detected by a full rescan.
|
||||
- **No automated test suite.** Verification so far was done with the demo library (`tools/make-dummy.js`), API smoke scripts and manual browser testing on desktop and mobile viewports. `parseKey` and `sizeFromBuffer` are pure functions, and `scanLibrary` only needs a folder. Together with the book pairing they are the natural first targets for unit tests.
|
||||
- **Reader layout remains mostly smoke/manual.** Reading analytics has focused Node tests
|
||||
for its persisted merge rules, while DOM layout, authentication and mobile browser
|
||||
lifecycle behavior still need browser-level/manual verification.
|
||||
- **Adding content by API** (instead of files) would be a new route plus the staging and renaming rules from [import.md §7](import.md#7-consistency-writing-into-a-live-library).
|
||||
@@ -0,0 +1,141 @@
|
||||
# Collections and relationships
|
||||
|
||||
Collections are a presentation layer, not a filesystem migration. Creating, reordering,
|
||||
or removing one never renames manga/chapter folders or changes reading IDs.
|
||||
|
||||
## Using the UI
|
||||
|
||||
Open **Settings → Admin → Collections & metadata** (or **Manage collections** on the dashboard).
|
||||
Create a collection, add library manga, choose each member's role and reading order,
|
||||
and save. Choose a member cover, a mosaic of up to four covers, or upload a custom
|
||||
image under 600 KB. Dashboard display can be grouped, expanded, or individual; the
|
||||
choice is remembered per browser. A collection page shows aggregate progress and
|
||||
continues the first unfinished member in the chosen order.
|
||||
|
||||
**New collection** starts an unsaved draft using an ungrouped library manga. Rename
|
||||
it, add members and choose **Save collection** (or **Save all changes**). Each manga
|
||||
can belong to one collection. If all manga are grouped, remove a member from another
|
||||
collection first; a collection's final member is retained until you delete the
|
||||
collection. Deletion never removes manga or reading progress.
|
||||
|
||||
Save feedback and errors appear above the editor. Controls are locked during a save;
|
||||
failed saves retain drafts. If another admin has changed the catalog, the stale write
|
||||
is rejected. **Reload catalog** fetches the latest revision, with confirmation before
|
||||
discarding edits; copy any wanted changes first. Unsaved JSON survives collection
|
||||
selection and must use **Validate & save JSON**, not an ordinary collection save.
|
||||
Metadata imports/research are blocked while drafts exist so they cannot silently
|
||||
replace them. Relationship edits save immediately when there is no collection draft.
|
||||
|
||||
Creation works on HTTPS, localhost and ordinary HTTP LAN/Tailscale addresses. The
|
||||
browser uses secure random bytes when its secure-context-only `randomUUID` API is
|
||||
unavailable. This does not change networking or the server's revision/validation rules.
|
||||
|
||||
Tag combinations match a single member, not tags scattered across unrelated members.
|
||||
Reading-time totals only include the current account's visible Reading Story. Collection
|
||||
publication status is ongoing if any member is ongoing, completed if all are completed,
|
||||
otherwise mixed. Collections do not control scraping or merge editions.
|
||||
|
||||
## Three distinct concepts
|
||||
|
||||
| Concept | Meaning | Owner |
|
||||
|---|---|---|
|
||||
| Relationship | Directed fact: target is a prequel/sequel/adaptation/etc. of source | Reader's Catalog Curator |
|
||||
| Collection | Which library manga share a dashboard tile | Admin decision |
|
||||
| Reading order | Recommended order within that collection | Admin decision, independent of chronology |
|
||||
|
||||
One manga belongs to at most one collection, preventing duplicate dashboard tiles and
|
||||
double-counted progress. Relationships can connect any number of works across groups.
|
||||
Only confirmed relationships appear on reader manga pages. Suggestions, rejections
|
||||
and conflicts remain visible to the admin. Removing a collection leaves its manga,
|
||||
reading history and relationships intact.
|
||||
|
||||
## Metadata and agent responsibilities
|
||||
|
||||
The metadata provider, **not the image scraper**, supplies relationships. AniList
|
||||
metadata now retains manga relationship edges in `manga.json` → `x-anilist.relations`.
|
||||
Existing metadata will not magically gain these fields: use **Research relationships**
|
||||
for a manga with a verified AniList ID, or import already-cached suggestions.
|
||||
Research uses one public metadata lookup with a 30-day cache, never image-host traffic.
|
||||
Only relationships whose two exact AniList identities already exist in the library
|
||||
become suggestions. This does not auto-enqueue missing manga or guess title matches.
|
||||
|
||||
An agent is a **catalog researcher/curator assistant**. It can read the admin JSON export,
|
||||
research public evidence, identify conflicting claims and propose relationships or
|
||||
membership edits. It should preserve rejected decisions and report uncertainty.
|
||||
It must not turn a shared-universe link into an automatic collection or an unquestioned
|
||||
reading order. No embedded LLM, API credentials, automatic research loop, or autonomous
|
||||
catalog approval is enabled in this release. That optional capability belongs behind
|
||||
this same evidence/review boundary, not inside adapters or Sentinel.
|
||||
|
||||
## Persistence and API contract
|
||||
|
||||
`data/catalog.json` is reader-owned, version 1, atomically replaced on save. Its
|
||||
revision supports optimistic concurrency; stale writes return 409. Unknown members,
|
||||
duplicate membership, invalid relationships and cover paths are rejected. Existing
|
||||
members absent from a temporary mount are retained rather than erased.
|
||||
Custom covers are content-addressed in `data/collection-covers/`. Include both in backups.
|
||||
|
||||
Authenticated reader APIs:
|
||||
|
||||
- `GET /api/collections`, `GET /api/collection?id=…` — account-specific public views.
|
||||
- `GET /api/admin/catalog` — admin export plus available manga, roles and relation types.
|
||||
- `POST /api/admin/catalog` — validated full replacement, with current `revision`.
|
||||
- `POST /api/admin/catalog/suggest` — import cached edges, with current revision.
|
||||
- `POST /api/admin/catalog/research` — `{m: libraryMangaId, revision}`; verified metadata lookup.
|
||||
- `POST /api/admin/collection-cover` — `{id, revision, base64}`; validated image upload.
|
||||
|
||||
Example proposal (use real IDs and the latest revision from the export):
|
||||
|
||||
```json
|
||||
{
|
||||
"version": 1,
|
||||
"revision": 0,
|
||||
"collections": [{
|
||||
"id": "moon-garden",
|
||||
"title": "Moon Garden Universe",
|
||||
"description": "Main story, then the prequel.",
|
||||
"grouped": true,
|
||||
"coverMode": "mosaic",
|
||||
"coverMangaId": "Moon Garden",
|
||||
"members": [
|
||||
{"mangaId": "Moon Garden", "role": "main", "readingOrder": 1},
|
||||
{"mangaId": "Before the Rain", "role": "prequel", "readingOrder": 2}
|
||||
]
|
||||
}],
|
||||
"relations": [{
|
||||
"from": "Moon Garden", "to": "Before the Rain", "type": "prequel",
|
||||
"state": "suggested", "method": "agent-research", "confidence": 0.8,
|
||||
"explanation": "Verify the publisher's chronology before accepting.",
|
||||
"evidence": [{"provider": "Publisher", "url": "https://example.com/work", "note": "Chronology"}]
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
Provider contract: [AniList relationship connections](https://docs.anilist.co/guide/graphql/connections).
|
||||
|
||||
## Repeatable functional acceptance
|
||||
|
||||
Run `npm.cmd test`, build `docker compose build reader`, then:
|
||||
|
||||
```powershell
|
||||
# Playwright is optional developer tooling; not a deployed application dependency.
|
||||
# Set NODE_PATH to its installed package root if it is not on normal resolution paths.
|
||||
node tools/test-compose.js --collections-ui --settings-ui
|
||||
```
|
||||
|
||||
The runner creates an isolated Compose project on loopback 8180 with three synthetic
|
||||
manga, disposable accounts, paused scraper and disabled discovery. It never mounts
|
||||
the real library, DB or queue. Headless Edge (override `QA_BROWSER_CHANNEL` if needed)
|
||||
proxies a synthetic `http://manga-qa.test:8180` origin to that fixture, reproducing the
|
||||
browser restrictions of a non-localhost HTTP address without using a real tailnet
|
||||
account. Screenshots: `data/qa/collections/desktop-editor.png` and `mobile-editor.png`.
|
||||
|
||||
Coverage: creation, validation, persistence after reload, member add/remove/role/order,
|
||||
selected/mosaic/custom covers and upload limits, JSON round-trip/draft protection,
|
||||
relationship creation/rejection/duplicates/removal, cached import, missing verified
|
||||
metadata identity, desktop/mobile edits, dashboard grouping and collection reading
|
||||
order, delete/cancel, overlapping saves, transient save failure/retry, concurrent-admin
|
||||
revision conflicts, and ordinary-reader read/write denial. No external manga or
|
||||
metadata research is performed; provider lookup success remains a separate test.
|
||||
The earlier settings test checked collections navigation only; this suite exercises
|
||||
the actual collection workflow and the insecure-origin failure it missed.
|
||||
+2
-2
@@ -91,7 +91,7 @@ Each chapter folder is one **version**:
|
||||
| `id` | folder name |
|
||||
| `source`, `sourceUrl` | `chapter.json.source` (`{name,url}` or a string), else the `[…]` part of the folder name |
|
||||
| `added` | `Date.parse(chapter.json.added)`, else the folder mtime |
|
||||
| `direction` | first valid of chapter.json → book.json → manga.json → `rtl`. Aliases: `vertical`, `webtoon` → `ttb` |
|
||||
| `direction` | manual chapter → book → manga; else coherent tall-strip dimensions → `ttb`; else prior generated vertical series direction; else fallback `rtl`. Aliases: `vertical`, `webtoon` → `ttb`. Legacy plain values remain manual |
|
||||
|
||||
6. A version without images is dropped silently.
|
||||
|
||||
@@ -196,7 +196,7 @@ The index is read by the API at request time. Nothing is precomputed per user.
|
||||
|
||||
| Endpoint | Built from |
|
||||
|----------|------------|
|
||||
| `GET /api/library` | manga list + the user's progress, follows and unread notifications |
|
||||
| `GET /api/library` | manga list + searchable metadata (`altTitles`, `authors`, `tags`, `status`, normalized first release), the user's progress, follows and unread notifications |
|
||||
| `GET /api/manga?m=` | manga incl. `meta` (the raw `manga.json`), books, chapters with ranked versions and the user's progress, "Continue" target |
|
||||
| `GET /api/chapter?m=&c=&v=` | pages of the chosen version (`v`, else the best ranked), direction, prev/next chapter (skipping extras if the user wants that), the user's progress |
|
||||
| `GET /lib/<path>` | the image file itself. Logged-in users only, image extensions only, path must stay inside `LIBRARY`. ETag + `Cache-Control: private, max-age=86400` |
|
||||
|
||||
+28
-1
@@ -72,6 +72,16 @@ Ignored, with a warning in the log:
|
||||
|
||||
### The key
|
||||
|
||||
The additive unnumbered MangaDex chapter form is
|
||||
`special-mangadex-<chapter-uuid> Pilot [MangaDex · Group]`. It is accepted only for
|
||||
chapters, never books. Its logical key is the complete lower-case prefix/UUID;
|
||||
`chapter.json` supplies `special:true`, `label` (Pilot/One-shot/Special), `sourceId`
|
||||
and nullable `sourceNumber`. The reader shows that label with no fabricated number.
|
||||
Such chapters sort after numbered entries and are not inherently skippable extras.
|
||||
Different source UUIDs remain distinct releases; no content equivalence is guessed.
|
||||
Numeric IDs and source-version folders are unchanged. Older reader releases ignore
|
||||
these new folders but retain them on disk; upgrade again to restore visibility.
|
||||
|
||||
The number is normalised: leading zeros and trailing decimal zeros are dropped (`0010.50` → `10.5`), and an `x` prefix is kept. The result is the **key**:
|
||||
|
||||
| Folder | Key |
|
||||
@@ -165,10 +175,17 @@ UTF-8 JSON (a BOM is tolerated). All fields are optional, and unknown fields are
|
||||
| `tags` | string[] | Genres / tags |
|
||||
| `description` | string | Synopsis (line breaks are kept) |
|
||||
| `direction` | `"rtl"` \| `"ltr"` \| `"ttb"` | How the pages are read. `rtl` (the default): manga, right to left. `ltr`: comics, printed manhwa. `ttb`: webtoon, one vertical strip cut into slices (aliases `vertical`, `webtoon`) |
|
||||
| `directionOverride` | same values | Explicit manual choice. Use this even when choosing `rtl`, which is also the fallback. May be set at manga, book, or chapter level |
|
||||
| `x-reading-direction` | object | Generated provenance: `{ "version": 1, "value": "ttb", "source": "detected" }` or `source: "fallback"`, with optional numeric evidence. The scraper owns this field; it is not a confidence score |
|
||||
| `sources` | `{name, url}[]` | Upstream sources the manga was scraped from. Several are allowed |
|
||||
| `x-anilist` | object | AniList work ID and linked metadata. The ID identifies a work, not necessarily a particular pilot or edition |
|
||||
| `x-metadata.imported` | object | Last imported values for title, alternative titles, authors, status, year, tags and description. The scraper updates a displayed field only when it is empty or still equals its previous imported value; edits in legacy files without this snapshot are preserved |
|
||||
| `x-scraper.autoUpdate` | boolean | Optional explicit admin choice. When absent, `status: "ongoing"` enables nightly checks and other statuses disable them |
|
||||
|
||||
The whole file is sent to logged-in users as part of the manga page. **Don't put secrets in it** (scraper tokens, cookies).
|
||||
|
||||
An AniList refresh with a different ID is stopped for review before chapter publication. Existing files that already contain `x-scraper.autoUpdate` keep that choice; the scraper does not guess whether an older value was automatic or manual. New files leave the choice absent so the default can follow status changes.
|
||||
|
||||
### `book.json`
|
||||
|
||||
| Field | Type | Meaning |
|
||||
@@ -188,7 +205,17 @@ The whole file is sent to logged-in users as part of the manga page. **Don't put
|
||||
| `priority` | number | Tie-breaker between versions with equal votes. Higher wins (default `0`) |
|
||||
| `direction` | as above | Overrides book/manga direction, e.g. a colour webtoon special inside a printed series |
|
||||
|
||||
`direction` is inherited **chapter → book → manga → `rtl`**, and each version can carry its own.
|
||||
Manual directions have priority at **chapter → book → manga**. A `directionOverride`
|
||||
or a plain `direction` without matching generated provenance is manual. Otherwise,
|
||||
the scanner checks the version's existing image dimensions for coherent tall strips
|
||||
and uses `ttb` when they are present. A prior generated vertical series direction
|
||||
applies to chapters with insufficient shape evidence. The final fallback is `rtl`;
|
||||
this does **not** classify a work as Japanese or prove its reading order. A direct
|
||||
edit to a generated `direction` becomes manual when it differs from its provenance.
|
||||
Existing plain directions are treated as manual because old scraper defaults and
|
||||
human edits cannot be distinguished safely. Set `directionOverride` to make a
|
||||
lasting choice, or explicitly review and migrate a legacy entry. Image files and
|
||||
chapter IDs do not change when direction metadata changes.
|
||||
|
||||
## Picture-viewer compatibility
|
||||
|
||||
|
||||
@@ -0,0 +1,244 @@
|
||||
# Reading Story concept
|
||||
|
||||
Status: first release implemented on 2026-10-04. The Active Reading Clock records the
|
||||
underlying per-user, per-chapter measurements and MangaCircus now returns the compact
|
||||
library, manga and chapter summaries described below. The bounded daily Reading Journal
|
||||
remains a later phase.
|
||||
|
||||
Policy update 2026-10-06: a timed read now means a visit with at least 60% viewed
|
||||
page coverage, not exhaustive 100% verification. No speed ceiling applies. Below-
|
||||
threshold lookup visits do not enter these summaries; existing history is retained.
|
||||
First/last durations continue growing until departure. See `reading-time-concept.md`.
|
||||
|
||||
## Product idea
|
||||
|
||||
Call the retrospective feature **Your Reading Story**. The Active Reading Clock is the
|
||||
instrument visible while reading; Reading Story is the quiet, personal record it builds.
|
||||
|
||||
The feature should answer four pleasant questions:
|
||||
|
||||
1. How much time have I spent with this manga?
|
||||
2. What did this chapter take me to read?
|
||||
3. Where am I in the story, including partially timed chapters?
|
||||
4. What does my reading habit look like over time?
|
||||
|
||||
It is not a productivity score. Do not rank users, award points for longer sessions, or
|
||||
describe a reader as fast or slow. Those patterns would reward timer inflation and turn a
|
||||
private leisure activity into a competition.
|
||||
|
||||
## Display language
|
||||
|
||||
Use **active time**, **reading time**, and **timed**. Never call it screen time or claim
|
||||
that MangaCircus proved comprehension.
|
||||
|
||||
Durations are friendly rather than stopwatch-precise outside the live reader:
|
||||
|
||||
| Stored measurement | Display |
|
||||
|---|---|
|
||||
| no reading record | `Not timed` |
|
||||
| 1–59 seconds | `Under a minute` |
|
||||
| 60 seconds–59 minutes | rounded to the nearest minute, for example `7 min` |
|
||||
| 1 hour or more | hours and minutes, for example `3 h 42 min` |
|
||||
| incomplete current pass | `4 min active · in progress` |
|
||||
| one completed pass | `6 min first read` |
|
||||
| completed rereads | `5 min last read · 18 min total · 3 passes` |
|
||||
|
||||
Zero must not mean “read instantly.” A chapter with no record is **Not timed**, because
|
||||
the setting may have been disabled or the chapter may have been marked read manually.
|
||||
Loading wait and unqualified visit drafts remain diagnostic details; they are not mixed
|
||||
into the headline reading duration.
|
||||
|
||||
## 1. Reader: the live clock
|
||||
|
||||
Keep the implemented bottom-toolbar clock as the immediate view:
|
||||
|
||||
```text
|
||||
[ ⏱ 06:14 · Reading ]
|
||||
```
|
||||
|
||||
Tapping it opens the detail panel. At 60% coverage show a gentle confirmation:
|
||||
**Reading time is now counted · 60% of pages viewed**. Time continues accumulating;
|
||||
this does not claim the entire chapter has been read or alter ordinary progress.
|
||||
|
||||
For subsequent qualifying visits say **Reread time is now counted**.
|
||||
“Together” makes the lifetime number feel connected to the story rather than to work.
|
||||
|
||||
## 2. Manga page: Your Reading Story card
|
||||
|
||||
Place one card after the hero and before metadata. Hide it entirely when the user has no
|
||||
timing records for this manga; do not show a large empty dashboard.
|
||||
|
||||
```text
|
||||
┌ Your Reading Story ─────────────────────────────────────────┐
|
||||
│ 3 h 42 min together 31 chapters timed 4 rereads │
|
||||
│ Typical chapter 7 min Last read Ch. 45 · 18 min ago │
|
||||
│ │
|
||||
│ Reading rhythm │
|
||||
│ 01 ▂▂▃▂▅▃▂ 02 ▃▄▂▂▃ 03 ▂▂▆▃▂ │
|
||||
│ Each bar is one completed chapter; height is active time. │
|
||||
└────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The headline values are:
|
||||
|
||||
- **time together:** sum of `totalActiveMs` for every chapter in this manga;
|
||||
- **chapters timed:** chapters with any credible active time;
|
||||
- **completed passes:** sum of completed passes, phrased as rereads only after subtracting
|
||||
the first completed pass per chapter;
|
||||
- **typical chapter:** median `lastCompleteMs` across completed chapters, never the mean;
|
||||
- **last read:** chapter with the newest `updatedAt`.
|
||||
|
||||
The **Reading rhythm** is a compact chapter sequence, grouped by the existing volume
|
||||
boundaries. It visualises only completed-pass duration. A gap means “not timed,” not zero.
|
||||
Bars use a log-like capped scale so one very long chapter does not flatten the rest.
|
||||
Hovering or tapping a bar reveals chapter number, title, last completed duration, lifetime
|
||||
time and pass count. Selecting a volume with the existing filter also filters this rhythm.
|
||||
|
||||
On narrow screens, show the first four headline values in a two-column grid and make the
|
||||
rhythm horizontally scrollable. The card is collapsible and remembers its state per user.
|
||||
|
||||
## 3. Chapter list: useful, quiet timing
|
||||
|
||||
Add the duration between the page count and chapter title, using a fixed-width column so
|
||||
the current chapter-row alignment stays clean:
|
||||
|
||||
```text
|
||||
Ch. 45 35 p. 7 min The Twin Swords ✓ read
|
||||
Ch. 44 31 p. 4 min The Visitor ✓ read
|
||||
Ch. 43 29 p. Not timed A Quiet Morning ✓ read
|
||||
```
|
||||
|
||||
The compact column shows the last completed pass. An incomplete chapter shows its current
|
||||
active time with a small hollow clock, such as `◷ 3 min`. A chapter with multiple passes
|
||||
gets a subtle repeat marker: `7 min ↻2`. Its accessible label and tooltip expand this to
|
||||
`Last read 7 minutes; 2 completed passes; 19 minutes total`.
|
||||
|
||||
Do not put lifetime time directly into every row; that makes reread chapters visually
|
||||
noisy. Lifetime detail belongs in the tooltip/tap sheet and Reading Story card.
|
||||
|
||||
The chapter timing column is hidden when the clock setting is disabled. Existing history
|
||||
is preserved and becomes visible again if the user re-enables the setting.
|
||||
|
||||
## 4. Library dashboard: one personal summary strip
|
||||
|
||||
Above the current filters and cover grid, show a compact strip only when at least one
|
||||
chapter has been timed:
|
||||
|
||||
```text
|
||||
Your Reading Story 18 h 26 min together · 146 chapters timed
|
||||
Most time with Berserk · Continue your last chapter →
|
||||
```
|
||||
|
||||
Add two optional library sorts:
|
||||
|
||||
- **Recently timed** — newest reading `updatedAt` first;
|
||||
- **Time together** — manga with the largest personal active-time total first.
|
||||
|
||||
Do not put duration badges on every cover by default. They compete with new-chapter badges
|
||||
and reading progress. A future `Show reading time on covers` preference may opt into a
|
||||
small `3 h 42 min` line below the existing read count.
|
||||
|
||||
The first release should not claim “today,” “this week,” streaks, or a calendar. The
|
||||
current compact database has totals and latest timestamps, but no honest time-series
|
||||
history from which those values can be reconstructed.
|
||||
|
||||
## 5. Later: Reading Journal
|
||||
|
||||
If readers enjoy Reading Story, add bounded daily aggregates rather than raw activity
|
||||
events:
|
||||
|
||||
```jsonc
|
||||
"daily": {
|
||||
"2026-10-04": { "activeMs": 2840000, "chapters": 6, "completedPasses": 5 }
|
||||
}
|
||||
```
|
||||
|
||||
Keep at most 400 daily buckets per user, folding older values into yearly totals. This
|
||||
enables a private activity calendar, `this week`, `this year`, and “you returned to this
|
||||
story after 3 months” without storing pointer movements, individual page timestamps or a
|
||||
surveillance-style event log.
|
||||
|
||||
Recommended journal messages are descriptive, not judgemental:
|
||||
|
||||
- **A long evening with Berserk · 1 h 18 min**
|
||||
- **You returned to Gantz after 42 days**
|
||||
- **Volume 4 was your longest volume · 2 h 06 min**
|
||||
- **12 chapters reread this year**
|
||||
|
||||
Avoid streak pressure, “goal missed,” speed comparisons, and global leaderboards.
|
||||
|
||||
## API and derived-data design
|
||||
|
||||
Keep the raw `reading` records private to their owner. Add derived summaries at the
|
||||
existing authenticated boundaries:
|
||||
|
||||
### `GET /api/library`
|
||||
|
||||
Each manga may include:
|
||||
|
||||
```jsonc
|
||||
"reading": {
|
||||
"activeMs": 13320000,
|
||||
"timedChapters": 31,
|
||||
"completedChapters": 29,
|
||||
"completedPasses": 35,
|
||||
"rereads": 6,
|
||||
"typicalChapterMs": 421000,
|
||||
"lastChapterId": "3/45",
|
||||
"updatedAt": 1791130200000
|
||||
}
|
||||
```
|
||||
|
||||
Also return a top-level personal total derived in the same pass. This avoids an extra
|
||||
request and lets library sorting work locally like the existing filters.
|
||||
|
||||
### `GET /api/manga`
|
||||
|
||||
Return the manga aggregate plus a small public summary on each chapter:
|
||||
|
||||
```jsonc
|
||||
"reading": {
|
||||
"totalActiveMs": 1140000,
|
||||
"lastCompleteMs": 421000,
|
||||
"completedPasses": 2,
|
||||
"currentPassMs": 0,
|
||||
"currentPassComplete": true,
|
||||
"updatedAt": 1791130200000
|
||||
}
|
||||
```
|
||||
|
||||
Do not send input-mode totals, rejected views or page ranges to the manga page. They are
|
||||
reader diagnostics and would make the overview payload and interface unnecessarily noisy.
|
||||
|
||||
Derive all aggregates on the server from the compact records during these requests. The
|
||||
library is small and JSON-backed; a second denormalised cache would create invalidation
|
||||
work without a current need. If measurement proves this slow, cache by a monotonic
|
||||
reading-data revision rather than writing duplicate totals into `db.json`.
|
||||
|
||||
## Controls, privacy and corrections
|
||||
|
||||
Under **Settings → Reading**, explain: “Reading Story is private to your account.” Add:
|
||||
|
||||
- **Show Reading Story** — controls retrospective UI separately from collecting the clock;
|
||||
- **Forget one manga’s reading time** — available on that manga's card menu;
|
||||
- **Delete all reading-time history** — destructive, confirmed with the account password;
|
||||
- **Export my reading-time summary** — compact JSON, with no raw input events.
|
||||
|
||||
Turning off the Active Reading Clock stops collection but does not silently erase history.
|
||||
The retrospective display setting may hide existing history without changing it. An admin
|
||||
must not see another user’s Reading Story through the normal admin interface.
|
||||
|
||||
## Delivery status
|
||||
|
||||
Implemented:
|
||||
|
||||
1. Server-side summary helpers and tests for missing, partial, complete and reread records.
|
||||
2. Manga-page card, volume-aware rhythm and aligned chapter-duration column.
|
||||
3. Library summary strip and the two personal-time sorts.
|
||||
4. User controls for hide, per-manga deletion, password-confirmed complete deletion and export.
|
||||
5. A gentle reader completion message for first reads and rereads.
|
||||
|
||||
Deferred: observe real use before adding bounded daily aggregates and the Reading Journal.
|
||||
|
||||
This order immediately makes the existing measurements useful without changing the clock
|
||||
protocol or committing MangaCircus to a larger analytics history.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Active Reading Clock
|
||||
|
||||
Current policy: relaxed personal estimates, implemented 2026-10-06. This is not an
|
||||
anti-cheat system, proof of comprehension, or a basis for competitive rewards.
|
||||
Enable it in User → Reading time. Disabled means no clock requests.
|
||||
|
||||
## What counts
|
||||
|
||||
Each chapter visit measures foreground, loaded-page viewing time. After viewing
|
||||
**at least 60% of eligible pages**, rounded up, the whole visit's active time counts,
|
||||
including time before reaching that threshold. It continues counting through the
|
||||
rest of the visit and is saved periodically and when leaving the reader.
|
||||
|
||||
- There is **no pages-per-second ceiling** and no skimming penalty.
|
||||
- A loaded image needs **100 ms cumulative visible exposure**. Flow exposure uses
|
||||
10% of the smaller of page/viewport area and samples scroll positions plus a
|
||||
50 ms timer. This detects brief views; it does not assess reading speed.
|
||||
- Only actually observed pages count. A scrollbar/slider jump does not credit
|
||||
all pages between the starting position and destination.
|
||||
- Hidden covers/extras are excluded from the denominator.
|
||||
- Source versions have separate visit coverage; qualifying time combines into
|
||||
existing chapter totals without changing reading identities.
|
||||
- Below-60% visits remain bounded diagnostic drafts, excluded from Reading Story,
|
||||
first/last-read duration, reread counts and chapter headline totals.
|
||||
- New navigation starts a fresh visit. Old coverage does not qualify later picture
|
||||
lookups. Separate sub-60% visits are not combined. Back/forward-cache restoration
|
||||
continues the same visit.
|
||||
|
||||
Example: view 12 of 20 pages, then leave → all active time from that visit counts.
|
||||
Jump to page 19, dwell, leave → no new reading total. Historical times are retained;
|
||||
this update neither deletes old measurements nor invents times for missed reads.
|
||||
|
||||
Intent cannot be inferred perfectly: slowly searching most of a chapter can look
|
||||
like reading. This rule filters quick-search-and-one-picture visits without
|
||||
penalizing honest fast readers. Ordinary read/unread progress is independent.
|
||||
|
||||
## Pauses and display
|
||||
|
||||
Pause when hidden/unfocused; resume on reader input after returning. Pause after
|
||||
**30 seconds** without activity (updated at the user's request on 2026-10-06). The first loaded page can start
|
||||
without an extra click. Mouse/pen motion, wheel, touch movement, momentum scrolling
|
||||
and keyboard controls—including held Up/Down arrows—refresh reader activity.
|
||||
|
||||
Loading wait stays separate. An unloaded neighbour no longer pauses the clock if
|
||||
another visible image is readable. Failed/unloaded images do not earn coverage.
|
||||
An end card pauses time. Other reader tabs/devices coordinate to avoid accidental
|
||||
duplicate clocks; this is not an anti-cheat score.
|
||||
|
||||
The toolbar shows **this visit's** time. Its panel shows pages viewed, the 60%
|
||||
requirement, whether the visit counts, saved chapter total, loading wait and
|
||||
pause/save-pending state. “Verified”, “rejects” and “skimming” labels are removed.
|
||||
|
||||
## Reliable saves and compatibility
|
||||
|
||||
New clients POST cumulative snapshots to authenticated `/api/reading/visit` every
|
||||
10 seconds, on attention changes, and using `sendBeacon` (keepalive fallback) on
|
||||
`pagehide`. Final snapshots dispatch even with an ordinary request in flight.
|
||||
Visit ID plus increasing sequence makes retries/out-of-order packets idempotent.
|
||||
Delayed saves avoid the legacy 12-second packet truncation. Final saves don't
|
||||
reclaim the next chapter's lease, nor do stale pulses reclaim an exited lease.
|
||||
|
||||
First/last duration keeps growing after qualification until departure. Late finals
|
||||
update their own receipt, not a newer visit. The latest eight visit receipts are
|
||||
retained per chapter; extremely old delayed requests after eviction are outside
|
||||
this bounded retry window. There is no durable offline queue: closing an offline
|
||||
tab or a browser/OS crash can still lose undelivered time. “Save pending” is explicit.
|
||||
|
||||
Legacy `/api/reading/pulse` remains for already-open old readers. Existing records
|
||||
and totals are retained. Reload open readers for this policy. On code rollback the
|
||||
separate new endpoint returns 404 instead of interpreting cumulative totals as
|
||||
legacy deltas. Refresh open readers after rollback as well.
|
||||
|
||||
## Privacy and verification
|
||||
|
||||
Only chapter/version IDs, visit/sequence IDs, cumulative active/loading/input-type
|
||||
times, viewed page indexes and exit flags are sent. No coordinates, key contents,
|
||||
recordings, screenshots or raw event stream are saved. Exposure geometry stays
|
||||
local. History is private to the user and existing export/reset controls remain.
|
||||
|
||||
`test/reading-visits.test.js` covers thresholds, historical preservation, fast
|
||||
slices, lookup exclusion, attention/loading, late/duplicate snapshots and delayed
|
||||
delivery. `node tools/test-compose.js --reading-ui` adds actual headless Edge
|
||||
mobile-layout QA with disposable users/generated images/isolated storage. It does
|
||||
not observe production readers or query image hosters.
|
||||
@@ -0,0 +1,113 @@
|
||||
# Runtime resilience and agent operations
|
||||
|
||||
## Ownership and failure boundaries
|
||||
|
||||
```text
|
||||
Browser → Reader ──private API──→ Scraper → library (atomic chapter publication)
|
||||
│ ↑
|
||||
└──health──→ Sentinel ───┘ bounded checkpoint recovery only
|
||||
|
||||
Supervisor(reader) Supervisor(scraper) Supervisor(sentinel)
|
||||
↑
|
||||
Coding agent: inspect incidents → reproduce → patch → test → release
|
||||
```
|
||||
|
||||
The reader is the only writer of users/progress and the catalog. The scraper is the
|
||||
only queue owner and content publisher. Sentinel owns only observations and incidents.
|
||||
Each service's supervisor owns only that child process. The coding agent is an
|
||||
**escalation/maintenance operator**, not a necessary part of normal downloading.
|
||||
The recurring Codex check is an additional backstop, not the runtime watchdog.
|
||||
|
||||
## Automatic recovery implemented
|
||||
|
||||
| Failure | Action | Automatic limit |
|
||||
|---|---|---|
|
||||
| Process exit or repeated failed liveness checks | Supervisor restarts only that service, with backoff | Three failures per rolling hour; then parked for operator action |
|
||||
| Network/HTTP 5xx | Preserve checkpoint; open source circuit, wait five minutes, retry job | One job-level retry; request-level retries remain bounded |
|
||||
| Title unavailable/no English chapters | Record unavailable, continue unrelated work | No blind retry |
|
||||
| Parser/local adapter failure | Quarantine job, keep partial data, continue other titles | Two different affected jobs in an hour pause source as systemic failure |
|
||||
| Storage/permission failure | Pause all downloads and report | No automatic retry |
|
||||
| HTTP 401/403/429 or access challenge | Pause affected source lane | No automatic retry/bypass |
|
||||
| Active job with no progress for 15 minutes | Sentinel requests cooperative checkpoint-safe retry | One stall recovery; next stall quarantined |
|
||||
|
||||
The daemon remains authoritative: a recovery request includes the observed progress
|
||||
fingerprint and is rejected if the job has moved, paused, or is no longer stale.
|
||||
Sentinel also observes completed/page counts; a deliberately paused queue is not a
|
||||
stall. Timeouts bound source fetches to 60 seconds and metadata calls to 30 seconds.
|
||||
Publication remains one complete chapter at a time, on the same filesystem.
|
||||
No recovery action raises pacing, creates another worker for the same source,
|
||||
changes identities or deletes partial images.
|
||||
|
||||
The publisher retains validated pages in `.staging-mcscraper-resume-<sha256>` folders
|
||||
on pause/failure. It checks the full chapter/source/page-manifest fingerprint before
|
||||
reuse and revalidates each image; completed chapters still publish with one rename.
|
||||
A changed manifest creates separate staging rather than mixing releases. Old random
|
||||
stages remain untouched for manual review: they cannot be safely associated from
|
||||
page filenames alone. Hidden pending image files are never treated as completed pages.
|
||||
|
||||
`retry_wait`, `unavailable` and `quarantined` are explicit queue states in schema 4.
|
||||
`ready` is a healthy inspected job awaiting operator download confirmation, not a
|
||||
stall. Inspection shares the source lane; it never downloads images or bypasses a
|
||||
source circuit. Source coverage and job completion are separate observations.
|
||||
Legacy unavailable-title errors are migrated; only pauses solely attributable to
|
||||
these unavailable titles are released. Access and unknown failures remain paused.
|
||||
Nightly discovery avoids duplicating outstanding, unavailable or quarantined jobs;
|
||||
use an explicit operator retry after fixing their cause.
|
||||
|
||||
## Observability
|
||||
|
||||
**Settings → Service health** shows Sentinel status, active checkpoints, failed jobs
|
||||
and incident history. **Settings → Scraper** retains queue, pacing and discovery controls.
|
||||
Service errors are classified rather than buried in a single historical `lastError`.
|
||||
`npm run check` performs read-only checks. Sentinel's private, token-protected `/metrics`
|
||||
exports reader-up, scraper-up and open-incident gauges for a future Prometheus collector.
|
||||
It is not a public endpoint.
|
||||
|
||||
Scraper status is independent of expensive library scans. Concurrent admin/Sentinel
|
||||
probes share one background inventory refresh and receive the previous summary
|
||||
immediately. The dashboard summary refreshes at most once per minute unless a
|
||||
publication or metadata change invalidates it; duplicate-sensitive decisions still
|
||||
request a fresh inventory. `libraryStatus` reports ready/refreshing/stale/updated/error, and failed
|
||||
background refreshes do not spawn a new scan on every UI poll. Authoritative
|
||||
discovery waits for the current inventory; publication/metadata invalidation during
|
||||
a scan forces another scan before it can be used for those decisions. Cold start
|
||||
may temporarily show no library targets, but queue and rate controls remain usable.
|
||||
|
||||
Storage:
|
||||
|
||||
- `data/catalog.json`, `data/collection-covers/`: reader catalog and covers.
|
||||
- `data/scraper/queue.json`, `events.log`: queue, circuits, failures and incidents.
|
||||
- `data/sentinel/state.json`: Sentinel incidents and last observation.
|
||||
- `data/runtime-reader.json`, `data/scraper/runtime.json`, `data/sentinel/runtime.json`:
|
||||
supervisor status and persisted crash budget.
|
||||
- `data/scraper/supervisor.lock`, `daemon.lock`: native process ownership; transient.
|
||||
|
||||
Scraper/Sentinel/supervisor emit structured logs. Queue and Sentinel histories retain
|
||||
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows
|
||||
supervisor logs remain append-only: archive them during stopped maintenance if needed.
|
||||
Never export user databases, tokens, cookies, signed query strings or unredacted environments.
|
||||
|
||||
## What self-repair intentionally does not mean
|
||||
|
||||
Sentinel cannot execute a shell, patch adapters, query image sources, access a Docker
|
||||
socket, or issue unrestricted restart commands. It cannot repair broken markup or
|
||||
solve source challenges. The agent handles those escalations in the repository:
|
||||
|
||||
1. Inspect one bounded incident and relevant checkpoint; preserve the active lane.
|
||||
2. Add a fixture reproducing the failure, patch the smallest adapter/logic surface.
|
||||
3. Run `npm test`; test UI/API in isolated data if affected.
|
||||
4. Build a traceable image, deploy checkpoint-safely, verify resumed progress.
|
||||
5. Record release, backup and rollback compatibility. Stop after repeated identical failures.
|
||||
|
||||
A VM-local monitor cannot alert on total VM/network failure. Before VM cutover choose
|
||||
an external health check and notification destination (e.g. an existing monitoring
|
||||
service and private ntfy/Gotify). No notification credentials or destination are invented.
|
||||
|
||||
## Crash-budget operation
|
||||
|
||||
A parked supervisor stays alive and reports `repair-required`; Docker's restart policy
|
||||
does not create an unlimited process restart storm. Its health check fails while the
|
||||
child is absent. Investigate first. Once repaired, restart the wrapper after its
|
||||
rolling one-hour budget has expired. For an intentional earlier reset, first stop the
|
||||
service, back up its `runtime*.json`, and clear only that service's `failures` array;
|
||||
never clear the queue or daemon checkpoint. Explicit deployments do not erase budgets.
|
||||
+37
-2
@@ -83,9 +83,13 @@
|
||||
```
|
||||
c<NNNN>[.<d>][ <title>][ [<Source>]] regular chapter
|
||||
x<NN>[ <title>][ [<Source>]] extra chapter outside the numbering
|
||||
special-mangadex-<chapter-uuid>[ <title>][ [<Source>]] unnumbered source release
|
||||
```
|
||||
- A **space** MUST follow the number when anything follows. `c0012_Title` sorts wrongly in file browsers; `c0012a` and `c0012Title` are not recognised at all.
|
||||
- Recommended check: `^(c\d{4}(\.\d*[1-9])?|x\d{2})( [^\[\]]+?)?( \[[^\[\]]+\])?$`
|
||||
- For the additive `special-mangadex-` form, validate the complete UUID using
|
||||
`chapter-identity.js`. Do not apply the numeric-only check above. The key is the
|
||||
lower-case source UUID prefix, not an invented numeric chapter. Books remain numeric.
|
||||
- **R-C2** Pad the chapter number's integer part to 4 digits. Never change the padding of an existing manga: chapter folder names are version ids.
|
||||
- **R-C3** Map upstream numbering like this:
|
||||
|
||||
@@ -102,6 +106,14 @@
|
||||
| Volume extras (afterword, sketches) | `x<NN>` in that volume's book | |
|
||||
| One-shot | its own manga (`v01/c0001`), or `x<NN>` inside an `x01 Oneshots` book of the related series | |
|
||||
| Negative or non-numeric only | `x<NN>` | Never invent a number that collides with a real one |
|
||||
| Unnumbered MangaDex release | `special-mangadex-<chapter-uuid>` | Stable upstream release identity; label Pilot, One-shot or Special; do not assign chapter 0/1 |
|
||||
|
||||
Unnumbered source-ID releases sort after numeric chapters. Their `chapter.json`
|
||||
stores `special:true`, `label`, `sourceId` and the original `sourceNumber` (usually
|
||||
null). They are not automatically skippable extras: an entire pilot may be story
|
||||
content. Keep different release UUIDs separate rather than guessing that they are
|
||||
the same logical chapter. Reimporting the same key is duplicate-safe. Existing
|
||||
numeric and `x` folder conventions and progress IDs MUST NOT be migrated or renamed.
|
||||
|
||||
- **R-C4** The title part of the folder name is for humans only.
|
||||
- The authoritative title is `chapter.json` → `title`, without the "Chapter 12:" prefix.
|
||||
@@ -174,7 +186,7 @@ All are UTF-8 JSON **without** BOM, pretty-printing optional.
|
||||
}
|
||||
```
|
||||
|
||||
- `direction`:
|
||||
- `direction` when an operator or reliable source explicitly provides the reading order:
|
||||
|
||||
| Upstream type | Value |
|
||||
|---------------|-------|
|
||||
@@ -186,6 +198,24 @@ All are UTF-8 JSON **without** BOM, pretty-printing optional.
|
||||
- Unknown fields are kept and ignored by the app. Keep your own state in one namespaced field such as `"x-scraper"`.
|
||||
- The whole file is shown to logged-in users. It MUST NOT contain secrets (tokens, cookies, credentials).
|
||||
- **Read-modify-write:** keep fields you don't own, because people may have edited them. Write atomically (R-W5).
|
||||
- **R-J4** The current adapters do not infer printed reading order from host,
|
||||
language, country, or catalog labels. They emit `rtl` with
|
||||
`"x-reading-direction":{"version":1,"value":"rtl","source":"fallback"}`.
|
||||
The publisher checks already downloaded image dimensions: at least three pages
|
||||
with height/width ≥ 4, at least 75% tall among known pages, consistent strip
|
||||
widths within 25%, and dimensions known for at least 75% of pages yield
|
||||
`direction: "ttb"` with `source: "detected"` and numeric evidence. A credits
|
||||
card can coexist with the strips. This detects layout, not content or language.
|
||||
Plain legacy directions and explicit `directionOverride` at any level are
|
||||
preserved across later imports. Do not bulk-convert old `rtl` values: a past
|
||||
scraper default and a human choice are indistinguishable. Use the reviewed
|
||||
migration tool on a named series after checking representative chapters.
|
||||
`node tools/review-reading-direction.js <absolute-library-manga-folder>
|
||||
<weebcentral-series-id> --chapter=c0056` is a read-only review. To apply a
|
||||
supported single-series correction, first pause and stop every writer and
|
||||
create a stopped-writer snapshot, then add `--apply --writers-stopped`.
|
||||
The tool verifies source identity, unchanged legacy RTL metadata, several
|
||||
chapter samples, and the requested chapter before an atomic metadata write.
|
||||
|
||||
- **R-J2** `chapter.json`, one per version folder:
|
||||
|
||||
@@ -240,7 +270,12 @@ MangaCircus can scan at any moment. Everything it can see must always be complet
|
||||
4. delete `.old-<random>`.
|
||||
The chapter is briefly absent, which is harmless: it won't be re-announced, and progress is kept.
|
||||
- **R-W5** Updating a JSON file: write `.<name>.tmp` in the same folder, then `rename` it over the target.
|
||||
- **R-W6** At start-up, delete your own leftover `.staging-*` / `.old-*` folders older than a few hours (from crashed runs).
|
||||
- **R-W6** Preserve incomplete staging folders: validated images are recovery assets,
|
||||
not disposable garbage. The managed publisher resumes a hidden folder only when
|
||||
its complete chapter/source/page-manifest fingerprint matches, and revalidates
|
||||
stored images. Legacy random stages lack this identity and require manual review;
|
||||
do not infer ownership or delete them merely because they are old. Completed
|
||||
replacement `.old-*` backups may be cleaned after successful publication.
|
||||
- **R-W7** On Windows, `rename` can fail with `EPERM`/`EBUSY` while another process reads the folder. Retry a few times with a short delay.
|
||||
- **R-W8** Files MUST be readable by the MangaCircus process. In Docker that is uid 1000, so use mode `0644` for files and `0755` for folders.
|
||||
- **R-W9** You MUST NOT write anything into MangaCircus's `DATA` folder (`db.json`, `scan-cache.json`).
|
||||
|
||||
@@ -0,0 +1,228 @@
|
||||
# Managed scraper service
|
||||
|
||||
The scraper is an optional Node.js daemon that runs beside the MangaCircus reader.
|
||||
It owns the durable queue and is the only process allowed to run a queued archive.
|
||||
The browser never contacts it directly: authenticated admin routes in `server.js`
|
||||
proxy commands to its private HTTP API.
|
||||
|
||||
## Start and storage
|
||||
|
||||
```bash
|
||||
npm run start:all # reader + daemon under one local supervisor
|
||||
npm start # reader only
|
||||
npm run start:scraper # daemon only
|
||||
```
|
||||
|
||||
These commands are for native development only. The current Windows installation
|
||||
runs the three-service Docker Compose stack; the old Scheduled Task is disabled.
|
||||
Never start a native supervisor or daemon against its mounted state. Its observed
|
||||
ports, logs, persisted settings, restart procedure, and recovery snapshot are recorded in
|
||||
[`local-deployment.md`](local-deployment.md).
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `SCRAPER_HOST` | `127.0.0.1` | Daemon bind address |
|
||||
| `SCRAPER_PORT` | `8091` | Daemon port |
|
||||
| `SCRAPER_TOKEN` | empty | Bearer token; mandatory when the bind address is not loopback |
|
||||
| `SCRAPER_DATA` | `<DATA>/scraper` | Durable queue, lock, and event log |
|
||||
| `LIBRARY` | `./library` | Shared writable library for the daemon |
|
||||
| `SCRAPER_URL` | `http://127.0.0.1:8091` | Daemon URL used by the reader |
|
||||
|
||||
The authoritative queue is `data/scraper/queue.json` (currently schema version 4),
|
||||
written atomically. Individual
|
||||
archives retain their existing resumable checkpoints under
|
||||
`library/.mcscraper-work-<source-id>/job.json`. `events.log` is append-only.
|
||||
|
||||
On its first start, the daemon imports the old `.mcscraper-queue/queue.json` wishlist
|
||||
when present. Completed chapter ids in each archive checkpoint remain authoritative,
|
||||
so migration does not redownload published chapters.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
The daemon owns one durable queue and runs at most one sequential worker per source. Different sources may run side by side, but two jobs from the same source never overlap:
|
||||
|
||||
1. Select the first `queued` item for every idle, unpaused source.
|
||||
2. Resolve its URL through the adapter registry in `scraper/adapters/index.js`.
|
||||
3. Run one adapter job per source with that source's request and chapter delays.
|
||||
4. For legacy in-progress checkpoints, continue publishing into their private work directory and expose the complete manga at the end.
|
||||
5. For every new job, stage one chapter beside its final book and expose that complete chapter immediately with one rename.
|
||||
6. Increment `publishedRevision` after every newly visible chapter and start the next item for that source when the selected scope is complete.
|
||||
|
||||
Legacy work can be converted without redownloading. While the queue is paused and no
|
||||
worker is active, `publishLegacyReady` validates every completed chapter, quarantines
|
||||
abandoned hidden partial stages, atomically exposes the accumulated manga folder, and
|
||||
changes its checkpoint to incremental mode. The resumed job then derives its remaining
|
||||
work from the visible library and publishes every later chapter immediately.
|
||||
|
||||
A source failure pauses only that source, allowing unrelated source workers to continue. Manual pause remains global. Request and chapter delays are configured independently for every installed host in **Settings → Scraper**. Both current adapters default to the fast profile: 1–4 seconds between sequential requests and 5–15 seconds between chapters.
|
||||
|
||||
The reader polls `publishedRevision` every 15 seconds and rescans after it increases.
|
||||
A source error pauses only that source lane for human review. Pause and shutdown requests
|
||||
are checked between requests; validated partial pages are retained in hidden,
|
||||
fingerprinted staging and reused only when the complete manifest still matches.
|
||||
|
||||
## Inspect, confirm, and interpret completion
|
||||
|
||||
The admin form uses **Inspect & queue** (`inspect:true`). This queues a catalog-only
|
||||
`kind:"inspection"` item on the normal sequential source lane, at the configured
|
||||
delays. No image or cover requests occur during inspection. Successful inspection
|
||||
sets `status:"ready"`, displays the canonical `sourceTitle` and source availability,
|
||||
and waits for **Confirm download**. Ready items survive daemon restarts and suppress
|
||||
duplicate nightly checks. Other jobs may proceed while confirmation is pending.
|
||||
Confirmation does not resume a deliberately paused queue or clear a source block.
|
||||
The download rediscovers the catalog; an edition title change requires re-inspection
|
||||
and confirmation. Direct API clients can still enqueue authorized downloads without
|
||||
inspection by omitting the flag. Existing download/discovery jobs are unchanged.
|
||||
|
||||
**Inspect again** operates on an inactive ready, completed, unavailable or quarantined
|
||||
item. It never overrides a source access block and does not fetch external-link pages.
|
||||
Old MangaDex unavailable messages are marked as needing re-inspection, not silently
|
||||
retried. No remote diagnosis is inferred from that old numeric-only adapter result.
|
||||
If the item was a completed download, inspection compares its selected chapters with
|
||||
the shared library inventory. An already-present selection remains **Complete** with
|
||||
its original completion and publication counts; it does not become an awaiting-download
|
||||
item. Newly available chapters instead remain **Ready** for explicit confirmation.
|
||||
WeebCentral catalog inspection accepts numbered Chapter, Episode, Rating, Night,
|
||||
Log and Act labels. If chapter links exist but none can be numbered, the parser
|
||||
reports a label-format problem; an actually empty catalog is reported separately.
|
||||
|
||||
`complete` means the selected **job** finished, not that a series is fully loaded.
|
||||
`publishedCount` counts chapters newly exposed by that run; `selectedCount` is its
|
||||
remaining selected scope. Resume may reduce those counts because the inventory skips
|
||||
chapters already published. MangaDex availability records hosted release/selection
|
||||
counts, unnumbered release counts, metadata's last chapter and the highest hosted
|
||||
number. A lower hosted maximum proves partial coverage; an equal maximum does **not**
|
||||
prove completeness. Other cases say completeness is unverified. The last chapter
|
||||
number is never used as a chapter count.
|
||||
|
||||
Empty English hosted catalogs distinguish no English releases listed, English
|
||||
external-link-only releases, and no hosted English releases with pages. External
|
||||
availability is checked through a separate, paced `includeExternalUrl=1` feed request
|
||||
only when hosted selections are empty; downloads use `includeExternalUrl=0`.
|
||||
|
||||
## Queue scopes and duplicate policy
|
||||
|
||||
Every new queue item has its own UUID plus the stable upstream `seriesId`. This allows
|
||||
several jobs for one manga (for example volume 2 followed by volume 3) without sharing
|
||||
queue identity. Its `scope` is one of:
|
||||
|
||||
| Mode | Meaning |
|
||||
|---|---|
|
||||
| `missing` | Every eligible chapter absent from the library |
|
||||
| `latest` | The newest eligible missing chapter |
|
||||
| `last` / `first` | The last/first `count` eligible missing chapters |
|
||||
| `chapter` | One numeric chapter |
|
||||
| `after` | Every numeric chapter after the supplied chapter |
|
||||
| `volume` | Chapters whose source catalog explicitly reports that volume |
|
||||
| `ids` | Internal exact source chapter ids produced by nightly discovery |
|
||||
|
||||
Unnumbered MangaDex releases use `special-mangadex-<chapter-uuid>` as their logical
|
||||
key, never an invented chapter zero or one. Numbered release deduplication is unchanged;
|
||||
unnumbered releases stay distinct unless their source UUID is identical. Two pilot
|
||||
translations therefore remain two labeled, group-credited releases, not a guessed
|
||||
numbered chapter or silently merged translation. Numeric `chapter`/`after` scopes
|
||||
exclude them; missing, exact-ID and applicable volume scopes include them. Catalog
|
||||
ordering puts numbered chapters first and source-ID-ordered specials afterward;
|
||||
first/last/latest operate on that ordering, not on publication timestamps.
|
||||
|
||||
`existingPolicy:"skip-any"` skips a chapter when any version already exists.
|
||||
`existingPolicy:"add-source"` permits another host's version but still skips a version
|
||||
already present from the selected host. Existing version folders are never overwritten.
|
||||
|
||||
Before downloading, the scraper reads a lightweight library inventory. Work identity is
|
||||
matched by exact source id, exact source URL, explicit library manga id, or verified
|
||||
AniList id—never by fuzzy title. The filesystem library remains authoritative: a stale
|
||||
checkpoint cannot make a missing published chapter look complete.
|
||||
|
||||
## Nightly discovery
|
||||
|
||||
`manga.json` stores the admin override at `x-scraper.autoUpdate`. When it is absent,
|
||||
`status:"ongoing"` defaults to enabled and other statuses default to disabled. At the
|
||||
configured local `autoUpdateTime`, the daemon queues one catalog-only discovery item for
|
||||
the preferred supported source of every enabled manga. Discovery shares the normal
|
||||
sequential source lane and pacing. If strictly newer eligible chapters are found, that
|
||||
same item is promoted to a download job containing their source ids. A successful
|
||||
no-change check records `lastCheckedAt` and its result in the per-manga/source
|
||||
update summary, appends history, and retires the discovery queue row without page
|
||||
requests. `lastDownloadedAt` is recorded when a chapter is published or a download
|
||||
job completes. These summaries persist in `queue.json` as `updateChecks`; on load,
|
||||
legacy completed no-change discovery rows migrate into the summary and are retired.
|
||||
Failed checks remain visible for attention. Duplicate pending checks for the same
|
||||
source series are suppressed.
|
||||
Only numbered chapters strictly newer than the library's highest regular chapter,
|
||||
plus unseen source-ID specials, are considered updates. Historical numbered gaps
|
||||
are never backfilled by the nightly task; use an explicit queue
|
||||
scope when a gap is intentional work.
|
||||
|
||||
## Private daemon API
|
||||
|
||||
All responses are JSON. When `SCRAPER_TOKEN` is set, send
|
||||
`Authorization: Bearer <token>`.
|
||||
|
||||
| Method | Path | Body / result |
|
||||
|---|---|---|
|
||||
| `GET` | `/health` | Basic liveness |
|
||||
| `GET` | `/api/state` | Queue, all active source checkpoints, settings, errors, history, adapters |
|
||||
| `POST` | `/api/queue` | `{url,title?,metadataSearch?,scope?,existingPolicy?,libraryMangaId?,inspect?}` |
|
||||
| `POST` | `/api/queue/remove` | `{id}`; active downloads cannot be removed |
|
||||
| `POST` | `/api/queue/clear-completed` | `{source?}`; remove inactive successful download job rows only; preserve library, history, checkpoints and other statuses |
|
||||
| `POST` | `/api/queue/reorder` | `{ids}` containing every queue id exactly once |
|
||||
| `POST` | `/api/control` | `{action:"pause"|"resume"|"retry"|"inspect"|"confirm"|"skip",id?}` |
|
||||
| `POST` | `/api/settings` | `sourceRates.{adapter}.{minMs,maxMs,chapterMinMs,chapterMaxMs}` plus `autoUpdateEnabled` and local `autoUpdateTime` (`HH:MM`) |
|
||||
| `POST` | `/api/discovery/run` | Queue an immediate update check for every enabled manga |
|
||||
| `POST` | `/api/library/auto-update` | `{id,enabled}` updates `manga.json` atomically |
|
||||
| `POST` | `/api/legacy/publish-ready` | Paused-only validation/publication of completed legacy work; converts its checkpoint to incremental mode |
|
||||
|
||||
Legacy top-level rate fields remain a fallback for adapters without a per-source entry. Settings also accept `autoUpdateEnabled` and a local `autoUpdateTime` in `HH:MM` form.
|
||||
|
||||
The reader exposes only `GET/POST /api/admin/scraper`, protected by the existing
|
||||
admin session. It maps a small command allowlist to the private API and never returns
|
||||
the shared token.
|
||||
|
||||
## Adapter contract
|
||||
|
||||
Each source plugin is registered in `scraper/adapters/index.js` with:
|
||||
|
||||
```js
|
||||
{
|
||||
id: 'host-id',
|
||||
label: 'Human name',
|
||||
match(url) { /* return {source,id,title,url} or null */ },
|
||||
async discover(item, { settings, shouldStop, onEvent }) { /* catalog only */ },
|
||||
async run(item, {
|
||||
libraryRoot, settings, shouldStop, onPublished, onCatalog
|
||||
}) { /* publish selected chapters and return counts/final path */ }
|
||||
}
|
||||
```
|
||||
|
||||
Adapters discover host-specific catalogs and pages, but reuse the central pacing,
|
||||
checkpoint, metadata, and atomic publisher modules. They must stop on HTTP 401, 403,
|
||||
429, CAPTCHA, challenge, or access-denied responses. Never add protection bypasses.
|
||||
|
||||
`onCatalog(catalog)` persists source title and optional availability before any page
|
||||
acquisition. The MangaDex adapter accepts only explicit title UUID URLs, verifies
|
||||
AniList cross-references, selects one English release per numeric chapter, preserves
|
||||
unnumbered source identities, credits groups and downloads sequential full-resolution
|
||||
MangaDex@Home pages. Special-only editions do not use fuzzy AniList lookup or AniList
|
||||
fallback library matching; their canonical source title names new folders. Explicit
|
||||
library targets remain operator choices; existing folders are never renamed.
|
||||
|
||||
## Admin UI
|
||||
|
||||
**Settings → Admin → Scraper** polls the authenticated proxy every three seconds
|
||||
only while that section is active. Activity, Queue, Updates and Sources are
|
||||
separate local tabs with a shared source filter. The section shows:
|
||||
|
||||
- every active source, manga, book/season, chapter, page, and overall progress;
|
||||
- source warnings and fatal errors;
|
||||
- queue ordering and removal controls;
|
||||
- add-by-URL, pause, resume, and retry controls;
|
||||
- queue scope, exact chapter/count/volume, target manga, and alternate-source controls;
|
||||
- per-manga auto-update checkboxes, nightly time, and a manual **Check now** action;
|
||||
- last checked and last downloaded times per manga/source, with old check events
|
||||
collapsed into recent history instead of occupying queue space;
|
||||
- a collapsed completed-jobs section with confirmation before bulk cleanup;
|
||||
- persistent image and chapter delay ranges.
|
||||
|
||||
Rate changes apply on the next manga. To apply them to the active manga, pause and
|
||||
resume it; the current chapter restarts safely if it was incomplete.
|
||||
@@ -0,0 +1,81 @@
|
||||
# Settings workspace
|
||||
|
||||
The settings page has two menu levels: ownership (User/Admin), then a focused
|
||||
section. Only one section is shown at a time. Scraper tabs are local controls,
|
||||
not a third global menu level.
|
||||
|
||||
```text
|
||||
Settings
|
||||
User
|
||||
Reading
|
||||
Reading time & privacy
|
||||
Notifications
|
||||
Account
|
||||
Admin
|
||||
Scraper
|
||||
[Activity | Queue | Updates | Sources]
|
||||
Server & library
|
||||
Users & access
|
||||
Collections & metadata
|
||||
Service health
|
||||
```
|
||||
|
||||
Desktop uses a sticky sidebar; mobile uses a wrapping menu above the content.
|
||||
Active sections have a visual marker and `aria-current`. Navigation uses real
|
||||
links, supports keyboard use and browser back/forward, and focuses the section
|
||||
heading when switching within Settings.
|
||||
|
||||
## Routes and ownership
|
||||
|
||||
Personal routes: `settings.html#user/reading`, `#user/reading-time`,
|
||||
`#user/notifications`, `#user/account`.
|
||||
|
||||
Admin routes: `settings.html#admin/scraper`, `#admin/library`, `#admin/users`.
|
||||
Scraper deep links append `/activity`, `/queue`, `/updates` or `/sources`.
|
||||
The old `/hosters` link opens Sources. The scraper default is Activity.
|
||||
Existing `collections.html` and `health.html` retain their URLs but share the
|
||||
same menu. `#admin/collections` and `#admin/health` redirect to those pages.
|
||||
Unknown routes default to Reading. Non-admin accounts never render the Admin menu
|
||||
or load admin data from Settings; admin hash routes normalize to Reading.
|
||||
Server-side authorization remains authoritative and still rejects non-admin API calls.
|
||||
|
||||
`public/settings-nav.js` defines the shared menu and route model. `settings.js`
|
||||
preserves personal form nodes across section changes and loads admin sections lazily.
|
||||
Scraper polling runs only while its section is active. Draft inputs survive polling
|
||||
and tab switches; successful commands reload state. Command failures are visible
|
||||
without deleting the user's draft. Source chips filter the four scraper views
|
||||
without changing queue ownership or per-source pacing.
|
||||
The Add manga, Completed downloads and Recent check activity disclosure boxes
|
||||
also retain their open/closed state across polling and command redraws; a page
|
||||
reload still starts them collapsed.
|
||||
|
||||
Activity shows active jobs, source attention, and the next queued items. Queue
|
||||
keeps reorder/inspect/confirm/remove controls, scopes, and an Add manga form;
|
||||
completed jobs are collapsed by default. **Clear completed downloads** removes
|
||||
only inactive, successful download job rows, optionally limited to the selected
|
||||
source. It never deletes library content, checkpoints, history or other statuses.
|
||||
Updates shows the nightly schedule, per-manga switches, the latest check and
|
||||
download times by manga/source, and a collapsed recent-check log. Successful
|
||||
no-change nightly checks update this summary and leave the queue. Sources owns
|
||||
the per-hoster request and chapter delay settings.
|
||||
|
||||
Reading-time collection controls, export and deletion live under Reading time &
|
||||
privacy; reader layout controls stay under Reading. Rescan controls and user
|
||||
administration are separate. Server & library describes existing maintenance,
|
||||
not an editor for Docker environment variables or an unrestricted restart console.
|
||||
|
||||
## Verification
|
||||
|
||||
`npm test` covers route defaults, deep links, ownership and existing service/API
|
||||
behavior. `node tools/test-compose.js --settings-ui` adds optional Playwright
|
||||
headless browser QA against the disposable Compose fixture, never the real user
|
||||
database or queue. Make Playwright available on the development machine; Windows
|
||||
uses installed headless Edge by default, overridable with `QA_BROWSER_CHANNEL`.
|
||||
The application itself remains dependency-free.
|
||||
|
||||
The UI check verifies desktop and 390×844 mobile navigation, form saves, draft
|
||||
retention, tab visibility, back/forward, stopping hidden scraper polling, shared
|
||||
collections/health navigation, non-admin denial and horizontal overflow.
|
||||
`--queue-ui` additionally checks the source filter, edition confirmation, add form,
|
||||
completed cleanup and mobile layout. Screenshots are generated under ignored
|
||||
`data/qa/settings/` and `data/qa/queue/`. No image sources are contacted.
|
||||
Reference in new issue
Block a user