Maashkinoozhe/MangaCircus
0
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Consolidate MangaCircus app, scraper, and Compose baseline

Browse Source
This commit is contained in:
Maashkinoozhe committed 2026-10-07 23:28:20 +02:00
1 parent cfb2d79103
commit eef4ffa07c
103 files changed
+9316 -172
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/architecture.md
+81 -17
View File
@@ -1,11 +1,20 @@
# MangaCircus architecture
MangaCircus is a self-hosted reader for manga, comics and webtoons. One Node.js process serves a handful of static pages and a small JSON API. The content is a folder tree on disk, and everything user-related lives in one JSON file.
MangaCircus is a self-hosted reader for manga, comics and webtoons. The reader owns
accounts/progress and a separate collection catalog. The scraper owns the persistent
queue and chapter publication. Sentinel observes health/progress and requests bounded
checkpoint recovery. Separate supervisors isolate service failures. All content remains
a plain folder tree; there is no database server or embedded autonomous LLM.
Related documents:
- [library-format.md](library-format.md): the folder format.
- [import.md](import.md): how the folder tree is indexed.
- [scraper-guide.md](scraper-guide.md): rules for tools that write the library.
- [scraper-service.md](scraper-service.md): managed queue, private API, and adapter contract.
- [collections.md](collections.md): relations, groups, curator UI and agent research.
- [resilience.md](resilience.md): supervision, recovery limits and incidents.
- [compose-deployment.md](compose-deployment.md): local/VM deployment and release workflow.
- [settings-navigation.md](settings-navigation.md): User/Admin workspace, scraper tabs and deep links.
## 1. Principles
@@ -13,7 +22,7 @@ Related documents:
|-----------|-------------|
| **The file system is the source of truth for content** | No import step, no copies, no content database. The library can be mounted read-only. Anything that writes files can feed it |
| **As dumb as possible** | No dependencies (`package.json` has none), no framework, no build step, no transpiler. Plain ES2019+ JavaScript in the browser, plain Node on the server |
| **One process, one data file** | In-memory library index, user data in `data/db.json`. Back up one file, run one container |
| **Small independent services** | The reader stays dependency-free; the optional scraper daemon can stop or restart without taking reading offline |
| **Readable without the app** | The folder format works in any picture viewer. The app adds progress, layout and convenience, not lock-in |
## 2. System context
@@ -35,6 +44,18 @@ Related documents:
## 3. Components
Runtime topology (2026-10-05): native `tools/start-all.js` launches reader, scraper and
Sentinel independently. In Compose, each container runs `runtime/launch.js <service>`
and exactly one service child. A failure does not stop unrelated services. Exactly one
reader and queue-owning scraper may write each shared state tree; do not scale replicas.
The catalog and relation store is reader-owned `data/catalog.json`. Sentinel writes only
`data/sentinel/`; it has no Docker socket or arbitrary code-repair authority.
The admin browser never talks to the scraper daemon directly. Admin-only reader
routes proxy commands to the daemon's private API, so its shared token stays on the
server. The daemon publishes complete folders atomically and increments a durable
revision; the reader polls that revision and rescans the library after publication.
| Path | Lines | Responsibility |
|------|------:|----------------|
| `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks |
@@ -43,7 +64,7 @@ Related documents:
| `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) |
| `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle |
| `public/read.html` + `reader.js` | ~470 | The reader (§7.2) |
| `public/settings.html` + `settings.js` | ~115 | Reading preferences, webhook, password, admin (rescan, users) |
| `public/settings.html` + `settings.js` + `settings-nav.js` | — | User/Admin navigation, personal preferences, lazy admin sections and scraper tabs |
| `public/login.html` + `login.js` | ~50 | Sign-in, first-run admin creation, optional sign-up |
| `public/app.css` | ~150 | All styles (dark theme, responsive) |
| `tools/png.js`, `tools/make-dummy.js` | ~310 | Demo library generator: a tiny PNG encoder with a bitmap font, no image libraries |
@@ -51,14 +72,15 @@ Related documents:
## 4. Runtime model
- **One process, event loop only.** All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests.
- **Reader process:** one event loop. All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests.
- **Optional scraper process:** one queue worker, one remote request at a time, durable checkpoints, and a private control API.
- **In-memory state:**
- `lib` is the current library index, replaced as a whole after each scan.
- `db` is all user data, loaded from `db.json` at start.
- **Timers:**
- a library scan at start and every `SCAN_INTERVAL` minutes,
- a debounced `db.json` write, 500 ms after the last change.
- **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json` synchronously, then exit. That matters for `docker stop`.
- **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json`; the daemon cooperatively stops at a page boundary and keeps its archive checkpoint.
- **Durability:** every file write goes to `*.tmp` first and is then renamed over the target, so a file is never half-written. A hard kill loses at most the changes of the last ~0.5 s. A corrupt `db.json` makes the server **refuse to start** rather than start empty and overwrite it.
## 5. Data model
@@ -93,6 +115,9 @@ This keeps ids human-readable and derivable without a database. The price: renam
"lastMode": "book", // the user's intended mode; changed only by clicking a mode
"adapt": true, // use a fitting mode for incompatible content (webtoon → flow)
"skipExtras": false, "skipCovers": false,
"flowPageGap": true, // show the 8 px separator between pages in Flow
"readingClock": false, // opt in to credible per-chapter active-time tracking
"readingStory": true, // show private retrospective summaries when history exists
"zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 }, // page zoom per mode
"webhook": ""
}
@@ -106,6 +131,12 @@ This keeps ids human-readable and derivable without a database. The price: renam
"done": false, // sticky once true (reset via "mark unread")
"shift": [] // book mode: run starts shown alone
} } } },
"reading": { "admin": { "Moon Garden": { "1/2": {
"schema": 1, "totalActiveMs": 412000, "loadingMs": 7400,
"sessions": 3, "completedPasses": 2,
"inputMs": { "mouse": 12000, "touch": 380000, "keyboard": 20000 },
"currentPass": { "version": "c0002 Roots [Official]", "seenRanges": [[0, 13]], "activeMs": 151000 }
} } } },
"follows": { "admin": { "Moon Garden": true } },
"notes": { "admin": [ { "id": "a1b2c3d4e5f6", "t": 1790797300000, "m": "Moon Garden", "c": "2/6",
"text": "Moon Garden · Ch. 6: Frost", "read": false } ] },
@@ -142,10 +173,13 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
| `GET /api/me` | user | Name, admin flag, settings, unread count |
| `POST /api/settings` `{…partial}` | user | Validated update of the settings keys above |
| `POST /api/password` `{old, new}` | user | Change password |
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read |
| `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions and the user's progress, `continue` target |
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read, and optional personal Reading Story aggregates |
| `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions, progress, optional chapter-time summaries, and `continue` target |
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, optional Active Reading Clock summary, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
| `POST /api/progress` `{m, c, v, p, n, done, shift}` | user | Save the reading position |
| `POST /api/reading/pulse` `{m, c, v, session, seq, …deltas}` | user | When opted in, idempotently merge a clamped active-time pulse, verified page coverage and input totals under a single-reader lease |
| `GET /api/reading/export` | user | Export the user's private derived reading-time summary without raw input or page-exposure diagnostics |
| `POST /api/reading/delete` `{m}` or `{all:true,password}` | user | Delete one manga's reading history, or all history after password confirmation, without changing normal reading progress |
| `POST /api/unread` `{m, c}` | user | Forget progress for a chapter |
| `POST /api/vote` `{m, c, v, value: 1\|-1\|0}` | user | Vote a version; returns the re-ranked versions |
| `POST /api/follow` `{m, on}` | user | Follow a manga for notifications |
@@ -237,9 +271,9 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
| Click or tap on a page, top/bottom 35 % of the screen (flow) | Scroll to the previous/next page top, at most 90 % of a screen (85 % steps for webtoons). At the very bottom → next chapter |
| Click or tap on a page, middle band | Toggle the bars |
| Horizontal swipe ≥ 50 px within 800 ms (paged, not zoomed in) | Previous/next |
| Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Page zoom 20–600 %, snapping to 100 % within ±3 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages |
| Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Continuous page zoom 20–600 %, snapping to 100 % within ±1 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages |
| Mouse drag (moved ≥ 6 px) | Pan; the following click is swallowed. Hand cursor when the content is larger than the screen |
| Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` `-` `0` zoom; `m`/Esc bars; `f` fullscreen |
| Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` and `-` zoom in 5-percentage-point steps, `0` fits at 100 %; `m`/Esc bars; `f` fullscreen |
**Bars:**
- Shown on open.
@@ -260,22 +294,50 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
- Stores the page index in the full page list, so skipping extras doesn't corrupt it.
- On open, the saved page is used for the same version, and mapped proportionally for another version. A finished chapter reopens at the start.
**Active Reading Clock (opt-in):**
- Runs only while the reader is visible and focused, a sufficiently exposed page is decoded, a server lease is held, and accepted reader input occurred in the last 50 seconds.
- Backgrounding or blurring pauses immediately and requires fresh input to resume. A delayed callback can never add time beyond the 50-second input deadline.
- A page is verified after at least 35% meaningful exposure for 200 ms cumulatively. Exposure is closed and evaluated when the page leaves the viewport, so a fast but plausible webtoon scroll is not lost between samples. A five-pages-per-second token bucket rejects implausible coverage without discarding otherwise credible active time.
- `BroadcastChannel` coordinates tabs on one browser. The authenticated API adds a 15-second per-user lease for other devices, and releases it when the tab pauses or leaves.
- Pulses are sent at most every 10 seconds and on pause/page exit. The server validates the manga/chapter/version and page indices, de-duplicates `(session, seq)`, clamps time to server-observed intervals and stores compact page ranges separately from ordinary progress.
- The bottom toolbar chip shows active time and the current state; its panel adds pass time, verified pages, predominant input and loading wait.
**Zoom persistence:** per mode key (`flow`, `page`, `book`, `strip`), saved 800 ms after the last change.
## 8. Deployment
### Current Windows deployment
The observed local installation uses the Windows Scheduled Task
`\MangaCircus Supervisor`. At logon it runs `tools/run-supervisor.ps1`, which appends
logs under `data/scraper/` and starts one `tools/start-all.js` supervisor. That parent
owns exactly one reader (`server.js`, port 8080) and one scraper daemon
(`scraper/daemon.js`, private port 8091). Starting `npm run start:all` manually at the
same time would create conflicting writers and is not supported.
The scraper's authoritative `data/scraper/queue.json` is schema version 3. It supports
legacy archive checkpoints and paused conversion of completed legacy work while adding
source-aware lanes, scoped queue jobs, incremental chapter publication, and nightly catalog discovery. See
[local-deployment.md](local-deployment.md) for operational checks and restart/recovery,
and [scraper-service.md](scraper-service.md) for the queue/API contract.
### Docker or manual deployment
```bash
docker compose up -d --build
```
- **Image:** `node:24-alpine` plus the app files. It runs as user `node` (uid 1000) and has a health check against `/api/auth`.
- **Images/services:** the same `node:24-alpine` image runs a reader service and a scraper service. The reader health check targets `/api/auth`.
- **Volumes:**
- `/library`: read-only is enough. Files must be readable by uid 1000.
- `/data`: must be writable.
- **Configuration:** only environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `COOKIE_SECURE`).
- **Without Docker:** `node server.js` with Node ≥ 18.
- `/library`: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000.
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
- **Upgrades:** replace the code and restart. There are no migrations: unknown fields in `db.json` are kept, and missing settings fall back to defaults.
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
unknown fields in `db.json` are kept. The scraper normalizes old queue entries into
schema version 3 on startup and preserves legacy job checkpoint behavior. Take a
queue/checkpoint recovery copy before upgrading a live scraper.
## 9. Capacity and limits
@@ -308,5 +370,7 @@ docker compose up -d --build
- **Notifications:** in-app plus webhook only. Web Push would need VAPID keys and a service worker.
- **No login rate limiting and no CSP header.** Both are small additions in `server.js`.
- **In-place image overwrites** are only detected by a full rescan.
- **No automated test suite.** Verification so far was done with the demo library (`tools/make-dummy.js`), API smoke scripts and manual browser testing on desktop and mobile viewports. `parseKey` and `sizeFromBuffer` are pure functions, and `scanLibrary` only needs a folder. Together with the book pairing they are the natural first targets for unit tests.
- **Reader layout remains mostly smoke/manual.** Reading analytics has focused Node tests
for its persisted merge rules, while DOM layout, authentication and mobile browser
lifecycle behavior still need browser-level/manual verification.
- **Adding content by API** (instead of files) would be a new route plus the staging and renaming rules from [import.md §7](import.md#7-consistency-writing-into-a-live-library).