# MangaCircus architecture MangaCircus is a self-hosted reader for manga, comics and webtoons. The reader owns accounts/progress and a separate collection catalog. The scraper owns the persistent queue and chapter publication. Sentinel observes health/progress and requests bounded checkpoint recovery. Separate supervisors isolate service failures. All content remains a plain folder tree; there is no database server or embedded autonomous LLM. Related documents: - [library-format.md](library-format.md): the folder format. - [import.md](import.md): how the folder tree is indexed. - [scraper-guide.md](scraper-guide.md): rules for tools that write the library. - [scraper-service.md](scraper-service.md): managed queue, private API, and adapter contract. - [collections.md](collections.md): relations, groups, curator UI and agent research. - [resilience.md](resilience.md): supervision, recovery limits and incidents. - [compose-deployment.md](compose-deployment.md): local/VM deployment and release workflow. - [settings-navigation.md](settings-navigation.md): User/Admin workspace, scraper tabs and deep links. - [peering.md](peering.md): optional private server-to-server catalog and transfer. - [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits. - [source-modules.md](source-modules.md): adding and maintaining first-party hosters inside the single scraper process. ## 1. Principles | Principle | Consequence | |-----------|-------------| | **The file system is the source of truth for content** | No import step, no copies, no content database. The library can be mounted read-only. Anything that writes files can feed it | | **As dumb as possible** | No dependencies (`package.json` has none), no framework, no build step, no transpiler. Plain ES2019+ JavaScript in the browser, plain Node on the server | | **Small independent services** | The reader stays dependency-free; the optional scraper daemon can stop or restart without taking reading offline | | **Readable without the app** | The folder format works in any picture viewer. The app adds progress, layout and convenience, not lock-in | ## 2. System context ``` ┌──────────────┐ writes folders (staged + renamed) scraper / ───▶│ library/ │◀──────────────────────────────────┐ person │ (read-only │ │ │ for the app)│ │ └──────┬───────┘ │ │ readdir / stat / small reads │ ┌──────▼─────────────────────────────┐ │ browser ─────▶│ node server.js │ webhook │ (desktop, │ ├ library.js scanner → index │────────────▶ ntfy / Gotify / … phone, PWA) │ ├ HTTP: / /lib/… /api/… │ (optional) │ └ data/db.json data/scan-cache.json│ └─────────────────────────────────────┘ ``` ## 3. Components Runtime topology (2026-10-05): native `tools/start-all.js` launches reader, scraper and Sentinel independently. In Compose, each container runs `runtime/launch.js ` and exactly one service child. A failure does not stop unrelated services. Exactly one reader and queue-owning scraper may write each shared state tree; do not scale replicas. The catalog and relation store is reader-owned `data/catalog.json`. Sentinel writes only `data/sentinel/`; it has no Docker socket or arbitrary code-repair authority. The admin browser never talks to the scraper daemon directly. Admin-only reader routes proxy commands to the daemon's private API, so its shared token stays on the server. The daemon publishes complete folders atomically and increments a durable revision; the reader polls that revision and refreshes changed manga after publication. The reader restores a disposable full-index snapshot at startup so browsing need not wait for a filesystem walk. Shallow directory stamps identify changed manga; a daily full reconciliation catches in-place manual edits. The last full-scan time survives restarts, so restarting does not itself force another complete walk. | Path | Lines | Responsibility | |------|------:|----------------| | `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks | | `library.js` | ~170 | Folder scanner, name parsing, image-size reader, scan cache | | `public/common.js` | ~85 | Browser helpers: `h()` element builder, `api()` fetch wrapper, top navigation with the notification bell | | `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) | | `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle | | `public/read.html` + `reader.js` | ~470 | The reader (§7.2) | | `public/settings.html` + `settings.js` + `settings-nav.js` | — | User/Admin navigation, personal preferences, lazy admin sections and scraper tabs | | `public/login.html` + `login.js` | ~50 | Sign-in, first-run admin creation, optional sign-up | | `public/app.css` | ~150 | All styles (dark theme, responsive) | | `tools/png.js`, `tools/make-dummy.js` | ~310 | Demo library generator: a tiny PNG encoder with a bitmap font, no image libraries | | `Dockerfile`, `docker-compose.yml` | | `node:24-alpine`, non-root, `/library` + `/data` | ## 4. Runtime model - **Reader process:** one event loop. All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests. - **Optional scraper process:** one queue worker, one remote request at a time, durable checkpoints, and a private control API. - **In-memory state:** - `lib` is the current library index, replaced as a whole after each scan. - `db` is all user data, loaded from `db.json` at start. - **Timers:** - a saved-index restore and shallow check at start (full scan when no snapshot), - a shallow check every `SCAN_INTERVAL` minutes, full reconciliation every `FULL_SCAN_INTERVAL` minutes, - a debounced `db.json` write, 500 ms after the last change. - **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json`; the daemon cooperatively stops at a page boundary and keeps its archive checkpoint. - **Durability:** every file write goes to `*.tmp` first and is then renamed over the target, so a file is never half-written. A hard kill loses at most the changes of the last ~0.5 s. A corrupt `db.json` makes the server **refuse to start** rather than start empty and overwrite it. ## 5. Data model ### 5.1 Content: the library index The index is built by `library.js` from the folder tree (see [import.md §2.6](import.md#26-the-result)). It is never written anywhere, except for the scan cache. ### 5.2 Identity Every reference from user data to content uses ids derived from names on disk: | Entity | Id | Example | Stable under | |--------|----|---------|--------------| | Manga | manga folder name | `Moon Garden` | JSON edits, anything inside the folder | | Book | normalised book number (`x` prefix for extras) | `1`, `x1` | title changes | | Chapter | `/` | `1/2.5` | title changes, adding/removing versions | | Version | chapter folder name | `c0002 Roots [MangaDex]` | nothing: renaming the folder creates a new version | This keeps ids human-readable and derivable without a database. The price: renaming a manga folder, renumbering a chapter or moving it to another book creates a new entity (see [library-format.md → Renaming](library-format.md#renaming-what-survives)). ### 5.3 User data: `data/db.json` ```jsonc { "users": { "admin": { "name": "admin", "admin": true, "created": 1790796794000, "hash": ":", "settings": { "startMode": "last", // "last" | "flow" | "page" | "book" "lastMode": "book", // the user's intended mode; changed only by clicking a mode "adapt": true, // use a fitting mode for incompatible content (webtoon → flow) "skipExtras": false, "skipCovers": false, "flowPageGap": true, // show the 8 px separator between pages in Flow "readingClock": false, // opt in to credible per-chapter active-time tracking "readingStory": true, // show private retrospective summaries when history exists "zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 }, // page zoom per mode "webhook": "" } } }, "sessions": { "": { "u": "admin", "t": 1790796794000 } }, "progress": { "admin": { "Moon Garden": { "1/2": { "v": "c0002 Roots [MangaDex]", // version read "p": 4, "n": 14, // page index in that version (all pages), page count "t": 1790796800000, // last activity "done": false, // sticky once true (reset via "mark unread") "shift": [] // book mode: run starts shown alone } } } }, "reading": { "admin": { "Moon Garden": { "1/2": { "schema": 1, "totalActiveMs": 412000, "loadingMs": 7400, "sessions": 3, "completedPasses": 2, "inputMs": { "mouse": 12000, "touch": 380000, "keyboard": 20000 }, "currentPass": { "version": "c0002 Roots [Official]", "seenRanges": [[0, 13]], "activeMs": 151000 } } } } }, "follows": { "admin": { "Moon Garden": true } }, "notes": { "admin": [ { "id": "a1b2c3d4e5f6", "t": 1790797300000, "m": "Moon Garden", "c": "2/6", "text": "Moon Garden · Ch. 6: Frost", "read": false } ] }, "votes": { "Moon Garden": { "1/2": { "c0002 Roots [Official]": { "admin": 1, "reader": -1 } } } }, "known": { "Moon Garden": ["1/1", "1/2", "1/2.5", "..."] } // every chapter id ever seen (append-only) } ``` - `data/scan-cache.json` holds `{ "//": { m: , p: [[file, w, h], …] } }`. It is disposable. - `data/library-index.json` holds a disposable, versioned index snapshot and shallow directory stamps. The library folders remain authoritative; a missing or invalid snapshot triggers a full scan. Rollback-compatible `scan-cache.json` is retained. - **Backups:** copy `data/db.json`. The atomic writes make a copy at any time consistent. ## 6. Server ### 6.1 Request routing | Path | Handling | |------|----------| | `/api/*` | `routes[" "]`. Handlers get `{ u, b, q, req, res }` and return an object, which is sent as JSON. Throwing `HttpError` sends `{ error }` with that status | | `/lib/*` | Image files from `LIBRARY`. Requires a session, allows image extensions only, and the path must resolve inside `LIBRARY`. `ETag` + `private, max-age=86400` | | everything else | Static files from `public/` (`/` → `index.html`). `ETag` + `no-cache` | Pages are static. Each page's script calls the API and redirects to `login.html?next=…` on `401`. All URLs are relative, so the app also works under a sub-path behind a reverse proxy. ### 6.2 API All requests and responses are JSON. POSTs require `Content-Type: application/json`. | Method & path | Auth | Purpose | |---------------|------|---------| | `GET /api/auth` | public | `{ firstRun, signup }` for the login page | | `POST /api/signup` `{name, password}` | public | First account (becomes admin), or anyone when `SIGNUP=open` | | `POST /api/login` `{name, password}` | public | Starts a session (cookie `sid`) | | `POST /api/logout` | user | Ends the session | | `GET /api/me` | user | Name, admin flag, settings, unread count | | `POST /api/settings` `{…partial}` | user | Validated update of the settings keys above | | `POST /api/password` `{old, new}` | user | Change password | | `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read, and optional personal Reading Story aggregates | | `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions, progress, optional chapter-time summaries, and `continue` target | | `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, optional Active Reading Clock summary, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read | | `POST /api/progress` `{m, c, v, p, n, done, shift}` | user | Save the reading position | | `POST /api/reading/pulse` `{m, c, v, session, seq, …deltas}` | user | When opted in, idempotently merge a clamped active-time pulse, verified page coverage and input totals under a single-reader lease | | `GET /api/reading/export` | user | Export the user's private derived reading-time summary without raw input or page-exposure diagnostics | | `POST /api/reading/delete` `{m}` or `{all:true,password}` | user | Delete one manga's reading history, or all history after password confirmation, without changing normal reading progress | | `POST /api/unread` `{m, c}` | user | Forget progress for a chapter | | `POST /api/vote` `{m, c, v, value: 1\|-1\|0}` | user | Vote a version; returns the re-ranked versions | | `POST /api/follow` `{m, on}` | user | Follow a manga for notifications | | `GET /api/notifications` · `POST /api/notifications/read` `{id \| all}` | user | Notification list and read state | | `GET /api/admin` | admin | Users, last scan result, library path | | `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result | | `POST /api/admin/user` `{name, password, admin}` | admin | Create a user or reset a password | | `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data | ### 6.3 Authentication and security | Concern | Measure | |---------|---------| | Passwords | `scrypt` (32-byte key, 16-byte random salt), constant-time compare. Unknown users are checked against a dummy hash to keep timing similar | | Sessions | 256-bit random token in cookie `sid` (`HttpOnly`, `SameSite=Lax`, 1 year, `Secure` with `COOKIE_SECURE=1`). Only its SHA-256 is stored. Pruned at start after a year or when the user is gone | | CSRF | POSTs must be `application/json` (plain cross-site forms can't send that) and must not carry `Sec-Fetch-Site: cross-site` | | Path traversal | Paths are decoded, resolved and checked against the root directory. `/lib/` serves only image extensions, never the JSON files | | XSS | The UI builds DOM nodes and inserts all data as text (`h()` helper). The only markup inserted is the app's own constant SVG icons. External links are rendered only for `http(s)` URLs | | Request size | JSON bodies are limited to 1 MB | | Authorisation | `admin` flag checked per admin route. Users only ever see their own progress, follows and notifications. Votes are shown as totals plus the user's own | | Not included | Login rate limiting, a CSP header, TLS. Run behind a reverse proxy if exposed. Webhook URLs are user-chosen, so any user can make the server send a POST to an address they choose. Fine for a household; consider it before `SIGNUP=open` | ### 6.4 Version ranking At request time: the sum of all users' votes, then `chapter.json.priority`, then folder order. The top version is what "open chapter" shows. An explicit `v=` (version picker, "Continue" of a manually chosen version) overrides it. ### 6.5 Notifications In-app notifications and optional webhooks are produced by `announce()` after each scan, from the append-only `known` list. See [import.md §5](import.md#5-change-detection-and-notifications). ## 7. Frontend ### 7.1 Pages Every page is a static HTML file plus one script, loaded with `defer`. All pages share `common.js` and `app.css`. There is no client-side router, no state library and no component system. Each script fetches what it needs, builds DOM with `h()`, and re-renders small parts on change. A web app manifest (`display: standalone`) makes "Add to home screen" open the app without browser bars on phones. ### 7.2 The reader (`reader.js`) **Inputs:** - the chapter payload (pages with pixel sizes, tags and rotation; direction; versions; neighbours; progress), - the user's settings. **Page model.** Each page's size is known before it loads, so the reader computes the whole layout itself and positions every page absolutely: ``` all pages ──filter (skipExtras / skipCovers)──▶ list list ──mode──▶ build(): create page elements (flow: all, lazy · page/book: current + preload next) └──▶ layout(): compute a box {x, y, w, h} per element from the viewport, zoom and page aspect └─▶ place(): apply the boxes; rotated images are sized swapped and turned with CSS ``` `layout()` repeats up to three times until the viewport size is stable, because scrollbars that appear or disappear change the viewport. **Layout rules** (z = page zoom of the current mode): | Mode | Box sizes | |------|-----------| | Flow (paged content) | `W = min(vw, vh · refAspect) · z`: a normal single page fits the screen at 100 %. Wide pages keep the single page height, capped at `max(vw, W)`. 8 px gap | | Flow (webtoon, zoom key `strip`) | `W = min(vw, median image width) · z`, no gaps, so slices join seamlessly | | Page | The page fits the viewport (`contain`) · z, centred. The canvas grows beyond the viewport when zoomed in | | Book | The pages of the spread share one height `H = min(vh, vw / Σaspect) · z`. Order is right-to-left for `rtl` | **Anchoring.** Before any re-layout (zoom, resize, pinch), `anchor(x, y)` records which page is under that point and where inside it, as a fraction. `restore()` scrolls so the same point is under the finger or cursor again. That is why zooming feels pinned to the cursor and a resize keeps the reading position. **Book pairing:** - Covers and wide pages stand alone. - The single pages between them form *runs*, and each run pairs from its start. - "⇄ Shift" toggles a run to start with a lone page. It is stored as the run's first page index in `progress.shift`. - One broken spot can therefore never shift the pairing of the whole chapter. **Mode resolution:** ``` intended = startMode == "last" ? lastMode : startMode compatible = direction == "ttb" ? [flow] : [flow, page, book] mode = adapt && intended ∉ compatible ? compatible[0] : intended ``` `lastMode` is written only when the user clicks a mode, never on an automatic switch. **Input:** | Input | Action | |-------|--------| | Click or tap outside the pages | Toggle the bars (the end card uses the whole screen as zones instead) | | Click or tap on the page(s), outer 35 % of the visible page area (paged) | Previous/next, following the reading direction (`rtl`: left = next) | | Click or tap on a page, top/bottom 35 % of the screen (flow) | Scroll to the previous/next page top, at most 90 % of a screen (85 % steps for webtoons). At the very bottom → next chapter | | Click or tap on a page, middle band | Toggle the bars | | Horizontal swipe ≥ 50 px within 800 ms (paged, not zoomed in) | Previous/next | | Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Continuous page zoom 20–600 %, snapping to 100 % within ±1 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages | | Mouse drag (moved ≥ 6 px) | Pan; the following click is swallowed. Hand cursor when the content is larger than the screen | | Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` and `-` zoom in 5-percentage-point steps, `0` fits at 100 %; `m`/Esc bars; `f` fullscreen | **Bars:** - Shown on open. - Hidden on the first page turn or after 100 px of scrolling. The first automatic hide (per device, once for Flow and once for Page/Book) shows a hint on how to get them back. - **Never shown by scrolling**, only by a tap beside the pages or in the middle of a page (or `m`). Showing them stops a running smooth scroll, so it can't hide them again right away. - The top bar has the chapter picker (every chapter, grouped by book, ✓ = read). **Loading:** - Flow mode loads images through an `IntersectionObserver` on the scroll container, 1.5 screens ahead. - Page and Book mode keep a small pool (12) of `Image` objects: - Once the pages on screen have loaded, the next 3 pages (Book: next 2 spreads) and the previous one are fetched at low priority and **decoded** (`img.decode()`). - A page turn then inserts those same, already decoded elements (`decoding="sync"`), so the old and new page swap in one frame. A fresh `` would be painted empty while it decodes, which is what caused the flicker on the first view of each page. - A page that is still loading shows a spinner (only after 150 ms, so fast loads never flash it). A page that fails shows a message. - A page with unknown size is laid out at 1000×1414 and re-laid out when it loads. **Progress:** - Saved 700 ms after the position settles, and on `pagehide` (a `fetch` with `keepalive`). - Stores the page index in the full page list, so skipping extras doesn't corrupt it. - On open, the saved page is used for the same version, and mapped proportionally for another version. A finished chapter reopens at the start. **Active Reading Clock (opt-in):** - Runs only while the reader is visible and focused, a sufficiently exposed page is decoded, a server lease is held, and accepted reader input occurred in the last 50 seconds. - Backgrounding or blurring pauses immediately and requires fresh input to resume. A delayed callback can never add time beyond the 50-second input deadline. - A page is verified after at least 35% meaningful exposure for 200 ms cumulatively. Exposure is closed and evaluated when the page leaves the viewport, so a fast but plausible webtoon scroll is not lost between samples. A five-pages-per-second token bucket rejects implausible coverage without discarding otherwise credible active time. - `BroadcastChannel` coordinates tabs on one browser. The authenticated API adds a 15-second per-user lease for other devices, and releases it when the tab pauses or leaves. - Pulses are sent at most every 10 seconds and on pause/page exit. The server validates the manga/chapter/version and page indices, de-duplicates `(session, seq)`, clamps time to server-observed intervals and stores compact page ranges separately from ordinary progress. - The bottom toolbar chip shows active time and the current state; its panel adds pass time, verified pages, predominant input and loading wait. **Zoom persistence:** per mode key (`flow`, `page`, `book`, `strip`), saved 800 ms after the last change. ## 8. Deployment ### Current Windows deployment The observed local installation uses the Windows Scheduled Task `\MangaCircus Supervisor`. At logon it runs `tools/run-supervisor.ps1`, which appends logs under `data/scraper/` and starts one `tools/start-all.js` supervisor. That parent owns exactly one reader (`server.js`, port 8080) and one scraper daemon (`scraper/daemon.js`, private port 8091). Starting `npm run start:all` manually at the same time would create conflicting writers and is not supported. The scraper's authoritative `data/scraper/queue.json` is schema version 3. It supports legacy archive checkpoints and paused conversion of completed legacy work while adding source-aware lanes, scoped queue jobs, incremental chapter publication, and nightly catalog discovery. See [local-deployment.md](local-deployment.md) for operational checks and restart/recovery, and [scraper-service.md](scraper-service.md) for the queue/API contract. ### Docker or manual deployment ```bash docker compose up -d --build ``` - **Images/services:** the same `node:24-alpine` image runs a reader service and a scraper service. The reader health check targets `/api/auth`. - **Volumes:** - `/library`: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000. - `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state. - **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI. - **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`. - **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works. - **Upgrades:** replace the code and restart. Reader state has no explicit migration; unknown fields in `db.json` are kept. The scraper normalizes old queue entries into schema version 3 on startup and preserves legacy job checkpoint behavior. Take a queue/checkpoint recovery copy before upgrading a live scraper. ## 9. Capacity and limits | Aspect | Behaviour | |--------|-----------| | Library size | The index keeps a small object per page (file name, size, tags), in the order of a few hundred bytes. Hundreds of thousands of pages fit in a modest container. Unchanged chapters cost one `stat` + one small read per scan | | First scan | Reads ≤ 256 KB per image once. Later scans use the cache | | User data | `db.json` is rewritten as a whole, at most about twice per second while people read. Comfortable up to a few MB (a household or small group, thousands of chapters read) | | Concurrency | One process, one writer. No clustering or horizontal scaling by design | | Images | Served as stored. No resizing or thumbnails, so large covers make the library page heavy | ## 10. Design decisions | Decision | Alternatives | Why | |----------|--------------|-----| | Index the folder tree in place | Import into a DB / managed storage | Stays readable by any viewer and writable by any tool. Nothing to keep in sync | | No dependencies, no framework, no build | Express, React, bundler | Minimal code, maximum longevity, runs anywhere Node runs | | JSON file for user data | SQLite (`node:sqlite` is still experimental in Node 22/24), Postgres | Zero dependencies, trivial backup, enough for the target size. Easy to swap: all access goes through the `db` object | | Page roles as file-name tags | Page lists in JSON | Visible in any file browser, survives copying, no second source of truth. Unknown tags are harmless | | Image sizes read on the server | Measure in the browser after load | Layout before load: exact scroll restore, zoom anchoring, auto-detected double pages, no layout jumps | | Absolute layout engine in the reader | CSS flow/flex layout | Deterministic geometry, needed for pointer-anchored zoom and exact position restore | | Chapter id = book number + chapter number | Random ids stored in JSON | Human-readable and needs no writes into the library. Downside: moving a chapter between books changes its identity | | Global votes, per-visit version choice | Per-user preferred version | "The better version is shown to everyone", while any user can still pick another one explicitly | | Bars never reappear on scroll | Show on scroll-up (common pattern) | Explicit requirement: readers scroll up and down constantly | ## 11. Known limitations and extension points - **Identity is name-based.** An optional stable id in `manga.json` (with a migration of the user data keys) would make manga renames safe. - **No thumbnails.** Would need an image library (e.g. `sharp`) or pre-generated `cover` files. - **Notifications:** in-app plus webhook only. Web Push would need VAPID keys and a service worker. - **No login rate limiting and no CSP header.** Both are small additions in `server.js`. - **In-place image overwrites** are only detected by a full rescan. - **Reader layout remains mostly smoke/manual.** Reading analytics has focused Node tests for its persisted merge rules, while DOM layout, authentication and mobile browser lifecycle behavior still need browser-level/manual verification. - **Adding content by API** (instead of files) would be a new route plus the staging and renaming rules from [import.md §7](import.md#7-consistency-writing-into-a-live-library).