MangaCircus/docs/architecture.md

MangaCircus architecture

MangaCircus is a self-hosted reader for manga, comics and webtoons. The reader owns accounts/progress and a separate collection catalog. The scraper owns the persistent queue and chapter publication, and its built-in stall watchdog requeues a stuck job once at a checkpoint. Separate supervisors isolate service failures. All content remains a plain folder tree; there is no database server or embedded autonomous LLM.

Related documents:

• library-format.md: the folder format.
• import.md: how the folder tree is indexed.
• scraper-guide.md: rules for tools that write the library.
• scraper-service.md: managed queue, private API, and adapter contract.
• collections.md: relations, groups, curator UI and agent research.
• resilience.md: supervision, recovery limits and incidents.
• compose-deployment.md: local/VM deployment and release workflow.
• settings-navigation.md: User/Admin workspace, scraper tabs and deep links.
• peering.md: optional private server-to-server catalog and transfer.
• discovery.md: reader-owned, opt-in deterministic discovery feed and its limits.
• source-modules.md: adding and maintaining first-party hosters inside the single scraper process.
• backup.md: built-in scheduled backup of data/ and the restore procedure.

1. Principles

Principle	Consequence
The file system is the source of truth for content	No import step, no copies, no content database. The library can be mounted read-only. Anything that writes files can feed it
As dumb as possible	No dependencies (package.json has none), no framework, no build step, no transpiler. Plain ES2019+ JavaScript in the browser, plain Node on the server
Small independent services	The reader stays dependency-free; the optional scraper daemon can stop or restart without taking reading offline
Readable without the app	The folder format works in any picture viewer. The app adds progress, layout and convenience, not lock-in

2. System context

                ┌──────────────┐    writes folders (staged + renamed)
  scraper / ───▶│   library/   │◀──────────────────────────────────┐
  person        │ (read-only   │                                   │
                │  for the app)│                                   │
                └──────┬───────┘                                   │
                       │ readdir / stat / small reads              │
                ┌──────▼─────────────────────────────┐             │
  browser ─────▶│ node server.js                      │   webhook   │
  (desktop,     │  ├ library.js   scanner → index     │────────────▶ ntfy / Gotify / …
   phone, PWA)  │  ├ HTTP: /  /lib/…  /api/…          │  (optional)
                │  └ data/db.json  data/scan-cache.json│
                └─────────────────────────────────────┘

3. Components

Runtime topology (2026-10-22): native tools/start-all.js launches reader and scraper independently. In Compose, each container runs runtime/launch.js <service> and exactly one service child. A failure does not stop unrelated services. Exactly one reader and queue-owning scraper may write each shared state tree; do not scale replicas. The catalog and relation store is reader-owned data/catalog.json. No service has a Docker socket or arbitrary code-repair authority.

The admin browser never talks to the scraper daemon directly. Admin-only reader routes proxy commands to the daemon's private API, so its shared token stays on the server. The daemon publishes complete folders atomically and increments a durable revision; the reader polls that revision and refreshes changed manga after publication. The reader restores a disposable full-index snapshot at startup so browsing need not wait for a filesystem walk. Shallow directory stamps identify changed manga; a daily full reconciliation catches in-place manual edits. The last full-scan time survives restarts, so restarting does not itself force another complete walk.

Path	Lines	Responsibility
server.js	~350	HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks
library.js	~170	Folder scanner, name parsing, image-size reader, scan cache
backup.js	~330	Built-in DATA backup: queue pause/resume, tar.gz writer and verifier, schedule, retention
public/common.js	~85	Browser helpers: h() element builder, api() fetch wrapper, top navigation with the notification bell
public/index.html + library.js	~60	Library overview (cover grid, filter, sort)
public/manga.html + manga.js	~110	Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle
public/read.html + reader.js	~470	The reader (§7.2)
public/settings.html + settings.js + settings-nav.js	—	User/Admin navigation, personal preferences, lazy admin sections and scraper tabs
public/login.html + login.js	~50	Sign-in, first-run admin creation, optional sign-up
public/app.css	~150	All styles (dark theme, responsive)
tools/png.js, tools/make-dummy.js	~310	Demo library generator: a tiny PNG encoder with a bitmap font, no image libraries
Dockerfile, docker-compose.yml		node:24-alpine, non-root, /library + /data

4. Runtime model

• Reader process: one event loop. All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests.
• Optional scraper process: one queue worker, one remote request at a time, durable checkpoints, and a private control API.
• In-memory state:
  • lib is the current library index, replaced as a whole after each scan.
  • db is all user data, loaded from db.json at start.
• Timers:
  • a saved-index restore and shallow check at start (full scan when no snapshot),
  • a shallow check every SCAN_INTERVAL minutes, full reconciliation every FULL_SCAN_INTERVAL minutes,
  • a debounced db.json write, 500 ms after the last change.
• Shutdown: SIGINT / SIGTERM flush db.json; the daemon cooperatively stops at a page boundary and keeps its archive checkpoint.
• Durability: every file write goes to *.tmp first and is then renamed over the target, so a file is never half-written. A hard kill loses at most the changes of the last ~0.5 s. A corrupt db.json makes the server refuse to start rather than start empty and overwrite it.

5. Data model

5.1 Content: the library index

The index is built by library.js from the folder tree (see import.md §2.6). It is never written anywhere, except for the scan cache.

5.2 Identity

Every reference from user data to content uses ids derived from names on disk:

Entity	Id	Example	Stable under
Manga	manga folder name	Moon Garden	JSON edits, anything inside the folder
Book	normalised book number (x prefix for extras)	1, x1	title changes
Chapter	<book key>/<chapter key>	1/2.5	title changes, adding/removing versions
Version	chapter folder name	c0002 Roots [MangaDex]	nothing: renaming the folder creates a new version

This keeps ids human-readable and derivable without a database. The price: renaming a manga folder, renumbering a chapter or moving it to another book creates a new entity (see library-format.md → Renaming).

5.3 User data: data/db.json

{
  "users": {
    "admin": {
      "name": "admin", "admin": true, "created": 1790796794000,
      "hash": "<salt hex>:<scrypt hex>",
      "settings": {
        "startMode": "last",            // "last" | "flow" | "page" | "book"
        "lastMode": "book",             // the user's intended mode; changed only by clicking a mode
        "adapt": true,                  // use a fitting mode for incompatible content (webtoon → flow)
        "skipExtras": false, "skipCovers": false,
        "flowPageGap": true,            // show the 8 px separator between pages in Flow
        "readingClock": false,           // opt in to credible per-chapter active-time tracking
        "readingStory": true,            // show private retrospective summaries when history exists
        "zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 },   // page zoom per mode
        "webhook": ""
      }
    }
  },
  "sessions":  { "<sha256 of token>": { "u": "admin", "t": 1790796794000 } },
  "progress":  { "admin": { "Moon Garden": { "1/2": {
                  "v": "c0002 Roots [MangaDex]",  // version read
                  "p": 4, "n": 14,                // page index in that version (all pages), page count
                  "t": 1790796800000,             // last activity
                  "done": false,                  // sticky once true (reset via "mark unread")
                  "shift": []                     // book mode: run starts shown alone
               } } } },
  "reading":   { "admin": { "Moon Garden": { "1/2": {
                  "schema": 1, "totalActiveMs": 412000, "loadingMs": 7400,
                  "sessions": 3, "completedPasses": 2,
                  "inputMs": { "mouse": 12000, "touch": 380000, "keyboard": 20000 },
                  "currentPass": { "version": "c0002 Roots [Official]", "seenRanges": [[0, 13]], "activeMs": 151000 }
                } } } },
  "follows":   { "admin": { "Moon Garden": true } },
  "notes":     { "admin": [ { "id": "a1b2c3d4e5f6", "t": 1790797300000, "m": "Moon Garden", "c": "2/6",
                              "text": "Moon Garden · Ch. 6: Frost", "read": false } ] },
  "votes":     { "Moon Garden": { "1/2": { "c0002 Roots [Official]": { "admin": 1, "reader": -1 } } } },
  "known":     { "Moon Garden": ["1/1", "1/2", "1/2.5", "..."] }   // every chapter id ever seen (append-only)
}

• data/scan-cache.json holds { "<Manga>/<Book>/<Chapter>": { m: <folder mtimeMs>, p: [[file, w, h], …] } }. It is disposable.
• data/library-index.json holds a disposable, versioned index snapshot and shallow directory stamps. The library folders remain authoritative; a missing or invalid snapshot triggers a full scan. Rollback-compatible scan-cache.json is retained.
• data/backup.json holds the built-in backup schedule, retention and recent results.
• Backups: the reader's built-in backup (backup.md) archives the whole data/ tree into BACKUP on a schedule, with db.json taken from memory and the queue paused around the copy. A plain copy of data/db.json also works: atomic writes make it consistent at any time.

6. Server

6.1 Request routing

Path	Handling
/api/*	routes["<METHOD> <path>"]. Handlers get { u, b, q, req, res } and return an object, which is sent as JSON. Throwing HttpError sends { error } with that status
/lib/*	Image files from LIBRARY. Requires a session, allows image extensions only, and the path must resolve inside LIBRARY. ETag + private, max-age=86400
everything else	Static files from public/ (/ → index.html). ETag + no-cache

Pages are static. Each page's script calls the API and redirects to login.html?next=… on 401. All URLs are relative, so the app also works under a sub-path behind a reverse proxy.

6.2 API

All requests and responses are JSON. POSTs require Content-Type: application/json.

Method & path	Auth	Purpose
GET /api/auth	public	{ firstRun, signup } for the login page
POST /api/signup {name, password}	public	First account (becomes admin), or anyone when SIGNUP=open
POST /api/login {name, password}	public	Starts a session (cookie sid)
POST /api/logout	user	Ends the session
GET /api/me	user	Name, admin flag, settings, unread count
POST /api/settings {…partial}	user	Validated update of the settings keys above
POST /api/password {old, new}	user	Change password
GET /api/library	user	Manga list with covers, counts, the user's read count, unread notifications, last read, and optional personal Reading Story aggregates
GET /api/manga?m=	user	Manga details, books, chapters with ranked versions, progress, optional chapter-time summaries, and continue target
GET /api/chapter?m=&c=[&v=]	user	Pages (u, w, h, t, r), direction, versions, prev/next, progress, optional Active Reading Clock summary, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read
POST /api/progress {m, c, v, p, n, done, shift}	user	Save the reading position
POST /api/reading/pulse {m, c, v, session, seq, …deltas}	user	When opted in, idempotently merge a clamped active-time pulse, verified page coverage and input totals under a single-reader lease
GET /api/reading/export	user	Export the user's private derived reading-time summary without raw input or page-exposure diagnostics
POST /api/reading/delete {m} or {all:true,password}	user	Delete one manga's reading history, or all history after password confirmation, without changing normal reading progress
POST /api/unread {m, c}	user	Forget progress for a chapter
POST /api/vote {m, c, v, value: 1|-1|0}	user	Vote a version; returns the re-ranked versions
POST /api/follow {m, on}	user	Follow a manga for notifications
GET /api/notifications · POST /api/notifications/read {id | all}	user	Notification list and read state
GET /api/admin	admin	Users, last scan result, library path
POST /api/admin/rescan {full}	admin	Scan now, and wait for the result
POST /api/admin/user {name, password, admin}	admin	Create a user or reset a password
POST /api/admin/user/role {name, admin}	admin	Grant or revoke admin rights; keeps the password and sessions. Revoking your own rights is refused, so one admin always remains
POST /api/admin/user/delete {name}	admin	Delete a user and their data
GET /api/admin/backup · POST /api/admin/backup {command:"settings"|"run"}	admin	Built-in backup status, schedule/retention, "back up now" (backup.md)

6.3 Authentication and security

Concern	Measure
Passwords	scrypt (32-byte key, 16-byte random salt), constant-time compare. Unknown users are checked against a dummy hash to keep timing similar
Sessions	256-bit random token in cookie sid (HttpOnly, SameSite=Lax, 1 year, Secure with COOKIE_SECURE=1). Only its SHA-256 is stored. Pruned at start after a year or when the user is gone
CSRF	POSTs must be application/json (plain cross-site forms can't send that) and must not carry Sec-Fetch-Site: cross-site
Path traversal	Paths are decoded, resolved and checked against the root directory. /lib/ serves only image extensions, never the JSON files
XSS	The UI builds DOM nodes and inserts all data as text (h() helper). The only markup inserted is the app's own constant SVG icons. External links are rendered only for http(s) URLs
Request size	JSON bodies are limited to 1 MB
Authorisation	admin flag checked per admin route. Users only ever see their own progress, follows and notifications. Votes are shown as totals plus the user's own
Not included	Login rate limiting, a CSP header, TLS. Run behind a reverse proxy if exposed. Webhook URLs are user-chosen, so any user can make the server send a POST to an address they choose. Fine for a household; consider it before SIGNUP=open

6.4 Version ranking

At request time: the sum of all users' votes, then chapter.json.priority, then folder order. The top version is what "open chapter" shows. An explicit v= (version picker, "Continue" of a manually chosen version) overrides it.

6.5 Notifications

In-app notifications and optional webhooks are produced by announce() after each scan, from the append-only known list. See import.md §5.

7. Frontend

7.1 Pages

Every page is a static HTML file plus one script, loaded with defer. All pages share common.js and app.css. There is no client-side router, no state library and no component system. Each script fetches what it needs, builds DOM with h(), and re-renders small parts on change.

A web app manifest (display: standalone) makes "Add to home screen" open the app without browser bars on phones.

7.2 The reader (reader.js)

Inputs:

• the chapter payload (pages with pixel sizes, tags and rotation; direction; versions; neighbours; progress),
• the user's settings.

Page model. Each page's size is known before it loads, so the reader computes the whole layout itself and positions every page absolutely:

all pages ──filter (skipExtras / skipCovers)──▶ list
list ──mode──▶ build(): create page elements (flow: all, lazy · page/book: current + preload next)
          └──▶ layout(): compute a box {x, y, w, h} per element from the viewport, zoom and page aspect
                   └─▶ place(): apply the boxes; rotated images are sized swapped and turned with CSS

layout() repeats up to three times until the viewport size is stable, because scrollbars that appear or disappear change the viewport.

Layout rules (z = page zoom of the current mode):

Mode	Box sizes
Flow (paged content)	W = min(vw, vh · refAspect) · z: a normal single page fits the screen at 100 %. Wide pages keep the single page height, capped at max(vw, W). 8 px gap
Flow (webtoon, zoom key strip)	W = min(vw, median image width) · z, no gaps, so slices join seamlessly
Page	The page fits the viewport (contain) · z, centred. The canvas grows beyond the viewport when zoomed in
Book	The pages of the spread share one height H = min(vh, vw / Σaspect) · z. Order is right-to-left for rtl

Anchoring. Before any re-layout (zoom, resize, pinch), anchor(x, y) records which page is under that point and where inside it, as a fraction. restore() scrolls so the same point is under the finger or cursor again. That is why zooming feels pinned to the cursor and a resize keeps the reading position.

Book pairing:

• Covers and wide pages stand alone.
• The single pages between them form runs, and each run pairs from its start.
• "⇄ Shift" toggles a run to start with a lone page. It is stored as the run's first page index in progress.shift.
• One broken spot can therefore never shift the pairing of the whole chapter.

Mode resolution:

intended   = startMode == "last" ? lastMode : startMode
compatible = direction == "ttb" ? [flow] : [flow, page, book]
mode       = adapt && intended ∉ compatible ? compatible[0] : intended

lastMode is written only when the user clicks a mode, never on an automatic switch.

Input:

Input	Action
Click or tap outside the pages	Toggle the bars (the end card uses the whole screen as zones instead)
Click or tap on the page(s), outer 35 % of the visible page area (paged)	Previous/next, following the reading direction (rtl: left = next)
Click or tap on a page, top/bottom 35 % of the screen (flow)	Scroll to the previous/next page top, at most 90 % of a screen (85 % steps for webtoons). At the very bottom → next chapter
Click or tap on a page, middle band	Toggle the bars
Horizontal swipe ≥ 50 px within 800 ms (paged, not zoomed in)	Previous/next
Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch	Continuous page zoom 20–600 %, snapping to 100 % within ±1 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages
Mouse drag (moved ≥ 6 px)	Pan; the following click is swallowed. Hand cursor when the content is larger than the screen
Keys	Arrows follow the direction; Space/PageDown/PageUp (paged); + and - zoom in 5-percentage-point steps, 0 fits at 100 %; m/Esc bars; f fullscreen

Bars:

• Shown on open.
• Hidden on the first page turn or after 100 px of scrolling. The first automatic hide (per device, once for Flow and once for Page/Book) shows a hint on how to get them back.
• Never shown by scrolling, only by a tap beside the pages or in the middle of a page (or m). Showing them stops a running smooth scroll, so it can't hide them again right away.
• The top bar has the chapter picker (every chapter, grouped by book, ✓ = read).

Loading:

• Flow mode loads images through an IntersectionObserver on the scroll container, 1.5 screens ahead.
• Page and Book mode keep a small pool (12) of Image objects:
  • Once the pages on screen have loaded, the next 3 pages (Book: next 2 spreads) and the previous one are fetched at low priority and decoded (img.decode()).
  • A page turn then inserts those same, already decoded elements (decoding="sync"), so the old and new page swap in one frame. A fresh <img> would be painted empty while it decodes, which is what caused the flicker on the first view of each page.
• A page that is still loading shows a spinner (only after 150 ms, so fast loads never flash it). A page that fails shows a message.
• A page with unknown size is laid out at 1000×1414 and re-laid out when it loads.

Progress:

• Saved 700 ms after the position settles, and on pagehide (a fetch with keepalive).
• Stores the page index in the full page list, so skipping extras doesn't corrupt it.
• On open, the saved page is used for the same version, and mapped proportionally for another version. A finished chapter reopens at the start.

Active Reading Clock (opt-in):

• Runs only while the reader is visible and focused, a sufficiently exposed page is decoded, a server lease is held, and accepted reader input occurred in the last 50 seconds.
• Backgrounding or blurring pauses immediately and requires fresh input to resume. A delayed callback can never add time beyond the 50-second input deadline.
• A page is verified after at least 35% meaningful exposure for 200 ms cumulatively. Exposure is closed and evaluated when the page leaves the viewport, so a fast but plausible webtoon scroll is not lost between samples. A five-pages-per-second token bucket rejects implausible coverage without discarding otherwise credible active time.
• BroadcastChannel coordinates tabs on one browser. The authenticated API adds a 15-second per-user lease for other devices, and releases it when the tab pauses or leaves.
• Pulses are sent at most every 10 seconds and on pause/page exit. The server validates the manga/chapter/version and page indices, de-duplicates (session, seq), clamps time to server-observed intervals and stores compact page ranges separately from ordinary progress.
• The bottom toolbar chip shows active time and the current state; its panel adds pass time, verified pages, predominant input and loading wait.

Zoom persistence: per mode key (flow, page, book, strip), saved 800 ms after the last change.

8. Deployment

Current Windows deployment

The observed local installation uses the Windows Scheduled Task \MangaCircus Supervisor. At logon it runs tools/run-supervisor.ps1, which appends logs under data/scraper/ and starts one tools/start-all.js supervisor. That parent owns exactly one reader (server.js, port 8080) and one scraper daemon (scraper/daemon.js, private port 8091). Starting npm run start:all manually at the same time would create conflicting writers and is not supported.

The scraper's authoritative data/scraper/queue.json is schema version 3. It supports legacy archive checkpoints and paused conversion of completed legacy work while adding source-aware lanes, scoped queue jobs, incremental chapter publication, and nightly catalog discovery. See local-deployment.md for operational checks and restart/recovery, and scraper-service.md for the queue/API contract.

Docker or manual deployment

docker compose up -d --build

• Images/services: the same node:24-alpine image runs a reader service and a scraper service. The reader health check targets /api/auth.
• Volumes:
  • /library: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000.
  • /data: reader state must be writable; the scraper also needs its writable /data/scraper state.
  • /backup: reader only, writable target of the built-in backup (BACKUP_PATH; off when unset).
• Configuration: process wiring uses environment variables (PORT, LIBRARY, DATA, BACKUP, SIGNUP, SCAN_INTERVAL, FULL_SCAN_INTERVAL, COOKIE_SECURE, SCRAPER_URL, and SCRAPER_TOKEN). Queue pacing and automatic-update schedule are persisted in data/scraper/queue.json and edited through the authenticated admin UI.
• Without Docker: npm run start:all with Node ≥ 18, or supervise npm start and npm run start:scraper independently. On the current Windows machine that supervision is already owned by \MangaCircus Supervisor.
• Behind a reverse proxy: terminate TLS, set COOKIE_SECURE=1. A sub-path works.
• Upgrades: replace the code and restart. Reader state has no explicit migration; unknown fields in db.json are kept. The scraper normalizes old queue entries into schema version 3 on startup and preserves legacy job checkpoint behavior. Take a queue/checkpoint recovery copy before upgrading a live scraper.

9. Capacity and limits

Aspect	Behaviour
Library size	The index keeps a small object per page (file name, size, tags), in the order of a few hundred bytes. Hundreds of thousands of pages fit in a modest container. Unchanged chapters cost one stat + one small read per scan
First scan	Reads 32 KB per image once (≤ 256 KB, or the whole file for JPEGs with huge EXIF/ICC blocks), SCAN_CONCURRENCY (8) at a time. Later scans use the cache
User data	db.json is rewritten as a whole, at most about twice per second while people read. Comfortable up to a few MB (a household or small group, thousands of chapters read)
Concurrency	One process, one writer. No clustering or horizontal scaling by design
Images	Served as stored. No resizing or thumbnails, so large covers make the library page heavy

10. Design decisions

Decision	Alternatives	Why
Index the folder tree in place	Import into a DB / managed storage	Stays readable by any viewer and writable by any tool. Nothing to keep in sync
No dependencies, no framework, no build	Express, React, bundler	Minimal code, maximum longevity, runs anywhere Node runs
Built-in backup in the reader, queue paused, no container stop	Host job that stops the stack; backup in a separate service	Only the reader sees all of /data and the in-memory db.json; the scraper's pause API gives a consistent point without downtime
JSON file for user data	SQLite (node:sqlite is still experimental in Node 22/24), Postgres	Zero dependencies, trivial backup, enough for the target size. Easy to swap: all access goes through the db object
Page roles as file-name tags	Page lists in JSON	Visible in any file browser, survives copying, no second source of truth. Unknown tags are harmless
Image sizes read on the server	Measure in the browser after load	Layout before load: exact scroll restore, zoom anchoring, auto-detected double pages, no layout jumps
Absolute layout engine in the reader	CSS flow/flex layout	Deterministic geometry, needed for pointer-anchored zoom and exact position restore
Chapter id = book number + chapter number	Random ids stored in JSON	Human-readable and needs no writes into the library. Downside: moving a chapter between books changes its identity
Global votes, per-visit version choice	Per-user preferred version	"The better version is shown to everyone", while any user can still pick another one explicitly
Bars never reappear on scroll	Show on scroll-up (common pattern)	Explicit requirement: readers scroll up and down constantly

11. Known limitations and extension points

• Identity is name-based. An optional stable id in manga.json (with a migration of the user data keys) would make manga renames safe.
• No thumbnails. Would need an image library (e.g. sharp) or pre-generated cover files.
• Notifications: in-app plus webhook only. Web Push would need VAPID keys and a service worker.
• No login rate limiting and no CSP header. Both are small additions in server.js.
• In-place image overwrites are only detected by a full rescan.
• Reader layout remains mostly smoke/manual. Reading analytics has focused Node tests for its persisted merge rules, while DOM layout, authentication and mobile browser lifecycle behavior still need browser-level/manual verification.
• Adding content by API (instead of files) would be a new route plus the staging and renaming rules from import.md §7.