MangaCircus/docs/architecture.md

# MangaCircus architecture

MangaCircus is a self-hosted reader for manga, comics and webtoons. One Node.js process serves a handful of static pages and a small JSON API. The content is a folder tree on disk, and everything user-related lives in one JSON file.

Related documents:
- [library-format.md](library-format.md): the folder format.
- [import.md](import.md): how the folder tree is indexed.
- [scraper-guide.md](scraper-guide.md): rules for tools that write the library.

## 1. Principles

| Principle | Consequence |
|-----------|-------------|
| **The file system is the source of truth for content** | No import step, no copies, no content database. The library can be mounted read-only. Anything that writes files can feed it |
| **As dumb as possible** | No dependencies (`package.json` has none), no framework, no build step, no transpiler. Plain ES2019+ JavaScript in the browser, plain Node on the server |
| **One process, one data file** | In-memory library index, user data in `data/db.json`. Back up one file, run one container |
| **Readable without the app** | The folder format works in any picture viewer. The app adds progress, layout and convenience, not lock-in |

## 2. System context

```
                ┌──────────────┐    writes folders (staged + renamed)
  scraper / ───▶│   library/   │◀──────────────────────────────────┐
  person        │ (read-only   │                                   │
                │  for the app)│                                   │
                └──────┬───────┘                                   │
                       │ readdir / stat / small reads              │
                ┌──────▼─────────────────────────────┐             │
  browser ─────▶│ node server.js                      │   webhook   │
  (desktop,     │  ├ library.js   scanner → index     │────────────▶ ntfy / Gotify / …
   phone, PWA)  │  ├ HTTP: /  /lib/…  /api/…          │  (optional)
                │  └ data/db.json  data/scan-cache.json│
                └─────────────────────────────────────┘
```

## 3. Components

| Path | Lines | Responsibility |
|------|------:|----------------|
| `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks |
| `library.js` | ~170 | Folder scanner, name parsing, image-size reader, scan cache |
| `public/common.js` | ~85 | Browser helpers: `h()` element builder, `api()` fetch wrapper, top navigation with the notification bell |
| `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) |
| `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle |
| `public/read.html` + `reader.js` | ~470 | The reader (§7.2) |
| `public/settings.html` + `settings.js` | ~115 | Reading preferences, webhook, password, admin (rescan, users) |
| `public/login.html` + `login.js` | ~50 | Sign-in, first-run admin creation, optional sign-up |
| `public/app.css` | ~150 | All styles (dark theme, responsive) |
| `tools/png.js`, `tools/make-dummy.js` | ~310 | Demo library generator: a tiny PNG encoder with a bitmap font, no image libraries |
| `Dockerfile`, `docker-compose.yml` | | `node:24-alpine`, non-root, `/library` + `/data` |

## 4. Runtime model

- **One process, event loop only.** All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests.
- **In-memory state:**
  - `lib` is the current library index, replaced as a whole after each scan.
  - `db` is all user data, loaded from `db.json` at start.
- **Timers:**
  - a library scan at start and every `SCAN_INTERVAL` minutes,
  - a debounced `db.json` write, 500 ms after the last change.
- **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json` synchronously, then exit. That matters for `docker stop`.
- **Durability:** every file write goes to `*.tmp` first and is then renamed over the target, so a file is never half-written. A hard kill loses at most the changes of the last ~0.5 s. A corrupt `db.json` makes the server **refuse to start** rather than start empty and overwrite it.

## 5. Data model

### 5.1 Content: the library index

The index is built by `library.js` from the folder tree (see [import.md §2.6](import.md#26-the-result)). It is never written anywhere, except for the scan cache.

### 5.2 Identity

Every reference from user data to content uses ids derived from names on disk:

| Entity | Id | Example | Stable under |
|--------|----|---------|--------------|
| Manga | manga folder name | `Moon Garden` | JSON edits, anything inside the folder |
| Book | normalised book number (`x` prefix for extras) | `1`, `x1` | title changes |
| Chapter | `<book key>/<chapter key>` | `1/2.5` | title changes, adding/removing versions |
| Version | chapter folder name | `c0002 Roots [MangaDex]` | nothing: renaming the folder creates a new version |

This keeps ids human-readable and derivable without a database. The price: renaming a manga folder, renumbering a chapter or moving it to another book creates a new entity (see [library-format.md → Renaming](library-format.md#renaming-what-survives)).

### 5.3 User data: `data/db.json`

```jsonc
{
  "users": {
    "admin": {
      "name": "admin", "admin": true, "created": 1790796794000,
      "hash": "<salt hex>:<scrypt hex>",
      "settings": {
        "startMode": "last",            // "last" | "flow" | "page" | "book"
        "lastMode": "book",             // the user's intended mode; changed only by clicking a mode
        "adapt": true,                  // use a fitting mode for incompatible content (webtoon → flow)
        "skipExtras": false, "skipCovers": false,
        "zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 },   // page zoom per mode
        "webhook": ""
      }
    }
  },
  "sessions":  { "<sha256 of token>": { "u": "admin", "t": 1790796794000 } },
  "progress":  { "admin": { "Moon Garden": { "1/2": {
                  "v": "c0002 Roots [MangaDex]",  // version read
                  "p": 4, "n": 14,                // page index in that version (all pages), page count
                  "t": 1790796800000,             // last activity
                  "done": false,                  // sticky once true (reset via "mark unread")
                  "shift": [],                    // book mode: run starts shown alone
                  "top": 4                        // furthest page read in this version (Senpai horizon)
               } } } },
  "follows":   { "admin": { "Moon Garden": true } },
  "notes":     { "admin": [ { "id": "a1b2c3d4e5f6", "t": 1790797300000, "m": "Moon Garden", "c": "2/6",
                              "text": "Moon Garden · Ch. 6: Frost", "read": false } ] },
  "votes":     { "Moon Garden": { "1/2": { "c0002 Roots [Official]": { "admin": 1, "reader": -1 } } } },
  "known":     { "Moon Garden": ["1/1", "1/2", "1/2.5", "..."] }   // every chapter id ever seen (append-only)
}
```

- `data/scan-cache.json` holds `{ "<Manga>/<Book>/<Chapter>": { m: <folder mtimeMs>, p: [[file, w, h], …] } }`. It is disposable.
- **Backups:** copy `data/db.json`. The atomic writes make a copy at any time consistent.

## 6. Server

### 6.1 Request routing

| Path | Handling |
|------|----------|
| `/api/*` | `routes["<METHOD> <path>"]`. Handlers get `{ u, b, q, req, res }` and return an object, which is sent as JSON. Throwing `HttpError` sends `{ error }` with that status |
| `/lib/*` | Image files from `LIBRARY`. Requires a session, allows image extensions only, and the path must resolve inside `LIBRARY`. `ETag` + `private, max-age=86400` |
| everything else | Static files from `public/` (`/` → `index.html`). `ETag` + `no-cache` |

Pages are static. Each page's script calls the API and redirects to `login.html?next=…` on `401`. All URLs are relative, so the app also works under a sub-path behind a reverse proxy.

### 6.2 API

All requests and responses are JSON. POSTs require `Content-Type: application/json`.

| Method & path | Auth | Purpose |
|---------------|------|---------|
| `GET /api/auth` | public | `{ firstRun, signup }` for the login page |
| `POST /api/signup` `{name, password}` | public | First account (becomes admin), or anyone when `SIGNUP=open` |
| `POST /api/login` `{name, password}` | public | Starts a session (cookie `sid`) |
| `POST /api/logout` | user | Ends the session |
| `GET /api/me` | user | Name, admin flag, settings, unread count |
| `POST /api/settings` `{…partial}` | user | Validated update of the settings keys above |
| `POST /api/password` `{old, new}` | user | Change password |
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read |
| `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions and the user's progress, `continue` target |
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
| `POST /api/progress` `{m, c, v, p, n, done, shift}` | user | Save the reading position |
| `POST /api/unread` `{m, c}` | user | Forget progress for a chapter |
| `POST /api/vote` `{m, c, v, value: 1\|-1\|0}` | user | Vote a version; returns the re-ranked versions |
| `POST /api/follow` `{m, on}` | user | Follow a manga for notifications |
| `GET /api/notifications` · `POST /api/notifications/read` `{id \| all}` | user | Notification list and read state |
| `GET /api/admin` | admin | Users, last scan result, library path |
| `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result |
| `POST /api/admin/user` `{name, password, admin}` | admin | Create a user or reset a password |
| `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data |
| `GET /api/senpai/v1/{locate,xray,characters/<ref>[/timeline],portraits/<ref>}?m=…` | user | Optional proxy to Senpai (`SENPAI_URL`): adds `Authorization: Bearer SENPAI_TOKEN`, `X-Senpai-User` and `X-Senpai-Horizon` (furthest chapter with progress in this manga, its version and `max(p, top)`, percent-encoded). 404 when `SENPAI_URL` is unset; other paths are not forwarded |

### 6.3 Authentication and security

| Concern | Measure |
|---------|---------|
| Passwords | `scrypt` (32-byte key, 16-byte random salt), constant-time compare. Unknown users are checked against a dummy hash to keep timing similar |
| Sessions | 256-bit random token in cookie `sid` (`HttpOnly`, `SameSite=Lax`, 1 year, `Secure` with `COOKIE_SECURE=1`). Only its SHA-256 is stored. Pruned at start after a year or when the user is gone |
| CSRF | POSTs must be `application/json` (plain cross-site forms can't send that) and must not carry `Sec-Fetch-Site: cross-site` |
| Path traversal | Paths are decoded, resolved and checked against the root directory. `/lib/` serves only image extensions, never the JSON files |
| XSS | The UI builds DOM nodes and inserts all data as text (`h()` helper). The only markup inserted is the app's own constant SVG icons. External links are rendered only for `http(s)` URLs |
| Request size | JSON bodies are limited to 1 MB |
| Authorisation | `admin` flag checked per admin route. Users only ever see their own progress, follows and notifications. Votes are shown as totals plus the user's own |
| Not included | Login rate limiting, a CSP header, TLS. Run behind a reverse proxy if exposed. Webhook URLs are user-chosen, so any user can make the server send a POST to an address they choose. Fine for a household; consider it before `SIGNUP=open` |

### 6.4 Version ranking

At request time: the sum of all users' votes, then `chapter.json.priority`, then folder order. The top version is what "open chapter" shows. An explicit `v=` (version picker, "Continue" of a manually chosen version) overrides it.

### 6.5 Notifications

In-app notifications and optional webhooks are produced by `announce()` after each scan, from the append-only `known` list. See [import.md §5](import.md#5-change-detection-and-notifications).

## 7. Frontend

### 7.1 Pages

Every page is a static HTML file plus one script, loaded with `defer`. All pages share `common.js` and `app.css`. There is no client-side router, no state library and no component system. Each script fetches what it needs, builds DOM with `h()`, and re-renders small parts on change.

A web app manifest (`display: standalone`) makes "Add to home screen" open the app without browser bars on phones.

### 7.2 The reader (`reader.js`)

**Inputs:**
- the chapter payload (pages with pixel sizes, tags and rotation; direction; versions; neighbours; progress),
- the user's settings.

**Page model.** Each page's size is known before it loads, so the reader computes the whole layout itself and positions every page absolutely:

```
all pages ──filter (skipExtras / skipCovers)──▶ list
list ──mode──▶ build(): create page elements (flow: all, lazy · page/book: current + preload next)
          └──▶ layout(): compute a box {x, y, w, h} per element from the viewport, zoom and page aspect
                   └─▶ place(): apply the boxes; rotated images are sized swapped and turned with CSS
```

`layout()` repeats up to three times until the viewport size is stable, because scrollbars that appear or disappear change the viewport.

**Layout rules** (z = page zoom of the current mode):

| Mode | Box sizes |
|------|-----------|
| Flow (paged content) | `W = min(vw, vh · refAspect) · z`: a normal single page fits the screen at 100 %. Wide pages keep the single page height, capped at `max(vw, W)`. 8 px gap |
| Flow (webtoon, zoom key `strip`) | `W = min(vw, median image width) · z`, no gaps, so slices join seamlessly |
| Page | The page fits the viewport (`contain`) · z, centred. The canvas grows beyond the viewport when zoomed in |
| Book | The pages of the spread share one height `H = min(vh, vw / Σaspect) · z`. Order is right-to-left for `rtl` |

**Anchoring.** Before any re-layout (zoom, resize, pinch), `anchor(x, y)` records which page is under that point and where inside it, as a fraction. `restore()` scrolls so the same point is under the finger or cursor again. That is why zooming feels pinned to the cursor and a resize keeps the reading position.

**Book pairing:**
- Covers and wide pages stand alone.
- The single pages between them form *runs*, and each run pairs from its start.
- "⇄ Shift" toggles a run to start with a lone page. It is stored as the run's first page index in `progress.shift`.
- One broken spot can therefore never shift the pairing of the whole chapter.

**Mode resolution:**

```
intended   = startMode == "last" ? lastMode : startMode
compatible = direction == "ttb" ? [flow] : [flow, page, book]
mode       = adapt && intended ∉ compatible ? compatible[0] : intended
```

`lastMode` is written only when the user clicks a mode, never on an automatic switch.

**Input:**

| Input | Action |
|-------|--------|
| Click or tap outside the pages | Toggle the bars (the end card uses the whole screen as zones instead) |
| Click or tap on the page(s), outer 35 % of the visible page area (paged) | Previous/next, following the reading direction (`rtl`: left = next) |
| Click or tap on a page, top/bottom 35 % of the screen (flow) | Scroll to the previous/next page top, at most 90 % of a screen (85 % steps for webtoons). At the very bottom → next chapter |
| Click or tap on a page, middle band | Toggle the bars |
| Horizontal swipe ≥ 50 px within 800 ms (paged, not zoomed in) | Previous/next |
| Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Page zoom 20–600 %, snapping to 100 % within ±3 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages |
| Mouse drag (moved ≥ 6 px) | Pan; the following click is swallowed. Hand cursor when the content is larger than the screen |
| Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` `-` `0` zoom; `m`/Esc bars; `f` fullscreen; `x` Who's here (Esc closes it) |

**Bars:**
- Shown on open.
- Hidden on the first page turn or after 100 px of scrolling. The first automatic hide (per device, once for Flow and once for Page/Book) shows a hint on how to get them back.
- **Never shown by scrolling**, only by a tap beside the pages or in the middle of a page (or `m`). Showing them stops a running smooth scroll, so it can't hide them again right away.
- The top bar has the chapter picker (every chapter, grouped by book, ✓ = read).

**Loading:**
- Flow mode loads images through an `IntersectionObserver` on the scroll container, 1.5 screens ahead.
- Page and Book mode keep a small pool (12) of `Image` objects:
  - Once the pages on screen have loaded, the next 3 pages (Book: next 2 spreads) and the previous one are fetched at low priority and **decoded** (`img.decode()`).
  - A page turn then inserts those same, already decoded elements (`decoding="sync"`), so the old and new page swap in one frame. A fresh `<img>` would be painted empty while it decodes, which is what caused the flicker on the first view of each page.
- A page that is still loading shows a spinner (only after 150 ms, so fast loads never flash it). A page that fails shows a message.
- A page with unknown size is laid out at 1000×1414 and re-laid out when it loads.

**Progress:**
- Saved 700 ms after the position settles, and on `pagehide` (a `fetch` with `keepalive`).
- Stores the page index in the full page list, so skipping extras doesn't corrupt it.
- On open, the saved page is used for the same version, and mapped proportionally for another version. A finished chapter reopens at the start.

**Zoom persistence:** per mode key (`flow`, `page`, `book`, `strip`), saved 800 ms after the last change.

## 8. Deployment

```bash
docker compose up -d --build
```

- **Image:** `node:24-alpine` plus the app files. It runs as user `node` (uid 1000) and has a health check against `/api/auth`.
- **Volumes:**
  - `/library`: read-only is enough. Files must be readable by uid 1000.
  - `/data`: must be writable.
- **Configuration:** only environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `COOKIE_SECURE`, optional `SENPAI_URL` + `SENPAI_TOKEN`).
- **With Senpai:** the Senpai repository's `docker-compose.circus.yml` builds this branch next to Senpai and sets `SENPAI_URL=http://senpai-api:8780`; Senpai is reached only through the proxy, never by the browser.
- **Without Docker:** `node server.js` with Node ≥ 18.
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
- **Upgrades:** replace the code and restart. There are no migrations: unknown fields in `db.json` are kept, and missing settings fall back to defaults.

## 9. Capacity and limits

| Aspect | Behaviour |
|--------|-----------|
| Library size | The index keeps a small object per page (file name, size, tags), in the order of a few hundred bytes. Hundreds of thousands of pages fit in a modest container. Unchanged chapters cost one `stat` + one small read per scan |
| First scan | Reads ≤ 256 KB per image once. Later scans use the cache |
| User data | `db.json` is rewritten as a whole, at most about twice per second while people read. Comfortable up to a few MB (a household or small group, thousands of chapters read) |
| Concurrency | One process, one writer. No clustering or horizontal scaling by design |
| Images | Served as stored. No resizing or thumbnails, so large covers make the library page heavy |

## 10. Design decisions

| Decision | Alternatives | Why |
|----------|--------------|-----|
| Index the folder tree in place | Import into a DB / managed storage | Stays readable by any viewer and writable by any tool. Nothing to keep in sync |
| No dependencies, no framework, no build | Express, React, bundler | Minimal code, maximum longevity, runs anywhere Node runs |
| JSON file for user data | SQLite (`node:sqlite` is still experimental in Node 22/24), Postgres | Zero dependencies, trivial backup, enough for the target size. Easy to swap: all access goes through the `db` object |
| Page roles as file-name tags | Page lists in JSON | Visible in any file browser, survives copying, no second source of truth. Unknown tags are harmless |
| Image sizes read on the server | Measure in the browser after load | Layout before load: exact scroll restore, zoom anchoring, auto-detected double pages, no layout jumps |
| Absolute layout engine in the reader | CSS flow/flex layout | Deterministic geometry, needed for pointer-anchored zoom and exact position restore |
| Chapter id = book number + chapter number | Random ids stored in JSON | Human-readable and needs no writes into the library. Downside: moving a chapter between books changes its identity |
| Global votes, per-visit version choice | Per-user preferred version | "The better version is shown to everyone", while any user can still pick another one explicitly |
| Bars never reappear on scroll | Show on scroll-up (common pattern) | Explicit requirement: readers scroll up and down constantly |

## 11. Known limitations and extension points

- **Identity is name-based.** An optional stable id in `manga.json` (with a migration of the user data keys) would make manga renames safe.
- **No thumbnails.** Would need an image library (e.g. `sharp`) or pre-generated `cover` files.
- **Notifications:** in-app plus webhook only. Web Push would need VAPID keys and a service worker.
- **No login rate limiting and no CSP header.** Both are small additions in `server.js`.
- **In-place image overwrites** are only detected by a full rescan.
- **No automated test suite.** Verification so far was done with the demo library (`tools/make-dummy.js`), API smoke scripts and manual browser testing on desktop and mobile viewports. `parseKey` and `sizeFromBuffer` are pure functions, and `scanLibrary` only needs a folder. Together with the book pairing they are the natural first targets for unit tests.
- **Adding content by API** (instead of files) would be a new route plus the staging and renaming rules from [import.md §7](import.md#7-consistency-writing-into-a-live-library).