MangaCircus/docs/import.md

# How the import works

MangaCircus does not copy or convert anything. "Import" means **indexing the library folder in place**. The scanner (`library.js`) walks the folder tree described in [library-format.md](library-format.md) and builds an in-memory index. The server (`server.js`) swaps that index in, compares it with what it has seen before, and notifies followers of new chapters. The library itself is only ever read.

```
                 ┌───────────── triggers ─────────────┐
                 │ start-up · shallow interval · daily full │
                 │ admin "Rescan" · admin "Full rescan"│
                 └────────────────┬───────────────────┘
                                  ▼
 data/library-index.json ──▶ immediate in-memory index at start (if valid)
 library/ ──read──▶ shallow changed-manga check or full scan   library.js
                      ├─ scanManga      (per manga folder)
                      │    ├─ book folders  → parseKey, book.json, cover
                      │    └─ chapter folders → scanVersion
                      │          ├─ chapter.json (always read)
                      │          └─ page list + image sizes (only if the folder changed)
                      └─ returns { index, cache }
                                  ▼
                      announce(index)   new chapter ids → notifications, webhook   server.js
                                  ▼
                      lib = index       (atomic swap, requests see old or new, never half)
                                  ▼
                      data/scan-cache.json + data/library-index.json (atomic)
```

## 1. Triggers

| Trigger | When | Cache |
|---------|------|-------|
| Start-up | Restore the saved index immediately if its library mount is present, then check changed manga. Without a valid snapshot, scan the whole library; the page shows "Scanning…" until done | used |
| Interval | Shallow change check every `SCAN_INTERVAL` minutes (default 10, `0` = off); full reconciliation every `FULL_SCAN_INTERVAL` minutes (default 1440, `0` = off) | used |
| **Rescan** (Settings → Admin, or `POST /api/admin/rescan {}`) | On demand; the request returns when the scan is done | used |
| **Full rescan** (`POST /api/admin/rescan {"full":true}`) | On demand | ignored: every image header is read again |

The saved index records when its last full reconciliation finished. A warm restart
does a shallow check but does not reset the daily full-scan clock; if that scan is
overdue, the reader schedules it within a minute. Set `FULL_SCAN_INTERVAL=0` to
disable automatic full reconciliations, while retaining admin Rescan.

Scans are **single-flight**. A trigger during a scan queues one coalesced follow-up,
with a full reconciliation taking precedence over a shallow check. Publication
revisions trigger a shallow check; if no changed directory is visible, the reader
falls back to a full scan. In-place edits to `chapter.json` are found by the full
reconciliation or an admin Rescan, not by shallow parent-directory stamps.

## 2. Walking the tree

All directory listings go through one helper:
- It skips every entry whose name starts with `.`. That makes hidden folders the safe place for half-written chapters (see §7).
- It follows symbolic links to directories.
- It sorts names with a natural, case-insensitive collator (`c2` before `c10`).

### 2.1 Manga folders

For each folder in the library root:

1. List it and read `manga.json`.
   - A missing file becomes `{}`.
   - Broken JSON is logged and becomes `{}`.
   - A UTF-8 BOM is stripped.
2. `id` = folder name. `title` = `manga.json.title`, else the folder name. `direction` = `manga.json.direction`.
3. The cover is the first file matching `cover.(jpg|jpeg|png|webp|gif|avif|bmp)`.
4. Scan its book folders (2.2).
5. Drop the manga (with a log line) if it ends up without chapters. An exception inside one manga skips only that manga.

### 2.2 Book folders

For each sub-folder:

1. `parseKey(name)` with `^([a-z]*) ?(\d+(?:\.\d+)?)(?=$|[\s_\-[(])[\s_-]*(.*)$` (case-insensitive).
   - No match → ignored and logged.
   - Letters exactly `x` → extra book.
   - key = `x?` + `parseFloat(number)`.
   - A trailing `[…]` is split off as the source name. The rest is the title.
2. A **duplicate book key** within the manga is ignored and logged. The first folder in sort order wins.
3. Read `book.json`:
   - `title` overrides the folder title.
   - `extra: true` marks the book as extra.
   - `direction` is used for inheritance.
4. Cover: `cover.*` in the folder.
5. Scan its chapter folders (2.3). A book without chapters is dropped.

### 2.3 Chapter version folders

Each chapter folder is one **version**:

1. `parseKey(name)` → the chapter key; no match → ignored and logged.
2. `stat` the folder and **always** read `chapter.json`. It is small, and edits to it must show up even though they don't change the folder's mtime.
3. **Cache check.** The cache key is `Manga/Book/Chapter` (always `/`). If the cached folder mtime equals the current one, the cached page list is reused. Otherwise:
   - List the files and keep those matching `\.(jpe?g|png|webp|gif|avif|bmp)$`.
   - Read the pixel size of each (§3).
   - Store `{ m: mtimeMs, p: [[file, width, height], …] }`.
4. Per page, derive:
   - **tags**: the name without extension, lower-cased, split on `_`, without the first part (`0012_double_rot90.jpg` → `double, rot90`),
   - **rotation**: the first `rot90|rot180|rot270` tag.
5. Version fields:

| Field | Source |
|-------|--------|
| `id` | folder name |
| `source`, `sourceUrl` | `chapter.json.source` (`{name,url}` or a string), else the `[…]` part of the folder name |
| `added` | `Date.parse(chapter.json.added)`, else the folder mtime |
| `direction` | manual chapter → book → manga; else coherent tall-strip dimensions → `ttb`; else prior generated vertical series direction; else fallback `rtl`. Aliases: `vertical`, `webtoon` → `ttb`. Legacy plain values remain manual |

6. A version without images is dropped silently.

### 2.4 Grouping versions into chapters

Versions in the same book with the same chapter key form one chapter:

| Field | Rule |
|-------|------|
| `id` | `<book key>/<chapter key>`, e.g. `1/2.5`, `x1/x1` |
| `label` | `Ch. <n>` or `Extra <n>` |
| `extra` | book is extra, or letters `x`, or any version's `chapter.json` has `extra: true` |
| `title` | first non-empty `chapter.json.title` / folder title, in folder order |
| `versions` | in folder order. Ranking by votes happens later, per request (§6) |

The manga's `updated` is the newest `added` of all versions.

### 2.5 Sorting and fallbacks

- Books, and chapters within a book, sort by `(extra-by-x-prefix, number)`. `extra: true` from JSON does **not** move anything.
- The manga's reading order is the concatenation of its books' chapters.
- Mangas sort by title.
- Covers fall back as follows:
  - book → first `cover`-tagged page (else the first page) of the first chapter's first version,
  - manga → first book's cover.

### 2.6 The result

```js
index = {
  mangas: [{
    id, title, meta /* manga.json as is */, direction, cover /* relative path */, updated,
    books:    [{ key, num, x, extra, label, title, cover, chapters: [...] }],
    chapters: [{ id, book, key, num, x, extra, label, title,
                 versions: [{ id, rel, meta, source, sourceUrl, added, direction,
                              pages: [{ f, w, h, t: [tags], r }] }] }],
    chapterById: Map,
  }],
  byId: Map,
}
```

Everything is plain data, and the whole index lives in memory.

## 3. Image sizes

The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the **first 32 KB** and parses the header; only if the size isn't there yet does it read on to 256 KB (and, for JPEG, the whole file). Parsing a prefix either finds nothing or the same size as the whole file, so the small first read never changes a result.

Uncached chapter folders and their images are probed **`SCAN_CONCURRENCY` at a time** (default 8, 1–64; chapters of a book in parallel, then assembled in folder order). A first scan over a NAS is bound by per-file latency, not bandwidth: on a VM reading a Synology NAS over NFS the old sequential 256 KB probe took about 20 ms per image; see §3.1 for the measurements. `SCAN_CONCURRENCY=1` restores the sequential order.

| Format | Where the size is |
|--------|-------------------|
| PNG | IHDR (bytes 16–23) |
| GIF | logical screen (bytes 6–9) |
| BMP | DIB header (bytes 18–25) |
| WebP | `VP8 ` / `VP8L` / `VP8X` chunk |
| JPEG | first SOFn marker (the scan skips APPn/EXIF/ICC segments). If it isn't within 32 KB, then 256 KB, the whole file is read once |
| AVIF / HEIF | first `ispe` box |

### 3.1 Measured first-scan speed

A Debian VM (4 vCPU) reading a Synology NAS over NFS 4.0, page cache dropped before each run, production
running alongside. Each variant scanned a different manga of ~5,000 pages (the NAS caches what it has just read).

| Variant | ms per image (round 1 / round 2) |
|---|---|
| sequential, 256 KB (before) | 7.4 / 9.7 |
| 32 KB first read, `SCAN_CONCURRENCY=1` | 3.6 / 6.0 |
| 32 KB first read, `SCAN_CONCURRENCY=8` (default) | 2.9 / 2.1 |
| 32 KB first read, `SCAN_CONCURRENCY=16` | 2.4 / 3.3 |

Whole library without cache (56 manga, 10,766 chapters, 299,467 pages) with the default: **825 s (13.8 min)**,
against an estimated 37–48 min before (about 4 h while the NAS was busy with a large copy). The page lists and
sizes of all 10,775 cached chapter folders were identical to those of the previous code.

- If the size can't be read, it is stored as `0×0`. The reader then assumes 1000×1414 (2000×1414 for `double`) and corrects the layout when the image arrives.
- EXIF orientation is **not** applied. Use `rot…` tags instead.

## 4. The scan cache

- Stored in `DATA/scan-cache.json`. It is rewritten atomically after every successful scan and holds only the folders seen in that scan, so deleted chapters drop out.
- `DATA/library-index.json` also stores the complete rebuildable index and shallow
  manga/book directory stamps. It lets a restarted reader serve the dashboard and
  manga pages before the background filesystem check completes. Both files can be
  deleted safely; without the index the next startup walks the library, and without
  the size cache it also rereads image headers.
- Invalidation is **per chapter folder, by its mtime**. Adding, removing or renaming a file changes the folder mtime, so that folder is re-read on the next scan.
- **Blind spot:** overwriting an image *in place* (same name) usually doesn't change the folder's mtime, so the old size stays cached. Run a **Full rescan** after such edits, or write new files and rename them (§7).
- Deleting the cache file is harmless. The next scan is simply slower.

Measured on the demo library (5 manga, 22 chapters, 213 pages):
- ~100 ms without cache on a local disk,
- 1.4 s without cache and 0.11 s with cache through a Docker Desktop bind mount on Windows.

The cost per chapter version is:
- unchanged: one `stat` + one small JSON read,
- changed: + one directory listing + ≤ 256 KB read per image.

The scan uses asynchronous I/O one file at a time, so it never blocks request handling.

## 5. Change detection and notifications

`db.json` keeps `known[mangaId]`, the list of chapter ids ever seen for a manga. After every scan, `announce(index)` runs:

1. **First sight of a manga** (no `known` entry): record all its chapter ids and **don't notify**. A newly added series must not flood followers.
2. Otherwise, a chapter id not in `known` is new. It is appended; the list **only grows**, so a temporarily missing folder or an unmounted disk never causes a second round of notifications when it comes back.
3. For every user following the manga:
   - one in-app notification per new chapter (text `Title · Ch. 7: Thaw`; at most 200 kept per user),
   - if the user has a webhook URL, **one** `POST` per manga per scan, with a plain-text body (`New in Moon Garden: Ch. 7 Thaw, Ch. 8 Bloom`), header `Title: MangaCircus`, and a 10 s timeout. Failures are logged, not retried.
4. Opening a chapter marks its notifications as read.

What counts as "new" follows from the chapter id (`book key/chapter key`):

| Change | Notifies? |
|--------|-----------|
| A new chapter number appears | yes |
| Another **version** of an existing chapter appears | no |
| A chapter is moved to another book or renumbered | yes (new id) |
| A manga folder is renamed | no (it is a new manga) |
| A chapter folder is created before its images are complete | yes, immediately. Hence §7 |

## 6. From index to API

The index is read by the API at request time. Nothing is precomputed per user.

| Endpoint | Built from |
|----------|------------|
| `GET /api/library` | manga list + searchable metadata (`altTitles`, `authors`, `tags`, `status`, normalized first release), the user's progress, follows and unread notifications |
| `GET /api/manga?m=` | manga incl. `meta` (the raw `manga.json`), books, chapters with ranked versions and the user's progress, "Continue" target |
| `GET /api/chapter?m=&c=&v=` | pages of the chosen version (`v`, else the best ranked), direction, prev/next chapter (skipping extras if the user wants that), the user's progress |
| `GET /lib/<path>` | the image file itself. Logged-in users only, image extensions only, path must stay inside `LIBRARY`. ETag + `Cache-Control: private, max-age=86400` |

**Version ranking** happens per request:
1. the sum of all users' votes,
2. then `chapter.json.priority` (higher first),
3. then folder order.

Votes, progress and follows are keyed by manga id, chapter id and version id. They live in `db.json`, never in the library.

## 7. Consistency: writing into a live library

The scanner can run at any moment, including while a tool is still writing a chapter. Without care, it indexes a half-copied chapter, and followers get notified about it. The rules that keep this safe:

1. **Stage in a hidden folder, publish with one rename.**
   - Write the chapter into a sibling folder whose name starts with `.` (for example `v02 Summer/.incoming-c0007/`). The scanner never looks at it.
   - Rename it to its final name when it is complete.
   - A rename within one file system is atomic: the next scan sees all of the chapter or none of it.
2. **Replace versions the same way**: stage the new folder, move the old one out (into a hidden name), move the new one in, then delete the old one.
3. **Don't overwrite files in place** (see §4). If you must, trigger a Full rescan afterwards.

On the server side:
- The new index replaces the old one in a single assignment, so requests never see a half-built index.
- If the whole scan fails (for example, the library folder is missing), the previous index stays active and the error is shown under Settings → Admin.

The full rule set for tools is in [scraper-guide.md](scraper-guide.md).

## 8. Operations

**Trigger a rescan from a script:**

```bash
curl -s -c jar.txt -H 'Content-Type: application/json' -d '{"name":"admin","password":"…"}' http://localhost:8080/api/login
curl -s -b jar.txt -H 'Content-Type: application/json' -d '{}' http://localhost:8080/api/admin/rescan
```

The response contains `scan: { at, ms, mangas, chapters, pages, error }`. Use `{"full":true}` for a full rescan. POST requests must send `Content-Type: application/json` (part of the CSRF protection).

**Log lines** (stdout, `docker logs`):

| Message | Meaning / fix |
|---------|---------------|
| `[scan] 5 manga, 22 chapters, 213 pages in 103 ms` | Scan finished |
| `[scan] ignoring "M/B": book folders must be named <letters><number>[ title]…` | Folder name has no valid key: letters, optional space, number, then space/end. Rename it |
| `[scan] ignoring "M/B/C": chapter folders must be named …` | Same for a chapter (typical: `c0012a`, `Ch.12`) |
| `[scan] ignoring "M/B": another book folder already uses number N` | Two book folders with the same number. Merge them or renumber one |
| `[scan] ignoring broken …/chapter.json: …` | Invalid JSON. The folder still works with name-based defaults |
| `[scan] "M" has no chapters (expected <manga>/<book>/<chapter>/<images>)` | Usually a missing book level (images or chapters directly in the manga folder) |
| `[scan] skipped "M": …` | Unexpected I/O error in that manga (permissions?) |
| `[scan] failed: …` | The library root can't be read. The old index stays active |
| `[notify] webhook of <user> failed: …` | The webhook URL is unreachable or returned an error |

**Troubleshooting:**

| Symptom | Likely cause |
|---------|--------------|
| New chapter doesn't show up | Next interval hasn't come yet (rescan manually), hidden or invalid folder name, or no image files |
| Pages in the wrong order | Numbers not zero-padded, or a mix of padding widths |
| A page looks squashed or stretched after being replaced | Image overwritten in place with other dimensions. Run a Full rescan |
| Followers were notified about an empty or partial chapter | The chapter was written in place instead of staged and renamed |
| Reading progress "lost" | Manga folder renamed, or chapter moved or renumbered (new id) |