MangaParty: self-hosted manga/comic/webtoon reader
Zero-dependency Node server + plain HTML/JS frontend. - Library is a folder tree (Manga/Book/Chapter/pages), indexed in place; page roles (cover, double, extra, rotation) are file name tags - Accounts with per-user progress, reading settings, follows and new-chapter notifications (in-app + optional webhook) - Reader with Flow, Page and Book modes, pointer-anchored page zoom, tap zones, auto-hiding controls, chapter picker, page preloading - Multiple versions per chapter with community votes - Docker image and compose file - Demo library generator (tools/make-dummy.js) - Docs: architecture, library format, import, scraper guide
This commit is contained in:
Maashkinoozhe committed
commit
8dd776b7f4
29 files changed
+3499
No files matched your search
+261
@@ -0,0 +1,261 @@
|
||||
# How the import works
|
||||
|
||||
MangaParty does not copy or convert anything. "Import" means **indexing the library folder in place**. The scanner (`library.js`) walks the folder tree described in [library-format.md](library-format.md) and builds an in-memory index. The server (`server.js`) swaps that index in, compares it with what it has seen before, and notifies followers of new chapters. The library itself is only ever read.
|
||||
|
||||
```
|
||||
┌───────────── triggers ─────────────┐
|
||||
│ start-up · every SCAN_INTERVAL min │
|
||||
│ admin "Rescan" · admin "Full rescan"│
|
||||
└────────────────┬───────────────────┘
|
||||
▼
|
||||
library/ ──read──▶ scanLibrary(root, oldCache) library.js
|
||||
├─ scanManga (per manga folder)
|
||||
│ ├─ book folders → parseKey, book.json, cover
|
||||
│ └─ chapter folders → scanVersion
|
||||
│ ├─ chapter.json (always read)
|
||||
│ └─ page list + image sizes (only if the folder changed)
|
||||
└─ returns { index, cache }
|
||||
▼
|
||||
announce(index) new chapter ids → notifications, webhook server.js
|
||||
▼
|
||||
lib = index (atomic swap, requests see old or new, never half)
|
||||
▼
|
||||
data/scan-cache.json (written atomically)
|
||||
```
|
||||
|
||||
## 1. Triggers
|
||||
|
||||
| Trigger | When | Cache |
|
||||
|---------|------|-------|
|
||||
| Start-up | Right after the HTTP server starts listening. Requests are served meanwhile; the library page shows "Scanning…" until the first scan is done | used |
|
||||
| Interval | Every `SCAN_INTERVAL` minutes (default 10, `0` = off) | used |
|
||||
| **Rescan** (Settings → Admin, or `POST /api/admin/rescan {}`) | On demand; the request returns when the scan is done | used |
|
||||
| **Full rescan** (`POST /api/admin/rescan {"full":true}`) | On demand | ignored: every image header is read again |
|
||||
|
||||
Scans are **single-flight**. A trigger that fires while a scan is running joins that scan instead of starting a second one.
|
||||
|
||||
## 2. Walking the tree
|
||||
|
||||
All directory listings go through one helper:
|
||||
- It skips every entry whose name starts with `.`. That makes hidden folders the safe place for half-written chapters (see §7).
|
||||
- It follows symbolic links to directories.
|
||||
- It sorts names with a natural, case-insensitive collator (`c2` before `c10`).
|
||||
|
||||
### 2.1 Manga folders
|
||||
|
||||
For each folder in the library root:
|
||||
|
||||
1. List it and read `manga.json`.
|
||||
- A missing file becomes `{}`.
|
||||
- Broken JSON is logged and becomes `{}`.
|
||||
- A UTF-8 BOM is stripped.
|
||||
2. `id` = folder name. `title` = `manga.json.title`, else the folder name. `direction` = `manga.json.direction`.
|
||||
3. The cover is the first file matching `cover.(jpg|jpeg|png|webp|gif|avif|bmp)`.
|
||||
4. Scan its book folders (2.2).
|
||||
5. Drop the manga (with a log line) if it ends up without chapters. An exception inside one manga skips only that manga.
|
||||
|
||||
### 2.2 Book folders
|
||||
|
||||
For each sub-folder:
|
||||
|
||||
1. `parseKey(name)` with `^([a-z]*) ?(\d+(?:\.\d+)?)(?=$|[\s_\-[(])[\s_-]*(.*)$` (case-insensitive).
|
||||
- No match → ignored and logged.
|
||||
- Letters exactly `x` → extra book.
|
||||
- key = `x?` + `parseFloat(number)`.
|
||||
- A trailing `[…]` is split off as the source name. The rest is the title.
|
||||
2. A **duplicate book key** within the manga is ignored and logged. The first folder in sort order wins.
|
||||
3. Read `book.json`:
|
||||
- `title` overrides the folder title.
|
||||
- `extra: true` marks the book as extra.
|
||||
- `direction` is used for inheritance.
|
||||
4. Cover: `cover.*` in the folder.
|
||||
5. Scan its chapter folders (2.3). A book without chapters is dropped.
|
||||
|
||||
### 2.3 Chapter version folders
|
||||
|
||||
Each chapter folder is one **version**:
|
||||
|
||||
1. `parseKey(name)` → the chapter key; no match → ignored and logged.
|
||||
2. `stat` the folder and **always** read `chapter.json`. It is small, and edits to it must show up even though they don't change the folder's mtime.
|
||||
3. **Cache check.** The cache key is `Manga/Book/Chapter` (always `/`). If the cached folder mtime equals the current one, the cached page list is reused. Otherwise:
|
||||
- List the files and keep those matching `\.(jpe?g|png|webp|gif|avif|bmp)$`.
|
||||
- Read the pixel size of each (§3).
|
||||
- Store `{ m: mtimeMs, p: [[file, width, height], …] }`.
|
||||
4. Per page, derive:
|
||||
- **tags**: the name without extension, lower-cased, split on `_`, without the first part (`0012_double_rot90.jpg` → `double, rot90`),
|
||||
- **rotation**: the first `rot90|rot180|rot270` tag.
|
||||
5. Version fields:
|
||||
|
||||
| Field | Source |
|
||||
|-------|--------|
|
||||
| `id` | folder name |
|
||||
| `source`, `sourceUrl` | `chapter.json.source` (`{name,url}` or a string), else the `[…]` part of the folder name |
|
||||
| `added` | `Date.parse(chapter.json.added)`, else the folder mtime |
|
||||
| `direction` | first valid of chapter.json → book.json → manga.json → `rtl`. Aliases: `vertical`, `webtoon` → `ttb` |
|
||||
|
||||
6. A version without images is dropped silently.
|
||||
|
||||
### 2.4 Grouping versions into chapters
|
||||
|
||||
Versions in the same book with the same chapter key form one chapter:
|
||||
|
||||
| Field | Rule |
|
||||
|-------|------|
|
||||
| `id` | `<book key>/<chapter key>`, e.g. `1/2.5`, `x1/x1` |
|
||||
| `label` | `Ch. <n>` or `Extra <n>` |
|
||||
| `extra` | book is extra, or letters `x`, or any version's `chapter.json` has `extra: true` |
|
||||
| `title` | first non-empty `chapter.json.title` / folder title, in folder order |
|
||||
| `versions` | in folder order. Ranking by votes happens later, per request (§6) |
|
||||
|
||||
The manga's `updated` is the newest `added` of all versions.
|
||||
|
||||
### 2.5 Sorting and fallbacks
|
||||
|
||||
- Books, and chapters within a book, sort by `(extra-by-x-prefix, number)`. `extra: true` from JSON does **not** move anything.
|
||||
- The manga's reading order is the concatenation of its books' chapters.
|
||||
- Mangas sort by title.
|
||||
- Covers fall back as follows:
|
||||
- book → first `cover`-tagged page (else the first page) of the first chapter's first version,
|
||||
- manga → first book's cover.
|
||||
|
||||
### 2.6 The result
|
||||
|
||||
```js
|
||||
index = {
|
||||
mangas: [{
|
||||
id, title, meta /* manga.json as is */, direction, cover /* relative path */, updated,
|
||||
books: [{ key, num, x, extra, label, title, cover, chapters: [...] }],
|
||||
chapters: [{ id, book, key, num, x, extra, label, title,
|
||||
versions: [{ id, rel, meta, source, sourceUrl, added, direction,
|
||||
pages: [{ f, w, h, t: [tags], r }] }] }],
|
||||
chapterById: Map,
|
||||
}],
|
||||
byId: Map,
|
||||
}
|
||||
```
|
||||
|
||||
Everything is plain data, and the whole index lives in memory.
|
||||
|
||||
## 3. Image sizes
|
||||
|
||||
The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the **first 256 KB** and parses the header:
|
||||
|
||||
| Format | Where the size is |
|
||||
|--------|-------------------|
|
||||
| PNG | IHDR (bytes 16–23) |
|
||||
| GIF | logical screen (bytes 6–9) |
|
||||
| BMP | DIB header (bytes 18–25) |
|
||||
| WebP | `VP8 ` / `VP8L` / `VP8X` chunk |
|
||||
| JPEG | first SOFn marker (the scan skips APPn/EXIF/ICC segments). If it isn't within 256 KB, the whole file is read once |
|
||||
| AVIF / HEIF | first `ispe` box |
|
||||
|
||||
- If the size can't be read, it is stored as `0×0`. The reader then assumes 1000×1414 (2000×1414 for `double`) and corrects the layout when the image arrives.
|
||||
- EXIF orientation is **not** applied. Use `rot…` tags instead.
|
||||
|
||||
## 4. The scan cache
|
||||
|
||||
- Stored in `DATA/scan-cache.json`. It is rewritten atomically after every successful scan and holds only the folders seen in that scan, so deleted chapters drop out.
|
||||
- Invalidation is **per chapter folder, by its mtime**. Adding, removing or renaming a file changes the folder mtime, so that folder is re-read on the next scan.
|
||||
- **Blind spot:** overwriting an image *in place* (same name) usually doesn't change the folder's mtime, so the old size stays cached. Run a **Full rescan** after such edits, or write new files and rename them (§7).
|
||||
- Deleting the cache file is harmless. The next scan is simply slower.
|
||||
|
||||
Measured on the demo library (5 manga, 22 chapters, 213 pages):
|
||||
- ~100 ms without cache on a local disk,
|
||||
- 1.4 s without cache and 0.11 s with cache through a Docker Desktop bind mount on Windows.
|
||||
|
||||
The cost per chapter version is:
|
||||
- unchanged: one `stat` + one small JSON read,
|
||||
- changed: + one directory listing + ≤ 256 KB read per image.
|
||||
|
||||
The scan uses asynchronous I/O one file at a time, so it never blocks request handling.
|
||||
|
||||
## 5. Change detection and notifications
|
||||
|
||||
`db.json` keeps `known[mangaId]`, the list of chapter ids ever seen for a manga. After every scan, `announce(index)` runs:
|
||||
|
||||
1. **First sight of a manga** (no `known` entry): record all its chapter ids and **don't notify**. A newly added series must not flood followers.
|
||||
2. Otherwise, a chapter id not in `known` is new. It is appended; the list **only grows**, so a temporarily missing folder or an unmounted disk never causes a second round of notifications when it comes back.
|
||||
3. For every user following the manga:
|
||||
- one in-app notification per new chapter (text `Title · Ch. 7: Thaw`; at most 200 kept per user),
|
||||
- if the user has a webhook URL, **one** `POST` per manga per scan, with a plain-text body (`New in Moon Garden: Ch. 7 Thaw, Ch. 8 Bloom`), header `Title: MangaParty`, and a 10 s timeout. Failures are logged, not retried.
|
||||
4. Opening a chapter marks its notifications as read.
|
||||
|
||||
What counts as "new" follows from the chapter id (`book key/chapter key`):
|
||||
|
||||
| Change | Notifies? |
|
||||
|--------|-----------|
|
||||
| A new chapter number appears | yes |
|
||||
| Another **version** of an existing chapter appears | no |
|
||||
| A chapter is moved to another book or renumbered | yes (new id) |
|
||||
| A manga folder is renamed | no (it is a new manga) |
|
||||
| A chapter folder is created before its images are complete | yes, immediately. Hence §7 |
|
||||
|
||||
## 6. From index to API
|
||||
|
||||
The index is read by the API at request time. Nothing is precomputed per user.
|
||||
|
||||
| Endpoint | Built from |
|
||||
|----------|------------|
|
||||
| `GET /api/library` | manga list + the user's progress, follows and unread notifications |
|
||||
| `GET /api/manga?m=` | manga incl. `meta` (the raw `manga.json`), books, chapters with ranked versions and the user's progress, "Continue" target |
|
||||
| `GET /api/chapter?m=&c=&v=` | pages of the chosen version (`v`, else the best ranked), direction, prev/next chapter (skipping extras if the user wants that), the user's progress |
|
||||
| `GET /lib/<path>` | the image file itself. Logged-in users only, image extensions only, path must stay inside `LIBRARY`. ETag + `Cache-Control: private, max-age=86400` |
|
||||
|
||||
**Version ranking** happens per request:
|
||||
1. the sum of all users' votes,
|
||||
2. then `chapter.json.priority` (higher first),
|
||||
3. then folder order.
|
||||
|
||||
Votes, progress and follows are keyed by manga id, chapter id and version id. They live in `db.json`, never in the library.
|
||||
|
||||
## 7. Consistency: writing into a live library
|
||||
|
||||
The scanner can run at any moment, including while a tool is still writing a chapter. Without care, it indexes a half-copied chapter, and followers get notified about it. The rules that keep this safe:
|
||||
|
||||
1. **Stage in a hidden folder, publish with one rename.**
|
||||
- Write the chapter into a sibling folder whose name starts with `.` (for example `v02 Summer/.incoming-c0007/`). The scanner never looks at it.
|
||||
- Rename it to its final name when it is complete.
|
||||
- A rename within one file system is atomic: the next scan sees all of the chapter or none of it.
|
||||
2. **Replace versions the same way**: stage the new folder, move the old one out (into a hidden name), move the new one in, then delete the old one.
|
||||
3. **Don't overwrite files in place** (see §4). If you must, trigger a Full rescan afterwards.
|
||||
|
||||
On the server side:
|
||||
- The new index replaces the old one in a single assignment, so requests never see a half-built index.
|
||||
- If the whole scan fails (for example, the library folder is missing), the previous index stays active and the error is shown under Settings → Admin.
|
||||
|
||||
The full rule set for tools is in [scraper-guide.md](scraper-guide.md).
|
||||
|
||||
## 8. Operations
|
||||
|
||||
**Trigger a rescan from a script:**
|
||||
|
||||
```bash
|
||||
curl -s -c jar.txt -H 'Content-Type: application/json' -d '{"name":"admin","password":"…"}' http://localhost:8080/api/login
|
||||
curl -s -b jar.txt -H 'Content-Type: application/json' -d '{}' http://localhost:8080/api/admin/rescan
|
||||
```
|
||||
|
||||
The response contains `scan: { at, ms, mangas, chapters, pages, error }`. Use `{"full":true}` for a full rescan. POST requests must send `Content-Type: application/json` (part of the CSRF protection).
|
||||
|
||||
**Log lines** (stdout, `docker logs`):
|
||||
|
||||
| Message | Meaning / fix |
|
||||
|---------|---------------|
|
||||
| `[scan] 5 manga, 22 chapters, 213 pages in 103 ms` | Scan finished |
|
||||
| `[scan] ignoring "M/B": book folders must be named <letters><number>[ title]…` | Folder name has no valid key: letters, optional space, number, then space/end. Rename it |
|
||||
| `[scan] ignoring "M/B/C": chapter folders must be named …` | Same for a chapter (typical: `c0012a`, `Ch.12`) |
|
||||
| `[scan] ignoring "M/B": another book folder already uses number N` | Two book folders with the same number. Merge them or renumber one |
|
||||
| `[scan] ignoring broken …/chapter.json: …` | Invalid JSON. The folder still works with name-based defaults |
|
||||
| `[scan] "M" has no chapters (expected <manga>/<book>/<chapter>/<images>)` | Usually a missing book level (images or chapters directly in the manga folder) |
|
||||
| `[scan] skipped "M": …` | Unexpected I/O error in that manga (permissions?) |
|
||||
| `[scan] failed: …` | The library root can't be read. The old index stays active |
|
||||
| `[notify] webhook of <user> failed: …` | The webhook URL is unreachable or returned an error |
|
||||
|
||||
**Troubleshooting:**
|
||||
|
||||
| Symptom | Likely cause |
|
||||
|---------|--------------|
|
||||
| New chapter doesn't show up | Next interval hasn't come yet (rescan manually), hidden or invalid folder name, or no image files |
|
||||
| Pages in the wrong order | Numbers not zero-padded, or a mix of padding widths |
|
||||
| A page looks squashed or stretched after being replaced | Image overwritten in place with other dimensions. Run a Full rescan |
|
||||
| Followers were notified about an empty or partial chapter | The chapter was written in place instead of staged and renamed |
|
||||
| Reading progress "lost" | Manga folder renamed, or chapter moved or renumbered (new id) |
|
||||
Reference in new issue
Block a user