MangaParty: self-hosted manga/comic/webtoon reader

Zero-dependency Node server + plain HTML/JS frontend. - Library is a folder tree (Manga/Book/Chapter/pages), indexed in place; page roles (cover, double, extra, rotation) are file name tags - Accounts with per-user progress, reading settings, follows and new-chapter notifications (in-app + optional webhook) - Reader with Flow, Page and Book modes, pointer-anchored page zoom, tap zones, auto-hiding controls, chapter picker, page preloading - Multiple versions per chapter with community votes - Docker image and compose file - Demo library generator (tools/make-dummy.js) - Docs: architecture, library format, import, scraper guide
2026-10-01 09:35:05 +02:00
commit 8dd776b7f4
@@ -0,0 +1,369 @@
+# Scraper guide: supplying the MangaParty library
+
+**Audience:** an automated agent or program that downloads manga, comics or webtoons from upstream sources and writes them into a MangaParty library folder.
+
+**Normative words:** MUST, MUST NOT, SHOULD, SHOULD NOT and MAY as in RFC 2119. Rules have ids (`R-…`) so you can cite them in logs and reviews.
+
+**Background reading** (optional; this guide is self-contained):
+- [library-format.md](library-format.md): the format.
+- [import.md](import.md): how MangaParty reads it.
+
+---
+
+## 0. The contract in short
+
+- You write **plain folders and image files**. There is no upload API.
+- MangaParty scans the library folder every `SCAN_INTERVAL` minutes (default 10), and on demand (§9).
+- The layout is `<library>/<Manga>/<Book>/<Chapter>/<pages>`, always three levels.
+- Names carry meaning:
+  - the manga folder name is the manga's id,
+  - the book/chapter **numbers** in folder names are ids and order,
+  - the chapter folder name is the version id,
+  - page file names carry order and page roles.
+- MangaParty notifies followers as soon as a new chapter number becomes visible. So a chapter MUST become visible **complete and in one step**: stage it in a hidden folder, then rename it (§10).
+- MangaParty never writes to the library. You own it; MangaParty only reads it.
+
+## 1. Inputs you need
+
+| Input | Required | Notes |
+|-------|----------|-------|
+| Library root path | yes | The folder MangaParty mounts as `LIBRARY` (`/library` in Docker) |
+| Upstream manga metadata | yes | Title, reading direction/type, chapter list with numbers |
+| Per chapter: ordered page images | yes | Plus, where available: which pages are covers, spreads, credits/notes |
+| MangaParty URL + an admin account | no | Only to trigger an immediate rescan (§9) |
+
+## 2. Target layout (template)
+
+```
+<library>/
+  <Manga folder>/                         R-M*
+    manga.json                            R-J1
+    cover.jpg                             R-M5
+    v01[ <Book title>]/                   R-B*
+      book.json                           optional
+      c0001[ <Chapter title>] [<Source>]/ R-C*, R-V*
+        chapter.json                      R-J2
+        0001_cover.jpg                    R-P*
+        0002.jpg
+        …
+```
+
+## 3. Manga folders
+
+- **R-M1** The manga folder name MUST NOT start with `.` and MUST be a valid name on Windows, macOS and Linux:
+  - no `< > : " / \ | ? *` and no control characters,
+  - no trailing space or dot,
+  - not a reserved Windows name (`CON`, `PRN`, `AUX`, `NUL`, `COM1`–`COM9`, `LPT1`–`LPT9`),
+  - Unicode normalised to NFC,
+  - SHOULD be ≤ 120 characters.
+- **R-M2** Derive the name from the title once (e.g. the English or romanised title, sanitised per R-M1). After that you MUST NOT rename it: the folder name is the manga's id, and renaming loses every user's progress, follows and votes. Put title changes into `manga.json` → `title`.
+- **R-M3** Before creating a manga, you MUST look for an existing folder for the same work, so one work never ends up in two folders. Match on `manga.json` → `sources[].url` (or your own id field, R-J1). Only if nothing matches, create a new folder.
+- **R-M4** A new manga MUST be published as a whole: stage it as `<library>/.staging-<random>/` with at least one complete chapter, then rename it to its final name (§10).
+- **R-M5** Put the series cover in the manga folder as `cover.<ext>` (`jpg`, `jpeg`, `png`, `webp`, `gif`, `avif`, `bmp`). It SHOULD be at most about 1000 px tall, because it is shown in the library grid at original size. If you have no cover, omit it: the first chapter's cover page is used instead.
+- **R-M6** You MUST NOT put page images directly into the manga folder. Only `manga.json`, `cover.*` and book folders belong there.
+
+## 4. Books (middle level)
+
+- **R-B1** Book folder name pattern:
+  - `v<NN>[ <title>]` for volumes, or `s<NN>[ <title>]` for seasons,
+  - `x<NN>[ <title>]` for extra books (specials, art books, one-shot collections) that sit outside the numbering.
+  - Example: `v01 Spring`, `s02 Season 2`, `x01 Specials`.
+- **R-B2** Pad numbers to 2 digits, or to 3 digits for series that can exceed 99 volumes, and keep the width the same within a manga. Re-padding a book folder later (`v01` → `v001`) is safe: the number, not the text, is the id.
+- **R-B3** Book numbers MUST be unique within a manga. MangaParty ignores a second folder with the same number.
+- **R-B4** A chapter's id includes its book number (`<book>/<chapter>`). Moving a chapter to another book later creates a "new" chapter: progress and votes stay behind, and followers get notified again. Therefore you MUST choose a book strategy per manga that never needs to move chapters:
+  - **Flat (default):** every chapter goes into `v01` (for example `v01 Chapters`). Use it for ongoing series and for sources without reliable volume data.
+  - **Volumes:** chapters go into their volume, **only** if the source gives a final volume number at the time you first write the chapter, and for every chapter.
+  - **Seasons (webtoons):** only if the source states the season.
+  Record the strategy for yourself (e.g. in your own field in `manga.json`, R-J1) so later runs follow it.
+- **R-B5** The book folder MUST contain only `book.json`, `cover.*` and chapter folders.
+
+## 5. Chapters
+
+- **R-C1** Chapter folder name:
+  ```
+  c<NNNN>[.<d>][ <title>][ [<Source>]]     regular chapter
+  x<NN>[ <title>][ [<Source>]]             extra chapter outside the numbering
+  ```
+  - A **space** MUST follow the number when anything follows. `c0012_Title` sorts wrongly in file browsers; `c0012a` and `c0012Title` are not recognised at all.
+  - Recommended check: `^(c\d{4}(\.\d*[1-9])?|x\d{2})( [^\[\]]+?)?( \[[^\[\]]+\])?$`
+- **R-C2** Pad the chapter number's integer part to 4 digits. Never change the padding of an existing manga: chapter folder names are version ids.
+- **R-C3** Map upstream numbering like this:
+
+| Upstream | Folder key | Notes |
+|----------|------------|-------|
+| `12` | `c0012` | |
+| `12.5` | `c0012.5` | In-between chapter, stays in reading order |
+| `12.50` | `c0012.5` | MUST drop trailing decimal zeros. `12.10` would collide with `12.1` |
+| `12a`, `12b` | `c0012.1`, `c0012.2` | Keep "12a" in `chapter.json` → `title` if it matters. Check for collisions with real `12.1` |
+| `12 part 1/2` | `c0012.1`, `c0012.2` | As above |
+| `0`, `Prologue` | `c0000` | |
+| `Omake`, `Extra`, `Side story` placed after chapter 12 | `c0012.5` + `"extra": true` in chapter.json | In order, but skippable |
+| `Special`, `Bonus`, `Character profiles`, not tied to a position | `x01`, `x02`, … in the book they belong to | Listed after the book's regular chapters, skippable |
+| Volume extras (afterword, sketches) | `x<NN>` in that volume's book | |
+| One-shot | its own manga (`v01/c0001`), or `x<NN>` inside an `x01 Oneshots` book of the related series | |
+| Negative or non-numeric only | `x<NN>` | Never invent a number that collides with a real one |
+
+- **R-C4** The title part of the folder name is for humans only.
+  - The authoritative title is `chapter.json` → `title`, without the "Chapter 12:" prefix.
+  - Remove `[` and `]` from titles in folder names (a trailing `[…]` is read as the source).
+- **R-C5** Chapter keys MUST be unique within a book, **except** for deliberate versions (§6).
+- **R-C6** Use the upstream's canonical chapter numbering (preferably the official one). If two sources number the same content differently, map both to one numbering; otherwise they won't be recognised as versions of each other.
+
+## 6. Versions
+
+- **R-V1** Each source's copy of a chapter is its own **version folder** in the same book, with the same chapter number and a source suffix:
+  ```
+  c0012 The Storm [MangaDex]/
+  c0012 The Storm [Official]/
+  ```
+  The suffix SHOULD always be present when a manga has, or may get, more than one source.
+- **R-V2** The version folder name is the version's id, and user votes attach to it. You MUST NOT rename an existing version folder. To update its content, replace it under the **same name** (R-W4).
+- **R-V3** Set `chapter.json` → `source` = `{ "name", "url" }` for every version.
+- **R-V4** You MAY set `chapter.json` → `priority` (number, default `0`) to decide which version wins while users haven't voted, e.g. `1` for official releases. Votes always beat priority.
+- **R-V5** You MUST NOT delete another source's version to "replace" it. Add your version next to it.
+
+## 7. Pages
+
+- **R-P1** Page file name:
+  ```
+  <NNNN>[_<tag>]…<.ext>
+  ```
+  - Numbers are 4 digits, contiguous, starting at `0001`, in reading order.
+  - Use 5 digits for the whole chapter if it has more than 9999 files.
+  - Recommended check: `^\d{4,5}(_[a-z0-9]+)*\.(jpg|jpeg|png|webp|gif|avif|bmp)$`
+- **R-P2** Allowed formats: JPEG, PNG, WebP, GIF, AVIF, BMP. The extension MUST match the actual format (it decides the `Content-Type`).
+  - SHOULD keep the upstream bytes as they are; don't re-encode.
+  - Convert only formats that are not allowed.
+- **R-P3** Tags: add only what you know. When unsure, add no tag.
+
+| Tag | Add when | Don't add when |
+|-----|----------|----------------|
+| `_cover` | The page is a chapter cover or title splash (usually the first page, often in colour, with the chapter title) | A normal first story page |
+| `_double` | One image shows two facing pages (a spread). Usually the width is > 1.2 × the height of normal pages | The source split the spread into two images: keep those as two normal pages, untagged, in order |
+| `_extra` | Credits, recruitment or "support us" pages, translator notes, scanlator ads, previews of other works | Story content |
+| `_rot90` / `_rot180` / `_rot270` | The source says, or you can detect with certainty, that the stored image is turned. The value is the **clockwise** angle needed to show it upright | You only guess |
+
+- **R-P4** Webtoons (`direction: "ttb"`):
+  - Keep the slices as delivered, in order, preferably all the same width. Don't tag spreads.
+  - SHOULD NOT stitch slices into one huge image; very tall images fail on some devices.
+  - SHOULD NOT cut slices smaller.
+- **R-P5** The chapter folder MUST contain only page images and `chapter.json`. Other files are ignored, and sub-folders are not read.
+- **R-P6** You SHOULD keep extra pages (tagged `_extra`) instead of deleting them; readers can skip them. You MAY drop pure advertising.
+
+## 8. Metadata files
+
+All are UTF-8 JSON **without** BOM, pretty-printing optional.
+
+- **R-J0** Every JSON file MUST be valid JSON. A broken file is ignored with a warning; the folder then falls back to name-based defaults.
+- **R-J1** `manga.json`
+
+  ```json
+  {
+    "title": "Moon Garden",
+    "altTitles": ["Tsuki no Niwa"],
+    "authors": ["Aki Tanaka"],
+    "status": "ongoing",
+    "year": 2021,
+    "tags": ["Fantasy", "Slice of Life"],
+    "description": "Synopsis, line breaks allowed.",
+    "direction": "rtl",
+    "sources": [
+      { "name": "MangaDex", "url": "https://mangadex.org/title/…" }
+    ],
+    "x-scraper": { "strategy": "flat", "ids": { "mangadex": "…" } }
+  }
+  ```
+
+  - `direction`:
+
+    | Upstream type | Value |
+    |---------------|-------|
+    | Japanese manga | `rtl` |
+    | Western comic, printed manhwa or manhua | `ltr` |
+    | Webtoon / vertical scroll (also most digital manhwa and manhua) | `ttb` |
+
+  - `sources`: **merge**. Add your entry if its `url` is missing, update your own entry, and never remove others.
+  - Unknown fields are kept and ignored by the app. Keep your own state in one namespaced field such as `"x-scraper"`.
+  - The whole file is shown to logged-in users. It MUST NOT contain secrets (tokens, cookies, credentials).
+  - **Read-modify-write:** keep fields you don't own, because people may have edited them. Write atomically (R-W5).
+
+- **R-J2** `chapter.json`, one per version folder:
+
+  ```json
+  {
+    "title": "The Storm",
+    "source": { "name": "MangaDex", "url": "https://mangadex.org/chapter/…" },
+    "added": "2024-03-15T10:00:00Z",
+    "priority": 0,
+    "extra": false
+  }
+  ```
+
+  | Field | Meaning |
+  |-------|---------|
+  | `added` | Upstream release date, else the download date (ISO 8601). It drives "recently updated" |
+  | `extra` | Only for in-order omake (see R-C3) |
+  | `direction` | Only if this chapter differs from the manga (e.g. a vertical colour special) |
+
+- **R-J3** `book.json`: `{ "title": "Spring" }`. Optional; only write it if the source has volume titles.
+
+## 9. Triggering the import (optional)
+
+New content shows up by itself at the next scheduled scan. For immediate pickup, trigger **one** rescan after a batch, not per chapter:
+
+```bash
+curl -s -c jar.txt -H 'Content-Type: application/json' -d '{"name":"<admin>","password":"<password>"}' "$MANGAPARTY/api/login"
+curl -s -b jar.txt -H 'Content-Type: application/json' -d '{}' "$MANGAPARTY/api/admin/rescan"
+```
+
+- The response contains `scan.error` (null on success) and counts.
+- Use a dedicated admin account for the scraper, and keep its password out of the library.
+
+## 10. Writing safely
+
+MangaParty can scan at any moment. Everything it can see must always be complete.
+
+- **R-W1** All staging paths MUST start with `.` and MUST be on the **same file system** as their final location; only then is `rename` atomic. Put staging folders next to the target:
+
+  | Target | Staging location |
+  |--------|------------------|
+  | new chapter or version | `<book>/.staging-<random>/` |
+  | new book | the book folder itself may be created early (an empty book is ignored), then stage chapters inside it |
+  | new manga | `<library>/.staging-<random>/` |
+
+- **R-W2** Publish = one `rename` of the finished staging folder to its final name. Write every image and `chapter.json` before that rename.
+- **R-W3** You MUST NOT overwrite image files in place. MangaParty caches image sizes per folder mtime, and an in-place overwrite goes unnoticed. Replace whole version folders instead (R-W4).
+- **R-W4** Replacing an existing version (same folder name):
+  1. stage the new copy,
+  2. `rename(final → .old-<random>)`,
+  3. `rename(staging → final)`,
+  4. delete `.old-<random>`.
+  The chapter is briefly absent, which is harmless: it won't be re-announced, and progress is kept.
+- **R-W5** Updating a JSON file: write `.<name>.tmp` in the same folder, then `rename` it over the target.
+- **R-W6** At start-up, delete your own leftover `.staging-*` / `.old-*` folders older than a few hours (from crashed runs).
+- **R-W7** On Windows, `rename` can fail with `EPERM`/`EBUSY` while another process reads the folder. Retry a few times with a short delay.
+- **R-W8** Files MUST be readable by the MangaParty process. In Docker that is uid 1000, so use mode `0644` for files and `0755` for folders.
+- **R-W9** You MUST NOT write anything into MangaParty's `DATA` folder (`db.json`, `scan-cache.json`).
+
+### 10.1 Reference implementation (Node.js, no dependencies)
+
+```js
+const fs = require('fs'), path = require('path'), crypto = require('crypto');
+const rnd = () => crypto.randomBytes(4).toString('hex');
+
+function renameRetry(from, to, tries = 5) {
+  for (let i = 0; ; i++) {
+    try { return fs.renameSync(from, to); }
+    catch (e) {
+      if (i >= tries || !['EPERM', 'EBUSY', 'EACCES'].includes(e.code)) throw e;
+      Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 200 * (i + 1)); // sleep
+    }
+  }
+}
+
+// pages: [{ name: '0001_cover.jpg', data: Buffer }, …] in reading order
+function publishChapter(bookDir, folderName, pages, chapterJson) {
+  fs.mkdirSync(bookDir, { recursive: true });
+  const stage = path.join(bookDir, '.staging-' + rnd());
+  fs.mkdirSync(stage);
+  for (const p of pages) fs.writeFileSync(path.join(stage, p.name), p.data);
+  fs.writeFileSync(path.join(stage, 'chapter.json'), JSON.stringify(chapterJson, null, 2) + '\n');
+  const target = path.join(bookDir, folderName);
+  let old = null;
+  if (fs.existsSync(target)) { old = path.join(bookDir, '.old-' + rnd()); renameRetry(target, old); }
+  renameRetry(stage, target);
+  if (old) fs.rmSync(old, { recursive: true, force: true });
+}
+
+function writeJsonAtomic(file, obj) {
+  const tmp = path.join(path.dirname(file), '.' + path.basename(file) + '.tmp');
+  fs.writeFileSync(tmp, JSON.stringify(obj, null, 2) + '\n');
+  renameRetry(tmp, file);
+}
+```
+
+## 11. Procedure per run
+
+```
+for each manga to scrape:
+    folder = find existing manga folder whose manga.json.sources[].url (or x-scraper id) matches   # R-M3
+    if none: folder = sanitize(title)                                                           # R-M1/2
+             make sure it does not exist yet; if it does and belongs to another work, add a suffix
+    strategy = folder's x-scraper.strategy or choose one (flat unless final volumes are known)  # R-B4
+    upstream_chapters = fetch chapter list
+
+    for each upstream chapter:
+        book   = strategy == flat ? "v01" : "v%02d" % volume                                   # R-B1/2
+        key    = map_number(upstream number)                                                     # R-C3
+        name   = key + (" " + sanitize_title(title) if title) + " [" + source_name + "]"         # R-C1/C4/V1
+        if <book>/<name> exists and is complete and unchanged upstream: skip                    # idempotent
+        pages  = download in reading order; tag cover/double/extra/rot only when certain        # R-P*
+        validate(pages, name)                                                                    # §12
+        publishChapter(<manga>/<book>, name, pages, chapter.json)                                # §10
+
+    merge + write manga.json atomically, update cover.* if missing                               # R-J1, R-M5
+    (new manga: do all of the above inside <library>/.staging-<rnd>/, then rename once)          # R-M4
+
+after the batch: optionally trigger one rescan                                                   # §9
+```
+
+## 12. Validation checklist (before every publish)
+
+- [ ] Manga folder name passes R-M1 and is the existing folder for this work (R-M3).
+- [ ] Book folder matches `^(v|s|x)\d{2,3}( [^\[\]]+)?$`, and its number is unique in the manga.
+- [ ] Chapter folder matches R-C1, has no trailing decimal zeros, and its number follows R-C3.
+- [ ] The same chapter number exists in this book only as deliberate versions with distinct `[Source]` suffixes.
+- [ ] ≥ 1 page. Names `0001…N` are contiguous and match R-P1. Extensions match the real formats.
+- [ ] Tags come only from `cover`, `double`, `extra`, `rot90`, `rot180`, `rot270` (plus your own on purpose).
+- [ ] At most one or two `_cover` pages, and they are at the start.
+- [ ] `chapter.json` parses, and `title` has no "Chapter N" prefix.
+- [ ] `source` is `{name, url}`, and `added` is ISO 8601.
+- [ ] No other files in the chapter folder.
+- [ ] Staging folder starts with `.` and lives next to the target.
+- [ ] Publishing is a single rename.
+- [ ] No secrets in `manga.json`.
+
+## 13. What MangaParty does with your output
+
+| You do | MangaParty does |
+|--------|-----------------|
+| Publish a manga folder for the first time | Adds it to the library. **No** notifications (first sight) |
+| Publish a new chapter number in a known manga | Lists it, and notifies followers in the app and by webhook within one scan |
+| Publish another version of an existing chapter | Adds it to the version picker. No notification. The best-voted (then highest `priority`) version is the default |
+| Make a chapter visible before it is complete | Indexes it as it is, notifies followers, and readers see missing pages |
+| Rename a manga folder | Treats it as a new manga. User data stays with the old name |
+| Move or renumber a chapter | Treats it as a new chapter and notifies again. Old progress and votes are orphaned |
+| Rename a version folder | Votes for it are lost |
+| Remove a chapter | It disappears from the list. It is not re-announced if it comes back |
+| Write invalid names or JSON | Skips the folder or ignores the JSON, and logs a `[scan]` warning in MangaParty's output |
+
+## 14. Worked example
+
+Upstream (simplified):
+
+```json
+{ "title": "Moon Garden", "type": "manga", "url": "https://src.example/t/42",
+  "chapters": [
+    { "num": "1",    "title": "Chapter 1: The Seed", "vol": 1, "date": "2024-01-10", "pages": ["a.jpg", "b.jpg", "c.jpg"] },
+    { "num": "2",    "title": "Roots",               "vol": 1, "date": "2024-01-24", "pages": ["…"] },
+    { "num": "2.50", "title": "Omake: Tea Time",      "vol": 1, "date": "2024-02-01", "pages": ["…"] },
+    { "num": "Special", "title": "Character Sheets",  "vol": 1, "date": "2024-02-02", "pages": ["…"] }
+  ] }
+```
+
+The volume numbers are final, so the **volumes** strategy applies. Result:
+
+```
+Moon Garden/
+  manga.json            {"title":"Moon Garden","direction":"rtl","sources":[{"name":"SrcExample","url":"https://src.example/t/42"}],"x-scraper":{"strategy":"volumes"}}
+  cover.jpg
+  v01/
+    c0001 The Seed [SrcExample]/          chapter.json {"title":"The Seed", …}      0001_cover.jpg 0002.jpg 0003.jpg
+    c0002 Roots [SrcExample]/             chapter.json {"title":"Roots", …}         0001.jpg …
+    c0002.5 Tea Time [SrcExample]/        chapter.json {"title":"Omake: Tea Time","extra":true, …}
+    x01 Character Sheets [SrcExample]/    chapter.json {"title":"Character Sheets", …}   0001_extra.jpg …
+```
+
+- `a.jpg` became `0001_cover.jpg` only because the upstream marks it as a cover, or it was verified to be one.
+- `2.50` became `c0002.5` (R-C3).
+- "Special" without a position became `x01` in its volume (R-C3).