MangaParty: self-hosted manga/comic/webtoon reader
Zero-dependency Node server + plain HTML/JS frontend. - Library is a folder tree (Manga/Book/Chapter/pages), indexed in place; page roles (cover, double, extra, rotation) are file name tags - Accounts with per-user progress, reading settings, follows and new-chapter notifications (in-app + optional webhook) - Reader with Flow, Page and Book modes, pointer-anchored page zoom, tap zones, auto-hiding controls, chapter picker, page preloading - Multiple versions per chapter with community votes - Docker image and compose file - Demo library generator (tools/make-dummy.js) - Docs: architecture, library format, import, scraper guide
This commit is contained in:
Maashkinoozhe committed
commit
8dd776b7f4
29 files changed
+3499
No files matched your search
@@ -0,0 +1,312 @@
|
||||
# MangaParty architecture
|
||||
|
||||
MangaParty is a self-hosted reader for manga, comics and webtoons. One Node.js process serves a handful of static pages and a small JSON API. The content is a folder tree on disk, and everything user-related lives in one JSON file.
|
||||
|
||||
Related documents:
|
||||
- [library-format.md](library-format.md): the folder format.
|
||||
- [import.md](import.md): how the folder tree is indexed.
|
||||
- [scraper-guide.md](scraper-guide.md): rules for tools that write the library.
|
||||
|
||||
## 1. Principles
|
||||
|
||||
| Principle | Consequence |
|
||||
|-----------|-------------|
|
||||
| **The file system is the source of truth for content** | No import step, no copies, no content database. The library can be mounted read-only. Anything that writes files can feed it |
|
||||
| **As dumb as possible** | No dependencies (`package.json` has none), no framework, no build step, no transpiler. Plain ES2019+ JavaScript in the browser, plain Node on the server |
|
||||
| **One process, one data file** | In-memory library index, user data in `data/db.json`. Back up one file, run one container |
|
||||
| **Readable without the app** | The folder format works in any picture viewer. The app adds progress, layout and convenience, not lock-in |
|
||||
|
||||
## 2. System context
|
||||
|
||||
```
|
||||
┌──────────────┐ writes folders (staged + renamed)
|
||||
scraper / ───▶│ library/ │◀──────────────────────────────────┐
|
||||
person │ (read-only │ │
|
||||
│ for the app)│ │
|
||||
└──────┬───────┘ │
|
||||
│ readdir / stat / small reads │
|
||||
┌──────▼─────────────────────────────┐ │
|
||||
browser ─────▶│ node server.js │ webhook │
|
||||
(desktop, │ ├ library.js scanner → index │────────────▶ ntfy / Gotify / …
|
||||
phone, PWA) │ ├ HTTP: / /lib/… /api/… │ (optional)
|
||||
│ └ data/db.json data/scan-cache.json│
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 3. Components
|
||||
|
||||
| Path | Lines | Responsibility |
|
||||
|------|------:|----------------|
|
||||
| `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks |
|
||||
| `library.js` | ~170 | Folder scanner, name parsing, image-size reader, scan cache |
|
||||
| `public/common.js` | ~85 | Browser helpers: `h()` element builder, `api()` fetch wrapper, top navigation with the notification bell |
|
||||
| `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) |
|
||||
| `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle |
|
||||
| `public/read.html` + `reader.js` | ~470 | The reader (§7.2) |
|
||||
| `public/settings.html` + `settings.js` | ~115 | Reading preferences, webhook, password, admin (rescan, users) |
|
||||
| `public/login.html` + `login.js` | ~50 | Sign-in, first-run admin creation, optional sign-up |
|
||||
| `public/app.css` | ~150 | All styles (dark theme, responsive) |
|
||||
| `tools/png.js`, `tools/make-dummy.js` | ~310 | Demo library generator: a tiny PNG encoder with a bitmap font, no image libraries |
|
||||
| `Dockerfile`, `docker-compose.yml` | | `node:24-alpine`, non-root, `/library` + `/data` |
|
||||
|
||||
## 4. Runtime model
|
||||
|
||||
- **One process, event loop only.** All disk I/O in the scanner is asynchronous and sequential, so a scan never blocks requests.
|
||||
- **In-memory state:**
|
||||
- `lib` is the current library index, replaced as a whole after each scan.
|
||||
- `db` is all user data, loaded from `db.json` at start.
|
||||
- **Timers:**
|
||||
- a library scan at start and every `SCAN_INTERVAL` minutes,
|
||||
- a debounced `db.json` write, 500 ms after the last change.
|
||||
- **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json` synchronously, then exit. That matters for `docker stop`.
|
||||
- **Durability:** every file write goes to `*.tmp` first and is then renamed over the target, so a file is never half-written. A hard kill loses at most the changes of the last ~0.5 s. A corrupt `db.json` makes the server **refuse to start** rather than start empty and overwrite it.
|
||||
|
||||
## 5. Data model
|
||||
|
||||
### 5.1 Content: the library index
|
||||
|
||||
The index is built by `library.js` from the folder tree (see [import.md §2.6](import.md#26-the-result)). It is never written anywhere, except for the scan cache.
|
||||
|
||||
### 5.2 Identity
|
||||
|
||||
Every reference from user data to content uses ids derived from names on disk:
|
||||
|
||||
| Entity | Id | Example | Stable under |
|
||||
|--------|----|---------|--------------|
|
||||
| Manga | manga folder name | `Moon Garden` | JSON edits, anything inside the folder |
|
||||
| Book | normalised book number (`x` prefix for extras) | `1`, `x1` | title changes |
|
||||
| Chapter | `<book key>/<chapter key>` | `1/2.5` | title changes, adding/removing versions |
|
||||
| Version | chapter folder name | `c0002 Roots [MangaDex]` | nothing: renaming the folder creates a new version |
|
||||
|
||||
This keeps ids human-readable and derivable without a database. The price: renaming a manga folder, renumbering a chapter or moving it to another book creates a new entity (see [library-format.md → Renaming](library-format.md#renaming-what-survives)).
|
||||
|
||||
### 5.3 User data: `data/db.json`
|
||||
|
||||
```jsonc
|
||||
{
|
||||
"users": {
|
||||
"admin": {
|
||||
"name": "admin", "admin": true, "created": 1790796794000,
|
||||
"hash": "<salt hex>:<scrypt hex>",
|
||||
"settings": {
|
||||
"startMode": "last", // "last" | "flow" | "page" | "book"
|
||||
"lastMode": "book", // the user's intended mode; changed only by clicking a mode
|
||||
"adapt": true, // use a fitting mode for incompatible content (webtoon → flow)
|
||||
"skipExtras": false, "skipCovers": false,
|
||||
"zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 }, // page zoom per mode
|
||||
"webhook": ""
|
||||
}
|
||||
}
|
||||
},
|
||||
"sessions": { "<sha256 of token>": { "u": "admin", "t": 1790796794000 } },
|
||||
"progress": { "admin": { "Moon Garden": { "1/2": {
|
||||
"v": "c0002 Roots [MangaDex]", // version read
|
||||
"p": 4, "n": 14, // page index in that version (all pages), page count
|
||||
"t": 1790796800000, // last activity
|
||||
"done": false, // sticky once true (reset via "mark unread")
|
||||
"shift": [] // book mode: run starts shown alone
|
||||
} } } },
|
||||
"follows": { "admin": { "Moon Garden": true } },
|
||||
"notes": { "admin": [ { "id": "a1b2c3d4e5f6", "t": 1790797300000, "m": "Moon Garden", "c": "2/6",
|
||||
"text": "Moon Garden · Ch. 6: Frost", "read": false } ] },
|
||||
"votes": { "Moon Garden": { "1/2": { "c0002 Roots [Official]": { "admin": 1, "reader": -1 } } } },
|
||||
"known": { "Moon Garden": ["1/1", "1/2", "1/2.5", "..."] } // every chapter id ever seen (append-only)
|
||||
}
|
||||
```
|
||||
|
||||
- `data/scan-cache.json` holds `{ "<Manga>/<Book>/<Chapter>": { m: <folder mtimeMs>, p: [[file, w, h], …] } }`. It is disposable.
|
||||
- **Backups:** copy `data/db.json`. The atomic writes make a copy at any time consistent.
|
||||
|
||||
## 6. Server
|
||||
|
||||
### 6.1 Request routing
|
||||
|
||||
| Path | Handling |
|
||||
|------|----------|
|
||||
| `/api/*` | `routes["<METHOD> <path>"]`. Handlers get `{ u, b, q, req, res }` and return an object, which is sent as JSON. Throwing `HttpError` sends `{ error }` with that status |
|
||||
| `/lib/*` | Image files from `LIBRARY`. Requires a session, allows image extensions only, and the path must resolve inside `LIBRARY`. `ETag` + `private, max-age=86400` |
|
||||
| everything else | Static files from `public/` (`/` → `index.html`). `ETag` + `no-cache` |
|
||||
|
||||
Pages are static. Each page's script calls the API and redirects to `login.html?next=…` on `401`. All URLs are relative, so the app also works under a sub-path behind a reverse proxy.
|
||||
|
||||
### 6.2 API
|
||||
|
||||
All requests and responses are JSON. POSTs require `Content-Type: application/json`.
|
||||
|
||||
| Method & path | Auth | Purpose |
|
||||
|---------------|------|---------|
|
||||
| `GET /api/auth` | public | `{ firstRun, signup }` for the login page |
|
||||
| `POST /api/signup` `{name, password}` | public | First account (becomes admin), or anyone when `SIGNUP=open` |
|
||||
| `POST /api/login` `{name, password}` | public | Starts a session (cookie `sid`) |
|
||||
| `POST /api/logout` | user | Ends the session |
|
||||
| `GET /api/me` | user | Name, admin flag, settings, unread count |
|
||||
| `POST /api/settings` `{…partial}` | user | Validated update of the settings keys above |
|
||||
| `POST /api/password` `{old, new}` | user | Change password |
|
||||
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read |
|
||||
| `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions and the user's progress, `continue` target |
|
||||
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
|
||||
| `POST /api/progress` `{m, c, v, p, n, done, shift}` | user | Save the reading position |
|
||||
| `POST /api/unread` `{m, c}` | user | Forget progress for a chapter |
|
||||
| `POST /api/vote` `{m, c, v, value: 1\|-1\|0}` | user | Vote a version; returns the re-ranked versions |
|
||||
| `POST /api/follow` `{m, on}` | user | Follow a manga for notifications |
|
||||
| `GET /api/notifications` · `POST /api/notifications/read` `{id \| all}` | user | Notification list and read state |
|
||||
| `GET /api/admin` | admin | Users, last scan result, library path |
|
||||
| `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result |
|
||||
| `POST /api/admin/user` `{name, password, admin}` | admin | Create a user or reset a password |
|
||||
| `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data |
|
||||
|
||||
### 6.3 Authentication and security
|
||||
|
||||
| Concern | Measure |
|
||||
|---------|---------|
|
||||
| Passwords | `scrypt` (32-byte key, 16-byte random salt), constant-time compare. Unknown users are checked against a dummy hash to keep timing similar |
|
||||
| Sessions | 256-bit random token in cookie `sid` (`HttpOnly`, `SameSite=Lax`, 1 year, `Secure` with `COOKIE_SECURE=1`). Only its SHA-256 is stored. Pruned at start after a year or when the user is gone |
|
||||
| CSRF | POSTs must be `application/json` (plain cross-site forms can't send that) and must not carry `Sec-Fetch-Site: cross-site` |
|
||||
| Path traversal | Paths are decoded, resolved and checked against the root directory. `/lib/` serves only image extensions, never the JSON files |
|
||||
| XSS | The UI builds DOM nodes and inserts all data as text (`h()` helper). The only markup inserted is the app's own constant SVG icons. External links are rendered only for `http(s)` URLs |
|
||||
| Request size | JSON bodies are limited to 1 MB |
|
||||
| Authorisation | `admin` flag checked per admin route. Users only ever see their own progress, follows and notifications. Votes are shown as totals plus the user's own |
|
||||
| Not included | Login rate limiting, a CSP header, TLS. Run behind a reverse proxy if exposed. Webhook URLs are user-chosen, so any user can make the server send a POST to an address they choose. Fine for a household; consider it before `SIGNUP=open` |
|
||||
|
||||
### 6.4 Version ranking
|
||||
|
||||
At request time: the sum of all users' votes, then `chapter.json.priority`, then folder order. The top version is what "open chapter" shows. An explicit `v=` (version picker, "Continue" of a manually chosen version) overrides it.
|
||||
|
||||
### 6.5 Notifications
|
||||
|
||||
In-app notifications and optional webhooks are produced by `announce()` after each scan, from the append-only `known` list. See [import.md §5](import.md#5-change-detection-and-notifications).
|
||||
|
||||
## 7. Frontend
|
||||
|
||||
### 7.1 Pages
|
||||
|
||||
Every page is a static HTML file plus one script, loaded with `defer`. All pages share `common.js` and `app.css`. There is no client-side router, no state library and no component system. Each script fetches what it needs, builds DOM with `h()`, and re-renders small parts on change.
|
||||
|
||||
A web app manifest (`display: standalone`) makes "Add to home screen" open the app without browser bars on phones.
|
||||
|
||||
### 7.2 The reader (`reader.js`)
|
||||
|
||||
**Inputs:**
|
||||
- the chapter payload (pages with pixel sizes, tags and rotation; direction; versions; neighbours; progress),
|
||||
- the user's settings.
|
||||
|
||||
**Page model.** Each page's size is known before it loads, so the reader computes the whole layout itself and positions every page absolutely:
|
||||
|
||||
```
|
||||
all pages ──filter (skipExtras / skipCovers)──▶ list
|
||||
list ──mode──▶ build(): create page elements (flow: all, lazy · page/book: current + preload next)
|
||||
└──▶ layout(): compute a box {x, y, w, h} per element from the viewport, zoom and page aspect
|
||||
└─▶ place(): apply the boxes; rotated images are sized swapped and turned with CSS
|
||||
```
|
||||
|
||||
`layout()` repeats up to three times until the viewport size is stable, because scrollbars that appear or disappear change the viewport.
|
||||
|
||||
**Layout rules** (z = page zoom of the current mode):
|
||||
|
||||
| Mode | Box sizes |
|
||||
|------|-----------|
|
||||
| Flow (paged content) | `W = min(vw, vh · refAspect) · z`: a normal single page fits the screen at 100 %. Wide pages keep the single page height, capped at `max(vw, W)`. 8 px gap |
|
||||
| Flow (webtoon, zoom key `strip`) | `W = min(vw, median image width) · z`, no gaps, so slices join seamlessly |
|
||||
| Page | The page fits the viewport (`contain`) · z, centred. The canvas grows beyond the viewport when zoomed in |
|
||||
| Book | The pages of the spread share one height `H = min(vh, vw / Σaspect) · z`. Order is right-to-left for `rtl` |
|
||||
|
||||
**Anchoring.** Before any re-layout (zoom, resize, pinch), `anchor(x, y)` records which page is under that point and where inside it, as a fraction. `restore()` scrolls so the same point is under the finger or cursor again. That is why zooming feels pinned to the cursor and a resize keeps the reading position.
|
||||
|
||||
**Book pairing:**
|
||||
- Covers and wide pages stand alone.
|
||||
- The single pages between them form *runs*, and each run pairs from its start.
|
||||
- "⇄ Shift" toggles a run to start with a lone page. It is stored as the run's first page index in `progress.shift`.
|
||||
- One broken spot can therefore never shift the pairing of the whole chapter.
|
||||
|
||||
**Mode resolution:**
|
||||
|
||||
```
|
||||
intended = startMode == "last" ? lastMode : startMode
|
||||
compatible = direction == "ttb" ? [flow] : [flow, page, book]
|
||||
mode = adapt && intended ∉ compatible ? compatible[0] : intended
|
||||
```
|
||||
|
||||
`lastMode` is written only when the user clicks a mode, never on an automatic switch.
|
||||
|
||||
**Input:**
|
||||
|
||||
| Input | Action |
|
||||
|-------|--------|
|
||||
| Click or tap outside the pages | Toggle the bars (the end card uses the whole screen as zones instead) |
|
||||
| Click or tap on the page(s), outer 35 % of the visible page area (paged) | Previous/next, following the reading direction (`rtl`: left = next) |
|
||||
| Click or tap on a page, top/bottom 35 % of the screen (flow) | Scroll to the previous/next page top, at most 90 % of a screen (85 % steps for webtoons). At the very bottom → next chapter |
|
||||
| Click or tap on a page, middle band | Toggle the bars |
|
||||
| Horizontal swipe ≥ 50 px within 800 ms (paged, not zoomed in) | Previous/next |
|
||||
| Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Page zoom 20–600 %, snapping to 100 % within ±3 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages |
|
||||
| Mouse drag (moved ≥ 6 px) | Pan; the following click is swallowed. Hand cursor when the content is larger than the screen |
|
||||
| Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` `-` `0` zoom; `m`/Esc bars; `f` fullscreen |
|
||||
|
||||
**Bars:**
|
||||
- Shown on open.
|
||||
- Hidden on the first page turn or after 100 px of scrolling. The first automatic hide (per device, once for Flow and once for Page/Book) shows a hint on how to get them back.
|
||||
- **Never shown by scrolling**, only by a tap beside the pages or in the middle of a page (or `m`). Showing them stops a running smooth scroll, so it can't hide them again right away.
|
||||
- The top bar has the chapter picker (every chapter, grouped by book, ✓ = read).
|
||||
|
||||
**Loading:**
|
||||
- Flow mode loads images through an `IntersectionObserver` on the scroll container, 1.5 screens ahead.
|
||||
- Page and Book mode keep a small pool (12) of `Image` objects:
|
||||
- Once the pages on screen have loaded, the next 3 pages (Book: next 2 spreads) and the previous one are fetched at low priority and **decoded** (`img.decode()`).
|
||||
- A page turn then inserts those same, already decoded elements (`decoding="sync"`), so the old and new page swap in one frame. A fresh `<img>` would be painted empty while it decodes, which is what caused the flicker on the first view of each page.
|
||||
- A page that is still loading shows a spinner (only after 150 ms, so fast loads never flash it). A page that fails shows a message.
|
||||
- A page with unknown size is laid out at 1000×1414 and re-laid out when it loads.
|
||||
|
||||
**Progress:**
|
||||
- Saved 700 ms after the position settles, and on `pagehide` (a `fetch` with `keepalive`).
|
||||
- Stores the page index in the full page list, so skipping extras doesn't corrupt it.
|
||||
- On open, the saved page is used for the same version, and mapped proportionally for another version. A finished chapter reopens at the start.
|
||||
|
||||
**Zoom persistence:** per mode key (`flow`, `page`, `book`, `strip`), saved 800 ms after the last change.
|
||||
|
||||
## 8. Deployment
|
||||
|
||||
```bash
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
- **Image:** `node:24-alpine` plus the app files. It runs as user `node` (uid 1000) and has a health check against `/api/auth`.
|
||||
- **Volumes:**
|
||||
- `/library`: read-only is enough. Files must be readable by uid 1000.
|
||||
- `/data`: must be writable.
|
||||
- **Configuration:** only environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `COOKIE_SECURE`).
|
||||
- **Without Docker:** `node server.js` with Node ≥ 18.
|
||||
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
|
||||
- **Upgrades:** replace the code and restart. There are no migrations: unknown fields in `db.json` are kept, and missing settings fall back to defaults.
|
||||
|
||||
## 9. Capacity and limits
|
||||
|
||||
| Aspect | Behaviour |
|
||||
|--------|-----------|
|
||||
| Library size | The index keeps a small object per page (file name, size, tags), in the order of a few hundred bytes. Hundreds of thousands of pages fit in a modest container. Unchanged chapters cost one `stat` + one small read per scan |
|
||||
| First scan | Reads ≤ 256 KB per image once. Later scans use the cache |
|
||||
| User data | `db.json` is rewritten as a whole, at most about twice per second while people read. Comfortable up to a few MB (a household or small group, thousands of chapters read) |
|
||||
| Concurrency | One process, one writer. No clustering or horizontal scaling by design |
|
||||
| Images | Served as stored. No resizing or thumbnails, so large covers make the library page heavy |
|
||||
|
||||
## 10. Design decisions
|
||||
|
||||
| Decision | Alternatives | Why |
|
||||
|----------|--------------|-----|
|
||||
| Index the folder tree in place | Import into a DB / managed storage | Stays readable by any viewer and writable by any tool. Nothing to keep in sync |
|
||||
| No dependencies, no framework, no build | Express, React, bundler | Minimal code, maximum longevity, runs anywhere Node runs |
|
||||
| JSON file for user data | SQLite (`node:sqlite` is still experimental in Node 22/24), Postgres | Zero dependencies, trivial backup, enough for the target size. Easy to swap: all access goes through the `db` object |
|
||||
| Page roles as file-name tags | Page lists in JSON | Visible in any file browser, survives copying, no second source of truth. Unknown tags are harmless |
|
||||
| Image sizes read on the server | Measure in the browser after load | Layout before load: exact scroll restore, zoom anchoring, auto-detected double pages, no layout jumps |
|
||||
| Absolute layout engine in the reader | CSS flow/flex layout | Deterministic geometry, needed for pointer-anchored zoom and exact position restore |
|
||||
| Chapter id = book number + chapter number | Random ids stored in JSON | Human-readable and needs no writes into the library. Downside: moving a chapter between books changes its identity |
|
||||
| Global votes, per-visit version choice | Per-user preferred version | "The better version is shown to everyone", while any user can still pick another one explicitly |
|
||||
| Bars never reappear on scroll | Show on scroll-up (common pattern) | Explicit requirement: readers scroll up and down constantly |
|
||||
|
||||
## 11. Known limitations and extension points
|
||||
|
||||
- **Identity is name-based.** An optional stable id in `manga.json` (with a migration of the user data keys) would make manga renames safe.
|
||||
- **No thumbnails.** Would need an image library (e.g. `sharp`) or pre-generated `cover` files.
|
||||
- **Notifications:** in-app plus webhook only. Web Push would need VAPID keys and a service worker.
|
||||
- **No login rate limiting and no CSP header.** Both are small additions in `server.js`.
|
||||
- **In-place image overwrites** are only detected by a full rescan.
|
||||
- **No automated test suite.** Verification so far was done with the demo library (`tools/make-dummy.js`), API smoke scripts and manual browser testing on desktop and mobile viewports. `parseKey` and `sizeFromBuffer` are pure functions, and `scanLibrary` only needs a folder. Together with the book pairing they are the natural first targets for unit tests.
|
||||
- **Adding content by API** (instead of files) would be a new route plus the staging and renaming rules from [import.md §7](import.md#7-consistency-writing-into-a-live-library).
|
||||
+261
@@ -0,0 +1,261 @@
|
||||
# How the import works
|
||||
|
||||
MangaParty does not copy or convert anything. "Import" means **indexing the library folder in place**. The scanner (`library.js`) walks the folder tree described in [library-format.md](library-format.md) and builds an in-memory index. The server (`server.js`) swaps that index in, compares it with what it has seen before, and notifies followers of new chapters. The library itself is only ever read.
|
||||
|
||||
```
|
||||
┌───────────── triggers ─────────────┐
|
||||
│ start-up · every SCAN_INTERVAL min │
|
||||
│ admin "Rescan" · admin "Full rescan"│
|
||||
└────────────────┬───────────────────┘
|
||||
▼
|
||||
library/ ──read──▶ scanLibrary(root, oldCache) library.js
|
||||
├─ scanManga (per manga folder)
|
||||
│ ├─ book folders → parseKey, book.json, cover
|
||||
│ └─ chapter folders → scanVersion
|
||||
│ ├─ chapter.json (always read)
|
||||
│ └─ page list + image sizes (only if the folder changed)
|
||||
└─ returns { index, cache }
|
||||
▼
|
||||
announce(index) new chapter ids → notifications, webhook server.js
|
||||
▼
|
||||
lib = index (atomic swap, requests see old or new, never half)
|
||||
▼
|
||||
data/scan-cache.json (written atomically)
|
||||
```
|
||||
|
||||
## 1. Triggers
|
||||
|
||||
| Trigger | When | Cache |
|
||||
|---------|------|-------|
|
||||
| Start-up | Right after the HTTP server starts listening. Requests are served meanwhile; the library page shows "Scanning…" until the first scan is done | used |
|
||||
| Interval | Every `SCAN_INTERVAL` minutes (default 10, `0` = off) | used |
|
||||
| **Rescan** (Settings → Admin, or `POST /api/admin/rescan {}`) | On demand; the request returns when the scan is done | used |
|
||||
| **Full rescan** (`POST /api/admin/rescan {"full":true}`) | On demand | ignored: every image header is read again |
|
||||
|
||||
Scans are **single-flight**. A trigger that fires while a scan is running joins that scan instead of starting a second one.
|
||||
|
||||
## 2. Walking the tree
|
||||
|
||||
All directory listings go through one helper:
|
||||
- It skips every entry whose name starts with `.`. That makes hidden folders the safe place for half-written chapters (see §7).
|
||||
- It follows symbolic links to directories.
|
||||
- It sorts names with a natural, case-insensitive collator (`c2` before `c10`).
|
||||
|
||||
### 2.1 Manga folders
|
||||
|
||||
For each folder in the library root:
|
||||
|
||||
1. List it and read `manga.json`.
|
||||
- A missing file becomes `{}`.
|
||||
- Broken JSON is logged and becomes `{}`.
|
||||
- A UTF-8 BOM is stripped.
|
||||
2. `id` = folder name. `title` = `manga.json.title`, else the folder name. `direction` = `manga.json.direction`.
|
||||
3. The cover is the first file matching `cover.(jpg|jpeg|png|webp|gif|avif|bmp)`.
|
||||
4. Scan its book folders (2.2).
|
||||
5. Drop the manga (with a log line) if it ends up without chapters. An exception inside one manga skips only that manga.
|
||||
|
||||
### 2.2 Book folders
|
||||
|
||||
For each sub-folder:
|
||||
|
||||
1. `parseKey(name)` with `^([a-z]*) ?(\d+(?:\.\d+)?)(?=$|[\s_\-[(])[\s_-]*(.*)$` (case-insensitive).
|
||||
- No match → ignored and logged.
|
||||
- Letters exactly `x` → extra book.
|
||||
- key = `x?` + `parseFloat(number)`.
|
||||
- A trailing `[…]` is split off as the source name. The rest is the title.
|
||||
2. A **duplicate book key** within the manga is ignored and logged. The first folder in sort order wins.
|
||||
3. Read `book.json`:
|
||||
- `title` overrides the folder title.
|
||||
- `extra: true` marks the book as extra.
|
||||
- `direction` is used for inheritance.
|
||||
4. Cover: `cover.*` in the folder.
|
||||
5. Scan its chapter folders (2.3). A book without chapters is dropped.
|
||||
|
||||
### 2.3 Chapter version folders
|
||||
|
||||
Each chapter folder is one **version**:
|
||||
|
||||
1. `parseKey(name)` → the chapter key; no match → ignored and logged.
|
||||
2. `stat` the folder and **always** read `chapter.json`. It is small, and edits to it must show up even though they don't change the folder's mtime.
|
||||
3. **Cache check.** The cache key is `Manga/Book/Chapter` (always `/`). If the cached folder mtime equals the current one, the cached page list is reused. Otherwise:
|
||||
- List the files and keep those matching `\.(jpe?g|png|webp|gif|avif|bmp)$`.
|
||||
- Read the pixel size of each (§3).
|
||||
- Store `{ m: mtimeMs, p: [[file, width, height], …] }`.
|
||||
4. Per page, derive:
|
||||
- **tags**: the name without extension, lower-cased, split on `_`, without the first part (`0012_double_rot90.jpg` → `double, rot90`),
|
||||
- **rotation**: the first `rot90|rot180|rot270` tag.
|
||||
5. Version fields:
|
||||
|
||||
| Field | Source |
|
||||
|-------|--------|
|
||||
| `id` | folder name |
|
||||
| `source`, `sourceUrl` | `chapter.json.source` (`{name,url}` or a string), else the `[…]` part of the folder name |
|
||||
| `added` | `Date.parse(chapter.json.added)`, else the folder mtime |
|
||||
| `direction` | first valid of chapter.json → book.json → manga.json → `rtl`. Aliases: `vertical`, `webtoon` → `ttb` |
|
||||
|
||||
6. A version without images is dropped silently.
|
||||
|
||||
### 2.4 Grouping versions into chapters
|
||||
|
||||
Versions in the same book with the same chapter key form one chapter:
|
||||
|
||||
| Field | Rule |
|
||||
|-------|------|
|
||||
| `id` | `<book key>/<chapter key>`, e.g. `1/2.5`, `x1/x1` |
|
||||
| `label` | `Ch. <n>` or `Extra <n>` |
|
||||
| `extra` | book is extra, or letters `x`, or any version's `chapter.json` has `extra: true` |
|
||||
| `title` | first non-empty `chapter.json.title` / folder title, in folder order |
|
||||
| `versions` | in folder order. Ranking by votes happens later, per request (§6) |
|
||||
|
||||
The manga's `updated` is the newest `added` of all versions.
|
||||
|
||||
### 2.5 Sorting and fallbacks
|
||||
|
||||
- Books, and chapters within a book, sort by `(extra-by-x-prefix, number)`. `extra: true` from JSON does **not** move anything.
|
||||
- The manga's reading order is the concatenation of its books' chapters.
|
||||
- Mangas sort by title.
|
||||
- Covers fall back as follows:
|
||||
- book → first `cover`-tagged page (else the first page) of the first chapter's first version,
|
||||
- manga → first book's cover.
|
||||
|
||||
### 2.6 The result
|
||||
|
||||
```js
|
||||
index = {
|
||||
mangas: [{
|
||||
id, title, meta /* manga.json as is */, direction, cover /* relative path */, updated,
|
||||
books: [{ key, num, x, extra, label, title, cover, chapters: [...] }],
|
||||
chapters: [{ id, book, key, num, x, extra, label, title,
|
||||
versions: [{ id, rel, meta, source, sourceUrl, added, direction,
|
||||
pages: [{ f, w, h, t: [tags], r }] }] }],
|
||||
chapterById: Map,
|
||||
}],
|
||||
byId: Map,
|
||||
}
|
||||
```
|
||||
|
||||
Everything is plain data, and the whole index lives in memory.
|
||||
|
||||
## 3. Image sizes
|
||||
|
||||
The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the **first 256 KB** and parses the header:
|
||||
|
||||
| Format | Where the size is |
|
||||
|--------|-------------------|
|
||||
| PNG | IHDR (bytes 16–23) |
|
||||
| GIF | logical screen (bytes 6–9) |
|
||||
| BMP | DIB header (bytes 18–25) |
|
||||
| WebP | `VP8 ` / `VP8L` / `VP8X` chunk |
|
||||
| JPEG | first SOFn marker (the scan skips APPn/EXIF/ICC segments). If it isn't within 256 KB, the whole file is read once |
|
||||
| AVIF / HEIF | first `ispe` box |
|
||||
|
||||
- If the size can't be read, it is stored as `0×0`. The reader then assumes 1000×1414 (2000×1414 for `double`) and corrects the layout when the image arrives.
|
||||
- EXIF orientation is **not** applied. Use `rot…` tags instead.
|
||||
|
||||
## 4. The scan cache
|
||||
|
||||
- Stored in `DATA/scan-cache.json`. It is rewritten atomically after every successful scan and holds only the folders seen in that scan, so deleted chapters drop out.
|
||||
- Invalidation is **per chapter folder, by its mtime**. Adding, removing or renaming a file changes the folder mtime, so that folder is re-read on the next scan.
|
||||
- **Blind spot:** overwriting an image *in place* (same name) usually doesn't change the folder's mtime, so the old size stays cached. Run a **Full rescan** after such edits, or write new files and rename them (§7).
|
||||
- Deleting the cache file is harmless. The next scan is simply slower.
|
||||
|
||||
Measured on the demo library (5 manga, 22 chapters, 213 pages):
|
||||
- ~100 ms without cache on a local disk,
|
||||
- 1.4 s without cache and 0.11 s with cache through a Docker Desktop bind mount on Windows.
|
||||
|
||||
The cost per chapter version is:
|
||||
- unchanged: one `stat` + one small JSON read,
|
||||
- changed: + one directory listing + ≤ 256 KB read per image.
|
||||
|
||||
The scan uses asynchronous I/O one file at a time, so it never blocks request handling.
|
||||
|
||||
## 5. Change detection and notifications
|
||||
|
||||
`db.json` keeps `known[mangaId]`, the list of chapter ids ever seen for a manga. After every scan, `announce(index)` runs:
|
||||
|
||||
1. **First sight of a manga** (no `known` entry): record all its chapter ids and **don't notify**. A newly added series must not flood followers.
|
||||
2. Otherwise, a chapter id not in `known` is new. It is appended; the list **only grows**, so a temporarily missing folder or an unmounted disk never causes a second round of notifications when it comes back.
|
||||
3. For every user following the manga:
|
||||
- one in-app notification per new chapter (text `Title · Ch. 7: Thaw`; at most 200 kept per user),
|
||||
- if the user has a webhook URL, **one** `POST` per manga per scan, with a plain-text body (`New in Moon Garden: Ch. 7 Thaw, Ch. 8 Bloom`), header `Title: MangaParty`, and a 10 s timeout. Failures are logged, not retried.
|
||||
4. Opening a chapter marks its notifications as read.
|
||||
|
||||
What counts as "new" follows from the chapter id (`book key/chapter key`):
|
||||
|
||||
| Change | Notifies? |
|
||||
|--------|-----------|
|
||||
| A new chapter number appears | yes |
|
||||
| Another **version** of an existing chapter appears | no |
|
||||
| A chapter is moved to another book or renumbered | yes (new id) |
|
||||
| A manga folder is renamed | no (it is a new manga) |
|
||||
| A chapter folder is created before its images are complete | yes, immediately. Hence §7 |
|
||||
|
||||
## 6. From index to API
|
||||
|
||||
The index is read by the API at request time. Nothing is precomputed per user.
|
||||
|
||||
| Endpoint | Built from |
|
||||
|----------|------------|
|
||||
| `GET /api/library` | manga list + the user's progress, follows and unread notifications |
|
||||
| `GET /api/manga?m=` | manga incl. `meta` (the raw `manga.json`), books, chapters with ranked versions and the user's progress, "Continue" target |
|
||||
| `GET /api/chapter?m=&c=&v=` | pages of the chosen version (`v`, else the best ranked), direction, prev/next chapter (skipping extras if the user wants that), the user's progress |
|
||||
| `GET /lib/<path>` | the image file itself. Logged-in users only, image extensions only, path must stay inside `LIBRARY`. ETag + `Cache-Control: private, max-age=86400` |
|
||||
|
||||
**Version ranking** happens per request:
|
||||
1. the sum of all users' votes,
|
||||
2. then `chapter.json.priority` (higher first),
|
||||
3. then folder order.
|
||||
|
||||
Votes, progress and follows are keyed by manga id, chapter id and version id. They live in `db.json`, never in the library.
|
||||
|
||||
## 7. Consistency: writing into a live library
|
||||
|
||||
The scanner can run at any moment, including while a tool is still writing a chapter. Without care, it indexes a half-copied chapter, and followers get notified about it. The rules that keep this safe:
|
||||
|
||||
1. **Stage in a hidden folder, publish with one rename.**
|
||||
- Write the chapter into a sibling folder whose name starts with `.` (for example `v02 Summer/.incoming-c0007/`). The scanner never looks at it.
|
||||
- Rename it to its final name when it is complete.
|
||||
- A rename within one file system is atomic: the next scan sees all of the chapter or none of it.
|
||||
2. **Replace versions the same way**: stage the new folder, move the old one out (into a hidden name), move the new one in, then delete the old one.
|
||||
3. **Don't overwrite files in place** (see §4). If you must, trigger a Full rescan afterwards.
|
||||
|
||||
On the server side:
|
||||
- The new index replaces the old one in a single assignment, so requests never see a half-built index.
|
||||
- If the whole scan fails (for example, the library folder is missing), the previous index stays active and the error is shown under Settings → Admin.
|
||||
|
||||
The full rule set for tools is in [scraper-guide.md](scraper-guide.md).
|
||||
|
||||
## 8. Operations
|
||||
|
||||
**Trigger a rescan from a script:**
|
||||
|
||||
```bash
|
||||
curl -s -c jar.txt -H 'Content-Type: application/json' -d '{"name":"admin","password":"…"}' http://localhost:8080/api/login
|
||||
curl -s -b jar.txt -H 'Content-Type: application/json' -d '{}' http://localhost:8080/api/admin/rescan
|
||||
```
|
||||
|
||||
The response contains `scan: { at, ms, mangas, chapters, pages, error }`. Use `{"full":true}` for a full rescan. POST requests must send `Content-Type: application/json` (part of the CSRF protection).
|
||||
|
||||
**Log lines** (stdout, `docker logs`):
|
||||
|
||||
| Message | Meaning / fix |
|
||||
|---------|---------------|
|
||||
| `[scan] 5 manga, 22 chapters, 213 pages in 103 ms` | Scan finished |
|
||||
| `[scan] ignoring "M/B": book folders must be named <letters><number>[ title]…` | Folder name has no valid key: letters, optional space, number, then space/end. Rename it |
|
||||
| `[scan] ignoring "M/B/C": chapter folders must be named …` | Same for a chapter (typical: `c0012a`, `Ch.12`) |
|
||||
| `[scan] ignoring "M/B": another book folder already uses number N` | Two book folders with the same number. Merge them or renumber one |
|
||||
| `[scan] ignoring broken …/chapter.json: …` | Invalid JSON. The folder still works with name-based defaults |
|
||||
| `[scan] "M" has no chapters (expected <manga>/<book>/<chapter>/<images>)` | Usually a missing book level (images or chapters directly in the manga folder) |
|
||||
| `[scan] skipped "M": …` | Unexpected I/O error in that manga (permissions?) |
|
||||
| `[scan] failed: …` | The library root can't be read. The old index stays active |
|
||||
| `[notify] webhook of <user> failed: …` | The webhook URL is unreachable or returned an error |
|
||||
|
||||
**Troubleshooting:**
|
||||
|
||||
| Symptom | Likely cause |
|
||||
|---------|--------------|
|
||||
| New chapter doesn't show up | Next interval hasn't come yet (rescan manually), hidden or invalid folder name, or no image files |
|
||||
| Pages in the wrong order | Numbers not zero-padded, or a mix of padding widths |
|
||||
| A page looks squashed or stretched after being replaced | Image overwritten in place with other dimensions. Run a Full rescan |
|
||||
| Followers were notified about an empty or partial chapter | The chapter was written in place instead of staged and renamed |
|
||||
| Reading progress "lost" | Manga folder renamed, or chapter moved or renumbered (new id) |
|
||||
@@ -0,0 +1,248 @@
|
||||
# The file system library
|
||||
|
||||
MangaParty has no import database for content. The **library folder is the source of truth**. The app reads it in place (read-only access is enough), and anything that can write files can add manga: a scraper, a sync tool, or a person with a file browser.
|
||||
|
||||
This document is the format specification. [import.md](import.md) describes how the app reads it, and [scraper-guide.md](scraper-guide.md) is a rule book for tools that write it.
|
||||
|
||||
## Goals of the format
|
||||
|
||||
| Goal | How the format meets it |
|
||||
|------|-------------------------|
|
||||
| Readable without the app | Plain image files in plain folders. Zero-padded numbers make alphabetical order equal reading order in every file browser and picture viewer |
|
||||
| Fast, simple indexing | Everything the app needs is in folder names, file names and a few small JSON files. Image contents are never decoded. Only the first bytes of each image are read once to learn its size |
|
||||
| Self-describing | Page roles (cover, double page, extra, rotated) are visible in the file name, not hidden in a database |
|
||||
| Several sources | One manga can list several upstream sources. One chapter can exist in several versions side by side |
|
||||
| Room for irregular releases | In-between chapters (`10.5`), extra chapters and extra books outside the numbering |
|
||||
|
||||
## Layout
|
||||
|
||||
Always three folder levels, **Manga / Book / Chapter**, with the pages inside the chapter folder:
|
||||
|
||||
```
|
||||
<library>/
|
||||
<Manga>/ manga folder; its name is the manga's id
|
||||
manga.json optional
|
||||
cover.<ext> optional
|
||||
<Book>/ e.g. "v01 Spring"
|
||||
book.json optional
|
||||
cover.<ext> optional
|
||||
<Chapter>/ e.g. "c0001 The Seed" (one folder per chapter version)
|
||||
chapter.json recommended
|
||||
0001_cover.<ext>
|
||||
0002.<ext>
|
||||
...
|
||||
```
|
||||
|
||||
Series without volumes (most webtoons, ongoing magazine series) still need the book level. Use a single `v01`, or seasons (`s01 Season 1`).
|
||||
|
||||
Anything else is ignored:
|
||||
- files at the library root,
|
||||
- files other than `manga.json` / `cover.*` in a manga folder,
|
||||
- files other than `book.json` / `cover.*` in a book folder,
|
||||
- non-image files and sub-folders inside a chapter folder,
|
||||
- every file or folder whose name starts with `.`.
|
||||
|
||||
## Manga folders
|
||||
|
||||
- Any name that does not start with `.`. The folder name is the manga's **id**: URLs, reading progress, follows, votes and the "already seen chapters" list all refer to it. **Renaming the folder turns it into a new manga for the app.**
|
||||
- The displayed title comes from `manga.json` → `title`, and falls back to the folder name.
|
||||
- Manga without a single readable chapter are left out (and logged).
|
||||
|
||||
## Book and chapter folder names
|
||||
|
||||
```
|
||||
<letters>[ ]<number>[<separator><title>][ [<source>]]
|
||||
```
|
||||
|
||||
| Part | Rule | Examples |
|
||||
|------|------|----------|
|
||||
| letters | Only letters (any case), optionally followed by **one** space. Free choice and just a hint for humans, **except `x`, which means "extra"** | `v`, `c`, `s`, `vol`, `Ch `, `Chapter `, `x` |
|
||||
| number | Digits, optionally one decimal part. It decides the order and the identity. Zero-pad it so file browsers sort it right (4 digits for chapters, 2 for books) | `0001`, `12`, `0010.5` |
|
||||
| separator | The number must be followed by the end of the name, a space, `_`, `-`, `[` or `(`. **Use a space**: `_` sorts after `.`, which breaks the order of decimal chapters in file browsers | ` `, ` - ` |
|
||||
| title | Anything. Used as the title when the JSON file has none. Leading `-`, `:` or `.` are trimmed | `The Seed` |
|
||||
| `[source]` | Optional, at the very end. Names the version (scan group, site) when `chapter.json` has no `source` | `[MangaDex]` |
|
||||
|
||||
Valid: `v01`, `v01 Spring`, `c0001 The Seed`, `c0002 Roots [MangaDex]`, `c0002.5 Tea Time`, `x01 Character Sheets`, `Vol 3`, `Chapter 12 - Title`, `c0010.5[Official]`.
|
||||
|
||||
Ignored, with a warning in the log:
|
||||
- `c0012a`: a letter right after the number.
|
||||
- `Ch.12`: a dot is not a letter.
|
||||
- `Specials`: no number.
|
||||
- `- c0001`: the name must start with the key.
|
||||
|
||||
### The key
|
||||
|
||||
The number is normalised: leading zeros and trailing decimal zeros are dropped (`0010.50` → `10.5`), and an `x` prefix is kept. The result is the **key**:
|
||||
|
||||
| Folder | Key |
|
||||
|--------|-----|
|
||||
| `v01 Spring` | `1` |
|
||||
| `c0002.5 Tea Time` | `2.5` |
|
||||
| `x01 Artbook` | `x1` |
|
||||
|
||||
- **Book keys must be unique within a manga.** A second folder with the same book number is ignored and logged.
|
||||
- **Chapter id = `<book key>/<chapter key>`**, for example `1/2.5` or `x1/x1`. Moving a chapter to another book, or renumbering it, therefore makes it a different chapter: reading progress and votes stay behind, and followers are told about a "new" chapter.
|
||||
- Because the key is a number, `c0010.1` and `c0010.10` are the **same** key (`10.1`). Avoid trailing zeros in decimals.
|
||||
|
||||
### Order
|
||||
|
||||
Inside a manga, books sort by (regular before extra, number). Inside a book, chapters sort the same way. The reading order of the whole manga is books in order, each with its chapters in order:
|
||||
|
||||
```
|
||||
v01: c0001, c0002, c0002.5, x01, x02 <- extras of book 1 come after its regular chapters
|
||||
v02: c0003, c0004
|
||||
x01: x01 <- extra books come after all regular books
|
||||
```
|
||||
|
||||
### Extras
|
||||
|
||||
Something "extra" can be skipped while reading (user setting *Skip extra pages and extra chapters*) and gets an "extra" badge in the chapter list:
|
||||
|
||||
- A chapter whose letters are exactly `x` (`x01 Character Sheets`) is extra. It sits outside the continuous numbering and is listed after the regular chapters of its book.
|
||||
- Every chapter in an extra book (`x01 Specials`) is extra.
|
||||
- `"extra": true` in `chapter.json` or `book.json` marks something as extra **without moving it**. Use it for an in-order omake such as `c0010.5`.
|
||||
|
||||
## Chapter versions
|
||||
|
||||
Several chapter folders **in the same book** with the **same chapter key** are versions of one chapter, for example two scan groups or an official release plus a fan translation:
|
||||
|
||||
```
|
||||
c0002 Roots [MangaDex]/
|
||||
c0002 Roots [Official]/
|
||||
```
|
||||
|
||||
- The **version id is the folder name**. Votes and reading progress refer to it, so don't rename version folders.
|
||||
- Users vote versions up or down. The chapter opens in the version with the highest total. Ties go to the higher `"priority"` in `chapter.json` (default `0`), then to the first folder name in sort order.
|
||||
- Users can still open any other version from the chapter list or the reader's version picker.
|
||||
- Each version has its own `chapter.json`, page count and page tags. When versions have different titles, the chapter title comes from the first version (in folder order) that has one.
|
||||
- Versions without any image are ignored.
|
||||
|
||||
## Pages
|
||||
|
||||
A chapter folder contains its pages as plain images: `jpg`, `jpeg`, `png`, `webp`, `gif`, `avif` or `bmp` (extension case doesn't matter).
|
||||
|
||||
```
|
||||
<number>[_<tag>]...<.ext>
|
||||
```
|
||||
|
||||
- **Order** is the natural sort order of the file names. Zero-pad the number (`0001` … `9999`) so every picture viewer agrees. For more than 9999 pages, use 5 digits throughout.
|
||||
- **Tags** follow the number, each introduced by `_`, and are case-insensitive: `0012_double.jpg`, `0030_extra.png`, `0007_double_rot90.jpg`.
|
||||
|
||||
| Tag | Meaning | Effect in the reader |
|
||||
|-----|---------|----------------------|
|
||||
| `cover` | Chapter cover / title page. It is also **the chapter's image**: the fallback cover for the book or manga when those have none | Stands alone in Book mode. Can be skipped (*Skip chapter cover pages*) |
|
||||
| `double` | One image showing two facing pages (a spread) | Stands alone in Book mode. In Flow mode it has the height of a single page, so it is twice as wide, but never wider than the screen |
|
||||
| `extra` | Extra content: translator notes, credits, recruitment pages, ads | Can be skipped (*Skip extra pages…*) |
|
||||
| `rot90`, `rot180`, `rot270` | The image is stored turned. The viewer turns it **clockwise** by this angle | Width and height swap for `rot90` / `rot270` |
|
||||
|
||||
- A page whose (rotated) image is **wider than tall** counts as a double page even without the tag. Tag it anyway; it documents the intent.
|
||||
- Unknown tags are ignored, so tools may add their own (`0005_color.jpg`) without breaking anything.
|
||||
- A file name without a leading number (`page-a.jpg`) is still a page. It is just sorted by name, which you rarely want.
|
||||
|
||||
### Covers
|
||||
|
||||
| Level | Cover | Fallback |
|
||||
|-------|-------|----------|
|
||||
| Manga (library tile, manga page) | `cover.<ext>` in the manga folder | cover of the first book |
|
||||
| Book | `cover.<ext>` in the book folder | first `cover`-tagged page (else the first page) of the first chapter's first version |
|
||||
| Chapter | first page tagged `cover` | first page |
|
||||
|
||||
Covers are served at their original size. A cover around 600×900 px keeps the library page light.
|
||||
|
||||
## Metadata files
|
||||
|
||||
UTF-8 JSON (a BOM is tolerated). All fields are optional, and unknown fields are kept and ignored. A broken file is logged and treated as empty, so the folder still works with its name-based defaults.
|
||||
|
||||
### `manga.json`
|
||||
|
||||
| Field | Type | Meaning |
|
||||
|-------|------|---------|
|
||||
| `title` | string | Display title (default: folder name) |
|
||||
| `altTitles` | string[] | Other titles |
|
||||
| `authors` | string[] | Authors / artists |
|
||||
| `status` | string | Free text, e.g. `ongoing`, `completed`, `hiatus` |
|
||||
| `year` | number | First release year |
|
||||
| `tags` | string[] | Genres / tags |
|
||||
| `description` | string | Synopsis (line breaks are kept) |
|
||||
| `direction` | `"rtl"` \| `"ltr"` \| `"ttb"` | How the pages are read. `rtl` (the default): manga, right to left. `ltr`: comics, printed manhwa. `ttb`: webtoon, one vertical strip cut into slices (aliases `vertical`, `webtoon`) |
|
||||
| `sources` | `{name, url}[]` | Upstream sources the manga was scraped from. Several are allowed |
|
||||
|
||||
The whole file is sent to logged-in users as part of the manga page. **Don't put secrets in it** (scraper tokens, cookies).
|
||||
|
||||
### `book.json`
|
||||
|
||||
| Field | Type | Meaning |
|
||||
|-------|------|---------|
|
||||
| `title` | string | Book title (default: title part of the folder name) |
|
||||
| `direction` | as above | Overrides the manga's direction for this book |
|
||||
| `extra` | boolean | Marks all its chapters as extra without renaming the folder |
|
||||
|
||||
### `chapter.json` (one per version folder, recommended)
|
||||
|
||||
| Field | Type | Meaning |
|
||||
|-------|------|---------|
|
||||
| `title` | string | Chapter title without "Chapter N" (default: title part of the folder name) |
|
||||
| `source` | `{name, url}` or string | Where this version came from (default: `[source]` in the folder name) |
|
||||
| `added` | ISO date/time string | Release (or download) date, used for "recently updated" (default: the folder's modification time) |
|
||||
| `extra` | boolean | Marks the chapter as extra without moving it |
|
||||
| `priority` | number | Tie-breaker between versions with equal votes. Higher wins (default `0`) |
|
||||
| `direction` | as above | Overrides book/manga direction, e.g. a colour webtoon special inside a printed series |
|
||||
|
||||
`direction` is inherited **chapter → book → manga → `rtl`**, and each version can carry its own.
|
||||
|
||||
## Picture-viewer compatibility
|
||||
|
||||
- Alphabetical order of chapter folders and page files equals reading order, as long as numbers are zero-padded and separated from titles by a space.
|
||||
- In-between chapters sort correctly: `c0010 Title` < `c0010.5 Omake` < `c0011` (a space sorts before `.`).
|
||||
- Extras (`x…`) sort after the numbered folders. That is also where the app lists them.
|
||||
- The one thing a plain viewer can't do is apply `rot…` tags.
|
||||
|
||||
## Examples
|
||||
|
||||
**Manga with everything**
|
||||
|
||||
```
|
||||
Moon Garden/
|
||||
manga.json cover.png
|
||||
v01 Spring/
|
||||
book.json cover.png
|
||||
c0001 The Seed/ chapter.json 0001_cover.png 0002.png … 0006_double.png … 0012.png
|
||||
c0002 Roots [MangaDex]/ chapter.json 0001_cover.png … 0014.png
|
||||
c0002 Roots [Official]/ chapter.json 0001_cover.png … 0011.png <- second version of ch. 2
|
||||
c0002.5 Tea Time/ chapter.json 0001.png … 0004.png <- in-between chapter
|
||||
x01 Character Sheets/ chapter.json 0001_extra.png … <- extra chapter
|
||||
v02 Summer/
|
||||
c0003 The Tide/ … 0009_rot90.png … 0014_extra.png 0015_extra.png
|
||||
x01 Specials/
|
||||
x01 Artbook/ … <- extra book
|
||||
```
|
||||
|
||||
**Webtoon** (`manga.json`: `"direction": "ttb"`)
|
||||
|
||||
```
|
||||
Skyline Tower/
|
||||
v01 Season 1/
|
||||
c0001 Episode 1/ 0001.png … 0008.png <- slices of one strip, same width, no gaps
|
||||
```
|
||||
|
||||
**Bare minimum** (no JSON at all: titles and ids come from the folder names)
|
||||
|
||||
```
|
||||
Blank Slate/
|
||||
v01/
|
||||
c0001/ 0001.png 0002.png …
|
||||
```
|
||||
|
||||
**One-shot**: a manga with one book and one chapter (`Oneshot Title/v01/c0001/…`), or an extra book inside the related series (`x01 Oneshots/x01 Title/`).
|
||||
|
||||
## Renaming: what survives
|
||||
|
||||
| Change | Effect |
|
||||
|--------|--------|
|
||||
| Title part of a **book** folder (`v01 Spring` → `v01 Early Spring`) | Nothing lost |
|
||||
| Title part of a **chapter** folder (`c0003 Tide` → `c0003 The Tide`) | Same chapter (the key is unchanged), so progress and notifications are fine. The full folder name is the version id, though, so votes for that version are lost, and saved progress is re-mapped proportionally |
|
||||
| `manga.json` / `chapter.json` content | Picked up on the next scan. Nothing lost |
|
||||
| Manga folder name | New manga: progress, follows, votes and seen-chapters are gone |
|
||||
| Book number, chapter number, or moving a chapter to another book | New chapter id: progress/votes stay with the old id, and followers are notified again |
|
||||
| Version folder name | Votes for that version are lost. Progress falls back to the best version at a proportional page |
|
||||
| Page files replaced **in place** (same file names) | Cached image sizes may be stale until an admin runs **Full rescan** |
|
||||
@@ -0,0 +1,369 @@
|
||||
# Scraper guide: supplying the MangaParty library
|
||||
|
||||
**Audience:** an automated agent or program that downloads manga, comics or webtoons from upstream sources and writes them into a MangaParty library folder.
|
||||
|
||||
**Normative words:** MUST, MUST NOT, SHOULD, SHOULD NOT and MAY as in RFC 2119. Rules have ids (`R-…`) so you can cite them in logs and reviews.
|
||||
|
||||
**Background reading** (optional; this guide is self-contained):
|
||||
- [library-format.md](library-format.md): the format.
|
||||
- [import.md](import.md): how MangaParty reads it.
|
||||
|
||||
---
|
||||
|
||||
## 0. The contract in short
|
||||
|
||||
- You write **plain folders and image files**. There is no upload API.
|
||||
- MangaParty scans the library folder every `SCAN_INTERVAL` minutes (default 10), and on demand (§9).
|
||||
- The layout is `<library>/<Manga>/<Book>/<Chapter>/<pages>`, always three levels.
|
||||
- Names carry meaning:
|
||||
- the manga folder name is the manga's id,
|
||||
- the book/chapter **numbers** in folder names are ids and order,
|
||||
- the chapter folder name is the version id,
|
||||
- page file names carry order and page roles.
|
||||
- MangaParty notifies followers as soon as a new chapter number becomes visible. So a chapter MUST become visible **complete and in one step**: stage it in a hidden folder, then rename it (§10).
|
||||
- MangaParty never writes to the library. You own it; MangaParty only reads it.
|
||||
|
||||
## 1. Inputs you need
|
||||
|
||||
| Input | Required | Notes |
|
||||
|-------|----------|-------|
|
||||
| Library root path | yes | The folder MangaParty mounts as `LIBRARY` (`/library` in Docker) |
|
||||
| Upstream manga metadata | yes | Title, reading direction/type, chapter list with numbers |
|
||||
| Per chapter: ordered page images | yes | Plus, where available: which pages are covers, spreads, credits/notes |
|
||||
| MangaParty URL + an admin account | no | Only to trigger an immediate rescan (§9) |
|
||||
|
||||
## 2. Target layout (template)
|
||||
|
||||
```
|
||||
<library>/
|
||||
<Manga folder>/ R-M*
|
||||
manga.json R-J1
|
||||
cover.jpg R-M5
|
||||
v01[ <Book title>]/ R-B*
|
||||
book.json optional
|
||||
c0001[ <Chapter title>] [<Source>]/ R-C*, R-V*
|
||||
chapter.json R-J2
|
||||
0001_cover.jpg R-P*
|
||||
0002.jpg
|
||||
…
|
||||
```
|
||||
|
||||
## 3. Manga folders
|
||||
|
||||
- **R-M1** The manga folder name MUST NOT start with `.` and MUST be a valid name on Windows, macOS and Linux:
|
||||
- no `< > : " / \ | ? *` and no control characters,
|
||||
- no trailing space or dot,
|
||||
- not a reserved Windows name (`CON`, `PRN`, `AUX`, `NUL`, `COM1`–`COM9`, `LPT1`–`LPT9`),
|
||||
- Unicode normalised to NFC,
|
||||
- SHOULD be ≤ 120 characters.
|
||||
- **R-M2** Derive the name from the title once (e.g. the English or romanised title, sanitised per R-M1). After that you MUST NOT rename it: the folder name is the manga's id, and renaming loses every user's progress, follows and votes. Put title changes into `manga.json` → `title`.
|
||||
- **R-M3** Before creating a manga, you MUST look for an existing folder for the same work, so one work never ends up in two folders. Match on `manga.json` → `sources[].url` (or your own id field, R-J1). Only if nothing matches, create a new folder.
|
||||
- **R-M4** A new manga MUST be published as a whole: stage it as `<library>/.staging-<random>/` with at least one complete chapter, then rename it to its final name (§10).
|
||||
- **R-M5** Put the series cover in the manga folder as `cover.<ext>` (`jpg`, `jpeg`, `png`, `webp`, `gif`, `avif`, `bmp`). It SHOULD be at most about 1000 px tall, because it is shown in the library grid at original size. If you have no cover, omit it: the first chapter's cover page is used instead.
|
||||
- **R-M6** You MUST NOT put page images directly into the manga folder. Only `manga.json`, `cover.*` and book folders belong there.
|
||||
|
||||
## 4. Books (middle level)
|
||||
|
||||
- **R-B1** Book folder name pattern:
|
||||
- `v<NN>[ <title>]` for volumes, or `s<NN>[ <title>]` for seasons,
|
||||
- `x<NN>[ <title>]` for extra books (specials, art books, one-shot collections) that sit outside the numbering.
|
||||
- Example: `v01 Spring`, `s02 Season 2`, `x01 Specials`.
|
||||
- **R-B2** Pad numbers to 2 digits, or to 3 digits for series that can exceed 99 volumes, and keep the width the same within a manga. Re-padding a book folder later (`v01` → `v001`) is safe: the number, not the text, is the id.
|
||||
- **R-B3** Book numbers MUST be unique within a manga. MangaParty ignores a second folder with the same number.
|
||||
- **R-B4** A chapter's id includes its book number (`<book>/<chapter>`). Moving a chapter to another book later creates a "new" chapter: progress and votes stay behind, and followers get notified again. Therefore you MUST choose a book strategy per manga that never needs to move chapters:
|
||||
- **Flat (default):** every chapter goes into `v01` (for example `v01 Chapters`). Use it for ongoing series and for sources without reliable volume data.
|
||||
- **Volumes:** chapters go into their volume, **only** if the source gives a final volume number at the time you first write the chapter, and for every chapter.
|
||||
- **Seasons (webtoons):** only if the source states the season.
|
||||
Record the strategy for yourself (e.g. in your own field in `manga.json`, R-J1) so later runs follow it.
|
||||
- **R-B5** The book folder MUST contain only `book.json`, `cover.*` and chapter folders.
|
||||
|
||||
## 5. Chapters
|
||||
|
||||
- **R-C1** Chapter folder name:
|
||||
```
|
||||
c<NNNN>[.<d>][ <title>][ [<Source>]] regular chapter
|
||||
x<NN>[ <title>][ [<Source>]] extra chapter outside the numbering
|
||||
```
|
||||
- A **space** MUST follow the number when anything follows. `c0012_Title` sorts wrongly in file browsers; `c0012a` and `c0012Title` are not recognised at all.
|
||||
- Recommended check: `^(c\d{4}(\.\d*[1-9])?|x\d{2})( [^\[\]]+?)?( \[[^\[\]]+\])?$`
|
||||
- **R-C2** Pad the chapter number's integer part to 4 digits. Never change the padding of an existing manga: chapter folder names are version ids.
|
||||
- **R-C3** Map upstream numbering like this:
|
||||
|
||||
| Upstream | Folder key | Notes |
|
||||
|----------|------------|-------|
|
||||
| `12` | `c0012` | |
|
||||
| `12.5` | `c0012.5` | In-between chapter, stays in reading order |
|
||||
| `12.50` | `c0012.5` | MUST drop trailing decimal zeros. `12.10` would collide with `12.1` |
|
||||
| `12a`, `12b` | `c0012.1`, `c0012.2` | Keep "12a" in `chapter.json` → `title` if it matters. Check for collisions with real `12.1` |
|
||||
| `12 part 1/2` | `c0012.1`, `c0012.2` | As above |
|
||||
| `0`, `Prologue` | `c0000` | |
|
||||
| `Omake`, `Extra`, `Side story` placed after chapter 12 | `c0012.5` + `"extra": true` in chapter.json | In order, but skippable |
|
||||
| `Special`, `Bonus`, `Character profiles`, not tied to a position | `x01`, `x02`, … in the book they belong to | Listed after the book's regular chapters, skippable |
|
||||
| Volume extras (afterword, sketches) | `x<NN>` in that volume's book | |
|
||||
| One-shot | its own manga (`v01/c0001`), or `x<NN>` inside an `x01 Oneshots` book of the related series | |
|
||||
| Negative or non-numeric only | `x<NN>` | Never invent a number that collides with a real one |
|
||||
|
||||
- **R-C4** The title part of the folder name is for humans only.
|
||||
- The authoritative title is `chapter.json` → `title`, without the "Chapter 12:" prefix.
|
||||
- Remove `[` and `]` from titles in folder names (a trailing `[…]` is read as the source).
|
||||
- **R-C5** Chapter keys MUST be unique within a book, **except** for deliberate versions (§6).
|
||||
- **R-C6** Use the upstream's canonical chapter numbering (preferably the official one). If two sources number the same content differently, map both to one numbering; otherwise they won't be recognised as versions of each other.
|
||||
|
||||
## 6. Versions
|
||||
|
||||
- **R-V1** Each source's copy of a chapter is its own **version folder** in the same book, with the same chapter number and a source suffix:
|
||||
```
|
||||
c0012 The Storm [MangaDex]/
|
||||
c0012 The Storm [Official]/
|
||||
```
|
||||
The suffix SHOULD always be present when a manga has, or may get, more than one source.
|
||||
- **R-V2** The version folder name is the version's id, and user votes attach to it. You MUST NOT rename an existing version folder. To update its content, replace it under the **same name** (R-W4).
|
||||
- **R-V3** Set `chapter.json` → `source` = `{ "name", "url" }` for every version.
|
||||
- **R-V4** You MAY set `chapter.json` → `priority` (number, default `0`) to decide which version wins while users haven't voted, e.g. `1` for official releases. Votes always beat priority.
|
||||
- **R-V5** You MUST NOT delete another source's version to "replace" it. Add your version next to it.
|
||||
|
||||
## 7. Pages
|
||||
|
||||
- **R-P1** Page file name:
|
||||
```
|
||||
<NNNN>[_<tag>]…<.ext>
|
||||
```
|
||||
- Numbers are 4 digits, contiguous, starting at `0001`, in reading order.
|
||||
- Use 5 digits for the whole chapter if it has more than 9999 files.
|
||||
- Recommended check: `^\d{4,5}(_[a-z0-9]+)*\.(jpg|jpeg|png|webp|gif|avif|bmp)$`
|
||||
- **R-P2** Allowed formats: JPEG, PNG, WebP, GIF, AVIF, BMP. The extension MUST match the actual format (it decides the `Content-Type`).
|
||||
- SHOULD keep the upstream bytes as they are; don't re-encode.
|
||||
- Convert only formats that are not allowed.
|
||||
- **R-P3** Tags: add only what you know. When unsure, add no tag.
|
||||
|
||||
| Tag | Add when | Don't add when |
|
||||
|-----|----------|----------------|
|
||||
| `_cover` | The page is a chapter cover or title splash (usually the first page, often in colour, with the chapter title) | A normal first story page |
|
||||
| `_double` | One image shows two facing pages (a spread). Usually the width is > 1.2 × the height of normal pages | The source split the spread into two images: keep those as two normal pages, untagged, in order |
|
||||
| `_extra` | Credits, recruitment or "support us" pages, translator notes, scanlator ads, previews of other works | Story content |
|
||||
| `_rot90` / `_rot180` / `_rot270` | The source says, or you can detect with certainty, that the stored image is turned. The value is the **clockwise** angle needed to show it upright | You only guess |
|
||||
|
||||
- **R-P4** Webtoons (`direction: "ttb"`):
|
||||
- Keep the slices as delivered, in order, preferably all the same width. Don't tag spreads.
|
||||
- SHOULD NOT stitch slices into one huge image; very tall images fail on some devices.
|
||||
- SHOULD NOT cut slices smaller.
|
||||
- **R-P5** The chapter folder MUST contain only page images and `chapter.json`. Other files are ignored, and sub-folders are not read.
|
||||
- **R-P6** You SHOULD keep extra pages (tagged `_extra`) instead of deleting them; readers can skip them. You MAY drop pure advertising.
|
||||
|
||||
## 8. Metadata files
|
||||
|
||||
All are UTF-8 JSON **without** BOM, pretty-printing optional.
|
||||
|
||||
- **R-J0** Every JSON file MUST be valid JSON. A broken file is ignored with a warning; the folder then falls back to name-based defaults.
|
||||
- **R-J1** `manga.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Moon Garden",
|
||||
"altTitles": ["Tsuki no Niwa"],
|
||||
"authors": ["Aki Tanaka"],
|
||||
"status": "ongoing",
|
||||
"year": 2021,
|
||||
"tags": ["Fantasy", "Slice of Life"],
|
||||
"description": "Synopsis, line breaks allowed.",
|
||||
"direction": "rtl",
|
||||
"sources": [
|
||||
{ "name": "MangaDex", "url": "https://mangadex.org/title/…" }
|
||||
],
|
||||
"x-scraper": { "strategy": "flat", "ids": { "mangadex": "…" } }
|
||||
}
|
||||
```
|
||||
|
||||
- `direction`:
|
||||
|
||||
| Upstream type | Value |
|
||||
|---------------|-------|
|
||||
| Japanese manga | `rtl` |
|
||||
| Western comic, printed manhwa or manhua | `ltr` |
|
||||
| Webtoon / vertical scroll (also most digital manhwa and manhua) | `ttb` |
|
||||
|
||||
- `sources`: **merge**. Add your entry if its `url` is missing, update your own entry, and never remove others.
|
||||
- Unknown fields are kept and ignored by the app. Keep your own state in one namespaced field such as `"x-scraper"`.
|
||||
- The whole file is shown to logged-in users. It MUST NOT contain secrets (tokens, cookies, credentials).
|
||||
- **Read-modify-write:** keep fields you don't own, because people may have edited them. Write atomically (R-W5).
|
||||
|
||||
- **R-J2** `chapter.json`, one per version folder:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "The Storm",
|
||||
"source": { "name": "MangaDex", "url": "https://mangadex.org/chapter/…" },
|
||||
"added": "2024-03-15T10:00:00Z",
|
||||
"priority": 0,
|
||||
"extra": false
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Meaning |
|
||||
|-------|---------|
|
||||
| `added` | Upstream release date, else the download date (ISO 8601). It drives "recently updated" |
|
||||
| `extra` | Only for in-order omake (see R-C3) |
|
||||
| `direction` | Only if this chapter differs from the manga (e.g. a vertical colour special) |
|
||||
|
||||
- **R-J3** `book.json`: `{ "title": "Spring" }`. Optional; only write it if the source has volume titles.
|
||||
|
||||
## 9. Triggering the import (optional)
|
||||
|
||||
New content shows up by itself at the next scheduled scan. For immediate pickup, trigger **one** rescan after a batch, not per chapter:
|
||||
|
||||
```bash
|
||||
curl -s -c jar.txt -H 'Content-Type: application/json' -d '{"name":"<admin>","password":"<password>"}' "$MANGAPARTY/api/login"
|
||||
curl -s -b jar.txt -H 'Content-Type: application/json' -d '{}' "$MANGAPARTY/api/admin/rescan"
|
||||
```
|
||||
|
||||
- The response contains `scan.error` (null on success) and counts.
|
||||
- Use a dedicated admin account for the scraper, and keep its password out of the library.
|
||||
|
||||
## 10. Writing safely
|
||||
|
||||
MangaParty can scan at any moment. Everything it can see must always be complete.
|
||||
|
||||
- **R-W1** All staging paths MUST start with `.` and MUST be on the **same file system** as their final location; only then is `rename` atomic. Put staging folders next to the target:
|
||||
|
||||
| Target | Staging location |
|
||||
|--------|------------------|
|
||||
| new chapter or version | `<book>/.staging-<random>/` |
|
||||
| new book | the book folder itself may be created early (an empty book is ignored), then stage chapters inside it |
|
||||
| new manga | `<library>/.staging-<random>/` |
|
||||
|
||||
- **R-W2** Publish = one `rename` of the finished staging folder to its final name. Write every image and `chapter.json` before that rename.
|
||||
- **R-W3** You MUST NOT overwrite image files in place. MangaParty caches image sizes per folder mtime, and an in-place overwrite goes unnoticed. Replace whole version folders instead (R-W4).
|
||||
- **R-W4** Replacing an existing version (same folder name):
|
||||
1. stage the new copy,
|
||||
2. `rename(final → .old-<random>)`,
|
||||
3. `rename(staging → final)`,
|
||||
4. delete `.old-<random>`.
|
||||
The chapter is briefly absent, which is harmless: it won't be re-announced, and progress is kept.
|
||||
- **R-W5** Updating a JSON file: write `.<name>.tmp` in the same folder, then `rename` it over the target.
|
||||
- **R-W6** At start-up, delete your own leftover `.staging-*` / `.old-*` folders older than a few hours (from crashed runs).
|
||||
- **R-W7** On Windows, `rename` can fail with `EPERM`/`EBUSY` while another process reads the folder. Retry a few times with a short delay.
|
||||
- **R-W8** Files MUST be readable by the MangaParty process. In Docker that is uid 1000, so use mode `0644` for files and `0755` for folders.
|
||||
- **R-W9** You MUST NOT write anything into MangaParty's `DATA` folder (`db.json`, `scan-cache.json`).
|
||||
|
||||
### 10.1 Reference implementation (Node.js, no dependencies)
|
||||
|
||||
```js
|
||||
const fs = require('fs'), path = require('path'), crypto = require('crypto');
|
||||
const rnd = () => crypto.randomBytes(4).toString('hex');
|
||||
|
||||
function renameRetry(from, to, tries = 5) {
|
||||
for (let i = 0; ; i++) {
|
||||
try { return fs.renameSync(from, to); }
|
||||
catch (e) {
|
||||
if (i >= tries || !['EPERM', 'EBUSY', 'EACCES'].includes(e.code)) throw e;
|
||||
Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 200 * (i + 1)); // sleep
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// pages: [{ name: '0001_cover.jpg', data: Buffer }, …] in reading order
|
||||
function publishChapter(bookDir, folderName, pages, chapterJson) {
|
||||
fs.mkdirSync(bookDir, { recursive: true });
|
||||
const stage = path.join(bookDir, '.staging-' + rnd());
|
||||
fs.mkdirSync(stage);
|
||||
for (const p of pages) fs.writeFileSync(path.join(stage, p.name), p.data);
|
||||
fs.writeFileSync(path.join(stage, 'chapter.json'), JSON.stringify(chapterJson, null, 2) + '\n');
|
||||
const target = path.join(bookDir, folderName);
|
||||
let old = null;
|
||||
if (fs.existsSync(target)) { old = path.join(bookDir, '.old-' + rnd()); renameRetry(target, old); }
|
||||
renameRetry(stage, target);
|
||||
if (old) fs.rmSync(old, { recursive: true, force: true });
|
||||
}
|
||||
|
||||
function writeJsonAtomic(file, obj) {
|
||||
const tmp = path.join(path.dirname(file), '.' + path.basename(file) + '.tmp');
|
||||
fs.writeFileSync(tmp, JSON.stringify(obj, null, 2) + '\n');
|
||||
renameRetry(tmp, file);
|
||||
}
|
||||
```
|
||||
|
||||
## 11. Procedure per run
|
||||
|
||||
```
|
||||
for each manga to scrape:
|
||||
folder = find existing manga folder whose manga.json.sources[].url (or x-scraper id) matches # R-M3
|
||||
if none: folder = sanitize(title) # R-M1/2
|
||||
make sure it does not exist yet; if it does and belongs to another work, add a suffix
|
||||
strategy = folder's x-scraper.strategy or choose one (flat unless final volumes are known) # R-B4
|
||||
upstream_chapters = fetch chapter list
|
||||
|
||||
for each upstream chapter:
|
||||
book = strategy == flat ? "v01" : "v%02d" % volume # R-B1/2
|
||||
key = map_number(upstream number) # R-C3
|
||||
name = key + (" " + sanitize_title(title) if title) + " [" + source_name + "]" # R-C1/C4/V1
|
||||
if <book>/<name> exists and is complete and unchanged upstream: skip # idempotent
|
||||
pages = download in reading order; tag cover/double/extra/rot only when certain # R-P*
|
||||
validate(pages, name) # §12
|
||||
publishChapter(<manga>/<book>, name, pages, chapter.json) # §10
|
||||
|
||||
merge + write manga.json atomically, update cover.* if missing # R-J1, R-M5
|
||||
(new manga: do all of the above inside <library>/.staging-<rnd>/, then rename once) # R-M4
|
||||
|
||||
after the batch: optionally trigger one rescan # §9
|
||||
```
|
||||
|
||||
## 12. Validation checklist (before every publish)
|
||||
|
||||
- [ ] Manga folder name passes R-M1 and is the existing folder for this work (R-M3).
|
||||
- [ ] Book folder matches `^(v|s|x)\d{2,3}( [^\[\]]+)?$`, and its number is unique in the manga.
|
||||
- [ ] Chapter folder matches R-C1, has no trailing decimal zeros, and its number follows R-C3.
|
||||
- [ ] The same chapter number exists in this book only as deliberate versions with distinct `[Source]` suffixes.
|
||||
- [ ] ≥ 1 page. Names `0001…N` are contiguous and match R-P1. Extensions match the real formats.
|
||||
- [ ] Tags come only from `cover`, `double`, `extra`, `rot90`, `rot180`, `rot270` (plus your own on purpose).
|
||||
- [ ] At most one or two `_cover` pages, and they are at the start.
|
||||
- [ ] `chapter.json` parses, and `title` has no "Chapter N" prefix.
|
||||
- [ ] `source` is `{name, url}`, and `added` is ISO 8601.
|
||||
- [ ] No other files in the chapter folder.
|
||||
- [ ] Staging folder starts with `.` and lives next to the target.
|
||||
- [ ] Publishing is a single rename.
|
||||
- [ ] No secrets in `manga.json`.
|
||||
|
||||
## 13. What MangaParty does with your output
|
||||
|
||||
| You do | MangaParty does |
|
||||
|--------|-----------------|
|
||||
| Publish a manga folder for the first time | Adds it to the library. **No** notifications (first sight) |
|
||||
| Publish a new chapter number in a known manga | Lists it, and notifies followers in the app and by webhook within one scan |
|
||||
| Publish another version of an existing chapter | Adds it to the version picker. No notification. The best-voted (then highest `priority`) version is the default |
|
||||
| Make a chapter visible before it is complete | Indexes it as it is, notifies followers, and readers see missing pages |
|
||||
| Rename a manga folder | Treats it as a new manga. User data stays with the old name |
|
||||
| Move or renumber a chapter | Treats it as a new chapter and notifies again. Old progress and votes are orphaned |
|
||||
| Rename a version folder | Votes for it are lost |
|
||||
| Remove a chapter | It disappears from the list. It is not re-announced if it comes back |
|
||||
| Write invalid names or JSON | Skips the folder or ignores the JSON, and logs a `[scan]` warning in MangaParty's output |
|
||||
|
||||
## 14. Worked example
|
||||
|
||||
Upstream (simplified):
|
||||
|
||||
```json
|
||||
{ "title": "Moon Garden", "type": "manga", "url": "https://src.example/t/42",
|
||||
"chapters": [
|
||||
{ "num": "1", "title": "Chapter 1: The Seed", "vol": 1, "date": "2024-01-10", "pages": ["a.jpg", "b.jpg", "c.jpg"] },
|
||||
{ "num": "2", "title": "Roots", "vol": 1, "date": "2024-01-24", "pages": ["…"] },
|
||||
{ "num": "2.50", "title": "Omake: Tea Time", "vol": 1, "date": "2024-02-01", "pages": ["…"] },
|
||||
{ "num": "Special", "title": "Character Sheets", "vol": 1, "date": "2024-02-02", "pages": ["…"] }
|
||||
] }
|
||||
```
|
||||
|
||||
The volume numbers are final, so the **volumes** strategy applies. Result:
|
||||
|
||||
```
|
||||
Moon Garden/
|
||||
manga.json {"title":"Moon Garden","direction":"rtl","sources":[{"name":"SrcExample","url":"https://src.example/t/42"}],"x-scraper":{"strategy":"volumes"}}
|
||||
cover.jpg
|
||||
v01/
|
||||
c0001 The Seed [SrcExample]/ chapter.json {"title":"The Seed", …} 0001_cover.jpg 0002.jpg 0003.jpg
|
||||
c0002 Roots [SrcExample]/ chapter.json {"title":"Roots", …} 0001.jpg …
|
||||
c0002.5 Tea Time [SrcExample]/ chapter.json {"title":"Omake: Tea Time","extra":true, …}
|
||||
x01 Character Sheets [SrcExample]/ chapter.json {"title":"Character Sheets", …} 0001_extra.jpg …
|
||||
```
|
||||
|
||||
- `a.jpg` became `0001_cover.jpg` only because the upstream marks it as a cover, or it was verified to be one.
|
||||
- `2.50` became `c0002.5` (R-C3).
|
||||
- "Special" without a position became `x01` in its volume (R-C3).
|
||||
Reference in new issue
Block a user