MangaCircus/docs/scraper-guide.md

# Scraper guide: supplying the MangaCircus library

**Audience:** an automated agent or program that downloads manga, comics or webtoons from upstream sources and writes them into a MangaCircus library folder.

**Normative words:** MUST, MUST NOT, SHOULD, SHOULD NOT and MAY as in RFC 2119. Rules have ids (`R-…`) so you can cite them in logs and reviews.

**Background reading** (optional; this guide is self-contained):
- [library-format.md](library-format.md): the format.
- [import.md](import.md): how MangaCircus reads it.

---

## 0. The contract in short

- You write **plain folders and image files**. There is no upload API.
- MangaCircus scans the library folder every `SCAN_INTERVAL` minutes (default 10), and on demand (§9).
- The layout is `<library>/<Manga>/<Book>/<Chapter>/<pages>`, always three levels.
- Names carry meaning:
  - the manga folder name is the manga's id,
  - the book/chapter **numbers** in folder names are ids and order,
  - the chapter folder name is the version id,
  - page file names carry order and page roles.
- MangaCircus notifies followers as soon as a new chapter number becomes visible. So a chapter MUST become visible **complete and in one step**: stage it in a hidden folder, then rename it (§10).
- MangaCircus never writes to the library. You own it; MangaCircus only reads it.

## 1. Inputs you need

| Input | Required | Notes |
|-------|----------|-------|
| Library root path | yes | The folder MangaCircus mounts as `LIBRARY` (`/library` in Docker) |
| Upstream manga metadata | yes | Title, reading direction/type, chapter list with numbers |
| Per chapter: ordered page images | yes | Plus, where available: which pages are covers, spreads, credits/notes |
| MangaCircus URL + an admin account | no | Only to trigger an immediate rescan (§9) |

## 2. Target layout (template)

```
<library>/
  <Manga folder>/                         R-M*
    manga.json                            R-J1
    cover.jpg                             R-M5
    v01[ <Book title>]/                   R-B*
      book.json                           optional
      c0001[ <Chapter title>] [<Source>]/ R-C*, R-V*
        chapter.json                      R-J2
        0001_cover.jpg                    R-P*
        0002.jpg
        …
```

## 3. Manga folders

- **R-M1** The manga folder name MUST NOT start with `.` and MUST be a valid name on Windows, macOS and Linux:
  - no `< > : " / \ | ? *` and no control characters,
  - no trailing space or dot,
  - not a reserved Windows name (`CON`, `PRN`, `AUX`, `NUL`, `COM1`–`COM9`, `LPT1`–`LPT9`),
  - Unicode normalised to NFC,
  - SHOULD be ≤ 120 characters.
- **R-M2** Derive the name from the title once (e.g. the English or romanised title, sanitised per R-M1). After that you MUST NOT rename it: the folder name is the manga's id, and renaming loses every user's progress, follows and votes. Put title changes into `manga.json` → `title`.
- **R-M3** Before creating a manga, you MUST look for an existing folder for the same work, so one work never ends up in two folders. Match on `manga.json` → `sources[].url` (or your own id field, R-J1). Only if nothing matches, create a new folder.
- **R-M4** A new manga MUST be published as a whole: stage it as `<library>/.staging-<random>/` with at least one complete chapter, then rename it to its final name (§10).
- **R-M5** Put the series cover in the manga folder as `cover.<ext>` (`jpg`, `jpeg`, `png`, `webp`, `gif`, `avif`, `bmp`). It SHOULD be at most about 1000 px tall, because it is shown in the library grid at original size. If you have no cover, omit it: the first chapter's cover page is used instead.
- **R-M6** You MUST NOT put page images directly into the manga folder. Only `manga.json`, `cover.*` and book folders belong there.

## 4. Books (middle level)

- **R-B1** Book folder name pattern:
  - `v<NN>[ <title>]` for volumes, or `s<NN>[ <title>]` for seasons,
  - `x<NN>[ <title>]` for extra books (specials, art books, one-shot collections) that sit outside the numbering.
  - Example: `v01 Spring`, `s02 Season 2`, `x01 Specials`.
- **R-B2** Pad numbers to 2 digits, or to 3 digits for series that can exceed 99 volumes, and keep the width the same within a manga. Re-padding a book folder later (`v01` → `v001`) is safe: the number, not the text, is the id.
- **R-B3** Book numbers MUST be unique within a manga. MangaCircus ignores a second folder with the same number.
- **R-B4** A chapter's id includes its book number (`<book>/<chapter>`). Moving a chapter to another book later creates a "new" chapter: progress and votes stay behind, and followers get notified again. Therefore you MUST choose a book strategy per manga that never needs to move chapters:
  - **Flat (default):** every chapter goes into `v01` (for example `v01 Chapters`). Use it for ongoing series and for sources without reliable volume data.
  - **Volumes:** chapters go into their volume, **only** if the source gives a final volume number at the time you first write the chapter, and for every chapter.
  - **Seasons (webtoons):** only if the source states the season.
  Record the strategy for yourself (e.g. in your own field in `manga.json`, R-J1) so later runs follow it.
- **R-B5** The book folder MUST contain only `book.json`, `cover.*` and chapter folders.

## 5. Chapters

- **R-C1** Chapter folder name:
  ```
  c<NNNN>[.<d>][ <title>][ [<Source>]]     regular chapter
  x<NN>[ <title>][ [<Source>]]             extra chapter outside the numbering
  special-mangadex-<chapter-uuid>[ <title>][ [<Source>]]  unnumbered source release
  ```
  - A **space** MUST follow the number when anything follows. `c0012_Title` sorts wrongly in file browsers; `c0012a` and `c0012Title` are not recognised at all.
  - Recommended check: `^(c\d{4}(\.\d*[1-9])?|x\d{2})( [^\[\]]+?)?( \[[^\[\]]+\])?$`
  - For the additive `special-mangadex-` form, validate the complete UUID using
    `chapter-identity.js`. Do not apply the numeric-only check above. The key is the
    lower-case source UUID prefix, not an invented numeric chapter. Books remain numeric.
- **R-C2** Pad the chapter number's integer part to 4 digits. Never change the padding of an existing manga: chapter folder names are version ids.
- **R-C3** Map upstream numbering like this:

| Upstream | Folder key | Notes |
|----------|------------|-------|
| `12` | `c0012` | |
| `12.5` | `c0012.5` | In-between chapter, stays in reading order |
| `12.50` | `c0012.5` | MUST drop trailing decimal zeros. `12.10` would collide with `12.1` |
| `12a`, `12b` | `c0012.1`, `c0012.2` | Keep "12a" in `chapter.json` → `title` if it matters. Check for collisions with real `12.1` |
| `12 part 1/2` | `c0012.1`, `c0012.2` | As above |
| `0`, `Prologue` | `c0000` | |
| `Omake`, `Extra`, `Side story` placed after chapter 12 | `c0012.5` + `"extra": true` in chapter.json | In order, but skippable |
| `Special`, `Bonus`, `Character profiles`, not tied to a position | `x01`, `x02`, … in the book they belong to | Listed after the book's regular chapters, skippable |
| Volume extras (afterword, sketches) | `x<NN>` in that volume's book | |
| One-shot | its own manga (`v01/c0001`), or `x<NN>` inside an `x01 Oneshots` book of the related series | |
| Negative or non-numeric only | `x<NN>` | Never invent a number that collides with a real one |
| Unnumbered MangaDex release | `special-mangadex-<chapter-uuid>` | Stable upstream release identity; label Pilot, One-shot or Special; do not assign chapter 0/1 |

Unnumbered source-ID releases sort after numeric chapters. Their `chapter.json`
stores `special:true`, `label`, `sourceId` and the original `sourceNumber` (usually
null). They are not automatically skippable extras: an entire pilot may be story
content. Keep different release UUIDs separate rather than guessing that they are
the same logical chapter. Reimporting the same key is duplicate-safe. Existing
numeric and `x` folder conventions and progress IDs MUST NOT be migrated or renamed.

- **R-C4** The title part of the folder name is for humans only.
  - The authoritative title is `chapter.json` → `title`, without the "Chapter 12:" prefix.
  - Remove `[` and `]` from titles in folder names (a trailing `[…]` is read as the source).
- **R-C5** Chapter keys MUST be unique within a book, **except** for deliberate versions (§6).
- **R-C6** Use the upstream's canonical chapter numbering (preferably the official one). If two sources number the same content differently, map both to one numbering; otherwise they won't be recognised as versions of each other.

## 6. Versions

- **R-V1** Each source's copy of a chapter is its own **version folder** in the same book, with the same chapter number and a source suffix:
  ```
  c0012 The Storm [MangaDex]/
  c0012 The Storm [Official]/
  ```
  The suffix SHOULD always be present when a manga has, or may get, more than one source.
- **R-V2** The version folder name is the version's id, and user votes attach to it. You MUST NOT rename an existing version folder. To update its content, replace it under the **same name** (R-W4).
- **R-V3** Set `chapter.json` → `source` = `{ "name", "url" }` for every version.
- **R-V4** You MAY set `chapter.json` → `priority` (number, default `0`) to decide which version wins while users haven't voted, e.g. `1` for official releases. Votes always beat priority.
- **R-V5** You MUST NOT delete another source's version to "replace" it. Add your version next to it.

## 7. Pages

- **R-P1** Page file name:
  ```
  <NNNN>[_<tag>]…<.ext>
  ```
  - Numbers are 4 digits, contiguous, starting at `0001`, in reading order.
  - Use 5 digits for the whole chapter if it has more than 9999 files.
  - Recommended check: `^\d{4,5}(_[a-z0-9]+)*\.(jpg|jpeg|png|webp|gif|avif|bmp)$`
- **R-P2** Allowed formats: JPEG, PNG, WebP, GIF, AVIF, BMP. The extension MUST match the actual format (it decides the `Content-Type`).
  - SHOULD keep the upstream bytes as they are; don't re-encode.
  - Convert only formats that are not allowed.
- **R-P3** Tags: add only what you know. When unsure, add no tag.

| Tag | Add when | Don't add when |
|-----|----------|----------------|
| `_cover` | The page is a chapter cover or title splash (usually the first page, often in colour, with the chapter title) | A normal first story page |
| `_double` | One image shows two facing pages (a spread). Usually the width is > 1.2 × the height of normal pages | The source split the spread into two images: keep those as two normal pages, untagged, in order |
| `_extra` | Credits, recruitment or "support us" pages, translator notes, scanlator ads, previews of other works | Story content |
| `_rot90` / `_rot180` / `_rot270` | The source says, or you can detect with certainty, that the stored image is turned. The value is the **clockwise** angle needed to show it upright | You only guess |

- **R-P4** Webtoons (`direction: "ttb"`):
  - Keep the slices as delivered, in order, preferably all the same width. Don't tag spreads.
  - SHOULD NOT stitch slices into one huge image; very tall images fail on some devices.
  - SHOULD NOT cut slices smaller.
- **R-P5** The chapter folder MUST contain only page images and `chapter.json`. Other files are ignored, and sub-folders are not read.
- **R-P6** You SHOULD keep extra pages (tagged `_extra`) instead of deleting them; readers can skip them. You MAY drop pure advertising.

## 8. Metadata files

All are UTF-8 JSON **without** BOM, pretty-printing optional.

- **R-J0** Every JSON file MUST be valid JSON. A broken file is ignored with a warning; the folder then falls back to name-based defaults.
- **R-J1** `manga.json`

  ```json
  {
    "title": "Moon Garden",
    "altTitles": ["Tsuki no Niwa"],
    "authors": ["Aki Tanaka"],
    "status": "ongoing",
    "year": 2021,
    "tags": ["Fantasy", "Slice of Life"],
    "description": "Synopsis, line breaks allowed.",
    "direction": "rtl",
    "sources": [
      { "name": "MangaDex", "url": "https://mangadex.org/title/…" }
    ],
    "x-scraper": { "strategy": "flat", "ids": { "mangadex": "…" } }
  }
  ```

  - `direction` when an operator or reliable source explicitly provides the reading order:

    | Upstream type | Value |
    |---------------|-------|
    | Japanese manga | `rtl` |
    | Western comic, printed manhwa or manhua | `ltr` |
    | Webtoon / vertical scroll (also most digital manhwa and manhua) | `ttb` |

  - `sources`: **merge**. Add your entry if its `url` is missing, update your own entry, and never remove others.
  - Unknown fields are kept and ignored by the app. Keep your own state in one namespaced field such as `"x-scraper"`.
  - The whole file is shown to logged-in users. It MUST NOT contain secrets (tokens, cookies, credentials).
  - **Read-modify-write:** keep fields you don't own, because people may have edited them. Write atomically (R-W5).
  - **R-J4** The current adapters do not infer printed reading order from host,
    language, country, or catalog labels. They emit `rtl` with
    `"x-reading-direction":{"version":1,"value":"rtl","source":"fallback"}`.
    The publisher checks already downloaded image dimensions: at least three pages
    with height/width ≥ 4, at least 75% tall among known pages, consistent strip
    widths within 25%, and dimensions known for at least 75% of pages yield
    `direction: "ttb"` with `source: "detected"` and numeric evidence. A credits
    card can coexist with the strips. This detects layout, not content or language.
    Plain legacy directions and explicit `directionOverride` at any level are
    preserved across later imports. Do not bulk-convert old `rtl` values: a past
    scraper default and a human choice are indistinguishable. Use the reviewed
    migration tool on a named series after checking representative chapters.
    `node tools/review-reading-direction.js <absolute-library-manga-folder>
    <weebcentral-series-id> --chapter=c0056` is a read-only review. To apply a
    supported single-series correction, first pause and stop every writer and
    create a stopped-writer snapshot, then add `--apply --writers-stopped`.
    The tool verifies source identity, unchanged legacy RTL metadata, several
    chapter samples, and the requested chapter before an atomic metadata write.

- **R-J2** `chapter.json`, one per version folder:

  ```json
  {
    "title": "The Storm",
    "source": { "name": "MangaDex", "url": "https://mangadex.org/chapter/…" },
    "added": "2024-03-15T10:00:00Z",
    "priority": 0,
    "extra": false
  }
  ```

  | Field | Meaning |
  |-------|---------|
  | `added` | Upstream release date, else the download date (ISO 8601). It drives "recently updated" |
  | `extra` | Only for in-order omake (see R-C3) |
  | `direction` | Only if this chapter differs from the manga (e.g. a vertical colour special) |

- **R-J3** `book.json`: `{ "title": "Spring" }`. Optional; only write it if the source has volume titles.

## 9. Triggering the import (optional)

New content shows up by itself at the next scheduled scan. For immediate pickup, trigger **one** rescan after a batch, not per chapter:

```bash
curl -s -c jar.txt -H 'Content-Type: application/json' -d '{"name":"<admin>","password":"<password>"}' "$MANGACIRCUS/api/login"
curl -s -b jar.txt -H 'Content-Type: application/json' -d '{}' "$MANGACIRCUS/api/admin/rescan"
```

- The response contains `scan.error` (null on success) and counts.
- Use a dedicated admin account for the scraper, and keep its password out of the library.

## 10. Writing safely

MangaCircus can scan at any moment. Everything it can see must always be complete.

- **R-W1** All staging paths MUST start with `.` and MUST be on the **same file system** as their final location; only then is `rename` atomic. Put staging folders next to the target:

  | Target | Staging location |
  |--------|------------------|
  | new chapter or version | `<book>/.staging-<random>/` |
  | new book | the book folder itself may be created early (an empty book is ignored), then stage chapters inside it |
  | new manga | `<library>/.staging-<random>/` |

- **R-W2** Publish = one `rename` of the finished staging folder to its final name. Write every image and `chapter.json` before that rename.
- **R-W3** You MUST NOT overwrite image files in place. MangaCircus caches image sizes per folder mtime, and an in-place overwrite goes unnoticed. Replace whole version folders instead (R-W4).
- **R-W4** Replacing an existing version (same folder name):
  1. stage the new copy,
  2. `rename(final → .old-<random>)`,
  3. `rename(staging → final)`,
  4. delete `.old-<random>`.
  The chapter is briefly absent, which is harmless: it won't be re-announced, and progress is kept.
- **R-W5** Updating a JSON file: write `.<name>.tmp` in the same folder, then `rename` it over the target.
- **R-W6** Preserve incomplete staging folders: validated images are recovery assets,
  not disposable garbage. The managed publisher resumes a hidden folder only when
  its complete chapter/source/page-manifest fingerprint matches, and revalidates
  stored images. Legacy random stages lack this identity and require manual review;
  do not infer ownership or delete them merely because they are old. Completed
  replacement `.old-*` backups may be cleaned after successful publication.
- **R-W7** On Windows, `rename` can fail with `EPERM`/`EBUSY` while another process reads the folder. Retry a few times with a short delay.
- **R-W8** Files MUST be readable by the MangaCircus process. In Docker that is uid 1000, so use mode `0644` for files and `0755` for folders.
- **R-W9** You MUST NOT write anything into MangaCircus's `DATA` folder (`db.json`, `scan-cache.json`).

### 10.1 Reference implementation (Node.js, no dependencies)

```js
const fs = require('fs'), path = require('path'), crypto = require('crypto');
const rnd = () => crypto.randomBytes(4).toString('hex');

function renameRetry(from, to, tries = 5) {
  for (let i = 0; ; i++) {
    try { return fs.renameSync(from, to); }
    catch (e) {
      if (i >= tries || !['EPERM', 'EBUSY', 'EACCES'].includes(e.code)) throw e;
      Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 200 * (i + 1)); // sleep
    }
  }
}

// pages: [{ name: '0001_cover.jpg', data: Buffer }, …] in reading order
function publishChapter(bookDir, folderName, pages, chapterJson) {
  fs.mkdirSync(bookDir, { recursive: true });
  const stage = path.join(bookDir, '.staging-' + rnd());
  fs.mkdirSync(stage);
  for (const p of pages) fs.writeFileSync(path.join(stage, p.name), p.data);
  fs.writeFileSync(path.join(stage, 'chapter.json'), JSON.stringify(chapterJson, null, 2) + '\n');
  const target = path.join(bookDir, folderName);
  let old = null;
  if (fs.existsSync(target)) { old = path.join(bookDir, '.old-' + rnd()); renameRetry(target, old); }
  renameRetry(stage, target);
  if (old) fs.rmSync(old, { recursive: true, force: true });
}

function writeJsonAtomic(file, obj) {
  const tmp = path.join(path.dirname(file), '.' + path.basename(file) + '.tmp');
  fs.writeFileSync(tmp, JSON.stringify(obj, null, 2) + '\n');
  renameRetry(tmp, file);
}
```

## 11. Procedure per run

```
for each manga to scrape:
    folder = find existing manga folder whose manga.json.sources[].url (or x-scraper id) matches   # R-M3
    if none: folder = sanitize(title)                                                           # R-M1/2
             make sure it does not exist yet; if it does and belongs to another work, add a suffix
    strategy = folder's x-scraper.strategy or choose one (flat unless final volumes are known)  # R-B4
    upstream_chapters = fetch chapter list

    for each upstream chapter:
        book   = strategy == flat ? "v01" : "v%02d" % volume                                   # R-B1/2
        key    = map_number(upstream number)                                                     # R-C3
        name   = key + (" " + sanitize_title(title) if title) + " [" + source_name + "]"         # R-C1/C4/V1
        if <book>/<name> exists and is complete and unchanged upstream: skip                    # idempotent
        pages  = download in reading order; tag cover/double/extra/rot only when certain        # R-P*
        validate(pages, name)                                                                    # §12
        publishChapter(<manga>/<book>, name, pages, chapter.json)                                # §10

    merge + write manga.json atomically, update cover.* if missing                               # R-J1, R-M5
    (new manga: do all of the above inside <library>/.staging-<rnd>/, then rename once)          # R-M4

after the batch: optionally trigger one rescan                                                   # §9
```

## 12. Validation checklist (before every publish)

- [ ] Manga folder name passes R-M1 and is the existing folder for this work (R-M3).
- [ ] Book folder matches `^(v|s|x)\d{2,3}( [^\[\]]+)?$`, and its number is unique in the manga.
- [ ] Chapter folder matches R-C1, has no trailing decimal zeros, and its number follows R-C3.
- [ ] The same chapter number exists in this book only as deliberate versions with distinct `[Source]` suffixes.
- [ ] ≥ 1 page. Names `0001…N` are contiguous and match R-P1. Extensions match the real formats.
- [ ] Tags come only from `cover`, `double`, `extra`, `rot90`, `rot180`, `rot270` (plus your own on purpose).
- [ ] At most one or two `_cover` pages, and they are at the start.
- [ ] `chapter.json` parses, and `title` has no "Chapter N" prefix.
- [ ] `source` is `{name, url}`, and `added` is ISO 8601.
- [ ] No other files in the chapter folder.
- [ ] Staging folder starts with `.` and lives next to the target.
- [ ] Publishing is a single rename.
- [ ] No secrets in `manga.json`.

## 13. What MangaCircus does with your output

| You do | MangaCircus does |
|--------|-----------------|
| Publish a manga folder for the first time | Adds it to the library. **No** notifications (first sight) |
| Publish a new chapter number in a known manga | Lists it, and notifies followers in the app and by webhook within one scan |
| Publish another version of an existing chapter | Adds it to the version picker. No notification. The best-voted (then highest `priority`) version is the default |
| Make a chapter visible before it is complete | Indexes it as it is, notifies followers, and readers see missing pages |
| Rename a manga folder | Treats it as a new manga. User data stays with the old name |
| Move or renumber a chapter | Treats it as a new chapter and notifies again. Old progress and votes are orphaned |
| Rename a version folder | Votes for it are lost |
| Remove a chapter | It disappears from the list. It is not re-announced if it comes back |
| Write invalid names or JSON | Skips the folder or ignores the JSON, and logs a `[scan]` warning in MangaCircus's output |

## 14. Worked example

Upstream (simplified):

```json
{ "title": "Moon Garden", "type": "manga", "url": "https://src.example/t/42",
  "chapters": [
    { "num": "1",    "title": "Chapter 1: The Seed", "vol": 1, "date": "2024-01-10", "pages": ["a.jpg", "b.jpg", "c.jpg"] },
    { "num": "2",    "title": "Roots",               "vol": 1, "date": "2024-01-24", "pages": ["…"] },
    { "num": "2.50", "title": "Omake: Tea Time",      "vol": 1, "date": "2024-02-01", "pages": ["…"] },
    { "num": "Special", "title": "Character Sheets",  "vol": 1, "date": "2024-02-02", "pages": ["…"] }
  ] }
```

The volume numbers are final, so the **volumes** strategy applies. Result:

```
Moon Garden/
  manga.json            {"title":"Moon Garden","direction":"rtl","sources":[{"name":"SrcExample","url":"https://src.example/t/42"}],"x-scraper":{"strategy":"volumes"}}
  cover.jpg
  v01/
    c0001 The Seed [SrcExample]/          chapter.json {"title":"The Seed", …}      0001_cover.jpg 0002.jpg 0003.jpg
    c0002 Roots [SrcExample]/             chapter.json {"title":"Roots", …}         0001.jpg …
    c0002.5 Tea Time [SrcExample]/        chapter.json {"title":"Omake: Tea Time","extra":true, …}
    x01 Character Sheets [SrcExample]/    chapter.json {"title":"Character Sheets", …}   0001_extra.jpg …
```

- `a.jpg` became `0001_cover.jpg` only because the upstream marks it as a cover, or it was verified to be one.
- `2.50` became `c0002.5` (R-C3).
- "Special" without a position became `x01` in its volume (R-C3).