0 Fork 0

Files

T

Maashkinoozhe eef4ffa07c Consolidate MangaCircus app, scraper, and Compose baseline

2026-10-07 23:28:20 +02:00

24 KiB

Raw Permalink Blame History

Scraper guide: supplying the MangaCircus library

Audience: an automated agent or program that downloads manga, comics or webtoons from upstream sources and writes them into a MangaCircus library folder.

Normative words: MUST, MUST NOT, SHOULD, SHOULD NOT and MAY as in RFC 2119. Rules have ids (R-…) so you can cite them in logs and reviews.

Background reading (optional; this guide is self-contained):

library-format.md: the format.
import.md: how MangaCircus reads it.

0. The contract in short

You write plain folders and image files. There is no upload API.
MangaCircus scans the library folder every SCAN_INTERVAL minutes (default 10), and on demand (§9).
The layout is <library>/<Manga>/<Book>/<Chapter>/<pages>, always three levels.
Names carry meaning:
- the manga folder name is the manga's id,
- the book/chapter numbers in folder names are ids and order,
- the chapter folder name is the version id,
- page file names carry order and page roles.
MangaCircus notifies followers as soon as a new chapter number becomes visible. So a chapter MUST become visible complete and in one step: stage it in a hidden folder, then rename it (§10).
MangaCircus never writes to the library. You own it; MangaCircus only reads it.

1. Inputs you need

Input	Required	Notes
Library root path	yes	The folder MangaCircus mounts as `LIBRARY` (`/library` in Docker)
Upstream manga metadata	yes	Title, reading direction/type, chapter list with numbers
Per chapter: ordered page images	yes	Plus, where available: which pages are covers, spreads, credits/notes
MangaCircus URL + an admin account	no	Only to trigger an immediate rescan (§9)

2. Target layout (template)

<library>/
  <Manga folder>/                         R-M*
    manga.json                            R-J1
    cover.jpg                             R-M5
    v01[ <Book title>]/                   R-B*
      book.json                           optional
      c0001[ <Chapter title>] [<Source>]/ R-C*, R-V*
        chapter.json                      R-J2
        0001_cover.jpg                    R-P*
        0002.jpg
        …

3. Manga folders

R-M1 The manga folder name MUST NOT start with . and MUST be a valid name on Windows, macOS and Linux:
- no < > : " / \ | ? * and no control characters,
- no trailing space or dot,
- not a reserved Windows name (CON, PRN, AUX, NUL, COM1–COM9, LPT1–LPT9),
- Unicode normalised to NFC,
- SHOULD be ≤ 120 characters.
R-M2 Derive the name from the title once (e.g. the English or romanised title, sanitised per R-M1). After that you MUST NOT rename it: the folder name is the manga's id, and renaming loses every user's progress, follows and votes. Put title changes into manga.json → title.
R-M3 Before creating a manga, you MUST look for an existing folder for the same work, so one work never ends up in two folders. Match on manga.json → sources[].url (or your own id field, R-J1). Only if nothing matches, create a new folder.
R-M4 A new manga MUST be published as a whole: stage it as <library>/.staging-<random>/ with at least one complete chapter, then rename it to its final name (§10).
R-M5 Put the series cover in the manga folder as cover.<ext> (jpg, jpeg, png, webp, gif, avif, bmp). It SHOULD be at most about 1000 px tall, because it is shown in the library grid at original size. If you have no cover, omit it: the first chapter's cover page is used instead.
R-M6 You MUST NOT put page images directly into the manga folder. Only manga.json, cover.* and book folders belong there.

4. Books (middle level)

R-B1 Book folder name pattern:
- v<NN>[ <title>] for volumes, or s<NN>[ <title>] for seasons,
- x<NN>[ <title>] for extra books (specials, art books, one-shot collections) that sit outside the numbering.
- Example: v01 Spring, s02 Season 2, x01 Specials.
R-B2 Pad numbers to 2 digits, or to 3 digits for series that can exceed 99 volumes, and keep the width the same within a manga. Re-padding a book folder later (v01 → v001) is safe: the number, not the text, is the id.
R-B3 Book numbers MUST be unique within a manga. MangaCircus ignores a second folder with the same number.
R-B4 A chapter's id includes its book number (<book>/<chapter>). Moving a chapter to another book later creates a "new" chapter: progress and votes stay behind, and followers get notified again. Therefore you MUST choose a book strategy per manga that never needs to move chapters:
- Flat (default): every chapter goes into v01 (for example v01 Chapters). Use it for ongoing series and for sources without reliable volume data.
- Volumes: chapters go into their volume, only if the source gives a final volume number at the time you first write the chapter, and for every chapter.
- Seasons (webtoons): only if the source states the season. Record the strategy for yourself (e.g. in your own field in manga.json, R-J1) so later runs follow it.
R-B5 The book folder MUST contain only book.json, cover.* and chapter folders.

5. Chapters

R-C1 Chapter folder name:
```
c<NNNN>[.<d>][ <title>][ [<Source>]]     regular chapter
x<NN>[ <title>][ [<Source>]]             extra chapter outside the numbering
special-mangadex-<chapter-uuid>[ <title>][ [<Source>]]  unnumbered source release
```
- A space MUST follow the number when anything follows. c0012_Title sorts wrongly in file browsers; c0012a and c0012Title are not recognised at all.
- Recommended check: ^(c\d{4}(\.\d*[1-9])?|x\d{2})( [^\[\]]+?)?( \[[^\[\]]+\])?$
- For the additive special-mangadex- form, validate the complete UUID using chapter-identity.js. Do not apply the numeric-only check above. The key is the lower-case source UUID prefix, not an invented numeric chapter. Books remain numeric.
R-C2 Pad the chapter number's integer part to 4 digits. Never change the padding of an existing manga: chapter folder names are version ids.
R-C3 Map upstream numbering like this:

Upstream	Folder key	Notes
`12`	`c0012`
`12.5`	`c0012.5`	In-between chapter, stays in reading order
`12.50`	`c0012.5`	MUST drop trailing decimal zeros. `12.10` would collide with `12.1`
`12a`, `12b`	`c0012.1`, `c0012.2`	Keep "12a" in `chapter.json` → `title` if it matters. Check for collisions with real `12.1`
`12 part 1/2`	`c0012.1`, `c0012.2`	As above
`0`, `Prologue`	`c0000`
`Omake`, `Extra`, `Side story` placed after chapter 12	`c0012.5` + `"extra": true` in chapter.json	In order, but skippable
`Special`, `Bonus`, `Character profiles`, not tied to a position	`x01`, `x02`, … in the book they belong to	Listed after the book's regular chapters, skippable
Volume extras (afterword, sketches)	`x<NN>` in that volume's book
One-shot	its own manga (`v01/c0001`), or `x<NN>` inside an `x01 Oneshots` book of the related series
Negative or non-numeric only	`x<NN>`	Never invent a number that collides with a real one
Unnumbered MangaDex release	`special-mangadex-<chapter-uuid>`	Stable upstream release identity; label Pilot, One-shot or Special; do not assign chapter 0/1

Unnumbered source-ID releases sort after numeric chapters. Their chapter.json stores special:true, label, sourceId and the original sourceNumber (usually null). They are not automatically skippable extras: an entire pilot may be story content. Keep different release UUIDs separate rather than guessing that they are the same logical chapter. Reimporting the same key is duplicate-safe. Existing numeric and x folder conventions and progress IDs MUST NOT be migrated or renamed.

R-C4 The title part of the folder name is for humans only.
- The authoritative title is chapter.json → title, without the "Chapter 12:" prefix.
- Remove [ and ] from titles in folder names (a trailing […] is read as the source).
R-C5 Chapter keys MUST be unique within a book, except for deliberate versions (§6).
R-C6 Use the upstream's canonical chapter numbering (preferably the official one). If two sources number the same content differently, map both to one numbering; otherwise they won't be recognised as versions of each other.

6. Versions

R-V1 Each source's copy of a chapter is its own version folder in the same book, with the same chapter number and a source suffix:
```
c0012 The Storm [MangaDex]/
c0012 The Storm [Official]/
```
The suffix SHOULD always be present when a manga has, or may get, more than one source.
R-V2 The version folder name is the version's id, and user votes attach to it. You MUST NOT rename an existing version folder. To update its content, replace it under the same name (R-W4).
R-V3 Set chapter.json → source = { "name", "url" } for every version.
R-V4 You MAY set chapter.json → priority (number, default 0) to decide which version wins while users haven't voted, e.g. 1 for official releases. Votes always beat priority.
R-V5 You MUST NOT delete another source's version to "replace" it. Add your version next to it.

7. Pages

R-P1 Page file name:
```
<NNNN>[_<tag>]…<.ext>
```
- Numbers are 4 digits, contiguous, starting at 0001, in reading order.
- Use 5 digits for the whole chapter if it has more than 9999 files.
- Recommended check: ^\d{4,5}(_[a-z0-9]+)*\.(jpg|jpeg|png|webp|gif|avif|bmp)$
R-P2 Allowed formats: JPEG, PNG, WebP, GIF, AVIF, BMP. The extension MUST match the actual format (it decides the Content-Type).
- SHOULD keep the upstream bytes as they are; don't re-encode.
- Convert only formats that are not allowed.
R-P3 Tags: add only what you know. When unsure, add no tag.

Tag	Add when	Don't add when
`_cover`	The page is a chapter cover or title splash (usually the first page, often in colour, with the chapter title)	A normal first story page
`_double`	One image shows two facing pages (a spread). Usually the width is > 1.2 × the height of normal pages	The source split the spread into two images: keep those as two normal pages, untagged, in order
`_extra`	Credits, recruitment or "support us" pages, translator notes, scanlator ads, previews of other works	Story content
`_rot90` / `_rot180` / `_rot270`	The source says, or you can detect with certainty, that the stored image is turned. The value is the clockwise angle needed to show it upright	You only guess

R-P4 Webtoons (direction: "ttb"):
- Keep the slices as delivered, in order, preferably all the same width. Don't tag spreads.
- SHOULD NOT stitch slices into one huge image; very tall images fail on some devices.
- SHOULD NOT cut slices smaller.
R-P5 The chapter folder MUST contain only page images and chapter.json. Other files are ignored, and sub-folders are not read.
R-P6 You SHOULD keep extra pages (tagged _extra) instead of deleting them; readers can skip them. You MAY drop pure advertising.

8. Metadata files

All are UTF-8 JSON without BOM, pretty-printing optional.

R-J0 Every JSON file MUST be valid JSON. A broken file is ignored with a warning; the folder then falls back to name-based defaults.
R-J1 manga.json
```
{
  "title": "Moon Garden",
  "altTitles": ["Tsuki no Niwa"],
  "authors": ["Aki Tanaka"],
  "status": "ongoing",
  "year": 2021,
  "tags": ["Fantasy", "Slice of Life"],
  "description": "Synopsis, line breaks allowed.",
  "direction": "rtl",
  "sources": [
    { "name": "MangaDex", "url": "https://mangadex.org/title/…" }
  ],
  "x-scraper": { "strategy": "flat", "ids": { "mangadex": "…" } }
}
```
- direction when an operator or reliable source explicitly provides the reading order:
  
  Upstream type Value
  
  Japanese manga rtl
  
  Western comic, printed manhwa or manhua ltr
  
  Webtoon / vertical scroll (also most digital manhwa and manhua) ttb
- sources: merge. Add your entry if its url is missing, update your own entry, and never remove others.
- Unknown fields are kept and ignored by the app. Keep your own state in one namespaced field such as "x-scraper".
- The whole file is shown to logged-in users. It MUST NOT contain secrets (tokens, cookies, credentials).
- Read-modify-write: keep fields you don't own, because people may have edited them. Write atomically (R-W5).
- R-J4 The current adapters do not infer printed reading order from host, language, country, or catalog labels. They emit rtl with "x-reading-direction":{"version":1,"value":"rtl","source":"fallback"}. The publisher checks already downloaded image dimensions: at least three pages with height/width ≥ 4, at least 75% tall among known pages, consistent strip widths within 25%, and dimensions known for at least 75% of pages yield direction: "ttb" with source: "detected" and numeric evidence. A credits card can coexist with the strips. This detects layout, not content or language. Plain legacy directions and explicit directionOverride at any level are preserved across later imports. Do not bulk-convert old rtl values: a past scraper default and a human choice are indistinguishable. Use the reviewed migration tool on a named series after checking representative chapters. node tools/review-reading-direction.js <absolute-library-manga-folder> <weebcentral-series-id> --chapter=c0056 is a read-only review. To apply a supported single-series correction, first pause and stop every writer and create a stopped-writer snapshot, then add --apply --writers-stopped. The tool verifies source identity, unchanged legacy RTL metadata, several chapter samples, and the requested chapter before an atomic metadata write.

Upstream type	Value
Japanese manga	`rtl`
Western comic, printed manhwa or manhua	`ltr`
Webtoon / vertical scroll (also most digital manhwa and manhua)	`ttb`

R-J2 chapter.json, one per version folder:

{
  "title": "The Storm",
  "source": { "name": "MangaDex", "url": "https://mangadex.org/chapter/…" },
  "added": "2024-03-15T10:00:00Z",
  "priority": 0,
  "extra": false
}

Field	Meaning
`added`	Upstream release date, else the download date (ISO 8601). It drives "recently updated"
`extra`	Only for in-order omake (see R-C3)
`direction`	Only if this chapter differs from the manga (e.g. a vertical colour special)

R-J3 book.json: { "title": "Spring" }. Optional; only write it if the source has volume titles.

9. Triggering the import (optional)

New content shows up by itself at the next scheduled scan. For immediate pickup, trigger one rescan after a batch, not per chapter:

curl -s -c jar.txt -H 'Content-Type: application/json' -d '{"name":"<admin>","password":"<password>"}' "$MANGACIRCUS/api/login"
curl -s -b jar.txt -H 'Content-Type: application/json' -d '{}' "$MANGACIRCUS/api/admin/rescan"

The response contains scan.error (null on success) and counts.
Use a dedicated admin account for the scraper, and keep its password out of the library.

10. Writing safely

MangaCircus can scan at any moment. Everything it can see must always be complete.

R-W1 All staging paths MUST start with . and MUST be on the same file system as their final location; only then is rename atomic. Put staging folders next to the target:

Target	Staging location
new chapter or version	`<book>/.staging-<random>/`
new book	the book folder itself may be created early (an empty book is ignored), then stage chapters inside it
new manga	`<library>/.staging-<random>/`

R-W2 Publish = one rename of the finished staging folder to its final name. Write every image and chapter.json before that rename.
R-W3 You MUST NOT overwrite image files in place. MangaCircus caches image sizes per folder mtime, and an in-place overwrite goes unnoticed. Replace whole version folders instead (R-W4).
R-W4 Replacing an existing version (same folder name):
1. stage the new copy,
2. rename(final → .old-<random>),
3. rename(staging → final),
4. delete .old-<random>. The chapter is briefly absent, which is harmless: it won't be re-announced, and progress is kept.
R-W5 Updating a JSON file: write .<name>.tmp in the same folder, then rename it over the target.
R-W6 Preserve incomplete staging folders: validated images are recovery assets, not disposable garbage. The managed publisher resumes a hidden folder only when its complete chapter/source/page-manifest fingerprint matches, and revalidates stored images. Legacy random stages lack this identity and require manual review; do not infer ownership or delete them merely because they are old. Completed replacement .old-* backups may be cleaned after successful publication.
R-W7 On Windows, rename can fail with EPERM/EBUSY while another process reads the folder. Retry a few times with a short delay.
R-W8 Files MUST be readable by the MangaCircus process. In Docker that is uid 1000, so use mode 0644 for files and 0755 for folders.
R-W9 You MUST NOT write anything into MangaCircus's DATA folder (db.json, scan-cache.json).

10.1 Reference implementation (Node.js, no dependencies)

const fs = require('fs'), path = require('path'), crypto = require('crypto');
const rnd = () => crypto.randomBytes(4).toString('hex');

function renameRetry(from, to, tries = 5) {
  for (let i = 0; ; i++) {
    try { return fs.renameSync(from, to); }
    catch (e) {
      if (i >= tries || !['EPERM', 'EBUSY', 'EACCES'].includes(e.code)) throw e;
      Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0, 200 * (i + 1)); // sleep
    }
  }
}

// pages: [{ name: '0001_cover.jpg', data: Buffer }, …] in reading order
function publishChapter(bookDir, folderName, pages, chapterJson) {
  fs.mkdirSync(bookDir, { recursive: true });
  const stage = path.join(bookDir, '.staging-' + rnd());
  fs.mkdirSync(stage);
  for (const p of pages) fs.writeFileSync(path.join(stage, p.name), p.data);
  fs.writeFileSync(path.join(stage, 'chapter.json'), JSON.stringify(chapterJson, null, 2) + '\n');
  const target = path.join(bookDir, folderName);
  let old = null;
  if (fs.existsSync(target)) { old = path.join(bookDir, '.old-' + rnd()); renameRetry(target, old); }
  renameRetry(stage, target);
  if (old) fs.rmSync(old, { recursive: true, force: true });
}

function writeJsonAtomic(file, obj) {
  const tmp = path.join(path.dirname(file), '.' + path.basename(file) + '.tmp');
  fs.writeFileSync(tmp, JSON.stringify(obj, null, 2) + '\n');
  renameRetry(tmp, file);
}

11. Procedure per run

for each manga to scrape:
    folder = find existing manga folder whose manga.json.sources[].url (or x-scraper id) matches   # R-M3
    if none: folder = sanitize(title)                                                           # R-M1/2
             make sure it does not exist yet; if it does and belongs to another work, add a suffix
    strategy = folder's x-scraper.strategy or choose one (flat unless final volumes are known)  # R-B4
    upstream_chapters = fetch chapter list

    for each upstream chapter:
        book   = strategy == flat ? "v01" : "v%02d" % volume                                   # R-B1/2
        key    = map_number(upstream number)                                                     # R-C3
        name   = key + (" " + sanitize_title(title) if title) + " [" + source_name + "]"         # R-C1/C4/V1
        if <book>/<name> exists and is complete and unchanged upstream: skip                    # idempotent
        pages  = download in reading order; tag cover/double/extra/rot only when certain        # R-P*
        validate(pages, name)                                                                    # §12
        publishChapter(<manga>/<book>, name, pages, chapter.json)                                # §10

    merge + write manga.json atomically, update cover.* if missing                               # R-J1, R-M5
    (new manga: do all of the above inside <library>/.staging-<rnd>/, then rename once)          # R-M4

after the batch: optionally trigger one rescan                                                   # §9

12. Validation checklist (before every publish)

Manga folder name passes R-M1 and is the existing folder for this work (R-M3).
Book folder matches ^(v|s|x)\d{2,3}( [^\[\]]+)?$, and its number is unique in the manga.
Chapter folder matches R-C1, has no trailing decimal zeros, and its number follows R-C3.
The same chapter number exists in this book only as deliberate versions with distinct [Source] suffixes.
≥ 1 page. Names 0001…N are contiguous and match R-P1. Extensions match the real formats.
Tags come only from cover, double, extra, rot90, rot180, rot270 (plus your own on purpose).
At most one or two _cover pages, and they are at the start.
chapter.json parses, and title has no "Chapter N" prefix.
source is {name, url}, and added is ISO 8601.
No other files in the chapter folder.
Staging folder starts with . and lives next to the target.
Publishing is a single rename.
No secrets in manga.json.

13. What MangaCircus does with your output

You do	MangaCircus does
Publish a manga folder for the first time	Adds it to the library. No notifications (first sight)
Publish a new chapter number in a known manga	Lists it, and notifies followers in the app and by webhook within one scan
Publish another version of an existing chapter	Adds it to the version picker. No notification. The best-voted (then highest `priority`) version is the default
Make a chapter visible before it is complete	Indexes it as it is, notifies followers, and readers see missing pages
Rename a manga folder	Treats it as a new manga. User data stays with the old name
Move or renumber a chapter	Treats it as a new chapter and notifies again. Old progress and votes are orphaned
Rename a version folder	Votes for it are lost
Remove a chapter	It disappears from the list. It is not re-announced if it comes back
Write invalid names or JSON	Skips the folder or ignores the JSON, and logs a `[scan]` warning in MangaCircus's output

14. Worked example

Upstream (simplified):

{ "title": "Moon Garden", "type": "manga", "url": "https://src.example/t/42",
  "chapters": [
    { "num": "1",    "title": "Chapter 1: The Seed", "vol": 1, "date": "2024-01-10", "pages": ["a.jpg", "b.jpg", "c.jpg"] },
    { "num": "2",    "title": "Roots",               "vol": 1, "date": "2024-01-24", "pages": ["…"] },
    { "num": "2.50", "title": "Omake: Tea Time",      "vol": 1, "date": "2024-02-01", "pages": ["…"] },
    { "num": "Special", "title": "Character Sheets",  "vol": 1, "date": "2024-02-02", "pages": ["…"] }
  ] }

The volume numbers are final, so the volumes strategy applies. Result:

Moon Garden/
  manga.json            {"title":"Moon Garden","direction":"rtl","sources":[{"name":"SrcExample","url":"https://src.example/t/42"}],"x-scraper":{"strategy":"volumes"}}
  cover.jpg
  v01/
    c0001 The Seed [SrcExample]/          chapter.json {"title":"The Seed", …}      0001_cover.jpg 0002.jpg 0003.jpg
    c0002 Roots [SrcExample]/             chapter.json {"title":"Roots", …}         0001.jpg …
    c0002.5 Tea Time [SrcExample]/        chapter.json {"title":"Omake: Tea Time","extra":true, …}
    x01 Character Sheets [SrcExample]/    chapter.json {"title":"Character Sheets", …}   0001_extra.jpg …

a.jpg became 0001_cover.jpg only because the upstream marks it as a cover, or it was verified to be one.
2.50 became c0002.5 (R-C3).
"Special" without a position became x01 in its volume (R-C3).

24 KiB Raw Permalink Blame History Unescape Escape