0 Fork 0

Files

T

Codex c8dd46d0fb Avoid redundant full scan after warm restart

2026-10-08 21:58:28 +02:00

16 KiB

Raw Permalink Blame History

How the import works

MangaCircus does not copy or convert anything. "Import" means indexing the library folder in place. The scanner (library.js) walks the folder tree described in library-format.md and builds an in-memory index. The server (server.js) swaps that index in, compares it with what it has seen before, and notifies followers of new chapters. The library itself is only ever read.

                 ┌───────────── triggers ─────────────┐
                 │ start-up · shallow interval · daily full │
                 │ admin "Rescan" · admin "Full rescan"│
                 └────────────────┬───────────────────┘
                                  ▼
 data/library-index.json ──▶ immediate in-memory index at start (if valid)
 library/ ──read──▶ shallow changed-manga check or full scan   library.js
                      ├─ scanManga      (per manga folder)
                      │    ├─ book folders  → parseKey, book.json, cover
                      │    └─ chapter folders → scanVersion
                      │          ├─ chapter.json (always read)
                      │          └─ page list + image sizes (only if the folder changed)
                      └─ returns { index, cache }
                                  ▼
                      announce(index)   new chapter ids → notifications, webhook   server.js
                                  ▼
                      lib = index       (atomic swap, requests see old or new, never half)
                                  ▼
                      data/scan-cache.json + data/library-index.json (atomic)

1. Triggers

Trigger	When	Cache
Start-up	Restore the saved index immediately if its library mount is present, then check changed manga. Without a valid snapshot, scan the whole library; the page shows "Scanning…" until done	used
Interval	Shallow change check every `SCAN_INTERVAL` minutes (default 10, `0` = off); full reconciliation every `FULL_SCAN_INTERVAL` minutes (default 1440, `0` = off)	used
Rescan (Settings → Admin, or `POST /api/admin/rescan {}`)	On demand; the request returns when the scan is done	used
Full rescan (`POST /api/admin/rescan {"full":true}`)	On demand	ignored: every image header is read again

The saved index records when its last full reconciliation finished. A warm restart does a shallow check but does not reset the daily full-scan clock; if that scan is overdue, the reader schedules it within a minute. Set FULL_SCAN_INTERVAL=0 to disable automatic full reconciliations, while retaining admin Rescan.

Scans are single-flight. A trigger during a scan queues one coalesced follow-up, with a full reconciliation taking precedence over a shallow check. Publication revisions trigger a shallow check; if no changed directory is visible, the reader falls back to a full scan. In-place edits to chapter.json are found by the full reconciliation or an admin Rescan, not by shallow parent-directory stamps.

2. Walking the tree

All directory listings go through one helper:

It skips every entry whose name starts with .. That makes hidden folders the safe place for half-written chapters (see §7).
It follows symbolic links to directories.
It sorts names with a natural, case-insensitive collator (c2 before c10).

2.1 Manga folders

For each folder in the library root:

List it and read manga.json.
- A missing file becomes {}.
- Broken JSON is logged and becomes {}.
- A UTF-8 BOM is stripped.
id = folder name. title = manga.json.title, else the folder name. direction = manga.json.direction.
The cover is the first file matching cover.(jpg|jpeg|png|webp|gif|avif|bmp).
Scan its book folders (2.2).
Drop the manga (with a log line) if it ends up without chapters. An exception inside one manga skips only that manga.

2.2 Book folders

For each sub-folder:

parseKey(name) with ^([a-z]*) ?(\d+(?:\.\d+)?)(?=$|[\s_\-[(])[\s_-]*(.*)$ (case-insensitive).
- No match → ignored and logged.
- Letters exactly x → extra book.
- key = x? + parseFloat(number).
- A trailing […] is split off as the source name. The rest is the title.
A duplicate book key within the manga is ignored and logged. The first folder in sort order wins.
Read book.json:
- title overrides the folder title.
- extra: true marks the book as extra.
- direction is used for inheritance.
Cover: cover.* in the folder.
Scan its chapter folders (2.3). A book without chapters is dropped.

2.3 Chapter version folders

Each chapter folder is one version:

parseKey(name) → the chapter key; no match → ignored and logged.
stat the folder and always read chapter.json. It is small, and edits to it must show up even though they don't change the folder's mtime.
Cache check. The cache key is Manga/Book/Chapter (always /). If the cached folder mtime equals the current one, the cached page list is reused. Otherwise:
- List the files and keep those matching \.(jpe?g|png|webp|gif|avif|bmp)$.
- Read the pixel size of each (§3).
- Store { m: mtimeMs, p: [[file, width, height], …] }.
Per page, derive:
- tags: the name without extension, lower-cased, split on _, without the first part (0012_double_rot90.jpg → double, rot90),
- rotation: the first rot90|rot180|rot270 tag.
Version fields:

Field	Source
`id`	folder name
`source`, `sourceUrl`	`chapter.json.source` (`{name,url}` or a string), else the `[…]` part of the folder name
`added`	`Date.parse(chapter.json.added)`, else the folder mtime
`direction`	manual chapter → book → manga; else coherent tall-strip dimensions → `ttb`; else prior generated vertical series direction; else fallback `rtl`. Aliases: `vertical`, `webtoon` → `ttb`. Legacy plain values remain manual

A version without images is dropped silently.

2.4 Grouping versions into chapters

Versions in the same book with the same chapter key form one chapter:

Field	Rule
`id`	`<book key>/<chapter key>`, e.g. `1/2.5`, `x1/x1`
`label`	`Ch. <n>` or `Extra <n>`
`extra`	book is extra, or letters `x`, or any version's `chapter.json` has `extra: true`
`title`	first non-empty `chapter.json.title` / folder title, in folder order
`versions`	in folder order. Ranking by votes happens later, per request (§6)

The manga's updated is the newest added of all versions.

2.5 Sorting and fallbacks

Books, and chapters within a book, sort by (extra-by-x-prefix, number). extra: true from JSON does not move anything.
The manga's reading order is the concatenation of its books' chapters.
Mangas sort by title.
Covers fall back as follows:
- book → first cover-tagged page (else the first page) of the first chapter's first version,
- manga → first book's cover.

2.6 The result

index = {
  mangas: [{
    id, title, meta /* manga.json as is */, direction, cover /* relative path */, updated,
    books:    [{ key, num, x, extra, label, title, cover, chapters: [...] }],
    chapters: [{ id, book, key, num, x, extra, label, title,
                 versions: [{ id, rel, meta, source, sourceUrl, added, direction,
                              pages: [{ f, w, h, t: [tags], r }] }] }],
    chapterById: Map,
  }],
  byId: Map,
}

Everything is plain data, and the whole index lives in memory.

3. Image sizes

The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the first 256 KB and parses the header:

Format	Where the size is
PNG	IHDR (bytes 16–23)
GIF	logical screen (bytes 6–9)
BMP	DIB header (bytes 18–25)
WebP	`VP8` / `VP8L` / `VP8X` chunk
JPEG	first SOFn marker (the scan skips APPn/EXIF/ICC segments). If it isn't within 256 KB, the whole file is read once
AVIF / HEIF	first `ispe` box

If the size can't be read, it is stored as 0×0. The reader then assumes 1000×1414 (2000×1414 for double) and corrects the layout when the image arrives.
EXIF orientation is not applied. Use rot… tags instead.

4. The scan cache

Stored in DATA/scan-cache.json. It is rewritten atomically after every successful scan and holds only the folders seen in that scan, so deleted chapters drop out.
DATA/library-index.json also stores the complete rebuildable index and shallow manga/book directory stamps. It lets a restarted reader serve the dashboard and manga pages before the background filesystem check completes. Both files can be deleted safely; without the index the next startup walks the library, and without the size cache it also rereads image headers.
Invalidation is per chapter folder, by its mtime. Adding, removing or renaming a file changes the folder mtime, so that folder is re-read on the next scan.
Blind spot: overwriting an image in place (same name) usually doesn't change the folder's mtime, so the old size stays cached. Run a Full rescan after such edits, or write new files and rename them (§7).
Deleting the cache file is harmless. The next scan is simply slower.

Measured on the demo library (5 manga, 22 chapters, 213 pages):

~100 ms without cache on a local disk,
1.4 s without cache and 0.11 s with cache through a Docker Desktop bind mount on Windows.

The cost per chapter version is:

unchanged: one stat + one small JSON read,
changed: + one directory listing + ≤ 256 KB read per image.

The scan uses asynchronous I/O one file at a time, so it never blocks request handling.

5. Change detection and notifications

db.json keeps known[mangaId], the list of chapter ids ever seen for a manga. After every scan, announce(index) runs:

First sight of a manga (no known entry): record all its chapter ids and don't notify. A newly added series must not flood followers.
Otherwise, a chapter id not in known is new. It is appended; the list only grows, so a temporarily missing folder or an unmounted disk never causes a second round of notifications when it comes back.
For every user following the manga:
- one in-app notification per new chapter (text Title · Ch. 7: Thaw; at most 200 kept per user),
- if the user has a webhook URL, one POST per manga per scan, with a plain-text body (New in Moon Garden: Ch. 7 Thaw, Ch. 8 Bloom), header Title: MangaCircus, and a 10 s timeout. Failures are logged, not retried.
Opening a chapter marks its notifications as read.

What counts as "new" follows from the chapter id (book key/chapter key):

Change	Notifies?
A new chapter number appears	yes
Another version of an existing chapter appears	no
A chapter is moved to another book or renumbered	yes (new id)
A manga folder is renamed	no (it is a new manga)
A chapter folder is created before its images are complete	yes, immediately. Hence §7

6. From index to API

The index is read by the API at request time. Nothing is precomputed per user.

Endpoint	Built from
`GET /api/library`	manga list + searchable metadata (`altTitles`, `authors`, `tags`, `status`, normalized first release), the user's progress, follows and unread notifications
`GET /api/manga?m=`	manga incl. `meta` (the raw `manga.json`), books, chapters with ranked versions and the user's progress, "Continue" target
`GET /api/chapter?m=&c=&v=`	pages of the chosen version (`v`, else the best ranked), direction, prev/next chapter (skipping extras if the user wants that), the user's progress
`GET /lib/<path>`	the image file itself. Logged-in users only, image extensions only, path must stay inside `LIBRARY`. ETag + `Cache-Control: private, max-age=86400`

Version ranking happens per request:

the sum of all users' votes,
then chapter.json.priority (higher first),
then folder order.

Votes, progress and follows are keyed by manga id, chapter id and version id. They live in db.json, never in the library.

7. Consistency: writing into a live library

The scanner can run at any moment, including while a tool is still writing a chapter. Without care, it indexes a half-copied chapter, and followers get notified about it. The rules that keep this safe:

Stage in a hidden folder, publish with one rename.
- Write the chapter into a sibling folder whose name starts with . (for example v02 Summer/.incoming-c0007/). The scanner never looks at it.
- Rename it to its final name when it is complete.
- A rename within one file system is atomic: the next scan sees all of the chapter or none of it.
Replace versions the same way: stage the new folder, move the old one out (into a hidden name), move the new one in, then delete the old one.
Don't overwrite files in place (see §4). If you must, trigger a Full rescan afterwards.

On the server side:

The new index replaces the old one in a single assignment, so requests never see a half-built index.
If the whole scan fails (for example, the library folder is missing), the previous index stays active and the error is shown under Settings → Admin.

The full rule set for tools is in scraper-guide.md.

8. Operations

Trigger a rescan from a script:

curl -s -c jar.txt -H 'Content-Type: application/json' -d '{"name":"admin","password":"…"}' http://localhost:8080/api/login
curl -s -b jar.txt -H 'Content-Type: application/json' -d '{}' http://localhost:8080/api/admin/rescan

The response contains scan: { at, ms, mangas, chapters, pages, error }. Use {"full":true} for a full rescan. POST requests must send Content-Type: application/json (part of the CSRF protection).

Log lines (stdout, docker logs):

Message	Meaning / fix
`[scan] 5 manga, 22 chapters, 213 pages in 103 ms`	Scan finished
`[scan] ignoring "M/B": book folders must be named <letters><number>[ title]…`	Folder name has no valid key: letters, optional space, number, then space/end. Rename it
`[scan] ignoring "M/B/C": chapter folders must be named …`	Same for a chapter (typical: `c0012a`, `Ch.12`)
`[scan] ignoring "M/B": another book folder already uses number N`	Two book folders with the same number. Merge them or renumber one
`[scan] ignoring broken …/chapter.json: …`	Invalid JSON. The folder still works with name-based defaults
`[scan] "M" has no chapters (expected <manga>/<book>/<chapter>/<images>)`	Usually a missing book level (images or chapters directly in the manga folder)
`[scan] skipped "M": …`	Unexpected I/O error in that manga (permissions?)
`[scan] failed: …`	The library root can't be read. The old index stays active
`[notify] webhook of <user> failed: …`	The webhook URL is unreachable or returned an error

Troubleshooting:

Symptom	Likely cause
New chapter doesn't show up	Next interval hasn't come yet (rescan manually), hidden or invalid folder name, or no image files
Pages in the wrong order	Numbers not zero-padded, or a mix of padding widths
A page looks squashed or stretched after being replaced	Image overwritten in place with other dimensions. Run a Full rescan
Followers were notified about an empty or partial chapter	The chapter was written in place instead of staged and renamed
Reading progress "lost"	Manga folder renamed, or chapter moved or renumbered (new id)

16 KiB Raw Permalink Blame History Unescape Escape