MaashkinoozheandClaude Opus 5.5
b2bb34174c
The stall watchdog now runs inside the scraper (ScraperService.checkStalls, every 30 s from the work loop): same rule as before, one checkpoint-safe restart, quarantine on the second stall, missing checkpoint is no evidence, and paused time no longer counts. Stalls are recorded as scraper incidents (failure.kind 'stall'). The private /api/recovery endpoint is gone. Dropped with the service: reader/scraper-unreachable incidents (supervisor and Docker handle crashes; the health page shows "Scraper: unreachable"), duplicate job-failure incidents and the unused /metrics endpoint. The stack is now reader + scraper (+ optional peer-gateway). The supervisor test moved to runtime/test (npm run test:runtime stays). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
156 lines
10 KiB
Markdown
156 lines
10 KiB
Markdown
# MangaCircus scraper agent runbook
|
||
|
||
This document is normative for automated agents operating the scraper. Read it together with `../docs/scraper-guide.md`.
|
||
|
||
## 1. Non-negotiable safety policy
|
||
|
||
1. Scrape only works the operator is authorised to copy.
|
||
2. Never bypass a CAPTCHA, Cloudflare challenge, login, paywall, geographic restriction, signed-URL check, or other access control.
|
||
3. If a normal browser session displays a challenge, stop. A human may complete it normally; the agent may resume afterward. Repeated challenges end the run.
|
||
4. Never request pages concurrently. Use the persisted per-host settings. The current operator-selected fast profile is 1–4 seconds between requests/images and 5–15 seconds between chapters. A host-specific published limit overrides these settings when it is slower.
|
||
5. Stop immediately on HTTP 401, 403, or 429. On 5xx, retry at most twice after 30 and 60 seconds. Do not rotate IP addresses, identities, user agents, or mirrors to evade a block.
|
||
6. Inspect metadata through AniList and its local 30-day cache. Do not revisit a reader/image host merely for authors, genres, descriptions, status, year, or cover metadata.
|
||
7. Download only missing chapters. Never probe every historical chapter merely to discover the newest one when a chapter list is available.
|
||
8. Publish only through the shared publisher, either `scraper/cli.js publish` or the
|
||
managed daemon. Both validate images and use MangaCircus-safe staging and rename
|
||
operations. Legacy conversion must be paused, validate every completed chapter,
|
||
quarantine incomplete stages, and use an atomic manga-folder rename.
|
||
|
||
## 2. Pipeline
|
||
|
||
Runtime recovery is documented in `../docs/resilience.md`. Request-level 5xx retries
|
||
remain limited to 30/60 seconds; after those are exhausted the daemon can resume the
|
||
same checkpoint once after a five-minute source cooldown. No automatic access-error
|
||
retry is permitted. Unavailable titles and parser failures are job-scoped; repeated
|
||
independent parser failures stop the source lane. The stall watchdog never changes adapters.
|
||
|
||
```text
|
||
public metadata -> cached series metadata
|
||
chapter host -> newest chapter IDs only
|
||
reader/browser -> ordered page manifest or local browser asset bundle
|
||
manifest validation -> slow sequential acquisition -> hidden staging folder
|
||
complete chapter -> one atomic rename -> optional one MangaCircus rescan per batch
|
||
```
|
||
|
||
The interchange format is `manifest.example.json`. Each page has exactly one of:
|
||
|
||
- `path`: an already downloaded browser asset; preferred for protected/browser-only hosts.
|
||
- `url`: a public image URL fetched sequentially by the publisher.
|
||
|
||
Do not store cookies, tokens, signed URLs, account identifiers, or other secrets in `manga.json`. Temporary manifests containing expiring URLs must stay outside the library and should be deleted after a successful import.
|
||
|
||
## 3. Metadata procedure
|
||
|
||
Run one AniList lookup per new series:
|
||
|
||
```bash
|
||
node scraper/cli.js metadata-anilist "series title"
|
||
```
|
||
|
||
Verify the result using title, native title, author and year. Copy the returned fields into `series.metadata`. Keep the chapter host URLs only in `series.sources`. If the lookup is ambiguous, stop for human selection; never silently attach metadata from a similarly named work.
|
||
|
||
For the selected Heavy Knight manga, the verified AniList identity is `150193` / MAL `147799`. AniList `150194` is the light novel and MUST NOT be used for manga metadata or cover art. For One Piece it is AniList `30013` / MAL `13`.
|
||
|
||
## 4. WeebCentral adapter
|
||
|
||
Observed 2026-10-01:
|
||
|
||
- Series: `https://weebcentral.com/series/<id>`
|
||
- Full list: `/series/<id>/full-chapter-list`
|
||
- Chapter: `/chapters/<id>`
|
||
- Page fragment: `/chapters/<id>/images?is_prev=False&reading_style=long_strip¤t_page=1`
|
||
|
||
The fragment contains the ordered image elements. Fetch the series page once, the full list once, and only the selected chapter/page fragments. Current chapter 177 exposed 15 PNG pages. Do not fetch cover art from this host when AniList metadata is available.
|
||
|
||
```bash
|
||
node scraper/cli.js discover-weebcentral "SERIES_URL" --limit 3 --out manifest.json
|
||
node scraper/cli.js publish manifest.json --library library
|
||
```
|
||
|
||
If the HTML no longer matches, stop and use a normal browser to inspect the visible reader. Do not guess endpoint grids.
|
||
|
||
## 5. ReiManga adapter
|
||
|
||
Observed 2026-10-01: anonymous scripted requests receive a managed Cloudflare challenge. The indexed series page lists chapters 177 downward, but that index is not sufficient to acquire pages.
|
||
|
||
Required approach:
|
||
|
||
1. Open the supplied series page in a normal browser.
|
||
2. If a challenge appears, stop for the human. Do not automate it.
|
||
3. After the user reaches the normal series page, open only the selected chapter.
|
||
4. Read the visible chapter number/title/date and inventory the image assets already rendered by the page.
|
||
5. Export only assets belonging to that chapter into a local temporary folder.
|
||
6. Create a manifest with local `path` entries and source name `ReiManga`.
|
||
7. Publish locally. Wait 30–60 seconds before opening another chapter.
|
||
|
||
If the site sends the chapter to another domain, record the ReiManga chapter URL as the source URL, but accept assets only from domains actually loaded by that reader page. Any login, paywall, error, or renewed challenge ends the run.
|
||
|
||
## 5A. MangaDex adapter
|
||
|
||
Observed 2026-10-03:
|
||
|
||
- Title: `https://mangadex.org/title/<uuid>`
|
||
- API: `https://api.mangadex.org`
|
||
- English feed: `/manga/<uuid>/feed`
|
||
- Page manifest: `/at-home/server/<chapter-uuid>`
|
||
- Images: the HTTPS MangaDex@Home host returned by that manifest
|
||
|
||
Only explicit MangaDex title UUIDs may enter the managed queue. Never automatically replace a queued source from fuzzy title search: verify the MangaDex AniList/MAL link against the selected work first. Select one stable release per numeric English chapter, credit MangaDex and the scanlation group in the chapter source, and ignore external-link or zero-page releases. Honor group removal requests. MangaDex-backed content must not be used to support advertising or paid access.
|
||
|
||
Unnumbered hosted releases are supported with `special-mangadex-<chapter-uuid>` keys
|
||
and human labels, never guessed chapter numbers. Keep distinct source UUIDs separate.
|
||
Use the queued catalog-only inspection and confirm the canonical edition before
|
||
downloading a new title. Empty hosted-English catalogs distinguish external-only,
|
||
no-English-listed and no-hosted-pages; none is an access error or a reason to probe
|
||
external hosts. A finished job does not prove whole-series coverage. See
|
||
`../docs/scraper-service.md` for inspection, scopes and availability semantics.
|
||
|
||
Use full-resolution `data` pages, validate that the manifest page count still matches the feed, and accept only HTTPS `uploads.mangadex.org` or `*.mangadex.network` image hosts. Keep one sequential MangaDex worker using its configured per-host rate (currently 1–4 seconds between requests and 5–15 seconds between chapters). Stop on 401, 403, or 429 exactly as for other sources; never add authentication or rotate hosts to evade a response.
|
||
|
||
## 6. MANGA Plus adapter
|
||
|
||
Observed 2026-10-01 for title `100020` (One Piece): the title page states that only the latest three chapters remain viewable. They are:
|
||
|
||
| Chapter | Viewer id | Published | Title |
|
||
|---|---:|---|---|
|
||
| 1192 | 1030379 | 2026-09-06 | We Won't Stand For It!!! |
|
||
| 1193 | 1030380 | 2026-09-13 | Still Practicing |
|
||
| 1194 | 1030381 | 2026-09-27 | All Things Are In Flux |
|
||
|
||
The normal web app calls `title_detailV4` and `manga_viewer_v3`, then renders signed `jumpg-assets*.tokyo-cdn.com` payloads. Direct non-browser API calls currently return an account-ban response. A browser asset export on 2026-10-01 produced payloads labelled `image/jpeg`, but their bytes were obfuscated and failed JPEG signature validation. Therefore:
|
||
|
||
1. Use only the normal MANGA Plus browser UI.
|
||
2. Read the title page and select the last three chapters explicitly marked viewable.
|
||
3. Open one viewer at a time, wait for its page count to stabilise, and inventory the rendered assets.
|
||
4. Export only URLs matching `/secure/title/100020/chapter/<viewer-id>/manga_page/` from that page's observed assets.
|
||
5. Validate the first exported payload by file signature before exporting or processing the rest. A `.jpg` name or HTTP content type is not proof of a JPEG.
|
||
6. If valid images are available, preserve numeric order and use local `path` entries in the import manifest. Set source name `MANGA Plus`, source URL `https://mangaplus.shueisha.co.jp/viewer/<viewer-id>`, priority `1`, direction `rtl`.
|
||
7. If payloads are obfuscated, abort the source. Do not decrypt them, extract pixels from the rendering canvas, or reconstruct chapters with automated screenshots.
|
||
8. Wait at least 30 seconds between chapters. If the UI says a chapter is unavailable, do not attempt alternate APIs or URLs.
|
||
|
||
Do not implement API protobuf decoding, image deobfuscation, signed-URL generation, device-id synthesis, canvas extraction, screenshot reconstruction, or retry identity rotation. The browser-exported file boundary plus ordinary image validation is the allowed boundary. As of the observation date, chapters 1192–1194 are identified but **not importable by this compliant adapter**.
|
||
|
||
## 7. MangaCircus publication rules
|
||
|
||
- Use the flat book strategy: `v01`.
|
||
- Chapter folders include stable source suffixes, for example `c1194 All Things Are In Flux [MANGA Plus]`.
|
||
- Existing chapters are skipped by default. `--replace` replaces the entire version folder atomically; never overwrite image files in place.
|
||
- New manga are built wholly under `.staging-mcscraper-*` and published with one rename.
|
||
- The publisher sniffs image bytes; filename extensions are never trusted.
|
||
- Page numbers are contiguous from `0001`.
|
||
- Add `cover`, `double`, `extra`, or rotation tags only when certain. Do not infer tags from aspect ratio alone.
|
||
- Trigger at most one MangaCircus rescan after the whole batch.
|
||
|
||
## 8. Abort checklist
|
||
|
||
Abort without workaround when any of these occurs:
|
||
|
||
- CAPTCHA, challenge, access-denied, rate-limit, account-ban, or login/paywall prompt.
|
||
- Page count changes during acquisition.
|
||
- Duplicate/non-numeric page order, unsupported image bytes, or an HTML error page disguised as an image.
|
||
- Metadata match is ambiguous.
|
||
- The target manga folder exists but belongs to another source identity.
|
||
- A chapter would need to move books or rename an existing version folder.
|
||
|
||
Log the URL, status, and phase, but never log cookies, query signatures, passwords, or browser storage.
|