MangaCircus/docs/scraper-service.md

Managed scraper service

The scraper is an optional Node.js daemon that runs beside the MangaCircus reader. It owns the durable queue and is the only process allowed to run a queued archive. The browser never contacts it directly: authenticated admin routes in server.js proxy commands to its private HTTP API.

In-process source modules

There is one scraper daemon/container and one durable queue. WeebCentral and MangaDex source modules are registered explicitly in scraper/adapters/index.js. Their descriptors live in scraper/sources/; they call source-specific catalog and archive code in the same process. Pure parsing functions in scraper/plugins/ handle source page/API shapes. No source parser has a separate container, HTTP endpoint, runtime download, or independent queue. Only reviewed first-party modules ship in the image; Uchiyomi packages are not executed.

The scraper's paced HTTP client fetches source content. Parsing is an ordinary function call. The registry checks source URLs and validates parsed output before the scraper follows links. WeebCentral catalog requests use a pinned-DNS HTTPS transport that validates redirects and stops on access errors. Image hosts are not restricted to a reviewed CDN allowlist yet. Existing queue state, source delays, chapter delays, and partial-download checkpoints are unchanged by this code organization. See adding a source for the full path.

Start and storage

npm run start:all       # reader + daemon under one local supervisor
npm start               # reader only
npm run start:scraper   # daemon only

These commands are for native development. Deployments run the Docker Compose stack (deployment.md); never start a native supervisor or daemon against the state a container has mounted.

Variable	Default	Purpose
SCRAPER_HOST	127.0.0.1	Daemon bind address
SCRAPER_PORT	8091	Daemon port
SCRAPER_TOKEN	empty	Bearer token; mandatory when the bind address is not loopback
SCRAPER_DATA	<DATA>/scraper	Durable queue, lock, and event log
LIBRARY	./library	Shared writable library for the daemon
SCRAPER_URL	http://127.0.0.1:8091	Daemon URL used by the reader

The authoritative queue is data/scraper/queue.json (currently schema version 4), written atomically. Individual archives retain their existing resumable checkpoints under library/.mcscraper-work-<source-id>/job.json. events.log is append-only.

The same state file also holds a sourceLedger keyed by adapter and source-series ID. It keeps source leads after completed queue rows are cleared. Existing queue URLs and recognized URLs in library metadata are backfilled without contacting a host; a successful catalog inspection records the selected chapter count, source edition, coverage signal, and observation time. Errors remain visible beside the last observation until a later successful inspection. Catalog found, adapter compatible, and download rights are separate states: a URL alone verifies none of them, and download rights remain unverified by scraper inspection. The admin-only Settings → Scraper → Sources tab shows this log. It is evidence for review, not an automatic source switch or a whole-series completeness claim.

On its first start, the daemon imports the old .mcscraper-queue/queue.json wishlist when present. Completed chapter ids in each archive checkpoint remain authoritative, so migration does not redownload published chapters.

Lifecycle

The daemon owns one durable queue and runs at most one sequential worker per source. Different sources may run side by side, but two jobs from the same source never overlap:

1. Select the first queued item for every idle, unpaused source.
2. Resolve its URL through the adapter registry in scraper/adapters/index.js.
3. Run one adapter job per source with that source's request and chapter delays.
4. For legacy in-progress checkpoints, continue publishing into their private work directory and expose the complete manga at the end.
5. For every new job, stage one chapter beside its final book and expose that complete chapter immediately with one rename.
6. Increment publishedRevision after every newly visible chapter and start the next item for that source when the selected scope is complete.

Legacy work can be converted without redownloading. While the queue is paused and no worker is active, publishLegacyReady validates every completed chapter, quarantines abandoned hidden partial stages, atomically exposes the accumulated manga folder, and changes its checkpoint to incremental mode. The resumed job then derives its remaining work from the visible library and publishes every later chapter immediately.

A source failure pauses only that source, allowing unrelated source workers to continue. Manual pause remains global. Request and chapter delays are configured independently for every installed host in Settings → Scraper. The current adapters default to the fast profile: 1–4 seconds between sequential requests and 5–15 seconds between chapters. The in-process request gate is shared across successive jobs on the same source, so changing titles does not reset the request gap.

The reader polls publishedRevision every 15 seconds and rescans after it increases. A source error pauses only that source lane for human review. Pause and shutdown requests are checked between requests; validated partial pages are retained in hidden, fingerprinted staging and reused only when the complete manifest still matches.

Inspect, confirm, and interpret completion

The admin form uses Inspect & queue (inspect:true). This queues a catalog-only kind:"inspection" item on the normal sequential source lane, at the configured delays. No image or cover requests occur during inspection. Successful inspection sets status:"ready", displays the canonical sourceTitle and source availability, and waits for Confirm download. Ready items survive daemon restarts and suppress duplicate nightly checks. Other jobs may proceed while confirmation is pending. Confirmation can save an inspected edition even while its source lane is paused; the job then remains queued and makes no source requests until that lane is reviewed and explicitly resumed. Confirmation does not resume a deliberately paused queue or clear a source block. Inspect again remains unavailable while the source lane is paused. The download rediscovers the catalog; an edition title change requires re-inspection and confirmation. Direct API clients can still enqueue authorized downloads without inspection by omitting the flag. Existing download/discovery jobs are unchanged.

Inspect again operates on an inactive ready, completed, unavailable or quarantined item. It never overrides a source access block and does not fetch external-link pages. Old MangaDex unavailable messages are marked as needing re-inspection, not silently retried. No remote diagnosis is inferred from that old numeric-only adapter result. If the item was a completed download, inspection compares its selected chapters with the shared library inventory. An already-present selection remains Complete with its original completion and publication counts; it does not become an awaiting-download item. Newly available chapters instead remain Ready for explicit confirmation. WeebCentral catalog inspection accepts numbered Chapter, Episode, Rating, Night, Log and Act labels. If chapter links exist but none can be numbered, the parser reports a label-format problem; an actually empty catalog is reported separately.

complete means the selected job finished, not that a series is fully loaded. publishedCount counts chapters newly exposed by that run; selectedCount is its remaining selected scope. Resume may reduce those counts because the inventory skips chapters already published. MangaDex availability records hosted release/selection counts, unnumbered release counts, metadata's last chapter and the highest hosted number. A lower hosted maximum proves partial coverage; an equal maximum does not prove completeness. Other cases say completeness is unverified. The last chapter number is never used as a chapter count.

Empty English hosted catalogs distinguish no English releases listed, English external-link-only releases, and no hosted English releases with pages. External availability is checked through a separate, paced includeExternalUrl=1 feed request only when hosted selections are empty; downloads use includeExternalUrl=0.

Queue scopes and duplicate policy

Every new queue item has its own UUID plus the stable upstream seriesId. This allows several jobs for one manga (for example volume 2 followed by volume 3) without sharing queue identity. Its scope is one of:

Mode	Meaning
missing	Every eligible chapter absent from the library
latest	The newest eligible missing chapter
last / first	The last/first count eligible missing chapters
chapter	One numeric chapter
after	Every numeric chapter after the supplied chapter
volume	Chapters whose source catalog explicitly reports that volume
ids	Internal exact source chapter ids produced by nightly discovery

Unnumbered MangaDex releases use special-mangadex-<chapter-uuid> as their logical key, never an invented chapter zero or one. Numbered release deduplication is unchanged; unnumbered releases stay distinct unless their source UUID is identical. Two pilot translations therefore remain two labeled, group-credited releases, not a guessed numbered chapter or silently merged translation. Numeric chapter/after scopes exclude them; missing, exact-ID and applicable volume scopes include them. Catalog ordering puts numbered chapters first and source-ID-ordered specials afterward; first/last/latest operate on that ordering, not on publication timestamps.

WeebCentral whole-volume releases use special-weebcentral-volume-<release-id> and a Volume N label rather than an invented chapter number. MangaPill is a first-party module in the same scraper process; its catalog and chapter HTML must remain on mangapill.com, and its page URLs must match the reviewed image host. It uses the ordinary source lane, pacing, checkpoint and publication rules.

existingPolicy:"skip-any" skips a chapter when any version already exists. existingPolicy:"add-source" permits another host's version but still skips a version already present from the selected host. Existing version folders are never overwritten.

Before downloading, the scraper reads a lightweight library inventory. Work identity is matched by exact source id, exact source URL, explicit library manga id, or verified AniList id—never by fuzzy title. The filesystem library remains authoritative: a stale checkpoint cannot make a missing published chapter look complete.

Nightly discovery

manga.json stores the admin override at x-scraper.autoUpdate. When it is absent, status:"ongoing" defaults to enabled and other statuses default to disabled. At the configured local autoUpdateTime, the daemon queues one catalog-only discovery item for the preferred supported source of every enabled manga. Discovery shares the normal sequential source lane and pacing. If strictly newer eligible chapters are found, that same item is promoted to a download job containing their source ids. A successful no-change check records lastCheckedAt and its result in the per-manga/source update summary, appends history, and retires the discovery queue row without page requests. lastDownloadedAt is recorded when a chapter is published or a download job completes. These summaries persist in queue.json as updateChecks; on load, legacy completed no-change discovery rows migrate into the summary and are retired. Failed checks remain visible for attention. Duplicate pending checks for the same source series are suppressed. Only numbered chapters strictly newer than the library's highest regular chapter, plus unseen source-ID specials, are considered updates. Historical numbered gaps are never backfilled by the nightly task; use an explicit queue scope when a gap is intentional work.

Private daemon API

All responses are JSON. When SCRAPER_TOKEN is set, send Authorization: Bearer <token>.

Method	Path	Body / result
GET	/health	Basic liveness
GET	/api/state	Queue, all active source checkpoints, settings, errors, history, adapters. The library summary in it is rebuilt in the background after a publish, or when it is 15 minutes old (the walk reads every chapter.json)
GET	/api/revision	{ publishedRevision, status, paused } only; the reader polls this every 15 s
POST	/api/queue	{url,title?,metadataSearch?,scope?,existingPolicy?,libraryMangaId?,inspect?}
POST	/api/queue/remove	{id}; active downloads cannot be removed
POST	/api/queue/clear-completed	{source?}; remove inactive successful download job rows only; preserve library, history, checkpoints and other statuses
POST	/api/queue/reorder	{ids} containing every queue id exactly once
POST	/api/control	`{action:"pause"
POST	/api/settings	sourceRates.{adapter}.{minMs,maxMs,chapterMinMs,chapterMaxMs}, sourcesEnabled.{weebcentral,mangadex,mangapill} (booleans; WeebCentral and MangaPill default to off, except on installations whose queue already used them; a switched-off source refuses new jobs, keeps queued ones waiting and is skipped by update checks), plus autoUpdateEnabled and local autoUpdateTime (HH:MM)
POST	/api/discovery/run	Queue an immediate update check for every enabled manga
POST	/api/library/auto-update	{id,enabled} updates manga.json atomically
POST	/api/legacy/publish-ready	Paused-only validation/publication of completed legacy work; converts its checkpoint to incremental mode

Legacy top-level rate fields remain a fallback for adapters without a per-source entry. Settings also accept autoUpdateEnabled and a local autoUpdateTime in HH:MM form.

The reader exposes only GET/POST /api/admin/scraper, protected by the existing admin session. It maps a small command allowlist to the private API and never returns the shared token.

Adapter contract

Each first-party source module is registered in scraper/adapters/index.js with:

{
  id: 'host-id',
  label: 'Human name',
  match(url) { /* return {source,id,title,url} or null */ },
  async discover(item, { settings, shouldStop, onEvent }) { /* catalog only */ },
  async run(item, {
    libraryRoot, settings, shouldStop, onPublished, onCatalog
  }) { /* publish selected chapters and return counts/final path */ }
}

Adapters discover host-specific catalogs and pages, but reuse the central pacing, checkpoint, metadata, and atomic publisher modules. They must stop on HTTP 401, 403, 429, CAPTCHA, challenge, or access-denied responses. Never add protection bypasses.

onCatalog(catalog) persists source title and optional availability before any page acquisition. The MangaDex adapter accepts only explicit title UUID URLs, verifies AniList cross-references, selects one English release per numeric chapter, preserves unnumbered source identities, credits groups and downloads sequential full-resolution MangaDex@Home pages. Special-only editions do not use fuzzy AniList lookup or AniList fallback library matching; their canonical source title names new folders. Explicit library targets remain operator choices; existing folders are never renamed.

Admin UI

Settings → Admin → Scraper polls the authenticated proxy every three seconds only while that section is active. Activity, Queue, Updates and Sources are separate local tabs with a shared source filter. The section shows:

• every active source, manga, book/season, chapter, page, and overall progress;
• source warnings and fatal errors;
• queue ordering and removal controls;
• add-by-URL, pause, resume, and retry controls;
• queue scope, exact chapter/count/volume, target manga, and alternate-source controls;
• per-manga auto-update checkboxes, nightly time, and a manual Check now action;
• last checked and last downloaded times per manga/source, with old check events collapsed into recent history instead of occupying queue space;
• a collapsed completed-jobs section with confirmation before bulk cleanup;
• persistent image and chapter delay ranges.

Rate changes apply on the next manga. To apply them to the active manga, pause and resume it; the current chapter restarts safely if it was incomplete.