# Managed scraper service

The scraper is an optional Node.js daemon that runs beside the MangaCircus reader.
It owns the durable queue and is the only process allowed to run a queued archive.
The browser never contacts it directly: authenticated admin routes in `server.js`
proxy commands to its private HTTP API.

## In-process source modules

There is one scraper daemon/container and one durable queue. WeebCentral and
MangaDex source modules are registered explicitly in `scraper/adapters/index.js`.
Their descriptors live in `scraper/sources/`; they call source-specific catalog
and archive code in the same process. Pure parsing functions in
`scraper/plugins/` handle source page/API shapes. No source parser has a separate
container, HTTP endpoint, runtime download, or independent queue. Only reviewed
first-party modules ship in the image; Uchiyomi packages are not executed.

The scraper's paced HTTP client fetches source content. Parsing is an ordinary
function call. The registry checks source URLs and validates parsed output before
the scraper follows links. WeebCentral catalog requests use a pinned-DNS HTTPS
transport that validates redirects and stops on access errors. Image hosts are
not restricted to a reviewed CDN allowlist yet. Existing queue state, source
delays, chapter delays, and partial-download checkpoints are unchanged by this
code organization. See [adding a source](source-modules.md) for the full path.

## Start and storage

```bash
npm run start:all       # reader + daemon under one local supervisor
npm start               # reader only
npm run start:scraper   # daemon only
```

These commands are for native development. Deployments run the Docker Compose stack
([`deployment.md`](deployment.md)); never start a native supervisor or daemon against
the state a container has mounted.

| Variable | Default | Purpose |
|---|---|---|
| `SCRAPER_HOST` | `127.0.0.1` | Daemon bind address |
| `SCRAPER_PORT` | `8091` | Daemon port |
| `SCRAPER_TOKEN` | empty | Bearer token; mandatory when the bind address is not loopback |
| `SCRAPER_DATA` | `<DATA>/scraper` | Durable queue, lock, and event log |
| `LIBRARY` | `./library` | Shared writable library for the daemon |
| `SCRAPER_URL` | `http://127.0.0.1:8091` | Daemon URL used by the reader |

The authoritative queue is `data/scraper/queue.json` (currently schema version 4),
written atomically. Individual
archives retain their existing resumable checkpoints under
`library/.mcscraper-work-<source-id>/job.json`. `events.log` is append-only.

The same state file also holds a `sourceLedger` keyed by adapter and source-series
ID. It keeps source leads after completed queue rows are cleared. Existing queue
URLs and recognized URLs in library metadata are backfilled without contacting a
host; a successful catalog inspection records the selected chapter count, source
edition, coverage signal, and observation time. Errors remain visible beside the
last observation until a later successful inspection. **Catalog found**, **adapter
compatible**, and **download rights** are separate states: a URL alone verifies
none of them, and download rights remain unverified by scraper inspection. The
admin-only **Settings → Scraper → Sources** tab shows this log. It is evidence for
review, not an automatic source switch or a whole-series completeness claim.

On its first start, the daemon imports the old `.mcscraper-queue/queue.json` wishlist
when present. Completed chapter ids in each archive checkpoint remain authoritative,
so migration does not redownload published chapters.

## Lifecycle

The daemon owns one durable queue and runs at most one sequential worker per source. Different sources may run side by side, but two jobs from the same source never overlap:

1. Select the first `queued` item for every idle, unpaused source.
2. Resolve its URL through the adapter registry in `scraper/adapters/index.js`.
3. Run one adapter job per source with that source's request and chapter delays.
4. For legacy in-progress checkpoints, continue publishing into their private work directory and expose the complete manga at the end.
5. For every new job, stage one chapter beside its final book and expose that complete chapter immediately with one rename.
6. Increment `publishedRevision` after every newly visible chapter and start the next item for that source when the selected scope is complete.

Legacy work can be converted without redownloading. While the queue is paused and no
worker is active, `publishLegacyReady` validates every completed chapter, quarantines
abandoned hidden partial stages, atomically exposes the accumulated manga folder, and
changes its checkpoint to incremental mode. The resumed job then derives its remaining
work from the visible library and publishes every later chapter immediately.

A source failure pauses only that source, allowing unrelated source workers to continue. Manual pause remains global. Request and chapter delays are configured independently for every installed host in **Settings → Scraper**. The current adapters default to the fast profile: 1–4 seconds between sequential requests and 5–15 seconds between chapters. The in-process request gate is shared across successive jobs on the same source, so changing titles does not reset the request gap.

The reader polls `publishedRevision` every 15 seconds and rescans after it increases.
A source error pauses only that source lane for human review. Pause and shutdown requests
are checked between requests; validated partial pages are retained in hidden,
fingerprinted staging and reused only when the complete manifest still matches.

## Inspect, confirm, and interpret completion

The admin form uses **Inspect & queue** (`inspect:true`). This queues a catalog-only
`kind:"inspection"` item on the normal sequential source lane, at the configured
delays. No image or cover requests occur during inspection. Successful inspection
sets `status:"ready"`, displays the canonical `sourceTitle` and source availability,
and waits for **Confirm download**. Ready items survive daemon restarts and suppress
duplicate nightly checks. Other jobs may proceed while confirmation is pending.
Confirmation can save an inspected edition even while its source lane is paused;
the job then remains queued and makes no source requests until that lane is
reviewed and explicitly resumed. Confirmation does not resume a deliberately
paused queue or clear a source block. **Inspect again** remains unavailable
while the source lane is paused.
The download rediscovers the catalog; an edition title change requires re-inspection
and confirmation. Direct API clients can still enqueue authorized downloads without
inspection by omitting the flag. Existing download/discovery jobs are unchanged.

**Inspect again** operates on an inactive ready, completed, unavailable or quarantined
item. It never overrides a source access block and does not fetch external-link pages.
Old MangaDex unavailable messages are marked as needing re-inspection, not silently
retried. No remote diagnosis is inferred from that old numeric-only adapter result.
If the item was a completed download, inspection compares its selected chapters with
the shared library inventory. An already-present selection remains **Complete** with
its original completion and publication counts; it does not become an awaiting-download
item. Newly available chapters instead remain **Ready** for explicit confirmation.
WeebCentral catalog inspection accepts numbered Chapter, Episode, Rating, Night,
Log and Act labels. If chapter links exist but none can be numbered, the parser
reports a label-format problem; an actually empty catalog is reported separately.

`complete` means the selected **job** finished, not that a series is fully loaded.
`publishedCount` counts chapters newly exposed by that run; `selectedCount` is its
remaining selected scope. Resume may reduce those counts because the inventory skips
chapters already published. MangaDex availability records hosted release/selection
counts, unnumbered release counts, metadata's last chapter and the highest hosted
number. A lower hosted maximum proves partial coverage; an equal maximum does **not**
prove completeness. Other cases say completeness is unverified. The last chapter
number is never used as a chapter count.

Empty English hosted catalogs distinguish no English releases listed, English
external-link-only releases, and no hosted English releases with pages. External
availability is checked through a separate, paced `includeExternalUrl=1` feed request
only when hosted selections are empty; downloads use `includeExternalUrl=0`.

## Queue scopes and duplicate policy

Every new queue item has its own UUID plus the stable upstream `seriesId`. This allows
several jobs for one manga (for example volume 2 followed by volume 3) without sharing
queue identity. Its `scope` is one of:

| Mode | Meaning |
|---|---|
| `missing` | Every eligible chapter absent from the library |
| `latest` | The newest eligible missing chapter |
| `last` / `first` | The last/first `count` eligible missing chapters |
| `chapter` | One numeric chapter |
| `after` | Every numeric chapter after the supplied chapter |
| `volume` | Chapters whose source catalog explicitly reports that volume |
| `ids` | Internal exact source chapter ids produced by nightly discovery |

Unnumbered MangaDex releases use `special-mangadex-<chapter-uuid>` as their logical
key, never an invented chapter zero or one. Numbered release deduplication is unchanged;
unnumbered releases stay distinct unless their source UUID is identical. Two pilot
translations therefore remain two labeled, group-credited releases, not a guessed
numbered chapter or silently merged translation. Numeric `chapter`/`after` scopes
exclude them; missing, exact-ID and applicable volume scopes include them. Catalog
ordering puts numbered chapters first and source-ID-ordered specials afterward;
first/last/latest operate on that ordering, not on publication timestamps.

WeebCentral whole-volume releases use `special-weebcentral-volume-<release-id>`
and a `Volume N` label rather than an invented chapter number. MangaPill is a
first-party module in the same scraper process; its catalog and chapter HTML
must remain on `mangapill.com`, and its page URLs must match the reviewed image
host. It uses the ordinary source lane, pacing, checkpoint and publication rules.

`existingPolicy:"skip-any"` skips a chapter when any version already exists.
`existingPolicy:"add-source"` permits another host's version but still skips a version
already present from the selected host. Existing version folders are never overwritten.

Before downloading, the scraper reads a lightweight library inventory. Work identity is
matched by exact source id, exact source URL, explicit library manga id, or verified
AniList id—never by fuzzy title. The filesystem library remains authoritative: a stale
checkpoint cannot make a missing published chapter look complete.

## Nightly discovery

`manga.json` stores the admin override at `x-scraper.autoUpdate`. When it is absent,
`status:"ongoing"` defaults to enabled and other statuses default to disabled. At the
configured local `autoUpdateTime`, the daemon queues one catalog-only discovery item for
the preferred supported source of every enabled manga. Discovery shares the normal
sequential source lane and pacing. If strictly newer eligible chapters are found, that
same item is promoted to a download job containing their source ids. A successful
no-change check records `lastCheckedAt` and its result in the per-manga/source
update summary, appends history, and retires the discovery queue row without page
requests. `lastDownloadedAt` is recorded when a chapter is published or a download
job completes. These summaries persist in `queue.json` as `updateChecks`; on load,
legacy completed no-change discovery rows migrate into the summary and are retired.
Failed checks remain visible for attention. Duplicate pending checks for the same
source series are suppressed.
Only numbered chapters strictly newer than the library's highest regular chapter,
plus unseen source-ID specials, are considered updates. Historical numbered gaps
are never backfilled by the nightly task; use an explicit queue
scope when a gap is intentional work.

## Private daemon API

All responses are JSON. When `SCRAPER_TOKEN` is set, send
`Authorization: Bearer <token>`.

| Method | Path | Body / result |
|---|---|---|
| `GET` | `/health` | Basic liveness |
| `GET` | `/api/state` | Queue, all active source checkpoints, settings, errors, history, adapters |
| `POST` | `/api/queue` | `{url,title?,metadataSearch?,scope?,existingPolicy?,libraryMangaId?,inspect?}` |
| `POST` | `/api/queue/remove` | `{id}`; active downloads cannot be removed |
| `POST` | `/api/queue/clear-completed` | `{source?}`; remove inactive successful download job rows only; preserve library, history, checkpoints and other statuses |
| `POST` | `/api/queue/reorder` | `{ids}` containing every queue id exactly once |
| `POST` | `/api/control` | `{action:"pause"|"resume"|"retry"|"inspect"|"confirm"|"skip",id?}`; `pause-source`/`resume-source` use `id` as the source name and do not change the global pause |
| `POST` | `/api/settings` | `sourceRates.{adapter}.{minMs,maxMs,chapterMinMs,chapterMaxMs}`, `sourcesEnabled.{weebcentral,mangadex,mangapill}` (booleans; WeebCentral and MangaPill default to off, except on installations whose queue already used them; a switched-off source refuses new jobs, keeps queued ones waiting and is skipped by update checks), plus `autoUpdateEnabled` and local `autoUpdateTime` (`HH:MM`) |
| `POST` | `/api/discovery/run` | Queue an immediate update check for every enabled manga |
| `POST` | `/api/library/auto-update` | `{id,enabled}` updates `manga.json` atomically |
| `POST` | `/api/legacy/publish-ready` | Paused-only validation/publication of completed legacy work; converts its checkpoint to incremental mode |

Legacy top-level rate fields remain a fallback for adapters without a per-source entry. Settings also accept `autoUpdateEnabled` and a local `autoUpdateTime` in `HH:MM` form.

The reader exposes only `GET/POST /api/admin/scraper`, protected by the existing
admin session. It maps a small command allowlist to the private API and never returns
the shared token.

## Adapter contract

Each first-party source module is registered in `scraper/adapters/index.js` with:

```js
{
  id: 'host-id',
  label: 'Human name',
  match(url) { /* return {source,id,title,url} or null */ },
  async discover(item, { settings, shouldStop, onEvent }) { /* catalog only */ },
  async run(item, {
    libraryRoot, settings, shouldStop, onPublished, onCatalog
  }) { /* publish selected chapters and return counts/final path */ }
}
```

Adapters discover host-specific catalogs and pages, but reuse the central pacing,
checkpoint, metadata, and atomic publisher modules. They must stop on HTTP 401, 403,
429, CAPTCHA, challenge, or access-denied responses. Never add protection bypasses.

`onCatalog(catalog)` persists source title and optional availability before any page
acquisition. The MangaDex adapter accepts only explicit title UUID URLs, verifies
AniList cross-references, selects one English release per numeric chapter, preserves
unnumbered source identities, credits groups and downloads sequential full-resolution
MangaDex@Home pages. Special-only editions do not use fuzzy AniList lookup or AniList
fallback library matching; their canonical source title names new folders. Explicit
library targets remain operator choices; existing folders are never renamed.

## Admin UI

**Settings → Admin → Scraper** polls the authenticated proxy every three seconds
only while that section is active. Activity, Queue, Updates and Sources are
separate local tabs with a shared source filter. The section shows:

- every active source, manga, book/season, chapter, page, and overall progress;
- source warnings and fatal errors;
- queue ordering and removal controls;
- add-by-URL, pause, resume, and retry controls;
- queue scope, exact chapter/count/volume, target manga, and alternate-source controls;
- per-manga auto-update checkboxes, nightly time, and a manual **Check now** action;
- last checked and last downloaded times per manga/source, with old check events
  collapsed into recent history instead of occupying queue space;
- a collapsed completed-jobs section with confirmation before bulk cleanup;
- persistent image and chapter delay ranges.

Rate changes apply on the next manga. To apply them to the active manga, pause and
resume it; the current chapter restarts safely if it was incomplete.