# MangaCircus scraper

The scraper uses small host adapters to create a common manifest, followed by one conservative publisher that writes the MangaCircus library atomically.

Start with [AGENT_RUNBOOK.md](AGENT_RUNBOOK.md). It contains the host-specific procedures, safety limits, and stop conditions. `manifest.example.json` documents the interchange format.

Once the managed daemon has started, `data/scraper/queue.json` is the authoritative
queue and is edited through the admin UI.

## Managed service

Run MangaCircus and the scraper daemon together:

```bash
npm run start:all
```

Or run them as two independently supervised processes:

```bash
npm start
npm run start:scraper
```

The daemon binds to `127.0.0.1:8091` by default, keeps its durable queue in
`data/scraper/queue.json`, and resumes incomplete archive checkpoints. An admin can
control it from **Settings → Scraper**. Set `SCRAPER_TOKEN` on both processes when
the daemon is not restricted to loopback.

Queue schema version 3 supports scoped jobs, multiple jobs for the same upstream
series, one sequential worker per source, incremental chapter publication, library
duplicate/version checks, and configurable nightly update discovery. See
[`docs/scraper-service.md`](../docs/scraper-service.md) for the service contract and
[`docs/deployment.md`](../docs/deployment.md) for running it in Docker. Run only one
supervisor or container against the same library and data.

```bash
npm run test:scraper
node scraper/cli.js metadata-anilist "One Piece"
node scraper/cli.js discover-weebcentral "https://weebcentral.com/series/..." --limit 3 --out manifest.json
node scraper/cli.js publish manifest.json --library library --metadata-search "Series title"
node scraper/cli.js archive-weebcentral "https://weebcentral.com/series/..." --library library --metadata-search "Series title"
node scraper/cli.js status-weebcentral "https://weebcentral.com/series/..." --library library
```

The implementation uses only Node.js built-ins. Browser-only sources must first export observed page assets to local files; the publisher consumes those files and does not automate access-control challenges.