49 lines
2.3 KiB
Markdown
49 lines
2.3 KiB
Markdown
# MangaCircus scraper
|
|
|
|
The scraper uses small host adapters to create a common manifest, followed by one conservative publisher that writes the MangaCircus library atomically.
|
|
|
|
Start with [AGENT_RUNBOOK.md](AGENT_RUNBOOK.md). It contains the host-specific procedures, safety limits, and stop conditions. `manifest.example.json` documents the interchange format.
|
|
|
|
The historical human-readable wishlist is [WISHLIST.md](WISHLIST.md). Once the
|
|
managed daemon has started, `data/scraper/queue.json` is the authoritative queue and
|
|
is edited through the admin UI.
|
|
|
|
## Managed service
|
|
|
|
Run MangaCircus and the scraper daemon together:
|
|
|
|
```bash
|
|
npm run start:all
|
|
```
|
|
|
|
Or run them as two independently supervised processes:
|
|
|
|
```bash
|
|
npm start
|
|
npm run start:scraper
|
|
```
|
|
|
|
The daemon binds to `127.0.0.1:8091` by default, keeps its durable queue in
|
|
`data/scraper/queue.json`, and resumes incomplete archive checkpoints. An admin can
|
|
control it from **Settings → Scraper**. Set `SCRAPER_TOKEN` on both processes when
|
|
the daemon is not restricted to loopback.
|
|
|
|
Queue schema version 3 supports scoped jobs, multiple jobs for the same upstream
|
|
series, one sequential worker per source, incremental chapter publication, library
|
|
duplicate/version checks, and configurable nightly update discovery. See
|
|
[`docs/scraper-service.md`](../docs/scraper-service.md) for the service contract and
|
|
[`docs/local-deployment.md`](../docs/local-deployment.md) for the currently running
|
|
Windows Scheduled Task deployment. On that machine the scheduled task owns the single
|
|
supervisor; do not run `npm run start:all` beside it.
|
|
|
|
```bash
|
|
npm run test:scraper
|
|
node scraper/cli.js metadata-anilist "One Piece"
|
|
node scraper/cli.js discover-weebcentral "https://weebcentral.com/series/..." --limit 3 --out manifest.json
|
|
node scraper/cli.js publish manifest.json --library library --metadata-search "Series title"
|
|
node scraper/cli.js archive-weebcentral "https://weebcentral.com/series/..." --library library --metadata-search "Series title"
|
|
node scraper/cli.js status-weebcentral "https://weebcentral.com/series/..." --library library
|
|
```
|
|
|
|
The implementation uses only Node.js built-ins. Browser-only sources must first export observed page assets to local files; the publisher consumes those files and does not automate access-control challenges.
|