MangaCircus/scraper/README.md

MangaCircus scraper

The scraper uses small host adapters to create a common manifest, followed by one conservative publisher that writes the MangaCircus library atomically.

Start with AGENT_RUNBOOK.md. It contains the host-specific procedures, safety limits, and stop conditions. manifest.example.json documents the interchange format.

The historical human-readable wishlist is WISHLIST.md. Once the managed daemon has started, data/scraper/queue.json is the authoritative queue and is edited through the admin UI.

Managed service

Run MangaCircus and the scraper daemon together:

npm run start:all

Or run them as two independently supervised processes:

npm start
npm run start:scraper

The daemon binds to 127.0.0.1:8091 by default, keeps its durable queue in data/scraper/queue.json, and resumes incomplete archive checkpoints. An admin can control it from Settings → Scraper. Set SCRAPER_TOKEN on both processes when the daemon is not restricted to loopback.

Queue schema version 3 supports scoped jobs, multiple jobs for the same upstream series, one sequential worker per source, incremental chapter publication, library duplicate/version checks, and configurable nightly update discovery. See docs/scraper-service.md for the service contract and docs/local-deployment.md for the currently running Windows Scheduled Task deployment. On that machine the scheduled task owns the single supervisor; do not run npm run start:all beside it.

npm run test:scraper
node scraper/cli.js metadata-anilist "One Piece"
node scraper/cli.js discover-weebcentral "https://weebcentral.com/series/..." --limit 3 --out manifest.json
node scraper/cli.js publish manifest.json --library library --metadata-search "Series title"
node scraper/cli.js archive-weebcentral "https://weebcentral.com/series/..." --library library --metadata-search "Series title"
node scraper/cli.js status-weebcentral "https://weebcentral.com/series/..." --library library

The implementation uses only Node.js built-ins. Browser-only sources must first export observed page assets to local files; the publisher consumes those files and does not automate access-control challenges.