MaashkinoozheandClaude Opus 5.5
f53b92a7dc
- docs/deployment.md: step-by-step Docker deployment for anyone (setup, configuration, LAN/HTTPS/VPN access, adding manga, operation, prebuilt images, backups, peering, troubleshooting, rules for agents). - deploy/circus.sh: host helper without Node (status, up/down with queue pause, logs, pause/resume, update, rescan, restore-test); deploy/compose.https.yml + Caddyfile for automatic HTTPS. - README rewritten for public readers (Docker first, sources/copyright note, docs index); AGENTS.md + CLAUDE.md for contributors and coding agents. - Removed notes about one particular installation (Windows host, handoffs, audits of a private library, benchmarks with local paths, wishlist, reading map) and the unused ops/proxmox templates; generalized remaining mentions of hosts, VPN and Windows-only commands. - Defaults: TZ and reading time zone UTC; .gitattributes keeps shell/YAML LF. - Personal agent coordination files are git-ignored (CLAUDE.local.md, .claude/skills/manga-org/). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
47 lines
2.1 KiB
Markdown
47 lines
2.1 KiB
Markdown
# MangaCircus scraper
|
|
|
|
The scraper uses small host adapters to create a common manifest, followed by one conservative publisher that writes the MangaCircus library atomically.
|
|
|
|
Start with [AGENT_RUNBOOK.md](AGENT_RUNBOOK.md). It contains the host-specific procedures, safety limits, and stop conditions. `manifest.example.json` documents the interchange format.
|
|
|
|
Once the managed daemon has started, `data/scraper/queue.json` is the authoritative
|
|
queue and is edited through the admin UI.
|
|
|
|
## Managed service
|
|
|
|
Run MangaCircus and the scraper daemon together:
|
|
|
|
```bash
|
|
npm run start:all
|
|
```
|
|
|
|
Or run them as two independently supervised processes:
|
|
|
|
```bash
|
|
npm start
|
|
npm run start:scraper
|
|
```
|
|
|
|
The daemon binds to `127.0.0.1:8091` by default, keeps its durable queue in
|
|
`data/scraper/queue.json`, and resumes incomplete archive checkpoints. An admin can
|
|
control it from **Settings → Scraper**. Set `SCRAPER_TOKEN` on both processes when
|
|
the daemon is not restricted to loopback.
|
|
|
|
Queue schema version 3 supports scoped jobs, multiple jobs for the same upstream
|
|
series, one sequential worker per source, incremental chapter publication, library
|
|
duplicate/version checks, and configurable nightly update discovery. See
|
|
[`docs/scraper-service.md`](../docs/scraper-service.md) for the service contract and
|
|
[`docs/deployment.md`](../docs/deployment.md) for running it in Docker. Run only one
|
|
supervisor or container against the same library and data.
|
|
|
|
```bash
|
|
npm run test:scraper
|
|
node scraper/cli.js metadata-anilist "One Piece"
|
|
node scraper/cli.js discover-weebcentral "https://weebcentral.com/series/..." --limit 3 --out manifest.json
|
|
node scraper/cli.js publish manifest.json --library library --metadata-search "Series title"
|
|
node scraper/cli.js archive-weebcentral "https://weebcentral.com/series/..." --library library --metadata-search "Series title"
|
|
node scraper/cli.js status-weebcentral "https://weebcentral.com/series/..." --library library
|
|
```
|
|
|
|
The implementation uses only Node.js built-ins. Browser-only sources must first export observed page assets to local files; the publisher consumes those files and does not automate access-control challenges.
|