Consolidate MangaCircus app, scraper, and Compose baseline
This commit is contained in:
1 parent
cfb2d79103
commit
eef4ffa07c
103 files changed
+9316
-172
No files matched your search
@@ -0,0 +1,48 @@
|
||||
# MangaCircus scraper
|
||||
|
||||
The scraper uses small host adapters to create a common manifest, followed by one conservative publisher that writes the MangaCircus library atomically.
|
||||
|
||||
Start with [AGENT_RUNBOOK.md](AGENT_RUNBOOK.md). It contains the host-specific procedures, safety limits, and stop conditions. `manifest.example.json` documents the interchange format.
|
||||
|
||||
The historical human-readable wishlist is [WISHLIST.md](WISHLIST.md). Once the
|
||||
managed daemon has started, `data/scraper/queue.json` is the authoritative queue and
|
||||
is edited through the admin UI.
|
||||
|
||||
## Managed service
|
||||
|
||||
Run MangaCircus and the scraper daemon together:
|
||||
|
||||
```bash
|
||||
npm run start:all
|
||||
```
|
||||
|
||||
Or run them as two independently supervised processes:
|
||||
|
||||
```bash
|
||||
npm start
|
||||
npm run start:scraper
|
||||
```
|
||||
|
||||
The daemon binds to `127.0.0.1:8091` by default, keeps its durable queue in
|
||||
`data/scraper/queue.json`, and resumes incomplete archive checkpoints. An admin can
|
||||
control it from **Settings → Scraper**. Set `SCRAPER_TOKEN` on both processes when
|
||||
the daemon is not restricted to loopback.
|
||||
|
||||
Queue schema version 3 supports scoped jobs, multiple jobs for the same upstream
|
||||
series, one sequential worker per source, incremental chapter publication, library
|
||||
duplicate/version checks, and configurable nightly update discovery. See
|
||||
[`docs/scraper-service.md`](../docs/scraper-service.md) for the service contract and
|
||||
[`docs/local-deployment.md`](../docs/local-deployment.md) for the currently running
|
||||
Windows Scheduled Task deployment. On that machine the scheduled task owns the single
|
||||
supervisor; do not run `npm run start:all` beside it.
|
||||
|
||||
```bash
|
||||
npm run test:scraper
|
||||
node scraper/cli.js metadata-anilist "One Piece"
|
||||
node scraper/cli.js discover-weebcentral "https://weebcentral.com/series/..." --limit 3 --out manifest.json
|
||||
node scraper/cli.js publish manifest.json --library library --metadata-search "Series title"
|
||||
node scraper/cli.js archive-weebcentral "https://weebcentral.com/series/..." --library library --metadata-search "Series title"
|
||||
node scraper/cli.js status-weebcentral "https://weebcentral.com/series/..." --library library
|
||||
```
|
||||
|
||||
The implementation uses only Node.js built-ins. Browser-only sources must first export observed page assets to local files; the publisher consumes those files and does not automate access-control challenges.
|
||||
Reference in new issue
Block a user