43 lines
2.7 KiB
Markdown
43 lines
2.7 KiB
Markdown
# Adding a source to the single scraper
|
|
|
|
MangaCircus uses **one scraper process**. A source module is reviewed first-party
|
|
code bundled into that process, not a container or a downloaded extension.
|
|
`scraper/adapters/index.js` is the explicit registration point. The library,
|
|
queue, pacing settings, checkpointing, source-error policy, and publication
|
|
remain shared infrastructure. Do not add another queue writer for a new hoster.
|
|
|
|
## Source boundary
|
|
|
|
| Concern | Location | New-source responsibility |
|
|
| --- | --- | --- |
|
|
| URL recognition and adapter contract | `scraper/sources/<hoster>.js` | `id`, `label`, `match`, `discover`, `run` |
|
|
| Catalog and chapter discovery | `scraper/adapters/<hoster>.js` | Map source responses to stable chapter IDs, metadata and page URLs |
|
|
| Source HTML/API parsing | `scraper/plugins/<hoster>.js` | Pure functions; no network or filesystem writes |
|
|
| Parser operation/URL validation | `scraper/plugins/manifest.js`, `registry.js`, `validation.js` | Allow only known operations and expected hosts/links |
|
|
| Resumable archive workflow | `scraper/jobs/<hoster>-archive.js` | Use the common publisher, inventory, pacing and checkpoints |
|
|
| Registration | `scraper/adapters/index.js` | Add one explicit `require` to `adapters` |
|
|
|
|
The module contract is shown in [scraper-service.md](scraper-service.md#adapter-contract).
|
|
Start from the closest existing hoster. `match(url)` returns `null` for unrelated
|
|
URLs and `{ source, id, title, url }` for an exact supported series URL. `discover`
|
|
must be read-only and return a catalog. `run` must honor scope, existing-content
|
|
policy, `shouldStop`, source-specific delays, and publication callbacks. Stable
|
|
source series/chapter IDs are essential for retries and duplicate detection.
|
|
|
|
## Before enabling a hoster
|
|
|
|
1. Document its permitted access method, terms and source-specific pacing. Do
|
|
not work around 401, 403, 429, CAPTCHA, challenges or source protections.
|
|
2. Add fixture-backed tests for URL matching, catalog, empty/partial catalogs,
|
|
chapter pages, unsafe links, and duplicate release handling. No test should
|
|
contact the live hoster by default.
|
|
3. Test pause/resume from a partial chapter and an existing library target with
|
|
the common publisher; verify one source lane and no parallel image workers.
|
|
4. Run `npm run test:scraper` and an isolated Compose smoke test. Review and deploy
|
|
the single scraper image with a checkpoint-safe procedure; never start another
|
|
daemon against the live queue.
|
|
|
|
Removing a hoster from registration stops new matches; it does **not** delete its
|
|
library chapters or queued/checkpointed work. Inspect affected queue items and
|
|
preserve checkpoints before disabling a source in a release.
|