Keep source plugins in single scraper process
This commit is contained in:
1 parent
0fc1a7b52a
commit
6d713e0a8c
19 files changed
+202
-467
No files matched your search
@@ -0,0 +1,42 @@
|
||||
# Adding a source to the single scraper
|
||||
|
||||
MangaCircus uses **one scraper process**. A source module is reviewed first-party
|
||||
code bundled into that process, not a container or a downloaded extension.
|
||||
`scraper/adapters/index.js` is the explicit registration point. The library,
|
||||
queue, pacing settings, checkpointing, source-error policy, and publication
|
||||
remain shared infrastructure. Do not add another queue writer for a new hoster.
|
||||
|
||||
## Source boundary
|
||||
|
||||
| Concern | Location | New-source responsibility |
|
||||
| --- | --- | --- |
|
||||
| URL recognition and adapter contract | `scraper/sources/<hoster>.js` | `id`, `label`, `match`, `discover`, `run` |
|
||||
| Catalog and chapter discovery | `scraper/adapters/<hoster>.js` | Map source responses to stable chapter IDs, metadata and page URLs |
|
||||
| Source HTML/API parsing | `scraper/plugins/<hoster>.js` | Pure functions; no network or filesystem writes |
|
||||
| Parser operation/URL validation | `scraper/plugins/manifest.js`, `registry.js`, `validation.js` | Allow only known operations and expected hosts/links |
|
||||
| Resumable archive workflow | `scraper/jobs/<hoster>-archive.js` | Use the common publisher, inventory, pacing and checkpoints |
|
||||
| Registration | `scraper/adapters/index.js` | Add one explicit `require` to `adapters` |
|
||||
|
||||
The module contract is shown in [scraper-service.md](scraper-service.md#adapter-contract).
|
||||
Start from the closest existing hoster. `match(url)` returns `null` for unrelated
|
||||
URLs and `{ source, id, title, url }` for an exact supported series URL. `discover`
|
||||
must be read-only and return a catalog. `run` must honor scope, existing-content
|
||||
policy, `shouldStop`, source-specific delays, and publication callbacks. Stable
|
||||
source series/chapter IDs are essential for retries and duplicate detection.
|
||||
|
||||
## Before enabling a hoster
|
||||
|
||||
1. Document its permitted access method, terms and source-specific pacing. Do
|
||||
not work around 401, 403, 429, CAPTCHA, challenges or source protections.
|
||||
2. Add fixture-backed tests for URL matching, catalog, empty/partial catalogs,
|
||||
chapter pages, unsafe links, and duplicate release handling. No test should
|
||||
contact the live hoster by default.
|
||||
3. Test pause/resume from a partial chapter and an existing library target with
|
||||
the common publisher; verify one source lane and no parallel image workers.
|
||||
4. Run `npm run test:scraper` and an isolated Compose smoke test. Review and deploy
|
||||
the single scraper image with a checkpoint-safe procedure; never start another
|
||||
daemon against the live queue.
|
||||
|
||||
Removing a hoster from registration stops new matches; it does **not** delete its
|
||||
library chapters or queued/checkpointed work. Inspect affected queue items and
|
||||
preserve checkpoints before disabling a source in a release.
|
||||
Reference in new issue
Block a user