MangaCircus/docs/source-modules.md

Adding a source to the single scraper

MangaCircus uses one scraper process. A source module is reviewed first-party code bundled into that process, not a container or a downloaded extension. scraper/adapters/index.js is the explicit registration point. The library, queue, pacing settings, checkpointing, source-error policy, and publication remain shared infrastructure. Do not add another queue writer for a new hoster.

Source boundary

Concern	Location	New-source responsibility
URL recognition and adapter contract	scraper/sources/<hoster>.js	id, label, match, discover, run
Catalog and chapter discovery	scraper/adapters/<hoster>.js	Map source responses to stable chapter IDs, metadata and page URLs
Source HTML/API parsing	scraper/plugins/<hoster>.js	Pure functions; no network or filesystem writes
Parser operation/URL validation	scraper/plugins/manifest.js, registry.js, validation.js	Allow only known operations and expected hosts/links
Resumable archive workflow	scraper/jobs/<hoster>-archive.js	Use the common publisher, inventory, pacing and checkpoints
Registration	scraper/adapters/index.js	Add one explicit require to adapters

The module contract is shown in scraper-service.md. Start from the closest existing hoster. match(url) returns null for unrelated URLs and { source, id, title, url } for an exact supported series URL. discover must be read-only and return a catalog. run must honor scope, existing-content policy, shouldStop, source-specific delays, and publication callbacks. Stable source series/chapter IDs are essential for retries and duplicate detection.

Before enabling a hoster

1. Document its permitted access method, terms and source-specific pacing. Do not work around 401, 403, 429, CAPTCHA, challenges or source protections.
2. Add fixture-backed tests for URL matching, catalog, empty/partial catalogs, chapter pages, unsafe links, and duplicate release handling. No test should contact the live hoster by default.
3. Test pause/resume from a partial chapter and an existing library target with the common publisher; verify one source lane and no parallel image workers.
4. Run npm run test:scraper and an isolated Compose smoke test. Review and deploy the single scraper image with a checkpoint-safe procedure; never start another daemon against the live queue.

Removing a hoster from registration stops new matches; it does not delete its library chapters or queued/checkpointed work. Inspect affected queue items and preserve checkpoints before disabling a source in a release.