Keep source plugins in single scraper process

docs/architecture.md

@@ -17,6 +17,7 @@ Related documents:
 - [settings-navigation.md](settings-navigation.md): User/Admin workspace, scraper tabs and deep links.
 - [peering.md](peering.md): optional private server-to-server catalog and transfer.
 - [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
+- [source-modules.md](source-modules.md): adding and maintaining first-party hosters inside the single scraper process.
 
 ## 1. Principles
 

docs/scraper-service.md

@@ -5,31 +5,23 @@ It owns the durable queue and is the only process allowed to run a queued archiv
 The browser never contacts it directly: authenticated admin routes in `server.js`
 proxy commands to its private HTTP API.
 
-## First-party source parser pilot
+## In-process source modules
 
-The WeebCentral and MangaDex parsers have reviewed first-party modules in
-`scraper/plugins/`. By default their existing in-process paths remain in use.
-To test process isolation in Compose, start the `source-plugins` profile and
-set `SOURCE_PLUGIN_URL=http://source-plugin-weebcentral:8094` and/or
-`MANGADEX_PLUGIN_URL=http://source-plugin-mangadex:8094` for the scraper.
-Each URL selects its source independently. A selected worker that is unavailable
-causes that source's discovery to fail closed and retain its checkpoint. Remove
-its URL to roll back. Do not restart a live scraper merely to try this option;
-use the normal checkpoint-safe deployment procedure.
+There is one scraper daemon/container and one durable queue. WeebCentral and
+MangaDex source modules are registered explicitly in `scraper/adapters/index.js`.
+Their descriptors live in `scraper/sources/`; they call source-specific catalog
+and archive code in the same process. Pure parsing functions in
+`scraper/plugins/` handle source page/API shapes. No source parser has a separate
+container, HTTP endpoint, runtime download, or independent queue. Only reviewed
+first-party modules ship in the image; Uchiyomi packages are not executed.
 
-Each worker has no published port, egress network, library/data mount, or token.
-Only the existing rate-limited scraper HTTP client fetches remote HTML/images.
-The worker accepts bounded HTML for four parsing steps over its internal-only
-network. It loads only modules listed in the static first-party registry; there
-is no runtime plugin download or execution of Uchiyomi packages. This is a
-parser-isolation pilot, **not yet** a complete generic source-plugin contract
-or migration of either archive runner. Parsed catalog and image URLs are checked
-before use, and parser requests/responses are size-bounded. When the worker is
-enabled, WeebCentral requests use a pinned-DNS HTTPS transport that validates
-each redirect and stops on access errors; the ordinary in-process path is not
-changed. Public image hosts are not yet restricted to a reviewed CDN allowlist.
-Keep the profile off in production until a separate deployment validation is
-completed.
+The scraper's paced HTTP client fetches source content. Parsing is an ordinary
+function call. The registry checks source URLs and validates parsed output before
+the scraper follows links. WeebCentral catalog requests use a pinned-DNS HTTPS
+transport that validates redirects and stops on access errors. Image hosts are
+not restricted to a reviewed CDN allowlist yet. Existing queue state, source
+delays, chapter delays, and partial-download checkpoints are unchanged by this
+code organization. See [adding a source](source-modules.md) for the full path.
 
 ## Start and storage
 
@@ -218,7 +210,7 @@ the shared token.
 
 ## Adapter contract
 
-Each source plugin is registered in `scraper/adapters/index.js` with:
+Each first-party source module is registered in `scraper/adapters/index.js` with:
 
 ```js
 {

docs/source-modules.md

@@ -0,0 +1,42 @@
+# Adding a source to the single scraper
+
+MangaCircus uses **one scraper process**. A source module is reviewed first-party
+code bundled into that process, not a container or a downloaded extension.
+`scraper/adapters/index.js` is the explicit registration point. The library,
+queue, pacing settings, checkpointing, source-error policy, and publication
+remain shared infrastructure. Do not add another queue writer for a new hoster.
+
+## Source boundary
+
+| Concern | Location | New-source responsibility |
+| --- | --- | --- |
+| URL recognition and adapter contract | `scraper/sources/<hoster>.js` | `id`, `label`, `match`, `discover`, `run` |
+| Catalog and chapter discovery | `scraper/adapters/<hoster>.js` | Map source responses to stable chapter IDs, metadata and page URLs |
+| Source HTML/API parsing | `scraper/plugins/<hoster>.js` | Pure functions; no network or filesystem writes |
+| Parser operation/URL validation | `scraper/plugins/manifest.js`, `registry.js`, `validation.js` | Allow only known operations and expected hosts/links |
+| Resumable archive workflow | `scraper/jobs/<hoster>-archive.js` | Use the common publisher, inventory, pacing and checkpoints |
+| Registration | `scraper/adapters/index.js` | Add one explicit `require` to `adapters` |
+
+The module contract is shown in [scraper-service.md](scraper-service.md#adapter-contract).
+Start from the closest existing hoster. `match(url)` returns `null` for unrelated
+URLs and `{ source, id, title, url }` for an exact supported series URL. `discover`
+must be read-only and return a catalog. `run` must honor scope, existing-content
+policy, `shouldStop`, source-specific delays, and publication callbacks. Stable
+source series/chapter IDs are essential for retries and duplicate detection.
+
+## Before enabling a hoster
+
+1. Document its permitted access method, terms and source-specific pacing. Do
+   not work around 401, 403, 429, CAPTCHA, challenges or source protections.
+2. Add fixture-backed tests for URL matching, catalog, empty/partial catalogs,
+   chapter pages, unsafe links, and duplicate release handling. No test should
+   contact the live hoster by default.
+3. Test pause/resume from a partial chapter and an existing library target with
+   the common publisher; verify one source lane and no parallel image workers.
+4. Run `npm run test:scraper` and an isolated Compose smoke test. Review and deploy
+   the single scraper image with a checkpoint-safe procedure; never start another
+   daemon against the live queue.
+
+Removing a hoster from registration stops new matches; it does **not** delete its
+library chapters or queued/checkpointed work. Inspect affected queue items and
+preserve checkpoints before disabling a source in a release.