Keep source plugins in single scraper process

This commit is contained in:
Codex committed 2026-10-13 22:42:20 +02:00
1 parent 0fc1a7b52a
commit 6d713e0a8c
19 files changed
+202 -467

No files matched your search

+16 -24
View File
@@ -5,31 +5,23 @@ It owns the durable queue and is the only process allowed to run a queued archiv
The browser never contacts it directly: authenticated admin routes in `server.js`
proxy commands to its private HTTP API.
## First-party source parser pilot
## In-process source modules
The WeebCentral and MangaDex parsers have reviewed first-party modules in
`scraper/plugins/`. By default their existing in-process paths remain in use.
To test process isolation in Compose, start the `source-plugins` profile and
set `SOURCE_PLUGIN_URL=http://source-plugin-weebcentral:8094` and/or
`MANGADEX_PLUGIN_URL=http://source-plugin-mangadex:8094` for the scraper.
Each URL selects its source independently. A selected worker that is unavailable
causes that source's discovery to fail closed and retain its checkpoint. Remove
its URL to roll back. Do not restart a live scraper merely to try this option;
use the normal checkpoint-safe deployment procedure.
There is one scraper daemon/container and one durable queue. WeebCentral and
MangaDex source modules are registered explicitly in `scraper/adapters/index.js`.
Their descriptors live in `scraper/sources/`; they call source-specific catalog
and archive code in the same process. Pure parsing functions in
`scraper/plugins/` handle source page/API shapes. No source parser has a separate
container, HTTP endpoint, runtime download, or independent queue. Only reviewed
first-party modules ship in the image; Uchiyomi packages are not executed.
Each worker has no published port, egress network, library/data mount, or token.
Only the existing rate-limited scraper HTTP client fetches remote HTML/images.
The worker accepts bounded HTML for four parsing steps over its internal-only
network. It loads only modules listed in the static first-party registry; there
is no runtime plugin download or execution of Uchiyomi packages. This is a
parser-isolation pilot, **not yet** a complete generic source-plugin contract
or migration of either archive runner. Parsed catalog and image URLs are checked
before use, and parser requests/responses are size-bounded. When the worker is
enabled, WeebCentral requests use a pinned-DNS HTTPS transport that validates
each redirect and stops on access errors; the ordinary in-process path is not
changed. Public image hosts are not yet restricted to a reviewed CDN allowlist.
Keep the profile off in production until a separate deployment validation is
completed.
The scraper's paced HTTP client fetches source content. Parsing is an ordinary
function call. The registry checks source URLs and validates parsed output before
the scraper follows links. WeebCentral catalog requests use a pinned-DNS HTTPS
transport that validates redirects and stops on access errors. Image hosts are
not restricted to a reviewed CDN allowlist yet. Existing queue state, source
delays, chapter delays, and partial-download checkpoints are unchanged by this
code organization. See [adding a source](source-modules.md) for the full path.
## Start and storage
@@ -218,7 +210,7 @@ the shared token.
## Adapter contract
Each source plugin is registered in `scraper/adapters/index.js` with:
Each first-party source module is registered in `scraper/adapters/index.js` with:
```js
{