Keep source plugins in single scraper process
This commit is contained in:
1 parent
0fc1a7b52a
commit
6d713e0a8c
19 files changed
+202
-467
No files matched your search
+16
-24
@@ -5,31 +5,23 @@ It owns the durable queue and is the only process allowed to run a queued archiv
|
||||
The browser never contacts it directly: authenticated admin routes in `server.js`
|
||||
proxy commands to its private HTTP API.
|
||||
|
||||
## First-party source parser pilot
|
||||
## In-process source modules
|
||||
|
||||
The WeebCentral and MangaDex parsers have reviewed first-party modules in
|
||||
`scraper/plugins/`. By default their existing in-process paths remain in use.
|
||||
To test process isolation in Compose, start the `source-plugins` profile and
|
||||
set `SOURCE_PLUGIN_URL=http://source-plugin-weebcentral:8094` and/or
|
||||
`MANGADEX_PLUGIN_URL=http://source-plugin-mangadex:8094` for the scraper.
|
||||
Each URL selects its source independently. A selected worker that is unavailable
|
||||
causes that source's discovery to fail closed and retain its checkpoint. Remove
|
||||
its URL to roll back. Do not restart a live scraper merely to try this option;
|
||||
use the normal checkpoint-safe deployment procedure.
|
||||
There is one scraper daemon/container and one durable queue. WeebCentral and
|
||||
MangaDex source modules are registered explicitly in `scraper/adapters/index.js`.
|
||||
Their descriptors live in `scraper/sources/`; they call source-specific catalog
|
||||
and archive code in the same process. Pure parsing functions in
|
||||
`scraper/plugins/` handle source page/API shapes. No source parser has a separate
|
||||
container, HTTP endpoint, runtime download, or independent queue. Only reviewed
|
||||
first-party modules ship in the image; Uchiyomi packages are not executed.
|
||||
|
||||
Each worker has no published port, egress network, library/data mount, or token.
|
||||
Only the existing rate-limited scraper HTTP client fetches remote HTML/images.
|
||||
The worker accepts bounded HTML for four parsing steps over its internal-only
|
||||
network. It loads only modules listed in the static first-party registry; there
|
||||
is no runtime plugin download or execution of Uchiyomi packages. This is a
|
||||
parser-isolation pilot, **not yet** a complete generic source-plugin contract
|
||||
or migration of either archive runner. Parsed catalog and image URLs are checked
|
||||
before use, and parser requests/responses are size-bounded. When the worker is
|
||||
enabled, WeebCentral requests use a pinned-DNS HTTPS transport that validates
|
||||
each redirect and stops on access errors; the ordinary in-process path is not
|
||||
changed. Public image hosts are not yet restricted to a reviewed CDN allowlist.
|
||||
Keep the profile off in production until a separate deployment validation is
|
||||
completed.
|
||||
The scraper's paced HTTP client fetches source content. Parsing is an ordinary
|
||||
function call. The registry checks source URLs and validates parsed output before
|
||||
the scraper follows links. WeebCentral catalog requests use a pinned-DNS HTTPS
|
||||
transport that validates redirects and stops on access errors. Image hosts are
|
||||
not restricted to a reviewed CDN allowlist yet. Existing queue state, source
|
||||
delays, chapter delays, and partial-download checkpoints are unchanged by this
|
||||
code organization. See [adding a source](source-modules.md) for the full path.
|
||||
|
||||
## Start and storage
|
||||
|
||||
@@ -218,7 +210,7 @@ the shared token.
|
||||
|
||||
## Adapter contract
|
||||
|
||||
Each source plugin is registered in `scraper/adapters/index.js` with:
|
||||
Each first-party source module is registered in `scraper/adapters/index.js` with:
|
||||
|
||||
```js
|
||||
{
|
||||
|
||||
Reference in new issue
Block a user