Checkpoint opt-in MangaDex parser worker
This commit is contained in:
1 parent
93f07e838e
commit
1c61686c9b
11 files changed
+298
-24
No files matched your search
+12
-12
@@ -7,23 +7,23 @@ proxy commands to its private HTTP API.
|
||||
|
||||
## First-party source parser pilot
|
||||
|
||||
The WeebCentral parser is now a reviewed first-party module in
|
||||
`scraper/plugins/weebcentral.js`. By default it runs in the scraper process,
|
||||
preserving the existing deployment. To test process isolation in Compose, start
|
||||
the `source-plugins` profile and set `SOURCE_PLUGIN_URL` for the scraper to
|
||||
`http://source-plugin-weebcentral:8094` (for example via the Compose environment).
|
||||
Both settings are required; enabling the URL without a healthy worker causes
|
||||
WeebCentral discovery to fail closed and retain its checkpoint. Remove the URL
|
||||
to roll back. Do not restart a live scraper merely to try this option; use the
|
||||
normal checkpoint-safe deployment procedure.
|
||||
The WeebCentral and MangaDex parsers have reviewed first-party modules in
|
||||
`scraper/plugins/`. By default their existing in-process paths remain in use.
|
||||
To test process isolation in Compose, start the `source-plugins` profile and
|
||||
set `SOURCE_PLUGIN_URL=http://source-plugin-weebcentral:8094` and/or
|
||||
`MANGADEX_PLUGIN_URL=http://source-plugin-mangadex:8094` for the scraper.
|
||||
Each URL selects its source independently. A selected worker that is unavailable
|
||||
causes that source's discovery to fail closed and retain its checkpoint. Remove
|
||||
its URL to roll back. Do not restart a live scraper merely to try this option;
|
||||
use the normal checkpoint-safe deployment procedure.
|
||||
|
||||
The worker has no published port, egress network, library/data mount, or token.
|
||||
Each worker has no published port, egress network, library/data mount, or token.
|
||||
Only the existing rate-limited scraper HTTP client fetches remote HTML/images.
|
||||
The worker accepts bounded HTML for four parsing steps over its internal-only
|
||||
network. It loads only modules listed in the static first-party registry; there
|
||||
is no runtime plugin download or execution of Uchiyomi packages. This is a
|
||||
parser-isolation pilot, **not yet** a complete generic source-plugin contract,
|
||||
or migration of the MangaDex runner. Parsed catalog and image URLs are checked
|
||||
parser-isolation pilot, **not yet** a complete generic source-plugin contract
|
||||
or migration of either archive runner. Parsed catalog and image URLs are checked
|
||||
before use, and parser requests/responses are size-bounded. When the worker is
|
||||
enabled, WeebCentral requests use a pinned-DNS HTTPS transport that validates
|
||||
each redirect and stops on access errors; the ordinary in-process path is not
|
||||
|
||||
Reference in new issue
Block a user