Document and package opt-in source worker
This commit is contained in:
1 parent
f4123b32e4
commit
cc2fd05062
2 files changed
+57
-1
No files matched your search
+31
-1
@@ -75,7 +75,8 @@ services:
|
|||||||
SCRAPER_HOST: 0.0.0.0
|
SCRAPER_HOST: 0.0.0.0
|
||||||
SCRAPER_PORT: "8091"
|
SCRAPER_PORT: "8091"
|
||||||
SCRAPER_TOKEN_FILE: /run/secrets/scraper_token
|
SCRAPER_TOKEN_FILE: /run/secrets/scraper_token
|
||||||
networks: [backend, egress]
|
SOURCE_PLUGIN_URL: ${SOURCE_PLUGIN_URL:-}
|
||||||
|
networks: [backend, egress, plugin_backend]
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: [CMD, node, tools/healthcheck.js, scraper]
|
test: [CMD, node, tools/healthcheck.js, scraper]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
@@ -108,9 +109,38 @@ services:
|
|||||||
start_period: 60s
|
start_period: 60s
|
||||||
retries: 3
|
retries: 3
|
||||||
|
|
||||||
|
source-plugin-weebcentral:
|
||||||
|
image: ${APP_IMAGE:-mangacircus:local}
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
args:
|
||||||
|
VCS_REF: ${VCS_REF:-development}
|
||||||
|
profiles: [source-plugins]
|
||||||
|
command: [node, scraper/plugins/worker.js]
|
||||||
|
init: true
|
||||||
|
restart: unless-stopped
|
||||||
|
read_only: true
|
||||||
|
tmpfs: [/tmp]
|
||||||
|
security_opt: [no-new-privileges:true]
|
||||||
|
cap_drop: [ALL]
|
||||||
|
pids_limit: 64
|
||||||
|
mem_limit: 256m
|
||||||
|
environment:
|
||||||
|
SOURCE_PLUGIN_HOST: 0.0.0.0
|
||||||
|
SOURCE_PLUGIN_PORT: "8094"
|
||||||
|
networks: [plugin_backend]
|
||||||
|
healthcheck:
|
||||||
|
test: [CMD, node, tools/healthcheck-source-plugin.js]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 5s
|
||||||
|
start_period: 15s
|
||||||
|
retries: 3
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
backend:
|
backend:
|
||||||
internal: true
|
internal: true
|
||||||
|
plugin_backend:
|
||||||
|
internal: true
|
||||||
egress: {}
|
egress: {}
|
||||||
|
|
||||||
secrets:
|
secrets:
|
||||||
|
|||||||
@@ -5,6 +5,32 @@ It owns the durable queue and is the only process allowed to run a queued archiv
|
|||||||
The browser never contacts it directly: authenticated admin routes in `server.js`
|
The browser never contacts it directly: authenticated admin routes in `server.js`
|
||||||
proxy commands to its private HTTP API.
|
proxy commands to its private HTTP API.
|
||||||
|
|
||||||
|
## First-party source parser pilot
|
||||||
|
|
||||||
|
The WeebCentral parser is now a reviewed first-party module in
|
||||||
|
`scraper/plugins/weebcentral.js`. By default it runs in the scraper process,
|
||||||
|
preserving the existing deployment. To test process isolation in Compose, start
|
||||||
|
the `source-plugins` profile and set `SOURCE_PLUGIN_URL` for the scraper to
|
||||||
|
`http://source-plugin-weebcentral:8094` (for example via the Compose environment).
|
||||||
|
Both settings are required; enabling the URL without a healthy worker causes
|
||||||
|
WeebCentral discovery to fail closed and retain its checkpoint. Remove the URL
|
||||||
|
to roll back. Do not restart a live scraper merely to try this option; use the
|
||||||
|
normal checkpoint-safe deployment procedure.
|
||||||
|
|
||||||
|
The worker has no published port, egress network, library/data mount, or token.
|
||||||
|
Only the existing rate-limited scraper HTTP client fetches remote HTML/images.
|
||||||
|
The worker accepts bounded HTML for four parsing steps over its internal-only
|
||||||
|
network. It loads only modules listed in the static first-party registry; there
|
||||||
|
is no runtime plugin download or execution of Uchiyomi packages. This is a
|
||||||
|
parser-isolation pilot, **not yet** a complete generic source-plugin contract,
|
||||||
|
or migration of the MangaDex runner. Parsed catalog and image URLs are checked
|
||||||
|
before use, and parser requests/responses are size-bounded. When the worker is
|
||||||
|
enabled, WeebCentral requests use a pinned-DNS HTTPS transport that validates
|
||||||
|
each redirect and stops on access errors; the ordinary in-process path is not
|
||||||
|
changed. Public image hosts are not yet restricted to a reviewed CDN allowlist.
|
||||||
|
Keep the profile off in production until a separate deployment validation is
|
||||||
|
completed.
|
||||||
|
|
||||||
## Start and storage
|
## Start and storage
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|||||||
Reference in new issue
Block a user