MangaCircus/vision/first-party-source-plugins.md

First-party source plugins for MangaCircus

Prepared 2026-10-13. The first implementation slice extracts the WeebCentral parser into a static first-party registry and adds an opt-in, network-isolated Compose worker. No external plugin was installed, no host was probed, and no live queue, delay, or container was changed. The broker, generic runner, new hoster pilot, admin controls, and full deployment validation below remain work to do; the diagram describes the target rather than the current implementation. Parser-result URL validation and bounded worker messages were added next; full redirect/DNS enforcement and source-specific CDN allowlists are still pending.

Decision

Keep MangaCircus's no-database library, source-aware queue, checkpoints, source ledger, and chapter publisher. Extract source-specific discovery/parsing into versioned, first-party plugins. Review and build every plugin from our repository; never install an arbitrary repository, APK, JavaScript package, or remotely supplied parser at runtime. An admin may enable/disable a reviewed plugin, not supply executable code. A catalog match remains distinct from adapter compatibility, actual chapter coverage, and download rights.

The upstream model is Uchiyomi v0.61.1, commit 726bd9fa6fe0090fd4b5de1cdb2b6926e5382127. Uchiyomi registers a common search/getSeries/listChapters/getPageUrls interface across built-in MangaDex, generic site engines, mounted JS source packs, and Suwayomi-backed Mihon/Tachiyomi extensions. We want the interface idea and selected reviewed parsers, not its executable-code loader or Suwayomi/FlareSolverr runtime.

Existing seam and problem

• scraper/adapters/index.js already dispatches match/discover/run; scraper/service.js schedules one job at a time per source, applies scopes and source settings, records evidence, and owns the durable queue.
• scraper/adapters/{weebcentral,mangadex}.js parse source catalogs and pages. scraper/jobs/{weebcentral,mangadex}-archive.js repeat transport, checkpoint, metadata, and publication steps. scraper/lib/http.js stops on 401/403/429. scraper/lib/publisher.js makes chapters visible atomically.
• Adding another hoster today means copying a whole archive runner. The plugin work must eliminate that repetition without allowing plugin code to write the library or bypass request controls.

Boundary

Admin/UI -> queue + source ledger -> scraper scheduler -> first-party plugin worker
                                   |                  ^
                                   v                  | request/response IPC
                              HTTP broker ----------+
                                   |
                                   v
                        host / approved image CDN

scraper scheduler -> library inventory + scope/dedupe -> common downloader
                  -> existing checkpoint/publisher -> library

The worker describes requests and parses replies. It has no library or state mount and no internet-facing network. The scraper-owned broker makes every source request, including image requests. It enforces one source lane, configured request/chapter delays, declared hostname allowlists, redirect and resolved-address checks, response size/time limits, and stop-on-401/403/429/challenge behavior. HTTP 5xx and network retry remain centrally bounded. No proxy/identity/user-agent rotation and no challenge solver. A source that requires a protection bypass is reported as unsupported/blocked, not silently routed elsewhere.

For production Compose, run one small plugin-worker instance per enabled plugin from the same pinned first-party image. Give each a dedicated internal-only network shared with the scraper (not the general backend or egress network); mount no library/data/secrets; use a read-only filesystem, dropped capabilities, no-new-privileges, and memory/PID limits. Do not inherit Compose's existing x-app anchor, which mounts the scraper token. Only the scraper has egress and library write access. On Windows local development, use the same Compose worker: an ordinary Node child process is crash isolation but not a network sandbox. No exposed plugin port or browser-to-worker route.

Proposed v1 IPC contract

Transport: versioned JSON messages over a private internal HTTP endpoint or stdio; bounded payloads and request IDs. The exact transport is an implementation choice, not exposed to users. A worker is pinned to one pluginId and code revision. The host validates every returned object.

{ "v": 1, "id": "req-1", "op": "catalog", "pluginId": "example", "seriesUrl": "https://allowed.example/series/123" }
{ "v": 1, "id": "req-2", "op": "fetch", "url": "https://allowed.example/series/123", "accept": "text/html" }
{ "v": 1, "id": "req-2", "ok": true, "status": 200, "contentType": "text/html", "body": "...bounded..." }
{ "v": 1, "id": "req-1", "ok": true, "title": "...", "chapters": [{ "id": "source-chapter-id", "number": "1", "volume": "1", "url": "https://allowed.example/chapter/1" }], "coverage": "unknown" }

Operations: match from a declarative manifest, catalog, chapterPages, optional search/seriesMetadata later. chapterPages returns ordered image descriptors; only the common downloader fetches bytes. Plugin output may report unrecognized-labels, external-only, and confidence/evidence, but may not mark rights verified or override the library's work identity. Plugin failures are typed (parser, catalog-empty, network, access-block, rate-limit) and visible in the source ledger. Version/schema mismatch disables that plugin lane without losing checkpoints.

Manifest, reviewed with code: stable plugin ID/version, exact series URL matcher, primary hostnames, separately reviewed image/CDN hostnames, optional language, parser fixture set, upstream provenance, license, and support state (experimental/supported/disabled). Hostnames are not administrator-provided at runtime in v1. Redirects and DNS answers must be checked again by the broker; declared hostnames alone do not protect against SSRF.

Uchiyomi code triage

Upstream implementation	Use in MangaCircus	Rationale
sources/types.ts	Adapt concept, write our own contract	Its narrow catalog/page operations are valuable; its Cloudflare/image-header capabilities and database-specific assumptions are not our policy.
sources/engines/madara.ts	Candidate for pure parser extraction	Broad WordPress/Madara markup; separate parsing from cfGet/cfPost, test with fixtures, then trial only on a specifically reviewed accessible host. Not a blanket license to scrape every Madara site.
sources/engines/mangathemesia.ts	Later candidate	Similar, but source-specific page and image hosts need review; currently solver-coupled.
sources/engines/manganato.ts	Later candidate, not first pilot	More variant/fallback logic and an explicit challenge dependency; higher maintenance and source-policy risk.
sources/mangadex.ts	Compare behavior/tests only	MangaCircus already has a MangaDex adapter. Evaluate its language and rate-limit edge cases without replacing working code wholesale.
sources/suwayomi/*, loader.ts	Do not port	Designed to run operator-supplied APKs/JS. Conflicts with the first-party-only requirement.
downloader.ts	Learn from tests/failure cases, do not transplant	CBZ/database assumptions differ from MangaCircus's folder/checkpoint publisher; its challenge session handling does not fit our policy.

The upstream sources/ directory contains no bundled plugin implementations beyond a placeholder; the generic engines above live in its main server. Do not mistake its extension catalog for code licensed by Uchiyomi: separately authored APKs have separately determined provenance and licenses.

Code provenance and license gate

Uchiyomi is MPL-2.0. No verbatim or adapted upstream code has been copied into MangaCircus in this preparation. Before copying any implementation, identify the exact upstream file/revision and third-party notices, preserve attribution/notices, add the MPL license and a THIRD_PARTY_SOURCES.md ledger, and keep copied/modified MPL-covered code in clearly marked files. If distributing a Docker image with those files, provide the corresponding source for the covered files as the license requires. Mozilla describes MPL as file-level copyleft; it does not require unrelated MangaCircus files to become MPL. This is an implementation checklist, not legal advice; review the final distribution arrangement before release. Reimplementing an idea from documented behavior is distinct from copying its expressive code, but provenance should still be documented.

Incremental implementation passes

1. Contract and broker, no new host. Add manifest validation and IPC schema, central fetch broker, disabled-by-default worker, fixture-driven contract tests, worker crash/timeouts, SSRF/redirect tests, 401/403/429 stop tests, and checkpoint-preservation tests. Existing adapters continue unchanged.
2. First-party compatibility plugins. Migrate MangaDex and WeebCentral catalog/page parsing one at a time behind the contract, with old-vs-new fixture parity. Keep existing job IDs, source IDs, queue order, delays, and work directory paths. The common runner calls the existing inventory/scope/publisher. Rollback is a plugin enable switch, not a queue rewrite.
3. One reviewed generic parser pilot. Start with a pure Madara-family parser based on upstream lessons. Choose a single source only after checking accessibility, terms, stable hostnames, and parser fixtures. Inspection/catalog-only first; no auto-queue. Record source evidence, compare with library identity, then test a small chapter in isolated A/B fixtures. Add further families only if they solve a concrete catalog gap.
4. Admin UI. Under Scraper → Sources, show installed first-party plugins, version/provenance, enabled state, supported hosts, last probe stage, observed catalog/coverage, rights state, and blocked reason. Enabling requires admin confirmation; no repository URL/APK upload/install button. Source candidates stay leads until reviewed and explicitly queued.
5. Deployment. Add the worker to local and VM Compose with internal-only networking and resource limits; run compose checks and end-to-end fixture tests, then stage on a test instance. Leave the live stack untouched until jobs are checkpoint-safe and rollback verified. Document plugin upgrades and recovery.

Acceptance: no plugin has direct internet or library-write capability; one worker per plugin may crash without taking down the reader/scraper; no source request can evade host limits or 401/403/429 handling; existing jobs resume unchanged across a rollback; the library still needs no database; source provenance and MPL obligations are traceable; no dynamic third-party code execution is introduced.