Checkpoint first-party source parser pilot

vision/first-party-source-plugins.md

@@ -0,0 +1,82 @@
+# First-party source plugins for MangaCircus
+
+Prepared 2026-10-13. The first implementation slice extracts the WeebCentral
+parser into a static first-party registry and adds an opt-in, network-isolated
+Compose worker. No external plugin was installed, no host was probed, and no
+live queue, delay, or container was changed. The broker, generic runner, new
+hoster pilot, admin controls, and full deployment validation below remain work
+to do; the diagram describes the target rather than the current implementation.
+
+## Decision
+
+Keep MangaCircus's no-database library, source-aware queue, checkpoints, source ledger, and chapter publisher. Extract source-specific discovery/parsing into versioned, first-party plugins. Review and build every plugin from **our repository**; never install an arbitrary repository, APK, JavaScript package, or remotely supplied parser at runtime. An admin may enable/disable a reviewed plugin, not supply executable code. A catalog match remains distinct from adapter compatibility, actual chapter coverage, and download rights.
+
+The upstream model is Uchiyomi v0.61.1, commit [`726bd9fa6fe0090fd4b5de1cdb2b6926e5382127`](https://github.com/AngeloSha/uchiyomi/tree/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127). Uchiyomi registers a common `search/getSeries/listChapters/getPageUrls` interface across built-in MangaDex, generic site engines, mounted JS source packs, and Suwayomi-backed Mihon/Tachiyomi extensions. We want the interface idea and selected reviewed parsers, **not** its executable-code loader or Suwayomi/FlareSolverr runtime.
+
+## Existing seam and problem
+
+- `scraper/adapters/index.js` already dispatches `match/discover/run`; `scraper/service.js` schedules one job at a time per source, applies scopes and source settings, records evidence, and owns the durable queue.
+- `scraper/adapters/{weebcentral,mangadex}.js` parse source catalogs and pages. `scraper/jobs/{weebcentral,mangadex}-archive.js` repeat transport, checkpoint, metadata, and publication steps. `scraper/lib/http.js` stops on 401/403/429. `scraper/lib/publisher.js` makes chapters visible atomically.
+- Adding another hoster today means copying a whole archive runner. The plugin work must eliminate that repetition **without** allowing plugin code to write the library or bypass request controls.
+
+## Boundary
+
+```text
+Admin/UI -> queue + source ledger -> scraper scheduler -> first-party plugin worker
+                                   |                  ^
+                                   v                  | request/response IPC
+                              HTTP broker ----------+
+                                   |
+                                   v
+                        host / approved image CDN
+
+scraper scheduler -> library inventory + scope/dedupe -> common downloader
+                  -> existing checkpoint/publisher -> library
+```
+
+The worker describes requests and parses replies. It has no library or state mount and no internet-facing network. The scraper-owned broker makes **every** source request, including image requests. It enforces one source lane, configured request/chapter delays, declared hostname allowlists, redirect and resolved-address checks, response size/time limits, and stop-on-401/403/429/challenge behavior. HTTP 5xx and network retry remain centrally bounded. No proxy/identity/user-agent rotation and no challenge solver. A source that requires a protection bypass is reported as unsupported/blocked, not silently routed elsewhere.
+
+For production Compose, run one small `plugin-worker` instance per enabled plugin from the same pinned first-party image. Give each a dedicated internal-only network shared with the scraper (not the general backend or `egress` network); mount no library/data/secrets; use a read-only filesystem, dropped capabilities, no-new-privileges, and memory/PID limits. Do **not** inherit Compose's existing `x-app` anchor, which mounts the scraper token. Only the scraper has egress and library write access. On Windows local development, use the same Compose worker: an ordinary Node child process is crash isolation but **not** a network sandbox. No exposed plugin port or browser-to-worker route.
+
+### Proposed v1 IPC contract
+
+Transport: versioned JSON messages over a private internal HTTP endpoint or stdio; bounded payloads and request IDs. The exact transport is an implementation choice, not exposed to users. A worker is pinned to one `pluginId` and code revision. The host validates every returned object.
+
+```json
+{ "v": 1, "id": "req-1", "op": "catalog", "pluginId": "example", "seriesUrl": "https://allowed.example/series/123" }
+{ "v": 1, "id": "req-2", "op": "fetch", "url": "https://allowed.example/series/123", "accept": "text/html" }
+{ "v": 1, "id": "req-2", "ok": true, "status": 200, "contentType": "text/html", "body": "...bounded..." }
+{ "v": 1, "id": "req-1", "ok": true, "title": "...", "chapters": [{ "id": "source-chapter-id", "number": "1", "volume": "1", "url": "https://allowed.example/chapter/1" }], "coverage": "unknown" }
+```
+
+Operations: `match` from a declarative manifest, `catalog`, `chapterPages`, optional `search`/`seriesMetadata` later. `chapterPages` returns ordered image descriptors; **only** the common downloader fetches bytes. Plugin output may report `unrecognized-labels`, `external-only`, and confidence/evidence, but may not mark rights verified or override the library's work identity. Plugin failures are typed (`parser`, `catalog-empty`, `network`, `access-block`, `rate-limit`) and visible in the source ledger. Version/schema mismatch disables that plugin lane without losing checkpoints.
+
+Manifest, reviewed with code: stable plugin ID/version, exact series URL matcher, primary hostnames, separately reviewed image/CDN hostnames, optional language, parser fixture set, upstream provenance, license, and support state (`experimental`/`supported`/`disabled`). Hostnames are **not** administrator-provided at runtime in v1. Redirects and DNS answers must be checked again by the broker; declared hostnames alone do not protect against SSRF.
+
+## Uchiyomi code triage
+
+| Upstream implementation | Use in MangaCircus | Rationale |
+| --- | --- | --- |
+| [`sources/types.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/types.ts) | Adapt **concept**, write our own contract | Its narrow catalog/page operations are valuable; its Cloudflare/image-header capabilities and database-specific assumptions are not our policy. |
+| [`sources/engines/madara.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/engines/madara.ts) | Candidate for pure parser extraction | Broad WordPress/Madara markup; separate parsing from `cfGet/cfPost`, test with fixtures, then trial only on a specifically reviewed accessible host. Not a blanket license to scrape every Madara site. |
+| [`sources/engines/mangathemesia.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/engines/mangathemesia.ts) | Later candidate | Similar, but source-specific page and image hosts need review; currently solver-coupled. |
+| [`sources/engines/manganato.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/engines/manganato.ts) | Later candidate, not first pilot | More variant/fallback logic and an explicit challenge dependency; higher maintenance and source-policy risk. |
+| [`sources/mangadex.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/mangadex.ts) | Compare behavior/tests only | MangaCircus already has a MangaDex adapter. Evaluate its language and rate-limit edge cases without replacing working code wholesale. |
+| [`sources/suwayomi/*`](https://github.com/AngeloSha/uchiyomi/tree/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/suwayomi), [`loader.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/sources/loader.ts) | Do not port | Designed to run operator-supplied APKs/JS. Conflicts with the first-party-only requirement. |
+| [`downloader.ts`](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/bff/src/lib/downloader.ts) | Learn from tests/failure cases, do not transplant | CBZ/database assumptions differ from MangaCircus's folder/checkpoint publisher; its challenge session handling does not fit our policy. |
+
+The upstream `sources/` directory contains no bundled plugin implementations beyond a placeholder; the generic engines above live in its main server. Do not mistake its extension catalog for code licensed by Uchiyomi: separately authored APKs have separately determined provenance and licenses.
+
+### Code provenance and license gate
+
+Uchiyomi is [MPL-2.0](https://github.com/AngeloSha/uchiyomi/blob/726bd9fa6fe0090fd4b5de1cdb2b6926e5382127/LICENSE). **No verbatim or adapted upstream code has been copied into MangaCircus in this preparation.** Before copying any implementation, identify the exact upstream file/revision and third-party notices, preserve attribution/notices, add the MPL license and a `THIRD_PARTY_SOURCES.md` ledger, and keep copied/modified MPL-covered code in clearly marked files. If distributing a Docker image with those files, provide the corresponding source for the covered files as the license requires. Mozilla describes MPL as file-level copyleft; it does not require unrelated MangaCircus files to become MPL. This is an implementation checklist, not legal advice; review the final distribution arrangement before release. Reimplementing an idea from documented behavior is distinct from copying its expressive code, but provenance should still be documented.
+
+## Incremental implementation passes
+
+1. **Contract and broker, no new host.** Add manifest validation and IPC schema, central fetch broker, disabled-by-default worker, fixture-driven contract tests, worker crash/timeouts, SSRF/redirect tests, 401/403/429 stop tests, and checkpoint-preservation tests. Existing adapters continue unchanged.
+2. **First-party compatibility plugins.** Migrate MangaDex and WeebCentral catalog/page parsing one at a time behind the contract, with old-vs-new fixture parity. Keep existing job IDs, source IDs, queue order, delays, and work directory paths. The common runner calls the existing inventory/scope/publisher. Rollback is a plugin enable switch, not a queue rewrite.
+3. **One reviewed generic parser pilot.** Start with a pure Madara-family parser based on upstream lessons. Choose a single source only after checking accessibility, terms, stable hostnames, and parser fixtures. Inspection/catalog-only first; no auto-queue. Record source evidence, compare with library identity, then test a small chapter in isolated A/B fixtures. Add further families only if they solve a concrete catalog gap.
+4. **Admin UI.** Under Scraper → Sources, show installed first-party plugins, version/provenance, enabled state, supported hosts, last probe stage, observed catalog/coverage, rights state, and blocked reason. Enabling requires admin confirmation; no repository URL/APK upload/install button. Source candidates stay leads until reviewed and explicitly queued.
+5. **Deployment.** Add the worker to local and VM Compose with internal-only networking and resource limits; run compose checks and end-to-end fixture tests, then stage on a test instance. Leave the live stack untouched until jobs are checkpoint-safe and rollback verified. Document plugin upgrades and recovery.
+
+Acceptance: no plugin has direct internet or library-write capability; one worker per plugin may crash without taking down the reader/scraper; no source request can evade host limits or 401/403/429 handling; existing jobs resume unchanged across a rollback; the library still needs no database; source provenance and MPL obligations are traceable; no dynamic third-party code execution is introduced.