Consolidate MangaCircus app, scraper, and Compose baseline
This commit is contained in:
Maashkinoozhe committed
1 parent
cfb2d79103
commit
eef4ffa07c
103 files changed
+9316
-172
No files matched your search
@@ -0,0 +1,228 @@
|
||||
# Managed scraper service
|
||||
|
||||
The scraper is an optional Node.js daemon that runs beside the MangaCircus reader.
|
||||
It owns the durable queue and is the only process allowed to run a queued archive.
|
||||
The browser never contacts it directly: authenticated admin routes in `server.js`
|
||||
proxy commands to its private HTTP API.
|
||||
|
||||
## Start and storage
|
||||
|
||||
```bash
|
||||
npm run start:all # reader + daemon under one local supervisor
|
||||
npm start # reader only
|
||||
npm run start:scraper # daemon only
|
||||
```
|
||||
|
||||
These commands are for native development only. The current Windows installation
|
||||
runs the three-service Docker Compose stack; the old Scheduled Task is disabled.
|
||||
Never start a native supervisor or daemon against its mounted state. Its observed
|
||||
ports, logs, persisted settings, restart procedure, and recovery snapshot are recorded in
|
||||
[`local-deployment.md`](local-deployment.md).
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `SCRAPER_HOST` | `127.0.0.1` | Daemon bind address |
|
||||
| `SCRAPER_PORT` | `8091` | Daemon port |
|
||||
| `SCRAPER_TOKEN` | empty | Bearer token; mandatory when the bind address is not loopback |
|
||||
| `SCRAPER_DATA` | `<DATA>/scraper` | Durable queue, lock, and event log |
|
||||
| `LIBRARY` | `./library` | Shared writable library for the daemon |
|
||||
| `SCRAPER_URL` | `http://127.0.0.1:8091` | Daemon URL used by the reader |
|
||||
|
||||
The authoritative queue is `data/scraper/queue.json` (currently schema version 4),
|
||||
written atomically. Individual
|
||||
archives retain their existing resumable checkpoints under
|
||||
`library/.mcscraper-work-<source-id>/job.json`. `events.log` is append-only.
|
||||
|
||||
On its first start, the daemon imports the old `.mcscraper-queue/queue.json` wishlist
|
||||
when present. Completed chapter ids in each archive checkpoint remain authoritative,
|
||||
so migration does not redownload published chapters.
|
||||
|
||||
## Lifecycle
|
||||
|
||||
The daemon owns one durable queue and runs at most one sequential worker per source. Different sources may run side by side, but two jobs from the same source never overlap:
|
||||
|
||||
1. Select the first `queued` item for every idle, unpaused source.
|
||||
2. Resolve its URL through the adapter registry in `scraper/adapters/index.js`.
|
||||
3. Run one adapter job per source with that source's request and chapter delays.
|
||||
4. For legacy in-progress checkpoints, continue publishing into their private work directory and expose the complete manga at the end.
|
||||
5. For every new job, stage one chapter beside its final book and expose that complete chapter immediately with one rename.
|
||||
6. Increment `publishedRevision` after every newly visible chapter and start the next item for that source when the selected scope is complete.
|
||||
|
||||
Legacy work can be converted without redownloading. While the queue is paused and no
|
||||
worker is active, `publishLegacyReady` validates every completed chapter, quarantines
|
||||
abandoned hidden partial stages, atomically exposes the accumulated manga folder, and
|
||||
changes its checkpoint to incremental mode. The resumed job then derives its remaining
|
||||
work from the visible library and publishes every later chapter immediately.
|
||||
|
||||
A source failure pauses only that source, allowing unrelated source workers to continue. Manual pause remains global. Request and chapter delays are configured independently for every installed host in **Settings → Scraper**. Both current adapters default to the fast profile: 1–4 seconds between sequential requests and 5–15 seconds between chapters.
|
||||
|
||||
The reader polls `publishedRevision` every 15 seconds and rescans after it increases.
|
||||
A source error pauses only that source lane for human review. Pause and shutdown requests
|
||||
are checked between requests; validated partial pages are retained in hidden,
|
||||
fingerprinted staging and reused only when the complete manifest still matches.
|
||||
|
||||
## Inspect, confirm, and interpret completion
|
||||
|
||||
The admin form uses **Inspect & queue** (`inspect:true`). This queues a catalog-only
|
||||
`kind:"inspection"` item on the normal sequential source lane, at the configured
|
||||
delays. No image or cover requests occur during inspection. Successful inspection
|
||||
sets `status:"ready"`, displays the canonical `sourceTitle` and source availability,
|
||||
and waits for **Confirm download**. Ready items survive daemon restarts and suppress
|
||||
duplicate nightly checks. Other jobs may proceed while confirmation is pending.
|
||||
Confirmation does not resume a deliberately paused queue or clear a source block.
|
||||
The download rediscovers the catalog; an edition title change requires re-inspection
|
||||
and confirmation. Direct API clients can still enqueue authorized downloads without
|
||||
inspection by omitting the flag. Existing download/discovery jobs are unchanged.
|
||||
|
||||
**Inspect again** operates on an inactive ready, completed, unavailable or quarantined
|
||||
item. It never overrides a source access block and does not fetch external-link pages.
|
||||
Old MangaDex unavailable messages are marked as needing re-inspection, not silently
|
||||
retried. No remote diagnosis is inferred from that old numeric-only adapter result.
|
||||
If the item was a completed download, inspection compares its selected chapters with
|
||||
the shared library inventory. An already-present selection remains **Complete** with
|
||||
its original completion and publication counts; it does not become an awaiting-download
|
||||
item. Newly available chapters instead remain **Ready** for explicit confirmation.
|
||||
WeebCentral catalog inspection accepts numbered Chapter, Episode, Rating, Night,
|
||||
Log and Act labels. If chapter links exist but none can be numbered, the parser
|
||||
reports a label-format problem; an actually empty catalog is reported separately.
|
||||
|
||||
`complete` means the selected **job** finished, not that a series is fully loaded.
|
||||
`publishedCount` counts chapters newly exposed by that run; `selectedCount` is its
|
||||
remaining selected scope. Resume may reduce those counts because the inventory skips
|
||||
chapters already published. MangaDex availability records hosted release/selection
|
||||
counts, unnumbered release counts, metadata's last chapter and the highest hosted
|
||||
number. A lower hosted maximum proves partial coverage; an equal maximum does **not**
|
||||
prove completeness. Other cases say completeness is unverified. The last chapter
|
||||
number is never used as a chapter count.
|
||||
|
||||
Empty English hosted catalogs distinguish no English releases listed, English
|
||||
external-link-only releases, and no hosted English releases with pages. External
|
||||
availability is checked through a separate, paced `includeExternalUrl=1` feed request
|
||||
only when hosted selections are empty; downloads use `includeExternalUrl=0`.
|
||||
|
||||
## Queue scopes and duplicate policy
|
||||
|
||||
Every new queue item has its own UUID plus the stable upstream `seriesId`. This allows
|
||||
several jobs for one manga (for example volume 2 followed by volume 3) without sharing
|
||||
queue identity. Its `scope` is one of:
|
||||
|
||||
| Mode | Meaning |
|
||||
|---|---|
|
||||
| `missing` | Every eligible chapter absent from the library |
|
||||
| `latest` | The newest eligible missing chapter |
|
||||
| `last` / `first` | The last/first `count` eligible missing chapters |
|
||||
| `chapter` | One numeric chapter |
|
||||
| `after` | Every numeric chapter after the supplied chapter |
|
||||
| `volume` | Chapters whose source catalog explicitly reports that volume |
|
||||
| `ids` | Internal exact source chapter ids produced by nightly discovery |
|
||||
|
||||
Unnumbered MangaDex releases use `special-mangadex-<chapter-uuid>` as their logical
|
||||
key, never an invented chapter zero or one. Numbered release deduplication is unchanged;
|
||||
unnumbered releases stay distinct unless their source UUID is identical. Two pilot
|
||||
translations therefore remain two labeled, group-credited releases, not a guessed
|
||||
numbered chapter or silently merged translation. Numeric `chapter`/`after` scopes
|
||||
exclude them; missing, exact-ID and applicable volume scopes include them. Catalog
|
||||
ordering puts numbered chapters first and source-ID-ordered specials afterward;
|
||||
first/last/latest operate on that ordering, not on publication timestamps.
|
||||
|
||||
`existingPolicy:"skip-any"` skips a chapter when any version already exists.
|
||||
`existingPolicy:"add-source"` permits another host's version but still skips a version
|
||||
already present from the selected host. Existing version folders are never overwritten.
|
||||
|
||||
Before downloading, the scraper reads a lightweight library inventory. Work identity is
|
||||
matched by exact source id, exact source URL, explicit library manga id, or verified
|
||||
AniList id—never by fuzzy title. The filesystem library remains authoritative: a stale
|
||||
checkpoint cannot make a missing published chapter look complete.
|
||||
|
||||
## Nightly discovery
|
||||
|
||||
`manga.json` stores the admin override at `x-scraper.autoUpdate`. When it is absent,
|
||||
`status:"ongoing"` defaults to enabled and other statuses default to disabled. At the
|
||||
configured local `autoUpdateTime`, the daemon queues one catalog-only discovery item for
|
||||
the preferred supported source of every enabled manga. Discovery shares the normal
|
||||
sequential source lane and pacing. If strictly newer eligible chapters are found, that
|
||||
same item is promoted to a download job containing their source ids. A successful
|
||||
no-change check records `lastCheckedAt` and its result in the per-manga/source
|
||||
update summary, appends history, and retires the discovery queue row without page
|
||||
requests. `lastDownloadedAt` is recorded when a chapter is published or a download
|
||||
job completes. These summaries persist in `queue.json` as `updateChecks`; on load,
|
||||
legacy completed no-change discovery rows migrate into the summary and are retired.
|
||||
Failed checks remain visible for attention. Duplicate pending checks for the same
|
||||
source series are suppressed.
|
||||
Only numbered chapters strictly newer than the library's highest regular chapter,
|
||||
plus unseen source-ID specials, are considered updates. Historical numbered gaps
|
||||
are never backfilled by the nightly task; use an explicit queue
|
||||
scope when a gap is intentional work.
|
||||
|
||||
## Private daemon API
|
||||
|
||||
All responses are JSON. When `SCRAPER_TOKEN` is set, send
|
||||
`Authorization: Bearer <token>`.
|
||||
|
||||
| Method | Path | Body / result |
|
||||
|---|---|---|
|
||||
| `GET` | `/health` | Basic liveness |
|
||||
| `GET` | `/api/state` | Queue, all active source checkpoints, settings, errors, history, adapters |
|
||||
| `POST` | `/api/queue` | `{url,title?,metadataSearch?,scope?,existingPolicy?,libraryMangaId?,inspect?}` |
|
||||
| `POST` | `/api/queue/remove` | `{id}`; active downloads cannot be removed |
|
||||
| `POST` | `/api/queue/clear-completed` | `{source?}`; remove inactive successful download job rows only; preserve library, history, checkpoints and other statuses |
|
||||
| `POST` | `/api/queue/reorder` | `{ids}` containing every queue id exactly once |
|
||||
| `POST` | `/api/control` | `{action:"pause"|"resume"|"retry"|"inspect"|"confirm"|"skip",id?}` |
|
||||
| `POST` | `/api/settings` | `sourceRates.{adapter}.{minMs,maxMs,chapterMinMs,chapterMaxMs}` plus `autoUpdateEnabled` and local `autoUpdateTime` (`HH:MM`) |
|
||||
| `POST` | `/api/discovery/run` | Queue an immediate update check for every enabled manga |
|
||||
| `POST` | `/api/library/auto-update` | `{id,enabled}` updates `manga.json` atomically |
|
||||
| `POST` | `/api/legacy/publish-ready` | Paused-only validation/publication of completed legacy work; converts its checkpoint to incremental mode |
|
||||
|
||||
Legacy top-level rate fields remain a fallback for adapters without a per-source entry. Settings also accept `autoUpdateEnabled` and a local `autoUpdateTime` in `HH:MM` form.
|
||||
|
||||
The reader exposes only `GET/POST /api/admin/scraper`, protected by the existing
|
||||
admin session. It maps a small command allowlist to the private API and never returns
|
||||
the shared token.
|
||||
|
||||
## Adapter contract
|
||||
|
||||
Each source plugin is registered in `scraper/adapters/index.js` with:
|
||||
|
||||
```js
|
||||
{
|
||||
id: 'host-id',
|
||||
label: 'Human name',
|
||||
match(url) { /* return {source,id,title,url} or null */ },
|
||||
async discover(item, { settings, shouldStop, onEvent }) { /* catalog only */ },
|
||||
async run(item, {
|
||||
libraryRoot, settings, shouldStop, onPublished, onCatalog
|
||||
}) { /* publish selected chapters and return counts/final path */ }
|
||||
}
|
||||
```
|
||||
|
||||
Adapters discover host-specific catalogs and pages, but reuse the central pacing,
|
||||
checkpoint, metadata, and atomic publisher modules. They must stop on HTTP 401, 403,
|
||||
429, CAPTCHA, challenge, or access-denied responses. Never add protection bypasses.
|
||||
|
||||
`onCatalog(catalog)` persists source title and optional availability before any page
|
||||
acquisition. The MangaDex adapter accepts only explicit title UUID URLs, verifies
|
||||
AniList cross-references, selects one English release per numeric chapter, preserves
|
||||
unnumbered source identities, credits groups and downloads sequential full-resolution
|
||||
MangaDex@Home pages. Special-only editions do not use fuzzy AniList lookup or AniList
|
||||
fallback library matching; their canonical source title names new folders. Explicit
|
||||
library targets remain operator choices; existing folders are never renamed.
|
||||
|
||||
## Admin UI
|
||||
|
||||
**Settings → Admin → Scraper** polls the authenticated proxy every three seconds
|
||||
only while that section is active. Activity, Queue, Updates and Sources are
|
||||
separate local tabs with a shared source filter. The section shows:
|
||||
|
||||
- every active source, manga, book/season, chapter, page, and overall progress;
|
||||
- source warnings and fatal errors;
|
||||
- queue ordering and removal controls;
|
||||
- add-by-URL, pause, resume, and retry controls;
|
||||
- queue scope, exact chapter/count/volume, target manga, and alternate-source controls;
|
||||
- per-manga auto-update checkboxes, nightly time, and a manual **Check now** action;
|
||||
- last checked and last downloaded times per manga/source, with old check events
|
||||
collapsed into recent history instead of occupying queue space;
|
||||
- a collapsed completed-jobs section with confirmation before bulk cleanup;
|
||||
- persistent image and chapter delay ranges.
|
||||
|
||||
Rate changes apply on the next manga. To apply them to the active manga, pause and
|
||||
resume it; the current chapter restarts safely if it was incomplete.
|
||||
Reference in new issue
Block a user