Consolidate MangaCircus app, scraper, and Compose baseline
This commit is contained in:
Maashkinoozhe committed
1 parent
cfb2d79103
commit
eef4ffa07c
103 files changed
+9316
-172
No files matched your search
@@ -0,0 +1,155 @@
|
||||
# MangaCircus scraper agent runbook
|
||||
|
||||
This document is normative for automated agents operating the scraper. Read it together with `../docs/scraper-guide.md`.
|
||||
|
||||
## 1. Non-negotiable safety policy
|
||||
|
||||
1. Scrape only works the operator is authorised to copy.
|
||||
2. Never bypass a CAPTCHA, Cloudflare challenge, login, paywall, geographic restriction, signed-URL check, or other access control.
|
||||
3. If a normal browser session displays a challenge, stop. A human may complete it normally; the agent may resume afterward. Repeated challenges end the run.
|
||||
4. Never request pages concurrently. Use the persisted per-host settings. The current operator-selected fast profile is 1–4 seconds between requests/images and 5–15 seconds between chapters. A host-specific published limit overrides these settings when it is slower.
|
||||
5. Stop immediately on HTTP 401, 403, or 429. On 5xx, retry at most twice after 30 and 60 seconds. Do not rotate IP addresses, identities, user agents, or mirrors to evade a block.
|
||||
6. Inspect metadata through AniList and its local 30-day cache. Do not revisit a reader/image host merely for authors, genres, descriptions, status, year, or cover metadata.
|
||||
7. Download only missing chapters. Never probe every historical chapter merely to discover the newest one when a chapter list is available.
|
||||
8. Publish only through the shared publisher, either `scraper/cli.js publish` or the
|
||||
managed daemon. Both validate images and use MangaCircus-safe staging and rename
|
||||
operations. Legacy conversion must be paused, validate every completed chapter,
|
||||
quarantine incomplete stages, and use an atomic manga-folder rename.
|
||||
|
||||
## 2. Pipeline
|
||||
|
||||
Runtime recovery is documented in `../docs/resilience.md`. Request-level 5xx retries
|
||||
remain limited to 30/60 seconds; after those are exhausted the daemon can resume the
|
||||
same checkpoint once after a five-minute source cooldown. No automatic access-error
|
||||
retry is permitted. Unavailable titles and parser failures are job-scoped; repeated
|
||||
independent parser failures stop the source lane. Sentinel never changes adapters.
|
||||
|
||||
```text
|
||||
public metadata -> cached series metadata
|
||||
chapter host -> newest chapter IDs only
|
||||
reader/browser -> ordered page manifest or local browser asset bundle
|
||||
manifest validation -> slow sequential acquisition -> hidden staging folder
|
||||
complete chapter -> one atomic rename -> optional one MangaCircus rescan per batch
|
||||
```
|
||||
|
||||
The interchange format is `manifest.example.json`. Each page has exactly one of:
|
||||
|
||||
- `path`: an already downloaded browser asset; preferred for protected/browser-only hosts.
|
||||
- `url`: a public image URL fetched sequentially by the publisher.
|
||||
|
||||
Do not store cookies, tokens, signed URLs, account identifiers, or other secrets in `manga.json`. Temporary manifests containing expiring URLs must stay outside the library and should be deleted after a successful import.
|
||||
|
||||
## 3. Metadata procedure
|
||||
|
||||
Run one AniList lookup per new series:
|
||||
|
||||
```bash
|
||||
node scraper/cli.js metadata-anilist "series title"
|
||||
```
|
||||
|
||||
Verify the result using title, native title, author and year. Copy the returned fields into `series.metadata`. Keep the chapter host URLs only in `series.sources`. If the lookup is ambiguous, stop for human selection; never silently attach metadata from a similarly named work.
|
||||
|
||||
For the selected Heavy Knight manga, the verified AniList identity is `150193` / MAL `147799`. AniList `150194` is the light novel and MUST NOT be used for manga metadata or cover art. For One Piece it is AniList `30013` / MAL `13`.
|
||||
|
||||
## 4. WeebCentral adapter
|
||||
|
||||
Observed 2026-10-01:
|
||||
|
||||
- Series: `https://weebcentral.com/series/<id>`
|
||||
- Full list: `/series/<id>/full-chapter-list`
|
||||
- Chapter: `/chapters/<id>`
|
||||
- Page fragment: `/chapters/<id>/images?is_prev=False&reading_style=long_strip¤t_page=1`
|
||||
|
||||
The fragment contains the ordered image elements. Fetch the series page once, the full list once, and only the selected chapter/page fragments. Current chapter 177 exposed 15 PNG pages. Do not fetch cover art from this host when AniList metadata is available.
|
||||
|
||||
```bash
|
||||
node scraper/cli.js discover-weebcentral "SERIES_URL" --limit 3 --out manifest.json
|
||||
node scraper/cli.js publish manifest.json --library library
|
||||
```
|
||||
|
||||
If the HTML no longer matches, stop and use a normal browser to inspect the visible reader. Do not guess endpoint grids.
|
||||
|
||||
## 5. ReiManga adapter
|
||||
|
||||
Observed 2026-10-01: anonymous scripted requests receive a managed Cloudflare challenge. The indexed series page lists chapters 177 downward, but that index is not sufficient to acquire pages.
|
||||
|
||||
Required approach:
|
||||
|
||||
1. Open the supplied series page in a normal browser.
|
||||
2. If a challenge appears, stop for the human. Do not automate it.
|
||||
3. After the user reaches the normal series page, open only the selected chapter.
|
||||
4. Read the visible chapter number/title/date and inventory the image assets already rendered by the page.
|
||||
5. Export only assets belonging to that chapter into a local temporary folder.
|
||||
6. Create a manifest with local `path` entries and source name `ReiManga`.
|
||||
7. Publish locally. Wait 30–60 seconds before opening another chapter.
|
||||
|
||||
If the site sends the chapter to another domain, record the ReiManga chapter URL as the source URL, but accept assets only from domains actually loaded by that reader page. Any login, paywall, error, or renewed challenge ends the run.
|
||||
|
||||
## 5A. MangaDex adapter
|
||||
|
||||
Observed 2026-10-03:
|
||||
|
||||
- Title: `https://mangadex.org/title/<uuid>`
|
||||
- API: `https://api.mangadex.org`
|
||||
- English feed: `/manga/<uuid>/feed`
|
||||
- Page manifest: `/at-home/server/<chapter-uuid>`
|
||||
- Images: the HTTPS MangaDex@Home host returned by that manifest
|
||||
|
||||
Only explicit MangaDex title UUIDs may enter the managed queue. Never automatically replace a queued source from fuzzy title search: verify the MangaDex AniList/MAL link against the selected work first. Select one stable release per numeric English chapter, credit MangaDex and the scanlation group in the chapter source, and ignore external-link or zero-page releases. Honor group removal requests. MangaDex-backed content must not be used to support advertising or paid access.
|
||||
|
||||
Unnumbered hosted releases are supported with `special-mangadex-<chapter-uuid>` keys
|
||||
and human labels, never guessed chapter numbers. Keep distinct source UUIDs separate.
|
||||
Use the queued catalog-only inspection and confirm the canonical edition before
|
||||
downloading a new title. Empty hosted-English catalogs distinguish external-only,
|
||||
no-English-listed and no-hosted-pages; none is an access error or a reason to probe
|
||||
external hosts. A finished job does not prove whole-series coverage. See
|
||||
`../docs/scraper-service.md` for inspection, scopes and availability semantics.
|
||||
|
||||
Use full-resolution `data` pages, validate that the manifest page count still matches the feed, and accept only HTTPS `uploads.mangadex.org` or `*.mangadex.network` image hosts. Keep one sequential MangaDex worker using its configured per-host rate (currently 1–4 seconds between requests and 5–15 seconds between chapters). Stop on 401, 403, or 429 exactly as for other sources; never add authentication or rotate hosts to evade a response.
|
||||
|
||||
## 6. MANGA Plus adapter
|
||||
|
||||
Observed 2026-10-01 for title `100020` (One Piece): the title page states that only the latest three chapters remain viewable. They are:
|
||||
|
||||
| Chapter | Viewer id | Published | Title |
|
||||
|---|---:|---|---|
|
||||
| 1192 | 1030379 | 2026-09-06 | We Won't Stand For It!!! |
|
||||
| 1193 | 1030380 | 2026-09-13 | Still Practicing |
|
||||
| 1194 | 1030381 | 2026-09-27 | All Things Are In Flux |
|
||||
|
||||
The normal web app calls `title_detailV4` and `manga_viewer_v3`, then renders signed `jumpg-assets*.tokyo-cdn.com` payloads. Direct non-browser API calls currently return an account-ban response. A browser asset export on 2026-10-01 produced payloads labelled `image/jpeg`, but their bytes were obfuscated and failed JPEG signature validation. Therefore:
|
||||
|
||||
1. Use only the normal MANGA Plus browser UI.
|
||||
2. Read the title page and select the last three chapters explicitly marked viewable.
|
||||
3. Open one viewer at a time, wait for its page count to stabilise, and inventory the rendered assets.
|
||||
4. Export only URLs matching `/secure/title/100020/chapter/<viewer-id>/manga_page/` from that page's observed assets.
|
||||
5. Validate the first exported payload by file signature before exporting or processing the rest. A `.jpg` name or HTTP content type is not proof of a JPEG.
|
||||
6. If valid images are available, preserve numeric order and use local `path` entries in the import manifest. Set source name `MANGA Plus`, source URL `https://mangaplus.shueisha.co.jp/viewer/<viewer-id>`, priority `1`, direction `rtl`.
|
||||
7. If payloads are obfuscated, abort the source. Do not decrypt them, extract pixels from the rendering canvas, or reconstruct chapters with automated screenshots.
|
||||
8. Wait at least 30 seconds between chapters. If the UI says a chapter is unavailable, do not attempt alternate APIs or URLs.
|
||||
|
||||
Do not implement API protobuf decoding, image deobfuscation, signed-URL generation, device-id synthesis, canvas extraction, screenshot reconstruction, or retry identity rotation. The browser-exported file boundary plus ordinary image validation is the allowed boundary. As of the observation date, chapters 1192–1194 are identified but **not importable by this compliant adapter**.
|
||||
|
||||
## 7. MangaCircus publication rules
|
||||
|
||||
- Use the flat book strategy: `v01`.
|
||||
- Chapter folders include stable source suffixes, for example `c1194 All Things Are In Flux [MANGA Plus]`.
|
||||
- Existing chapters are skipped by default. `--replace` replaces the entire version folder atomically; never overwrite image files in place.
|
||||
- New manga are built wholly under `.staging-mcscraper-*` and published with one rename.
|
||||
- The publisher sniffs image bytes; filename extensions are never trusted.
|
||||
- Page numbers are contiguous from `0001`.
|
||||
- Add `cover`, `double`, `extra`, or rotation tags only when certain. Do not infer tags from aspect ratio alone.
|
||||
- Trigger at most one MangaCircus rescan after the whole batch.
|
||||
|
||||
## 8. Abort checklist
|
||||
|
||||
Abort without workaround when any of these occurs:
|
||||
|
||||
- CAPTCHA, challenge, access-denied, rate-limit, account-ban, or login/paywall prompt.
|
||||
- Page count changes during acquisition.
|
||||
- Duplicate/non-numeric page order, unsupported image bytes, or an HTML error page disguised as an image.
|
||||
- Metadata match is ambiguous.
|
||||
- The target manga folder exists but belongs to another source identity.
|
||||
- A chapter would need to move books or rename an existing version folder.
|
||||
|
||||
Log the URL, status, and phase, but never log cookies, query signatures, passwords, or browser storage.
|
||||
@@ -0,0 +1,48 @@
|
||||
# MangaCircus scraper
|
||||
|
||||
The scraper uses small host adapters to create a common manifest, followed by one conservative publisher that writes the MangaCircus library atomically.
|
||||
|
||||
Start with [AGENT_RUNBOOK.md](AGENT_RUNBOOK.md). It contains the host-specific procedures, safety limits, and stop conditions. `manifest.example.json` documents the interchange format.
|
||||
|
||||
The historical human-readable wishlist is [WISHLIST.md](WISHLIST.md). Once the
|
||||
managed daemon has started, `data/scraper/queue.json` is the authoritative queue and
|
||||
is edited through the admin UI.
|
||||
|
||||
## Managed service
|
||||
|
||||
Run MangaCircus and the scraper daemon together:
|
||||
|
||||
```bash
|
||||
npm run start:all
|
||||
```
|
||||
|
||||
Or run them as two independently supervised processes:
|
||||
|
||||
```bash
|
||||
npm start
|
||||
npm run start:scraper
|
||||
```
|
||||
|
||||
The daemon binds to `127.0.0.1:8091` by default, keeps its durable queue in
|
||||
`data/scraper/queue.json`, and resumes incomplete archive checkpoints. An admin can
|
||||
control it from **Settings → Scraper**. Set `SCRAPER_TOKEN` on both processes when
|
||||
the daemon is not restricted to loopback.
|
||||
|
||||
Queue schema version 3 supports scoped jobs, multiple jobs for the same upstream
|
||||
series, one sequential worker per source, incremental chapter publication, library
|
||||
duplicate/version checks, and configurable nightly update discovery. See
|
||||
[`docs/scraper-service.md`](../docs/scraper-service.md) for the service contract and
|
||||
[`docs/local-deployment.md`](../docs/local-deployment.md) for the currently running
|
||||
Windows Scheduled Task deployment. On that machine the scheduled task owns the single
|
||||
supervisor; do not run `npm run start:all` beside it.
|
||||
|
||||
```bash
|
||||
npm run test:scraper
|
||||
node scraper/cli.js metadata-anilist "One Piece"
|
||||
node scraper/cli.js discover-weebcentral "https://weebcentral.com/series/..." --limit 3 --out manifest.json
|
||||
node scraper/cli.js publish manifest.json --library library --metadata-search "Series title"
|
||||
node scraper/cli.js archive-weebcentral "https://weebcentral.com/series/..." --library library --metadata-search "Series title"
|
||||
node scraper/cli.js status-weebcentral "https://weebcentral.com/series/..." --library library
|
||||
```
|
||||
|
||||
The implementation uses only Node.js built-ins. Browser-only sources must first export observed page assets to local files; the publisher consumes those files and does not automate access-control challenges.
|
||||
@@ -0,0 +1,95 @@
|
||||
'use strict';
|
||||
|
||||
const { runArchive } = require('../jobs/weebcentral-archive');
|
||||
const { runMangaDexArchive } = require('../jobs/mangadex-archive');
|
||||
const { SlowHttp } = require('../lib/http');
|
||||
const { discoverWeebCatalog } = require('./weebcentral');
|
||||
const { mangaDexInfo, discoverMangaDexCatalog } = require('./mangadex');
|
||||
const { ArchiveControlError } = require('../lib/control');
|
||||
|
||||
function controlledHttp(http, shouldStop) {
|
||||
return { async get(...args) {
|
||||
const check = () => { const reason = shouldStop?.(); if (reason) throw new ArchiveControlError(reason === true ? 'paused' : String(reason)); };
|
||||
check(); const response = await http.get(...args); check(); return response;
|
||||
} };
|
||||
}
|
||||
|
||||
function weebCentralInfo(value) {
|
||||
let url;
|
||||
try { url = new URL(String(value)); } catch { return null; }
|
||||
if (url.protocol !== 'https:' || url.hostname !== 'weebcentral.com') return null;
|
||||
const match = /^\/series\/([A-Z0-9]+)(?:\/([^/?#]+))?\/?$/i.exec(url.pathname);
|
||||
if (!match) return null;
|
||||
const title = decodeURIComponent(match[2] || match[1]).replace(/[-_]+/g, ' ').trim();
|
||||
return { source: 'weebcentral', id: match[1].toUpperCase(), title, url: `https://weebcentral.com/series/${match[1]}${match[2] ? `/${match[2]}` : ''}` };
|
||||
}
|
||||
|
||||
const adapters = [{
|
||||
id: 'weebcentral',
|
||||
label: 'WeebCentral',
|
||||
match: weebCentralInfo,
|
||||
async discover(item, options) {
|
||||
const http = new SlowHttp({
|
||||
minMs: options.settings.minMs, maxMs: options.settings.maxMs, maxRetries: 2,
|
||||
onEvent: options.onEvent,
|
||||
});
|
||||
return discoverWeebCatalog(item.url, controlledHttp(http, options.shouldStop));
|
||||
},
|
||||
run(item, options) {
|
||||
return runArchive({
|
||||
seriesUrl: item.url,
|
||||
libraryRoot: options.libraryRoot,
|
||||
metadataSearch: item.metadataSearch || item.title,
|
||||
minMs: options.settings.minMs,
|
||||
maxMs: options.settings.maxMs,
|
||||
chapterMinMs: options.settings.chapterMinMs,
|
||||
chapterMaxMs: options.settings.chapterMaxMs,
|
||||
shouldStop: options.shouldStop,
|
||||
scope: item.scope,
|
||||
existingPolicy: item.existingPolicy,
|
||||
libraryMangaId: item.libraryMangaId,
|
||||
onPublished: options.onPublished,
|
||||
onCatalog: options.onCatalog,
|
||||
});
|
||||
},
|
||||
}, {
|
||||
id: 'mangadex',
|
||||
label: 'MangaDex',
|
||||
match: mangaDexInfo,
|
||||
async discover(item, options) {
|
||||
const http = new SlowHttp({
|
||||
minMs: options.settings.minMs, maxMs: options.settings.maxMs, maxRetries: 2,
|
||||
userAgent: 'MangaCircus-Scraper/1.0 (personal archival; sequential MangaDex requests)',
|
||||
onEvent: options.onEvent,
|
||||
});
|
||||
return discoverMangaDexCatalog(item.url, controlledHttp(http, options.shouldStop));
|
||||
},
|
||||
run(item, options) {
|
||||
return runMangaDexArchive({
|
||||
seriesUrl: item.url,
|
||||
title: item.title,
|
||||
libraryRoot: options.libraryRoot,
|
||||
metadataSearch: item.metadataSearch || item.title,
|
||||
minMs: options.settings.minMs,
|
||||
maxMs: options.settings.maxMs,
|
||||
chapterMinMs: options.settings.chapterMinMs,
|
||||
chapterMaxMs: options.settings.chapterMaxMs,
|
||||
shouldStop: options.shouldStop,
|
||||
scope: item.scope,
|
||||
existingPolicy: item.existingPolicy,
|
||||
libraryMangaId: item.libraryMangaId,
|
||||
onPublished: options.onPublished,
|
||||
onCatalog: options.onCatalog,
|
||||
});
|
||||
},
|
||||
}];
|
||||
|
||||
function adapterFor(url) {
|
||||
for (const adapter of adapters) {
|
||||
const info = adapter.match(url);
|
||||
if (info) return { adapter, info };
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
module.exports = { adapters, adapterFor, weebCentralInfo, mangaDexInfo };
|
||||
@@ -0,0 +1,161 @@
|
||||
'use strict';
|
||||
|
||||
const API = 'https://api.mangadex.org';
|
||||
const { specialKey } = require('../../chapter-identity');
|
||||
|
||||
function mangaDexInfo(value) {
|
||||
let url;
|
||||
try { url = new URL(String(value)); } catch { return null; }
|
||||
if (url.protocol !== 'https:' || !['mangadex.org', 'www.mangadex.org'].includes(url.hostname)) return null;
|
||||
const match = /^\/title\/([0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12})(?:\/([^/?#]+))?\/?$/i.exec(url.pathname);
|
||||
if (!match) return null;
|
||||
const id = match[1].toLowerCase();
|
||||
const title = decodeURIComponent(match[2] || id).replace(/[-_]+/g, ' ').trim();
|
||||
return { source: 'mangadex', id, title, url: `https://mangadex.org/title/${id}${match[2] ? `/${match[2]}` : ''}` };
|
||||
}
|
||||
|
||||
function localizedTitle(attributes = {}) {
|
||||
const titles = attributes.title || {};
|
||||
return titles.en || titles['ja-ro'] || titles.ja || Object.values(titles).find(Boolean) || 'Untitled';
|
||||
}
|
||||
|
||||
function chapterNumber(value) {
|
||||
const raw = String(value ?? '').trim();
|
||||
if (!/^\d+(?:\.\d+)?$/.test(raw)) return null;
|
||||
const [whole, decimal = ''] = raw.split('.');
|
||||
const normalizedWhole = whole.replace(/^0+(?=\d)/, '') || '0';
|
||||
const normalizedDecimal = decimal.replace(/0+$/, '');
|
||||
return normalizedDecimal ? `${normalizedWhole}.${normalizedDecimal}` : normalizedWhole;
|
||||
}
|
||||
|
||||
function groupFor(resource) {
|
||||
const relation = (resource.relationships || []).find(item => item.type === 'scanlation_group');
|
||||
return relation ? { id: relation.id, name: relation.attributes?.name || 'Unknown group' } : null;
|
||||
}
|
||||
|
||||
function releaseRank(resource) {
|
||||
const group = groupFor(resource)?.name || '';
|
||||
const credited = /official|license/i.test(group) ? 0 : group ? 1 : 2;
|
||||
const date = Date.parse(resource.attributes?.publishAt || resource.attributes?.readableAt || resource.attributes?.createdAt || '') || Number.MAX_SAFE_INTEGER;
|
||||
return [credited, date, resource.id];
|
||||
}
|
||||
|
||||
function compareRank(left, right) {
|
||||
const a = releaseRank(left), b = releaseRank(right);
|
||||
return a[0] - b[0] || a[1] - b[1] || String(a[2]).localeCompare(String(b[2]));
|
||||
}
|
||||
|
||||
function selectChapterReleases(resources = [], specialLabel = 'Special') {
|
||||
const selected = new Map();
|
||||
for (const resource of resources) {
|
||||
if (resource.attributes?.externalUrl || Number(resource.attributes?.pages || 0) < 1) continue;
|
||||
const number = chapterNumber(resource.attributes?.chapter) || specialKey(resource.id);
|
||||
const prior = selected.get(number);
|
||||
if (!prior || compareRank(resource, prior) < 0) selected.set(number, resource);
|
||||
}
|
||||
return [...selected.entries()].map(([number, resource]) => {
|
||||
const group = groupFor(resource);
|
||||
return {
|
||||
id: resource.id, number, title: resource.attributes?.title || '',
|
||||
...(!chapterNumber(resource.attributes?.chapter) ? { special: true, label: specialLabel, sourceId: resource.id, sourceNumber: resource.attributes?.chapter ?? null } : {}),
|
||||
...(resource.attributes?.volume ? { volume: chapterNumber(resource.attributes.volume) || String(resource.attributes.volume) } : {}),
|
||||
pages: Number(resource.attributes?.pages || 0),
|
||||
added: resource.attributes?.publishAt || resource.attributes?.readableAt || resource.attributes?.createdAt,
|
||||
group,
|
||||
url: `https://mangadex.org/chapter/${resource.id}`,
|
||||
};
|
||||
}).sort((a, b) => Number(!!a.special) - Number(!!b.special) || (a.special ? a.number.localeCompare(b.number) : Number(a.number) - Number(b.number) || a.number.localeCompare(b.number)));
|
||||
}
|
||||
|
||||
function availabilitySummary(attributes, releases, chapters, externalReleases = null) {
|
||||
const numbered = chapters.filter(ref => !ref.special);
|
||||
const expected = chapterNumber(attributes.lastChapter);
|
||||
const maximum = numbered.length ? Math.max(...numbered.map(ref => Number(ref.number))) : null;
|
||||
const reason = chapters.length ? null : externalReleases > 0 ? 'external-only'
|
||||
: Array.isArray(attributes.availableTranslatedLanguages) && !attributes.availableTranslatedLanguages.includes('en') ? 'no-english' : 'no-hosted-pages';
|
||||
const message = reason === 'external-only' ? `MangaDex lists ${externalReleases} English external-link releases, but no hosted English pages. External links are not downloaded.`
|
||||
: reason === 'no-english' ? 'This MangaDex edition has no English releases listed.'
|
||||
: reason ? 'This MangaDex edition has no hosted English releases with pages.' : null;
|
||||
return { language: 'en', hostedReleases: releases.filter(r => !r.attributes?.externalUrl && Number(r.attributes?.pages) > 0).length,
|
||||
selectedChapters: chapters.length, numberedChapters: numbered.length, specialReleases: chapters.length - numbered.length,
|
||||
externalReleases, expectedLastChapter: expected, highestHostedChapter: maximum,
|
||||
coverage: expected && (maximum === null || maximum < Number(expected)) ? 'partial' : 'unknown', reason, message };
|
||||
}
|
||||
|
||||
async function jsonGet(http, url) {
|
||||
const response = await http.get(url, { accept: 'application/json' });
|
||||
const body = await response.json();
|
||||
if (body.result && body.result !== 'ok') throw new Error(`MangaDex API result: ${body.result}`);
|
||||
return body;
|
||||
}
|
||||
|
||||
async function discoverMangaDexCatalog(seriesUrl, http) {
|
||||
const info = mangaDexInfo(seriesUrl);
|
||||
if (!info) throw new Error('Invalid MangaDex title URL');
|
||||
const manga = (await jsonGet(http, `${API}/manga/${info.id}`)).data;
|
||||
if (!manga || manga.type !== 'manga') throw new Error('MangaDex title was not found');
|
||||
|
||||
const releases = [];
|
||||
let offset = 0;
|
||||
for (;;) {
|
||||
const params = new URLSearchParams({ limit: '100', offset: String(offset), includeFutureUpdates: '0', includeExternalUrl: '0' });
|
||||
params.append('translatedLanguage[]', 'en');
|
||||
params.append('includes[]', 'scanlation_group');
|
||||
params.append('order[chapter]', 'asc');
|
||||
const page = await jsonGet(http, `${API}/manga/${info.id}/feed?${params}`);
|
||||
releases.push(...(page.data || []));
|
||||
offset += page.data?.length || 0;
|
||||
if (!page.data?.length || offset >= Number(page.total || 0)) break;
|
||||
}
|
||||
|
||||
const title = localizedTitle(manga.attributes);
|
||||
const specialLabel = /pre.serialization|pilot/i.test(title) ? 'Pilot'
|
||||
: (manga.attributes?.tags || []).some(tag => tag.attributes?.name?.en === 'Oneshot') ? 'One-shot' : 'Special';
|
||||
const chapters = selectChapterReleases(releases, specialLabel);
|
||||
// External-only is a separate feed mode, not a combined hosted+external catalog.
|
||||
let externalReleases = null;
|
||||
if (!chapters.length) {
|
||||
const params = new URLSearchParams({ limit: '1', includeFutureUpdates: '0', includeExternalUrl: '1' });
|
||||
params.append('translatedLanguage[]', 'en');
|
||||
externalReleases = Number((await jsonGet(http, `${API}/manga/${info.id}/feed?${params}`)).total || 0);
|
||||
}
|
||||
return {
|
||||
id: info.id,
|
||||
title,
|
||||
originalLanguage: manga.attributes?.originalLanguage || null,
|
||||
links: manga.attributes?.links || {},
|
||||
availability: availabilitySummary(manga.attributes || {}, releases, chapters, externalReleases),
|
||||
chapters,
|
||||
};
|
||||
}
|
||||
|
||||
function pageUrls(atHome, chapterUrl, expectedPages) {
|
||||
const base = new URL(atHome.baseUrl);
|
||||
if (base.protocol !== 'https:' || !(base.hostname === 'uploads.mangadex.org' || base.hostname.endsWith('.mangadex.network'))) {
|
||||
throw new Error(`Unexpected MangaDex@Home host: ${base.hostname}`);
|
||||
}
|
||||
const hash = String(atHome.chapter?.hash || '');
|
||||
const files = atHome.chapter?.data;
|
||||
if (!hash || !Array.isArray(files) || !files.length) throw new Error('MangaDex@Home returned no pages');
|
||||
if (expectedPages && files.length !== expectedPages) throw new Error(`MangaDex page count changed: expected ${expectedPages}, received ${files.length}`);
|
||||
return files.map(file => ({
|
||||
url: new URL(`/data/${encodeURIComponent(hash)}/${encodeURIComponent(file)}`, base).href,
|
||||
headers: { referer: chapterUrl },
|
||||
}));
|
||||
}
|
||||
|
||||
async function discoverMangaDexChapter(ref, http) {
|
||||
const atHome = await jsonGet(http, `${API}/at-home/server/${ref.id}?forcePort443=true`);
|
||||
const groupName = ref.group?.name ? ` · ${ref.group.name}` : '';
|
||||
return {
|
||||
number: ref.number, title: ref.title || '', added: ref.added,
|
||||
...(ref.special ? { special: true, label: ref.label, sourceId: ref.id, sourceNumber: ref.sourceNumber ?? null } : {}),
|
||||
source: { name: `MangaDex${groupName}`, url: ref.url },
|
||||
pages: pageUrls(atHome, ref.url, ref.pages),
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
API, mangaDexInfo, localizedTitle, chapterNumber, selectChapterReleases, availabilitySummary,
|
||||
discoverMangaDexCatalog, discoverMangaDexChapter, pageUrls,
|
||||
};
|
||||
@@ -0,0 +1,88 @@
|
||||
'use strict';
|
||||
|
||||
const { SlowHttp } = require('../lib/http');
|
||||
const { fallbackDirection } = require('../../reading-direction');
|
||||
|
||||
const decode = value => value.replace(/&/g, '&').replace(/'/g, "'").replace(/"/g, '"').replace(/<[^>]+>/g, '').trim();
|
||||
|
||||
function absolute(base, value) { return new URL(value, base).href; }
|
||||
|
||||
function parseChapters(html, baseUrl) {
|
||||
const found = [];
|
||||
const re = /<a\b[^>]*href="([^"]*\/chapters\/([A-Z0-9]+)[^"]*)"[^>]*>([\s\S]*?)<\/a>/gi;
|
||||
for (const match of html.matchAll(re)) {
|
||||
const text = decode(match[3]).replace(/\s+/g, ' ');
|
||||
const season = Number(text.match(/^S(\d+)\s*-/i)?.[1] || 0) || null;
|
||||
// A few WeebCentral catalogs use a series-specific label (for example
|
||||
// "Rating 442", "Episode 128", "Night 182", "Log. 65", or "Act 60.6") in the same place where most catalogs
|
||||
// say "Chapter 442". Treat those labels as chapter numbering instead of
|
||||
// discarding the entire catalog.
|
||||
const regular = text.match(/\b(?:Chapter|Rating|Episode|Night|Log|Act)\s*[.:]?\s*([0-9]+(?:\.[0-9]+)?)/i)?.[1];
|
||||
const prologue = text.match(/Prologue(?:\s+([0-9]+))?/i)?.[1];
|
||||
const number = regular || (prologue ? `0.${prologue.padStart(2, '0')}` : /Prologue/i.test(text) ? '0' : null);
|
||||
if (number) found.push({
|
||||
id: match[2], number, title: regular ? '' : `Prologue ${prologue || ''}`.trim(),
|
||||
...(season ? { book: { number: season, title: `Season ${season}` } } : {}),
|
||||
url: absolute(baseUrl, match[1]), raw: text,
|
||||
});
|
||||
}
|
||||
return [...new Map(found.map(chapter => [chapter.id, chapter])).values()];
|
||||
}
|
||||
|
||||
async function discoverWeebCatalog(seriesUrl, http) {
|
||||
const seriesHtml = await (await http.get(seriesUrl, { accept: 'text/html' })).text();
|
||||
const title = decode(seriesHtml.match(/<h1\b[^>]*>([\s\S]*?)<\/h1>/i)?.[1] || 'Untitled');
|
||||
const listUrl = seriesHtml.match(/hx-get="([^"]+\/full-chapter-list)"/i)?.[1];
|
||||
const listHtml = listUrl ? await (await http.get(absolute(seriesUrl, listUrl), { accept: 'text/html' })).text() : seriesHtml;
|
||||
const chapterLinks = [...listHtml.matchAll(/<a\b[^>]*href="[^"]*\/chapters\/[A-Z0-9]+[^"]*"/gi)].length;
|
||||
const chapters = parseChapters(listHtml, seriesUrl).sort((a, b) =>
|
||||
Number(a.book?.number || 1) - Number(b.book?.number || 1) || Number(a.number) - Number(b.number));
|
||||
return { title, chapters, chapterLinks,
|
||||
...(chapterLinks && !chapters.length ? { availability: { reason: 'unrecognized-labels',
|
||||
message: `WeebCentral lists ${chapterLinks} chapter links, but their numbering labels are not recognized by the parser.` } } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
async function discoverWeebChapter(chapter, http) {
|
||||
const pageHtml = await (await http.get(chapter.url, { accept: 'text/html' })).text();
|
||||
const manifestUrl = pageHtml.match(/['"](https?:\/\/weebcentral\.com\/chapters\/[^'"]+\/images\?is_prev=[^'"]+)['"]/i)?.[1]
|
||||
|| `${chapter.url}/images?is_prev=False`;
|
||||
const fragment = await (await http.get(`${manifestUrl}&reading_style=long_strip¤t_page=1`, { accept: 'text/html' })).text();
|
||||
const pages = parsePageUrls(fragment, manifestUrl).map(url => ({ url, headers: { referer: chapter.url } }));
|
||||
if (!pages.length) throw new Error(`No pages found for chapter ${chapter.number}`);
|
||||
return {
|
||||
number: chapter.number, title: chapter.title || '',
|
||||
...(chapter.book ? { book: chapter.book } : {}),
|
||||
added: pageHtml.match(/datetime="([^"]+)"/i)?.[1],
|
||||
source: { name: 'WeebCentral', url: chapter.url }, pages,
|
||||
};
|
||||
}
|
||||
|
||||
function parsePageUrls(html, baseUrl) {
|
||||
const urls = [];
|
||||
for (const match of html.matchAll(/<img\b[^>]*(?:src|data-src)="([^"]+)"/gi)) {
|
||||
const url = absolute(baseUrl, match[1].replace(/&/g, '&'));
|
||||
if (!url.startsWith('data:')) urls.push(url);
|
||||
}
|
||||
return [...new Set(urls)];
|
||||
}
|
||||
|
||||
async function discoverWeebCentral(seriesUrl, options = {}) {
|
||||
const http = new SlowHttp(options.http || {});
|
||||
const catalog = await discoverWeebCatalog(seriesUrl, http);
|
||||
const chapters = catalog.chapters
|
||||
.sort((a, b) => Number(b.number) - Number(a.number))
|
||||
.slice(0, options.limit || 3)
|
||||
.sort((a, b) => Number(a.number) - Number(b.number));
|
||||
const output = [];
|
||||
for (const chapter of chapters) {
|
||||
output.push(await discoverWeebChapter(chapter, http));
|
||||
}
|
||||
return {
|
||||
version: 1,
|
||||
series: { title: catalog.title, ...fallbackDirection(), sources: [{ name: 'WeebCentral', url: seriesUrl }], scraper: { ids: { weebcentral: seriesUrl.split('/series/')[1]?.split('/')[0] } } },
|
||||
http: options.http || {}, chapters: output,
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = { discoverWeebCentral, discoverWeebCatalog, discoverWeebChapter, parseChapters, parsePageUrls };
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
const { publishManifest } = require('./lib/publisher');
|
||||
const { discoverWeebCentral } = require('./adapters/weebcentral');
|
||||
const { lookupAniList } = require('./metadata/anilist');
|
||||
const { runArchive } = require('./jobs/weebcentral-archive');
|
||||
const { runQueue } = require('./jobs/weebcentral-queue');
|
||||
|
||||
function flag(args, name, fallback) {
|
||||
const index = args.indexOf(name);
|
||||
return index < 0 ? fallback : args[index + 1];
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const args = process.argv.slice(2);
|
||||
const command = args.shift();
|
||||
if (command === 'publish') {
|
||||
const file = args[0];
|
||||
if (!file) throw new Error('Usage: publish <manifest.json> [--library path] [--metadata-search title] [--replace]');
|
||||
const manifest = JSON.parse(await fs.readFile(path.resolve(file), 'utf8'));
|
||||
const metadataSearch = flag(args, '--metadata-search');
|
||||
if (metadataSearch) {
|
||||
const metadata = await lookupAniList(metadataSearch);
|
||||
manifest.series.metadata = { ...metadata, title: manifest.series.title };
|
||||
manifest.series.scraper = { ...(manifest.series.scraper || {}), ids: { ...(manifest.series.scraper?.ids || {}), anilist: metadata['x-anilist']?.id } };
|
||||
}
|
||||
const result = await publishManifest(manifest, { libraryRoot: flag(args, '--library'), replace: args.includes('--replace') });
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
return;
|
||||
}
|
||||
if (command === 'discover-weebcentral') {
|
||||
const url = args[0];
|
||||
if (!url) throw new Error('Usage: discover-weebcentral <series-url> --out manifest.json [--limit 3]');
|
||||
const manifest = await discoverWeebCentral(url, { limit: Number(flag(args, '--limit', 3)) });
|
||||
const out = flag(args, '--out');
|
||||
if (out) await fs.writeFile(path.resolve(out), JSON.stringify(manifest, null, 2) + '\n');
|
||||
else console.log(JSON.stringify(manifest, null, 2));
|
||||
return;
|
||||
}
|
||||
if (command === 'metadata-anilist') {
|
||||
const search = args.join(' ');
|
||||
if (!search) throw new Error('Usage: metadata-anilist <title>');
|
||||
console.log(JSON.stringify(await lookupAniList(search), null, 2));
|
||||
return;
|
||||
}
|
||||
if (command === 'archive-weebcentral') {
|
||||
const url = args[0];
|
||||
if (!url) throw new Error('Usage: archive-weebcentral <series-url> [--library path] [--metadata-search title]');
|
||||
await runArchive({
|
||||
seriesUrl: url,
|
||||
libraryRoot: flag(args, '--library', 'library'),
|
||||
metadataSearch: flag(args, '--metadata-search'),
|
||||
minMs: Number(flag(args, '--min-ms', 8000)),
|
||||
maxMs: Number(flag(args, '--max-ms', 15000)),
|
||||
chapterMinMs: Number(flag(args, '--chapter-min-ms', 30000)),
|
||||
chapterMaxMs: Number(flag(args, '--chapter-max-ms', 60000)),
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (command === 'status-weebcentral') {
|
||||
const url = args[0];
|
||||
if (!url) throw new Error('Usage: status-weebcentral <series-url> [--library path]');
|
||||
const sourceId = url.split('/series/')[1]?.split('/')[0];
|
||||
if (!sourceId) throw new Error('Could not read WeebCentral series id');
|
||||
const workRoot = path.resolve(flag(args, '--library', 'library'), `.mcscraper-work-${sourceId}`);
|
||||
const state = JSON.parse(await fs.readFile(path.join(workRoot, 'job.json'), 'utf8'));
|
||||
console.log(JSON.stringify({
|
||||
status: state.status || 'unknown',
|
||||
completed: state.completed?.length || 0,
|
||||
total: state.total || null,
|
||||
percent: state.total ? Math.round((state.completed?.length || 0) * 1000 / state.total) / 10 : null,
|
||||
lastChapter: state.lastChapter || null,
|
||||
current: state.current || null,
|
||||
lastProblem: state.lastProblem || null,
|
||||
lastError: state.lastError || null,
|
||||
problemCount: state.problemCount || 0,
|
||||
rate: state.rate || null,
|
||||
updated: state.updated || null,
|
||||
log: path.join(workRoot, 'job.log'),
|
||||
}, null, 2));
|
||||
return;
|
||||
}
|
||||
if (command === 'run-wishlist') {
|
||||
const entries = [
|
||||
['The Gamer', '01J76XY9DYH4MH2HZG2PPF74A6', 'The-Gamer'],
|
||||
['Kanojo Okarishimasu', '01J76XYC7KP5M48JAY8S4ZNDX5', 'Kanojo-Okarishimasu'],
|
||||
['Toradora', '01J76XY7J4Z4G6E42GED6QEQXZ', 'Toradora'],
|
||||
['Solo Leveling', '01J76XYCPSY3C4BNPBRY8JMCBE', 'Solo-Leveling'],
|
||||
['Gantz', '01J76XY7EREQ4B1Q5Z5RJ749V6', 'Gantz'],
|
||||
['One Piece', '01J76XY7E9FNDZ1DBBM6PBJPFK', 'One-Piece'],
|
||||
].map(([title, id, slug]) => ({ id, title, url: `https://weebcentral.com/series/${id}/${slug}` }));
|
||||
await runQueue({
|
||||
entries,
|
||||
waitForFolder: flag(args, '--wait-for', 'Berserk'),
|
||||
libraryRoot: flag(args, '--library', 'library'),
|
||||
minMs: Number(flag(args, '--min-ms', 3000)),
|
||||
maxMs: Number(flag(args, '--max-ms', 7000)),
|
||||
chapterMinMs: Number(flag(args, '--chapter-min-ms', 12000)),
|
||||
chapterMaxMs: Number(flag(args, '--chapter-max-ms', 25000)),
|
||||
});
|
||||
return;
|
||||
}
|
||||
if (command === 'status-queue') {
|
||||
const queueRoot = path.resolve(flag(args, '--library', 'library'), '.mcscraper-queue');
|
||||
const state = JSON.parse(await fs.readFile(path.join(queueRoot, 'queue.json'), 'utf8'));
|
||||
console.log(JSON.stringify({ ...state, log: path.join(queueRoot, 'queue.log') }, null, 2));
|
||||
return;
|
||||
}
|
||||
console.log('Commands:\n publish <manifest.json> [--library path] [--metadata-search title] [--replace]\n discover-weebcentral <url> [--limit 3] [--out file]\n archive-weebcentral <url> [--library path] [--metadata-search title]\n status-weebcentral <url> [--library path]\n run-wishlist [--library path] [--wait-for Berserk]\n status-queue [--library path]\n metadata-anilist <title>');
|
||||
}
|
||||
|
||||
main().catch(error => { console.error(`[scraper] ${error.message}`); process.exitCode = 1; });
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
const http = require('http');
|
||||
const path = require('path');
|
||||
const { ScraperService } = require('./service');
|
||||
const { secret } = require('../runtime/config');
|
||||
|
||||
const HOST = process.env.SCRAPER_HOST || '127.0.0.1';
|
||||
const PORT = Number(process.env.SCRAPER_PORT || 8091);
|
||||
const TOKEN = secret('SCRAPER_TOKEN');
|
||||
const LIBRARY = path.resolve(process.env.LIBRARY || 'library');
|
||||
const DATA = path.resolve(process.env.SCRAPER_DATA || path.join(process.env.DATA || 'data', 'scraper'));
|
||||
|
||||
if (!['127.0.0.1', 'localhost', '::1'].includes(HOST) && !TOKEN) {
|
||||
throw new Error('SCRAPER_TOKEN is required when SCRAPER_HOST is not loopback');
|
||||
}
|
||||
|
||||
function json(res, code, value) {
|
||||
res.writeHead(code, { 'content-type': 'application/json; charset=utf-8', 'cache-control': 'no-store' });
|
||||
res.end(JSON.stringify(value));
|
||||
}
|
||||
|
||||
async function body(req) {
|
||||
let text = '';
|
||||
for await (const chunk of req) { text += chunk; if (text.length > 1000000) throw Object.assign(new Error('Request too large'), { code: 413 }); }
|
||||
try { return text ? JSON.parse(text) : {}; } catch { throw Object.assign(new Error('Invalid JSON'), { code: 400 }); }
|
||||
}
|
||||
|
||||
(async () => {
|
||||
const service = await new ScraperService({ libraryRoot: LIBRARY, dataRoot: DATA }).init();
|
||||
const server = http.createServer(async (req, res) => {
|
||||
try {
|
||||
const url = new URL(req.url, 'http://localhost');
|
||||
if (url.pathname === '/health') return json(res, 200, { ok: true, status: service.state.status });
|
||||
if (TOKEN && req.headers.authorization !== `Bearer ${TOKEN}`) return json(res, 401, { error: 'Unauthorized' });
|
||||
if (req.method === 'GET' && url.pathname === '/api/state') return json(res, 200, await service.publicState());
|
||||
if (req.method !== 'POST') return json(res, 404, { error: 'Not found' });
|
||||
const input = await body(req);
|
||||
if (url.pathname === '/api/queue') return json(res, 200, { item: await service.enqueue(input), state: await service.publicState() });
|
||||
if (url.pathname === '/api/queue/remove') { await service.remove(String(input.id || '')); return json(res, 200, await service.publicState()); }
|
||||
if (url.pathname === '/api/queue/clear-completed') { const result = await service.clearCompleted(String(input.source || '')); return json(res, 200, { ...result, state: await service.publicState() }); }
|
||||
if (url.pathname === '/api/queue/reorder') { await service.reorder(input.ids); return json(res, 200, await service.publicState()); }
|
||||
if (url.pathname === '/api/control') { await service.control(String(input.action || ''), input.id); return json(res, 200, await service.publicState()); }
|
||||
if (url.pathname === '/api/recovery') return json(res, 200, await service.recover(input));
|
||||
if (url.pathname === '/api/settings') { await service.updateSettings(input); return json(res, 200, await service.publicState()); }
|
||||
if (url.pathname === '/api/discovery/run') { await service.enqueueDiscoveryChecks('manual'); return json(res, 200, await service.publicState()); }
|
||||
if (url.pathname === '/api/library/auto-update') { await service.setMangaAutoUpdate(String(input.id || ''), !!input.enabled); return json(res, 200, await service.publicState()); }
|
||||
if (url.pathname === '/api/legacy/publish-ready') return json(res, 200, { result: await service.publishLegacyReady(), state: await service.publicState() });
|
||||
return json(res, 404, { error: 'Not found' });
|
||||
} catch (error) {
|
||||
console.error('[scraper-daemon]', error.message);
|
||||
json(res, error.code || (/not found/i.test(error.message) ? 404 : 400), { error: error.message });
|
||||
}
|
||||
});
|
||||
server.listen(PORT, HOST, () => console.log(`MangaCircus scraper on http://${HOST}:${PORT} (library: ${LIBRARY}, data: ${DATA})`));
|
||||
|
||||
let closing = false;
|
||||
const close = async () => {
|
||||
if (closing) return; closing = true;
|
||||
server.close(); await service.close(); process.exit(0);
|
||||
};
|
||||
process.on('SIGINT', close);
|
||||
process.on('SIGTERM', close);
|
||||
})().catch(error => { console.error(`[scraper-daemon] ${error.message}`); process.exitCode = 1; });
|
||||
@@ -0,0 +1,215 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
const { SlowHttp } = require('../lib/http');
|
||||
const { sleep } = require('../lib/delay');
|
||||
const { ArchiveControlError } = require('../lib/control');
|
||||
const { cleanPart } = require('../lib/names');
|
||||
const { publishManifest } = require('../lib/publisher');
|
||||
const { selectByScope } = require('../lib/scope');
|
||||
const { scanLibrary, findManga, shouldDownload, sourceKey } = require('../lib/library-inventory');
|
||||
const { lookupAniList, lookupAniListById } = require('../metadata/anilist');
|
||||
const { mangaDexInfo, discoverMangaDexCatalog, discoverMangaDexChapter } = require('../adapters/mangadex');
|
||||
|
||||
const randomBetween = (min, max) => min + Math.floor(Math.random() * (max - min + 1));
|
||||
const exists = async file => { try { await fs.access(file); return true; } catch { return false; } };
|
||||
|
||||
async function writeJsonAtomic(file, value) {
|
||||
const temp = `${file}.tmp`;
|
||||
await fs.writeFile(temp, JSON.stringify(value, null, 2) + '\n');
|
||||
await fs.rename(temp, file).catch(async error => {
|
||||
if (!['EPERM', 'EEXIST'].includes(error.code)) throw error;
|
||||
await fs.rm(file, { force: true });
|
||||
await fs.rename(temp, file);
|
||||
});
|
||||
}
|
||||
|
||||
async function renameRetry(from, to) {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
try { await fs.rename(from, to); return; } catch (error) {
|
||||
if (attempt >= 5 || !['EPERM', 'EBUSY', 'EACCES'].includes(error.code)) throw error;
|
||||
await sleep(200 * (attempt + 1));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function runMangaDexArchive(options) {
|
||||
const info = mangaDexInfo(options.seriesUrl);
|
||||
if (!info) throw new Error('Could not read MangaDex title id');
|
||||
const checkControl = () => {
|
||||
const reason = options.shouldStop && options.shouldStop();
|
||||
if (reason) throw new ArchiveControlError(reason === true ? 'paused' : String(reason));
|
||||
};
|
||||
const libraryRoot = path.resolve(options.libraryRoot || 'library');
|
||||
const workRoot = path.join(libraryRoot, `.mcscraper-work-${info.id}`);
|
||||
const stateFile = path.join(workRoot, 'job.json');
|
||||
const logFile = path.join(workRoot, 'job.log');
|
||||
await fs.mkdir(workRoot, { recursive: true });
|
||||
|
||||
const resumed = await exists(stateFile);
|
||||
let state = { version: 2, seriesUrl: options.seriesUrl, source: 'mangadex', publicationMode: 'incremental', completed: [], started: new Date().toISOString() };
|
||||
if (resumed) state = JSON.parse(await fs.readFile(stateFile, 'utf8'));
|
||||
const incremental = state.publicationMode === 'incremental';
|
||||
const completed = new Set(state.completed || []);
|
||||
const event = async (level, message, details = {}) => {
|
||||
const entry = { time: new Date().toISOString(), level, message, ...details };
|
||||
await fs.appendFile(logFile, JSON.stringify(entry) + '\n');
|
||||
state.lastEvent = entry;
|
||||
if (level === 'warning' || level === 'error') {
|
||||
state.lastProblem = entry;
|
||||
state.problemCount = Number(state.problemCount || 0) + 1;
|
||||
}
|
||||
state.updated = entry.time;
|
||||
await writeJsonAtomic(stateFile, state);
|
||||
};
|
||||
|
||||
const minMs = Number(options.minMs);
|
||||
const maxMs = Number(options.maxMs);
|
||||
const chapterMinMs = Number(options.chapterMinMs);
|
||||
const chapterMaxMs = Number(options.chapterMaxMs);
|
||||
const http = new SlowHttp({
|
||||
minMs, maxMs, maxRetries: 2,
|
||||
userAgent: 'MangaCircus-Scraper/1.0 (personal archival; sequential MangaDex requests)',
|
||||
onEvent: item => event(item.level, item.message, { status: item.status }),
|
||||
});
|
||||
|
||||
checkControl();
|
||||
console.log(`[mangadex] discovering catalog ${info.id}`);
|
||||
const catalog = await discoverMangaDexCatalog(options.seriesUrl, http);
|
||||
if (options.onCatalog) await options.onCatalog(catalog);
|
||||
if (!catalog.chapters.length) throw Object.assign(new Error(catalog.availability.message), { failureKind: 'unavailable' });
|
||||
const specialOnly = catalog.chapters.every(ref => ref.special);
|
||||
const folderTitle = String(specialOnly ? catalog.title : options.title || catalog.title).trim() || catalog.title;
|
||||
console.log(`[mangadex] ${folderTitle}: ${catalog.chapters.length} unique English chapters`);
|
||||
const expectedAniList = catalog.links.al ? String(catalog.links.al) : null;
|
||||
const metadata = expectedAniList
|
||||
? await lookupAniListById(expectedAniList)
|
||||
: options.metadataSearch && !specialOnly ? await lookupAniList(options.metadataSearch) : {};
|
||||
const actualAniList = metadata['x-anilist']?.id ? String(metadata['x-anilist'].id) : null;
|
||||
if (expectedAniList && actualAniList && expectedAniList !== actualAniList) {
|
||||
throw new Error(`AniList identity mismatch: MangaDex ${expectedAniList}, lookup ${actualAniList}`);
|
||||
}
|
||||
const series = {
|
||||
title: folderTitle, ...require('../../reading-direction').fallbackDirection(),
|
||||
sources: [{ name: 'MangaDex', url: options.seriesUrl }],
|
||||
metadata: { ...metadata, title: specialOnly ? catalog.title : metadata.title || folderTitle },
|
||||
coverUrl: metadata['x-anilist']?.coverUrl,
|
||||
scraper: { ids: { mangadex: info.id, ...(expectedAniList ? { anilist: expectedAniList } : {}) } },
|
||||
};
|
||||
|
||||
let inventory = await scanLibrary(libraryRoot);
|
||||
let manga = findManga(inventory, {
|
||||
source: 'mangadex', seriesId: info.id, sourceUrl: options.seriesUrl,
|
||||
anilist: specialOnly ? null : expectedAniList || actualAniList, libraryMangaId: options.libraryMangaId,
|
||||
});
|
||||
if (options.libraryMangaId && !manga) throw new Error(`Library manga target not found: ${options.libraryMangaId}`);
|
||||
const eligible = incremental
|
||||
? catalog.chapters.filter(ref => shouldDownload(manga, ref, 'mangadex', options.existingPolicy))
|
||||
: catalog.chapters;
|
||||
const refs = incremental ? selectByScope(eligible, options.scope) : eligible;
|
||||
state.status = 'running';
|
||||
state.catalogTotal = catalog.chapters.length;
|
||||
state.total = incremental ? refs.length : catalog.chapters.length;
|
||||
state.jobCompleted = 0;
|
||||
state.scope = options.scope || { mode: 'missing' };
|
||||
state.existingPolicy = options.existingPolicy || 'skip-any';
|
||||
state.rate = { minMs, maxMs, chapterMinMs, chapterMaxMs };
|
||||
state.lastError = null;
|
||||
await event('info', `Catalog ready: ${catalog.chapters.length} chapters; ${refs.length} selected`);
|
||||
|
||||
if (incremental && !refs.length) {
|
||||
state.status = 'complete'; state.current = null;
|
||||
await event('info', 'No eligible chapters to download');
|
||||
return { finalPath: manga?.path || null, publishedCount: 0, selectedCount: 0 };
|
||||
}
|
||||
|
||||
let publishedCount = 0;
|
||||
for (let index = 0; index < refs.length; index++) {
|
||||
checkControl();
|
||||
const ref = refs[index];
|
||||
if (!incremental && completed.has(ref.id)) {
|
||||
if (!incremental) console.log(`[mangadex] skip completed ${ref.number} (${index + 1}/${refs.length})`);
|
||||
continue;
|
||||
}
|
||||
if (incremental && !shouldDownload(manga, ref, 'mangadex', options.existingPolicy)) {
|
||||
state.jobCompleted++;
|
||||
await event('info', `Skipped existing chapter ${ref.number}`, { chapter: ref.number });
|
||||
continue;
|
||||
}
|
||||
let chapter, result;
|
||||
try {
|
||||
state.status = 'inspecting';
|
||||
state.current = { number: ref.number, label: ref.special ? ref.label : null, ...(ref.volume ? { volume: ref.volume } : {}), sourceId: ref.id, position: index + 1, total: refs.length, downloadedPages: 0, expectedPages: ref.pages || null };
|
||||
await event('info', `Inspecting chapter ${ref.number}`, { chapter: ref.number });
|
||||
chapter = await discoverMangaDexChapter(ref, http);
|
||||
state.status = 'downloading';
|
||||
state.current.expectedPages = chapter.pages.length;
|
||||
await event('info', `Downloading chapter ${ref.number}: ${chapter.pages.length} pages`, { chapter: ref.number });
|
||||
console.log(`[mangadex] download ${ref.number}: ${chapter.pages.length} pages`);
|
||||
result = await publishManifest({ version: 1, series, chapters: [chapter] }, {
|
||||
libraryRoot: incremental ? libraryRoot : workRoot,
|
||||
...(incremental && manga ? { mangaDir: manga.path } : {}),
|
||||
httpClient: http,
|
||||
onPage: async progress => {
|
||||
state.current.downloadedPages = progress.completed;
|
||||
state.updated = new Date().toISOString();
|
||||
await writeJsonAtomic(stateFile, state);
|
||||
checkControl();
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof ArchiveControlError) {
|
||||
state.status = error.reason;
|
||||
state.current = null;
|
||||
await event('info', `Archive ${error.reason}`, { chapter: ref.number });
|
||||
throw error;
|
||||
}
|
||||
state.status = 'error';
|
||||
state.lastError = { time: new Date().toISOString(), chapter: ref.number, message: error.message };
|
||||
await event('error', error.message, { chapter: ref.number });
|
||||
throw error;
|
||||
}
|
||||
completed.add(ref.id);
|
||||
state.completed = [...completed];
|
||||
state.jobCompleted = Number(state.jobCompleted || 0) + 1;
|
||||
state.lastChapter = ref.number;
|
||||
state.status = 'running';
|
||||
state.current = null;
|
||||
await writeJsonAtomic(stateFile, state);
|
||||
const published = result.results[0].status === 'published';
|
||||
if (published && incremental) {
|
||||
publishedCount++;
|
||||
if (!manga) manga = { id: path.basename(result.mangaDir), path: result.mangaDir, meta: { 'x-scraper': { strategy: 'flat' } }, chapters: [] };
|
||||
manga.chapters.push({ book: '1', number: String(ref.number), sourceKey: sourceKey(chapter.source.url, chapter.source.name) });
|
||||
if (options.onPublished) await options.onPublished({ mangaDir: result.mangaDir, chapter: ref.number });
|
||||
}
|
||||
await event('info', `Completed chapter ${ref.number} (${state.jobCompleted}/${state.total})`, { chapter: ref.number });
|
||||
console.log(`[mangadex] complete ${ref.number}: ${result.results[0].status} (${state.jobCompleted}/${state.total})`);
|
||||
if (index < refs.length - 1) {
|
||||
checkControl();
|
||||
await sleep(randomBetween(chapterMinMs, chapterMaxMs));
|
||||
checkControl();
|
||||
}
|
||||
}
|
||||
|
||||
if (incremental) {
|
||||
state.status = 'complete'; state.current = null;
|
||||
await event('info', `Published ${publishedCount} chapter${publishedCount === 1 ? '' : 's'} incrementally`);
|
||||
return { finalPath: manga?.path || null, publishedCount, selectedCount: refs.length };
|
||||
}
|
||||
|
||||
const stagedManga = path.join(workRoot, cleanPart(folderTitle));
|
||||
const finalManga = path.join(libraryRoot, cleanPart(folderTitle));
|
||||
if (await exists(finalManga)) throw new Error(`Final manga folder already exists: ${finalManga}`);
|
||||
await renameRetry(stagedManga, finalManga);
|
||||
state.status = 'complete';
|
||||
state.current = null;
|
||||
await event('info', `Published ${finalManga}`);
|
||||
await fs.rm(workRoot, { recursive: true, force: true });
|
||||
if (options.onPublished) await options.onPublished({ mangaDir: finalManga, completeManga: true });
|
||||
console.log(`[mangadex] PUBLISHED ${finalManga}`);
|
||||
return { finalPath: finalManga, publishedCount: 1, selectedCount: catalog.chapters.length };
|
||||
}
|
||||
|
||||
module.exports = { runMangaDexArchive };
|
||||
@@ -0,0 +1,220 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
const { SlowHttp } = require('../lib/http');
|
||||
const { sleep } = require('../lib/delay');
|
||||
const { ArchiveControlError } = require('../lib/control');
|
||||
const { cleanPart } = require('../lib/names');
|
||||
const { publishManifest } = require('../lib/publisher');
|
||||
const { selectByScope } = require('../lib/scope');
|
||||
const { scanLibrary, findManga, shouldDownload, sourceKey } = require('../lib/library-inventory');
|
||||
const { lookupAniList } = require('../metadata/anilist');
|
||||
const { discoverWeebCatalog, discoverWeebChapter } = require('../adapters/weebcentral');
|
||||
|
||||
function randomBetween(min, max) {
|
||||
return min + Math.floor(Math.random() * (max - min + 1));
|
||||
}
|
||||
|
||||
async function exists(file) {
|
||||
try { await fs.access(file); return true; } catch { return false; }
|
||||
}
|
||||
|
||||
async function writeJsonAtomic(file, value) {
|
||||
const temp = `${file}.tmp`;
|
||||
await fs.writeFile(temp, JSON.stringify(value, null, 2) + '\n');
|
||||
await fs.rename(temp, file).catch(async error => {
|
||||
if (!['EPERM', 'EEXIST'].includes(error.code)) throw error;
|
||||
await fs.rm(file, { force: true });
|
||||
await fs.rename(temp, file);
|
||||
});
|
||||
}
|
||||
|
||||
async function renameRetry(from, to) {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
try { await fs.rename(from, to); return; } catch (error) {
|
||||
if (attempt >= 5 || !['EPERM', 'EBUSY', 'EACCES'].includes(error.code)) throw error;
|
||||
await sleep(200 * (attempt + 1));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function runArchive(options) {
|
||||
const checkControl = () => {
|
||||
const reason = options.shouldStop && options.shouldStop();
|
||||
if (reason) throw new ArchiveControlError(reason === true ? 'paused' : String(reason));
|
||||
};
|
||||
const libraryRoot = path.resolve(options.libraryRoot || 'library');
|
||||
const sourceId = options.seriesUrl.split('/series/')[1]?.split('/')[0];
|
||||
if (!sourceId) throw new Error('Could not read WeebCentral series id');
|
||||
const workRoot = path.join(libraryRoot, `.mcscraper-work-${sourceId}`);
|
||||
const stateFile = path.join(workRoot, 'job.json');
|
||||
const logFile = path.join(workRoot, 'job.log');
|
||||
await fs.mkdir(workRoot, { recursive: true });
|
||||
|
||||
const resumed = await exists(stateFile);
|
||||
let state = { version: 2, seriesUrl: options.seriesUrl, publicationMode: 'incremental', completed: [], started: new Date().toISOString() };
|
||||
if (resumed) state = JSON.parse(await fs.readFile(stateFile, 'utf8'));
|
||||
const incremental = state.publicationMode === 'incremental';
|
||||
const completed = new Set(state.completed || []);
|
||||
const event = async (level, message, details = {}) => {
|
||||
const entry = { time: new Date().toISOString(), level, message, ...details };
|
||||
await fs.appendFile(logFile, JSON.stringify(entry) + '\n');
|
||||
state.lastEvent = entry;
|
||||
if (level === 'warning' || level === 'error') {
|
||||
state.lastProblem = entry;
|
||||
state.problemCount = Number(state.problemCount || 0) + 1;
|
||||
}
|
||||
state.updated = entry.time;
|
||||
await writeJsonAtomic(stateFile, state);
|
||||
};
|
||||
const http = new SlowHttp({
|
||||
minMs: options.minMs, maxMs: options.maxMs, maxRetries: 2,
|
||||
onEvent: item => event(item.level, item.message, { url: item.url, ...(item.status ? { status: item.status } : {}) }),
|
||||
});
|
||||
|
||||
checkControl();
|
||||
console.log(`[archive] discovering catalog ${options.seriesUrl}`);
|
||||
const catalog = await discoverWeebCatalog(options.seriesUrl, http);
|
||||
if (options.onCatalog) await options.onCatalog(catalog);
|
||||
if (!catalog.chapters.length) throw Object.assign(new Error(catalog.availability?.message || 'WeebCentral lists no chapter links for this series.'), {
|
||||
failureKind: catalog.chapterLinks ? 'adapter' : 'unavailable',
|
||||
});
|
||||
const keys = new Map();
|
||||
for (const chapter of catalog.chapters) {
|
||||
const key = `${chapter.book?.number || 1}:${chapter.number}`;
|
||||
const prior = keys.get(key);
|
||||
if (prior) throw new Error(`Ambiguous mapped chapter ${key}: ${prior.url} and ${chapter.url}`);
|
||||
keys.set(key, chapter);
|
||||
}
|
||||
console.log(`[archive] ${catalog.title}: ${catalog.chapters.length} source chapters`);
|
||||
const metadata = options.metadataSearch ? await lookupAniList(options.metadataSearch) : {};
|
||||
const series = {
|
||||
title: catalog.title, ...require('../../reading-direction').fallbackDirection(),
|
||||
sources: [{ name: 'WeebCentral', url: options.seriesUrl }],
|
||||
metadata: { ...metadata, title: catalog.title },
|
||||
coverUrl: metadata['x-anilist']?.coverUrl,
|
||||
scraper: { ids: { weebcentral: sourceId, ...(metadata['x-anilist']?.id ? { anilist: metadata['x-anilist'].id } : {}) } },
|
||||
};
|
||||
|
||||
let inventory = await scanLibrary(libraryRoot);
|
||||
let manga = findManga(inventory, {
|
||||
source: 'weebcentral', seriesId: sourceId, sourceUrl: options.seriesUrl,
|
||||
anilist: metadata['x-anilist']?.id, libraryMangaId: options.libraryMangaId,
|
||||
});
|
||||
if (options.libraryMangaId && !manga) throw new Error(`Library manga target not found: ${options.libraryMangaId}`);
|
||||
const eligible = incremental
|
||||
? catalog.chapters.filter(ref => shouldDownload(manga, ref, 'weebcentral', options.existingPolicy))
|
||||
: catalog.chapters;
|
||||
const refs = incremental ? selectByScope(eligible, options.scope) : eligible;
|
||||
state.status = 'running';
|
||||
state.catalogTotal = catalog.chapters.length;
|
||||
state.total = incremental ? refs.length : catalog.chapters.length;
|
||||
state.jobCompleted = 0;
|
||||
state.scope = options.scope || { mode: 'missing' };
|
||||
state.existingPolicy = options.existingPolicy || 'skip-any';
|
||||
state.rate = { minMs: options.minMs, maxMs: options.maxMs, chapterMinMs: options.chapterMinMs, chapterMaxMs: options.chapterMaxMs };
|
||||
state.lastError = null;
|
||||
await event('info', `Catalog ready: ${catalog.chapters.length} chapters; ${refs.length} selected`);
|
||||
|
||||
if (incremental && !refs.length) {
|
||||
state.status = 'complete'; state.current = null;
|
||||
await event('info', 'No eligible chapters to download');
|
||||
return { finalPath: manga?.path || null, publishedCount: 0, selectedCount: 0 };
|
||||
}
|
||||
|
||||
let publishedCount = 0;
|
||||
for (let index = 0; index < refs.length; index++) {
|
||||
checkControl();
|
||||
const ref = refs[index];
|
||||
if (!incremental && completed.has(ref.id)) {
|
||||
if (!incremental) console.log(`[archive] skip completed ${ref.number} (${index + 1}/${refs.length})`);
|
||||
continue;
|
||||
}
|
||||
if (incremental && !shouldDownload(manga, ref, 'weebcentral', options.existingPolicy)) {
|
||||
state.jobCompleted++;
|
||||
await event('info', `Skipped existing chapter ${ref.number}`, { chapter: ref.number });
|
||||
continue;
|
||||
}
|
||||
console.log(`[archive] inspect ${ref.number} (${index + 1}/${refs.length})`);
|
||||
let chapter;
|
||||
let result;
|
||||
try {
|
||||
state.status = 'inspecting';
|
||||
state.current = { number: ref.number, ...(ref.book ? { book: ref.book } : {}), sourceId: ref.id, position: index + 1, total: refs.length, downloadedPages: 0, expectedPages: null };
|
||||
await event('info', `Inspecting chapter ${ref.number}`, { chapter: ref.number });
|
||||
chapter = await discoverWeebChapter(ref, http);
|
||||
state.status = 'downloading';
|
||||
state.current.expectedPages = chapter.pages.length;
|
||||
await event('info', `Downloading chapter ${ref.number}: ${chapter.pages.length} pages`, { chapter: ref.number });
|
||||
console.log(`[archive] download ${ref.number}: ${chapter.pages.length} pages`);
|
||||
result = await publishManifest({ version: 1, series, chapters: [chapter] }, {
|
||||
libraryRoot: incremental ? libraryRoot : workRoot,
|
||||
...(incremental && manga ? { mangaDir: manga.path } : {}),
|
||||
httpClient: http,
|
||||
onPage: async progress => {
|
||||
state.current.downloadedPages = progress.completed;
|
||||
state.updated = new Date().toISOString();
|
||||
await writeJsonAtomic(stateFile, state);
|
||||
checkControl();
|
||||
},
|
||||
});
|
||||
} catch (error) {
|
||||
if (error instanceof ArchiveControlError) {
|
||||
state.status = error.reason;
|
||||
state.current = null;
|
||||
await event('info', `Archive ${error.reason}`, { chapter: ref.number });
|
||||
throw error;
|
||||
}
|
||||
state.status = 'error';
|
||||
state.lastError = { time: new Date().toISOString(), chapter: ref.number, message: error.message };
|
||||
await event('error', error.message, { chapter: ref.number });
|
||||
throw error;
|
||||
}
|
||||
completed.add(ref.id);
|
||||
state.completed = [...completed];
|
||||
state.jobCompleted = Number(state.jobCompleted || 0) + 1;
|
||||
state.lastChapter = ref.number;
|
||||
state.updated = new Date().toISOString();
|
||||
state.total = incremental ? refs.length : catalog.chapters.length;
|
||||
state.status = 'running';
|
||||
state.current = null;
|
||||
await writeJsonAtomic(stateFile, state);
|
||||
const published = result.results[0].status === 'published';
|
||||
if (published && incremental) {
|
||||
publishedCount++;
|
||||
if (!manga) manga = { id: path.basename(result.mangaDir), path: result.mangaDir, meta: { 'x-scraper': { strategy: 'flat' } }, chapters: [] };
|
||||
manga.chapters.push({ book: String(ref.book?.number || 1), number: String(ref.number), sourceKey: sourceKey(chapter.source.url, chapter.source.name) });
|
||||
if (options.onPublished) await options.onPublished({ mangaDir: result.mangaDir, chapter: ref.number });
|
||||
}
|
||||
await event('info', `Completed chapter ${ref.number} (${state.jobCompleted}/${state.total})`, { chapter: ref.number });
|
||||
console.log(`[archive] complete ${ref.number}: ${result.results[0].status} (${state.jobCompleted}/${state.total})`);
|
||||
if (index < refs.length - 1) {
|
||||
checkControl();
|
||||
const pause = randomBetween(options.chapterMinMs, options.chapterMaxMs);
|
||||
console.log(`[archive] chapter pause ${Math.round(pause / 1000)}s`);
|
||||
await sleep(pause);
|
||||
checkControl();
|
||||
}
|
||||
}
|
||||
|
||||
if (incremental) {
|
||||
state.status = 'complete'; state.current = null;
|
||||
await event('info', `Published ${publishedCount} chapter${publishedCount === 1 ? '' : 's'} incrementally`);
|
||||
return { finalPath: manga?.path || null, publishedCount, selectedCount: refs.length };
|
||||
}
|
||||
|
||||
const stagedManga = path.join(workRoot, cleanPart(catalog.title));
|
||||
const finalManga = path.join(libraryRoot, cleanPart(catalog.title));
|
||||
if (await exists(finalManga)) throw new Error(`Final manga folder already exists: ${finalManga}`);
|
||||
await renameRetry(stagedManga, finalManga);
|
||||
state.status = 'complete';
|
||||
state.current = null;
|
||||
await event('info', `Published ${finalManga}`);
|
||||
await fs.rm(workRoot, { recursive: true, force: true });
|
||||
if (options.onPublished) await options.onPublished({ mangaDir: finalManga, completeManga: true });
|
||||
console.log(`[archive] PUBLISHED ${finalManga}`);
|
||||
return { finalPath: finalManga, publishedCount: 1, selectedCount: catalog.chapters.length };
|
||||
}
|
||||
|
||||
module.exports = { runArchive };
|
||||
@@ -0,0 +1,124 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
const { runArchive } = require('./weebcentral-archive');
|
||||
const { sleep } = require('../lib/delay');
|
||||
|
||||
async function exists(file) {
|
||||
try { await fs.access(file); return true; } catch { return false; }
|
||||
}
|
||||
|
||||
async function writeJsonAtomic(file, value) {
|
||||
const temp = `${file}.tmp`;
|
||||
await fs.writeFile(temp, JSON.stringify(value, null, 2) + '\n');
|
||||
await fs.rename(temp, file).catch(async error => {
|
||||
if (!['EPERM', 'EEXIST'].includes(error.code)) throw error;
|
||||
await fs.rm(file, { force: true });
|
||||
await fs.rename(temp, file);
|
||||
});
|
||||
}
|
||||
|
||||
async function acquireLock(lockFile) {
|
||||
try {
|
||||
const handle = await fs.open(lockFile, 'wx');
|
||||
await handle.writeFile(String(process.pid));
|
||||
await handle.close();
|
||||
return;
|
||||
} catch (error) {
|
||||
if (error.code !== 'EEXIST') throw error;
|
||||
}
|
||||
const priorPid = Number((await fs.readFile(lockFile, 'utf8').catch(() => '')).trim());
|
||||
try {
|
||||
if (priorPid) process.kill(priorPid, 0);
|
||||
throw new Error(`Queue runner is already active (PID ${priorPid || 'unknown'})`);
|
||||
} catch (error) {
|
||||
if (error.message.startsWith('Queue runner is already active')) throw error;
|
||||
await fs.rm(lockFile, { force: true });
|
||||
const handle = await fs.open(lockFile, 'wx');
|
||||
await handle.writeFile(String(process.pid));
|
||||
await handle.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function runQueue(options) {
|
||||
const libraryRoot = path.resolve(options.libraryRoot || 'library');
|
||||
const queueRoot = path.join(libraryRoot, '.mcscraper-queue');
|
||||
const stateFile = path.join(queueRoot, 'queue.json');
|
||||
const logFile = path.join(queueRoot, 'queue.log');
|
||||
const lockFile = path.join(queueRoot, 'queue.lock');
|
||||
await fs.mkdir(queueRoot, { recursive: true });
|
||||
await acquireLock(lockFile);
|
||||
|
||||
let state = {
|
||||
version: 1,
|
||||
status: 'waiting',
|
||||
waitForFolder: options.waitForFolder || null,
|
||||
entries: options.entries,
|
||||
completed: [],
|
||||
created: new Date().toISOString(),
|
||||
};
|
||||
if (await exists(stateFile)) {
|
||||
const saved = JSON.parse(await fs.readFile(stateFile, 'utf8'));
|
||||
state = { ...state, ...saved, entries: saved.entries?.length ? saved.entries : options.entries };
|
||||
}
|
||||
state.pid = process.pid;
|
||||
|
||||
const event = async (level, message, details = {}) => {
|
||||
const entry = { time: new Date().toISOString(), level, message, ...details };
|
||||
await fs.appendFile(logFile, JSON.stringify(entry) + '\n');
|
||||
state.lastEvent = entry;
|
||||
state.updated = entry.time;
|
||||
if (level === 'error') state.lastError = entry;
|
||||
await writeJsonAtomic(stateFile, state);
|
||||
};
|
||||
|
||||
const cleanup = async () => { await fs.rm(lockFile, { force: true }).catch(() => {}); };
|
||||
process.once('exit', () => {});
|
||||
try {
|
||||
await event('info', `Queue runner started (PID ${process.pid})`);
|
||||
if (state.waitForFolder) {
|
||||
const prerequisite = path.join(libraryRoot, state.waitForFolder);
|
||||
state.status = 'waiting';
|
||||
await event('info', `Waiting for ${state.waitForFolder} to finish`);
|
||||
while (!(await exists(prerequisite))) await sleep(60000);
|
||||
await event('info', `${state.waitForFolder} is complete; starting queue`);
|
||||
}
|
||||
|
||||
const completed = new Set(state.completed || []);
|
||||
for (let index = 0; index < state.entries.length; index++) {
|
||||
const item = state.entries[index];
|
||||
if (completed.has(item.id)) continue;
|
||||
state.status = 'downloading';
|
||||
state.current = { ...item, position: index + 1, total: state.entries.length };
|
||||
state.lastError = null;
|
||||
await event('info', `Starting ${item.title}`, { title: item.title });
|
||||
try {
|
||||
await runArchive({
|
||||
seriesUrl: item.url,
|
||||
libraryRoot,
|
||||
metadataSearch: item.metadataSearch || item.title,
|
||||
minMs: options.minMs,
|
||||
maxMs: options.maxMs,
|
||||
chapterMinMs: options.chapterMinMs,
|
||||
chapterMaxMs: options.chapterMaxMs,
|
||||
});
|
||||
} catch (error) {
|
||||
state.status = 'error';
|
||||
await event('error', error.message, { title: item.title });
|
||||
throw error;
|
||||
}
|
||||
completed.add(item.id);
|
||||
state.completed = [...completed];
|
||||
state.current = null;
|
||||
await event('info', `Completed ${item.title}`, { title: item.title });
|
||||
}
|
||||
state.status = 'complete';
|
||||
state.current = null;
|
||||
await event('info', 'Queue complete');
|
||||
} finally {
|
||||
await cleanup();
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { runQueue };
|
||||
@@ -0,0 +1,11 @@
|
||||
'use strict';
|
||||
|
||||
class ArchiveControlError extends Error {
|
||||
constructor(reason = 'paused') {
|
||||
super(`Archive ${reason}`);
|
||||
this.name = 'ArchiveControlError';
|
||||
this.reason = reason;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { ArchiveControlError };
|
||||
@@ -0,0 +1,27 @@
|
||||
'use strict';
|
||||
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
|
||||
class PoliteGate {
|
||||
constructor({ minMs = 8000, maxMs = 15000 } = {}) {
|
||||
if (!Number.isFinite(minMs) || !Number.isFinite(maxMs) || minMs < 0 || maxMs < minMs) {
|
||||
throw new Error('Invalid delay range');
|
||||
}
|
||||
this.minMs = minMs;
|
||||
this.maxMs = maxMs;
|
||||
this.lastRequestAt = 0;
|
||||
}
|
||||
|
||||
async wait() {
|
||||
if (!this.lastRequestAt) return;
|
||||
const target = this.minMs + Math.floor(Math.random() * (this.maxMs - this.minMs + 1));
|
||||
const remaining = target - (Date.now() - this.lastRequestAt);
|
||||
if (remaining > 0) await sleep(remaining);
|
||||
}
|
||||
|
||||
mark() {
|
||||
this.lastRequestAt = Date.now();
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { PoliteGate, sleep };
|
||||
@@ -0,0 +1,23 @@
|
||||
'use strict';
|
||||
|
||||
function safeMessage(error) {
|
||||
return String(error?.message || error || 'Unknown error').replace(/https?:\/\/[^\s]+/gi, raw => {
|
||||
try { const u = new URL(raw); return u.origin + u.pathname; } catch { return '[source URL]'; }
|
||||
}).slice(0, 1500);
|
||||
}
|
||||
function classifyFailure(error) {
|
||||
const message = safeMessage(error);
|
||||
const status = Number(error?.status || /(?:HTTP|returned)\s+(\d{3})/i.exec(message)?.[1]);
|
||||
const code = error?.code || error?.cause?.code;
|
||||
const failure = (kind, scope, retryable = false) => ({ kind, scope, retryable, message, ...(status ? { httpStatus: status } : {}) });
|
||||
if ([401, 403, 429].includes(status) || /captcha|cloudflare|challenge|access.denied|account.ban|paywall|login required/i.test(message)) return failure('access', 'source');
|
||||
if (['ENOSPC', 'EROFS', 'EACCES', 'EPERM', 'EIO'].includes(code)) return failure('storage', 'global');
|
||||
if (error?.failureKind === 'unavailable' || /No downloadable English|title was not found|HTTP 404/i.test(message)) return failure('unavailable', 'job');
|
||||
if (error?.failureKind === 'transient' || status >= 500 || /network|fetch failed|timed? ?out|timeout|socket|connection reset/i.test(message)
|
||||
|| ['ETIMEDOUT', 'ECONNRESET', 'ENOTFOUND', 'EAI_AGAIN', 'UND_ERR_CONNECT_TIMEOUT'].includes(code)) return failure('transient', 'source', true);
|
||||
return failure('adapter', 'job');
|
||||
}
|
||||
function progressKey(job) {
|
||||
return JSON.stringify([job?.completed || 0, job?.total || 0, job?.current?.sourceId || '', job?.current?.downloadedPages || 0, job?.lastChapter || '']);
|
||||
}
|
||||
module.exports = { classifyFailure, safeMessage, progressKey };
|
||||
@@ -0,0 +1,52 @@
|
||||
'use strict';
|
||||
|
||||
const { PoliteGate, sleep } = require('./delay');
|
||||
|
||||
class StopScrapeError extends Error {}
|
||||
|
||||
class SlowHttp {
|
||||
constructor(options = {}) {
|
||||
this.gate = new PoliteGate(options);
|
||||
this.userAgent = options.userAgent || 'MangaCircus-Scraper/1.0 (personal archival; sequential requests)';
|
||||
this.maxRetries = options.maxRetries ?? 2;
|
||||
this.onEvent = options.onEvent;
|
||||
this.timeoutMs = options.timeoutMs || 60000;
|
||||
}
|
||||
|
||||
async get(url, options = {}) {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
await this.gate.wait();
|
||||
let response;
|
||||
try {
|
||||
response = await fetch(url, {
|
||||
redirect: 'follow',
|
||||
signal: AbortSignal.timeout(this.timeoutMs),
|
||||
headers: { 'user-agent': this.userAgent, accept: options.accept || '*/*', ...(options.headers || {}) },
|
||||
});
|
||||
this.gate.mark();
|
||||
} catch (error) {
|
||||
this.gate.mark();
|
||||
if (attempt >= this.maxRetries) throw new Error(`Network request failed after ${attempt + 1} attempts: ${url}`, { cause: error });
|
||||
const backoff = 30000 * (2 ** attempt);
|
||||
const message = `Network failure; retrying in ${backoff / 1000}s (${attempt + 1}/${this.maxRetries})`;
|
||||
console.warn(`[http] ${message}`);
|
||||
if (this.onEvent) await this.onEvent({ level: 'warning', message, url });
|
||||
await sleep(backoff);
|
||||
continue;
|
||||
}
|
||||
|
||||
if ([401, 403, 429].includes(response.status)) {
|
||||
if (this.onEvent) await this.onEvent({ level: 'error', message: `Host returned ${response.status}; stopping`, url, status: response.status });
|
||||
throw Object.assign(new StopScrapeError(`Host returned ${response.status}; stop and require human review: ${url}`), { status: response.status });
|
||||
}
|
||||
if (response.status >= 500 && attempt < this.maxRetries) {
|
||||
await sleep(30000 * (2 ** attempt));
|
||||
continue;
|
||||
}
|
||||
if (!response.ok) throw Object.assign(new Error(`HTTP ${response.status}: ${url}`), { status: response.status });
|
||||
return response;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { SlowHttp, StopScrapeError };
|
||||
@@ -0,0 +1,16 @@
|
||||
'use strict';
|
||||
|
||||
const TYPES = [
|
||||
{ ext: 'jpg', test: b => b.length >= 3 && b[0] === 0xff && b[1] === 0xd8 && b[2] === 0xff },
|
||||
{ ext: 'png', test: b => b.length >= 8 && b.subarray(0, 8).equals(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10])) },
|
||||
{ ext: 'gif', test: b => b.length >= 6 && /^GIF8[79]a$/.test(b.subarray(0, 6).toString('ascii')) },
|
||||
{ ext: 'webp', test: b => b.length >= 12 && b.subarray(0, 4).toString('ascii') === 'RIFF' && b.subarray(8, 12).toString('ascii') === 'WEBP' },
|
||||
{ ext: 'bmp', test: b => b.length >= 2 && b.subarray(0, 2).toString('ascii') === 'BM' },
|
||||
{ ext: 'avif', test: b => b.length >= 12 && b.subarray(4, 12).toString('ascii').includes('ftyp') && /avif|avis/.test(b.subarray(8, 32).toString('ascii')) },
|
||||
];
|
||||
|
||||
function imageExtension(buffer) {
|
||||
return TYPES.find(type => type.test(buffer))?.ext || null;
|
||||
}
|
||||
|
||||
module.exports = { imageExtension };
|
||||
@@ -0,0 +1,30 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
|
||||
async function writeJsonAtomic(file, value, io = fs) {
|
||||
const folder = path.dirname(file), name = path.basename(file);
|
||||
const temp = path.join(folder, `.${name}.${crypto.randomUUID()}.tmp`);
|
||||
await io.writeFile(temp, JSON.stringify(value, null, 2) + '\n');
|
||||
try {
|
||||
try { await io.rename(temp, file); return; }
|
||||
catch (error) {
|
||||
if (!['EPERM', 'EACCES', 'EBUSY', 'EEXIST'].includes(error.code)) throw error;
|
||||
}
|
||||
// Windows may refuse rename-over-existing. Keep the old JSON recoverable
|
||||
// until the replacement is safely in place; never delete it first.
|
||||
const backup = path.join(folder, `.old-${name}-${crypto.randomUUID()}`);
|
||||
await io.rename(file, backup);
|
||||
try { await io.rename(temp, file); }
|
||||
catch (error) {
|
||||
try { await io.rename(backup, file); }
|
||||
catch (restoreError) { error.message += `; previous JSON remains at ${backup}: ${restoreError.message}`; }
|
||||
throw error;
|
||||
}
|
||||
await io.rm(backup, { force: true });
|
||||
} finally { await io.rm(temp, { force: true }); }
|
||||
}
|
||||
|
||||
module.exports = { writeJsonAtomic };
|
||||
@@ -0,0 +1,179 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
const { imageExtension } = require('./images');
|
||||
const { assertInside } = require('./names');
|
||||
|
||||
const PAGE = /^(\d{4,5})(?:_[a-z0-9]+)*\.(jpe?g|png|webp|gif|avif|bmp)$/i;
|
||||
const BOOK = /^(?:v|s|x)\d{2,3}(?: |$)/i;
|
||||
const CHAPTER = /^(?:c\d{4}(?:\.\d*[1-9])?|x\d{2})(?: |$)/i;
|
||||
const OWN_STAGE = /^\.staging-mcscraper-[a-f0-9]+$/;
|
||||
|
||||
async function exists(target) {
|
||||
try { await fs.access(target); return true; } catch { return false; }
|
||||
}
|
||||
|
||||
async function renameRetry(from, to, tries = 5) {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
try { await fs.rename(from, to); return; } catch (error) {
|
||||
if (attempt >= tries || !['EPERM', 'EBUSY', 'EACCES'].includes(error.code)) throw error;
|
||||
await new Promise(resolve => setTimeout(resolve, 200 * (attempt + 1)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function writeJsonAtomic(file, value) {
|
||||
const temp = `${file}.migration.tmp`;
|
||||
const old = `${file}.migration.old`;
|
||||
await fs.writeFile(temp, JSON.stringify(value, null, 2) + '\n', { mode: 0o644 });
|
||||
await fs.rm(old, { force: true });
|
||||
await renameRetry(file, old);
|
||||
try {
|
||||
await renameRetry(temp, file);
|
||||
} catch (error) {
|
||||
await renameRetry(old, file).catch(() => {});
|
||||
await fs.rm(temp, { force: true });
|
||||
throw error;
|
||||
}
|
||||
await fs.rm(old, { force: true }).catch(() => {});
|
||||
}
|
||||
|
||||
async function directories(folder) {
|
||||
return (await fs.readdir(folder, { withFileTypes: true })).filter(entry => entry.isDirectory());
|
||||
}
|
||||
|
||||
async function validatePage(file, expected) {
|
||||
const handle = await fs.open(file, 'r');
|
||||
try {
|
||||
const prefix = Buffer.alloc(16);
|
||||
const { bytesRead } = await handle.read(prefix, 0, prefix.length, 0);
|
||||
const actual = imageExtension(prefix.subarray(0, bytesRead));
|
||||
if (!actual) throw new Error(`Unsupported image bytes: ${file}`);
|
||||
const normalized = expected.toLowerCase() === 'jpeg' ? 'jpg' : expected.toLowerCase();
|
||||
if (actual !== normalized) throw new Error(`Image extension does not match bytes: ${file}`);
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
}
|
||||
|
||||
async function validateChapter(folder) {
|
||||
const entries = await fs.readdir(folder, { withFileTypes: true });
|
||||
if (entries.some(entry => entry.isDirectory())) throw new Error(`Chapter contains a subdirectory: ${folder}`);
|
||||
const chapterFile = path.join(folder, 'chapter.json');
|
||||
const chapter = JSON.parse(await fs.readFile(chapterFile, 'utf8'));
|
||||
if (!chapter.source?.name || !chapter.source?.url) throw new Error(`Invalid chapter source: ${chapterFile}`);
|
||||
const pages = entries.filter(entry => entry.isFile() && entry.name !== 'chapter.json').map(entry => {
|
||||
const match = PAGE.exec(entry.name);
|
||||
if (!match) throw new Error(`Invalid chapter file: ${path.join(folder, entry.name)}`);
|
||||
return { name: entry.name, number: Number(match[1]), extension: match[2] };
|
||||
}).sort((a, b) => a.number - b.number);
|
||||
if (!pages.length) throw new Error(`Chapter has no pages: ${folder}`);
|
||||
for (let index = 0; index < pages.length; index++) {
|
||||
if (pages[index].number !== index + 1) throw new Error(`Non-contiguous pages in ${folder}`);
|
||||
await validatePage(path.join(folder, pages[index].name), pages[index].extension);
|
||||
}
|
||||
return pages.length;
|
||||
}
|
||||
|
||||
async function inspectLegacyWork({ libraryRoot, seriesId }) {
|
||||
const id = String(seriesId || '');
|
||||
if (!/^[A-Za-z0-9-]+$/.test(id)) throw new Error(`Unsafe legacy series id: ${id}`);
|
||||
const root = path.resolve(libraryRoot);
|
||||
const workRoot = assertInside(root, path.join(root, `.mcscraper-work-${id}`));
|
||||
const stateFile = path.join(workRoot, 'job.json');
|
||||
if (!(await exists(stateFile))) return null;
|
||||
const state = JSON.parse(await fs.readFile(stateFile, 'utf8'));
|
||||
if (state.publicationMode === 'incremental') return null;
|
||||
const mangaDirs = (await directories(workRoot)).filter(entry => !entry.name.startsWith('.'));
|
||||
if (!mangaDirs.length) return null;
|
||||
if (mangaDirs.length !== 1) throw new Error(`Legacy work ${id} must contain exactly one manga folder`);
|
||||
const mangaDir = path.join(workRoot, mangaDirs[0].name);
|
||||
const target = assertInside(root, path.join(root, mangaDirs[0].name));
|
||||
if (await exists(target)) throw new Error(`Cannot publish legacy work; target already exists: ${target}`);
|
||||
const meta = JSON.parse(await fs.readFile(path.join(mangaDir, 'manga.json'), 'utf8'));
|
||||
if (!meta.title || !Array.isArray(meta.sources) || !meta.sources.length) throw new Error(`Invalid legacy manga metadata: ${mangaDir}`);
|
||||
|
||||
let chapterCount = 0, pageCount = 0;
|
||||
const incompleteStages = [];
|
||||
for (const book of await directories(mangaDir)) {
|
||||
if (book.name.startsWith('.')) throw new Error(`Unexpected hidden folder remains in ${mangaDir}`);
|
||||
if (!BOOK.test(book.name)) throw new Error(`Invalid legacy book folder: ${book.name}`);
|
||||
const bookDir = path.join(mangaDir, book.name);
|
||||
for (const chapter of await directories(bookDir)) {
|
||||
if (chapter.name.startsWith('.')) {
|
||||
if (!OWN_STAGE.test(chapter.name)) throw new Error(`Unexpected hidden folder remains in ${bookDir}: ${chapter.name}`);
|
||||
incompleteStages.push(path.join(bookDir, chapter.name));
|
||||
continue;
|
||||
}
|
||||
if (!CHAPTER.test(chapter.name)) throw new Error(`Invalid legacy chapter folder: ${chapter.name}`);
|
||||
pageCount += await validateChapter(path.join(bookDir, chapter.name));
|
||||
chapterCount++;
|
||||
}
|
||||
}
|
||||
const completedCount = new Set(state.completed || []).size;
|
||||
if (!chapterCount || chapterCount !== completedCount) {
|
||||
throw new Error(`Legacy work ${id} has ${chapterCount} complete folders but ${completedCount} completed checkpoint entries`);
|
||||
}
|
||||
return {
|
||||
id, root, workRoot, stateFile, state, mangaDir, target,
|
||||
mangaId: path.basename(target), chapterCount, pageCount, incompleteStages,
|
||||
};
|
||||
}
|
||||
|
||||
async function promoteLegacyWork(inspection) {
|
||||
const { stateFile, mangaDir, target, state, workRoot } = inspection;
|
||||
const quarantineRoot = path.join(workRoot, '.legacy-incomplete');
|
||||
const quarantined = [];
|
||||
if (inspection.incompleteStages.length) await fs.mkdir(quarantineRoot, { recursive: true, mode: 0o755 });
|
||||
for (const stage of inspection.incompleteStages) {
|
||||
const relative = path.relative(mangaDir, stage).split(path.sep).join('__');
|
||||
const quarantine = path.join(quarantineRoot, relative);
|
||||
await renameRetry(stage, quarantine);
|
||||
quarantined.push({ stage, quarantine });
|
||||
}
|
||||
try {
|
||||
await renameRetry(mangaDir, target);
|
||||
} catch (error) {
|
||||
for (const item of quarantined.reverse()) await renameRetry(item.quarantine, item.stage).catch(() => {});
|
||||
throw error;
|
||||
}
|
||||
const migrated = {
|
||||
...state,
|
||||
publicationMode: 'incremental',
|
||||
status: 'paused',
|
||||
current: null,
|
||||
jobCompleted: 0,
|
||||
total: 0,
|
||||
migratedToIncremental: {
|
||||
time: new Date().toISOString(),
|
||||
mangaId: inspection.mangaId,
|
||||
chaptersPublished: inspection.chapterCount,
|
||||
pagesPublished: inspection.pageCount,
|
||||
incompleteStagesQuarantined: quarantined.length,
|
||||
},
|
||||
};
|
||||
try {
|
||||
await writeJsonAtomic(stateFile, migrated);
|
||||
} catch (error) {
|
||||
await renameRetry(target, mangaDir).catch(() => {});
|
||||
for (const item of quarantined.reverse()) await renameRetry(item.quarantine, item.stage).catch(() => {});
|
||||
throw error;
|
||||
}
|
||||
await fs.appendFile(path.join(workRoot, 'job.log'), JSON.stringify({
|
||||
time: migrated.migratedToIncremental.time,
|
||||
level: 'info',
|
||||
message: `Published ${inspection.chapterCount} completed legacy chapters and switched to incremental mode`,
|
||||
mangaId: inspection.mangaId,
|
||||
}) + '\n').catch(() => {});
|
||||
return {
|
||||
seriesId: inspection.id,
|
||||
mangaId: inspection.mangaId,
|
||||
finalPath: target,
|
||||
chaptersPublished: inspection.chapterCount,
|
||||
pagesPublished: inspection.pageCount,
|
||||
incompleteStagesQuarantined: quarantined.length,
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = { inspectLegacyWork, promoteLegacyWork };
|
||||
@@ -0,0 +1,158 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
const { assertInside } = require('./names');
|
||||
const { chapterNumber } = require('./scope');
|
||||
const { isSpecialKey, parseSpecialFolder, parseKey } = require('../../chapter-identity');
|
||||
const { writeJsonAtomic } = require('./json-file');
|
||||
|
||||
async function readJson(file) {
|
||||
try { return JSON.parse(await fs.readFile(file, 'utf8')); } catch { return {}; }
|
||||
}
|
||||
|
||||
async function directories(folder) {
|
||||
try { return (await fs.readdir(folder, { withFileTypes: true })).filter(entry => entry.isDirectory() && !entry.name.startsWith('.')); }
|
||||
catch { return []; }
|
||||
}
|
||||
|
||||
function keyFromName(name) { return parseKey(name)?.key || null; }
|
||||
|
||||
function sourceKey(url, name = '') {
|
||||
try {
|
||||
const host = new URL(String(url)).hostname.replace(/^www\./, '').toLowerCase();
|
||||
if (host === 'weebcentral.com') return 'weebcentral';
|
||||
if (host === 'mangadex.org') return 'mangadex';
|
||||
return host;
|
||||
} catch {}
|
||||
const label = String(name).toLowerCase();
|
||||
if (label.startsWith('mangadex')) return 'mangadex';
|
||||
if (label.startsWith('weebcentral')) return 'weebcentral';
|
||||
return label.replace(/[^a-z0-9]+/g, '-').replace(/^-|-$/g, '');
|
||||
}
|
||||
|
||||
function autoUpdateFor(meta) {
|
||||
const configured = meta?.['x-scraper']?.autoUpdate;
|
||||
return configured === undefined ? String(meta?.status || '').toLowerCase() === 'ongoing' : !!configured;
|
||||
}
|
||||
|
||||
async function scanLibrary(libraryRoot) {
|
||||
const root = path.resolve(libraryRoot);
|
||||
const mangas = [];
|
||||
for (const mangaEntry of await directories(root)) {
|
||||
const mangaPath = path.join(root, mangaEntry.name);
|
||||
const meta = await readJson(path.join(mangaPath, 'manga.json'));
|
||||
const chapters = [];
|
||||
for (const bookEntry of await directories(mangaPath)) {
|
||||
const book = keyFromName(bookEntry.name);
|
||||
if (!book) continue;
|
||||
const bookPath = path.join(mangaPath, bookEntry.name);
|
||||
for (const versionEntry of await directories(bookPath)) {
|
||||
const number = parseSpecialFolder(versionEntry.name)?.key || keyFromName(versionEntry.name);
|
||||
if (!number) continue;
|
||||
const chapterMeta = await readJson(path.join(bookPath, versionEntry.name, 'chapter.json'));
|
||||
const source = typeof chapterMeta.source === 'string' ? { name: chapterMeta.source, url: '' } : (chapterMeta.source || {});
|
||||
chapters.push({
|
||||
book, number, version: versionEntry.name,
|
||||
sourceName: String(source.name || ''), sourceUrl: String(source.url || ''),
|
||||
sourceKey: sourceKey(source.url, source.name),
|
||||
});
|
||||
}
|
||||
}
|
||||
mangas.push({
|
||||
id: mangaEntry.name, path: mangaPath, title: String(meta.title || mangaEntry.name), meta,
|
||||
status: String(meta.status || 'unknown').toLowerCase(), autoUpdate: autoUpdateFor(meta),
|
||||
ids: { ...(meta['x-scraper']?.ids || {}) }, sources: Array.isArray(meta.sources) ? meta.sources : [], chapters,
|
||||
});
|
||||
}
|
||||
return { root, mangas };
|
||||
}
|
||||
|
||||
function findManga(inventory, identity = {}) {
|
||||
const source = String(identity.source || '');
|
||||
const seriesId = identity.seriesId == null ? '' : String(identity.seriesId);
|
||||
const sourceUrl = String(identity.sourceUrl || '');
|
||||
const anilist = identity.anilist == null ? '' : String(identity.anilist);
|
||||
const explicit = identity.libraryMangaId == null ? '' : String(identity.libraryMangaId);
|
||||
if (explicit) return inventory.mangas.find(manga => manga.id === explicit) || null;
|
||||
if (source && seriesId) {
|
||||
const bySourceId = inventory.mangas.find(manga => String(manga.ids[source] ?? '') === seriesId);
|
||||
if (bySourceId) return bySourceId;
|
||||
}
|
||||
if (sourceUrl) {
|
||||
const byUrl = inventory.mangas.find(manga => manga.sources.some(item => item.url === sourceUrl));
|
||||
if (byUrl) return byUrl;
|
||||
}
|
||||
if (anilist) {
|
||||
const matches = inventory.mangas.filter(manga => String(manga.ids.anilist ?? manga.meta?.['x-anilist']?.id ?? '') === anilist);
|
||||
// AniList identifies the work, not every pilot/edition on a source. Never
|
||||
// silently merge a different listing from that same host into its sibling.
|
||||
if (matches.length !== 1) return null;
|
||||
if (source && seriesId && matches[0].ids[source] != null && String(matches[0].ids[source]) !== seriesId) return null;
|
||||
return matches[0];
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function chapterBook(ref, manga) {
|
||||
if (manga?.meta?.['x-scraper']?.strategy === 'volumes' && ref.volume != null) return chapterNumber(ref.volume, 'volume');
|
||||
return chapterNumber(ref.book?.number || 1, 'book');
|
||||
}
|
||||
|
||||
function existingVersions(manga, ref) {
|
||||
if (!manga) return [];
|
||||
const book = chapterBook(ref, manga);
|
||||
const number = isSpecialKey(ref.number) ? String(ref.number).toLowerCase() : chapterNumber(ref.number);
|
||||
return manga.chapters.filter(item => item.book === book && item.number === number);
|
||||
}
|
||||
|
||||
function shouldDownload(manga, ref, source, existingPolicy = 'skip-any') {
|
||||
const versions = existingVersions(manga, ref);
|
||||
if (!versions.length) return true;
|
||||
if (existingPolicy === 'add-source') return !versions.some(item => item.sourceKey === source);
|
||||
return false;
|
||||
}
|
||||
|
||||
function isNewerThanLibrary(manga, ref) {
|
||||
if (isSpecialKey(ref.number)) return !existingVersions(manga, ref).length;
|
||||
if (!manga?.chapters?.length) return true;
|
||||
const regular = manga.chapters.filter(item => !String(item.book).startsWith('x') && /^\d+(?:\.\d+)?$/.test(String(item.number)));
|
||||
if (!regular.length) return true;
|
||||
const latest = regular.reduce((best, item) => {
|
||||
if (!best) return item;
|
||||
return Number(item.book) > Number(best.book)
|
||||
|| (Number(item.book) === Number(best.book) && Number(item.number) > Number(best.number)) ? item : best;
|
||||
}, null);
|
||||
const candidate = { book: chapterBook(ref, manga), number: chapterNumber(ref.number) };
|
||||
return Number(candidate.book) > Number(latest.book)
|
||||
|| (Number(candidate.book) === Number(latest.book) && Number(candidate.number) > Number(latest.number));
|
||||
}
|
||||
|
||||
function summarizeInventory(inventory, supported) {
|
||||
return inventory.mangas.map(manga => ({
|
||||
id: manga.id, title: manga.title, status: manga.status, autoUpdate: manga.autoUpdate,
|
||||
chapters: new Set(manga.chapters.map(item => `${item.book}/${item.number}`)).size,
|
||||
versions: manga.chapters.length,
|
||||
sources: manga.sources.map(source => {
|
||||
const found = supported(source.url);
|
||||
return { name: source.name, url: source.url, adapter: found?.adapter.id || null };
|
||||
}),
|
||||
}));
|
||||
}
|
||||
|
||||
async function setAutoUpdate(libraryRoot, mangaId, enabled) {
|
||||
const root = path.resolve(libraryRoot);
|
||||
const folder = assertInside(root, path.join(root, String(mangaId)));
|
||||
const stat = await fs.stat(folder).catch(() => null);
|
||||
if (!stat?.isDirectory() || path.basename(folder).startsWith('.')) throw new Error('Library manga not found');
|
||||
const file = path.join(folder, 'manga.json');
|
||||
const meta = await readJson(file);
|
||||
meta['x-scraper'] = { ...(meta['x-scraper'] || {}), autoUpdate: !!enabled };
|
||||
await writeJsonAtomic(file, meta);
|
||||
return { id: mangaId, autoUpdate: !!enabled };
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
scanLibrary, findManga, existingVersions, shouldDownload, summarizeInventory,
|
||||
isNewerThanLibrary, setAutoUpdate, sourceKey, autoUpdateFor, keyFromName,
|
||||
};
|
||||
@@ -0,0 +1,34 @@
|
||||
'use strict';
|
||||
|
||||
const IMPORTED_FIELDS = ['title', 'altTitles', 'authors', 'status', 'year', 'tags', 'description'];
|
||||
const same = (a, b) => JSON.stringify(a) === JSON.stringify(b);
|
||||
|
||||
// The effective fields remain at the top of manga.json. A snapshot of the last
|
||||
// imported values lets later imports refresh unchanged fields without replacing
|
||||
// values somebody edited in the library. Legacy files have no snapshot, so their
|
||||
// existing values are conservatively treated as intentional.
|
||||
function mergeImportedMetadata(prior, imported, fallbackTitle) {
|
||||
const previous = prior['x-metadata']?.imported || {};
|
||||
const result = { ...prior };
|
||||
for (const field of IMPORTED_FIELDS) {
|
||||
if (!Object.hasOwn(imported, field)) continue;
|
||||
if (result[field] === undefined || result[field] === null || result[field] === ''
|
||||
|| (Object.hasOwn(previous, field) && same(result[field], previous[field]))) {
|
||||
result[field] = imported[field];
|
||||
}
|
||||
}
|
||||
if (!result.title) result.title = fallbackTitle;
|
||||
|
||||
const oldId = prior['x-anilist']?.id;
|
||||
const newId = imported['x-anilist']?.id;
|
||||
if (oldId != null && newId != null && String(oldId) !== String(newId)) {
|
||||
throw new Error(`AniList identity conflict: existing ${oldId}, imported ${newId}; review the manga match`);
|
||||
}
|
||||
if (imported['x-anilist']) result['x-anilist'] = { ...(prior['x-anilist'] || {}), ...imported['x-anilist'] };
|
||||
const snapshot = { ...previous };
|
||||
for (const field of IMPORTED_FIELDS) if (Object.hasOwn(imported, field)) snapshot[field] = imported[field];
|
||||
if (Object.keys(snapshot).length) result['x-metadata'] = { ...(prior['x-metadata'] || {}), imported: snapshot };
|
||||
return result;
|
||||
}
|
||||
|
||||
module.exports = { mergeImportedMetadata, IMPORTED_FIELDS };
|
||||
@@ -0,0 +1,50 @@
|
||||
'use strict';
|
||||
|
||||
const path = require('path');
|
||||
const { isSpecialKey } = require('../../chapter-identity');
|
||||
|
||||
const RESERVED = /^(con|prn|aux|nul|com[1-9]|lpt[1-9])$/i;
|
||||
|
||||
function cleanPart(value, max = 120) {
|
||||
let result = String(value || '')
|
||||
.normalize('NFC')
|
||||
.replace(/[<>:"/\\|?*\x00-\x1f]/g, '')
|
||||
.replace(/[\[\]]/g, '')
|
||||
.replace(/\s+/g, ' ')
|
||||
.trim()
|
||||
.replace(/[ .]+$/g, '');
|
||||
if (!result) result = 'Untitled';
|
||||
if (RESERVED.test(result)) result += '-manga';
|
||||
return result.slice(0, max).replace(/[ .]+$/g, '');
|
||||
}
|
||||
|
||||
function chapterKey(value) {
|
||||
const raw = String(value).trim();
|
||||
if (isSpecialKey(raw)) return raw.toLowerCase();
|
||||
if (/^\d+(?:\.\d+)?$/.test(raw)) {
|
||||
const [integer, decimal = ''] = raw.split('.');
|
||||
const normalized = decimal.replace(/0+$/g, '');
|
||||
return `c${integer.padStart(4, '0')}${normalized ? `.${normalized}` : ''}`;
|
||||
}
|
||||
const extra = raw.match(/^x(\d+)$/i);
|
||||
if (extra) return `x${extra[1].padStart(2, '0')}`;
|
||||
throw new Error(`Unsupported chapter number: ${value}`);
|
||||
}
|
||||
|
||||
function chapterFolder(chapter) {
|
||||
const parts = [chapterKey(chapter.number)];
|
||||
if (chapter.title) parts.push(cleanPart(chapter.title));
|
||||
parts.push(`[${cleanPart(chapter.source.name, 50)}]`);
|
||||
return parts.join(' ');
|
||||
}
|
||||
|
||||
function assertInside(parent, child) {
|
||||
const root = path.resolve(parent);
|
||||
const target = path.resolve(child);
|
||||
if (target !== root && !target.startsWith(root + path.sep)) {
|
||||
throw new Error(`Path escapes library root: ${target}`);
|
||||
}
|
||||
return target;
|
||||
}
|
||||
|
||||
module.exports = { cleanPart, chapterKey, chapterFolder, assertInside };
|
||||
@@ -0,0 +1,279 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs');
|
||||
const fsp = fs.promises;
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const { SlowHttp } = require('./http');
|
||||
const { imageExtension } = require('./images');
|
||||
const { cleanPart, chapterFolder, assertInside } = require('./names');
|
||||
const { sizeFromBuffer, imageSize } = require('../../library');
|
||||
const { directionFields } = require('../../reading-direction');
|
||||
const { mergeImportedMetadata } = require('./metadata-merge');
|
||||
|
||||
const token = () => crypto.randomBytes(5).toString('hex');
|
||||
// Staging is a recovery asset, not disposable garbage. Legacy stages without an
|
||||
// identity must be reviewed manually; never infer a chapter from page filenames.
|
||||
const ownTemp = name => /^\.old-mcscraper-[a-f0-9]+$/.test(name);
|
||||
const digest = value => crypto.createHash('sha256').update(JSON.stringify(value)).digest('hex');
|
||||
const coverFile = name => /^cover\.(?:jpe?g|png|webp|gif|avif|bmp)$/i.test(name);
|
||||
|
||||
async function exists(file) {
|
||||
try { await fsp.access(file); return true; } catch { return false; }
|
||||
}
|
||||
|
||||
async function readMetadata(file) {
|
||||
try { return JSON.parse((await fsp.readFile(file, 'utf8')).replace(/^\uFEFF/, '')); }
|
||||
catch (error) { if (error.code === 'ENOENT') return {}; throw error; }
|
||||
}
|
||||
|
||||
function pageDimensions(size, tags = []) {
|
||||
const rotation = tags.find(tag => /^rot(90|180|270)$/.test(tag));
|
||||
return { w: size?.w || 0, h: size?.h || 0, r: rotation ? Number(rotation.slice(3)) : 0 };
|
||||
}
|
||||
|
||||
async function publishedDimensions(folder) {
|
||||
const pages = [];
|
||||
for (const name of (await fsp.readdir(folder)).sort()) {
|
||||
if (!/\.(?:jpe?g|png|webp|gif|avif|bmp)$/i.test(name)) continue;
|
||||
const tags = name.slice(0, name.lastIndexOf('.')).split('_').slice(1);
|
||||
pages.push(pageDimensions(await imageSize(path.join(folder, name)).catch(() => null), tags));
|
||||
}
|
||||
return pages;
|
||||
}
|
||||
|
||||
async function renameRetry(from, to, tries = 5) {
|
||||
for (let attempt = 0; ; attempt++) {
|
||||
try { await fsp.rename(from, to); return; } catch (error) {
|
||||
if (attempt >= tries || !['EPERM', 'EBUSY', 'EACCES'].includes(error.code)) throw error;
|
||||
await new Promise(resolve => setTimeout(resolve, 200 * (attempt + 1)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function writeJsonAtomic(file, value) {
|
||||
const temp = path.join(path.dirname(file), `.${path.basename(file)}.tmp`);
|
||||
await fsp.writeFile(temp, JSON.stringify(value, null, 2) + '\n', { mode: 0o644 });
|
||||
if (!(await exists(file))) return renameRetry(temp, file);
|
||||
try {
|
||||
await renameRetry(temp, file);
|
||||
} catch (error) {
|
||||
if (!['EPERM', 'EBUSY', 'EACCES', 'EEXIST'].includes(error.code)) throw error;
|
||||
const old = path.join(path.dirname(file), `.old-mcscraper-${token()}`);
|
||||
await renameRetry(file, old);
|
||||
try { await renameRetry(temp, file); } catch (moveError) { await renameRetry(old, file); throw moveError; }
|
||||
await fsp.rm(old, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function cleanupOwnTemps(folder, maxAgeMs = 6 * 60 * 60 * 1000) {
|
||||
if (!(await exists(folder))) return;
|
||||
const now = Date.now();
|
||||
for (const entry of await fsp.readdir(folder, { withFileTypes: true })) {
|
||||
if (!entry.isDirectory() || !ownTemp(entry.name)) continue;
|
||||
const target = path.join(folder, entry.name);
|
||||
const stat = await fsp.stat(target);
|
||||
if (now - stat.mtimeMs > maxAgeMs) await fsp.rm(target, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
async function findManga(libraryRoot, sourceUrls) {
|
||||
if (!(await exists(libraryRoot))) return null;
|
||||
for (const entry of await fsp.readdir(libraryRoot, { withFileTypes: true })) {
|
||||
if (!entry.isDirectory() || entry.name.startsWith('.')) continue;
|
||||
const file = path.join(libraryRoot, entry.name, 'manga.json');
|
||||
try {
|
||||
const json = JSON.parse(await fsp.readFile(file, 'utf8'));
|
||||
if ((json.sources || []).some(source => sourceUrls.has(source.url))) return path.join(libraryRoot, entry.name);
|
||||
} catch {}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function validateManifest(manifest) {
|
||||
if (!manifest || manifest.version !== 1) throw new Error('Manifest version must be 1');
|
||||
if (!manifest.series?.title) throw new Error('series.title is required');
|
||||
if (!['rtl', 'ltr', 'ttb'].includes(manifest.series.direction)) throw new Error('series.direction must be rtl, ltr, or ttb');
|
||||
if (!Array.isArray(manifest.series.sources) || !manifest.series.sources.length) throw new Error('series.sources is required');
|
||||
if (!Array.isArray(manifest.chapters) || !manifest.chapters.length) throw new Error('At least one chapter is required');
|
||||
for (const chapter of manifest.chapters) {
|
||||
if (chapter.number == null || chapter.number === '' || !chapter.source?.name || !chapter.source?.url) throw new Error('Every chapter needs number and source {name,url}');
|
||||
if (!Array.isArray(chapter.pages) || !chapter.pages.length) throw new Error(`Chapter ${chapter.number} has no pages`);
|
||||
for (const page of chapter.pages) {
|
||||
if (!!page.url === !!page.path) throw new Error('Every page must have exactly one of url or path');
|
||||
for (const tag of page.tags || []) if (!/^[a-z0-9]+$/.test(tag)) throw new Error(`Invalid page tag: ${tag}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function pageBytes(page, http) {
|
||||
if (page.path) return fsp.readFile(path.resolve(page.path));
|
||||
const response = await http.get(page.url, { headers: page.headers || {} });
|
||||
return Buffer.from(await response.arrayBuffer());
|
||||
}
|
||||
|
||||
function validateCoverUrl(value) {
|
||||
const url = new URL(value);
|
||||
if (url.protocol !== 'https:' || !(url.hostname === 'anilist.co' || url.hostname.endsWith('.anilist.co'))) {
|
||||
throw new Error('Series cover must use the AniList HTTPS image host');
|
||||
}
|
||||
return url.href;
|
||||
}
|
||||
|
||||
async function installSeriesCover(mangaDir, coverUrl, http) {
|
||||
const files = await fsp.readdir(mangaDir);
|
||||
if (files.some(coverFile)) return { status: 'existing' };
|
||||
const response = await http.get(validateCoverUrl(coverUrl));
|
||||
const data = Buffer.from(await response.arrayBuffer());
|
||||
if (data.length > 10 * 1024 * 1024) throw new Error('Series cover exceeds 10 MiB');
|
||||
const ext = imageExtension(data);
|
||||
if (!ext) throw new Error('Series cover is not a supported image');
|
||||
const temp = path.join(mangaDir, `.cover-mcscraper-${token()}.${ext}`);
|
||||
const target = path.join(mangaDir, `cover.${ext}`);
|
||||
await fsp.writeFile(temp, data, { mode: 0o644 });
|
||||
try { await renameRetry(temp, target); } catch (error) { await fsp.rm(temp, { force: true }); throw error; }
|
||||
return { status: 'installed', file: path.basename(target) };
|
||||
}
|
||||
|
||||
async function stageChapter(bookDir, chapter, http, onPage, priorMeta = {}) {
|
||||
// Full manifest fingerprint prevents reusing a page from a changed release.
|
||||
// Store only the hash, never image URLs/headers (which may contain signatures).
|
||||
const identity = digest([chapterFolder(chapter), chapter.source, chapter.pages.map(p => [p.url || path.resolve(p.path), p.tags || []])]);
|
||||
const stage = path.join(bookDir, `.staging-mcscraper-resume-${identity}`);
|
||||
await fsp.mkdir(stage, { recursive: true, mode: 0o755 });
|
||||
const dimensions = [];
|
||||
{
|
||||
for (let index = 0; index < chapter.pages.length; index++) {
|
||||
const tags = (chapter.pages[index].tags || []).map(tag => `_${tag}`).join('');
|
||||
const prefix = `${String(index + 1).padStart(4, '0')}${tags}`;
|
||||
const prior = (await fsp.readdir(stage)).filter(name => name.startsWith(prefix + '.') && !name.endsWith('.tmp'));
|
||||
if (prior.length === 1) {
|
||||
const bytes = await fsp.readFile(path.join(stage, prior[0]));
|
||||
if (imageExtension(bytes) === path.extname(prior[0]).slice(1)) {
|
||||
dimensions.push(pageDimensions(sizeFromBuffer(bytes), chapter.pages[index].tags));
|
||||
if (onPage) await onPage({ completed: index + 1, total: chapter.pages.length });
|
||||
continue;
|
||||
}
|
||||
}
|
||||
if (prior.length) throw new Error(`Partial page ${index + 1} requires review: invalid or ambiguous image`);
|
||||
const data = await pageBytes(chapter.pages[index], http);
|
||||
const ext = imageExtension(data);
|
||||
if (!ext) throw new Error(`Page ${index + 1} is not a supported image`);
|
||||
dimensions.push(pageDimensions(sizeFromBuffer(data), chapter.pages[index].tags));
|
||||
const target = path.join(stage, `${prefix}.${ext}`);
|
||||
const pending = path.join(stage, `.${prefix}.tmp`);
|
||||
await fsp.writeFile(pending, data, { mode: 0o644 });
|
||||
await renameRetry(pending, target);
|
||||
if (onPage) await onPage({ completed: index + 1, total: chapter.pages.length });
|
||||
}
|
||||
await fsp.writeFile(path.join(stage, 'chapter.json'), JSON.stringify({
|
||||
...priorMeta,
|
||||
title: chapter.title || '', source: chapter.source,
|
||||
added: chapter.added || new Date().toISOString(),
|
||||
priority: Number(chapter.priority || 0), extra: !!chapter.extra,
|
||||
...(chapter.special ? { special: true, label: chapter.label || 'Special', sourceId: chapter.sourceId, sourceNumber: chapter.sourceNumber ?? null } : {}),
|
||||
...directionFields([priorMeta, chapter], dimensions),
|
||||
}, null, 2) + '\n', { mode: 0o644 });
|
||||
// A process interruption during a write can leave a hidden pending file.
|
||||
for (const name of await fsp.readdir(stage)) {
|
||||
if (/^\.\d{4,5}(?:_[a-z0-9]+)*\.tmp$/.test(name)) await fsp.rm(path.join(stage, name), { force: true });
|
||||
}
|
||||
return { stage, dimensions };
|
||||
}
|
||||
}
|
||||
|
||||
async function publishStaged(stage, target, replace) {
|
||||
if (!(await exists(target))) return renameRetry(stage, target);
|
||||
if (!replace) { await fsp.rm(stage, { recursive: true, force: true }); return false; }
|
||||
const old = path.join(path.dirname(target), `.old-mcscraper-${token()}`);
|
||||
await renameRetry(target, old);
|
||||
try { await renameRetry(stage, target); } catch (error) { await renameRetry(old, target); throw error; }
|
||||
await fsp.rm(old, { recursive: true, force: true });
|
||||
return true;
|
||||
}
|
||||
|
||||
async function publishManifest(manifest, options = {}) {
|
||||
validateManifest(manifest);
|
||||
const libraryRoot = path.resolve(options.libraryRoot || manifest.libraryRoot || 'library');
|
||||
await fsp.mkdir(libraryRoot, { recursive: true, mode: 0o755 });
|
||||
await cleanupOwnTemps(libraryRoot);
|
||||
const sourceUrls = new Set(manifest.series.sources.map(source => source.url));
|
||||
let mangaDir = options.mangaDir ? assertInside(libraryRoot, options.mangaDir) : await findManga(libraryRoot, sourceUrls);
|
||||
if (mangaDir && !(await exists(mangaDir))) throw new Error(`Target manga folder does not exist: ${mangaDir}`);
|
||||
if (mangaDir && path.basename(mangaDir).startsWith('.')) throw new Error('Target manga folder must be published');
|
||||
const isNew = !mangaDir;
|
||||
const finalName = cleanPart(manifest.series.folder || manifest.series.title);
|
||||
const finalDir = assertInside(libraryRoot, path.join(libraryRoot, finalName));
|
||||
if (!mangaDir && await exists(finalDir)) throw new Error(`Folder already belongs to another work: ${finalDir}`);
|
||||
mangaDir = mangaDir || path.join(libraryRoot, `.staging-mcscraper-resume-${digest([finalName, [...sourceUrls].sort()])}`);
|
||||
await fsp.mkdir(mangaDir, { recursive: true, mode: 0o755 });
|
||||
const http = options.httpClient || new SlowHttp(options.http || manifest.http || {});
|
||||
const results = [];
|
||||
const dimensions = [];
|
||||
let cover = null;
|
||||
const prior = await readMetadata(path.join(mangaDir, 'manga.json'));
|
||||
const owned = manifest.series.metadata || {};
|
||||
// Validate identity before fetching or exposing any chapter pages.
|
||||
const merged = mergeImportedMetadata(prior, owned, manifest.series.title);
|
||||
|
||||
try {
|
||||
for (const chapter of manifest.chapters) {
|
||||
const bookNumber = Number(chapter.book?.number || 1);
|
||||
if (!Number.isInteger(bookNumber) || bookNumber < 1) throw new Error(`Invalid book number for chapter ${chapter.number}`);
|
||||
const bookName = `v${String(bookNumber).padStart(2, '0')}${chapter.book?.title ? ` ${cleanPart(chapter.book.title)}` : ''}`;
|
||||
const bookDir = path.join(mangaDir, bookName);
|
||||
await fsp.mkdir(bookDir, { recursive: true, mode: 0o755 });
|
||||
await cleanupOwnTemps(bookDir);
|
||||
if (chapter.book?.title) {
|
||||
const file = path.join(bookDir, 'book.json'), priorBook = await readMetadata(file);
|
||||
await writeJsonAtomic(file, { ...priorBook, title: chapter.book.title,
|
||||
...(chapter.book.direction || chapter.book.directionOverride
|
||||
? directionFields([priorBook, chapter.book], []) : {}) });
|
||||
}
|
||||
const folder = chapterFolder(chapter);
|
||||
const target = path.join(bookDir, folder);
|
||||
if (await exists(target) && !options.replace) {
|
||||
dimensions.push(...await publishedDimensions(target));
|
||||
results.push({ chapter: chapter.number, status: 'skipped' }); continue;
|
||||
}
|
||||
const priorChapter = await readMetadata(path.join(target, 'chapter.json'));
|
||||
const staged = await stageChapter(bookDir, chapter, http, options.onPage, priorChapter);
|
||||
dimensions.push(...staged.dimensions);
|
||||
await publishStaged(staged.stage, target, !!options.replace);
|
||||
results.push({ chapter: chapter.number, status: 'published', folder });
|
||||
}
|
||||
|
||||
const sources = [...(prior.sources || [])];
|
||||
for (const source of manifest.series.sources) {
|
||||
const index = sources.findIndex(item => item.url === source.url);
|
||||
if (index < 0) sources.push(source); else sources[index] = { ...sources[index], ...source };
|
||||
}
|
||||
const priorScraper = prior['x-scraper'] || {};
|
||||
const nextScraper = { ...priorScraper, strategy: priorScraper.strategy || 'flat', ...(manifest.series.scraper || {}) };
|
||||
await writeJsonAtomic(path.join(mangaDir, 'manga.json'), {
|
||||
...merged,
|
||||
...directionFields([prior, merged, manifest.series], dimensions),
|
||||
sources,
|
||||
'x-scraper': nextScraper,
|
||||
});
|
||||
|
||||
if (manifest.series.coverUrl && !manifest.series.coverAttempted) {
|
||||
manifest.series.coverAttempted = true;
|
||||
try {
|
||||
cover = await installSeriesCover(mangaDir, manifest.series.coverUrl, http);
|
||||
} catch (error) {
|
||||
cover = { status: 'error', message: error.message };
|
||||
console.warn(`[publisher] cover unavailable for ${manifest.series.title}: ${error.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (isNew) await renameRetry(mangaDir, finalDir);
|
||||
return { mangaDir: isNew ? finalDir : mangaDir, results, cover };
|
||||
} catch (error) {
|
||||
// Preserve hidden validated pages on pause, transient failure or shutdown.
|
||||
// Publication still occurs only after the whole chapter validates.
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { publishManifest, validateManifest, cleanupOwnTemps, installSeriesCover, validateCoverUrl, writeJsonAtomic };
|
||||
@@ -0,0 +1,74 @@
|
||||
'use strict';
|
||||
|
||||
const PUBLIC_MODES = new Set(['missing', 'latest', 'last', 'first', 'chapter', 'after', 'volume']);
|
||||
|
||||
function chapterNumber(value, field = 'chapter') {
|
||||
const raw = String(value ?? '').trim();
|
||||
if (!/^\d+(?:\.\d+)?$/.test(raw)) throw new Error(`${field} must be a numeric chapter`);
|
||||
const [whole, decimal = ''] = raw.split('.');
|
||||
const normalizedWhole = whole.replace(/^0+(?=\d)/, '') || '0';
|
||||
const normalizedDecimal = decimal.replace(/0+$/, '');
|
||||
return normalizedDecimal ? `${normalizedWhole}.${normalizedDecimal}` : normalizedWhole;
|
||||
}
|
||||
|
||||
function positiveCount(value) {
|
||||
const count = Number(value);
|
||||
if (!Number.isInteger(count) || count < 1 || count > 10000) throw new Error('count must be an integer from 1 to 10000');
|
||||
return count;
|
||||
}
|
||||
|
||||
function normalizeScope(input = {}) {
|
||||
const raw = input && typeof input === 'object' ? input : {};
|
||||
const mode = String(raw.mode || 'missing').toLowerCase();
|
||||
if (mode === 'ids') {
|
||||
if (!Array.isArray(raw.ids) || !raw.ids.length || raw.ids.length > 10000) throw new Error('ids scope requires 1 to 10000 source chapter ids');
|
||||
return { mode, ids: [...new Set(raw.ids.map(value => String(value).trim()).filter(Boolean))] };
|
||||
}
|
||||
if (!PUBLIC_MODES.has(mode)) throw new Error(`Unsupported queue scope: ${mode}`);
|
||||
if (mode === 'last' || mode === 'first') return { mode, count: positiveCount(raw.count) };
|
||||
if (mode === 'chapter' || mode === 'after') return { mode, chapter: chapterNumber(raw.chapter) };
|
||||
if (mode === 'volume') {
|
||||
const volume = String(raw.volume ?? '').trim();
|
||||
if (!/^\d+(?:\.\d+)?$/.test(volume)) throw new Error('volume must be numeric');
|
||||
return { mode, volume: chapterNumber(volume, 'volume') };
|
||||
}
|
||||
return { mode };
|
||||
}
|
||||
|
||||
function volumeOf(ref) {
|
||||
const value = ref.volume ?? ref.book?.number;
|
||||
if (value === undefined || value === null || value === '') return null;
|
||||
try { return chapterNumber(value, 'volume'); } catch { return null; }
|
||||
}
|
||||
|
||||
function selectByScope(chapters, input) {
|
||||
const scope = normalizeScope(input);
|
||||
const list = [...chapters];
|
||||
if (scope.mode === 'missing') return list;
|
||||
if (scope.mode === 'latest') return list.length ? [list[list.length - 1]] : [];
|
||||
if (scope.mode === 'last') return list.slice(-scope.count);
|
||||
if (scope.mode === 'first') return list.slice(0, scope.count);
|
||||
const numbered = list.filter(ref => /^\d+(?:\.\d+)?$/.test(String(ref.number)));
|
||||
if (scope.mode === 'chapter') return numbered.filter(ref => chapterNumber(ref.number) === scope.chapter);
|
||||
if (scope.mode === 'after') return numbered.filter(ref => Number(chapterNumber(ref.number)) > Number(scope.chapter));
|
||||
if (scope.mode === 'volume') return list.filter(ref => volumeOf(ref) === scope.volume);
|
||||
if (scope.mode === 'ids') {
|
||||
const ids = new Set(scope.ids);
|
||||
return list.filter(ref => ids.has(String(ref.id)));
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
function scopeLabel(input) {
|
||||
const scope = normalizeScope(input);
|
||||
if (scope.mode === 'missing') return 'all missing chapters';
|
||||
if (scope.mode === 'latest') return 'newest chapter';
|
||||
if (scope.mode === 'last') return `last ${scope.count} chapters`;
|
||||
if (scope.mode === 'first') return `first ${scope.count} chapters`;
|
||||
if (scope.mode === 'chapter') return `chapter ${scope.chapter}`;
|
||||
if (scope.mode === 'after') return `chapters after ${scope.chapter}`;
|
||||
if (scope.mode === 'volume') return `volume ${scope.volume}`;
|
||||
return `${scope.ids.length} discovered chapters`;
|
||||
}
|
||||
|
||||
module.exports = { normalizeScope, selectByScope, scopeLabel, chapterNumber, volumeOf };
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"version": 1,
|
||||
"libraryRoot": "library",
|
||||
"series": {
|
||||
"title": "Example Manga",
|
||||
"direction": "rtl",
|
||||
"sources": [{ "name": "Example", "url": "https://example.invalid/series/1" }],
|
||||
"metadata": { "authors": ["Example Author"], "status": "ongoing" },
|
||||
"scraper": { "ids": { "example": "1" } }
|
||||
},
|
||||
"http": { "minMs": 8000, "maxMs": 15000, "maxRetries": 2 },
|
||||
"chapters": [{
|
||||
"number": "1",
|
||||
"title": "Opening",
|
||||
"added": "2026-01-01T00:00:00Z",
|
||||
"source": { "name": "Example", "url": "https://example.invalid/chapter/1" },
|
||||
"pages": [
|
||||
{ "path": "C:/absolute/path/0001.jpg", "tags": ["cover"] },
|
||||
{ "url": "https://example.invalid/images/2.jpg", "headers": { "referer": "https://example.invalid/chapter/1" } }
|
||||
]
|
||||
}]
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const { SlowHttp } = require('../lib/http');
|
||||
|
||||
const QUERY = `query ($search: String) {
|
||||
Media(search: $search, type: MANGA, format: MANGA) {
|
||||
id idMal format title { romaji english native } synonyms
|
||||
staff(perPage: 20) { edges { role node { name { full } } } }
|
||||
status startDate { year } genres description(asHtml: false)
|
||||
countryOfOrigin siteUrl coverImage { extraLarge }
|
||||
relations { edges { relationType node { id type format title { romaji english native } siteUrl } } }
|
||||
}
|
||||
}`;
|
||||
|
||||
const QUERY_BY_ID = `query ($id: Int) {
|
||||
Media(id: $id, type: MANGA, format: MANGA) {
|
||||
id idMal format title { romaji english native } synonyms
|
||||
staff(perPage: 20) { edges { role node { name { full } } } }
|
||||
status startDate { year } genres description(asHtml: false)
|
||||
countryOfOrigin siteUrl coverImage { extraLarge }
|
||||
relations { edges { relationType node { id type format title { romaji english native } siteUrl } } }
|
||||
}
|
||||
}`;
|
||||
|
||||
function stripHtml(value) {
|
||||
return String(value || '').replace(/<br\s*\/?>/gi, '\n').replace(/<[^>]+>/g, '').trim();
|
||||
}
|
||||
|
||||
function metadataFromMedia(media) {
|
||||
return {
|
||||
title: media.title.english || media.title.romaji,
|
||||
altTitles: [...new Set([media.title.romaji, media.title.native, ...(media.synonyms || [])].filter(Boolean))],
|
||||
authors: [...new Set((media.staff?.edges || [])
|
||||
.filter(edge => /^(?:original story|story(?: & art)?|art|illustration)$/i.test(edge.role || ''))
|
||||
.map(edge => edge.node?.name?.full).filter(Boolean))],
|
||||
status: media.status === 'FINISHED' ? 'completed' : media.status === 'RELEASING' ? 'ongoing' : 'unknown',
|
||||
year: media.startDate?.year || undefined,
|
||||
tags: media.genres || [], description: stripHtml(media.description),
|
||||
'x-anilist': { id: media.id, idMal: media.idMal, format: media.format, url: media.siteUrl, coverUrl: media.coverImage?.extraLarge,
|
||||
relations: (media.relations?.edges || []).filter(edge => edge.node?.type === 'MANGA').map(edge => ({ id: edge.node.id, type: edge.relationType,
|
||||
format: edge.node.format, title: edge.node.title?.english || edge.node.title?.romaji || edge.node.title?.native, url: edge.node.siteUrl })) },
|
||||
};
|
||||
}
|
||||
|
||||
async function lookup(query, variables, cacheKey, options = {}) {
|
||||
const cacheDir = path.resolve(options.cacheDir || process.env.METADATA_CACHE || path.join(__dirname, '..', '.cache', 'anilist'));
|
||||
await fs.mkdir(cacheDir, { recursive: true });
|
||||
const cacheFile = path.join(cacheDir, `${cacheKey}.json`);
|
||||
try {
|
||||
const stat = await fs.stat(cacheFile);
|
||||
const cached = JSON.parse(await fs.readFile(cacheFile, 'utf8'));
|
||||
if (cached._cacheVersion === 4 && Date.now() - stat.mtimeMs < (options.maxAgeMs || 30 * 86400000)) return cached.metadata;
|
||||
} catch {}
|
||||
|
||||
const http = new SlowHttp({ minMs: 0, maxMs: 0, maxRetries: 1 });
|
||||
const response = await fetch('https://graphql.anilist.co', {
|
||||
method: 'POST', headers: { 'content-type': 'application/json', 'user-agent': http.userAgent },
|
||||
body: JSON.stringify({ query, variables }),
|
||||
signal: AbortSignal.timeout(30000),
|
||||
});
|
||||
if (!response.ok) throw Object.assign(new Error(`AniList HTTP ${response.status}`), { status: response.status });
|
||||
const body = await response.json();
|
||||
if (body.errors?.length) throw new Error(body.errors[0].message);
|
||||
const metadata = metadataFromMedia(body.data.Media);
|
||||
const temporary = `${cacheFile}.${crypto.randomUUID()}.tmp`;
|
||||
try {
|
||||
await fs.writeFile(temporary, JSON.stringify({ _cacheVersion: 4, metadata }, null, 2) + '\n');
|
||||
await fs.rename(temporary, cacheFile);
|
||||
} finally { await fs.rm(temporary, { force: true }); }
|
||||
return metadata;
|
||||
}
|
||||
|
||||
async function lookupAniList(search, options = {}) {
|
||||
const key = Buffer.from(search).toString('base64url');
|
||||
return lookup(QUERY, { search }, key, options);
|
||||
}
|
||||
|
||||
async function lookupAniListById(id, options = {}) {
|
||||
const value = Number(id);
|
||||
if (!Number.isInteger(value) || value < 1) throw new Error('AniList id must be a positive integer');
|
||||
return lookup(QUERY_BY_ID, { id: value }, `id-${value}`, options);
|
||||
}
|
||||
|
||||
module.exports = { lookupAniList, lookupAniListById, metadataFromMedia };
|
||||
@@ -0,0 +1,812 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('fs').promises;
|
||||
const path = require('path');
|
||||
const crypto = require('crypto');
|
||||
const { adapterFor, adapters } = require('./adapters');
|
||||
const { ArchiveControlError } = require('./lib/control');
|
||||
const { sleep } = require('./lib/delay');
|
||||
const { normalizeScope, selectByScope, scopeLabel } = require('./lib/scope');
|
||||
const { scanLibrary, findManga, shouldDownload, isNewerThanLibrary, summarizeInventory, setAutoUpdate } = require('./lib/library-inventory');
|
||||
const { inspectLegacyWork, promoteLegacyWork } = require('./lib/legacy-publication');
|
||||
const { classifyFailure, safeMessage, progressKey } = require('./lib/failures');
|
||||
const { writeJsonAtomic } = require('./lib/json-file');
|
||||
|
||||
const RATE_KEYS = ['minMs', 'maxMs', 'chapterMinMs', 'chapterMaxMs'];
|
||||
const FAST_RATE_SETTINGS = {
|
||||
minMs: 1000, maxMs: 4000, chapterMinMs: 5000, chapterMaxMs: 15000,
|
||||
};
|
||||
const DEFAULT_SOURCE_RATES = {
|
||||
weebcentral: { ...FAST_RATE_SETTINGS },
|
||||
mangadex: { ...FAST_RATE_SETTINGS },
|
||||
};
|
||||
const DEFAULT_SETTINGS = {
|
||||
...FAST_RATE_SETTINGS,
|
||||
sourceRates: Object.fromEntries(Object.entries(DEFAULT_SOURCE_RATES).map(([source, rate]) => [source, { ...rate }])),
|
||||
autoUpdateEnabled: true, autoUpdateTime: '03:00',
|
||||
};
|
||||
|
||||
async function exists(file) {
|
||||
try { await fs.access(file); return true; } catch { return false; }
|
||||
}
|
||||
|
||||
function validateRateSettings(input, prior, prefix = '') {
|
||||
const next = { ...prior };
|
||||
for (const key of RATE_KEYS) {
|
||||
if (input[key] === undefined) continue;
|
||||
const value = Number(input[key]);
|
||||
if (!Number.isFinite(value) || value < 500 || value > 3600000) throw new Error(`${prefix}${key} must be between 500 and 3600000 ms`);
|
||||
next[key] = Math.round(value);
|
||||
}
|
||||
if (next.maxMs < next.minMs) throw new Error(`${prefix}maxMs must be at least ${prefix}minMs`);
|
||||
if (next.chapterMaxMs < next.chapterMinMs) throw new Error(`${prefix}chapterMaxMs must be at least ${prefix}chapterMinMs`);
|
||||
return next;
|
||||
}
|
||||
|
||||
function validateSettings(input = {}, prior = DEFAULT_SETTINGS) {
|
||||
const priorBase = { ...FAST_RATE_SETTINGS, ...prior };
|
||||
const next = validateRateSettings(input, priorBase);
|
||||
if (input.sourceRates !== undefined && (!input.sourceRates || typeof input.sourceRates !== 'object' || Array.isArray(input.sourceRates))) {
|
||||
throw new Error('sourceRates must be an object');
|
||||
}
|
||||
const requestedRates = input.sourceRates || {};
|
||||
const sourceIds = new Set([
|
||||
...Object.keys(DEFAULT_SOURCE_RATES),
|
||||
...Object.keys(prior.sourceRates || {}),
|
||||
...Object.keys(requestedRates),
|
||||
]);
|
||||
next.sourceRates = {};
|
||||
for (const source of sourceIds) {
|
||||
if (!/^[a-z0-9][a-z0-9-]*$/.test(source)) throw new Error(`Invalid source id: ${source}`);
|
||||
const patch = requestedRates[source];
|
||||
if (patch !== undefined && (!patch || typeof patch !== 'object' || Array.isArray(patch))) {
|
||||
throw new Error(`sourceRates.${source} must be an object`);
|
||||
}
|
||||
const base = {
|
||||
...FAST_RATE_SETTINGS,
|
||||
...(DEFAULT_SOURCE_RATES[source] || {}),
|
||||
...((prior.sourceRates || {})[source] || {}),
|
||||
};
|
||||
next.sourceRates[source] = validateRateSettings(patch || {}, base, `sourceRates.${source}.`);
|
||||
}
|
||||
if (input.autoUpdateEnabled !== undefined) next.autoUpdateEnabled = !!input.autoUpdateEnabled;
|
||||
if (input.autoUpdateTime !== undefined) {
|
||||
const value = String(input.autoUpdateTime).trim();
|
||||
if (!/^(?:[01]\d|2[0-3]):[0-5]\d$/.test(value)) throw new Error('autoUpdateTime must be HH:MM');
|
||||
next.autoUpdateTime = value;
|
||||
}
|
||||
return next;
|
||||
}
|
||||
|
||||
function settingsForSource(settings, source) {
|
||||
return { ...settings, ...((settings.sourceRates || {})[source] || {}) };
|
||||
}
|
||||
|
||||
function existingPolicy(value) {
|
||||
const policy = String(value || 'skip-any');
|
||||
if (!['skip-any', 'add-source'].includes(policy)) throw new Error('existingPolicy must be skip-any or add-source');
|
||||
return policy;
|
||||
}
|
||||
|
||||
function nextSchedule(time, from = new Date()) {
|
||||
const [hours, minutes] = String(time).split(':').map(Number);
|
||||
const next = new Date(from);
|
||||
next.setHours(hours, minutes, 0, 0);
|
||||
if (next <= from) next.setDate(next.getDate() + 1);
|
||||
return next;
|
||||
}
|
||||
|
||||
class ScraperService {
|
||||
constructor(options = {}) {
|
||||
this.libraryRoot = path.resolve(options.libraryRoot || 'library');
|
||||
this.dataRoot = path.resolve(options.dataRoot || path.join('data', 'scraper'));
|
||||
this.stateFile = path.join(this.dataRoot, 'queue.json');
|
||||
this.logFile = path.join(this.dataRoot, 'events.log');
|
||||
this.lockFile = path.join(this.dataRoot, 'daemon.lock');
|
||||
this.legacyStateFile = path.join(this.libraryRoot, '.mcscraper-queue', 'queue.json');
|
||||
this.state = null;
|
||||
this.stopping = false;
|
||||
this.loopPromise = null;
|
||||
this.active = new Map();
|
||||
this.writeChain = Promise.resolve();
|
||||
this.discoveryTimer = null;
|
||||
this.inventoryCache = null;
|
||||
this.inventoryCachedAt = 0;
|
||||
this.inventoryGeneration = 0;
|
||||
this.inventoryCacheGeneration = -1;
|
||||
this.inventoryRefresh = null;
|
||||
this.inventoryAttemptedAt = 0;
|
||||
this.inventoryError = null;
|
||||
this.scanInventory = options.scanLibrary || scanLibrary;
|
||||
this.resolveAdapter = options.adapterFor || adapterFor;
|
||||
this.adapterList = options.adapters || adapters;
|
||||
this.retryDelayMs = options.retryDelayMs ?? 300000;
|
||||
this.runtimeHeartbeat = Date.now();
|
||||
}
|
||||
|
||||
async init() {
|
||||
await fs.mkdir(this.dataRoot, { recursive: true });
|
||||
await fs.mkdir(this.libraryRoot, { recursive: true });
|
||||
await this.acquireLock();
|
||||
this.state = await this.loadState();
|
||||
this.state.incidents = this.state.incidents || [];
|
||||
this.state.circuits = this.state.circuits || {};
|
||||
this.state.updateChecks = this.state.updateChecks || {};
|
||||
// Older releases retained successful no-change checks in the work queue.
|
||||
// Keep their latest result before retiring those finished rows.
|
||||
for (const item of this.state.queue) {
|
||||
if (item.kind !== 'discovery' || item.status !== 'complete' || !item.checked || !item.libraryMangaId) continue;
|
||||
const key = JSON.stringify([item.libraryMangaId, item.source]);
|
||||
const prior = this.state.updateChecks[key];
|
||||
if (!prior?.lastCheckedAt || Date.parse(item.checked) > Date.parse(prior.lastCheckedAt)) {
|
||||
this.state.updateChecks[key] = { ...prior, mangaId: item.libraryMangaId, source: item.source,
|
||||
lastCheckedAt: item.checked, lastResult: 'no-updates' };
|
||||
}
|
||||
}
|
||||
this.state.queue = this.state.queue.filter(item => item.kind !== 'discovery' || item.status !== 'complete' || !item.libraryMangaId);
|
||||
this.state.queue.forEach((item, index) => { item.order = index; });
|
||||
for (const item of this.state.queue) {
|
||||
delete item.stopReason;
|
||||
if (item.status === 'downloading' || item.status === 'paused' || item.status === 'discovering') item.status = 'queued';
|
||||
const found = this.resolveAdapter(item.url);
|
||||
item.source = found?.adapter.id || item.source || 'unknown';
|
||||
item.seriesId = item.seriesId || found?.info.id || item.id;
|
||||
item.kind = ['discovery', 'inspection'].includes(item.kind) ? item.kind : 'download';
|
||||
item.scope = normalizeScope(item.scope || {});
|
||||
item.scopeLabel = item.scopeLabel || scopeLabel(item.scope);
|
||||
item.existingPolicy = existingPolicy(item.existingPolicy);
|
||||
if (item.status === 'error') {
|
||||
item.failure = item.failure || classifyFailure(new Error(item.error));
|
||||
if (item.failure.kind === 'unavailable') item.status = 'unavailable';
|
||||
}
|
||||
if (item.source === 'mangadex' && item.status === 'unavailable' && !item.availability
|
||||
&& item.error === 'No downloadable English MangaDex chapters found') {
|
||||
item.error = 'The previous adapter selected no numbered hosted English chapters. Inspect this source again to verify its edition and availability, including pilots and specials.';
|
||||
if (item.failure) item.failure = { ...item.failure, message: item.error };
|
||||
}
|
||||
}
|
||||
this.state.version = 4;
|
||||
this.state.currentId = null;
|
||||
this.state.currentIds = {};
|
||||
this.state.sourcePaused = this.state.sourcePaused || {};
|
||||
// A legacy title-unavailable pause can be released; access and unknown failures remain paused.
|
||||
for (const source of Object.keys(this.state.sourcePaused)) {
|
||||
const failed = this.state.queue.filter(item => item.source === source && ['error', 'unavailable', 'quarantined'].includes(item.status));
|
||||
if (failed.length && failed.every(item => item.failure?.kind === 'unavailable')) delete this.state.sourcePaused[source];
|
||||
}
|
||||
this.state.status = this.state.paused ? 'paused' : 'idle';
|
||||
this.state.pid = process.pid;
|
||||
await this.event('info', 'Scraper daemon started', { pid: process.pid });
|
||||
this.scheduleDiscovery();
|
||||
this.loopPromise = this.workLoop();
|
||||
return this;
|
||||
}
|
||||
|
||||
async acquireLock() {
|
||||
try {
|
||||
const handle = await fs.open(this.lockFile, 'wx');
|
||||
await handle.writeFile(String(process.pid));
|
||||
return await handle.close();
|
||||
} catch (error) {
|
||||
if (error.code !== 'EEXIST') throw error;
|
||||
}
|
||||
const oldPid = Number((await fs.readFile(this.lockFile, 'utf8').catch(() => '')).trim());
|
||||
try {
|
||||
if (oldPid) process.kill(oldPid, 0);
|
||||
throw new Error(`Scraper daemon is already running (PID ${oldPid || 'unknown'})`);
|
||||
} catch (error) {
|
||||
if (error.code !== 'ESRCH') throw error;
|
||||
await fs.rm(this.lockFile, { force: true });
|
||||
const handle = await fs.open(this.lockFile, 'wx');
|
||||
await handle.writeFile(String(process.pid));
|
||||
await handle.close();
|
||||
}
|
||||
}
|
||||
|
||||
async loadState() {
|
||||
if (await exists(this.stateFile)) {
|
||||
const saved = JSON.parse(await fs.readFile(this.stateFile, 'utf8'));
|
||||
return {
|
||||
version: 3, paused: false, status: 'idle', queue: [], history: [], publishedRevision: 0,
|
||||
currentIds: {}, sourcePaused: {},
|
||||
settings: DEFAULT_SETTINGS, ...saved,
|
||||
settings: validateSettings(saved.settings || {}, DEFAULT_SETTINGS),
|
||||
};
|
||||
}
|
||||
const state = {
|
||||
version: 3, paused: false, status: 'idle', currentId: null, currentIds: {}, sourcePaused: {},
|
||||
settings: { ...DEFAULT_SETTINGS }, queue: [], history: [], publishedRevision: 0,
|
||||
created: new Date().toISOString(), lastError: null,
|
||||
};
|
||||
if (await exists(this.legacyStateFile)) {
|
||||
const legacy = JSON.parse(await fs.readFile(this.legacyStateFile, 'utf8'));
|
||||
const completed = new Set(legacy.completed || []);
|
||||
state.queue = (legacy.entries || []).map((item, index) => ({
|
||||
id: item.id, seriesId: item.id, source: 'weebcentral', title: item.title, url: item.url,
|
||||
metadataSearch: item.metadataSearch || item.title,
|
||||
kind: 'download', scope: { mode: 'missing' }, existingPolicy: 'skip-any',
|
||||
status: completed.has(item.id) ? 'complete' : 'queued', order: index,
|
||||
added: legacy.created || new Date().toISOString(),
|
||||
}));
|
||||
state.settings = validateSettings(legacy.settings || {}, DEFAULT_SETTINGS);
|
||||
state.migratedFrom = this.legacyStateFile;
|
||||
}
|
||||
return state;
|
||||
}
|
||||
|
||||
async persist() {
|
||||
const write = async () => {
|
||||
this.state.updated = new Date().toISOString();
|
||||
await writeJsonAtomic(this.stateFile, this.state);
|
||||
};
|
||||
const pending = this.writeChain.then(write, write);
|
||||
this.writeChain = pending.catch(() => {});
|
||||
return pending;
|
||||
}
|
||||
|
||||
async event(level, message, details = {}) {
|
||||
const entry = { time: new Date().toISOString(), level, message: safeMessage(message), ...details };
|
||||
if (entry.url) entry.url = safeMessage(entry.url);
|
||||
await fs.appendFile(this.logFile, JSON.stringify(entry) + '\n');
|
||||
console.log(JSON.stringify({ service: 'scraper', ...entry }));
|
||||
this.state.lastEvent = entry;
|
||||
if (level === 'error') this.state.lastError = entry;
|
||||
await this.persist();
|
||||
}
|
||||
|
||||
async recentEvents(limit = 30) {
|
||||
const handle = await fs.open(this.logFile, 'r').catch(error => {
|
||||
if (error.code === 'ENOENT') return null;
|
||||
throw error;
|
||||
});
|
||||
if (!handle) return [];
|
||||
let text;
|
||||
try {
|
||||
const { size } = await handle.stat();
|
||||
const length = Math.min(size, 512 * 1024), start = size - length;
|
||||
const buffer = Buffer.alloc(length);
|
||||
const { bytesRead } = await handle.read(buffer, 0, length, start);
|
||||
text = buffer.subarray(0, bytesRead).toString('utf8');
|
||||
if (start > 0) {
|
||||
const boundary = text.indexOf('\n');
|
||||
text = boundary < 0 ? '' : text.slice(boundary + 1);
|
||||
}
|
||||
} finally { await handle.close(); }
|
||||
return text.trim().split(/\r?\n/).filter(Boolean).slice(-Math.max(1, Math.min(200, limit)))
|
||||
.map(line => { try { return JSON.parse(line); } catch { return { level: 'warning', message: line }; } });
|
||||
}
|
||||
|
||||
invalidateInventory() {
|
||||
// Keep the last summary visible while rebuilding. Download/discovery decisions
|
||||
// still await an authoritative scan through libraryInventory(), never this stale view.
|
||||
this.inventoryGeneration++;
|
||||
}
|
||||
|
||||
refreshInventory() {
|
||||
if (this.inventoryRefresh) return this.inventoryRefresh;
|
||||
const generation = this.inventoryGeneration;
|
||||
this.inventoryAttemptedAt = Date.now();
|
||||
this.inventoryRefresh = Promise.resolve().then(() => this.scanInventory(this.libraryRoot))
|
||||
.then(inventory => {
|
||||
// A chapter or metadata edit during the scan invalidates its result.
|
||||
if (generation === this.inventoryGeneration) {
|
||||
this.inventoryCache = inventory;
|
||||
this.inventoryCachedAt = Date.now();
|
||||
this.inventoryCacheGeneration = generation;
|
||||
}
|
||||
this.inventoryError = null;
|
||||
return inventory;
|
||||
}).catch(error => {
|
||||
this.inventoryError = safeMessage(error.message);
|
||||
throw error;
|
||||
}).finally(() => { this.inventoryRefresh = null; });
|
||||
return this.inventoryRefresh;
|
||||
}
|
||||
|
||||
async libraryInventory(force = false) {
|
||||
if (!force && this.inventoryCache && this.inventoryCacheGeneration === this.inventoryGeneration
|
||||
&& Date.now() - this.inventoryCachedAt < 60000) return this.inventoryCache;
|
||||
do {
|
||||
await this.refreshInventory();
|
||||
} while (this.inventoryCacheGeneration !== this.inventoryGeneration);
|
||||
return this.inventoryCache;
|
||||
}
|
||||
|
||||
scheduleDiscovery() {
|
||||
if (this.discoveryTimer) clearTimeout(this.discoveryTimer);
|
||||
this.discoveryTimer = null;
|
||||
if (!this.state?.settings?.autoUpdateEnabled || this.stopping) {
|
||||
if (this.state) this.state.nextDiscoveryAt = null;
|
||||
return;
|
||||
}
|
||||
const next = nextSchedule(this.state.settings.autoUpdateTime);
|
||||
this.state.nextDiscoveryAt = next.toISOString();
|
||||
const delay = Math.max(1000, next.getTime() - Date.now());
|
||||
this.discoveryTimer = setTimeout(() => {
|
||||
this.discoveryTimer = null;
|
||||
this.enqueueDiscoveryChecks('schedule').catch(error => this.event('error', `Nightly discovery failed: ${error.message}`))
|
||||
.finally(() => { if (!this.stopping) this.scheduleDiscovery(); });
|
||||
}, delay);
|
||||
this.discoveryTimer.unref();
|
||||
}
|
||||
|
||||
pendingSeries(source, seriesId) {
|
||||
return this.state.queue.some(item => item.source === source && String(item.seriesId) === String(seriesId)
|
||||
&& ['queued', 'downloading', 'discovering', 'ready', 'error', 'retry_wait', 'quarantined', 'unavailable'].includes(item.status));
|
||||
}
|
||||
|
||||
recordUpdateCheck(item, patch) {
|
||||
if (!item.libraryMangaId || !item.source) return;
|
||||
const key = JSON.stringify([item.libraryMangaId, item.source]);
|
||||
this.state.updateChecks[key] = { ...(this.state.updateChecks[key] || {}),
|
||||
mangaId: item.libraryMangaId, source: item.source, ...patch };
|
||||
}
|
||||
|
||||
async enqueueDiscoveryChecks(reason = 'manual') {
|
||||
const inventory = await this.libraryInventory(true);
|
||||
let queued = 0, skipped = 0;
|
||||
for (const manga of inventory.mangas) {
|
||||
if (!manga.autoUpdate) continue;
|
||||
const supported = manga.sources.map(source => ({ source, found: this.resolveAdapter(source.url) })).filter(item => item.found);
|
||||
if (!supported.length) { skipped++; continue; }
|
||||
const preferred = manga.meta?.['x-scraper']?.preferredSource;
|
||||
const selected = supported.find(item => item.found.adapter.id === preferred) || supported[0];
|
||||
const source = selected.found.adapter.id, seriesId = selected.found.info.id;
|
||||
if (this.pendingSeries(source, seriesId)) { skipped++; continue; }
|
||||
const item = {
|
||||
id: crypto.randomUUID(), seriesId, source, kind: 'discovery',
|
||||
title: manga.title, url: selected.found.info.url, metadataSearch: manga.title,
|
||||
libraryMangaId: manga.id, scope: { mode: 'missing' }, existingPolicy: 'skip-any',
|
||||
status: 'queued', order: this.state.queue.length, added: new Date().toISOString(), reason,
|
||||
};
|
||||
this.state.queue.push(item); queued++;
|
||||
}
|
||||
this.state.lastDiscovery = { time: new Date().toISOString(), reason, queued, skipped };
|
||||
await this.event('info', `Queued ${queued} update check${queued === 1 ? '' : 's'}`, { reason, queued, skipped });
|
||||
return { queued, skipped };
|
||||
}
|
||||
|
||||
async setMangaAutoUpdate(id, enabled) {
|
||||
const result = await setAutoUpdate(this.libraryRoot, id, enabled);
|
||||
this.invalidateInventory();
|
||||
await this.event('info', `${enabled ? 'Enabled' : 'Disabled'} auto update for ${id}`, { mangaId: id, enabled: !!enabled });
|
||||
return result;
|
||||
}
|
||||
|
||||
async publishLegacyReady() {
|
||||
if (!this.state.paused || this.active.size || Object.keys(this.state.currentIds || {}).length) {
|
||||
throw new Error('Pause the queue and wait for active workers to stop before publishing legacy work');
|
||||
}
|
||||
const candidates = [];
|
||||
const seen = new Set();
|
||||
for (const item of this.state.queue) {
|
||||
const seriesId = String(item.seriesId || item.id || '');
|
||||
if (!seriesId || seen.has(seriesId) || item.status === 'complete') continue;
|
||||
seen.add(seriesId);
|
||||
const inspection = await inspectLegacyWork({ libraryRoot: this.libraryRoot, seriesId });
|
||||
if (inspection) candidates.push({ item, inspection });
|
||||
}
|
||||
const published = [];
|
||||
for (const candidate of candidates) {
|
||||
const result = await promoteLegacyWork(candidate.inspection);
|
||||
candidate.item.libraryMangaId = result.mangaId;
|
||||
candidate.item.legacyPromotedAt = new Date().toISOString();
|
||||
published.push(result);
|
||||
this.state.publishedRevision = Number(this.state.publishedRevision || 0) + 1;
|
||||
this.invalidateInventory();
|
||||
await this.event('info', `Published ${result.chaptersPublished} completed chapters for ${candidate.item.title}`, {
|
||||
id: candidate.item.id, seriesId: candidate.item.seriesId, title: candidate.item.title,
|
||||
mangaId: result.mangaId, chaptersPublished: result.chaptersPublished,
|
||||
});
|
||||
}
|
||||
return { published };
|
||||
}
|
||||
|
||||
async runDiscovery(item, found, source) {
|
||||
if (this.stopping || this.state.paused) throw new ArchiveControlError(this.stopping ? 'stopped' : 'paused');
|
||||
item.status = 'discovering';
|
||||
const catalog = await found.adapter.discover(item, {
|
||||
settings: settingsForSource(this.state.settings, source),
|
||||
shouldStop: () => this.stopping ? 'stopped' : this.state.paused ? 'paused' : item.stopReason || false,
|
||||
onEvent: event => this.event(event.level, event.message, { id: item.id, title: item.title, source, ...(event.status ? { status: event.status } : {}) }),
|
||||
});
|
||||
if (this.stopping || this.state.paused || item.stopReason) throw new ArchiveControlError(item.stopReason || (this.stopping ? 'stopped' : 'paused'));
|
||||
await this.recordCatalog(item, catalog);
|
||||
if (item.kind === 'inspection') {
|
||||
if (!catalog.chapters?.length) throw Object.assign(new Error(catalog.availability?.message || 'This source edition has no downloadable chapters'), {
|
||||
failureKind: catalog.availability?.reason === 'unrecognized-labels' ? 'adapter' : 'unavailable',
|
||||
});
|
||||
if (item.inspectionFromComplete) {
|
||||
const inventory = await this.libraryInventory(true);
|
||||
const manga = findManga(inventory, {
|
||||
source, seriesId: item.seriesId, sourceUrl: item.url,
|
||||
anilist: catalog.links?.al, libraryMangaId: item.libraryMangaId,
|
||||
});
|
||||
const missing = selectByScope(catalog.chapters, item.scope)
|
||||
.filter(ref => shouldDownload(manga, ref, source, item.existingPolicy));
|
||||
if (!missing.length) {
|
||||
item.kind = 'download'; item.status = 'complete'; item.checked = new Date().toISOString();
|
||||
delete item.inspectionFromComplete;
|
||||
for (const incident of this.state.incidents) if (incident.jobId === item.id && incident.state === 'open') {
|
||||
incident.state = 'resolved'; incident.resolution = 'inspected-complete';
|
||||
}
|
||||
await this.event('info', `Source inspected: ${item.sourceTitle}; completed download is already in the library`, { id: item.id, source });
|
||||
return;
|
||||
}
|
||||
}
|
||||
delete item.inspectionFromComplete;
|
||||
item.status = 'ready'; item.checked = new Date().toISOString();
|
||||
for (const incident of this.state.incidents) if (incident.jobId === item.id && incident.state === 'open') {
|
||||
incident.state = 'resolved'; incident.resolution = 'inspected-ready';
|
||||
}
|
||||
await this.event('info', `Source inspected: ${item.sourceTitle}; awaiting download confirmation`, { id: item.id, source });
|
||||
return;
|
||||
}
|
||||
const inventory = await this.libraryInventory(true);
|
||||
if (this.stopping || this.state.paused) throw new ArchiveControlError(this.stopping ? 'stopped' : 'paused');
|
||||
const manga = findManga(inventory, {
|
||||
source, seriesId: item.seriesId, sourceUrl: item.url,
|
||||
anilist: catalog.links?.al, libraryMangaId: item.libraryMangaId,
|
||||
});
|
||||
if (item.libraryMangaId && !manga) throw new Error(`Library manga target not found: ${item.libraryMangaId}`);
|
||||
const candidates = selectByScope(catalog.chapters || [], item.scope)
|
||||
.filter(ref => shouldDownload(manga, ref, source, item.existingPolicy))
|
||||
.filter(ref => isNewerThanLibrary(manga, ref));
|
||||
item.checked = new Date().toISOString();
|
||||
item.catalogTotal = catalog.chapters?.length || 0;
|
||||
this.recordUpdateCheck(item, { lastCheckedAt: item.checked,
|
||||
...(candidates.length ? { lastNewContentAt: item.checked, lastResult: 'new-content' } : { lastResult: 'no-updates' }) });
|
||||
if (!candidates.length) {
|
||||
item.status = 'complete'; item.completed = item.checked; item.discovered = 0;
|
||||
this.state.history.push({ id: item.id, title: item.title, source, status: 'checked', time: item.completed, discovered: 0 });
|
||||
this.state.queue = this.state.queue.filter(entry => entry.id !== item.id);
|
||||
this.state.queue.forEach((entry, index) => { entry.order = index; });
|
||||
await this.event('info', `No updates found for ${item.title}`, { id: item.id, title: item.title, source });
|
||||
return;
|
||||
}
|
||||
item.kind = 'download'; item.status = 'queued'; item.discovered = candidates.length;
|
||||
item.scope = normalizeScope({ mode: 'ids', ids: candidates.map(ref => ref.id) });
|
||||
item.scopeLabel = scopeLabel(item.scope);
|
||||
await this.event('info', `Found ${candidates.length} new chapter${candidates.length === 1 ? '' : 's'} for ${item.title}`, {
|
||||
id: item.id, title: item.title, source, discovered: candidates.length,
|
||||
});
|
||||
}
|
||||
|
||||
async recordCatalog(item, catalog) {
|
||||
const title = String(catalog.title || item.title).slice(0, 200);
|
||||
const changedEdition = item.confirmedSourceTitle && item.confirmedSourceTitle !== title;
|
||||
item.sourceTitle = title;
|
||||
item.catalogTotal = catalog.chapters?.length || 0;
|
||||
item.availability = catalog.availability || { selectedChapters: item.catalogTotal, coverage: 'unknown' };
|
||||
await this.persist();
|
||||
if (changedEdition) throw Object.assign(new Error('The source edition title changed since inspection. Inspect and confirm it again before downloading.'), { failureKind: 'unavailable' });
|
||||
}
|
||||
|
||||
async jobState(item) {
|
||||
if (!item) return null;
|
||||
if (item.kind === 'inspection') return { status: 'discovering', updated: item.started, completed: 0, total: null, current: null };
|
||||
const file = path.join(this.libraryRoot, `.mcscraper-work-${item.seriesId || item.id}`, 'job.json');
|
||||
try {
|
||||
const job = JSON.parse(await fs.readFile(file, 'utf8'));
|
||||
return {
|
||||
status: job.status || 'unknown', completed: job.publicationMode === 'incremental' ? Number(job.jobCompleted || 0) : job.completed?.length || 0, total: job.total || null,
|
||||
percent: job.total ? Math.round((job.publicationMode === 'incremental' ? Number(job.jobCompleted || 0) : job.completed?.length || 0) * 1000 / job.total) / 10 : null,
|
||||
lastChapter: job.lastChapter || null, current: job.current || null,
|
||||
lastProblem: job.lastProblem || null, lastError: job.lastError || null,
|
||||
problemCount: job.problemCount || 0, rate: job.rate || null, updated: job.updated || null,
|
||||
};
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
async publicState() {
|
||||
const activeItems = this.state.queue.filter(item => Object.values(this.state.currentIds || {}).includes(item.id));
|
||||
const active = await Promise.all(activeItems.map(async current => ({
|
||||
source: this.resolveAdapter(current.url)?.adapter.id || current.source,
|
||||
current, job: await this.jobState(current),
|
||||
})));
|
||||
const current = active[0]?.current || null;
|
||||
const job = active[0]?.job || null;
|
||||
// Status and watchdog probes must not wait for thousands of bind-mount reads.
|
||||
// Share one background scan, serve the previous summary, and surface its age/error.
|
||||
if (!this.inventoryRefresh && (this.inventoryCacheGeneration !== this.inventoryGeneration || Date.now() - this.inventoryCachedAt >= 60000)
|
||||
&& Date.now() - this.inventoryAttemptedAt >= 10000) {
|
||||
this.refreshInventory().catch(() => {});
|
||||
}
|
||||
const inventory = this.inventoryCache;
|
||||
return {
|
||||
version: this.state.version, status: this.state.status, paused: this.state.paused,
|
||||
pid: this.state.pid, settings: this.state.settings, publishedRevision: this.state.publishedRevision || 0,
|
||||
current, job, active, sourcePaused: this.state.sourcePaused || {}, queue: this.state.queue,
|
||||
history: (this.state.history || []).slice(-20).reverse(), lastError: this.state.lastError || null,
|
||||
lastEvent: this.state.lastEvent || null, updated: this.state.updated || null,
|
||||
nextDiscoveryAt: this.state.nextDiscoveryAt || null, lastDiscovery: this.state.lastDiscovery || null,
|
||||
library: inventory ? summarizeInventory(inventory, value => this.resolveAdapter(value)) : [],
|
||||
updateChecks: Object.values(this.state.updateChecks || {}),
|
||||
libraryStatus: {
|
||||
ready: !!inventory, refreshing: !!this.inventoryRefresh,
|
||||
stale: this.inventoryCacheGeneration !== this.inventoryGeneration || Date.now() - this.inventoryCachedAt >= 60000,
|
||||
updated: this.inventoryCachedAt ? new Date(this.inventoryCachedAt).toISOString() : null,
|
||||
error: this.inventoryError,
|
||||
},
|
||||
adapters: this.adapterList.map(item => ({ id: item.id, label: item.label })),
|
||||
events: await this.recentEvents(25),
|
||||
incidents: (this.state.incidents || []).slice(-50).reverse(), circuits: this.state.circuits || {},
|
||||
runtimeHeartbeat: new Date(this.runtimeHeartbeat).toISOString(),
|
||||
};
|
||||
}
|
||||
|
||||
async enqueue(input) {
|
||||
const found = this.resolveAdapter(input.url);
|
||||
if (!found) throw new Error('No installed scraper adapter accepts this URL');
|
||||
const scope = normalizeScope(input.scope || {});
|
||||
const policy = existingPolicy(input.existingPolicy);
|
||||
const signature = JSON.stringify([found.adapter.id, found.info.id, scope, policy, input.libraryMangaId || null]);
|
||||
if (this.state.queue.some(item => item.status !== 'complete'
|
||||
&& JSON.stringify([item.source, item.seriesId, item.scope, item.existingPolicy, item.libraryMangaId || null]) === signature)) {
|
||||
throw new Error('That exact manga scope is already queued');
|
||||
}
|
||||
const item = {
|
||||
...found.info, id: crypto.randomUUID(), seriesId: found.info.id, kind: input.inspect === true ? 'inspection' : 'download',
|
||||
title: String(input.title || found.info.title).trim().slice(0, 200),
|
||||
metadataSearch: String(input.metadataSearch || input.title || found.info.title).trim().slice(0, 200),
|
||||
scope, scopeLabel: scopeLabel(scope), existingPolicy: policy,
|
||||
...(input.libraryMangaId ? { libraryMangaId: String(input.libraryMangaId) } : {}),
|
||||
status: 'queued', order: this.state.queue.length, added: new Date().toISOString(),
|
||||
};
|
||||
this.state.queue.push(item);
|
||||
await this.event('info', `Queued ${item.title}: ${item.scopeLabel}`, { id: item.id, seriesId: item.seriesId, title: item.title });
|
||||
return item;
|
||||
}
|
||||
|
||||
async remove(id) {
|
||||
const item = this.state.queue.find(entry => entry.id === id);
|
||||
if (!item) throw new Error('Queue item not found');
|
||||
if (Object.values(this.state.currentIds || {}).includes(item.id)) throw new Error('Pause the active download before removing it');
|
||||
this.state.queue = this.state.queue.filter(entry => entry.id !== id);
|
||||
this.state.queue.forEach((entry, index) => { entry.order = index; });
|
||||
await this.event('info', `Removed ${item.title} from queue`, { id: item.id, title: item.title });
|
||||
}
|
||||
|
||||
async clearCompleted(source = '') {
|
||||
if (source && !this.adapterList.some(adapter => adapter.id === source)) throw new Error('Unknown source');
|
||||
const before = this.state.queue.length;
|
||||
const activeIds = new Set(Object.values(this.state.currentIds || {}));
|
||||
this.state.queue = this.state.queue.filter(item => !(item.status === 'complete' && item.kind === 'download'
|
||||
&& !activeIds.has(item.id) && (!source || item.source === source)));
|
||||
this.state.queue.forEach((item, index) => { item.order = index; });
|
||||
const cleared = before - this.state.queue.length;
|
||||
if (cleared) await this.event('info', `Cleared ${cleared} completed download job${cleared === 1 ? '' : 's'} from queue`, { source: source || 'all', cleared });
|
||||
return { cleared };
|
||||
}
|
||||
|
||||
async reorder(ids) {
|
||||
if (!Array.isArray(ids) || ids.length !== this.state.queue.length || new Set(ids).size !== ids.length) throw new Error('Order must contain every queue id exactly once');
|
||||
const byId = new Map(this.state.queue.map(item => [item.id, item]));
|
||||
if (ids.some(id => !byId.has(id))) throw new Error('Order contains an unknown queue id');
|
||||
for (const current of Object.values(this.state.currentIds || {})) {
|
||||
if (ids.indexOf(current) !== this.state.queue.findIndex(item => item.id === current)) throw new Error('Active items cannot be moved while downloading');
|
||||
}
|
||||
this.state.queue = ids.map((id, index) => ({ ...byId.get(id), order: index }));
|
||||
await this.event('info', 'Queue reordered');
|
||||
}
|
||||
|
||||
async updateSettings(patch) {
|
||||
this.state.settings = validateSettings(patch, this.state.settings);
|
||||
this.scheduleDiscovery();
|
||||
await this.event('info', 'Scraper settings updated', { settings: this.state.settings, nextDiscoveryAt: this.state.nextDiscoveryAt });
|
||||
return this.state.settings;
|
||||
}
|
||||
|
||||
async control(action, id) {
|
||||
if (action === 'inspect' || action === 'confirm') {
|
||||
const item = this.state.queue.find(entry => entry.id === id);
|
||||
if (!item || Object.values(this.state.currentIds || {}).includes(item.id)) throw new Error('Job not found or already active');
|
||||
if (this.state.sourcePaused[item.source] || item.failure?.kind === 'access') throw new Error('Review the source access block before inspecting or confirming');
|
||||
if (action === 'confirm') {
|
||||
if (item.status !== 'ready' || item.kind !== 'inspection') throw new Error('Inspect the source first');
|
||||
item.confirmedSourceTitle = item.sourceTitle;
|
||||
item.kind = 'download';
|
||||
} else {
|
||||
if (!['unavailable', 'quarantined', 'complete', 'ready'].includes(item.status)) throw new Error('Only inactive finished, unavailable or quarantined jobs can be inspected again');
|
||||
item.inspectionFromComplete = !!item.completed && Number(item.publishedCount || 0) > 0;
|
||||
item.kind = 'inspection'; delete item.confirmedSourceTitle;
|
||||
}
|
||||
item.status = 'queued'; delete item.error; delete item.failure; delete item.nextAttemptAt;
|
||||
await this.event('info', action === 'confirm' ? `Download confirmed: ${item.sourceTitle}` : `Source inspection queued: ${item.title}`, { id: item.id, source: item.source });
|
||||
return;
|
||||
}
|
||||
if (action === 'pause') {
|
||||
this.state.paused = true;
|
||||
this.state.status = 'pausing';
|
||||
await this.event('info', 'Pause requested');
|
||||
return;
|
||||
}
|
||||
if (action === 'resume') {
|
||||
this.state.paused = false;
|
||||
this.recomputeStatus();
|
||||
await this.event('info', 'Queue resumed');
|
||||
return;
|
||||
}
|
||||
if (action === 'retry') {
|
||||
const item = this.state.queue.find(entry => entry.id === id) || this.state.queue.find(entry => entry.status === 'error');
|
||||
if (!item) throw new Error('No failed queue item to retry');
|
||||
if (Object.values(this.state.currentIds || {}).includes(item.id)) throw new Error('That item is already active');
|
||||
const source = this.resolveAdapter(item.url)?.adapter.id || item.source;
|
||||
const otherAccessError = this.state.queue.some(entry => entry.id !== item.id && entry.source === source && entry.status === 'error' && entry.failure?.kind === 'access');
|
||||
if (otherAccessError) throw new Error('Review the access-blocked job for this source first');
|
||||
item.status = 'queued'; delete item.error; delete item.failure; delete item.nextAttemptAt;
|
||||
item.recoveryAttempts = 0;
|
||||
delete this.state.sourcePaused[source]; delete this.state.circuits[source];
|
||||
for (const incident of this.state.incidents) if (incident.jobId === item.id && incident.state === 'open') { incident.state = 'resolved'; incident.resolution = 'operator-retry'; }
|
||||
this.state.lastError = null; this.state.paused = false; this.recomputeStatus();
|
||||
await this.event('info', `Retrying ${item.title}`, { id: item.id, title: item.title });
|
||||
return;
|
||||
}
|
||||
if (action === 'skip') {
|
||||
const item = this.state.queue.find(entry => entry.id === id);
|
||||
if (!item || this.state.currentIds[item.source] === item.id) throw new Error('Job not found or it is active');
|
||||
item.status = 'skipped';
|
||||
if (item.failure?.scope === 'job') delete this.state.sourcePaused[item.source];
|
||||
await this.event('info', `Skipped ${item.title}`, { id: item.id, source: item.source });
|
||||
return;
|
||||
}
|
||||
throw new Error('Unknown control action');
|
||||
}
|
||||
|
||||
async recover(input) {
|
||||
const item = this.state.queue.find(entry => entry.id === input.id);
|
||||
if (!item) throw new Error('Job not found');
|
||||
// Compare observations so a stale watcher cannot disturb a progressing worker.
|
||||
if (Object.values(this.state.currentIds).includes(item.id)) {
|
||||
const job = await this.jobState(item);
|
||||
if (progressKey(job) !== input.progressKey || this.state.paused || item.stopReason) return { changed: false };
|
||||
if (Date.now() - Date.parse(job?.updated || item.started) < 900000) return { changed: false };
|
||||
if ((item.stallRecoveries || 0) >= 1) {
|
||||
item.stopReason = 'quarantined';
|
||||
} else { item.stallRecoveries = (item.stallRecoveries || 0) + 1; item.stopReason = 'recovering'; }
|
||||
await this.event('warning', `Sentinel requested recovery at the next checkpoint for ${item.title}`, { id: item.id, source: item.source });
|
||||
return { changed: true };
|
||||
}
|
||||
return { changed: false };
|
||||
}
|
||||
|
||||
recomputeStatus() {
|
||||
const activeIds = Object.values(this.state.currentIds || {});
|
||||
this.state.currentId = activeIds[0] || null;
|
||||
if (this.stopping) this.state.status = 'stopping';
|
||||
else if (this.state.paused) this.state.status = activeIds.length ? 'pausing' : 'paused';
|
||||
else if (activeIds.length) this.state.status = 'downloading';
|
||||
else if (this.state.queue.some(item => item.status === 'retry_wait')) this.state.status = 'retry_wait';
|
||||
else if (this.state.queue.some(item => ['error', 'quarantined'].includes(item.status))) this.state.status = 'attention';
|
||||
else this.state.status = 'idle';
|
||||
}
|
||||
|
||||
startItem(item, found) {
|
||||
const source = found.adapter.id;
|
||||
item.status = 'downloading'; item.started = item.kind === 'inspection' ? new Date().toISOString() : item.started || new Date().toISOString(); delete item.error;
|
||||
this.state.currentIds[source] = item.id;
|
||||
this.state.lastError = null;
|
||||
this.recomputeStatus();
|
||||
const entry = { id: item.id, source, promise: null };
|
||||
this.active.set(source, entry);
|
||||
entry.promise = this.runItem(item, found, source).finally(async () => {
|
||||
this.active.delete(source);
|
||||
if (this.state.currentIds[source] === item.id) delete this.state.currentIds[source];
|
||||
this.recomputeStatus();
|
||||
await this.persist();
|
||||
}).catch(error => { console.error('[scraper] persistence failed:', safeMessage(error)); this.state.paused = true; });
|
||||
}
|
||||
|
||||
async runItem(item, found, source) {
|
||||
await this.event('info', `${item.kind === 'discovery' ? 'Checking' : 'Starting'} ${item.title}`, { id: item.id, title: item.title, source, kind: item.kind });
|
||||
try {
|
||||
if (item.kind === 'discovery' || item.kind === 'inspection') {
|
||||
await this.runDiscovery(item, found, source);
|
||||
return;
|
||||
}
|
||||
let revisionNotifications = 0;
|
||||
const result = await found.adapter.run(item, {
|
||||
libraryRoot: this.libraryRoot, settings: settingsForSource(this.state.settings, source),
|
||||
shouldStop: () => this.stopping ? 'stopped' : this.state.paused ? 'paused' : item.stopReason || false,
|
||||
onCatalog: catalog => this.recordCatalog(item, catalog),
|
||||
onPublished: async details => {
|
||||
revisionNotifications++;
|
||||
this.state.publishedRevision = Number(this.state.publishedRevision || 0) + 1;
|
||||
const mangaId = item.libraryMangaId || (details.mangaDir && path.basename(details.mangaDir));
|
||||
if (mangaId) this.recordUpdateCheck({ ...item, libraryMangaId: mangaId }, { lastDownloadedAt: new Date().toISOString() });
|
||||
this.invalidateInventory();
|
||||
await this.event('info', `Published ${item.title}${details.chapter ? ` chapter ${details.chapter}` : ''}`, {
|
||||
id: item.id, title: item.title, source, ...(details.chapter ? { chapter: details.chapter } : {}),
|
||||
});
|
||||
},
|
||||
});
|
||||
const finalPath = typeof result === 'string' ? result : result?.finalPath || null;
|
||||
if (!revisionNotifications && (typeof result === 'string' || Number(result?.publishedCount || 0) > 0)) {
|
||||
this.state.publishedRevision = Number(this.state.publishedRevision || 0) + 1;
|
||||
this.invalidateInventory();
|
||||
}
|
||||
item.status = 'complete'; item.completed = new Date().toISOString(); item.finalPath = finalPath;
|
||||
delete this.state.circuits[source];
|
||||
for (const incident of this.state.incidents) if (incident.jobId === item.id && incident.state === 'open') { incident.state = 'resolved'; incident.resolution = 'recovered'; }
|
||||
item.publishedCount = Number(result?.publishedCount ?? 0);
|
||||
if (item.publishedCount > 0 && !revisionNotifications) {
|
||||
const mangaId = item.libraryMangaId || (finalPath && path.basename(finalPath));
|
||||
if (mangaId) this.recordUpdateCheck({ ...item, libraryMangaId: mangaId }, { lastDownloadedAt: item.completed });
|
||||
}
|
||||
item.selectedCount = Number(result?.selectedCount ?? 0);
|
||||
this.state.history.push({ id: item.id, title: item.title, sourceTitle: item.sourceTitle, availability: item.availability, source, status: 'complete', time: item.completed, finalPath, publishedCount: item.publishedCount, selectedCount: item.selectedCount });
|
||||
await this.event('info', `Completed ${item.title}`, { id: item.id, title: item.title, source, finalPath, publishedCount: item.publishedCount });
|
||||
} catch (error) {
|
||||
if (error instanceof ArchiveControlError) {
|
||||
item.status = error.reason === 'quarantined' ? 'quarantined' : 'queued';
|
||||
delete item.stopReason;
|
||||
await this.event('info', `${item.title} ${error.reason}`, { id: item.id, title: item.title, source });
|
||||
return;
|
||||
}
|
||||
const failure = classifyFailure(error);
|
||||
item.failure = failure; item.error = failure.message;
|
||||
const incident = { id: crypto.randomUUID(), at: new Date().toISOString(), jobId: item.id, source, title: item.title, failure, state: 'open' };
|
||||
this.state.incidents.push(incident); this.state.incidents = this.state.incidents.slice(-200);
|
||||
if (failure.kind === 'unavailable') { item.status = 'unavailable'; incident.state = 'resolved'; incident.resolution = 'skipped-unavailable'; }
|
||||
else if (failure.retryable && (item.recoveryAttempts || 0) < 1) {
|
||||
item.recoveryAttempts = (item.recoveryAttempts || 0) + 1;
|
||||
item.status = 'retry_wait'; item.nextAttemptAt = new Date(Date.now() + this.retryDelayMs).toISOString();
|
||||
this.state.circuits[source] = { state: 'open', until: item.nextAttemptAt, jobId: item.id, reason: 'transient' };
|
||||
} else if (failure.scope === 'source') { item.status = 'error'; this.state.sourcePaused[source] = true; }
|
||||
else if (failure.scope === 'global') { item.status = 'error'; this.state.paused = true; }
|
||||
else {
|
||||
item.status = 'quarantined';
|
||||
const similar = this.state.incidents.filter(i => i.source === source && i.failure.kind === 'adapter' && Date.now() - Date.parse(i.at) < 3600000);
|
||||
if (new Set(similar.map(i => i.jobId)).size >= 2) this.state.sourcePaused[source] = true;
|
||||
}
|
||||
await this.event(failure.kind === 'unavailable' ? 'warning' : 'error', failure.message, { id: item.id, title: item.title, source, failure, nextAttemptAt: item.nextAttemptAt || null });
|
||||
}
|
||||
}
|
||||
|
||||
async workLoop() {
|
||||
while (!this.stopping) {
|
||||
this.runtimeHeartbeat = Date.now();
|
||||
if (!this.state.paused) {
|
||||
for (const item of this.state.queue) {
|
||||
if (item.status === 'retry_wait' && Date.parse(item.nextAttemptAt) <= Date.now()) { item.status = 'queued'; delete this.state.circuits[item.source]; }
|
||||
if (item.status !== 'queued') continue;
|
||||
const found = this.resolveAdapter(item.url);
|
||||
if (!found) {
|
||||
item.status = 'quarantined'; item.error = 'Adapter no longer available';
|
||||
await this.event('error', item.error, { id: item.id, title: item.title, source: item.source });
|
||||
continue;
|
||||
}
|
||||
const source = found.adapter.id;
|
||||
if (this.active.has(source) || this.state.sourcePaused[source]) continue;
|
||||
if (this.state.circuits[source]?.state === 'open') continue;
|
||||
if (item.libraryMangaId && [...this.active.values()].some(entry => {
|
||||
const activeItem = this.state.queue.find(candidate => candidate.id === entry.id);
|
||||
return activeItem?.libraryMangaId === item.libraryMangaId;
|
||||
})) continue;
|
||||
this.startItem(item, found);
|
||||
}
|
||||
}
|
||||
const prior = this.state.status;
|
||||
this.recomputeStatus();
|
||||
if (prior !== this.state.status) await this.persist();
|
||||
await sleep(500);
|
||||
}
|
||||
}
|
||||
|
||||
async close() {
|
||||
if (!this.state) return;
|
||||
this.stopping = true;
|
||||
if (this.discoveryTimer) clearTimeout(this.discoveryTimer);
|
||||
this.discoveryTimer = null;
|
||||
this.recomputeStatus();
|
||||
await this.persist();
|
||||
const timeout = new Promise(resolve => { const timer = setTimeout(resolve, 30000); timer.unref(); });
|
||||
const workers = [...this.active.values()].map(entry => entry.promise).filter(Boolean);
|
||||
const drained = await Promise.race([Promise.allSettled([this.loopPromise || Promise.resolve(), ...workers]).then(() => true), timeout.then(() => false)]);
|
||||
await this.writeChain;
|
||||
// Keep the lock until the process exits if an in-flight worker did not drain.
|
||||
if (drained) await fs.rm(this.lockFile, { force: true });
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { ScraperService, DEFAULT_SETTINGS, validateSettings, settingsForSource };
|
||||
@@ -0,0 +1,71 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs').promises;
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { ScraperService } = require('../service');
|
||||
|
||||
const adapter = { id: 'alpha', label: 'Alpha',
|
||||
match(value) { return value.startsWith('https://alpha.test/') ? { id: '42', source: 'alpha', title: 'Test', url: value } : null; },
|
||||
async discover() { return { title: 'Test', chapters: [{ id: 'one', number: '1' }] }; },
|
||||
async run() { throw new Error('A no-change check must not download'); },
|
||||
};
|
||||
const resolve = value => { const info = adapter.match(value); return info ? { adapter, info } : null; };
|
||||
const waitFor = async predicate => {
|
||||
const deadline = Date.now() + 3000;
|
||||
while (!predicate()) {
|
||||
if (Date.now() > deadline) throw new Error('Timed out waiting for local discovery');
|
||||
await new Promise(resolveWait => setTimeout(resolveWait, 25));
|
||||
}
|
||||
};
|
||||
|
||||
test('finished-job cleanup preserves unfinished jobs, legacy check summaries, history and library files', async t => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-scraper-dashboard-'));
|
||||
const dataRoot = path.join(root, 'data'), libraryRoot = path.join(root, 'library');
|
||||
await fs.mkdir(dataRoot); await fs.mkdir(path.join(libraryRoot, 'Test'), { recursive: true });
|
||||
const chapterFile = path.join(libraryRoot, 'Test', 'keep.jpg');
|
||||
await fs.writeFile(chapterFile, 'library-bytes');
|
||||
const queue = [
|
||||
{ id: 'done-a', kind: 'download', source: 'alpha', url: 'https://alpha.test/42', status: 'complete' },
|
||||
{ id: 'checked', kind: 'discovery', source: 'alpha', url: 'https://alpha.test/42', status: 'complete', libraryMangaId: 'Test', checked: '2026-10-06T12:00:00Z' },
|
||||
{ id: 'ready', kind: 'inspection', source: 'alpha', url: 'https://alpha.test/42', status: 'ready' },
|
||||
{ id: 'failed', kind: 'download', source: 'alpha', url: 'https://alpha.test/42', status: 'quarantined' },
|
||||
{ id: 'queued', kind: 'download', source: 'alpha', url: 'https://alpha.test/42', status: 'queued' },
|
||||
{ id: 'done-b', kind: 'download', source: 'beta', url: 'https://beta.test/42', status: 'complete' },
|
||||
];
|
||||
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({ paused: true, queue, history: [{ id: 'done-a', status: 'complete' }] }));
|
||||
const service = await new ScraperService({ libraryRoot, dataRoot, adapterFor: resolve, adapters: [adapter, { id: 'beta', label: 'Beta' }] }).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
|
||||
assert.deepEqual(service.state.queue.map(item => item.id), ['done-a', 'ready', 'failed', 'queued', 'done-b']);
|
||||
assert.equal((await service.publicState()).updateChecks[0].lastCheckedAt, '2026-10-06T12:00:00Z');
|
||||
assert.equal((await service.clearCompleted('alpha')).cleared, 1);
|
||||
assert.deepEqual(service.state.queue.map(item => item.id), ['ready', 'failed', 'queued', 'done-b']);
|
||||
assert.equal((await service.clearCompleted()).cleared, 1);
|
||||
assert.equal((await service.clearCompleted()).cleared, 0);
|
||||
assert.deepEqual(service.state.queue.map(item => item.id), ['ready', 'failed', 'queued']);
|
||||
assert.deepEqual(service.state.queue.map(item => item.order), [0, 1, 2]);
|
||||
assert.equal((await fs.readFile(chapterFile, 'utf8')), 'library-bytes');
|
||||
assert.equal(service.state.history[0].id, 'done-a');
|
||||
assert.equal(JSON.parse(await fs.readFile(path.join(dataRoot, 'queue.json'), 'utf8')).updateChecks['["Test","alpha"]'].lastCheckedAt, '2026-10-06T12:00:00Z');
|
||||
});
|
||||
|
||||
test('a nightly no-change check updates the latest timestamp and retires its queue row', async t => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-scraper-check-summary-'));
|
||||
const dataRoot = path.join(root, 'data'), libraryRoot = path.join(root, 'library');
|
||||
const chapter = path.join(libraryRoot, 'Test', 'v01', 'c0001 [Alpha]');
|
||||
await fs.mkdir(dataRoot); await fs.mkdir(chapter, { recursive: true });
|
||||
await fs.writeFile(path.join(libraryRoot, 'Test', 'manga.json'), JSON.stringify({ title: 'Test', status: 'ongoing', sources: [{ name: 'Alpha', url: 'https://alpha.test/title/42' }] }));
|
||||
await fs.writeFile(path.join(chapter, 'chapter.json'), JSON.stringify({ source: { name: 'Alpha', url: 'https://alpha.test/chapter/one' } }));
|
||||
await fs.writeFile(path.join(chapter, '0001.png'), Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
|
||||
const service = await new ScraperService({ libraryRoot, dataRoot, adapterFor: resolve, adapters: [adapter] }).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
|
||||
await service.enqueueDiscoveryChecks('schedule');
|
||||
await waitFor(() => service.state.updateChecks['["Test","alpha"]']?.lastCheckedAt);
|
||||
assert.equal(service.state.queue.length, 0);
|
||||
const summary = (await service.publicState()).updateChecks[0];
|
||||
assert.equal(summary.lastResult, 'no-updates');
|
||||
assert.equal(summary.lastNewContentAt, undefined);
|
||||
assert.equal(summary.lastDownloadedAt, undefined);
|
||||
assert.ok(service.state.history.some(item => item.status === 'checked'));
|
||||
});
|
||||
@@ -0,0 +1,92 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { ScraperService, validateSettings } = require('../service');
|
||||
|
||||
function fixture(scanLibrary) {
|
||||
const service = new ScraperService({ scanLibrary, adapters: [] });
|
||||
service.state = { version: 4, paused: true, status: 'paused', queue: [], settings: validateSettings({}), currentIds: {} };
|
||||
service.recentEvents = async () => [];
|
||||
return service;
|
||||
}
|
||||
const inventory = title => ({ mangas: title ? [{ id: title, title, status: 'ongoing', autoUpdate: true, chapters: [], sources: [] }] : [] });
|
||||
|
||||
test('cold status returns controls immediately and simultaneous probes share one slow scan', { timeout: 1000 }, async () => {
|
||||
let finish, scans = 0;
|
||||
const service = fixture(() => { scans++; return new Promise(resolve => { finish = resolve; }); });
|
||||
const states = await Promise.all(Array.from({ length: 8 }, () => service.publicState()));
|
||||
assert.equal(scans, 1);
|
||||
for (const state of states) {
|
||||
assert.equal(state.paused, true);
|
||||
assert.equal(state.settings.sourceRates.mangadex.minMs, 1000);
|
||||
assert.deepEqual(state.library, []);
|
||||
assert.equal(state.libraryStatus.ready, false);
|
||||
assert.equal(state.libraryStatus.refreshing, true);
|
||||
}
|
||||
finish(inventory('Demo'));
|
||||
await service.inventoryRefresh;
|
||||
const ready = await service.publicState();
|
||||
assert.equal(ready.library[0].title, 'Demo');
|
||||
assert.equal(ready.libraryStatus.ready, true);
|
||||
assert.equal(ready.libraryStatus.stale, false);
|
||||
assert.equal(scans, 1);
|
||||
});
|
||||
|
||||
test('status keeps previous summary but authoritative callers await scans and reject invalidated results', { timeout: 1000 }, async () => {
|
||||
const completions = [];
|
||||
const service = fixture(() => new Promise(resolve => completions.push(resolve)));
|
||||
const first = service.libraryInventory(true);
|
||||
await Promise.resolve();
|
||||
completions[0](inventory('Old'));
|
||||
await first;
|
||||
service.invalidateInventory();
|
||||
const fresh = service.libraryInventory(true);
|
||||
await Promise.resolve();
|
||||
let finished = false;
|
||||
fresh.then(() => { finished = true; });
|
||||
let state = await service.publicState();
|
||||
assert.equal(state.library[0].title, 'Old');
|
||||
assert.equal(state.libraryStatus.stale, true);
|
||||
assert.equal(state.libraryStatus.refreshing, true);
|
||||
assert.equal(finished, false);
|
||||
// Publication during an in-flight scan requires another authoritative scan.
|
||||
service.invalidateInventory();
|
||||
const scan = service.inventoryRefresh;
|
||||
completions[1](inventory('Obsolete'));
|
||||
await scan;
|
||||
await Promise.resolve();
|
||||
assert.equal(completions.length, 3);
|
||||
assert.equal(finished, false);
|
||||
completions[2](inventory('Published'));
|
||||
assert.equal((await fresh).mangas[0].title, 'Published');
|
||||
state = await service.publicState();
|
||||
assert.equal(state.library[0].title, 'Published');
|
||||
assert.equal(state.libraryStatus.stale, false);
|
||||
});
|
||||
|
||||
test('background inventory failures are visible without breaking controls or launching a poll storm', { timeout: 1000 }, async () => {
|
||||
let scans = 0;
|
||||
const service = fixture(async () => { scans++; throw new Error('Scan failed https://test.example/a?token=private'); });
|
||||
await service.publicState();
|
||||
await assert.rejects(service.inventoryRefresh, /Scan failed/);
|
||||
for (let n = 0; n < 3; n++) {
|
||||
const state = await service.publicState();
|
||||
assert.equal(state.paused, true);
|
||||
assert.match(state.libraryStatus.error, /Scan failed/);
|
||||
assert.ok(!state.libraryStatus.error.includes('private'));
|
||||
}
|
||||
assert.equal(scans, 1);
|
||||
await assert.rejects(service.libraryInventory(true), /Scan failed/);
|
||||
assert.equal(scans, 2);
|
||||
});
|
||||
|
||||
test('dashboard polling reuses a recent snapshot while decisions still force a fresh scan', async () => {
|
||||
let scans = 0;
|
||||
const service = fixture(async () => { scans++; return inventory('Demo'); });
|
||||
await service.libraryInventory(true);
|
||||
service.inventoryCachedAt = Date.now() - 30000;
|
||||
for (let n = 0; n < 5; n++) assert.equal((await service.publicState()).library[0].title, 'Demo');
|
||||
assert.equal(scans, 1);
|
||||
await service.libraryInventory(true);
|
||||
assert.equal(scans, 2);
|
||||
});
|
||||
@@ -0,0 +1,41 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs/promises');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { writeJsonAtomic } = require('../lib/json-file');
|
||||
|
||||
test('Windows rename fallback replaces JSON without deleting the old state first', async t => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-json-'));
|
||||
t.after(() => fs.rm(root, { recursive: true, force: true }));
|
||||
const file = path.join(root, 'queue.json');
|
||||
await fs.writeFile(file, '{"old":true}\n');
|
||||
let failedOnce = false;
|
||||
const io = { ...fs, async rename(from, to) {
|
||||
if (to === file && !failedOnce) {
|
||||
failedOnce = true;
|
||||
const error = new Error('destination busy'); error.code = 'EEXIST'; throw error;
|
||||
}
|
||||
return fs.rename(from, to);
|
||||
} };
|
||||
await writeJsonAtomic(file, { next: true }, io);
|
||||
assert.deepEqual(JSON.parse(await fs.readFile(file, 'utf8')), { next: true });
|
||||
assert.deepEqual(await fs.readdir(root), ['queue.json']);
|
||||
});
|
||||
|
||||
test('failed replacement restores the old JSON', async t => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-json-'));
|
||||
t.after(() => fs.rm(root, { recursive: true, force: true }));
|
||||
const file = path.join(root, 'queue.json');
|
||||
await fs.writeFile(file, '{"old":true}\n');
|
||||
const io = { ...fs, async rename(from, to) {
|
||||
if (to === file && from.endsWith('.tmp')) {
|
||||
const error = new Error('replacement blocked'); error.code = 'EEXIST'; throw error;
|
||||
}
|
||||
return fs.rename(from, to);
|
||||
} };
|
||||
await assert.rejects(writeJsonAtomic(file, { next: true }, io), /replacement blocked/);
|
||||
assert.deepEqual(JSON.parse(await fs.readFile(file, 'utf8')), { old: true });
|
||||
assert.deepEqual(await fs.readdir(root), ['queue.json']);
|
||||
});
|
||||
@@ -0,0 +1,135 @@
|
||||
'use strict';
|
||||
const test = require('node:test'), assert = require('node:assert/strict');
|
||||
const fs = require('node:fs/promises'), os = require('node:os'), path = require('node:path');
|
||||
const { specialKey, parseSpecialFolder } = require('../../chapter-identity');
|
||||
const { selectChapterReleases, availabilitySummary, discoverMangaDexCatalog, discoverMangaDexChapter } = require('../adapters/mangadex');
|
||||
const { selectByScope } = require('../lib/scope');
|
||||
const inventory = require('../lib/library-inventory');
|
||||
const { publishManifest } = require('../lib/publisher');
|
||||
const reader = require('../../library');
|
||||
const { ScraperService } = require('../service');
|
||||
const ids = ['37d6949a-c7ce-4855-b838-4fd572e87629', '27dd738c-eca2-488d-a791-53ab4b24313b'];
|
||||
const titleId = '06a37094-b3ba-4f25-bf2b-54c9751e0e91';
|
||||
const release = (id, chapter = null) => ({ id, attributes: { chapter, pages: 46, title: 'Pilot', externalUrl: null }, relationships: [] });
|
||||
const pause = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
async function until(predicate) {
|
||||
const end = Date.now() + 5000;
|
||||
while (!predicate()) { if (Date.now() > end) throw new Error('Fixture timed out'); await pause(20); }
|
||||
}
|
||||
|
||||
test('unnumbered releases retain source IDs, do not invent chapter zero, and remain distinct', async () => {
|
||||
const selected = selectChapterReleases(ids.map(id => release(id)), 'Pilot');
|
||||
assert.equal(selected.length, 2);
|
||||
assert.deepEqual(new Set(selected.map(r => r.number)), new Set(ids.map(specialKey)));
|
||||
assert.ok(selected.every(r => r.special && r.label === 'Pilot' && r.sourceNumber === null));
|
||||
assert.equal(selectByScope(selected, { mode: 'chapter', chapter: '0' }).length, 0);
|
||||
assert.equal(selectByScope(selected, { mode: 'after', chapter: '1' }).length, 0);
|
||||
assert.equal(selectByScope(selected, { mode: 'missing' }).length, 2);
|
||||
assert.equal(selectByScope(selected, { mode: 'ids', ids: [ids[0]] })[0].id, ids[0]);
|
||||
assert.throws(() => specialKey('../escape'));
|
||||
assert.equal(parseSpecialFolder(specialKey(ids[0]) + ' Pilot [MangaDex]').src, 'MangaDex');
|
||||
const chapter = await discoverMangaDexChapter(selected[0], { async get() { return { async json() { return { result: 'ok', baseUrl: 'https://uploads.mangadex.org', chapter: { hash: 'fixture', data: Array.from({ length: 46 }, (_, n) => `${n}.png`) } }; } }; } });
|
||||
assert.equal(chapter.label, 'Pilot'); assert.equal(chapter.sourceId, selected[0].id);
|
||||
});
|
||||
|
||||
test('availability distinguishes no English, external-only, and partial hosted catalogs', () => {
|
||||
assert.equal(availabilitySummary({ availableTranslatedLanguages: ['ar'] }, [], [], 0).reason, 'no-english');
|
||||
const external = availabilitySummary({}, [], [], 8);
|
||||
assert.equal(external.reason, 'external-only'); assert.match(external.message, /8 English external-link/);
|
||||
const chapters = selectChapterReleases([release('regular-fixture', '336')]);
|
||||
const summary = availabilitySummary({ lastChapter: '686' }, [release('regular-fixture', '336')], chapters);
|
||||
assert.equal(summary.coverage, 'partial'); assert.equal(summary.selectedChapters, 1);
|
||||
assert.equal(availabilitySummary({ lastChapter: '336' }, [], chapters).coverage, 'unknown', 'Highest number alone does not prove completeness');
|
||||
});
|
||||
|
||||
test('catalog uses separate external-only feed only when hosted selections are empty', async () => {
|
||||
const modes = [];
|
||||
const http = { async get(url) {
|
||||
const parsed = new URL(url), mode = parsed.searchParams.get('includeExternalUrl');
|
||||
if (!mode) return { async json() { return { data: { type: 'manga', attributes: { title: { en: 'CLAYMORE' }, lastChapter: '155' } } }; } };
|
||||
modes.push(mode);
|
||||
return { async json() { return { result: 'ok', data: [], total: mode === '1' ? 8 : 0 }; } };
|
||||
} };
|
||||
const catalog = await discoverMangaDexCatalog(`https://mangadex.org/title/${titleId}`, http);
|
||||
assert.deepEqual(modes, ['0', '1']); assert.equal(catalog.availability.reason, 'external-only');
|
||||
modes.length = 0;
|
||||
http.get = async url => {
|
||||
const mode = new URL(url).searchParams.get('includeExternalUrl');
|
||||
if (!mode) return { async json() { return { data: { type: 'manga', attributes: { title: { en: 'Bleach (Pre-Serialization)' } } } }; } };
|
||||
modes.push(mode); return { async json() { return { result: 'ok', data: ids.map(id => release(id)), total: 2 }; } };
|
||||
};
|
||||
const pilot = await discoverMangaDexCatalog(`https://mangadex.org/title/${titleId}`, http);
|
||||
assert.deepEqual(modes, ['0']); assert.equal(pilot.availability.specialReleases, 2); assert.equal(pilot.chapters[0].label, 'Pilot');
|
||||
});
|
||||
|
||||
test('special publication is atomic, readable, stable across rescans and duplicate-safe', async t => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-special-'));
|
||||
t.after(() => fs.rm(root, { recursive: true, force: true }));
|
||||
// Real fixture PNG; no upstream image requests.
|
||||
const { Canvas } = require('../../tools/png');
|
||||
const page = path.join(root, '.fixture.png'); await fs.writeFile(page, new Canvas(10, 20, [1, 2, 3]).png());
|
||||
const manifest = { version: 1, series: { title: 'Pilot edition', direction: 'rtl', sources: [{ name: 'MangaDex', url: `https://mangadex.org/title/${titleId}` }] },
|
||||
chapters: selectChapterReleases(ids.map(id => release(id)), 'Pilot').map(ref => ({ ...ref, source: { name: 'MangaDex', url: ref.url }, pages: [{ path: page }] })) };
|
||||
const first = await publishManifest(manifest, { libraryRoot: root });
|
||||
assert.ok(first.results.every(result => result.status === 'published'));
|
||||
const scanned = (await reader.scanLibrary(root)).index.mangas[0];
|
||||
assert.equal(scanned.chapters.length, 2); assert.ok(scanned.chapters.every(c => c.label === 'Pilot' && c.num === null && !c.extra));
|
||||
const stableIds = scanned.chapters.map(c => c.id);
|
||||
assert.deepEqual((await publishManifest(manifest, { libraryRoot: root })).results.map(r => r.status), ['skipped', 'skipped']);
|
||||
assert.deepEqual((await reader.scanLibrary(root)).index.mangas[0].chapters.map(c => c.id), stableIds);
|
||||
const manga = (await inventory.scanLibrary(root)).mangas[0];
|
||||
assert.equal(inventory.shouldDownload(manga, manifest.chapters[0], 'mangadex'), false);
|
||||
assert.equal(inventory.isNewerThanLibrary(manga, manifest.chapters[0]), false);
|
||||
assert.equal(inventory.isNewerThanLibrary(manga, { number: '1' }), true);
|
||||
assert.equal(reader.parseKey('c0012.5 Existing [Host]').key, '12.5');
|
||||
assert.equal(reader.parseKey('v01').key, '1');
|
||||
const regular = { ...manifest, chapters: [{ number: '12.5', title: 'Existing chapter', source: { name: 'Host', url: 'https://host.test/12' }, pages: [{ path: page }] }] };
|
||||
await publishManifest(regular, { libraryRoot: root });
|
||||
assert.ok((await reader.scanLibrary(root)).index.mangas[0].chapterById.has('1/12.5'));
|
||||
});
|
||||
|
||||
test('legacy unavailable messages become unverified without retrying or downloading', async t => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-unavailable-')), dataRoot = path.join(root, 'data');
|
||||
await fs.mkdir(dataRoot); await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({ paused: true,
|
||||
queue: [{ id: 'old-job', source: 'mangadex', url: `https://mangadex.org/title/${titleId}`, status: 'unavailable', error: 'No downloadable English MangaDex chapters found', failure: { kind: 'unavailable', scope: 'job' } }],
|
||||
settings: { autoUpdateEnabled: false } }));
|
||||
const service = await new ScraperService({ libraryRoot: path.join(root, 'library'), dataRoot }).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
|
||||
assert.equal(service.state.queue[0].status, 'unavailable'); assert.equal(service.active.size, 0);
|
||||
assert.match(service.state.queue[0].error, /Inspect this source again/);
|
||||
assert.equal(service.state.queue[0].failure.kind, 'unavailable');
|
||||
});
|
||||
|
||||
test('inspection waits for confirmation, survives restart, and uses one normal source lane', async t => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-inspect-')), dataRoot = path.join(root, 'data');
|
||||
await fs.mkdir(dataRoot); await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({ paused: true, queue: [], settings: { autoUpdateEnabled: false } }));
|
||||
let running = 0, maxRunning = 0, downloads = 0;
|
||||
const catalog = { title: 'Canonical Pilot', chapters: ids.map(id => ({ id, number: specialKey(id), special: true })), availability: { selectedChapters: 2, specialReleases: 2, coverage: 'unknown' } };
|
||||
const adapter = { id: 'fixture', label: 'Fixture', match: url => ({ source: 'fixture', id: url.split('/').pop(), url, title: 'Misleading slug' }),
|
||||
async discover() { maxRunning = Math.max(maxRunning, ++running); await pause(50); running--; return catalog; },
|
||||
async run(item, options) { downloads++; await options.onCatalog(catalog); return { publishedCount: 1, selectedCount: 1 }; } };
|
||||
const options = { libraryRoot: path.join(root, 'library'), dataRoot, adapters: [adapter], adapterFor: url => ({ adapter, info: adapter.match(url) }) };
|
||||
let service = await new ScraperService(options).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(root, { recursive: true, force: true }); });
|
||||
const item = await service.enqueue({ url: 'https://fixture.test/one', inspect: true });
|
||||
const other = await service.enqueue({ url: 'https://fixture.test/two', inspect: true });
|
||||
await assert.rejects(service.control('confirm', item.id), /Inspect the source first/);
|
||||
await service.control('resume'); await until(() => item.status === 'ready' && other.status === 'ready');
|
||||
assert.equal(downloads, 0); assert.equal(maxRunning, 1); assert.equal(item.sourceTitle, 'Canonical Pilot');
|
||||
await service.close(); service = await new ScraperService(options).init();
|
||||
const restored = service.state.queue.find(q => q.id === item.id);
|
||||
assert.equal(restored.kind, 'inspection'); assert.equal(restored.status, 'ready');
|
||||
await service.control('confirm', restored.id); await until(() => restored.status === 'complete');
|
||||
await until(() => service.active.size === 0);
|
||||
assert.equal(downloads, 1); assert.equal(restored.selectedCount, 1);
|
||||
assert.equal(service.state.history.at(-1).availability.specialReleases, 2);
|
||||
service.state.sourcePaused.fixture = true;
|
||||
await assert.rejects(service.control('inspect', restored.id), /access block/);
|
||||
});
|
||||
|
||||
test('edition change after confirmation cannot silently download another edition', async () => {
|
||||
const service = Object.create(ScraperService.prototype);
|
||||
service.persist = async () => {};
|
||||
const item = { title: 'slug', confirmedSourceTitle: 'Pilot edition' };
|
||||
await assert.rejects(service.recordCatalog(item, { title: 'Main edition', chapters: [] }), /edition title changed/);
|
||||
});
|
||||
@@ -0,0 +1,111 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs').promises;
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { detectVerticalStrip, resolveReadingDirection, fallbackDirection } = require('../../reading-direction');
|
||||
const { publishManifest } = require('../lib/publisher');
|
||||
const { scanLibrary } = require('../../library');
|
||||
|
||||
const strip = (w = 800) => ({ w, h: 12000 });
|
||||
|
||||
test('strip detection tolerates a landscape title card and rejects weak evidence', () => {
|
||||
const pages = [{ w: 1200, h: 800 }, strip(), strip(), strip()];
|
||||
assert.equal(detectVerticalStrip(pages).value, 'ttb');
|
||||
assert.equal(detectVerticalStrip(pages).evidence.tallPages, 3);
|
||||
assert.equal(detectVerticalStrip([strip()]), null);
|
||||
assert.equal(detectVerticalStrip([strip(), strip(), { w: 800, h: 1200 }, { w: 800, h: 1200 }]), null);
|
||||
assert.equal(detectVerticalStrip([strip(), strip(), strip(1400), { w: 1200, h: 800 }]), null);
|
||||
assert.equal(detectVerticalStrip([{ w: 800, h: 12000, r: 90 }, strip(), strip()]), null);
|
||||
assert.equal(detectVerticalStrip([{ w: 12000, h: 800, r: 90 }, strip(), strip()]).value, 'ttb');
|
||||
});
|
||||
|
||||
test('manual choices at chapter, book or manga level beat detection and fallback', () => {
|
||||
const pages = [strip(), strip(), strip()];
|
||||
const automatic = fallbackDirection();
|
||||
assert.equal(resolveReadingDirection([automatic], pages).value, 'ttb');
|
||||
assert.equal(resolveReadingDirection([automatic]).source, 'fallback');
|
||||
assert.equal(resolveReadingDirection([{ direction: 'ltr' }, automatic], pages).value, 'ltr');
|
||||
assert.equal(resolveReadingDirection([{}, { direction: 'rtl' }, automatic], pages).value, 'rtl');
|
||||
assert.equal(resolveReadingDirection([{ directionOverride: 'rtl', ...automatic }], pages).value, 'rtl');
|
||||
assert.equal(resolveReadingDirection([{ ...automatic, direction: 'ltr' }], pages).value, 'ltr');
|
||||
assert.equal(resolveReadingDirection([{ direction: 'rtl' }], pages).source, 'manual');
|
||||
assert.equal(resolveReadingDirection([], pages).value, 'ttb');
|
||||
});
|
||||
|
||||
function png(w, h) {
|
||||
const b = Buffer.alloc(24);
|
||||
Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]).copy(b);
|
||||
b.writeUInt32BE(w, 16); b.writeUInt32BE(h, 20);
|
||||
return b;
|
||||
}
|
||||
|
||||
async function json(file) { return JSON.parse(await fs.readFile(file, 'utf8')); }
|
||||
|
||||
test('publisher detects new strips, preserves updates and manual metadata, and scanner agrees', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-direction-'));
|
||||
t.after(() => fs.rm(temp, { recursive: true, force: true }));
|
||||
const libraryRoot = path.join(temp, 'library');
|
||||
const sizes = [[1200, 800], [800, 12000], [800, 12500], [800, 14210]];
|
||||
const files = [];
|
||||
for (const [i, size] of sizes.entries()) {
|
||||
const file = path.join(temp, `page${i}.png`);
|
||||
await fs.writeFile(file, png(...size)); files.push(file);
|
||||
}
|
||||
const series = { title: 'Strip Work', ...fallbackDirection(),
|
||||
sources: [{ name: 'Test', url: 'https://example.test/series' }] };
|
||||
const chapter = number => ({ number, source: { name: 'Test', url: `https://example.test/${number}` },
|
||||
book: { number: 1, title: 'Season 1' }, pages: files.map(file => ({ path: file })) });
|
||||
const first = await publishManifest({ version: 1, series, chapters: [chapter('1')] }, { libraryRoot });
|
||||
const mangaDir = first.mangaDir, bookDir = path.join(mangaDir, 'v01 Season 1');
|
||||
const mangaFile = path.join(mangaDir, 'manga.json');
|
||||
assert.equal((await json(mangaFile)).direction, 'ttb');
|
||||
assert.equal((await json(mangaFile))['x-reading-direction'].source, 'detected');
|
||||
const scan = (await scanLibrary(libraryRoot)).index.mangas[0];
|
||||
assert.equal(scan.chapters[0].versions[0].direction, 'ttb');
|
||||
assert.equal(scan.chapters[0].versions[0].pages[0].w, 1200);
|
||||
|
||||
const same = await publishManifest({ version: 1, series, chapters: [chapter('2')] }, { libraryRoot });
|
||||
assert.equal(same.mangaDir, mangaDir);
|
||||
assert.equal((await json(mangaFile)).direction, 'ttb');
|
||||
|
||||
const bookFile = path.join(bookDir, 'book.json');
|
||||
await fs.writeFile(bookFile, JSON.stringify({ title: 'Season 1', direction: 'ltr', note: 'human' }));
|
||||
const chapterDir = path.join(bookDir, first.results[0].folder);
|
||||
const chapterFile = path.join(chapterDir, 'chapter.json');
|
||||
await fs.writeFile(chapterFile, JSON.stringify({ ...await json(chapterFile), directionOverride: 'rtl', note: 'human' }));
|
||||
await publishManifest({ version: 1, series, chapters: [chapter('1')] }, { libraryRoot, replace: true });
|
||||
assert.equal((await json(bookFile)).direction, 'ltr');
|
||||
assert.equal((await json(bookFile)).note, 'human');
|
||||
assert.equal((await json(chapterFile)).direction, 'rtl');
|
||||
assert.equal((await json(chapterFile)).directionOverride, 'rtl');
|
||||
assert.equal((await json(chapterFile)).note, 'human');
|
||||
const rescanned = (await scanLibrary(libraryRoot)).index.mangas[0];
|
||||
assert.equal(rescanned.chapters[0].versions[0].direction, 'rtl');
|
||||
assert.equal(rescanned.chapters[1].versions[0].direction, 'ltr');
|
||||
|
||||
await fs.writeFile(mangaFile, JSON.stringify({ ...await json(mangaFile), direction: 'rtl' }));
|
||||
await publishManifest({ version: 1, series, chapters: [chapter('3')] }, { libraryRoot });
|
||||
assert.equal((await json(mangaFile)).direction, 'rtl');
|
||||
assert.equal((await json(mangaFile))['x-reading-direction'].source, 'manual');
|
||||
assert.equal((await scanLibrary(libraryRoot)).index.mangas[0].chapters[2].versions[0].direction, 'ltr');
|
||||
});
|
||||
|
||||
test('legacy plain RTL stays protected across automatic imports', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-legacy-direction-'));
|
||||
t.after(() => fs.rm(temp, { recursive: true, force: true }));
|
||||
const libraryRoot = path.join(temp, 'library');
|
||||
const page = path.join(temp, 'tall.png'); await fs.writeFile(page, png(800, 12000));
|
||||
const series = { title: 'Legacy', ...fallbackDirection(), sources: [{ name: 'Test', url: 'https://example.test/legacy' }] };
|
||||
const chapter = { number: '1', source: { name: 'Test', url: 'https://example.test/legacy/1' },
|
||||
pages: [1, 2, 3].map(() => ({ path: page })) };
|
||||
const first = await publishManifest({ version: 1, series, chapters: [chapter] }, { libraryRoot });
|
||||
const file = path.join(first.mangaDir, 'manga.json');
|
||||
const meta = await json(file); delete meta['x-reading-direction']; meta.direction = 'rtl';
|
||||
await fs.writeFile(file, JSON.stringify(meta));
|
||||
await publishManifest({ version: 1, series, chapters: [{ ...chapter, number: '2', source: { name: 'Test', url: 'https://example.test/legacy/2' } }] }, { libraryRoot });
|
||||
assert.equal((await json(file)).direction, 'rtl');
|
||||
assert.equal((await json(file))['x-reading-direction'].source, 'manual');
|
||||
});
|
||||
@@ -0,0 +1,48 @@
|
||||
'use strict';
|
||||
const test = require('node:test'), assert = require('node:assert/strict'), fs = require('fs').promises, os = require('os'), path = require('path');
|
||||
const { ScraperService, validateSettings } = require('../service');
|
||||
const { classifyFailure, safeMessage } = require('../lib/failures');
|
||||
test('failure classification protects access challenges and distinguishes job failures', () => {
|
||||
for (const status of [401, 403, 429]) assert.equal(classifyFailure(Object.assign(new Error('Host response'), { status })).kind, 'access');
|
||||
assert.equal(classifyFailure(new Error('No downloadable English MangaDex chapters found')).scope, 'job');
|
||||
assert.equal(classifyFailure(Object.assign(new Error('Disk full'), { code: 'ENOSPC' })).scope, 'global');
|
||||
assert.equal(classifyFailure(new Error('HTTP 503')).retryable, true);
|
||||
assert.equal(safeMessage('HTTP 403 https://test.example/a?token=secret'), 'HTTP 403 https://test.example/a');
|
||||
});
|
||||
async function fixture(t, errors, options = {}) {
|
||||
const dir = await fs.mkdtemp(path.join(os.tmpdir(), 'mc-recovery-'));
|
||||
const dataRoot = path.join(dir, 'data'); await fs.mkdir(dataRoot);
|
||||
const counts = {};
|
||||
const adapter = { id: 'test', label: 'Test', run: async item => {
|
||||
counts[item.id] = (counts[item.id] || 0) + 1;
|
||||
if (errors[item.id]) throw errors[item.id]; return { publishedCount: 0 };
|
||||
} };
|
||||
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({ version: 3, paused: true, settings: validateSettings({ autoUpdateEnabled: false }),
|
||||
queue: Object.keys(errors).concat('healthy').map(id => ({ id, seriesId: id, source: 'test', title: id, url: 'https://test.example/' + id, status: 'queued' })), history: [] }));
|
||||
const service = await new ScraperService({ libraryRoot: path.join(dir, 'library'), dataRoot, adapterFor: url => ({ adapter, info: { id: url.split('/').pop() } }), adapters: [adapter], ...options }).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(dir, { recursive: true, force: true }); });
|
||||
await service.control('resume');
|
||||
return { service, counts };
|
||||
}
|
||||
async function wait(predicate) {
|
||||
const end = Date.now() + 6000;
|
||||
while (!predicate()) { if (Date.now() > end) throw new Error('Timeout'); await new Promise(r => setTimeout(r, 25)); }
|
||||
}
|
||||
test('unavailable titles and one parser failure do not stop healthy jobs', async t => {
|
||||
const { service } = await fixture(t, { absent: new Error('No downloadable English MangaDex chapters found'), parser: new Error('Catalog markup changed') });
|
||||
await wait(() => service.state.queue.find(i => i.id === 'healthy').status === 'complete');
|
||||
assert.equal(service.state.queue.find(i => i.id === 'absent').status, 'unavailable');
|
||||
assert.equal(service.state.queue.find(i => i.id === 'parser').status, 'quarantined');
|
||||
assert.equal(service.state.sourcePaused.test, undefined);
|
||||
});
|
||||
test('transient recovery is bounded and access errors are never retried automatically', async t => {
|
||||
const { service, counts } = await fixture(t, { broken: new Error('HTTP 503') }, { retryDelayMs: 10 });
|
||||
await wait(() => service.state.sourcePaused.test);
|
||||
assert.equal(counts.broken, 2); assert.equal(service.state.queue[0].status, 'error');
|
||||
});
|
||||
test('protection failures pause only the affected lane and keep checkpoints', async t => {
|
||||
const { service, counts } = await fixture(t, { blocked: new Error('HTTP 403') });
|
||||
await wait(() => service.state.sourcePaused.test);
|
||||
assert.equal(counts.blocked, 1); assert.equal(counts.healthy, undefined);
|
||||
assert.equal(service.state.incidents[0].failure.kind, 'access');
|
||||
});
|
||||
@@ -0,0 +1,556 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs/promises');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { cleanPart, chapterKey, chapterFolder, assertInside } = require('../lib/names');
|
||||
const { imageExtension } = require('../lib/images');
|
||||
const { publishManifest } = require('../lib/publisher');
|
||||
const { mergeImportedMetadata } = require('../lib/metadata-merge');
|
||||
const { metadataFromMedia } = require('../metadata/anilist');
|
||||
const { normalizeScope, selectByScope, scopeLabel } = require('../lib/scope');
|
||||
const { scanLibrary, findManga, shouldDownload, isNewerThanLibrary, setAutoUpdate } = require('../lib/library-inventory');
|
||||
const { inspectLegacyWork } = require('../lib/legacy-publication');
|
||||
const { discoverWeebCatalog, parseChapters, parsePageUrls } = require('../adapters/weebcentral');
|
||||
const { chapterNumber, selectChapterReleases, pageUrls } = require('../adapters/mangadex');
|
||||
const { adapterFor } = require('../adapters');
|
||||
const { ScraperService, validateSettings, settingsForSource } = require('../service');
|
||||
|
||||
test('MangaCircus chapter keys are normalized', () => {
|
||||
assert.equal(chapterKey('12'), 'c0012');
|
||||
assert.equal(chapterKey('12.50'), 'c0012.5');
|
||||
assert.equal(chapterKey('x2'), 'x02');
|
||||
assert.throws(() => chapterKey('special'));
|
||||
});
|
||||
|
||||
test('folder names are portable and versioned', () => {
|
||||
assert.equal(cleanPart('Bad:<Title>. '), 'BadTitle');
|
||||
assert.equal(cleanPart('CON'), 'CON-manga');
|
||||
assert.equal(chapterFolder({ number: '7', title: 'A [Test]', source: { name: 'Host' } }), 'c0007 A Test [Host]');
|
||||
assert.throws(() => assertInside('C:/library', 'C:/outside'));
|
||||
});
|
||||
|
||||
test('image signatures are recognized', () => {
|
||||
assert.equal(imageExtension(Buffer.from([0xff, 0xd8, 0xff, 0x00])), 'jpg');
|
||||
assert.equal(imageExtension(Buffer.from([137, 80, 78, 71, 13, 10, 26, 10])), 'png');
|
||||
assert.equal(imageExtension(Buffer.from('not an image')), null);
|
||||
});
|
||||
|
||||
test('per-source rate settings default fast and update independently', () => {
|
||||
const defaults = validateSettings({});
|
||||
const fast = { minMs: 1000, maxMs: 4000, chapterMinMs: 5000, chapterMaxMs: 15000 };
|
||||
assert.deepEqual(Object.fromEntries(Object.entries(settingsForSource(defaults, 'weebcentral')).filter(([key]) => key.endsWith('Ms'))), fast);
|
||||
assert.deepEqual(Object.fromEntries(Object.entries(settingsForSource(defaults, 'mangadex')).filter(([key]) => key.endsWith('Ms'))), fast);
|
||||
|
||||
const changed = validateSettings({ sourceRates: { mangadex: {
|
||||
minMs: 1500, maxMs: 3500, chapterMinMs: 6000, chapterMaxMs: 12000,
|
||||
} } }, defaults);
|
||||
assert.deepEqual(Object.fromEntries(Object.entries(settingsForSource(changed, 'mangadex')).filter(([key]) => key.endsWith('Ms'))), {
|
||||
minMs: 1500, maxMs: 3500, chapterMinMs: 6000, chapterMaxMs: 12000,
|
||||
});
|
||||
assert.deepEqual(Object.fromEntries(Object.entries(settingsForSource(changed, 'weebcentral')).filter(([key]) => key.endsWith('Ms'))), fast);
|
||||
assert.throws(() => validateSettings({ sourceRates: { mangadex: { minMs: 5000, maxMs: 1000 } } }, defaults));
|
||||
});
|
||||
|
||||
test('AniList metadata keeps manga format and original story credits', () => {
|
||||
const metadata = metadataFromMedia({
|
||||
id: 150193, idMal: null, format: 'MANGA',
|
||||
title: { english: 'Heavy Knight', romaji: 'Juukishi', native: '重騎士' }, synonyms: [],
|
||||
staff: { edges: [
|
||||
{ role: 'Original Story', node: { name: { full: 'Nekoko' } } },
|
||||
{ role: 'Story & Art', node: { name: { full: 'Lee Brocco' } } },
|
||||
{ role: 'Character Design', node: { name: { full: 'Jaian' } } },
|
||||
] },
|
||||
status: 'RELEASING', startDate: { year: 2022 }, genres: ['Fantasy'], description: '',
|
||||
siteUrl: 'https://anilist.co/manga/150193', coverImage: { extraLarge: 'https://s4.anilist.co/cover.jpg' },
|
||||
});
|
||||
assert.deepEqual(metadata.authors, ['Nekoko', 'Lee Brocco']);
|
||||
assert.equal(metadata['x-anilist'].format, 'MANGA');
|
||||
});
|
||||
|
||||
test('metadata refresh preserves legacy and manual fields while updating untouched imports', () => {
|
||||
const first = mergeImportedMetadata({ title: 'My preferred title', status: 'ongoing' }, {
|
||||
title: 'Provider title', status: 'completed', tags: ['Fantasy'], 'x-anilist': { id: 42 },
|
||||
}, 'Fallback');
|
||||
assert.equal(first.title, 'My preferred title');
|
||||
assert.equal(first.status, 'ongoing');
|
||||
assert.deepEqual(first.tags, ['Fantasy']);
|
||||
assert.equal(first['x-metadata'].imported.title, 'Provider title');
|
||||
const second = mergeImportedMetadata({ ...first, tags: ['Fantasy', 'Favorite'] }, {
|
||||
title: 'New provider title', status: 'ongoing', tags: ['Action'], 'x-anilist': { id: 42 },
|
||||
}, 'Fallback');
|
||||
assert.equal(second.title, 'My preferred title');
|
||||
assert.equal(second.status, 'ongoing');
|
||||
assert.deepEqual(second.tags, ['Fantasy', 'Favorite']);
|
||||
const third = mergeImportedMetadata(second, { description: 'First', 'x-anilist': { id: 42 } }, 'Fallback');
|
||||
assert.equal(mergeImportedMetadata(third, { description: 'Revised', 'x-anilist': { id: 42 } }, 'Fallback').description, 'Revised');
|
||||
assert.throws(() => mergeImportedMetadata(first, { 'x-anilist': { id: 99 } }, 'Fallback'), /identity conflict/);
|
||||
});
|
||||
|
||||
test('publisher rejects a conflicting work ID before fetching any pages', async t => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-identity-'));
|
||||
t.after(() => fs.rm(root, { recursive: true, force: true }));
|
||||
const manga = path.join(root, 'Existing');
|
||||
await fs.mkdir(manga);
|
||||
await fs.writeFile(path.join(manga, 'manga.json'), JSON.stringify({ title: 'Existing',
|
||||
sources: [{ name: 'Host', url: 'https://example.test/series' }], 'x-anilist': { id: 42 } }));
|
||||
let requested = 0;
|
||||
await assert.rejects(publishManifest({ version: 1,
|
||||
series: { title: 'Existing', direction: 'rtl', metadata: { 'x-anilist': { id: 99 } },
|
||||
sources: [{ name: 'Host', url: 'https://example.test/series' }] },
|
||||
chapters: [{ number: '1', source: { name: 'Host', url: 'https://example.test/chapter' },
|
||||
pages: [{ url: 'https://example.test/page.png' }] }],
|
||||
}, { libraryRoot: root, httpClient: { async get() { requested++; throw new Error('unexpected request'); } } }), /identity conflict/);
|
||||
assert.equal(requested, 0);
|
||||
assert.equal((await fs.readdir(manga)).includes('v01'), false);
|
||||
});
|
||||
|
||||
test('queue scopes select exact catalog slices', () => {
|
||||
const chapters = [
|
||||
{ id: 'a', number: '1', volume: '1' }, { id: 'b', number: '2', volume: '1' },
|
||||
{ id: 'c', number: '3', volume: '2' }, { id: 'd', number: '4', volume: '2' },
|
||||
];
|
||||
assert.deepEqual(selectByScope(chapters, { mode: 'latest' }).map(x => x.id), ['d']);
|
||||
assert.deepEqual(selectByScope(chapters, { mode: 'last', count: 2 }).map(x => x.id), ['c', 'd']);
|
||||
assert.deepEqual(selectByScope(chapters, { mode: 'first', count: 2 }).map(x => x.id), ['a', 'b']);
|
||||
assert.deepEqual(selectByScope(chapters, { mode: 'chapter', chapter: '003.0' }).map(x => x.id), ['c']);
|
||||
assert.deepEqual(selectByScope(chapters, { mode: 'after', chapter: '2' }).map(x => x.id), ['c', 'd']);
|
||||
assert.deepEqual(selectByScope(chapters, { mode: 'volume', volume: '2' }).map(x => x.id), ['c', 'd']);
|
||||
assert.deepEqual(selectByScope(chapters, { mode: 'ids', ids: ['b', 'd'] }).map(x => x.id), ['b', 'd']);
|
||||
assert.equal(scopeLabel(normalizeScope({ mode: 'last', count: 5 })), 'last 5 chapters');
|
||||
});
|
||||
|
||||
test('library inventory distinguishes missing chapters from alternate source versions', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-inventory-'));
|
||||
t.after(() => fs.rm(temp, { recursive: true, force: true }));
|
||||
const manga = path.join(temp, 'Library Manga'), book = path.join(manga, 'v01');
|
||||
await fs.mkdir(path.join(book, 'c0001 [WeebCentral]'), { recursive: true });
|
||||
await fs.mkdir(path.join(book, 'c0002 [MangaDex]'), { recursive: true });
|
||||
await fs.writeFile(path.join(manga, 'manga.json'), JSON.stringify({
|
||||
title: 'Library Manga', status: 'ongoing',
|
||||
sources: [{ name: 'WeebCentral', url: 'https://weebcentral.com/series/SERIES/Test' }],
|
||||
'x-scraper': { ids: { weebcentral: 'SERIES', anilist: 42 } },
|
||||
}));
|
||||
await fs.writeFile(path.join(book, 'c0001 [WeebCentral]', 'chapter.json'), JSON.stringify({ source: { name: 'WeebCentral', url: 'https://weebcentral.com/chapters/A' } }));
|
||||
await fs.writeFile(path.join(book, 'c0002 [MangaDex]', 'chapter.json'), JSON.stringify({ source: { name: 'MangaDex · Group', url: 'https://mangadex.org/chapter/B' } }));
|
||||
const inventory = await scanLibrary(temp);
|
||||
const found = findManga(inventory, { source: 'weebcentral', seriesId: 'SERIES' });
|
||||
assert.equal(found.id, 'Library Manga');
|
||||
assert.equal(findManga(inventory, { source: 'mangadex', seriesId: 'other', anilist: 42 }).id, 'Library Manga');
|
||||
assert.equal(found.autoUpdate, true);
|
||||
assert.equal(shouldDownload(found, { number: '1' }, 'mangadex', 'skip-any'), false);
|
||||
assert.equal(shouldDownload(found, { number: '1' }, 'mangadex', 'add-source'), true);
|
||||
assert.equal(shouldDownload(found, { number: '1' }, 'weebcentral', 'add-source'), false);
|
||||
assert.equal(shouldDownload(found, { number: '3' }, 'weebcentral', 'skip-any'), true);
|
||||
assert.equal(isNewerThanLibrary(found, { number: '1' }), false);
|
||||
assert.equal(isNewerThanLibrary(found, { number: '3' }), true);
|
||||
await setAutoUpdate(temp, 'Library Manga', false);
|
||||
assert.equal((await scanLibrary(temp)).mangas[0].autoUpdate, false);
|
||||
});
|
||||
|
||||
test('a shared work ID cannot merge distinct source listings or ambiguous editions', () => {
|
||||
const original = { id: 'Main', ids: { weebcentral: 'MAIN', anilist: 42 }, sources: [] };
|
||||
assert.equal(findManga({ mangas: [original] }, { source: 'weebcentral', seriesId: 'PILOT', anilist: 42 }), null);
|
||||
assert.equal(findManga({ mangas: [original] }, { source: 'mangadex', seriesId: 'OTHER', anilist: 42 }), original);
|
||||
assert.equal(findManga({ mangas: [original, { id: 'Pilot', ids: { anilist: 42 }, sources: [] }] },
|
||||
{ source: 'mangadex', seriesId: 'OTHER', anilist: 42 }), null);
|
||||
});
|
||||
|
||||
test('recent events read the bounded log tail without losing the newest incident', async t => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-events-'));
|
||||
t.after(() => fs.rm(root, { recursive: true, force: true }));
|
||||
const service = new ScraperService({ dataRoot: root });
|
||||
await fs.writeFile(service.logFile, `${'x'.repeat(600 * 1024)}\n${JSON.stringify({ level: 'error', message: 'recent' })}\n`);
|
||||
assert.deepEqual(await service.recentEvents(1), [{ level: 'error', message: 'recent' }]);
|
||||
assert.equal((await service.recentEvents(10)).at(-1).message, 'recent');
|
||||
});
|
||||
|
||||
test('paused legacy work is validated, published, and converted to incremental mode', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-legacy-publish-'));
|
||||
const dataRoot = path.join(temp, 'data'), libraryRoot = path.join(temp, 'library');
|
||||
const workRoot = path.join(libraryRoot, '.mcscraper-work-SERIES42');
|
||||
const stagedManga = path.join(workRoot, 'Legacy Manga'), book = path.join(stagedManga, 'v01');
|
||||
await fs.mkdir(dataRoot, { recursive: true });
|
||||
await fs.mkdir(book, { recursive: true });
|
||||
await fs.writeFile(path.join(stagedManga, 'manga.json'), JSON.stringify({
|
||||
title: 'Legacy Manga', status: 'ongoing',
|
||||
sources: [{ name: 'Alpha', url: 'https://alpha.test/title/SERIES42' }],
|
||||
'x-scraper': { ids: { alpha: 'SERIES42' } },
|
||||
}));
|
||||
for (const number of [1, 2]) {
|
||||
const chapter = path.join(book, `c000${number} [Alpha]`);
|
||||
await fs.mkdir(chapter, { recursive: true });
|
||||
await fs.writeFile(path.join(chapter, 'chapter.json'), JSON.stringify({ source: { name: 'Alpha', url: `https://alpha.test/chapter/${number}` } }));
|
||||
await fs.writeFile(path.join(chapter, '0001.png'), Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
|
||||
}
|
||||
const incomplete = path.join(book, '.staging-mcscraper-abc123');
|
||||
await fs.mkdir(incomplete);
|
||||
await fs.writeFile(path.join(incomplete, '0001.png'), Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
|
||||
await fs.writeFile(path.join(workRoot, 'job.json'), JSON.stringify({
|
||||
version: 2, status: 'paused', completed: ['one', 'two'], total: 4,
|
||||
}));
|
||||
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
|
||||
version: 3, paused: true, status: 'paused', settings: validateSettings({}), history: [], publishedRevision: 0,
|
||||
queue: [{ id: 'old-id', seriesId: 'SERIES42', source: 'alpha', title: 'Legacy Manga', url: 'https://alpha.test/title/SERIES42', status: 'queued', order: 0 }],
|
||||
}));
|
||||
const alpha = {
|
||||
id: 'alpha', label: 'Alpha',
|
||||
match(value) { return String(value).includes('alpha.test/title/') ? { source: 'alpha', id: 'SERIES42', title: 'Legacy Manga', url: value } : null; },
|
||||
async discover() { return { chapters: [] }; }, async run() { return null; },
|
||||
};
|
||||
const service = await new ScraperService({
|
||||
libraryRoot, dataRoot, adapters: [alpha],
|
||||
adapterFor: value => { const info = alpha.match(value); return info ? { adapter: alpha, info } : null; },
|
||||
}).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
|
||||
|
||||
const inspection = await inspectLegacyWork({ libraryRoot, seriesId: 'SERIES42' });
|
||||
assert.equal(inspection.chapterCount, 2);
|
||||
assert.equal(inspection.pageCount, 2);
|
||||
assert.equal(inspection.incompleteStages.length, 1);
|
||||
const result = await service.publishLegacyReady();
|
||||
assert.equal(result.published.length, 1);
|
||||
assert.equal(result.published[0].chaptersPublished, 2);
|
||||
assert.equal(result.published[0].incompleteStagesQuarantined, 1);
|
||||
assert.equal(await fs.stat(path.join(libraryRoot, 'Legacy Manga', 'v01', 'c0002 [Alpha]')).then(stat => stat.isDirectory()), true);
|
||||
await assert.rejects(fs.stat(stagedManga));
|
||||
const migrated = JSON.parse(await fs.readFile(path.join(workRoot, 'job.json'), 'utf8'));
|
||||
assert.equal(migrated.publicationMode, 'incremental');
|
||||
assert.equal(migrated.migratedToIncremental.chaptersPublished, 2);
|
||||
assert.equal(migrated.migratedToIncremental.incompleteStagesQuarantined, 1);
|
||||
assert.equal(await fs.stat(path.join(workRoot, '.legacy-incomplete', 'v01__.staging-mcscraper-abc123')).then(stat => stat.isDirectory()), true);
|
||||
assert.equal(service.state.queue[0].libraryMangaId, 'Legacy Manga');
|
||||
assert.equal(service.state.publishedRevision, 1);
|
||||
});
|
||||
|
||||
test('WeebCentral fragments preserve chapter and page order', () => {
|
||||
const chapters = parseChapters('<a href="/chapters/P1">Prologue 1</a><a href="/chapters/ABC">Chapter 12</a><a href="/chapters/DEF">Chapter 12.5</a><a href="/chapters/S2">S2 - Chapter 1</a><a href="/chapters/R442"><span>Rating 442</span><span>Last Read</span></a><a href="/chapters/E128"><span>Episode 128</span><span>Last Read</span></a><a href="/chapters/E78"><span>Episode: 78</span><span>Last Read</span></a><a href="/chapters/N182"><span>Night 182</span><span>Last Read</span></a><a href="/chapters/L65">Log. 65</a><a href="/chapters/A606">Act 60.6</a>', 'https://weebcentral.com/s');
|
||||
assert.deepEqual(chapters.map(item => item.number), ['0.01', '12', '12.5', '1', '442', '128', '78', '182', '65', '60.6']);
|
||||
assert.deepEqual(chapters[3].book, { number: 2, title: 'Season 2' });
|
||||
const pages = parsePageUrls('<img src="/001.png"><img data-src="https://img.example/002.webp">', 'https://weebcentral.com/c');
|
||||
assert.deepEqual(pages, ['https://weebcentral.com/001.png', 'https://img.example/002.webp']);
|
||||
});
|
||||
|
||||
test('WeebCentral reports unrecognized chapter labels separately from an empty catalog', async () => {
|
||||
const series = '<h1>Example</h1><div hx-get="/series/X/full-chapter-list"></div>';
|
||||
const httpFor = list => ({ get: async url => ({ text: async () => url.endsWith('full-chapter-list') ? list : series }) });
|
||||
const unknown = await discoverWeebCatalog('https://weebcentral.com/series/X/Example', httpFor('<a href="/chapters/ABC">Part One</a>'));
|
||||
assert.equal(unknown.chapterLinks, 1);
|
||||
assert.equal(unknown.chapters.length, 0);
|
||||
assert.equal(unknown.availability.reason, 'unrecognized-labels');
|
||||
assert.match(unknown.availability.message, /1 chapter links/);
|
||||
const empty = await discoverWeebCatalog('https://weebcentral.com/series/X/Example', httpFor('<div>No chapters</div>'));
|
||||
assert.equal(empty.chapterLinks, 0);
|
||||
assert.equal(empty.availability, undefined);
|
||||
});
|
||||
|
||||
test('completed download reinspection preserves complete state when all scoped chapters are present', async () => {
|
||||
const service = new ScraperService({ libraryRoot: 'unused', dataRoot: 'unused' });
|
||||
const item = { id: 'done', title: 'Done', source: 'weebcentral', url: 'https://weebcentral.com/series/EXAMPLE/Done',
|
||||
seriesId: 'EXAMPLE', kind: 'inspection', status: 'queued', completed: '2026-10-01T00:00:00Z',
|
||||
publishedCount: 2, selectedCount: 2, scope: { mode: 'missing' }, inspectionFromComplete: true };
|
||||
service.state = { paused: false, settings: validateSettings({}), queue: [item], incidents: [], sourcePaused: {}, currentIds: {} };
|
||||
service.recordCatalog = async (entry, catalog) => { entry.sourceTitle = catalog.title; entry.catalogTotal = catalog.chapters.length; };
|
||||
service.libraryInventory = async () => ({ mangas: [{ id: 'Done', ids: { weebcentral: 'EXAMPLE' }, sources: [],
|
||||
chapters: [{ book: '1', number: '1' }, { book: '1', number: '2' }] }] });
|
||||
service.event = async () => {};
|
||||
const adapter = { discover: async () => ({ title: 'Done', chapters: [{ id: 'one', number: '1' }, { id: 'two', number: '2' }] }) };
|
||||
await service.runDiscovery(item, { adapter }, 'weebcentral');
|
||||
assert.equal(item.status, 'complete');
|
||||
assert.equal(item.kind, 'download');
|
||||
assert.equal(item.completed, '2026-10-01T00:00:00Z');
|
||||
assert.equal(item.publishedCount, 2);
|
||||
assert.equal(item.inspectionFromComplete, undefined);
|
||||
});
|
||||
|
||||
test('publisher keeps season chapter numbers in separate books', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-books-'));
|
||||
t.after(() => fs.rm(temp, { recursive: true, force: true }));
|
||||
const page = path.join(temp, 'page.png');
|
||||
await fs.writeFile(page, Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
|
||||
const base = { title: '', source: { name: 'Host', url: 'https://example.test/chapter' }, pages: [{ path: page }] };
|
||||
await publishManifest({
|
||||
version: 1,
|
||||
series: { title: 'Season Test', direction: 'ttb', sources: [{ name: 'Host', url: 'https://example.test/series' }] },
|
||||
chapters: [
|
||||
{ ...base, number: '1', book: { number: 1, title: 'Season 1' } },
|
||||
{ ...base, number: '1', book: { number: 2, title: 'Season 2' } },
|
||||
],
|
||||
}, { libraryRoot: path.join(temp, 'library') });
|
||||
assert.equal(await fs.readFile(path.join(temp, 'library', 'Season Test', 'v01 Season 1', 'book.json'), 'utf8').then(JSON.parse).then(x => x.title), 'Season 1');
|
||||
assert.equal(await fs.readFile(path.join(temp, 'library', 'Season Test', 'v02 Season 2', 'book.json'), 'utf8').then(JSON.parse).then(x => x.title), 'Season 2');
|
||||
});
|
||||
|
||||
test('publisher stages and atomically exposes a complete new manga', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-test-'));
|
||||
t.after(() => fs.rm(temp, { recursive: true, force: true }));
|
||||
const page = path.join(temp, 'page.png');
|
||||
await fs.writeFile(page, Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]));
|
||||
const result = await publishManifest({
|
||||
version: 1,
|
||||
series: { title: 'Test Manga', direction: 'rtl', metadata: { status: 'ongoing' },
|
||||
sources: [{ name: 'Test', url: 'https://example.test/manga' }] },
|
||||
chapters: [{ number: '1', title: 'Start', source: { name: 'Test', url: 'https://example.test/chapter/1' }, pages: [{ path: page }] }],
|
||||
}, { libraryRoot: path.join(temp, 'library') });
|
||||
assert.equal(result.results[0].status, 'published');
|
||||
const chapter = path.join(temp, 'library', 'Test Manga', 'v01', 'c0001 Start [Test]');
|
||||
assert.deepEqual((await fs.readdir(chapter)).sort(), ['0001.png', 'chapter.json']);
|
||||
const meta = JSON.parse(await fs.readFile(path.join(temp, 'library', 'Test Manga', 'manga.json'), 'utf8'));
|
||||
assert.equal(meta['x-scraper'].autoUpdate, undefined, 'ongoing is a derived default, not a frozen override');
|
||||
assert.equal((await scanLibrary(path.join(temp, 'library'))).mangas[0].autoUpdate, true);
|
||||
assert.equal((await fs.readdir(path.join(temp, 'library'))).some(name => name.startsWith('.staging-')), false);
|
||||
});
|
||||
|
||||
test('publisher installs a validated AniList series cover', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-cover-'));
|
||||
t.after(() => fs.rm(temp, { recursive: true, force: true }));
|
||||
const page = path.join(temp, 'page.png');
|
||||
const png = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]);
|
||||
await fs.writeFile(page, png);
|
||||
let requests = 0;
|
||||
const httpClient = {
|
||||
async get(url) {
|
||||
requests++;
|
||||
assert.equal(url, 'https://s4.anilist.co/example.png');
|
||||
return { arrayBuffer: async () => png };
|
||||
},
|
||||
};
|
||||
const series = {
|
||||
title: 'Covered Manga', direction: 'rtl',
|
||||
sources: [{ name: 'Test', url: 'https://example.test/covered' }],
|
||||
coverUrl: 'https://s4.anilist.co/example.png',
|
||||
};
|
||||
const result = await publishManifest({
|
||||
version: 1, series,
|
||||
chapters: [{ number: '1', source: { name: 'Test', url: 'https://example.test/chapter/1' }, pages: [{ path: page }] }],
|
||||
}, { libraryRoot: path.join(temp, 'library'), httpClient });
|
||||
assert.equal(result.cover.status, 'installed');
|
||||
assert.deepEqual(await fs.readFile(path.join(temp, 'library', 'Covered Manga', 'cover.png')), png);
|
||||
assert.equal(requests, 1);
|
||||
});
|
||||
|
||||
test('publisher retains validated partial pages and resumes without duplicate requests', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-resume-'));
|
||||
t.after(() => fs.rm(temp, { recursive: true, force: true }));
|
||||
const png = Buffer.from([137, 80, 78, 71, 13, 10, 26, 10, 0]);
|
||||
const manifest = {
|
||||
version: 1, series: { title: 'Resumable', direction: 'rtl', sources: [{ name: 'Test', url: 'https://example.test/series' }] },
|
||||
chapters: [{ number: '1', source: { name: 'Test', url: 'https://example.test/chapter/1' },
|
||||
pages: [1, 2].map(n => ({ url: `https://example.test/${n}.png` })) }],
|
||||
};
|
||||
const requests = [];
|
||||
const httpClient = { async get(url) { requests.push(url); return { arrayBuffer: async () => png }; } };
|
||||
const libraryRoot = path.join(temp, 'library');
|
||||
await assert.rejects(publishManifest(manifest, { libraryRoot, httpClient,
|
||||
onPage() { throw new Error('cooperative pause'); } }), /cooperative pause/);
|
||||
await assert.rejects(fs.stat(path.join(libraryRoot, 'Resumable')));
|
||||
const hidden = (await fs.readdir(libraryRoot))[0];
|
||||
assert.match(hidden, /^\.staging-mcscraper-resume-/);
|
||||
const { cleanupOwnTemps } = require('../lib/publisher');
|
||||
await cleanupOwnTemps(libraryRoot, -1);
|
||||
assert.ok(await fs.stat(path.join(libraryRoot, hidden)));
|
||||
await publishManifest(manifest, { libraryRoot, httpClient });
|
||||
assert.deepEqual(requests, ['https://example.test/1.png', 'https://example.test/2.png']);
|
||||
const chapter = path.join(libraryRoot, 'Resumable', 'v01', 'c0001 [Test]');
|
||||
assert.deepEqual((await fs.readdir(chapter)).sort(), ['0001.png', '0002.png', 'chapter.json']);
|
||||
});
|
||||
|
||||
test('adapter registry recognizes supported series URLs', () => {
|
||||
const found = adapterFor('https://weebcentral.com/series/01ABCDEF/My-Series');
|
||||
assert.equal(found.adapter.id, 'weebcentral');
|
||||
assert.equal(found.info.id, '01ABCDEF');
|
||||
assert.equal(found.info.title, 'My Series');
|
||||
const dex = adapterFor('https://mangadex.org/title/5bdb093f-9707-4a30-b02e-c8b66554ecf0/biomega');
|
||||
assert.equal(dex.adapter.id, 'mangadex');
|
||||
assert.equal(dex.info.id, '5bdb093f-9707-4a30-b02e-c8b66554ecf0');
|
||||
assert.equal(adapterFor('https://example.test/series/1'), null);
|
||||
});
|
||||
|
||||
test('MangaDex releases are normalized, deduplicated, credited, and host-validated', () => {
|
||||
assert.equal(chapterNumber('0012.50'), '12.5');
|
||||
assert.equal(chapterNumber('Special'), null);
|
||||
const release = (id, chapter, group, publishAt) => ({
|
||||
id, attributes: { chapter, title: '', pages: 2, publishAt, externalUrl: null },
|
||||
relationships: group ? [{ id: `group-${id}`, type: 'scanlation_group', attributes: { name: group } }] : [],
|
||||
});
|
||||
const selected = selectChapterReleases([
|
||||
release('fan', '1', 'Fan Group', '2020-01-01T00:00:00Z'),
|
||||
release('official', '1.0', 'Publisher Official', '2021-01-01T00:00:00Z'),
|
||||
release('two', '2', 'Fan Group', '2020-01-02T00:00:00Z'),
|
||||
]);
|
||||
assert.deepEqual(selected.map(item => item.id), ['official', 'two']);
|
||||
assert.equal(selected[0].group.name, 'Publisher Official');
|
||||
const pages = pageUrls({
|
||||
baseUrl: 'https://node.mangadex.network',
|
||||
chapter: { hash: 'abc', data: ['1.jpg', '2.png'] },
|
||||
}, 'https://mangadex.org/chapter/test', 2);
|
||||
assert.deepEqual(pages.map(item => item.url), [
|
||||
'https://node.mangadex.network/data/abc/1.jpg',
|
||||
'https://node.mangadex.network/data/abc/2.png',
|
||||
]);
|
||||
assert.throws(() => pageUrls({ baseUrl: 'https://evil.example', chapter: { hash: 'abc', data: ['1.jpg'] } }, 'https://mangadex.org/chapter/test', 1));
|
||||
});
|
||||
|
||||
test('daemon queue state is durable while paused', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-service-'));
|
||||
const dataRoot = path.join(temp, 'data');
|
||||
await fs.mkdir(dataRoot, { recursive: true });
|
||||
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
|
||||
version: 1, paused: true, status: 'paused', settings: validateSettings({}), queue: [], history: [], publishedRevision: 0,
|
||||
}));
|
||||
const service = await new ScraperService({ libraryRoot: path.join(temp, 'library'), dataRoot }).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
|
||||
const item = await service.enqueue({ url: 'https://weebcentral.com/series/01ABCDEF/My-Series' });
|
||||
assert.equal(item.status, 'queued');
|
||||
const scoped = await service.enqueue({ url: 'https://weebcentral.com/series/01ABCDEF/My-Series', scope: { mode: 'latest' } });
|
||||
assert.notEqual(item.id, scoped.id);
|
||||
assert.equal(item.seriesId, scoped.seriesId);
|
||||
assert.equal((await service.publicState()).queue.length, 2);
|
||||
await service.updateSettings({ minMs: 1500, maxMs: 4500 });
|
||||
assert.equal(JSON.parse(await fs.readFile(path.join(dataRoot, 'queue.json'), 'utf8')).settings.minMs, 1500);
|
||||
await service.remove(item.id);
|
||||
await service.remove(scoped.id);
|
||||
assert.equal((await service.publicState()).queue.length, 0);
|
||||
});
|
||||
|
||||
test('retry recovers stale downloading entries before work resumes', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-retry-'));
|
||||
const dataRoot = path.join(temp, 'data');
|
||||
await fs.mkdir(dataRoot, { recursive: true });
|
||||
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
|
||||
version: 1, paused: true, status: 'error', currentId: 'FAILED', settings: validateSettings({}),
|
||||
queue: [
|
||||
{ id: 'STALE', title: 'Stale', url: 'https://weebcentral.com/series/STALE/Stale', status: 'downloading', order: 0 },
|
||||
{ id: 'FAILED', title: 'Failed', url: 'https://weebcentral.com/series/FAILED/Failed', status: 'error', order: 1 },
|
||||
], history: [], publishedRevision: 0,
|
||||
}));
|
||||
const service = await new ScraperService({ libraryRoot: path.join(temp, 'library'), dataRoot }).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
|
||||
await service.control('retry', 'FAILED');
|
||||
assert.deepEqual(service.state.queue.map(item => item.status), ['queued', 'queued']);
|
||||
assert.equal(service.state.currentId, null);
|
||||
assert.equal(service.state.paused, false);
|
||||
});
|
||||
|
||||
test('nightly discovery queues one preferred supported source for ongoing manga', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-discovery-'));
|
||||
const dataRoot = path.join(temp, 'data'), libraryRoot = path.join(temp, 'library');
|
||||
const manga = path.join(libraryRoot, 'Ongoing Manga');
|
||||
await fs.mkdir(manga, { recursive: true });
|
||||
await fs.mkdir(dataRoot, { recursive: true });
|
||||
await fs.writeFile(path.join(manga, 'manga.json'), JSON.stringify({
|
||||
title: 'Ongoing Manga', status: 'ongoing',
|
||||
sources: [{ name: 'Alpha', url: 'https://alpha.test/title/42' }],
|
||||
'x-scraper': { ids: { alpha: '42' } },
|
||||
}));
|
||||
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
|
||||
version: 3, paused: true, status: 'paused', settings: validateSettings({}), queue: [], history: [], publishedRevision: 0,
|
||||
}));
|
||||
const alpha = {
|
||||
id: 'alpha', label: 'Alpha',
|
||||
match(value) { const url = new URL(value); return url.hostname === 'alpha.test' ? { source: 'alpha', id: url.pathname.split('/').pop(), title: 'Alpha', url: value } : null; },
|
||||
async discover() { return { chapters: [] }; }, async run() { return null; },
|
||||
};
|
||||
const resolve = value => { const info = alpha.match(value); return info ? { adapter: alpha, info } : null; };
|
||||
const service = await new ScraperService({ libraryRoot, dataRoot, adapterFor: resolve, adapters: [alpha] }).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
|
||||
const first = await service.enqueueDiscoveryChecks('test');
|
||||
const second = await service.enqueueDiscoveryChecks('test');
|
||||
assert.deepEqual(first, { queued: 1, skipped: 0 });
|
||||
assert.deepEqual(second, { queued: 0, skipped: 1 });
|
||||
assert.equal(service.state.queue[0].kind, 'discovery');
|
||||
assert.equal(service.state.queue[0].libraryMangaId, 'Ongoing Manga');
|
||||
});
|
||||
|
||||
test('discovery promotes only chapters newer than the library frontier', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-discovery-run-'));
|
||||
const dataRoot = path.join(temp, 'data'), libraryRoot = path.join(temp, 'library');
|
||||
const chapter = path.join(libraryRoot, 'Ongoing Manga', 'v01', 'c0001 [Alpha]');
|
||||
const laterChapter = path.join(libraryRoot, 'Ongoing Manga', 'v01', 'c0003 [Alpha]');
|
||||
await fs.mkdir(chapter, { recursive: true });
|
||||
await fs.mkdir(laterChapter, { recursive: true });
|
||||
await fs.mkdir(dataRoot, { recursive: true });
|
||||
await fs.writeFile(path.join(libraryRoot, 'Ongoing Manga', 'manga.json'), JSON.stringify({
|
||||
title: 'Ongoing Manga', status: 'ongoing', sources: [{ name: 'Alpha', url: 'https://alpha.test/title/42' }],
|
||||
'x-scraper': { ids: { alpha: '42' } },
|
||||
}));
|
||||
await fs.writeFile(path.join(chapter, 'chapter.json'), JSON.stringify({ source: { name: 'Alpha', url: 'https://alpha.test/chapter/one' } }));
|
||||
await fs.writeFile(path.join(laterChapter, 'chapter.json'), JSON.stringify({ source: { name: 'Alpha', url: 'https://alpha.test/chapter/three' } }));
|
||||
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
|
||||
version: 3, paused: true, status: 'paused', settings: validateSettings({}), queue: [], history: [], publishedRevision: 0,
|
||||
}));
|
||||
let downloadedScope = null;
|
||||
const alpha = {
|
||||
id: 'alpha', label: 'Alpha',
|
||||
match(value) { const url = new URL(value); return url.hostname === 'alpha.test' ? { source: 'alpha', id: '42', title: 'Alpha', url: value } : null; },
|
||||
async discover() { return { chapters: [{ id: 'one', number: '1' }, { id: 'two', number: '2' }, { id: 'three', number: '3' }, { id: 'four', number: '4' }] }; },
|
||||
async run(item, options) { downloadedScope = item.scope; await options.onPublished({ chapter: '4' }); return { finalPath: path.join(libraryRoot, 'Ongoing Manga'), publishedCount: 1, selectedCount: 1 }; },
|
||||
};
|
||||
const resolve = value => { const info = alpha.match(value); return info ? { adapter: alpha, info } : null; };
|
||||
const service = await new ScraperService({ libraryRoot, dataRoot, adapterFor: resolve, adapters: [alpha] }).init();
|
||||
const waitFor = async predicate => {
|
||||
const deadline = Date.now() + 3000;
|
||||
while (!predicate()) { if (Date.now() > deadline) throw new Error('Timed out waiting for discovery'); await new Promise(resolveWait => setTimeout(resolveWait, 20)); }
|
||||
};
|
||||
try {
|
||||
await service.enqueueDiscoveryChecks('test');
|
||||
await service.control('resume');
|
||||
await waitFor(() => service.state.queue[0]?.status === 'complete');
|
||||
assert.deepEqual(downloadedScope, { mode: 'ids', ids: ['four'] });
|
||||
assert.equal(service.state.queue[0].publishedCount, 1);
|
||||
const summary = (await service.publicState()).updateChecks.find(item => item.mangaId === 'Ongoing Manga' && item.source === 'alpha');
|
||||
assert.ok(summary.lastCheckedAt && summary.lastNewContentAt && summary.lastDownloadedAt);
|
||||
assert.equal(summary.lastResult, 'new-content');
|
||||
} finally {
|
||||
await service.close();
|
||||
await fs.rm(temp, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('source-aware scheduler runs one item per source in parallel', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-sources-'));
|
||||
const dataRoot = path.join(temp, 'data');
|
||||
await fs.mkdir(dataRoot, { recursive: true });
|
||||
const queue = [
|
||||
{ id: 'A1', source: 'alpha', title: 'Alpha One', url: 'https://alpha.test/1', status: 'queued', order: 0 },
|
||||
{ id: 'A2', source: 'alpha', title: 'Alpha Two', url: 'https://alpha.test/2', status: 'queued', order: 1 },
|
||||
{ id: 'B1', source: 'beta', title: 'Beta One', url: 'https://beta.test/1', status: 'queued', order: 2 },
|
||||
];
|
||||
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
|
||||
version: 2, paused: true, status: 'paused', settings: validateSettings({}), queue, history: [], publishedRevision: 0,
|
||||
}));
|
||||
const resolvers = new Map(), started = [];
|
||||
const fakeAdapter = id => ({
|
||||
id, label: id,
|
||||
match(value) { const url = new URL(value); return url.hostname === `${id}.test` ? { source: id, id: url.pathname.slice(1), title: id, url: value } : null; },
|
||||
run(item) { started.push(item.id); return new Promise(resolve => resolvers.set(item.id, () => resolve(path.join(temp, item.id)))); },
|
||||
});
|
||||
const fakeAdapters = [fakeAdapter('alpha'), fakeAdapter('beta')];
|
||||
const resolveAdapter = value => {
|
||||
for (const adapter of fakeAdapters) { const info = adapter.match(value); if (info) return { adapter, info }; }
|
||||
return null;
|
||||
};
|
||||
const service = await new ScraperService({ libraryRoot: path.join(temp, 'library'), dataRoot, adapterFor: resolveAdapter, adapters: fakeAdapters }).init();
|
||||
const waitFor = async predicate => {
|
||||
const deadline = Date.now() + 3000;
|
||||
while (!predicate()) { if (Date.now() > deadline) throw new Error('Timed out waiting for scheduler'); await new Promise(resolve => setTimeout(resolve, 20)); }
|
||||
};
|
||||
try {
|
||||
await service.control('resume');
|
||||
await waitFor(() => resolvers.has('A1') && resolvers.has('B1'));
|
||||
assert.deepEqual(new Set(started), new Set(['A1', 'B1']));
|
||||
assert.equal(resolvers.has('A2'), false);
|
||||
resolvers.get('A1')();
|
||||
await waitFor(() => resolvers.has('A2'));
|
||||
assert.equal(service.state.queue.find(item => item.id === 'A2').status, 'downloading');
|
||||
resolvers.get('A2')(); resolvers.get('B1')();
|
||||
await waitFor(() => service.state.queue.every(item => item.status === 'complete'));
|
||||
assert.equal(service.state.publishedRevision, 3);
|
||||
} finally {
|
||||
for (const resolve of resolvers.values()) resolve();
|
||||
await service.close();
|
||||
await fs.rm(temp, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
Reference in new issue
Block a user