Compare commits
27
Commits
No files matched your search
@@ -5,10 +5,12 @@
|
||||
!library.js
|
||||
!reading-direction.js
|
||||
!chapter-identity.js
|
||||
!chapter-dates.js
|
||||
!catalog.js
|
||||
!backup.js
|
||||
!reading-analytics.js
|
||||
!reading-access.js
|
||||
!moments.js
|
||||
!public/
|
||||
!public/**
|
||||
!greatest/
|
||||
|
||||
+7
-2
@@ -11,13 +11,18 @@ SCRAPER_TOKEN_PATH=./.secrets/scraper_token
|
||||
# Built-in backups of DATA_PATH (docs/backup.md): an existing folder, ideally on another
|
||||
# disk/NAS, writable by uid 1000. Unset = backups off (./backup is then mounted unused).
|
||||
# BACKUP_PATH=./backup
|
||||
TZ=Europe/Berlin
|
||||
TZ=UTC
|
||||
COOKIE_SECURE=0
|
||||
# Optional server-enforced reading windows. Day keys: 0=Sunday through 6=Saturday.
|
||||
# Ranges may cross midnight. Multiple ranges per day are supported.
|
||||
# READING_WINDOWS_JSON='{"0":[["08:00","01:00"]],"1":[["07:00","24:00"]],"2":[["07:00","24:00"]],"3":[["07:00","24:00"]],"4":[["07:00","24:00"]],"5":[["07:00","24:00"]],"6":[["08:00","01:00"]]}'
|
||||
READING_TIMEZONE=Europe/Berlin
|
||||
READING_TIMEZONE=UTC
|
||||
READING_QUIZ_UNLOCK_MINUTES=60
|
||||
# Optional deployment-managed JSON file under DATA_PATH, e.g. /data/reading-quiz.json.
|
||||
# READING_QUIZ_CATALOG_PATH=/data/reading-quiz.json
|
||||
PROXY_NETWORK=mangacircus-edge
|
||||
# Extra Compose files, ':'-separated (docs/deployment.md), e.g. HTTPS with the bundled Caddy:
|
||||
# COMPOSE_FILE=docker-compose.yml:deploy/compose.https.yml
|
||||
# MANGACIRCUS_DOMAIN=manga.example.org
|
||||
# PEER_DATA_PATH=./data/peering
|
||||
# PEER_USAGE_PATH=./data/peering-usage
|
||||
@@ -0,0 +1,4 @@
|
||||
# Shell scripts and config used inside Linux containers keep LF on every platform
|
||||
*.sh text eol=lf
|
||||
deploy/Caddyfile text eol=lf
|
||||
*.yml text eol=lf
|
||||
+7
-1
@@ -21,7 +21,6 @@ pnpm-debug.log*
|
||||
!.env.example
|
||||
!.env.*.example
|
||||
.secrets/
|
||||
/ops/proxmox/environment.json
|
||||
*.pem
|
||||
*.key
|
||||
|
||||
@@ -52,3 +51,10 @@ $RECYCLE.BIN/
|
||||
*.swp
|
||||
*.swo
|
||||
*~
|
||||
|
||||
# --- deploy/circus.sh state ---
|
||||
.queue-was-paused
|
||||
|
||||
# --- personal agent coordination (synced by a private tool; never published) ---
|
||||
CLAUDE.local.md
|
||||
.claude/skills/manga-org/
|
||||
@@ -0,0 +1,54 @@
|
||||
# Working on MangaCircus
|
||||
|
||||
Guide for contributors and coding agents. Users who only want to run it: [docs/deployment.md](docs/deployment.md).
|
||||
|
||||
## Layout
|
||||
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| `server.js`, `library.js` | Reader: HTTP server, accounts, progress, JSON storage; folder scanner and image-size reader |
|
||||
| `chapter-identity.js`, `chapter-dates.js`, `reading-*.js`, `catalog.js`, `backup.js` | Shared modules (chapter keys, chapter dates, reading clock/limits, collections, built-in backup) |
|
||||
| `public/` | The web pages: plain HTML, CSS and JavaScript, no build step |
|
||||
| `scraper/` | Download queue daemon (`service.js`, `daemon.js`), source adapters and parsers, the publisher that writes the library, metadata lookups |
|
||||
| `peering/` | Optional server-to-server sharing (gateway, client, adapter) |
|
||||
| `runtime/` | In-container supervisor and configuration |
|
||||
| `discovery/`, `greatest/` | Discovery Scout and the Greatest of All Time dashboard |
|
||||
| `tools/` | Demo library, health checks, maintenance tools (dry run by default), UI and Compose test drivers |
|
||||
| `deploy/` | `circus.sh` operations helper, HTTPS with Caddy |
|
||||
| `docs/`, `vision/` | Contracts and guides; design notes for planned features |
|
||||
|
||||
## Rules
|
||||
|
||||
- **No runtime dependencies.** Node.js built-ins only. Playwright is an optional dev tool for UI tests.
|
||||
- **One writer per store.** The reader alone writes `DATA` (users, progress, catalog). The scraper alone
|
||||
writes the library and its queue. Tools that edit `chapter.json` files are dry runs by default and need
|
||||
`--apply --writers-stopped`.
|
||||
- **Library writes go through the publisher** (`scraper/lib/publisher.js`): staging folder, validated
|
||||
images, one atomic rename. The folder contract is [docs/scraper-guide.md](docs/scraper-guide.md).
|
||||
- **Sources are polite.** Sequential requests, persisted per-host delays, stop on 401/403/429, never bypass
|
||||
access controls ([scraper/AGENT_RUNBOOK.md](scraper/AGENT_RUNBOOK.md)).
|
||||
- **No secrets or personal data in the repository**: no `.env`, `.secrets/`, `data/`, libraries, host
|
||||
names, IP addresses or local paths. Tests use disposable fixtures and never a real library or user
|
||||
database.
|
||||
- Keep the docs current in the same change: a new file format field goes into
|
||||
[docs/library-format.md](docs/library-format.md), a new setting into the README or
|
||||
[docs/deployment.md](docs/deployment.md), an API change into [docs/architecture.md](docs/architecture.md).
|
||||
- Match the surrounding style: small functions, short comments that say why, no framework.
|
||||
|
||||
## Tests
|
||||
|
||||
```bash
|
||||
npm test # reader, scraper, runtime and peering suites (node --test)
|
||||
```
|
||||
|
||||
Without Node on the machine, run them in Docker (the peering test needs `openssl`, present in `node:24`):
|
||||
|
||||
```bash
|
||||
docker run --rm -v "$PWD:/app" -w /app node:24 npm test
|
||||
```
|
||||
|
||||
Browser and Compose acceptance tests (Playwright, disposable data, never the live stack):
|
||||
`npm run test:ui`, `npm run test:discovery-ui`, `npm run test:direction-ui`, `npm run test:peer-ab`.
|
||||
|
||||
Try changes against demo data: `node tools/make-dummy.js library && npm run start:all`, or
|
||||
`docker compose -f docker-compose.yml -f compose.dev.yml up -d --build` (mounts the editable source).
|
||||
@@ -0,0 +1,3 @@
|
||||
# MangaCircus
|
||||
|
||||
Read and follow @AGENTS.md (layout, rules, tests). Deployment: docs/deployment.md.
|
||||
+2
-2
@@ -3,7 +3,7 @@ RUN apk add --no-cache tzdata openssl
|
||||
WORKDIR /app
|
||||
ARG VCS_REF=development
|
||||
LABEL org.opencontainers.image.title="MangaCircus" org.opencontainers.image.revision=$VCS_REF
|
||||
COPY package.json server.js library.js reading-direction.js chapter-identity.js reading-analytics.js reading-access.js catalog.js backup.js ./
|
||||
COPY package.json server.js library.js reading-direction.js chapter-identity.js chapter-dates.js reading-analytics.js reading-access.js catalog.js backup.js moments.js ./
|
||||
COPY public ./public
|
||||
COPY greatest ./greatest
|
||||
COPY discovery ./discovery
|
||||
@@ -12,7 +12,7 @@ COPY scraper ./scraper
|
||||
COPY peering ./peering
|
||||
COPY runtime ./runtime
|
||||
RUN mkdir -p /library /data && chown node:node /data /library
|
||||
ENV PORT=8080 LIBRARY=/library DATA=/data TZ=Europe/Berlin
|
||||
ENV PORT=8080 LIBRARY=/library DATA=/data TZ=UTC
|
||||
EXPOSE 8080
|
||||
USER node
|
||||
HEALTHCHECK --interval=30s --timeout=5s --start-period=60s CMD node tools/healthcheck.js reader
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2026 maashkinoozhe
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -8,59 +8,53 @@ A no-frills, self-hosted manga / comic / webtoon reader.
|
||||
- Reader with three view modes (**Flow**, **Page**, **Book**), a page zoom that leaves the controls alone, and tap zones for mobile.
|
||||
- A separate **Greatest of All Time** manga-history dashboard, with five-year periods, per-user wish ratings, community ranking, and local-library badges. Its curated titles do not enter the download queue.
|
||||
|
||||
## Run it
|
||||
## Run it with Docker
|
||||
|
||||
The supported way to run MangaCircus is Docker Compose on a Linux host: one image, a **reader**
|
||||
service (the web app) and a **scraper** service (the managed download queue). Short version:
|
||||
|
||||
```bash
|
||||
node server.js
|
||||
git clone <repository-url> mangacircus && cd mangacircus
|
||||
mkdir -p library data/scraper data/peering data/peering-usage backup .secrets
|
||||
openssl rand -hex 32 > .secrets/scraper_token && chmod 600 .secrets/scraper_token
|
||||
cp .env.example .env # set LIBRARY_PATH etc.; see the guide
|
||||
sudo chown -R 1000:1000 library data backup
|
||||
docker compose build
|
||||
bash deploy/circus.sh up # reader first, then the scraper
|
||||
```
|
||||
|
||||
Open http://localhost:8080. The first account you create becomes the admin. Try it with demo content:
|
||||
Open http://127.0.0.1:8080. The first account you create becomes the admin. The full guide covers
|
||||
configuration, LAN and HTTPS access (a ready Caddy setup is included), adding manga, updates, backups,
|
||||
peering and troubleshooting: **[docs/deployment.md](docs/deployment.md)**.
|
||||
|
||||
## Try it without Docker
|
||||
|
||||
Needs Node.js 18 or newer, nothing else:
|
||||
|
||||
```bash
|
||||
node tools/make-dummy.js library
|
||||
node tools/make-dummy.js library # five demo manga
|
||||
npm run start:all # reader on :8080 and the scraper daemon on 127.0.0.1:8091
|
||||
```
|
||||
|
||||
To run the reader and managed scraper together, use `npm run start:all`. Admins can
|
||||
then manage the persistent download queue, progress, pauses, retries, and rate limits
|
||||
under **Settings → Admin → Scraper**. The reader continues working if the scraper is stopped.
|
||||
**Settings → Admin → Collections & metadata** manages grouped tiles, reading order and relationships;
|
||||
**Settings → Admin → Service health** shows recovery incidents. See [collections](docs/collections.md)
|
||||
and [resilience](docs/resilience.md) for the ownership and agent-maintenance contracts.
|
||||
The current Windows installation uses Docker Compose; its old `MangaCircus Supervisor`
|
||||
Scheduled Task is disabled. Do not run a native duplicate writer. See
|
||||
[docs/local-deployment.md](docs/local-deployment.md) for its observed layout, health
|
||||
checks, restart procedure, and recovery point.
|
||||
Deployment-controlled reading hours and the optional progress-aware quiz are
|
||||
documented in [reading limiter](docs/reading-limiter.md). They are disabled
|
||||
unless configured on the reader container.
|
||||
See [Greatest of All Time](docs/greatest.md) for catalog ownership, voting, and metadata refresh.
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
node tools/prepare-compose.js
|
||||
docker compose config -q
|
||||
docker compose up -d --build
|
||||
```
|
||||
|
||||
Stop existing native writers before using their data in containers. `docker-compose.yml`
|
||||
runs reader and scraper with explicit persistent bind mounts (`library/`,
|
||||
`data/`), file-backed secrets and only a loopback reader port. No named data volume is
|
||||
used. See [Compose deployment](docs/compose-deployment.md) for local development, VM
|
||||
recommendations, cutover, image releases and backup/rollback. The local Docker stack
|
||||
is verified; VM deployment is still pending.
|
||||
`npm start` runs the reader alone. Admins manage the persistent download queue, pauses, retries and rate
|
||||
limits under **Settings → Admin → Scraper**. The reader keeps working when the scraper is stopped.
|
||||
**Settings → Admin → Collections & metadata** manages grouped tiles, reading order and relationships, and
|
||||
**Settings → Admin → Service health** shows recovery incidents and backups ([collections](docs/collections.md),
|
||||
[resilience](docs/resilience.md)). Deployment-controlled reading hours and the optional progress-aware quiz
|
||||
are described in [reading limiter](docs/reading-limiter.md); they are off unless configured.
|
||||
See [Greatest of All Time](docs/greatest.md) for that dashboard's catalog and voting.
|
||||
|
||||
### Settings workspace
|
||||
|
||||
Settings has **User** and **Admin** menu groups. User sections cover Reading,
|
||||
Reading time & privacy, Notifications and Account. Admin sections cover Scraper,
|
||||
Server & library, Users & access, Collections & metadata and Service health.
|
||||
Scraper has local Queue, Automatic updates and Hoster settings tabs. Only admins
|
||||
see administration; personal settings do not poll scraper services. See
|
||||
[settings navigation](docs/settings-navigation.md) for routes and verification.
|
||||
Settings has **User** and **Admin** menu groups. User sections cover Reading, Reading time & privacy,
|
||||
Notifications and Account. Admin sections cover Scraper, Server & library, Users & access, Collections &
|
||||
metadata and Service health. Only admins see administration ([settings navigation](docs/settings-navigation.md)).
|
||||
|
||||
### Settings (environment variables)
|
||||
|
||||
These configure the reader process. In Docker most are set for you by `docker-compose.yml`; the variables
|
||||
you put in `.env` are listed in [docs/deployment.md](docs/deployment.md#4-configure-env).
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|-----------------|-------------|---------|
|
||||
| `PORT` | `8080` | HTTP port |
|
||||
@@ -76,7 +70,17 @@ see administration; personal settings do not poll scraper services. See
|
||||
| `SCRAPER_TOKEN_FILE` | empty | Alternative secret file (used in Compose) |
|
||||
| `BACKUP` | – | Existing folder for built-in backups of `DATA`; unset = off. Compose: set `BACKUP_PATH`. Schedule and retention under Settings → Service health ([backup](docs/backup.md)) |
|
||||
|
||||
Serve it behind a reverse proxy with HTTPS if it is reachable from the internet. All pages use relative URLs, so a sub-path like `/manga/` works.
|
||||
Serve it behind a reverse proxy with HTTPS if it is reachable from the internet. All pages use relative
|
||||
URLs, so a sub-path like `/manga/` works.
|
||||
|
||||
### Sources and copyright
|
||||
|
||||
The scraper can download from MangaDex (on by default; official API) and from WeebCentral and MangaPill.
|
||||
The last two are unofficial sites that mostly host unlicensed copies, so they are **off by default**. An admin
|
||||
can switch each one on under **Settings → Admin → Scraper → Hoster settings**. Peering can copy chapters
|
||||
between servers. You are responsible for only downloading and sharing what you are allowed to copy where you live,
|
||||
and for respecting each site's terms. The scraper is deliberately slow and stops on access errors; it never
|
||||
bypasses CAPTCHAs, logins or other access controls ([scraper/AGENT_RUNBOOK.md](scraper/AGENT_RUNBOOK.md)).
|
||||
|
||||
---
|
||||
|
||||
@@ -101,23 +105,27 @@ library/
|
||||
|
||||
## Documentation
|
||||
|
||||
Unnumbered MangaDex pilots/one-shots use stable `special-mangadex-<chapter-uuid>`
|
||||
chapter keys and human labels, not fabricated numbers. Existing numeric IDs remain
|
||||
unchanged. In Settings → Admin → Scraper, **Inspect & queue** shows the source edition
|
||||
before **Confirm download**. “Job finished” refers to its selected scope, not verified
|
||||
whole-series completeness. See the scraper service and library format contracts below.
|
||||
|
||||
| Document | For |
|
||||
|----------|-----|
|
||||
| [docs/library-format.md](docs/library-format.md) | The folder format: naming rules, page tags, metadata fields, versions, extras, what survives a rename |
|
||||
| [docs/import.md](docs/import.md) | How the library is scanned and indexed, caching, change detection and notifications, troubleshooting |
|
||||
| [docs/architecture.md](docs/architecture.md) | Components, data model, API, security, the reader's layout engine, deployment, design decisions |
|
||||
| [docs/local-deployment.md](docs/local-deployment.md) | Current local Compose deployment, Tailscale access, persistent state, checks, restart, and recovery |
|
||||
| [docs/settings-navigation.md](docs/settings-navigation.md) | User/Admin settings sections, scraper tabs, routes and UI checks |
|
||||
| [docs/backup.md](docs/backup.md) | Built-in scheduled backup of `data/`, what it contains, retention and the restore procedure |
|
||||
| [docs/reading-insights-concept.md](docs/reading-insights-concept.md) | Active-time presentation, private summaries, deletion/export controls, and the future journal design |
|
||||
| [docs/deployment.md](docs/deployment.md) | Docker deployment: setup, configuration, HTTPS, operation (`deploy/circus.sh`), updates, backups, troubleshooting |
|
||||
| [docs/library-format.md](docs/library-format.md) | The folder format: naming rules, page tags, metadata fields, chapter dates, versions, extras, what survives a rename |
|
||||
| [docs/import.md](docs/import.md) | How the library is scanned and indexed, caching, change detection and notifications |
|
||||
| [docs/architecture.md](docs/architecture.md) | Components, data model, API, security, the reader's layout engine, design decisions |
|
||||
| [docs/scraper-service.md](docs/scraper-service.md) | The managed download queue, private API and adapter contract |
|
||||
| [docs/scraper-guide.md](docs/scraper-guide.md) | Rule book (agent-readable) for scrapers and other tools that write into the library |
|
||||
| [scraper/AGENT_RUNBOOK.md](scraper/AGENT_RUNBOOK.md) | Conservative scraper workflow and host-specific procedures for automated agents |
|
||||
| [scraper/AGENT_RUNBOOK.md](scraper/AGENT_RUNBOOK.md) | Conservative scraper workflow, rate limits and stop conditions for automated agents |
|
||||
| [docs/source-modules.md](docs/source-modules.md) | Adding a new source to the scraper |
|
||||
| [docs/peering.md](docs/peering.md) | Optional server-to-server sharing over mutual TLS |
|
||||
| [docs/backup.md](docs/backup.md) | Built-in scheduled backup of `data/`, what it contains, retention and restore |
|
||||
| [docs/settings-navigation.md](docs/settings-navigation.md) | User/Admin settings sections, scraper tabs and routes |
|
||||
| [docs/reading-insights-concept.md](docs/reading-insights-concept.md) | Active-time presentation, private summaries, export and deletion |
|
||||
| [docs/resilience.md](docs/resilience.md) | Supervisors, automatic recovery limits and incidents |
|
||||
| [AGENTS.md](AGENTS.md) | Working on the code: layout, tests, rules for contributors and coding agents |
|
||||
|
||||
Unnumbered MangaDex pilots and one-shots use stable `special-mangadex-<chapter-uuid>` chapter keys and
|
||||
human labels, not invented numbers. In Settings → Admin → Scraper, **Inspect & queue** shows the source
|
||||
edition before **Confirm download**. "Job finished" refers to its selected scope, not verified whole-series
|
||||
completeness.
|
||||
|
||||
---
|
||||
|
||||
@@ -134,6 +142,7 @@ whole-series completeness. See the scraper service and library format contracts
|
||||
- **Flow page gap:** Settings → Reading lets each user keep the normal separator between Flow pages or remove it for continuous webtoon artwork and text that crosses image edges.
|
||||
- **Page tone:** the reader's **Tone** button cycles Original → Inverted → Sepia → Dim → Extra dim in Flow, Page and Book. It changes only the artwork, never the controls or site theme. The choice is saved per user; it can also be selected in Settings → User → Reading.
|
||||
- **Active Reading Clock:** opt in under Settings → User → Reading time & privacy. No scrolling-speed limit; a visit counts at 60% viewed-page coverage and saves on exit. Pauses in the background or after 30 seconds without reader activity. Tap the clock for this visit's time, coverage, qualification and loading wait.
|
||||
- **Continue where you left off:** the first line of the library content (below the search, filters and tags) is one row of the five manga you read most recently and have not finished (same tiles as the grid; they stay in the grid too). It hides while a search is typed, a status filter is set, or the library is sorted by *Recently read*; tags and collection grouping do not affect it. Turn it off under Settings → User → Reading → *Library page*.
|
||||
- **Your Reading Story:** private active-time summaries appear on the library and manga pages when history exists. Chapter rows show the last timed read (60%-coverage visit), rereads remain visible as lifetime time, and Settings lets each user hide, export, or delete their own history. No leaderboards or speed ranking are used.
|
||||
- **Tap / click zones:**
|
||||
- Tapping **beside the pages** always shows or hides the controls.
|
||||
@@ -141,8 +150,9 @@ whole-series completeness. See the scraper service and library format contracts
|
||||
- In Flow: the bottom part of the screen goes forward, the top part goes back.
|
||||
- The middle of the page also toggles the controls (useful when a zoomed page fills the screen).
|
||||
- A horizontal swipe also turns pages on touch screens.
|
||||
- Keys: arrows, Space / PageDown, `m` = controls, `f` = fullscreen.
|
||||
- **Controls stay out of the way:** they hide as soon as you turn a page or scroll, and come back only with a tap (middle of the page or beside it). Scrolling up never brings them back. The first time they hide, a short hint says so.
|
||||
- Keys: arrows, Space / PageDown, `i` = controls, `m` = save a moment, `f` = fullscreen, `?` = all shortcuts (also under Settings → User → Keyboard shortcuts).
|
||||
- **Moments:** the bookmark button in the top bar (or `m`) saves the page you are on (in Book mode the spread) as a moment. A picture that runs over several pages can take up to four: *+ Previous page* / *+ Next page* add them, stacked in Flow and side by side in Page and Book. Drag over the preview to keep only a region (pages it doesn't touch are left out), and pick exactly one category: Twist, Hot, Awesome, Cute, Funny or Feels, plus an optional note. The button is filled on pages that already have one. Each manga page shows *Your moments* as a strip of crops, sorted by story or by when you saved them; click one to step through them, see the full page, edit, delete or jump into the reader at that page. Moments are private to each user.
|
||||
- **Controls stay out of the way:** they hide as soon as you turn a page or scroll, and come back only with a tap (middle of the page or beside it) or `i`. Scrolling up never brings them back. The first time they hide, a short hint says so.
|
||||
- **Controls:** the top bar has a chapter picker with every chapter of the manga, grouped by book, with ✓ for read ones. The bottom bar has the page slider, the mode switch, Shift, zoom, the version picker and previous/next chapter.
|
||||
- **Loading:** the next pages (next 3 in Page mode, next 2 spreads in Book mode) are downloaded and decoded in advance, so page turns don't flash. A page that still isn't there shows a spinner. On phones, "Add to home screen" opens the app without browser bars, and the ⛶ button (where supported) goes fullscreen.
|
||||
- **Progress** is saved per user and chapter version: the last page, whether the chapter is done, and book-mode shifts. The manga page offers *Continue*. Tap a chapter's status to mark it read/unread.
|
||||
@@ -156,5 +166,12 @@ whole-series completeness. See the scraper service and library format contracts
|
||||
server.js HTTP server, accounts, progress, votes, notifications (JSON file storage)
|
||||
library.js folder scanner + image size reader
|
||||
public/ the web pages (library, manga, reader, settings, login)
|
||||
tools/ demo library generator (pure Node, draws PNGs itself)
|
||||
scraper/ download queue daemon, source adapters, publisher
|
||||
peering/ optional server-to-server sharing
|
||||
deploy/ Docker operations helper (circus.sh) and HTTPS with Caddy
|
||||
tools/ demo library generator, maintenance and test tools
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT, see [LICENSE](LICENSE). Manga you read or download with it are not covered by this license and belong to their rights holders.
|
||||
@@ -0,0 +1,109 @@
|
||||
'use strict';
|
||||
// Chapter dates (BL-0047). chapter.json keeps three independent dates:
|
||||
// added when the chapter entered this library (kept on replace)
|
||||
// x-dates.source when the source site uploaded it; batch=true marks a bulk import, not a release
|
||||
// x-dates.release the official public release (filled by a metadata job; precision "day")
|
||||
// x-dates.ecosystem when it first entered any Manga Circus library; travels over peering, earliest wins
|
||||
// The reader shows one release date per chapter by tier: official > source (not batch) > ecosystem > library.
|
||||
|
||||
const BATCH_SIZE = 3; // this many uploads ...
|
||||
const BATCH_GAP_MS = 10 * 60e3; // ... each within 10 minutes of the next form a bulk import
|
||||
|
||||
const iso = value => {
|
||||
const time = typeof value === 'number' ? value : Date.parse(value);
|
||||
return Number.isFinite(time) && time > 0 ? new Date(time).toISOString() : null;
|
||||
};
|
||||
const text = (value, max = 120) => (typeof value === 'string' ? value.trim().slice(0, max) : '');
|
||||
|
||||
// Returns one flag per item: true when its upload time belongs to a bulk import, null without a time.
|
||||
function batchFlags(times) {
|
||||
const order = times.map((value, index) => ({ index, time: Date.parse(value) }))
|
||||
.filter(item => Number.isFinite(item.time)).sort((a, b) => a.time - b.time);
|
||||
const flags = times.map(() => null);
|
||||
for (let start = 0; start < order.length;) {
|
||||
let end = start + 1;
|
||||
while (end < order.length && order[end].time - order[end - 1].time <= BATCH_GAP_MS) end++;
|
||||
for (let i = start; i < end; i++) flags[order[i].index] = end - start >= BATCH_SIZE;
|
||||
start = end;
|
||||
}
|
||||
return flags;
|
||||
}
|
||||
|
||||
function sourceDate(at, name, batch) {
|
||||
const value = iso(at);
|
||||
return value ? { at: value, name: text(name), batch: !!batch } : null;
|
||||
}
|
||||
|
||||
function cleanSource(value) {
|
||||
return value && typeof value === 'object' ? sourceDate(value.at, value.name, value.batch) : null;
|
||||
}
|
||||
function cleanRelease(value) {
|
||||
if (!value || typeof value !== 'object' || !iso(value.at)) return null;
|
||||
const precision = ['day', 'month', 'year'].includes(value.precision) ? value.precision : 'day';
|
||||
return { at: String(value.at).slice(0, 30), precision, source: text(value.source) };
|
||||
}
|
||||
function cleanEcosystem(value) {
|
||||
if (!value || typeof value !== 'object' || !iso(value.at)) return null;
|
||||
return { at: iso(value.at), via: text(value.via) || 'this server' };
|
||||
}
|
||||
|
||||
// chapter.json fields for a (re)published chapter. A prior "added" and an earlier ecosystem
|
||||
// date always survive; a peer's ecosystem claim may lower the date but never raise it.
|
||||
function publishedDates(priorMeta = {}, incoming = {}, now = Date.now()) {
|
||||
const prior = priorMeta['x-dates'] && typeof priorMeta['x-dates'] === 'object' ? priorMeta['x-dates'] : {};
|
||||
const added = iso(priorMeta.added) || iso(incoming.added) || iso(now);
|
||||
const source = cleanSource(incoming.source) || cleanSource(prior.source);
|
||||
const release = cleanRelease(incoming.release) || cleanRelease(prior.release);
|
||||
let claim = cleanEcosystem(incoming.ecosystem);
|
||||
if (claim) {
|
||||
// Nobody can have had the chapter before the source uploaded it, or in the future.
|
||||
const floor = source ? Date.parse(source.at) : 0, time = Math.min(Date.parse(claim.at), now);
|
||||
claim = { ...claim, at: iso(Math.max(time, floor)) };
|
||||
}
|
||||
const candidates = [cleanEcosystem(prior.ecosystem), claim, { at: added, via: 'this server' }].filter(Boolean);
|
||||
const ecosystem = candidates.reduce((best, item) => (Date.parse(item.at) < Date.parse(best.at) ? item : best));
|
||||
return { added, 'x-dates': { version: 1, ...(source ? { source } : {}), ...(release ? { release } : {}), ecosystem } };
|
||||
}
|
||||
|
||||
// Dates of one library chapter from its versions ({ added: ms, meta: chapter.json }).
|
||||
function chapterDates(versions) {
|
||||
const known = {};
|
||||
const keep = (slot, time, value) => {
|
||||
if (Number.isFinite(time) && (!known[slot] || time < known[slot].time)) known[slot] = { time, ...value };
|
||||
};
|
||||
for (const version of versions) {
|
||||
const dates = (version.meta && version.meta['x-dates']) || {};
|
||||
const release = cleanRelease(dates.release), source = cleanSource(dates.source), ecosystem = cleanEcosystem(dates.ecosystem);
|
||||
if (release) keep('official', Date.parse(release.at), { at: release.at, precision: release.precision, name: release.source });
|
||||
if (source) keep(source.batch ? 'batch' : 'source', Date.parse(source.at), { at: source.at, name: source.name });
|
||||
if (ecosystem) keep('ecosystem', Date.parse(ecosystem.at), { at: ecosystem.at, name: ecosystem.via });
|
||||
if (Number.isFinite(version.added)) keep('library', version.added, { at: iso(version.added) });
|
||||
}
|
||||
const tier = ['official', 'source', 'ecosystem', 'library'].find(slot => known[slot]) || null;
|
||||
const out = {};
|
||||
for (const [slot, { time, ...value }] of Object.entries(known)) out[slot] = value;
|
||||
return { tier, release: tier ? out[tier] : null, known: out };
|
||||
}
|
||||
|
||||
// Ecosystem dates offered to peers: the earliest per chapter number, falling back to "added"
|
||||
// for chapters published before x-dates existed (this server was then the one that had it).
|
||||
function ecosystemOf(meta, fallbackMs) {
|
||||
const dates = (meta && meta['x-dates']) || {};
|
||||
return cleanEcosystem(dates.ecosystem)?.at || iso(meta && meta.added) || iso(fallbackMs);
|
||||
}
|
||||
// What a gateway sends with a chapter: only cleaned, non-identifying date fields.
|
||||
function peerDates(meta, fallbackMs) {
|
||||
const dates = (meta && meta['x-dates']) || {};
|
||||
const source = cleanSource(dates.source), release = cleanRelease(dates.release), ecosystem = ecosystemOf(meta, fallbackMs);
|
||||
return { ...(ecosystem ? { ecosystem } : {}), ...(source ? { source } : {}), ...(release ? { release } : {}) };
|
||||
}
|
||||
// Incoming peer dates in publishedDates() shape.
|
||||
function fromPeer(value, alias) {
|
||||
if (!value || typeof value !== 'object') return {};
|
||||
const ecosystem = iso(value.ecosystem);
|
||||
return { ...(ecosystem ? { ecosystem: { at: ecosystem, via: `peer ${text(alias, 40)}`.trim() } } : {}),
|
||||
...(cleanSource(value.source) ? { source: cleanSource(value.source) } : {}),
|
||||
...(cleanRelease(value.release) ? { release: cleanRelease(value.release) } : {}) };
|
||||
}
|
||||
|
||||
module.exports = { BATCH_SIZE, BATCH_GAP_MS, batchFlags, sourceDate, publishedDates, chapterDates, ecosystemOf, peerDates, fromPeer };
|
||||
@@ -0,0 +1,5 @@
|
||||
# Used by deploy/compose.https.yml. Caddy obtains and renews the certificate on its own.
|
||||
{$MANGACIRCUS_DOMAIN} {
|
||||
encode zstd gzip
|
||||
reverse_proxy reader:8080
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
#!/bin/bash
|
||||
# Operate a MangaCircus Docker Compose deployment. Run it from anywhere; it works in the repository
|
||||
# folder that contains it. Needs Docker with Compose, nothing else on the host (no Node).
|
||||
# Extra Compose files (e.g. compose.lan.yml) come from COMPOSE_FILE in .env, like plain `docker compose`.
|
||||
#
|
||||
# status services, reader health, scraper queue
|
||||
# up reader first (waits for health), then the scraper; resumes the queue if `down` paused it
|
||||
# down pause the queue, wait for the running chapter (<= 15 min), stop all services
|
||||
# restart [service] down + up, or restart one service
|
||||
# logs [service] [lines] recent logs (default: all services, 100 lines)
|
||||
# pause | resume pause or resume the scraper queue
|
||||
# update git pull --ff-only, rebuild the image, recreate safely (queue paused meanwhile)
|
||||
# rescan make the reader re-read every chapter.json (full reconciliation) without an admin login
|
||||
# restore-test restore the newest built-in backup into a throwaway reader and check it starts
|
||||
#
|
||||
# Safety: never `docker compose down -v`. Only the scraper writes the library; only the reader writes users.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$(readlink -f "$0")")/.."
|
||||
log() { echo "[circus] $*"; }
|
||||
[ -f .env ] || { log "no .env here; see docs/deployment.md" >&2; exit 1; }
|
||||
env_value() { sed -n "s/^$1=//p" .env | tail -n 1 | sed -e 's/^["'\'']//' -e 's/["'\'']$//'; }
|
||||
running() { [ -n "$(docker compose ps -q --status running "$1" 2>/dev/null)" ]; }
|
||||
|
||||
# Scraper control through the reader container: the token never leaves it and nothing secret is printed.
|
||||
control() {
|
||||
docker compose exec -T -e ACTION="$1" reader node -e '
|
||||
const {secret}=require("./runtime/config");
|
||||
(async()=>{
|
||||
const a=process.env.ACTION, state=a==="state";
|
||||
const r=await fetch(process.env.SCRAPER_URL+(state?"/api/state":"/api/control"),{method:state?"GET":"POST",
|
||||
signal:AbortSignal.timeout(20000),headers:{authorization:"Bearer "+secret("SCRAPER_TOKEN"),"content-type":"application/json"},
|
||||
...(state?{}:{body:JSON.stringify({action:a})})});
|
||||
if(!r.ok)throw new Error("scraper HTTP "+r.status);
|
||||
const s=await r.json();
|
||||
console.log(JSON.stringify({status:s.status,paused:s.paused,sourcePaused:s.sourcePaused,active:(s.active||[]).length,
|
||||
queue:(s.queue||[]).length,nextDiscoveryAt:s.nextDiscoveryAt,library:s.libraryStatus&&{state:s.libraryStatus.state,error:s.libraryStatus.error}}));
|
||||
})().catch(e=>{console.error(e.message);process.exitCode=1;});'
|
||||
}
|
||||
field() { sed -n "s/.*\"$1\":\([^,}]*\).*/\1/p"; }
|
||||
health() { docker compose exec -T reader node -e 'fetch("http://127.0.0.1:8080/health").then(r=>r.text()).then(t=>console.log(t))' 2>/dev/null; }
|
||||
wait_reader() {
|
||||
for _ in $(seq 1 60); do health | grep -q '"ok":true' && return 0; sleep 5; done
|
||||
log "reader not healthy after 5 minutes" >&2; return 1
|
||||
}
|
||||
# Pause the queue and wait for the running chapter; prints whether the queue was paused before.
|
||||
pause_wait() {
|
||||
local was=true
|
||||
if running scraper && running reader; then
|
||||
was=$(control state | field paused)
|
||||
[ "$was" = true ] || control pause >/dev/null
|
||||
for _ in $(seq 1 90); do [ "$(control state | field active)" = 0 ] && break; sleep 10; done
|
||||
fi
|
||||
echo "$was"
|
||||
}
|
||||
resume_if() {
|
||||
[ "$1" = false ] || { log "queue stays paused"; return 0; }
|
||||
for _ in $(seq 1 30); do control resume >/dev/null 2>&1 && { log "queue resumed"; return 0; }; sleep 5; done
|
||||
log "could not resume the queue" >&2; return 1
|
||||
}
|
||||
start_all() {
|
||||
docker compose up -d reader
|
||||
wait_reader
|
||||
docker compose up -d --remove-orphans
|
||||
sleep 15
|
||||
}
|
||||
|
||||
cmd=${1:-status}; shift || true
|
||||
case "$cmd" in
|
||||
status)
|
||||
docker compose ps --format 'table {{.Service}}\t{{.Status}}\t{{.Ports}}'
|
||||
running reader && { health || echo "reader: health check failed"; } || echo "reader: not running"
|
||||
running scraper && running reader && control state || echo "scraper: not running"
|
||||
;;
|
||||
up)
|
||||
start_all
|
||||
# Only a queue that `down` paused is resumed; without that marker the queue is left as it is.
|
||||
if [ -f .queue-was-paused ]; then was=$(cat .queue-was-paused); rm -f .queue-was-paused; resume_if "$was"; fi
|
||||
docker compose ps --format 'table {{.Service}}\t{{.Status}}'
|
||||
;;
|
||||
down)
|
||||
was=$(pause_wait); echo "$was" > .queue-was-paused
|
||||
docker compose stop
|
||||
log "stopped (queue was paused before: $was)"
|
||||
;;
|
||||
restart)
|
||||
if [ -n "${1:-}" ]; then docker compose restart "$1"; [ "$1" != reader ] || wait_reader
|
||||
else "$0" down && "$0" up; fi
|
||||
;;
|
||||
logs)
|
||||
docker compose logs --no-color --tail "${2:-100}" ${1:+"$1"}
|
||||
;;
|
||||
pause|resume)
|
||||
control "$cmd"
|
||||
;;
|
||||
update)
|
||||
git pull --ff-only
|
||||
VCS_REF=$(git rev-parse --short HEAD) docker compose build
|
||||
was=$(pause_wait)
|
||||
start_all # recreates the containers whose image or configuration changed
|
||||
resume_if "$was"
|
||||
log "now running $(git log -1 --format='%h %s')"
|
||||
;;
|
||||
rescan)
|
||||
# The reader re-reads in-place chapter.json edits only on a full reconciliation. Marking the saved
|
||||
# index as overdue makes it run one within a minute while it keeps serving the current index.
|
||||
data=$(env_value DATA_PATH); data=${data:-./data}
|
||||
image=$(docker compose images -q reader | head -n 1)
|
||||
[ -f "$data/library-index.json" ] || { log "no saved index yet; the reader scans on its own" >&2; exit 1; }
|
||||
docker compose stop reader
|
||||
docker run --rm --user 1000:1000 -v "$(readlink -f "$data")":/data "$image" node -e '
|
||||
const fs=require("fs"),f="/data/library-index.json",j=JSON.parse(fs.readFileSync(f,"utf8"));
|
||||
j.fullAt=1; fs.writeFileSync(f+".tmp",JSON.stringify(j)); fs.renameSync(f+".tmp",f);'
|
||||
docker compose up -d reader
|
||||
wait_reader
|
||||
log "full reconciliation scheduled; it logs \"[scan] full: …\" when done (docker compose logs reader)"
|
||||
;;
|
||||
restore-test)
|
||||
backup=$(env_value BACKUP_PATH); library=$(env_value LIBRARY_PATH); library=${library:-./library}
|
||||
[ -n "$backup" ] || { log "BACKUP_PATH is not set in .env" >&2; exit 1; }
|
||||
archive=$(ls -1t "$backup"/circus-*.tar.gz 2>/dev/null | head -n 1)
|
||||
[ -n "$archive" ] || { log "no circus-*.tar.gz in $backup" >&2; exit 1; }
|
||||
image=$(docker compose images -q reader | head -n 1)
|
||||
tmp=$(mktemp -d); mkdir "$tmp/data"
|
||||
trap 'docker rm -f circus-restore-test >/dev/null 2>&1 || true; rm -rf "$tmp"' EXIT
|
||||
tar -C "$tmp/data" -xzf "$archive"
|
||||
chmod -R a+rwX "$tmp/data"
|
||||
log "archive: $archive"
|
||||
docker run -d --name circus-restore-test --network none --user 1000:1000 --read-only --tmpfs /tmp \
|
||||
-v "$(readlink -f "$library")":/library:ro -v "$tmp/data":/data -e DATA=/data -e LIBRARY=/library \
|
||||
-e SIGNUP=closed -e SCAN_INTERVAL=0 "$image" node server.js >/dev/null
|
||||
out=""
|
||||
for _ in $(seq 1 30); do
|
||||
out=$(docker exec circus-restore-test node -e 'fetch("http://127.0.0.1:8080/api/auth").then(r=>r.text()).then(console.log)' 2>/dev/null || true)
|
||||
[ -n "$out" ] && break; sleep 2
|
||||
done
|
||||
echo "auth: $out"
|
||||
echo "$out" | grep -q '"firstRun":false' && echo "RESTORE TEST OK" || { echo "RESTORE TEST FAILED" >&2; exit 1; }
|
||||
;;
|
||||
*)
|
||||
sed -n '2,17p' "$0"; exit 2
|
||||
;;
|
||||
esac
|
||||
@@ -0,0 +1,27 @@
|
||||
# HTTPS in front of the reader with Caddy and automatic Let's Encrypt certificates.
|
||||
# Needs a DNS name for this host (MANGACIRCUS_DOMAIN in .env) and ports 80 and 443 reachable
|
||||
# from the internet. Use it with the main file, e.g. in .env:
|
||||
# COMPOSE_FILE=docker-compose.yml:deploy/compose.https.yml
|
||||
# Already running a reverse proxy? Skip this file and see docs/deployment.md, "Your own proxy".
|
||||
services:
|
||||
reader:
|
||||
ports: !reset []
|
||||
environment:
|
||||
COOKIE_SECURE: "1"
|
||||
|
||||
caddy:
|
||||
image: caddy:2.8.4-alpine
|
||||
restart: unless-stopped
|
||||
ports: ["80:80", "443:443", "443:443/udp"]
|
||||
environment:
|
||||
MANGACIRCUS_DOMAIN: ${MANGACIRCUS_DOMAIN:?Set MANGACIRCUS_DOMAIN in .env}
|
||||
volumes:
|
||||
- ./deploy/Caddyfile:/etc/caddy/Caddyfile:ro
|
||||
- caddy-data:/data
|
||||
- caddy-config:/config
|
||||
networks: [egress]
|
||||
depends_on: [reader]
|
||||
|
||||
volumes:
|
||||
caddy-data: {}
|
||||
caddy-config: {}
|
||||
@@ -0,0 +1,65 @@
|
||||
'use strict';
|
||||
|
||||
// Anchors are the titles a user has shown taste for; their tags steer the wider AniList pool.
|
||||
const ANCHOR_WEIGHTS = { flame: 5, good: 3, interested: 4, maybe: 2, read: 3, owned: 1 };
|
||||
const MAX_ANCHORS = 15;
|
||||
const MIN_TAG_RANK = 60;
|
||||
const MAX_TOP_TAGS = 3;
|
||||
|
||||
function anchorsFor({ user, db, library, greatest }) {
|
||||
if (!user?.settings?.discoveryEnabled) return [];
|
||||
const byId = new Map();
|
||||
const add = (anilistId, source, title) => {
|
||||
if (!Number.isSafeInteger(anilistId) || anilistId <= 0) return;
|
||||
const weight = ANCHOR_WEIGHTS[source];
|
||||
const prior = byId.get(anilistId);
|
||||
if (!prior || weight > prior.weight) byId.set(anilistId, { anilistId, weight, source, title: title ?? null });
|
||||
};
|
||||
|
||||
const ratings = db?.greatestRatings?.[user.name] || {};
|
||||
for (const entry of greatest?.entries || []) {
|
||||
const vote = ratings[entry.id];
|
||||
if (vote === 'flame' || vote === 'good') add(Number(entry.metadata?.id), vote, entry.title);
|
||||
}
|
||||
|
||||
const feedback = db?.discoveryFeedback?.[user.name] || {};
|
||||
for (const [key, value] of Object.entries(feedback)) {
|
||||
const match = /^anilist:(\d+)$/.exec(key);
|
||||
if (match && (value === 'interested' || value === 'maybe')) add(Number(match[1]), value, null);
|
||||
}
|
||||
|
||||
const progress = db?.progress?.[user.name] || {};
|
||||
const reading = Boolean(user.settings.discoveryReadingActivity);
|
||||
for (const manga of library || []) {
|
||||
const id = Number(manga.meta?.['x-anilist']?.id);
|
||||
if (!id) continue;
|
||||
const chapters = progress[manga.id];
|
||||
const done = reading && chapters && Object.values(chapters).some(chapter => chapter?.done);
|
||||
add(id, done ? 'read' : 'owned', manga.title);
|
||||
}
|
||||
|
||||
return [...byId.values()]
|
||||
.sort((a, b) => b.weight - a.weight || a.anilistId - b.anilistId)
|
||||
.slice(0, MAX_ANCHORS);
|
||||
}
|
||||
|
||||
function topTagsFor(anchors, recordOf, { mature = false } = {}) {
|
||||
const scores = new Map();
|
||||
for (const anchor of anchors) {
|
||||
const record = recordOf(anchor.anilistId);
|
||||
for (const tag of Array.isArray(record?.tags) ? record.tags : []) {
|
||||
// A tag without a rank is no evidence of taste, so it must not pass the rank filter.
|
||||
if (!tag || typeof tag !== 'object' || typeof tag.name !== 'string') continue;
|
||||
if (!(tag.rank >= MIN_TAG_RANK) || (tag.isAdult && !mature)) continue;
|
||||
const entry = scores.get(tag.name) || { name: tag.name, kind: tag.kind, score: 0 };
|
||||
entry.score += anchor.weight;
|
||||
scores.set(tag.name, entry);
|
||||
}
|
||||
}
|
||||
return [...scores.values()]
|
||||
.sort((a, b) => b.score - a.score || (a.name < b.name ? -1 : a.name > b.name ? 1 : 0))
|
||||
.slice(0, MAX_TOP_TAGS)
|
||||
.map(({ name, kind }) => ({ name, kind }));
|
||||
}
|
||||
|
||||
module.exports = { ANCHOR_WEIGHTS, MAX_ANCHORS, anchorsFor, topTagsFor };
|
||||
@@ -0,0 +1,165 @@
|
||||
'use strict';
|
||||
|
||||
// Polite AniList GraphQL client for discovery seeds. Sequential, rate limited, stops on access controls.
|
||||
|
||||
const STAFF_ROLE = /^(?:original story|story(?: & art)?|art|illustration)$/i;
|
||||
const USER_AGENT = 'MangaCircus-Discovery/1.0';
|
||||
|
||||
// Stop: the whole run should end. Skip: this one request is lost, carry on.
|
||||
class AniListStop extends Error {
|
||||
constructor(reason, message) { super(message || `AniList stop: ${reason}`); this.name = 'AniListStop'; this.reason = reason; }
|
||||
}
|
||||
class AniListSkip extends Error {
|
||||
constructor(reason, message) { super(message || `AniList skip: ${reason}`); this.name = 'AniListSkip'; this.reason = reason; }
|
||||
}
|
||||
|
||||
function createLimiter({ perMinute = 25, now = Date.now, sleep = ms => new Promise(resolve => setTimeout(resolve, ms)) } = {}) {
|
||||
const interval = 60000 / perMinute;
|
||||
let nextAllowed = 0;
|
||||
let tail = Promise.resolve();
|
||||
// Chained so concurrent callers queue up instead of all reading the same clock.
|
||||
function wait() {
|
||||
const turn = tail.then(async () => {
|
||||
await sleep(Math.max(0, nextAllowed - now()));
|
||||
nextAllowed = now() + interval;
|
||||
});
|
||||
tail = turn.catch(() => {});
|
||||
return turn;
|
||||
}
|
||||
return { wait };
|
||||
}
|
||||
|
||||
const invalid = message => new AniListSkip('invalid', `AniList response is malformed: ${message}`);
|
||||
const isObject = value => !!value && typeof value === 'object' && !Array.isArray(value);
|
||||
const optionalArray = (value, what) => { if (value != null && !Array.isArray(value)) throw invalid(what); };
|
||||
|
||||
// Malformed input is a Skip('invalid'), never a plain TypeError from deep inside the mapping.
|
||||
function assertShape(media) {
|
||||
if (media.title != null && !isObject(media.title)) throw invalid('title');
|
||||
for (const key of ['synonyms', 'genres', 'tags']) optionalArray(media[key], key);
|
||||
if (media.staff != null && !isObject(media.staff)) throw invalid('staff');
|
||||
optionalArray(media.staff?.edges, 'staff edges');
|
||||
if ((media.tags || []).some(tag => !isObject(tag))) throw invalid('tags');
|
||||
if ((media.staff?.edges || []).some(edge => !isObject(edge) || (edge.node != null && !isObject(edge.node)))) throw invalid('staff edges');
|
||||
}
|
||||
|
||||
function normalizeMedia(media) {
|
||||
if (!isObject(media) || !Number.isInteger(media.id)) throw invalid('media without integer id');
|
||||
if (media.format !== 'MANGA' && media.format !== 'ONE_SHOT') return null;
|
||||
assertShape(media);
|
||||
const title = media.title || {};
|
||||
const names = [title.english || title.romaji || title.native, title.romaji, title.native, ...(media.synonyms || [])].filter(Boolean);
|
||||
return {
|
||||
anilistId: media.id,
|
||||
title: names[0] || '',
|
||||
altTitles: [...new Set(names.slice(1))].filter(name => name !== names[0]),
|
||||
format: media.format,
|
||||
isAdult: !!media.isAdult,
|
||||
status: media.status === 'RELEASING' ? 'ongoing' : media.status === 'FINISHED' ? 'completed' : 'unknown',
|
||||
year: media.startDate?.year ?? null,
|
||||
averageScore: media.averageScore ?? null,
|
||||
popularity: media.popularity ?? null,
|
||||
trending: media.trending ?? null,
|
||||
tags: [
|
||||
...(media.genres || []).map(name => ({ name, rank: 100, kind: 'genre', isAdult: false })),
|
||||
...(media.tags || []).map(tag => ({ name: tag.name, rank: tag.rank, kind: 'tag', isAdult: !!tag.isAdult }))],
|
||||
authors: (media.staff?.edges || [])
|
||||
// An author without an integer id could never match a creator seed.
|
||||
.filter(edge => STAFF_ROLE.test(edge?.role || '') && edge.node?.name?.full && Number.isInteger(edge.node.id))
|
||||
.map(edge => ({ staffId: edge.node.id, name: edge.node.name.full, role: edge.role })),
|
||||
coverUrl: media.coverImage?.extraLarge || null,
|
||||
siteUrl: media.siteUrl || null,
|
||||
};
|
||||
}
|
||||
|
||||
const MEDIA_FIELDS = `fragment MediaFields on Media {
|
||||
id format isAdult title { romaji english native } synonyms status startDate { year }
|
||||
averageScore popularity trending genres tags { name rank isAdult }
|
||||
staff(perPage: 20) { edges { role node { id name { full } } } }
|
||||
coverImage { extraLarge } siteUrl
|
||||
}`;
|
||||
|
||||
const QUERIES = {
|
||||
title: `query ($id: Int) { Media(id: $id, type: MANGA) { ...MediaFields } } ${MEDIA_FIELDS}`,
|
||||
recs: `query ($id: Int) { Media(id: $id, type: MANGA) {
|
||||
recommendations(perPage: 25, sort: RATING_DESC) { nodes { rating mediaRecommendation { ...MediaFields } } } } } ${MEDIA_FIELDS}`,
|
||||
creator: `query ($id: Int) { Staff(id: $id) {
|
||||
staffMedia(type: MANGA, perPage: 25, sort: POPULARITY_DESC) { edges { staffRole node { ...MediaFields } } } } } ${MEDIA_FIELDS}`,
|
||||
tagTopGenre: `query ($genre_in: [String]) { Page(perPage: 25) {
|
||||
media(type: MANGA, genre_in: $genre_in, popularity_greater: 1000, sort: SCORE_DESC) { ...MediaFields } } } ${MEDIA_FIELDS}`,
|
||||
tagTopTag: `query ($tag_in: [String]) { Page(perPage: 25) {
|
||||
media(type: MANGA, tag_in: $tag_in, popularity_greater: 1000, sort: SCORE_DESC) { ...MediaFields } } } ${MEDIA_FIELDS}`,
|
||||
fresh: `query ($startDate_greater: FuzzyDateInt, $perPage: Int) { Page(perPage: $perPage) {
|
||||
media(type: MANGA, status: RELEASING, startDate_greater: $startDate_greater, sort: TRENDING_DESC) { ...MediaFields } } } ${MEDIA_FIELDS}`,
|
||||
};
|
||||
|
||||
const isTimeout = error => error && (error.name === 'TimeoutError' || error.name === 'AbortError');
|
||||
const need = (value, check = Array.isArray) => { if (!check(value)) throw new AniListSkip('invalid', 'AniList response is missing data'); return value; };
|
||||
|
||||
function createClient({ endpoint = process.env.DISCOVERY_ANILIST_URL || 'https://graphql.anilist.co', limiter = createLimiter(),
|
||||
fetchImpl = fetch, timeoutMs = 30000, today = () => new Date() } = {}) {
|
||||
async function post(query, variables) {
|
||||
await limiter.wait();
|
||||
let response, text;
|
||||
try {
|
||||
response = await fetchImpl(endpoint, {
|
||||
method: 'POST', headers: { 'content-type': 'application/json', 'user-agent': USER_AGENT },
|
||||
body: JSON.stringify({ query, variables }), signal: AbortSignal.timeout(timeoutMs),
|
||||
redirect: 'error' }); // never follow a redirect to another host
|
||||
if (response.status === 429) throw new AniListStop('rate-limited');
|
||||
if (response.status === 401 || response.status === 403) throw new AniListStop('forbidden');
|
||||
text = await response.text();
|
||||
} catch (error) {
|
||||
if (error instanceof AniListStop || error instanceof AniListSkip) throw error;
|
||||
if (isTimeout(error)) throw new AniListSkip('timeout');
|
||||
throw new AniListSkip('upstream', `AniList request failed: ${error.message}`);
|
||||
}
|
||||
// An HTML or non-JSON page means a bot-check in front of the API, even on a 5xx; never work around it.
|
||||
if (/text\/html/i.test(response.headers.get('content-type') || '')) throw new AniListStop('challenge');
|
||||
let body;
|
||||
try { body = JSON.parse(text); } catch { throw new AniListStop('challenge'); }
|
||||
if (response.status >= 500) throw new AniListSkip('upstream', `AniList HTTP ${response.status}`);
|
||||
if (!response.ok || !body || typeof body !== 'object') throw new AniListSkip('invalid', `AniList HTTP ${response.status}`);
|
||||
return body.data || {};
|
||||
}
|
||||
|
||||
// Every item is normalized (and so validated) before any is dropped: one malformed item invalidates the response.
|
||||
const mediaOf = data => need(data.Media, isObject);
|
||||
const entries = (list, pick) => need(list).map(item => {
|
||||
if (!isObject(item)) throw invalid('list item');
|
||||
const media = pick(item);
|
||||
return { item, media, record: normalizeMedia(media) };
|
||||
}).filter(entry => entry.record);
|
||||
|
||||
return {
|
||||
async title(id) {
|
||||
return normalizeMedia(mediaOf(await post(QUERIES.title, { id })));
|
||||
},
|
||||
async recs(id) {
|
||||
const nodes = mediaOf(await post(QUERIES.recs, { id })).recommendations?.nodes;
|
||||
// AniList returns a null mediaRecommendation when the recommended media was removed: valid, just drop it.
|
||||
const present = need(nodes).filter(node => !isObject(node) || node.mediaRecommendation !== null);
|
||||
return entries(present, node => node.mediaRecommendation).filter(({ item }) => item.rating > 0)
|
||||
.map(({ item, record }) => ({ anilistId: record.anilistId, rating: item.rating, record }));
|
||||
},
|
||||
async creator(staffId) {
|
||||
const data = await post(QUERIES.creator, { id: staffId });
|
||||
return entries(need(data.Staff, isObject).staffMedia?.edges, edge => edge.node)
|
||||
.map(({ item, record }) => ({ anilistId: record.anilistId, role: item.staffRole, record }));
|
||||
},
|
||||
async tagTop({ name, kind }) {
|
||||
const genre = kind === 'genre';
|
||||
const data = await post(genre ? QUERIES.tagTopGenre : QUERIES.tagTopTag, genre ? { genre_in: [name] } : { tag_in: [name] });
|
||||
return entries(data.Page?.media, media => media).map(({ record }) => ({ anilistId: record.anilistId, record }));
|
||||
},
|
||||
async fresh() {
|
||||
const date = today();
|
||||
// AniList FuzzyDateInt: YYYYMMDD, two years back.
|
||||
const startDate = (date.getUTCFullYear() - 2) * 10000 + (date.getUTCMonth() + 1) * 100 + date.getUTCDate();
|
||||
const data = await post(QUERIES.fresh, { startDate_greater: startDate, perPage: 50 });
|
||||
return entries(data.Page?.media, media => media).map(({ record }) => ({ anilistId: record.anilistId, record }));
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = { createLimiter, createClient, normalizeMedia, AniListStop, AniListSkip };
|
||||
@@ -0,0 +1,93 @@
|
||||
'use strict';
|
||||
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const crypto = require('node:crypto');
|
||||
|
||||
const DAY = 86400000;
|
||||
const KINDS = ['title', 'recs', 'creator', 'tagTop', 'fresh'];
|
||||
const TTL_MS = { title: 30 * DAY, recs: 30 * DAY, creator: 30 * DAY, tagTop: 7 * DAY, fresh: 7 * DAY };
|
||||
|
||||
function checkKind(kind) {
|
||||
if (!KINDS.includes(kind)) throw new TypeError(`Unknown discovery cache kind: ${kind}`);
|
||||
}
|
||||
|
||||
class DiscoveryCache {
|
||||
constructor(dir, { maxFiles = 5000, now = Date.now } = {}) {
|
||||
this.dir = path.resolve(dir);
|
||||
this.maxFiles = maxFiles;
|
||||
this.now = now;
|
||||
}
|
||||
// base64url keeps any key (slashes, dots, spaces) inside the kind folder.
|
||||
fileFor(kind, key) {
|
||||
checkKind(kind);
|
||||
return path.join(this.dir, kind, `${Buffer.from(String(key)).toString('base64url')}.json`);
|
||||
}
|
||||
readFile(file, kind) {
|
||||
let entry;
|
||||
try { entry = JSON.parse(fs.readFileSync(file, 'utf8')); }
|
||||
catch (error) {
|
||||
if (error.code === 'ENOENT') return null;
|
||||
entry = null;
|
||||
}
|
||||
const valid = entry && entry.schema === 1 && entry.kind === kind && typeof entry.fetchedAt === 'string'
|
||||
&& Number.isFinite(Date.parse(entry.fetchedAt)) && 'data' in entry;
|
||||
if (valid) return entry;
|
||||
// A bad file would otherwise be re-read on every lookup; drop it so the caller refetches.
|
||||
try { fs.rmSync(file, { force: true }); } catch {}
|
||||
return null;
|
||||
}
|
||||
get(kind, key) { return this.readFile(this.fileFor(kind, key), kind); }
|
||||
isFresh(kind, key) {
|
||||
const entry = this.get(kind, key);
|
||||
return !!entry && this.now() - Date.parse(entry.fetchedAt) < TTL_MS[kind];
|
||||
}
|
||||
put(kind, key, data) {
|
||||
const file = this.fileFor(kind, key);
|
||||
const entry = { schema: 1, kind, key, fetchedAt: new Date(this.now()).toISOString(), data };
|
||||
fs.mkdirSync(path.dirname(file), { recursive: true });
|
||||
const temp = `${file}.${crypto.randomBytes(6).toString('hex')}.tmp`;
|
||||
try {
|
||||
fs.writeFileSync(temp, JSON.stringify(entry), { mode: 0o600 });
|
||||
fs.renameSync(temp, file);
|
||||
} finally { try { fs.rmSync(temp, { force: true }); } catch {} }
|
||||
this.prune();
|
||||
return entry;
|
||||
}
|
||||
files(kind) {
|
||||
try {
|
||||
return fs.readdirSync(path.join(this.dir, kind)).filter((name) => name.endsWith('.json'))
|
||||
.map((name) => path.join(this.dir, kind, name));
|
||||
} catch { return []; }
|
||||
}
|
||||
list(kind) {
|
||||
checkKind(kind);
|
||||
return this.files(kind).map((file) => this.readFile(file, kind)).filter(Boolean);
|
||||
}
|
||||
stats() {
|
||||
let files = 0, bytes = 0;
|
||||
for (const kind of KINDS) for (const file of this.files(kind)) {
|
||||
try { bytes += fs.statSync(file).size; files++; } catch {}
|
||||
}
|
||||
return { files, bytes };
|
||||
}
|
||||
// Oldest entries go first so the cache never grows past maxFiles. Every put() calls this, so
|
||||
// under the cap it only counts directory entries; files are read only when something must go.
|
||||
prune() {
|
||||
const byKind = KINDS.map(kind => [kind, this.files(kind)]);
|
||||
if (byKind.reduce((sum, [, files]) => sum + files.length, 0) <= this.maxFiles) return;
|
||||
const all = [];
|
||||
for (const [kind, files] of byKind) for (const file of files) {
|
||||
// readFile drops corrupt and schema-mismatched files itself.
|
||||
const entry = this.readFile(file, kind);
|
||||
if (entry) all.push({ file, at: Date.parse(entry.fetchedAt) });
|
||||
}
|
||||
if (all.length <= this.maxFiles) return;
|
||||
all.sort((a, b) => a.at - b.at);
|
||||
for (const { file } of all.slice(0, all.length - this.maxFiles)) {
|
||||
try { fs.rmSync(file, { force: true }); } catch {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { KINDS, TTL_MS, DiscoveryCache };
|
||||
+7
-2
@@ -5,6 +5,7 @@ const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { loadCatalog } = require('../greatest');
|
||||
const { generate } = require('./engine');
|
||||
const { DiscoveryCache } = require('./cache');
|
||||
|
||||
if (!process.argv.includes('--dry-run')) {
|
||||
console.error('Usage: node discovery/cli.js run --dry-run [--data DIR] [--catalog FILE] [--show]');
|
||||
@@ -17,8 +18,12 @@ let library = [];
|
||||
try { library = JSON.parse(fs.readFileSync(path.join(data, 'library-index.json'), 'utf8')).mangas || []; }
|
||||
catch (error) { if (error.code !== 'ENOENT') throw error; }
|
||||
const greatest = loadCatalog(arg('--catalog', path.join(__dirname, '..', 'greatest', 'catalog.json')));
|
||||
const result = generate({ db, library, greatest, cacheDir: path.join(data, 'scraper', 'metadata', 'anilist') });
|
||||
const summary = { examined: result.examined, candidates: Object.keys(result.candidates).length,
|
||||
// The harvest cache is only read here (get/list, never put); note get/list still drop a corrupt entry file.
|
||||
const harvestDir = path.join(data, 'discovery-cache');
|
||||
const harvestCache = fs.existsSync(harvestDir) ? new DiscoveryCache(harvestDir) : null;
|
||||
const result = generate({ db, library, greatest, cacheDir: path.join(data, 'scraper', 'metadata', 'anilist'), harvestCache });
|
||||
// candidates: the ones some feed shows (what a run would publish); examined/clipped: the whole pool.
|
||||
const summary = { examined: result.examined, clipped: result.clipped, candidates: Object.keys(result.candidates).length,
|
||||
optedInUsers: Object.values(db.users || {}).filter(u => u.settings?.discoveryEnabled).length,
|
||||
feedSizes: Object.values(result.feeds).filter(f => !f.disabled).map(f => f.items.length) };
|
||||
if (process.argv.includes('--show')) summary.titles = Object.values(result.feeds).find(f => !f.disabled)?.items
|
||||
|
||||
+143
-49
@@ -3,12 +3,18 @@
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { normalize } = require('../greatest');
|
||||
const { anchorsFor } = require('./anchors');
|
||||
const { MAX_FEED, SLOTS, round2, seedIndexFor, seedNotes, groupFor, partRank, mixSlots } = require('./scoring');
|
||||
|
||||
const MAX_CANDIDATES = 50;
|
||||
const MAX_FEED = 20;
|
||||
const MAX_CANDIDATES = 2000;
|
||||
const SEED_KINDS = ['recs', 'creator', 'tagTop', 'fresh'];
|
||||
const MATURE_TAG_RANK = 60;
|
||||
const VALID_FORMATS = new Set(['MANGA', 'ONE_SHOT']);
|
||||
const safeUrl = value => /^https:\/\/[^\s]+$/i.test(String(value || '')) ? String(value) : null;
|
||||
const unique = values => [...new Set(values.filter(Boolean))];
|
||||
// manga.json and cached metadata are hand-editable files: a field that is not a list counts as empty.
|
||||
const listOf = value => Array.isArray(value) ? value : [];
|
||||
const stringsOf = value => unique(listOf(value).filter(item => typeof item === 'string'));
|
||||
|
||||
function cachedMetadata(dir, id) {
|
||||
if (!Number.isSafeInteger(Number(id)) || Number(id) < 1) return null;
|
||||
@@ -26,7 +32,7 @@ function candidateFromGreatest(entry, cacheDir) {
|
||||
type: cached?.['x-anilist']?.format || 'MANGA', status: cached?.status || entry.status || 'unknown',
|
||||
cover: safeUrl(cached?.['x-anilist']?.coverUrl) || safeUrl(entry.cover),
|
||||
metadataUrl: safeUrl(cached?.['x-anilist']?.url) || safeUrl(entry.metadata?.url) || null,
|
||||
tags: unique(cached?.tags || []), authors: unique(cached?.authors || []),
|
||||
tags: stringsOf(cached?.tags), authors: stringsOf(cached?.authors),
|
||||
period: entry.period, editorialRank: entry.rank, greatestId: entry.id,
|
||||
evidence: [{ id: `greatest:${entry.id}`, kind: 'curation', label: 'Greatest of All Time selection',
|
||||
note: String(entry.note || '').slice(0, 600), url: safeUrl(entry.source?.url) || safeUrl(entry.metadata?.url) }],
|
||||
@@ -34,7 +40,8 @@ function candidateFromGreatest(entry, cacheDir) {
|
||||
};
|
||||
}
|
||||
function addRelations(map, anchor, cacheDir) {
|
||||
for (const relation of anchor.relations || []) {
|
||||
for (const relation of listOf(anchor.relations)) {
|
||||
if (!relation || typeof relation !== 'object') continue;
|
||||
if (!VALID_FORMATS.has(relation.format) || !Number.isSafeInteger(Number(relation.id))) continue;
|
||||
const id = `anilist:${Number(relation.id)}`;
|
||||
if (map.has(id)) continue;
|
||||
@@ -44,16 +51,51 @@ function addRelations(map, anchor, cacheDir) {
|
||||
year: cached?.year || null, type: relation.format, status: cached?.status || 'unknown',
|
||||
cover: safeUrl(cached?.['x-anilist']?.coverUrl),
|
||||
metadataUrl: safeUrl(cached?.['x-anilist']?.url) || safeUrl(relation.url) || `https://anilist.co/manga/${relation.id}`,
|
||||
tags: unique(cached?.tags || []), authors: unique(cached?.authors || []), period: null,
|
||||
tags: stringsOf(cached?.tags), authors: stringsOf(cached?.authors), period: null,
|
||||
editorialRank: null, greatestId: null,
|
||||
evidence: [{ id: `relation:${anchor.id}:${id}`, kind: 'relation', label: `${relation.type || 'RELATED'} to ${anchor.title}`,
|
||||
note: '', url: safeUrl(relation.url) || `https://anilist.co/manga/${relation.id}` }],
|
||||
relatedTo: { id: anchor.id, title: anchor.title, type: relation.type || 'RELATED' },
|
||||
relatedTo: { id: anchor.id, title: anchor.title, type: String(relation.type || 'RELATED') },
|
||||
relations: cached?.['x-anilist']?.relations || [],
|
||||
});
|
||||
}
|
||||
}
|
||||
function buildCandidates({ greatest, library, cacheDir }) {
|
||||
const textList = values => unique(Array.isArray(values) ? values.map(value => value?.name).filter(name => typeof name === 'string') : []);
|
||||
const recordList = values => Array.isArray(values) ? values.filter(value => value && typeof value === 'object') : [];
|
||||
function candidateFromRecord(record, id, seed, fetchedAt) {
|
||||
return {
|
||||
id: `anilist:${id}`, title: String(record.title || `AniList ${id}`), year: Number(record.year) || null,
|
||||
type: record.format || 'MANGA', status: record.status || 'unknown', cover: safeUrl(record.coverUrl),
|
||||
metadataUrl: safeUrl(record.siteUrl) || `https://anilist.co/manga/${id}`,
|
||||
tags: textList(record.tags), authors: textList(record.authors), period: null, editorialRank: null, greatestId: null,
|
||||
evidence: [], relations: [], isAdult: record.isAdult === true,
|
||||
averageScore: Number.isFinite(record.averageScore) ? record.averageScore : null,
|
||||
tagRecords: recordList(record.tags), authorRecords: recordList(record.authors), fetchedAt, seed,
|
||||
};
|
||||
}
|
||||
// Greatest and relation candidates keep their own fields; AniList only fills what they lack.
|
||||
function fillFromRecord(candidate, record, fetchedAt) {
|
||||
if (candidate.isAdult === undefined) candidate.isAdult = record.isAdult === true;
|
||||
if (!candidate.tagRecords) candidate.tagRecords = recordList(record.tags);
|
||||
if (!candidate.authorRecords) candidate.authorRecords = recordList(record.authors);
|
||||
if (candidate.averageScore == null && Number.isFinite(record.averageScore)) candidate.averageScore = record.averageScore;
|
||||
if (!candidate.fetchedAt) candidate.fetchedAt = fetchedAt;
|
||||
}
|
||||
function addHarvested(map, harvestCache) {
|
||||
for (const kind of SEED_KINDS) {
|
||||
const entries = harvestCache.list(kind).sort((a, b) => String(a.key).localeCompare(String(b.key)));
|
||||
for (const entry of entries) for (const item of Array.isArray(entry.data) ? entry.data : []) {
|
||||
const record = item?.record;
|
||||
const id = Number(item?.anilistId);
|
||||
if (!record || typeof record !== 'object' || !Number.isSafeInteger(id) || id < 1) continue;
|
||||
if (record.format && !VALID_FORMATS.has(record.format)) continue;
|
||||
const existing = map.get(`anilist:${id}`);
|
||||
if (existing) fillFromRecord(existing, record, entry.fetchedAt);
|
||||
else map.set(`anilist:${id}`, candidateFromRecord(record, id, kind, entry.fetchedAt));
|
||||
}
|
||||
}
|
||||
}
|
||||
function buildCandidates({ greatest, library, cacheDir, harvestCache = null }) {
|
||||
const map = new Map();
|
||||
for (const entry of greatest.entries) {
|
||||
const candidate = candidateFromGreatest(entry, cacheDir);
|
||||
@@ -65,8 +107,15 @@ function buildCandidates({ greatest, library, cacheDir }) {
|
||||
addRelations(map, { id: `anilist:${identity.id}`, title: manga.title, relations: identity.relations }, cacheDir);
|
||||
}
|
||||
for (const candidate of [...map.values()]) if (candidate.greatestId) addRelations(map, candidate, cacheDir);
|
||||
if (harvestCache) addHarvested(map, harvestCache);
|
||||
return [...map.values()];
|
||||
}
|
||||
// Title-level isAdult or a strong adult tag; the Phase A regex only where AniList gave no isAdult.
|
||||
function isMature(candidate) {
|
||||
if (candidate.isAdult === undefined) return candidate.tags.some(tag => /adult|hentai|erotica/i.test(tag));
|
||||
return candidate.isAdult === true
|
||||
|| (candidate.tagRecords || []).some(tag => tag?.isAdult && Number(tag.rank) >= MATURE_TAG_RANK);
|
||||
}
|
||||
function ownedKeySet(library) {
|
||||
const ids = new Set(), names = new Set();
|
||||
for (const manga of library) {
|
||||
@@ -77,9 +126,10 @@ function ownedKeySet(library) {
|
||||
return { ids, names };
|
||||
}
|
||||
function labelForVote(greatest, id) { return greatest.entries.find(entry => entry.id === id)?.title || null; }
|
||||
function feedForUser({ user, db, candidates, greatest, library, queuedTitles = [] }) {
|
||||
function feedForUser({ user, db, candidates, greatest, library, queuedTitles = [], harvestCache = null, now = () => new Date() }) {
|
||||
const settings = user.settings || {};
|
||||
if (!settings.discoveryEnabled) return { generatedAt: new Date().toISOString(), items: [], disabled: true };
|
||||
const today = now();
|
||||
if (!settings.discoveryEnabled) return { generatedAt: today.toISOString(), items: [], disabled: true };
|
||||
const votes = db.greatestRatings?.[user.name] || {};
|
||||
const feedback = db.discoveryFeedback?.[user.name] || {};
|
||||
const owned = ownedKeySet(library);
|
||||
@@ -93,20 +143,21 @@ function feedForUser({ user, db, candidates, greatest, library, queuedTitles = [
|
||||
for (const tag of candidate?.tags || []) likedTags.set(tag.toLowerCase(), anchor);
|
||||
}
|
||||
const ownedTags = new Map();
|
||||
for (const manga of library) for (const tag of manga.meta?.tags || []) if (typeof tag === 'string') ownedTags.set(tag.toLowerCase(), manga.title);
|
||||
for (const manga of library) for (const tag of stringsOf(manga.meta?.tags)) ownedTags.set(tag.toLowerCase(), manga.title);
|
||||
const readTags = new Map();
|
||||
if (settings.discoveryReadingActivity) for (const manga of library) {
|
||||
const progress = db.progress?.[user.name]?.[manga.id] || {};
|
||||
if (!Object.values(progress).some(chapter => chapter?.done)) continue;
|
||||
for (const tag of manga.meta?.tags || []) if (typeof tag === 'string') readTags.set(tag.toLowerCase(), manga.title);
|
||||
for (const tag of stringsOf(manga.meta?.tags)) readTags.set(tag.toLowerCase(), manga.title);
|
||||
}
|
||||
const seeds = seedIndexFor({ anchors: anchorsFor({ user, db, library, greatest }), harvestCache, mature: !!settings.discoveryMature });
|
||||
const rows = [];
|
||||
for (const candidate of candidates) {
|
||||
if (candidate.greatestId && votes[candidate.greatestId]) continue;
|
||||
if (['dismissed', 'hidden'].includes(feedback[candidate.id])) continue;
|
||||
if (owned.ids.has(candidate.id) || owned.names.has(`${normalize(candidate.title)}:${candidate.year || '?'}`)) continue;
|
||||
if (queued.has(normalize(candidate.title))) continue;
|
||||
if (!settings.discoveryMature && candidate.tags.some(tag => /adult|hentai|erotica/i.test(tag))) continue;
|
||||
if (!settings.discoveryMature && isMature(candidate)) continue;
|
||||
const matchingLikedTag = candidate.tags.find(tag => likedTags.has(tag.toLowerCase()));
|
||||
const matchingOwnedTag = candidate.tags.find(tag => ownedTags.has(tag.toLowerCase()));
|
||||
const matchingReadTag = candidate.tags.find(tag => readTags.has(tag.toLowerCase()));
|
||||
@@ -115,47 +166,50 @@ function feedForUser({ user, db, candidates, greatest, library, queuedTitles = [
|
||||
const scoreParts = { editorial: candidate.greatestId ? Math.max(0, 6 - Math.min(6, Number(candidate.editorialRank) || 6)) : 0,
|
||||
relation: relation ? 3 : 0, sharedTaste: matchingLikedTag ? 4 : 0, sharedLibrary: matchingOwnedTag ? 1 : 0,
|
||||
era: era ? 2 : 0, reading: matchingReadTag ? 2 : 0,
|
||||
interest: feedback[candidate.id] === 'interested' ? 5 : feedback[candidate.id] === 'maybe' ? 2 : 0 };
|
||||
let reason, personalEvidence = null;
|
||||
if (relation) reason = `${candidate.title} is listed as ${relation.type.toLowerCase().replaceAll('_', ' ')} to ${relation.title}. Check the edition and reading order first.`;
|
||||
else if (matchingLikedTag) {
|
||||
interest: feedback[candidate.id] === 'interested' ? 5 : feedback[candidate.id] === 'maybe' ? 2 : 0,
|
||||
communityRec: 0, sameCreator: 0, tagTop: 0, fresh: 0 };
|
||||
// Every applicable part brings its reason and, for taste parts, its personal evidence.
|
||||
const notes = [];
|
||||
if (scoreParts.editorial && candidate.evidence[0]?.note) notes.push({ part: 'editorial', text: candidate.evidence[0].note });
|
||||
if (relation) notes.push({ part: 'relation', text: `${candidate.title} is listed as ${relation.type.toLowerCase().replaceAll('_', ' ')} to ${relation.title}. Check the edition and reading order first.` });
|
||||
if (matchingLikedTag) {
|
||||
const anchor = likedTags.get(matchingLikedTag.toLowerCase());
|
||||
reason = `You liked ${anchor.title}; this also carries the ${matchingLikedTag} tag.`;
|
||||
personalEvidence = { id: `taste:greatest:${anchor.id}`, kind: 'personal', label: `Your Greatest vote for ${anchor.title}`, note: `Shared tag: ${matchingLikedTag}`, url: null };
|
||||
} else if (era) {
|
||||
reason = `You liked ${era.title}; this is another curated work from ${candidate.period}.`;
|
||||
personalEvidence = { id: `taste:greatest:${era.id}`, kind: 'personal', label: `Your Greatest vote for ${era.title}`, note: `Shared period: ${candidate.period}`, url: null };
|
||||
} else if (matchingReadTag) {
|
||||
notes.push({ part: 'sharedTaste', text: `You liked ${anchor.title}; this also carries the ${matchingLikedTag} tag.`,
|
||||
evidence: { id: `taste:greatest:${anchor.id}`, kind: 'personal', label: `Your Greatest vote for ${anchor.title}`, note: `Shared tag: ${matchingLikedTag}`, url: null } });
|
||||
}
|
||||
if (era) notes.push({ part: 'era', text: `You liked ${era.title}; this is another curated work from ${candidate.period}.`,
|
||||
evidence: { id: `taste:greatest:${era.id}`, kind: 'personal', label: `Your Greatest vote for ${era.title}`, note: `Shared period: ${candidate.period}`, url: null } });
|
||||
if (matchingReadTag) {
|
||||
const title = readTags.get(matchingReadTag.toLowerCase());
|
||||
reason = `You have read ${title}; this also carries the ${matchingReadTag} tag.`;
|
||||
personalEvidence = { id: `taste:read:${normalize(title)}`, kind: 'personal', label: `Your completed reading in ${title}`, note: `Shared tag: ${matchingReadTag}`, url: null };
|
||||
} else if (matchingOwnedTag) {
|
||||
notes.push({ part: 'reading', text: `You have read ${title}; this also carries the ${matchingReadTag} tag.`,
|
||||
evidence: { id: `taste:read:${normalize(title)}`, kind: 'personal', label: `Your completed reading in ${title}`, note: `Shared tag: ${matchingReadTag}`, url: null } });
|
||||
}
|
||||
if (matchingOwnedTag) {
|
||||
const title = ownedTags.get(matchingOwnedTag.toLowerCase());
|
||||
reason = `This shares the ${matchingOwnedTag} tag with ${title} in your library.`;
|
||||
personalEvidence = { id: `taste:library:${normalize(title)}`, kind: 'personal', label: `${title} in your library`, note: `Shared tag: ${matchingOwnedTag}`, url: null };
|
||||
notes.push({ part: 'sharedLibrary', text: `This shares the ${matchingOwnedTag} tag with ${title} in your library.`,
|
||||
evidence: { id: `taste:library:${normalize(title)}`, kind: 'personal', label: `${title} in your library`, note: `Shared tag: ${matchingOwnedTag}`, url: null } });
|
||||
}
|
||||
else reason = candidate.evidence[0]?.note || 'A curated manga worth a closer look.';
|
||||
reason = reason.slice(0, 280);
|
||||
const evidence = [...candidate.evidence, ...(personalEvidence ? [personalEvidence] : [])];
|
||||
rows.push({ candidateId: candidate.id, reason, evidence, evidenceIds: evidence.map(e => e.id), scoreParts,
|
||||
score: Object.values(scoreParts).reduce((a, b) => a + b, 0), period: candidate.period || 'Related works' });
|
||||
for (const note of seedNotes(candidate, seeds, today.getFullYear())) { scoreParts[note.part] = note.value; notes.push(note); }
|
||||
notes.sort((a, b) => scoreParts[b.part] - scoreParts[a.part] || partRank(a.part) - partRank(b.part));
|
||||
const reasons = notes.map(note => String(note.text).slice(0, 280));
|
||||
const fallback = candidate.evidence[0]?.note || (candidate.greatestId || relation ? 'A curated manga worth a closer look.' : 'A manga from AniList worth a closer look.');
|
||||
const evidence = [...candidate.evidence];
|
||||
for (const note of notes) if (note.evidence && !evidence.some(e => e.id === note.evidence.id)) evidence.push(note.evidence);
|
||||
rows.push({ candidateId: candidate.id, reason: (reasons[0] || fallback).slice(0, 280), reasons, evidence,
|
||||
evidenceIds: evidence.map(e => e.id), scoreParts, score: round2(Object.values(scoreParts).reduce((a, b) => a + b, 0)),
|
||||
period: candidate.period || 'Related works', group: groupFor(scoreParts, candidate.period), tags: candidate.tags });
|
||||
}
|
||||
rows.sort((a, b) => b.score - a.score || a.candidateId.localeCompare(b.candidateId));
|
||||
const picked = [], periodCounts = new Map();
|
||||
for (const row of rows) {
|
||||
if (picked.length >= MAX_FEED) break;
|
||||
if ((periodCounts.get(row.period) || 0) >= 4) continue;
|
||||
picked.push(row); periodCounts.set(row.period, (periodCounts.get(row.period) || 0) + 1);
|
||||
// Tags only steer the slot mix; the published row does not repeat them.
|
||||
const items = mixSlots(rows, { topTags: seeds.topTags }).map(({ tags, ...row }) => row);
|
||||
return { generatedAt: today.toISOString(), items };
|
||||
}
|
||||
return { generatedAt: new Date().toISOString(), items: picked.map((row, index) => ({ ...row, rank: index + 1 })) };
|
||||
}
|
||||
function generate({ db, library, greatest, cacheDir, queuedTitles = [] }) {
|
||||
const all = buildCandidates({ greatest, library, cacheDir });
|
||||
// Sample across eras before the work cap; insertion order in the curated catalog
|
||||
// must not make recent manga invisible.
|
||||
// Sample across eras (and seed kinds) before the work cap; insertion order in the curated
|
||||
// catalog must not make recent manga invisible.
|
||||
function candidatePool({ greatest, library, cacheDir, harvestCache }) {
|
||||
const all = buildCandidates({ greatest, library, cacheDir, harvestCache });
|
||||
const buckets = new Map();
|
||||
for (const candidate of all) {
|
||||
const key = candidate.period || 'Related works';
|
||||
const key = candidate.period || candidate.seed || 'Related works';
|
||||
if (!buckets.has(key)) buckets.set(key, []);
|
||||
buckets.get(key).push(candidate);
|
||||
}
|
||||
@@ -165,10 +219,50 @@ function generate({ db, library, greatest, cacheDir, queuedTitles = [] }) {
|
||||
if (values.length && candidates.length < MAX_CANDIDATES) candidates.push(values.shift());
|
||||
}
|
||||
}
|
||||
const feeds = {};
|
||||
for (const user of Object.values(db.users || {})) feeds[user.name] = feedForUser({ user, db, candidates, greatest, library, queuedTitles });
|
||||
return { candidates: Object.fromEntries(candidates.map(c => [c.id, c])), feeds,
|
||||
return { all, candidates };
|
||||
}
|
||||
// Only candidates a feed shows are published; the rest of the pool (up to 2000, with AniList
|
||||
// records) would bloat discovery.json for nothing. examined/clipped still describe the whole pool.
|
||||
function runResult(all, candidates, feeds) {
|
||||
const shown = new Set();
|
||||
for (const feed of Object.values(feeds)) for (const item of feed?.items || []) shown.add(item.candidateId);
|
||||
return { candidates: Object.fromEntries(candidates.filter(c => shown.has(c.id)).map(c => [c.id, c])), feeds,
|
||||
examined: all.length, clipped: Math.max(0, all.length - candidates.length) };
|
||||
}
|
||||
function generate({ db, library, greatest, cacheDir, harvestCache = null, queuedTitles = [], now }) {
|
||||
const { all, candidates } = candidatePool({ greatest, library, cacheDir, harvestCache });
|
||||
const feeds = {};
|
||||
for (const user of Object.values(db.users || {}))
|
||||
feeds[user.name] = feedForUser({ user, db, candidates, greatest, library, queuedTitles, harvestCache, now });
|
||||
return runResult(all, candidates, feeds);
|
||||
}
|
||||
// Same result as generate(), but yields to the event loop between users so the reader stays responsive.
|
||||
async function generateAsync({ db, library, greatest, cacheDir, harvestCache = null, queuedTitles = [], onlyUser, now }) {
|
||||
const { all, candidates } = candidatePool({ greatest, library, cacheDir, harvestCache });
|
||||
const feeds = {};
|
||||
for (const user of Object.values(db.users || {})) {
|
||||
if (onlyUser !== undefined && user.name !== onlyUser) continue;
|
||||
await new Promise(setImmediate);
|
||||
feeds[user.name] = feedForUser({ user, db, candidates, greatest, library, queuedTitles, harvestCache, now });
|
||||
}
|
||||
return runResult(all, candidates, feeds);
|
||||
}
|
||||
// A per-user rescore replaces one feed; the others and the candidates they show are carried over.
|
||||
function mergeUserFeed(prior, result, userName) {
|
||||
const feeds = { ...(prior?.feeds || {}) };
|
||||
if (result.feeds?.[userName]) feeds[userName] = result.feeds[userName];
|
||||
else delete feeds[userName];
|
||||
const candidates = {};
|
||||
for (const [name, feed] of Object.entries(feeds)) {
|
||||
if (name === userName) continue;
|
||||
for (const item of feed?.items || []) {
|
||||
const candidate = prior?.candidates?.[item.candidateId];
|
||||
if (candidate) candidates[item.candidateId] = candidate;
|
||||
}
|
||||
}
|
||||
Object.assign(candidates, result.candidates || {});
|
||||
return { candidates, feeds };
|
||||
}
|
||||
|
||||
module.exports = { generate, buildCandidates, feedForUser, cachedMetadata, MAX_CANDIDATES, MAX_FEED };
|
||||
module.exports = { generate, generateAsync, buildCandidates, candidatePool, feedForUser, mixSlots, mergeUserFeed, cachedMetadata,
|
||||
isMature, MAX_CANDIDATES, MAX_FEED, SLOTS };
|
||||
@@ -0,0 +1,174 @@
|
||||
'use strict';
|
||||
|
||||
const crypto = require('node:crypto');
|
||||
const { AniListStop, AniListSkip } = require('./anilist');
|
||||
const { topTagsFor } = require('./anchors');
|
||||
|
||||
const BUDGETS = { nightly: 300, admin: 300, user: 60 };
|
||||
const REFRESH_COOLDOWN_MS = 3600000;
|
||||
const MAX_CONSECUTIVE_FAILURES = 5;
|
||||
const CREATOR_MIN_WEIGHT = 3;
|
||||
|
||||
// Thrown out of a run to unwind every stage at once.
|
||||
class StopRun extends Error {
|
||||
constructor(status, stopReason) { super(stopReason); this.status = status; this.stopReason = stopReason; }
|
||||
}
|
||||
|
||||
async function runHarvest({ kind, targets, cache, client, budget }) {
|
||||
let requests = 0, failures = 0;
|
||||
const seen = new Set();
|
||||
// One request: skipped when already done this run or still fresh in the cache, otherwise awaited in turn.
|
||||
const fetchInto = async (cacheKind, key, call) => {
|
||||
const id = `${cacheKind}\n${key}`;
|
||||
if (seen.has(id)) return;
|
||||
seen.add(id);
|
||||
if (cache.isFresh(cacheKind, key)) return;
|
||||
if (requests >= budget) throw new StopRun('success', 'budget');
|
||||
requests++;
|
||||
let data;
|
||||
try { data = await call(); }
|
||||
catch (error) {
|
||||
if (error instanceof AniListStop) throw new StopRun('stopped', error.reason);
|
||||
if (!(error instanceof AniListSkip)) throw error;
|
||||
if (++failures >= MAX_CONSECUTIVE_FAILURES) throw new StopRun('stopped', 'upstream-errors');
|
||||
return;
|
||||
}
|
||||
failures = 0;
|
||||
// A null title is not a manga format: nothing worth keeping.
|
||||
if (data !== null) cache.put(cacheKind, key, data);
|
||||
};
|
||||
|
||||
const anchors = targets.flatMap(target => target.anchors);
|
||||
const recordOf = id => cache.get('title', id)?.data || null;
|
||||
try {
|
||||
for (const { anilistId } of anchors) await fetchInto('title', anilistId, () => client.title(anilistId));
|
||||
for (const { anilistId } of anchors) await fetchInto('recs', anilistId, () => client.recs(anilistId));
|
||||
for (const anchor of anchors) {
|
||||
if (anchor.weight < CREATOR_MIN_WEIGHT) continue;
|
||||
for (const { staffId } of recordOf(anchor.anilistId)?.authors || [])
|
||||
await fetchInto('creator', staffId, () => client.creator(staffId));
|
||||
}
|
||||
for (const target of targets) {
|
||||
for (const tag of topTagsFor(target.anchors, recordOf, { mature: target.mature }))
|
||||
await fetchInto('tagTop', `${tag.kind}:${tag.name}`, () => client.tagTop(tag));
|
||||
}
|
||||
// With nobody to harvest for, even the global list is not worth a request.
|
||||
if (targets.length) await fetchInto('fresh', 'global', () => client.fresh());
|
||||
} catch (error) {
|
||||
// The caller records how many requests a local fault had already spent.
|
||||
if (!(error instanceof StopRun)) throw Object.assign(error, { requests });
|
||||
return { requests, budget, status: error.status, stopReason: error.stopReason };
|
||||
}
|
||||
return { requests, budget, status: 'success', stopReason: null };
|
||||
}
|
||||
|
||||
const localDate = date => [date.getFullYear(), String(date.getMonth() + 1).padStart(2, '0'),
|
||||
String(date.getDate()).padStart(2, '0')].join('-');
|
||||
|
||||
function shouldRunNightly({ now, harvestAt, lastNightlyDate, enabled }) {
|
||||
if (!enabled) return false;
|
||||
const match = /^(\d{2}):(\d{2})$/.exec(harvestAt || '');
|
||||
if (!match) return false;
|
||||
if (now.getHours() * 60 + now.getMinutes() < Number(match[1]) * 60 + Number(match[2])) return false;
|
||||
return localDate(now) !== lastNightlyDate;
|
||||
}
|
||||
|
||||
const httpError = (status, message, extra = {}) => Object.assign(new Error(message), { status }, extra);
|
||||
|
||||
// These reasons mean AniList pushed back: no further calls until a nightly or admin run.
|
||||
const BLOCKING_REASONS = ['rate-limited', 'forbidden', 'challenge'];
|
||||
const blockingReason = record => record?.status === 'stopped' && BLOCKING_REASONS.includes(record.stopReason)
|
||||
? record.stopReason : null;
|
||||
|
||||
class Harvester {
|
||||
constructor({ cache, client, store, targetsFor, onDone, now = Date.now }) {
|
||||
Object.assign(this, { cache, client, store, targetsFor, onDone, now });
|
||||
this.queue = [];
|
||||
this.active = null;
|
||||
this.lastAccepted = new Map();
|
||||
// The pause outlives a restart through the persisted harvest record.
|
||||
try { this.blockedReason = blockingReason(store.load?.()?.harvest); }
|
||||
catch { this.blockedReason = null; }
|
||||
}
|
||||
nextAllowedAt(user) {
|
||||
const last = this.lastAccepted.get(user);
|
||||
if (last === undefined || this.now() - last >= REFRESH_COOLDOWN_MS) return null;
|
||||
return new Date(last + REFRESH_COOLDOWN_MS).toISOString();
|
||||
}
|
||||
jobs() { return [this.active, ...this.queue].filter(Boolean); }
|
||||
request({ kind, user }) {
|
||||
if (!Object.hasOwn(BUDGETS, kind)) throw httpError(400, `Unknown harvest kind: ${kind}`);
|
||||
if (kind === 'user') {
|
||||
if (!user) throw httpError(400, 'A user refresh needs a user');
|
||||
if (this.jobs().some(job => job.kind === 'user' && job.user === user))
|
||||
throw httpError(409, 'A refresh for this user is already queued or running');
|
||||
if (this.blockedReason)
|
||||
throw httpError(503, `AniList paused the harvest: ${this.blockedReason}`, { blockedReason: this.blockedReason });
|
||||
const nextAllowedAt = this.nextAllowedAt(user);
|
||||
if (nextAllowedAt) throw httpError(429, 'Refresh is cooling down', { nextAllowedAt });
|
||||
this.lastAccepted.set(user, this.now());
|
||||
} else if (this.jobs().length) {
|
||||
throw httpError(409, 'A harvest is already queued or running');
|
||||
} else {
|
||||
// An accepted nightly or admin run is the one allowed retry after a pause.
|
||||
this.blockedReason = null;
|
||||
}
|
||||
const job = { id: crypto.randomBytes(8).toString('hex'), kind, ...(kind === 'user' ? { user } : {}) };
|
||||
const position = this.jobs().length;
|
||||
this.queue.push(job);
|
||||
if (!this.active) this.drain();
|
||||
return { harvestId: job.id, position };
|
||||
}
|
||||
state(userName) {
|
||||
const nextAllowedAt = this.nextAllowedAt(userName);
|
||||
const extra = { ...(nextAllowedAt ? { nextAllowedAt } : {}), ...(this.blockedReason ? { blockedReason: this.blockedReason } : {}) };
|
||||
const index = this.jobs().findIndex(job => job.kind === 'user' && job.user === userName);
|
||||
if (index === 0 && this.active) return { state: 'running', ...extra };
|
||||
if (index >= 0) return { state: 'queued', position: index, ...extra };
|
||||
return { state: 'idle', ...extra };
|
||||
}
|
||||
current() {
|
||||
if (!this.active) return null;
|
||||
const { id, kind, user, startedAt } = this.active;
|
||||
return { id, kind, ...(user ? { user } : {}), startedAt };
|
||||
}
|
||||
// Queued refreshes are dropped unrun; their users get their cooldown back for after the pause.
|
||||
pause(reason) {
|
||||
this.blockedReason = reason;
|
||||
for (const job of this.queue) if (job.kind === 'user') this.lastAccepted.delete(job.user);
|
||||
this.queue = this.queue.filter(job => job.kind !== 'user');
|
||||
}
|
||||
// Runs synchronously up to the first await, so a job is active before request() returns.
|
||||
async drain() {
|
||||
while (this.queue.length) {
|
||||
const job = this.queue.shift();
|
||||
job.startedAt = new Date(this.now()).toISOString();
|
||||
this.active = job;
|
||||
try { await this.run(job); }
|
||||
catch (error) { console.error(`Discovery harvest ${job.id} failed:`, error); }
|
||||
finally { this.active = null; }
|
||||
}
|
||||
}
|
||||
async run(job) {
|
||||
const budget = BUDGETS[job.kind];
|
||||
const base = { id: job.id, kind: job.kind, user: job.user ?? null };
|
||||
await this.store.patch({ harvest: { ...base, status: 'running', startedAt: job.startedAt, requests: 0, budget } });
|
||||
let result;
|
||||
try {
|
||||
const targets = await this.targetsFor(job.kind, job.user);
|
||||
result = await runHarvest({ kind: job.kind, targets, cache: this.cache, client: this.client, budget });
|
||||
} catch (error) {
|
||||
// A local fault (cache write, bad target) must not leave the record "running" until restart.
|
||||
console.error(`Discovery harvest ${job.id} failed:`, error);
|
||||
result = { requests: error?.requests ?? 0, budget, status: 'stopped', stopReason: 'error' };
|
||||
}
|
||||
const blocked = blockingReason(result);
|
||||
if (blocked) this.pause(blocked);
|
||||
await this.store.patch({ harvest: { ...base, status: result.status, startedAt: job.startedAt,
|
||||
finishedAt: new Date(this.now()).toISOString(), requests: result.requests, budget, stopReason: result.stopReason } });
|
||||
try { await this.onDone(job, result); }
|
||||
catch (error) { console.error(`Discovery harvest ${job.id} follow-up failed:`, error); }
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { BUDGETS, REFRESH_COOLDOWN_MS, MAX_CONSECUTIVE_FAILURES, runHarvest, shouldRunNightly, Harvester };
|
||||
@@ -0,0 +1,158 @@
|
||||
'use strict';
|
||||
|
||||
// Pure scoring helpers for the AniList seed parts and the feed slot mix (BL-0051).
|
||||
const { topTagsFor } = require('./anchors');
|
||||
|
||||
const MAX_FEED = 20;
|
||||
const SLOTS = { perKind: 8, perPeriod: 4, fresh: 4, outside: 4 };
|
||||
const CREATOR_MIN_WEIGHT = 3;
|
||||
const TAG_TOP_MIN_SCORE = 75;
|
||||
const FRESH_YEARS = 2;
|
||||
// Part name -> group (seed kind) for the four AniList seed parts.
|
||||
const SEED_PART_GROUP = { communityRec: 'recs', sameCreator: 'creator', tagTop: 'tagTop', fresh: 'fresh' };
|
||||
const SEED_GROUPS = new Set(Object.values(SEED_PART_GROUP));
|
||||
// Tie-break when two parts are equal: seed parts first, then Phase A order.
|
||||
const PART_ORDER = ['communityRec', 'sameCreator', 'tagTop', 'fresh',
|
||||
'interest', 'editorial', 'sharedTaste', 'relation', 'era', 'reading', 'sharedLibrary'];
|
||||
|
||||
const round2 = value => Math.round(value * 100) / 100;
|
||||
const listOf = value => Array.isArray(value) ? value : [];
|
||||
// Cached lists come from disk; anything that is not an object is skipped, never dereferenced.
|
||||
const objectsOf = value => listOf(value).filter(item => item && typeof item === 'object');
|
||||
const partRank = part => { const index = PART_ORDER.indexOf(part); return index < 0 ? PART_ORDER.length : index; };
|
||||
|
||||
function communityRecValue(rating, weight) {
|
||||
const safe = Math.max(0, Number(rating) || 0);
|
||||
return Math.min(5, 1 + Math.log2(1 + safe) / 2) * weight / 5;
|
||||
}
|
||||
const sameCreatorValue = weight => 4 * weight / 5;
|
||||
|
||||
// What the harvest cache says about this user's anchors: one lookup map per seed kind.
|
||||
function seedIndexFor({ anchors, harvestCache, mature = false }) {
|
||||
const index = { recs: new Map(), creators: new Map(), tagTop: new Map(), fresh: new Map(), topTags: [] };
|
||||
if (!harvestCache) return index;
|
||||
const recordOf = id => harvestCache.get('title', id)?.data || null;
|
||||
const nameOf = anchor => anchor.title || recordOf(anchor.anilistId)?.title || `AniList ${anchor.anilistId}`;
|
||||
for (const anchor of anchors) {
|
||||
const name = nameOf(anchor);
|
||||
const entry = harvestCache.get('recs', anchor.anilistId);
|
||||
for (const item of objectsOf(entry?.data)) {
|
||||
const id = `anilist:${Number(item.anilistId)}`;
|
||||
if (!index.recs.has(id)) index.recs.set(id, []);
|
||||
index.recs.get(id).push({ anchor, name, rating: Number(item.rating) || 0, record: item.record || null, fetchedAt: entry.fetchedAt });
|
||||
}
|
||||
if (anchor.weight < CREATOR_MIN_WEIGHT) continue;
|
||||
for (const author of objectsOf(recordOf(anchor.anilistId)?.authors)) {
|
||||
const prior = index.creators.get(author.staffId);
|
||||
if (prior && prior.anchor.weight >= anchor.weight) continue;
|
||||
index.creators.set(author.staffId, { anchor, name, author,
|
||||
fetchedAt: harvestCache.get('creator', author.staffId)?.fetchedAt || null });
|
||||
}
|
||||
}
|
||||
index.topTags = topTagsFor(anchors, recordOf, { mature });
|
||||
index.topTags.forEach((tag, place) => {
|
||||
const entry = harvestCache.get('tagTop', `${tag.kind}:${tag.name}`);
|
||||
for (const item of objectsOf(entry?.data)) {
|
||||
const id = `anilist:${Number(item.anilistId)}`;
|
||||
// Tags come best-first, so the first hit is the strongest.
|
||||
if (!index.tagTop.has(id)) index.tagTop.set(id, { tag, value: 3 - place, record: item.record || null, fetchedAt: entry.fetchedAt });
|
||||
}
|
||||
});
|
||||
const fresh = harvestCache.get('fresh', 'global');
|
||||
for (const item of objectsOf(fresh?.data)) {
|
||||
index.fresh.set(`anilist:${Number(item.anilistId)}`, { record: item.record || null, fetchedAt: fresh.fetchedAt });
|
||||
}
|
||||
return index;
|
||||
}
|
||||
|
||||
// Seed parts for one candidate with the reason and evidence each one brings.
|
||||
function seedNotes(candidate, seeds, thisYear) {
|
||||
const notes = [];
|
||||
const urlFor = record => record?.siteUrl && /^https:\/\/[^\s]+$/i.test(String(record.siteUrl)) ? String(record.siteUrl) : candidate.metadataUrl || null;
|
||||
|
||||
let bestRec = null;
|
||||
for (const hit of seeds.recs.get(candidate.id) || []) {
|
||||
const value = communityRecValue(hit.rating, hit.anchor.weight);
|
||||
if (!bestRec || value > bestRec.value) bestRec = { ...hit, value };
|
||||
}
|
||||
if (bestRec) notes.push({ part: 'communityRec', value: round2(bestRec.value),
|
||||
text: `AniList readers who liked ${bestRec.name} recommend this (${bestRec.rating}).`,
|
||||
evidence: { id: `rec:${bestRec.anchor.anilistId}:${candidate.id}`, kind: 'recommendation',
|
||||
label: `AniList recommendation from readers of ${bestRec.name}`, note: `Rating ${bestRec.rating}`,
|
||||
url: urlFor(bestRec.record), fetchedAt: bestRec.fetchedAt } });
|
||||
|
||||
let bestCreator = null;
|
||||
for (const author of objectsOf(candidate.authorRecords)) {
|
||||
const hit = seeds.creators.get(author.staffId);
|
||||
// An anchor is never its own "also by" suggestion.
|
||||
if (!hit || `anilist:${hit.anchor.anilistId}` === candidate.id) continue;
|
||||
if (!bestCreator || hit.anchor.weight > bestCreator.anchor.weight) bestCreator = { ...hit, candidateAuthor: author };
|
||||
}
|
||||
if (bestCreator) {
|
||||
const author = bestCreator.candidateAuthor.name || bestCreator.author.name;
|
||||
const verb = bestCreator.anchor.source === 'read' ? 'read' : 'liked';
|
||||
notes.push({ part: 'sameCreator', value: round2(sameCreatorValue(bestCreator.anchor.weight)),
|
||||
text: `Also by ${author} — you ${verb} ${bestCreator.name}.`,
|
||||
evidence: { id: `creator:${bestCreator.author.staffId}:${candidate.id}`, kind: 'creator',
|
||||
label: `Also by ${author}`, note: `Shared creator with ${bestCreator.name}`,
|
||||
url: urlFor(null), fetchedAt: bestCreator.fetchedAt || candidate.fetchedAt || null } });
|
||||
}
|
||||
|
||||
const top = seeds.tagTop.get(candidate.id);
|
||||
const averageScore = Number(candidate.averageScore ?? top?.record?.averageScore);
|
||||
if (top && averageScore >= TAG_TOP_MIN_SCORE) notes.push({ part: 'tagTop', value: top.value,
|
||||
text: `Top-rated in ${top.tag.name}, one of your most-liked tags.`,
|
||||
evidence: { id: `tagtop:${top.tag.kind}:${top.tag.name}:${candidate.id}`, kind: 'tag-top',
|
||||
label: `Top-rated in ${top.tag.name} on AniList`, note: `Average score ${averageScore}`,
|
||||
url: urlFor(top.record), fetchedAt: top.fetchedAt } });
|
||||
|
||||
const fresh = seeds.fresh.get(candidate.id);
|
||||
if (fresh) {
|
||||
const status = fresh.record?.status ?? candidate.status;
|
||||
const year = Number(fresh.record?.year ?? candidate.year);
|
||||
const isNew = status === 'ongoing' && year >= thisYear - FRESH_YEARS;
|
||||
notes.push({ part: 'fresh', value: isNew ? 2 : 1,
|
||||
// The exact template claims a start year and "ongoing", which only the value-2 case proves.
|
||||
text: isNew ? `New: started ${year}, ongoing and trending.` : 'Trending on AniList right now.',
|
||||
evidence: { id: `fresh:${candidate.id}`, kind: 'fresh', label: 'Trending new manga on AniList', note: '',
|
||||
url: urlFor(fresh.record), fetchedAt: fresh.fetchedAt } });
|
||||
}
|
||||
return notes;
|
||||
}
|
||||
|
||||
function largestPart(scoreParts) {
|
||||
let best = null;
|
||||
for (const [part, value] of Object.entries(scoreParts)) {
|
||||
if (!best || value > best.value || (value === best.value && partRank(part) < partRank(best.part))) best = { part, value };
|
||||
}
|
||||
return best;
|
||||
}
|
||||
function groupFor(scoreParts, period) {
|
||||
const top = largestPart(scoreParts);
|
||||
return top && top.value > 0 && SEED_PART_GROUP[top.part] ? SEED_PART_GROUP[top.part] : period || 'Related works';
|
||||
}
|
||||
const byScore = (a, b) => b.score - a.score || (a.candidateId < b.candidateId ? -1 : a.candidateId > b.candidateId ? 1 : 0);
|
||||
|
||||
// Reserve slots for fresh and outside-the-top-tags rows, then fill greedily; every step respects the group caps.
|
||||
function mixSlots(rows, { topTags = [], max = MAX_FEED } = {}) {
|
||||
const sorted = [...rows].sort(byScore);
|
||||
const picked = new Set(), counts = new Map();
|
||||
const capOf = group => SEED_GROUPS.has(group) ? SLOTS.perKind : SLOTS.perPeriod;
|
||||
const take = row => {
|
||||
if (picked.has(row) || picked.size >= max || (counts.get(row.group) || 0) >= capOf(row.group)) return false;
|
||||
picked.add(row); counts.set(row.group, (counts.get(row.group) || 0) + 1);
|
||||
return true;
|
||||
};
|
||||
const reserve = (limit, wanted) => {
|
||||
let taken = 0;
|
||||
for (const row of sorted) if (taken < limit && wanted(row) && take(row)) taken++;
|
||||
};
|
||||
const top = new Set(topTags.map(tag => String(tag.name).toLowerCase()));
|
||||
reserve(SLOTS.fresh, row => row.scoreParts?.fresh > 0);
|
||||
reserve(SLOTS.outside, row => !listOf(row.tags).some(tag => top.has(String(tag).toLowerCase())));
|
||||
for (const row of sorted) take(row);
|
||||
return [...picked].sort(byScore).map((row, index) => ({ ...row, rank: index + 1 }));
|
||||
}
|
||||
|
||||
module.exports = { MAX_FEED, SLOTS, PART_ORDER, round2, communityRecValue, sameCreatorValue,
|
||||
seedIndexFor, seedNotes, largestPart, groupFor, partRank, mixSlots };
|
||||
+19
-2
@@ -4,6 +4,8 @@ const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const crypto = require('node:crypto');
|
||||
|
||||
const META_KEYS = ['harvest', 'settings', 'nightly'];
|
||||
|
||||
class DiscoveryStore {
|
||||
constructor(dataDir) { this.file = path.join(path.resolve(dataDir), 'discovery.json'); }
|
||||
load() {
|
||||
@@ -11,9 +13,14 @@ class DiscoveryStore {
|
||||
const data = JSON.parse(fs.readFileSync(this.file, 'utf8'));
|
||||
if (data.schema !== 1 || !Number.isSafeInteger(data.revision) || !data.candidates || !data.feeds || !data.lastRun)
|
||||
throw new Error('Invalid discovery store schema');
|
||||
// Phase A files lack the metadata fields; default them.
|
||||
if (data.harvest === undefined) data.harvest = null;
|
||||
if (data.settings === undefined) data.settings = null;
|
||||
if (!data.nightly) data.nightly = { lastDate: null };
|
||||
return data;
|
||||
} catch (error) {
|
||||
if (error.code === 'ENOENT') return { schema: 1, revision: 0, candidates: {}, feeds: {}, lastRun: { status: 'never' } };
|
||||
if (error.code === 'ENOENT') return { schema: 1, revision: 0, candidates: {}, feeds: {}, lastRun: { status: 'never' },
|
||||
harvest: null, settings: null, nightly: { lastDate: null } };
|
||||
throw new Error(`Discovery store unavailable; refusing to overwrite it: ${error.message}`);
|
||||
}
|
||||
}
|
||||
@@ -21,7 +28,17 @@ class DiscoveryStore {
|
||||
const prior = this.load();
|
||||
const data = { schema: 1, revision: prior.revision + 1, candidates: value.candidates, feeds: value.feeds,
|
||||
lastRun: { status: 'success', startedAt: value.startedAt, finishedAt: new Date().toISOString(),
|
||||
candidateCount: Object.keys(value.candidates).length, examined: value.examined, clipped: value.clipped } };
|
||||
candidateCount: Object.keys(value.candidates).length, examined: value.examined, clipped: value.clipped },
|
||||
harvest: prior.harvest, settings: prior.settings, nightly: prior.nightly };
|
||||
return this.write(data);
|
||||
}
|
||||
// Metadata only: each given key replaces that field; revision stays.
|
||||
patch(fields) {
|
||||
const data = this.load();
|
||||
for (const key of META_KEYS) if (key in fields) data[key] = fields[key];
|
||||
return this.write(data);
|
||||
}
|
||||
write(data) {
|
||||
fs.mkdirSync(path.dirname(this.file), { recursive: true });
|
||||
const temp = `${this.file}.${crypto.randomBytes(6).toString('hex')}.tmp`;
|
||||
try {
|
||||
|
||||
+7
-4
@@ -47,17 +47,20 @@ services:
|
||||
DATA: /data
|
||||
LIBRARY: /library
|
||||
BACKUP: ${BACKUP_PATH:+/backup}
|
||||
TZ: ${TZ:-Europe/Berlin}
|
||||
TZ: ${TZ:-UTC}
|
||||
SIGNUP: closed
|
||||
SCAN_INTERVAL: "10"
|
||||
SCAN_CONCURRENCY: ${SCAN_CONCURRENCY:-8}
|
||||
COOKIE_SECURE: ${COOKIE_SECURE:-0}
|
||||
READING_WINDOWS_JSON: ${READING_WINDOWS_JSON:-}
|
||||
READING_TIMEZONE: ${READING_TIMEZONE:-Europe/Berlin}
|
||||
READING_TIMEZONE: ${READING_TIMEZONE:-UTC}
|
||||
READING_QUIZ_UNLOCK_MINUTES: ${READING_QUIZ_UNLOCK_MINUTES:-60}
|
||||
READING_QUIZ_CATALOG_PATH: ${READING_QUIZ_CATALOG_PATH:-}
|
||||
SCRAPER_URL: http://scraper:8091
|
||||
SCRAPER_TOKEN_FILE: /run/secrets/scraper_token
|
||||
DISCOVERY_HARVEST_AT: ${DISCOVERY_HARVEST_AT:-03:30}
|
||||
DISCOVERY_ANILIST_URL: ${DISCOVERY_ANILIST_URL:-}
|
||||
GREATEST_COVER_WARM: ${GREATEST_COVER_WARM:-1}
|
||||
networks: [backend, egress]
|
||||
healthcheck:
|
||||
test: [CMD, node, tools/healthcheck.js, reader]
|
||||
@@ -87,7 +90,7 @@ services:
|
||||
LIBRARY: /library
|
||||
SCRAPER_DATA: /data/scraper
|
||||
METADATA_CACHE: /data/scraper/metadata/anilist
|
||||
TZ: ${TZ:-Europe/Berlin}
|
||||
TZ: ${TZ:-UTC}
|
||||
SCRAPER_HOST: 0.0.0.0
|
||||
SCRAPER_PORT: "8091"
|
||||
SCRAPER_TOKEN_FILE: /run/secrets/scraper_token
|
||||
@@ -124,7 +127,7 @@ services:
|
||||
PEER_USAGE_FILE: /data/peering-usage/usage.json
|
||||
PEER_HOST: 0.0.0.0
|
||||
PEER_PORT: "8093"
|
||||
# Docker Desktop does not publish host ports from an internal-only network.
|
||||
# Docker (Desktop) does not publish host ports from an internal-only network.
|
||||
networks: [backend, egress]
|
||||
healthcheck:
|
||||
test: [CMD, node, tools/healthcheck-peer.js]
|
||||
|
||||
+51
-19
@@ -13,7 +13,7 @@ Related documents:
|
||||
- [scraper-service.md](scraper-service.md): managed queue, private API, and adapter contract.
|
||||
- [collections.md](collections.md): relations, groups, curator UI and agent research.
|
||||
- [resilience.md](resilience.md): supervision, recovery limits and incidents.
|
||||
- [compose-deployment.md](compose-deployment.md): local/VM deployment and release workflow.
|
||||
- [deployment.md](deployment.md): Docker deployment, HTTPS, operation, updates and backups.
|
||||
- [settings-navigation.md](settings-navigation.md): User/Admin workspace, scraper tabs and deep links.
|
||||
- [peering.md](peering.md): optional private server-to-server catalog and transfer.
|
||||
- [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
|
||||
@@ -128,6 +128,7 @@ This keeps ids human-readable and derivable without a database. The price: renam
|
||||
"flowPageGap": true, // show the 8 px separator between pages in Flow
|
||||
"readingClock": false, // opt in to credible per-chapter active-time tracking
|
||||
"readingStory": true, // show private retrospective summaries when history exists
|
||||
"continueRow": true, // library: "Continue where you left off" row at the top
|
||||
"zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 }, // page zoom per mode
|
||||
"webhook": ""
|
||||
}
|
||||
@@ -150,6 +151,11 @@ This keeps ids human-readable and derivable without a database. The price: renam
|
||||
"follows": { "admin": { "Moon Garden": true } },
|
||||
"notes": { "admin": [ { "id": "a1b2c3d4e5f6", "t": 1790797300000, "m": "Moon Garden", "c": "2/6",
|
||||
"text": "Moon Garden · Ch. 6: Frost", "read": false } ] },
|
||||
"moments": { "admin": [ { "id": "9f1c2b3a4d5e", "m": "Moon Garden", "c": "1/2", "v": "c0002 Roots [MangaDex]",
|
||||
"i": [4], "f": ["0005.png"], "sha": ["<sha256 of the file>"], // 1–4 pages in displayed order
|
||||
"layout": "row", // "row" side by side (Page/Book), "column" stacked (Flow)
|
||||
"region": { "x": 0.1, "y": 0.5, "w": 0.5, "h": 0.25 }, // fractions of the pages put together; null = all
|
||||
"cat": "awesome", "note": "", "t": 1790797400000 } ] },
|
||||
"votes": { "Moon Garden": { "1/2": { "c0002 Roots [Official]": { "admin": 1, "reader": -1 } } } },
|
||||
"known": { "Moon Garden": ["1/1", "1/2", "1/2.5", "..."] } // every chapter id ever seen (append-only)
|
||||
}
|
||||
@@ -171,7 +177,7 @@ This keeps ids human-readable and derivable without a database. The price: renam
|
||||
|
||||
| Path | Handling |
|
||||
|------|----------|
|
||||
| `/api/*` | `routes["<METHOD> <path>"]`. Handlers get `{ u, b, q, req, res }` and return an object, which is sent as JSON. Throwing `HttpError` sends `{ error }` with that status |
|
||||
| `/api/*` | `routes["<METHOD> <path>"]`. Handlers get `{ u, b, q, req, res }` and return an object, which is sent as JSON. Throwing `HttpError(code, message, extra)` sends `{ error, ...extra }` with that status (for example `nextAllowedAt` on a discovery `429`) |
|
||||
| `/lib/*` | Image files from `LIBRARY`. Requires a session, allows image extensions only, and the path must resolve inside `LIBRARY`. `ETag` + `private, max-age=86400` |
|
||||
| everything else | Static files from `public/` (`/` → `index.html`). `ETag` + `no-cache` |
|
||||
|
||||
@@ -200,6 +206,9 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
|
||||
| `POST /api/unread` `{m, c}` | user | Forget progress for a chapter |
|
||||
| `POST /api/vote` `{m, c, v, value: 1\|-1\|0}` | user | Vote a version; returns the re-ranked versions |
|
||||
| `POST /api/follow` `{m, on}` | user | Follow a manga for notifications |
|
||||
| `GET /api/moments?m=` | user | The caller's moments of that manga in story order: category, note, region, chapter, `order`, page images, `missing` when a page file is gone. `/api/chapter` also lists the caller's moments on the opened version |
|
||||
| `POST /api/moments` `{m, c, v, pages, layout, region, cat, note}` | user | Save a moment: `pages` are 1–4 distinct indexes in displayed order, `layout` `row` (default) or `column`, `cat` one of `twist hot awesome cute funny feels` (required), `region` fractions of the pages put together, or null |
|
||||
| `POST /api/moments/update` `{id, cat?, region?, note?}` · `POST /api/moments/delete` `{id}` | user | Edit or delete one of the caller's own moments (another user's id is `404`) |
|
||||
| `GET /api/notifications` · `POST /api/notifications/read` `{id \| all}` | user | Notification list and read state |
|
||||
| `GET /api/admin` | admin | Users, last scan result, library path |
|
||||
| `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result |
|
||||
@@ -207,6 +216,14 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
|
||||
| `POST /api/admin/user/role` `{name, admin}` | admin | Grant or revoke admin rights; keeps the password and sessions. Revoking your own rights is refused, so one admin always remains |
|
||||
| `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data |
|
||||
| `GET /api/admin/backup` · `POST /api/admin/backup` `{command:"settings"\|"run"}` | admin | Built-in backup status, schedule/retention, "back up now" ([backup.md](backup.md)) |
|
||||
| `GET /api/greatest-cover?id=<entry>&v=<version>` | user | The Greatest entry's cover from `data/greatest-covers/` (`greatest/covers.js`), cached by the browser for a year; `v` changes with the catalog URL. A `302` to the catalog URL until the copy exists. `/api/greatest` lists this URL as `cover` for titles not in the library |
|
||||
| `GET /api/discovery` | user | The caller's discovery feed, settings, feedback and `refresh: { state: idle\|queued\|running, position?, nextAllowedAt?, blockedReason? }` (§6.6) |
|
||||
| `POST /api/discovery/feedback` `{candidateId, value}` | user | The caller's choice for a candidate; requires opt-in |
|
||||
| `POST /api/discovery/refresh` | user | Queue a per-user AniList harvest for the caller (opt-in required, else `403`). `{ harvestId, position }`; `409` while one is queued or running for the caller; `429 { nextAllowedAt }` within an hour of the last accepted refresh; `503 { blockedReason }` after AniList pushed back, until a nightly or admin harvest |
|
||||
| `GET /api/admin/discovery` | admin | Last scoring run, `lastError` (a failed admin run or a failed re-score after a harvest), opted-in users, candidate count, plus `harvest` (last run record), `currentHarvest`, `cache: { files, bytes }`, `settings: { harvestEnabled, harvestAt }` and `nextNightlyAt` (`null` when off) |
|
||||
| `POST /api/admin/discovery/run` | admin | Re-score every user from local data; `409` while a run is active |
|
||||
| `POST /api/admin/discovery/harvest` | admin | Start a full harvest for all opted-in users: `{ harvestId, position }`, `409` while any harvest is queued or running |
|
||||
| `POST /api/admin/discovery/settings` `{harvestEnabled, harvestAt}` | admin | Exactly a boolean and an `HH:MM` time, anything else is `400`; stored in `discovery.json` `settings` |
|
||||
|
||||
### 6.3 Authentication and security
|
||||
|
||||
@@ -229,6 +246,28 @@ At request time: the sum of all users' votes, then `chapter.json.priority`, then
|
||||
|
||||
In-app notifications and optional webhooks are produced by `announce()` after each scan, from the append-only `known` list. See [import.md §5](import.md#5-change-detection-and-notifications).
|
||||
|
||||
### 6.6 Discovery harvest
|
||||
|
||||
The reader alone talks to AniList for discovery ([discovery.md](discovery.md)). A single-flight harvester
|
||||
(`discovery/harvest.js`) fetches seeds for the opted-in users' anchors into `data/discovery-cache/`; scoring
|
||||
(`discovery/engine.js`) is local and publishes `data/discovery.json` atomically.
|
||||
|
||||
- **Kinds and budgets:** `nightly` and `admin` (300 requests, every opted-in user), `user` (60 requests, the
|
||||
caller only). After a nightly or admin harvest every user is re-scored; after a per-user refresh only that
|
||||
user, and the other feeds are carried over unchanged into the same publish.
|
||||
- **Schedule:** checked every minute and once 30 s after startup. The time is `discovery.json` `settings`, else
|
||||
`DISCOVERY_HARVEST_AT` (`HH:MM`), else `03:30`; `nightly.lastDate` is written only when the run is accepted.
|
||||
- **Run record** `discovery.json` `harvest`: `{ id, kind, user, status: running|success|stopped|interrupted,
|
||||
startedAt, finishedAt, requests, budget, stopReason }`. `stopReason` is `budget`, `rate-limited`, `forbidden`,
|
||||
`challenge`, `upstream-errors` or `error` (a local fault). A run cut off by a restart is marked `interrupted`
|
||||
at startup; queued refreshes are not persisted.
|
||||
- **Published candidates:** only those at least one feed shows. `lastRun.candidateCount` (the admin
|
||||
candidate count) counts that set; `lastRun.examined` and `clipped` still describe the whole pool.
|
||||
- **Published fields:** candidates found through AniList carry `tagRecords`, `authorRecords`, `seed`
|
||||
(`recs|creator|tagTop|fresh`) and `fetchedAt`; feed items carry `group` (seed kind or period) and `reasons`
|
||||
(all reasons, strongest first) next to `reason`; seed evidence carries `fetchedAt`.
|
||||
- `DISCOVERY_ANILIST_URL` overrides the AniList endpoint (tests point it at a local fixture server).
|
||||
|
||||
## 7. Frontend
|
||||
|
||||
### 7.1 Pages
|
||||
@@ -292,12 +331,12 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
|
||||
| Horizontal swipe ≥ 50 px within 800 ms (paged, not zoomed in) | Previous/next |
|
||||
| Ctrl + wheel, touchpad pinch, Safari gesture events, two-finger pinch | Continuous page zoom 20–600 %, snapping to 100 % within ±1 %, anchored at the pointer or finger midpoint. Browser zoom is suppressed only over the pages |
|
||||
| Mouse drag (moved ≥ 6 px) | Pan; the following click is swallowed. Hand cursor when the content is larger than the screen |
|
||||
| Keys | Arrows follow the direction; Space/PageDown/PageUp (paged); `+` and `-` zoom in 5-percentage-point steps, `0` fits at 100 %; `m`/Esc bars; `f` fullscreen |
|
||||
| Keys | Dispatched through `READER_KEYS` in `public/keys.js`, which also feeds Settings → Keyboard shortcuts and the reader's `?` list. Arrows follow the direction; Space/PageDown/PageUp (paged); `+` and `-` zoom in 5-percentage-point steps, `0` fits at 100 %; `m` saves a moment; `i`/Esc bars; `f` fullscreen; `?` shortcuts |
|
||||
|
||||
**Bars:**
|
||||
- Shown on open.
|
||||
- Hidden on the first page turn or after 100 px of scrolling. The first automatic hide (per device, once for Flow and once for Page/Book) shows a hint on how to get them back.
|
||||
- **Never shown by scrolling**, only by a tap beside the pages or in the middle of a page (or `m`). Showing them stops a running smooth scroll, so it can't hide them again right away.
|
||||
- **Never shown by scrolling**, only by a tap beside the pages or in the middle of a page (or `i`). Showing them stops a running smooth scroll, so it can't hide them again right away.
|
||||
- The top bar has the chapter picker (every chapter, grouped by book, ✓ = read).
|
||||
|
||||
**Loading:**
|
||||
@@ -321,24 +360,17 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
|
||||
- Pulses are sent at most every 10 seconds and on pause/page exit. The server validates the manga/chapter/version and page indices, de-duplicates `(session, seq)`, clamps time to server-observed intervals and stores compact page ranges separately from ordinary progress.
|
||||
- The bottom toolbar chip shows active time and the current state; its panel adds pass time, verified pages, predominant input and loading wait.
|
||||
|
||||
**Moments** (`public/moments.js`): the top bar's bookmark button (or `m`) opens a sheet for the page(s) on screen: in Flow the page at the reading line, in Book the spread. "+ Previous page" / "+ Next page" add neighbours (up to four pages) for a picture that runs over several pages: stacked in Flow (`column`), side by side in Page and Book (`row`, right to left for `rtl`). A rectangle dragged over the preview keeps only that region, and pages it does not touch are left out. Exactly one category is required. The button is filled when the page already has a moment. The manga page shows the user's moments as a strip of crops (sort by story or by date saved, category filters from five moments), with a viewer to step through them, edit, delete or open the reader at `read.html?…&p=<page index>`. Crops position the full page images inside a clipping box; nothing is generated on the server.
|
||||
|
||||
**Zoom persistence:** per mode key (`flow`, `page`, `book`, `strip`), saved 800 ms after the last change.
|
||||
|
||||
## 8. Deployment
|
||||
|
||||
### Current Windows deployment
|
||||
|
||||
The observed local installation uses the Windows Scheduled Task
|
||||
`\MangaCircus Supervisor`. At logon it runs `tools/run-supervisor.ps1`, which appends
|
||||
logs under `data/scraper/` and starts one `tools/start-all.js` supervisor. That parent
|
||||
owns exactly one reader (`server.js`, port 8080) and one scraper daemon
|
||||
(`scraper/daemon.js`, private port 8091). Starting `npm run start:all` manually at the
|
||||
same time would create conflicting writers and is not supported.
|
||||
|
||||
The scraper's authoritative `data/scraper/queue.json` is schema version 3. It supports
|
||||
legacy archive checkpoints and paused conversion of completed legacy work while adding
|
||||
source-aware lanes, scoped queue jobs, incremental chapter publication, and nightly catalog discovery. See
|
||||
[local-deployment.md](local-deployment.md) for operational checks and restart/recovery,
|
||||
and [scraper-service.md](scraper-service.md) for the queue/API contract.
|
||||
The step-by-step guide is [deployment.md](deployment.md). The scraper's authoritative
|
||||
`data/scraper/queue.json` is schema version 3. It supports legacy archive checkpoints
|
||||
and paused conversion of completed legacy work while adding source-aware lanes, scoped
|
||||
queue jobs, incremental chapter publication, and nightly catalog discovery. See
|
||||
[scraper-service.md](scraper-service.md) for the queue/API contract.
|
||||
|
||||
### Docker or manual deployment
|
||||
|
||||
@@ -352,7 +384,7 @@ docker compose up -d --build
|
||||
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
|
||||
- `/backup`: reader only, writable target of the built-in backup (`BACKUP_PATH`; off when unset).
|
||||
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `BACKUP`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
|
||||
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
|
||||
- **Without Docker:** `npm run start:all` with Node ≥ 18 starts one supervisor that owns exactly one reader (`server.js`, port 8080) and one scraper daemon (`scraper/daemon.js`, private port 8091), or supervise `npm start` and `npm run start:scraper` independently. On Windows, `tools/run-supervisor.ps1` runs `start:all` with logs under `data/scraper/`, e.g. from a scheduled task at logon. Never start a second supervisor or a container against the same data.
|
||||
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
|
||||
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
|
||||
unknown fields in `db.json` are kept. The scraper normalizes old queue entries into
|
||||
|
||||
+2
-2
@@ -112,5 +112,5 @@ and queue changes made after that backup are lost.
|
||||
|
||||
The scraper token is not in `DATA`; Compose keeps it under `.secrets/` (or the
|
||||
deployment's `SCRAPER_TOKEN_PATH`). Restore tests should unpack into a separate
|
||||
folder and run an isolated stack (no second writer against the live library), as
|
||||
a restore-test script does for the host-level archives.
|
||||
folder and run an isolated reader (no second writer against the live library), as
|
||||
`deploy/circus.sh restore-test` does.
|
||||
+3
-3
@@ -26,7 +26,7 @@ selection and must use **Validate & save JSON**, not an ordinary collection save
|
||||
Metadata imports/research are blocked while drafts exist so they cannot silently
|
||||
replace them. Relationship edits save immediately when there is no collection draft.
|
||||
|
||||
Creation works on HTTPS, localhost and ordinary HTTP LAN/Tailscale addresses. The
|
||||
Creation works on HTTPS, localhost and ordinary HTTP LAN/VPN addresses. The
|
||||
browser uses secure random bytes when its secure-context-only `randomUUID` API is
|
||||
unavailable. This does not change networking or the server's revision/validation rules.
|
||||
|
||||
@@ -115,9 +115,9 @@ Provider contract: [AniList relationship connections](https://docs.anilist.co/gu
|
||||
|
||||
## Repeatable functional acceptance
|
||||
|
||||
Run `npm.cmd test`, build `docker compose build reader`, then:
|
||||
Run `npm test`, build `docker compose build reader`, then:
|
||||
|
||||
```powershell
|
||||
```bash
|
||||
# Playwright is optional developer tooling; not a deployed application dependency.
|
||||
# Set NODE_PATH to its installed package root if it is not on normal resolution paths.
|
||||
node tools/test-compose.js --collections-ui --settings-ui
|
||||
|
||||
@@ -0,0 +1,201 @@
|
||||
# Deploying MangaCircus with Docker
|
||||
|
||||
This guide sets up MangaCircus on one Linux host with Docker Compose: the reader (web app), the managed
|
||||
scraper, and optionally HTTPS and server peering. Every step is a command with what you should see. It is
|
||||
written so that a person or a coding agent can follow it without other context.
|
||||
|
||||
## 1. What runs
|
||||
|
||||
| Service | Port | Role |
|
||||
|---|---|---|
|
||||
| `reader` | 8080 (published on `READER_BIND`) | Web app: library, reader, accounts, settings. The only writer of users, progress and the catalog (`DATA_PATH`). Reads the library read-only |
|
||||
| `scraper` | 8091, internal only | Download queue and source adapters. The only writer of the library (`LIBRARY_PATH`) and of its own state (`SCRAPER_DATA_PATH`). Never publish or proxy this port |
|
||||
| `peer-gateway` | 8093, optional (`--profile peering`) | Shares chosen manga with other MangaCircus servers over mutual TLS ([peering.md](peering.md)) |
|
||||
|
||||
All services run from one image (`node:24-alpine`, no runtime dependencies) as uid/gid 1000. Each container
|
||||
has a read-only root file system, no capabilities, rotated logs, a health check and an in-container
|
||||
supervisor that restarts a crashed process. The reader works when the scraper is stopped.
|
||||
|
||||
Data on the host, all bind mounts (no named volumes):
|
||||
|
||||
| Path (`.env`) | Contents | Back up? |
|
||||
|---|---|---|
|
||||
| `LIBRARY_PATH` | The manga: `Manga/Book/Chapter/images` ([library-format.md](library-format.md)) | Yes, it is your collection |
|
||||
| `DATA_PATH` | `db.json` (users, progress, votes), catalog, discovery feeds, library index cache | Yes (built-in backup, section 8). `discovery-cache/` inside it is safe to delete and may be left out of backups: the next harvest rebuilds it. The same holds for `greatest-covers/` (local copies of the Greatest catalog's cover images, fetched again after the next start) |
|
||||
| `SCRAPER_DATA_PATH` | Queue, settings, source ledger, metadata cache | Included in the built-in backup |
|
||||
| `BACKUP_PATH` | Built-in backup archives | Keep it on another disk or NAS |
|
||||
| `SCRAPER_TOKEN_PATH` | Shared secret between reader and scraper | Regenerate if lost |
|
||||
|
||||
## 2. Requirements
|
||||
|
||||
- Linux x86_64 or arm64 with Docker Engine 24+ and the Compose plugin 2.24.4+ (`docker compose version`).
|
||||
Docker Desktop on Windows or macOS works for trying it out. Bind mounts are slow there for large libraries.
|
||||
- 1–2 CPU cores and 1 GB RAM are enough for a few readers. The first scan of a large library is
|
||||
I/O-bound (about 14 minutes for 300,000 pages on a NAS over NFS). Later starts reuse the index cache.
|
||||
- Disk: the library plus room for downloads in progress. The image is about 60 MB.
|
||||
- Git to fetch the code. Node is **not** needed on the host.
|
||||
|
||||
## 3. Get the code and prepare folders
|
||||
|
||||
```bash
|
||||
git clone <repository-url> mangacircus && cd mangacircus
|
||||
mkdir -p library data/scraper data/peering data/peering-usage backup .secrets
|
||||
openssl rand -hex 32 > .secrets/scraper_token && chmod 600 .secrets/scraper_token
|
||||
cp .env.example .env
|
||||
sudo chown -R 1000:1000 library data backup
|
||||
```
|
||||
|
||||
The containers run as uid 1000: the reader must be able to read the library and write `data/`, and the
|
||||
scraper must be able to write the library. On an NFS share with squashed ids, map all users to one account
|
||||
that owns the files. Don't use `chmod 777`. Bind mounts never create missing folders: a typo in a path
|
||||
fails the start instead of showing an empty library.
|
||||
|
||||
Already have a library? Point `LIBRARY_PATH` at it (folder format in [library-format.md](library-format.md)).
|
||||
Starting empty? Fill it from the web UI's scraper queue (section 6). To try the reader first, create five
|
||||
demo manga in an empty library after the build in section 5:
|
||||
`docker compose run --rm --no-deps scraper node tools/make-dummy.js /library`.
|
||||
|
||||
## 4. Configure `.env`
|
||||
|
||||
| Variable | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `LIBRARY_PATH` | `./library` | Library folder (absolute path recommended) |
|
||||
| `DATA_PATH` | `./data` | Reader data |
|
||||
| `SCRAPER_DATA_PATH` | `./data/scraper` | Scraper state |
|
||||
| `PEER_DATA_PATH`, `PEER_USAGE_PATH` | `./data/peering`, `./data/peering-usage` | Peering identity and transfer counters (folders must exist even when peering is off) |
|
||||
| `SCRAPER_TOKEN_PATH` | `./.secrets/scraper_token` | The secret file from section 3 |
|
||||
| `BACKUP_PATH` | unset = backups off | Folder for built-in backups (section 8) |
|
||||
| `READER_BIND` / `READER_PORT` | `127.0.0.1` / `8080` | Where the reader listens on the host. `0.0.0.0` = every interface (plain HTTP, LAN only) |
|
||||
| `TZ` | `UTC` | Time zone of the containers; affects the nightly update time and backup schedule |
|
||||
| `COOKIE_SECURE` | `0` | `1` when readers reach the site over HTTPS |
|
||||
| `READING_WINDOWS_JSON`, `READING_TIMEZONE`, `READING_QUIZ_*` | off | Optional server-enforced reading hours ([reading-limiter.md](reading-limiter.md)) |
|
||||
| `DISCOVERY_HARVEST_AT` | `03:30` | Daily time (`HH:MM`) of the nightly Discover harvest from AniList, in the container's local time zone (`TZ`). Only users who opted in are harvested; the admin can change the time or switch it off on the Discover page ([discovery.md](discovery.md)) |
|
||||
| `DISCOVERY_ANILIST_URL` | AniList's public API | Tests only: points the discovery harvest at a fixture server. Leave unset in production |
|
||||
| `GREATEST_COVER_WARM` | `1` | `0` stops the reader from downloading the Greatest catalog's covers (about 100 images, one per second, 10 s after start and once a day for missing ones). Tests and offline installs set `0`; the page then loads covers from the catalog's own URLs |
|
||||
| `APP_IMAGE`, `VCS_REF` | `mangacircus:local`, `development` | Image to run; set a pinned registry image to deploy prebuilt releases (section 7) |
|
||||
| `COMPOSE_FILE` | `docker-compose.yml` | Extra Compose files, `:`-separated (e.g. `docker-compose.yml:deploy/compose.https.yml`) |
|
||||
| `MANGACIRCUS_DOMAIN` | – | Host name for `deploy/compose.https.yml` |
|
||||
|
||||
Sign-up is closed: the first account becomes the admin, who then adds the other users. Reader settings
|
||||
not listed here (scan intervals, scan concurrency) are in the README's environment table.
|
||||
|
||||
## 5. Build and start
|
||||
|
||||
```bash
|
||||
docker compose config -q # no output = the configuration is valid
|
||||
docker compose build
|
||||
bash deploy/circus.sh up # reader first, waits for its health check, then the scraper
|
||||
bash deploy/circus.sh status
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
```
|
||||
SERVICE STATUS PORTS
|
||||
reader Up 1 minute (healthy) 127.0.0.1:8080->8080/tcp
|
||||
scraper Up 40 seconds (healthy) 8080/tcp
|
||||
{"ok":true,"scan":{"mangas":…,"stale":false,"error":null}, …}
|
||||
{"status":"idle","paused":false,"active":0,"queue":0, …}
|
||||
```
|
||||
|
||||
Open `http://127.0.0.1:8080` (or an SSH tunnel: `ssh -L 8080:127.0.0.1:8080 <host>`) and create the admin
|
||||
account. The library page fills as soon as the first scan finishes.
|
||||
|
||||
## 6. Access and adding manga
|
||||
|
||||
- **On the LAN:** set `READER_BIND=0.0.0.0` (or the host's LAN address), or use `compose.lan.yml` with
|
||||
`READER_LAN_BIND`. This is plain HTTP: logins and cookies travel unencrypted, so never forward it to the
|
||||
internet.
|
||||
- **HTTPS with the bundled Caddy:** point a DNS name at the host, open ports 80 and 443, set
|
||||
`MANGACIRCUS_DOMAIN=manga.example.org` and `COMPOSE_FILE=docker-compose.yml:deploy/compose.https.yml`,
|
||||
then `bash deploy/circus.sh up`. Caddy fetches the certificate. The reader gets no host port and secure
|
||||
cookies.
|
||||
- **Your own proxy:** keep `READER_BIND=127.0.0.1` and proxy `http://127.0.0.1:8080`. Alternatively, use
|
||||
`compose.vm.yml`, which joins the reader to an external Docker network (`PROXY_NETWORK`, default
|
||||
`mangacircus-edge`) and removes the host port. Set `COOKIE_SECURE=1`. All pages use relative URLs, so a
|
||||
sub-path such as `/manga/` works.
|
||||
- **Private access over a VPN** (WireGuard, Tailscale, …): bind the reader to the VPN address, or keep it on
|
||||
loopback and forward from the VPN client.
|
||||
- **Adding manga:** admins use **Settings → Admin → Scraper → Inspect & queue** with a series URL from a
|
||||
supported source, or copy folders into the library by hand. MangaDex is on by default. WeebCentral and
|
||||
MangaPill are unofficial sites that mostly host unlicensed copies, so they start switched off. Switch them
|
||||
on per source under **Hoster settings** if you may copy what you download there. While a source is off,
|
||||
its queued jobs wait and nightly update checks skip it. Upgraded installations keep the sources their
|
||||
queue already used. The reader
|
||||
notices new folders within `SCAN_INTERVAL` minutes. Rules for tools that write into the library:
|
||||
[scraper-guide.md](scraper-guide.md). Only download what you are allowed to copy; the scraper keeps
|
||||
conservative per-host rate limits and stops on 401/403/429 ([scraper/AGENT_RUNBOOK.md](../scraper/AGENT_RUNBOOK.md)).
|
||||
|
||||
## 7. Operating it
|
||||
|
||||
`deploy/circus.sh` wraps the safe sequences (run it from any folder; it finds the repository itself):
|
||||
|
||||
| Task | Command | Notes |
|
||||
|---|---|---|
|
||||
| Status | `bash deploy/circus.sh status` | services, reader health, queue |
|
||||
| Stop / start | `… down` / `… up` | `down` pauses the queue and waits for the running chapter (up to 15 minutes); `up` resumes only what `down` paused |
|
||||
| Restart | `… restart` or `… restart reader` | |
|
||||
| Logs | `… logs scraper 200` | |
|
||||
| Pause / resume downloads | `… pause` / `… resume` | Same as the buttons in Settings → Admin → Scraper |
|
||||
| Update to the newest code | `… update` | `git pull --ff-only`, rebuild, pause, recreate, resume |
|
||||
| Re-read edited `chapter.json` files | `… rescan` | Full reconciliation without an admin login; admins can also use Settings → Admin → Rescan |
|
||||
| Check the newest backup restores | `… restore-test` | Throwaway reader without network; prints `RESTORE TEST OK` |
|
||||
|
||||
**Deploying prebuilt images** (several hosts, or no building on the server): build once, push to your
|
||||
registry and pin the digest:
|
||||
|
||||
```bash
|
||||
docker build --build-arg VCS_REF=$(git rev-parse --short HEAD) -t registry.example.org/mangacircus:$(git rev-parse --short HEAD) .
|
||||
docker push registry.example.org/mangacircus:<commit> # note the sha256 digest it prints
|
||||
# on the server, in .env: APP_IMAGE=registry.example.org/mangacircus@sha256:<digest>
|
||||
docker compose pull && bash deploy/circus.sh restart
|
||||
```
|
||||
|
||||
Pinned digests make rollbacks exact: put the previous `APP_IMAGE` back and restart. Data formats only grow
|
||||
(new fields are ignored by older releases), but a rollback does not undo queue or catalog changes made by
|
||||
the newer release.
|
||||
|
||||
**Rules for operators and agents:**
|
||||
- Never run two scrapers, or a container and a native `node` process, against the same library or data
|
||||
folder.
|
||||
- Never run `docker compose down -v`. Stop with `deploy/circus.sh down`.
|
||||
- Never publish port 8091, and never print or commit `.secrets/` or `.env`.
|
||||
- One-off maintenance tools that write `chapter.json` files are dry runs by default. Run them with
|
||||
`--apply --writers-stopped` only after `docker compose stop scraper`, then `deploy/circus.sh rescan`.
|
||||
Examples: `tools/backfill-chapter-dates.js` and `tools/release-dates.js`, run inside the image with
|
||||
`docker compose run --rm --no-deps scraper node tools/<tool>.js /library …`.
|
||||
|
||||
## 8. Backups
|
||||
|
||||
- **Reader and scraper data:** set `BACKUP_PATH` to an existing folder on another disk or NAS (owned by uid
|
||||
1000), restart, and choose the schedule and retention under **Settings → Service health → Backups**. A
|
||||
run pauses the queue, waits for the current chapter, archives the data folder without stopping anything
|
||||
and resumes. Details, archive contents and the restore procedure: [backup.md](backup.md). Check it now
|
||||
and then with `deploy/circus.sh restore-test`.
|
||||
- **The library:** back it up with your usual file backup (it is plain folders and images), including
|
||||
hidden `.mcscraper-*` folders if downloads are in progress.
|
||||
- **Restore:** `deploy/circus.sh down`, move the data folder aside, unpack the archive into a fresh data
|
||||
folder (`tar -xzf circus-….tar.gz -C data`, owned by uid 1000), `deploy/circus.sh up`. Restored archives
|
||||
have the queue paused; resume it after checking.
|
||||
|
||||
## 9. Optional: server peering
|
||||
|
||||
Share selected manga with friends' MangaCircus servers: enable the `peering` profile
|
||||
(`docker compose --profile peering up -d peer-gateway`) and follow [peering.md](peering.md). The gateway
|
||||
needs its own reachable port (`PEER_BIND`, `PEER_PORT`) and uses mutual TLS with pinned certificates.
|
||||
|
||||
## 10. Troubleshooting
|
||||
|
||||
| Symptom | Cause / fix |
|
||||
|---|---|
|
||||
| `docker compose up` fails with "bind source path does not exist" | A folder from `.env` is missing; create it (section 3) |
|
||||
| Reader healthy but the library is empty | First scan still running (`docker compose logs reader` shows `[scan] …` when done), or `LIBRARY_PATH` points at the wrong folder |
|
||||
| Scraper jobs fail with `EACCES` | The library or `SCRAPER_DATA_PATH` is not writable by uid 1000 |
|
||||
| Backups fail with `EACCES` on a NAS | Map the share's users to the uid that owns the folder; the backup probes with a real write |
|
||||
| Settings → Scraper says the service is unavailable | Scraper container stopped or unhealthy: `deploy/circus.sh status`, `logs scraper` |
|
||||
| Edited `chapter.json` changes don't show | Run `deploy/circus.sh rescan` (in-place edits are picked up by full reconciliations, every 24 hours by default) |
|
||||
| Logins fail behind HTTPS | `COOKIE_SECURE=1` needs HTTPS end to end at the browser; with plain HTTP keep it `0` |
|
||||
|
||||
More background: [architecture.md](architecture.md) (components, API, security model),
|
||||
[import.md](import.md) (scanning and caching), [scraper-service.md](scraper-service.md) (queue, adapters,
|
||||
recovery) and [resilience.md](resilience.md) (supervisors and incidents).
|
||||
+93
-33
@@ -1,23 +1,28 @@
|
||||
# Discovery Scout (deterministic first release)
|
||||
# Discovery Scout
|
||||
|
||||
The reader now has a separate **Discover** page. This first slice deliberately uses
|
||||
the curated Greatest catalog, already-cached AniList identities/relations, library
|
||||
metadata, each user's Greatest votes, and explicit Discover feedback. It does not
|
||||
call an LLM, fetch new metadata, inspect chapter/image hosters, edit a collection,
|
||||
The reader has a separate **Discover** page. Suggestions come from the curated
|
||||
Greatest catalog, cached AniList relations of Greatest and library titles, and a
|
||||
wider pool that the reader harvests from AniList's public API for the users who
|
||||
opted in (design: [the wider-pool spec](superpowers/specs/2026-10-25-discovery-wider-pool-design.md)).
|
||||
Discovery does not call an LLM, inspect chapter/image hosters, edit a collection,
|
||||
or enqueue downloads. The reader remains the only writer of account and discovery
|
||||
state; no fourth Compose service, credential, or database is required.
|
||||
state; no further Compose service, credential, or database is required.
|
||||
|
||||
## Use
|
||||
|
||||
1. Open `discover.html`. Each account independently enables **Personalized
|
||||
discovery** (off by default). Reading activity is a separate opt-in. The
|
||||
mature-title switch only applies to candidates with a known mature tag; it is
|
||||
not a content-safety guarantee.
|
||||
2. An admin presses **Run discovery now** on the same page. Existing suggestions
|
||||
remain visible until the new feed is published. Runs are bounded to 50
|
||||
candidates and 20 suggestions per opted-in user, with no more than four from
|
||||
one period. A second concurrent run is rejected.
|
||||
3. A reader can open **Why this is here · evidence** and record Interested, Maybe,
|
||||
mature-title switch only affects titles AniList marks as adult or with a known
|
||||
mature tag; it is not a content-safety guarantee.
|
||||
2. **Refresh my suggestions** queues a harvest for your own account (at most once an
|
||||
hour). The button shows the queue position, `Fetching…` while it runs, and when
|
||||
the next refresh is possible. Your existing suggestions stay visible until the
|
||||
new feed is published.
|
||||
3. An admin can press **Run discovery now** (re-score from local data, no network)
|
||||
or **Harvest now** (fetch from AniList for every opted-in user, then re-score).
|
||||
A second concurrent run or harvest is rejected. A nightly harvest runs on its
|
||||
own; the admin panel can switch it off or change its time.
|
||||
4. A reader can open **Why this is here · evidence** and record Interested, Maybe,
|
||||
No thanks, or Hide. Feedback is private to the account. Dismissed/hidden
|
||||
titles disappear at once; other feedback influences the next run. The user
|
||||
can undo a choice by clicking its active button again.
|
||||
@@ -28,17 +33,72 @@ the existing source inspection and confirmation process to download anything.
|
||||
The title/year/format matching is intentionally conservative but not a verified
|
||||
identity merge. Missing year, cover, or status remains missing.
|
||||
|
||||
## Seeds from AniList
|
||||
|
||||
A harvest starts from each opted-in user's **anchors**: titles they voted
|
||||
`flame`/`good` in Greatest, marked Interested/Maybe in Discover, read (only with
|
||||
reading activity opted in), or own with an AniList ID. The 15 strongest anchors
|
||||
count. It then asks AniList for four seed kinds:
|
||||
|
||||
| Seed | What | Reason shown on the card |
|
||||
|---|---|---|
|
||||
| Community recommendations | "readers who liked X recommend Y" for each anchor | AniList readers who liked {anchor} recommend this ({rating}). |
|
||||
| Same creator | other works by the story/art authors of strong anchors | Also by {author} — you read/liked {anchor}. |
|
||||
| Top in your tags | well-rated titles in the user's three strongest tags | Top-rated in {tag}, one of your most-liked tags. |
|
||||
| Fresh | recently started, releasing and trending titles | New: started {year}, ongoing and trending. — or, when the cached record does not show an ongoing title started in the last two years, Trending on AniList right now. |
|
||||
|
||||
A title found by several seeds keeps every reason and evidence item; the card
|
||||
shows the strongest, the details list all of them, each with the date it was
|
||||
fetched. A feed holds at most 20 suggestions, at most 8 from one seed kind, at most
|
||||
4 from one Greatest period (or from related works), up to 4 fresh titles, and up to 4
|
||||
from outside your strongest tags.
|
||||
|
||||
- **Budgets:** a nightly or admin harvest makes at most 300 requests, a
|
||||
per-user refresh 60. Requests are sequential and limited to 25 a minute. When
|
||||
the budget is spent the harvest stops; what was fetched is still used.
|
||||
- **Cache:** `data/discovery-cache/`, one file per response. Titles,
|
||||
recommendations and creators stay fresh for 30 days, tag lists and the fresh
|
||||
list for 7 days; at most 5,000 files, oldest pruned first. Scoring reads only
|
||||
this cache and never the network, so a failed harvest means older candidates,
|
||||
never an empty feed.
|
||||
- **Nightly time:** `03:30` by default, or `DISCOVERY_HARVEST_AT`, in the
|
||||
container's local time zone (`TZ`). The admin panel overrides both and can turn
|
||||
the nightly harvest off. A night missed while the reader was down runs once
|
||||
after startup.
|
||||
- **Privacy:** for opted-in users only, the server sends AniList the IDs of
|
||||
their anchor titles (liked, read or owned), the AniList staff IDs of those titles'
|
||||
authors and the names of their top tags, without login, cookies or user names. Feeds and feedback stay private to this server; cache entries are
|
||||
not tied to a user.
|
||||
|
||||
### Stop reasons
|
||||
|
||||
The admin panel shows the last harvest as `status · requests/budget · stop
|
||||
reason · cache files`.
|
||||
|
||||
| Stop reason | Meaning | What happens next |
|
||||
|---|---|---|
|
||||
| `budget` | The request budget ran out | Normal; the next harvest continues where the cache is stale |
|
||||
| `rate-limited` | AniList answered 429 | Harvesting pauses; refresh buttons show "Paused by AniList" until the next nightly or admin harvest |
|
||||
| `forbidden` | AniList answered 401/403 | Same pause |
|
||||
| `challenge` | AniList answered with an HTML or challenge page | Same pause |
|
||||
| `upstream-errors` | Five timeouts or server errors in a row | The harvest ends; nothing is paused |
|
||||
| `error` | A local fault (for example a cache write) | See the reader log |
|
||||
|
||||
Discovery never retries with a different identity, user agent or host. A
|
||||
re-scoring failure after a harvest is shown as the admin panel's error line.
|
||||
|
||||
## Data and API
|
||||
|
||||
- `data/discovery.json`: atomic, reader-owned candidate and per-user feed
|
||||
snapshots. A previous valid version is retained as `discovery.json.prev`.
|
||||
A corrupt file makes reads/runs fail closed rather than overwrite it.
|
||||
- `data/discovery.json`: atomic, reader-owned per-user feed snapshots and the
|
||||
candidates those feeds show (the rest of the pool is only counted:
|
||||
`examined`, `clipped`), plus the last harvest record and the admin's harvest settings. A
|
||||
previous valid version is retained as `discovery.json.prev`. A corrupt file
|
||||
makes reads/runs/harvests fail closed rather than overwrite it.
|
||||
- `data/discovery-cache/`: harvested AniList responses; safe to delete.
|
||||
- `data/db.json`: consent in user settings and explicit feedback under
|
||||
`discoveryFeedback`. The account-deletion route removes that feedback.
|
||||
- `GET /api/discovery`: caller's own feed, settings, evidence, and feedback only.
|
||||
- `POST /api/discovery/feedback`: caller's candidate choice; requires opt-in.
|
||||
- `GET /api/admin/discovery` and `POST /api/admin/discovery/run`: admin-only
|
||||
status and manual generation. A run keeps the last good feed on failure.
|
||||
- Routes: [architecture.md](architecture.md) (`/api/discovery*`,
|
||||
`/api/admin/discovery*`) and §6.6 there for the harvester.
|
||||
|
||||
The job asks the private scraper state API for queued titles to avoid recommending
|
||||
them. If that API is unavailable, the run still uses only cached sources; the
|
||||
@@ -49,24 +109,24 @@ No scraper queue edit occurs.
|
||||
|
||||
From the project directory:
|
||||
|
||||
```powershell
|
||||
```bash
|
||||
node discovery/cli.js run --dry-run --data data --show
|
||||
npm run test:reader
|
||||
docker build -t mangacircus:discovery-test .
|
||||
$env:APP_IMAGE='mangacircus:discovery-test'; npm run test:discovery-ui
|
||||
APP_IMAGE=mangacircus:discovery-test npm run test:discovery-ui
|
||||
```
|
||||
|
||||
The dry run reads `db.json`, a saved library index if present, Greatest, and
|
||||
cached AniList metadata. It writes nothing and does not contact providers. The
|
||||
Compose/browser suite uses a disposable data tree and queue, not the live one.
|
||||
cached AniList metadata. It does not contact AniList or any other provider and does
|
||||
not publish `discovery.json`; reading the harvest cache may still delete a corrupt
|
||||
cache file, as the reader would. The
|
||||
Compose/browser suite uses a disposable data tree and queue, not the live one,
|
||||
and points `DISCOVERY_ANILIST_URL` at a closed local port so no harvest reaches
|
||||
AniList.
|
||||
|
||||
## Deliberate limits and next stage
|
||||
## Limits
|
||||
|
||||
This slice has no scheduled run, external metadata discovery, LLM, admin seed
|
||||
form, edition-finding action, or explicit purge control for old feed snapshots.
|
||||
Disabling consent stops feed display and future personalized generation, but
|
||||
existing snapshot/feedback is retained until account deletion or a later data
|
||||
management feature. Candidate quality is limited by the curated list and local
|
||||
metadata cache. Add external public metadata feeds and optional LLM only behind
|
||||
the existing bounded candidate/evidence contract after this deterministic path
|
||||
has been used and reviewed.
|
||||
There is no admin seed form, edition-finding action, LLM rationale, or explicit
|
||||
purge control for old feed snapshots. Disabling consent stops feed display,
|
||||
harvesting and future personalized generation for that account, but existing
|
||||
snapshot/feedback is retained until account deletion.
|
||||
+1
-1
@@ -23,7 +23,7 @@ Within each cohort, **Rank by my vote** sorts flame → good → neutral → sku
|
||||
|
||||
The one-time seed is curated from the research document. All 106 catalog entries now have a cover URL. Most were matched to public AniList manga metadata; the five initial gaps were checked individually. Phoenix (1954) and Nausicaä use matching AniList records; Kitarō uses Drawn & Quarterly's Kitaro edition, Someone Hertz uses Shonen Jump's first-volume cover, and Hitoner uses Shueisha's 2026 first-volume cover. Hitoner's AniList 2024 entry represents an earlier work, so it was **not** used for the 2026 series. The five added URLs returned image content on 2026-10-09, but remote hosts can change them; the UI still shows a title-letter fallback if an image fails.
|
||||
|
||||
`node tools/enrich-greatest-catalog.js` is an **optional manual** public-AniList metadata refresh, paced at one request every 1.5 seconds. It only accepts matching manga title and **original serialization year**, leaving uncertain results untouched rather than using a sequel or remake cover. The dashboard makes **no metadata calls when loading**; only an admin pressing **Search MangaDex** triggers a bounded lookup. Browser covers are lazy-loaded from their listed source; local library covers take precedence.
|
||||
`node tools/enrich-greatest-catalog.js` is an **optional manual** public-AniList metadata refresh, paced at one request every 1.5 seconds. It only accepts matching manga title and **original serialization year**, leaving uncertain results untouched rather than using a sequel or remake cover. The dashboard makes **no metadata calls when loading**; only an admin pressing **Search MangaDex** triggers a bounded lookup. Local library covers take precedence. For the rest, the reader keeps its own copy of each catalog cover in `data/greatest-covers/` (`greatest/covers.js`): it downloads missing ones one at a time, about one a second, 10 seconds after start and once a day, stops on HTTP 401/403/429, and accepts only JPEG/PNG/WebP/GIF/AVIF images up to 2 MB from the catalog's own HTTPS URLs. The page loads covers from `/api/greatest-cover`, which the browser caches for a year; until a copy exists that route redirects to the catalog URL. Hotlinking about 100 covers per page load made the image CDN drop some of them at random. `GREATEST_COVER_WARM=0` switches the downloads off.
|
||||
|
||||
Public catalog chapter counts are often absent; the UI says “Chapters unknown” rather than inventing one. For owned titles, the local library count wins. Status comes from AniList's releasing/finished flag or local library metadata, and may be unknown. Official source links may be publisher editions, chapter services, or an award/info lead; they do not imply all chapters are available in every country. No unverified hoster coverage is claimed.
|
||||
|
||||
|
||||
+2
-2
@@ -150,7 +150,7 @@ Everything is plain data, and the whole index lives in memory.
|
||||
|
||||
The reader lays pages out before they load, so it needs every image's size. Instead of decoding images, the scanner reads the **first 32 KB** and parses the header; only if the size isn't there yet does it read on to 256 KB (and, for JPEG, the whole file). Parsing a prefix either finds nothing or the same size as the whole file, so the small first read never changes a result.
|
||||
|
||||
Uncached chapter folders and their images are probed **`SCAN_CONCURRENCY` at a time** (default 8, 1–64; chapters of a book in parallel, then assembled in folder order). A first scan over a NAS is bound by per-file latency, not bandwidth: on the server against a Synology (NFS) the old sequential 256 KB probe took about 20 ms per image; see §3.1 for the measurements. `SCAN_CONCURRENCY=1` restores the sequential order.
|
||||
Uncached chapter folders and their images are probed **`SCAN_CONCURRENCY` at a time** (default 8, 1–64; chapters of a book in parallel, then assembled in folder order). A first scan over a NAS is bound by per-file latency, not bandwidth: on a VM reading a Synology NAS over NFS the old sequential 256 KB probe took about 20 ms per image; see §3.1 for the measurements. `SCAN_CONCURRENCY=1` restores the sequential order.
|
||||
|
||||
| Format | Where the size is |
|
||||
|--------|-------------------|
|
||||
@@ -163,7 +163,7 @@ Uncached chapter folders and their images are probed **`SCAN_CONCURRENCY` at a t
|
||||
|
||||
### 3.1 Measured first-scan speed
|
||||
|
||||
the server (Debian VM, 4 vCPU) reading a Synology over NFS 4.0, page cache dropped before each run, production
|
||||
A Debian VM (4 vCPU) reading a Synology NAS over NFS 4.0, page cache dropped before each run, production
|
||||
running alongside. Each variant scanned a different manga of ~5,000 pages (the NAS caches what it has just read).
|
||||
|
||||
| Variant | ms per image (round 1 / round 2) |
|
||||
|
||||
+43
-1
@@ -208,11 +208,53 @@ An AniList refresh with a different ID is stopped for review before chapter publ
|
||||
|-------|------|---------|
|
||||
| `title` | string | Chapter title without "Chapter N" (default: title part of the folder name) |
|
||||
| `source` | `{name, url}` or string | Where this version came from (default: `[source]` in the folder name) |
|
||||
| `added` | ISO date/time string | Release (or download) date, used for "recently updated" (default: the folder's modification time) |
|
||||
| `added` | ISO date/time string | When this version entered **this** library, used for "recently updated" (default: the folder's modification time). The scraper keeps it when it replaces a version |
|
||||
| `x-dates` | object | Other chapter dates, written by the scraper (see below) |
|
||||
| `extra` | boolean | Marks the chapter as extra without moving it |
|
||||
| `priority` | number | Tie-breaker between versions with equal votes. Higher wins (default `0`) |
|
||||
| `direction` | as above | Overrides book/manga direction, e.g. a colour webtoon special inside a printed series |
|
||||
|
||||
#### Chapter dates (`x-dates`)
|
||||
|
||||
```json
|
||||
"added": "2026-10-17T15:04:03.424Z",
|
||||
"x-dates": {
|
||||
"version": 1,
|
||||
"source": { "at": "2026-10-18T15:06:05.348Z", "name": "WeebCentral", "batch": false },
|
||||
"release": { "at": "2023-09-17", "precision": "day", "source": "MangaUpdates · Viz" },
|
||||
"ecosystem": { "at": "2026-10-17T15:04:03.424Z", "via": "this server" }
|
||||
}
|
||||
```
|
||||
|
||||
- `source`: when the source site uploaded this version (WeebCentral's full chapter list,
|
||||
MangaDex `publishAt`). `batch: true` means it arrived with 3 or more other uploads
|
||||
less than 10 minutes apart, which is a site import, not a release.
|
||||
- `release`: the official public release (`precision` `day`, `month` or `year`). It is
|
||||
filled from MangaUpdates releases by publishers such as Viz, MANGA Plus or Webtoon
|
||||
(`scraper/metadata/release-dates.js`). Those lists include English re-releases of old
|
||||
chapters, so a date is kept only when all of these hold:
|
||||
- it lies within the work's original run on AniList;
|
||||
- no scanlation or source upload shows the chapter more than 4 days earlier;
|
||||
- it fits the release order: walking down from the newest chapter, a lower chapter
|
||||
can't be dated more than 2 days after a higher one.
|
||||
|
||||
Series whose chapter numbers restart in a later book are skipped.
|
||||
- The scraper refreshes ongoing series nightly before its update checks, but only
|
||||
while no download is running.
|
||||
- `node tools/release-dates.js <library>` covers a whole library: a dry run by
|
||||
default, then `--apply --writers-stopped`.
|
||||
- A date that this job wrote and no longer accepts is removed. Release dates from any
|
||||
other source stay.
|
||||
- `ecosystem`: when the chapter first entered any Manga Circus library. Peer transfers
|
||||
carry it, and the earliest known date always wins. A peer's claim can't be earlier than
|
||||
the source upload or later than now.
|
||||
|
||||
The chapter list shows one release date per chapter, choosing the first tier that has
|
||||
a date: `release` → `source` unless `batch` (shown as ≈) → `ecosystem` (◎) → `added` (↓).
|
||||
Each tier takes the earliest value across the chapter's versions. Existing libraries get
|
||||
these fields from `node tools/backfill-chapter-dates.js <library>`: a dry run by default,
|
||||
then `--apply --writers-stopped` and a deep Rescan.
|
||||
|
||||
Manual directions have priority at **chapter → book → manga**. A `directionOverride`
|
||||
or a plain `direction` without matching generated provenance is manual. Otherwise,
|
||||
the scanner checks the version's existing image dimensions for coherent tall strips
|
||||
|
||||
+26
-18
@@ -41,24 +41,24 @@ separate lifetimes.
|
||||
The optional `peer-gateway` Compose profile publishes port 8093 on **127.0.0.1** by
|
||||
default. It uses the same image as reader/scraper, a read-only library mount, a
|
||||
read-only peering policy/identity mount, and its own writable usage-counter mount.
|
||||
The gateway needs an external-facing Docker network for its host port to work on
|
||||
Docker Desktop; it also joins the private backend. It has no reader account database,
|
||||
The gateway needs an external-facing Docker network for its host port to be
|
||||
published; it also joins the private backend. It has no reader account database,
|
||||
scraper queue, Docker socket, or writable library mount. It must never be scaled
|
||||
past one instance for the same usage-counter directory.
|
||||
|
||||
1. Build the image and start the normal stack as documented in
|
||||
[`compose-deployment.md`](compose-deployment.md). Set the local server name and
|
||||
[`deployment.md`](deployment.md). Set the local server name and
|
||||
generate its certificate in **Settings → Admin → Peering**. Keep
|
||||
`/data/peering/identity.key` private and backed up.
|
||||
2. Start the optional service with `docker compose --profile peering up -d
|
||||
peer-gateway`. Verify `docker compose --profile peering ps` shows healthy and
|
||||
`https://127.0.0.1:8093/health` responds. `/health` is deliberately unauthenticated
|
||||
and contains only `{ "ok": true }`.
|
||||
3. Give the other server a reachable HTTPS endpoint for port 8093. A private
|
||||
Tailscale TCP forward is recommended for the pilot; the default loopback binding
|
||||
is not remotely reachable. Do not expose port 8093 on the public internet merely
|
||||
to make pairing work. Use a stable tailnet DNS name, not a changing IP. Keep
|
||||
the scraper (8091) private.
|
||||
3. Give the other server a reachable HTTPS endpoint for port 8093. A private VPN
|
||||
(WireGuard, Tailscale or similar) with a TCP forward is recommended; the default
|
||||
loopback binding is not remotely reachable. Do not expose port 8093 on the public
|
||||
internet merely to make pairing work. Use a stable DNS name, not a changing IP.
|
||||
Keep the scraper (8091) private.
|
||||
|
||||
The `PEER_BIND`, `PEER_PORT`, `PEER_DATA_PATH`, and `PEER_USAGE_PATH` Compose variables
|
||||
control the bind and persistent paths. The gateway's internal port remains 8093.
|
||||
@@ -110,8 +110,18 @@ an admin can distinguish pairing from permission.
|
||||
| `GET /peer/v1/access` | Pairing state, certificate expiry and current grant status/scopes; no catalog |
|
||||
| `GET /peer/v1/catalog?cursor=N` | Up to 50 approved opaque title IDs, safe metadata, a stable work identity and cover availability |
|
||||
| `GET /peer/v1/covers/{mangaId}` | Approved cover image only; bounded to 5 MiB and charged to peer byte quota |
|
||||
| `GET /peer/v1/manga/{id}?cursor=N` | Up to 100 complete chapter IDs, public metadata and the same stable work identity |
|
||||
| `GET /peer/v1/manga/{id}/chapters/{id}` | Ordered page IDs, MIME, byte size, SHA-256 and ETag |
|
||||
| `GET /peer/v1/manga/{id}?cursor=N` | Up to 100 complete chapter IDs, public metadata, chapter `dates` and the same stable work identity |
|
||||
| `GET /peer/v1/manga/{id}/chapters/{id}` | Ordered page IDs, MIME, byte size, SHA-256 and ETag, plus chapter `dates` |
|
||||
|
||||
Chapter `dates` are `{ ecosystem, source?, release? }`. `ecosystem` is the earliest
|
||||
date at which any version of that chapter number entered a Manga Circus library known
|
||||
to the sender. Older chapters fall back to their `added` date. The receiver stores the
|
||||
date as `x-dates.ecosystem` (`via: "peer <alias>"`) and only lets it lower an existing
|
||||
date, never raise it. See `docs/library-format.md`. Older peers omit `dates`, and the
|
||||
receiver then uses its own arrival time. A sync also lowers the ecosystem date of
|
||||
chapters this library already has, including ones it scraped itself, when the peer
|
||||
knows an earlier date for the same book and chapter number. The job reports these as
|
||||
`ecosystemLowered`. The reader shows the new date after its next full reconciliation.
|
||||
| `GET /peer/v1/blobs/{id}` | One image, optional `Range: bytes=N-` and `If-Match`; 206 on resume |
|
||||
|
||||
Unknown/unshared IDs return 404. Revoked serving returns 403; bad/missing grants
|
||||
@@ -122,9 +132,10 @@ not shared. Catalog and chapter lists are short-lived in-memory caches. Turning
|
||||
serving blocks subsequent requests immediately, though an already streaming image
|
||||
may finish its current chunk/response.
|
||||
|
||||
The initial implementation sends original image bytes; it does not transcode them.
|
||||
See [`vision/server-peering-compression-test.md`](../vision/server-peering-compression-test.md)
|
||||
for the measured reason. The transport accepts a small amount of controlled page
|
||||
The initial implementation sends original image bytes; it does not transcode them:
|
||||
lossless compression of already encoded pages saved only about 6% of the transfer size
|
||||
in a measurement over 287 pages (webtoon strips about 11%, PNG chapters under 1%);
|
||||
`node tools/benchmark-peer-transfer.js` repeats it on your own library. The transport accepts a small amount of controlled page
|
||||
parallelism, not unbounded connections.
|
||||
|
||||
## Tests and safe operation
|
||||
@@ -139,11 +150,8 @@ checks the queue remains paused, and tears both stacks down. The UI suite checks
|
||||
guided setup, approvals, and non-admin boundary. An isolated single-stack Compose
|
||||
acceptance is:
|
||||
|
||||
```powershell
|
||||
$env:APP_IMAGE = 'mangacircus:local' # replace with the image just built
|
||||
$env:QA_PORT = '8188'
|
||||
$env:QA_PEER_PORT = '8193'
|
||||
node tools/test-compose.js --peer-gateway --peer-ui
|
||||
```bash
|
||||
APP_IMAGE=mangacircus:local QA_PORT=8188 QA_PEER_PORT=8193 node tools/test-compose.js --peer-gateway --peer-ui
|
||||
```
|
||||
|
||||
It makes disposable library/data, four containers, a certificate and a browser
|
||||
|
||||
+3
-3
@@ -15,7 +15,7 @@ The reader is the only writer of users/progress and the catalog. The scraper is
|
||||
only queue owner and content publisher, and it records its own failures and incidents.
|
||||
Each service's supervisor owns only that child process. The coding agent is an
|
||||
**escalation/maintenance operator**, not a necessary part of normal downloading.
|
||||
The recurring Codex check is an additional backstop, not the runtime watchdog.
|
||||
A periodic check by an operator or coding agent is an additional backstop, not the runtime watchdog.
|
||||
|
||||
## Automatic recovery implemented
|
||||
|
||||
@@ -89,8 +89,8 @@ Storage:
|
||||
- `data/scraper/supervisor.lock`, `daemon.lock`: native process ownership; transient.
|
||||
|
||||
Scraper and supervisor emit structured logs. The queue history retains
|
||||
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native Windows
|
||||
supervisor logs remain append-only: archive them during stopped maintenance if needed.
|
||||
200 incidents; the UI shows a bounded subset. Docker logs are rotated. Native supervisor logs
|
||||
(`tools/run-supervisor.ps1`) remain append-only: archive them during stopped maintenance if needed.
|
||||
Never export user databases, tokens, cookies, signed query strings or unredacted environments.
|
||||
|
||||
## What self-repair intentionally does not mean
|
||||
|
||||
@@ -31,11 +31,9 @@ npm start # reader only
|
||||
npm run start:scraper # daemon only
|
||||
```
|
||||
|
||||
These commands are for native development only. The current Windows installation
|
||||
runs the Docker Compose stack; the old Scheduled Task is disabled.
|
||||
Never start a native supervisor or daemon against its mounted state. Its observed
|
||||
ports, logs, persisted settings, restart procedure, and recovery snapshot are recorded in
|
||||
[`local-deployment.md`](local-deployment.md).
|
||||
These commands are for native development. Deployments run the Docker Compose stack
|
||||
([`deployment.md`](deployment.md)); never start a native supervisor or daemon against
|
||||
the state a container has mounted.
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
@@ -207,7 +205,7 @@ All responses are JSON. When `SCRAPER_TOKEN` is set, send
|
||||
| `POST` | `/api/queue/clear-completed` | `{source?}`; remove inactive successful download job rows only; preserve library, history, checkpoints and other statuses |
|
||||
| `POST` | `/api/queue/reorder` | `{ids}` containing every queue id exactly once |
|
||||
| `POST` | `/api/control` | `{action:"pause"|"resume"|"retry"|"inspect"|"confirm"|"skip",id?}`; `pause-source`/`resume-source` use `id` as the source name and do not change the global pause |
|
||||
| `POST` | `/api/settings` | `sourceRates.{adapter}.{minMs,maxMs,chapterMinMs,chapterMaxMs}` plus `autoUpdateEnabled` and local `autoUpdateTime` (`HH:MM`) |
|
||||
| `POST` | `/api/settings` | `sourceRates.{adapter}.{minMs,maxMs,chapterMinMs,chapterMaxMs}`, `sourcesEnabled.{weebcentral,mangadex,mangapill}` (booleans; WeebCentral and MangaPill default to off, except on installations whose queue already used them; a switched-off source refuses new jobs, keeps queued ones waiting and is skipped by update checks), plus `autoUpdateEnabled` and local `autoUpdateTime` (`HH:MM`) |
|
||||
| `POST` | `/api/discovery/run` | Queue an immediate update check for every enabled manga |
|
||||
| `POST` | `/api/library/auto-update` | `{id,enabled}` updates `manga.json` atomically |
|
||||
| `POST` | `/api/legacy/publish-ready` | Paused-only validation/publication of completed legacy work; converts its checkpoint to incremental mode |
|
||||
|
||||
@@ -73,8 +73,8 @@ not an editor for Docker environment variables or an unrestricted restart consol
|
||||
`npm test` covers route defaults, deep links, ownership and existing service/API
|
||||
behavior. `node tools/test-compose.js --settings-ui` adds optional Playwright
|
||||
headless browser QA against the disposable Compose fixture, never the real user
|
||||
database or queue. Make Playwright available on the development machine; Windows
|
||||
uses installed headless Edge by default, overridable with `QA_BROWSER_CHANNEL`.
|
||||
database or queue. Make Playwright available on the development machine; on Windows it
|
||||
uses the installed headless Edge by default, overridable with `QA_BROWSER_CHANNEL`.
|
||||
The application itself remains dependency-free.
|
||||
|
||||
The UI check verifies desktop and 390×844 mobile navigation, form saves, draft
|
||||
|
||||
@@ -0,0 +1,337 @@
|
||||
# Discovery wider pool Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Widen Discover's candidate pool with four AniList-sourced seed kinds (community recommendations, same creator, top in tags, fresh), harvested server-side nightly and per user, scored locally.
|
||||
|
||||
**Architecture:** New reader-side modules: a file cache, a rate-limited AniList client, anchor selection, and a single-flight harvester. The existing engine scores from Greatest + relations + harvest cache; `server.js` wires routes, the nightly scheduler and rescoring. Harvest (network) and scoring (local) stay separate, so a failed harvest never empties a feed.
|
||||
|
||||
**Tech Stack:** Node.js built-ins only (`node:test`, `fetch`, `fs`), plain browser JS, Playwright for the Compose UI test.
|
||||
|
||||
**Spec:** [`docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md`](../specs/2026-10-25-discovery-wider-pool-design.md)
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- No runtime dependencies; Node built-ins only (`AGENTS.md`).
|
||||
- One writer: only the reader writes `data/discovery.json` and `data/discovery-cache/`; never write under `data/scraper/`.
|
||||
- All fetching runs in the reader container, never in the browser.
|
||||
- AniList: at most 25 requests per minute, one at a time; stop the whole harvest on 429, 403 or a challenge; never retry with a changed identity, user agent or host.
|
||||
- Budgets: nightly and admin harvests 300 requests, per-user refresh 60. Per-user cooldown 1 hour. At most 15 anchors per user.
|
||||
- Cache TTL: `title`, `recs`, `creator` 30 days; `tagTop`, `fresh` 7 days. Cap 5,000 files, oldest `fetchedAt` pruned first.
|
||||
- Feed: 20 items; at most 8 per seed kind, at most 4 per Greatest period/`Related works`; up to 4 slots reserved for `fresh`, up to 4 for candidates outside the user's top-3 tags. Candidate cap 2,000.
|
||||
- Personalized discovery and reading activity remain separate opt-ins, both off by default.
|
||||
- Discovery never edits the scraper queue, library or collections.
|
||||
- Tests never call live AniList; fixtures in `test/fixtures/anilist/` are handcrafted and contain no personal data.
|
||||
- Public repo: no host names, IPs, local paths or personal seeds anywhere.
|
||||
- Commits end with `Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>`; push `feature/discovery-wider-pool` after each task.
|
||||
|
||||
## Review Focus
|
||||
|
||||
1. **A fresh install with an empty cache and no network** must behave exactly like Phase A: the feed comes from Greatest and relations, and the harvest record shows a stop reason. Test in Task 6 (`engine: empty harvest cache equals Phase A pool`) and Task 4 (`harvest: network error stops with upstream-errors after 5`).
|
||||
2. **A user with no anchors** (opted in, no votes, empty library) gets no harvest requests except the global `fresh` list, and still a feed. Test in Task 4 (`plan for anchorless user only fetches fresh`).
|
||||
3. **Old `discovery.json` files without `harvest`, `settings` or `nightly` fields** must load without the fail-closed error. Test in Task 5 (`store loads a Phase A file without new fields`).
|
||||
4. **A per-user rescore** must keep every other user's feed and the candidates those feeds reference. Test in Task 6 (`mergeUserFeed keeps other users' referenced candidates`).
|
||||
5. **AniList text containing HTML or script** (titles, tag names) is shown as text, never as markup. Test in Task 8 (UI test renders a fixture title `<img src=x onerror=alert(1)>` as text).
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Discovery cache
|
||||
|
||||
**Files:**
|
||||
- Create: `discovery/cache.js`
|
||||
- Test: `test/discovery-cache.test.js`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
- `KINDS = ['title', 'recs', 'creator', 'tagTop', 'fresh']`
|
||||
- `TTL_MS = { title: 30*86400000, recs: 30*86400000, creator: 30*86400000, tagTop: 7*86400000, fresh: 7*86400000 }`
|
||||
- `class DiscoveryCache { constructor(dir: string, { maxFiles = 5000, now = Date.now } = {}) }`
|
||||
- `get(kind, key) -> { schema: 1, kind, key, fetchedAt: ISO string, data } | null`. A corrupt or schema-mismatched file is deleted and returns `null`.
|
||||
- `isFresh(kind, key) -> boolean`: true when an entry exists and `now() - fetchedAt < TTL_MS[kind]`.
|
||||
- `put(kind, key, data) -> entry`: atomic write (temp file + rename), then pruning.
|
||||
- `list(kind) -> entry[]`, including expired entries.
|
||||
- `stats() -> { files: number, bytes: number }`
|
||||
- File path: `<dir>/<kind>/<base64url(String(key))>.json`. An unknown kind throws `TypeError`.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests** in `test/discovery-cache.test.js`, each using an `fs.mkdtempSync` dir and a fake `now`:
|
||||
- `cache round-trips an entry`: `put('title', 42, {a:1})`, then `get('title', 42).data` deep-equals `{a:1}` and `fetchedAt` equals `new Date(now()).toISOString()`.
|
||||
- `freshness follows TTL per kind`: with `now` advanced by 8 days, `isFresh('fresh','global')` is `false` and `isFresh('title',1)` is `true`. At 31 days `isFresh('title',1)` is `false`, while `get` still returns the entry.
|
||||
- `prune keeps the newest maxFiles`: with `maxFiles: 3` and 5 puts at increasing `now`, `stats().files === 3` and the first two keys return `null`.
|
||||
- `corrupt entry is deleted`: write `{broken` to the entry file; `get` returns `null` and the file no longer exists.
|
||||
- `odd keys are file-safe`: `put('tagTop', 'genre:Slice of Life/../x', …)` lands inside `<dir>/tagTop/`.
|
||||
- `unknown kind throws`.
|
||||
- [ ] **Step 2: Run** `node --test test/discovery-cache.test.js`. Expected: FAIL, `Cannot find module '../discovery/cache'`.
|
||||
- [ ] **Step 3: Implement `discovery/cache.js`** per the Interfaces. Prune by reading each entry's `fetchedAt` across all kinds, sorting oldest first, and deleting until `files <= maxFiles`. Follow `discovery/store.js` style.
|
||||
- [ ] **Step 4: Run** `node --test test/discovery-cache.test.js`. Expected: PASS.
|
||||
- [ ] **Step 5: Commit** `Discovery cache with per-kind TTL and size cap (BL-0051)` and push.
|
||||
|
||||
### Task 2: AniList client
|
||||
|
||||
**Files:**
|
||||
- Create: `discovery/anilist.js`, `test/fixtures/anilist/{title,recs,creator,tagtop,fresh,novel}.json`
|
||||
- Test: `test/discovery-anilist.test.js`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
- `createLimiter({ perMinute = 25, now = Date.now, sleep = ms => new Promise(r => setTimeout(r, ms)) }) -> { wait(): Promise<void> }`. Calls are serialized and spaced at least `60000 / perMinute` ms apart.
|
||||
- `class AniListStop extends Error { reason: 'rate-limited' | 'forbidden' | 'challenge' }`
|
||||
- `class AniListSkip extends Error { reason: 'timeout' | 'upstream' | 'invalid' }`
|
||||
- `normalizeMedia(media) -> Record | null`. Returns `null` unless `format` is `MANGA` or `ONE_SHOT`.
|
||||
- `Record = { anilistId, title, altTitles: string[], format, isAdult: boolean, status: 'ongoing'|'completed'|'unknown', year: number|null, averageScore: number|null, popularity: number|null, trending: number|null, tags: [{ name, rank, kind: 'genre'|'tag', isAdult: boolean }], authors: [{ staffId, name, role }], coverUrl: string|null, siteUrl: string|null }`.
|
||||
- Genres map to `{ rank: 100, kind: 'genre', isAdult: false }`.
|
||||
- Status: `RELEASING` → `ongoing`, `FINISHED` → `completed`, anything else → `unknown`.
|
||||
- `authors` keeps the staff roles matched by the regex in `scraper/metadata/anilist.js` (`original story|story(?: & art)?|art|illustration`).
|
||||
- `createClient({ endpoint = process.env.DISCOVERY_ANILIST_URL || 'https://graphql.anilist.co', limiter = createLimiter(), fetchImpl = fetch, timeoutMs = 30000, today = () => new Date() })` returns:
|
||||
- `title(id) -> Record`
|
||||
- `recs(id) -> [{ anilistId, rating, record }]`, keeping only `rating > 0`
|
||||
- `creator(staffId) -> [{ anilistId, role, record }]`
|
||||
- `tagTop({ name, kind }) -> [{ anilistId, record }]`
|
||||
- `fresh() -> [{ anilistId, record }]`
|
||||
- Entries whose `normalizeMedia` returns `null` are dropped.
|
||||
- Queries share one `MEDIA_FIELDS` fragment: `id format isAdult title{romaji english native} synonyms status startDate{year} averageScore popularity trending genres tags{name rank isAdult} staff(perPage:20){edges{role node{id name{full}}}} coverImage{extraLarge} siteUrl`. Page sizes and sort orders come from spec §2.1:
|
||||
- `tagTop` uses `popularity_greater: 1000`, with `genre_in` or `tag_in` by `kind`.
|
||||
- `fresh` uses `status: RELEASING, startDate_greater: <(today.year − 2) * 10000 + month * 100 + day>, sort: TRENDING_DESC, perPage: 50`.
|
||||
- All list queries use the full fragment, so no follow-up `title` fetches are needed. This replaces the spec's follow-up stage on purpose.
|
||||
- Error mapping, applied after `await limiter.wait()`:
|
||||
- HTTP 429 → `AniListStop('rate-limited')`
|
||||
- HTTP 401 or 403 → `AniListStop('forbidden')`
|
||||
- non-JSON body or `content-type` text/html → `AniListStop('challenge')`
|
||||
- abort or timeout → `AniListSkip('timeout')`
|
||||
- HTTP 5xx → `AniListSkip('upstream')`
|
||||
- missing `data.Media` / `data.Page.media` / `data.Staff`, or a media without an integer `id` → `AniListSkip('invalid')`
|
||||
- Requests send `content-type: application/json` and the fixed header `user-agent: MangaCircus-Discovery/1.0`. Do not reuse `SlowHttp` identity rotation.
|
||||
|
||||
- [ ] **Step 1: Write fixtures** with minimal handcrafted responses:
|
||||
- `recs.json` has 3 recommendations, one with `rating: -2` and one with `format: NOVEL`.
|
||||
- `title.json` has `isAdult: false`, 2 genres and 2 tags, one of them `isAdult: true`.
|
||||
- [ ] **Step 2: Write the failing tests** in `test/discovery-anilist.test.js`. Use a stub `fetchImpl` returning `new Response(JSON.stringify(fixture), { status, headers })` and a fake limiter `{ wait: async () => {} }`:
|
||||
- `normalizeMedia maps fields and drops novels`: asserts the `Record` shape on `title.json`; `normalizeMedia({ ...media, format: 'NOVEL' }) === null`.
|
||||
- `recs keeps positive ratings and manga formats`: exactly 1 entry with its rating.
|
||||
- `fresh query sends startDate two years back`: with `today = new Date('2026-10-25')`, the request body's variables contain `startDate_greater: 20241025`.
|
||||
- `tagTop uses genre_in for genres and tag_in for tags`.
|
||||
- `429 stops`, `403 stops`, `html body is a challenge`: each throws `AniListStop` with its reason.
|
||||
- `5xx and timeout skip`, `missing data is invalid`: each throws `AniListSkip` with its reason.
|
||||
- `limiter spaces calls 2400 ms apart`: the fake `sleep` records each duration and advances the fake `now` by it. Three `wait()` calls issued at the same instant record `[0, 2400, 2400]`.
|
||||
- [ ] **Step 3: Run** `node --test test/discovery-anilist.test.js`. Expected: FAIL (module missing).
|
||||
- [ ] **Step 4: Implement `discovery/anilist.js`** per the Interfaces.
|
||||
- [ ] **Step 5: Run** the same command. Expected: PASS.
|
||||
- [ ] **Step 6: Commit** `Rate-limited AniList client for discovery seeds (BL-0051)` and push.
|
||||
|
||||
### Task 3: Anchors and top tags
|
||||
|
||||
**Files:**
|
||||
- Create: `discovery/anchors.js`
|
||||
- Test: `test/discovery-anchors.test.js`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `Record` from Task 2.
|
||||
- Produces:
|
||||
- `ANCHOR_WEIGHTS = { flame: 5, good: 3, interested: 4, maybe: 2, read: 3, owned: 1 }`
|
||||
- `MAX_ANCHORS = 15`
|
||||
- `anchorsFor({ user, db, library, greatest }) -> [{ anilistId: number, weight, source: keyof ANCHOR_WEIGHTS, title }]`. Returns `[]` unless `user.settings.discoveryEnabled`. Sources:
|
||||
- Greatest votes: `db.greatestRatings[user.name][entryId]` is `flame` or `good`, and `entry.metadata.id` is set.
|
||||
- Feedback: `db.discoveryFeedback[user.name]['anilist:N']` is `interested` or `maybe`.
|
||||
- `read`: only with `settings.discoveryReadingActivity`; a library manga with `x-anilist.id` and any chapter in `db.progress[user.name][manga.id]` that has `done`.
|
||||
- `owned`: a library manga with `meta['x-anilist'].id`.
|
||||
- Dedupe by id, keeping the highest weight. Sort by weight desc, then `anilistId` asc. Cap at 15.
|
||||
- `topTagsFor(anchors, recordOf: id => Record|null, { mature = false }) -> [{ name, kind }]`, at most 3. Sums anchor weight per tag with `rank >= 60`. Skips `isAdult` tags unless `mature`. Ties are broken by name asc.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**:
|
||||
- `anchors are empty without opt-in`
|
||||
- `weights, dedupe and cap`: a title voted `good` that is also owned yields one anchor with weight 3. Twenty owned titles yield 15 anchors.
|
||||
- `reading anchors need the reading opt-in`
|
||||
- `feedback keys parse anilist ids and ignore others`: `greatest:x` and `anilist:abc` are ignored.
|
||||
- `top tags weigh anchors, skip low rank and adult tags`: an adult tag is included with `{ mature: true }`.
|
||||
- [ ] **Step 2: Run** `node --test test/discovery-anchors.test.js`. Expected: FAIL.
|
||||
- [ ] **Step 3: Implement `discovery/anchors.js`.**
|
||||
- [ ] **Step 4: Run** the same command. Expected: PASS.
|
||||
- [ ] **Step 5: Commit** `Discovery anchors and top tags (BL-0051)` and push.
|
||||
|
||||
### Task 4: Harvester
|
||||
|
||||
**Files:**
|
||||
- Create: `discovery/harvest.js`
|
||||
- Test: `test/discovery-harvest.test.js`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `DiscoveryCache` (Task 1), the client and its errors (Task 2), `anchorsFor`/`topTagsFor` (Task 3). It also consumes the store methods `load()` and `patch({ harvest })` from Task 5; tests pass a stub `{ patch(){} }`, so Task 5 can land later.
|
||||
- Produces:
|
||||
- `BUDGETS = { nightly: 300, admin: 300, user: 60 }`
|
||||
- `REFRESH_COOLDOWN_MS = 3600000`
|
||||
- `MAX_CONSECUTIVE_FAILURES = 5`
|
||||
- `async runHarvest({ kind, targets: [{ user, anchors, mature }], cache, client, budget }) -> { requests, budget, status: 'success'|'stopped', stopReason: null|'budget'|'rate-limited'|'forbidden'|'challenge'|'upstream-errors' }`. Stages, in order, each skipping requests whose `cache.isFresh(kind, key)`:
|
||||
1. `title` for every anchor.
|
||||
2. `recs` per anchor.
|
||||
3. `creator` per author `staffId` of anchors with `weight >= 3`, read from the cached `title` record.
|
||||
4. `tagTop` per `topTagsFor(...)` tag, keyed `${kind}:${name}`.
|
||||
5. One `fresh`, key `global`.
|
||||
|
||||
Requests are deduped across targets. Each success is `cache.put`. An `AniListSkip` increments a consecutive-failure counter that resets on success; at 5 the run returns `stopped`/`upstream-errors`. An `AniListStop` returns `stopped` with its reason immediately. Reaching `budget` returns `success`/`budget`.
|
||||
- `shouldRunNightly({ now: Date, harvestAt: 'HH:MM', lastNightlyDate: 'YYYY-MM-DD'|null, enabled }) -> boolean`: true when enabled, local `now` is at or after `harvestAt`, and the local date differs from `lastNightlyDate`.
|
||||
- `class Harvester`:
|
||||
- `constructor({ cache, client, store, targetsFor: (kind, userName?) => targets, onDone: async (job, result) => void, now = Date.now })`
|
||||
- `request({ kind: 'nightly'|'admin'|'user', user?: string }) -> { harvestId, position }`. Throws `Object.assign(new Error(msg), { status, nextAllowedAt? })` with:
|
||||
- `409` for an admin or nightly harvest while one is running or queued, and for a user who already has one queued or running.
|
||||
- `429` within `REFRESH_COOLDOWN_MS` of that user's last accepted request; the error carries `nextAllowedAt` as an ISO string.
|
||||
- `state(userName) -> { state: 'idle'|'queued'|'running', position?: number, nextAllowedAt?: string }`
|
||||
- `current() -> { id, kind, user?, startedAt } | null`
|
||||
- Jobs run one at a time, FIFO. Each run calls `store.patch({ harvest: { id, kind, user, status: 'running', startedAt, requests: 0, budget } })`, then the final record (`status`, `finishedAt`, `requests`, `stopReason`), then `onDone(job, result)`. Errors thrown in `onDone` are logged and do not break the queue.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests** using a fake client (methods resolve from fixture maps, count calls, and can throw scripted errors) and a real `DiscoveryCache` in a temp dir:
|
||||
- `plan order and dedupe`: two targets sharing an anchor. The call log starts with `title` calls, then `recs`, `creator`, `tagTop`, `fresh`; the shared anchor is fetched once.
|
||||
- `fresh cache entries are skipped`
|
||||
- `budget stops with reason budget`: budget 3 → `requests === 3`, `stopReason: 'budget'`.
|
||||
- `429 stops at once`: the call count does not grow after the throw.
|
||||
- `network error stops with upstream-errors after 5`: five consecutive `AniListSkip`; a success in between resets the counter.
|
||||
- `plan for anchorless user only fetches fresh`
|
||||
- `shouldRunNightly`: 03:29 → false; 03:30 → true; same date → false; disabled → false.
|
||||
- `harvester single-flight, positions, 409 and 429 cooldown`: with fake `now`, a second user request inside one hour throws `429` with `nextAllowedAt`. An admin request while one is running throws `409`. A queued user's `state()` returns `queued` with `position` 1.
|
||||
- `onDone failure does not stall the queue`
|
||||
- [ ] **Step 2: Run** `node --test test/discovery-harvest.test.js`. Expected: FAIL.
|
||||
- [ ] **Step 3: Implement `discovery/harvest.js`.**
|
||||
- [ ] **Step 4: Run** the same command. Expected: PASS.
|
||||
- [ ] **Step 5: Commit** `Single-flight discovery harvester with budgets and stop reasons (BL-0051)` and push.
|
||||
|
||||
### Task 5: Store metadata
|
||||
|
||||
**Files:**
|
||||
- Modify: `discovery/store.js`
|
||||
- Test: `test/discovery.test.js` (add tests)
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
- `load()` defaults `harvest: null`, `settings: null`, `nightly: { lastDate: null }` when the fields are absent. The validation of existing fields is unchanged.
|
||||
- `publish(value)` keeps the prior `harvest`, `settings` and `nightly`.
|
||||
- `patch({ harvest?, settings?, nightly? }) -> data` writes atomically through the same temp+`.prev`+rename path, leaves `revision` unchanged, and refuses on a corrupt file like `publish`.
|
||||
- Extract a private `write(data)` shared by `publish` and `patch`.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**:
|
||||
- `store loads a Phase A file without new fields`: write a Phase A-shaped JSON; `load().nightly.lastDate === null`.
|
||||
- `patch keeps revision and publish keeps harvest`: `patch({ harvest: { status: 'success' } })`, then `publish(...)`; `load().harvest.status === 'success'` and the revision went up by exactly 1.
|
||||
- `patch refuses a corrupt store`
|
||||
- [ ] **Step 2: Run** `node --test test/discovery.test.js`. Expected: the new tests FAIL.
|
||||
- [ ] **Step 3: Implement** the store changes.
|
||||
- [ ] **Step 4: Run** `npm run test:reader`. Expected: PASS.
|
||||
- [ ] **Step 5: Commit** `Discovery store keeps harvest, settings and nightly state (BL-0051)` and push.
|
||||
|
||||
### Task 6: Engine — wider pool, new scores, slot mix
|
||||
|
||||
**Files:**
|
||||
- Modify: `discovery/engine.js`, `discovery/cli.js` (pass `harvestCache` when `data/discovery-cache` exists)
|
||||
- Test: `test/discovery.test.js` (update the cap assertion; add tests), `test/fixtures/anilist/` reused through a temp `DiscoveryCache`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `DiscoveryCache` (Task 1), `Record` (Task 2), `anchorsFor`/`topTagsFor` (Task 3).
|
||||
- Produces:
|
||||
- `MAX_CANDIDATES = 2000`, `MAX_FEED = 20`
|
||||
- `SLOTS = { perKind: 8, perPeriod: 4, fresh: 4, outside: 4 }`
|
||||
- `buildCandidates({ greatest, library, cacheDir, harvestCache? })`: adds one candidate per distinct `anilistId` found in `recs`, `creator`, `tagTop` and `fresh` entries. New candidate fields: `isAdult`, `averageScore`, `tagRecords: Record.tags`, `authorRecords: Record.authors`, `fetchedAt`. `tags` and `authors` stay name arrays. Greatest candidates keep their fields; when a harvested record has the same id, it fills `isAdult` and `tagRecords`. Era bucketing stays, and harvested candidates bucket by their seed kind instead of `Related works`.
|
||||
- `feedForUser({ user, db, candidates, greatest, library, queuedTitles, harvestCache? })` adds these parts, with values from spec §4, rounded to 2 decimals:
|
||||
- `communityRec = min(5, 1 + log2(1 + rating) / 2) * anchorWeight / 5`, using the best (anchor, rating) pair among the user's anchors whose `recs` list contains the candidate.
|
||||
- `sameCreator = 4 * anchorWeight / 5`, when the candidate's `authorRecords` shares a `staffId` with an anchor of `weight >= 3`.
|
||||
- `tagTop = 3 | 2 | 1` for the user's 1st, 2nd or 3rd top tag, only if `averageScore >= 75` and the candidate is in that tag's `tagTop` list.
|
||||
- `fresh = 2` if `status === 'ongoing'` and `year >= today.year - 2`; `1` if only present in the `fresh` list.
|
||||
- Mature filter: exclude unless `settings.discoveryMature` when `isAdult === true` or any `tagRecords` entry has `isAdult && rank >= 60`. Use the Phase A regex only when `isAdult` is `undefined`.
|
||||
- Each row gets `group`. If the row's largest part is one of the four new parts, `group` is its kind (`recs`, `creator`, `tagTop`, `fresh`); otherwise it is `period || 'Related works'`.
|
||||
- `reason` comes from the largest part. `reasons: string[]` lists every applicable reason in descending part order. Exact templates:
|
||||
- recs: `AniList readers who liked ${anchor} recommend this (${rating}).`
|
||||
- creator: `Also by ${author} — you ${read|liked} ${anchor}.` Use `read` when the anchor source is `read`, otherwise `liked`.
|
||||
- tagTop: `Top-rated in ${tag}, one of your most-liked tags.`
|
||||
- fresh: `New: started ${year}, ongoing and trending.`
|
||||
- Evidence ids: `rec:<anchorId>:<candidateId>`, `creator:<staffId>:<candidateId>`, `tagtop:<kind>:<name>:<candidateId>`, `fresh:<candidateId>`. Kinds: `recommendation|creator|tag-top|fresh`. Each carries `url: record.siteUrl` and `fetchedAt`.
|
||||
- `mixSlots(rows, { topTags }) -> rows` (at most 20, ranked):
|
||||
1. Sort by score desc, then `candidateId`.
|
||||
2. Take up to 4 rows with `scoreParts.fresh > 0`.
|
||||
3. Take up to 4 rows whose `tags` intersect none of the `topTags` names.
|
||||
4. Fill greedily.
|
||||
- Steps 2–4 all respect the caps: `perKind` 8 for the four kinds and `perPeriod` 4 for other groups.
|
||||
- Never pad.
|
||||
- The final order is by score desc, then `rank = index + 1`.
|
||||
- `async generateAsync({ db, library, greatest, cacheDir, harvestCache, queuedTitles, onlyUser? }) -> same shape as generate()`. Awaits `new Promise(setImmediate)` between users; with `onlyUser`, scores only that user. `generate()` stays synchronous for `cli.js` and the existing tests.
|
||||
- `mergeUserFeed(prior: { candidates, feeds }, result, userName) -> { candidates, feeds }`: replaces `feeds[userName]`, keeps every other feed, and keeps exactly the candidates referenced by any kept feed plus `result.candidates`.
|
||||
|
||||
- [ ] **Step 1: Update and write the failing tests**:
|
||||
- Change `assert.equal(Object.keys(result.candidates).length, 50)` to `Math.min(2000, examined)`; keep the other Phase A assertions.
|
||||
- `engine: empty harvest cache equals Phase A pool`: with and without an empty `harvestCache`, the candidate ids are equal.
|
||||
- `community rec, creator, tagTop and fresh parts and reasons`: fixture cache plus a user with a `flame` anchor. Asserts exact part values, for example: rating 42, weight 5 → `min(5, 1 + log2(43)/2) = 3.71`. Also asserts the exact reason strings and evidence kinds with `fetchedAt`.
|
||||
- `isAdult filter`: an `isAdult` candidate is absent without the mature opt-in and present with it.
|
||||
- `slot mix`: 30 `recs`-group rows → at most 8 from that kind. Fresh rows are present even when low-scored. Rows outside the top tags are present. Fewer than 20 eligible rows → no padding.
|
||||
- `multiple seeds merge reasons`: one candidate found by recs and creator has both reasons and both evidence entries.
|
||||
- `owned, queued, voted, dismissed still excluded` for harvested candidates.
|
||||
- `mergeUserFeed keeps other users' referenced candidates`
|
||||
- `scoring one user stays under 200 ms on 2000 candidates`: synthetic records; measure `feedForUser` with `performance.now()`.
|
||||
- [ ] **Step 2: Run** `node --test test/discovery.test.js`. Expected: the new tests FAIL.
|
||||
- [ ] **Step 3: Implement** the engine changes and the `cli.js` pass-through.
|
||||
- [ ] **Step 4: Run** `npm run test:reader`. Expected: PASS.
|
||||
- [ ] **Step 5: Commit** `Discovery scores AniList seeds with slot mix and isAdult filter (BL-0051)` and push.
|
||||
|
||||
### Task 7: Server wiring
|
||||
|
||||
**Files:**
|
||||
- Modify: `server.js` (around the discovery block at lines 240–267, the routes at 490–517, `HttpError` at 81, the error response at 974, and the timers near 980–1012), `docker-compose.yml` (reader `environment`: `DISCOVERY_HARVEST_AT: ${DISCOVERY_HARVEST_AT:-03:30}`, `DISCOVERY_ANILIST_URL: ${DISCOVERY_ANILIST_URL:-}`), `docs/architecture.md`
|
||||
- Test: `test/discovery-server.test.js` (spawns `server.js` on a free port with a temp `DATA`/`LIBRARY` and `DISCOVERY_ANILIST_URL` pointing to an in-test `http.createServer` serving fixtures). Copy the helper pattern from an existing server-spawning test in `test/` if one exists; otherwise write a minimal spawn/wait-for-port helper inside this file.
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Tasks 1–6.
|
||||
- Produces:
|
||||
- `HttpError(code, msg, extra = {})`; the error response becomes `{ error, ...extra }`.
|
||||
- `DISCOVERY_HARVEST_CACHE = path.join(DATA, 'discovery-cache')`
|
||||
- `harvestAt` default: `process.env.DISCOVERY_HARVEST_AT` if it matches `/^([01]\d|2[0-3]):[0-5]\d$/`, else `'03:30'`. `store.settings` overrides it.
|
||||
- `async rescore({ onlyUser })`: refactors the body of `startDiscoveryRun`. With `onlyUser` it calls `generateAsync({ onlyUser })` then `mergeUserFeed` before `publish`; otherwise it runs a full `generateAsync`. `startDiscoveryRun` keeps its route behavior.
|
||||
- `targetsFor(kind, userName)`: all opted-in users, or only `userName` for `kind === 'user'`. Each target is `{ user, anchors: anchorsFor(...), mature: !!settings.discoveryMature }`.
|
||||
- Startup: if `store.load().harvest?.status === 'running'`, `patch` it to `interrupted` with `finishedAt`. Queued jobs are not persisted.
|
||||
- Scheduler: `setInterval(checkNightly, 60000).unref()`, plus one call 30 s after startup. When `shouldRunNightly` is true and no harvest is active, call `patch({ nightly: { lastDate } })` and then `harvester.request({ kind: 'nightly' })`.
|
||||
- Routes:
|
||||
- `POST /api/discovery/refresh`: caller only; `403` without opt-in. Returns `{ harvestId, position }`, `409`, or `429` with `{ nextAllowedAt }`.
|
||||
- `GET /api/discovery`: adds `refresh: harvester.state(u.name)`.
|
||||
- `POST /api/admin/discovery/harvest`: admin only; returns `{ harvestId, position }` or `409`.
|
||||
- `GET /api/admin/discovery`: adds `harvest`, `currentHarvest`, `cache: harvestCache.stats()`, `settings: { harvestEnabled, harvestAt }`, `nextNightlyAt`.
|
||||
- `POST /api/admin/discovery/settings`: admin only. Accepts exactly `{ harvestEnabled: boolean, harvestAt: 'HH:MM' }`; anything else → `400`. Saved via `patch({ settings })`.
|
||||
|
||||
- [ ] **Step 1: Write the failing integration tests**:
|
||||
- `refresh requires opt-in, queues, runs and rescores only the caller`: two opted-in users. Alice refreshes, and her feed afterwards contains a fixture rec candidate. Bob's `generatedAt` is unchanged.
|
||||
- `second refresh within an hour is 429 with nextAllowedAt`
|
||||
- `admin harvest is admin-only and single-flight`: `403` for a non-admin; `409` on a second call while one is running (the fixture server delays its responses by 500 ms).
|
||||
- `settings validation`: `harvestAt: '25:00'` → `400`; an extra key → `400`; a valid body round-trips through `GET /api/admin/discovery`.
|
||||
- `interrupted harvest is marked on startup`: pre-write a `discovery.json` with `harvest.status: 'running'`, start the server, and get `interrupted`.
|
||||
- `harvest never touches scraper queue or library`: no files change under the temp `LIBRARY` or `DATA/scraper`. Snapshot the directory listings and mtimes before and after.
|
||||
- [ ] **Step 2: Run** `node --test test/discovery-server.test.js`. Expected: FAIL.
|
||||
- [ ] **Step 3: Implement** the server changes, the compose env lines, and the new routes in `docs/architecture.md`.
|
||||
- [ ] **Step 4: Run** `npm test`. Expected: all suites PASS.
|
||||
- [ ] **Step 5: Commit** `Discovery harvest routes, nightly schedule and per-user refresh (BL-0051)` and push.
|
||||
|
||||
### Task 8: Discover UI, browser test and docs
|
||||
|
||||
**Files:**
|
||||
- Modify: `public/discover.js`, `public/discover.css` if present (else the stylesheet `discover.html` links), `tools/test-discovery-ui.js`, `tools/test-compose.js` (pass `DISCOVERY_ANILIST_URL=http://127.0.0.1:9/` for `--discovery-ui`, so a harvest fails fast without reaching AniList), `docs/discovery.md`, `docs/deployment.md`, `vision/discovery-scout.md`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: Task 7 routes and response fields.
|
||||
- UI, using the existing `h()`/`button()` helpers. Never use `innerHTML`; text goes in through `h()` children.
|
||||
- **Refresh my suggestions** button, shown when `state.enabled`. While `refresh.state` is `queued` it reads `Queued (position N)`, while `running` it reads `Fetching…`, and both disable it. During a cooldown it shows `Next refresh possible at <local HH:MM>` (from `nextAllowedAt`) and is disabled. It polls every 1.5 s while queued or running, like `run()`.
|
||||
- Card: show `item.reason`; the details list shows all `item.reasons` and evidence, plus ` · fetched <local date>` when `entry.fetchedAt` is set.
|
||||
- Settings hint for **Personalized discovery**, exact text: `Off by default. When on, the server sends AniList the IDs of titles you liked, without login, to find related work. Your feed and feedback stay private to this server.`
|
||||
- The mature hint becomes `Only affects titles AniList marks as adult or with a known mature tag.`
|
||||
- Admin panel:
|
||||
- adds a line `Last harvest: <status> · <requests>/<budget> requests · <stopReason or —> · cache <files> files`, plus the next nightly time
|
||||
- adds a **Harvest now** button
|
||||
- adds an **Automatic harvest** checkbox and a time `<input type="time">` that save through the settings route
|
||||
- replaces the old "This first version…" paragraph with: `Harvests ask AniList for public metadata only. They never ask image hosters, call an LLM, or edit the scraper queue.`
|
||||
- Docs:
|
||||
- `docs/discovery.md`: replace the deliberate-limits section with the seed kinds, budgets, cache, nightly time, privacy note and stop reasons.
|
||||
- `docs/deployment.md`: `data/discovery-cache/` is safe to delete and can be left out of backups; document `DISCOVERY_HARVEST_AT` and `DISCOVERY_ANILIST_URL` (test only).
|
||||
- `vision/discovery-scout.md` §4: mark the candidate input as implemented, with a link to the spec.
|
||||
|
||||
- [ ] **Step 1: Extend `tools/test-discovery-ui.js`**, after the existing Maybe click:
|
||||
- Click **Refresh my suggestions**; wait until the button text matches `/Next refresh possible at \d{1,2}:\d{2}/`.
|
||||
- The admin panel shows `Last harvest: stopped` and `upstream-errors`.
|
||||
- Turn on **Automatic harvest**, set the time to `04:15`, reload, and see `04:15`.
|
||||
- **Harvest now** while one is running shows the 409 message in the status note.
|
||||
- Review Focus 5: before the stack starts, seed the disposable data tree with a `discovery.json` whose candidate title is `<img src=x onerror=alert(1)>` and which is in the QA admin's feed. Assert that the card's `h2` `textContent` equals that string and that `h2 img` count is 0.
|
||||
- The existing non-admin checks additionally assert `Harvest now` count 0 and `POST /api/admin/discovery/harvest` → 403.
|
||||
- Mobile viewport (375×812): the refresh button is visible without horizontal scroll (`document.documentElement.scrollWidth <= 375`).
|
||||
- [ ] **Step 2: Run** `docker build -t mangacircus:discovery-test .` then `APP_IMAGE=mangacircus:discovery-test npm run test:discovery-ui`. Expected: FAIL on the missing button.
|
||||
- [ ] **Step 3: Implement** the UI and docs.
|
||||
- [ ] **Step 4: Run** the same two commands, then `npm test`. Expected: PASS.
|
||||
- [ ] **Step 5: Commit** `Discover page: refresh, harvest admin and seed reasons; docs (BL-0051)` and push.
|
||||
- [ ] **Step 6: Update the project's coordination backlog item and release the claim.** Merging into `main` waits for the user.
|
||||
@@ -0,0 +1,306 @@
|
||||
# Discovery: wider candidate pool — design
|
||||
|
||||
Status: implemented on branch feature/discovery-wider-pool. Date: 2026-10-25. Builds on the
|
||||
deterministic Discovery Scout ([`docs/discovery.md`](../../discovery.md),
|
||||
[`vision/discovery-scout.md`](../../../vision/discovery-scout.md)).
|
||||
|
||||
## 1. Goal
|
||||
|
||||
Today Discover can only suggest titles from the curated Greatest catalog plus
|
||||
AniList *relations* (sequels, spin-offs) of Greatest and library titles, and only
|
||||
when those are already in the local metadata cache. A run makes no network call.
|
||||
Users run out of new suggestions quickly.
|
||||
|
||||
This change widens the candidate pool with four seed kinds, fetched server-side
|
||||
from AniList:
|
||||
|
||||
1. **Community recommendations** — "readers who liked X recommend Y" for the
|
||||
user's anchor titles.
|
||||
2. **Same creator** — other works by the story/art authors of anchor titles.
|
||||
3. **Top in your tags** — well-rated titles in the user's strongest genres/tags.
|
||||
4. **Fresh** — recently started or currently releasing, trending titles.
|
||||
|
||||
Success: suggestions the user did not already know, each with a traceable
|
||||
reason and evidence link, without exceeding AniList's rate limits or making the
|
||||
reader noticeably slower.
|
||||
|
||||
### Unchanged rules from Phase A
|
||||
|
||||
- Personalized discovery and reading activity stay separate opt-ins, both off
|
||||
by default; feedback stays private to the account.
|
||||
- Discovery never edits the scraper queue, library or collections and never
|
||||
downloads anything.
|
||||
- Every suggestion carries evidence; missing data stays missing.
|
||||
- The reader is the only writer of discovery state; no new Compose service, no
|
||||
database, no LLM.
|
||||
- All fetching runs in the reader container, never in the browser.
|
||||
- The repository is public: no host names, paths or personal seeds in code,
|
||||
docs or fixtures.
|
||||
|
||||
### Non-goals
|
||||
|
||||
- No other metadata provider (AniList's API covers all four seed kinds).
|
||||
- No LLM rationale or rerank (vision Phase C).
|
||||
- No edition finding or queueing from a suggestion (vision Phase B).
|
||||
- No worker container; it is only a fallback if section 7's latency check fails.
|
||||
|
||||
## 2. Components
|
||||
|
||||
| Unit | Responsibility | Depends on |
|
||||
|---|---|---|
|
||||
| `discovery/anilist.js` (new) | GraphQL queries for the four seed kinds and for a title by ID; one shared rate limiter; schema check and normalization of responses | `fetch`, `discovery/cache.js` |
|
||||
| `discovery/cache.js` (new) | `data/discovery-cache/`: atomic writes, TTL per entry kind, size cap with oldest-first pruning | filesystem |
|
||||
| `discovery/harvest.js` (new) | Anchor selection per user, request plan, budgets, single-flight run state, stop reasons | `anilist.js`, `cache.js`, `db` (read-only) |
|
||||
| `discovery/engine.js` (changed) | Builds candidates from Greatest, relations **and the harvest cache**; new score parts, evidence kinds, slot mix, `isAdult` filter | `cache.js` (read-only) |
|
||||
| `server.js` (changed) | Nightly scheduler, admin "Harvest now", per-user "Refresh my suggestions", admin status fields; route handlers stay thin | `harvest.js`, `engine.js`, `discovery/store.js` |
|
||||
| `public/discover.*` (changed) | Refresh button states, new reason labels, fetch date in evidence, admin harvest panel | reader API |
|
||||
|
||||
The existing `discovery/store.js` (atomic `discovery.json`, `.prev` recovery,
|
||||
fail-closed on corruption) is reused unchanged. The reader keeps reading the
|
||||
scraper's AniList cache read-only for relations; the harvester never writes into
|
||||
`data/scraper/`.
|
||||
|
||||
### 2.1 `discovery/anilist.js`
|
||||
|
||||
Queries (all `type: MANGA`, unauthenticated, `https://graphql.anilist.co`):
|
||||
|
||||
| Kind | Query | Page size |
|
||||
|---|---|---|
|
||||
| `title` | `Media(id)` with the fields of `scraper/metadata/anilist.js` plus `isAdult averageScore popularity trending tags { name rank isAdult } recommendations staff` | 1 |
|
||||
| `recs` | `Media(id).recommendations(sort: RATING_DESC)` → `mediaRecommendation` + `rating` | 25 |
|
||||
| `creator` | `Staff(id).staffMedia(type: MANGA, sort: POPULARITY_DESC)` | 25 |
|
||||
| `tagTop` | `Page.media(tag_in or genre_in: [one tag], sort: SCORE_DESC, popularity_greater: 1000)` | 25 |
|
||||
| `fresh` | `Page.media(status: RELEASING, startDate_greater: <today − 2 years>, sort: TRENDING_DESC)` | 50 |
|
||||
|
||||
- **Rate limiter:** at most 25 requests per minute, one at a time, shared by all
|
||||
harvests in the process. AniList allows more, so this stays below its limit
|
||||
with headroom.
|
||||
- **Normalized record:** `{ anilistId, title, altTitles, format, isAdult,
|
||||
status, year, averageScore, popularity, trending, tags[{name, rank}],
|
||||
authors[{staffId, name, role}], coverUrl, siteUrl, fetchedAt }`.
|
||||
Each list response becomes a list of `{ anilistId, rating? }` plus one record
|
||||
per entry it contains.
|
||||
- **Formats:** only `MANGA` and `ONE_SHOT` are kept; novels and anime are dropped.
|
||||
- **Response check:** responses are checked against a schema before they are
|
||||
cached; an invalid one counts as a failed request. Titles, descriptions and
|
||||
tags are treated as untrusted text: stored as data, escaped when shown.
|
||||
|
||||
### 2.2 `discovery/cache.js`
|
||||
|
||||
- **Location:** `data/discovery-cache/<kind>/<key>.json`. Every entry has
|
||||
`{ schema: 1, kind, key, fetchedAt, data }`.
|
||||
- **Lifetime:** `title` and `recs` entries live 30 days, `creator` 30 days,
|
||||
`tagTop` and `fresh` 7 days. An expired entry is still used for scoring until
|
||||
it is refreshed. Only the harvest checks the TTL.
|
||||
- **Size cap:** 5,000 files. On write, the oldest `fetchedAt` entries are pruned
|
||||
first.
|
||||
- **Writes:** temporary file then rename, as in the existing stores. A corrupt
|
||||
entry is ignored and deleted.
|
||||
|
||||
### 2.3 `discovery/harvest.js`
|
||||
|
||||
**Anchors per opted-in user.** Each anchor has a weight; the list is
|
||||
deduplicated by AniList ID and capped at the 15 strongest:
|
||||
|
||||
| Source | Weight |
|
||||
|---|---|
|
||||
| Greatest vote `flame` | 5 |
|
||||
| Greatest vote `good` | 3 |
|
||||
| Discover feedback `interested` | 4 |
|
||||
| Discover feedback `maybe` | 2 |
|
||||
| Completed chapter in a library title (only with reading activity opted in) | 3 |
|
||||
| Owned library title with an AniList ID | 1 |
|
||||
|
||||
**User's top tags.** The 3 tags with the highest summed anchor weight, using
|
||||
the anchors' cached `title` records (tag rank ≥ 60, no adult tags unless the
|
||||
user enabled mature titles).
|
||||
|
||||
**Request plan, in this order:**
|
||||
|
||||
1. `title` for anchors that are not cached.
|
||||
2. `recs` for each anchor.
|
||||
3. `creator` for the story/art staff of anchors weighted 3 or more.
|
||||
4. `tagTop` for each top tag.
|
||||
5. One global `fresh` list.
|
||||
|
||||
Requests whose cache entry is still within its TTL are skipped. Then follow-up
|
||||
`title` requests run for new candidates that are missing tags or `isAdult`, up
|
||||
to the remaining budget.
|
||||
|
||||
**Budgets:** nightly harvest 300 requests, per-user refresh 60 requests. When
|
||||
the budget runs out, the plan stops; whatever was cached so far is still used.
|
||||
|
||||
**Run state:** one harvest at a time (single-flight).
|
||||
|
||||
- A queued per-user refresh waits behind a running harvest; a second refresh
|
||||
for the same user while one is queued returns `409`.
|
||||
- The run record is persisted in `discovery.json` `harvest`:
|
||||
`{ id, kind: nightly|admin|user, user?, status, startedAt, finishedAt,
|
||||
requests, budget, stopReason }`.
|
||||
|
||||
## 3. Data flow
|
||||
|
||||
```
|
||||
anchors ─▶ request plan ─▶ AniList (rate-limited) ─▶ discovery-cache
|
||||
│
|
||||
Greatest + scraper AniList cache (relations) ─────────────┤
|
||||
▼
|
||||
engine.generate() ─▶ discovery.json (atomic publish)
|
||||
```
|
||||
|
||||
Harvest and scoring are separate steps. Scoring is local and never touches the
|
||||
network, so a failed harvest means older candidates, never an empty or broken
|
||||
feed.
|
||||
|
||||
- **After a nightly or admin harvest:** re-score every opted-in user.
|
||||
- **After a per-user refresh:** re-score only that user. The other users' feed
|
||||
snapshots are carried over unchanged into the same publish.
|
||||
|
||||
## 4. Scoring
|
||||
|
||||
`engine.generate()` builds the candidate map from Greatest, relations, and every
|
||||
record referenced by the harvest cache, limited to 2,000 candidates; the
|
||||
replaced Phase A cap was 50. Exclusions stay as in Phase A:
|
||||
|
||||
- titles the user already voted on in Greatest
|
||||
- titles dismissed or hidden in Discover
|
||||
- owned titles, matched by ID or by normalized title + year
|
||||
- queued titles
|
||||
|
||||
The mature filter now uses `isAdult` (title-level, or an adult tag with
|
||||
rank ≥ 60). The Phase A tag regex is kept only for candidates with no `isAdult`
|
||||
data.
|
||||
|
||||
New score parts, all bounded:
|
||||
|
||||
| Part | Value | Rule |
|
||||
|---|---|---|
|
||||
| `communityRec` | 0–5 | `min(5, 1 + log2(1 + rating) / 2) × anchorWeight / 5`; strongest anchor counts |
|
||||
| `sameCreator` | 0–4 | `4 × anchorWeight / 5` for a shared story/art author with an anchor weighted ≥ 3 |
|
||||
| `tagTop` | 0–3 | `3` for the user's top tag, `2` second, `1` third, only if `averageScore ≥ 75` |
|
||||
| `fresh` | 0–2 | `2` if releasing and started within 2 years, `1` if only trending |
|
||||
|
||||
Existing parts (`editorial`, `relation`, `sharedTaste`, `era`, `reading`,
|
||||
`interest`) and their weights are unchanged.
|
||||
|
||||
**Slot mix** for the 20 feed slots, applied greedily by descending score:
|
||||
|
||||
- At most 8 items whose strongest part comes from the same seed kind.
|
||||
- Up to 4 slots reserved for `fresh` candidates when such candidates exist.
|
||||
- About 20% (4 slots) taken from candidates outside the user's strongest tag
|
||||
cluster, as in the vision doc.
|
||||
- If a rule cannot be met, the slot falls back to the next best candidate; the
|
||||
feed is never padded.
|
||||
|
||||
**Reasons.** A candidate found by several seeds collects every reason and
|
||||
evidence item. The tile shows the reason of the strongest part; the detail
|
||||
panel lists all of them. Reason templates:
|
||||
|
||||
| Kind | Template |
|
||||
|---|---|
|
||||
| `recs` | "AniList readers who liked {anchor} recommend this ({rating})." |
|
||||
| `creator` | "Also by {author} — you {read/liked} {anchor}." |
|
||||
| `tagTop` | "Top-rated in {tag}, one of your most-liked tags." |
|
||||
| `fresh` | "New: started {year}, ongoing and trending." |
|
||||
|
||||
Evidence entries use `kind: recommendation|creator|tag-top|fresh`, link to the
|
||||
AniList record, and carry `fetchedAt`.
|
||||
|
||||
## 5. API and UI
|
||||
|
||||
Routes:
|
||||
|
||||
- `POST /api/discovery/refresh` — caller only, requires opt-in. Returns
|
||||
`{ harvestId, position }`. Returns `429` with `nextAllowedAt` within 1 hour of
|
||||
the caller's last refresh, and `409` if one is already queued for the caller.
|
||||
- `GET /api/discovery` — adds
|
||||
`refresh: { state: idle|queued|running, position?, nextAllowedAt? }` and
|
||||
`fetchedAt` on evidence.
|
||||
- `POST /api/admin/discovery/harvest` — admin only; starts a full harvest.
|
||||
Returns `409` while one runs.
|
||||
- `GET /api/admin/discovery` — adds the last harvest record, cache file count
|
||||
and size, and the next nightly time.
|
||||
- `POST /api/admin/discovery/settings` (new) — `{ harvestEnabled: boolean,
|
||||
harvestAt: "HH:MM" }`. Defaults: `true` and `03:30`, overridable by
|
||||
`DISCOVERY_HARVEST_AT`. The value is validated; nothing else is accepted.
|
||||
|
||||
UI on `discover.html`:
|
||||
|
||||
- **Refresh my suggestions** button with three states: idle, queued/fetching
|
||||
(with queue position), and cooldown ("next refresh possible at 14:20").
|
||||
- Reason labels per section 4; evidence shows "fetched {date}".
|
||||
- The settings text for **Personalized discovery** says that the server sends
|
||||
AniList the IDs of titles you liked, without login, to find related work.
|
||||
- Admin panel: last harvest (time, requests / budget, stop reason), cache size,
|
||||
next nightly run, **Harvest now**, enable switch and time.
|
||||
|
||||
## 6. Failure handling
|
||||
|
||||
| Condition | Behaviour |
|
||||
|---|---|
|
||||
| HTTP 429, 403, or a challenge page | Stop the harvest at once with `stopReason`, shown in the admin panel. No retry until the next nightly slot or admin action; never change identity, user agent or host. |
|
||||
| Timeout or 5xx | Skip that request and count it. Five in a row abort the run (`stopReason: upstream-errors`). A timeout never counts as an empty result. |
|
||||
| Invalid response | Dropped, counted as a failed request, not cached. |
|
||||
| Restart during a harvest | On startup the run is marked `interrupted`. If the nightly slot was missed, one harvest runs after startup; there is never a catch-up storm. A queued per-user refresh is dropped and the user can press the button again. |
|
||||
| Corrupt `discovery.json` | Unchanged from Phase A: reads and runs fail closed; the file is never overwritten. |
|
||||
| No network | Harvest fails with a stop reason; scoring uses whatever is cached. A fresh install behaves exactly like Phase A. |
|
||||
| Account deletion | Unchanged (feedback removed). Cache entries are not user-specific and stay. |
|
||||
|
||||
## 7. Performance
|
||||
|
||||
- Harvesting is I/O-bound and single-flight. At 25 requests per minute, a full
|
||||
nightly harvest takes about 12 minutes.
|
||||
- Scoring 2,000 candidates for each user must keep the reader event loop
|
||||
responsive. Each user is scored in its own `setImmediate` turn, and a test
|
||||
asserts that one user's scoring stays below 200 ms on the fixture set.
|
||||
- If production shows reader stalls, moving the job to a worker container is
|
||||
recorded as a backlog item; it is not part of this change.
|
||||
|
||||
## 8. Testing
|
||||
|
||||
All network tests use recorded, anonymized AniList fixtures; no test calls the
|
||||
live API.
|
||||
|
||||
**Unit:**
|
||||
|
||||
- query builders and response normalization
|
||||
- schema rejection of malformed responses
|
||||
- dropping non-manga formats
|
||||
- the rate limiter, with a fake clock
|
||||
- stop on 429/403/challenge, and the consecutive-error abort
|
||||
- cache TTL, size pruning, and corrupt-entry handling
|
||||
- anchor weights, cap and dedup; reading activity used only with opt-in
|
||||
- top-tag selection
|
||||
- each new score part
|
||||
- slot-mix limits and fallback
|
||||
- merging reasons from several seeds
|
||||
- `isAdult` filter
|
||||
- exclusion of owned, queued, voted and dismissed titles
|
||||
|
||||
**Integration:**
|
||||
|
||||
- per-user refresh: queue position, `429` cooldown, `409` duplicate
|
||||
- admin harvest single-flight
|
||||
- re-scoring after a per-user refresh leaves other users' feeds unchanged
|
||||
- interrupted-run recovery
|
||||
- one missed-night run after startup
|
||||
- no harvest or score ever changes the scraper queue or library
|
||||
- settings validation
|
||||
|
||||
**Browser** (existing disposable Compose suite):
|
||||
|
||||
- refresh button states
|
||||
- reason labels for each seed kind
|
||||
- evidence fetch date
|
||||
- admin harvest panel
|
||||
- desktop and mobile layout
|
||||
|
||||
## 9. Documentation
|
||||
|
||||
- `docs/discovery.md`: seed kinds, budgets, cache, nightly time, privacy note,
|
||||
stop reasons.
|
||||
- `docs/deployment.md`: `data/discovery-cache/` (safe to delete, rebuilt by the
|
||||
next harvest; may be left out of backups) and `DISCOVERY_HARVEST_AT`.
|
||||
- `docs/architecture.md`: the new routes.
|
||||
- `vision/discovery-scout.md`: mark the candidate-input part of section 4 as
|
||||
implemented.
|
||||
@@ -0,0 +1,73 @@
|
||||
'use strict';
|
||||
|
||||
// Local copies of the Greatest catalog's cover images. Hotlinking ~100 covers per page
|
||||
// load makes the image CDN drop some of them at random; one polite download each fixes that.
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const crypto = require('node:crypto');
|
||||
|
||||
const TYPES = { 'image/jpeg': '.jpg', 'image/png': '.png', 'image/webp': '.webp', 'image/gif': '.gif', 'image/avif': '.avif' };
|
||||
const EXTENSIONS = new Set(Object.values(TYPES));
|
||||
const USER_AGENT = 'MangaCircus-Greatest/1.0';
|
||||
|
||||
class GreatestCovers {
|
||||
constructor(dir, { fetchImpl = fetch, sleep = ms => new Promise(resolve => setTimeout(resolve, ms)),
|
||||
delayMs = 1000, maxBytes = 2 * 1024 * 1024, timeoutMs = 20000 } = {}) {
|
||||
this.dir = path.resolve(dir);
|
||||
Object.assign(this, { fetchImpl, sleep, delayMs, maxBytes, timeoutMs });
|
||||
this.files = new Map(); // key -> file name
|
||||
try {
|
||||
for (const name of fs.readdirSync(this.dir)) {
|
||||
const ext = path.extname(name);
|
||||
if (EXTENSIONS.has(ext)) this.files.set(path.basename(name, ext), name);
|
||||
}
|
||||
} catch (error) { if (error.code !== 'ENOENT') throw error; }
|
||||
}
|
||||
// A hash of the URL names the file, so a changed catalog URL is fetched again.
|
||||
version(url) { return crypto.createHash('sha256').update(String(url)).digest('hex').slice(0, 16); }
|
||||
file(url) {
|
||||
const name = url ? this.files.get(this.version(url)) : null;
|
||||
return name ? path.join(this.dir, name) : null;
|
||||
}
|
||||
// Sequential, paced, and stops on 403/429 like every other outbound client here.
|
||||
async warm(urls) {
|
||||
const result = { fetched: 0, skipped: 0, failed: 0, stopped: null };
|
||||
let requested = false;
|
||||
for (const url of new Set(urls.filter(url => /^https:\/\/\S+$/i.test(String(url || ''))))) {
|
||||
if (this.file(url)) { result.skipped++; continue; }
|
||||
if (requested) await this.sleep(this.delayMs);
|
||||
requested = true;
|
||||
try {
|
||||
await this.download(url);
|
||||
result.fetched++;
|
||||
} catch (error) {
|
||||
if (error.stop) { result.stopped = error.message; break; }
|
||||
result.failed++;
|
||||
}
|
||||
}
|
||||
// Duplicates and repeats of stored covers count as skipped.
|
||||
result.skipped += urls.filter(Boolean).length - new Set(urls.filter(Boolean)).size;
|
||||
return result;
|
||||
}
|
||||
async download(url) {
|
||||
const response = await this.fetchImpl(url, { headers: { 'user-agent': USER_AGENT },
|
||||
signal: AbortSignal.timeout(this.timeoutMs) });
|
||||
if ([401, 403, 429].includes(response.status)) throw Object.assign(new Error(`HTTP ${response.status}`), { stop: true });
|
||||
if (!response.ok) throw new Error(`HTTP ${response.status}`);
|
||||
const ext = TYPES[String(response.headers.get('content-type') || '').split(';')[0].trim().toLowerCase()];
|
||||
if (!ext) throw new Error('Not an image');
|
||||
if (Number(response.headers.get('content-length')) > this.maxBytes) throw new Error('Image too large');
|
||||
const body = Buffer.from(await response.arrayBuffer());
|
||||
if (!body.length || body.length > this.maxBytes) throw new Error('Image too large or empty');
|
||||
const key = this.version(url), name = key + ext;
|
||||
fs.mkdirSync(this.dir, { recursive: true });
|
||||
const temp = path.join(this.dir, `.${key}.${crypto.randomBytes(4).toString('hex')}.tmp`);
|
||||
try {
|
||||
fs.writeFileSync(temp, body);
|
||||
fs.renameSync(temp, path.join(this.dir, name));
|
||||
} finally { fs.rmSync(temp, { force: true }); }
|
||||
this.files.set(key, name);
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { GreatestCovers };
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
'use strict';
|
||||
/*
|
||||
* Moments: a page, or a region across up to four neighbouring pages, that a reader saved with exactly one category.
|
||||
* `layout` says how those pages sit together: side by side ("row", Page and Book mode) or stacked ("column", Flow).
|
||||
* Stored per user in db.moments[user] as a list. A moment points at its pages by version folder and file
|
||||
* name (plus the file's SHA-256 for a later relocation), so a re-scan that shifts page numbers never makes it
|
||||
* point at the wrong picture: a file that is gone shows as "page moved" instead.
|
||||
*/
|
||||
const CATEGORIES = ['twist', 'hot', 'awesome', 'cute', 'funny', 'feels'];
|
||||
const NOTE_MAX = 200, MAX_PER_USER = 5000, MAX_PAGES = 4, LAYOUTS = ['row', 'column'];
|
||||
|
||||
class MomentError extends Error { constructor(msg) { super(msg); this.code = 400; } }
|
||||
const fail = msg => { throw new MomentError(msg); };
|
||||
|
||||
const clamp01 = x => Math.min(1, Math.max(0, x));
|
||||
// A region is a rectangle in fractions (0–1) of the displayed page, or of all its pages put together in `layout`.
|
||||
// Nothing or a rectangle covering (almost) everything means the whole page.
|
||||
function cleanRegion(r) {
|
||||
if (r == null) return null;
|
||||
const n = ['x', 'y', 'w', 'h'].map(k => Number(r[k]));
|
||||
if (n.some(v => !Number.isFinite(v))) fail('Invalid region.');
|
||||
const x = clamp01(n[0]), y = clamp01(n[1]), w = clamp01(Math.min(n[2], 1 - x)), h = clamp01(Math.min(n[3], 1 - y));
|
||||
if (w < 0.01 || h < 0.01) fail('The selected region is too small.');
|
||||
if (w > 0.995 && h > 0.995) return null;
|
||||
const round = v => Math.round(v * 1e4) / 1e4;
|
||||
return { x: round(x), y: round(y), w: round(w), h: round(h) };
|
||||
}
|
||||
function cleanCategory(cat) {
|
||||
return CATEGORIES.includes(cat) ? cat : fail('Pick one category: ' + CATEGORIES.join(', ') + '.');
|
||||
}
|
||||
const cleanNote = note => String(note == null ? '' : note).replace(/\s+/g, ' ').trim().slice(0, NOTE_MAX);
|
||||
|
||||
// Page indexes (into the version's full page list) in displayed order: left to right, or top to bottom.
|
||||
function cleanPages(pages, version) {
|
||||
if (!Array.isArray(pages) || pages.length < 1 || pages.length > MAX_PAGES) fail(`A moment covers 1 to ${MAX_PAGES} pages.`);
|
||||
const out = pages.map(Number);
|
||||
if (out.some(i => !Number.isInteger(i) || i < 0 || i >= version.pages.length)) fail('Page not found.');
|
||||
if (new Set(out).size !== out.length) fail('A page appears twice.');
|
||||
return out;
|
||||
}
|
||||
const cleanLayout = layout => (layout == null ? 'row' : LAYOUTS.includes(layout) ? layout : fail('Invalid page layout.'));
|
||||
|
||||
// Where a stored moment's pages are now. Returns null when the chapter, the version or a page file is gone.
|
||||
function locate(moment, manga) {
|
||||
const chapter = manga && manga.chapterById.get(moment.c), version = chapter && chapter.versions.find(v => v.id === moment.v);
|
||||
if (!version) return null;
|
||||
const pages = moment.f.map((file, k) => {
|
||||
const at = moment.i[k];
|
||||
return version.pages[at] && version.pages[at].f === file ? at : version.pages.findIndex(p => p.f === file);
|
||||
});
|
||||
return pages.every(i => i >= 0) ? { chapter, version, pages } : null;
|
||||
}
|
||||
|
||||
// The browser's view of a moment. `url` turns a library-relative path into an image URL.
|
||||
function view(moment, manga, url) {
|
||||
const at = locate(moment, manga);
|
||||
const chapter = at ? at.chapter : manga && manga.chapterById.get(moment.c);
|
||||
return {
|
||||
id: moment.id, c: moment.c, v: moment.v, cat: moment.cat, note: moment.note || '', t: moment.t, region: moment.region || null,
|
||||
layout: moment.layout || 'row',
|
||||
chapter: chapter ? { label: chapter.label, title: chapter.title, book: chapter.book } : { label: moment.c, title: '', book: '' },
|
||||
// Story order: chapter position in the manga, then page, then how far down the page the region starts.
|
||||
order: [chapter ? manga.chapters.indexOf(chapter) : 1e9, at ? Math.min(...at.pages) : moment.i[0], moment.region ? moment.region.y : 0],
|
||||
missing: !at,
|
||||
pages: at ? at.pages.map(i => {
|
||||
const p = at.version.pages[i];
|
||||
return { i, u: url(at.version.rel + '/' + p.f), w: p.w, h: p.h, r: p.r || 0 };
|
||||
}) : [],
|
||||
};
|
||||
}
|
||||
|
||||
function storyCompare(a, b) {
|
||||
for (let k = 0; k < 3; k++) if (a.order[k] !== b.order[k]) return a.order[k] - b.order[k];
|
||||
return a.t - b.t;
|
||||
}
|
||||
|
||||
module.exports = { CATEGORIES, NOTE_MAX, MAX_PER_USER, MAX_PAGES, MomentError, cleanRegion, cleanCategory, cleanNote, cleanPages, cleanLayout, locate, view, storyCompare };
|
||||
@@ -2,6 +2,7 @@
|
||||
"name": "mangacircus",
|
||||
"version": "1.0.0",
|
||||
"private": true,
|
||||
"license": "MIT",
|
||||
"description": "No-frills self-hosted manga reader with no runtime dependencies.",
|
||||
"main": "server.js",
|
||||
"scripts": {
|
||||
|
||||
+37
-2
@@ -12,6 +12,8 @@ const { publishManifest } = require('../scraper/lib/publisher');
|
||||
const { writeJsonAtomic } = require('../scraper/lib/json-file');
|
||||
const { sleep } = require('../scraper/lib/delay');
|
||||
const { normalizeWorkIdentity, sameWorkIdentity, resolveWorkIdentity } = require('../scraper/lib/work-identity');
|
||||
const { fromPeer, publishedDates } = require('../chapter-dates');
|
||||
const { parseKey, parseSpecialFolder } = require('../chapter-identity');
|
||||
|
||||
const dataRoot = () => path.resolve(process.env.PEER_DATA || path.join(process.env.DATA || 'data', 'peering'));
|
||||
const store = () => new PeeringState(dataRoot());
|
||||
@@ -27,7 +29,37 @@ const infoFor = value => {
|
||||
};
|
||||
function refsOf(result) {
|
||||
return result.chapters.map(ref => ({ id: ref.id, number: ref.number, volume: ref.volume,
|
||||
book: { number: Number(ref.volume) || 1 }, title: ref.title, pages: ref.pageCount }));
|
||||
book: { number: Number(ref.volume) || 1 }, title: ref.title, pages: ref.pageCount,
|
||||
...(ref.dates && typeof ref.dates === 'object' ? { dates: ref.dates } : {}) }));
|
||||
}
|
||||
// Chapters this library already has adopt a peer's earlier ecosystem date (BL-0049): the
|
||||
// chapter entered Manga Circus there first. Later claims never raise a date.
|
||||
async function lowerEcosystemDates(mangaPath, refs, alias) {
|
||||
const claims = new Map();
|
||||
for (const ref of refs) {
|
||||
const claim = fromPeer(ref.dates, alias).ecosystem;
|
||||
const key = `${ref.volume}/${ref.number}`;
|
||||
if (claim && (!claims.has(key) || Date.parse(claim.at) < Date.parse(claims.get(key).at))) claims.set(key, claim);
|
||||
}
|
||||
let lowered = 0;
|
||||
if (!claims.size) return lowered;
|
||||
for (const book of await fs.readdir(mangaPath, { withFileTypes: true })) {
|
||||
const bookKey = book.isDirectory() && parseKey(book.name)?.key;
|
||||
if (!bookKey) continue;
|
||||
for (const chapter of await fs.readdir(path.join(mangaPath, book.name), { withFileTypes: true })) {
|
||||
const parsed = chapter.isDirectory() && (parseSpecialFolder(chapter.name) || parseKey(chapter.name));
|
||||
const claim = parsed && claims.get(`${bookKey}/${parsed.key}`);
|
||||
if (!claim) continue;
|
||||
const file = path.join(mangaPath, book.name, chapter.name, 'chapter.json');
|
||||
let meta;
|
||||
try { meta = JSON.parse(await fs.readFile(file, 'utf8')); } catch { continue; }
|
||||
const next = publishedDates(meta, { ecosystem: claim });
|
||||
if (next['x-dates'].ecosystem.at === meta['x-dates']?.ecosystem?.at) continue;
|
||||
await writeJsonAtomic(file, { ...meta, ...next });
|
||||
lowered++;
|
||||
}
|
||||
}
|
||||
return lowered;
|
||||
}
|
||||
function sameTitle(left, right) {
|
||||
return String(left || '').normalize('NFKC').trim().replace(/\s+/g, ' ').toLowerCase()
|
||||
@@ -101,6 +133,7 @@ async function run(info, item, options) {
|
||||
await writeJsonAtomic(path.join(manga.path, 'manga.json'), manga.meta);
|
||||
}
|
||||
const selected = selectByScope(catalog.chapters.filter(ref => shouldDownload(manga, ref, source, item.existingPolicy)), item.scope);
|
||||
const ecosystemLowered = manga ? await lowerEcosystemDates(manga.path, catalog.chapters, peer.alias) : 0;
|
||||
job.total = selected.length; job.jobCompleted = 0; job.current = null; await record();
|
||||
let publishedCount = 0;
|
||||
const identityToPersist = localIdentity || workIdentity;
|
||||
@@ -132,6 +165,8 @@ async function run(info, item, options) {
|
||||
}
|
||||
const publication = await publishManifest({ version: 1, series, chapters: [{
|
||||
number: ref.number, title: ref.title, book: { number: Number(ref.volume) || 1 },
|
||||
// The peer's ecosystem date travels along: the chapter entered Circus there, not here.
|
||||
dates: fromPeer(chapter.dates || ref.dates, peer.alias),
|
||||
source: { name: `Peer ${peer.alias} ${peer.id.slice(1, 7)}`, url: sourceUrl },
|
||||
pages: pages.map(page => ({ path: page.path })),
|
||||
}] }, { libraryRoot: options.libraryRoot, ...(manga ? { mangaDir: manga.path } : {}) });
|
||||
@@ -147,7 +182,7 @@ async function run(info, item, options) {
|
||||
if (index < selected.length - 1) { stop(); await sleep(Math.max(500, Number(options.settings.chapterMinMs || 500))); }
|
||||
}
|
||||
job.status = 'complete'; job.current = null; await record();
|
||||
return { finalPath: manga?.path || null, publishedCount, selectedCount: selected.length };
|
||||
return { finalPath: manga?.path || null, publishedCount, selectedCount: selected.length, ecosystemLowered };
|
||||
}
|
||||
function adapterForPeer(value) {
|
||||
const info = infoFor(value);
|
||||
|
||||
+15
-3
@@ -9,6 +9,7 @@ const crypto = require('node:crypto');
|
||||
const { once } = require('node:events');
|
||||
const { PeeringState, fp, hash, requirePeer, grantStatus } = require('./state');
|
||||
const { parseKey, parseSpecialFolder } = require('../chapter-identity');
|
||||
const { peerDates } = require('../chapter-dates');
|
||||
const { resolveWorkIdentity } = require('../scraper/lib/work-identity');
|
||||
|
||||
const DATA = path.resolve(process.env.PEER_DATA || path.join(process.env.DATA || 'data', 'peering'));
|
||||
@@ -116,12 +117,23 @@ async function mangaIndex(peer, share) {
|
||||
if (!pages.length) continue;
|
||||
const id = `c${opaque(`${share.id}/${bookName}/${chapterName}`)}`;
|
||||
const chapterMeta = await jsonFile(path.join(chapterDir, 'chapter.json'));
|
||||
const dirStat = await fsp.stat(chapterDir);
|
||||
const item = { id, number: parsed.key, volume: book.key, title: String(chapterMeta.title || parsed.title || '').slice(0, 200),
|
||||
pageCount: pages.length, chapterDir, pages };
|
||||
pageCount: pages.length, chapterDir, pages, dates: peerDates(chapterMeta, dirStat.mtimeMs) };
|
||||
chapters.push(item); byId.set(id, item);
|
||||
if (chapters.length > 10000) fail(413, 'Shared manga chapter limit exceeded');
|
||||
}
|
||||
}
|
||||
// Every version of a chapter number offers the earliest ecosystem date known for it.
|
||||
const earliest = new Map();
|
||||
for (const item of chapters) {
|
||||
const key = `${item.volume}/${item.number}`, at = item.dates.ecosystem;
|
||||
if (at && (!earliest.has(key) || Date.parse(at) < Date.parse(earliest.get(key)))) earliest.set(key, at);
|
||||
}
|
||||
for (const item of chapters) {
|
||||
const at = earliest.get(`${item.volume}/${item.number}`);
|
||||
if (at) item.dates = { ...item.dates, ecosystem: at };
|
||||
}
|
||||
const result = { at: Date.now(), mangaDir, meta, chapters, byId };
|
||||
indexCache.set(key, result);
|
||||
return result;
|
||||
@@ -147,7 +159,7 @@ async function chapterManifest(peer, share, chapterId) {
|
||||
size: stat.size, mtimeMs: stat.mtimeMs, at: Date.now() });
|
||||
if (pages.length > 500) fail(413, 'Chapter page limit exceeded');
|
||||
}
|
||||
return { id: chapter.id, number: chapter.number, volume: chapter.volume, title: chapter.title, pages };
|
||||
return { id: chapter.id, number: chapter.number, volume: chapter.volume, title: chapter.title, dates: chapter.dates, pages };
|
||||
}
|
||||
async function streamBlob(req, res, peer, id) {
|
||||
const item = blobCache.get(`${peer.id}:${id}`);
|
||||
@@ -248,7 +260,7 @@ const server = https.createServer({ key: identity.key, cert: identity.cert, requ
|
||||
return json(res, 200, { id: share.id, title: String(index.meta.title || share.mangaId).slice(0, 200),
|
||||
workIdentity: resolveWorkIdentity(index.meta, share.mangaId, identity.id),
|
||||
...publicMetadata(index.meta), coverAvailable: !!coverFile(index.mangaDir),
|
||||
chapters: index.chapters.slice(cursor, cursor + 100).map(({ id, number, volume, title, pageCount }) => ({ id, number, volume, title, pageCount })),
|
||||
chapters: index.chapters.slice(cursor, cursor + 100).map(({ id, number, volume, title, pageCount, dates }) => ({ id, number, volume, title, pageCount, dates })),
|
||||
nextCursor: cursor + 100 < index.chapters.length ? String(cursor + 100) : null });
|
||||
}
|
||||
const chapterMatch = /^\/peer\/v1\/manga\/(m[0-9a-f]{24})\/chapters\/(c[0-9a-f]{32})$/.exec(url.pathname);
|
||||
|
||||
@@ -68,7 +68,8 @@ test('mTLS pairing, opt-in catalog, peer queue publication, and revocation', { t
|
||||
await fsp.mkdir(chapterDir, { recursive: true });
|
||||
await fsp.writeFile(path.join(mangaDir, 'manga.json'), JSON.stringify({ title: 'Sample Manga', direction: 'ltr' }));
|
||||
await fsp.writeFile(path.join(mangaDir, 'cover.png'), PNG);
|
||||
await fsp.writeFile(path.join(chapterDir, 'chapter.json'), JSON.stringify({ title: 'The first chapter', source: { name: 'Local' } }));
|
||||
await fsp.writeFile(path.join(chapterDir, 'chapter.json'), JSON.stringify({ title: 'The first chapter', source: { name: 'Local' },
|
||||
added: '2026-10-01T08:00:00.000Z', 'x-dates': { version: 1, source: { at: '2026-09-30T15:00:00.000Z', name: 'WeebCentral', batch: false } } }));
|
||||
await fsp.writeFile(path.join(chapterDir, '0001.png'), PNG);
|
||||
const ga = await gateway(dataA, libA, portA, path.join(root, 'usage-a.json'));
|
||||
let gb = await gateway(dataB, libB, portB, path.join(root, 'usage-b.json'));
|
||||
@@ -134,10 +135,22 @@ test('mTLS pairing, opt-in catalog, peer queue publication, and revocation', { t
|
||||
assert.equal(result.publishedCount, 1);
|
||||
assert.deepEqual(JSON.parse(await fsp.readFile(path.join(libB, 'Sample Manga', 'manga.json'), 'utf8'))['x-work-identity'], remote.workIdentity,
|
||||
'The stable work identity is stored with the imported peer edition');
|
||||
assert.equal(fs.existsSync(path.join(libB, 'Sample Manga', 'v01', 'c0001 The first chapter [Peer ServerA ' + inB.id.slice(1, 7) + ']', '0001.png')), true);
|
||||
const receivedDir = path.join(libB, 'Sample Manga', 'v01', 'c0001 The first chapter [Peer ServerA ' + inB.id.slice(1, 7) + ']');
|
||||
assert.equal(fs.existsSync(path.join(receivedDir, '0001.png')), true);
|
||||
const received = JSON.parse(await fsp.readFile(path.join(receivedDir, 'chapter.json'), 'utf8'));
|
||||
assert.deepEqual(received['x-dates'].ecosystem, { at: '2026-10-01T08:00:00.000Z', via: 'peer ServerA' },
|
||||
'The chapter keeps the date it first entered Manga Circus on A, not its arrival on B');
|
||||
assert.equal(received['x-dates'].source.at, '2026-09-30T15:00:00.000Z');
|
||||
assert.notEqual(received.added, '2026-10-01T08:00:00.000Z', '"added" is the arrival in B\'s own library');
|
||||
// B claims it got the chapter later on its own: the next sync lowers it to A's date again.
|
||||
await fsp.writeFile(path.join(receivedDir, 'chapter.json'), JSON.stringify({ ...received,
|
||||
'x-dates': { ...received['x-dates'], ecosystem: { at: '2026-10-20T00:00:00.000Z', via: 'this server' } } }));
|
||||
const second = await run(info, { scope: { mode: 'missing' }, existingPolicy: 'skip-any' },
|
||||
{ libraryRoot: libB, settings: { chapterMinMs: 500 }, shouldStop: () => false });
|
||||
assert.equal(second.publishedCount, 0);
|
||||
assert.equal(second.ecosystemLowered, 1, 'An existing chapter adopts the peer\'s earlier ecosystem date');
|
||||
assert.deepEqual(JSON.parse(await fsp.readFile(path.join(receivedDir, 'chapter.json'), 'utf8'))['x-dates'].ecosystem,
|
||||
{ at: '2026-10-01T08:00:00.000Z', via: 'peer ServerA' });
|
||||
|
||||
// Simulate B adding one chapter to its imported edition, then sharing it back.
|
||||
// A must identify its original folder by the carried origin identity and fetch
|
||||
|
||||
@@ -94,7 +94,18 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
|
||||
.prog i { display: block; height: 100%; background: var(--accent); }
|
||||
.badge { position: absolute; top: 7px; right: 7px; background: var(--accent); color: #fff; font-size: 12px; font-weight: 700; padding: 2px 8px; border-radius: 10px; }
|
||||
.empty { text-align: center; padding: 60px 10px; }
|
||||
/* One fixed row; tiles sized like a six-column .grid, extra tiles scroll sideways instead of wrapping. */
|
||||
.continue-row { margin-bottom: 20px; padding-bottom: 16px; border-bottom: 1px solid var(--line); }
|
||||
.continue-banner {
|
||||
display: inline-flex; gap: 6px; align-items: center; margin-bottom: 9px; padding: 2px 9px; border-radius: 10px;
|
||||
background: color-mix(in srgb, var(--accent) 22%, var(--card)); color: #ffd8de; font-size: 12px; font-weight: 650;
|
||||
}
|
||||
.continue-tiles {
|
||||
display: grid; grid-auto-flow: column; grid-auto-columns: max(140px, calc((100% - 80px) / 6)); gap: 16px;
|
||||
overflow-x: auto; padding-bottom: 4px; scrollbar-width: thin;
|
||||
}
|
||||
@media (max-width: 560px) {
|
||||
.continue-tiles { grid-auto-columns: calc((100% - 12px) / 2); gap: 12px; }
|
||||
main { padding: 12px; }
|
||||
.toolbar { flex-wrap: wrap; }
|
||||
.toolbar input { flex-basis: 100%; }
|
||||
@@ -180,6 +191,11 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
|
||||
.ch { display: flex; width: 100%; max-width: 100%; align-items: center; gap: 6px; border-bottom: 1px solid var(--line); white-space: nowrap; }
|
||||
.ch-main { flex: 1 1 auto; min-width: 0; display: grid; grid-template-columns: 76px 52px minmax(70px, 1fr) max-content; gap: 8px; align-items: baseline; padding: 6px 4px; text-decoration: none; }
|
||||
.ch-main.has-time { grid-template-columns: 76px 52px 72px minmax(70px, 1fr) max-content; }
|
||||
.ch-main.has-date { grid-template-columns: 76px 52px 104px minmax(70px, 1fr) max-content; }
|
||||
.ch-main.has-time.has-date { grid-template-columns: 76px 52px 104px 72px minmax(70px, 1fr) max-content; }
|
||||
.ch-date { min-width: 0; overflow: hidden; color: var(--muted); font-size: 13px; font-variant-numeric: tabular-nums; text-overflow: ellipsis; white-space: nowrap; }
|
||||
.ch-date.tier-official { color: inherit; }
|
||||
.ch-date.tier-ecosystem, .ch-date.tier-library { opacity: .75; }
|
||||
.ch-main:hover .ch-title { text-decoration: underline; }
|
||||
.ch-label { min-width: 0; overflow: hidden; text-overflow: ellipsis; font-weight: 650; font-variant-numeric: tabular-nums; }
|
||||
.ch-pages { min-width: 0; overflow: hidden; color: var(--muted); font-size: 13px; font-variant-numeric: tabular-nums; text-align: left; white-space: nowrap; }
|
||||
@@ -205,6 +221,10 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
|
||||
.ch { flex-wrap: nowrap; gap: 4px; overflow: hidden; }
|
||||
.ch-main { grid-template-columns: 64px 44px minmax(0, 1fr) max-content; gap: 6px; }
|
||||
.ch-main.has-time { grid-template-columns: 64px 44px 62px minmax(0, 1fr) max-content; }
|
||||
/* Phones: the date takes the page-count column. */
|
||||
.ch-main.has-date { grid-template-columns: 64px 64px minmax(0, 1fr) max-content; }
|
||||
.ch-main.has-time.has-date { grid-template-columns: 64px 64px 62px minmax(0, 1fr) max-content; }
|
||||
.ch-main.has-date .ch-pages { display: none; }
|
||||
.ch > .btn.small { min-width: 0; padding-left: 5px; padding-right: 5px; }
|
||||
.ch > .state { flex: none; }
|
||||
}
|
||||
@@ -309,6 +329,10 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
|
||||
.discovery-card-body details { margin: 10px 0; }
|
||||
.discovery-card-body ul { padding-left: 18px; }
|
||||
.discovery-actions { margin: 10px 0; }
|
||||
.discovery-refresh-row { margin: 0 0 14px; }
|
||||
/* The paused and cooldown labels are long; let them wrap on phones instead of widening the page. */
|
||||
.discovery-refresh { max-width: 100%; white-space: normal; text-align: center; }
|
||||
.discovery-harvest-settings { align-items: center; gap: 12px; margin: 10px 0; }
|
||||
.catalog-workspace { min-width: 0; margin: 0; padding: 0; border: 0; }
|
||||
.catalog-workspace > .row { margin-bottom: 16px; }
|
||||
.catalog-workspace input, .catalog-workspace select, .catalog-workspace textarea { min-width: 0; max-width: 100%; }
|
||||
@@ -478,3 +502,66 @@ body.nobars #bot { transform: translateY(100%); opacity: 0; pointer-events: none
|
||||
.seg button { padding: 7px 10px; } #bot .btn { padding: 6px 10px; } .wide-only { display: none; }
|
||||
.clock-panel { bottom: calc(124px + env(safe-area-inset-bottom)); }
|
||||
}
|
||||
|
||||
/* ---------------- moments (BL-0055) ---------------- */
|
||||
.moment-btn.on { color: var(--accent); border-color: var(--accent); }
|
||||
.moment-btn.on svg path:first-child { fill: currentColor; }
|
||||
.moment-dialog { width: min(640px, calc(100vw - 16px)); max-height: calc(100vh - 16px); overflow: auto; color: var(--fg); background: var(--bg);
|
||||
border: 1px solid var(--line); border-radius: 12px; padding: 16px; }
|
||||
.moment-dialog::backdrop { background: #000c; }
|
||||
.moment-dialog h2 { margin: 0; font-size: 18px; }
|
||||
.moment-head { display: flex; align-items: center; gap: 10px; margin-bottom: 8px; }
|
||||
.moment-head > :first-child { flex: 1; min-width: 0; }
|
||||
.moment-context { font-weight: 600; }
|
||||
.moment-preview-wrap { display: flex; justify-content: center; margin: 12px 0 8px; }
|
||||
.moment-preview { position: relative; line-height: 0; }
|
||||
.moment-composite { position: relative; }
|
||||
.moment-page { position: absolute; top: 0; overflow: hidden; background: #19191c; }
|
||||
.moment-page img { display: block; width: 100%; height: 100%; -webkit-user-drag: none; user-select: none; }
|
||||
.region-layer { position: absolute; inset: 0; cursor: crosshair; touch-action: none; overflow: hidden; }
|
||||
.region-box { position: absolute; border: 2px solid #fff; outline: 1px solid #0008; box-shadow: 0 0 0 9999px #0009; pointer-events: none; }
|
||||
.moment-region-row { display: flex; align-items: center; justify-content: space-between; gap: 10px; min-height: 32px; }
|
||||
.moment-chips { display: flex; flex-wrap: wrap; gap: 8px; margin: 12px 0; }
|
||||
.moment-chip { display: inline-flex; align-items: center; gap: 6px; padding: 6px 12px; border-radius: 18px; border: 1px solid var(--line);
|
||||
background: var(--card); color: var(--fg); font: inherit; cursor: pointer; }
|
||||
.moment-chip:hover { background: var(--card2); }
|
||||
.moment-chip[aria-checked="true"] { border-color: var(--cat); background: color-mix(in srgb, var(--cat) 22%, var(--card)); box-shadow: 0 0 0 1px var(--cat); }
|
||||
.moment-dialog input[type=text] { width: 100%; }
|
||||
.moment-actions { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; margin-top: 12px; }
|
||||
.moment-dialog .btn[disabled] { opacity: .45; pointer-events: none; }
|
||||
.moment-crop { position: relative; overflow: hidden; border-radius: 6px; background: #19191c; flex: none; }
|
||||
.moment-crop.missing { display: flex; align-items: center; justify-content: center; color: var(--muted); font-size: 13px; line-height: 1.2; text-align: center; }
|
||||
.moment-viewer { width: auto; max-width: calc(100vw - 16px); }
|
||||
.moment-viewer-body { display: flex; align-items: center; justify-content: center; gap: 8px; }
|
||||
.moment-viewer-stage { display: flex; justify-content: center; touch-action: pan-y; user-select: none; }
|
||||
.moment-viewer-info { display: flex; flex-wrap: wrap; align-items: center; gap: 4px 12px; }
|
||||
.moment-tag { display: inline-flex; align-items: center; gap: 5px; padding: 2px 10px; border-radius: 12px; font-size: 13px;
|
||||
border: 1px solid var(--cat); background: color-mix(in srgb, var(--cat) 18%, var(--card)); }
|
||||
.moment-note { flex-basis: 100%; color: var(--muted); font-style: italic; }
|
||||
.moment-nav { font-size: 22px; min-width: 40px; }
|
||||
@media (max-width: 600px) {
|
||||
.moment-viewer-body { position: relative; }
|
||||
.moment-nav { position: absolute; z-index: 1; background: #000a; }
|
||||
.moment-nav.prev { left: 2px; } .moment-nav.next { right: 2px; }
|
||||
}
|
||||
.moments-card { max-width: none; }
|
||||
.moments-head { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; margin-bottom: 8px; }
|
||||
.moments-card h2 { margin: 0; }
|
||||
.moment-filters { display: flex; gap: 7px; overflow-x: auto; padding-bottom: 6px; }
|
||||
.moment-filter { display: inline-flex; align-items: center; gap: 5px; }
|
||||
.moment-strip { display: flex; gap: 10px; overflow-x: auto; padding: 4px 2px 10px; scrollbar-width: thin; }
|
||||
.moment-thumb { position: relative; flex: none; display: flex; flex-direction: column; gap: 4px; padding: 0; border: 0; background: none;
|
||||
color: var(--fg); font: inherit; cursor: pointer; text-align: left; }
|
||||
.moment-thumb .moment-crop { transition: transform .15s; }
|
||||
.moment-thumb:hover .moment-crop, .moment-thumb:focus-visible .moment-crop { transform: scale(1.03); }
|
||||
.moment-thumb-cat { position: absolute; top: 5px; left: 5px; display: flex; padding: 3px; border-radius: 50%; background: #000b; }
|
||||
.moment-thumb-label { font-size: 12px; color: var(--muted); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; max-width: 200px; }
|
||||
.key-group h3 { margin: 4px 0 8px; font-size: 15px; }
|
||||
.key-group + .key-group { margin-top: 16px; }
|
||||
.key-table { border-collapse: collapse; width: 100%; }
|
||||
.key-table td { padding: 5px 8px; border-top: 1px solid var(--line); vertical-align: top; }
|
||||
.key-table td:first-child { width: 1%; white-space: nowrap; }
|
||||
kbd { display: inline-block; min-width: 24px; padding: 1px 7px; border: 1px solid var(--line); border-bottom-width: 2px; border-radius: 5px;
|
||||
background: var(--card2); font: 13px/1.5 ui-monospace, monospace; text-align: center; }
|
||||
@media (max-width: 480px) { .moment-dialog { padding: 12px; } .moment-actions .moment-hint { flex-basis: 100%; } .moment-actions .moment-hint:empty { display: none; } }
|
||||
.moment-extend { display: flex; flex-wrap: wrap; justify-content: center; gap: 8px; margin-bottom: 8px; }
|
||||
+61
-18
@@ -5,13 +5,25 @@
|
||||
topNav(me);
|
||||
const main = $('main'), note = h('p', { role: 'status', 'aria-live': 'polite', class: 'muted small' });
|
||||
const button = (label, action, className = '') => h('button', { type: 'button', class: `btn ${className}`, onclick: action }, label);
|
||||
const clock = iso => new Date(iso).toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' });
|
||||
const fail = error => { note.textContent = error.message; note.className = 'error'; };
|
||||
async function refresh() {
|
||||
try { state = await api('api/discovery'); if (me.admin) adminState = await api('api/admin/discovery'); render(); }
|
||||
catch (error) { note.textContent = error.message; note.className = 'error'; }
|
||||
catch (error) { fail(error); }
|
||||
}
|
||||
// Something is still working server-side: a scoring run, a harvest, or this user's refresh.
|
||||
const busy = () => adminState?.currentRun?.status === 'running' || !!adminState?.currentHarvest
|
||||
|| ['queued', 'running'].includes(state.refresh?.state);
|
||||
function watch() {
|
||||
if (active || !busy()) return;
|
||||
active = setInterval(async () => {
|
||||
await refresh();
|
||||
if (!busy()) { clearInterval(active); active = null; }
|
||||
}, 1500);
|
||||
}
|
||||
async function preference(key, checked) {
|
||||
try { await api('api/settings', { [key]: checked }); await refresh(); }
|
||||
catch (error) { note.textContent = error.message; note.className = 'error'; }
|
||||
catch (error) { fail(error); }
|
||||
}
|
||||
async function feedback(candidateId, value) {
|
||||
try {
|
||||
@@ -19,19 +31,27 @@
|
||||
state.feedback[candidateId] = value;
|
||||
note.textContent = value ? 'Preference saved. The next run will use it.' : 'Preference removed.';
|
||||
render();
|
||||
} catch (error) { note.textContent = error.message; note.className = 'error'; }
|
||||
} catch (error) { fail(error); }
|
||||
}
|
||||
async function run() {
|
||||
// Start a server job, then poll until it finishes; on refusal re-read the state (cooldown, pause).
|
||||
async function start(endpoint, message) {
|
||||
try {
|
||||
await api('api/admin/discovery/run', {});
|
||||
note.textContent = 'Discovery running. Existing suggestions remain visible until it finishes.';
|
||||
await api(endpoint, {});
|
||||
note.textContent = message; note.className = 'muted small';
|
||||
} catch (error) { fail(error); }
|
||||
await refresh();
|
||||
if (active) clearInterval(active);
|
||||
active = setInterval(async () => {
|
||||
await refresh();
|
||||
if (adminState?.currentRun?.status !== 'running') { clearInterval(active); active = null; }
|
||||
}, 1500);
|
||||
} catch (error) { note.textContent = error.message; note.className = 'error'; }
|
||||
watch();
|
||||
}
|
||||
const run = () => start('api/admin/discovery/run', 'Discovery running. Existing suggestions remain visible until it finishes.');
|
||||
const refreshMine = () => start('api/discovery/refresh', 'Refresh queued. Your current suggestions stay visible until it finishes.');
|
||||
const harvestNow = () => start('api/admin/discovery/harvest', 'Harvest started. Suggestions are re-scored when it finishes.');
|
||||
function refreshButton() {
|
||||
const refreshState = state.refresh || { state: 'idle' };
|
||||
const label = refreshState.state === 'queued' ? `Queued (position ${refreshState.position})`
|
||||
: refreshState.state === 'running' ? 'Fetching…'
|
||||
: refreshState.blockedReason ? `Paused by AniList (${refreshState.blockedReason}) until the next nightly or admin harvest`
|
||||
: refreshState.nextAllowedAt ? `Next refresh possible at ${clock(refreshState.nextAllowedAt)}` : null;
|
||||
return h('button', { type: 'button', class: 'btn discovery-refresh', disabled: !!label, onclick: refreshMine }, label || 'Refresh my suggestions');
|
||||
}
|
||||
function toggle(label, key, hint) {
|
||||
return h('label', { class: 'check' }, h('input', { type: 'checkbox', checked: !!state.settings[key],
|
||||
@@ -40,18 +60,37 @@
|
||||
}
|
||||
function preferences() {
|
||||
return h('section', { class: 'card discovery-settings' }, h('h2', {}, 'Your discovery preferences'),
|
||||
toggle('Personalized discovery', 'discoveryEnabled', 'Off by default. Your feed and feedback stay private to this server.'),
|
||||
toggle('Personalized discovery', 'discoveryEnabled', 'Off by default. When on, the server sends AniList the IDs of titles you liked, read or own, their authors\' IDs and your top tags, without login, to find related work. Your feed and feedback stay private to this server.'),
|
||||
toggle('Use reading activity', 'discoveryReadingActivity', 'Separately opt in to using completed chapters as a small taste signal.'),
|
||||
toggle('Include mature titles', 'discoveryMature', 'Only affects titles with a known mature tag.'));
|
||||
toggle('Include mature titles', 'discoveryMature', 'Only affects titles AniList marks as adult or with a known mature tag.'));
|
||||
}
|
||||
function harvestSettings() {
|
||||
const settings = adminState?.settings || { harvestEnabled: false, harvestAt: '03:30' };
|
||||
const enabled = h('input', { type: 'checkbox', checked: settings.harvestEnabled, onchange: save });
|
||||
const time = h('input', { type: 'time', value: settings.harvestAt, 'aria-label': 'Harvest time', required: true, onchange: save });
|
||||
async function save() {
|
||||
if (!time.value) return;
|
||||
try {
|
||||
const saved = await api('api/admin/discovery/settings', { harvestEnabled: enabled.checked, harvestAt: time.value });
|
||||
note.textContent = `Harvest settings saved (${saved.settings.harvestEnabled ? 'on' : 'off'}, ${saved.settings.harvestAt})`;
|
||||
note.className = 'muted small';
|
||||
await refresh();
|
||||
} catch (error) { fail(error); }
|
||||
}
|
||||
return h('div', { class: 'row discovery-harvest-settings' }, h('label', { class: 'check' }, enabled, h('span', {}, 'Automatic harvest')), time);
|
||||
}
|
||||
function adminPanel() {
|
||||
if (!me.admin) return null;
|
||||
const runState = adminState?.currentRun?.status || adminState?.lastRun?.status || 'never';
|
||||
const harvest = adminState?.harvest, cache = adminState?.cache;
|
||||
return h('section', { class: 'card discovery-admin' }, h('h2', {}, 'Discovery job'),
|
||||
h('p', { class: 'muted small' }, `Status: ${runState} · ${adminState?.candidateCount || 0} candidates · ${adminState?.optedInUsers || 0} opted-in users`),
|
||||
h('p', { class: 'muted small' }, `Last harvest: ${harvest?.status || 'never'} · ${harvest?.requests ?? 0}/${harvest?.budget ?? 0} requests · ${harvest?.stopReason || '—'} · cache ${cache?.files ?? 0} files`),
|
||||
h('p', { class: 'muted small' }, adminState?.nextNightlyAt ? `Next nightly harvest: ${new Date(adminState.nextNightlyAt).toLocaleString()}` : 'Automatic harvest is off.'),
|
||||
adminState?.lastError ? h('p', { class: 'error' }, adminState.lastError) : null,
|
||||
button('Run discovery now', run, 'primary'),
|
||||
h('p', { class: 'muted small' }, 'This first version uses cached public metadata and the curated catalog. It does not ask image hosters, call an LLM, or edit the scraper queue.'));
|
||||
h('div', { class: 'row' }, button('Run discovery now', run, 'primary'), button('Harvest now', harvestNow)),
|
||||
harvestSettings(),
|
||||
h('p', { class: 'muted small' }, 'Harvests ask AniList for public metadata only. They never ask image hosters, call an LLM, or edit the scraper queue.'));
|
||||
}
|
||||
function card(item) {
|
||||
const candidate = item.candidate, vote = state.feedback[candidate.id] || null;
|
||||
@@ -64,8 +103,10 @@
|
||||
h('p', { class: 'muted small' }, [candidate.year || 'Year unknown', candidate.status || 'Status unknown', candidate.period].filter(Boolean).join(' · ')),
|
||||
h('p', {}, item.reason),
|
||||
h('details', {}, h('summary', {}, 'Why this is here · evidence'),
|
||||
h('ul', {}, evidence.map(entry => h('li', {}, entry.url ? h('a', { href: entry.url, target: '_blank', rel: 'noopener noreferrer' }, entry.label) : entry.label,
|
||||
entry.note ? h('span', { class: 'muted' }, ` — ${entry.note}`) : null))),
|
||||
h('ul', {}, (item.reasons || []).map(reason => h('li', {}, reason)),
|
||||
evidence.map(entry => h('li', {}, entry.url ? h('a', { href: entry.url, target: '_blank', rel: 'noopener noreferrer' }, entry.label) : entry.label,
|
||||
entry.note ? h('span', { class: 'muted' }, ` — ${entry.note}`) : null,
|
||||
entry.fetchedAt ? h('span', { class: 'muted' }, ` · fetched ${new Date(entry.fetchedAt).toLocaleDateString()}`) : null))),
|
||||
h('p', { class: 'muted small' }, 'A recommendation does not verify a particular edition or English chapter availability.')),
|
||||
h('div', { class: 'row discovery-actions' }, votes.map(([value, label]) => button(label, () => feedback(candidate.id, vote === value ? null : value), vote === value ? 'on' : ''))),
|
||||
candidate.metadataUrl ? h('a', { class: 'small', href: candidate.metadataUrl, target: '_blank', rel: 'noopener noreferrer' }, 'Public metadata ↗') : null));
|
||||
@@ -75,10 +116,12 @@
|
||||
main.append(h('h1', {}, 'Discover'), h('p', { class: 'muted' }, 'A small, evidence-backed shelf of manga you might like. Suggestions are not downloads.'), note,
|
||||
preferences(), adminPanel());
|
||||
if (!state.enabled) return main.append(h('p', { class: 'muted' }, 'Turn on personalized discovery to see your suggestions.'));
|
||||
main.append(h('div', { class: 'row discovery-refresh-row' }, refreshButton()));
|
||||
if (!state.items.length) return main.append(h('p', { class: 'muted' }, state.lastRun?.status === 'never'
|
||||
? 'No discovery run yet. An admin can run one above.' : 'No current suggestions passed the filters. Your previous feedback is retained.'));
|
||||
main.append(h('p', { class: 'muted small' }, `Last built ${new Date(state.generatedAt).toLocaleString()} · ${state.items.length} suggestions`),
|
||||
h('div', { class: 'discovery-grid' }, state.items.filter(item => !['hidden', 'dismissed'].includes(state.feedback[item.candidateId])).map(card)));
|
||||
}
|
||||
await refresh();
|
||||
watch();
|
||||
})();
|
||||
+4
-2
@@ -65,8 +65,10 @@
|
||||
|
||||
function cover(item) {
|
||||
const box = h('span', { class: 'greatest-cover' }, h('span', { class: 'greatest-cover-fallback', 'aria-hidden': 'true' }, item.title.slice(0, 1)));
|
||||
if (item.cover && safeHref(item.cover)) {
|
||||
const image = h('img', { src: item.cover, alt: '', loading: 'lazy' });
|
||||
// Covers come from the server (cached catalog covers, owned library covers) as relative URLs.
|
||||
const src = /^(api\/greatest-cover\?|lib\/)/.test(item.cover || '') ? item.cover : safeHref(item.cover);
|
||||
if (src) {
|
||||
const image = h('img', { src, alt: '', loading: 'lazy' });
|
||||
image.onerror = () => image.remove();
|
||||
box.append(image);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
'use strict';
|
||||
/*
|
||||
* Keyboard shortcuts. The reader dispatches keys through READER_KEYS, and Settings and the reader's "?" list
|
||||
* are drawn from the same tables, so the documentation cannot drift from what the keys do.
|
||||
*/
|
||||
const READER_KEYS = [
|
||||
{ keys: ['ArrowLeft', 'ArrowRight'], label: '← →', action: 'turn', text: 'Previous / next page, following the reading direction' },
|
||||
{ keys: [' ', 'PageDown'], label: 'Space · Page Down', action: 'forward', text: 'Next page (Shift + Space: previous) · Page and Book mode' },
|
||||
{ keys: ['PageUp'], label: 'Page Up', action: 'back', text: 'Previous page · Page and Book mode' },
|
||||
{ keys: ['+', '='], label: '+', action: 'zoomIn', text: 'Zoom in 5 %' },
|
||||
{ keys: ['-'], label: '−', action: 'zoomOut', text: 'Zoom out 5 %' },
|
||||
{ keys: ['0'], label: '0', action: 'zoomFit', text: 'Fit the screen (100 %)' },
|
||||
{ keys: ['m', 'M'], label: 'M', action: 'moment', text: 'Save a moment of the page you are on' },
|
||||
{ keys: ['i', 'I'], label: 'I', action: 'bars', text: 'Show or hide the controls' },
|
||||
{ keys: ['Escape'], label: 'Esc', action: 'bars', text: 'Show or hide the controls (closes an open dialog first)' },
|
||||
{ keys: ['f', 'F'], label: 'F', action: 'fullscreen', text: 'Fullscreen' },
|
||||
{ keys: ['?'], label: '?', action: 'help', text: 'Show these shortcuts' },
|
||||
];
|
||||
const MOMENT_VIEWER_KEYS = [
|
||||
{ keys: ['ArrowLeft', 'ArrowRight'], label: '← →', action: 'step', text: 'Previous / next moment' },
|
||||
{ keys: ['Escape'], label: 'Esc', action: 'close', text: 'Close the viewer' },
|
||||
];
|
||||
const KEY_GROUPS = [
|
||||
{ title: 'Reader', keys: READER_KEYS },
|
||||
{ title: 'Moments viewer (manga page)', keys: MOMENT_VIEWER_KEYS },
|
||||
];
|
||||
const keyAction = (table, key) => (table.find(entry => entry.keys.includes(key)) || {}).action || null;
|
||||
|
||||
function keyTable(groups = KEY_GROUPS) {
|
||||
return groups.map(group => h('section', { class: 'key-group' }, h('h3', {}, group.title),
|
||||
h('table', { class: 'key-table' }, h('tbody', {}, group.keys.map(entry =>
|
||||
h('tr', {}, h('td', {}, h('kbd', {}, entry.label)), h('td', {}, entry.text)))))));
|
||||
}
|
||||
|
||||
if (typeof module !== 'undefined') module.exports = { READER_KEYS, MOMENT_VIEWER_KEYS, KEY_GROUPS, keyAction };
|
||||
+23
-12
@@ -55,6 +55,12 @@
|
||||
h('option', { value: 'grouped' }, 'Group collections'), h('option', { value: 'expanded' }, 'Expand collections'), h('option', { value: 'individual' }, 'Individual manga'));
|
||||
grouping.value = ['grouped', 'expanded', 'individual'].includes(store.get('grouping')) ? store.get('grouping') : 'grouped';
|
||||
grouping.onchange = render;
|
||||
// Started but unfinished manga, most recently read first; the same manga still appear in the grid.
|
||||
const continueItems = data.mangas.filter(m => m.lastRead && m.read < m.chapters)
|
||||
.sort((a, b) => b.lastRead - a.lastRead).slice(0, 5);
|
||||
const continueRow = h('section', { class: 'continue-row', 'aria-label': 'Continue where you left off', hidden: true },
|
||||
h('div', { class: 'continue-banner' }, h('span', { 'aria-hidden': 'true' }, '▶'), 'Continue where you left off'),
|
||||
h('div', { class: 'continue-tiles' }, continueItems.map(m => tile(m))));
|
||||
if (storyAvailable) {
|
||||
const most = data.mangas.find(manga => manga.id === data.reading.mostTimeMangaId);
|
||||
const latest = data.mangas.find(manga => manga.id === data.reading.lastMangaId);
|
||||
@@ -76,7 +82,7 @@
|
||||
h('div', { class: 'filter-heading' }, h('b', {}, 'Tags'), h('span', { class: 'muted small' }, 'Manga must match every selected tag')),
|
||||
tagBar) : null,
|
||||
resultCount),
|
||||
grid);
|
||||
continueRow, grid);
|
||||
|
||||
function syncTags() {
|
||||
store.set('tags', JSON.stringify([...selectedTags]));
|
||||
@@ -95,8 +101,24 @@
|
||||
return direction * (at - bt) || a.title.localeCompare(b.title);
|
||||
}
|
||||
|
||||
function tile(m) {
|
||||
const cover = m.isCollection && m.coverMode === 'mosaic' && m.covers.length
|
||||
? h('div', { class: 'cv collection-mosaic' }, m.covers.map(src => h('img', { loading: 'lazy', src, alt: '' })))
|
||||
: h('div', { class: 'cv' }, m.cover ? h('img', { loading: 'lazy', src: m.cover, alt: '' }) : null);
|
||||
return h('a', { class: 'tile', href: m.isCollection ? 'collection.html?' + qsa({ id: m.id }) : mangaUrl(m.id) },
|
||||
cover, m.isCollection ? h('span', { class: 'collection-count' }, `${m.members.length} manga`) : null,
|
||||
m.unread ? h('span', { class: 'badge' }, m.unread + ' new') : null,
|
||||
h('div', { class: 't' }, m.title),
|
||||
h('div', { class: 's' }, [m.firstRelease, m.status, m.read ? `${m.read} / ${m.chapters} read` : `${m.chapters} chapters`].filter(Boolean).join(' · ')),
|
||||
m.read ? h('div', { class: 'prog' }, h('i', { style: `width:${Math.round((100 * m.read) / Math.max(1, m.chapters))}%` })) : null);
|
||||
}
|
||||
|
||||
function render() {
|
||||
store.set('filter', filter.value); store.set('sort', sort.value); store.set('status', status.value); store.set('grouping', grouping.value);
|
||||
// Only on the plain overview: a search or status filter means the reader is looking for something else,
|
||||
// and "Recently read" already puts these first. Tags and grouping keep it.
|
||||
continueRow.hidden = !me.settings.continueRow || !continueItems.length || sort.value === 'read'
|
||||
|| !!filter.value.trim() || !!status.value;
|
||||
const q = filter.value.trim().toLowerCase();
|
||||
const groups = (data.collections || []).filter(c => c.grouped);
|
||||
const membership = new Set(groups.flatMap(c => c.members.map(m => m.id)));
|
||||
@@ -125,17 +147,6 @@
|
||||
}
|
||||
resultCount.textContent = `${list.length} results · ${data.mangas.length} manga in the library`;
|
||||
grid.textContent = '';
|
||||
const tile = m => {
|
||||
const cover = m.isCollection && m.coverMode === 'mosaic' && m.covers.length
|
||||
? h('div', { class: 'cv collection-mosaic' }, m.covers.map(src => h('img', { loading: 'lazy', src, alt: '' })))
|
||||
: h('div', { class: 'cv' }, m.cover ? h('img', { loading: 'lazy', src: m.cover, alt: '' }) : null);
|
||||
return h('a', { class: 'tile', href: m.isCollection ? 'collection.html?' + qsa({ id: m.id }) : mangaUrl(m.id) },
|
||||
cover, m.isCollection ? h('span', { class: 'collection-count' }, `${m.members.length} manga`) : null,
|
||||
m.unread ? h('span', { class: 'badge' }, m.unread + ' new') : null,
|
||||
h('div', { class: 't' }, m.title),
|
||||
h('div', { class: 's' }, [m.firstRelease, m.status, m.read ? `${m.read} / ${m.chapters} read` : `${m.chapters} chapters`].filter(Boolean).join(' · ')),
|
||||
m.read ? h('div', { class: 'prog' }, h('i', { style: `width:${Math.round((100 * m.read) / Math.max(1, m.chapters))}%` })) : null);
|
||||
};
|
||||
if (grouping.value === 'expanded') {
|
||||
const visible = new Set(list.map(m => m.id));
|
||||
for (const c of groups) {
|
||||
|
||||
@@ -9,6 +9,8 @@
|
||||
<link rel="manifest" href="manifest.webmanifest">
|
||||
<meta name="theme-color" content="#121214">
|
||||
<script src="common.js" defer></script>
|
||||
<script src="keys.js" defer></script>
|
||||
<script src="moments.js" defer></script>
|
||||
<script src="manga.js" defer></script>
|
||||
</head>
|
||||
<body>
|
||||
|
||||
+94
-5
@@ -164,6 +164,7 @@
|
||||
main.append(h('details', { class: 'metadata card', open: metadataOpen },
|
||||
h('summary', { class: 'metadata-summary' }, h('h2', {}, 'Metadata')),
|
||||
metadata, rawMeta));
|
||||
main.append(momentsGallery());
|
||||
|
||||
const volumeButtons = new Map();
|
||||
const volumeBar = h('div', { class: 'volume-filter', 'aria-label': 'Filter chapters by volume' });
|
||||
@@ -213,14 +214,16 @@
|
||||
const panel = h('div', { class: 'versions', hidden: true });
|
||||
function draw() {
|
||||
const p = ch.progress, done = p && p.done;
|
||||
const state = h('button', { class: 'state' + (done ? ' done' : ''), title: done ? 'Mark as unread' : 'Mark as read' },
|
||||
done ? '✓ read' : p ? `page ${p.p + 1} / ${p.n}` : 'unread');
|
||||
const readAt = done ? readDate(ch.readAt) : null;
|
||||
const state = h('button', { class: 'state' + (done ? ' done' : ''), title: (readAt ? readAt.title + ' · ' : '') + (done ? 'Mark as unread' : 'Mark as read') },
|
||||
done ? (readAt ? '✓ ' + readAt.text : '✓ read') : p ? `page ${p.p + 1} / ${p.n}` : 'unread');
|
||||
state.onclick = async () => {
|
||||
if (done) { await api('api/unread', { m: M.id, c: ch.id }); ch.progress = null; }
|
||||
if (done) { await api('api/unread', { m: M.id, c: ch.id }); ch.progress = null; ch.readAt = null; }
|
||||
else {
|
||||
const version = ch.versions[0];
|
||||
await api('api/progress', { m: M.id, c: ch.id, v: version.id, p: version.pages - 1, n: version.pages, done: true });
|
||||
ch.progress = { v: version.id, p: version.pages - 1, n: version.pages, done: true };
|
||||
ch.readAt = { at: Date.now(), approx: false };
|
||||
}
|
||||
draw();
|
||||
};
|
||||
@@ -228,12 +231,14 @@
|
||||
? h('button', { class: 'btn small', onclick: () => { panel.hidden = !panel.hidden; } }, ch.versions.length + ' versions')
|
||||
: null;
|
||||
const pages = ch.versions[0] ? ch.versions[0].pages : 0;
|
||||
const timing = chapterTiming(ch);
|
||||
const timing = chapterTiming(ch), released = chapterRelease(ch.dates);
|
||||
wrap.textContent = '';
|
||||
wrap.append(h('div', { class: 'ch' + (done ? ' read' : '') },
|
||||
h('a', { class: 'ch-main' + (timing ? ' has-time' : ''), href: readUrl(M.id, ch.id) },
|
||||
h('a', { class: 'ch-main' + (timing ? ' has-time' : '') + (M.chapterDates?.release ? ' has-date' : ''), href: readUrl(M.id, ch.id) },
|
||||
h('span', { class: 'ch-label' }, ch.label),
|
||||
h('span', { class: 'ch-pages' }, `${pages} p.`),
|
||||
M.chapterDates?.release ? h('span', { class: 'ch-date' + (released ? ' tier-' + released.tier : ''), title: released ? released.title : 'No date known', 'aria-label': released ? released.title : 'No date known' },
|
||||
released ? released.text : '–') : null,
|
||||
timing ? h('span', { class: 'ch-time', title: timing.title, 'aria-label': timing.title }, timing.text) : null,
|
||||
h('span', { class: 'ch-title', title: ch.title || '' }, ch.title || 'Untitled'),
|
||||
ch.extra ? h('span', { class: 'tag' }, 'extra') : null),
|
||||
@@ -261,6 +266,42 @@
|
||||
return wrap;
|
||||
}
|
||||
|
||||
// Phones (the chapter-row breakpoint in app.css) get numeric dates to leave room for titles.
|
||||
const compact = matchMedia('(max-width: 640px)').matches;
|
||||
|
||||
// Read date: the day and the hour (local time) the chapter was first completed.
|
||||
function readDate(value) {
|
||||
const mode = M.chapterDates?.read;
|
||||
if (!value || !value.at || mode === 'off') return null;
|
||||
const when = new Date(value.at), thisYear = when.getFullYear() === new Date().getFullYear();
|
||||
const day = when.toLocaleDateString(undefined, compact ? { day: '2-digit', month: '2-digit', ...(thisYear ? {} : { year: '2-digit' }) }
|
||||
: { day: 'numeric', month: 'short', ...(thisYear ? {} : { year: 'numeric' }) });
|
||||
const text = (value.approx ? '≤ ' : '') + day + (mode === 'hour' ? `, ${String(when.getHours()).padStart(2, '0')}h` : '');
|
||||
const full = when.toLocaleDateString(undefined, { dateStyle: 'long' }) + (mode === 'hour' ? `, ${String(when.getHours()).padStart(2, '0')}:00–${String((when.getHours() + 1) % 24).padStart(2, '0')}:00` : '');
|
||||
return { text, title: (value.approx ? 'Read by ' : 'Read on ') + full + (value.approx ? ' (earlier reads were not recorded)' : '') };
|
||||
}
|
||||
|
||||
// Release date by tier: official > source upload > Manga Circus ecosystem > this library.
|
||||
const TIER = { official: { mark: '' }, source: { mark: '≈' }, ecosystem: { mark: '◎' }, library: { mark: '↓' } };
|
||||
const dayText = (at, precision, long) => {
|
||||
// Day-precision dates are calendar dates, not instants: show them in UTC so they do not shift.
|
||||
const dayOnly = !String(at).includes('T');
|
||||
const opts = precision === 'year' ? { year: 'numeric' } : precision === 'month' ? { year: 'numeric', month: 'short' }
|
||||
: long ? { dateStyle: 'long' } : compact ? { day: '2-digit', month: '2-digit', year: '2-digit' } : { day: 'numeric', month: 'short', year: 'numeric' };
|
||||
return new Date(at).toLocaleDateString(undefined, { ...opts, ...(dayOnly ? { timeZone: 'UTC' } : {}) });
|
||||
};
|
||||
function chapterRelease(dates) {
|
||||
if (!dates || !dates.release) return null;
|
||||
const known = dates.known || {}, lines = [];
|
||||
if (known.official) lines.push(`Released ${dayText(known.official.at, known.official.precision, true)}${known.official.name ? ' (' + known.official.name + ')' : ''}`);
|
||||
if (known.source) lines.push(`Uploaded at ${known.source.name || 'the source'} ${dayText(known.source.at, 'day', true)}`);
|
||||
if (known.batch) lines.push(`Bulk upload at ${known.batch.name || 'the source'} ${dayText(known.batch.at, 'day', true)} (not a release date)`);
|
||||
if (known.ecosystem) lines.push(`First in Manga Circus ${dayText(known.ecosystem.at, 'day', true)}${known.ecosystem.name && known.ecosystem.name !== 'this server' ? ' (via ' + known.ecosystem.name + ')' : ''}`);
|
||||
if (known.library) lines.push(`Added to this library ${dayText(known.library.at, 'day', true)}`);
|
||||
const tier = TIER[dates.tier] || TIER.library;
|
||||
return { tier: dates.tier, text: tier.mark + dayText(dates.release.at, dates.release.precision), title: lines.join('\n') };
|
||||
}
|
||||
|
||||
function chapterTiming(chapter) {
|
||||
if (!M.reading) return null;
|
||||
const value = chapter.reading;
|
||||
@@ -276,6 +317,54 @@
|
||||
return { text: `◷ ${readingDuration(active, true)}`, title: `${readingDuration(active)} active; current pass in progress` };
|
||||
}
|
||||
|
||||
// Your moments (BL-0055): hidden until there is at least one. Only the user's own are ever returned.
|
||||
function momentsGallery() {
|
||||
const sortKey = `moments-sort.${me.name}`;
|
||||
let list = [], sort = store.get(sortKey) === 'recent' ? 'recent' : 'story', filter = null;
|
||||
const section = h('section', { class: 'card moments-card', hidden: true });
|
||||
const count = h('span', { class: 'muted small' });
|
||||
const sortButtons = ['story', 'recent'].map(key => h('button', { type: 'button', 'data-sort': key,
|
||||
onclick: () => { sort = key; store.set(sortKey, key); draw(); } }, key === 'story' ? 'Story' : 'Recent'));
|
||||
const filters = h('div', { class: 'moment-filters', role: 'group', 'aria-label': 'Filter moments by category' });
|
||||
const strip = h('div', { class: 'moment-strip' });
|
||||
section.append(h('div', { class: 'moments-head' }, h('h2', {}, 'Your moments'), count, h('div', { class: 'grow' }),
|
||||
h('div', { class: 'seg', role: 'group', 'aria-label': 'Sort moments' }, sortButtons)), filters, strip);
|
||||
const shown = () => {
|
||||
const items = list.filter(m => !filter || m.cat === filter);
|
||||
return sort === 'recent' ? items.sort((a, b) => b.t - a.t) : items.sort((a, b) => {
|
||||
for (let k = 0; k < 3; k++) if (a.order[k] !== b.order[k]) return a.order[k] - b.order[k];
|
||||
return a.t - b.t;
|
||||
});
|
||||
};
|
||||
function draw() {
|
||||
section.hidden = !list.length;
|
||||
count.textContent = `${list.length} saved`;
|
||||
for (const b of sortButtons) b.classList.toggle('on', b.dataset.sort === sort);
|
||||
// Category filters only help once there are a few moments in more than one category.
|
||||
const cats = MOMENT_CATS.filter(c => list.some(m => m.cat === c.id));
|
||||
if (filter && !cats.some(c => c.id === filter)) filter = null;
|
||||
filters.hidden = list.length < 5 || cats.length < 2;
|
||||
filters.replaceChildren(...cats.map(c => h('button', { type: 'button', class: 'filter-chip moment-filter' + (filter === c.id ? ' on' : ''),
|
||||
'aria-pressed': String(filter === c.id), onclick: () => { filter = filter === c.id ? null : c.id; draw(); } },
|
||||
momentIcon(c.id, 14), `${c.label} ${list.filter(m => m.cat === c.id).length}`)));
|
||||
const items = shown();
|
||||
strip.replaceChildren(...items.map((m, k) => h('button', { type: 'button', class: 'moment-thumb', title: `${momentCat(m.cat).label} · ${m.chapter.label} · ${m.missing ? 'page moved' : momentPageLabel(m.pages)}${m.note ? '\n' + m.note : ''}`,
|
||||
onclick: () => momentViewer({ mangaId: M.id, mangaTitle: M.title, list: items, index: k, onChange: changed }) },
|
||||
momentCrop(m.pages, m.region, 200, 120, m.layout),
|
||||
h('span', { class: 'moment-thumb-cat' }, momentIcon(m.cat, 14)),
|
||||
h('span', { class: 'moment-thumb-label' }, `${m.chapter.label} · ${m.missing ? 'moved' : momentPageLabel(m.pages)}`))));
|
||||
}
|
||||
// The viewer edits its (sorted, filtered) copy; fold the changes back into the full list.
|
||||
function changed(items) {
|
||||
const byId = new Map(items.map(m => [m.id, m]));
|
||||
const visible = new Set(shown().map(m => m.id));
|
||||
list = list.filter(m => !visible.has(m.id) || byId.has(m.id)).map(m => byId.get(m.id) || m);
|
||||
draw();
|
||||
}
|
||||
api('api/moments?' + qsa({ m: M.id })).then(data => { list = data.moments; draw(); }).catch(() => {});
|
||||
return section;
|
||||
}
|
||||
|
||||
drawVolumeFilter();
|
||||
drawChapters();
|
||||
})();
|
||||
@@ -0,0 +1,302 @@
|
||||
'use strict';
|
||||
/*
|
||||
* Moments (BL-0055): a page, or a region of one or two facing pages, saved with exactly one category.
|
||||
* Shared by the reader (save sheet) and the manga page (gallery and viewer). Crops are drawn by positioning the
|
||||
* full page images inside a clipping box, so no thumbnails are generated on the server.
|
||||
*/
|
||||
const MOMENT_CATS = [
|
||||
{ id: 'twist', label: 'Twist', color: '#a29bf2', svg: '<path d="M12 1.5L13.8 4.6L17.3 3.2L17.4 6.8L21 7.6L19.2 10.8L22 13.2L18.8 14.9L19.6 18.5L16 18.2L14.6 21.6L12 19.2L9.4 21.6L8 18.2L4.4 18.5L5.2 14.9L2 13.2L4.8 10.8L3 7.6L6.6 6.8L6.7 3.2L10.2 4.6Z"/><path d="M9.3 8.3v4.4"/><path d="M12.6 9.2a1.9 1.9 0 1 1 2.6 1.8c-.6.3-.8.7-.8 1.5"/><circle cx="9.3" cy="15.3" r=".5" fill="currentColor"/><circle cx="14.4" cy="15.3" r=".5" fill="currentColor"/>' },
|
||||
{ id: 'hot', label: 'Hot', color: '#f0845c', svg: '<path d="M12 2.5c.5 3 3 4.5 4.5 7a6.5 6.5 0 1 1-11 1.5c.8 1.2 1.8 1.8 2.8 1.8C7.5 9 9.5 5.5 12 2.5z"/><path d="M12 21a3 3 0 0 1-3-3c0-1.8 1.5-3 3-5c1.5 2 3 3.2 3 5a3 3 0 0 1-3 3z"/><path d="M19.5 3l-1 1.6M21.5 5.5l-1.6.8"/>' },
|
||||
{ id: 'awesome', label: 'Awesome', color: '#f2b443', svg: '<path d="M12 5.5L13.5 10.5L18.5 12L13.5 13.5L12 18.5L10.5 13.5L5.5 12L10.5 10.5Z"/><path d="M3 3l2.6 2.6M21 3l-2.6 2.6M3 21l2.6-2.6M21 21l-2.6-2.6M12 1.5v1.8M12 20.7v1.8M1.5 12h1.8M20.7 12h1.8"/>' },
|
||||
{ id: 'cute', label: 'Cute', color: '#f07aa5', svg: '<path d="M11 21.5s-7-4.4-7-9.5a4 4 0 0 1 7-2.6a4 4 0 0 1 7 2.6c0 5.1-7 9.5-7 9.5z"/><path d="M8 13.5l1 1M13 13.5l1 1"/><path d="M19.5 2v4M17.5 4h4"/><path d="M4 3.5v2.4M2.8 4.7h2.4"/>' },
|
||||
{ id: 'funny', label: 'Funny', color: '#4fcfa0', svg: '<circle cx="11" cy="13" r="8"/><path d="M6.8 11.6l1.6-1.6l1.6 1.6M12 11.6l1.6-1.6l1.6 1.6"/><path d="M7.3 14.5h7.4a3.7 3.7 0 0 1-7.4 0z"/><path d="M20.2 2.5c-1 1.6-1.6 2.6-1.6 3.4a1.6 1.6 0 0 0 3.2 0c0-.8-.6-1.8-1.6-3.4z"/>' },
|
||||
{ id: 'feels', label: 'Feels', color: '#6aa8ec', svg: '<path d="M2.5 10c2.5-4 6-5.5 9.5-5.5S19 6 21.5 10"/><path d="M5 11.5a7 7 0 0 0 14 0"/><circle cx="12" cy="11" r="4"/><circle cx="13.5" cy="9.5" r="1" fill="currentColor"/><path d="M18 16.5c-1 1.5-1.5 2.4-1.5 3.1a1.5 1.5 0 0 0 3 0c0-.7-.5-1.6-1.5-3.1z"/>' },
|
||||
];
|
||||
const momentCat = id => MOMENT_CATS.find(c => c.id === id) || MOMENT_CATS[0];
|
||||
function momentIcon(id, size = 18) {
|
||||
const el = icon(`<svg viewBox="0 0 24 24" width="${size}" height="${size}" fill="none" stroke="currentColor" stroke-width="${size < 20 ? 2 : 1.6}" stroke-linecap="round" stroke-linejoin="round" aria-hidden="true">${momentCat(id).svg}</svg>`);
|
||||
el.style.color = momentCat(id).color;
|
||||
return el;
|
||||
}
|
||||
const momentPageLabel = pages => {
|
||||
const ns = pages.map(p => p.i + 1).sort((a, b) => a - b);
|
||||
return 'p. ' + (ns.length > 1 ? ns[0] + '–' + ns[ns.length - 1] : ns[0]);
|
||||
};
|
||||
|
||||
// Displayed size after rotation, as in the reader.
|
||||
const shapeOf = p => {
|
||||
const turned = (p.r || 0) % 180, w = (turned ? p.h : p.w) || 1000, hh = (turned ? p.w : p.h) || 1414;
|
||||
return { ...p, a: w / hh };
|
||||
};
|
||||
const FULL = { x: 0, y: 0, w: 1, h: 1 };
|
||||
|
||||
// Each page's box in units: side by side ("row", Page/Book) pages share a height of 1,
|
||||
// stacked ("column", Flow) pages share a width of 1.
|
||||
function arrange(pages, layout) {
|
||||
const column = layout === 'column';
|
||||
let at = 0;
|
||||
const boxes = pages.map(shapeOf).map((p, k) => {
|
||||
const b = column ? { k, p, x: 0, y: at, w: 1, h: 1 / p.a } : { k, p, x: at, y: 0, w: p.a, h: 1 };
|
||||
at += column ? b.h : b.w;
|
||||
return b;
|
||||
});
|
||||
return { boxes, w: column ? 1 : at, h: column ? at : 1 };
|
||||
}
|
||||
|
||||
// The pages put together at `scale` px per unit.
|
||||
function composite(pages, layout, scale) {
|
||||
const { boxes, w, h: hh } = arrange(pages, layout), el = h('div', { class: 'moment-composite' });
|
||||
for (const b of boxes) {
|
||||
const W = b.w * scale, H = b.h * scale, r = b.p.r || 0;
|
||||
const img = h('img', { src: b.p.u, alt: '', draggable: 'false', decoding: 'async', loading: 'lazy' });
|
||||
if (r) img.style.cssText = `position:absolute;left:50%;top:50%;max-width:none;width:${r % 180 ? H : W}px;height:${r % 180 ? W : H}px;transform:translate(-50%,-50%) rotate(${r}deg)`;
|
||||
el.append(h('div', { class: 'moment-page', style: `left:${b.x * scale}px;top:${b.y * scale}px;width:${W}px;height:${H}px` }, img));
|
||||
}
|
||||
el.style.width = w * scale + 'px';
|
||||
el.style.height = hh * scale + 'px';
|
||||
return el;
|
||||
}
|
||||
|
||||
// A box of at most maxW × maxH showing `region` (or everything) of the pages.
|
||||
function momentCrop(pages, region, maxW, maxH, layout = 'row') {
|
||||
if (!pages.length) return h('div', { class: 'moment-crop missing', style: `width:${Math.round(maxH * 0.7)}px;height:${maxH}px` }, 'Page moved');
|
||||
const r = region || FULL, { w: TW, h: TH } = arrange(pages, layout), aspect = (r.w * TW) / (r.h * TH);
|
||||
const boxW = Math.min(maxW, maxH * aspect), boxH = boxW / aspect, scale = boxW / (r.w * TW);
|
||||
const inner = composite(pages, layout, scale);
|
||||
inner.style.transform = `translate(${-r.x * TW * scale}px, ${-r.y * TH * scale}px)`;
|
||||
return h('div', { class: 'moment-crop', style: `width:${Math.round(boxW)}px;height:${Math.round(boxH)}px` }, inner);
|
||||
}
|
||||
|
||||
// Drag a rectangle over `target` (position: relative). Coordinates are fractions of the target box.
|
||||
function regionPicker(target, initial, onChange) {
|
||||
let region = initial || null, start = null;
|
||||
const box = h('div', { class: 'region-box', hidden: true });
|
||||
const layer = h('div', { class: 'region-layer' }, box);
|
||||
const draw = () => {
|
||||
box.hidden = !region;
|
||||
if (region) Object.assign(box.style, { left: region.x * 100 + '%', top: region.y * 100 + '%', width: region.w * 100 + '%', height: region.h * 100 + '%' });
|
||||
layer.classList.toggle('has-region', !!region);
|
||||
};
|
||||
const at = e => {
|
||||
const rect = layer.getBoundingClientRect();
|
||||
return { x: Math.min(1, Math.max(0, (e.clientX - rect.left) / rect.width)), y: Math.min(1, Math.max(0, (e.clientY - rect.top) / rect.height)) };
|
||||
};
|
||||
layer.addEventListener('pointerdown', e => { e.preventDefault(); layer.setPointerCapture(e.pointerId); start = at(e); });
|
||||
layer.addEventListener('pointermove', e => {
|
||||
if (!start) return;
|
||||
const p = at(e);
|
||||
region = { x: Math.min(start.x, p.x), y: Math.min(start.y, p.y), w: Math.abs(p.x - start.x), h: Math.abs(p.y - start.y) };
|
||||
draw();
|
||||
});
|
||||
const end = () => {
|
||||
if (!start) return;
|
||||
start = null;
|
||||
if (region && (region.w < 0.03 || region.h < 0.03)) region = null; // a tap, not a selection
|
||||
draw();
|
||||
onChange(region);
|
||||
};
|
||||
layer.addEventListener('pointerup', end);
|
||||
layer.addEventListener('pointercancel', end);
|
||||
target.append(layer);
|
||||
draw();
|
||||
return { get: () => region, clear() { region = null; draw(); onChange(null); } };
|
||||
}
|
||||
|
||||
// Pages the region does not touch are dropped and the region is mapped onto the pages that remain,
|
||||
// so a selection inside one page of a spread or a stack saves that page alone.
|
||||
function trimPages(pages, region, layout = 'row') {
|
||||
if (pages.length < 2 || !region) return { pages, region };
|
||||
const column = layout === 'column', { boxes, w: TW, h: TH } = arrange(pages, layout), total = column ? TH : TW;
|
||||
const from = (column ? region.y : region.x) * total, to = from + (column ? region.h : region.w) * total, eps = 0.002 * total;
|
||||
const startOf = b => (column ? b.y : b.x), endOf = b => startOf(b) + (column ? b.h : b.w);
|
||||
const keep = boxes.filter(b => endOf(b) > from + eps && startOf(b) < to - eps);
|
||||
if (!keep.length || keep.length === boxes.length) return { pages, region };
|
||||
const s0 = startOf(keep[0]), len = endOf(keep[keep.length - 1]) - s0;
|
||||
const a = Math.max(0, (from - s0) / len), b = Math.min(1, (to - s0) / len);
|
||||
return { pages: keep.map(x => pages[x.k]), region: column ? { ...region, y: a, h: b - a } : { ...region, x: a, w: b - a } };
|
||||
}
|
||||
|
||||
// `dialog.done()` closes and cleans up at once. The "close" event (Esc) does the same, but it is a queued task
|
||||
// that a backgrounded page may run late, so the app never waits for it.
|
||||
function openDialog(className, onClose) {
|
||||
const dialog = h('dialog', { class: 'moment-dialog ' + className });
|
||||
let finished = false;
|
||||
dialog.done = () => {
|
||||
if (finished) return;
|
||||
finished = true;
|
||||
if (dialog.open) dialog.close();
|
||||
dialog.remove();
|
||||
if (onClose) onClose();
|
||||
};
|
||||
dialog.addEventListener('close', dialog.done);
|
||||
dialog.addEventListener('click', e => { if (e.target === dialog) dialog.done(); }); // click on the backdrop
|
||||
document.body.append(dialog);
|
||||
return dialog;
|
||||
}
|
||||
|
||||
/*
|
||||
* The save / edit sheet. `context` [manga title, chapter text]; `initial` {region, cat, note}.
|
||||
* Pages come either fixed (`pages` in displayed order, for an edit) or from `sequence` (pages in reading order)
|
||||
* with the shown `range` [first, last], which "+ Previous page" / "+ Next page" widen up to four pages.
|
||||
* `layout` "row" puts them side by side (right to left when `rtl`), "column" stacks them as in Flow.
|
||||
* `save({pages, layout, region, cat, note})` returns a promise. Resolves with its result, or null when cancelled.
|
||||
*/
|
||||
function momentSheet({ heading, context, pages: fixed, sequence, range, layout = 'row', rtl = false, initial = {}, save }) {
|
||||
return new Promise(resolve => {
|
||||
let result = null, cat = initial.cat || null, region = initial.region || null, picker = null;
|
||||
let [lo, hi] = range || [0, 0];
|
||||
const start = [lo, hi], MAX = 4;
|
||||
const dialog = openDialog('moment-sheet', () => resolve(result));
|
||||
const pagesNow = () => {
|
||||
if (fixed) return fixed;
|
||||
const run = sequence.slice(lo, hi + 1);
|
||||
return layout === 'row' && rtl ? run.reverse() : run;
|
||||
};
|
||||
const wrap = h('div', { class: 'moment-preview-wrap' });
|
||||
const where = h('div', { class: 'muted small' });
|
||||
const whole = h('button', { class: 'btn small', type: 'button', hidden: true, onclick: () => picker.clear() }, 'Whole selection');
|
||||
const help = h('span', { class: 'muted small' });
|
||||
const earlier = h('button', { class: 'btn small', type: 'button', onclick: () => { lo--; drawPreview(); } }, '+ Previous page');
|
||||
const later = h('button', { class: 'btn small', type: 'button', onclick: () => { hi++; drawPreview(); } }, '+ Next page');
|
||||
const reset = h('button', { class: 'btn small', type: 'button', onclick: () => { [lo, hi] = start; drawPreview(); } }, 'Reset pages');
|
||||
const extend = h('div', { class: 'moment-extend', hidden: !sequence },
|
||||
layout === 'row' && rtl ? [later, reset, earlier] : [earlier, reset, later]);
|
||||
// Changing the pages changes the geometry, so a selection starts over.
|
||||
function drawPreview() {
|
||||
const pages = pagesNow(), { w: TW, h: TH } = arrange(pages, layout);
|
||||
const maxW = Math.min(560, innerWidth - 48);
|
||||
const maxH = Math.min(innerHeight * (layout === 'column' ? 0.62 : innerWidth < 600 ? 0.42 : 0.5), layout === 'column' ? 900 : 520);
|
||||
const preview = h('div', { class: 'moment-preview' }, composite(pages, layout, Math.min(maxW / TW, maxH / TH)));
|
||||
if (picker) region = null;
|
||||
picker = regionPicker(preview, region, r => { region = r; whole.hidden = !r; });
|
||||
whole.hidden = !region;
|
||||
wrap.replaceChildren(preview);
|
||||
where.textContent = `${context[1]} · ${momentPageLabel(pages)}`;
|
||||
help.textContent = 'Drag over the picture to keep just a part; pages it does not touch are left out. Without a selection everything shown is saved.';
|
||||
if (sequence) {
|
||||
const count = hi - lo + 1;
|
||||
earlier.disabled = lo === 0 || count >= MAX;
|
||||
later.disabled = hi === sequence.length - 1 || count >= MAX;
|
||||
reset.hidden = lo === start[0] && hi === start[1];
|
||||
}
|
||||
}
|
||||
const hint = h('span', { class: 'muted small moment-hint' });
|
||||
const saveBtn = h('button', { class: 'btn primary', type: 'submit' }, 'Save moment');
|
||||
const chips = MOMENT_CATS.map(c => {
|
||||
const chip = h('button', { class: 'moment-chip', type: 'button', role: 'radio', 'data-cat': c.id, style: `--cat:${c.color}` }, momentIcon(c.id, 18), c.label);
|
||||
chip.onclick = () => { cat = c.id; drawChips(); };
|
||||
return chip;
|
||||
});
|
||||
const drawChips = () => {
|
||||
for (const chip of chips) chip.setAttribute('aria-checked', String(chip.dataset.cat === cat));
|
||||
saveBtn.disabled = !cat;
|
||||
hint.textContent = cat ? '' : 'Pick a category to save';
|
||||
};
|
||||
const note = h('input', { type: 'text', maxlength: '200', placeholder: 'Optional note', 'aria-label': 'Note', value: initial.note || '' });
|
||||
const error = h('p', { class: 'error small', hidden: true });
|
||||
const form = h('form', { method: 'dialog' },
|
||||
h('div', { class: 'moment-head' }, h('h2', {}, heading),
|
||||
h('button', { class: 'btn icon', type: 'button', 'aria-label': 'Close', onclick: () => dialog.done() }, '×')),
|
||||
h('div', { class: 'moment-context' }, context[0]), where,
|
||||
wrap, extend,
|
||||
h('div', { class: 'moment-region-row' }, help, whole),
|
||||
h('div', { class: 'moment-chips', role: 'radiogroup', 'aria-label': 'Category' }, chips),
|
||||
note, error,
|
||||
h('div', { class: 'moment-actions' }, hint, h('div', { class: 'grow' }),
|
||||
h('button', { class: 'btn', type: 'button', onclick: () => dialog.done() }, 'Cancel'), saveBtn));
|
||||
form.addEventListener('submit', async e => {
|
||||
e.preventDefault();
|
||||
if (!cat) return;
|
||||
saveBtn.disabled = true; error.hidden = true;
|
||||
try {
|
||||
const kept = fixed ? { pages: fixed, region } : trimPages(pagesNow(), region, layout);
|
||||
result = await save({ pages: kept.pages, layout, region: kept.region, cat, note: note.value });
|
||||
dialog.done();
|
||||
} catch (err) { error.textContent = err.message; error.hidden = false; saveBtn.disabled = false; }
|
||||
});
|
||||
dialog.append(form);
|
||||
drawPreview();
|
||||
drawChips();
|
||||
dialog.showModal();
|
||||
});
|
||||
}
|
||||
|
||||
const momentReadUrl = (mangaId, m) => 'read.html?' + qsa({ m: mangaId, c: m.c, v: m.v, p: m.pages.length ? Math.min(...m.pages.map(p => p.i)) : null });
|
||||
|
||||
/*
|
||||
* The large viewer: `list` of moment views, starting at `index`. `onChange(list)` after an edit or a delete.
|
||||
*/
|
||||
function momentViewer({ mangaId, mangaTitle, list, index, onChange }) {
|
||||
let k = index, full = false;
|
||||
const dialog = openDialog('moment-viewer', () => removeEventListener('resize', draw));
|
||||
const stage = h('div', { class: 'moment-viewer-stage' });
|
||||
const info = h('div', { class: 'moment-viewer-info' });
|
||||
const prevBtn = h('button', { class: 'btn icon moment-nav prev', type: 'button', 'aria-label': 'Previous moment', onclick: () => step(-1) }, '‹');
|
||||
const nextBtn = h('button', { class: 'btn icon moment-nav next', type: 'button', 'aria-label': 'Next moment', onclick: () => step(1) }, '›');
|
||||
const counter = h('span', { class: 'muted small' });
|
||||
const fullBtn = h('button', { class: 'btn small', type: 'button', onclick: () => { full = !full; draw(); } });
|
||||
const readLink = h('a', { class: 'btn primary small' }, 'Open in reader');
|
||||
const actions = h('div', { class: 'moment-actions' }, fullBtn,
|
||||
h('button', { class: 'btn small', type: 'button', onclick: edit }, 'Edit'),
|
||||
h('button', { class: 'btn small danger', type: 'button', onclick: remove }, 'Delete'),
|
||||
h('div', { class: 'grow' }), readLink);
|
||||
dialog.append(
|
||||
h('div', { class: 'moment-head' }, info, counter, h('button', { class: 'btn icon', type: 'button', 'aria-label': 'Close', onclick: () => dialog.done() }, '×')),
|
||||
h('div', { class: 'moment-viewer-body' }, prevBtn, stage, nextBtn), actions);
|
||||
|
||||
function step(d) {
|
||||
if (!list.length) return;
|
||||
k = (k + d + list.length) % list.length; full = false; draw();
|
||||
}
|
||||
function draw() {
|
||||
if (!list.length) return dialog.done();
|
||||
const m = list[k], c = momentCat(m.cat);
|
||||
const maxW = Math.min(1100, innerWidth - (innerWidth < 600 ? 32 : 140)), maxH = innerHeight - (innerWidth < 600 ? 230 : 200);
|
||||
stage.replaceChildren(momentCrop(m.pages, full ? null : m.region, maxW, maxH, m.layout));
|
||||
info.replaceChildren(h('span', { class: 'moment-tag', style: `--cat:${c.color}` }, momentIcon(m.cat, 16), c.label),
|
||||
h('span', {}, `${m.chapter.label}${m.chapter.title ? ' · ' + m.chapter.title : ''} · ${m.missing ? 'page moved' : momentPageLabel(m.pages)}`),
|
||||
h('span', { class: 'muted small' }, ago(m.t)),
|
||||
...(m.note ? [h('span', { class: 'moment-note' }, m.note)] : []));
|
||||
counter.textContent = `${k + 1} / ${list.length}`;
|
||||
prevBtn.hidden = nextBtn.hidden = list.length < 2;
|
||||
fullBtn.hidden = !m.region || m.missing;
|
||||
fullBtn.textContent = full ? 'Show selection' : (m.pages.length > 1 ? 'Show all pages' : 'Show full page');
|
||||
readLink.href = momentReadUrl(mangaId, m); // a moved page still opens its chapter
|
||||
}
|
||||
async function edit() {
|
||||
const m = list[k];
|
||||
if (m.missing) return;
|
||||
const saved = await momentSheet({ heading: 'Edit moment', context: [mangaTitle, `${m.chapter.label}${m.chapter.title ? ' · ' + m.chapter.title : ''}`],
|
||||
pages: m.pages, layout: m.layout, initial: m,
|
||||
save: async ({ region, cat, note }) => (await api('api/moments/update', { id: m.id, region, cat, note })).moment });
|
||||
if (!saved) return;
|
||||
list[k] = saved; draw(); onChange(list);
|
||||
}
|
||||
async function remove() {
|
||||
if (!confirm('Delete this moment?')) return;
|
||||
await api('api/moments/delete', { id: list[k].id });
|
||||
list.splice(k, 1);
|
||||
k = Math.min(k, list.length - 1);
|
||||
onChange(list);
|
||||
draw();
|
||||
}
|
||||
|
||||
dialog.addEventListener('keydown', e => {
|
||||
if (e.target.closest('input, textarea') || dialog.querySelector('dialog[open]')) return;
|
||||
const action = keyAction(MOMENT_VIEWER_KEYS, e.key);
|
||||
if (action === 'step') { e.preventDefault(); step(e.key === 'ArrowRight' ? 1 : -1); }
|
||||
});
|
||||
let swipe = null;
|
||||
stage.addEventListener('pointerdown', e => { swipe = { x: e.clientX, y: e.clientY, t: Date.now() }; });
|
||||
stage.addEventListener('pointerup', e => {
|
||||
const s = swipe; swipe = null;
|
||||
if (!s || Date.now() - s.t > 800) return;
|
||||
const dx = e.clientX - s.x, dy = e.clientY - s.y;
|
||||
if (Math.abs(dx) >= 50 && Math.abs(dx) > 2 * Math.abs(dy)) step(dx < 0 ? 1 : -1);
|
||||
});
|
||||
addEventListener('resize', draw);
|
||||
draw();
|
||||
dialog.showModal();
|
||||
}
|
||||
|
||||
if (typeof module !== 'undefined') module.exports = { MOMENT_CATS, trimPages, arrange, momentPageLabel };
|
||||
@@ -14,6 +14,8 @@
|
||||
<script src="common.js" defer></script>
|
||||
<script src="reading-clock-core.js" defer></script>
|
||||
<script src="reading-clock.js" defer></script>
|
||||
<script src="keys.js" defer></script>
|
||||
<script src="moments.js" defer></script>
|
||||
<script src="reader.js" defer></script>
|
||||
</head>
|
||||
<body class="reader">
|
||||
@@ -23,6 +25,7 @@
|
||||
<a id="back" class="btn icon" href="./" title="Back to the manga" aria-label="Back">‹</a>
|
||||
<div id="title" class="grow"></div>
|
||||
<select id="chsel" title="Jump to a chapter" aria-label="Chapter"></select>
|
||||
<button id="moment" class="btn icon moment-btn" title="Save a moment of this page (M)" aria-label="Save a moment"><svg viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M5 3.5h10v17l-5-3.6l-5 3.6z"/><path d="M19.5 2.5v5M17 5h5"/></svg></button>
|
||||
<button id="fs" class="btn icon" title="Fullscreen" aria-label="Fullscreen"><svg viewBox="0 0 24 24" width="18" height="18" fill="none" stroke="currentColor" stroke-width="2"><path d="M4 9V4h5M20 9V4h-5M4 15v5h5M20 15v5h-5"/></svg></button>
|
||||
</div>
|
||||
|
||||
|
||||
+68
-11
@@ -149,6 +149,8 @@
|
||||
const k = list.findIndex(x => x.i >= p);
|
||||
cur = k < 0 ? list.length - 1 : k;
|
||||
}
|
||||
// A link to a page (e.g. "Open in reader" from a moment) wins over the saved position.
|
||||
if (P.get('p') != null) { const k = list.findIndex(x => x.i >= +P.get('p')); if (k >= 0) cur = k; }
|
||||
|
||||
// Book mode: pages pair up inside each run of single pages. Runs restart after every double page / cover,
|
||||
// so one missing pair never shifts the rest of the chapter. "Shift" makes a run start with a lone page.
|
||||
@@ -480,18 +482,69 @@
|
||||
else toggleBars();
|
||||
});
|
||||
|
||||
// Keys are looked up in READER_KEYS (keys.js), the same table Settings and "?" show.
|
||||
const keyActions = {
|
||||
turn: e => { e.preventDefault(); (e.key === (rtl ? 'ArrowLeft' : 'ArrowRight') ? next : prev)(); },
|
||||
forward: e => { if (mode === 'flow') return; e.preventDefault(); (e.shiftKey ? prev : next)(); },
|
||||
back: e => { if (mode === 'flow') return; e.preventDefault(); prev(); },
|
||||
zoomIn: () => stepZoom(1),
|
||||
zoomOut: () => stepZoom(-1),
|
||||
zoomFit: () => setZoom(1),
|
||||
moment: e => { e.preventDefault(); saveMoment(); },
|
||||
bars: () => toggleBars(),
|
||||
fullscreen: () => toggleFullscreen(),
|
||||
help: () => showKeys(),
|
||||
};
|
||||
addEventListener('keydown', e => {
|
||||
if ((e.target.closest && e.target.closest('input, select, textarea')) || e.ctrlKey || e.metaKey || e.altKey) return;
|
||||
const k = e.key;
|
||||
if (k === 'ArrowLeft' || k === 'ArrowRight') { e.preventDefault(); (k === (rtl ? 'ArrowLeft' : 'ArrowRight') ? next : prev)(); }
|
||||
else if ((k === ' ' || k === 'PageDown') && mode !== 'flow') { e.preventDefault(); (e.shiftKey ? prev : next)(); }
|
||||
else if (k === 'PageUp' && mode !== 'flow') { e.preventDefault(); prev(); }
|
||||
else if (k === '+' || k === '=') stepZoom(1);
|
||||
else if (k === '-') stepZoom(-1);
|
||||
else if (k === '0') setZoom(1);
|
||||
else if (k === 'm' || k === 'Escape') toggleBars();
|
||||
else if (k === 'f') toggleFullscreen();
|
||||
if (document.querySelector('dialog[open]')) return; // an open dialog handles its own keys (Esc closes it)
|
||||
const action = keyAction(READER_KEYS, e.key);
|
||||
if (action) keyActions[action](e);
|
||||
});
|
||||
function showKeys() {
|
||||
const dialog = openDialog('key-dialog', () => stage.focus({ preventScroll: true }));
|
||||
dialog.append(
|
||||
h('div', { class: 'moment-head' }, h('h2', {}, 'Keyboard shortcuts'),
|
||||
h('button', { class: 'btn icon', type: 'button', 'aria-label': 'Close', onclick: () => dialog.done() }, '×')),
|
||||
keyTable([{ title: 'Reader', keys: READER_KEYS }]),
|
||||
h('p', { class: 'muted small' }, 'Also listed in Settings → Keyboard shortcuts.'));
|
||||
dialog.showModal();
|
||||
}
|
||||
|
||||
// ---------- moments ----------
|
||||
const momentPages = new Set((D.moments || []).flatMap(x => x.pages));
|
||||
const momentPage = p => ({ i: p.i, u: p.u, w: D.pages[p.i].w, h: D.pages[p.i].h, r: p.r });
|
||||
// Positions in `list` of the page(s) on screen, in reading order. In Flow the page at the reading line.
|
||||
function currentRange() {
|
||||
if (atEnd && mode !== 'flow') return null;
|
||||
const ks = (mode === 'book' ? shown : [cur]).filter(k => list[k]);
|
||||
return ks.length ? [Math.min(...ks), Math.max(...ks)] : null;
|
||||
}
|
||||
function currentPages() {
|
||||
const range = currentRange();
|
||||
return range ? list.slice(range[0], range[1] + 1).map(momentPage) : [];
|
||||
}
|
||||
let momentOpen = false;
|
||||
async function saveMoment() {
|
||||
const range = currentRange();
|
||||
if (!range) return toast('No page here');
|
||||
if (momentOpen) return;
|
||||
momentOpen = true;
|
||||
const chapter = D.chapters.find(c => c.id === D.chapter.id), book = D.books.length > 1 && chapter && D.books.find(b => b.key === chapter.book);
|
||||
// A picture can run over several pages: the sheet can add neighbours, stacked in Flow, side by side in Page/Book.
|
||||
const saved = await momentSheet({
|
||||
heading: 'Save moment', sequence: list.map(momentPage), range, layout: mode === 'flow' ? 'column' : 'row', rtl,
|
||||
context: [D.manga.title, [book && book.label, D.chapter.label + (D.chapter.title ? ' · ' + D.chapter.title : '')].filter(Boolean).join(' · ')],
|
||||
save: async v => (await api('api/moments', { m: D.manga.id, c: D.chapter.id, v: D.version, pages: v.pages.map(p => p.i), layout: v.layout,
|
||||
region: v.region, cat: v.cat, note: v.note })).moment,
|
||||
});
|
||||
momentOpen = false;
|
||||
stage.focus({ preventScroll: true });
|
||||
if (!saved) return;
|
||||
for (const p of saved.pages) momentPages.add(p.i);
|
||||
updateUI();
|
||||
toast(`Saved · ${momentCat(saved.cat).label} moment`, 1500);
|
||||
}
|
||||
|
||||
let scrollRaf = 0;
|
||||
stage.addEventListener('scroll', () => {
|
||||
@@ -598,8 +651,8 @@
|
||||
const key = mode === 'flow' ? 'hint.flow' : 'hint.paged';
|
||||
if (store.get(key)) return;
|
||||
store.set(key, '1');
|
||||
toast(mode === 'flow' ? 'Controls hidden · tap the middle of the screen or beside the pages to show them'
|
||||
: 'Controls hidden · tap the middle of the page or beside it to show them', 3500);
|
||||
toast(mode === 'flow' ? 'Controls hidden · tap the middle of the screen or beside the pages (or press I) to show them'
|
||||
: 'Controls hidden · tap the middle of the page or beside it (or press I) to show them', 3500);
|
||||
}
|
||||
|
||||
let toastTimer = 0;
|
||||
@@ -625,6 +678,9 @@
|
||||
sh.hidden = mode !== 'book';
|
||||
sh.classList.toggle('on', shown.length > 0 && !solo(list[shown[0]]) && shift.has(list[runStart(shown[0])].i));
|
||||
$('#zoom').textContent = Math.round(zoom() * 100) + '%';
|
||||
const hasMoment = currentPages().some(p => momentPages.has(p.i));
|
||||
$('#moment').classList.toggle('on', hasMoment);
|
||||
$('#moment').title = (hasMoment ? 'This page has a moment · save another' : 'Save a moment of this page') + ' (M)';
|
||||
}
|
||||
|
||||
let saveTimer = 0;
|
||||
@@ -669,6 +725,7 @@
|
||||
ver.onchange = () => { saveProgress(true); location.href = readUrl(D.manga.id, D.chapter.id, ver.value); };
|
||||
for (const b of modeBtns) b.onclick = () => { if (b.dataset.mode !== mode) setMode(b.dataset.mode, true); };
|
||||
toneButton.onclick = cycleTone;
|
||||
$('#moment').onclick = saveMoment;
|
||||
$('#shift').onclick = toggleShift;
|
||||
$('#zoomout').onclick = () => stepZoom(-1);
|
||||
$('#zoom').onclick = () => setZoom(1);
|
||||
|
||||
@@ -5,6 +5,7 @@ const SETTINGS_SECTIONS = [
|
||||
{ id: 'user/reading', group: 'User', title: 'Reading', hint: 'Reader modes, page spacing and zoom.' },
|
||||
{ id: 'user/reading-hours', group: 'User', title: 'Reading hours', hint: 'When this server allows chapter reading.' },
|
||||
{ id: 'user/reading-time', group: 'User', title: 'Reading time & privacy', hint: 'Your clock, Reading Story and personal data.' },
|
||||
{ id: 'user/shortcuts', group: 'User', title: 'Keyboard shortcuts', hint: 'Every key the reader understands. Press ? in the reader for the same list.' },
|
||||
{ id: 'user/notifications', group: 'User', title: 'Notifications', hint: 'Chapter alerts and your optional webhook.' },
|
||||
{ id: 'user/account', group: 'User', title: 'Account', hint: 'Your account and password.' },
|
||||
{ id: 'admin/scraper', group: 'Admin', title: 'Scraper', hint: 'Live activity, downloads, update checks and sources.' },
|
||||
|
||||
@@ -9,6 +9,7 @@
|
||||
<link rel="manifest" href="manifest.webmanifest">
|
||||
<meta name="theme-color" content="#121214">
|
||||
<script src="common.js" defer></script>
|
||||
<script src="keys.js" defer></script>
|
||||
<script src="settings-nav.js" defer></script>
|
||||
<script src="settings.js" defer></script>
|
||||
</head>
|
||||
|
||||
+30
-2
@@ -38,6 +38,11 @@
|
||||
h('option', { value: 'dim' }, 'Dim (70% brightness)'),
|
||||
h('option', { value: 'extra-dim' }, 'Extra dim (45% brightness)'));
|
||||
tone.value = S.pageTone;
|
||||
const readDate = h('select', { onchange: e => save({ readDate: e.target.value }) },
|
||||
h('option', { value: 'hour' }, 'Date and hour (e.g. 23 Oct, 21h)'),
|
||||
h('option', { value: 'date' }, 'Date only'),
|
||||
h('option', { value: 'off' }, 'Do not show'));
|
||||
readDate.value = S.readDate;
|
||||
const LABEL = { flow: 'Flow', page: 'Page', book: 'Book', strip: 'Webtoon flow' };
|
||||
const zooms = () => Object.keys(LABEL).map(k => `${LABEL[k]} ${Math.round((S.zoom[k] || 1) * 100)}%`).join(' · ');
|
||||
const zoomText = h('span', { class: 'muted' }, zooms());
|
||||
@@ -85,6 +90,18 @@
|
||||
h('button', { class: 'btn small', onclick: async () => { await save({ zoom: {} }); zoomText.textContent = zooms(); } }, 'Reset to fit'),
|
||||
h('div', { class: 'muted small' }, '100% = the page fits the screen. Change it in the reader with Ctrl + mouse wheel, a pinch, or the + / - keys.'))));
|
||||
|
||||
section('user/reading').append(h('section', { class: 'card' },
|
||||
h('h2', {}, 'Library page'),
|
||||
check('continueRow', 'Show "Continue where you left off" at the top',
|
||||
'One row with the five manga you read most recently and have not finished. It hides itself while you search, filter by status, or sort by "Recently read".')));
|
||||
|
||||
section('user/reading').append(h('section', { class: 'card' },
|
||||
h('h2', {}, 'Chapter list dates'),
|
||||
h('label', { class: 'field' }, h('span', {}, 'When I read a chapter'), readDate,
|
||||
h('div', { class: 'muted small' }, 'Shown next to ✓ in the chapter list: when you first finished the chapter, in your local time. Marking it unread clears it.')),
|
||||
check('releaseDate', 'Show when each chapter was released',
|
||||
'Best known date first: the official release, else the upload at the source (≈), else when it first reached Manga Circus (◎), else when it was added here (↓). Hover a date for all known dates.')));
|
||||
|
||||
const readingHoursBody = h('div', {}, h('p', { class: 'muted' }, 'Loading server reading hours…'));
|
||||
section('user/reading-hours').append(h('section', { class: 'card' },
|
||||
h('h2', {}, 'Server reading hours'), readingHoursBody,
|
||||
@@ -123,6 +140,7 @@
|
||||
const oldPw = h('input', { type: 'password', autocomplete: 'current-password', placeholder: 'Current password' });
|
||||
const newPw = h('input', { type: 'password', autocomplete: 'new-password', placeholder: 'New password' });
|
||||
const pwMsg = h('span', { class: 'small' });
|
||||
section('user/shortcuts').append(h('section', { class: 'card' }, keyTable()));
|
||||
section('user/account').append(h('section', { class: 'card' },
|
||||
h('h2', {}, 'Account · ' + me.name),
|
||||
h('div', { class: 'row' }, oldPw, newPw, h('button', { class: 'btn', onclick: async () => {
|
||||
@@ -194,7 +212,8 @@
|
||||
return;
|
||||
}
|
||||
const activeEntries = state.active?.length ? state.active : state.current ? [{ source: state.current.source, current: state.current, job: state.job }] : [];
|
||||
const url = h('input', { type: 'url', placeholder: 'WeebCentral or MangaDex title URL', class: 'grow' });
|
||||
const usable = (state.adapters || []).filter(adapter => state.settings?.sourcesEnabled?.[adapter.id] !== false).map(adapter => adapter.label);
|
||||
const url = h('input', { type: 'url', placeholder: `${usable.join(', ') || 'Source'} title URL`, class: 'grow' });
|
||||
const title = h('input', { placeholder: 'Title override (optional)' });
|
||||
const scope = h('select', {},
|
||||
h('option', { value: 'missing' }, 'All missing chapters'),
|
||||
@@ -217,8 +236,16 @@
|
||||
const max = h('input', { type: 'number', min: '.5', step: '.5', value: (sourceRate.maxMs ?? 4000) / 1000, title: 'Maximum seconds between image and API requests' });
|
||||
const chapterMin = h('input', { type: 'number', min: '.5', step: '1', value: (sourceRate.chapterMinMs ?? 5000) / 1000, title: 'Minimum seconds between chapters' });
|
||||
const chapterMax = h('input', { type: 'number', min: '.5', step: '1', value: (sourceRate.chapterMaxMs ?? 15000) / 1000, title: 'Maximum seconds between chapters' });
|
||||
// Sources with a switch (sourcesEnabled) start switched off on new installations.
|
||||
const switchable = Object.hasOwn(rate.sourcesEnabled || {}, adapter.id);
|
||||
const enabled = switchable ? h('input', { type: 'checkbox', checked: rate.sourcesEnabled[adapter.id] === true,
|
||||
onchange: e => command({ command: 'settings', settings: { sourcesEnabled: { [adapter.id]: e.target.checked } } }) }) : null;
|
||||
return h('div', { class: 'source-rate' },
|
||||
h('b', {}, adapter.label),
|
||||
switchable ? h('label', { class: 'check' }, enabled, h('span', {}, `Use ${adapter.label}`,
|
||||
h('div', { class: 'muted small' }, adapter.id === 'mangadex'
|
||||
? 'Official API; titles removed at a publisher\'s request are not offered.'
|
||||
: 'Unofficial site that mostly hosts unlicensed copies. Switch it on only for what you may copy. While it is off, its queued jobs wait and update checks skip it.'))) : null,
|
||||
h('div', { class: 'row rate-row' },
|
||||
h('label', {}, h('small', {}, 'Requests min (s)'), min), h('label', {}, h('small', {}, 'Requests max (s)'), max),
|
||||
h('label', {}, h('small', {}, 'Chapters min (s)'), chapterMin), h('label', {}, h('small', {}, 'Chapters max (s)'), chapterMax),
|
||||
@@ -236,7 +263,8 @@
|
||||
const completed = visible(state.queue.filter(item => item.status === 'complete' && item.kind === 'download'));
|
||||
const when = value => value && !Number.isNaN(Date.parse(value)) ? new Date(value).toLocaleString() : 'Never';
|
||||
const chips = h('div', { class: 'row scraper-filters', role: 'group', 'aria-label': 'Filter by source' },
|
||||
[{ id: '', label: 'All sources' }, ...(state.adapters || []).map(adapter => ({ id: adapter.id, label: adapter.label }))].map(option =>
|
||||
[{ id: '', label: 'All sources' }, ...(state.adapters || []).map(adapter => ({ id: adapter.id,
|
||||
label: adapter.label + (state.settings?.sourcesEnabled?.[adapter.id] === false ? ' (off)' : '') }))].map(option =>
|
||||
h('button', { class: 'btn small', type: 'button', 'aria-pressed': sourceFilter === option.id ? 'true' : 'false',
|
||||
onclick: () => { sourceFilter = option.id; draw(lastState); } }, option.label)));
|
||||
card.append(chips);
|
||||
|
||||
+12
-1
@@ -115,6 +115,17 @@ function chapterInsight(record) {
|
||||
};
|
||||
}
|
||||
|
||||
// Earliest completion time the clock still remembers (only the last few visits are kept).
|
||||
// approx: older completed passes were already dropped, so the first read may be earlier.
|
||||
function earliestCompletion(record) {
|
||||
if (!record || typeof record !== 'object') return null;
|
||||
const passes = [record.currentPass, ...(record.recentSessions || []).map(item => item && item.pass)];
|
||||
const times = passes.map(pass => pass && whole(pass.completedAt)).filter(time => time > 0);
|
||||
if (!times.length) return null;
|
||||
const counted = new Set(times).size;
|
||||
return { at: Math.min(...times), approx: whole(record.completedPasses) > counted };
|
||||
}
|
||||
|
||||
function median(values) {
|
||||
const sorted = values.filter(value => Number.isFinite(value) && value > 0).sort((a, b) => a - b);
|
||||
if (!sorted.length) return null;
|
||||
@@ -295,6 +306,6 @@ function applyPulse(record, pulse, { version, pageCount, eligible, now = Date.no
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
INPUT_TYPES, applyPulse, applyVisit, chapterInsight, compactRanges, eligiblePages, emptyRecord,
|
||||
INPUT_TYPES, applyPulse, applyVisit, chapterInsight, compactRanges, earliestCompletion, eligiblePages, emptyRecord,
|
||||
libraryInsight, mangaInsight, publicSummary, rangeSet,
|
||||
};
|
||||
+4
-6
@@ -4,9 +4,8 @@ The scraper uses small host adapters to create a common manifest, followed by on
|
||||
|
||||
Start with [AGENT_RUNBOOK.md](AGENT_RUNBOOK.md). It contains the host-specific procedures, safety limits, and stop conditions. `manifest.example.json` documents the interchange format.
|
||||
|
||||
The historical human-readable wishlist is [WISHLIST.md](WISHLIST.md). Once the
|
||||
managed daemon has started, `data/scraper/queue.json` is the authoritative queue and
|
||||
is edited through the admin UI.
|
||||
Once the managed daemon has started, `data/scraper/queue.json` is the authoritative
|
||||
queue and is edited through the admin UI.
|
||||
|
||||
## Managed service
|
||||
|
||||
@@ -32,9 +31,8 @@ Queue schema version 3 supports scoped jobs, multiple jobs for the same upstream
|
||||
series, one sequential worker per source, incremental chapter publication, library
|
||||
duplicate/version checks, and configurable nightly update discovery. See
|
||||
[`docs/scraper-service.md`](../docs/scraper-service.md) for the service contract and
|
||||
[`docs/local-deployment.md`](../docs/local-deployment.md) for the currently running
|
||||
Windows Scheduled Task deployment. On that machine the scheduled task owns the single
|
||||
supervisor; do not run `npm run start:all` beside it.
|
||||
[`docs/deployment.md`](../docs/deployment.md) for running it in Docker. Run only one
|
||||
supervisor or container against the same library and data.
|
||||
|
||||
```bash
|
||||
npm run test:scraper
|
||||
|
||||
@@ -3,6 +3,7 @@
|
||||
const API = 'https://api.mangadex.org';
|
||||
const { parse } = require('../plugins/registry');
|
||||
const { localizedTitle, chapterNumber, selectChapterReleases, availabilitySummary, pageUrls } = require('../plugins/mangadex');
|
||||
const { sourceDate } = require('../../chapter-dates');
|
||||
|
||||
function mangaDexInfo(value) {
|
||||
let url;
|
||||
@@ -71,8 +72,9 @@ async function discoverMangaDexChapter(ref, http) {
|
||||
const atHome = await jsonGet(http, `${API}/at-home/server/${ref.id}?forcePort443=true`);
|
||||
const groupName = ref.group?.name ? ` · ${ref.group.name}` : '';
|
||||
const parsed = parse('mangadex', 'atHome', { data: { response: atHome, chapterUrl: ref.url, expectedPages: ref.pages }, url: `${API}/at-home/server/${ref.id}` });
|
||||
const source = sourceDate(ref.uploadedAt, `MangaDex${groupName}`, ref.uploadBatch);
|
||||
return {
|
||||
number: ref.number, title: ref.title || '', added: ref.added,
|
||||
number: ref.number, title: ref.title || '', ...(source ? { dates: { source } } : {}),
|
||||
...(ref.special ? { special: true, label: ref.label, sourceId: ref.id, sourceNumber: ref.sourceNumber ?? null } : {}),
|
||||
source: { name: `MangaDex${groupName}`, url: ref.url },
|
||||
pages: parsed.pages,
|
||||
|
||||
@@ -4,6 +4,7 @@ const { SlowHttp } = require('../lib/http');
|
||||
const { fallbackDirection } = require('../../reading-direction');
|
||||
const parser = require('../plugins/weebcentral');
|
||||
const { parse } = require('../plugins/registry');
|
||||
const { sourceDate } = require('../../chapter-dates');
|
||||
|
||||
function weebCentralInfo(value) {
|
||||
let url;
|
||||
@@ -36,7 +37,8 @@ async function discoverWeebCatalog(seriesUrl, http) {
|
||||
|
||||
async function discoverWeebChapter(chapter, http) {
|
||||
const pageHtml = await (await http.get(chapter.url, { accept: 'text/html' })).text();
|
||||
const { manifestUrl, added } = parse('weebcentral', 'chapter', { html: pageHtml, url: chapter.url });
|
||||
const { manifestUrl } = parse('weebcentral', 'chapter', { html: pageHtml, url: chapter.url });
|
||||
const source = sourceDate(chapter.uploadedAt, 'WeebCentral', chapter.uploadBatch);
|
||||
const fragment = await (await http.get(`${manifestUrl}&reading_style=long_strip¤t_page=1`, { accept: 'text/html' })).text();
|
||||
const parsed = parse('weebcentral', 'pages', { html: fragment, url: manifestUrl });
|
||||
const pages = parsed.pages.map(url => ({ url, headers: { referer: chapter.url } }));
|
||||
@@ -46,7 +48,7 @@ async function discoverWeebChapter(chapter, http) {
|
||||
...(chapter.special ? { special: true, label: chapter.label, sourceId: chapter.id,
|
||||
sourceNumber: chapter.volume, releaseKind: chapter.releaseKind, volume: chapter.volume } : {}),
|
||||
...(chapter.book ? { book: chapter.book } : {}),
|
||||
added,
|
||||
...(source ? { dates: { source } } : {}),
|
||||
source: { name: 'WeebCentral', url: chapter.url }, pages,
|
||||
};
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ const { directionFields } = require('../../reading-direction');
|
||||
const { mergeImportedMetadata } = require('./metadata-merge');
|
||||
const { normalizeWorkIdentity } = require('./work-identity');
|
||||
const { parseKey } = require('../../chapter-identity');
|
||||
const { publishedDates } = require('../../chapter-dates');
|
||||
|
||||
const token = () => crypto.randomBytes(5).toString('hex');
|
||||
// Staging is a recovery asset, not disposable garbage. Legacy stages without an
|
||||
@@ -180,7 +181,8 @@ async function stageChapter(bookDir, chapter, http, onPage, priorMeta = {}) {
|
||||
await fsp.writeFile(path.join(stage, 'chapter.json'), JSON.stringify({
|
||||
...priorMeta,
|
||||
title: chapter.title || '', source: chapter.source,
|
||||
added: chapter.added || new Date().toISOString(),
|
||||
// "added" stays the first arrival in this library; x-dates keeps source/release/ecosystem.
|
||||
...publishedDates(priorMeta, { ...(chapter.dates || {}), added: chapter.added }),
|
||||
priority: Number(chapter.priority || 0), extra: !!chapter.extra,
|
||||
...(chapter.special ? { special: true, label: chapter.label || 'Special', sourceId: chapter.sourceId,
|
||||
sourceNumber: chapter.sourceNumber ?? null,
|
||||
|
||||
@@ -0,0 +1,180 @@
|
||||
'use strict';
|
||||
// Official chapter release dates (BL-0048) from MangaUpdates publisher releases.
|
||||
// MangaUpdates also lists English digital re-releases of old chapters as "official"
|
||||
// (Naruto ch. 700 shows 2020; it came out in Japan in 2014). A date is accepted only when
|
||||
// 1. it lies inside the work's original run (AniList start - 7 days ... end + 30 days), and
|
||||
// 2. nothing shows the chapter earlier: no scanlation release in MangaUpdates and no
|
||||
// source upload (bulk or not) more than EARLY_TOLERANCE_DAYS before it, and
|
||||
// 3. it fits the release order (see officialDates).
|
||||
// Simulpub releases pass all three; back-catalog re-releases fail at least one.
|
||||
const fs = require('node:fs').promises;
|
||||
const path = require('node:path');
|
||||
const crypto = require('node:crypto');
|
||||
const { parseKey } = require('../../chapter-identity');
|
||||
const { writeJsonAtomic } = require('../lib/json-file');
|
||||
|
||||
const DAY = 864e5;
|
||||
const EARLY_TOLERANCE_DAYS = 4; // early scans and source uploads of weekly series arrive 1-3 days before the official date
|
||||
const ORDER_TOLERANCE_DAYS = 2; // double releases and day-granular dates may put a lower chapter a day or two later
|
||||
const OFFICIAL = new Set(['viz', 'viz media', 'manga plus', 'mangaplus', 'shueisha', 'kodansha', 'kodansha usa', 'k manga',
|
||||
'yen press', 'seven seas', 'square enix', 'square enix manga', 'manga up!', 'comikey', 'azuki', 'tappytoon', 'webtoon',
|
||||
'line webtoon', 'webtoons', 'tapas', 'pocket comics', 'manta', 'lezhin', 'lezhin comics', 'j-novel club', 'dark horse',
|
||||
'vertical', 'denpa', 'tokyopop', 'ize press', 'one peace books', 'inky pen', 'kakaopage', 'kakao webtoon', 'naver webtoon']);
|
||||
const UA = 'MangaCircus/1.0 (self-hosted manga reader; chapter release dates)';
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
const norm = value => String(value || '').toLowerCase().normalize('NFKD').replace(/[^a-z0-9]+/g, '');
|
||||
// "001", "1.50" and "1" are the same chapter; ranges, volumes and letters are not single chapters.
|
||||
const chapterKey = value => (/^\s*\d+(?:\.\d+)?\s*$/.test(String(value ?? '')) ? String(Number(value)) : null);
|
||||
const isOfficial = groups => groups.some(name => OFFICIAL.has(String(name).toLowerCase().trim()));
|
||||
|
||||
async function cached(cacheDir, key, maxAgeMs, load) {
|
||||
const file = cacheDir && path.join(cacheDir, crypto.createHash('sha256').update(key).digest('hex').slice(0, 32) + '.json');
|
||||
if (file) {
|
||||
try {
|
||||
const stat = await fs.stat(file);
|
||||
if (Date.now() - stat.mtimeMs < maxAgeMs) return JSON.parse(await fs.readFile(file, 'utf8'));
|
||||
} catch { /* not cached */ }
|
||||
}
|
||||
const value = await load();
|
||||
if (file) { await fs.mkdir(cacheDir, { recursive: true }); await writeJsonAtomic(file, value); }
|
||||
return value;
|
||||
}
|
||||
|
||||
class ReleaseDateSources {
|
||||
constructor(options = {}) {
|
||||
this.cacheDir = options.cacheDir || null;
|
||||
this.minGapMs = options.minGapMs ?? 1100;
|
||||
this.fetch = options.fetch || globalThis.fetch;
|
||||
this.last = 0;
|
||||
}
|
||||
async request(url, body) {
|
||||
const wait = this.last + this.minGapMs - Date.now();
|
||||
if (wait > 0) await sleep(wait);
|
||||
this.last = Date.now();
|
||||
const response = await this.fetch(url, { method: 'POST', headers: { 'content-type': 'application/json', 'user-agent': UA },
|
||||
body: JSON.stringify(body), signal: AbortSignal.timeout(30000) });
|
||||
if (!response.ok) throw new Error(`${new URL(url).hostname} HTTP ${response.status}`);
|
||||
return response.json();
|
||||
}
|
||||
// MangaUpdates series whose matched name equals one of the manga's titles and whose year fits.
|
||||
async findSeries(meta) {
|
||||
const titles = [meta.title, ...(Array.isArray(meta.altTitles) ? meta.altTitles : [])].map(norm).filter(Boolean);
|
||||
const year = Number(meta.year) || null;
|
||||
const result = await cached(this.cacheDir, `mu-search:${meta.title}`, 30 * DAY,
|
||||
() => this.request('https://api.mangaupdates.com/v1/series/search', { search: String(meta.title || ''), perpage: 10 }));
|
||||
for (const hit of result.results || []) {
|
||||
const record = hit.record || {}, hitYear = Number(record.year) || null;
|
||||
if (year && hitYear && Math.abs(hitYear - year) > 1) continue;
|
||||
if ([record.title, hit.hit_title].map(norm).some(name => name && titles.includes(name))) {
|
||||
return { id: String(record.series_id), title: record.title, url: record.url || null };
|
||||
}
|
||||
}
|
||||
return null;
|
||||
}
|
||||
async releases(seriesId, { pages = Infinity, maxAgeMs = 7 * DAY } = {}) {
|
||||
return cached(this.cacheDir, `mu-releases:${seriesId}:${pages}`, maxAgeMs, async () => {
|
||||
const rows = [];
|
||||
for (let page = 1; page <= pages; page++) {
|
||||
const result = await this.request('https://api.mangaupdates.com/v1/releases/search',
|
||||
{ search: String(seriesId), search_type: 'series', perpage: 100, page, orderby: 'date' });
|
||||
for (const { record } of result.results || []) {
|
||||
if (record?.release_date) rows.push({ chapter: record.chapter, date: record.release_date, groups: (record.groups || []).map(g => g.name) });
|
||||
}
|
||||
if (!result.results?.length || page * 100 >= (result.total_hits || 0)) break;
|
||||
}
|
||||
return rows;
|
||||
});
|
||||
}
|
||||
async originalRun(anilistId) {
|
||||
if (!anilistId) return null;
|
||||
const result = await cached(this.cacheDir, `anilist-run:${anilistId}`, 30 * DAY, () => this.request('https://graphql.anilist.co', {
|
||||
query: 'query ($id: Int) { Media(id: $id, type: MANGA) { status startDate { year month day } endDate { year month day } } }',
|
||||
variables: { id: Number(anilistId) },
|
||||
}));
|
||||
const media = result.data?.Media;
|
||||
if (!media) return null;
|
||||
const date = (d, end) => (d?.year ? Date.UTC(d.year, (d.month || (end ? 12 : 1)) - 1, d.day || (end ? 28 : 1)) : null);
|
||||
return { status: media.status, start: date(media.startDate, false), end: media.status === 'RELEASING' ? null : date(media.endDate, true) };
|
||||
}
|
||||
}
|
||||
|
||||
// Pure decision: official dates per chapter key, with the reasons others were rejected.
|
||||
// evidence: Map chapterKey -> earliest other time (ms) the library knows for that chapter.
|
||||
function officialDates(releases, run, evidence = new Map()) {
|
||||
const official = new Map(), scans = new Map();
|
||||
for (const row of releases) {
|
||||
const key = chapterKey(row.chapter), time = Date.parse(row.date);
|
||||
if (!key || !Number.isFinite(time)) continue;
|
||||
const target = isOfficial(row.groups) ? official : scans;
|
||||
const prior = target.get(key);
|
||||
if (!prior || time < prior.time) target.set(key, { time, date: String(row.date).slice(0, 10), groups: row.groups });
|
||||
}
|
||||
const accepted = new Map(), rejected = { outsideRun: 0, earlier: 0, outOfOrder: 0, noRun: 0 };
|
||||
// 3. Walking down from the newest chapter, release dates never rise again: a lower chapter
|
||||
// dated after a higher one is a back-catalog re-release (One Piece ch. 8 on 2019-02-06).
|
||||
let ceiling = Infinity;
|
||||
for (const [key, item] of [...official].sort((a, b) => Number(b[0]) - Number(a[0]))) {
|
||||
if (!run || !run.start) { rejected.noRun++; continue; }
|
||||
if (item.time < run.start - 7 * DAY || (run.end && item.time > run.end + 30 * DAY)) { rejected.outsideRun++; continue; }
|
||||
const earliest = Math.min(scans.get(key)?.time ?? Infinity, evidence.get(key) ?? Infinity);
|
||||
if (earliest < item.time - EARLY_TOLERANCE_DAYS * DAY) { rejected.earlier++; continue; }
|
||||
if (item.time > ceiling + ORDER_TOLERANCE_DAYS * DAY) { rejected.outOfOrder++; continue; }
|
||||
ceiling = Math.min(ceiling, item.time);
|
||||
accepted.set(key, { at: item.date, precision: 'day', source: `MangaUpdates · ${item.groups.join(', ')}` });
|
||||
}
|
||||
return { accepted, rejected, officialCount: official.size };
|
||||
}
|
||||
|
||||
// Chapter version folders of one manga with their numeric chapter key.
|
||||
async function chapterVersions(mangaDir) {
|
||||
const out = [];
|
||||
const dirs = async dir => (await fs.readdir(dir, { withFileTypes: true })).filter(e => e.isDirectory() && !e.name.startsWith('.')).map(e => e.name);
|
||||
for (const book of await dirs(mangaDir)) {
|
||||
if (!parseKey(book)) continue;
|
||||
for (const chapter of await dirs(path.join(mangaDir, book))) {
|
||||
const parsed = parseKey(chapter);
|
||||
if (!parsed || parsed.x) continue; // extras and specials have no chapter number to match
|
||||
const file = path.join(mangaDir, book, chapter, 'chapter.json');
|
||||
let meta;
|
||||
try { meta = JSON.parse(await fs.readFile(file, 'utf8')); } catch { continue; }
|
||||
out.push({ key: chapterKey(parsed.num), book, file, meta });
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Look up, decide and (with apply) write x-dates.release for one manga folder.
|
||||
async function refreshMangaReleaseDates(mangaDir, sources, { apply = false, pages = Infinity, maxAgeMs } = {}) {
|
||||
const meta = JSON.parse(await fs.readFile(path.join(mangaDir, 'manga.json'), 'utf8').catch(() => '{}'));
|
||||
const versions = await chapterVersions(mangaDir);
|
||||
const summary = { manga: path.basename(mangaDir), series: null, chapters: new Set(versions.map(v => v.key)).size, official: 0, accepted: 0, written: 0, cleared: 0, rejected: null, ambiguous: false };
|
||||
// Seasons that restart at chapter 1 cannot be matched to MangaUpdates' running numbers.
|
||||
const books = new Map();
|
||||
for (const version of versions) books.set(version.key, (books.get(version.key) || new Set()).add(version.book));
|
||||
if ([...books.values()].some(set => set.size > 1)) { summary.ambiguous = true; return summary; }
|
||||
const series = await sources.findSeries(meta);
|
||||
if (!series) return summary;
|
||||
summary.series = series;
|
||||
const run = await sources.originalRun(meta['x-anilist']?.id || meta['x-scraper']?.ids?.anilist);
|
||||
const evidence = new Map();
|
||||
for (const version of versions) {
|
||||
const time = Date.parse(version.meta['x-dates']?.source?.at);
|
||||
if (version.key && Number.isFinite(time) && time < (evidence.get(version.key) ?? Infinity)) evidence.set(version.key, time);
|
||||
}
|
||||
const { accepted, rejected, officialCount } = officialDates(await sources.releases(series.id, { pages, ...(maxAgeMs ? { maxAgeMs } : {}) }), run, evidence);
|
||||
summary.official = officialCount; summary.rejected = rejected;
|
||||
const have = new Set(versions.map(v => v.key));
|
||||
summary.accepted = [...accepted.keys()].filter(key => have.has(key)).length;
|
||||
for (const version of versions) {
|
||||
const dates = version.meta['x-dates'] || { version: 1 }, next = accepted.get(version.key);
|
||||
const prior = dates.release;
|
||||
let changed = false;
|
||||
if (next && JSON.stringify(prior) !== JSON.stringify(next)) { dates.release = next; changed = true; summary.written++; }
|
||||
// A date this job wrote earlier and no longer accepts is withdrawn; other release dates stay.
|
||||
else if (!next && prior && /^MangaUpdates\b/.test(prior.source || '') && pages === Infinity) { delete dates.release; changed = true; summary.cleared++; }
|
||||
if (changed && apply) await writeJsonAtomic(version.file, { ...version.meta, 'x-dates': dates });
|
||||
}
|
||||
return summary;
|
||||
}
|
||||
|
||||
module.exports = { ReleaseDateSources, officialDates, refreshMangaReleaseDates, chapterKey, isOfficial, EARLY_TOLERANCE_DAYS };
|
||||
@@ -1,6 +1,7 @@
|
||||
'use strict';
|
||||
|
||||
const { specialKey } = require('../../chapter-identity');
|
||||
const { batchFlags } = require('../../chapter-dates');
|
||||
|
||||
function localizedTitle(attributes = {}) {
|
||||
const titles = attributes.title || {};
|
||||
@@ -41,14 +42,18 @@ function selectChapterReleases(resources = [], specialLabel = 'Special') {
|
||||
const prior = selected.get(number);
|
||||
if (!prior || compareRank(resource, prior) < 0) selected.set(number, resource);
|
||||
}
|
||||
return [...selected.entries()].map(([number, resource]) => {
|
||||
const chosen = [...selected.entries()];
|
||||
const uploaded = chosen.map(([, resource]) => resource.attributes?.publishAt || resource.attributes?.readableAt || resource.attributes?.createdAt);
|
||||
const batches = batchFlags(uploaded);
|
||||
return chosen.map(([number, resource], index) => {
|
||||
const group = groupFor(resource);
|
||||
return {
|
||||
id: resource.id, number, title: resource.attributes?.title || '',
|
||||
...(!chapterNumber(resource.attributes?.chapter) ? { special: true, label: specialLabel, sourceId: resource.id, sourceNumber: resource.attributes?.chapter ?? null } : {}),
|
||||
...(resource.attributes?.volume ? { volume: chapterNumber(resource.attributes.volume) || String(resource.attributes.volume) } : {}),
|
||||
pages: Number(resource.attributes?.pages || 0),
|
||||
added: resource.attributes?.publishAt || resource.attributes?.readableAt || resource.attributes?.createdAt,
|
||||
// A group's upload time, not an official release; bulk uploads are flagged.
|
||||
...(uploaded[index] ? { uploadedAt: uploaded[index], uploadBatch: !!batches[index] } : {}),
|
||||
group,
|
||||
url: `https://mangadex.org/chapter/${resource.id}`,
|
||||
};
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
'use strict';
|
||||
const { volumeKey } = require('../../chapter-identity');
|
||||
const { batchFlags } = require('../../chapter-dates');
|
||||
|
||||
// First-party, pure parsing functions. This module never fetches remote content.
|
||||
const decode = value => value.replace(/&/g, '&').replace(/'/g, "'").replace(/"/g, '"').replace(/<[^>]+>/g, '').trim();
|
||||
@@ -21,19 +22,26 @@ function catalog({ html, url }) {
|
||||
const volume = !regular ? text.match(/^Volume\s+([0-9]+(?:\.[0-9]+)?)(?!\d)/i)?.[1] : null;
|
||||
const prologue = text.match(/Prologue(?:\s+([0-9]+))?/i)?.[1];
|
||||
const number = regular || (prologue ? `0.${prologue.padStart(2, '0')}` : /Prologue/i.test(text) ? '0' : null);
|
||||
// The full chapter list carries each upload time; the chapter page itself has none.
|
||||
const uploaded = match[3].match(/datetime="([^"]+)"/i)?.[1];
|
||||
const uploadedAt = uploaded && Number.isFinite(Date.parse(uploaded)) ? uploaded : undefined;
|
||||
if (number) found.push({
|
||||
id: match[2], number, title: regular ? '' : `Prologue ${prologue || ''}`.trim(),
|
||||
...(season ? { book: { number: season, title: `Season ${season}` } } : {}),
|
||||
url: absolute(url, match[1]), raw: text,
|
||||
url: absolute(url, match[1]), raw: text, ...(uploadedAt ? { uploadedAt } : {}),
|
||||
});
|
||||
else if (volume) found.push({
|
||||
id: match[2], number: volumeKey(match[2]), title: '', special: true,
|
||||
label: `Volume ${volume}`, releaseKind: 'volume', volume,
|
||||
book: { number: Number(volume), title: `Volume ${volume}` },
|
||||
url: absolute(url, match[1]), raw: text,
|
||||
url: absolute(url, match[1]), raw: text, ...(uploadedAt ? { uploadedAt } : {}),
|
||||
});
|
||||
}
|
||||
const chapters = [...new Map(found.map(chapter => [chapter.id, chapter])).values()];
|
||||
// Back-catalog chapters were imported together; their upload time is not a release date.
|
||||
batchFlags(chapters.map(chapter => chapter.uploadedAt)).forEach((batch, index) => {
|
||||
if (batch !== null) chapters[index].uploadBatch = batch;
|
||||
});
|
||||
const chapterLinks = [...html.matchAll(/<a\b[^>]*href="[^"]*\/chapters\/[A-Z0-9]+[^"]*"/gi)].length;
|
||||
return { chapters, chapterLinks };
|
||||
}
|
||||
@@ -42,7 +50,6 @@ function chapter({ html, url }) {
|
||||
return {
|
||||
manifestUrl: html.match(/['"](https?:\/\/weebcentral\.com\/chapters\/[^'"]+\/images\?is_prev=[^'"]+)['"]/i)?.[1]
|
||||
|| `${url}/images?is_prev=False`,
|
||||
added: html.match(/datetime="([^"]+)"/i)?.[1],
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+61
-5
@@ -13,6 +13,7 @@ const { scanLibrary, findManga, shouldDownload, isNewerThanLibrary, summarizeInv
|
||||
const { inspectLegacyWork, promoteLegacyWork } = require('./lib/legacy-publication');
|
||||
const { classifyFailure, safeMessage, progressKey } = require('./lib/failures');
|
||||
const { writeJsonAtomic } = require('./lib/json-file');
|
||||
const { ReleaseDateSources, refreshMangaReleaseDates } = require('./metadata/release-dates');
|
||||
|
||||
const RATE_KEYS = ['minMs', 'maxMs', 'chapterMinMs', 'chapterMaxMs'];
|
||||
const FAST_RATE_SETTINGS = {
|
||||
@@ -23,9 +24,13 @@ const DEFAULT_SOURCE_RATES = {
|
||||
mangadex: { ...FAST_RATE_SETTINGS },
|
||||
mangapill: { ...FAST_RATE_SETTINGS },
|
||||
};
|
||||
// Sources an admin must switch on first. WeebCentral and MangaPill mostly host unlicensed
|
||||
// copies; MangaDex has an official API and handles takedowns. Peers are always available.
|
||||
const DEFAULT_SOURCES_ENABLED = { weebcentral: false, mangadex: true, mangapill: false };
|
||||
const DEFAULT_SETTINGS = {
|
||||
...FAST_RATE_SETTINGS,
|
||||
sourceRates: Object.fromEntries(Object.entries(DEFAULT_SOURCE_RATES).map(([source, rate]) => [source, { ...rate }])),
|
||||
sourcesEnabled: { ...DEFAULT_SOURCES_ENABLED },
|
||||
autoUpdateEnabled: true, autoUpdateTime: '03:00',
|
||||
};
|
||||
|
||||
@@ -72,6 +77,15 @@ function validateSettings(input = {}, prior = DEFAULT_SETTINGS) {
|
||||
};
|
||||
next.sourceRates[source] = validateRateSettings(patch || {}, base, `sourceRates.${source}.`);
|
||||
}
|
||||
if (input.sourcesEnabled !== undefined && (!input.sourcesEnabled || typeof input.sourcesEnabled !== 'object' || Array.isArray(input.sourcesEnabled))) {
|
||||
throw new Error('sourcesEnabled must be an object');
|
||||
}
|
||||
next.sourcesEnabled = { ...DEFAULT_SOURCES_ENABLED, ...(prior.sourcesEnabled || {}) };
|
||||
for (const [source, enabled] of Object.entries(input.sourcesEnabled || {})) {
|
||||
if (!Object.hasOwn(DEFAULT_SOURCES_ENABLED, source)) throw new Error(`Unknown source: ${source}`);
|
||||
if (typeof enabled !== 'boolean') throw new Error(`sourcesEnabled.${source} must be true or false`);
|
||||
next.sourcesEnabled[source] = enabled;
|
||||
}
|
||||
if (input.autoUpdateEnabled !== undefined) next.autoUpdateEnabled = !!input.autoUpdateEnabled;
|
||||
if (input.autoUpdateTime !== undefined) {
|
||||
const value = String(input.autoUpdateTime).trim();
|
||||
@@ -85,6 +99,13 @@ function settingsForSource(settings, source) {
|
||||
return { ...settings, ...((settings.sourceRates || {})[source] || {}) };
|
||||
}
|
||||
|
||||
// Installations from before the switch keep every source they already use: off-by-default
|
||||
// applies to new installations, it must not silently stop a working queue.
|
||||
function inheritedSourcesEnabled(saved) {
|
||||
const used = new Set([...(saved.queue || []), ...(saved.history || [])].map(item => item && item.source));
|
||||
return Object.fromEntries(Object.entries(DEFAULT_SOURCES_ENABLED).map(([source, enabled]) => [source, enabled || used.has(source)]));
|
||||
}
|
||||
|
||||
function existingPolicy(value) {
|
||||
const policy = String(value || 'skip-any');
|
||||
if (!['skip-any', 'add-source'].includes(policy)) throw new Error('existingPolicy must be skip-any or add-source');
|
||||
@@ -216,11 +237,13 @@ class ScraperService {
|
||||
async loadState() {
|
||||
if (await exists(this.stateFile)) {
|
||||
const saved = JSON.parse(await fs.readFile(this.stateFile, 'utf8'));
|
||||
const savedSettings = saved.settings || {};
|
||||
if (!savedSettings.sourcesEnabled) savedSettings.sourcesEnabled = inheritedSourcesEnabled(saved);
|
||||
return {
|
||||
version: 3, paused: false, status: 'idle', queue: [], history: [], publishedRevision: 0,
|
||||
currentIds: {}, sourcePaused: {},
|
||||
settings: DEFAULT_SETTINGS, ...saved,
|
||||
settings: validateSettings(saved.settings || {}, DEFAULT_SETTINGS),
|
||||
settings: validateSettings(savedSettings, DEFAULT_SETTINGS),
|
||||
};
|
||||
}
|
||||
const state = {
|
||||
@@ -238,7 +261,7 @@ class ScraperService {
|
||||
status: completed.has(item.id) ? 'complete' : 'queued', order: index,
|
||||
added: legacy.created || new Date().toISOString(),
|
||||
}));
|
||||
state.settings = validateSettings(legacy.settings || {}, DEFAULT_SETTINGS);
|
||||
state.settings = validateSettings({ ...(legacy.settings || {}), sourcesEnabled: inheritedSourcesEnabled(state) }, DEFAULT_SETTINGS);
|
||||
state.migratedFrom = this.legacyStateFile;
|
||||
}
|
||||
return state;
|
||||
@@ -353,12 +376,36 @@ class ScraperService {
|
||||
const delay = Math.max(1000, next.getTime() - Date.now());
|
||||
this.discoveryTimer = setTimeout(() => {
|
||||
this.discoveryTimer = null;
|
||||
this.enqueueDiscoveryChecks('schedule').catch(error => this.event('error', `Nightly discovery failed: ${error.message}`))
|
||||
this.refreshReleaseDates('schedule').catch(error => this.event('error', `Nightly release dates failed: ${error.message}`))
|
||||
.then(() => this.enqueueDiscoveryChecks('schedule'))
|
||||
.catch(error => this.event('error', `Nightly discovery failed: ${error.message}`))
|
||||
.finally(() => { if (!this.stopping) this.scheduleDiscovery(); });
|
||||
}, delay);
|
||||
this.discoveryTimer.unref();
|
||||
}
|
||||
|
||||
// Nightly, before the update checks: official release dates of ongoing series from their
|
||||
// newest MangaUpdates releases (BL-0048). Runs only while no download is active, so the
|
||||
// scraper stays the single writer of chapter.json.
|
||||
async refreshReleaseDates(reason = 'manual') {
|
||||
if (this.active.size) { await this.event('info', 'Release dates skipped: a download is running', { reason }); return null; }
|
||||
const inventory = await this.libraryInventory(true);
|
||||
const sources = new ReleaseDateSources({ cacheDir: path.join(this.dataRoot, 'release-dates-cache') });
|
||||
let checked = 0, written = 0;
|
||||
for (const manga of inventory.mangas) {
|
||||
if (this.stopping || this.active.size) break;
|
||||
if (manga.meta?.status !== 'ongoing') continue;
|
||||
try {
|
||||
const result = await refreshMangaReleaseDates(manga.path, sources, { apply: true, pages: 1, maxAgeMs: 20 * 3600e3 });
|
||||
checked++; written += result.written;
|
||||
} catch (error) { await this.event('warning', `Release dates for ${manga.title} failed: ${error.message}`, { mangaId: manga.id }); }
|
||||
}
|
||||
// The reader reconciles fully when the revision rises without a visible folder change.
|
||||
if (written) { this.state.publishedRevision = Number(this.state.publishedRevision || 0) + 1; await this.persist(); }
|
||||
await this.event('info', `Release dates: ${written} chapter${written === 1 ? '' : 's'} dated in ${checked} ongoing series`, { reason, checked, written });
|
||||
return { checked, written };
|
||||
}
|
||||
|
||||
pendingSeries(source, seriesId) {
|
||||
return this.state.queue.some(item => item.source === source && String(item.seriesId) === String(seriesId)
|
||||
&& ['queued', 'downloading', 'discovering', 'ready', 'error', 'retry_wait', 'quarantined', 'unavailable'].includes(item.status));
|
||||
@@ -376,7 +423,8 @@ class ScraperService {
|
||||
let queued = 0, skipped = 0;
|
||||
for (const manga of inventory.mangas) {
|
||||
if (!manga.autoUpdate) continue;
|
||||
const supported = manga.sources.map(source => ({ source, found: this.resolveAdapter(source.url) })).filter(item => item.found);
|
||||
const supported = manga.sources.map(source => ({ source, found: this.resolveAdapter(source.url) }))
|
||||
.filter(item => item.found && this.sourceEnabled(item.found.adapter.id));
|
||||
if (!supported.length) { skipped++; continue; }
|
||||
const preferred = manga.meta?.['x-scraper']?.preferredSource;
|
||||
const selected = supported.find(item => item.found.adapter.id === preferred) || supported[0];
|
||||
@@ -573,9 +621,16 @@ class ScraperService {
|
||||
};
|
||||
}
|
||||
|
||||
sourceEnabled(source) {
|
||||
return !Object.hasOwn(DEFAULT_SOURCES_ENABLED, source) || this.state.settings.sourcesEnabled?.[source] === true;
|
||||
}
|
||||
|
||||
async enqueue(input) {
|
||||
const found = this.resolveAdapter(input.url);
|
||||
if (!found) throw new Error('No installed scraper adapter accepts this URL');
|
||||
if (!this.sourceEnabled(found.adapter.id)) {
|
||||
throw new Error(`${found.adapter.label || found.adapter.id} is switched off. An admin can switch it on under Settings → Admin → Scraper → Hoster settings.`);
|
||||
}
|
||||
const workIdentity = normalizeWorkIdentity(input.workIdentity);
|
||||
if (input.workIdentity != null && !workIdentity) throw new Error('Invalid manga work identity');
|
||||
const scope = normalizeScope(input.scope || {});
|
||||
@@ -877,7 +932,8 @@ class ScraperService {
|
||||
continue;
|
||||
}
|
||||
const source = found.adapter.id;
|
||||
if (this.active.has(source) || this.state.sourcePaused[source]) continue;
|
||||
// A switched-off source keeps its queued jobs; they wait until it is switched on again.
|
||||
if (this.active.has(source) || this.state.sourcePaused[source] || !this.sourceEnabled(source)) continue;
|
||||
if (this.state.circuits[source]?.state === 'open') continue;
|
||||
if (item.libraryMangaId && [...this.active.values()].some(entry => {
|
||||
const activeItem = this.state.queue.find(candidate => candidate.id === entry.id);
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs').promises;
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { officialDates, refreshMangaReleaseDates, chapterKey } = require('../metadata/release-dates');
|
||||
|
||||
const DAY = 864e5;
|
||||
const run = (start, end = null) => ({ start: Date.parse(start), end: end ? Date.parse(end) : null });
|
||||
const viz = (chapter, date) => ({ chapter, date, groups: ['Viz', 'MANGA Plus'] });
|
||||
const scan = (chapter, date) => ({ chapter, date, groups: ['Some Scans'] });
|
||||
|
||||
test('chapter keys only match single numbered chapters', () => {
|
||||
assert.equal(chapterKey('001'), '1');
|
||||
assert.equal(chapterKey('10.50'), '10.5');
|
||||
assert.equal(chapterKey('1-3'), null);
|
||||
assert.equal(chapterKey('7a'), null);
|
||||
});
|
||||
|
||||
test('simulpub dates are accepted, re-releases after the run are not', () => {
|
||||
const ongoing = officialDates([viz('1', '2023-09-17'), viz('2', '2023-09-24'), scan('2', '2023-09-23')], run('2023-09-18'));
|
||||
assert.deepEqual([...ongoing.accepted.keys()].sort(), ['1', '2'], 'scans a day early do not disqualify');
|
||||
assert.equal(ongoing.accepted.get('1').at, '2023-09-17');
|
||||
assert.equal(ongoing.accepted.get('1').source, 'MangaUpdates · Viz, MANGA Plus');
|
||||
const ended = officialDates([viz('700', '2020-12-20')], run('1999-09-21', '2014-11-10'));
|
||||
assert.equal(ended.accepted.size, 0);
|
||||
assert.equal(ended.rejected.outsideRun, 1);
|
||||
assert.equal(officialDates([viz('1', '2020-01-01')], null).rejected.noRun, 1);
|
||||
});
|
||||
|
||||
test('an earlier scanlation or source upload means the official date is a re-release', () => {
|
||||
const scanned = officialDates([viz('5', '2021-06-01'), scan('5', '2019-01-01')], run('2018-01-01'));
|
||||
assert.equal(scanned.rejected.earlier, 1);
|
||||
const uploaded = officialDates([viz('5', '2021-06-01')], run('2018-01-01'), new Map([['5', Date.parse('2021-05-01')]]));
|
||||
assert.equal(uploaded.rejected.earlier, 1);
|
||||
const leak = officialDates([viz('5', '2021-06-04')], run('2018-01-01'), new Map([['5', Date.parse('2021-06-04') - 2 * DAY]]));
|
||||
assert.equal(leak.accepted.size, 1, 'an upload two days early is a leak, not an older release');
|
||||
});
|
||||
|
||||
test('back-catalog chapters dated after newer chapters are out of order', () => {
|
||||
const result = officialDates([
|
||||
viz('8', '2019-02-06'), viz('33', '2019-07-31'), // weekly back-catalog re-release
|
||||
viz('928', '2018-12-21'), viz('930', '2019-01-20'), viz('931', '2019-02-03'), viz('1196', '2026-10-18'),
|
||||
viz('929', '2019-01-21'), // a day after 930: a double release is still in order
|
||||
], run('1997-07-22'));
|
||||
assert.deepEqual([...result.accepted.keys()].sort((a, b) => a - b), ['928', '929', '930', '931', '1196']);
|
||||
assert.equal(result.rejected.outOfOrder, 2);
|
||||
});
|
||||
|
||||
async function library(chapters) {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), 'release-dates-'));
|
||||
const manga = path.join(root, 'Kagurabachi');
|
||||
await fs.mkdir(manga, { recursive: true });
|
||||
await fs.writeFile(path.join(manga, 'manga.json'), JSON.stringify({ title: 'Kagurabachi', year: 2023, 'x-anilist': { id: 169355 } }));
|
||||
for (const [folder, meta] of Object.entries(chapters)) {
|
||||
await fs.mkdir(path.join(manga, folder), { recursive: true });
|
||||
await fs.writeFile(path.join(manga, folder, 'chapter.json'), JSON.stringify(meta));
|
||||
}
|
||||
return manga;
|
||||
}
|
||||
const fakeSources = releases => ({
|
||||
async findSeries(meta) { return meta.title === 'Kagurabachi' ? { id: '60441951323', title: 'Kagurabachi' } : null; },
|
||||
async originalRun() { return run('2023-09-18'); },
|
||||
async releases() { return releases; },
|
||||
});
|
||||
|
||||
test('refresh writes x-dates.release, keeps other dates and withdraws its own stale dates', async () => {
|
||||
const manga = await library({
|
||||
'v01/c0001 [WeebCentral]': { title: '', added: '2026-10-01T00:00:00Z', 'x-dates': { version: 1, source: { at: '2024-09-07T10:00:00Z', name: 'WeebCentral', batch: true } } },
|
||||
'v01/c0002 [WeebCentral]': { title: '', 'x-dates': { version: 1, release: { at: '2020-01-01', precision: 'day', source: 'MangaUpdates · Viz' } } },
|
||||
'v01/c0003 [WeebCentral]': { title: '', 'x-dates': { version: 1, release: { at: '2023-10-01', precision: 'day', source: 'Hand-entered' } } },
|
||||
});
|
||||
const dry = await refreshMangaReleaseDates(manga, fakeSources([viz('1', '2023-09-17')]));
|
||||
assert.equal(dry.written, 1);
|
||||
assert.equal(JSON.parse(await fs.readFile(path.join(manga, 'v01/c0001 [WeebCentral]/chapter.json'), 'utf8'))['x-dates'].release, undefined, 'dry run writes nothing');
|
||||
const applied = await refreshMangaReleaseDates(manga, fakeSources([viz('1', '2023-09-17')]), { apply: true });
|
||||
assert.deepEqual([applied.written, applied.cleared], [1, 1]);
|
||||
const one = JSON.parse(await fs.readFile(path.join(manga, 'v01/c0001 [WeebCentral]/chapter.json'), 'utf8'));
|
||||
assert.deepEqual(one['x-dates'].release, { at: '2023-09-17', precision: 'day', source: 'MangaUpdates · Viz, MANGA Plus' });
|
||||
assert.equal(one['x-dates'].source.batch, true);
|
||||
assert.equal(one.added, '2026-10-01T00:00:00Z');
|
||||
assert.equal(JSON.parse(await fs.readFile(path.join(manga, 'v01/c0002 [WeebCentral]/chapter.json'), 'utf8'))['x-dates'].release, undefined);
|
||||
assert.equal(JSON.parse(await fs.readFile(path.join(manga, 'v01/c0003 [WeebCentral]/chapter.json'), 'utf8'))['x-dates'].release.source, 'Hand-entered');
|
||||
});
|
||||
|
||||
test('seasons that restart their numbering are skipped', async () => {
|
||||
const manga = await library({ 'v01 Season 1/c0001': { title: '' }, 'v02 Season 2/c0001': { title: '' } });
|
||||
const result = await refreshMangaReleaseDates(manga, fakeSources([viz('1', '2023-09-17')]), { apply: true });
|
||||
assert.equal(result.ambiguous, true);
|
||||
assert.equal(result.written, 0);
|
||||
});
|
||||
@@ -426,6 +426,9 @@ test('daemon queue state is durable while paused', async t => {
|
||||
}));
|
||||
const service = await new ScraperService({ libraryRoot: path.join(temp, 'library'), dataRoot }).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
|
||||
// WeebCentral is off on a fresh installation until an admin switches it on.
|
||||
await assert.rejects(() => service.enqueue({ url: 'https://weebcentral.com/series/01ABCDEF/My-Series' }), /switched off/);
|
||||
await service.updateSettings({ sourcesEnabled: { weebcentral: true } });
|
||||
const item = await service.enqueue({ url: 'https://weebcentral.com/series/01ABCDEF/My-Series' });
|
||||
assert.equal(item.status, 'queued');
|
||||
const scoped = await service.enqueue({ url: 'https://weebcentral.com/series/01ABCDEF/My-Series', scope: { mode: 'latest' } });
|
||||
@@ -439,6 +442,30 @@ test('daemon queue state is durable while paused', async t => {
|
||||
assert.equal((await service.publicState()).queue.length, 0);
|
||||
});
|
||||
|
||||
test('sources are off by default but existing installations keep the ones they use', async t => {
|
||||
assert.deepEqual(validateSettings({}).sourcesEnabled, { weebcentral: false, mangadex: true, mangapill: false });
|
||||
assert.throws(() => validateSettings({ sourcesEnabled: { weebcentral: 'yes' } }), /true or false/);
|
||||
assert.throws(() => validateSettings({ sourcesEnabled: { other: true } }), /Unknown source/);
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-sources-'));
|
||||
const dataRoot = path.join(temp, 'data');
|
||||
await fs.mkdir(dataRoot, { recursive: true });
|
||||
const { sourcesEnabled, ...older } = validateSettings({});
|
||||
await fs.writeFile(path.join(dataRoot, 'queue.json'), JSON.stringify({
|
||||
version: 3, paused: true, status: 'paused', settings: older, publishedRevision: 0,
|
||||
queue: [{ id: 'A', title: 'A', url: 'https://weebcentral.com/series/AAA/A', source: 'weebcentral', status: 'queued', order: 0 }],
|
||||
history: [],
|
||||
}));
|
||||
const service = await new ScraperService({ libraryRoot: path.join(temp, 'library'), dataRoot }).init();
|
||||
t.after(async () => { await service.close(); await fs.rm(temp, { recursive: true, force: true }); });
|
||||
assert.deepEqual(service.state.settings.sourcesEnabled, { weebcentral: true, mangadex: true, mangapill: false });
|
||||
await assert.rejects(() => service.enqueue({ url: 'https://mangapill.com/manga/2/one-piece' }), /MangaPill is switched off/);
|
||||
// Switching a source off keeps its jobs queued instead of failing them.
|
||||
await service.updateSettings({ sourcesEnabled: { weebcentral: false } });
|
||||
assert.equal(service.sourceEnabled('weebcentral'), false);
|
||||
assert.equal(service.sourceEnabled('peer-abc'), true, 'peers have no switch');
|
||||
assert.equal(service.state.queue[0].status, 'queued');
|
||||
});
|
||||
|
||||
test('retry recovers stale downloading entries before work resumes', async t => {
|
||||
const temp = await fs.mkdtemp(path.join(os.tmpdir(), 'mcscraper-retry-'));
|
||||
const dataRoot = path.join(temp, 'data');
|
||||
|
||||
@@ -10,8 +10,8 @@ const { adapters, adapterFor } = require('../adapters');
|
||||
const seriesUrl = 'https://weebcentral.com/series/ABC/title';
|
||||
const chapterUrl = 'https://weebcentral.com/chapters/XYZ';
|
||||
const seriesHtml = '<h1>Test Manga</h1><div hx-get="/series/ABC/full-chapter-list"></div>';
|
||||
const listHtml = '<a href="/chapters/XYZ">Chapter 1</a><a href="/chapters/LMN">Log. 2</a>';
|
||||
const chapterHtml = '<time datetime="2026-01-01"></time>';
|
||||
const listHtml = '<a href="/chapters/XYZ"><span>Chapter 1</span><time datetime="2026-01-01T15:06:05.348Z">x</time></a><a href="/chapters/LMN">Log. 2</a>';
|
||||
const chapterHtml = '<div>no dates on the chapter page</div>';
|
||||
const imagesHtml = '<img src="https://img.example/1.jpg"><img data-src="https://img.example/2.jpg">';
|
||||
|
||||
test('first-party registry rejects unknown plugins and malformed input', () => {
|
||||
@@ -45,7 +45,9 @@ test('in-process parser preserves WeebCentral catalog and page discovery', async
|
||||
assert.deepEqual(catalog.chapters.map(chapter => chapter.number), ['1', '2']);
|
||||
const chapter = await discoverWeebChapter(catalog.chapters[0], http);
|
||||
assert.equal(chapter.pages.length, 2);
|
||||
assert.equal(chapter.added, '2026-01-01');
|
||||
assert.equal(chapter.added, undefined);
|
||||
assert.deepEqual(chapter.dates, { source: { at: '2026-01-01T15:06:05.348Z', name: 'WeebCentral', batch: false } });
|
||||
assert.equal(catalog.chapters[1].uploadedAt, undefined);
|
||||
assert.equal(requested.length, 4);
|
||||
});
|
||||
|
||||
|
||||
@@ -19,16 +19,23 @@ const { Catalog, CatalogError, collectionViews, TYPES, ROLES } = require('./cata
|
||||
const { RATINGS: GREATEST_RATINGS, loadCatalog: loadGreatestCatalog, findOwned: findGreatestOwned,
|
||||
countsFor: greatestCounts, scoreFor: greatestScore } = require('./greatest');
|
||||
const { hosterLinks } = require('./greatest/hosters');
|
||||
const { GreatestCovers } = require('./greatest/covers');
|
||||
const { secret } = require('./runtime/config');
|
||||
const { PeeringState, requirePeer } = require('./peering/state');
|
||||
const peerClient = require('./peering/client');
|
||||
const { DiscoveryStore } = require('./discovery/store');
|
||||
const { generate: generateDiscovery } = require('./discovery/engine');
|
||||
const { generateAsync: generateDiscovery, mergeUserFeed } = require('./discovery/engine');
|
||||
const { DiscoveryCache } = require('./discovery/cache');
|
||||
const { createClient: createAniListClient } = require('./discovery/anilist');
|
||||
const { anchorsFor } = require('./discovery/anchors');
|
||||
const { shouldRunNightly, Harvester } = require('./discovery/harvest');
|
||||
const {
|
||||
applyPulse, applyVisit, chapterInsight, eligiblePages, emptyRecord, libraryInsight, mangaInsight, publicSummary,
|
||||
applyPulse, applyVisit, chapterInsight, earliestCompletion, eligiblePages, emptyRecord, libraryInsight, mangaInsight, publicSummary,
|
||||
} = require('./reading-analytics');
|
||||
const { chapterDates } = require('./chapter-dates');
|
||||
const { parseSchedule, publicSchedule, loadQuestionBank, createReadingAccess } = require('./reading-access');
|
||||
const { BackupService, BackupError } = require('./backup');
|
||||
const moments = require('./moments');
|
||||
|
||||
const PORT = +process.env.PORT || 8080;
|
||||
const LIB = path.resolve(process.env.LIBRARY || 'library');
|
||||
@@ -43,6 +50,9 @@ const PEER_DATA = path.resolve(process.env.PEER_DATA || path.join(DATA, 'peering
|
||||
const peerStore = new PeeringState(PEER_DATA);
|
||||
const discoveryStore = new DiscoveryStore(DATA);
|
||||
const DISCOVERY_CACHE = path.join(DATA, 'scraper', 'metadata', 'anilist');
|
||||
const DISCOVERY_HARVEST_CACHE = path.join(DATA, 'discovery-cache');
|
||||
const HARVEST_TIME = /^([01]\d|2[0-3]):[0-5]\d$/;
|
||||
const DISCOVERY_HARVEST_AT = HARVEST_TIME.test(process.env.DISCOVERY_HARVEST_AT || '') ? process.env.DISCOVERY_HARVEST_AT : '03:30';
|
||||
let discoveryRun = null, discoveryError = null;
|
||||
const PUB = path.join(__dirname, 'public');
|
||||
const IMG = /\.(jpe?g|png|webp|gif|avif|bmp)$/i;
|
||||
@@ -57,8 +67,12 @@ fs.mkdirSync(DATA, { recursive: true });
|
||||
const catalog = new Catalog(DATA);
|
||||
const greatestCatalog = loadGreatestCatalog();
|
||||
const greatestById = new Map(greatestCatalog.entries.map(entry => [entry.id, entry]));
|
||||
const greatestCovers = new GreatestCovers(path.join(DATA, 'greatest-covers'));
|
||||
// The versioned URL lets browsers keep a cover for a year; a new catalog URL changes `v`.
|
||||
const greatestCoverUrl = entry => entry.cover
|
||||
? `api/greatest-cover?id=${encodeURIComponent(entry.id)}&v=${greatestCovers.version(entry.cover)}` : null;
|
||||
const DB_FILE = path.join(DATA, 'db.json'), CACHE_FILE = path.join(DATA, 'scan-cache.json'), INDEX_FILE = path.join(DATA, 'library-index.json');
|
||||
const db = { users: {}, sessions: {}, progress: {}, reading: {}, readingAccess: {}, follows: {}, notes: {}, votes: {}, greatestRatings: {}, discoveryFeedback: {}, known: {} };
|
||||
const db = { users: {}, sessions: {}, progress: {}, reading: {}, readingAccess: {}, follows: {}, notes: {}, votes: {}, greatestRatings: {}, discoveryFeedback: {}, moments: {}, known: {} };
|
||||
// A corrupt db.json stops the server instead of silently starting empty and overwriting it.
|
||||
if (fs.existsSync(DB_FILE)) Object.assign(db, JSON.parse(fs.readFileSync(DB_FILE, 'utf8')));
|
||||
function writeAtomic(file, text) { fs.writeFileSync(file + '.tmp', text); fs.renameSync(file + '.tmp', file); }
|
||||
@@ -77,8 +91,9 @@ const readingAccess = createReadingAccess({ schedule: readingSchedule,
|
||||
getProgress: (user, mangaId) => db.progress[user]?.[mangaId] || {},
|
||||
getMangaTitle: mangaId => lib.byId.get(mangaId)?.title || null });
|
||||
|
||||
class HttpError extends Error { constructor(code, msg) { super(msg); this.code = code; } }
|
||||
function bad(code, msg) { throw new HttpError(code, msg); }
|
||||
// `extra` fields are sent next to `error`, e.g. `nextAllowedAt` on a 429.
|
||||
class HttpError extends Error { constructor(code, msg, extra = {}) { super(msg); this.code = code; this.extra = extra; } }
|
||||
function bad(code, msg, extra) { throw new HttpError(code, msg, extra); }
|
||||
function peerForRoute(id) {
|
||||
try { return requirePeer(peerStore.load(), id); }
|
||||
catch { bad(404, 'Peer not found'); }
|
||||
@@ -97,7 +112,7 @@ function checkPw(pw, stored) {
|
||||
}
|
||||
const DUMMY_HASH = hashPw('x');
|
||||
const DEFAULTS = { startMode: 'last', lastMode: 'page', adapt: true, skipExtras: false, skipCovers: false, flowPageGap: true, pageTone: 'original', readingClock: false, readingStory: true,
|
||||
discoveryEnabled: false, discoveryReadingActivity: false, discoveryMature: false, zoom: {}, webhook: '' };
|
||||
continueRow: true, readDate: 'hour', releaseDate: true, discoveryEnabled: false, discoveryReadingActivity: false, discoveryMature: false, zoom: {}, webhook: '' };
|
||||
const settingsOf = u => ({ ...DEFAULTS, ...u.settings, zoom: { ...(u.settings && u.settings.zoom) } });
|
||||
|
||||
function createUser(name, pw, admin, replace) {
|
||||
@@ -217,6 +232,15 @@ const progressOf = (u, m) => (db.progress[u.name] && db.progress[u.name][m.id])
|
||||
const mangaReadingOf = (u, m) => (db.reading[u.name] && db.reading[u.name][m.id]) || {};
|
||||
const readingOf = (u, m, ch) => (db.reading[u.name] && db.reading[u.name][m.id] && db.reading[u.name][m.id][ch.id]) || null;
|
||||
const unreadOf = u => (db.notes[u.name] || []).filter(n => !n.read).length;
|
||||
const momentsOf = u => (db.moments[u.name] ||= []);
|
||||
const momentOf = (u, id) => momentsOf(u).find(x => x.id === String(id || '')) || bad(404, 'Moment not found');
|
||||
const fileSha = async rel => sha256File(path.join(LIB, rel)).catch(() => null);
|
||||
function sha256File(file) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const hash = crypto.createHash('sha256');
|
||||
fs.createReadStream(file).on('error', reject).on('data', d => hash.update(d)).on('end', () => resolve(hash.digest('hex')));
|
||||
});
|
||||
}
|
||||
const admin = u => u.admin || bad(403, 'Admins only');
|
||||
const READING_LEASE_MS = 15000;
|
||||
const readingLeases = new Map();
|
||||
@@ -240,6 +264,26 @@ function readDiscovery() {
|
||||
try { return discoveryStore.load(); }
|
||||
catch (error) { bad(503, error.message); }
|
||||
}
|
||||
// Scoring is local: it reads the harvest cache and never calls AniList. With `onlyUser` only that
|
||||
// feed is rebuilt and the other users' feeds are carried over into the same publish.
|
||||
async function rescoreNow(onlyUser) {
|
||||
const startedAt = new Date().toISOString();
|
||||
let queuedTitles = [], queueAvailable = true;
|
||||
try { queuedTitles = (await scraperRequest('/api/state')).queue?.filter(item => !['complete', 'removed'].includes(item.status)).map(item => item.title) || []; }
|
||||
catch { queueAvailable = false; }
|
||||
const result = await generateDiscovery({ db, library: lib.mangas, greatest: greatestCatalog,
|
||||
cacheDir: DISCOVERY_CACHE, harvestCache, queuedTitles, onlyUser });
|
||||
// Read the prior store only now, right before the synchronous publish, so no other publish slips between.
|
||||
const value = onlyUser === undefined ? result : { ...result, ...mergeUserFeed(discoveryStore.load(), result, onlyUser) };
|
||||
return { published: discoveryStore.publish({ ...value, startedAt }), queueAvailable };
|
||||
}
|
||||
// One rescore at a time: an admin run and a harvest follow-up must not publish over each other.
|
||||
let rescoring = Promise.resolve();
|
||||
function rescore({ onlyUser } = {}) {
|
||||
const run = rescoring.then(() => rescoreNow(onlyUser));
|
||||
rescoring = run.catch(() => {});
|
||||
return run;
|
||||
}
|
||||
function startDiscoveryRun() {
|
||||
if (discoveryRun?.status === 'running') bad(409, 'A discovery run is already active');
|
||||
readDiscovery(); // Refuse a new run if the on-disk store is corrupt.
|
||||
@@ -248,12 +292,7 @@ function startDiscoveryRun() {
|
||||
discoveryError = null;
|
||||
setImmediate(async () => {
|
||||
try {
|
||||
let queuedTitles = [], queueAvailable = true;
|
||||
try { queuedTitles = (await scraperRequest('/api/state')).queue?.filter(item => !['complete', 'removed'].includes(item.status)).map(item => item.title) || []; }
|
||||
catch { queueAvailable = false; }
|
||||
const result = generateDiscovery({ db, library: lib.mangas, greatest: greatestCatalog,
|
||||
cacheDir: DISCOVERY_CACHE, queuedTitles });
|
||||
const published = discoveryStore.publish({ ...result, startedAt });
|
||||
const { published, queueAvailable } = await rescore();
|
||||
discoveryRun = { id, status: 'success', startedAt, finishedAt: published.lastRun.finishedAt,
|
||||
candidates: published.lastRun.candidateCount, queueAvailable };
|
||||
} catch (error) {
|
||||
@@ -265,6 +304,62 @@ function startDiscoveryRun() {
|
||||
return discoveryRun;
|
||||
}
|
||||
|
||||
// ---------- discovery harvest (AniList seeds, BL-0051) ----------
|
||||
const harvestCache = new DiscoveryCache(DISCOVERY_HARVEST_CACHE);
|
||||
const discoverySettings = state => ({ harvestEnabled: state.settings?.harvestEnabled ?? true,
|
||||
harvestAt: state.settings?.harvestAt ?? DISCOVERY_HARVEST_AT });
|
||||
const localDate = date => [date.getFullYear(), String(date.getMonth() + 1).padStart(2, '0'), String(date.getDate()).padStart(2, '0')].join('-');
|
||||
function nextNightlyAt(state, now = new Date()) {
|
||||
const { harvestEnabled, harvestAt } = discoverySettings(state);
|
||||
if (!harvestEnabled) return null;
|
||||
const [hours, minutes] = harvestAt.split(':').map(Number);
|
||||
const slot = new Date(now);
|
||||
slot.setHours(hours, minutes, 0, 0);
|
||||
if (state.nightly?.lastDate === localDate(now)) slot.setDate(slot.getDate() + 1);
|
||||
else if (slot < now) return now.toISOString(); // due: the next minute's check starts it
|
||||
return slot.toISOString();
|
||||
}
|
||||
function harvestTargets(kind, userName) {
|
||||
const users = kind === 'user' ? [db.users[userName]].filter(Boolean) : Object.values(db.users);
|
||||
return users.filter(user => settingsOf(user).discoveryEnabled).map(user => ({ user,
|
||||
anchors: anchorsFor({ user, db, library: lib.mangas, greatest: greatestCatalog }), mature: !!settingsOf(user).discoveryMature }));
|
||||
}
|
||||
// Queued jobs live in memory only; a run cut off by a restart is recorded as interrupted.
|
||||
try {
|
||||
const { harvest } = discoveryStore.load();
|
||||
if (harvest?.status === 'running') discoveryStore.patch({ harvest: { ...harvest, status: 'interrupted', finishedAt: new Date().toISOString() } });
|
||||
} catch (error) { console.error('[discovery]', error.message); }
|
||||
const harvester = new Harvester({ cache: harvestCache, client: createAniListClient(), store: discoveryStore,
|
||||
targetsFor: harvestTargets, onDone: async job => {
|
||||
// A failed follow-up rescore must reach the admin panel (lastError), not only the log.
|
||||
try {
|
||||
await rescore(job.kind === 'user' ? { onlyUser: job.user } : {});
|
||||
discoveryError = null; // a later success must not leave an old failure on the admin panel
|
||||
} catch (error) { discoveryError = `Re-scoring after harvest ${job.id} failed: ${String(error.message || error)}`.slice(0, 300); throw error; }
|
||||
} });
|
||||
function requestHarvest(args) {
|
||||
readDiscovery(); // A corrupt store fails closed before any AniList call.
|
||||
try { return harvester.request(args); }
|
||||
catch (error) {
|
||||
if (!error.status) throw error;
|
||||
const extra = {};
|
||||
for (const key of ['nextAllowedAt', 'blockedReason']) if (error[key]) extra[key] = error[key];
|
||||
bad(error.status, error.message, extra);
|
||||
}
|
||||
}
|
||||
function checkNightly() {
|
||||
let state;
|
||||
try { state = discoveryStore.load(); } catch { return; }
|
||||
const now = new Date(), { harvestEnabled, harvestAt } = discoverySettings(state);
|
||||
if (!shouldRunNightly({ now, harvestAt, lastNightlyDate: state.nightly?.lastDate, enabled: harvestEnabled })) return;
|
||||
if (harvester.current()) return;
|
||||
// The date is written only once the run was accepted; a 409 (refresh in flight) retries next minute.
|
||||
try { harvester.request({ kind: 'nightly' }); }
|
||||
catch (error) { if (error.status !== 409) console.error('[discovery] nightly harvest:', error.message); return; }
|
||||
try { discoveryStore.patch({ nightly: { lastDate: localDate(now) } }); }
|
||||
catch (error) { console.error('[discovery]', error.message); }
|
||||
}
|
||||
|
||||
const mangaDexSearchCache = new Map();
|
||||
let lastMangaDexSearch = 0;
|
||||
async function searchGreatestMangaDex(entry) {
|
||||
@@ -322,6 +417,13 @@ function firstReleaseOf(meta) {
|
||||
}
|
||||
return { label: label || '', time: Number.isFinite(time) ? time : 0 };
|
||||
}
|
||||
// When a done chapter was completed: the progress record, or for reads saved before doneAt
|
||||
// existed, the earliest completion the reading clock still remembers.
|
||||
function readAtOf(progress, record) {
|
||||
if (!progress || !progress.done) return null;
|
||||
if (progress.doneAt) return { at: progress.doneAt, approx: false };
|
||||
return earliestCompletion(record);
|
||||
}
|
||||
function continueOf(m, prog, s) {
|
||||
const ok = c => !(s.skipExtras && c.extra);
|
||||
let last = null;
|
||||
@@ -437,7 +539,8 @@ const routes = {
|
||||
if (['last', 'flow', 'page', 'book'].includes(b.startMode)) s.startMode = b.startMode;
|
||||
if (['flow', 'page', 'book'].includes(b.lastMode)) s.lastMode = b.lastMode;
|
||||
if (['original', 'inverted', 'sepia', 'dim', 'extra-dim'].includes(b.pageTone)) s.pageTone = b.pageTone;
|
||||
for (const k of ['adapt', 'skipExtras', 'skipCovers', 'flowPageGap', 'readingClock', 'readingStory',
|
||||
if (['off', 'date', 'hour'].includes(b.readDate)) s.readDate = b.readDate;
|
||||
for (const k of ['adapt', 'skipExtras', 'skipCovers', 'flowPageGap', 'readingClock', 'readingStory', 'releaseDate', 'continueRow',
|
||||
'discoveryEnabled', 'discoveryReadingActivity', 'discoveryMature']) if (typeof b[k] === 'boolean') s[k] = b[k];
|
||||
if (b.zoom && typeof b.zoom === 'object') {
|
||||
s.zoom = {};
|
||||
@@ -485,6 +588,7 @@ const routes = {
|
||||
discoveryReadingActivity: !!settingsOf(u).discoveryReadingActivity,
|
||||
discoveryMature: !!settingsOf(u).discoveryMature },
|
||||
generatedAt: feed?.generatedAt || null, revision: state.revision, lastRun: state.lastRun,
|
||||
refresh: harvester.state(u.name),
|
||||
feedback: db.discoveryFeedback?.[u.name] || {},
|
||||
items: (feed?.items || []).map(item => ({ ...item, candidate: state.candidates[item.candidateId] || null })).filter(item => item.candidate) };
|
||||
},
|
||||
@@ -498,14 +602,30 @@ const routes = {
|
||||
save();
|
||||
return { candidateId: id, value };
|
||||
},
|
||||
'POST /api/discovery/refresh': ({ u }) => {
|
||||
if (!settingsOf(u).discoveryEnabled) bad(403, 'Enable personalized discovery first');
|
||||
return requestHarvest({ kind: 'user', user: u.name });
|
||||
},
|
||||
'GET /api/admin/discovery': ({ u }) => {
|
||||
admin(u);
|
||||
const state = readDiscovery();
|
||||
return { revision: state.revision, lastRun: state.lastRun, currentRun: discoveryRun,
|
||||
lastError: discoveryError, optedInUsers: Object.values(db.users).filter(user => user.settings?.discoveryEnabled).length,
|
||||
candidateCount: Object.keys(state.candidates).length };
|
||||
candidateCount: Object.keys(state.candidates).length,
|
||||
harvest: state.harvest, currentHarvest: harvester.current(), cache: harvestCache.stats(),
|
||||
settings: discoverySettings(state), nextNightlyAt: nextNightlyAt(state) };
|
||||
},
|
||||
'POST /api/admin/discovery/run': ({ u }) => { admin(u); return startDiscoveryRun(); },
|
||||
'POST /api/admin/discovery/harvest': ({ u }) => { admin(u); return requestHarvest({ kind: 'admin' }); },
|
||||
'POST /api/admin/discovery/settings': ({ u, b }) => {
|
||||
admin(u);
|
||||
const keys = b && typeof b === 'object' && !Array.isArray(b) ? Object.keys(b).sort().join() : '';
|
||||
if (keys !== 'harvestAt,harvestEnabled' || typeof b.harvestEnabled !== 'boolean' || typeof b.harvestAt !== 'string' || !HARVEST_TIME.test(b.harvestAt))
|
||||
bad(400, 'Send exactly { harvestEnabled: true|false, harvestAt: "HH:MM" }');
|
||||
readDiscovery();
|
||||
const state = discoveryStore.patch({ settings: { harvestEnabled: b.harvestEnabled, harvestAt: b.harvestAt } });
|
||||
return { settings: discoverySettings(state), nextNightlyAt: nextNightlyAt(state) };
|
||||
},
|
||||
|
||||
'GET /api/greatest': ({ u }) => ({
|
||||
title: greatestCatalog.title, updatedAt: greatestCatalog.updatedAt, periods: greatestCatalog.periods,
|
||||
@@ -515,7 +635,7 @@ const routes = {
|
||||
const counts = greatestCounts(entry.id, db.greatestRatings || {}, db.users);
|
||||
return {
|
||||
...entry,
|
||||
cover: owned?.cover ? url(owned.cover) : entry.cover,
|
||||
cover: owned?.cover ? url(owned.cover) : greatestCoverUrl(entry),
|
||||
chapters: owned ? owned.chapters.length : entry.chapters,
|
||||
status: owned?.meta?.status || entry.status,
|
||||
library: owned ? { id: owned.id, chapters: owned.chapters.length } : null,
|
||||
@@ -557,9 +677,10 @@ const routes = {
|
||||
},
|
||||
|
||||
'GET /api/manga': ({ u, q }) => {
|
||||
const m = mangaOf(q.get('m')), prog = progressOf(u, m), showStory = settingsOf(u).readingStory;
|
||||
const m = mangaOf(q.get('m')), prog = progressOf(u, m), s = settingsOf(u), showStory = s.readingStory;
|
||||
const records = mangaReadingOf(u, m);
|
||||
return {
|
||||
chapterDates: { read: s.readDate, release: s.releaseDate },
|
||||
id: m.id, title: m.title, meta: m.meta, cover: url(m.cover), direction: m.direction,
|
||||
collections: catalog.state.collections.filter(c => c.members.some(member => member.mangaId === m.id)).map(c => ({ id: c.id, title: c.title })),
|
||||
relations: catalog.state.relations.filter(r => r.state === 'confirmed' && (r.from === m.id || r.to === m.id)).map(r => ({ ...r,
|
||||
@@ -569,6 +690,8 @@ const routes = {
|
||||
chapters: m.chapters.map(ch => ({
|
||||
id: ch.id, book: ch.book, label: ch.label, title: ch.title, extra: ch.extra, progress: prog[ch.id] || null, versions: versionsOf(m, ch, u),
|
||||
reading: showStory ? chapterInsight(records[ch.id]) : null,
|
||||
readAt: s.readDate === 'off' ? null : readAtOf(prog[ch.id], records[ch.id]),
|
||||
dates: s.releaseDate ? chapterDates(ch.versions) : null,
|
||||
})),
|
||||
reading: showStory ? mangaInsight(records, m.chapters.map(ch => ch.id)) : null,
|
||||
continue: continueOf(m, prog, settingsOf(u)),
|
||||
@@ -628,6 +751,9 @@ const routes = {
|
||||
prev: neighbour(m, ch, -1, s), next: neighbour(m, ch, 1, s),
|
||||
progress: prog[ch.id] || null,
|
||||
reading: s.readingClock ? publicSummary(readingOf(u, m, ch), v.id, eligiblePages(v, s)) : null,
|
||||
// the user's moments on this version, so the reader can mark pages that already have one
|
||||
moments: momentsOf(u).filter(x => x.m === m.id && x.c === ch.id && x.v === v.id)
|
||||
.map(x => ({ id: x.id, cat: x.cat, pages: (moments.locate(x, m) || { pages: [] }).pages })),
|
||||
// for the chapter picker: the whole manga across all books
|
||||
books: m.books.map(b => ({ key: b.key, label: b.label, title: b.title })),
|
||||
chapters: m.chapters.map(c => ({ id: c.id, book: c.book, label: c.label, title: c.title, extra: c.extra, done: !!(prog[c.id] && prog[c.id].done) })),
|
||||
@@ -637,8 +763,11 @@ const routes = {
|
||||
'POST /api/progress': ({ u, b }) => {
|
||||
const m = mangaOf(b.m), ch = chapterOf(m, b.c);
|
||||
const all = (db.progress[u.name] = db.progress[u.name] || {}), mp = (all[m.id] = all[m.id] || {}), old = mp[ch.id] || {};
|
||||
const now = Date.now(), done = !!(old.done || b.done);
|
||||
// doneAt: when the chapter was first completed; kept until it is marked unread.
|
||||
const doneAt = old.done ? old.doneAt : done ? now : undefined;
|
||||
mp[ch.id] = {
|
||||
v: String(b.v || ''), p: Math.max(0, b.p | 0), n: Math.max(0, b.n | 0), t: Date.now(), done: !!(old.done || b.done),
|
||||
v: String(b.v || ''), p: Math.max(0, b.p | 0), n: Math.max(0, b.n | 0), t: now, done, ...(doneAt ? { doneAt } : {}),
|
||||
shift: Array.isArray(b.shift) ? b.shift.filter(Number.isInteger).slice(0, 1000) : old.shift || [],
|
||||
};
|
||||
save();
|
||||
@@ -711,6 +840,41 @@ const routes = {
|
||||
save();
|
||||
return { following: !!f[m.id] };
|
||||
},
|
||||
|
||||
// Moments (BL-0055): private to their user; one category each, no uncategorised saves.
|
||||
'GET /api/moments': ({ u, q }) => {
|
||||
const m = mangaOf(q.get('m'));
|
||||
return { moments: momentsOf(u).filter(x => x.m === m.id).map(x => moments.view(x, m, url)).sort(moments.storyCompare) };
|
||||
},
|
||||
'POST /api/moments': async ({ u, b }) => {
|
||||
const m = mangaOf(b.m), ch = chapterOf(m, b.c), v = ch.versions.find(x => x.id === b.v) || bad(404, 'Version not found');
|
||||
const pages = moments.cleanPages(b.pages, v);
|
||||
const moment = { id: crypto.randomBytes(6).toString('hex'), m: m.id, c: ch.id, v: v.id, i: pages, f: pages.map(i => v.pages[i].f),
|
||||
layout: moments.cleanLayout(b.layout),
|
||||
region: moments.cleanRegion(b.region), cat: moments.cleanCategory(b.cat), note: moments.cleanNote(b.note), t: Date.now() };
|
||||
const list = momentsOf(u);
|
||||
if (list.length >= moments.MAX_PER_USER) bad(409, `You already have ${moments.MAX_PER_USER} moments.`);
|
||||
moment.sha = await Promise.all(moment.f.map(f => fileSha(v.rel + '/' + f)));
|
||||
list.push(moment);
|
||||
save();
|
||||
return { moment: moments.view(moment, m, url) };
|
||||
},
|
||||
'POST /api/moments/update': ({ u, b }) => {
|
||||
const moment = momentOf(u, b.id);
|
||||
if ('cat' in b) moment.cat = moments.cleanCategory(b.cat);
|
||||
if ('region' in b) moment.region = moments.cleanRegion(b.region);
|
||||
if ('note' in b) moment.note = moments.cleanNote(b.note);
|
||||
moment.edited = Date.now();
|
||||
save();
|
||||
return { moment: moments.view(moment, lib.byId.get(moment.m), url) };
|
||||
},
|
||||
'POST /api/moments/delete': ({ u, b }) => {
|
||||
const list = momentsOf(u), moment = momentOf(u, b.id);
|
||||
list.splice(list.indexOf(moment), 1);
|
||||
save();
|
||||
return { ok: true };
|
||||
},
|
||||
|
||||
'GET /api/notifications': ({ u }) => ({ items: (db.notes[u.name] || []).slice().reverse() }),
|
||||
'POST /api/notifications/read': ({ u, b }) => {
|
||||
for (const n of db.notes[u.name] || []) if (b.all || n.id === b.id) n.read = true;
|
||||
@@ -752,7 +916,7 @@ const routes = {
|
||||
admin(u);
|
||||
const name = String(b.name || '');
|
||||
if (name === u.name) bad(400, "You can't delete your own account.");
|
||||
delete db.users[name]; delete db.progress[name]; delete db.reading[name]; delete db.follows[name]; delete db.notes[name];
|
||||
delete db.users[name]; delete db.progress[name]; delete db.reading[name]; delete db.follows[name]; delete db.notes[name]; delete db.moments[name];
|
||||
if (db.discoveryFeedback) delete db.discoveryFeedback[name];
|
||||
if (db.greatestRatings) delete db.greatestRatings[name];
|
||||
for (const [k, s] of Object.entries(db.sessions)) if (s.u === name) delete db.sessions[k];
|
||||
@@ -923,6 +1087,16 @@ http.createServer(async (req, res) => {
|
||||
if (!c?.customCover) bad(404, 'Cover not found');
|
||||
return serveFile(req, res, path.join(DATA, 'collection-covers'), c.customCover, 'private, no-cache', IMG);
|
||||
}
|
||||
if (pathname === '/api/greatest-cover' && req.method === 'GET') {
|
||||
if (!u) bad(401, 'Please log in');
|
||||
const entry = greatestById.get(searchParams.get('id'));
|
||||
if (!entry?.cover) bad(404, 'Cover not found');
|
||||
const file = greatestCovers.file(entry.cover);
|
||||
if (file) return serveFile(req, res, greatestCovers.dir, path.basename(file), 'private, max-age=31536000, immutable', IMG);
|
||||
// Not downloaded yet: the browser fetches the catalog's own URL this once.
|
||||
res.writeHead(302, { Location: entry.cover, 'Cache-Control': 'no-store' });
|
||||
return res.end();
|
||||
}
|
||||
if (pathname === '/api/peering/cover' && req.method === 'GET') {
|
||||
if (!u) bad(401, 'Please log in');
|
||||
admin(u);
|
||||
@@ -954,15 +1128,27 @@ http.createServer(async (req, res) => {
|
||||
}
|
||||
serveFile(req, res, PUB, pathname === '/' ? 'index.html' : pathname.slice(1), 'no-cache');
|
||||
} catch (e) {
|
||||
const expected = e instanceof HttpError || e instanceof CatalogError;
|
||||
const expected = e instanceof HttpError || e instanceof CatalogError || e instanceof moments.MomentError;
|
||||
if (!expected) console.error(e);
|
||||
if (!res.headersSent) sendJson(res, expected ? e.code : 500, { error: expected ? e.message : 'Internal error' });
|
||||
if (!res.headersSent) sendJson(res, expected ? e.code : 500, expected ? { error: e.message, ...e.extra } : { error: 'Internal error' });
|
||||
}
|
||||
}).listen(PORT, () => console.log(`MangaCircus on http://localhost:${PORT} (library: ${LIB}, data: ${DATA})`));
|
||||
|
||||
if (lib.mangas.length) refreshChanged();
|
||||
else rescan();
|
||||
if (SCAN_MINUTES > 0) setInterval(() => refreshChanged(), SCAN_MINUTES * 60000).unref();
|
||||
// Fetch missing Greatest covers shortly after start and retry failures once a day;
|
||||
// covers already on disk cost nothing.
|
||||
async function warmGreatestCovers() {
|
||||
try {
|
||||
const result = await greatestCovers.warm(greatestCatalog.entries.map(entry => entry.cover));
|
||||
if (result.fetched || result.failed || result.stopped) console.log('[greatest] covers', JSON.stringify(result));
|
||||
} catch (error) { console.error('[greatest] covers', error.message); }
|
||||
}
|
||||
if (process.env.GREATEST_COVER_WARM !== '0') {
|
||||
setTimeout(warmGreatestCovers, 10000).unref();
|
||||
setInterval(warmGreatestCovers, 86400000).unref();
|
||||
}
|
||||
// Persist the last reconciliation time. Restarting a healthy reader should not
|
||||
// trigger another full tree walk; overdue reconciliations run within a minute.
|
||||
if (FULL_SCAN_MINUTES > 0) setInterval(() => {
|
||||
@@ -995,3 +1181,7 @@ async function pollScraper() {
|
||||
}
|
||||
pollScraper();
|
||||
setInterval(pollScraper, 15000).unref();
|
||||
|
||||
// Nightly discovery harvest; the delayed first check also covers a slot missed while the reader was down.
|
||||
setInterval(checkNightly, 60000).unref();
|
||||
setTimeout(checkNightly, 30000).unref();
|
||||
@@ -21,7 +21,8 @@ test('only admins can promote or demote, nobody can demote themselves, and chang
|
||||
const port = await freePort(), output = [];
|
||||
child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
|
||||
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
|
||||
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '' } });
|
||||
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '',
|
||||
GREATEST_COVER_WARM: '0' } });
|
||||
child.stdout.on('data', d => output.push(String(d)));
|
||||
child.stderr.on('data', d => output.push(String(d)));
|
||||
const base = `http://127.0.0.1:${port}`;
|
||||
|
||||
+1
-1
@@ -251,7 +251,7 @@ test('reader and scraper: admin backup API pauses and resumes the real queue', {
|
||||
const base = `http://127.0.0.1:${rp}`, token = 'backup-test-only';
|
||||
const env = { ...process.env, DATA: f.data, LIBRARY: library, SCRAPER_DATA: path.join(f.data, 'scraper'), BACKUP: f.target,
|
||||
PORT: String(rp), SCRAPER_PORT: String(sp), SCRAPER_HOST: '127.0.0.1', SCRAPER_URL: `http://127.0.0.1:${sp}`, SENTINEL_URL: 'http://127.0.0.1:9',
|
||||
SCRAPER_TOKEN: token, SCRAPER_TOKEN_FILE: '', SCAN_INTERVAL: '0' };
|
||||
SCRAPER_TOKEN: token, SCRAPER_TOKEN_FILE: '', SCAN_INTERVAL: '0', GREATEST_COVER_WARM: '0' };
|
||||
for (const script of ['server.js', 'scraper/daemon.js']) {
|
||||
const child = spawn(process.execPath, [script], { cwd: root, env, windowsHide: true }); children.push(child);
|
||||
for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => output.push(chunk.toString()));
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { batchFlags, publishedDates, chapterDates, peerDates, fromPeer } = require('../chapter-dates');
|
||||
const { earliestCompletion } = require('../reading-analytics');
|
||||
|
||||
test('bulk imports are flagged, weekly uploads are not', () => {
|
||||
const flags = batchFlags([
|
||||
'2024-09-07T10:00:00Z', '2024-09-07T10:00:03Z', '2024-09-07T10:04:00Z', // one import
|
||||
'2026-10-11T15:05:58Z', '2026-10-18T15:06:05Z', // weekly releases
|
||||
'2026-10-20T09:00:00Z', '2026-10-20T09:01:00Z', // a double release stays a release
|
||||
undefined,
|
||||
]);
|
||||
assert.deepEqual(flags, [true, true, true, false, false, false, false, null]);
|
||||
});
|
||||
|
||||
test('publishing keeps the first arrival and the earliest ecosystem date', () => {
|
||||
const now = Date.parse('2026-10-24T12:00:00Z');
|
||||
const fresh = publishedDates({}, { source: { at: '2026-10-18T15:06:05Z', name: 'WeebCentral', batch: false } }, now);
|
||||
assert.equal(fresh.added, '2026-10-24T12:00:00.000Z');
|
||||
assert.deepEqual(fresh['x-dates'].ecosystem, { at: '2026-10-24T12:00:00.000Z', via: 'this server' });
|
||||
assert.equal(fresh['x-dates'].source.name, 'WeebCentral');
|
||||
|
||||
// Replacing a version later: "added" and the ecosystem date survive.
|
||||
const replaced = publishedDates(fresh, {}, now + 864e5);
|
||||
assert.equal(replaced.added, fresh.added);
|
||||
assert.deepEqual(replaced['x-dates'].ecosystem, fresh['x-dates'].ecosystem);
|
||||
assert.equal(replaced['x-dates'].source.at, '2026-10-18T15:06:05.000Z');
|
||||
|
||||
// A peer that had it earlier lowers the date; a later claim never raises it.
|
||||
const earlier = publishedDates(fresh, { ecosystem: { at: '2026-10-19T08:00:00Z', via: 'peer bob' } }, now);
|
||||
assert.deepEqual(earlier['x-dates'].ecosystem, { at: '2026-10-19T08:00:00.000Z', via: 'peer bob' });
|
||||
const later = publishedDates(earlier, { ecosystem: { at: '2026-10-23T00:00:00Z', via: 'peer eve' } }, now);
|
||||
assert.equal(later['x-dates'].ecosystem.via, 'peer bob');
|
||||
});
|
||||
|
||||
test('peer ecosystem claims cannot predate the source upload or lie in the future', () => {
|
||||
const now = Date.parse('2026-10-24T12:00:00Z');
|
||||
const source = { at: '2026-10-18T15:06:05Z', name: 'WeebCentral', batch: false };
|
||||
const early = publishedDates({}, { source, ecosystem: { at: '2001-01-01T00:00:00Z', via: 'peer x' } }, now);
|
||||
assert.equal(early['x-dates'].ecosystem.at, '2026-10-18T15:06:05.000Z');
|
||||
const future = publishedDates({}, { ecosystem: { at: '2099-01-01T00:00:00Z', via: 'peer x' } }, now);
|
||||
assert.equal(future['x-dates'].ecosystem.at, '2026-10-24T12:00:00.000Z');
|
||||
});
|
||||
|
||||
test('release date tiers: official, source upload, ecosystem, library', () => {
|
||||
const added = Date.parse('2026-10-17T15:04:03Z');
|
||||
const version = dates => ({ added, meta: { 'x-dates': dates } });
|
||||
const ecosystem = { at: '2026-10-10T00:00:00Z', via: 'peer bob' };
|
||||
assert.equal(chapterDates([version({})]).tier, 'library');
|
||||
assert.equal(chapterDates([version({ ecosystem })]).release.at, '2026-10-10T00:00:00.000Z');
|
||||
const batch = chapterDates([version({ ecosystem, source: { at: '2024-09-07T10:00:00Z', name: 'WeebCentral', batch: true } })]);
|
||||
assert.equal(batch.tier, 'ecosystem', 'a bulk upload is no release date');
|
||||
assert.equal(batch.known.batch.at, '2024-09-07T10:00:00.000Z');
|
||||
const source = chapterDates([version({ ecosystem }), version({ source: { at: '2026-10-04T15:06:05Z', name: 'MangaDex', batch: false } })]);
|
||||
assert.equal(source.tier, 'source');
|
||||
assert.equal(source.known.ecosystem.at, '2026-10-10T00:00:00.000Z', 'earliest across versions');
|
||||
const official = chapterDates([version({ release: { at: '2023-09-17', precision: 'day', source: 'MangaUpdates · Viz' }, ecosystem })]);
|
||||
assert.deepEqual(official.release, { at: '2023-09-17', precision: 'day', name: 'MangaUpdates · Viz' });
|
||||
assert.equal(official.tier, 'official');
|
||||
});
|
||||
|
||||
test('peers exchange cleaned dates and the receiver records the peer', () => {
|
||||
const meta = { added: '2026-10-17T15:04:03Z', secret: 'x',
|
||||
'x-dates': { source: { at: '2026-10-18T15:06:05Z', name: 'WeebCentral', batch: false, extra: 'drop' } } };
|
||||
const sent = peerDates(meta, 0);
|
||||
assert.deepEqual(sent, { ecosystem: '2026-10-17T15:04:03.000Z', source: { at: '2026-10-18T15:06:05.000Z', name: 'WeebCentral', batch: false } });
|
||||
const received = fromPeer(sent, 'bob');
|
||||
assert.deepEqual(received.ecosystem, { at: '2026-10-17T15:04:03.000Z', via: 'peer bob' });
|
||||
assert.deepEqual(fromPeer('nonsense', 'bob'), {});
|
||||
assert.deepEqual(fromPeer({ ecosystem: 'not a date' }, 'bob'), {});
|
||||
});
|
||||
|
||||
test('legacy reads get the earliest completion the reading clock remembers', () => {
|
||||
assert.equal(earliestCompletion(null), null);
|
||||
const record = { completedPasses: 3, currentPass: { completedAt: 5000 },
|
||||
recentSessions: [{ pass: { completedAt: 3000 } }, { pass: { completedAt: null } }, { pass: { completedAt: 5000 } }] };
|
||||
assert.deepEqual(earliestCompletion(record), { at: 3000, approx: true });
|
||||
assert.deepEqual(earliestCompletion({ ...record, completedPasses: 2 }), { at: 3000, approx: false });
|
||||
});
|
||||
@@ -0,0 +1,73 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const { anchorsFor, topTagsFor, ANCHOR_WEIGHTS, MAX_ANCHORS } = require('../discovery/anchors');
|
||||
|
||||
const user = (settings = { discoveryEnabled: true }) => ({ name: 'ann', settings });
|
||||
const manga = (id, anilist, title = `Manga ${id}`) => ({ id, title, meta: anilist ? { 'x-anilist': { id: anilist } } : {} });
|
||||
|
||||
test('anchors are empty without opt-in', () => {
|
||||
const library = [manga('a', 1)];
|
||||
assert.deepEqual(anchorsFor({ user: user({}), db: {}, library, greatest: { entries: [] } }), []);
|
||||
assert.deepEqual(anchorsFor({ user: user({ discoveryEnabled: false }), db: {}, library, greatest: { entries: [] } }), []);
|
||||
});
|
||||
|
||||
test('weights, dedupe and cap', () => {
|
||||
const greatest = { entries: [{ id: 'g1', title: 'Great One', metadata: { id: 7 } }, { id: 'g2', title: 'No Meta' }] };
|
||||
const db = { greatestRatings: { ann: { g1: 'good', g2: 'flame' } } };
|
||||
const anchors = anchorsFor({ user: user(), db, library: [manga('m7', 7, 'Owned Seven')], greatest });
|
||||
assert.deepEqual(anchors, [{ anilistId: 7, weight: 3, source: 'good', title: 'Great One' }]);
|
||||
assert.equal(ANCHOR_WEIGHTS.flame, 5);
|
||||
const many = Array.from({ length: 20 }, (_, i) => manga(`m${i}`, i + 1));
|
||||
const capped = anchorsFor({ user: user(), db: {}, library: many, greatest: { entries: [] } });
|
||||
assert.equal(capped.length, MAX_ANCHORS);
|
||||
assert.equal(capped.length, 15);
|
||||
assert.deepEqual(capped.map(a => a.anilistId), Array.from({ length: 15 }, (_, i) => i + 1));
|
||||
assert.equal(capped[0].source, 'owned');
|
||||
});
|
||||
|
||||
test('reading anchors need the reading opt-in', () => {
|
||||
const library = [manga('m1', 1), manga('m2', 2)];
|
||||
const db = { progress: { ann: { m1: { c1: { done: true } }, m2: { c1: { done: false } } } } };
|
||||
const off = anchorsFor({ user: user(), db, library, greatest: { entries: [] } });
|
||||
assert.deepEqual(off.map(a => [a.anilistId, a.source]), [[1, 'owned'], [2, 'owned']]);
|
||||
const on = anchorsFor({ user: user({ discoveryEnabled: true, discoveryReadingActivity: true }), db, library, greatest: { entries: [] } });
|
||||
assert.deepEqual(on.map(a => [a.anilistId, a.source, a.weight, a.title]), [[1, 'read', 3, 'Manga m1'], [2, 'owned', 1, 'Manga m2']]);
|
||||
});
|
||||
|
||||
test('feedback keys parse anilist ids and ignore others', () => {
|
||||
const db = { discoveryFeedback: { ann: {
|
||||
'anilist:10': 'interested', 'anilist:11': 'maybe', 'anilist:12': 'dismissed', 'anilist:13': 'hidden',
|
||||
'greatest:x': 'interested', 'anilist:abc': 'interested' } } };
|
||||
const anchors = anchorsFor({ user: user(), db, library: [], greatest: { entries: [] } });
|
||||
assert.deepEqual(anchors, [
|
||||
{ anilistId: 10, weight: 4, source: 'interested', title: null },
|
||||
{ anilistId: 11, weight: 2, source: 'maybe', title: null }]);
|
||||
});
|
||||
|
||||
test('top tags weigh anchors, skip low rank and adult tags', () => {
|
||||
const records = {
|
||||
1: { tags: [{ name: 'Action', rank: 100, kind: 'genre', isAdult: false }, { name: 'Ecchi', rank: 90, kind: 'tag', isAdult: true },
|
||||
{ name: 'Faint', rank: 59, kind: 'tag', isAdult: false }] },
|
||||
2: { tags: [{ name: 'Action', rank: 100, kind: 'genre', isAdult: false }, { name: 'Zeta', rank: 60, kind: 'tag', isAdult: false },
|
||||
{ name: 'Alpha', rank: 80, kind: 'tag', isAdult: false }] }
|
||||
};
|
||||
const anchors = [{ anilistId: 1, weight: 5 }, { anilistId: 2, weight: 3 }, { anilistId: 3, weight: 5 }];
|
||||
const recordOf = id => records[id] || null;
|
||||
assert.deepEqual(topTagsFor(anchors, recordOf), [
|
||||
{ name: 'Action', kind: 'genre' }, { name: 'Alpha', kind: 'tag' }, { name: 'Zeta', kind: 'tag' }]);
|
||||
assert.deepEqual(topTagsFor(anchors, recordOf, { mature: true }), [
|
||||
{ name: 'Action', kind: 'genre' }, { name: 'Ecchi', kind: 'tag' }, { name: 'Alpha', kind: 'tag' }]);
|
||||
});
|
||||
|
||||
test('top tags drop tags without a rank and survive malformed records', () => {
|
||||
const records = {
|
||||
1: { tags: [{ name: 'NoRank', kind: 'tag', isAdult: false }, { name: 'NullRank', rank: null, kind: 'tag', isAdult: false },
|
||||
null, 7, { name: 'Kept', rank: 60, kind: 'tag', isAdult: false }] },
|
||||
2: { tags: 5 },
|
||||
3: { tags: { name: 'Object', rank: 90 } },
|
||||
};
|
||||
const anchors = [{ anilistId: 1, weight: 5 }, { anilistId: 2, weight: 3 }, { anilistId: 3, weight: 3 }];
|
||||
assert.deepEqual(topTagsFor(anchors, id => records[id]), [{ name: 'Kept', kind: 'tag' }]);
|
||||
});
|
||||
@@ -0,0 +1,175 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const path = require('node:path');
|
||||
const { createLimiter, createClient, normalizeMedia, AniListStop, AniListSkip } = require('../discovery/anilist');
|
||||
|
||||
const fixture = name => JSON.parse(fs.readFileSync(path.join(__dirname, 'fixtures', 'anilist', `${name}.json`), 'utf8'));
|
||||
const json = (body, status = 200) => new Response(JSON.stringify(body), { status, headers: { 'content-type': 'application/json' } });
|
||||
const noWait = { wait: async () => {} };
|
||||
|
||||
function stub(respond) {
|
||||
const calls = [];
|
||||
const fetchImpl = async (url, init) => { calls.push({ url, init, body: JSON.parse(init.body) }); return respond(); };
|
||||
return { calls, client: createClient({ endpoint: 'http://localhost/graphql', limiter: noWait, fetchImpl, today: () => new Date('2026-10-25') }) };
|
||||
}
|
||||
const clientFor = respond => stub(respond).client;
|
||||
|
||||
test('normalizeMedia maps fields and drops novels', () => {
|
||||
const media = fixture('title').data.Media;
|
||||
assert.deepEqual(normalizeMedia(media), {
|
||||
anilistId: 101, title: 'Moon Traveller', altTitles: ['Tsuki no Tabibito', '月の旅人', 'Lunar Wanderer'],
|
||||
format: 'MANGA', isAdult: false, status: 'ongoing', year: 2019, averageScore: 81, popularity: 5400, trending: 37,
|
||||
tags: [
|
||||
{ name: 'Adventure', rank: 100, kind: 'genre', isAdult: false },
|
||||
{ name: 'Fantasy', rank: 100, kind: 'genre', isAdult: false },
|
||||
{ name: 'Travel', rank: 88, kind: 'tag', isAdult: false },
|
||||
{ name: 'Nudity', rank: 40, kind: 'tag', isAdult: true }],
|
||||
authors: [{ staffId: 900, name: 'Aoi Example', role: 'Story & Art' }],
|
||||
coverUrl: 'https://example.invalid/cover/101.jpg', siteUrl: 'https://anilist.co/manga/101' });
|
||||
assert.equal(normalizeMedia({ ...media, format: 'NOVEL' }), null);
|
||||
assert.equal(normalizeMedia(fixture('novel').data.Media), null);
|
||||
assert.equal(normalizeMedia({ ...media, status: 'HIATUS' }).status, 'unknown');
|
||||
});
|
||||
|
||||
test('normalizeMedia drops authors without an integer staff id', () => {
|
||||
const media = fixture('title').data.Media;
|
||||
const edge = (id, full) => ({ role: 'Story', node: { id, name: { full } } });
|
||||
const record = normalizeMedia({ ...media, staff: { edges: [edge(5, 'Has Id'), edge(undefined, 'No Id'),
|
||||
edge('6', 'String Id'), edge(1.5, 'Float Id'), edge(null, 'Null Id')] } });
|
||||
assert.deepEqual(record.authors, [{ staffId: 5, name: 'Has Id', role: 'Story' }]);
|
||||
});
|
||||
|
||||
test('title returns a record and sends the fixed headers', async () => {
|
||||
const { client, calls } = stub(() => json(fixture('title')));
|
||||
const record = await client.title(101);
|
||||
assert.equal(record.anilistId, 101);
|
||||
assert.equal(calls[0].body.variables.id, 101);
|
||||
assert.equal(calls[0].init.headers['user-agent'], 'MangaCircus-Discovery/1.0');
|
||||
assert.equal(calls[0].init.headers['content-type'], 'application/json');
|
||||
});
|
||||
|
||||
test('recs keeps positive ratings and manga formats', async () => {
|
||||
const out = await clientFor(() => json(fixture('recs'))).recs(101);
|
||||
assert.equal(out.length, 1);
|
||||
assert.equal(out[0].anilistId, 201);
|
||||
assert.equal(out[0].rating, 12);
|
||||
assert.equal(out[0].record.title, 'Sky Town');
|
||||
});
|
||||
|
||||
test('recs query follows spec sort and page size', async () => {
|
||||
const { client, calls } = stub(() => json(fixture('recs')));
|
||||
await client.recs(101);
|
||||
assert.match(calls[0].body.query, /recommendations\(perPage: 25, sort: RATING_DESC\)/);
|
||||
});
|
||||
|
||||
test('creator query follows spec sort and page size', async () => {
|
||||
const { client, calls } = stub(() => json(fixture('creator')));
|
||||
await client.creator(900);
|
||||
assert.match(calls[0].body.query, /staffMedia\(type: MANGA, perPage: 25, sort: POPULARITY_DESC\)/);
|
||||
});
|
||||
|
||||
test('title on a novel resolves to null', async () => {
|
||||
assert.equal(await clientFor(() => json(fixture('novel'))).title(777), null);
|
||||
});
|
||||
|
||||
test('requests refuse redirects', async () => {
|
||||
const { client, calls } = stub(() => json(fixture('title')));
|
||||
await client.title(101);
|
||||
assert.equal(calls[0].init.redirect, 'error');
|
||||
const redirected = () => { throw new TypeError('fetch failed: unexpected redirect'); };
|
||||
await assert.rejects(clientFor(redirected).title(1), e => e instanceof AniListSkip && e.reason === 'upstream');
|
||||
});
|
||||
|
||||
test('recs drops recommendations whose media was removed', async () => {
|
||||
const recs = fixture('recs');
|
||||
recs.data.Media.recommendations.nodes = [recs.data.Media.recommendations.nodes[0], { rating: 10, mediaRecommendation: null }];
|
||||
const out = await clientFor(() => json(recs)).recs(101);
|
||||
assert.deepEqual(out.map(e => e.anilistId), [201]);
|
||||
});
|
||||
|
||||
test('one malformed item invalidates a list response', async () => {
|
||||
const invalid = e => e instanceof AniListSkip && e.reason === 'invalid';
|
||||
const recs = fixture('recs');
|
||||
recs.data.Media.recommendations.nodes.push(null);
|
||||
await assert.rejects(clientFor(() => json(recs)).recs(1), invalid);
|
||||
const tagtop = fixture('tagtop');
|
||||
tagtop.data.Page.media[0].id = 'x';
|
||||
await assert.rejects(clientFor(() => json(tagtop)).tagTop({ name: 'Mystery', kind: 'genre' }), invalid);
|
||||
for (const patch of [{ genres: 'Drama' }, { tags: [null] }, { title: 'x' }, { staff: { edges: {} } }, { staff: { edges: [null] } }]) {
|
||||
const fresh = fixture('fresh');
|
||||
Object.assign(fresh.data.Page.media[0], patch);
|
||||
await assert.rejects(clientFor(() => json(fresh)).fresh(), invalid, JSON.stringify(patch));
|
||||
}
|
||||
await assert.rejects(async () => normalizeMedia({ id: 1, format: 'MANGA', title: 5 }), invalid);
|
||||
});
|
||||
|
||||
test('creator returns roles and drops novels', async () => {
|
||||
const out = await clientFor(() => json(fixture('creator'))).creator(900);
|
||||
assert.deepEqual(out.map(e => [e.anilistId, e.role]), [[301, 'Story & Art']]);
|
||||
});
|
||||
|
||||
test('fresh query sends startDate two years back', async () => {
|
||||
const { client, calls } = stub(() => json(fixture('fresh')));
|
||||
const out = await client.fresh();
|
||||
assert.equal(calls[0].body.variables.startDate_greater, 20241025);
|
||||
assert.equal(calls[0].body.variables.perPage, 50);
|
||||
assert.match(calls[0].body.query, /TRENDING_DESC/);
|
||||
assert.deepEqual(out.map(e => e.anilistId), [501]);
|
||||
});
|
||||
|
||||
test('tagTop uses genre_in for genres and tag_in for tags', async () => {
|
||||
const g = stub(() => json(fixture('tagtop')));
|
||||
const out = await g.client.tagTop({ name: 'Mystery', kind: 'genre' });
|
||||
assert.deepEqual(g.calls[0].body.variables.genre_in, ['Mystery']);
|
||||
assert.equal(g.calls[0].body.variables.tag_in, undefined);
|
||||
assert.match(g.calls[0].body.query, /genre_in: \$genre_in, popularity_greater: 1000, sort: SCORE_DESC/);
|
||||
assert.match(g.calls[0].body.query, /Page\(perPage: 25\)/);
|
||||
assert.deepEqual(out.map(e => e.anilistId), [401]);
|
||||
const t = stub(() => json(fixture('tagtop')));
|
||||
await t.client.tagTop({ name: 'Detective', kind: 'tag' });
|
||||
assert.match(t.calls[0].body.query, /tag_in: \$tag_in, popularity_greater: 1000, sort: SCORE_DESC/);
|
||||
assert.deepEqual(t.calls[0].body.variables.tag_in, ['Detective']);
|
||||
assert.equal(t.calls[0].body.variables.genre_in, undefined);
|
||||
});
|
||||
|
||||
test('429 stops, 403 stops, html body is a challenge', async () => {
|
||||
await assert.rejects(clientFor(() => json({}, 429)).title(1), e => e instanceof AniListStop && e.reason === 'rate-limited');
|
||||
await assert.rejects(clientFor(() => json({}, 403)).title(1), e => e instanceof AniListStop && e.reason === 'forbidden');
|
||||
await assert.rejects(clientFor(() => json({}, 401)).title(1), e => e instanceof AniListStop && e.reason === 'forbidden');
|
||||
const html = () => new Response('<html>checking your browser</html>', { status: 200, headers: { 'content-type': 'text/html' } });
|
||||
await assert.rejects(clientFor(html).title(1), e => e instanceof AniListStop && e.reason === 'challenge');
|
||||
const garbage = () => new Response('not json', { status: 200 });
|
||||
await assert.rejects(clientFor(garbage).title(1), e => e instanceof AniListStop && e.reason === 'challenge');
|
||||
});
|
||||
|
||||
test('5xx and timeout skip', async () => {
|
||||
await assert.rejects(clientFor(() => json({}, 503)).title(1), e => e instanceof AniListSkip && e.reason === 'upstream');
|
||||
// A challenge page served with a 5xx is still a challenge, not an outage.
|
||||
const html503 = () => new Response('<html>just a moment</html>', { status: 503, headers: { 'content-type': 'text/html' } });
|
||||
await assert.rejects(clientFor(html503).title(1), e => e instanceof AniListStop && e.reason === 'challenge');
|
||||
const text502 = () => new Response('Bad Gateway', { status: 502 });
|
||||
await assert.rejects(clientFor(text502).title(1), e => e instanceof AniListStop && e.reason === 'challenge');
|
||||
const timeout = () => { throw Object.assign(new Error('timed out'), { name: 'TimeoutError' }); };
|
||||
await assert.rejects(clientFor(timeout).title(1), e => e instanceof AniListSkip && e.reason === 'timeout');
|
||||
const abort = () => { throw Object.assign(new Error('aborted'), { name: 'AbortError' }); };
|
||||
await assert.rejects(clientFor(abort).title(1), e => e instanceof AniListSkip && e.reason === 'timeout');
|
||||
});
|
||||
|
||||
test('missing data is invalid', async () => {
|
||||
const invalid = e => e instanceof AniListSkip && e.reason === 'invalid';
|
||||
await assert.rejects(clientFor(() => json({ data: {} })).title(1), invalid);
|
||||
await assert.rejects(clientFor(() => json({ data: { Page: {} } })).fresh(), invalid);
|
||||
await assert.rejects(clientFor(() => json({ data: {} })).creator(1), invalid);
|
||||
await assert.rejects(clientFor(() => json({ data: { Media: { ...fixture('title').data.Media, id: 'x' } } })).title(1), invalid);
|
||||
});
|
||||
|
||||
test('limiter spaces calls 2400 ms apart', async () => {
|
||||
let clock = 1000;
|
||||
const sleeps = [];
|
||||
const limiter = createLimiter({ perMinute: 25, now: () => clock, sleep: async ms => { sleeps.push(ms); clock += ms; } });
|
||||
await Promise.all([limiter.wait(), limiter.wait(), limiter.wait()]);
|
||||
assert.deepEqual(sleeps, [0, 2400, 2400]);
|
||||
});
|
||||
@@ -0,0 +1,116 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { DiscoveryCache } = require('../discovery/cache');
|
||||
|
||||
const DAY = 86400000;
|
||||
|
||||
function setup(t, options = {}) {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'circus-dcache-'));
|
||||
t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
|
||||
const clock = { t: Date.UTC(2026, 0, 1) };
|
||||
const cache = new DiscoveryCache(dir, { ...options, now: () => clock.t });
|
||||
return { dir, clock, cache };
|
||||
}
|
||||
|
||||
test('cache round-trips an entry', (t) => {
|
||||
const { clock, cache } = setup(t);
|
||||
cache.put('title', 42, { a: 1 });
|
||||
const entry = cache.get('title', 42);
|
||||
assert.deepEqual(entry.data, { a: 1 });
|
||||
assert.equal(entry.fetchedAt, new Date(clock.t).toISOString());
|
||||
assert.equal(entry.schema, 1);
|
||||
assert.equal(entry.kind, 'title');
|
||||
});
|
||||
|
||||
test('freshness follows TTL per kind', (t) => {
|
||||
const { clock, cache } = setup(t);
|
||||
cache.put('fresh', 'global', []);
|
||||
cache.put('title', 1, {});
|
||||
clock.t += 8 * DAY;
|
||||
assert.equal(cache.isFresh('fresh', 'global'), false);
|
||||
assert.equal(cache.isFresh('title', 1), true);
|
||||
clock.t += 23 * DAY;
|
||||
assert.equal(cache.isFresh('title', 1), false);
|
||||
assert.notEqual(cache.get('title', 1), null);
|
||||
assert.equal(cache.list('title').length, 1);
|
||||
});
|
||||
|
||||
test('prune keeps the newest maxFiles', (t) => {
|
||||
const { clock, cache } = setup(t, { maxFiles: 3 });
|
||||
for (let i = 1; i <= 5; i++) { clock.t += 1000; cache.put('title', i, { i }); }
|
||||
assert.equal(cache.stats().files, 3);
|
||||
assert.equal(cache.get('title', 1), null);
|
||||
assert.equal(cache.get('title', 2), null);
|
||||
assert.notEqual(cache.get('title', 5), null);
|
||||
});
|
||||
|
||||
test('prune removes the oldest entries across kinds', (t) => {
|
||||
const { clock, cache } = setup(t, { maxFiles: 3 });
|
||||
const kinds = ['title', 'recs', 'creator', 'fresh', 'tagTop'];
|
||||
kinds.forEach((kind, i) => { clock.t += 1000; cache.put(kind, i, { i }); });
|
||||
assert.equal(cache.stats().files, 3);
|
||||
assert.equal(cache.get('title', 0), null);
|
||||
assert.equal(cache.get('recs', 1), null);
|
||||
for (const [i, kind] of kinds.entries()) if (i >= 2) assert.notEqual(cache.get(kind, i), null, kind);
|
||||
});
|
||||
|
||||
test('a write under the cap does not read other entries', (t) => {
|
||||
const { dir, cache } = setup(t, { maxFiles: 10 });
|
||||
cache.put('recs', 7, {});
|
||||
const file = path.join(dir, 'recs', fs.readdirSync(path.join(dir, 'recs'))[0]);
|
||||
fs.writeFileSync(file, '{broken');
|
||||
cache.put('title', 1, {});
|
||||
assert.equal(fs.readFileSync(file, 'utf8'), '{broken');
|
||||
});
|
||||
|
||||
test('over the cap, corrupt and schema-mismatch entries go before valid ones', (t) => {
|
||||
const { dir, clock, cache } = setup(t, { maxFiles: 2 });
|
||||
cache.put('recs', 1, {});
|
||||
cache.put('recs', 2, {});
|
||||
const [first, second] = fs.readdirSync(path.join(dir, 'recs')).map(name => path.join(dir, 'recs', name));
|
||||
fs.writeFileSync(first, '{broken');
|
||||
fs.writeFileSync(second, JSON.stringify({ schema: 2, kind: 'recs', key: 2, fetchedAt: new Date(clock.t).toISOString(), data: {} }));
|
||||
clock.t += 1000;
|
||||
cache.put('title', 3, {});
|
||||
assert.equal(fs.existsSync(first), false);
|
||||
assert.equal(fs.existsSync(second), false);
|
||||
assert.notEqual(cache.get('title', 3), null);
|
||||
});
|
||||
|
||||
test('schema-mismatch entry is deleted like a corrupt one', (t) => {
|
||||
const { dir, clock, cache } = setup(t);
|
||||
cache.put('recs', 7, {});
|
||||
const file = path.join(dir, 'recs', fs.readdirSync(path.join(dir, 'recs'))[0]);
|
||||
fs.writeFileSync(file, JSON.stringify({ schema: 2, kind: 'recs', key: 7, fetchedAt: new Date(clock.t).toISOString(), data: {} }));
|
||||
assert.equal(cache.get('recs', 7), null);
|
||||
assert.equal(fs.existsSync(file), false);
|
||||
});
|
||||
|
||||
test('corrupt entry is deleted', (t) => {
|
||||
const { dir, cache } = setup(t);
|
||||
cache.put('recs', 7, {});
|
||||
const file = path.join(dir, 'recs', fs.readdirSync(path.join(dir, 'recs'))[0]);
|
||||
fs.writeFileSync(file, '{broken');
|
||||
assert.equal(cache.get('recs', 7), null);
|
||||
assert.equal(fs.existsSync(file), false);
|
||||
});
|
||||
|
||||
test('odd keys are file-safe', (t) => {
|
||||
const { dir, cache } = setup(t);
|
||||
cache.put('tagTop', 'genre:Slice of Life/../x', [1]);
|
||||
const files = fs.readdirSync(path.join(dir, 'tagTop'));
|
||||
assert.equal(files.length, 1);
|
||||
assert.deepEqual(fs.readdirSync(dir), ['tagTop']);
|
||||
assert.deepEqual(cache.get('tagTop', 'genre:Slice of Life/../x').data, [1]);
|
||||
});
|
||||
|
||||
test('unknown kind throws', (t) => {
|
||||
const { cache } = setup(t);
|
||||
assert.throws(() => cache.put('nope', 1, {}), TypeError);
|
||||
assert.throws(() => cache.get('nope', 1), TypeError);
|
||||
});
|
||||
@@ -0,0 +1,342 @@
|
||||
'use strict';
|
||||
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { DiscoveryCache } = require('../discovery/cache');
|
||||
const { AniListStop, AniListSkip } = require('../discovery/anilist');
|
||||
const {
|
||||
BUDGETS, REFRESH_COOLDOWN_MS, MAX_CONSECUTIVE_FAILURES, runHarvest, shouldRunNightly, Harvester,
|
||||
} = require('../discovery/harvest');
|
||||
|
||||
const tempCache = (t) => {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'discovery-harvest-'));
|
||||
t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
|
||||
return new DiscoveryCache(dir);
|
||||
};
|
||||
|
||||
const record = (id, { tags = [], authors = [] } = {}) => ({ anilistId: id, title: `T${id}`, tags, authors });
|
||||
|
||||
// Fake client: answers from fixture maps, logs every call, and throws scripted errors by call number.
|
||||
function fakeClient({ titles = {}, errors = {} } = {}) {
|
||||
const calls = [];
|
||||
const answer = (name, key, value) => {
|
||||
calls.push(`${name}:${key}`);
|
||||
const error = errors[calls.length];
|
||||
if (error) throw error;
|
||||
return value;
|
||||
};
|
||||
return {
|
||||
calls,
|
||||
async title(id) { return answer('title', id, id in titles ? titles[id] : record(id)); },
|
||||
async recs(id) { return answer('recs', id, [{ anilistId: id * 100 }]); },
|
||||
async creator(staffId) { return answer('creator', staffId, [{ anilistId: staffId * 10 }]); },
|
||||
async tagTop({ name, kind }) { return answer('tagTop', `${kind}:${name}`, [{ anilistId: 1 }]); },
|
||||
async fresh() { return answer('fresh', 'global', [{ anilistId: 2 }]); },
|
||||
};
|
||||
}
|
||||
|
||||
const anchor = (anilistId, weight = 1) => ({ anilistId, weight, source: 'owned', title: null });
|
||||
|
||||
test('constants', () => {
|
||||
assert.deepEqual(BUDGETS, { nightly: 300, admin: 300, user: 60 });
|
||||
assert.equal(REFRESH_COOLDOWN_MS, 3600000);
|
||||
assert.equal(MAX_CONSECUTIVE_FAILURES, 5);
|
||||
});
|
||||
|
||||
test('plan order and dedupe', async (t) => {
|
||||
const cache = tempCache(t);
|
||||
const titles = {
|
||||
1: record(1, { tags: [{ name: 'Drama', kind: 'genre', rank: 100, isAdult: false }], authors: [{ staffId: 9, name: 'A', role: 'Story' }] }),
|
||||
2: record(2, { tags: [{ name: 'Drama', kind: 'genre', rank: 100, isAdult: false }] }),
|
||||
3: record(3, { tags: [{ name: 'Gore', kind: 'tag', rank: 80, isAdult: false }], authors: [{ staffId: 8, name: 'B', role: 'Art' }] }),
|
||||
};
|
||||
const client = fakeClient({ titles });
|
||||
const targets = [
|
||||
{ user: 'ann', anchors: [anchor(1, 5), anchor(2, 1)], mature: false },
|
||||
{ user: 'bob', anchors: [anchor(1, 3), anchor(3, 1)], mature: false },
|
||||
];
|
||||
const result = await runHarvest({ kind: 'admin', targets, cache, client, budget: 300 });
|
||||
assert.deepEqual(client.calls, [
|
||||
'title:1', 'title:2', 'title:3',
|
||||
'recs:1', 'recs:2', 'recs:3',
|
||||
'creator:9',
|
||||
'tagTop:genre:Drama', 'tagTop:tag:Gore',
|
||||
'fresh:global',
|
||||
]);
|
||||
assert.deepEqual(result, { requests: 10, budget: 300, status: 'success', stopReason: null });
|
||||
assert.equal(cache.get('title', 1).data.anilistId, 1);
|
||||
assert.deepEqual(cache.get('recs', 2).data, [{ anilistId: 200 }]);
|
||||
assert.deepEqual(cache.get('creator', 9).data, [{ anilistId: 90 }]);
|
||||
assert.deepEqual(cache.get('tagTop', 'genre:Drama').data, [{ anilistId: 1 }]);
|
||||
assert.deepEqual(cache.get('fresh', 'global').data, [{ anilistId: 2 }]);
|
||||
});
|
||||
|
||||
test('non-manga titles are counted but not cached', async (t) => {
|
||||
const cache = tempCache(t);
|
||||
const client = fakeClient({ titles: { 4: null } });
|
||||
const result = await runHarvest({ kind: 'user', targets: [{ user: 'ann', anchors: [anchor(4, 5)], mature: false }], cache, client, budget: 60 });
|
||||
assert.equal(cache.get('title', 4), null);
|
||||
assert.deepEqual(client.calls, ['title:4', 'recs:4', 'fresh:global']);
|
||||
assert.equal(result.requests, 3);
|
||||
});
|
||||
|
||||
test('fresh cache entries are skipped', async (t) => {
|
||||
const cache = tempCache(t);
|
||||
cache.put('title', 1, record(1));
|
||||
cache.put('recs', 1, []);
|
||||
cache.put('fresh', 'global', []);
|
||||
const client = fakeClient();
|
||||
const result = await runHarvest({ kind: 'user', targets: [{ user: 'ann', anchors: [anchor(1), anchor(2)], mature: false }], cache, client, budget: 60 });
|
||||
assert.deepEqual(client.calls, ['title:2', 'recs:2']);
|
||||
assert.equal(result.requests, 2);
|
||||
assert.equal(result.stopReason, null);
|
||||
});
|
||||
|
||||
test('budget stops with reason budget', async (t) => {
|
||||
const cache = tempCache(t);
|
||||
const client = fakeClient();
|
||||
const targets = [{ user: 'ann', anchors: [anchor(1), anchor(2), anchor(3)], mature: false }];
|
||||
const result = await runHarvest({ kind: 'user', targets, cache, client, budget: 3 });
|
||||
assert.deepEqual(result, { requests: 3, budget: 3, status: 'success', stopReason: 'budget' });
|
||||
assert.equal(client.calls.length, 3);
|
||||
});
|
||||
|
||||
test('429 stops at once', async (t) => {
|
||||
const cache = tempCache(t);
|
||||
const client = fakeClient({ errors: { 2: new AniListStop('rate-limited') } });
|
||||
const targets = [{ user: 'ann', anchors: [anchor(1), anchor(2), anchor(3)], mature: false }];
|
||||
const result = await runHarvest({ kind: 'admin', targets, cache, client, budget: 300 });
|
||||
assert.deepEqual(result, { requests: 2, budget: 300, status: 'stopped', stopReason: 'rate-limited' });
|
||||
assert.equal(client.calls.length, 2);
|
||||
assert.equal(cache.get('title', 2), null);
|
||||
});
|
||||
|
||||
test('network error stops with upstream-errors after 5', async (t) => {
|
||||
const skip = () => new AniListSkip('timeout');
|
||||
const targets = [{ user: 'ann', anchors: [1, 2, 3, 4, 5, 6, 7, 8].map((id) => anchor(id)), mature: false }];
|
||||
|
||||
const client = fakeClient({ errors: { 1: skip(), 2: skip(), 3: skip(), 4: skip(), 5: skip() } });
|
||||
const result = await runHarvest({ kind: 'admin', targets, cache: tempCache(t), client, budget: 300 });
|
||||
assert.deepEqual(result, { requests: 5, budget: 300, status: 'stopped', stopReason: 'upstream-errors' });
|
||||
assert.equal(client.calls.length, 5);
|
||||
|
||||
// A success in the middle resets the counter, so the run gets past five failures in total.
|
||||
const mixed = fakeClient({ errors: { 1: skip(), 2: skip(), 3: skip(), 4: skip(), 6: skip(), 7: skip(), 8: skip() } });
|
||||
const reset = await runHarvest({ kind: 'admin', targets, cache: tempCache(t), client: mixed, budget: 300 });
|
||||
assert.equal(reset.status, 'success');
|
||||
assert.equal(reset.stopReason, null);
|
||||
assert.ok(mixed.calls.length > 8);
|
||||
});
|
||||
|
||||
test('plan for anchorless user only fetches fresh', async (t) => {
|
||||
const client = fakeClient();
|
||||
const result = await runHarvest({ kind: 'user', targets: [{ user: 'ann', anchors: [], mature: false }], cache: tempCache(t), client, budget: 60 });
|
||||
assert.deepEqual(client.calls, ['fresh:global']);
|
||||
assert.deepEqual(result, { requests: 1, budget: 60, status: 'success', stopReason: null });
|
||||
});
|
||||
|
||||
test('shouldRunNightly', () => {
|
||||
const at = (h, m) => new Date(2026, 9, 25, h, m);
|
||||
assert.equal(shouldRunNightly({ now: at(3, 29), harvestAt: '03:30', lastNightlyDate: null, enabled: true }), false);
|
||||
assert.equal(shouldRunNightly({ now: at(3, 30), harvestAt: '03:30', lastNightlyDate: null, enabled: true }), true);
|
||||
assert.equal(shouldRunNightly({ now: at(23, 0), harvestAt: '03:30', lastNightlyDate: '2026-10-24', enabled: true }), true);
|
||||
assert.equal(shouldRunNightly({ now: at(3, 30), harvestAt: '03:30', lastNightlyDate: '2026-10-25', enabled: true }), false);
|
||||
assert.equal(shouldRunNightly({ now: at(3, 30), harvestAt: '03:30', lastNightlyDate: null, enabled: false }), false);
|
||||
});
|
||||
|
||||
// A run that waits until the test releases it, so queue state can be observed.
|
||||
function gatedHarvester({ onDone = async () => {}, now, load }) {
|
||||
const gates = [];
|
||||
const patches = [];
|
||||
const done = [];
|
||||
const client = {
|
||||
calls: [],
|
||||
async fresh() {
|
||||
this.calls.push('fresh');
|
||||
await new Promise((resolve, reject) => gates.push({ resolve, reject }));
|
||||
return [];
|
||||
},
|
||||
};
|
||||
// Nothing stays cached, so every job makes its one fresh call and waits at the gate.
|
||||
const cache = { isFresh: () => false, get: () => null, put() {} };
|
||||
const harvester = new Harvester({
|
||||
cache,
|
||||
client,
|
||||
store: { patch(value) { patches.push(structuredClone(value)); }, ...(load ? { load } : {}) },
|
||||
targetsFor: (kind, user) => [{ user: user || 'all', anchors: [], mature: false }],
|
||||
onDone: async (job, result) => { done.push({ job, result }); await onDone(job, result); },
|
||||
now,
|
||||
});
|
||||
// Lets the waiting request finish, or fail with `error`.
|
||||
const release = async (error) => {
|
||||
for (let i = 0; !gates.length; i++) {
|
||||
if (i > 1000) throw new Error('no harvest request is waiting');
|
||||
await new Promise(setImmediate);
|
||||
}
|
||||
const gate = gates.shift();
|
||||
if (error) gate.reject(error); else gate.resolve();
|
||||
await new Promise(setImmediate);
|
||||
};
|
||||
return { harvester, patches, done, release, client };
|
||||
}
|
||||
|
||||
const settle = async () => { for (let i = 0; i < 20; i++) await new Promise(setImmediate); };
|
||||
|
||||
test('harvester single-flight, positions, 409 and 429 cooldown', async (t) => {
|
||||
let clock = Date.parse('2026-10-25T10:00:00Z');
|
||||
const h = gatedHarvester({ now: () => clock });
|
||||
const { harvester } = h;
|
||||
|
||||
assert.deepEqual(harvester.state('ann'), { state: 'idle' });
|
||||
assert.equal(harvester.current(), null);
|
||||
|
||||
const first = harvester.request({ kind: 'user', user: 'ann' });
|
||||
assert.equal(first.position, 0);
|
||||
assert.equal(typeof first.harvestId, 'string');
|
||||
assert.equal(harvester.state('ann').state, 'running');
|
||||
assert.deepEqual(harvester.current(), { id: first.harvestId, kind: 'user', user: 'ann', startedAt: '2026-10-25T10:00:00.000Z' });
|
||||
|
||||
assert.throws(() => harvester.request({ kind: 'admin' }), (error) => error.status === 409);
|
||||
assert.throws(() => harvester.request({ kind: 'nightly' }), (error) => error.status === 409);
|
||||
assert.throws(() => harvester.request({ kind: 'user', user: 'ann' }), (error) => error.status === 409);
|
||||
|
||||
const second = harvester.request({ kind: 'user', user: 'bob' });
|
||||
assert.equal(second.position, 1);
|
||||
assert.deepEqual(harvester.state('bob'), { state: 'queued', position: 1, nextAllowedAt: '2026-10-25T11:00:00.000Z' });
|
||||
assert.throws(() => harvester.request({ kind: 'user', user: 'bob' }), (error) => error.status === 409);
|
||||
|
||||
await h.release();
|
||||
await settle();
|
||||
assert.equal(harvester.state('bob').state, 'running');
|
||||
assert.equal(harvester.state('ann').state, 'idle');
|
||||
|
||||
clock += 30 * 60000;
|
||||
assert.throws(() => harvester.request({ kind: 'user', user: 'ann' }), (error) => {
|
||||
assert.equal(error.status, 429);
|
||||
assert.equal(error.nextAllowedAt, '2026-10-25T11:00:00.000Z');
|
||||
return true;
|
||||
});
|
||||
assert.deepEqual(harvester.state('ann'), { state: 'idle', nextAllowedAt: '2026-10-25T11:00:00.000Z' });
|
||||
|
||||
await h.release();
|
||||
await settle();
|
||||
assert.equal(harvester.current(), null);
|
||||
|
||||
// The persisted records: running first, then the final one, for each job in order.
|
||||
assert.deepEqual(h.patches.map((p) => [p.harvest.user, p.harvest.status]),
|
||||
[['ann', 'running'], ['ann', 'success'], ['bob', 'running'], ['bob', 'success']]);
|
||||
assert.deepEqual(h.patches[0].harvest, { id: first.harvestId, kind: 'user', user: 'ann', status: 'running',
|
||||
startedAt: '2026-10-25T10:00:00.000Z', requests: 0, budget: 60 });
|
||||
assert.equal(h.patches[1].harvest.requests, 1);
|
||||
assert.equal(h.patches[1].harvest.stopReason, null);
|
||||
assert.equal(typeof h.patches[1].harvest.finishedAt, 'string');
|
||||
assert.deepEqual(h.done.map((d) => d.job.user), ['ann', 'bob']);
|
||||
|
||||
clock += 30 * 60000;
|
||||
const admin = harvester.request({ kind: 'admin' });
|
||||
assert.equal(admin.position, 0);
|
||||
const ann = harvester.request({ kind: 'user', user: 'ann' });
|
||||
assert.equal(ann.position, 1);
|
||||
await h.release();
|
||||
await h.release();
|
||||
await settle();
|
||||
assert.equal(h.patches.at(-4).harvest.budget, 300);
|
||||
});
|
||||
|
||||
test('onDone failure does not stall the queue', async (t) => {
|
||||
const errors = [];
|
||||
const original = console.error;
|
||||
console.error = (...args) => errors.push(args);
|
||||
t.after(() => { console.error = original; });
|
||||
const h = gatedHarvester({ now: () => Date.parse('2026-10-25T10:00:00Z'), onDone: async () => { throw new Error('score failed'); } });
|
||||
h.harvester.request({ kind: 'nightly' });
|
||||
h.harvester.request({ kind: 'user', user: 'ann' });
|
||||
await h.release();
|
||||
await h.release();
|
||||
await settle();
|
||||
assert.deepEqual(h.done.map((d) => d.job.kind), ['nightly', 'user']);
|
||||
assert.equal(h.harvester.current(), null);
|
||||
assert.ok(errors.length >= 2);
|
||||
});
|
||||
|
||||
test('AniList stop pauses user refreshes until an admin or nightly run', async () => {
|
||||
let clock = Date.parse('2026-10-25T10:00:00Z');
|
||||
const h = gatedHarvester({ now: () => clock });
|
||||
const { harvester, client } = h;
|
||||
harvester.request({ kind: 'user', user: 'ann' });
|
||||
harvester.request({ kind: 'user', user: 'bob' });
|
||||
assert.equal(harvester.state('bob').state, 'queued');
|
||||
|
||||
// (a) 429 while ann runs: bob's queued refresh is dropped without touching AniList.
|
||||
await h.release(new AniListStop('rate-limited'));
|
||||
await settle();
|
||||
assert.equal(client.calls.length, 1);
|
||||
assert.equal(harvester.current(), null);
|
||||
assert.deepEqual(harvester.state('bob'), { state: 'idle', blockedReason: 'rate-limited' });
|
||||
assert.deepEqual(h.patches.map((p) => [p.harvest.user, p.harvest.status, p.harvest.stopReason ?? null]),
|
||||
[['ann', 'running', null], ['ann', 'stopped', 'rate-limited']]);
|
||||
assert.deepEqual(h.done.map((d) => d.job.user), ['ann']);
|
||||
|
||||
// (b) a new refresh is refused while paused, even for a user outside any cooldown.
|
||||
clock += 2 * REFRESH_COOLDOWN_MS;
|
||||
assert.throws(() => harvester.request({ kind: 'user', user: 'cid' }), (error) => {
|
||||
assert.equal(error.status, 503);
|
||||
assert.equal(error.blockedReason, 'rate-limited');
|
||||
assert.match(error.message, /AniList paused the harvest: rate-limited/);
|
||||
return true;
|
||||
});
|
||||
assert.equal(client.calls.length, 1);
|
||||
|
||||
// (c) an accepted admin run is the allowed retry and lifts the pause.
|
||||
harvester.request({ kind: 'admin' });
|
||||
assert.equal(harvester.state('cid').blockedReason, undefined);
|
||||
await h.release();
|
||||
await settle();
|
||||
assert.equal(client.calls.length, 2);
|
||||
assert.equal(harvester.request({ kind: 'user', user: 'cid' }).position, 0);
|
||||
await h.release();
|
||||
await settle();
|
||||
});
|
||||
|
||||
test('pause survives a restart through the stored harvest record', () => {
|
||||
const load = () => ({ harvest: { status: 'stopped', stopReason: 'forbidden' } });
|
||||
const { harvester } = gatedHarvester({ now: () => 0, load });
|
||||
assert.throws(() => harvester.request({ kind: 'user', user: 'ann' }),
|
||||
(error) => error.status === 503 && error.blockedReason === 'forbidden');
|
||||
assert.equal(harvester.state('ann').blockedReason, 'forbidden');
|
||||
|
||||
const budgetStop = gatedHarvester({ now: () => 0, load: () => ({ harvest: { status: 'stopped', stopReason: 'budget' } }) });
|
||||
assert.deepEqual(budgetStop.harvester.state('ann'), { state: 'idle' });
|
||||
const broken = gatedHarvester({ now: () => 0, load: () => { throw new Error('corrupt'); } });
|
||||
assert.deepEqual(broken.harvester.state('ann'), { state: 'idle' });
|
||||
});
|
||||
|
||||
test('unknown kinds are refused', () => {
|
||||
const { harvester } = gatedHarvester({ now: () => 0 });
|
||||
for (const kind of ['constructor', 'toString', 'other'])
|
||||
assert.throws(() => harvester.request({ kind }), (error) => error.status === 400);
|
||||
});
|
||||
|
||||
test('local fault keeps the real request count and async targets work', async (t) => {
|
||||
const original = console.error;
|
||||
console.error = () => {};
|
||||
t.after(() => { console.error = original; });
|
||||
let puts = 0;
|
||||
const cache = { isFresh: () => false, get: () => null, put() { if (++puts === 2) throw new Error('disk full'); } };
|
||||
const patches = [];
|
||||
const done = [];
|
||||
const harvester = new Harvester({
|
||||
cache, client: fakeClient(), store: { patch(value) { patches.push(value); } },
|
||||
targetsFor: async () => [{ user: 'ann', anchors: [anchor(1)], mature: false }],
|
||||
onDone: async (job, result) => { done.push(result); },
|
||||
});
|
||||
harvester.request({ kind: 'admin' });
|
||||
await settle();
|
||||
assert.deepEqual(patches.at(-1).harvest.status, 'stopped');
|
||||
assert.equal(patches.at(-1).harvest.stopReason, 'error');
|
||||
assert.equal(patches.at(-1).harvest.requests, 2);
|
||||
assert.equal(done.length, 1);
|
||||
});
|
||||
@@ -0,0 +1,318 @@
|
||||
'use strict';
|
||||
// BL-0051: discovery harvest routes, per-user refresh and nightly state, against a fixture AniList server.
|
||||
const test = require('node:test'), assert = require('node:assert/strict');
|
||||
const fs = require('fs'), os = require('os'), path = require('path'), net = require('net'), http = require('http'), crypto = require('crypto');
|
||||
const { spawn } = require('child_process');
|
||||
const { Canvas } = require('../tools/png');
|
||||
|
||||
const FIXTURES = path.join(__dirname, 'fixtures', 'anilist');
|
||||
const fixture = name => JSON.parse(fs.readFileSync(path.join(FIXTURES, `${name}.json`), 'utf8'));
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
|
||||
// The anchor title carries no tags or staff, so a run stays at title + recs + fresh (the client allows 25 a minute).
|
||||
function fixtureBodies() {
|
||||
const title = fixture('title');
|
||||
Object.assign(title.data.Media, { genres: [], tags: [], staff: { edges: [] } });
|
||||
const recs = fixture('recs');
|
||||
recs.data.Media.recommendations.nodes[0].rating = 500;
|
||||
return { title, recs, creator: fixture('creator'), tagtop: fixture('tagtop'), fresh: fixture('fresh') };
|
||||
}
|
||||
|
||||
async function startAniList({ delayMs = 0 } = {}) {
|
||||
const bodies = fixtureBodies(), seen = [];
|
||||
const server = http.createServer((req, res) => {
|
||||
let raw = '';
|
||||
req.on('data', chunk => { raw += chunk; });
|
||||
req.on('end', async () => {
|
||||
const query = String(JSON.parse(raw || '{}').query || '');
|
||||
const kind = query.includes('recommendations') ? 'recs' : query.includes('Staff(') ? 'creator'
|
||||
: /genre_in|tag_in/.test(query) ? 'tagtop' : query.includes('startDate_greater') ? 'fresh' : 'title';
|
||||
seen.push(kind);
|
||||
if (delayMs) await sleep(delayMs);
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify(bodies[kind]));
|
||||
});
|
||||
});
|
||||
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
|
||||
return { url: `http://127.0.0.1:${server.address().port}/`, seen, close: () => new Promise(resolve => server.close(resolve)) };
|
||||
}
|
||||
|
||||
async function freePort() {
|
||||
const listener = net.createServer();
|
||||
await new Promise(resolve => listener.listen(0, '127.0.0.1', resolve));
|
||||
const port = listener.address().port;
|
||||
await new Promise(resolve => listener.close(resolve));
|
||||
return port;
|
||||
}
|
||||
|
||||
const hashPw = pw => { const salt = crypto.randomBytes(16).toString('hex'); return salt + ':' + crypto.scryptSync(pw, salt, 32).toString('hex'); };
|
||||
const sha = s => crypto.createHash('sha256').update(s).digest('hex');
|
||||
const USERS = { boss: { admin: true, discovery: false }, alice: { discovery: true }, bob: { discovery: true }, carol: { discovery: false } };
|
||||
|
||||
// A disposable reader with ready-made accounts and sessions; nightly harvests are off unless the store says otherwise.
|
||||
async function startReader({ anilist, discovery = {}, mangaMeta = null, scraperUrl = 'http://127.0.0.1:9' } = {}) {
|
||||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-discovery-server-'));
|
||||
const library = path.join(root, 'library'), data = path.join(root, 'data');
|
||||
fs.mkdirSync(path.join(library, 'Example Manga'), { recursive: true });
|
||||
fs.writeFileSync(path.join(library, 'Example Manga', 'notes.txt'), 'not a chapter');
|
||||
if (mangaMeta) {
|
||||
const chapter = path.join(library, mangaMeta.title, 'v01', 'c0001');
|
||||
fs.mkdirSync(chapter, { recursive: true });
|
||||
fs.writeFileSync(path.join(chapter, '0001.png'), new Canvas(20, 30, [60, 90, 140]).png());
|
||||
fs.writeFileSync(path.join(library, mangaMeta.title, 'manga.json'), JSON.stringify(mangaMeta));
|
||||
}
|
||||
fs.mkdirSync(path.join(data, 'scraper'), { recursive: true });
|
||||
fs.writeFileSync(path.join(data, 'scraper', 'queue.json'), JSON.stringify({ queue: [] }));
|
||||
const users = {}, sessions = {}, cookies = {};
|
||||
for (const [name, spec] of Object.entries(USERS)) {
|
||||
users[name] = { name, hash: hashPw(`${name}-password`), admin: !!spec.admin, created: Date.now(),
|
||||
settings: { discoveryEnabled: spec.discovery } };
|
||||
const token = crypto.randomBytes(32).toString('base64url');
|
||||
sessions[sha(token)] = { u: name, t: Date.now() };
|
||||
cookies[name] = `sid=${token}`;
|
||||
}
|
||||
fs.writeFileSync(path.join(data, 'db.json'), JSON.stringify({ users, sessions,
|
||||
discoveryFeedback: { alice: { 'anilist:101': 'interested' } } }));
|
||||
fs.writeFileSync(path.join(data, 'discovery.json'), JSON.stringify({ schema: 1, revision: 0, candidates: {}, feeds: {},
|
||||
lastRun: { status: 'never' }, harvest: null, settings: { harvestEnabled: false, harvestAt: '03:30' },
|
||||
nightly: { lastDate: null }, ...discovery }));
|
||||
|
||||
const port = await freePort(), output = [];
|
||||
const child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
|
||||
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
|
||||
SCRAPER_URL: scraperUrl, SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '',
|
||||
DISCOVERY_ANILIST_URL: anilist ? anilist.url : 'http://127.0.0.1:9/', DISCOVERY_HARVEST_AT: '', GREATEST_COVER_WARM: '0' } });
|
||||
child.stdout.on('data', d => output.push(String(d)));
|
||||
child.stderr.on('data', d => output.push(String(d)));
|
||||
const base = `http://127.0.0.1:${port}`;
|
||||
let ready = false;
|
||||
for (let n = 0; n < 100 && !ready; n++) {
|
||||
ready = await fetch(base + '/health').then(r => r.ok).catch(() => false);
|
||||
if (!ready) await sleep(50);
|
||||
}
|
||||
assert.ok(ready, output.join(''));
|
||||
const call = async (who, method, url, body) => {
|
||||
const response = await fetch(base + url, { method, headers: { cookie: cookies[who], ...(body ? { 'content-type': 'application/json' } : {}) },
|
||||
body: body === undefined ? undefined : JSON.stringify(body) });
|
||||
return { status: response.status, body: await response.json().catch(() => null) };
|
||||
};
|
||||
return {
|
||||
root, library, data, output,
|
||||
get: (who, url) => call(who, 'GET', url),
|
||||
post: (who, url, body = {}) => call(who, 'POST', url, body),
|
||||
async stop() {
|
||||
await new Promise(resolve => { if (child.exitCode !== null) return resolve(); child.once('exit', resolve); child.kill(); });
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
async function until(check, what, timeoutMs = 20000) {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
let last;
|
||||
while (Date.now() < deadline) {
|
||||
last = await check();
|
||||
if (last) return last;
|
||||
await sleep(100);
|
||||
}
|
||||
assert.fail(`Timed out waiting for ${what}`);
|
||||
}
|
||||
const idle = (reader, who) => until(async () => {
|
||||
const { body } = await reader.get(who, '/api/discovery');
|
||||
return body.refresh.state === 'idle' ? body : null;
|
||||
}, `${who}'s refresh to finish`);
|
||||
const harvestDone = reader => until(async () => {
|
||||
const { body } = await reader.get('boss', '/api/admin/discovery');
|
||||
return !body.currentHarvest && body.harvest && body.harvest.status !== 'running' ? body : null;
|
||||
}, 'the harvest to finish');
|
||||
|
||||
function snapshot(dir) {
|
||||
const out = {};
|
||||
const walk = rel => {
|
||||
for (const entry of fs.readdirSync(path.join(dir, rel), { withFileTypes: true })) {
|
||||
const child = path.join(rel, entry.name), stat = fs.statSync(path.join(dir, child));
|
||||
out[child] = entry.isDirectory() ? 'dir' : `${stat.size}:${stat.mtimeMs}`;
|
||||
if (entry.isDirectory()) walk(child);
|
||||
}
|
||||
};
|
||||
walk('');
|
||||
return out;
|
||||
}
|
||||
|
||||
test('refresh requires opt-in, queues, runs and rescores only the caller', { timeout: 60000 }, async () => {
|
||||
const anilist = await startAniList();
|
||||
const reader = await startReader({ anilist });
|
||||
try {
|
||||
assert.equal((await reader.post('boss', '/api/admin/discovery/run')).status, 200);
|
||||
await until(async () => (await reader.get('boss', '/api/admin/discovery')).body.currentRun?.status === 'success', 'the first run');
|
||||
const bobBefore = (await reader.get('bob', '/api/discovery')).body;
|
||||
assert.ok(bobBefore.generatedAt);
|
||||
assert.deepEqual(bobBefore.refresh, { state: 'idle' });
|
||||
|
||||
const refused = await reader.post('carol', '/api/discovery/refresh');
|
||||
assert.equal(refused.status, 403);
|
||||
|
||||
const accepted = await reader.post('alice', '/api/discovery/refresh');
|
||||
assert.equal(accepted.status, 200, JSON.stringify(accepted.body));
|
||||
assert.match(accepted.body.harvestId, /^[0-9a-f]{16}$/);
|
||||
assert.equal(accepted.body.position, 0);
|
||||
|
||||
const alice = await idle(reader, 'alice');
|
||||
assert.ok(alice.items.some(item => item.candidateId === 'anilist:201'), JSON.stringify(alice.items.map(i => i.candidateId)));
|
||||
const rec = alice.items.find(item => item.candidateId === 'anilist:201');
|
||||
assert.ok(rec.reasons.some(reason => /recommend/.test(reason)));
|
||||
assert.ok(rec.candidate.fetchedAt);
|
||||
assert.deepEqual(anilist.seen.sort(), ['fresh', 'recs', 'title']);
|
||||
|
||||
const bobAfter = (await reader.get('bob', '/api/discovery')).body;
|
||||
assert.equal(bobAfter.generatedAt, bobBefore.generatedAt);
|
||||
assert.deepEqual(bobAfter.items.map(i => i.candidateId), bobBefore.items.map(i => i.candidateId));
|
||||
|
||||
const admin = (await reader.get('boss', '/api/admin/discovery')).body;
|
||||
assert.equal(admin.harvest.kind, 'user');
|
||||
assert.equal(admin.harvest.user, 'alice');
|
||||
assert.equal(admin.harvest.status, 'success');
|
||||
assert.equal(admin.harvest.requests, 3);
|
||||
assert.equal(admin.harvest.budget, 60);
|
||||
assert.ok(admin.cache.files >= 3 && admin.cache.bytes > 0, JSON.stringify(admin.cache));
|
||||
assert.ok(fs.existsSync(path.join(reader.data, 'discovery-cache')));
|
||||
} finally { await reader.stop(); await anilist.close(); }
|
||||
});
|
||||
|
||||
test('second refresh within an hour is 429 with nextAllowedAt', { timeout: 60000 }, async () => {
|
||||
const anilist = await startAniList({ delayMs: 400 });
|
||||
const reader = await startReader({ anilist });
|
||||
try {
|
||||
assert.equal((await reader.post('bob', '/api/discovery/refresh')).status, 200);
|
||||
const duplicate = await reader.post('bob', '/api/discovery/refresh');
|
||||
assert.equal(duplicate.status, 409);
|
||||
const state = await idle(reader, 'bob');
|
||||
assert.ok(Date.parse(state.refresh.nextAllowedAt) > Date.now() + 50 * 60000);
|
||||
|
||||
const cooling = await reader.post('bob', '/api/discovery/refresh');
|
||||
assert.equal(cooling.status, 429);
|
||||
assert.equal(cooling.body.nextAllowedAt, state.refresh.nextAllowedAt);
|
||||
assert.equal(typeof cooling.body.error, 'string');
|
||||
} finally { await reader.stop(); await anilist.close(); }
|
||||
});
|
||||
|
||||
test('admin harvest is admin-only and single-flight', { timeout: 60000 }, async () => {
|
||||
const anilist = await startAniList({ delayMs: 500 });
|
||||
const reader = await startReader({ anilist });
|
||||
try {
|
||||
assert.equal((await reader.post('alice', '/api/admin/discovery/harvest')).status, 403);
|
||||
const first = await reader.post('boss', '/api/admin/discovery/harvest');
|
||||
assert.equal(first.status, 200, JSON.stringify(first.body));
|
||||
assert.equal(first.body.position, 0);
|
||||
assert.equal((await reader.post('boss', '/api/admin/discovery/harvest')).status, 409);
|
||||
const running = (await reader.get('boss', '/api/admin/discovery')).body;
|
||||
assert.equal(running.currentHarvest.id, first.body.harvestId);
|
||||
assert.equal(running.currentHarvest.kind, 'admin');
|
||||
const done = await harvestDone(reader);
|
||||
assert.equal(done.harvest.kind, 'admin');
|
||||
assert.equal(done.harvest.budget, 300);
|
||||
// Afterwards every opted-in user was rescored.
|
||||
for (const who of ['alice', 'bob']) assert.ok((await reader.get(who, '/api/discovery')).body.generatedAt, who);
|
||||
} finally { await reader.stop(); await anilist.close(); }
|
||||
});
|
||||
|
||||
// A scraper stand-in. The rescore asks it for the queue after the harvest's final store patch and
|
||||
// right before publishing, so onState can break discovery.json at exactly that point.
|
||||
async function startScraper(onState) {
|
||||
const server = http.createServer((req, res) => {
|
||||
if (req.url.startsWith('/api/state')) onState();
|
||||
res.writeHead(200, { 'content-type': 'application/json' });
|
||||
res.end(JSON.stringify({ queue: [] }));
|
||||
});
|
||||
await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
|
||||
return { url: `http://127.0.0.1:${server.address().port}`, close: () => new Promise(resolve => server.close(resolve)) };
|
||||
}
|
||||
|
||||
test('a failed rescore after a harvest reaches the admin as lastError', { timeout: 60000 }, async () => {
|
||||
const anilist = await startAniList();
|
||||
let reader, armed = false, saved = null;
|
||||
const scraper = await startScraper(() => {
|
||||
if (!armed) return;
|
||||
armed = false;
|
||||
const file = path.join(reader.data, 'discovery.json');
|
||||
saved = fs.readFileSync(file, 'utf8');
|
||||
fs.writeFileSync(file, '{broken'); // the publish now refuses to overwrite it
|
||||
});
|
||||
// A malformed manga.json (tags not a list, relations an object) must not be what breaks scoring.
|
||||
reader = await startReader({ anilist, scraperUrl: scraper.url,
|
||||
mangaMeta: { title: 'Broken Tags', tags: 5, 'x-anilist': { id: 101, relations: {} } } });
|
||||
try {
|
||||
await until(async () => (await reader.get('boss', '/api/library')).body?.mangas?.length === 1, 'the library scan');
|
||||
armed = true;
|
||||
const first = await reader.post('boss', '/api/admin/discovery/harvest');
|
||||
assert.equal(first.status, 200);
|
||||
await until(async () => saved && reader.output.join('').includes(`${first.body.harvestId} follow-up failed`), 'the failed rescore');
|
||||
fs.writeFileSync(path.join(reader.data, 'discovery.json'), saved);
|
||||
const admin = await until(async () => {
|
||||
const { body } = await reader.get('boss', '/api/admin/discovery');
|
||||
return body.lastError ? body : null;
|
||||
}, 'the rescore error');
|
||||
assert.match(admin.lastError, /^Re-scoring after harvest [0-9a-f]{16} failed: Discovery store unavailable/);
|
||||
assert.equal(admin.currentRun, null);
|
||||
assert.equal(admin.harvest.kind, 'admin');
|
||||
|
||||
// The next harvest's successful rescore clears the stale error.
|
||||
const second = await reader.post('boss', '/api/admin/discovery/harvest');
|
||||
assert.equal(second.status, 200, JSON.stringify(second.body));
|
||||
const cleared = await until(async () => {
|
||||
const { body } = await reader.get('boss', '/api/admin/discovery');
|
||||
return body.harvest?.id === second.body.harvestId && !body.currentHarvest && body.lastError === null ? body : null;
|
||||
}, 'the error to clear');
|
||||
assert.ok(cleared.candidateCount > 0);
|
||||
} finally { await reader.stop(); await anilist.close(); await scraper.close(); }
|
||||
});
|
||||
|
||||
test('settings validation', { timeout: 30000 }, async () => {
|
||||
const reader = await startReader();
|
||||
try {
|
||||
const before = (await reader.get('boss', '/api/admin/discovery')).body;
|
||||
assert.deepEqual(before.settings, { harvestEnabled: false, harvestAt: '03:30' });
|
||||
assert.equal(before.nextNightlyAt, null);
|
||||
assert.equal((await reader.post('alice', '/api/admin/discovery/settings', { harvestEnabled: true, harvestAt: '04:00' })).status, 403);
|
||||
for (const body of [{ harvestEnabled: true, harvestAt: '25:00' }, { harvestEnabled: true, harvestAt: '4:00' },
|
||||
{ harvestEnabled: 'yes', harvestAt: '04:00' }, { harvestEnabled: true }, { harvestEnabled: true, harvestAt: '04:00', extra: 1 }]) {
|
||||
const response = await reader.post('boss', '/api/admin/discovery/settings', body);
|
||||
assert.equal(response.status, 400, JSON.stringify(body));
|
||||
}
|
||||
const saved = await reader.post('boss', '/api/admin/discovery/settings', { harvestEnabled: true, harvestAt: '23:59' });
|
||||
assert.equal(saved.status, 200);
|
||||
const after = (await reader.get('boss', '/api/admin/discovery')).body;
|
||||
assert.deepEqual(after.settings, { harvestEnabled: true, harvestAt: '23:59' });
|
||||
const next = new Date(after.nextNightlyAt);
|
||||
// The next slot today at 23:59, or "due now" when the test happens to run inside that minute.
|
||||
const atSlot = next.getHours() === 23 && next.getMinutes() === 59 && next.getTime() > Date.now() - 120000;
|
||||
assert.ok(atSlot || Math.abs(next.getTime() - Date.now()) < 120000, after.nextNightlyAt);
|
||||
} finally { await reader.stop(); }
|
||||
});
|
||||
|
||||
test('interrupted harvest is marked on startup', { timeout: 30000 }, async () => {
|
||||
const reader = await startReader({ discovery: { harvest: { id: 'abc', kind: 'nightly', user: null, status: 'running',
|
||||
startedAt: '2026-10-24T03:30:00.000Z', requests: 12, budget: 300 } } });
|
||||
try {
|
||||
const { body } = await reader.get('boss', '/api/admin/discovery');
|
||||
assert.equal(body.harvest.status, 'interrupted');
|
||||
assert.equal(body.harvest.id, 'abc');
|
||||
assert.equal(body.harvest.requests, 12);
|
||||
assert.ok(Date.parse(body.harvest.finishedAt));
|
||||
assert.equal(body.currentHarvest, null);
|
||||
} finally { await reader.stop(); }
|
||||
});
|
||||
|
||||
test('harvest never touches scraper queue or library', { timeout: 60000 }, async () => {
|
||||
const anilist = await startAniList();
|
||||
const reader = await startReader({ anilist });
|
||||
try {
|
||||
const libraryBefore = snapshot(reader.library), scraperBefore = snapshot(path.join(reader.data, 'scraper'));
|
||||
assert.equal((await reader.post('alice', '/api/discovery/refresh')).status, 200);
|
||||
await idle(reader, 'alice');
|
||||
assert.equal((await reader.post('boss', '/api/admin/discovery/harvest')).status, 200);
|
||||
await harvestDone(reader);
|
||||
assert.deepEqual(snapshot(reader.library), libraryBefore);
|
||||
assert.deepEqual(snapshot(path.join(reader.data, 'scraper')), scraperBefore);
|
||||
} finally { await reader.stop(); await anilist.close(); }
|
||||
});
|
||||
+328
-4
@@ -4,8 +4,9 @@ const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { generate } = require('../discovery/engine');
|
||||
const { generate, generateAsync, buildCandidates, candidatePool, feedForUser, mixSlots, mergeUserFeed, MAX_CANDIDATES, SLOTS } = require('../discovery/engine');
|
||||
const { DiscoveryStore } = require('../discovery/store');
|
||||
const { DiscoveryCache } = require('../discovery/cache');
|
||||
|
||||
test('discovery is bounded, diversified, opt-in, evidence-linked and user-specific', () => {
|
||||
const { loadCatalog } = require('../greatest');
|
||||
@@ -15,12 +16,17 @@ test('discovery is bounded, diversified, opt-in, evidence-linked and user-specif
|
||||
off: { name: 'off', settings: {} } };
|
||||
const db = { users, greatestRatings: { alice: { [greatest.entries[0].id]: 'flame' },
|
||||
bob: { [greatest.entries[20].id]: 'flame' } }, discoveryFeedback: { alice: {} }, progress: {} };
|
||||
const result = generate({ db, library: [], greatest, cacheDir: path.join(os.tmpdir(), 'nonexistent-discovery-cache') });
|
||||
assert.equal(Object.keys(result.candidates).length, 50);
|
||||
const cacheDir = path.join(os.tmpdir(), 'nonexistent-discovery-cache');
|
||||
const result = generate({ db, library: [], greatest, cacheDir });
|
||||
// Only candidates a feed shows are published; the pool itself stays bounded at 2000.
|
||||
const shown = new Set(Object.values(result.feeds).flatMap(feed => (feed.items || []).map(item => item.candidateId)));
|
||||
assert.deepEqual(Object.keys(result.candidates).sort(), [...shown].sort());
|
||||
const pool = candidatePool({ greatest, library: [], cacheDir });
|
||||
assert.equal(pool.candidates.length, Math.min(2000, result.examined));
|
||||
assert.ok(result.feeds.alice.items.length > 0 && result.feeds.alice.items.length <= 20);
|
||||
assert.ok(result.feeds.bob.items.length > 0);
|
||||
assert.equal(result.feeds.off.disabled, true);
|
||||
assert.ok(new Set(Object.values(result.candidates).map(c => c.period)).size >= 10, 'An early catalog page must not crowd out later eras');
|
||||
assert.ok(new Set(pool.candidates.map(c => c.period)).size >= 10, 'An early catalog page must not crowd out later eras');
|
||||
assert.ok(result.feeds.alice.items.every(item => item.evidenceIds.every(id => item.evidence.some(e => e.id === id))));
|
||||
assert.ok(result.feeds.alice.items.some(item => item.evidence.some(e => e.kind === 'personal')));
|
||||
assert.ok(!result.feeds.bob.items.some(item => item.evidence.some(e => e.label?.includes(greatest.entries[0].title) && e.kind === 'personal')));
|
||||
@@ -46,3 +52,321 @@ test('discovery store publishes atomically and refuses to overwrite corrupt data
|
||||
assert.throws(() => store.publish({ candidates: {}, feeds: {} }), /refusing to overwrite/);
|
||||
assert.equal(fs.readFileSync(store.file, 'utf8'), '{broken');
|
||||
});
|
||||
|
||||
function tempStore(t) {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-discovery-meta-'));
|
||||
t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
|
||||
return new DiscoveryStore(dir);
|
||||
}
|
||||
const emptyRun = () => ({ candidates: {}, feeds: {}, startedAt: new Date().toISOString(), examined: 0, clipped: 0 });
|
||||
|
||||
test('store loads a Phase A file without new fields', t => {
|
||||
const store = tempStore(t);
|
||||
fs.writeFileSync(store.file, JSON.stringify({ schema: 1, revision: 3, candidates: {}, feeds: {}, lastRun: { status: 'success' } }));
|
||||
const data = store.load();
|
||||
assert.equal(data.revision, 3);
|
||||
assert.equal(data.harvest, null);
|
||||
assert.equal(data.settings, null);
|
||||
assert.deepEqual(data.nightly, { lastDate: null });
|
||||
});
|
||||
|
||||
test('patch keeps revision and publish keeps harvest', t => {
|
||||
const store = tempStore(t);
|
||||
store.publish(emptyRun());
|
||||
store.patch({ harvest: { status: 'success' } });
|
||||
assert.equal(store.load().revision, 1);
|
||||
store.patch({ settings: { a: 1 } });
|
||||
store.patch({ nightly: { lastDate: '2026-10-25' } });
|
||||
store.publish(emptyRun());
|
||||
const data = store.load();
|
||||
assert.equal(data.harvest.status, 'success');
|
||||
assert.deepEqual(data.settings, { a: 1 });
|
||||
assert.equal(data.nightly.lastDate, '2026-10-25');
|
||||
assert.equal(data.revision, 2);
|
||||
});
|
||||
|
||||
test('patch refuses a corrupt store', t => {
|
||||
const store = tempStore(t);
|
||||
fs.writeFileSync(store.file, '{broken');
|
||||
assert.throws(() => store.patch({ harvest: {} }), /refusing to overwrite/);
|
||||
assert.equal(fs.readFileSync(store.file, 'utf8'), '{broken');
|
||||
});
|
||||
|
||||
// ---- Wider pool (BL-0051): AniList seeds, new score parts, slot mix ----
|
||||
const NOW = () => new Date('2026-10-25T12:00:00Z');
|
||||
const FETCHED_AT = '2026-10-20T00:00:00.000Z';
|
||||
const NO_SCRAPER_CACHE = path.join(os.tmpdir(), 'nonexistent-discovery-cache');
|
||||
const AUTHOR = { staffId: 7, name: 'Author Seven', role: 'Story' };
|
||||
function seedRecord(id, extra = {}) {
|
||||
return { anilistId: id, title: `Seed ${id}`, altTitles: [], format: 'MANGA', isAdult: false, status: 'completed',
|
||||
year: 2010, averageScore: 70, popularity: 1000, trending: 0,
|
||||
tags: [{ name: 'Action', rank: 90, kind: 'genre', isAdult: false }],
|
||||
authors: [{ staffId: 900 + id, name: `Writer ${id}`, role: 'Story' }],
|
||||
coverUrl: `https://img.example/${id}.jpg`, siteUrl: `https://anilist.co/manga/${id}`, ...extra };
|
||||
}
|
||||
const seed = (id, extra) => ({ anilistId: id, record: seedRecord(id, extra) });
|
||||
function harvestFixture(t) {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-discovery-harvest-'));
|
||||
t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
|
||||
const cache = new DiscoveryCache(dir, { now: () => Date.parse(FETCHED_AT) });
|
||||
cache.put('title', 100, seedRecord(100, { title: 'Anchor One', authors: [AUTHOR],
|
||||
tags: [{ name: 'Drama', rank: 100, kind: 'genre', isAdult: false }, { name: 'Time Skip', rank: 80, kind: 'tag', isAdult: false }] }));
|
||||
cache.put('recs', 100, [
|
||||
{ ...seed(201), rating: 42 },
|
||||
{ ...seed(209, { authors: [AUTHOR] }), rating: 10 },
|
||||
{ ...seed(207, { isAdult: true }), rating: 5 },
|
||||
{ ...seed(208, { tags: [{ name: 'Nudity', rank: 65, kind: 'tag', isAdult: true }] }), rating: 5 },
|
||||
{ ...seed(212), rating: 3 },
|
||||
]);
|
||||
cache.put('creator', 7, [{ ...seed(202, { authors: [AUTHOR] }), role: 'Story' }, { ...seed(209, { authors: [AUTHOR] }), role: 'Story' }]);
|
||||
cache.put('tagTop', 'genre:Drama', [seed(203, { averageScore: 80, tags: [{ name: 'Drama', rank: 90, kind: 'genre', isAdult: false }] })]);
|
||||
cache.put('tagTop', 'tag:Time Skip', [seed(204, { averageScore: 70 })]);
|
||||
cache.put('fresh', 'global', [seed(205, { status: 'ongoing', year: 2025 }), seed(206, { status: 'ongoing', year: 2020 }),
|
||||
seed(210, { title: 'Owned Title', year: 2020 }), seed(211, { title: 'Queued Title' }), seed(213, { status: 'ongoing', year: 2026 })]);
|
||||
return cache;
|
||||
}
|
||||
const fixtureGreatest = { entries: [
|
||||
{ id: 'g1', title: 'Anchor One', period: '2000s', rank: 1, note: '', metadata: { id: 100 } },
|
||||
{ id: 'g2', title: 'Voted Two', period: '1990s', rank: 2, note: '', metadata: { id: 212 } },
|
||||
] };
|
||||
function fixtureWorld(t, settings = {}) {
|
||||
const user = { name: 'alice', settings: { discoveryEnabled: true, ...settings } };
|
||||
const db = { users: { alice: user }, greatestRatings: { alice: { g1: 'flame', g2: 'good' } },
|
||||
discoveryFeedback: { alice: { 'anilist:213': 'dismissed' } }, progress: {} };
|
||||
const library = [{ id: 'm1', title: 'Owned Title', meta: { year: 2020, 'x-anilist': { id: 210 } } }];
|
||||
return { user, db, library, greatest: fixtureGreatest, harvestCache: harvestFixture(t), queuedTitles: ['Queued Title'] };
|
||||
}
|
||||
function fixtureFeed(t, settings) {
|
||||
const world = fixtureWorld(t, settings);
|
||||
const candidates = buildCandidates({ greatest: world.greatest, library: world.library, cacheDir: NO_SCRAPER_CACHE, harvestCache: world.harvestCache });
|
||||
const feed = feedForUser({ ...world, candidates, now: NOW });
|
||||
return { world, candidates, feed, item: id => feed.items.find(item => item.candidateId === `anilist:${id}`) };
|
||||
}
|
||||
|
||||
test('engine: empty harvest cache equals Phase A pool', t => {
|
||||
const { loadCatalog } = require('../greatest');
|
||||
const greatest = loadCatalog();
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-discovery-empty-'));
|
||||
t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
|
||||
const db = { users: {}, greatestRatings: {}, discoveryFeedback: {}, progress: {} };
|
||||
const without = candidatePool({ greatest, library: [], cacheDir: NO_SCRAPER_CACHE });
|
||||
const withEmpty = candidatePool({ greatest, library: [], cacheDir: NO_SCRAPER_CACHE, harvestCache: new DiscoveryCache(dir) });
|
||||
assert.equal(MAX_CANDIDATES, 2000);
|
||||
assert.deepEqual(withEmpty.candidates.map(c => c.id).sort(), without.candidates.map(c => c.id).sort());
|
||||
const users = { alice: { name: 'alice', settings: { discoveryEnabled: true } } };
|
||||
const run = harvestCache => generate({ db: { ...db, users }, library: [], greatest, cacheDir: NO_SCRAPER_CACHE, harvestCache });
|
||||
assert.deepEqual(Object.keys(run(new DiscoveryCache(dir)).candidates).sort(), Object.keys(run(null).candidates).sort());
|
||||
});
|
||||
|
||||
test('published candidates are only those referenced by a feed', async t => {
|
||||
const world = fixtureWorld(t);
|
||||
world.db.users.bob = { name: 'bob', settings: { discoveryEnabled: true } };
|
||||
world.db.users.off = { name: 'off', settings: {} };
|
||||
const args = { db: world.db, library: world.library, greatest: world.greatest, cacheDir: NO_SCRAPER_CACHE,
|
||||
harvestCache: world.harvestCache, queuedTitles: world.queuedTitles, now: NOW };
|
||||
const referenced = result => new Set(Object.values(result.feeds).flatMap(feed => (feed.items || []).map(item => item.candidateId)));
|
||||
for (const result of [generate(args), await generateAsync(args), await generateAsync({ ...args, onlyUser: 'alice' })]) {
|
||||
const shown = referenced(result);
|
||||
assert.ok(shown.size > 0);
|
||||
for (const id of Object.keys(result.candidates)) assert.ok(shown.has(id), `${id} is published but not shown`);
|
||||
for (const id of shown) assert.ok(result.candidates[id], `${id} is shown but not published`);
|
||||
assert.ok(result.examined > Object.keys(result.candidates).length, 'examined still counts the whole pool');
|
||||
assert.equal(result.clipped, 0);
|
||||
}
|
||||
});
|
||||
|
||||
test('malformed library metadata and cache records score without throwing', t => {
|
||||
const world = fixtureWorld(t, { discoveryReadingActivity: true });
|
||||
world.library.push(
|
||||
{ id: 'm2', title: 'Bad Tags', meta: { tags: 5, 'x-anilist': { id: 220, relations: {} } } },
|
||||
{ id: 'm3', title: 'Bad Relations', meta: { tags: [7, null, { name: 'Drama' }, 'Drama'],
|
||||
'x-anilist': { id: 221, relations: [null, 3, 'x', { id: 222, format: 'MANGA', type: 'SEQUEL', title: 'Sequel Title' }] } } },
|
||||
{ id: 'm4', title: 'Bad Identity', meta: { tags: { a: 1 }, 'x-anilist': 5 } });
|
||||
world.db.progress = { alice: { m2: { c1: { done: true } }, m3: { c1: { done: true } } } };
|
||||
world.harvestCache.put('title', 220, seedRecord(220, { authors: [null, 5, AUTHOR], tags: [null, 'Drama', { name: 'Drama', kind: 'genre' }] }));
|
||||
world.harvestCache.put('recs', 220, [null, 5, { ...seed(230), rating: 7 }]);
|
||||
const args = { db: world.db, library: world.library, greatest: world.greatest, cacheDir: NO_SCRAPER_CACHE,
|
||||
harvestCache: world.harvestCache, queuedTitles: world.queuedTitles, now: NOW };
|
||||
const result = generate(args);
|
||||
const ids = result.feeds.alice.items.map(item => item.candidateId);
|
||||
assert.ok(ids.includes('anilist:222'), JSON.stringify(ids));
|
||||
assert.ok(ids.includes('anilist:230'), JSON.stringify(ids));
|
||||
});
|
||||
|
||||
test('harvested records join the pool and fill Greatest candidates', t => {
|
||||
const { candidates } = fixtureFeed(t);
|
||||
const byId = new Map(candidates.map(c => [c.id, c]));
|
||||
for (const id of [201, 202, 203, 204, 205, 206, 207, 208, 209]) assert.ok(byId.has(`anilist:${id}`), `anilist:${id}`);
|
||||
const fromRecs = byId.get('anilist:201');
|
||||
assert.equal(fromRecs.isAdult, false);
|
||||
assert.equal(fromRecs.averageScore, 70);
|
||||
assert.deepEqual(fromRecs.tags, ['Action']);
|
||||
assert.deepEqual(fromRecs.authors, ['Writer 201']);
|
||||
assert.equal(fromRecs.tagRecords[0].rank, 90);
|
||||
assert.equal(fromRecs.authorRecords[0].staffId, 1101);
|
||||
assert.equal(fromRecs.fetchedAt, FETCHED_AT);
|
||||
const greatestOne = byId.get('anilist:212');
|
||||
assert.equal(greatestOne.greatestId, 'g2');
|
||||
assert.equal(greatestOne.title, 'Voted Two');
|
||||
assert.equal(greatestOne.isAdult, false);
|
||||
assert.equal(greatestOne.tagRecords.length, 1);
|
||||
});
|
||||
|
||||
test('community rec, creator, tagTop and fresh parts and reasons', t => {
|
||||
const { item } = fixtureFeed(t);
|
||||
const rec = item(201);
|
||||
assert.equal(rec.scoreParts.communityRec, 3.71);
|
||||
assert.equal(rec.group, 'recs');
|
||||
assert.equal(rec.reason, 'AniList readers who liked Anchor One recommend this (42).');
|
||||
const recEvidence = rec.evidence.find(e => e.id === 'rec:100:anilist:201');
|
||||
assert.equal(recEvidence.kind, 'recommendation');
|
||||
assert.equal(recEvidence.fetchedAt, FETCHED_AT);
|
||||
assert.equal(recEvidence.url, 'https://anilist.co/manga/201');
|
||||
assert.ok(rec.evidenceIds.includes('rec:100:anilist:201'));
|
||||
|
||||
const creator = item(202);
|
||||
assert.equal(creator.scoreParts.sameCreator, 4);
|
||||
assert.equal(creator.group, 'creator');
|
||||
assert.equal(creator.reason, 'Also by Author Seven — you liked Anchor One.');
|
||||
const creatorEvidence = creator.evidence.find(e => e.id === 'creator:7:anilist:202');
|
||||
assert.equal(creatorEvidence.kind, 'creator');
|
||||
assert.equal(creatorEvidence.fetchedAt, FETCHED_AT);
|
||||
|
||||
const top = item(203);
|
||||
assert.equal(top.scoreParts.tagTop, 3);
|
||||
assert.equal(top.group, 'tagTop');
|
||||
assert.equal(top.reason, 'Top-rated in Drama, one of your most-liked tags.');
|
||||
assert.equal(top.evidence.find(e => e.id === 'tagtop:genre:Drama:anilist:203').kind, 'tag-top');
|
||||
assert.equal(item(204).scoreParts.tagTop, 0, 'averageScore below 75 earns no tagTop part');
|
||||
|
||||
const fresh = item(205);
|
||||
assert.equal(fresh.scoreParts.fresh, 2);
|
||||
assert.equal(fresh.group, 'fresh');
|
||||
assert.equal(fresh.reason, 'New: started 2025, ongoing and trending.');
|
||||
const freshEvidence = fresh.evidence.find(e => e.id === 'fresh:anilist:205');
|
||||
assert.equal(freshEvidence.kind, 'fresh');
|
||||
assert.equal(freshEvidence.fetchedAt, FETCHED_AT);
|
||||
assert.equal(item(206).scoreParts.fresh, 1);
|
||||
for (const row of [rec, creator, top, fresh]) assert.ok(row.reason.length <= 280);
|
||||
});
|
||||
|
||||
test('isAdult filter', t => {
|
||||
const plain = fixtureFeed(t);
|
||||
assert.equal(plain.item(207), undefined);
|
||||
assert.equal(plain.item(208), undefined, 'an adult tag with rank >= 60 counts as mature');
|
||||
const mature = fixtureFeed(t, { discoveryMature: true });
|
||||
assert.ok(mature.item(207));
|
||||
assert.ok(mature.item(208));
|
||||
});
|
||||
|
||||
test('multiple seeds merge reasons', t => {
|
||||
const { item } = fixtureFeed(t);
|
||||
const both = item(209);
|
||||
assert.equal(both.scoreParts.sameCreator, 4);
|
||||
assert.equal(both.scoreParts.communityRec, 2.73);
|
||||
assert.equal(both.score, 6.73);
|
||||
assert.equal(both.group, 'creator');
|
||||
assert.deepEqual(both.reasons, ['Also by Author Seven — you liked Anchor One.',
|
||||
'AniList readers who liked Anchor One recommend this (10).']);
|
||||
assert.equal(both.reason, both.reasons[0]);
|
||||
assert.ok(both.evidenceIds.includes('rec:100:anilist:209'));
|
||||
assert.ok(both.evidenceIds.includes('creator:7:anilist:209'));
|
||||
});
|
||||
|
||||
test('owned, queued, voted, dismissed still excluded', t => {
|
||||
const { item, candidates } = fixtureFeed(t);
|
||||
for (const id of [210, 211, 212, 213]) {
|
||||
assert.ok(candidates.some(c => c.id === `anilist:${id}`), `anilist:${id} is in the pool`);
|
||||
assert.equal(item(id), undefined, `anilist:${id} must not be suggested`);
|
||||
}
|
||||
});
|
||||
|
||||
test('slot mix', () => {
|
||||
const row = (id, score, group, extra = {}) => ({ candidateId: `anilist:${id}`, score, group, tags: ['Drama'],
|
||||
scoreParts: { fresh: 0 }, ...extra });
|
||||
const rows = [];
|
||||
for (let i = 0; i < 30; i++) rows.push(row(1000 + i, 30 - i, 'recs'));
|
||||
rows.push(row(2001, 0.5, 'fresh', { scoreParts: { fresh: 1 } }), row(2002, 0.4, 'fresh', { scoreParts: { fresh: 1 } }));
|
||||
rows.push(row(3001, 0.2, '1990s', { tags: ['Sports'] }), row(3002, 0.1, '1990s', { tags: [] }));
|
||||
const picked = mixSlots(rows, { topTags: [{ name: 'Drama', kind: 'genre' }] });
|
||||
assert.equal(SLOTS.perKind, 8);
|
||||
assert.equal(picked.filter(r => r.group === 'recs').length, 8);
|
||||
assert.ok(picked.some(r => r.candidateId === 'anilist:2001') && picked.some(r => r.candidateId === 'anilist:2002'));
|
||||
assert.ok(picked.some(r => r.candidateId === 'anilist:3001') && picked.some(r => r.candidateId === 'anilist:3002'));
|
||||
assert.equal(picked.length, 12, 'never pads up to 20');
|
||||
assert.deepEqual(picked.map(r => r.rank), picked.map((_, i) => i + 1));
|
||||
for (let i = 1; i < picked.length; i++) assert.ok(picked[i - 1].score >= picked[i].score);
|
||||
|
||||
const many = [];
|
||||
for (let i = 0; i < 40; i++) many.push(row(4000 + i, 100 - i, ['recs', 'creator', 'tagTop', 'fresh', '1980s'][i % 5],
|
||||
{ scoreParts: { fresh: i % 5 === 3 ? 1 : 0 } }));
|
||||
const full = mixSlots(many, { topTags: [] });
|
||||
assert.equal(full.length, 20);
|
||||
assert.ok(full.filter(r => r.group === '1980s').length <= SLOTS.perPeriod);
|
||||
});
|
||||
|
||||
test('slot mix reservations let low-scored fresh and outside rows past a full feed', () => {
|
||||
const row = (id, score, group, extra = {}) => ({ candidateId: `anilist:${id}`, score, group, tags: ['Drama'],
|
||||
scoreParts: { fresh: 0 }, ...extra });
|
||||
// 32 higher-scored rows that the caps alone would allow: 8 per seed kind, 4 per period.
|
||||
const rows = [];
|
||||
const groups = ['recs', 'creator', 'tagTop', '1980s', '1990s'];
|
||||
const caps = { recs: 8, creator: 8, tagTop: 8, '1980s': 4, '1990s': 4 };
|
||||
let id = 5000;
|
||||
for (const group of groups) for (let i = 0; i < caps[group]; i++) rows.push(row(id++, 100 - rows.length, group));
|
||||
assert.ok(rows.length > 20);
|
||||
rows.push(row(2001, 1, 'fresh', { scoreParts: { fresh: 1 } }), row(2002, 0.9, 'fresh', { scoreParts: { fresh: 1 } }));
|
||||
rows.push(row(3001, 0.2, '2000s', { tags: ['Sports'] }), row(3002, 0.1, '2000s', { tags: [] }));
|
||||
const picked = mixSlots(rows, { topTags: [{ name: 'Drama', kind: 'genre' }] });
|
||||
assert.equal(picked.length, 20);
|
||||
for (const low of ['anilist:2001', 'anilist:2002', 'anilist:3001', 'anilist:3002'])
|
||||
assert.ok(picked.some(r => r.candidateId === low), `${low} needs its reserved slot`);
|
||||
for (const group of groups) assert.ok(picked.filter(r => r.group === group).length <= caps[group], group);
|
||||
});
|
||||
|
||||
test('mergeUserFeed keeps other users\' referenced candidates', () => {
|
||||
const prior = { candidates: { a: { id: 'a' }, b: { id: 'b' }, c: { id: 'c' } },
|
||||
feeds: { alice: { items: [{ candidateId: 'a' }] }, bob: { items: [{ candidateId: 'b' }] }, off: { items: [], disabled: true } } };
|
||||
const result = { candidates: { d: { id: 'd' }, b: { id: 'b', fresh: true } }, feeds: { alice: { items: [{ candidateId: 'd' }] } } };
|
||||
const merged = mergeUserFeed(prior, result, 'alice');
|
||||
assert.deepEqual(Object.keys(merged.feeds).sort(), ['alice', 'bob', 'off']);
|
||||
assert.deepEqual(merged.feeds.alice.items, [{ candidateId: 'd' }]);
|
||||
assert.deepEqual(merged.feeds.bob, prior.feeds.bob);
|
||||
assert.deepEqual(Object.keys(merged.candidates).sort(), ['b', 'd']);
|
||||
assert.equal(merged.candidates.b.fresh, true);
|
||||
});
|
||||
|
||||
test('generateAsync matches generate and can score one user', async t => {
|
||||
const world = fixtureWorld(t);
|
||||
world.db.users.bob = { name: 'bob', settings: { discoveryEnabled: true } };
|
||||
const args = { db: world.db, library: world.library, greatest: world.greatest, cacheDir: NO_SCRAPER_CACHE,
|
||||
harvestCache: world.harvestCache, queuedTitles: world.queuedTitles, now: NOW };
|
||||
const sync = generate(args);
|
||||
const all = await generateAsync(args);
|
||||
assert.deepEqual(all.feeds.alice.items, sync.feeds.alice.items);
|
||||
assert.deepEqual(Object.keys(all.candidates), Object.keys(sync.candidates));
|
||||
const one = await generateAsync({ ...args, onlyUser: 'alice' });
|
||||
assert.deepEqual(Object.keys(one.feeds), ['alice']);
|
||||
assert.deepEqual(one.feeds.alice.items, sync.feeds.alice.items);
|
||||
});
|
||||
|
||||
test('scoring one user stays under 200 ms on 2000 candidates', t => {
|
||||
const world = fixtureWorld(t);
|
||||
const tagNames = ['Drama', 'Action', 'Romance', 'Time Skip', 'Comedy', 'Horror'];
|
||||
const candidates = [];
|
||||
for (let i = 0; i < 2000; i++) {
|
||||
const id = 10000 + i;
|
||||
candidates.push({ id: `anilist:${id}`, title: `Synthetic ${id}`, year: 2000 + (i % 27), type: 'MANGA',
|
||||
status: i % 3 ? 'completed' : 'ongoing', cover: null, metadataUrl: `https://anilist.co/manga/${id}`,
|
||||
tags: [tagNames[i % 6], tagNames[(i + 1) % 6]], authors: [`Writer ${i % 50}`], period: null, editorialRank: null,
|
||||
greatestId: null, evidence: [], relations: [], isAdult: i % 97 === 0, averageScore: 60 + (i % 40),
|
||||
tagRecords: [{ name: tagNames[i % 6], rank: 80, kind: 'tag', isAdult: false }],
|
||||
authorRecords: [{ staffId: i % 50, name: `Writer ${i % 50}`, role: 'Story' }], fetchedAt: FETCHED_AT, seed: 'recs' });
|
||||
}
|
||||
feedForUser({ ...world, candidates, now: NOW });
|
||||
const started = performance.now();
|
||||
const feed = feedForUser({ ...world, candidates, now: NOW });
|
||||
const elapsed = performance.now() - started;
|
||||
assert.ok(feed.items.length > 0);
|
||||
assert.ok(elapsed < 200, `took ${elapsed.toFixed(1)} ms`);
|
||||
});
|
||||
@@ -0,0 +1,27 @@
|
||||
'use strict';
|
||||
// The image copies top-level modules by name; a new module missing there crashes the reader at start.
|
||||
const test = require('node:test'), assert = require('node:assert/strict');
|
||||
const fs = require('fs'), path = require('path');
|
||||
const root = path.resolve(__dirname, '..');
|
||||
|
||||
test('every top-level module the reader requires is copied into the image', () => {
|
||||
const dockerfile = fs.readFileSync(path.join(root, 'Dockerfile'), 'utf8');
|
||||
const copied = new Set(dockerfile.split('\n').filter(line => line.startsWith('COPY ')).flatMap(line => line.split(/\s+/).slice(1, -1)));
|
||||
const pending = ['server.js'], seen = new Set();
|
||||
while (pending.length) {
|
||||
const file = pending.pop();
|
||||
if (seen.has(file)) continue;
|
||||
seen.add(file);
|
||||
for (const [, name] of fs.readFileSync(path.join(root, file), 'utf8').matchAll(/require\('\.\/([\w.-]+?)(?:\.js)?'\)/g)) {
|
||||
// A folder (e.g. ./greatest) is copied as a whole; only single files need their own name in COPY.
|
||||
if (fs.existsSync(path.join(root, name)) && fs.statSync(path.join(root, name)).isDirectory()) assert.ok(copied.has(name), `${name}/ is not copied`);
|
||||
else pending.push(name + '.js');
|
||||
}
|
||||
}
|
||||
// .dockerignore is an allowlist too: a file not let through never reaches COPY.
|
||||
const allowed = new Set(fs.readFileSync(path.join(root, '.dockerignore'), 'utf8').split(/\r?\n/).filter(line => line.startsWith('!')).map(line => line.slice(1)));
|
||||
for (const file of seen) {
|
||||
assert.ok(copied.has(file), `${file} is required but not copied by the Dockerfile`);
|
||||
assert.ok(allowed.has(file), `${file} is required but not allowed through .dockerignore`);
|
||||
}
|
||||
});
|
||||
Vendored
+10
@@ -0,0 +1,10 @@
|
||||
{"data":{"Staff":{"staffMedia":{"edges":[
|
||||
{"staffRole":"Story & Art","node":{"id":301,"format":"MANGA","isAdult":false,
|
||||
"title":{"romaji":"Hoshi no Uta","english":"Star Song","native":null},"synonyms":[],
|
||||
"status":"FINISHED","startDate":{"year":2018},"averageScore":79,"popularity":2500,"trending":3,
|
||||
"genres":["Fantasy"],"tags":[],"staff":{"edges":[]},"coverImage":{"extraLarge":null},"siteUrl":null}},
|
||||
{"staffRole":"Original Story","node":{"id":777,"format":"NOVEL","isAdult":false,
|
||||
"title":{"romaji":"Hon no Tabi","english":null,"native":null},"synonyms":[],
|
||||
"status":"FINISHED","startDate":{"year":2010},"averageScore":70,"popularity":1200,"trending":1,
|
||||
"genres":[],"tags":[],"staff":{"edges":[]},"coverImage":{"extraLarge":null},"siteUrl":null}}
|
||||
]}}}}
|
||||
Vendored
+6
@@ -0,0 +1,6 @@
|
||||
{"data":{"Page":{"media":[
|
||||
{"id":501,"format":"MANGA","isAdult":false,
|
||||
"title":{"romaji":"Atarashii Kaze","english":"New Wind","native":null},"synonyms":[],
|
||||
"status":"RELEASING","startDate":{"year":2025},"averageScore":72,"popularity":2100,"trending":150,
|
||||
"genres":["Sports"],"tags":[],"staff":{"edges":[]},"coverImage":{"extraLarge":null},"siteUrl":null}
|
||||
]}}}
|
||||
Vendored
+5
@@ -0,0 +1,5 @@
|
||||
{"data":{"Media":{"id":777,"format":"NOVEL","isAdult":false,
|
||||
"title":{"romaji":"Hon no Tabi","english":null,"native":null},
|
||||
"synonyms":[],"status":"FINISHED","startDate":{"year":2010},
|
||||
"averageScore":70,"popularity":1200,"trending":1,"genres":[],"tags":[],
|
||||
"staff":{"edges":[]},"coverImage":{"extraLarge":null},"siteUrl":"https://anilist.co/manga/777"}}}
|
||||
Vendored
+14
@@ -0,0 +1,14 @@
|
||||
{"data":{"Media":{"recommendations":{"nodes":[
|
||||
{"rating":12,"mediaRecommendation":{"id":201,"format":"MANGA","isAdult":false,
|
||||
"title":{"romaji":"Sora no Machi","english":"Sky Town","native":null},"synonyms":[],
|
||||
"status":"FINISHED","startDate":{"year":2015},"averageScore":77,"popularity":3000,"trending":5,
|
||||
"genres":["Drama"],"tags":[],"staff":{"edges":[]},"coverImage":{"extraLarge":null},"siteUrl":"https://anilist.co/manga/201"}},
|
||||
{"rating":-2,"mediaRecommendation":{"id":202,"format":"MANGA","isAdult":false,
|
||||
"title":{"romaji":"Kuroi Mori","english":null,"native":null},"synonyms":[],
|
||||
"status":"HIATUS","startDate":{"year":2012},"averageScore":60,"popularity":900,"trending":0,
|
||||
"genres":[],"tags":[],"staff":{"edges":[]},"coverImage":{"extraLarge":null},"siteUrl":null}},
|
||||
{"rating":9,"mediaRecommendation":{"id":777,"format":"NOVEL","isAdult":false,
|
||||
"title":{"romaji":"Hon no Tabi","english":null,"native":null},"synonyms":[],
|
||||
"status":"FINISHED","startDate":{"year":2010},"averageScore":70,"popularity":1200,"trending":1,
|
||||
"genres":[],"tags":[],"staff":{"edges":[]},"coverImage":{"extraLarge":null},"siteUrl":null}}
|
||||
]}}}}
|
||||
Vendored
+11
@@ -0,0 +1,11 @@
|
||||
{"data":{"Page":{"media":[
|
||||
{"id":401,"format":"ONE_SHOT","isAdult":false,
|
||||
"title":{"romaji":"Ichi Wa","english":"One Tale","native":null},"synonyms":[],
|
||||
"status":"FINISHED","startDate":{"year":2020},"averageScore":74,"popularity":1800,"trending":2,
|
||||
"genres":["Mystery"],"tags":[{"name":"Detective","rank":90,"isAdult":false}],"staff":{"edges":[]},
|
||||
"coverImage":{"extraLarge":null},"siteUrl":null},
|
||||
{"id":777,"format":"NOVEL","isAdult":false,
|
||||
"title":{"romaji":"Hon no Tabi","english":null,"native":null},"synonyms":[],
|
||||
"status":"FINISHED","startDate":{"year":2010},"averageScore":70,"popularity":1200,"trending":1,
|
||||
"genres":[],"tags":[],"staff":{"edges":[]},"coverImage":{"extraLarge":null},"siteUrl":null}
|
||||
]}}}
|
||||
Vendored
+11
@@ -0,0 +1,11 @@
|
||||
{"data":{"Media":{"id":101,"format":"MANGA","isAdult":false,
|
||||
"title":{"romaji":"Tsuki no Tabibito","english":"Moon Traveller","native":"月の旅人"},
|
||||
"synonyms":["Lunar Wanderer"],"status":"RELEASING","startDate":{"year":2019},
|
||||
"averageScore":81,"popularity":5400,"trending":37,
|
||||
"genres":["Adventure","Fantasy"],
|
||||
"tags":[{"name":"Travel","rank":88,"isAdult":false},{"name":"Nudity","rank":40,"isAdult":true}],
|
||||
"staff":{"edges":[
|
||||
{"role":"Story & Art","node":{"id":900,"name":{"full":"Aoi Example"}}},
|
||||
{"role":"Translator","node":{"id":901,"name":{"full":"Not An Author"}}}]},
|
||||
"coverImage":{"extraLarge":"https://example.invalid/cover/101.jpg"},
|
||||
"siteUrl":"https://anilist.co/manga/101"}}}
|
||||
@@ -0,0 +1,75 @@
|
||||
'use strict';
|
||||
// BL-0053: Greatest covers are served from the reader's own copy, with a redirect until one exists.
|
||||
const test = require('node:test'), assert = require('node:assert/strict');
|
||||
const fs = require('fs'), os = require('os'), path = require('path'), net = require('net'), crypto = require('crypto');
|
||||
const { spawn } = require('child_process');
|
||||
const { loadCatalog } = require('../greatest');
|
||||
const { GreatestCovers } = require('../greatest/covers');
|
||||
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
const hashPw = pw => { const salt = crypto.randomBytes(16).toString('hex'); return salt + ':' + crypto.scryptSync(pw, salt, 32).toString('hex'); };
|
||||
const sha = s => crypto.createHash('sha256').update(s).digest('hex');
|
||||
const PNG = Buffer.from('89504e470d0a1a0a0000000d49484452', 'hex');
|
||||
|
||||
async function freePort() {
|
||||
const listener = net.createServer();
|
||||
await new Promise(resolve => listener.listen(0, '127.0.0.1', resolve));
|
||||
const port = listener.address().port;
|
||||
await new Promise(resolve => listener.close(resolve));
|
||||
return port;
|
||||
}
|
||||
|
||||
test('Greatest covers come from the local copy, else redirect to the catalog URL', async t => {
|
||||
const entries = loadCatalog().entries.filter(entry => entry.cover);
|
||||
const [cached, missing] = entries;
|
||||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-greatest-cover-'));
|
||||
const library = path.join(root, 'library'), data = path.join(root, 'data');
|
||||
fs.mkdirSync(library, { recursive: true });
|
||||
fs.mkdirSync(path.join(data, 'greatest-covers'), { recursive: true });
|
||||
const version = new GreatestCovers(path.join(data, 'greatest-covers')).version(cached.cover);
|
||||
fs.writeFileSync(path.join(data, 'greatest-covers', `${version}.png`), PNG);
|
||||
const token = crypto.randomBytes(32).toString('base64url');
|
||||
fs.writeFileSync(path.join(data, 'db.json'), JSON.stringify({
|
||||
users: { reader: { name: 'reader', hash: hashPw('reader-password'), admin: false, created: Date.now(), settings: {} } },
|
||||
sessions: { [sha(token)]: { u: 'reader', t: Date.now() } } }));
|
||||
|
||||
const port = await freePort(), output = [];
|
||||
const child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
|
||||
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
|
||||
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '',
|
||||
GREATEST_COVER_WARM: '0' } });
|
||||
child.stdout.on('data', d => output.push(String(d)));
|
||||
child.stderr.on('data', d => output.push(String(d)));
|
||||
t.after(async () => {
|
||||
await new Promise(resolve => { if (child.exitCode !== null) return resolve(); child.once('exit', resolve); child.kill(); });
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
const base = `http://127.0.0.1:${port}`;
|
||||
let ready = false;
|
||||
for (let n = 0; n < 100 && !ready; n++) {
|
||||
ready = await fetch(base + '/health').then(r => r.ok).catch(() => false);
|
||||
if (!ready) await sleep(50);
|
||||
}
|
||||
assert.ok(ready, output.join(''));
|
||||
const get = (url, cookie = `sid=${token}`) => fetch(base + '/' + url, { headers: cookie ? { cookie } : {}, redirect: 'manual' });
|
||||
|
||||
const listing = await (await get('api/greatest')).json();
|
||||
const coverOf = id => listing.entries.find(entry => entry.id === id).cover;
|
||||
assert.equal(coverOf(cached.id), `api/greatest-cover?id=${encodeURIComponent(cached.id)}&v=${version}`);
|
||||
assert.match(coverOf(missing.id), /^api\/greatest-cover\?id=.+&v=[0-9a-f]{16}$/);
|
||||
|
||||
const hit = await get(coverOf(cached.id));
|
||||
assert.equal(hit.status, 200);
|
||||
assert.equal(hit.headers.get('content-type'), 'image/png');
|
||||
assert.equal(hit.headers.get('cache-control'), 'private, max-age=31536000, immutable');
|
||||
assert.deepEqual(Buffer.from(await hit.arrayBuffer()), PNG);
|
||||
|
||||
const miss = await get(coverOf(missing.id));
|
||||
assert.equal(miss.status, 302);
|
||||
assert.equal(miss.headers.get('location'), missing.cover);
|
||||
assert.equal(miss.headers.get('cache-control'), 'no-store');
|
||||
|
||||
assert.equal((await get(coverOf(cached.id), null)).status, 401);
|
||||
assert.equal((await get('api/greatest-cover?id=not-in-the-catalog')).status, 404);
|
||||
assert.equal((await get('api/greatest-cover?id=../../db.json')).status, 404);
|
||||
});
|
||||
@@ -0,0 +1,84 @@
|
||||
'use strict';
|
||||
const test = require('node:test');
|
||||
const assert = require('node:assert/strict');
|
||||
const fs = require('node:fs');
|
||||
const os = require('node:os');
|
||||
const path = require('node:path');
|
||||
const { GreatestCovers } = require('../greatest/covers');
|
||||
|
||||
const PNG = Buffer.from('89504e470d0a1a0a0000000d49484452', 'hex');
|
||||
const image = (body = PNG, type = 'image/png') => new Response(body, { status: 200, headers: { 'content-type': type } });
|
||||
|
||||
function setup(t, responses, options = {}) {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-greatest-covers-'));
|
||||
t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
|
||||
const calls = [];
|
||||
const fetchImpl = async (url, init) => {
|
||||
calls.push({ url, init });
|
||||
const next = responses[url];
|
||||
if (next instanceof Error) throw next;
|
||||
return typeof next === 'function' ? next() : next || new Response('missing', { status: 404 });
|
||||
};
|
||||
const sleeps = [];
|
||||
const covers = new GreatestCovers(dir, { fetchImpl, sleep: async ms => { sleeps.push(ms); }, ...options });
|
||||
return { dir, calls, sleeps, covers };
|
||||
}
|
||||
|
||||
test('warm downloads each cover once and serves it from disk afterwards', async t => {
|
||||
const a = 'https://img.example/a.png', b = 'https://img.example/b.jpg';
|
||||
const { dir, calls, sleeps, covers } = setup(t, { [a]: () => image(), [b]: () => image(PNG, 'image/jpeg') });
|
||||
const result = await covers.warm([a, b, a]);
|
||||
assert.deepEqual(result, { fetched: 2, skipped: 1, failed: 0, stopped: null });
|
||||
assert.equal(calls.length, 2);
|
||||
assert.deepEqual(sleeps, [1000], 'one pause between two downloads, none before the first');
|
||||
assert.ok(covers.file(a).endsWith('.png') && fs.existsSync(covers.file(a)));
|
||||
assert.ok(covers.file(b).endsWith('.jpg'));
|
||||
assert.equal(covers.file('https://img.example/other.png'), null);
|
||||
|
||||
// A restarted reader finds the stored files and asks nobody.
|
||||
const reopened = new GreatestCovers(dir, { fetchImpl: async () => { throw new Error('must not fetch'); } });
|
||||
assert.deepEqual(await reopened.warm([a, b]), { fetched: 0, skipped: 2, failed: 0, stopped: null });
|
||||
assert.equal(reopened.file(a), covers.file(a));
|
||||
});
|
||||
|
||||
test('a changed cover URL is a new file', async t => {
|
||||
const { covers } = setup(t, { 'https://img.example/v1.png': () => image(), 'https://img.example/v2.png': () => image() });
|
||||
await covers.warm(['https://img.example/v1.png', 'https://img.example/v2.png']);
|
||||
assert.notEqual(covers.file('https://img.example/v1.png'), covers.file('https://img.example/v2.png'));
|
||||
assert.notEqual(covers.version('https://img.example/v1.png'), covers.version('https://img.example/v2.png'));
|
||||
});
|
||||
|
||||
test('non-images, oversized files, plain http and errors are skipped', async t => {
|
||||
const html = 'https://img.example/page', big = 'https://img.example/big.png', declaredBig = 'https://img.example/declared.png';
|
||||
const broken = 'https://img.example/broken.png', gone = 'https://img.example/gone.png';
|
||||
const { covers, calls } = setup(t, {
|
||||
[html]: () => new Response('<html>', { status: 200, headers: { 'content-type': 'text/html' } }),
|
||||
[big]: () => image(Buffer.alloc(64)),
|
||||
[declaredBig]: () => new Response(PNG, { status: 200, headers: { 'content-type': 'image/png', 'content-length': '999999' } }),
|
||||
[broken]: new TypeError('fetch failed'),
|
||||
}, { maxBytes: 32 });
|
||||
const result = await covers.warm([html, big, declaredBig, broken, gone, 'http://img.example/plain.png', '', null]);
|
||||
assert.deepEqual(result, { fetched: 0, skipped: 0, failed: 5, stopped: null });
|
||||
assert.equal(calls.length, 5, 'plain http and empty URLs are never requested');
|
||||
for (const url of [html, big, declaredBig, broken, gone]) assert.equal(covers.file(url), null);
|
||||
assert.deepEqual(fs.readdirSync(covers.dir).filter(name => !name.startsWith('.')), []);
|
||||
});
|
||||
|
||||
test('429 and 403 stop the run at once', async t => {
|
||||
for (const status of [429, 403]) {
|
||||
const first = 'https://img.example/1.png', second = 'https://img.example/2.png';
|
||||
const { covers, calls } = setup(t, { [first]: () => new Response('', { status }), [second]: () => image() });
|
||||
const result = await covers.warm([first, second]);
|
||||
assert.equal(result.stopped, `HTTP ${status}`);
|
||||
assert.equal(calls.length, 1);
|
||||
assert.equal(covers.file(second), null);
|
||||
}
|
||||
});
|
||||
|
||||
test('requests carry a timeout and a fixed user agent', async t => {
|
||||
const a = 'https://img.example/a.png';
|
||||
const { covers, calls } = setup(t, { [a]: () => image() });
|
||||
await covers.warm([a]);
|
||||
assert.ok(calls[0].init.signal instanceof AbortSignal);
|
||||
assert.match(calls[0].init.headers['user-agent'], /^MangaCircus/);
|
||||
});
|
||||
@@ -78,7 +78,7 @@ test('reader serves a saved index immediately after restart and verifies it in b
|
||||
const output = [];
|
||||
child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
|
||||
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
|
||||
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '' } });
|
||||
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', GREATEST_COVER_WARM: '0' } });
|
||||
child.stdout.on('data', data => output.push(String(data)));
|
||||
child.stderr.on('data', data => output.push(String(data)));
|
||||
const base = `http://127.0.0.1:${port}`;
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
'use strict';
|
||||
// BL-0055: Moments, a reader's saved pages or page regions with exactly one category.
|
||||
const test = require('node:test'), assert = require('node:assert/strict');
|
||||
const fs = require('fs'), os = require('os'), path = require('path'), net = require('net');
|
||||
const { spawn } = require('child_process');
|
||||
const { Canvas } = require('../tools/png');
|
||||
const moments = require('../moments');
|
||||
const { trimPages, momentPageLabel } = require('../public/moments');
|
||||
const { READER_KEYS, MOMENT_VIEWER_KEYS, keyAction } = require('../public/keys');
|
||||
|
||||
test('regions are clamped fractions; a whole-page rectangle means no region', () => {
|
||||
assert.deepEqual(moments.cleanRegion({ x: 0.1, y: 0.2, w: 0.3, h: 0.4 }), { x: 0.1, y: 0.2, w: 0.3, h: 0.4 });
|
||||
assert.deepEqual(moments.cleanRegion({ x: -1, y: 0.5, w: 3, h: 2 }), { x: 0, y: 0.5, w: 1, h: 0.5 });
|
||||
assert.equal(moments.cleanRegion({ x: 0, y: 0, w: 1, h: 1 }), null);
|
||||
assert.equal(moments.cleanRegion(null), null);
|
||||
assert.throws(() => moments.cleanRegion({ x: 0.5, y: 0.5, w: 0.001, h: 0.5 }), /too small/);
|
||||
assert.throws(() => moments.cleanRegion({ x: 'a', y: 0, w: 1, h: 1 }), /Invalid/);
|
||||
});
|
||||
|
||||
test('exactly one known category is required; notes are trimmed and capped', () => {
|
||||
for (const cat of ['twist', 'hot', 'awesome', 'cute', 'funny', 'feels']) assert.equal(moments.cleanCategory(cat), cat);
|
||||
assert.throws(() => moments.cleanCategory(undefined), /Pick one category/);
|
||||
assert.throws(() => moments.cleanCategory('meh'), /Pick one category/);
|
||||
assert.equal(moments.cleanNote(' so good \n'), 'so good');
|
||||
assert.equal(moments.cleanNote('x'.repeat(500)).length, moments.NOTE_MAX);
|
||||
});
|
||||
|
||||
const near = (a, b) => Math.abs(a - b) < 1e-9;
|
||||
test('side by side: a region inside one page of a spread becomes that page alone', () => {
|
||||
const left = { i: 5, w: 700, h: 1000 }, right = { i: 4, w: 700, h: 1000 };
|
||||
const one = trimPages([left, right], { x: 0.6, y: 0.1, w: 0.2, h: 0.3 });
|
||||
assert.deepEqual(one.pages, [right]);
|
||||
assert.ok(near(one.region.x, 0.2) && near(one.region.w, 0.4) && one.region.y === 0.1);
|
||||
assert.equal(trimPages([left, right], { x: 0.3, y: 0.1, w: 0.4, h: 0.3 }).pages.length, 2);
|
||||
});
|
||||
|
||||
test('stacked (Flow): a picture across the break keeps only the pages it touches', () => {
|
||||
const pages = [0, 1, 2, 3].map(i => ({ i, w: 800, h: 400 })); // each page is half as tall as wide: 0.5 of the width
|
||||
// The stack is 2 units tall; y 0.4–0.6 is the lower half of page 1 and the upper half of page 2.
|
||||
const cut = trimPages(pages, { x: 0.1, y: 0.4, w: 0.8, h: 0.2 }, 'column');
|
||||
assert.deepEqual(cut.pages.map(p => p.i), [1, 2]);
|
||||
assert.ok(near(cut.region.y, 0.3) && near(cut.region.h, 0.4) && cut.region.x === 0.1);
|
||||
assert.deepEqual(trimPages(pages, null, 'column').pages.length, 4);
|
||||
assert.equal(momentPageLabel(cut.pages), 'p. 2–3');
|
||||
});
|
||||
|
||||
test('every reader key is bound once and the M / I split holds', () => {
|
||||
const keys = READER_KEYS.flatMap(entry => entry.keys);
|
||||
assert.equal(new Set(keys).size, keys.length);
|
||||
assert.equal(keyAction(READER_KEYS, 'm'), 'moment');
|
||||
assert.equal(keyAction(READER_KEYS, 'i'), 'bars');
|
||||
assert.equal(keyAction(READER_KEYS, 'Escape'), 'bars');
|
||||
assert.equal(keyAction(READER_KEYS, '?'), 'help');
|
||||
assert.equal(keyAction(MOMENT_VIEWER_KEYS, 'ArrowLeft'), 'step');
|
||||
assert.ok(READER_KEYS.every(entry => entry.label && entry.text));
|
||||
});
|
||||
|
||||
async function freePort() {
|
||||
const listener = net.createServer();
|
||||
await new Promise(resolve => listener.listen(0, '127.0.0.1', resolve));
|
||||
const port = listener.address().port;
|
||||
await new Promise(resolve => listener.close(resolve));
|
||||
return port;
|
||||
}
|
||||
|
||||
test('moments are private, validated, editable and survive page renumbering as "moved"', { timeout: 30000 }, async () => {
|
||||
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-moments-'));
|
||||
let child;
|
||||
try {
|
||||
const library = path.join(root, 'library'), data = path.join(root, 'data');
|
||||
const chapterDir = path.join(library, 'Moon Garden', 'v01', 'c001 Roots');
|
||||
fs.mkdirSync(chapterDir, { recursive: true }); fs.mkdirSync(data);
|
||||
for (const n of [1, 2, 3]) fs.writeFileSync(path.join(chapterDir, `00${n}.png`), new Canvas(60, 90, [40 * n, 60, 120]).png());
|
||||
const port = await freePort(), output = [];
|
||||
child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
|
||||
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
|
||||
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '', GREATEST_COVER_WARM: '0' } });
|
||||
child.stdout.on('data', d => output.push(String(d)));
|
||||
child.stderr.on('data', d => output.push(String(d)));
|
||||
const base = `http://127.0.0.1:${port}`;
|
||||
let ready = false;
|
||||
for (let n = 0; n < 100 && !ready; n++) {
|
||||
ready = await fetch(base + '/health').then(r => r.ok).catch(() => false);
|
||||
if (!ready) await new Promise(resolve => setTimeout(resolve, 50));
|
||||
}
|
||||
assert.ok(ready, output.join(''));
|
||||
const post = (cookie, url, body) => fetch(base + url, { method: 'POST',
|
||||
headers: { 'content-type': 'application/json', ...(cookie ? { cookie } : {}) }, body: JSON.stringify(body) });
|
||||
const get = (cookie, url) => fetch(base + url, { headers: { cookie } }).then(r => r.json());
|
||||
const session = r => r.headers.get('set-cookie').split(';')[0];
|
||||
|
||||
const alice = session(await post(null, '/api/signup', { name: 'alice', password: 'alice-password' }));
|
||||
await post(alice, '/api/admin/user', { name: 'bob', password: 'bob-password' });
|
||||
const bob = session(await post(null, '/api/login', { name: 'bob', password: 'bob-password' }));
|
||||
await post(alice, '/api/admin/rescan', { full: true });
|
||||
|
||||
const manga = await get(alice, '/api/manga?m=' + encodeURIComponent('Moon Garden'));
|
||||
const c = manga.chapters[0].id, v = manga.chapters[0].versions[0].id;
|
||||
const base3 = { m: 'Moon Garden', c, v };
|
||||
|
||||
// A category is required: nothing is ever dumped into an uncategorised pile.
|
||||
assert.equal((await post(alice, '/api/moments', { ...base3, pages: [1] })).status, 400);
|
||||
assert.equal((await post(alice, '/api/moments', { ...base3, pages: [9], cat: 'hot' })).status, 400);
|
||||
|
||||
const created = await (await post(alice, '/api/moments', { ...base3, pages: [1], cat: 'awesome', region: { x: 0.1, y: 0.5, w: 0.5, h: 0.25 }, note: ' big fight ' })).json();
|
||||
assert.equal(created.moment.cat, 'awesome');
|
||||
assert.equal(created.moment.note, 'big fight');
|
||||
assert.equal(created.moment.pages[0].i, 1);
|
||||
assert.match(created.moment.pages[0].u, /^lib\/Moon%20Garden\/v01\/c001%20Roots\/002\.png$/);
|
||||
const spread = (await (await post(alice, '/api/moments', { ...base3, pages: [2, 0], cat: 'twist' })).json()).moment;
|
||||
assert.deepEqual(spread.pages.map(p => p.i), [2, 0]);
|
||||
assert.equal(spread.layout, 'row');
|
||||
// Flow: a picture stacked over three pages; more than four pages or an unknown layout is refused.
|
||||
const stacked = (await (await post(alice, '/api/moments', { ...base3, pages: [0, 1, 2], layout: 'column', cat: 'awesome', region: { x: 0, y: 0.2, w: 1, h: 0.5 } })).json()).moment;
|
||||
assert.equal(stacked.layout, 'column');
|
||||
assert.equal((await post(alice, '/api/moments', { ...base3, pages: [0, 1, 2, 0], cat: 'hot' })).status, 400);
|
||||
assert.equal((await post(alice, '/api/moments', { ...base3, pages: [0], layout: 'grid', cat: 'hot' })).status, 400);
|
||||
assert.equal((await post(alice, '/api/moments/delete', { id: stacked.id })).status, 200);
|
||||
|
||||
// Story order: page 0/2 spread starts before page 1.
|
||||
const listed = (await get(alice, '/api/moments?m=Moon%20Garden')).moments;
|
||||
assert.deepEqual(listed.map(x => x.id), [spread.id, created.moment.id]);
|
||||
const chapter = await get(alice, `/api/chapter?m=Moon%20Garden&c=${encodeURIComponent(c)}`);
|
||||
assert.deepEqual(chapter.moments.map(x => x.pages).sort(), [[1], [2, 0]].sort());
|
||||
|
||||
// Private: bob sees nothing and cannot touch alice's moments.
|
||||
assert.deepEqual((await get(bob, '/api/moments?m=Moon%20Garden')).moments, []);
|
||||
assert.equal((await get(bob, `/api/chapter?m=Moon%20Garden&c=${encodeURIComponent(c)}`)).moments.length, 0);
|
||||
assert.equal((await post(bob, '/api/moments/update', { id: created.moment.id, cat: 'hot' })).status, 404);
|
||||
assert.equal((await post(bob, '/api/moments/delete', { id: created.moment.id })).status, 404);
|
||||
|
||||
const edited = (await (await post(alice, '/api/moments/update', { id: created.moment.id, cat: 'feels', region: null })).json()).moment;
|
||||
assert.equal(edited.cat, 'feels'); assert.equal(edited.region, null);
|
||||
assert.equal((await post(alice, '/api/moments/update', { id: created.moment.id, cat: '' })).status, 400);
|
||||
|
||||
// The file of page 2 goes away: the moment shows as moved instead of pointing at another picture.
|
||||
fs.renameSync(path.join(chapterDir, '002.png'), path.join(chapterDir, '009.png'));
|
||||
await post(alice, '/api/admin/rescan', { full: true });
|
||||
const after = (await get(alice, '/api/moments?m=Moon%20Garden')).moments;
|
||||
assert.equal(after.find(x => x.id === created.moment.id).missing, true);
|
||||
assert.equal(after.find(x => x.id === spread.id).missing, false);
|
||||
|
||||
assert.equal((await post(alice, '/api/moments/delete', { id: spread.id })).status, 200);
|
||||
assert.equal((await get(alice, '/api/moments?m=Moon%20Garden')).moments.length, 1);
|
||||
} finally {
|
||||
if (child) child.kill();
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
@@ -29,7 +29,7 @@ test('three services integrate collections, auth, custom covers and private heal
|
||||
const token = 'integration-test-only';
|
||||
const env = { ...process.env, DATA: data, LIBRARY: library, SCRAPER_DATA: path.join(data, 'scraper'),
|
||||
PORT: String(rp), SCRAPER_PORT: String(sp), SCRAPER_HOST: '127.0.0.1',
|
||||
READER_URL: base, SCRAPER_URL: scraper, SCRAPER_TOKEN: token, SCRAPER_TOKEN_FILE: '', SCAN_INTERVAL: '0', COOKIE_SECURE: '0' };
|
||||
READER_URL: base, SCRAPER_URL: scraper, SCRAPER_TOKEN: token, SCRAPER_TOKEN_FILE: '', SCAN_INTERVAL: '0', COOKIE_SECURE: '0', GREATEST_COVER_WARM: '0' };
|
||||
for (const script of ['server.js', 'scraper/daemon.js']) {
|
||||
const child = spawn(process.execPath, [script], { cwd: root, env, windowsHide: true }); children.push(child);
|
||||
for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => output.push(chunk.toString()));
|
||||
@@ -67,6 +67,10 @@ test('three services integrate collections, auth, custom covers and private heal
|
||||
assert.equal((await (await request('/api/settings', { pageTone: 'sepia' })).json()).settings.pageTone, 'sepia');
|
||||
assert.equal((await (await request('/api/settings', { pageTone: 'invalid' })).json()).settings.pageTone, 'sepia');
|
||||
assert.equal((await (await anotherRequest('/api/me')).json()).settings.pageTone, 'original');
|
||||
assert.equal((await (await request('/api/me')).json()).settings.continueRow, true);
|
||||
assert.equal((await (await request('/api/settings', { continueRow: false })).json()).settings.continueRow, false);
|
||||
assert.equal((await (await request('/api/settings', { continueRow: 'no' })).json()).settings.continueRow, false);
|
||||
assert.equal((await (await anotherRequest('/api/me')).json()).settings.continueRow, true);
|
||||
assert.equal((await fetch(base + '/api/discovery')).status, 401);
|
||||
assert.equal((await anotherRequest('/api/admin/discovery/run', {})).status, 403);
|
||||
assert.equal((await request('/api/settings', { discoveryEnabled: true })).status, 200);
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
// One-time migration to chapter dates (BL-0047). Dry run by default; apply only with the
|
||||
// scraper stopped (--apply --writers-stopped), then run a deep "Rescan" in Settings.
|
||||
// - WeebCentral: x-dates.source from the series' full chapter list (one request per series).
|
||||
// - MangaDex: chapters published before x-dates stored the group's publishAt in "added";
|
||||
// it moves to x-dates.source and "added" becomes the folder's arrival time.
|
||||
// - Every chapter: x-dates.ecosystem = "added" unless an earlier one is already known.
|
||||
const fs = require('node:fs').promises;
|
||||
const path = require('node:path');
|
||||
const { batchFlags, sourceDate, publishedDates } = require('../chapter-dates');
|
||||
const weebcentral = require('../scraper/plugins/weebcentral');
|
||||
const { writeJsonAtomic } = require('../scraper/lib/publisher');
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
const apply = args.includes('--apply');
|
||||
const only = args.find(arg => arg.startsWith('--manga='))?.slice('--manga='.length);
|
||||
const offline = args.includes('--offline');
|
||||
const root = args.find(arg => !arg.startsWith('--'));
|
||||
if (!root || args.some(arg => arg.startsWith('--') && !['--apply', '--writers-stopped', '--offline'].includes(arg) && !arg.startsWith('--manga='))
|
||||
|| (apply && !args.includes('--writers-stopped'))) {
|
||||
console.error('Usage: node tools/backfill-chapter-dates.js <library> [--manga=<folder>] [--offline] [--apply --writers-stopped]');
|
||||
process.exit(2);
|
||||
}
|
||||
const UA = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130 Safari/537.36';
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
const readJson = async file => { try { return JSON.parse(await fs.readFile(file, 'utf8')); } catch { return null; } };
|
||||
const dirs = async dir => (await fs.readdir(dir, { withFileTypes: true })).filter(e => e.isDirectory() && !e.name.startsWith('.')).map(e => e.name).sort();
|
||||
|
||||
async function weebCentralUploads(seriesId) {
|
||||
const url = `https://weebcentral.com/series/${seriesId}/full-chapter-list`;
|
||||
const response = await fetch(url, { headers: { 'user-agent': UA, accept: 'text/html' }, signal: AbortSignal.timeout(30000) });
|
||||
if (!response.ok) throw new Error(`WeebCentral HTTP ${response.status}`);
|
||||
const { chapters } = weebcentral.catalog({ html: await response.text(), url });
|
||||
return new Map(chapters.filter(c => c.uploadedAt).map(c => [c.id, sourceDate(c.uploadedAt, 'WeebCentral', c.uploadBatch)]));
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const totals = { chapters: 0, changed: 0, alreadyDated: 0, weebSource: 0, weebBatch: 0, mangaDexMoved: 0, missing: 0 };
|
||||
for (const manga of await dirs(root)) {
|
||||
if (only && manga !== only) continue;
|
||||
const mangaDir = path.join(root, manga), meta = (await readJson(path.join(mangaDir, 'manga.json'))) || {};
|
||||
const versions = [];
|
||||
for (const book of await dirs(mangaDir)) {
|
||||
for (const chapter of await dirs(path.join(mangaDir, book))) {
|
||||
const dir = path.join(mangaDir, book, chapter), file = path.join(dir, 'chapter.json'), chapterMeta = await readJson(file);
|
||||
if (chapterMeta) versions.push({ dir, file, meta: chapterMeta, mtime: (await fs.stat(dir)).mtimeMs });
|
||||
}
|
||||
}
|
||||
let uploads = new Map();
|
||||
const seriesId = meta['x-scraper']?.ids?.weebcentral;
|
||||
if (seriesId && !offline && versions.some(v => /weebcentral\.com\/chapters\//.test(v.meta.source?.url || '') && !v.meta['x-dates'])) {
|
||||
try { uploads = await weebCentralUploads(seriesId); await sleep(2000); }
|
||||
catch (error) { console.warn(`[backfill] ${manga}: WeebCentral list failed: ${error.message}`); }
|
||||
}
|
||||
// Legacy MangaDex "added" values are group upload times; flag bulk uploads among them.
|
||||
const legacyDex = versions.filter(v => !v.meta['x-dates'] && /^MangaDex\b/.test(v.meta.source?.name || '') && Date.parse(v.meta.added));
|
||||
const dexBatch = new Map(batchFlags(legacyDex.map(v => v.meta.added)).map((flag, i) => [legacyDex[i], flag]));
|
||||
let changed = 0;
|
||||
for (const version of versions) {
|
||||
totals.chapters++;
|
||||
if (version.meta['x-dates']) { totals.alreadyDated++; continue; }
|
||||
const prior = { ...version.meta }, incoming = {};
|
||||
const weebId = /weebcentral\.com\/chapters\/([0-9A-Z]{26})/.exec(prior.source?.url || '')?.[1];
|
||||
if (weebId && uploads.has(weebId)) {
|
||||
incoming.source = uploads.get(weebId);
|
||||
totals[incoming.source.batch ? 'weebBatch' : 'weebSource']++;
|
||||
} else if (dexBatch.has(version)) {
|
||||
incoming.source = sourceDate(prior.added, prior.source.name, dexBatch.get(version));
|
||||
prior.added = new Date(version.mtime).toISOString();
|
||||
totals.mangaDexMoved++;
|
||||
} else totals.missing++;
|
||||
const next = { ...prior, ...publishedDates(prior, incoming, version.mtime) };
|
||||
changed++; totals.changed++;
|
||||
if (apply) await writeJsonAtomic(version.file, next);
|
||||
}
|
||||
if (changed) console.log(`[backfill] ${manga}: ${changed} of ${versions.length} chapter versions ${apply ? 'updated' : 'would change'}`);
|
||||
}
|
||||
console.log(JSON.stringify({ apply, ...totals }));
|
||||
if (apply) console.log('Run a deep "Rescan" in Settings so the reader picks up the edited chapter.json files.');
|
||||
}
|
||||
|
||||
main().catch(error => { console.error(error); process.exit(1); });
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
// Official chapter release dates from MangaUpdates (BL-0048) for a whole library or one manga.
|
||||
// Dry run by default. Apply only with the scraper stopped (--apply --writers-stopped), then let a
|
||||
// full reconciliation (or admin Rescan) pick up the edited chapter.json files.
|
||||
const fs = require('node:fs').promises;
|
||||
const path = require('node:path');
|
||||
const { ReleaseDateSources, refreshMangaReleaseDates } = require('../scraper/metadata/release-dates');
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
const apply = args.includes('--apply');
|
||||
const only = args.find(arg => arg.startsWith('--manga='))?.slice('--manga='.length);
|
||||
const cacheDir = args.find(arg => arg.startsWith('--cache='))?.slice('--cache='.length) || path.join(require('node:os').tmpdir(), 'mangacircus-release-dates');
|
||||
const root = args.find(arg => !arg.startsWith('--'));
|
||||
if (!root || args.some(arg => arg.startsWith('--') && !['--apply', '--writers-stopped'].includes(arg) && !/^--(manga|cache)=/.test(arg))
|
||||
|| (apply && !args.includes('--writers-stopped'))) {
|
||||
console.error('Usage: node tools/release-dates.js <library> [--manga=<folder>] [--cache=<dir>] [--apply --writers-stopped]');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const sources = new ReleaseDateSources({ cacheDir });
|
||||
const totals = { manga: 0, matched: 0, chapters: 0, accepted: 0, written: 0, cleared: 0, outsideRun: 0, earlier: 0, outOfOrder: 0, noRun: 0 };
|
||||
const names = (await fs.readdir(root, { withFileTypes: true })).filter(e => e.isDirectory() && !e.name.startsWith('.')).map(e => e.name).sort();
|
||||
for (const name of names) {
|
||||
if (only && name !== only) continue;
|
||||
try {
|
||||
const s = await refreshMangaReleaseDates(path.join(root, name), sources, { apply });
|
||||
totals.manga++; totals.chapters += s.chapters;
|
||||
if (s.ambiguous) { console.log(`${name.slice(0, 44).padEnd(44)} skipped: chapter numbers repeat across books`); continue; }
|
||||
if (!s.series) { console.log(`${name.slice(0, 44).padEnd(44)} no MangaUpdates match`); continue; }
|
||||
totals.matched++; totals.accepted += s.accepted; totals.written += s.written; totals.cleared += s.cleared;
|
||||
for (const k of ['outsideRun', 'earlier', 'outOfOrder', 'noRun']) totals[k] += s.rejected[k];
|
||||
console.log(`${name.slice(0, 44).padEnd(44)} ${String(s.chapters).padStart(5)} ch official ${String(s.official).padStart(4)} accepted ${String(s.accepted).padStart(4)}`
|
||||
+ ` rejected: outside run ${s.rejected.outsideRun}, earlier evidence ${s.rejected.earlier}, out of order ${s.rejected.outOfOrder}`
|
||||
+ `${s.rejected.noRun ? `, no AniList run ${s.rejected.noRun}` : ''} (${s.series.title})`);
|
||||
} catch (error) { console.warn(`${name}: ${error.message}`); }
|
||||
}
|
||||
console.log(JSON.stringify({ apply, ...totals }));
|
||||
}
|
||||
|
||||
main().catch(error => { console.error(error); process.exit(1); });
|
||||
+24
-2
@@ -18,6 +18,23 @@ if (peerGatewayQa) {
|
||||
if (process.platform === 'win32' && !process.env.OPENSSL_BIN) process.env.OPENSSL_BIN = 'C:\\Program Files\\Git\\usr\\bin\\openssl.exe';
|
||||
new PeeringState(peerData).generateIdentity('ComposeQA');
|
||||
}
|
||||
const discoveryQa = process.argv.includes('--discovery-ui');
|
||||
// Review focus: an untrusted AniList title in the QA admin's first feed must render as text, never markup.
|
||||
const XSS_TITLE = '<img src=x onerror=alert(1)>';
|
||||
if (discoveryQa) {
|
||||
const fetchedAt = '2026-10-20T10:00:00.000Z';
|
||||
const evidence = [{ id: 'recommendation:990001', kind: 'recommendation', label: 'AniList recommendation',
|
||||
note: 'Fixture recommendation', url: 'https://anilist.co/manga/990001', fetchedAt }];
|
||||
const reasons = ['AniList readers who liked Moon Garden recommend this (42).', 'New: started 2025, ongoing and trending.'];
|
||||
fs.writeFileSync(path.join(data, 'discovery.json'), JSON.stringify({ schema: 1, revision: 1,
|
||||
candidates: { 'anilist:990001': { id: 'anilist:990001', title: XSS_TITLE, year: 2025, status: 'RELEASING', cover: null,
|
||||
seed: 'recs', fetchedAt, evidence, metadataUrl: 'https://anilist.co/manga/990001' } },
|
||||
feeds: { qa: { generatedAt: fetchedAt, items: [{ candidateId: 'anilist:990001', reason: reasons[0], reasons, evidence,
|
||||
evidenceIds: evidence.map(entry => entry.id), scoreParts: { communityRec: 3, fresh: 2 }, score: 5, period: 'Related works', group: 'recs' }] } },
|
||||
lastRun: { status: 'success', startedAt: fetchedAt, finishedAt: fetchedAt, candidateCount: 1, examined: 1, clipped: 0 },
|
||||
// Nightly off, so only the test's own clicks start harvests.
|
||||
harvest: null, settings: { harvestEnabled: false, harvestAt: '03:30' }, nightly: { lastDate: null } }));
|
||||
}
|
||||
const pausedConfirmQa = process.argv.includes('--paused-confirm');
|
||||
const readingGateQa = process.argv.includes('--reading-limit-ui');
|
||||
if (readingGateQa) fs.writeFileSync(path.join(data, 'reading-quiz.json'), JSON.stringify({ version: 1, manga: {
|
||||
@@ -45,14 +62,19 @@ for (const [index, name] of ['Moon Garden', 'Before the Rain', 'Iron Harbor'].en
|
||||
fs.writeFileSync(path.join(library, name, 'manga.json'), JSON.stringify({ title: name,
|
||||
direction: directionQa && index === 1 ? 'ltr' : process.argv.includes('--reading-ui') && index === 0 ? 'ttb' : 'rtl',
|
||||
...(directionQa && index === 0 ? { 'x-reading-direction': { version: 1, value: 'rtl', source: 'fallback' } } : {}),
|
||||
// AniList IDs make these owned titles harvest anchors; the unreachable endpoint turns their requests into upstream errors.
|
||||
...(discoveryQa ? { 'x-anilist': { id: 990101 + index } } : {}),
|
||||
status: index ? 'completed' : 'ongoing', tags: ['Fantasy', 'Demo'] }));
|
||||
}
|
||||
const env = { ...process.env, LIBRARY_PATH: library, DATA_PATH: data, SCRAPER_DATA_PATH: path.join(data, 'scraper'),
|
||||
SCRAPER_TOKEN_PATH: tokenFile, BACKUP_PATH: backupDir, READER_PORT: port, READER_BIND: '127.0.0.1', COOKIE_SECURE: '0', APP_IMAGE: process.env.APP_IMAGE || 'mangacircus:local',
|
||||
READER_IMAGE: process.env.APP_IMAGE || 'mangacircus:local',
|
||||
GREATEST_COVER_WARM: '0', // QA stacks never download catalog covers
|
||||
...(readingGateQa ? { READING_WINDOWS_JSON: '{"0":[],"1":[],"2":[],"3":[],"4":[],"5":[],"6":[]}',
|
||||
READING_TIMEZONE: 'Europe/Berlin', READING_QUIZ_UNLOCK_MINUTES: '60', READING_QUIZ_CATALOG_PATH: '/data/reading-quiz.json' }
|
||||
: { READING_WINDOWS_JSON: '', READING_QUIZ_CATALOG_PATH: '' }),
|
||||
// A closed local port: every harvest request fails fast and AniList is never contacted.
|
||||
...(discoveryQa ? { DISCOVERY_ANILIST_URL: 'http://127.0.0.1:9/' } : {}),
|
||||
...(peerGatewayQa ? { PEER_DATA_PATH: path.join(data, 'peering'), PEER_USAGE_PATH: path.join(data, 'peering-usage'), PEER_PORT: process.env.QA_PEER_PORT || '8193' } : {}) };
|
||||
const args = ['compose', '-p', project, '-f', path.join(root, 'docker-compose.yml'), ...(peerGatewayQa ? ['--profile', 'peering'] : [])];
|
||||
const docker = (...command) => new Promise((resolve, reject) => {
|
||||
@@ -215,10 +237,10 @@ let keep = false;
|
||||
const image = await api('/lib/' + encodeURIComponent('Moon Garden') + '/cover.png'); assert.equal(image.status, 200);
|
||||
state = await (await api('/api/admin/scraper')).json(); assert.equal(state.paused, true);
|
||||
if (process.argv.includes('--settings-ui')) await require('./test-settings-ui')(url, cookie);
|
||||
if (process.argv.includes('--discovery-ui')) {
|
||||
if (discoveryQa) {
|
||||
const login = await api('/api/login', { name: 'qa', password: 'disposable-local-test' });
|
||||
assert.equal(login.status, 200);
|
||||
await require('./test-discovery-ui')(url, login.headers.get('set-cookie').split(';')[0]);
|
||||
await require('./test-discovery-ui')(url, login.headers.get('set-cookie').split(';')[0], { xssTitle: XSS_TITLE });
|
||||
}
|
||||
if (process.argv.includes('--peer-ui')) {
|
||||
const login = await api('/api/login', { name: 'qa', password: 'disposable-local-test' });
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
'use strict';
|
||||
const assert = require('node:assert/strict');
|
||||
|
||||
module.exports = async function testDiscoveryUI(url, cookie) {
|
||||
module.exports = async function testDiscoveryUI(url, cookie, { xssTitle } = {}) {
|
||||
const { chromium } = require('playwright');
|
||||
const browser = await chromium.launch({ channel: process.env.QA_BROWSER_CHANNEL || 'msedge', headless: true });
|
||||
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
|
||||
@@ -9,24 +9,72 @@ module.exports = async function testDiscoveryUI(url, cookie) {
|
||||
await context.addCookies([{ name, value: value.join('='), url }]);
|
||||
const page = await context.newPage(), errors = [];
|
||||
page.on('pageerror', error => errors.push(error.message));
|
||||
// An injected title must never run script; any dialog counts as a failure.
|
||||
page.on('dialog', dialog => { errors.push(`Unexpected dialog: ${dialog.message()}`); dialog.dismiss().catch(() => {}); });
|
||||
try {
|
||||
await page.goto(url + '/discover.html');
|
||||
await page.getByRole('heading', { name: 'Discover', exact: true }).waitFor();
|
||||
await page.getByText('Turn on personalized discovery').waitFor();
|
||||
await page.getByText('Off by default. When on, the server sends AniList the IDs of titles you liked, read or own, '
|
||||
+ 'their authors\' IDs and your top tags, without login, to find related work. Your feed and feedback stay private to this server.').waitFor();
|
||||
await page.getByLabel('Personalized discovery').check();
|
||||
// Review focus 5: the seeded feed is the first one shown, before any run replaces it.
|
||||
// The XSS check must never be skipped silently: the Compose driver always seeds the title.
|
||||
assert.ok(xssTitle, 'The discovery UI test needs the seeded XSS title (run it through tools/test-compose.js --discovery-ui)');
|
||||
const seeded = page.locator('.discovery-card').filter({ hasText: 'AniList readers who liked Moon Garden' });
|
||||
await seeded.waitFor();
|
||||
assert.equal(await seeded.locator('h2').textContent(), xssTitle);
|
||||
assert.equal(await page.locator('.discovery-card h2 img').count(), 0);
|
||||
await seeded.getByText('Why this is here · evidence').click();
|
||||
const details = await seeded.locator('details li').allTextContents();
|
||||
assert.ok(details.some(text => text.includes('New: started 2025')), `All reasons listed: ${details}`);
|
||||
assert.ok(details.some(text => /AniList recommendation.* · fetched \S+/.test(text)), `Evidence fetch date: ${details}`);
|
||||
await page.getByRole('button', { name: 'Run discovery now' }).click();
|
||||
// The seeded store already says "success"; wait until the run's own feed replaced the seeded card.
|
||||
await page.waitForFunction(title => /Status: success/.test(document.querySelector('.discovery-admin')?.textContent || '')
|
||||
&& ![...document.querySelectorAll('.discovery-card h2')].some(heading => heading.textContent === title), xssTitle, { timeout: 60000 });
|
||||
await page.getByText(/suggestions/, { exact: false }).first().waitFor();
|
||||
await page.waitForFunction(() => document.querySelectorAll('.discovery-card').length > 0);
|
||||
await page.locator('.discovery-card').first().getByText('Why this is here · evidence').click();
|
||||
assert.ok(await page.locator('.discovery-card').first().locator('details li').count() > 0);
|
||||
await page.locator('.discovery-card').first().getByRole('button', { name: '👍 Maybe' }).click();
|
||||
await page.getByText('Preference saved').waitFor();
|
||||
|
||||
// Per-user refresh: the unreachable AniList endpoint ends it with upstream errors, then the cooldown shows.
|
||||
const refresh = page.locator('.discovery-refresh');
|
||||
await page.getByRole('button', { name: 'Refresh my suggestions' }).click();
|
||||
await page.waitForFunction(() => /Next refresh possible at \d{1,2}:\d{2}/.test(document.querySelector('.discovery-refresh')?.textContent || ''),
|
||||
null, { timeout: 90000 });
|
||||
assert.equal(await refresh.isDisabled(), true);
|
||||
await page.locator('.discovery-admin').getByText('Last harvest: stopped').waitFor({ timeout: 30000 });
|
||||
await page.locator('.discovery-admin').getByText('upstream-errors').waitFor();
|
||||
|
||||
// Nightly settings save through the settings route and survive a reload.
|
||||
await page.getByLabel('Automatic harvest').check();
|
||||
await page.getByText('Harvest settings saved').waitFor();
|
||||
await page.getByLabel('Harvest time').fill('04:15');
|
||||
await page.getByText(/Harvest settings saved \(on, 04:15\)/).waitFor();
|
||||
await page.reload();
|
||||
await page.getByRole('heading', { name: 'Discovery job' }).waitFor();
|
||||
assert.equal(await page.getByLabel('Harvest time').inputValue(), '04:15');
|
||||
assert.equal(await page.getByLabel('Automatic harvest').isChecked(), true);
|
||||
|
||||
// Harvest now is single-flight: a second click while one runs shows the server's 409 message.
|
||||
await page.getByRole('button', { name: 'Harvest now' }).click();
|
||||
await page.getByText('Harvest started').waitFor();
|
||||
await page.getByRole('button', { name: 'Harvest now' }).click();
|
||||
await page.getByText('A harvest is already queued or running').waitFor();
|
||||
assert.equal(errors.length, 0, `Browser errors: ${errors.join('; ')}`);
|
||||
await page.setViewportSize({ width: 390, height: 844 });
|
||||
|
||||
await page.setViewportSize({ width: 375, height: 812 });
|
||||
await page.getByRole('button', { name: /Refresh my suggestions|Next refresh possible|Queued|Fetching|Paused by AniList/ }).waitFor();
|
||||
assert.ok(await refresh.isVisible());
|
||||
const box = await refresh.boundingBox();
|
||||
assert.ok(box && box.x >= 0 && box.x + box.width <= 375, `Refresh button off screen: ${JSON.stringify(box)}`);
|
||||
const overflow = await page.evaluate(() => ({ width: innerWidth, scroll: document.documentElement.scrollWidth,
|
||||
elements: [...document.querySelectorAll('body *')].filter(el => el.getBoundingClientRect().right > innerWidth + 1)
|
||||
.slice(0, 8).map(el => `${el.tagName}.${el.className}: ${Math.round(el.getBoundingClientRect().right)}`) }));
|
||||
assert.ok(overflow.scroll <= overflow.width, `Discover page overflows on mobile: ${JSON.stringify(overflow)}`);
|
||||
assert.ok(overflow.scroll <= 375, `Discover page overflows on mobile: ${JSON.stringify(overflow)}`);
|
||||
const create = await page.evaluate(() => fetch('/api/admin/user', { method: 'POST', headers: { 'content-type': 'application/json' },
|
||||
body: JSON.stringify({ name: 'discovery-reader-qa', password: 'disposable-reader-test', admin: false }) }).then(response => response.status));
|
||||
assert.equal(create, 200);
|
||||
@@ -36,7 +84,10 @@ module.exports = async function testDiscoveryUI(url, cookie) {
|
||||
await page.reload();
|
||||
await page.getByRole('heading', { name: 'Discover', exact: true }).waitFor();
|
||||
assert.equal(await page.getByRole('button', { name: 'Run discovery now' }).count(), 0);
|
||||
assert.equal(await page.getByRole('button', { name: 'Harvest now' }).count(), 0);
|
||||
assert.equal(await page.evaluate(() => fetch('/api/admin/discovery').then(response => response.status)), 403);
|
||||
assert.equal(await page.evaluate(() => fetch('/api/admin/discovery/harvest', { method: 'POST', headers: { 'content-type': 'application/json' }, body: '{}' })
|
||||
.then(response => response.status)), 403);
|
||||
assert.equal(errors.length, 0, `Browser errors: ${errors.join('; ')}`);
|
||||
} finally { await browser.close(); }
|
||||
};
|
||||
@@ -40,7 +40,7 @@ async function freePort() {
|
||||
const scraperPort = fakeScraper.address().port;
|
||||
const child = spawn(process.execPath, ['server.js'], { cwd: root, windowsHide: true,
|
||||
env: { ...process.env, PORT: String(port), LIBRARY: library, DATA: data, SCAN_INTERVAL: '0',
|
||||
FULL_SCAN_INTERVAL: '0', SCRAPER_URL: `http://127.0.0.1:${scraperPort}`, COOKIE_SECURE: '0' }, stdio: 'ignore' });
|
||||
FULL_SCAN_INTERVAL: '0', SCRAPER_URL: `http://127.0.0.1:${scraperPort}`, COOKIE_SECURE: '0', GREATEST_COVER_WARM: '0' }, stdio: 'ignore' });
|
||||
let browser;
|
||||
try {
|
||||
for (let i = 0; i < 80; i++) {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Discovery Scout — personalized manga discovery
|
||||
|
||||
Status: **deterministic first slice implemented; later phases remain a vision**. Updated 2026-10-09. See [`docs/discovery.md`](../docs/discovery.md) for the current contract and deliberate omissions.
|
||||
Status: **deterministic first slice and the wider AniList candidate pool implemented; later phases remain a vision**. Updated 2026-10-09. See [`docs/discovery.md`](../docs/discovery.md) for the current contract and deliberate omissions.
|
||||
|
||||
## 1. Outcome
|
||||
|
||||
@@ -41,6 +41,8 @@ Use a two-stage pipeline. **Code** gathers, identifies, filters, and computes re
|
||||
|
||||
### Candidate input, in priority order
|
||||
|
||||
**Implemented (2026-10-25):** items 1 and 2 — the reader harvests AniList recommendations, same-creator works, top titles in the user's tags and fresh titles into a local cache, with budgets, a nightly run and per-user refresh. See the [wider-pool spec](../docs/superpowers/specs/2026-10-25-discovery-wider-pool-design.md) and [`docs/discovery.md`](../docs/discovery.md). Item 3 (admin seed form) is still open.
|
||||
|
||||
1. Reuse public AniList metadata/relations already cached by MangaCircus, plus the read-only Greatest catalog, for cold start and sequel/creator leads. Existing *Greatest* votes are stronger preference signals than curator rank.
|
||||
2. Add one or two explicit, public metadata feeds only after checking their API terms and stable identity fields. Cache responses with timestamps. Publisher/award lists may seed editorial candidates, but only with source links and dates.
|
||||
3. Let an admin seed a candidate by metadata URL or exact provider ID. This is the fallback when public APIs are incomplete. It is **not** a scraper URL and creates no queue item.
|
||||
@@ -167,7 +169,7 @@ Keep the first code change small and reviewable:
|
||||
| `public/discover.html`, `discover.js`, `discover.css` (new) | Feed, evidence, feedback, empty/error states, admin run control |
|
||||
| `public/common.js` | One navigation link; no global discovery polling |
|
||||
| `test/discovery*.test.js`, disposable browser test | Identity, consent, isolation, budgets, action permissions and UI behavior |
|
||||
| `docs/architecture.md`, `docs/compose-deployment.md` | Record ownership, backup, local release and rollback once implemented |
|
||||
| `docs/architecture.md`, `docs/deployment.md` | Record ownership, backup, local release and rollback once implemented |
|
||||
|
||||
The exact file split is a proposal, not a requirement to create empty modules. Start with storage and identity tests, then the deterministic pipeline, then API/UI. Do not add model-provider plumbing, a new Compose service, or a nightly timer in the first implementation slice. A candidate must pass schema validation and duplicate/identity checks before it can appear in a published feed.
|
||||
|
||||
|
||||
@@ -74,7 +74,7 @@ Catalog snapshots use cursor/revision pagination and bounded page sizes. A revis
|
||||
|
||||
Input from a peer is untrusted even after pairing: cap manifests, chapter/page sizes and total transfer size; validate schemas and image MIME/content; reject traversal paths, archives/executables, and malformed names; stream rather than buffering a whole manga; verify each page hash before publication. Hashes detect corruption but do not prove editorial correctness. Never overwrite an existing chapter silently.
|
||||
|
||||
For transport efficiency, start with original image bytes and page-level resume. A [local compression test](server-peering-compression-test.md) found an average ~6% reduction with zstd-3 but a wide 0.7–10.7% spread by chapter, so codec negotiation is an optional later optimization, not a prerequisite for peering. Measure actual chapter-ready time over a real peer link before adding chapter packing or automatic compression.
|
||||
For transport efficiency, start with original image bytes and page-level resume. A local compression test (`tools/benchmark-peer-transfer.js`) found an average ~6% reduction with zstd-3 but a wide 0.7–10.7% spread by chapter, so codec negotiation is an optional later optimization, not a prerequisite for peering. Measure actual chapter-ready time over a real peer link before adding chapter packing or automatic compression.
|
||||
|
||||
## 5. Reader and queue experience
|
||||
|
||||
|
||||
Reference in new issue
Block a user