Compare commits

...
Author SHA1 Message Date
MaashkinoozheandClaude Opus 5.5 c760141636 Continue row: a run of more than N chapters in order also marks a reread
A done chapter after the head now counts as left again when it was finished
more than rereadDays before the head (default 7) or before a run of more
than rereadChapters chapters read in order up to the head (default 5).
Both are user settings under Library page; 0 turns a rule off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 11:47:45 +01:00
MaashkinoozheandClaude Opus 5.5 9b529d7709 Continue row: only manga with something left after the last chapter read
GET /api/library returns `left` per manga: chapters after the newest saved
position that are unfinished, unread, or were last read more than 7 days
before it (a reread from the start). The continue row lists manga with
left > 0, so caught-up series drop out until a new chapter arrives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 11:27:02 +01:00
Maashkinoozhe dddf9bb4ac Merge remote-tracking branch 'origin/main' into feature/stale-reading-session 2026-10-31 12:21:24 +01:00
MaashkinoozheandClaude Opus 5.5 618992ec14 Reader: stale reading sessions never move "Continue" back
A reader tab left open on one device (or restored by the browser) used to
save its old chapter as the newest position as soon as it was shown again.
Progress saves now carry the tab id and the newest progress time the tab
knows (`seen`); the server refuses a save when another tab or device saved
a newer position since. The reader then asks: jump to the newest position,
or keep reading here (which makes this chapter the head again). It checks on
open, when the tab becomes visible, and on every save. Reader links carry
`at` and the reader keeps it in the address bar, so reloads and restored
tabs still know what they had seen.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 11:49:52 +01:00
MaashkinoozheandClaude Opus 5.5 e3e57b60bd Reader settings per login session
Each sign-in keeps its own reader mode, zoom, page tone and skipping once
it changes one; sessions that never changed follow the latest set, and a
new sign-in starts from the session used last. Settings -> Reading shows
every session's state, lets a session follow the latest again or become
the latest, and turns the feature off for one shared set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 11:45:44 +01:00
MaashkinoozheandClaude Opus 5.5 03767d5704 Scraper: stop walking the whole library once a minute for status polls
The reader polled /api/state every 15 s; each call rebuilt the scraper's library
summary when it was over 60 s old, reading every chapter.json (on the NAS about
5,300 opens / 16,000 NFS calls and 1.3 s CPU per walk, once a minute, plus 115 KB
per poll). The reader now polls GET /api/revision (falls back to /api/state on an
older scraper), and the summary is rebuilt after a library change or after 15
minutes. Release-date writes invalidate it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 23:10:38 +01:00
MaashkinoozheandClaude Opus 5.5 b328ad98fa Moments test waits for the server to exit before removing its folder
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:52:29 +01:00
MaashkinoozheandClaude Opus 5.5 d7d3bf5b2b Read-ahead per reader; compose passes READ_AHEAD and PERF_TRACE; report fixes
Each user gets an own read-ahead queue, served in turn, so two people reading
at once no longer cancel each other's chapter. The report rounds phase times
and measures the quiet time before a slow storage probe correctly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:51:19 +01:00
MaashkinoozheandClaude Opus 5.5 896726a51b Performance trace, load test and chapter read-ahead
A switchable trace (DATA/perf/trace.on or PERF_TRACE=1) records every request's
phases (stat, open, first read, disk vs network wait), browser resource timings,
page-turn waits and flow stalls, a cold storage probe, event-loop lag, scans and
scraper calls as JSONL. tools/perf-report.js says which phase slow loads spend
their time in; tools/perf-bench.js simulates readers.

READ_AHEAD (default on) reads an opened chapter's pages, then the next chapter's,
into the OS cache so a NAS's cold reads stop reaching the browser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 22:28:50 +01:00
33 changed files with 1659 additions and 74 deletions

No files matched your search

+3
View File
@@ -11,6 +11,7 @@
!reading-analytics.js
!reading-access.js
!moments.js
!reader-sessions.js
!public/
!public/**
!greatest/
@@ -23,6 +24,8 @@
!scraper/**
!peering/
!peering/**
!perf/
!perf/**
!tools/
!tools/**
scraper/.cache/**
+4
View File
@@ -20,6 +20,10 @@ READING_TIMEZONE=UTC
READING_QUIZ_UNLOCK_MINUTES=60
# Optional deployment-managed JSON file under DATA_PATH, e.g. /data/reading-quiz.json.
# READING_QUIZ_CATALOG_PATH=/data/reading-quiz.json
# Read an opened chapter into the OS cache ahead of the browser (1 = on, the default); see docs/performance.md.
# READ_AHEAD=1
# Performance trace from the start (also switchable at runtime: touch DATA_PATH/perf/trace.on).
# PERF_TRACE=0
PROXY_NETWORK=mangacircus-edge
# Extra Compose files, ':'-separated (docs/deployment.md), e.g. HTTPS with the bundled Caddy:
# COMPOSE_FILE=docker-compose.yml:deploy/compose.https.yml
+1
View File
@@ -12,6 +12,7 @@ Guide for contributors and coding agents. Users who only want to run it: [docs/d
| `scraper/` | Download queue daemon (`service.js`, `daemon.js`), source adapters and parsers, the publisher that writes the library, metadata lookups |
| `peering/` | Optional server-to-server sharing (gateway, client, adapter) |
| `runtime/` | In-container supervisor and configuration |
| `perf/` | Switchable performance trace and chapter read-ahead ([docs/performance.md](docs/performance.md)) |
| `discovery/`, `greatest/` | Discovery Scout and the Greatest of All Time dashboard |
| `tools/` | Demo library, health checks, maintenance tools (dry run by default), UI and Compose test drivers |
| `deploy/` | `circus.sh` operations helper, HTTPS with Caddy |
+2 -1
View File
@@ -3,7 +3,7 @@ RUN apk add --no-cache tzdata openssl
WORKDIR /app
ARG VCS_REF=development
LABEL org.opencontainers.image.title="MangaCircus" org.opencontainers.image.revision=$VCS_REF
COPY package.json server.js library.js reading-direction.js chapter-identity.js chapter-dates.js reading-analytics.js reading-access.js catalog.js backup.js moments.js ./
COPY package.json server.js library.js reading-direction.js chapter-identity.js chapter-dates.js reading-analytics.js reading-access.js catalog.js backup.js moments.js reader-sessions.js ./
COPY public ./public
COPY greatest ./greatest
COPY discovery ./discovery
@@ -11,6 +11,7 @@ COPY tools ./tools
COPY scraper ./scraper
COPY peering ./peering
COPY runtime ./runtime
COPY perf ./perf
RUN mkdir -p /library /data && chown node:node /data /library
ENV PORT=8080 LIBRARY=/library DATA=/data TZ=UTC
EXPOSE 8080
+7 -1
View File
@@ -50,6 +50,10 @@ Settings has **User** and **Admin** menu groups. User sections cover Reading, Re
Notifications and Account. Admin sections cover Scraper, Server & library, Users & access, Collections &
metadata and Service health. Only admins see administration ([settings navigation](docs/settings-navigation.md)).
Reader settings (mode, zoom, page tone, skipping) are kept per sign-in: each device or browser keeps its own once you
change one there, and a new sign-in starts from the one you used last. **Settings → Reading → Reader settings per
session** shows every session's state and turns this off for one shared set ([details](docs/architecture.md#67-reader-settings-per-session)).
### Settings (environment variables)
These configure the reader process. In Docker most are set for you by `docker-compose.yml`; the variables
@@ -63,6 +67,8 @@ you put in `.env` are listed in [docs/deployment.md](docs/deployment.md#4-config
| `SIGNUP` | `closed` | `open` lets anyone create an account. Otherwise the admin adds users under Settings |
| `SCAN_INTERVAL` | `10` | Minutes between shallow library change checks (`0` disables this timer) |
| `FULL_SCAN_INTERVAL` | `1440` | Minutes between full library reconciliations (`0` disables this timer) |
| `READ_AHEAD` | `1` | Read an opened chapter's pages (then the next chapter's) into the OS cache ahead of the browser; `0` = off ([docs/performance.md](docs/performance.md)) |
| `PERF_TRACE` | `0` | `1` = performance trace from the start; it can also be switched at runtime with the file `DATA/perf/trace.on` ([docs/performance.md](docs/performance.md)) |
| `SCAN_CONCURRENCY` | `8` | Chapter folders and images probed at once by an uncached scan (1–64); higher helps on network storage |
| `COOKIE_SECURE` | – | `1` marks the login cookie `Secure` (use it when you serve over HTTPS) |
| `SCRAPER_URL` | `http://127.0.0.1:8091` | Private scraper daemon URL |
@@ -142,7 +148,7 @@ completeness.
- **Flow page gap:** Settings → Reading lets each user keep the normal separator between Flow pages or remove it for continuous webtoon artwork and text that crosses image edges.
- **Page tone:** the reader's **Tone** button cycles Original → Inverted → Sepia → Dim → Extra dim in Flow, Page and Book. It changes only the artwork, never the controls or site theme. The choice is saved per user; it can also be selected in Settings → User → Reading.
- **Active Reading Clock:** opt in under Settings → User → Reading time & privacy. No scrolling-speed limit; a visit counts at 60% viewed-page coverage and saves on exit. Pauses in the background or after 30 seconds without reader activity. Tap the clock for this visit's time, coverage, qualification and loading wait.
- **Continue where you left off:** the first line of the library content (below the search, filters and tags) is one row of the five manga you read most recently and have not finished (same tiles as the grid; they stay in the grid too). It hides while a search is typed, a status filter is set, or the library is sorted by *Recently read*; tags and collection grouping do not affect it. Turn it off under Settings → User → Reading → *Library page*.
- **Continue where you left off:** the first line of the library content (below the search, filters and tags) is one row of the five manga you read most recently that still have something left after the last chapter you read (same tiles as the grid; they stay in the grid too). Left means an unfinished last chapter, an unread later chapter, or — when you are rereading — a later chapter you finished in an earlier pass: more than *N days* before your current position (default 7), or before a run of more than *M chapters* you have since read in order up to it (default 5). Both are set under Settings → User → Reading → *Library page*; 0 turns a rule off. A manga you have caught up on drops out until a new chapter arrives. It hides while a search is typed, a status filter is set, or the library is sorted by *Recently read*; tags and collection grouping do not affect it. Turn it off under Settings → User → Reading → *Library page*.
- **Your Reading Story:** private active-time summaries appear on the library and manga pages when history exists. Chapter rows show the last timed read (60%-coverage visit), rereads remain visible as lifetime time, and Settings lets each user hide, export, or delete their own history. No leaderboards or speed ranking are used.
- **Tap / click zones:**
- Tapping **beside the pages** always shows or hides the controls.
+2
View File
@@ -61,6 +61,8 @@ services:
DISCOVERY_HARVEST_AT: ${DISCOVERY_HARVEST_AT:-03:30}
DISCOVERY_ANILIST_URL: ${DISCOVERY_ANILIST_URL:-}
GREATEST_COVER_WARM: ${GREATEST_COVER_WARM:-1}
READ_AHEAD: ${READ_AHEAD:-1}
PERF_TRACE: ${PERF_TRACE:-0}
networks: [backend, egress]
healthcheck:
test: [CMD, node, tools/healthcheck.js, reader]
+35 -8
View File
@@ -129,12 +129,17 @@ This keeps ids human-readable and derivable without a database. The price: renam
"readingClock": false, // opt in to credible per-chapter active-time tracking
"readingStory": true, // show private retrospective summaries when history exists
"continueRow": true, // library: "Continue where you left off" row at the top
"rereadDays": 7, // continue row: a later chapter finished this long before the head is left again (0 = off)
"rereadChapters": 5, // ... or one finished before a run of more than this many in-order chapters (0 = off)
"zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 }, // page zoom per mode
"webhook": ""
}
"webhook": "",
"readerPerSession": true // reader settings per login session (section 6.7)
},
"readerFrom": "3f9c…" // public id of the session the latest reader settings came from
}
},
"sessions": { "<sha256 of token>": { "u": "admin", "t": 1790796794000 } },
"sessions": { "<sha256 of token>": { "u": "admin", "t": 1790796794000, "seen": 1790796794000,
"agent": "<User-Agent>", "reader": { …locked reader settings, only after a change… } } },
"progress": { "admin": { "Moon Garden": { "1/2": {
"v": "c0002 Roots [MangaDex]", // version read
"p": 4, "n": 14, // page index in that version (all pages), page count
@@ -177,7 +182,7 @@ This keeps ids human-readable and derivable without a database. The price: renam
| Path | Handling |
|------|----------|
| `/api/*` | `routes["<METHOD> <path>"]`. Handlers get `{ u, b, q, req, res }` and return an object, which is sent as JSON. Throwing `HttpError(code, message, extra)` sends `{ error, ...extra }` with that status (for example `nextAllowedAt` on a discovery `429`) |
| `/api/*` | `routes["<METHOD> <path>"]`. Handlers get `{ u, b, q, req, res, sess, sk }` (`sess` = the login session record, `sk` its key) and return an object, which is sent as JSON. Throwing `HttpError(code, message, extra)` sends `{ error, ...extra }` with that status (for example `nextAllowedAt` on a discovery `429`) |
| `/lib/*` | Image files from `LIBRARY`. Requires a session, allows image extensions only, and the path must resolve inside `LIBRARY`. `ETag` + `private, max-age=86400` |
| everything else | Static files from `public/` (`/` → `index.html`). `ETag` + `no-cache` |
@@ -194,12 +199,15 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
| `POST /api/login` `{name, password}` | public | Starts a session (cookie `sid`) |
| `POST /api/logout` | user | Ends the session |
| `GET /api/me` | user | Name, admin flag, settings, unread count |
| `POST /api/settings` `{…partial}` | user | Validated update of the settings keys above |
| `POST /api/settings` `{…partial}` | user | Validated update of the settings keys above; reader keys go to the current session (6.7) |
| `GET /api/reader-sessions` | user | `{ enabled, current, latestFrom, latest, sessions: [{ id, device, created, seen, current, locked, reader }] }` |
| `POST /api/reader-sessions` `{id, action}` | user | `follow`: the session drops its own reader settings and follows the latest; `latest`: its settings become the latest |
| `POST /api/password` `{old, new}` | user | Change password |
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read, and optional personal Reading Story aggregates |
| `GET /api/library` | user | Manga list with covers, counts, the user's read count, unread notifications, last read, `left` (chapters after the newest saved position that are unfinished, unread, or finished in an earlier pass — over `rereadDays` before it, or before a run of more than `rereadChapters` chapters read in order up to it; 0 = caught up), and optional personal Reading Story aggregates |
| `GET /api/manga?m=` | user | Manga details, books, chapters with ranked versions, progress, optional chapter-time summaries, and `continue` target |
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, optional Active Reading Clock summary, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
| `POST /api/progress` `{m, c, v, p, n, done, shift}` | user | Save the reading position |
| `GET /api/chapter?m=&c=[&v=]` | user | Pages (`u, w, h, t, r`), direction, versions, prev/next, progress, `latest` (as in `/api/progress/latest`), optional Active Reading Clock summary, plus all books and chapters of the manga (with the user's read state) for the chapter picker. Also marks that chapter's notifications read |
| `POST /api/progress` `{m, c, v, p, n, done, shift[, tab, seen]}` | user | Save the reading position. With `seen`, a save is refused (`{ok:false, stale:true, latest}`) when another tab saved a newer position of this manga since; otherwise `{ok:true, t}` |
| `GET /api/progress/latest?m=` | user | The manga's newest saved position (`latest`: `t, w, c, label, v, p, n, done, continue`) or `null` |
| `POST /api/reading/pulse` `{m, c, v, session, seq, …deltas}` | user | When opted in, idempotently merge a clamped active-time pulse, verified page coverage and input totals under a single-reader lease |
| `GET /api/reading/export` | user | Export the user's private derived reading-time summary without raw input or page-exposure diagnostics |
| `POST /api/reading/delete` `{m}` or `{all:true,password}` | user | Delete one manga's reading history, or all history after password confirmation, without changing normal reading progress |
@@ -210,6 +218,7 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
| `POST /api/moments` `{m, c, v, pages, layout, region, cat, note}` | user | Save a moment: `pages` are 1–4 distinct indexes in displayed order, `layout` `row` (default) or `column`, `cat` one of `twist hot awesome cute funny feels` (required), `region` fractions of the pages put together, or null |
| `POST /api/moments/update` `{id, cat?, region?, note?}` · `POST /api/moments/delete` `{id}` | user | Edit or delete one of the caller's own moments (another user's id is `404`) |
| `GET /api/notifications` · `POST /api/notifications/read` `{id \| all}` | user | Notification list and read state |
| `POST /api/perf` `{page, events: […]}` | user | Browser timings for the performance trace; ignored with `{on: false}` while the trace is off. `/api/chapter` returns `perf: true` while it is on ([performance.md](performance.md)) |
| `GET /api/admin` | admin | Users, last scan result, library path |
| `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result |
| `POST /api/admin/user` `{name, password, admin}` | admin | Create a user or reset a password |
@@ -268,6 +277,22 @@ The reader alone talks to AniList for discovery ([discovery.md](discovery.md)).
(all reasons, strongest first) next to `reason`; seed evidence carries `fetchedAt`.
- `DISCOVERY_ANILIST_URL` overrides the AniList endpoint (tests point it at a local fixture server).
### 6.7 Reader settings per session
The reader keys (`startMode`, `lastMode`, `adapt`, `skipExtras`, `skipCovers`, `flowPageGap`, `pageTone`, `zoom`)
can differ per login session, so a phone and a desktop keep their own mode and zoom (`reader-sessions.js`):
- `user.settings` holds the **latest** reader set. A session without its own set follows it.
- The first reader change in a session copies the latest set into `session.reader` with the change applied: the session
is now **locked** and keeps that set whatever other sessions do. The change also becomes the latest set.
- Loading a page (`GET /api/me`) from a locked session makes its set the latest again, so a new sign-in starts from the
session used last. A new sign-in has no set of its own until it changes something.
- Other settings (Reading Story, dates, discovery, webhook, …) stay per user.
- `readerPerSession: false` (Settings → Reading) brings back one shared set: changes go to `user.settings` and apply to
every session. Locked sets are kept and apply again when the switch is turned back on.
Settings → Reading → *Reader settings per session* lists the user's sessions (device from the User-Agent, sign-in and last
activity, own or following) with "Follow latest again" and "Use for new sessions".
## 7. Frontend
### 7.1 Pages
@@ -351,6 +376,7 @@ mode = adapt && intended ∉ compatible ? compatible[0] : intended
- Saved 700 ms after the position settles, and on `pagehide` (a `fetch` with `keepalive`).
- Stores the page index in the full page list, so skipping extras doesn't corrupt it.
- On open, the saved page is used for the same version, and mapped proportionally for another version. A finished chapter reopens at the start.
- **Stale reading sessions.** A tab left open on one device (or restored by the browser) must not move "Continue" back after the user read on elsewhere. Each browser tab has an id (`sessionStorage`) that its saves store as `w`; reader links carry `at`, the newest progress time the linking page knew (`continue.at`), and the reader keeps `at` current in its address bar so a reload or restored tab still knows it. A link without `at` (bookmark, notification) counts as a deliberate open. When the manga's newest position is later than `at` and came from another tab, the reader saves nothing and asks: jump to the newest position, or keep reading here, which makes this chapter the head again. It checks on open, when the tab becomes visible again, and with every save (the server refuses the stale save). The comparison is by save time, not chapter order; the question names no device.
**Active Reading Clock (opt-in):**
- Runs only while the reader is visible and focused, a sufficiently exposed page is decoded, a server lease is held, and accepted reader input occurred in the last 50 seconds.
@@ -400,6 +426,7 @@ docker compose up -d --build
| User data | `db.json` is rewritten as a whole, at most about twice per second while people read. Comfortable up to a few MB (a household or small group, thousands of chapters read) |
| Concurrency | One process, one writer. No clustering or horizontal scaling by design |
| Images | Served as stored. No resizing or thumbnails, so large covers make the library page heavy |
| Slow storage | Opening a chapter reads its pages (then the next chapter's) into the OS cache, two at a time (`READ_AHEAD`). Where the time goes: [performance.md](performance.md) |
## 10. Design decisions
+78
View File
@@ -0,0 +1,78 @@
# Performance trace, load test and read-ahead
When pages load slowly, the trace shows where the time goes: the storage (a NAS, a disk waking up), the
reader process (CPU, event loop), library scans running at the same time, or the network to the browser.
## Switching the trace on and off
No restart is needed. The reader checks for a flag file every 5 seconds:
```bash
touch "$DATA/perf/trace.on" # on (Docker: the data folder mounted at /data)
rm "$DATA/perf/trace.on" # off
```
`PERF_TRACE=1` in the environment keeps it on from the start. While it is off, the reader does no extra work.
The trace goes to `DATA/perf/trace-YYYY-MM-DD.jsonl` (UTC days), one JSON object per line. Files older than
14 days are deleted; one day stops growing at 256 MB. The file contains request paths (manga and chapter
names) and user names, so treat it like the rest of `DATA`.
## What is recorded
Every line has `t` (server time, ms since the epoch) and `k` (kind):
| `k` | Fields | Meaning |
|---|---|---|
| `req` | `m`, `path`, `status`, `ms`, `bytes`, `conn`, `aborted`; for files `stat`, `open`, `first`, `readEnd`, `disk`, `netWait`; for pages also `access` | One HTTP request. Phase fields are ms since the request arrived: `stat` file lookup done, `open` file opened, `first` first chunk read, `readEnd` last chunk read. `disk` = time spent waiting for the storage, `netWait` = time the file stream was paused because the client did not take data fast enough. `conn` > 1 means a reused keep-alive connection |
| `client` | `e`: `hello`, `res`, `view`, `stall` | Reported by the reader page while the trace is on (`public/perf.js`). `res`: the browser's resource timing of a page image (`queue`, `dns`, `connect`, `ttfb`, `download`, `total`, `size`, `server` = the reader's `stat` time, `cached`). `view`: a page turn in page/book mode, `wait` until the new page was decoded and painted, `preloaded`. `stall`: flow mode, a page on screen before its image, `wait` until it loaded. `hello`: browser, viewport, connection type |
| `nas` | `statMs`, `firstMs`, `ms`, `bytes` or `error` | Every `PERF_NAS_PROBE_SECONDS` (60) one random library page is read. Out of a large library it is almost never cached, so this is the storage's cold latency over time; seconds after a quiet period mean disks were in standby |
| `loop` | `lagP50`, `lagP99`, `lagMax`, `cpu`, `rss`, `heap` | Every 5 s: event-loop delay (ms), CPU % of one core, memory (MB) |
| `scan` | `mode`, `ms`, `changed` | Library scans (change checks every `SCAN_INTERVAL`, full reconciliations) |
| `ahead` | `tag`, `files`, `done`, `skipped`, `bytes`, `ms`, `replaced` | A read-ahead batch (below) |
| `scraper` | `endpoint`, `ms`, `status`/`error` | Calls from the reader to the scraper daemon |
| `trace` | `on` | The trace was switched on or off |
## Reading the result
```bash
node tools/perf-report.js "$DATA/perf" # all days
node tools/perf-report.js "$DATA/perf" --since 2026-10-29T18:00Z --slow 300
docker compose exec reader node tools/perf-report.js /data/perf # inside the container
```
The report starts with a verdict (which phase dominates the slow loads), then percentiles for the server's
phases, the browser's timings, page turns, flow stalls, the storage probe, the event loop, scans, read-ahead
and the API routes. `--json` prints the same numbers for scripts and agents.
How to read it:
- **Slow `firstRead` / `readRest`, slow `nas` probes**: the storage is slow. Probes of several seconds after
minutes without reads point at disk standby (on a Synology: *Control Panel → Hardware & Power → HDD
Hibernation*).
- **Slow `stat`**: metadata lookups on the storage are slow (often during a scan, see `scan`).
- **High `netWait`, or browser `download` much longer than the server's `ms`**: the network between server
and browser (Wi-Fi, VPN, a proxy).
- **Browser `queue`**: the browser waited for a free connection (too many parallel requests).
- **High `lagMax`**: the reader process was busy and could not answer.
## Load test
`tools/perf-bench.js` simulates readers paging through random chapters with a think time and a small
look-ahead, and reports how long a reader looked at an empty page (`wait`), time to first byte and total
per page. It only reads (no progress is written). Use a separate account:
```bash
BENCH_PASSWORD=… node tools/perf-bench.js --url http://server:8080 --user bench --password-env BENCH_PASSWORD \
--readers 4 --chapters 3 --pages 40 --think 400 --ahead 2
```
Run it with the trace on to see the server's side of the same requests.
## Read-ahead
`READ_AHEAD=1` (default): when a chapter is opened, the reader reads its pages once in page order, starting
at the saved page, two at a time, and then the next chapter's. The data is thrown away; the OS page cache
keeps it, so the browser's requests are answered from memory instead of the storage. Opening another
chapter replaces whatever is still queued. `READ_AHEAD=0` turns it off (useful on a local SSD, or to compare
with the trace).
+2 -1
View File
@@ -199,7 +199,8 @@ All responses are JSON. When `SCRAPER_TOKEN` is set, send
| Method | Path | Body / result |
|---|---|---|
| `GET` | `/health` | Basic liveness |
| `GET` | `/api/state` | Queue, all active source checkpoints, settings, errors, history, adapters |
| `GET` | `/api/state` | Queue, all active source checkpoints, settings, errors, history, adapters. The library summary in it is rebuilt in the background after a publish, or when it is 15 minutes old (the walk reads every `chapter.json`) |
| `GET` | `/api/revision` | `{ publishedRevision, status, paused }` only; the reader polls this every 15 s |
| `POST` | `/api/queue` | `{url,title?,metadataSearch?,scope?,existingPolicy?,libraryMangaId?,inspect?}` |
| `POST` | `/api/queue/remove` | `{id}`; active downloads cannot be removed |
| `POST` | `/api/queue/clear-completed` | `{source?}`; remove inactive successful download job rows only; preserve library, history, checkpoints and other statuses |
+78
View File
@@ -0,0 +1,78 @@
'use strict';
/*
* Read-ahead for network libraries: when a chapter is opened, its pages (then the next chapter's) are read
* once in page order, a few at a time, so the OS page cache holds them before the browser asks. On a NAS
* with spinning disks a cold page costs 10-100 ms (seconds while a disk wakes up); a cached one well under
* 5 ms. The data is discarded; only the cache effect matters.
*
* One queue per reader (`key`, the user): opening another chapter replaces that reader's queue only, and
* the queues take turns so two people reading at once both get their pages ahead.
*/
const fs = require('fs');
class ReadAhead {
constructor({ concurrency = 2, remember = 4000, onBatch = () => {} } = {}) {
this.concurrency = concurrency; this.remember = remember; this.onBatch = onBatch;
this.batches = new Map(); this.turn = []; this.active = 0; this.recent = new Map(); this.buffers = [];
}
// `files` in reading order; `tag` names the batch in the trace.
warm(files, tag, key = '') {
const old = this.batches.get(key);
if (old) this.finish(key, old, true);
const queue = files.filter(f => !this.recent.has(f));
const batch = { tag, queue, files: queue.length, skipped: files.length - queue.length, bytes: 0, done: 0, active: 0, t0: performance.now() };
this.batches.set(key, batch);
if (!this.turn.includes(key)) this.turn.unshift(key); // a fresh request goes next
if (!queue.length) this.finish(key, batch, false);
this.pump();
}
// Runs after the reader's current batch (e.g. the next chapter) unless a newer warm() replaces it first.
then(files, tag, key = '') {
const batch = this.batches.get(key);
if (!batch) this.warm(files, tag, key); else batch.next = () => this.warm(files, tag, key);
}
pump() {
while (this.active < this.concurrency) {
const key = this.turn.find(k => this.batches.get(k)?.queue.length);
if (key === undefined) return;
this.turn.push(...this.turn.splice(this.turn.indexOf(key), 1)); // round robin between readers
const batch = this.batches.get(key), file = batch.queue.shift();
this.active++; batch.active++;
this.readFile(file).then(bytes => { this.note(file); batch.bytes += bytes; batch.done++; }, () => {}).finally(() => {
this.active--; batch.active--;
if (this.batches.get(key) === batch && !batch.queue.length && !batch.active) this.finish(key, batch, false);
this.pump();
});
}
}
finish(key, b, replaced) {
if (this.batches.get(key) === b) this.batches.delete(key);
if (!this.batches.has(key)) this.turn = this.turn.filter(k => k !== key);
b.queue = [];
this.onBatch({ tag: b.tag, files: b.files, done: b.done, skipped: b.skipped, bytes: b.bytes,
ms: Math.round(performance.now() - b.t0), replaced });
if (!replaced && b.next) b.next();
}
note(file) {
this.recent.delete(file); this.recent.set(file, 1);
while (this.recent.size > this.remember) this.recent.delete(this.recent.keys().next().value);
}
async readFile(file) {
const buf = this.buffers.pop() || Buffer.allocUnsafe(256 * 1024);
const fh = await fs.promises.open(file, 'r');
let total = 0;
try {
for (;;) {
const { bytesRead } = await fh.read(buf, 0, buf.length, total);
if (!bytesRead) break;
total += bytesRead;
}
} finally { await fh.close(); this.buffers.push(buf); }
return total;
}
}
module.exports = { ReadAhead };
+173
View File
@@ -0,0 +1,173 @@
'use strict';
/*
* Switchable performance trace: one JSON object per line in DATA/perf/trace-YYYY-MM-DD.jsonl.
*
* On while PERF_TRACE=1 or while the file DATA/perf/trace.on exists (checked every few seconds, so it can be
* switched on a running server with `touch` / `rm`). Off, every call is a cheap no-op.
*
* Event kinds (field `k`): `req` one HTTP request with its phases, `client` timings the reader page reports,
* `loop` event-loop lag / CPU / memory every SAMPLE_MS, `nas` a periodic read of one uncached library page,
* `scan` library scans, `scraper` calls to the scraper daemon, `ahead` read-ahead batches, `trace` on/off.
* The format and the report tool: docs/performance.md, tools/perf-report.js.
*/
const fs = require('fs'), path = require('path'), { monitorEventLoopDelay } = require('perf_hooks');
const CHECK_MS = 5000, FLUSH_MS = 1000, SAMPLE_MS = 5000;
const MAX_DAY_BYTES = 256 * 1024 * 1024; // a runaway trace never fills the data disk
const KEEP_DAYS = 14;
class Tracer {
constructor({ dir, env = process.env, now = Date.now } = {}) {
this.dir = dir; this.now = now;
this.forced = env.PERF_TRACE === '1';
this.flag = path.join(dir, 'trace.on');
this.on = false; this.buf = []; this.day = ''; this.dayBytes = 0; this.dropped = 0;
this.timers = []; this.probe = null; this.probeMs = Math.max(10, +env.PERF_NAS_PROBE_SECONDS || 60) * 1000;
}
start() {
this.check();
this.timers.push(setInterval(() => this.check(), CHECK_MS).unref());
this.timers.push(setInterval(() => this.flush(), FLUSH_MS).unref());
return this;
}
stop() { for (const t of this.timers) clearInterval(t); this.timers = []; this.setOn(false); this.flush(); }
check() {
let on = this.forced;
if (!on) try { fs.accessSync(this.flag); on = true; } catch { /* off */ }
this.setOn(on);
}
setOn(on) {
if (on === this.on) return;
if (on) {
this.on = true;
this.log({ k: 'trace', on: true, pid: process.pid, node: process.version });
this.startSampler();
} else {
this.log({ k: 'trace', on: false });
this.on = false;
this.stopSampler();
}
}
log(event) {
if (!this.on) return;
this.buf.push(JSON.stringify({ t: this.now(), ...event }));
if (this.buf.length >= 500) this.flush();
}
flush() {
if (!this.buf.length) return;
const lines = this.buf.join('\n') + '\n'; this.buf = [];
const day = new Date(this.now()).toISOString().slice(0, 10);
try {
if (day !== this.day) { this.day = day; this.dayBytes = 0; fs.mkdirSync(this.dir, { recursive: true }); this.prune(); }
if (this.dayBytes + lines.length > MAX_DAY_BYTES) { this.dropped += lines.length; return; }
fs.appendFileSync(path.join(this.dir, `trace-${day}.jsonl`), lines);
this.dayBytes += lines.length;
} catch (error) { console.warn('[perf] trace write failed:', error.message); }
}
prune() {
const cutoff = new Date(this.now() - KEEP_DAYS * 864e5).toISOString().slice(0, 10);
for (const name of fs.readdirSync(this.dir)) {
const m = /^trace-(\d{4}-\d\d-\d\d)\.jsonl$/.exec(name);
if (m && m[1] < cutoff) fs.rmSync(path.join(this.dir, name), { force: true });
}
}
// Event-loop delay shows whether the reader itself (CPU, sync work) holds requests back.
startSampler() {
const h = monitorEventLoopDelay({ resolution: 10 }); h.enable();
let cpu = process.cpuUsage(), at = process.hrtime.bigint();
this.sampler = { h, timer: setInterval(() => {
const nowCpu = process.cpuUsage(), nowAt = process.hrtime.bigint(), wall = Number(nowAt - at) / 1000;
const mem = process.memoryUsage();
this.log({ k: 'loop', lagP50: ms(h.percentile(50)), lagP99: ms(h.percentile(99)), lagMax: ms(h.max),
cpu: round(((nowCpu.user - cpu.user) + (nowCpu.system - cpu.system)) / wall * 100), rss: mb(mem.rss), heap: mb(mem.heapUsed) });
h.reset(); cpu = nowCpu; at = nowAt;
}, SAMPLE_MS).unref() };
this.startProbe();
}
stopSampler() {
if (this.sampler) { clearInterval(this.sampler.timer); this.sampler.h.disable(); this.sampler = null; }
if (this.probe) { clearInterval(this.probe); this.probe = null; }
}
// NAS probe: read one random library page now and then. A page picked at random out of a large library is
// almost never in the OS cache, so this measures the storage itself (and shows disks waking from standby).
setProbeSource(pick) { this.pick = pick; if (this.on) this.startProbe(); }
startProbe() {
if (this.probe || !this.pick) return;
const run = () => this.probeOnce().catch(() => {});
this.probe = setInterval(run, this.probeMs).unref();
setTimeout(run, 1000).unref();
}
async probeOnce() {
const file = this.pick && this.pick();
if (!file) return;
const t0 = performance.now();
let st, bytes = 0, firstMs = null, error = null;
try {
st = await fs.promises.stat(file);
const statMs = performance.now() - t0;
const fh = await fs.promises.open(file, 'r');
try {
const chunk = Buffer.allocUnsafe(256 * 1024);
for (;;) {
const { bytesRead } = await fh.read(chunk, 0, chunk.length, bytes);
if (firstMs === null) firstMs = performance.now() - t0;
if (!bytesRead) break;
bytes += bytesRead;
}
} finally { await fh.close(); }
this.log({ k: 'nas', statMs: round(statMs), firstMs: round(firstMs), ms: round(performance.now() - t0), bytes });
} catch (e) { error = e.code || e.message; this.log({ k: 'nas', ms: round(performance.now() - t0), error }); }
}
}
// Wraps one HTTP request: `mark(name)` stores the time since the request arrived, `set` adds fields, and the
// event is written when the response is done (or the client went away). Off, returns a shared no-op.
const NOOP = { mark() {}, set() {}, stream() {}, elapsed: () => 0, on: false };
function requestTrace(tracer, req, res) {
if (!tracer.on) return NOOP;
const t0 = performance.now(), ev = { k: 'req', m: req.method, path: req.url.split('?')[0].slice(0, 300) };
const trace = {
on: true,
mark(name) { ev[name] = round(performance.now() - t0); },
elapsed: () => round(performance.now() - t0),
set(fields) { Object.assign(ev, fields); },
// Splits a file stream's time into waiting for the disk and waiting for the client (TCP backpressure):
// pipe() pauses the file stream while the socket buffer is full and resumes it on 'drain'.
stream(src) {
let pausedAt = 0, waitNet = 0;
src.on('open', () => trace.mark('open'));
src.once('data', () => trace.mark('first'));
src.on('data', chunk => { ev.bytes = (ev.bytes || 0) + chunk.length; });
src.on('pause', () => { pausedAt = performance.now(); });
src.on('resume', () => { if (pausedAt) { waitNet += performance.now() - pausedAt; pausedAt = 0; } });
src.on('end', () => {
trace.mark('readEnd');
ev.netWait = round(waitNet);
ev.disk = round(Math.max(0, ev.readEnd - (ev.open || 0) - waitNet));
});
},
};
let done = false;
const finish = aborted => {
if (done) return; done = true;
ev.ms = round(performance.now() - t0); ev.status = res.statusCode;
if (aborted) ev.aborted = true;
ev.conn = req.socket._perfRequests = (req.socket._perfRequests || 0) + 1; // >1: keep-alive reuse
tracer.log(ev);
};
res.on('finish', () => finish(false));
res.on('close', () => finish(!res.writableFinished));
return trace;
}
const round = v => (v == null || Number.isNaN(v) ? null : Math.round(v * 10) / 10);
const ms = ns => round(ns / 1e6);
const mb = b => Math.round(b / 1048576);
module.exports = { Tracer, requestTrace };
+10
View File
@@ -387,6 +387,8 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
.login input { width: 100%; }
.saved { color: #7fd3a0; font-size: 13px; }
.reading-data { margin-top: 16px; padding-top: 12px; border-top: 1px solid var(--line); }
.reader-sessions { list-style: none; margin: 12px 0 0; padding: 0; }
.reader-sessions li { display: grid; gap: 4px; padding: 10px 0; border-top: 1px solid var(--line); overflow-wrap: anywhere; }
.reading-data > summary { cursor: pointer; font-weight: 600; }
.reading-danger { margin-top: 14px; padding-top: 12px; border-top: 1px solid var(--line); }
.reading-danger p { margin: 3px 0 8px; }
@@ -429,6 +431,14 @@ body.reader { position: fixed; top: 0; left: 0; right: 0; bottom: 0; overflow: h
.reading-gate-message[data-tone="retry"] { border-color: #9b7233; background: #382b1a; color: #ffdaa0; }
.reading-gate-message[data-tone="error"] { border-color: #944c55; background: #3c2025; color: #ffc0c6; }
@media (max-width: 660px) { .reading-gate { margin: 28px 12px 100px; padding: 18px; } }
/* Stale reading session: a newer position was saved elsewhere since this page was opened. */
.stale-session { position: fixed; inset: 0; z-index: 60; display: flex; align-items: center; justify-content: center;
padding: 16px; background: rgba(0, 0, 0, .6); }
.stale-box { box-sizing: border-box; width: min(440px, 100%); padding: 20px; border-radius: 12px; background: var(--card);
border: 1px solid var(--line); box-shadow: 0 12px 40px rgba(0, 0, 0, .6); }
.stale-box h2 { margin: 0 0 8px; font-size: 18px; }
.stale-actions { display: flex; flex-wrap: wrap; gap: 10px; margin: 16px 0 10px; }
.stale-actions .btn { flex: 1 1 auto; justify-content: center; text-align: center; }
.pg { position: absolute; overflow: hidden; background: #19191c; }
.pg img { display: block; width: 100%; height: 100%; -webkit-user-drag: none; }
#canvas[data-page-tone="inverted"] .pg img { filter: invert(1); }
+1 -1
View File
@@ -8,7 +8,7 @@
h('div', { class: 'info' }, h('h1', {}, c.title), h('p', { class: 'muted' }, `${c.members.length} manga · ${c.read}/${c.chapters} chapters read · ${c.unread} new`),
c.description ? h('p', {}, c.description) : null,
c.reading.activeMs ? h('p', {}, `${readingDuration(c.reading.activeMs)} together`) : null,
h('div', { class: 'actions' }, h('a', { class: 'btn primary', href: readUrl(c.continue.m, c.continue.c, c.continue.v) }, c.continue.text),
h('div', { class: 'actions' }, h('a', { class: 'btn primary', href: readUrl(c.continue.m, c.continue.c, c.continue.v, c.continue.at) }, c.continue.text),
me.admin ? h('a', { class: 'btn', href: 'collections.html?' + qsa({ id: c.id }) }, 'Edit collection') : null))));
main.append(h('h2', {}, 'Reading order'), h('div', { class: 'collection-members' }, c.members.map(m =>
h('a', { class: 'collection-member', href: mangaUrl(m.id) }, m.cover ? h('img', { src: m.cover, alt: '', loading: 'lazy' }) : null,
+16 -1
View File
@@ -39,6 +39,20 @@ const store = {
get(k) { try { return localStorage.getItem('mc.' + k); } catch (e) { return null; } },
set(k, v) { try { localStorage.setItem('mc.' + k, v); } catch (e) { /* private mode */ } },
};
// One id per browser tab (it survives navigation and reloads in that tab): progress saves carry it, so
// a reader can tell its own saves from those of another tab or device.
const tabId = (() => {
let id = null;
return () => {
if (id) return id;
try { id = sessionStorage.getItem('mc.tab'); } catch (e) { /* storage blocked */ }
if (!/^[a-zA-Z0-9_-]{8,40}$/.test(id || '')) {
id = Array.from(crypto.getRandomValues(new Uint8Array(12)), b => b.toString(16).padStart(2, '0')).join('');
try { sessionStorage.setItem('mc.tab', id); } catch (e) { /* this page keeps it in memory */ }
}
return id;
};
})();
function ago(t) {
const s = (Date.now() - t) / 1000;
@@ -60,7 +74,8 @@ function readingDuration(ms, compact = false) {
}
const qsa = obj => { const p = new URLSearchParams(); for (const k in obj) if (obj[k] != null && obj[k] !== '') p.set(k, obj[k]); return p.toString(); };
const readUrl = (m, c, v) => 'read.html?' + qsa({ m, c, v });
// `at`: the manga's latest progress time the linking page knew, so the reader can tell it opened from a stale page.
const readUrl = (m, c, v, at) => 'read.html?' + qsa({ m, c, v, at: at || null });
const mangaUrl = m => 'manga.html?' + qsa({ m });
const libraryUrlForTags = tags => {
const p = new URLSearchParams();
+4 -3
View File
@@ -55,8 +55,9 @@
h('option', { value: 'grouped' }, 'Group collections'), h('option', { value: 'expanded' }, 'Expand collections'), h('option', { value: 'individual' }, 'Individual manga'));
grouping.value = ['grouped', 'expanded', 'individual'].includes(store.get('grouping')) ? store.get('grouping') : 'grouped';
grouping.onchange = render;
// Started but unfinished manga, most recently read first; the same manga still appear in the grid.
const continueItems = data.mangas.filter(m => m.lastRead && m.read < m.chapters)
// Started manga with chapters left after the last one read (first read or reread, server `left`),
// most recently read first; the same manga still appear in the grid.
const continueItems = data.mangas.filter(m => m.lastRead && m.left > 0)
.sort((a, b) => b.lastRead - a.lastRead).slice(0, 5);
const continueRow = h('section', { class: 'continue-row', 'aria-label': 'Continue where you left off', hidden: true },
h('div', { class: 'continue-banner' }, h('span', { 'aria-hidden': 'true' }, '▶'), 'Continue where you left off'),
@@ -72,7 +73,7 @@
data.reading.rereads ? h('span', { class: 'story-pill' }, `${data.reading.rereads} reread${data.reading.rereads === 1 ? '' : 's'}`) : null),
h('div', { class: 'reading-summary-links' },
most ? h('a', { href: mangaUrl(most.id) }, `Most time with ${most.title}`) : null,
latest && data.reading.lastChapterId ? h('a', { href: readUrl(latest.id, data.reading.lastChapterId) }, `Continue ${latest.title} →`) : null)));
latest && data.reading.lastChapterId ? h('a', { href: readUrl(latest.id, data.reading.lastChapterId, null, latest.lastRead) }, `Continue ${latest.title} →`) : null)));
}
main.append(
h('div', { class: 'library-controls' },
+6 -6
View File
@@ -48,7 +48,7 @@
h('div', { class: 'chips' }, tags.map(tag => h('a', { class: 'chip chip-link', href: libraryUrlForTags([tag]), title: `Show manga tagged ${tag}` }, tag))),
meta.description ? h('p', { class: 'desc' }, meta.description) : null,
h('div', { class: 'actions' },
M.continue ? h('a', { class: 'btn primary', href: readUrl(M.id, M.continue.c, M.continue.v) }, M.continue.text) : null,
M.continue ? h('a', { class: 'btn primary', href: readUrl(M.id, M.continue.c, M.continue.v, M.continue.at) }, M.continue.text) : null,
follow))));
let storyRhythm = null;
@@ -73,7 +73,7 @@
storyStat(M.reading.typicalChapterMs ? readingDuration(M.reading.typicalChapterMs) : '—', 'typical chapter'),
storyStat(M.reading.rereads, `reread${M.reading.rereads === 1 ? '' : 's'}`)),
lastChapter ? h('p', { class: 'story-last muted' }, 'Last timed ',
h('a', { href: readUrl(M.id, lastChapter.id) }, `${lastChapter.label}${lastChapter.title ? ' · ' + lastChapter.title : ''}`),
h('a', { href: readUrl(M.id, lastChapter.id, null, M.continue?.at) }, `${lastChapter.label}${lastChapter.title ? ' · ' + lastChapter.title : ''}`),
` · ${ago(M.reading.updatedAt)}`) : null,
h('div', { class: 'story-rhythm-head' }, h('b', {}, 'Reading rhythm'), h('span', { class: 'muted small' }, 'Completed chapters · active time')),
storyRhythm,
@@ -110,7 +110,7 @@
const passes = chapter.reading.completedPasses;
const title = `${label} · ${readingDuration(value)}${passes > 1 ? ` · ${passes} timed reads · ${readingDuration(chapter.reading.totalActiveMs)} total` : ''}`;
storyRhythm.append(h('a', {
class: 'rhythm-bar', role: 'listitem', href: readUrl(M.id, chapter.id), title,
class: 'rhythm-bar', role: 'listitem', href: readUrl(M.id, chapter.id, null, M.continue?.at), title,
'aria-label': title, style: `height:${height}px`,
}));
}
@@ -221,7 +221,7 @@
if (done) { await api('api/unread', { m: M.id, c: ch.id }); ch.progress = null; ch.readAt = null; }
else {
const version = ch.versions[0];
await api('api/progress', { m: M.id, c: ch.id, v: version.id, p: version.pages - 1, n: version.pages, done: true });
await api('api/progress', { m: M.id, c: ch.id, v: version.id, p: version.pages - 1, n: version.pages, done: true, tab: tabId() });
ch.progress = { v: version.id, p: version.pages - 1, n: version.pages, done: true };
ch.readAt = { at: Date.now(), approx: false };
}
@@ -234,7 +234,7 @@
const timing = chapterTiming(ch), released = chapterRelease(ch.dates);
wrap.textContent = '';
wrap.append(h('div', { class: 'ch' + (done ? ' read' : '') },
h('a', { class: 'ch-main' + (timing ? ' has-time' : '') + (M.chapterDates?.release ? ' has-date' : ''), href: readUrl(M.id, ch.id) },
h('a', { class: 'ch-main' + (timing ? ' has-time' : '') + (M.chapterDates?.release ? ' has-date' : ''), href: readUrl(M.id, ch.id, null, M.continue?.at) },
h('span', { class: 'ch-label' }, ch.label),
h('span', { class: 'ch-pages' }, `${pages} p.`),
M.chapterDates?.release ? h('span', { class: 'ch-date' + (released ? ' tier-' + released.tier : ''), title: released ? released.title : 'No date known', 'aria-label': released ? released.title : 'No date known' },
@@ -259,7 +259,7 @@
h('button', { class: version.my > 0 ? 'on' : '', title: 'Better', onclick: () => vote(1) }, '▲'),
h('b', {}, version.score),
h('button', { class: version.my < 0 ? 'on' : '', title: 'Worse', onclick: () => vote(-1) }, '▼')),
h('a', { class: 'btn small', href: readUrl(M.id, ch.id, version.id) }, 'Read')));
h('a', { class: 'btn small', href: readUrl(M.id, ch.id, version.id, M.continue?.at) }, 'Read')));
});
}
draw();
+72
View File
@@ -0,0 +1,72 @@
'use strict';
/*
* Reader-side half of the performance trace (docs/performance.md). Only started when the server says its
* trace is on (`perf: true` in /api/chapter). It reports, in batches to POST api/perf:
* - `res` the browser's resource timing of every library image: queueing, connection, time to first byte,
* download, size, the server's own `stat` time and whether the connection was reused;
* - `view` a page turn (page/book mode): how long the new page(s) took to appear, and whether they were
* preloaded already;
* - `stall` flow mode: a page scrolled into view before its image had loaded, and how long it stayed empty.
*/
window.PerfTrace = {
start(pageName) {
let queue = [], on = true;
const push = e => { if (on) queue.push(e); };
const rel = url => { try { return decodeURIComponent(new URL(url).pathname); } catch { return String(url); } };
const flush = beacon => {
if (!queue.length || !on) return;
const body = JSON.stringify({ page: pageName, events: queue.splice(0, 300) });
if (beacon && navigator.sendBeacon) navigator.sendBeacon('api/perf', new Blob([body], { type: 'application/json' }));
else fetch('api/perf', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body, keepalive: true })
.then(r => r.json()).then(r => { if (r.on === false) on = false; }).catch(() => {});
};
setInterval(() => flush(false), 5000);
addEventListener('pagehide', () => flush(true));
push({ e: 'hello', ua: navigator.userAgent, conn: navigator.connection ? navigator.connection.effectiveType : '',
dpr: devicePixelRatio, vw: innerWidth, vh: innerHeight });
if (window.PerformanceObserver) {
new PerformanceObserver(list => {
for (const r of list.getEntries()) {
if (!/\/lib\//.test(r.name)) continue;
const st = (r.serverTiming || []).find(x => x.name === 'stat');
push({ e: 'res', url: rel(r.name), at: r.startTime,
queue: r.requestStart ? r.requestStart - r.startTime : null, // waiting for a free connection
connect: r.connectEnd - r.connectStart, dns: r.domainLookupEnd - r.domainLookupStart,
ttfb: r.responseStart && r.requestStart ? r.responseStart - r.requestStart : null,
download: r.responseStart ? r.responseEnd - r.responseStart : null,
total: r.duration, size: r.transferSize, body: r.encodedBodySize, server: st ? st.duration : null,
proto: r.nextHopProtocol || '', cached: r.transferSize === 0 && r.encodedBodySize > 0 });
}
}).observe({ type: 'resource', buffered: true });
}
return {
// Called right after a page turn put new <img> elements on screen.
view(imgs, info) {
const t0 = performance.now(), waiting = imgs.filter(img => !img.complete);
const event = { e: 'view', ...info, n: imgs.length, preloaded: !waiting.length };
if (!waiting.length) { push({ ...event, wait: 0 }); return; }
Promise.all(waiting.map(img => (img.decode ? img.decode() : Promise.resolve()).catch(() => {})))
.then(() => requestAnimationFrame(() => push({ ...event, wait: performance.now() - t0,
url: rel(waiting[0].currentSrc || waiting[0].src) })));
},
// Flow mode: watch pages enter the real viewport (not the preload margin) before their image is there.
watchFlow(root, els) {
if (!window.IntersectionObserver) return;
const io = new IntersectionObserver(entries => {
for (const en of entries) {
const img = en.target.firstChild;
if (!en.isIntersecting || (img.complete && img.naturalWidth)) continue;
const t0 = performance.now(), hadSrc = !!img.getAttribute('src');
const done = () => push({ e: 'stall', wait: performance.now() - t0, hadSrc, url: rel(img.currentSrc || img.src) });
img.addEventListener('load', done, { once: true });
io.unobserve(en.target);
}
}, { root });
for (const el of els) io.observe(el);
return io;
},
};
},
};
+1
View File
@@ -16,6 +16,7 @@
<script src="reading-clock.js" defer></script>
<script src="keys.js" defer></script>
<script src="moments.js" defer></script>
<script src="perf.js" defer></script>
<script src="reader.js" defer></script>
</head>
<body class="reader">
+67 -7
View File
@@ -74,6 +74,9 @@
D = payload;
}
catch (e) { canvas.textContent = ''; canvas.append(h('p', { class: 'loading' }, e.message + ' · ', h('a', { href: './' }, 'Library'))); return; }
// Timings for the server's performance trace, only while an admin has it switched on.
const perf = D.perf && window.PerfTrace ? PerfTrace.start('read') : null;
let perfFlow = null;
// An already-open chapter also closes when a window ends or an unlock expires.
const checkReadingAccess = async () => {
@@ -152,6 +155,55 @@
// A link to a page (e.g. "Open in reader" from a moment) wins over the saved position.
if (P.get('p') != null) { const k = list.findIndex(x => x.i >= +P.get('p')); if (k >= 0) cur = k; }
// Stale reading sessions: a tab left open (or restored by the browser) on another device must not move
// the "Continue" point back. `seen` is the newest progress time this tab knows: the link's `at`, kept
// current in the address bar, so a reload or restored tab still knows it. Without `at` the open was a
// deliberate one (bookmark, notification) and starts from the current head.
const tab = tabId();
let seen = Number(P.get('at')) || (D.latest ? D.latest.t : 0), stale = null, staleCard = null;
const chapterUrl = (c, v) => readUrl(D.manga.id, c, v, seen);
const newer = latest => !!latest && latest.t > seen && latest.w !== tab;
function markSeen(t) {
seen = Math.max(seen, t || 0);
const q = new URLSearchParams(location.search);
q.set('at', seen);
history.replaceState(history.state, '', '?' + q);
}
// Another tab or device saved a newer position. The same place is no conflict; anything else asks.
function checkLatest(latest) {
if (!newer(latest)) return;
if (latest.c === D.chapter.id && list[cur] && latest.p === list[cur].i) { markSeen(latest.t); return; }
showStale(latest);
}
function showStale(latest) {
stale = latest;
clearTimeout(saveTimer);
if (staleCard) staleCard.remove();
const where = latest.done ? `${latest.label} (finished)` : `${latest.label} · page ${latest.p + 1}`;
const go = h('a', { class: 'btn primary', href: readUrl(D.manga.id, latest.continue.c, latest.continue.v, latest.t) }, latest.continue.text);
const stay = h('button', { class: 'btn', type: 'button' }, 'Keep reading here');
stay.onclick = () => {
staleCard.remove(); staleCard = null; stale = null;
markSeen(latest.t);
saveProgress();
stage.focus({ preventScroll: true });
};
staleCard = h('div', { class: 'stale-session', role: 'dialog', 'aria-modal': 'true', 'aria-labelledby': 'stale-title' },
h('div', { class: 'stale-box' },
h('h2', { id: 'stale-title' }, 'You have read further since'),
h('p', {}, 'This page is an older reading session. Your newest reading position is ', h('b', {}, where), ` (${ago(latest.t)}).`),
h('div', { class: 'stale-actions' }, go, stay),
h('p', { class: 'muted small' }, `Keeping ${D.chapter.label} makes it your “Continue” point again.`)));
document.body.append(staleCard);
go.focus();
}
async function recheck() {
if (stale || document.hidden) return;
try { checkLatest((await api('api/progress/latest?' + qsa({ m: D.manga.id }))).latest); } catch { /* offline: the next save checks again */ }
}
document.addEventListener('visibilitychange', recheck);
window.addEventListener('pageshow', e => { if (e.persisted) recheck(); });
// Book mode: pages pair up inside each run of single pages. Runs restart after every double page / cover,
// so one missing pair never shifts the rest of the chapter. "Shift" makes a run start with a lone page.
function computeSpreads() {
@@ -237,7 +289,7 @@
const n = D.next;
return h('div', { class: 'end' }, h('div', { class: 'endbox' },
h('div', { class: 'muted' }, 'End of ' + D.chapter.label + (D.chapter.title ? ' · ' + D.chapter.title : '')),
n ? h('a', { class: 'btn primary', href: readUrl(D.manga.id, n.id), onclick: () => saveProgress(true) }, `Next: ${n.label}${n.title ? ' · ' + n.title : ''} ›`)
n ? h('a', { class: 'btn primary', href: chapterUrl(n.id), onclick: e => { e.currentTarget.href = chapterUrl(n.id); saveProgress(true); } }, `Next: ${n.label}${n.title ? ' · ' + n.title : ''} ›`)
: h('div', {}, 'You are up to date.'),
h('a', { class: 'btn', href: mangaUrl(D.manga.id) }, 'Back to ' + D.manga.title)));
}
@@ -250,11 +302,13 @@
if (mode === 'flow') {
els = list.map(p => canvas.appendChild(pageEl(p, true)));
endEl = canvas.appendChild(endCard());
if (perf) { if (perfFlow) perfFlow.disconnect(); perfFlow = perf.watchFlow(stage, els); }
} else if (atEnd) {
endEl = canvas.appendChild(endCard());
} else {
shown = mode === 'page' ? [cur] : spreads[spreadIdx];
els = shown.map(k => canvas.appendChild(pageEl(list[k])));
if (perf) perf.view(els.map(el => el.firstChild), { mode, page: list[shown[0]].i });
// The pages on screen get the bandwidth first, then the next ones are fetched.
const token = ++buildToken, waiting = els.map(el => el.firstChild).filter(img => !img.complete);
Promise.all(waiting.map(img => new Promise(done => {
@@ -584,7 +638,7 @@
function goChapter(c) {
if (!c) return toast('You are up to date');
saveProgress(true);
location.href = readUrl(D.manga.id, c.id);
location.href = chapterUrl(c.id);
}
// Flow: jump to the top of the next page, but never further than ~one screen.
function flowStep(dir) {
@@ -696,8 +750,13 @@
}
function saveProgress(keepalive) {
clearTimeout(saveTimer);
const body = JSON.stringify({ m: D.manga.id, c: D.chapter.id, v: D.version, p: list[cur].i, n: all.length, done, shift: [...shift] });
fetch('api/progress', { method: 'POST', keepalive: !!keepalive, headers: { 'Content-Type': 'application/json' }, body }).catch(() => {});
if (stale) return; // nothing is saved until the reader chose where to go on
const body = JSON.stringify({ m: D.manga.id, c: D.chapter.id, v: D.version, p: list[cur].i, n: all.length, done, shift: [...shift], tab, seen });
fetch('api/progress', { method: 'POST', keepalive: !!keepalive, headers: { 'Content-Type': 'application/json' }, body })
.then(r => r.json()).then(r => {
if (r.stale) checkLatest(r.latest);
else if (r.ok && r.t) markSeen(r.t);
}).catch(() => {});
}
addEventListener('pagehide', () => saveProgress(true));
@@ -712,17 +771,17 @@
groups[c.book].append(h('option', { value: c.id }, `${c.label}${c.title ? ' · ' + c.title : ''}${c.extra ? ' (extra)' : ''}${c.done ? ' ✓' : ''}`));
}
chsel.value = D.chapter.id;
chsel.onchange = () => { saveProgress(true); location.href = readUrl(D.manga.id, chsel.value); };
chsel.onchange = () => { saveProgress(true); location.href = chapterUrl(chsel.value); };
$('#back').href = mangaUrl(D.manga.id);
for (const [id, c] of [['#prevch', D.prev], ['#nextch', D.next]]) {
const a = $(id);
if (c) { a.href = readUrl(D.manga.id, c.id); a.title = c.label + (c.title ? ' · ' + c.title : ''); a.onclick = () => saveProgress(true); }
if (c) { a.href = chapterUrl(c.id); a.title = c.label + (c.title ? ' · ' + c.title : ''); a.onclick = () => { a.href = chapterUrl(c.id); saveProgress(true); }; }
else a.classList.add('disabled');
}
for (const v of D.versions) ver.append(h('option', { value: v.id }, (v.source || v.id) + (v.score ? ` (${v.score > 0 ? '+' : ''}${v.score})` : '')));
ver.value = D.version;
ver.hidden = D.versions.length < 2;
ver.onchange = () => { saveProgress(true); location.href = readUrl(D.manga.id, D.chapter.id, ver.value); };
ver.onchange = () => { saveProgress(true); location.href = chapterUrl(D.chapter.id, ver.value); };
for (const b of modeBtns) b.onclick = () => { if (b.dataset.mode !== mode) setMode(b.dataset.mode, true); };
toneButton.onclick = cycleTone;
$('#moment').onclick = saveMoment;
@@ -743,6 +802,7 @@
for (const bar of [$('#top'), $('#bot')]) bar.addEventListener('click', e => { if (e.target.closest('button')) stage.focus({ preventScroll: true }); });
setMode(mode, false);
if (newer(D.latest)) checkLatest(D.latest); else markSeen(seen);
if (S.readingClock) readingClock = createReadingClock({
button: $('#readclock'),
panel: $('#clockpanel'),
+59 -1
View File
@@ -15,15 +15,68 @@
const saved = h('span', { class: 'saved', hidden: true }, 'Saved');
shell.titleRow.append(saved);
let savedTimer = 0;
let refreshSessions = () => {};
async function save(patch) {
Object.assign(S, (await api('api/settings', patch)).settings);
saved.hidden = false;
clearTimeout(savedTimer);
savedTimer = setTimeout(() => { saved.hidden = true; }, 1500);
refreshSessions();
}
// Which login session holds which reader settings (reader-sessions.js on the server).
function readerSessionsCard() {
const TONE = { original: 'Original', inverted: 'Inverted', sepia: 'Sepia', dim: 'Dim', 'extra-dim': 'Extra dim' };
const START = { last: 'last mode', flow: 'always Flow', page: 'always Page', book: 'always Book' };
const summary = r => [`${LABEL[r.lastMode] || r.lastMode} (opens: ${START[r.startMode] || r.startMode})`, `tone ${TONE[r.pageTone] || r.pageTone}`,
'zoom ' + (Object.keys(r.zoom || {}).length ? Object.entries(r.zoom).map(([k, z]) => `${LABEL[k] || k} ${Math.round(z * 100)}%`).join(', ') : 'fit'),
r.skipExtras ? 'skips extras' : null, r.skipCovers ? 'skips covers' : null, r.flowPageGap === false ? 'no Flow gap' : null].filter(Boolean).join(' · ');
const state = h('div', {}, h('p', { class: 'muted' }, 'Loading sessions…'));
const message = h('span', { class: 'small' });
async function act(id, action, current) {
try {
draw(await api('api/reader-sessions', { id, action }));
// This page's controls show this session's settings; reload them when they changed.
if (current && action === 'follow') location.reload();
} catch (error) { message.textContent = error.message; message.className = 'small error'; }
}
function draw(o) {
const current = o.sessions.find(s => s.current), source = o.sessions.find(s => s.id === o.latestFrom);
const latestText = !o.latestFrom ? 'one shared set (changed while per-session settings were off, or never changed)'
: source ? (source.current ? 'this session' : `${source.device}, active ${ago(source.seen)}`) : 'a session that has since signed out';
const intent = !o.enabled ? 'Off: every session uses the same reader settings, and a change applies everywhere.'
: current?.locked ? 'This session has its own reader settings. Changes here stay here and become the starting point for new sessions.'
: 'This session follows the latest reader settings. Your first change here gives it its own copy, which stays put when other sessions change theirs.';
state.replaceChildren(
h('p', {}, h('b', {}, 'Now: '), intent),
o.enabled ? h('p', { class: 'muted small' }, 'New sessions start from the latest settings: ', h('b', {}, summary(o.latest)), ` — from ${latestText}.`)
: h('p', { class: 'muted small' }, 'Shared settings: ', h('b', {}, summary(o.latest))),
h('ul', { class: 'reader-sessions' }, o.sessions.map(s => h('li', {},
h('div', {}, h('b', {}, s.device), s.current ? ' (this session)' : '', ' · ',
h('span', { class: 'muted small' }, `signed in ${new Date(s.created).toLocaleDateString()}, active ${ago(s.seen)}`)),
h('div', { class: 'small' }, !o.enabled ? 'Uses the shared settings' + (s.locked ? ' (its own copy is kept for when this is on again)' : '')
: s.locked ? 'Own settings: ' + summary(s.reader) : 'Follows the latest settings'),
o.enabled ? h('div', { class: 'row' },
s.locked ? h('button', { class: 'btn small', onclick: () => act(s.id, 'follow', s.current) }, 'Follow latest again') : null,
s.locked && s.id !== o.latestFrom ? h('button', { class: 'btn small', onclick: () => act(s.id, 'latest', s.current) }, 'Use for new sessions') : null) : null))),
message);
}
refreshSessions = () => api('api/reader-sessions').then(draw, error => { message.textContent = error.message; message.className = 'small error'; });
refreshSessions();
return h('section', { class: 'card' },
h('h2', {}, 'Reader settings per session'),
// The controls above switch between this session's set and the shared one, so reload them.
h('label', { class: 'check' },
h('input', { type: 'checkbox', checked: !!S.readerPerSession, onchange: async e => { await save({ readerPerSession: e.target.checked }); location.reload(); } }),
h('span', {}, 'Keep reader settings per session', h('div', { class: 'muted small' }, 'Each sign-in (device or browser) keeps its own reader mode, zoom, tone and the other reader preferences above once you change one there. A new sign-in copies the settings of the session you used last. Turn this off for one shared set.'))),
state);
}
const check = (key, text, hint) => h('label', { class: 'check' },
h('input', { type: 'checkbox', checked: !!S[key], onchange: e => save({ [key]: e.target.checked }) }),
h('span', {}, text, hint ? h('div', { class: 'muted small' }, hint) : null));
const number = (key, text, max) => h('label', { class: 'field' }, h('span', {}, text),
h('input', { type: 'number', min: 0, max, step: 1, value: S[key], style: 'width:6em',
onchange: e => { const n = Math.round(+e.target.value); if (n >= 0 && n <= max) save({ [key]: n }); else e.target.value = S[key]; } }));
const startMode = h('select', { onchange: e => save({ startMode: e.target.value }) },
h('option', { value: 'last' }, 'The mode I used last'),
@@ -90,10 +143,15 @@
h('button', { class: 'btn small', onclick: async () => { await save({ zoom: {} }); zoomText.textContent = zooms(); } }, 'Reset to fit'),
h('div', { class: 'muted small' }, '100% = the page fits the screen. Change it in the reader with Ctrl + mouse wheel, a pinch, or the + / - keys.'))));
section('user/reading').append(readerSessionsCard());
section('user/reading').append(h('section', { class: 'card' },
h('h2', {}, 'Library page'),
check('continueRow', 'Show "Continue where you left off" at the top',
'One row with the five manga you read most recently and have not finished. It hides itself while you search, filter by status, or sort by "Recently read".')));
'One row with the five manga you read most recently that still have chapters left after the last one you read. It hides itself while you search, filter by status, or sort by "Recently read".'),
h('p', { class: 'muted small' }, 'Rereading: later chapters you finished in an earlier pass count as left again when either rule applies. 0 turns a rule off.'),
number('rereadDays', 'Finished more than this many days before my last read', 3650),
number('rereadChapters', 'Or I have since read more than this many chapters in a row', 1000)));
section('user/reading').append(h('section', { class: 'card' },
h('h2', {}, 'Chapter list dates'),
+88
View File
@@ -0,0 +1,88 @@
'use strict';
/* Reader settings per login session (docs/architecture.md, "Reader settings per session").
*
* The user's own reader settings (`user.settings`) are the "latest" set. A session that never changed a
* reader setting follows the latest set; the first change copies the latest set into the session and
* locks it there, and also makes it the new latest set. Using a locked session makes its set the latest
* again, so a new login starts from the last active session. With `readerPerSession: false` there is
* one set per user, as before; locked sets are kept and return when the switch is turned on again. */
const crypto = require('crypto');
const READER_KEYS = ['startMode', 'lastMode', 'adapt', 'skipExtras', 'skipCovers', 'flowPageGap', 'pageTone', 'zoom'];
const perSession = settings => settings.readerPerSession !== false;
const publicId = key => crypto.createHash('sha256').update('reader-session:' + key).digest('hex').slice(0, 16);
function pick(settings) {
const out = {};
for (const k of READER_KEYS) if (settings[k] !== undefined) out[k] = k === 'zoom' ? { ...settings[k] } : settings[k];
return out;
}
// `base` is the user's merged settings (defaults + user.settings); the session's locked set wins over it.
function effective(base, session) {
if (!perSession(base) || !session?.reader) return base;
return { ...base, ...pick(session.reader), zoom: { ...session.reader.zoom } };
}
// Splits a validated settings patch: reader keys go to the session (locking it) and to the latest set.
function applyPatch({ user, base, session, key, patch }) {
const s = (user.settings = user.settings || {});
const reader = pick(patch), rest = Object.fromEntries(Object.entries(patch).filter(([k]) => !READER_KEYS.includes(k)));
Object.assign(s, rest);
if (!Object.keys(reader).length) return;
if (perSession({ ...base, ...rest }) && session) {
session.reader = { ...pick(effective(base, session)), ...reader };
Object.assign(s, pick(session.reader));
user.readerFrom = publicId(key);
} else {
Object.assign(s, reader);
user.readerFrom = null;
}
}
// A request from a locked session makes its set the latest one. Returns true when something changed.
function noteActive(user, base, session, key) {
if (!perSession(base) || !session?.reader) return false;
const id = publicId(key);
if (user.readerFrom === id) return false;
user.settings = { ...(user.settings || {}), ...pick(session.reader) };
user.readerFrom = id;
return true;
}
function deviceLabel(agent) {
const a = String(agent || '');
if (!a) return 'Unknown device';
const os = /iPhone/.test(a) ? 'iPhone' : /iPad/.test(a) ? 'iPad' : /Android/.test(a) ? 'Android'
: /Windows/.test(a) ? 'Windows' : /Mac OS X|Macintosh/.test(a) ? 'Mac' : /CrOS/.test(a) ? 'ChromeOS' : /Linux/.test(a) ? 'Linux' : '';
const browser = /Edg\//.test(a) ? 'Edge' : /OPR\//.test(a) ? 'Opera' : /Firefox\//.test(a) ? 'Firefox'
: /Chrome\//.test(a) ? 'Chrome' : /Safari\//.test(a) ? 'Safari' : '';
return [browser, os].filter(Boolean).join(' on ') || a.slice(0, 60);
}
// What Settings shows: every login session of the user with its state and effective reader set.
function overview(user, base, sessions, currentKey) {
const list = Object.entries(sessions).filter(([, s]) => s.u === user.name).map(([key, s]) => ({
id: publicId(key), device: deviceLabel(s.agent), created: s.t, seen: s.seen || s.t,
current: key === currentKey, locked: !!s.reader, reader: pick(effective(base, s)),
})).sort((a, b) => b.current - a.current || b.seen - a.seen);
return { enabled: perSession(base), current: currentKey ? publicId(currentKey) : null,
latestFrom: user.readerFrom || null, latest: pick(base), sessions: list };
}
// Settings actions on one of the user's sessions: 'follow' drops its lock, 'latest' makes its set the latest.
function act(user, base, sessions, id, action) {
const entry = Object.entries(sessions).find(([key, s]) => s.u === user.name && publicId(key) === id);
if (!entry) return false;
const [key, session] = entry;
if (action === 'follow') { delete session.reader; if (user.readerFrom === id) user.readerFrom = null; return true; }
if (action === 'latest') {
user.settings = { ...(user.settings || {}), ...pick(effective(base, session)) };
user.readerFrom = session.reader ? publicId(key) : null;
return true;
}
return false;
}
module.exports = { READER_KEYS, publicId, pick, effective, applyPatch, noteActive, deviceLabel, overview, act };
+1
View File
@@ -35,6 +35,7 @@ async function body(req) {
if (url.pathname === '/health') return json(res, 200, { ok: true, status: service.state.status });
if (TOKEN && req.headers.authorization !== `Bearer ${TOKEN}`) return json(res, 401, { error: 'Unauthorized' });
if (req.method === 'GET' && url.pathname === '/api/state') return json(res, 200, await service.publicState());
if (req.method === 'GET' && url.pathname === '/api/revision') return json(res, 200, service.revision());
if (req.method !== 'POST') return json(res, 404, { error: 'Not found' });
const input = await body(req);
if (url.pathname === '/api/queue') return json(res, 200, { item: await service.enqueue(input), state: await service.publicState() });
+15 -4
View File
@@ -16,6 +16,7 @@ const { writeJsonAtomic } = require('./lib/json-file');
const { ReleaseDateSources, refreshMangaReleaseDates } = require('./metadata/release-dates');
const RATE_KEYS = ['minMs', 'maxMs', 'chapterMinMs', 'chapterMaxMs'];
const INVENTORY_SUMMARY_MS = 15 * 60000; // status summary age limit; decisions still scan (libraryInventory)
const FAST_RATE_SETTINGS = {
minMs: 1000, maxMs: 4000, chapterMinMs: 5000, chapterMaxMs: 15000,
};
@@ -401,7 +402,7 @@ class ScraperService {
} catch (error) { await this.event('warning', `Release dates for ${manga.title} failed: ${error.message}`, { mangaId: manga.id }); }
}
// The reader reconciles fully when the revision rises without a visible folder change.
if (written) { this.state.publishedRevision = Number(this.state.publishedRevision || 0) + 1; await this.persist(); }
if (written) { this.invalidateInventory(); this.state.publishedRevision = Number(this.state.publishedRevision || 0) + 1; await this.persist(); }
await this.event('info', `Release dates: ${written} chapter${written === 1 ? '' : 's'} dated in ${checked} ongoing series`, { reason, checked, written });
return { checked, written };
}
@@ -583,6 +584,15 @@ class ScraperService {
} catch { return null; }
}
inventoryStale() {
return this.inventoryCacheGeneration !== this.inventoryGeneration || Date.now() - this.inventoryCachedAt >= INVENTORY_SUMMARY_MS;
}
// What the reader polls every few seconds: no library walk, no event log read.
revision() {
return { publishedRevision: this.state.publishedRevision || 0, status: this.state.status, paused: this.state.paused };
}
async publicState() {
const activeItems = this.state.queue.filter(item => Object.values(this.state.currentIds || {}).includes(item.id));
const active = await Promise.all(activeItems.map(async current => ({
@@ -593,8 +603,9 @@ class ScraperService {
const job = active[0]?.job || null;
// Status and watchdog probes must not wait for thousands of bind-mount reads.
// Share one background scan, serve the previous summary, and surface its age/error.
if (!this.inventoryRefresh && (this.inventoryCacheGeneration !== this.inventoryGeneration || Date.now() - this.inventoryCachedAt >= 60000)
&& Date.now() - this.inventoryAttemptedAt >= 10000) {
// The scraper is the library's only writer and invalidates the summary on every publish, so the age
// limit only catches outside edits. A walk reads every chapter.json: on a NAS that is ~10,000 opens.
if (!this.inventoryRefresh && this.inventoryStale() && Date.now() - this.inventoryAttemptedAt >= 10000) {
this.refreshInventory().catch(() => {});
}
const inventory = this.inventoryCache;
@@ -609,7 +620,7 @@ class ScraperService {
updateChecks: Object.values(this.state.updateChecks || {}),
libraryStatus: {
ready: !!inventory, refreshing: !!this.inventoryRefresh,
stale: this.inventoryCacheGeneration !== this.inventoryGeneration || Date.now() - this.inventoryCachedAt >= 60000,
stale: this.inventoryStale(),
updated: this.inventoryCachedAt ? new Date(this.inventoryCachedAt).toISOString() : null,
error: this.inventoryError,
},
+17
View File
@@ -80,6 +80,23 @@ test('background inventory failures are visible without breaking controls or lau
assert.equal(scans, 2);
});
test('status polling walks the library again only after a change or after 15 minutes', async () => {
let scans = 0;
const service = fixture(async () => { scans++; return inventory('Demo'); });
await service.libraryInventory(true);
service.inventoryCachedAt = Date.now() - 10 * 60000; service.inventoryAttemptedAt = 0;
for (let n = 0; n < 5; n++) await service.publicState();
assert.equal(scans, 1);
assert.deepEqual(service.revision(), { publishedRevision: 0, status: 'paused', paused: true });
assert.equal(scans, 1);
service.invalidateInventory();
await service.publicState(); await service.inventoryRefresh;
assert.equal(scans, 2);
service.inventoryCachedAt = Date.now() - 16 * 60000; service.inventoryAttemptedAt = 0;
await service.publicState(); await service.inventoryRefresh;
assert.equal(scans, 3);
});
test('dashboard polling reuses a recent snapshot while decisions still force a fresh scan', async () => {
let scans = 0;
const service = fixture(async () => { scans++; return inventory('Demo'); });
+170 -37
View File
@@ -12,6 +12,8 @@
* SCRAPER_URL=http://127.0.0.1:8091 scraper daemon API
* SCRAPER_TOKEN=... shared daemon token (optional on loopback)
* BACKUP=/backup folder for built-in DATA backups (unset = backups off)
* READ_AHEAD=1 read an opened chapter's pages (then the next chapter's) into the OS cache, 0 = off
* PERF_TRACE=1 performance trace to DATA/perf/ (or touch DATA/perf/trace.on at runtime; docs/performance.md)
*/
const http = require('http'), fs = require('fs'), path = require('path'), crypto = require('crypto');
const { scanLibrary, scanChangedLibrary, libraryStamps, snapshotIndex, restoreIndex } = require('./library');
@@ -36,6 +38,9 @@ const { chapterDates } = require('./chapter-dates');
const { parseSchedule, publicSchedule, loadQuestionBank, createReadingAccess } = require('./reading-access');
const { BackupService, BackupError } = require('./backup');
const moments = require('./moments');
const { Tracer, requestTrace } = require('./perf/trace');
const { ReadAhead } = require('./perf/read-ahead');
const readerSessions = require('./reader-sessions');
const PORT = +process.env.PORT || 8080;
const LIB = path.resolve(process.env.LIBRARY || 'library');
@@ -64,6 +69,9 @@ const MIME = {
// ---------- storage: one JSON file, written atomically ----------
fs.mkdirSync(DATA, { recursive: true });
const tracer = new Tracer({ dir: path.join(DATA, 'perf') }).start();
const readAhead = process.env.READ_AHEAD === '0' ? null
: new ReadAhead({ onBatch: batch => tracer.log({ k: 'ahead', ...batch }) });
const catalog = new Catalog(DATA);
const greatestCatalog = loadGreatestCatalog();
const greatestById = new Map(greatestCatalog.entries.map(entry => [entry.id, entry]));
@@ -112,8 +120,10 @@ function checkPw(pw, stored) {
}
const DUMMY_HASH = hashPw('x');
const DEFAULTS = { startMode: 'last', lastMode: 'page', adapt: true, skipExtras: false, skipCovers: false, flowPageGap: true, pageTone: 'original', readingClock: false, readingStory: true,
continueRow: true, readDate: 'hour', releaseDate: true, discoveryEnabled: false, discoveryReadingActivity: false, discoveryMature: false, zoom: {}, webhook: '' };
const settingsOf = u => ({ ...DEFAULTS, ...u.settings, zoom: { ...(u.settings && u.settings.zoom) } });
continueRow: true, rereadDays: 7, rereadChapters: 5, readDate: 'hour', releaseDate: true, discoveryEnabled: false, discoveryReadingActivity: false, discoveryMature: false, zoom: {}, webhook: '', readerPerSession: true };
const userSettingsOf = u => ({ ...DEFAULTS, ...u.settings, zoom: { ...(u.settings && u.settings.zoom) } });
// With a login session, its locked reader settings (reader-sessions.js) win over the user's latest set.
const settingsOf = (u, sess) => readerSessions.effective(userSettingsOf(u), sess);
function createUser(name, pw, admin, replace) {
name = String(name || '').trim().toLowerCase(); pw = String(pw || '');
@@ -125,13 +135,21 @@ function createUser(name, pw, admin, replace) {
save();
return db.users[name];
}
function startSession(res, user) {
function startSession(req, res, user) {
const token = crypto.randomBytes(32).toString('base64url');
db.sessions[sha(token)] = { u: user.name, t: Date.now() };
db.sessions[sha(token)] = { u: user.name, t: Date.now(), agent: String(req.headers['user-agent'] || '').slice(0, 300) };
save();
res.setHeader('Set-Cookie', `sid=${token}; Path=/; HttpOnly; SameSite=Lax; Max-Age=31536000${SECURE_COOKIE ? '; Secure' : ''}`);
}
const tokenOf = req => { const m = /(?:^|;\s*)sid=([^;]+)/.exec(req.headers.cookie || ''); return m ? m[1] : null; };
function sessionKeyOf(req) { const t = tokenOf(req); return t ? sha(t) : null; }
// Last-seen time (coarse, to keep db.json writes rare) and device label of a login session.
function touchSession(u, sess, key, req) {
let changed = false;
if (!sess.agent && req.headers['user-agent']) { sess.agent = String(req.headers['user-agent']).slice(0, 300); changed = true; }
if (Date.now() - (sess.seen || 0) > 5 * 60e3) { sess.seen = Date.now(); changed = true; }
if (changed) save();
}
function userOf(req) { const t = tokenOf(req), s = t && db.sessions[sha(t)]; return (s && db.users[s.u]) || null; }
for (const [k, s] of Object.entries(db.sessions)) if (Date.now() - s.t > 365 * 864e5 || !db.users[s.u]) delete db.sessions[k];
@@ -175,8 +193,10 @@ async function performScan(mode) {
scanState = { at: Date.now(), ms: Date.now() - t0, mode, changedMangas: changed.length,
mangas: lib.mangas.length, chapters, pages, stale: false, error: null };
console.log(`[scan] ${mode}: ${changed.length} changed manga, ${lib.mangas.length} manga, ${chapters} chapters, ${pages} pages in ${scanState.ms} ms`);
tracer.log({ k: 'scan', mode, ms: scanState.ms, changed: changed.length });
return scanState;
} catch (e) {
tracer.log({ k: 'scan', mode, ms: Date.now() - t0, error: e.message });
scanState = { ...scanState, at: Date.now(), error: e.message };
console.error('[scan] failed:', e.message);
return scanState;
@@ -253,9 +273,14 @@ async function scraperRequest(endpoint, body) {
options.method = 'POST'; headers['content-type'] = 'application/json'; options.body = JSON.stringify(body);
}
let response;
const t0 = performance.now();
try { response = await fetch(SCRAPER_URL + endpoint, options); }
catch (error) { throw new HttpError(503, `Scraper service unavailable: ${error.message}`); }
catch (error) {
tracer.log({ k: 'scraper', endpoint, ms: Math.round(performance.now() - t0), error: error.message });
throw new HttpError(503, `Scraper service unavailable: ${error.message}`);
}
const data = await response.json().catch(() => ({}));
tracer.log({ k: 'scraper', endpoint, ms: Math.round(performance.now() - t0), status: response.status });
if (!response.ok) throw new HttpError(response.status >= 500 ? 502 : response.status, data.error || `Scraper HTTP ${response.status}`);
return data;
}
@@ -424,14 +449,58 @@ function readAtOf(progress, record) {
if (progress.doneAt) return { at: progress.doneAt, approx: false };
return earliestCompletion(record);
}
function continueOf(m, prog, s) {
const ok = c => !(s.skipExtras && c.extra);
// The most recently saved chapter position of a manga: the head that "Continue" follows.
function lastOf(m, prog) {
let last = null;
for (const ch of m.chapters) { const p = prog[ch.id]; if (p && (!last || p.t > last.p.t)) last = { ch, p }; }
if (!last) { const first = m.chapters.find(ok) || m.chapters[0]; return { c: first.id, text: 'Start reading · ' + first.label }; }
if (!last.p.done) return { c: last.ch.id, v: last.p.v, text: `Continue · ${last.ch.label} · page ${last.p.p + 1}` };
return last;
}
// `at` is the head's time; reader links carry it so a reader opened from a stale page notices a newer head.
function continueOf(m, prog, s) {
const ok = c => !(s.skipExtras && c.extra);
const last = lastOf(m, prog);
if (!last) { const first = m.chapters.find(ok) || m.chapters[0]; return { c: first.id, at: 0, text: 'Start reading · ' + first.label }; }
const at = last.p.t;
if (!last.p.done) return { c: last.ch.id, v: last.p.v, at, text: `Continue · ${last.ch.label} · page ${last.p.p + 1}` };
const next = m.chapters.slice(m.chapters.indexOf(last.ch) + 1).find(ok);
return next ? { c: next.id, text: 'Read next · ' + next.label } : { c: last.ch.id, v: last.p.v, text: 'Read again · ' + last.ch.label };
return next ? { c: next.id, at, text: 'Read next · ' + next.label } : { c: last.ch.id, v: last.p.v, at, text: 'Read again · ' + last.ch.label };
}
// Chapters still ahead of the head: never finished, or last read in an earlier pass. 0 = caught up.
// A done chapter after the head belongs to an earlier pass (the head is a reread) when it was saved
// more than `rereadDays` before the head, or before a run of more than `rereadChapters` chapters read
// in order up to the head. 0 turns either rule off.
function leftOf(m, prog, s) {
const last = lastOf(m, prog);
if (!last) return 0;
const chapters = m.chapters.filter(c => c === last.ch || !(s.skipExtras && c.extra)), at = chapters.indexOf(last.ch);
let runStart = last.p.t, run = 1;
for (let i = at - 1; i >= 0; i--) {
const p = prog[chapters[i].id];
if (!p || !p.done || p.t > runStart) break;
runStart = p.t; run++;
}
const gap = s.rereadDays > 0 ? s.rereadDays * 24 * 3600 * 1000 : Infinity;
const passStart = s.rereadChapters > 0 && run > s.rereadChapters ? runStart : -Infinity;
return chapters.slice(at + 1).filter(c => { const p = prog[c.id]; return !p || !p.done || last.p.t - p.t > gap || p.t < passStart; }).length
+ (last.p.done ? 0 : 1);
}
// What a reader tab compares itself with (stale reading sessions, docs/architecture.md).
function latestOf(m, prog, s) {
const last = lastOf(m, prog);
if (!last) return null;
const { t, w, v, p, n, done } = last.p;
return { t, w: w || '', c: last.ch.id, label: last.ch.label, v, p, n, done, continue: continueOf(m, prog, s) };
}
// From the saved page on, then the pages before it; afterwards the next chapter's preferred version.
function warmChapter(u, m, ch, v, progress, s) {
const from = Math.min(Math.max(0, +(progress && progress.v === v.id && progress.p) || 0), v.pages.length);
const pages = v.pages.slice(from).concat(v.pages.slice(0, from)).map(p => path.join(LIB, v.rel, p.f));
readAhead.warm(pages, `${m.id}/${ch.id}`, u.name);
const next = neighbour(m, ch, 1, s), nextCh = next && m.chapterById.get(next.id);
if (nextCh) {
const nv = ranked(m, nextCh)[0].v;
readAhead.then(nv.pages.map(p => path.join(LIB, nv.rel, p.f)), `${m.id}/${nextCh.id}`, u.name);
}
}
function neighbour(m, ch, step, s) {
for (let i = m.chapters.indexOf(ch) + step; i >= 0 && i < m.chapters.length; i += step) {
@@ -446,11 +515,11 @@ const PUBLIC = new Set(['/api/auth', '/api/login', '/api/signup']);
const routes = {
'GET /api/health': () => ({ ok: !scanState.error, service: 'reader', uptime: Math.round(process.uptime()), scan: scanState }),
'GET /api/collections': ({ u }) => ({ collections: routes['GET /api/library']({ u }).collections }),
'GET /api/collection': ({ u, q }) => {
'GET /api/collection': ({ u, q, sess }) => {
const view = routes['GET /api/library']({ u }).collections.find(c => c.id === q.get('id')) || bad(404, 'Collection not found');
const next = view.members.find(m => m.read < m.chapters) || view.members[0];
const manga = mangaOf(next.id);
return { ...view, continue: { m: next.id, ...continueOf(manga, progressOf(u, manga), settingsOf(u)) },
return { ...view, continue: { m: next.id, ...continueOf(manga, progressOf(u, manga), settingsOf(u, sess)) },
relations: catalog.state.relations.filter(r => r.state === 'confirmed' && view.members.some(m => m.id === r.from || m.id === r.to)) };
},
'GET /api/admin/catalog': ({ u }) => { admin(u); return { ...catalog.state, types: TYPES, roles: ROLES, mangas: lib.mangas.map(m => ({ id: m.id, title: m.title })) }; },
@@ -498,16 +567,16 @@ const routes = {
},
'GET /api/auth': () => ({ firstRun: !Object.keys(db.users).length, signup: OPEN_SIGNUP }),
'POST /api/signup': ({ b, res }) => {
'POST /api/signup': ({ req, b, res }) => {
const first = !Object.keys(db.users).length;
if (!first && !OPEN_SIGNUP) bad(403, 'Sign-up is closed. Ask an admin for an account.');
startSession(res, createUser(b.name, b.password, first));
startSession(req, res, createUser(b.name, b.password, first));
return { ok: true };
},
'POST /api/login': ({ b, res }) => {
'POST /api/login': ({ req, b, res }) => {
const user = db.users[String(b.name || '').trim().toLowerCase()], pw = String(b.password || '');
if (!checkPw(pw, user ? user.hash : DUMMY_HASH) || !user) bad(403, 'Wrong user name or password.');
startSession(res, user);
startSession(req, res, user);
return { ok: true };
},
'POST /api/logout': ({ req, res }) => {
@@ -516,7 +585,18 @@ const routes = {
res.setHeader('Set-Cookie', 'sid=; Path=/; HttpOnly; SameSite=Lax; Max-Age=0');
return { ok: true };
},
'GET /api/me': ({ u }) => ({ name: u.name, admin: !!u.admin, settings: settingsOf(u), unread: unreadOf(u) }),
// Every page loads /api/me, so a page view marks its session as the last active one for reader settings.
'GET /api/me': ({ u, sess, sk }) => {
if (sess && readerSessions.noteActive(u, userSettingsOf(u), sess, sk)) save();
return { name: u.name, admin: !!u.admin, settings: settingsOf(u, sess), unread: unreadOf(u) };
},
'GET /api/reader-sessions': ({ u, sk }) => readerSessions.overview(u, userSettingsOf(u), db.sessions, sk),
'POST /api/reader-sessions': ({ u, b, sk }) => {
if (!['follow', 'latest'].includes(b.action)) bad(400, 'Unknown action');
if (!readerSessions.act(u, userSettingsOf(u), db.sessions, String(b.id || ''), b.action)) bad(404, 'Session not found');
save();
return readerSessions.overview(u, userSettingsOf(u), db.sessions, sk);
},
'GET /api/reading-schedule': () => publicSchedule(readingSchedule),
'GET /api/reading-access': ({ u, q }) => { mangaOf(q.get('m')); return readingAccess.status(u.name, true); },
@@ -533,22 +613,25 @@ const routes = {
return { ok: true };
},
'POST /api/settings': ({ u, b }) => {
'POST /api/settings': ({ u, b, sess, sk }) => {
if (typeof b.webhook === 'string' && b.webhook.trim() && !/^https?:\/\/\S+$/i.test(b.webhook.trim())) bad(400, 'The webhook must be an http(s) URL.');
const s = (u.settings = u.settings || {});
const base = userSettingsOf(u), s = {};
if (['last', 'flow', 'page', 'book'].includes(b.startMode)) s.startMode = b.startMode;
if (['flow', 'page', 'book'].includes(b.lastMode)) s.lastMode = b.lastMode;
if (['original', 'inverted', 'sepia', 'dim', 'extra-dim'].includes(b.pageTone)) s.pageTone = b.pageTone;
if (['off', 'date', 'hour'].includes(b.readDate)) s.readDate = b.readDate;
if (Number.isInteger(b.rereadDays) && b.rereadDays >= 0 && b.rereadDays <= 3650) s.rereadDays = b.rereadDays;
if (Number.isInteger(b.rereadChapters) && b.rereadChapters >= 0 && b.rereadChapters <= 1000) s.rereadChapters = b.rereadChapters;
for (const k of ['adapt', 'skipExtras', 'skipCovers', 'flowPageGap', 'readingClock', 'readingStory', 'releaseDate', 'continueRow',
'discoveryEnabled', 'discoveryReadingActivity', 'discoveryMature']) if (typeof b[k] === 'boolean') s[k] = b[k];
'discoveryEnabled', 'discoveryReadingActivity', 'discoveryMature', 'readerPerSession']) if (typeof b[k] === 'boolean') s[k] = b[k];
if (b.zoom && typeof b.zoom === 'object') {
s.zoom = {};
for (const k of ['flow', 'page', 'book', 'strip']) if (+b.zoom[k] > 0) s.zoom[k] = Math.min(8, Math.max(0.1, +b.zoom[k]));
}
if (typeof b.webhook === 'string') s.webhook = b.webhook.trim().slice(0, 500);
readerSessions.applyPatch({ user: u, base, session: sess, key: sk, patch: s });
save();
return { settings: settingsOf(u) };
return { settings: settingsOf(u, sess) };
},
'POST /api/password': ({ u, b }) => {
if (!checkPw(String(b.old || ''), u.hash)) bad(403, 'The current password is wrong.');
@@ -558,7 +641,7 @@ const routes = {
'GET /api/library': ({ u }) => {
const prog = db.progress[u.name] || {}, fol = db.follows[u.name] || {}, notes = (db.notes[u.name] || []).filter(n => !n.read);
const showStory = settingsOf(u).readingStory;
const s = settingsOf(u), showStory = s.readingStory;
const mangas = lib.mangas.map(m => {
const p = prog[m.id] || {}, meta = m.meta || {}, release = firstReleaseOf(meta);
return {
@@ -568,7 +651,7 @@ const routes = {
tags: Array.isArray(meta.tags) ? meta.tags : (meta.tags ? [String(meta.tags)] : []), status: meta.status || '', year: meta.year || '',
firstRelease: release.label, firstReleaseAt: release.time,
read: m.chapters.filter(c => p[c.id] && p[c.id].done).length,
lastRead: Object.values(p).reduce((t, x) => Math.max(t, x.t || 0), 0),
lastRead: Object.values(p).reduce((t, x) => Math.max(t, x.t || 0), 0), left: leftOf(m, p, s),
following: !!fol[m.id], unread: notes.filter(n => n.m === m.id).length,
reading: showStory ? mangaInsight(mangaReadingOf(u, m), m.chapters.map(ch => ch.id)) : null,
};
@@ -676,8 +759,8 @@ const routes = {
return { results: await searchGreatestMangaDex(entry) };
},
'GET /api/manga': ({ u, q }) => {
const m = mangaOf(q.get('m')), prog = progressOf(u, m), s = settingsOf(u), showStory = s.readingStory;
'GET /api/manga': ({ u, q, sess }) => {
const m = mangaOf(q.get('m')), prog = progressOf(u, m), s = settingsOf(u, sess), showStory = s.readingStory;
const records = mangaReadingOf(u, m);
return {
chapterDates: { read: s.readDate, release: s.releaseDate },
@@ -694,7 +777,7 @@ const routes = {
dates: s.releaseDate ? chapterDates(ch.versions) : null,
})),
reading: showStory ? mangaInsight(records, m.chapters.map(ch => ch.id)) : null,
continue: continueOf(m, prog, settingsOf(u)),
continue: continueOf(m, prog, s),
};
},
@@ -735,21 +818,24 @@ const routes = {
return { ok: true, deleted: m.id };
},
'GET /api/chapter': ({ u, q }) => {
const m = mangaOf(q.get('m')), ch = chapterOf(m, q.get('c')), s = settingsOf(u), prog = progressOf(u, m);
'GET /api/chapter': ({ u, q, sess }) => {
const m = mangaOf(q.get('m')), ch = chapterOf(m, q.get('c')), s = settingsOf(u, sess), prog = progressOf(u, m);
if (!readingAccess.status(u.name).allowed) bad(423, 'Reading is outside the deployment schedule. Complete the five-question challenge to unlock it.');
const v = ch.versions.find(x => x.id === q.get('v')) || ranked(m, ch)[0].v;
// Opening a chapter clears its "new chapter" notification.
let seen = false;
for (const n of db.notes[u.name] || []) if (!n.read && n.m === m.id && n.c === ch.id) seen = n.read = true;
if (seen) save();
if (readAhead) warmChapter(u, m, ch, v, prog[ch.id], s);
return {
perf: tracer.on,
manga: { id: m.id, title: m.title },
chapter: { id: ch.id, label: ch.label, title: ch.title, extra: ch.extra },
version: v.id, versions: versionsOf(m, ch, u), direction: v.direction,
pages: v.pages.map(p => ({ u: url(v.rel + '/' + p.f), w: p.w, h: p.h, t: p.t, r: p.r })),
prev: neighbour(m, ch, -1, s), next: neighbour(m, ch, 1, s),
progress: prog[ch.id] || null,
latest: latestOf(m, prog, s),
reading: s.readingClock ? publicSummary(readingOf(u, m, ch), v.id, eligiblePages(v, s)) : null,
// the user's moments on this version, so the reader can mark pages that already have one
moments: momentsOf(u).filter(x => x.m === m.id && x.c === ch.id && x.v === v.id)
@@ -760,18 +846,30 @@ const routes = {
};
},
'GET /api/progress/latest': ({ u, q }) => {
const m = mangaOf(q.get('m'));
return { latest: latestOf(m, progressOf(u, m), settingsOf(u)) };
},
'POST /api/progress': ({ u, b }) => {
const m = mangaOf(b.m), ch = chapterOf(m, b.c);
const all = (db.progress[u.name] = db.progress[u.name] || {}), mp = (all[m.id] = all[m.id] || {}), old = mp[ch.id] || {};
const tab = /^[a-zA-Z0-9_-]{8,40}$/.test(String(b.tab || '')) ? String(b.tab) : '';
// A reader tab sends the head time it knows (`seen`). When another tab or device saved a newer
// position since, this tab is stale: it must not become the head until the reader chooses to stay.
if (Number.isFinite(b.seen)) {
const last = lastOf(m, mp);
if (last && last.p.t > b.seen && (!tab || last.p.w !== tab)) return { ok: false, stale: true, latest: latestOf(m, mp, settingsOf(u)) };
}
const now = Date.now(), done = !!(old.done || b.done);
// doneAt: when the chapter was first completed; kept until it is marked unread.
const doneAt = old.done ? old.doneAt : done ? now : undefined;
mp[ch.id] = {
v: String(b.v || ''), p: Math.max(0, b.p | 0), n: Math.max(0, b.n | 0), t: now, done, ...(doneAt ? { doneAt } : {}),
shift: Array.isArray(b.shift) ? b.shift.filter(Number.isInteger).slice(0, 1000) : old.shift || [],
...(tab ? { w: tab } : {}),
};
save();
return { ok: true };
return { ok: true, t: now };
},
// Separate endpoint: an older server on rollback cannot misread cumulative
// visit totals as legacy deltas. Old open tabs retain their original protocol.
@@ -875,6 +973,21 @@ const routes = {
return { ok: true };
},
// Timings the reader page measured (resource timing, decode, page-turn waits) while the trace is on.
'POST /api/perf': ({ u, b }) => {
if (!tracer.on) return { on: false };
const events = Array.isArray(b.events) ? b.events.slice(0, 300) : [];
for (const e of events) if (e && typeof e === 'object' && !Array.isArray(e)) {
const clean = {};
for (const [key, value] of Object.entries(e).slice(0, 30)) {
if (key === 't' || key === 'k') continue; // the server's clock and kind
if (typeof value === 'number' && Number.isFinite(value)) clean[key] = Math.round(value * 10) / 10;
else if (typeof value === 'string' || typeof value === 'boolean') clean[key] = typeof value === 'string' ? value.slice(0, 300) : value;
}
tracer.log({ ...clean, k: 'client', user: u.name, page: b.page ? String(b.page).slice(0, 40) : undefined });
}
return { on: true };
},
'GET /api/notifications': ({ u }) => ({ items: (db.notes[u.name] || []).slice().reverse() }),
'POST /api/notifications/read': ({ u, b }) => {
for (const n of db.notes[u.name] || []) if (b.all || n.id === b.id) n.read = true;
@@ -1060,23 +1173,32 @@ async function handleApi(req, res, u, route, q) {
if (!/^application\/json/.test(req.headers['content-type'] || '') || req.headers['sec-fetch-site'] === 'cross-site') bad(403, 'Bad request origin');
b = (await readBody(req)) || {};
}
sendJson(res, 200, await route({ req, res, u, b, q }));
const sk = u && sessionKeyOf(req), sess = sk && db.sessions[sk];
if (sess) touchSession(u, sess, sk, req);
sendJson(res, 200, await route({ req, res, u, b, q, sess, sk }));
}
function serveFile(req, res, root, rel, cacheControl, only) {
function serveFile(req, res, root, rel, cacheControl, only, trace) {
let file;
try { file = path.resolve(root, decodeURIComponent(rel)); } catch { return sendJson(res, 400, { error: 'Bad path' }); }
if (!file.startsWith(root + path.sep) || (only && !only.test(file))) return sendJson(res, 404, { error: 'Not found' });
fs.stat(file, (err, st) => {
if (trace) trace.mark('stat');
if (err || !st.isFile()) return sendJson(res, 404, { error: 'Not found' });
const etag = `"${st.size.toString(36)}-${Math.round(st.mtimeMs).toString(36)}"`;
if (req.headers['if-none-match'] === etag) { res.writeHead(304, { ETag: etag, 'Cache-Control': cacheControl }); return res.end(); }
res.writeHead(200, { 'Content-Type': MIME[path.extname(file).toLowerCase()] || 'application/octet-stream', 'Content-Length': st.size, ETag: etag, 'Cache-Control': cacheControl });
const headers = { 'Content-Type': MIME[path.extname(file).toLowerCase()] || 'application/octet-stream', 'Content-Length': st.size, ETag: etag, 'Cache-Control': cacheControl };
// The browser's resource timing shows this, so a slow page can be matched to the server's share.
if (trace && trace.on) headers['Server-Timing'] = `stat;dur=${trace.elapsed()}`;
res.writeHead(200, headers);
if (req.method === 'HEAD') return res.end();
fs.createReadStream(file).on('error', () => res.destroy()).pipe(res);
const stream = fs.createReadStream(file).on('error', () => res.destroy());
if (trace) trace.stream(stream);
stream.pipe(res);
});
}
http.createServer(async (req, res) => {
const trace = requestTrace(tracer, req, res);
try {
const { pathname, searchParams } = new URL(req.url, 'http://localhost');
const u = userOf(req);
@@ -1124,9 +1246,10 @@ http.createServer(async (req, res) => {
const parts = relative.split(path.sep);
if (parts.length >= 4 && !readingAccess.status(u.name).allowed)
bad(423, 'Reading is outside the deployment schedule.');
return serveFile(req, res, LIB, pathname.slice(5), readingSchedule && parts.length >= 4 ? 'private, no-store' : 'private, max-age=86400', IMG);
trace.mark('access');
return serveFile(req, res, LIB, pathname.slice(5), readingSchedule && parts.length >= 4 ? 'private, no-store' : 'private, max-age=86400', IMG, trace);
}
serveFile(req, res, PUB, pathname === '/' ? 'index.html' : pathname.slice(1), 'no-cache');
serveFile(req, res, PUB, pathname === '/' ? 'index.html' : pathname.slice(1), 'no-cache', null, trace);
} catch (e) {
const expected = e instanceof HttpError || e instanceof CatalogError || e instanceof moments.MomentError;
if (!expected) console.error(e);
@@ -1134,6 +1257,13 @@ http.createServer(async (req, res) => {
}
}).listen(PORT, () => console.log(`MangaCircus on http://localhost:${PORT} (library: ${LIB}, data: ${DATA})`));
// The trace's storage probe reads a random page of the library (rarely in the OS cache).
tracer.setProbeSource(() => {
const m = lib.mangas[Math.floor(Math.random() * lib.mangas.length)];
const ch = m && m.chapters[Math.floor(Math.random() * m.chapters.length)], v = ch && ch.versions[0];
const p = v && v.pages[Math.floor(Math.random() * v.pages.length)];
return p ? path.join(LIB, v.rel, p.f) : null;
});
if (lib.mangas.length) refreshChanged();
else rescan();
if (SCAN_MINUTES > 0) setInterval(() => refreshChanged(), SCAN_MINUTES * 60000).unref();
@@ -1161,10 +1291,13 @@ if (FULL_SCAN_MINUTES > 0) setInterval(() => {
// A completed scraper publish increments a durable revision. Polling it here keeps the
// reader index fresh without exposing an unauthenticated rescan endpoint to the daemon.
let scraperRevision = null;
let scraperRevision = null, revisionRoute = true;
async function pollScraper() {
try {
const state = await scraperRequest('/api/state');
// /api/revision is a few bytes; the full state is ~100 KB and makes the scraper summarise the library.
let state;
if (revisionRoute) state = await scraperRequest('/api/revision').catch(error => { if (error.code !== 404) throw error; revisionRoute = false; });
if (!revisionRoute) state = await scraperRequest('/api/state'); // a scraper from before /api/revision
const revision = Number(state.publishedRevision || 0);
if (scraperRevision === null || revision > scraperRevision) {
const checked = await refreshChanged();
+110
View File
@@ -0,0 +1,110 @@
'use strict';
// "Continue where you left off" only lists manga with something left after the last chapter read:
// unread chapters, or chapters last read in an earlier pass (a reread).
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('fs'), os = require('os'), path = require('path'), net = require('net');
const { spawn } = require('child_process');
const { Canvas } = require('../tools/png');
async function freePort() {
const listener = net.createServer();
await new Promise(resolve => listener.listen(0, '127.0.0.1', resolve));
const port = listener.address().port;
await new Promise(resolve => listener.close(resolve));
return port;
}
async function start(library, data) {
const port = await freePort(), output = [];
const child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '', GREATEST_COVER_WARM: '0' } });
child.stdout.on('data', d => output.push(String(d)));
child.stderr.on('data', d => output.push(String(d)));
const base = `http://127.0.0.1:${port}`;
let ready = false;
for (let n = 0; n < 100 && !ready; n++) {
ready = await fetch(base + '/health').then(r => r.ok).catch(() => false);
if (!ready) await new Promise(resolve => setTimeout(resolve, 50));
}
assert.ok(ready, output.join(''));
return { child, base };
}
async function stop(child) {
if (child && child.exitCode === null) { child.kill(); await new Promise(resolve => child.once('exit', resolve)); }
}
test('the continue row counts chapters left after the head, rereads included', { timeout: 30000 }, async () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-left-'));
let server;
try {
const library = path.join(root, 'library'), data = path.join(root, 'data');
for (const [title, count] of [['Moon Garden', 4], ['Sun Field', 4], ['Star Path', 8]])
for (let n = 1; n <= count; n++) {
const dir = path.join(library, title, 'v01', `c00${n}`);
fs.mkdirSync(dir, { recursive: true });
fs.writeFileSync(path.join(dir, '001.png'), new Canvas(60, 90, [40, 60, 120]).png());
}
fs.mkdirSync(data);
server = await start(library, data);
let { base } = server;
const post = (cookie, url, body) => fetch(base + url, { method: 'POST',
headers: { 'content-type': 'application/json', cookie }, body: JSON.stringify(body) }).then(r => r.json());
const get = (cookie, url) => fetch(base + url, { headers: { cookie } }).then(r => r.json());
const signup = await fetch(base + '/api/signup', { method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: 'alice', password: 'alice-password' }) });
const alice = signup.headers.get('set-cookie').split(';')[0];
await post(alice, '/api/admin/rescan', { full: true });
const ids = async m => (await get(alice, '/api/manga?m=' + encodeURIComponent(m))).chapters.map(c => ({ c: c.id, v: c.versions[0].id }));
const moon = await ids('Moon Garden'), sun = await ids('Sun Field');
const done = (m, ch) => post(alice, '/api/progress', { m, ...ch, p: 0, n: 1, done: true });
const left = async () => Object.fromEntries((await get(alice, '/api/library')).mangas.map(m => [m.id, m.left]));
// First read stopped after chapter 2: two chapters left. Finished: nothing left.
for (const ch of moon.slice(0, 2)) await done('Moon Garden', ch);
for (const ch of sun) await done('Sun Field', ch);
assert.deepEqual(await left(), { 'Moon Garden': 2, 'Sun Field': 0, 'Star Path': 0 });
// Read all 8 chapters of Star Path, then reread 1-6 right away: more than 5 in a row is a reread
// even within the time limit; with a longer run setting, or the run rule off, it is not.
const star = await ids('Star Path');
for (const ch of star) await done('Star Path', ch);
await new Promise(resolve => setTimeout(resolve, 5));
for (const ch of star.slice(0, 5)) await done('Star Path', ch);
assert.equal((await left())['Star Path'], 0, 'five in a row is not yet a reread');
await done('Star Path', star[5]);
assert.equal((await left())['Star Path'], 2);
await post(alice, '/api/settings', { rereadChapters: 6 });
assert.equal((await left())['Star Path'], 0);
await post(alice, '/api/settings', { rereadChapters: 0 });
assert.equal((await left())['Star Path'], 0);
await post(alice, '/api/settings', { rereadChapters: 5, rereadDays: -1 });
const settings = (await get(alice, '/api/me')).settings;
assert.equal(settings.rereadDays, 7, 'invalid values are ignored');
assert.equal((await left())['Star Path'], 2);
// Unfinished head: the rest of that chapter is still left.
await post(alice, '/api/progress', { m: 'Moon Garden', ...moon[2], p: 0, n: 1 });
assert.equal((await left())['Moon Garden'], 2);
// Glancing back at chapter 1 right after finishing is not a reread.
await done('Sun Field', sun[0]);
assert.equal((await left())['Sun Field'], 0);
// The finish was a month ago: rereading chapters 1-2 now leaves 3-4 from the earlier pass.
await new Promise(resolve => setTimeout(resolve, 700));
await stop(server.child);
const dbFile = path.join(data, 'db.json'), db = JSON.parse(fs.readFileSync(dbFile, 'utf8'));
const month = 30 * 24 * 3600 * 1000, sunProgress = db.progress.alice['Sun Field'];
for (const p of Object.values(sunProgress)) { p.t -= month; p.doneAt -= month; }
sunProgress[sun[0].c].t += month;
fs.writeFileSync(dbFile, JSON.stringify(db));
server = await start(library, data);
base = server.base;
await done('Sun Field', sun[1]);
assert.equal((await left())['Sun Field'], 2);
} finally {
await stop(server && server.child);
fs.rmSync(root, { recursive: true, force: true });
}
});
+8 -2
View File
@@ -3,6 +3,7 @@
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('fs'), path = require('path');
const root = path.resolve(__dirname, '..');
const allowedNames = () => new Set(fs.readFileSync(path.join(root, '.dockerignore'), 'utf8').split(/\r?\n/).filter(line => line.startsWith('!')).map(line => line.slice(1)));
test('every top-level module the reader requires is copied into the image', () => {
const dockerfile = fs.readFileSync(path.join(root, 'Dockerfile'), 'utf8');
@@ -12,14 +13,19 @@ test('every top-level module the reader requires is copied into the image', () =
const file = pending.pop();
if (seen.has(file)) continue;
seen.add(file);
for (const [, name] of fs.readFileSync(path.join(root, file), 'utf8').matchAll(/require\('\.\/([\w.-]+?)(?:\.js)?'\)/g)) {
const source = fs.readFileSync(path.join(root, file), 'utf8');
for (const [, dir] of source.matchAll(/require\('\.\/([\w.-]+)\/[^']+'\)/g)) {
assert.ok(copied.has(dir), `${dir}/ is not copied`);
assert.ok(allowedNames().has(dir + '/'), `${dir}/ is not allowed through .dockerignore`);
}
for (const [, name] of source.matchAll(/require\('\.\/([\w.-]+?)(?:\.js)?'\)/g)) {
// A folder (e.g. ./greatest) is copied as a whole; only single files need their own name in COPY.
if (fs.existsSync(path.join(root, name)) && fs.statSync(path.join(root, name)).isDirectory()) assert.ok(copied.has(name), `${name}/ is not copied`);
else pending.push(name + '.js');
}
}
// .dockerignore is an allowlist too: a file not let through never reaches COPY.
const allowed = new Set(fs.readFileSync(path.join(root, '.dockerignore'), 'utf8').split(/\r?\n/).filter(line => line.startsWith('!')).map(line => line.slice(1)));
const allowed = allowedNames();
for (const file of seen) {
assert.ok(copied.has(file), `${file} is required but not copied by the Dockerfile`);
assert.ok(allowed.has(file), `${file} is required but not allowed through .dockerignore`);
+2 -1
View File
@@ -143,7 +143,8 @@ test('moments are private, validated, editable and survive page renumbering as "
assert.equal((await post(alice, '/api/moments/delete', { id: spread.id })).status, 200);
assert.equal((await get(alice, '/api/moments?m=Moon%20Garden')).moments.length, 1);
} finally {
if (child) child.kill();
// The server writes db.json on SIGTERM; removing the folder before it exits races that write.
if (child && child.exitCode === null) await new Promise(resolve => { child.once('exit', resolve); child.kill(); });
fs.rmSync(root, { recursive: true, force: true });
}
});
+117
View File
@@ -0,0 +1,117 @@
'use strict';
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('fs'), os = require('os'), path = require('path'), http = require('http');
const { Tracer, requestTrace } = require('../perf/trace');
const { ReadAhead } = require('../perf/read-ahead');
const { analyse } = require('../tools/perf-report');
const tmp = () => fs.mkdtempSync(path.join(os.tmpdir(), 'circus-perf-'));
const lines = dir => fs.readdirSync(dir).filter(n => n.endsWith('.jsonl'))
.flatMap(n => fs.readFileSync(path.join(dir, n), 'utf8').trim().split('\n')).filter(Boolean).map(JSON.parse);
test('tracer is off without the flag file and writes nothing', () => {
const dir = tmp(), tracer = new Tracer({ dir, env: {} });
tracer.check(); tracer.log({ k: 'x' }); tracer.flush();
assert.equal(tracer.on, false);
assert.deepEqual(fs.readdirSync(dir), []);
});
test('the flag file switches the trace on and off at runtime', () => {
const dir = tmp(), tracer = new Tracer({ dir, env: {} });
fs.writeFileSync(path.join(dir, 'trace.on'), '');
tracer.check(); tracer.log({ k: 'x', a: 1 });
fs.rmSync(path.join(dir, 'trace.on'));
tracer.check(); tracer.log({ k: 'x', a: 2 }); tracer.flush();
const events = lines(dir);
assert.deepEqual(events.map(e => e.k), ['trace', 'x', 'trace']);
assert.equal(events[1].a, 1);
assert.equal(events[2].on, false);
assert.ok(events.every(e => typeof e.t === 'number'));
});
test('a served file is traced with storage and network phases', async () => {
const dir = tmp(), tracer = new Tracer({ dir, env: { PERF_TRACE: '1' } });
tracer.check();
const file = path.join(dir, 'page.jpg');
fs.writeFileSync(file, Buffer.alloc(3 * 1024 * 1024, 7));
const server = http.createServer((req, res) => {
const trace = requestTrace(tracer, req, res);
fs.stat(file, (err, st) => {
trace.mark('stat');
res.writeHead(200, { 'Content-Length': st.size });
const stream = fs.createReadStream(file);
trace.stream(stream);
stream.pipe(res);
});
});
await new Promise(r => server.listen(0, r));
const { port } = server.address();
const r = await fetch(`http://127.0.0.1:${port}/lib/M/B/C/page.jpg`);
assert.equal((await r.arrayBuffer()).byteLength, 3 * 1024 * 1024);
await new Promise(r => setTimeout(r, 50));
server.close(); tracer.stopSampler(); tracer.flush();
const req = lines(dir).find(e => e.k === 'req');
assert.equal(req.status, 200);
assert.equal(req.bytes, 3 * 1024 * 1024);
assert.equal(req.path, '/lib/M/B/C/page.jpg');
for (const field of ['stat', 'open', 'first', 'readEnd', 'disk', 'netWait', 'ms']) assert.equal(typeof req[field], 'number', field);
assert.ok(req.ms >= req.readEnd && req.readEnd >= req.first && req.first >= req.open);
});
test('read-ahead reads files in order and a newer chapter replaces the queued one', async () => {
const dir = tmp(), files = [];
for (let i = 0; i < 6; i++) { const f = path.join(dir, `p${i}.jpg`); fs.writeFileSync(f, Buffer.alloc(1000 + i)); files.push(f); }
const batches = [];
const ahead = new ReadAhead({ concurrency: 1, onBatch: b => batches.push(b) });
const done = new Promise(resolve => { const check = () => (batches.some(b => b.tag === 'B' && !b.replaced) ? resolve() : setTimeout(check, 5)); check(); });
ahead.warm(files.slice(0, 4), 'A');
ahead.warm(files.slice(3), 'B');
await done;
assert.equal(batches[0].tag, 'A'); assert.equal(batches[0].replaced, true);
const b = batches.find(x => x.tag === 'B');
assert.equal(b.done, 3); assert.equal(b.bytes, 1003 + 1004 + 1005);
// Files read recently are skipped the next time.
batches.length = 0;
ahead.warm(files.slice(4), 'C');
assert.deepEqual(batches.map(x => [x.tag, x.files, x.skipped]), [['C', 0, 2]]);
});
test('read-ahead keeps one queue per reader and serves them in turn', async () => {
const dir = tmp(), mk = n => { const f = path.join(dir, n); fs.writeFileSync(f, n); return f; };
const order = [], batches = [];
const ahead = new ReadAhead({ concurrency: 1, onBatch: b => batches.push(b) });
const read = ahead.readFile.bind(ahead);
ahead.readFile = f => { order.push(path.basename(f)); return read(f); };
ahead.warm([mk('a1'), mk('a2')], 'A', 'ann');
ahead.warm([mk('b1'), mk('b2')], 'B', 'bob');
await new Promise(r => { const check = () => (batches.length === 2 ? r() : setTimeout(check, 5)); check(); });
assert.deepEqual(order, ['a1', 'b1', 'a2', 'b2']);
assert.ok(batches.every(b => !b.replaced && b.done === 2));
});
test('read-ahead runs the follow-up batch after the current one', async () => {
const dir = tmp(), a = path.join(dir, 'a.jpg'), b = path.join(dir, 'b.jpg');
fs.writeFileSync(a, 'a'); fs.writeFileSync(b, 'bb');
const tags = [];
const ahead = new ReadAhead({ onBatch: x => tags.push(x.tag) });
ahead.warm([a], 'now'); ahead.then([b], 'next');
await new Promise(r => { const check = () => (tags.length === 2 ? r() : setTimeout(check, 5)); check(); });
assert.deepEqual(tags, ['now', 'next']);
});
test('the report blames the storage for slow first reads', () => {
const t = Date.parse('2026-10-29T20:00:00Z');
const events = [
{ t, k: 'req', m: 'GET', path: '/lib/M/B/C/1.jpg', status: 200, bytes: 300000, stat: 2, open: 3, first: 1900, readEnd: 1950, disk: 1947, netWait: 0, ms: 1955, conn: 1 },
{ t: t + 1000, k: 'req', m: 'GET', path: '/lib/M/B/C/2.jpg', status: 200, bytes: 300000, stat: 1, open: 2, first: 5, readEnd: 20, disk: 18, netWait: 0, ms: 22, conn: 2 },
{ t: t + 2000, k: 'client', e: 'res', url: '/lib/M/B/C/1.jpg', total: 2100, ttfb: 30, download: 2000, queue: 1, connect: 0, dns: 0 },
{ t: t + 3000, k: 'client', e: 'view', wait: 2050, preloaded: false },
{ t: t + 4000, k: 'nas', firstMs: 2500, ms: 2600, bytes: 1 },
];
const o = analyse(events, { slow: 500 });
assert.equal(o.slowServer.count, 1);
assert.deepEqual(o.slowServer.by, { firstRead: 1 });
assert.deepEqual(o.slowBrowser.by, { server: 1 });
assert.equal(o.pageTurns.over1s, 1);
assert.match(o.verdict.join('\n'), /first read from storage/);
});
+106
View File
@@ -0,0 +1,106 @@
'use strict';
// Reader settings per login session: a session follows the latest set until it changes one, then keeps its own.
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('fs'), os = require('os'), path = require('path'), net = require('net');
const { spawn } = require('child_process');
const readerSessions = require('../reader-sessions');
async function freePort() {
const listener = net.createServer();
await new Promise(resolve => listener.listen(0, '127.0.0.1', resolve));
const port = listener.address().port;
await new Promise(resolve => listener.close(resolve));
return port;
}
test('device labels name browser and system', () => {
assert.equal(readerSessions.deviceLabel('Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 Version/17.0 Mobile/15E148 Safari/604.1'), 'Safari on iPhone');
assert.equal(readerSessions.deviceLabel('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/130.0 Safari/537.36 Edg/130.0'), 'Edge on Windows');
assert.equal(readerSessions.deviceLabel(''), 'Unknown device');
});
test('sessions lock their reader settings on the first change, new ones copy the last active one', { timeout: 20000 }, async () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-reader-sessions-'));
let child;
try {
const library = path.join(root, 'library'), data = path.join(root, 'data');
fs.mkdirSync(library); fs.mkdirSync(data);
const port = await freePort(), output = [];
child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '',
GREATEST_COVER_WARM: '0' } });
child.stdout.on('data', d => output.push(String(d)));
child.stderr.on('data', d => output.push(String(d)));
const base = `http://127.0.0.1:${port}`;
let ready = false;
for (let n = 0; n < 100 && !ready; n++) {
ready = await fetch(base + '/health').then(r => r.ok).catch(() => false);
if (!ready) await new Promise(resolve => setTimeout(resolve, 50));
}
assert.ok(ready, output.join(''));
const json = async r => { const body = await r.json(); assert.equal(r.status, 200, JSON.stringify(body)); return body; };
const post = (cookie, url, body) => fetch(base + url, { method: 'POST',
headers: { 'content-type': 'application/json', 'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) Firefox/130.0', ...(cookie ? { cookie } : {}) },
body: JSON.stringify(body) }).then(async r => (url.endsWith('login') || url.endsWith('signup') ? r : json(r)));
const me = async cookie => (await json(await fetch(base + '/api/me', { headers: { cookie } }))).settings;
const overview = cookie => fetch(base + '/api/reader-sessions', { headers: { cookie } }).then(json);
const login = async () => (await post(null, '/api/login', { name: 'reader', password: 'reader-password' })).headers.get('set-cookie').split(';')[0];
const a = (await post(null, '/api/signup', { name: 'reader', password: 'reader-password' })).headers.get('set-cookie').split(';')[0];
// First session: defaults, following the latest set.
assert.equal((await me(a)).lastMode, 'page');
assert.equal((await overview(a)).sessions[0].locked, false);
// Session B exists before A changes anything: it follows A's change.
const b = await login();
await post(a, '/api/settings', { lastMode: 'flow', zoom: { flow: 1.5 } });
assert.equal((await me(b)).lastMode, 'flow');
assert.equal((await me(b)).zoom.flow, 1.5);
// B changes its own: B locks, A keeps its own set.
await post(b, '/api/settings', { pageTone: 'sepia' });
assert.equal((await me(b)).pageTone, 'sepia');
assert.equal((await me(b)).lastMode, 'flow');
assert.equal((await me(a)).pageTone, 'original');
// A non-reader setting is shared and does not lock anything.
const c = await login();
await post(c, '/api/settings', { continueRow: false });
assert.equal((await me(a)).continueRow, false);
assert.equal((await overview(c)).sessions.find(s => s.current).locked, false);
// C follows the last active session, which is A (it loaded /api/me last).
assert.equal((await me(c)).pageTone, 'original');
await me(b);
assert.equal((await me(c)).pageTone, 'sepia');
// Overview: three sessions, labels, intent.
const o = await overview(a);
assert.equal(o.enabled, true);
assert.equal(o.sessions.length, 3);
assert.equal(o.sessions[0].current, true);
assert.equal(o.sessions[0].device, 'Firefox on Linux');
assert.deepEqual(o.sessions.map(s => s.locked).sort(), [false, true, true]);
// "Follow latest again" drops A's lock; "Use for new sessions" makes a set the latest.
const idA = o.sessions[0].id, idB = o.sessions.find(s => !s.current && s.locked).id;
await post(a, '/api/reader-sessions', { id: idA, action: 'follow' });
assert.equal((await me(a)).pageTone, 'sepia');
assert.equal((await fetch(base + '/api/reader-sessions', { method: 'POST', headers: { 'content-type': 'application/json', cookie: a },
body: JSON.stringify({ id: 'nope', action: 'follow' }) })).status, 404);
assert.equal((await post(a, '/api/reader-sessions', { id: idB, action: 'latest' })).latestFrom, idB);
// Off: one shared set; a change applies everywhere and locks nothing. On again: kept locks return.
await post(b, '/api/settings', { readerPerSession: false });
await post(a, '/api/settings', { pageTone: 'dim' });
assert.equal((await me(b)).pageTone, 'dim');
assert.equal((await me(c)).pageTone, 'dim');
await post(a, '/api/settings', { readerPerSession: true });
assert.equal((await me(b)).pageTone, 'sepia');
assert.equal((await me(a)).pageTone, 'sepia');
} finally {
if (child && child.exitCode === null) { child.kill(); await new Promise(resolve => child.once('exit', resolve)); }
fs.rmSync(root, { recursive: true, force: true });
}
});
+97
View File
@@ -0,0 +1,97 @@
'use strict';
// Stale reading sessions: a reader tab left open on another device must not move "Continue" back.
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('fs'), os = require('os'), path = require('path'), net = require('net');
const { spawn } = require('child_process');
const { Canvas } = require('../tools/png');
async function freePort() {
const listener = net.createServer();
await new Promise(resolve => listener.listen(0, '127.0.0.1', resolve));
const port = listener.address().port;
await new Promise(resolve => listener.close(resolve));
return port;
}
test('a save from a tab that missed a newer position is refused until the reader stays', { timeout: 30000 }, async () => {
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-stale-'));
let child;
try {
const library = path.join(root, 'library'), data = path.join(root, 'data');
for (const ch of ['c001 One', 'c002 Two', 'c003 Three']) {
const dir = path.join(library, 'Moon Garden', 'v01', ch);
fs.mkdirSync(dir, { recursive: true });
for (const n of [1, 2, 3]) fs.writeFileSync(path.join(dir, `00${n}.png`), new Canvas(60, 90, [40 * n, 60, 120]).png());
}
fs.mkdirSync(data);
const port = await freePort(), output = [];
child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '', GREATEST_COVER_WARM: '0' } });
child.stdout.on('data', d => output.push(String(d)));
child.stderr.on('data', d => output.push(String(d)));
const base = `http://127.0.0.1:${port}`;
let ready = false;
for (let n = 0; n < 100 && !ready; n++) {
ready = await fetch(base + '/health').then(r => r.ok).catch(() => false);
if (!ready) await new Promise(resolve => setTimeout(resolve, 50));
}
assert.ok(ready, output.join(''));
const post = (cookie, url, body) => fetch(base + url, { method: 'POST',
headers: { 'content-type': 'application/json', cookie }, body: JSON.stringify(body) }).then(r => r.json());
const get = (cookie, url) => fetch(base + url, { headers: { cookie } }).then(r => r.json());
const signup = await fetch(base + '/api/signup', { method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: 'alice', password: 'alice-password' }) });
const alice = signup.headers.get('set-cookie').split(';')[0];
await post(alice, '/api/admin/rescan', { full: true });
const manga = await get(alice, '/api/manga?m=Moon%20Garden');
const [one, two, three] = manga.chapters.map(c => ({ c: c.id, v: c.versions[0].id }));
const save = (ch, extra) => post(alice, '/api/progress', { m: 'Moon Garden', ...ch, p: 1, n: 3, ...extra });
const latest = () => get(alice, '/api/progress/latest?m=Moon%20Garden').then(r => r.latest);
// Phone reads chapter 1, the tablet then reads on in chapter 3.
const first = await save(one, { tab: 'phone-tab-1', seen: 0 });
assert.equal(first.ok, true);
const phoneSeen = first.t;
await new Promise(resolve => setTimeout(resolve, 5));
const tablet = await save(three, { tab: 'tablet-tab-1', seen: phoneSeen });
assert.equal(tablet.ok, true);
assert.equal((await latest()).c, three.c);
// The phone's old tab wakes up and saves chapter 1 again: refused, the head stays on chapter 3.
const refused = await save(one, { tab: 'phone-tab-1', seen: phoneSeen, p: 2 });
assert.equal(refused.ok, false);
assert.equal(refused.stale, true);
assert.equal(refused.latest.c, three.c);
assert.equal(refused.latest.w, 'tablet-tab-1');
assert.equal(refused.latest.continue.c, three.c);
assert.match(refused.latest.continue.text, /Continue/);
const after = await get(alice, '/api/manga?m=Moon%20Garden');
assert.equal(after.continue.c, three.c);
assert.equal(after.continue.at, tablet.t);
assert.equal(after.chapters[0].progress.p, 1, 'the stale save changed nothing');
// The chapter view tells a reader which head it is compared with.
const view = await get(alice, `/api/chapter?m=Moon%20Garden&c=${encodeURIComponent(one.c)}`);
assert.equal(view.latest.c, three.c);
assert.equal(view.latest.t, tablet.t);
// The tablet's own saves never conflict with themselves, even with an older `seen` still in flight.
assert.equal((await save(three, { tab: 'tablet-tab-1', seen: phoneSeen, p: 2 })).ok, true);
// "Keep reading here": the phone now knows the newest time and its save becomes the head.
const head = await latest();
const stayed = await save(one, { tab: 'phone-tab-1', seen: head.t, p: 2 });
assert.equal(stayed.ok, true);
assert.equal((await latest()).c, one.c);
assert.equal((await get(alice, '/api/manga?m=Moon%20Garden')).continue.c, one.c);
// Saves without `seen` (mark as read on the manga page, older clients) behave as before.
assert.equal((await save(two, { done: true })).ok, true);
assert.equal((await latest()).c, two.c);
} finally {
if (child && child.exitCode === null) { child.kill(); await new Promise(resolve => child.once('exit', resolve)); }
fs.rmSync(root, { recursive: true, force: true });
}
});
+110
View File
@@ -0,0 +1,110 @@
#!/usr/bin/env node
'use strict';
/*
* Load test for the reader: N simulated readers each open random chapters and fetch the pages one after
* another with a think time, the way a person paging through a chapter does, plus a small look-ahead.
* Measures time to first byte and total per page from this machine. Read-only apart from the login; it
* never writes progress. Run it with the server's trace on to see the server's share (docs/performance.md).
*
* node tools/perf-bench.js --url http://host:8080 --user bench --password-env BENCH_PASSWORD \
* [--readers 4] [--chapters 3] [--pages 40] [--think 400] [--ahead 2] [--manga Berserk] [--json]
* --cookie 'sid=…' instead of user/password. --chapter-only skips the page fetches (API timings only).
*/
function parseArgs(argv) {
const o = { url: 'http://localhost:8080', readers: 4, chapters: 3, pages: 40, think: 400, ahead: 2, manga: '', json: false };
for (let i = 0; i < argv.length; i++) {
const key = argv[i].replace(/^--/, ''), next = () => argv[++i];
if (['readers', 'chapters', 'pages', 'think', 'ahead'].includes(key)) o[key] = +next();
else if (key === 'password-env') o.password = process.env[next()] || '';
else if (['url', 'user', 'cookie', 'manga'].includes(key)) o[key] = next();
else if (key === 'json' || key === 'chapter-only') o[key.replace('-o', 'O')] = true;
else if (key === 'h' || key === 'help') o.help = true;
}
o.url = o.url.replace(/\/$/, '');
return o;
}
async function login(o) {
if (o.cookie) return o.cookie;
const r = await fetch(o.url + '/api/login', { method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ name: o.user, password: o.password }) });
if (!r.ok) throw new Error(`login failed: HTTP ${r.status} ${await r.text()}`);
const cookie = (r.headers.get('set-cookie') || '').split(';')[0];
if (!cookie) throw new Error('login returned no session cookie');
return cookie;
}
async function timed(url, cookie) {
const t0 = performance.now();
const r = await fetch(url, { headers: { cookie } });
const ttfb = performance.now() - t0;
let bytes = 0;
for await (const chunk of r.body) bytes += chunk.length;
return { status: r.status, ttfb, ms: performance.now() - t0, bytes, server: r.headers.get('server-timing') || '' };
}
const json = async (o, cookie, path) => {
const t0 = performance.now(), r = await fetch(o.url + path, { headers: { cookie } });
if (!r.ok) throw new Error(`${path}: HTTP ${r.status}`);
return { data: await r.json(), ms: performance.now() - t0 };
};
const pick = list => list[Math.floor(Math.random() * list.length)];
const sleep = ms => new Promise(r => setTimeout(r, ms));
async function reader(o, cookie, mangas, out, id) {
for (let c = 0; c < o.chapters; c++) {
const m = pick(mangas);
const manga = await json(o, cookie, '/api/manga?m=' + encodeURIComponent(m.id));
out.api.push({ path: '/api/manga', ms: manga.ms });
const ch = pick(manga.data.chapters);
const chapter = await json(o, cookie, `/api/chapter?m=${encodeURIComponent(m.id)}&c=${encodeURIComponent(ch.id)}`);
out.api.push({ path: '/api/chapter', ms: chapter.ms });
if (o.chapterOnly) continue;
const pages = chapter.data.pages.slice(0, o.pages);
// The browser keeps `ahead` pages in flight while the reader looks at the current one.
const inflight = new Map();
const start = k => { if (k < pages.length && !inflight.has(k)) inflight.set(k, timed(o.url + '/' + pages[k].u, cookie)); };
for (let k = 0; k < pages.length; k++) {
const waitFrom = performance.now();
start(k);
for (let a = 1; a <= o.ahead; a++) start(k + a);
const r = await inflight.get(k);
out.pages.push({ reader: id, manga: m.id, chapter: ch.id, page: k, wait: performance.now() - waitFrom, ...r });
await sleep(o.think * (0.5 + Math.random()));
}
await Promise.all(inflight.values());
}
}
const pct = (v, p) => { const s = v.slice().sort((a, b) => a - b); return s.length ? Math.round(s[Math.min(s.length - 1, Math.floor(p / 100 * s.length))]) : null; };
const sum = v => ({ n: v.length, p50: pct(v, 50), p90: pct(v, 90), p99: pct(v, 99), max: v.length ? Math.round(Math.max(...v)) : null });
async function main() {
const o = parseArgs(process.argv.slice(2));
if (o.help || (!o.cookie && !o.user)) {
console.log('usage: node tools/perf-bench.js --url URL (--user NAME --password-env VAR | --cookie sid=…) [--readers N] [--chapters N] [--pages N] [--think ms] [--ahead N] [--manga text] [--chapter-only] [--json]');
process.exit(o.help ? 0 : 2);
}
const cookie = await login(o);
let mangas = (await json(o, cookie, '/api/library')).data.mangas;
if (o.manga) mangas = mangas.filter(m => m.title.toLowerCase().includes(o.manga.toLowerCase()));
if (!mangas.length) throw new Error('no manga to read');
const out = { pages: [], api: [] }, t0 = Date.now();
await Promise.all(Array.from({ length: o.readers }, (_, i) => reader(o, cookie, mangas, out, i)));
const ok = out.pages.filter(p => p.status === 200), mb = ok.reduce((a, p) => a + p.bytes, 0) / 1048576;
const result = {
readers: o.readers, seconds: Math.round((Date.now() - t0) / 1000), pages: out.pages.length, errors: out.pages.length - ok.length,
MB: Math.round(mb), wait: sum(ok.map(p => p.wait)), ttfb: sum(ok.map(p => p.ttfb)), total: sum(ok.map(p => p.ms)),
waitedOver300: ok.filter(p => p.wait >= 300).length, waitedOver1s: ok.filter(p => p.wait >= 1000).length,
api: Object.fromEntries([...new Set(out.api.map(a => a.path))].map(p => [p, sum(out.api.filter(a => a.path === p).map(a => a.ms))])),
worst: ok.sort((a, b) => b.wait - a.wait).slice(0, 8).map(p => ({ wait: Math.round(p.wait), ttfb: Math.round(p.ttfb), ms: Math.round(p.ms), kb: Math.round(p.bytes / 1024), at: `${p.manga} ${p.chapter} p${p.page}` })),
};
if (o.json) return console.log(JSON.stringify(result, null, 2));
console.log(`${result.readers} readers, ${result.pages} pages (${result.MB} MB) in ${result.seconds} s, ${result.errors} errors`);
for (const k of ['wait', 'ttfb', 'total']) console.log(` ${k.padEnd(6)} ms ${JSON.stringify(result[k])}`);
console.log(` page waits >= 300 ms: ${result.waitedOver300}, >= 1 s: ${result.waitedOver1s} (wait = time the reader looked at an empty page)`);
for (const [p, s] of Object.entries(result.api)) console.log(` ${p} ${JSON.stringify(s)}`);
for (const w of result.worst) console.log(` wait ${w.wait} ms (ttfb ${w.ttfb}, total ${w.ms}, ${w.kb} KB) ${w.at}`);
}
main().catch(e => { console.error(e.message); process.exit(1); });
+197
View File
@@ -0,0 +1,197 @@
#!/usr/bin/env node
'use strict';
/*
* Summarise a performance trace (docs/performance.md) and say where slow page loads spend their time.
*
* node tools/perf-report.js DATA/perf all trace files in the folder
* node tools/perf-report.js trace-2026-10-29.jsonl --since 2026-10-29T18:00 --until … --slow 500 --json
*
* Plain text by default (readable by people and agents); --json prints the numbers only.
*/
const fs = require('fs'), path = require('path');
function parseArgs(argv) {
const opts = { files: [], since: null, until: null, slow: 500, json: false };
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if (a === '--since') opts.since = Date.parse(argv[++i]);
else if (a === '--until') opts.until = Date.parse(argv[++i]);
else if (a === '--slow') opts.slow = +argv[++i];
else if (a === '--json') opts.json = true;
else if (a === '-h' || a === '--help') opts.help = true;
else opts.files.push(a);
}
return opts;
}
function readEvents(inputs, { since, until }) {
const files = inputs.flatMap(p => fs.statSync(p).isDirectory()
? fs.readdirSync(p).filter(n => /^trace-.*\.jsonl$/.test(n)).sort().map(n => path.join(p, n)) : [p]);
const events = [];
for (const file of files) for (const line of fs.readFileSync(file, 'utf8').split('\n')) {
if (!line) continue;
let e; try { e = JSON.parse(line); } catch { continue; }
if ((since && e.t < since) || (until && e.t > until)) continue;
events.push(e);
}
return events.sort((a, b) => a.t - b.t);
}
const pct = (values, p) => {
const v = values.filter(x => typeof x === 'number').sort((a, b) => a - b);
if (!v.length) return null;
return v[Math.min(v.length - 1, Math.floor((p / 100) * v.length))];
};
const stats = values => {
const v = values.filter(x => typeof x === 'number');
return v.length ? { n: v.length, p50: pct(v, 50), p90: pct(v, 90), p99: pct(v, 99), max: Math.max(...v) } : { n: 0 };
};
const isPageImage = e => e.k === 'req' && e.path.startsWith('/lib/') && e.path.split('/').length >= 6;
// Where one served image spent its time on the server.
function serverPhases(e) {
const stat = e.stat ?? 0, open = e.open != null ? e.open - stat : 0, firstRead = e.first != null && e.open != null ? e.first - e.open : 0;
const rest = Math.max(0, (e.disk ?? 0) - firstRead), net = e.netWait ?? 0;
const tail = e.readEnd != null ? Math.max(0, e.ms - e.readEnd) : 0;
const r = v => Math.round(v * 10) / 10;
return { stat: r(stat), open: r(open), firstRead: r(firstRead), readRest: r(rest), netWait: r(net), tail: r(tail) };
}
const dominant = phases => Object.entries(phases).sort((a, b) => b[1] - a[1])[0][0];
function analyse(events, { slow }) {
const out = { window: events.length ? { from: new Date(events[0].t).toISOString(), to: new Date(events.at(-1).t).toISOString() } : null };
const reqs = events.filter(e => e.k === 'req');
const imgs = reqs.filter(isPageImage), served = imgs.filter(e => e.status === 200 && !e.aborted);
out.requests = { total: reqs.length, pageImages: imgs.length, served: served.length,
notModified: imgs.filter(e => e.status === 304).length, aborted: imgs.filter(e => e.aborted).length,
errors: imgs.filter(e => e.status >= 400).length, keepAliveReuse: imgs.length ? round(imgs.filter(e => e.conn > 1).length / imgs.length) : null };
out.serverImage = { total: stats(served.map(e => e.ms)), stat: stats(served.map(e => e.stat)),
firstRead: stats(served.map(e => serverPhases(e).firstRead)), disk: stats(served.map(e => e.disk)),
netWait: stats(served.map(e => e.netWait)), kbytes: stats(served.map(e => e.bytes / 1024)) };
const slowServed = served.filter(e => e.ms >= slow);
out.slowServer = { count: slowServed.length, share: served.length ? round(slowServed.length / served.length) : null, by: {} };
for (const e of slowServed) { const d = dominant(serverPhases(e)); out.slowServer.by[d] = (out.slowServer.by[d] || 0) + 1; }
out.slowServer.worst = slowServed.sort((a, b) => b.ms - a.ms).slice(0, 10)
.map(e => ({ at: new Date(e.t).toISOString(), ms: e.ms, ...serverPhases(e), path: e.path }));
const api = {};
for (const e of reqs.filter(e => e.path.startsWith('/api/'))) (api[e.path] ||= []).push(e.ms);
out.api = Object.fromEntries(Object.entries(api).map(([p, v]) => [p, stats(v)]).sort((a, b) => (b[1].p99 || 0) - (a[1].p99 || 0)).slice(0, 12));
// Browser view, matched to the server's record of the same file where possible.
const res = events.filter(e => e.k === 'client' && e.e === 'res'), byPath = new Map();
const decoded = p => { try { return decodeURIComponent(p); } catch { return p; } };
for (const e of imgs) { const key = decoded(e.path); (byPath.get(key) || byPath.set(key, []).get(key)).push(e); }
const matched = res.map(r => {
const cands = (byPath.get(r.url) || []).filter(s => s.t <= r.t + 1000 && s.t >= r.t - 120000);
const s = cands.at(-1);
return { ...r, serverMs: s ? s.ms : null };
});
const loaded = matched.filter(r => !r.cached);
out.browserImage = { total: stats(loaded.map(r => r.total)), queue: stats(loaded.map(r => r.queue)),
ttfb: stats(loaded.map(r => r.ttfb)), download: stats(loaded.map(r => r.download)),
connect: stats(loaded.map(r => r.connect)), dns: stats(loaded.map(r => r.dns)), fromCache: matched.length - loaded.length };
const slowBrowser = loaded.filter(r => r.total >= slow);
out.slowBrowser = { count: slowBrowser.length, share: loaded.length ? round(slowBrowser.length / loaded.length) : null, by: {} };
for (const r of slowBrowser) {
const parts = { server: r.serverMs ?? 0, network: Math.max(0, (r.ttfb ?? 0) + (r.download ?? 0) - (r.serverMs ?? 0)),
queue: r.queue ?? 0, connect: (r.connect ?? 0) + (r.dns ?? 0) };
const d = r.serverMs == null ? 'unmatched' : dominant(parts);
out.slowBrowser.by[d] = (out.slowBrowser.by[d] || 0) + 1;
}
const views = events.filter(e => e.k === 'client' && e.e === 'view'), stalls = events.filter(e => e.k === 'client' && e.e === 'stall');
out.pageTurns = { count: views.length, preloaded: views.filter(v => v.preloaded).length, wait: stats(views.map(v => v.wait)),
over300ms: views.filter(v => v.wait >= 300).length, over1s: views.filter(v => v.wait >= 1000).length };
out.flowStalls = { count: stalls.length, wait: stats(stalls.map(s => s.wait)), over1s: stalls.filter(s => s.wait >= 1000).length };
out.clients = [...new Set(events.filter(e => e.k === 'client' && e.e === 'hello').map(e => `${e.user}: ${e.ua}`))].slice(0, 10);
// Storage probe: cold reads of random pages; a long read after quiet time points at disks waking up.
const nas = events.filter(e => e.k === 'nas' && !e.error);
// Library reads that finished before a probe started (a probe's own read does not count).
const ioEnds = events.filter(e => isPageImage(e) || e.k === 'nas' || e.k === 'ahead').map(e => e.t);
out.nas = { probes: nas.length, errors: events.filter(e => e.k === 'nas' && e.error).length, first: stats(nas.map(e => e.firstMs)), total: stats(nas.map(e => e.ms)) };
out.nas.slow = nas.filter(e => e.firstMs >= 1000).map(e => {
const start = e.t - e.ms, before = ioEnds.filter(t => t < start).at(-1);
return { at: new Date(e.t).toISOString(), firstMs: e.firstMs, quietMinBefore: before ? round((start - before) / 60000) : null };
}).slice(0, 30);
const loops = events.filter(e => e.k === 'loop');
out.eventLoop = { samples: loops.length, lagP99: stats(loops.map(e => e.lagP99)), lagMax: stats(loops.map(e => e.lagMax)), cpu: stats(loops.map(e => e.cpu)), rssMB: stats(loops.map(e => e.rss)) };
const scans = events.filter(e => e.k === 'scan');
out.scans = { count: scans.length, ms: stats(scans.map(e => e.ms)) };
const during = e => scans.some(s => e.t >= s.t - s.ms && e.t - e.ms <= s.t);
out.scans.slowImagesDuringScan = slowServed.filter(during).length;
const span = events.length ? events.at(-1).t - events[0].t : 0;
out.scans.timeShare = span ? round(scans.reduce((a, s) => a + s.ms, 0) / span) : null;
const ahead = events.filter(e => e.k === 'ahead');
out.readAhead = { batches: ahead.length, replaced: ahead.filter(e => e.replaced).length, files: ahead.reduce((a, e) => a + e.done, 0),
MBps: stats(ahead.filter(e => e.ms > 0 && e.bytes > 0).map(e => round(e.bytes / 1048.576 / e.ms))) };
const scraper = events.filter(e => e.k === 'scraper');
out.scraper = { calls: scraper.length, errors: scraper.filter(e => e.error).length, ms: stats(scraper.map(e => e.ms)) };
out.verdict = verdict(out, slow);
return out;
}
function verdict(o, slow) {
const lines = [];
const top = by => Object.entries(by).sort((a, b) => b[1] - a[1]);
if (o.slowServer.count) {
const [[phase, n]] = top(o.slowServer.by);
const name = { stat: 'the file lookup (stat) on the storage', open: 'opening the file', firstRead: 'the first read from storage',
readRest: 'reading the rest of the file from storage', netWait: 'the client receiving data (network)', tail: 'flushing to the client' }[phase];
lines.push(`${o.slowServer.count} of ${o.serverImage.total.n} served pages took >= ${slow} ms on the server; ${n} of them mostly in ${name}.`);
} else if (o.serverImage.total.n) lines.push(`No served page took >= ${slow} ms on the server (p99 ${o.serverImage.total.p99} ms).`);
if (o.slowBrowser.count) {
const [[part, n]] = top(o.slowBrowser.by);
lines.push(`${o.slowBrowser.count} of ${o.browserImage.total.n} page loads took >= ${slow} ms in the browser; mostly ${part} (${n}).`);
}
if (o.pageTurns.count) lines.push(`Page turns: ${o.pageTurns.preloaded}/${o.pageTurns.count} preloaded, ${o.pageTurns.over300ms} waited >= 300 ms, ${o.pageTurns.over1s} >= 1 s (p90 ${o.pageTurns.wait.p90 ?? 0} ms).`);
if (o.flowStalls.count) lines.push(`Flow mode: ${o.flowStalls.count} pages were on screen before their image (p90 ${o.flowStalls.wait.p90} ms, ${o.flowStalls.over1s} >= 1 s).`);
if (o.nas.probes) {
lines.push(`Storage probe (cold page): first byte p50 ${o.nas.first.p50} ms, p99 ${o.nas.first.p99} ms, max ${o.nas.first.max} ms; ${o.nas.slow.length} probes >= 1 s.`);
const sleepy = o.nas.slow.filter(s => s.quietMinBefore >= 5).length;
if (sleepy) lines.push(`${sleepy} slow probes followed >= 5 min without library reads: disks waking from standby is likely.`);
}
if (o.eventLoop.samples && (o.eventLoop.lagMax.max || 0) >= 200) lines.push(`Event loop blocked up to ${o.eventLoop.lagMax.max} ms (reader CPU work holds requests back).`);
if (o.scans.slowImagesDuringScan) lines.push(`${o.scans.slowImagesDuringScan} slow pages overlapped a library scan (scans ran ${Math.round((o.scans.timeShare || 0) * 100)}% of the time).`);
return lines;
}
function print(o) {
const s = x => (x && x.n ? `n=${x.n} p50=${x.p50} p90=${x.p90} p99=${x.p99} max=${x.max}` : 'n=0');
const out = [];
out.push(`Trace ${o.window ? `${o.window.from} .. ${o.window.to}` : '(empty)'}`, '', 'Verdict:', ...o.verdict.map(l => ' - ' + l), '');
out.push('Requests', ` ${JSON.stringify(o.requests)}`);
out.push('Server, page images (ms)');
for (const k of ['total', 'stat', 'firstRead', 'disk', 'netWait', 'kbytes']) out.push(` ${k.padEnd(10)} ${s(o.serverImage[k])}`);
out.push(` slow by phase ${JSON.stringify(o.slowServer.by)}`);
for (const w of o.slowServer.worst) out.push(` ${w.at} ${w.ms} ms stat=${w.stat} first=${w.firstRead} rest=${w.readRest} net=${w.netWait} ${w.path}`);
out.push('Browser, page images (ms)');
for (const k of ['total', 'queue', 'ttfb', 'download', 'connect', 'dns']) out.push(` ${k.padEnd(10)} ${s(o.browserImage[k])}`);
out.push(` from browser cache ${o.browserImage.fromCache}; slow by part ${JSON.stringify(o.slowBrowser.by)}`);
out.push(`Page turns ${o.pageTurns.count} (${o.pageTurns.preloaded} preloaded) wait ${s(o.pageTurns.wait)}`);
out.push(`Flow stalls ${o.flowStalls.count} wait ${s(o.flowStalls.wait)}`);
out.push(`Storage probe first byte ${s(o.nas.first)} total ${s(o.nas.total)} errors ${o.nas.errors}`);
for (const p of o.nas.slow) out.push(` ${p.at} first=${p.firstMs} ms after ${p.quietMinBefore ?? '?'} min quiet`);
out.push(`Event loop lagP99 ${s(o.eventLoop.lagP99)} lagMax ${s(o.eventLoop.lagMax)} cpu% ${s(o.eventLoop.cpu)} rssMB ${s(o.eventLoop.rssMB)}`);
out.push(`Scans ${o.scans.count} ${s(o.scans.ms)}`);
out.push(`Read-ahead ${JSON.stringify(o.readAhead)}`);
out.push(`Scraper calls ${o.scraper.calls} errors ${o.scraper.errors} ${s(o.scraper.ms)}`);
out.push('API (ms, slowest p99 first)');
for (const [p, v] of Object.entries(o.api)) out.push(` ${p.padEnd(28)} ${s(v)}`);
if (o.clients.length) out.push('Clients', ...o.clients.map(c => ' ' + c));
return out.join('\n');
}
const round = v => Math.round(v * 100) / 100;
if (require.main === module) {
const opts = parseArgs(process.argv.slice(2));
if (opts.help || !opts.files.length) { console.log('usage: node tools/perf-report.js <trace.jsonl|folder>... [--since ISO] [--until ISO] [--slow ms] [--json]'); process.exit(opts.help ? 0 : 2); }
const result = analyse(readEvents(opts.files, opts), opts);
console.log(opts.json ? JSON.stringify(result, null, 2) : print(result));
}
module.exports = { analyse, readEvents, print };