Compare commits

...
Author SHA1 Message Date
MaashkinoozheandClaude Opus 5.5 41e7ed063e Continue row: first line of the content, only on the plain overview
The row now sits below the search, filters and tags, directly above the
grid. It hides while a search is typed or a status filter is set (as well
as on "Recently read"); tags and collection grouping keep it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 12:08:35 +01:00
MaashkinoozheandClaude Opus 5.5 1ddefaa7ac Library: "Continue where you left off" row with a per-user setting
One fixed row above the library showing the five most recently read,
unfinished manga with the normal tiles (they stay in the grid as well).
It ignores search and filters, hides while sorted by "Recently read",
and can be turned off under Settings > Reading > Library page
(continueRow, default on).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-27 12:02:16 +01:00
MaashkinoozheandClaude Opus 5.5 e9da27e688 Greatest: serve catalog covers from a local copy (BL-0053)
The page hotlinked ~100 AniList CDN covers per load and some failed at random.
The reader now downloads each catalog cover once (sequential, ~1/s, stops on
401/403/429, images up to 2 MB) into data/greatest-covers/ and serves it from
/api/greatest-cover with a year-long browser cache; until a copy exists the
route redirects to the catalog URL. The page also accepts the server's
relative cover URLs, so owned titles show their library cover again.
GREATEST_COVER_WARM=0 turns the downloads off (tests, offline installs).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-26 00:38:55 +01:00
20 changed files with 324 additions and 25 deletions

No files matched your search

+1
View File
@@ -142,6 +142,7 @@ completeness.
- **Flow page gap:** Settings → Reading lets each user keep the normal separator between Flow pages or remove it for continuous webtoon artwork and text that crosses image edges.
- **Page tone:** the reader's **Tone** button cycles Original → Inverted → Sepia → Dim → Extra dim in Flow, Page and Book. It changes only the artwork, never the controls or site theme. The choice is saved per user; it can also be selected in Settings → User → Reading.
- **Active Reading Clock:** opt in under Settings → User → Reading time & privacy. No scrolling-speed limit; a visit counts at 60% viewed-page coverage and saves on exit. Pauses in the background or after 30 seconds without reader activity. Tap the clock for this visit's time, coverage, qualification and loading wait.
- **Continue where you left off:** the first line of the library content (below the search, filters and tags) is one row of the five manga you read most recently and have not finished (same tiles as the grid; they stay in the grid too). It hides while a search is typed, a status filter is set, or the library is sorted by *Recently read*; tags and collection grouping do not affect it. Turn it off under Settings → User → Reading → *Library page*.
- **Your Reading Story:** private active-time summaries appear on the library and manga pages when history exists. Chapter rows show the last timed read (60%-coverage visit), rereads remain visible as lifetime time, and Settings lets each user hide, export, or delete their own history. No leaderboards or speed ranking are used.
- **Tap / click zones:**
- Tapping **beside the pages** always shows or hides the controls.
+1
View File
@@ -60,6 +60,7 @@ services:
SCRAPER_TOKEN_FILE: /run/secrets/scraper_token
DISCOVERY_HARVEST_AT: ${DISCOVERY_HARVEST_AT:-03:30}
DISCOVERY_ANILIST_URL: ${DISCOVERY_ANILIST_URL:-}
GREATEST_COVER_WARM: ${GREATEST_COVER_WARM:-1}
networks: [backend, egress]
healthcheck:
test: [CMD, node, tools/healthcheck.js, reader]
+2
View File
@@ -128,6 +128,7 @@ This keeps ids human-readable and derivable without a database. The price: renam
"flowPageGap": true, // show the 8 px separator between pages in Flow
"readingClock": false, // opt in to credible per-chapter active-time tracking
"readingStory": true, // show private retrospective summaries when history exists
"continueRow": true, // library: "Continue where you left off" row at the top
"zoom": { "flow": 1, "page": 1.5, "book": 1, "strip": 0.8 }, // page zoom per mode
"webhook": ""
}
@@ -207,6 +208,7 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
| `POST /api/admin/user/role` `{name, admin}` | admin | Grant or revoke admin rights; keeps the password and sessions. Revoking your own rights is refused, so one admin always remains |
| `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data |
| `GET /api/admin/backup` · `POST /api/admin/backup` `{command:"settings"\|"run"}` | admin | Built-in backup status, schedule/retention, "back up now" ([backup.md](backup.md)) |
| `GET /api/greatest-cover?id=<entry>&v=<version>` | user | The Greatest entry's cover from `data/greatest-covers/` (`greatest/covers.js`), cached by the browser for a year; `v` changes with the catalog URL. A `302` to the catalog URL until the copy exists. `/api/greatest` lists this URL as `cover` for titles not in the library |
| `GET /api/discovery` | user | The caller's discovery feed, settings, feedback and `refresh: { state: idle\|queued\|running, position?, nextAllowedAt?, blockedReason? }` (§6.6) |
| `POST /api/discovery/feedback` `{candidateId, value}` | user | The caller's choice for a candidate; requires opt-in |
| `POST /api/discovery/refresh` | user | Queue a per-user AniList harvest for the caller (opt-in required, else `403`). `{ harvestId, position }`; `409` while one is queued or running for the caller; `429 { nextAllowedAt }` within an hour of the last accepted refresh; `503 { blockedReason }` after AniList pushed back, until a nightly or admin harvest |
+2 -1
View File
@@ -21,7 +21,7 @@ Data on the host, all bind mounts (no named volumes):
| Path (`.env`) | Contents | Back up? |
|---|---|---|
| `LIBRARY_PATH` | The manga: `Manga/Book/Chapter/images` ([library-format.md](library-format.md)) | Yes, it is your collection |
| `DATA_PATH` | `db.json` (users, progress, votes), catalog, discovery feeds, library index cache | Yes (built-in backup, section 8). `discovery-cache/` inside it is safe to delete and may be left out of backups: the next harvest rebuilds it |
| `DATA_PATH` | `db.json` (users, progress, votes), catalog, discovery feeds, library index cache | Yes (built-in backup, section 8). `discovery-cache/` inside it is safe to delete and may be left out of backups: the next harvest rebuilds it. The same holds for `greatest-covers/` (local copies of the Greatest catalog's cover images, fetched again after the next start) |
| `SCRAPER_DATA_PATH` | Queue, settings, source ledger, metadata cache | Included in the built-in backup |
| `BACKUP_PATH` | Built-in backup archives | Keep it on another disk or NAS |
| `SCRAPER_TOKEN_PATH` | Shared secret between reader and scraper | Regenerate if lost |
@@ -71,6 +71,7 @@ demo manga in an empty library after the build in section 5:
| `READING_WINDOWS_JSON`, `READING_TIMEZONE`, `READING_QUIZ_*` | off | Optional server-enforced reading hours ([reading-limiter.md](reading-limiter.md)) |
| `DISCOVERY_HARVEST_AT` | `03:30` | Daily time (`HH:MM`) of the nightly Discover harvest from AniList, in the container's local time zone (`TZ`). Only users who opted in are harvested; the admin can change the time or switch it off on the Discover page ([discovery.md](discovery.md)) |
| `DISCOVERY_ANILIST_URL` | AniList's public API | Tests only: points the discovery harvest at a fixture server. Leave unset in production |
| `GREATEST_COVER_WARM` | `1` | `0` stops the reader from downloading the Greatest catalog's covers (about 100 images, one per second, 10 s after start and once a day for missing ones). Tests and offline installs set `0`; the page then loads covers from the catalog's own URLs |
| `APP_IMAGE`, `VCS_REF` | `mangacircus:local`, `development` | Image to run; set a pinned registry image to deploy prebuilt releases (section 7) |
| `COMPOSE_FILE` | `docker-compose.yml` | Extra Compose files, `:`-separated (e.g. `docker-compose.yml:deploy/compose.https.yml`) |
| `MANGACIRCUS_DOMAIN` | – | Host name for `deploy/compose.https.yml` |
+1 -1
View File
@@ -23,7 +23,7 @@ Within each cohort, **Rank by my vote** sorts flame → good → neutral → sku
The one-time seed is curated from the research document. All 106 catalog entries now have a cover URL. Most were matched to public AniList manga metadata; the five initial gaps were checked individually. Phoenix (1954) and Nausicaä use matching AniList records; Kitarō uses Drawn & Quarterly's Kitaro edition, Someone Hertz uses Shonen Jump's first-volume cover, and Hitoner uses Shueisha's 2026 first-volume cover. Hitoner's AniList 2024 entry represents an earlier work, so it was **not** used for the 2026 series. The five added URLs returned image content on 2026-10-09, but remote hosts can change them; the UI still shows a title-letter fallback if an image fails.
`node tools/enrich-greatest-catalog.js` is an **optional manual** public-AniList metadata refresh, paced at one request every 1.5 seconds. It only accepts matching manga title and **original serialization year**, leaving uncertain results untouched rather than using a sequel or remake cover. The dashboard makes **no metadata calls when loading**; only an admin pressing **Search MangaDex** triggers a bounded lookup. Browser covers are lazy-loaded from their listed source; local library covers take precedence.
`node tools/enrich-greatest-catalog.js` is an **optional manual** public-AniList metadata refresh, paced at one request every 1.5 seconds. It only accepts matching manga title and **original serialization year**, leaving uncertain results untouched rather than using a sequel or remake cover. The dashboard makes **no metadata calls when loading**; only an admin pressing **Search MangaDex** triggers a bounded lookup. Local library covers take precedence. For the rest, the reader keeps its own copy of each catalog cover in `data/greatest-covers/` (`greatest/covers.js`): it downloads missing ones one at a time, about one a second, 10 seconds after start and once a day, stops on HTTP 401/403/429, and accepts only JPEG/PNG/WebP/GIF/AVIF images up to 2 MB from the catalog's own HTTPS URLs. The page loads covers from `/api/greatest-cover`, which the browser caches for a year; until a copy exists that route redirects to the catalog URL. Hotlinking about 100 covers per page load made the image CDN drop some of them at random. `GREATEST_COVER_WARM=0` switches the downloads off.
Public catalog chapter counts are often absent; the UI says “Chapters unknown” rather than inventing one. For owned titles, the local library count wins. Status comes from AniList's releasing/finished flag or local library metadata, and may be unknown. Official source links may be publisher editions, chapter services, or an award/info lead; they do not imply all chapters are available in every country. No unverified hoster coverage is claimed.
+73
View File
@@ -0,0 +1,73 @@
'use strict';
// Local copies of the Greatest catalog's cover images. Hotlinking ~100 covers per page
// load makes the image CDN drop some of them at random; one polite download each fixes that.
const fs = require('node:fs');
const path = require('node:path');
const crypto = require('node:crypto');
const TYPES = { 'image/jpeg': '.jpg', 'image/png': '.png', 'image/webp': '.webp', 'image/gif': '.gif', 'image/avif': '.avif' };
const EXTENSIONS = new Set(Object.values(TYPES));
const USER_AGENT = 'MangaCircus-Greatest/1.0';
class GreatestCovers {
constructor(dir, { fetchImpl = fetch, sleep = ms => new Promise(resolve => setTimeout(resolve, ms)),
delayMs = 1000, maxBytes = 2 * 1024 * 1024, timeoutMs = 20000 } = {}) {
this.dir = path.resolve(dir);
Object.assign(this, { fetchImpl, sleep, delayMs, maxBytes, timeoutMs });
this.files = new Map(); // key -> file name
try {
for (const name of fs.readdirSync(this.dir)) {
const ext = path.extname(name);
if (EXTENSIONS.has(ext)) this.files.set(path.basename(name, ext), name);
}
} catch (error) { if (error.code !== 'ENOENT') throw error; }
}
// A hash of the URL names the file, so a changed catalog URL is fetched again.
version(url) { return crypto.createHash('sha256').update(String(url)).digest('hex').slice(0, 16); }
file(url) {
const name = url ? this.files.get(this.version(url)) : null;
return name ? path.join(this.dir, name) : null;
}
// Sequential, paced, and stops on 403/429 like every other outbound client here.
async warm(urls) {
const result = { fetched: 0, skipped: 0, failed: 0, stopped: null };
let requested = false;
for (const url of new Set(urls.filter(url => /^https:\/\/\S+$/i.test(String(url || ''))))) {
if (this.file(url)) { result.skipped++; continue; }
if (requested) await this.sleep(this.delayMs);
requested = true;
try {
await this.download(url);
result.fetched++;
} catch (error) {
if (error.stop) { result.stopped = error.message; break; }
result.failed++;
}
}
// Duplicates and repeats of stored covers count as skipped.
result.skipped += urls.filter(Boolean).length - new Set(urls.filter(Boolean)).size;
return result;
}
async download(url) {
const response = await this.fetchImpl(url, { headers: { 'user-agent': USER_AGENT },
signal: AbortSignal.timeout(this.timeoutMs) });
if ([401, 403, 429].includes(response.status)) throw Object.assign(new Error(`HTTP ${response.status}`), { stop: true });
if (!response.ok) throw new Error(`HTTP ${response.status}`);
const ext = TYPES[String(response.headers.get('content-type') || '').split(';')[0].trim().toLowerCase()];
if (!ext) throw new Error('Not an image');
if (Number(response.headers.get('content-length')) > this.maxBytes) throw new Error('Image too large');
const body = Buffer.from(await response.arrayBuffer());
if (!body.length || body.length > this.maxBytes) throw new Error('Image too large or empty');
const key = this.version(url), name = key + ext;
fs.mkdirSync(this.dir, { recursive: true });
const temp = path.join(this.dir, `.${key}.${crypto.randomBytes(4).toString('hex')}.tmp`);
try {
fs.writeFileSync(temp, body);
fs.renameSync(temp, path.join(this.dir, name));
} finally { fs.rmSync(temp, { force: true }); }
this.files.set(key, name);
}
}
module.exports = { GreatestCovers };
+11
View File
@@ -94,7 +94,18 @@ main { max-width: 1200px; margin: 0 auto; padding: 16px; }
.prog i { display: block; height: 100%; background: var(--accent); }
.badge { position: absolute; top: 7px; right: 7px; background: var(--accent); color: #fff; font-size: 12px; font-weight: 700; padding: 2px 8px; border-radius: 10px; }
.empty { text-align: center; padding: 60px 10px; }
/* One fixed row; tiles sized like a six-column .grid, extra tiles scroll sideways instead of wrapping. */
.continue-row { margin-bottom: 20px; padding-bottom: 16px; border-bottom: 1px solid var(--line); }
.continue-banner {
display: inline-flex; gap: 6px; align-items: center; margin-bottom: 9px; padding: 2px 9px; border-radius: 10px;
background: color-mix(in srgb, var(--accent) 22%, var(--card)); color: #ffd8de; font-size: 12px; font-weight: 650;
}
.continue-tiles {
display: grid; grid-auto-flow: column; grid-auto-columns: max(140px, calc((100% - 80px) / 6)); gap: 16px;
overflow-x: auto; padding-bottom: 4px; scrollbar-width: thin;
}
@media (max-width: 560px) {
.continue-tiles { grid-auto-columns: calc((100% - 12px) / 2); gap: 12px; }
main { padding: 12px; }
.toolbar { flex-wrap: wrap; }
.toolbar input { flex-basis: 100%; }
+4 -2
View File
@@ -65,8 +65,10 @@
function cover(item) {
const box = h('span', { class: 'greatest-cover' }, h('span', { class: 'greatest-cover-fallback', 'aria-hidden': 'true' }, item.title.slice(0, 1)));
if (item.cover && safeHref(item.cover)) {
const image = h('img', { src: item.cover, alt: '', loading: 'lazy' });
// Covers come from the server (cached catalog covers, owned library covers) as relative URLs.
const src = /^(api\/greatest-cover\?|lib\/)/.test(item.cover || '') ? item.cover : safeHref(item.cover);
if (src) {
const image = h('img', { src, alt: '', loading: 'lazy' });
image.onerror = () => image.remove();
box.append(image);
}
+23 -12
View File
@@ -55,6 +55,12 @@
h('option', { value: 'grouped' }, 'Group collections'), h('option', { value: 'expanded' }, 'Expand collections'), h('option', { value: 'individual' }, 'Individual manga'));
grouping.value = ['grouped', 'expanded', 'individual'].includes(store.get('grouping')) ? store.get('grouping') : 'grouped';
grouping.onchange = render;
// Started but unfinished manga, most recently read first; the same manga still appear in the grid.
const continueItems = data.mangas.filter(m => m.lastRead && m.read < m.chapters)
.sort((a, b) => b.lastRead - a.lastRead).slice(0, 5);
const continueRow = h('section', { class: 'continue-row', 'aria-label': 'Continue where you left off', hidden: true },
h('div', { class: 'continue-banner' }, h('span', { 'aria-hidden': 'true' }, '▶'), 'Continue where you left off'),
h('div', { class: 'continue-tiles' }, continueItems.map(m => tile(m))));
if (storyAvailable) {
const most = data.mangas.find(manga => manga.id === data.reading.mostTimeMangaId);
const latest = data.mangas.find(manga => manga.id === data.reading.lastMangaId);
@@ -76,7 +82,7 @@
h('div', { class: 'filter-heading' }, h('b', {}, 'Tags'), h('span', { class: 'muted small' }, 'Manga must match every selected tag')),
tagBar) : null,
resultCount),
grid);
continueRow, grid);
function syncTags() {
store.set('tags', JSON.stringify([...selectedTags]));
@@ -95,8 +101,24 @@
return direction * (at - bt) || a.title.localeCompare(b.title);
}
function tile(m) {
const cover = m.isCollection && m.coverMode === 'mosaic' && m.covers.length
? h('div', { class: 'cv collection-mosaic' }, m.covers.map(src => h('img', { loading: 'lazy', src, alt: '' })))
: h('div', { class: 'cv' }, m.cover ? h('img', { loading: 'lazy', src: m.cover, alt: '' }) : null);
return h('a', { class: 'tile', href: m.isCollection ? 'collection.html?' + qsa({ id: m.id }) : mangaUrl(m.id) },
cover, m.isCollection ? h('span', { class: 'collection-count' }, `${m.members.length} manga`) : null,
m.unread ? h('span', { class: 'badge' }, m.unread + ' new') : null,
h('div', { class: 't' }, m.title),
h('div', { class: 's' }, [m.firstRelease, m.status, m.read ? `${m.read} / ${m.chapters} read` : `${m.chapters} chapters`].filter(Boolean).join(' · ')),
m.read ? h('div', { class: 'prog' }, h('i', { style: `width:${Math.round((100 * m.read) / Math.max(1, m.chapters))}%` })) : null);
}
function render() {
store.set('filter', filter.value); store.set('sort', sort.value); store.set('status', status.value); store.set('grouping', grouping.value);
// Only on the plain overview: a search or status filter means the reader is looking for something else,
// and "Recently read" already puts these first. Tags and grouping keep it.
continueRow.hidden = !me.settings.continueRow || !continueItems.length || sort.value === 'read'
|| !!filter.value.trim() || !!status.value;
const q = filter.value.trim().toLowerCase();
const groups = (data.collections || []).filter(c => c.grouped);
const membership = new Set(groups.flatMap(c => c.members.map(m => m.id)));
@@ -125,17 +147,6 @@
}
resultCount.textContent = `${list.length} results · ${data.mangas.length} manga in the library`;
grid.textContent = '';
const tile = m => {
const cover = m.isCollection && m.coverMode === 'mosaic' && m.covers.length
? h('div', { class: 'cv collection-mosaic' }, m.covers.map(src => h('img', { loading: 'lazy', src, alt: '' })))
: h('div', { class: 'cv' }, m.cover ? h('img', { loading: 'lazy', src: m.cover, alt: '' }) : null);
return h('a', { class: 'tile', href: m.isCollection ? 'collection.html?' + qsa({ id: m.id }) : mangaUrl(m.id) },
cover, m.isCollection ? h('span', { class: 'collection-count' }, `${m.members.length} manga`) : null,
m.unread ? h('span', { class: 'badge' }, m.unread + ' new') : null,
h('div', { class: 't' }, m.title),
h('div', { class: 's' }, [m.firstRelease, m.status, m.read ? `${m.read} / ${m.chapters} read` : `${m.chapters} chapters`].filter(Boolean).join(' · ')),
m.read ? h('div', { class: 'prog' }, h('i', { style: `width:${Math.round((100 * m.read) / Math.max(1, m.chapters))}%` })) : null);
};
if (grouping.value === 'expanded') {
const visible = new Set(list.map(m => m.id));
for (const c of groups) {
+5
View File
@@ -90,6 +90,11 @@
h('button', { class: 'btn small', onclick: async () => { await save({ zoom: {} }); zoomText.textContent = zooms(); } }, 'Reset to fit'),
h('div', { class: 'muted small' }, '100% = the page fits the screen. Change it in the reader with Ctrl + mouse wheel, a pinch, or the + / - keys.'))));
section('user/reading').append(h('section', { class: 'card' },
h('h2', {}, 'Library page'),
check('continueRow', 'Show "Continue where you left off" at the top',
'One row with the five manga you read most recently and have not finished. It hides itself while you search, filter by status, or sort by "Recently read".')));
section('user/reading').append(h('section', { class: 'card' },
h('h2', {}, 'Chapter list dates'),
h('label', { class: 'field' }, h('span', {}, 'When I read a chapter'), readDate,
+30 -3
View File
@@ -19,6 +19,7 @@ const { Catalog, CatalogError, collectionViews, TYPES, ROLES } = require('./cata
const { RATINGS: GREATEST_RATINGS, loadCatalog: loadGreatestCatalog, findOwned: findGreatestOwned,
countsFor: greatestCounts, scoreFor: greatestScore } = require('./greatest');
const { hosterLinks } = require('./greatest/hosters');
const { GreatestCovers } = require('./greatest/covers');
const { secret } = require('./runtime/config');
const { PeeringState, requirePeer } = require('./peering/state');
const peerClient = require('./peering/client');
@@ -65,6 +66,10 @@ fs.mkdirSync(DATA, { recursive: true });
const catalog = new Catalog(DATA);
const greatestCatalog = loadGreatestCatalog();
const greatestById = new Map(greatestCatalog.entries.map(entry => [entry.id, entry]));
const greatestCovers = new GreatestCovers(path.join(DATA, 'greatest-covers'));
// The versioned URL lets browsers keep a cover for a year; a new catalog URL changes `v`.
const greatestCoverUrl = entry => entry.cover
? `api/greatest-cover?id=${encodeURIComponent(entry.id)}&v=${greatestCovers.version(entry.cover)}` : null;
const DB_FILE = path.join(DATA, 'db.json'), CACHE_FILE = path.join(DATA, 'scan-cache.json'), INDEX_FILE = path.join(DATA, 'library-index.json');
const db = { users: {}, sessions: {}, progress: {}, reading: {}, readingAccess: {}, follows: {}, notes: {}, votes: {}, greatestRatings: {}, discoveryFeedback: {}, known: {} };
// A corrupt db.json stops the server instead of silently starting empty and overwriting it.
@@ -106,7 +111,7 @@ function checkPw(pw, stored) {
}
const DUMMY_HASH = hashPw('x');
const DEFAULTS = { startMode: 'last', lastMode: 'page', adapt: true, skipExtras: false, skipCovers: false, flowPageGap: true, pageTone: 'original', readingClock: false, readingStory: true,
readDate: 'hour', releaseDate: true, discoveryEnabled: false, discoveryReadingActivity: false, discoveryMature: false, zoom: {}, webhook: '' };
continueRow: true, readDate: 'hour', releaseDate: true, discoveryEnabled: false, discoveryReadingActivity: false, discoveryMature: false, zoom: {}, webhook: '' };
const settingsOf = u => ({ ...DEFAULTS, ...u.settings, zoom: { ...(u.settings && u.settings.zoom) } });
function createUser(name, pw, admin, replace) {
@@ -525,7 +530,7 @@ const routes = {
if (['flow', 'page', 'book'].includes(b.lastMode)) s.lastMode = b.lastMode;
if (['original', 'inverted', 'sepia', 'dim', 'extra-dim'].includes(b.pageTone)) s.pageTone = b.pageTone;
if (['off', 'date', 'hour'].includes(b.readDate)) s.readDate = b.readDate;
for (const k of ['adapt', 'skipExtras', 'skipCovers', 'flowPageGap', 'readingClock', 'readingStory', 'releaseDate',
for (const k of ['adapt', 'skipExtras', 'skipCovers', 'flowPageGap', 'readingClock', 'readingStory', 'releaseDate', 'continueRow',
'discoveryEnabled', 'discoveryReadingActivity', 'discoveryMature']) if (typeof b[k] === 'boolean') s[k] = b[k];
if (b.zoom && typeof b.zoom === 'object') {
s.zoom = {};
@@ -620,7 +625,7 @@ const routes = {
const counts = greatestCounts(entry.id, db.greatestRatings || {}, db.users);
return {
...entry,
cover: owned?.cover ? url(owned.cover) : entry.cover,
cover: owned?.cover ? url(owned.cover) : greatestCoverUrl(entry),
chapters: owned ? owned.chapters.length : entry.chapters,
status: owned?.meta?.status || entry.status,
library: owned ? { id: owned.id, chapters: owned.chapters.length } : null,
@@ -1034,6 +1039,16 @@ http.createServer(async (req, res) => {
if (!c?.customCover) bad(404, 'Cover not found');
return serveFile(req, res, path.join(DATA, 'collection-covers'), c.customCover, 'private, no-cache', IMG);
}
if (pathname === '/api/greatest-cover' && req.method === 'GET') {
if (!u) bad(401, 'Please log in');
const entry = greatestById.get(searchParams.get('id'));
if (!entry?.cover) bad(404, 'Cover not found');
const file = greatestCovers.file(entry.cover);
if (file) return serveFile(req, res, greatestCovers.dir, path.basename(file), 'private, max-age=31536000, immutable', IMG);
// Not downloaded yet: the browser fetches the catalog's own URL this once.
res.writeHead(302, { Location: entry.cover, 'Cache-Control': 'no-store' });
return res.end();
}
if (pathname === '/api/peering/cover' && req.method === 'GET') {
if (!u) bad(401, 'Please log in');
admin(u);
@@ -1074,6 +1089,18 @@ http.createServer(async (req, res) => {
if (lib.mangas.length) refreshChanged();
else rescan();
if (SCAN_MINUTES > 0) setInterval(() => refreshChanged(), SCAN_MINUTES * 60000).unref();
// Fetch missing Greatest covers shortly after start and retry failures once a day;
// covers already on disk cost nothing.
async function warmGreatestCovers() {
try {
const result = await greatestCovers.warm(greatestCatalog.entries.map(entry => entry.cover));
if (result.fetched || result.failed || result.stopped) console.log('[greatest] covers', JSON.stringify(result));
} catch (error) { console.error('[greatest] covers', error.message); }
}
if (process.env.GREATEST_COVER_WARM !== '0') {
setTimeout(warmGreatestCovers, 10000).unref();
setInterval(warmGreatestCovers, 86400000).unref();
}
// Persist the last reconciliation time. Restarting a healthy reader should not
// trigger another full tree walk; overdue reconciliations run within a minute.
if (FULL_SCAN_MINUTES > 0) setInterval(() => {
+2 -1
View File
@@ -21,7 +21,8 @@ test('only admins can promote or demote, nobody can demote themselves, and chang
const port = await freePort(), output = [];
child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '' } });
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '',
GREATEST_COVER_WARM: '0' } });
child.stdout.on('data', d => output.push(String(d)));
child.stderr.on('data', d => output.push(String(d)));
const base = `http://127.0.0.1:${port}`;
+1 -1
View File
@@ -251,7 +251,7 @@ test('reader and scraper: admin backup API pauses and resumes the real queue', {
const base = `http://127.0.0.1:${rp}`, token = 'backup-test-only';
const env = { ...process.env, DATA: f.data, LIBRARY: library, SCRAPER_DATA: path.join(f.data, 'scraper'), BACKUP: f.target,
PORT: String(rp), SCRAPER_PORT: String(sp), SCRAPER_HOST: '127.0.0.1', SCRAPER_URL: `http://127.0.0.1:${sp}`, SENTINEL_URL: 'http://127.0.0.1:9',
SCRAPER_TOKEN: token, SCRAPER_TOKEN_FILE: '', SCAN_INTERVAL: '0' };
SCRAPER_TOKEN: token, SCRAPER_TOKEN_FILE: '', SCAN_INTERVAL: '0', GREATEST_COVER_WARM: '0' };
for (const script of ['server.js', 'scraper/daemon.js']) {
const child = spawn(process.execPath, [script], { cwd: root, env, windowsHide: true }); children.push(child);
for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => output.push(chunk.toString()));
+1 -1
View File
@@ -81,7 +81,7 @@ async function startReader({ anilist, discovery = {}, mangaMeta = null, scraperU
const child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
SCRAPER_URL: scraperUrl, SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '',
DISCOVERY_ANILIST_URL: anilist ? anilist.url : 'http://127.0.0.1:9/', DISCOVERY_HARVEST_AT: '' } });
DISCOVERY_ANILIST_URL: anilist ? anilist.url : 'http://127.0.0.1:9/', DISCOVERY_HARVEST_AT: '', GREATEST_COVER_WARM: '0' } });
child.stdout.on('data', d => output.push(String(d)));
child.stderr.on('data', d => output.push(String(d)));
const base = `http://127.0.0.1:${port}`;
+75
View File
@@ -0,0 +1,75 @@
'use strict';
// BL-0053: Greatest covers are served from the reader's own copy, with a redirect until one exists.
const test = require('node:test'), assert = require('node:assert/strict');
const fs = require('fs'), os = require('os'), path = require('path'), net = require('net'), crypto = require('crypto');
const { spawn } = require('child_process');
const { loadCatalog } = require('../greatest');
const { GreatestCovers } = require('../greatest/covers');
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
const hashPw = pw => { const salt = crypto.randomBytes(16).toString('hex'); return salt + ':' + crypto.scryptSync(pw, salt, 32).toString('hex'); };
const sha = s => crypto.createHash('sha256').update(s).digest('hex');
const PNG = Buffer.from('89504e470d0a1a0a0000000d49484452', 'hex');
async function freePort() {
const listener = net.createServer();
await new Promise(resolve => listener.listen(0, '127.0.0.1', resolve));
const port = listener.address().port;
await new Promise(resolve => listener.close(resolve));
return port;
}
test('Greatest covers come from the local copy, else redirect to the catalog URL', async t => {
const entries = loadCatalog().entries.filter(entry => entry.cover);
const [cached, missing] = entries;
const root = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-greatest-cover-'));
const library = path.join(root, 'library'), data = path.join(root, 'data');
fs.mkdirSync(library, { recursive: true });
fs.mkdirSync(path.join(data, 'greatest-covers'), { recursive: true });
const version = new GreatestCovers(path.join(data, 'greatest-covers')).version(cached.cover);
fs.writeFileSync(path.join(data, 'greatest-covers', `${version}.png`), PNG);
const token = crypto.randomBytes(32).toString('base64url');
fs.writeFileSync(path.join(data, 'db.json'), JSON.stringify({
users: { reader: { name: 'reader', hash: hashPw('reader-password'), admin: false, created: Date.now(), settings: {} } },
sessions: { [sha(token)]: { u: 'reader', t: Date.now() } } }));
const port = await freePort(), output = [];
const child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', BACKUP: '',
GREATEST_COVER_WARM: '0' } });
child.stdout.on('data', d => output.push(String(d)));
child.stderr.on('data', d => output.push(String(d)));
t.after(async () => {
await new Promise(resolve => { if (child.exitCode !== null) return resolve(); child.once('exit', resolve); child.kill(); });
fs.rmSync(root, { recursive: true, force: true });
});
const base = `http://127.0.0.1:${port}`;
let ready = false;
for (let n = 0; n < 100 && !ready; n++) {
ready = await fetch(base + '/health').then(r => r.ok).catch(() => false);
if (!ready) await sleep(50);
}
assert.ok(ready, output.join(''));
const get = (url, cookie = `sid=${token}`) => fetch(base + '/' + url, { headers: cookie ? { cookie } : {}, redirect: 'manual' });
const listing = await (await get('api/greatest')).json();
const coverOf = id => listing.entries.find(entry => entry.id === id).cover;
assert.equal(coverOf(cached.id), `api/greatest-cover?id=${encodeURIComponent(cached.id)}&v=${version}`);
assert.match(coverOf(missing.id), /^api\/greatest-cover\?id=.+&v=[0-9a-f]{16}$/);
const hit = await get(coverOf(cached.id));
assert.equal(hit.status, 200);
assert.equal(hit.headers.get('content-type'), 'image/png');
assert.equal(hit.headers.get('cache-control'), 'private, max-age=31536000, immutable');
assert.deepEqual(Buffer.from(await hit.arrayBuffer()), PNG);
const miss = await get(coverOf(missing.id));
assert.equal(miss.status, 302);
assert.equal(miss.headers.get('location'), missing.cover);
assert.equal(miss.headers.get('cache-control'), 'no-store');
assert.equal((await get(coverOf(cached.id), null)).status, 401);
assert.equal((await get('api/greatest-cover?id=not-in-the-catalog')).status, 404);
assert.equal((await get('api/greatest-cover?id=../../db.json')).status, 404);
});
+84
View File
@@ -0,0 +1,84 @@
'use strict';
const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const os = require('node:os');
const path = require('node:path');
const { GreatestCovers } = require('../greatest/covers');
const PNG = Buffer.from('89504e470d0a1a0a0000000d49484452', 'hex');
const image = (body = PNG, type = 'image/png') => new Response(body, { status: 200, headers: { 'content-type': type } });
function setup(t, responses, options = {}) {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'mc-greatest-covers-'));
t.after(() => fs.rmSync(dir, { recursive: true, force: true }));
const calls = [];
const fetchImpl = async (url, init) => {
calls.push({ url, init });
const next = responses[url];
if (next instanceof Error) throw next;
return typeof next === 'function' ? next() : next || new Response('missing', { status: 404 });
};
const sleeps = [];
const covers = new GreatestCovers(dir, { fetchImpl, sleep: async ms => { sleeps.push(ms); }, ...options });
return { dir, calls, sleeps, covers };
}
test('warm downloads each cover once and serves it from disk afterwards', async t => {
const a = 'https://img.example/a.png', b = 'https://img.example/b.jpg';
const { dir, calls, sleeps, covers } = setup(t, { [a]: () => image(), [b]: () => image(PNG, 'image/jpeg') });
const result = await covers.warm([a, b, a]);
assert.deepEqual(result, { fetched: 2, skipped: 1, failed: 0, stopped: null });
assert.equal(calls.length, 2);
assert.deepEqual(sleeps, [1000], 'one pause between two downloads, none before the first');
assert.ok(covers.file(a).endsWith('.png') && fs.existsSync(covers.file(a)));
assert.ok(covers.file(b).endsWith('.jpg'));
assert.equal(covers.file('https://img.example/other.png'), null);
// A restarted reader finds the stored files and asks nobody.
const reopened = new GreatestCovers(dir, { fetchImpl: async () => { throw new Error('must not fetch'); } });
assert.deepEqual(await reopened.warm([a, b]), { fetched: 0, skipped: 2, failed: 0, stopped: null });
assert.equal(reopened.file(a), covers.file(a));
});
test('a changed cover URL is a new file', async t => {
const { covers } = setup(t, { 'https://img.example/v1.png': () => image(), 'https://img.example/v2.png': () => image() });
await covers.warm(['https://img.example/v1.png', 'https://img.example/v2.png']);
assert.notEqual(covers.file('https://img.example/v1.png'), covers.file('https://img.example/v2.png'));
assert.notEqual(covers.version('https://img.example/v1.png'), covers.version('https://img.example/v2.png'));
});
test('non-images, oversized files, plain http and errors are skipped', async t => {
const html = 'https://img.example/page', big = 'https://img.example/big.png', declaredBig = 'https://img.example/declared.png';
const broken = 'https://img.example/broken.png', gone = 'https://img.example/gone.png';
const { covers, calls } = setup(t, {
[html]: () => new Response('<html>', { status: 200, headers: { 'content-type': 'text/html' } }),
[big]: () => image(Buffer.alloc(64)),
[declaredBig]: () => new Response(PNG, { status: 200, headers: { 'content-type': 'image/png', 'content-length': '999999' } }),
[broken]: new TypeError('fetch failed'),
}, { maxBytes: 32 });
const result = await covers.warm([html, big, declaredBig, broken, gone, 'http://img.example/plain.png', '', null]);
assert.deepEqual(result, { fetched: 0, skipped: 0, failed: 5, stopped: null });
assert.equal(calls.length, 5, 'plain http and empty URLs are never requested');
for (const url of [html, big, declaredBig, broken, gone]) assert.equal(covers.file(url), null);
assert.deepEqual(fs.readdirSync(covers.dir).filter(name => !name.startsWith('.')), []);
});
test('429 and 403 stop the run at once', async t => {
for (const status of [429, 403]) {
const first = 'https://img.example/1.png', second = 'https://img.example/2.png';
const { covers, calls } = setup(t, { [first]: () => new Response('', { status }), [second]: () => image() });
const result = await covers.warm([first, second]);
assert.equal(result.stopped, `HTTP ${status}`);
assert.equal(calls.length, 1);
assert.equal(covers.file(second), null);
}
});
test('requests carry a timeout and a fixed user agent', async t => {
const a = 'https://img.example/a.png';
const { covers, calls } = setup(t, { [a]: () => image() });
await covers.warm([a]);
assert.ok(calls[0].init.signal instanceof AbortSignal);
assert.match(calls[0].init.headers['user-agent'], /^MangaCircus/);
});
+1 -1
View File
@@ -78,7 +78,7 @@ test('reader serves a saved index immediately after restart and verifies it in b
const output = [];
child = spawn(process.execPath, ['server.js'], { cwd: path.resolve(__dirname, '..'), windowsHide: true,
env: { ...process.env, LIBRARY: library, DATA: data, PORT: String(port), SCAN_INTERVAL: '0', FULL_SCAN_INTERVAL: '0',
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '' } });
SCRAPER_URL: 'http://127.0.0.1:9', SCRAPER_TOKEN: 'fixture-token', SCRAPER_TOKEN_FILE: '', GREATEST_COVER_WARM: '0' } });
child.stdout.on('data', data => output.push(String(data)));
child.stderr.on('data', data => output.push(String(data)));
const base = `http://127.0.0.1:${port}`;
+5 -1
View File
@@ -29,7 +29,7 @@ test('three services integrate collections, auth, custom covers and private heal
const token = 'integration-test-only';
const env = { ...process.env, DATA: data, LIBRARY: library, SCRAPER_DATA: path.join(data, 'scraper'),
PORT: String(rp), SCRAPER_PORT: String(sp), SCRAPER_HOST: '127.0.0.1',
READER_URL: base, SCRAPER_URL: scraper, SCRAPER_TOKEN: token, SCRAPER_TOKEN_FILE: '', SCAN_INTERVAL: '0', COOKIE_SECURE: '0' };
READER_URL: base, SCRAPER_URL: scraper, SCRAPER_TOKEN: token, SCRAPER_TOKEN_FILE: '', SCAN_INTERVAL: '0', COOKIE_SECURE: '0', GREATEST_COVER_WARM: '0' };
for (const script of ['server.js', 'scraper/daemon.js']) {
const child = spawn(process.execPath, [script], { cwd: root, env, windowsHide: true }); children.push(child);
for (const stream of [child.stdout, child.stderr]) stream.on('data', chunk => output.push(chunk.toString()));
@@ -67,6 +67,10 @@ test('three services integrate collections, auth, custom covers and private heal
assert.equal((await (await request('/api/settings', { pageTone: 'sepia' })).json()).settings.pageTone, 'sepia');
assert.equal((await (await request('/api/settings', { pageTone: 'invalid' })).json()).settings.pageTone, 'sepia');
assert.equal((await (await anotherRequest('/api/me')).json()).settings.pageTone, 'original');
assert.equal((await (await request('/api/me')).json()).settings.continueRow, true);
assert.equal((await (await request('/api/settings', { continueRow: false })).json()).settings.continueRow, false);
assert.equal((await (await request('/api/settings', { continueRow: 'no' })).json()).settings.continueRow, false);
assert.equal((await (await anotherRequest('/api/me')).json()).settings.continueRow, true);
assert.equal((await fetch(base + '/api/discovery')).status, 401);
assert.equal((await anotherRequest('/api/admin/discovery/run', {})).status, 403);
assert.equal((await request('/api/settings', { discoveryEnabled: true })).status, 200);
+1
View File
@@ -69,6 +69,7 @@ for (const [index, name] of ['Moon Garden', 'Before the Rain', 'Iron Harbor'].en
const env = { ...process.env, LIBRARY_PATH: library, DATA_PATH: data, SCRAPER_DATA_PATH: path.join(data, 'scraper'),
SCRAPER_TOKEN_PATH: tokenFile, BACKUP_PATH: backupDir, READER_PORT: port, READER_BIND: '127.0.0.1', COOKIE_SECURE: '0', APP_IMAGE: process.env.APP_IMAGE || 'mangacircus:local',
READER_IMAGE: process.env.APP_IMAGE || 'mangacircus:local',
GREATEST_COVER_WARM: '0', // QA stacks never download catalog covers
...(readingGateQa ? { READING_WINDOWS_JSON: '{"0":[],"1":[],"2":[],"3":[],"4":[],"5":[],"6":[]}',
READING_TIMEZONE: 'Europe/Berlin', READING_QUIZ_UNLOCK_MINUTES: '60', READING_QUIZ_CATALOG_PATH: '/data/reading-quiz.json' }
: { READING_WINDOWS_JSON: '', READING_QUIZ_CATALOG_PATH: '' }),
+1 -1
View File
@@ -40,7 +40,7 @@ async function freePort() {
const scraperPort = fakeScraper.address().port;
const child = spawn(process.execPath, ['server.js'], { cwd: root, windowsHide: true,
env: { ...process.env, PORT: String(port), LIBRARY: library, DATA: data, SCAN_INTERVAL: '0',
FULL_SCAN_INTERVAL: '0', SCRAPER_URL: `http://127.0.0.1:${scraperPort}`, COOKIE_SECURE: '0' }, stdio: 'ignore' });
FULL_SCAN_INTERVAL: '0', SCRAPER_URL: `http://127.0.0.1:${scraperPort}`, COOKIE_SECURE: '0', GREATEST_COVER_WARM: '0' }, stdio: 'ignore' });
let browser;
try {
for (let i = 0; i < 80; i++) {