Cache library index and refresh changed manga
This commit is contained in:
1 parent
eef4ffa07c
commit
8ac27e1f3d
6 files changed
+269
-30
No files matched your search
+10
-3
@@ -54,7 +54,10 @@ The catalog and relation store is reader-owned `data/catalog.json`. Sentinel wri
|
||||
The admin browser never talks to the scraper daemon directly. Admin-only reader
|
||||
routes proxy commands to the daemon's private API, so its shared token stays on the
|
||||
server. The daemon publishes complete folders atomically and increments a durable
|
||||
revision; the reader polls that revision and rescans the library after publication.
|
||||
revision; the reader polls that revision and refreshes changed manga after publication.
|
||||
The reader restores a disposable full-index snapshot at startup so browsing need not
|
||||
wait for a filesystem walk. Shallow directory stamps identify changed manga; a
|
||||
background and daily full reconciliation catches in-place manual edits.
|
||||
|
||||
| Path | Lines | Responsibility |
|
||||
|------|------:|----------------|
|
||||
@@ -78,7 +81,8 @@ revision; the reader polls that revision and rescans the library after publicati
|
||||
- `lib` is the current library index, replaced as a whole after each scan.
|
||||
- `db` is all user data, loaded from `db.json` at start.
|
||||
- **Timers:**
|
||||
- a library scan at start and every `SCAN_INTERVAL` minutes,
|
||||
- a saved-index restore and shallow check at start (full scan when no snapshot),
|
||||
- a shallow check every `SCAN_INTERVAL` minutes, full reconciliation every `FULL_SCAN_INTERVAL` minutes,
|
||||
- a debounced `db.json` write, 500 ms after the last change.
|
||||
- **Shutdown:** `SIGINT` / `SIGTERM` flush `db.json`; the daemon cooperatively stops at a page boundary and keeps its archive checkpoint.
|
||||
- **Durability:** every file write goes to `*.tmp` first and is then renamed over the target, so a file is never half-written. A hard kill loses at most the changes of the last ~0.5 s. A corrupt `db.json` makes the server **refuse to start** rather than start empty and overwrite it.
|
||||
@@ -146,6 +150,9 @@ This keeps ids human-readable and derivable without a database. The price: renam
|
||||
```
|
||||
|
||||
- `data/scan-cache.json` holds `{ "<Manga>/<Book>/<Chapter>": { m: <folder mtimeMs>, p: [[file, w, h], …] } }`. It is disposable.
|
||||
- `data/library-index.json` holds a disposable, versioned index snapshot and shallow
|
||||
directory stamps. The library folders remain authoritative; a missing or invalid
|
||||
snapshot triggers a full scan. Rollback-compatible `scan-cache.json` is retained.
|
||||
- **Backups:** copy `data/db.json`. The atomic writes make a copy at any time consistent.
|
||||
|
||||
## 6. Server
|
||||
@@ -331,7 +338,7 @@ docker compose up -d --build
|
||||
- **Volumes:**
|
||||
- `/library`: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000.
|
||||
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
|
||||
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
|
||||
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
|
||||
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
|
||||
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
|
||||
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
|
||||
|
||||
Reference in new issue
Block a user