Add built-in backup of the data folder (BL-0044)

The reader owns a scheduled backup into BACKUP (Compose: BACKUP_PATH mounted at
/backup, off when unset). A run pauses the scraper queue through its control
API, waits (bounded) until no job is active, writes a verified tar.gz of DATA
with db.json from memory and scraper/queue.json stored paused, excludes
runtime/lock/pid/temp files, resumes only a queue it paused, then applies
count- or day-based retention to its own archive names only. Schedule,
retention, "Back up now" and last result live in Settings > Service health.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
MaashkinoozheandClaude Opus 5.5 committed 2026-10-22 18:03:21 +02:00
1 parent fcc50d576c
commit 086f6353f8
15 files changed
+904 -11

No files matched your search

+11 -2
View File
@@ -18,6 +18,7 @@ Related documents:
- [peering.md](peering.md): optional private server-to-server catalog and transfer.
- [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
- [source-modules.md](source-modules.md): adding and maintaining first-party hosters inside the single scraper process.
- [backup.md](backup.md): built-in scheduled backup of `data/` and the restore procedure.
## 1. Principles
@@ -67,6 +68,7 @@ survives restarts, so restarting does not itself force another complete walk.
|------|------:|----------------|
| `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks |
| `library.js` | ~170 | Folder scanner, name parsing, image-size reader, scan cache |
| `backup.js` | ~330 | Built-in `DATA` backup: queue pause/resume, tar.gz writer and verifier, schedule, retention |
| `public/common.js` | ~85 | Browser helpers: `h()` element builder, `api()` fetch wrapper, top navigation with the notification bell |
| `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) |
| `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle |
@@ -157,7 +159,11 @@ This keeps ids human-readable and derivable without a database. The price: renam
- `data/library-index.json` holds a disposable, versioned index snapshot and shallow
directory stamps. The library folders remain authoritative; a missing or invalid
snapshot triggers a full scan. Rollback-compatible `scan-cache.json` is retained.
- **Backups:** copy `data/db.json`. The atomic writes make a copy at any time consistent.
- `data/backup.json` holds the built-in backup schedule, retention and recent results.
- **Backups:** the reader's built-in backup ([backup.md](backup.md)) archives the whole
`data/` tree into `BACKUP` on a schedule, with `db.json` taken from memory and the
queue paused around the copy. A plain copy of `data/db.json` also works: atomic
writes make it consistent at any time.
## 6. Server
@@ -199,6 +205,7 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
| `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result |
| `POST /api/admin/user` `{name, password, admin}` | admin | Create a user or reset a password |
| `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data |
| `GET /api/admin/backup` · `POST /api/admin/backup` `{command:"settings"\|"run"}` | admin | Built-in backup status, schedule/retention, "back up now" ([backup.md](backup.md)) |
### 6.3 Authentication and security
@@ -342,7 +349,8 @@ docker compose up -d --build
- **Volumes:**
- `/library`: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000.
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
- `/backup`: reader only, writable target of the built-in backup (`BACKUP_PATH`; off when unset).
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `BACKUP`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
@@ -366,6 +374,7 @@ docker compose up -d --build
|----------|--------------|-----|
| Index the folder tree in place | Import into a DB / managed storage | Stays readable by any viewer and writable by any tool. Nothing to keep in sync |
| No dependencies, no framework, no build | Express, React, bundler | Minimal code, maximum longevity, runs anywhere Node runs |
| Built-in backup in the reader, queue paused, no container stop | Host job that stops the stack; backup in Sentinel | Only the reader sees all of `/data` and the in-memory `db.json`; the scraper's pause API gives a consistent point without downtime |
| JSON file for user data | SQLite (`node:sqlite` is still experimental in Node 22/24), Postgres | Zero dependencies, trivial backup, enough for the target size. Easy to swap: all access goes through the `db` object |
| Page roles as file-name tags | Page lists in JSON | Visible in any file browser, survives copying, no second source of truth. Unknown tags are harmless |
| Image sizes read on the server | Measure in the browser after load | Layout before load: exact scroll restore, zoom anchoring, auto-detected double pages, no layout jumps |