Add built-in backup of the data folder (BL-0044)
The reader owns a scheduled backup into BACKUP (Compose: BACKUP_PATH mounted at /backup, off when unset). A run pauses the scraper queue through its control API, waits (bounded) until no job is active, writes a verified tar.gz of DATA with db.json from memory and scraper/queue.json stored paused, excludes runtime/lock/pid/temp files, resumes only a queue it paused, then applies count- or day-based retention to its own archive names only. Schedule, retention, "Back up now" and last result live in Settings > Service health. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
fcc50d576c
commit
086f6353f8
15 files changed
+904
-11
No files matched your search
+11
-2
@@ -18,6 +18,7 @@ Related documents:
|
||||
- [peering.md](peering.md): optional private server-to-server catalog and transfer.
|
||||
- [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
|
||||
- [source-modules.md](source-modules.md): adding and maintaining first-party hosters inside the single scraper process.
|
||||
- [backup.md](backup.md): built-in scheduled backup of `data/` and the restore procedure.
|
||||
|
||||
## 1. Principles
|
||||
|
||||
@@ -67,6 +68,7 @@ survives restarts, so restarting does not itself force another complete walk.
|
||||
|------|------:|----------------|
|
||||
| `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks |
|
||||
| `library.js` | ~170 | Folder scanner, name parsing, image-size reader, scan cache |
|
||||
| `backup.js` | ~330 | Built-in `DATA` backup: queue pause/resume, tar.gz writer and verifier, schedule, retention |
|
||||
| `public/common.js` | ~85 | Browser helpers: `h()` element builder, `api()` fetch wrapper, top navigation with the notification bell |
|
||||
| `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) |
|
||||
| `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle |
|
||||
@@ -157,7 +159,11 @@ This keeps ids human-readable and derivable without a database. The price: renam
|
||||
- `data/library-index.json` holds a disposable, versioned index snapshot and shallow
|
||||
directory stamps. The library folders remain authoritative; a missing or invalid
|
||||
snapshot triggers a full scan. Rollback-compatible `scan-cache.json` is retained.
|
||||
- **Backups:** copy `data/db.json`. The atomic writes make a copy at any time consistent.
|
||||
- `data/backup.json` holds the built-in backup schedule, retention and recent results.
|
||||
- **Backups:** the reader's built-in backup ([backup.md](backup.md)) archives the whole
|
||||
`data/` tree into `BACKUP` on a schedule, with `db.json` taken from memory and the
|
||||
queue paused around the copy. A plain copy of `data/db.json` also works: atomic
|
||||
writes make it consistent at any time.
|
||||
|
||||
## 6. Server
|
||||
|
||||
@@ -199,6 +205,7 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
|
||||
| `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result |
|
||||
| `POST /api/admin/user` `{name, password, admin}` | admin | Create a user or reset a password |
|
||||
| `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data |
|
||||
| `GET /api/admin/backup` · `POST /api/admin/backup` `{command:"settings"\|"run"}` | admin | Built-in backup status, schedule/retention, "back up now" ([backup.md](backup.md)) |
|
||||
|
||||
### 6.3 Authentication and security
|
||||
|
||||
@@ -342,7 +349,8 @@ docker compose up -d --build
|
||||
- **Volumes:**
|
||||
- `/library`: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000.
|
||||
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
|
||||
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
|
||||
- `/backup`: reader only, writable target of the built-in backup (`BACKUP_PATH`; off when unset).
|
||||
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `BACKUP`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
|
||||
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
|
||||
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
|
||||
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
|
||||
@@ -366,6 +374,7 @@ docker compose up -d --build
|
||||
|----------|--------------|-----|
|
||||
| Index the folder tree in place | Import into a DB / managed storage | Stays readable by any viewer and writable by any tool. Nothing to keep in sync |
|
||||
| No dependencies, no framework, no build | Express, React, bundler | Minimal code, maximum longevity, runs anywhere Node runs |
|
||||
| Built-in backup in the reader, queue paused, no container stop | Host job that stops the stack; backup in Sentinel | Only the reader sees all of `/data` and the in-memory `db.json`; the scraper's pause API gives a consistent point without downtime |
|
||||
| JSON file for user data | SQLite (`node:sqlite` is still experimental in Node 22/24), Postgres | Zero dependencies, trivial backup, enough for the target size. Easy to swap: all access goes through the `db` object |
|
||||
| Page roles as file-name tags | Page lists in JSON | Visible in any file browser, survives copying, no second source of truth. Unknown tags are harmless |
|
||||
| Image sizes read on the server | Measure in the browser after load | Layout before load: exact scroll restore, zoom anchoring, auto-detected double pages, no layout jumps |
|
||||
|
||||
Reference in new issue
Block a user