Add built-in backup of the data folder (BL-0044)
The reader owns a scheduled backup into BACKUP (Compose: BACKUP_PATH mounted at /backup, off when unset). A run pauses the scraper queue through its control API, waits (bounded) until no job is active, writes a verified tar.gz of DATA with db.json from memory and scraper/queue.json stored paused, excludes runtime/lock/pid/temp files, resumes only a queue it paused, then applies count- or day-based retention to its own archive names only. Schedule, retention, "Back up now" and last result live in Settings > Service health. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
fcc50d576c
commit
086f6353f8
15 files changed
+904
-11
No files matched your search
+11
-2
@@ -18,6 +18,7 @@ Related documents:
|
||||
- [peering.md](peering.md): optional private server-to-server catalog and transfer.
|
||||
- [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
|
||||
- [source-modules.md](source-modules.md): adding and maintaining first-party hosters inside the single scraper process.
|
||||
- [backup.md](backup.md): built-in scheduled backup of `data/` and the restore procedure.
|
||||
|
||||
## 1. Principles
|
||||
|
||||
@@ -67,6 +68,7 @@ survives restarts, so restarting does not itself force another complete walk.
|
||||
|------|------:|----------------|
|
||||
| `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks |
|
||||
| `library.js` | ~170 | Folder scanner, name parsing, image-size reader, scan cache |
|
||||
| `backup.js` | ~330 | Built-in `DATA` backup: queue pause/resume, tar.gz writer and verifier, schedule, retention |
|
||||
| `public/common.js` | ~85 | Browser helpers: `h()` element builder, `api()` fetch wrapper, top navigation with the notification bell |
|
||||
| `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) |
|
||||
| `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle |
|
||||
@@ -157,7 +159,11 @@ This keeps ids human-readable and derivable without a database. The price: renam
|
||||
- `data/library-index.json` holds a disposable, versioned index snapshot and shallow
|
||||
directory stamps. The library folders remain authoritative; a missing or invalid
|
||||
snapshot triggers a full scan. Rollback-compatible `scan-cache.json` is retained.
|
||||
- **Backups:** copy `data/db.json`. The atomic writes make a copy at any time consistent.
|
||||
- `data/backup.json` holds the built-in backup schedule, retention and recent results.
|
||||
- **Backups:** the reader's built-in backup ([backup.md](backup.md)) archives the whole
|
||||
`data/` tree into `BACKUP` on a schedule, with `db.json` taken from memory and the
|
||||
queue paused around the copy. A plain copy of `data/db.json` also works: atomic
|
||||
writes make it consistent at any time.
|
||||
|
||||
## 6. Server
|
||||
|
||||
@@ -199,6 +205,7 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
|
||||
| `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result |
|
||||
| `POST /api/admin/user` `{name, password, admin}` | admin | Create a user or reset a password |
|
||||
| `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data |
|
||||
| `GET /api/admin/backup` · `POST /api/admin/backup` `{command:"settings"\|"run"}` | admin | Built-in backup status, schedule/retention, "back up now" ([backup.md](backup.md)) |
|
||||
|
||||
### 6.3 Authentication and security
|
||||
|
||||
@@ -342,7 +349,8 @@ docker compose up -d --build
|
||||
- **Volumes:**
|
||||
- `/library`: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000.
|
||||
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
|
||||
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
|
||||
- `/backup`: reader only, writable target of the built-in backup (`BACKUP_PATH`; off when unset).
|
||||
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `BACKUP`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
|
||||
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
|
||||
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
|
||||
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
|
||||
@@ -366,6 +374,7 @@ docker compose up -d --build
|
||||
|----------|--------------|-----|
|
||||
| Index the folder tree in place | Import into a DB / managed storage | Stays readable by any viewer and writable by any tool. Nothing to keep in sync |
|
||||
| No dependencies, no framework, no build | Express, React, bundler | Minimal code, maximum longevity, runs anywhere Node runs |
|
||||
| Built-in backup in the reader, queue paused, no container stop | Host job that stops the stack; backup in Sentinel | Only the reader sees all of `/data` and the in-memory `db.json`; the scraper's pause API gives a consistent point without downtime |
|
||||
| JSON file for user data | SQLite (`node:sqlite` is still experimental in Node 22/24), Postgres | Zero dependencies, trivial backup, enough for the target size. Easy to swap: all access goes through the `db` object |
|
||||
| Page roles as file-name tags | Page lists in JSON | Visible in any file browser, survives copying, no second source of truth. Unknown tags are harmless |
|
||||
| Image sizes read on the server | Measure in the browser after load | Layout before load: exact scroll restore, zoom anchoring, auto-detected double pages, no layout jumps |
|
||||
|
||||
+114
@@ -0,0 +1,114 @@
|
||||
# Built-in backup
|
||||
|
||||
The reader can back up its own `DATA` folder on a schedule set in
|
||||
**Settings → Admin → Service health → Backups**, without stopping any container.
|
||||
The library is not part of it (it is plain files; back it up with the storage that
|
||||
holds it). Implementation: [`backup.js`](../backup.js); tests: `test/backup.test.js`.
|
||||
|
||||
## Turning it on
|
||||
|
||||
| Where | Setting |
|
||||
|---|---|
|
||||
| Native | `BACKUP=/path/to/backups` for `server.js`. Unset = feature off |
|
||||
| Compose | `BACKUP_PATH=/host/folder` in `.env`. The reader mounts it rw at `/backup` and gets `BACKUP=/backup`; without `BACKUP_PATH` the variable stays empty and backups are off |
|
||||
|
||||
The folder must already exist and be writable by uid 1000 (`create_host_path: false`:
|
||||
a missing folder makes `docker compose up` fail instead of silently creating an empty
|
||||
one on the wrong disk). Use another disk or a NAS share, never a folder inside `DATA`.
|
||||
Without `BACKUP_PATH`, Compose still mounts the unused default `./backup`, so that
|
||||
folder must exist too (`node tools/prepare-compose.js` creates it). Upgrading an
|
||||
existing deployment therefore needs either `BACKUP_PATH` in `.env` or an empty
|
||||
`backup/` beside `docker-compose.yml`.
|
||||
|
||||
Archives contain password hashes, sessions and the peering private key: keep the
|
||||
target private.
|
||||
|
||||
### Why the reader owns it
|
||||
|
||||
The reader is the only service that mounts all of `/data` (the scraper sees only
|
||||
`/data/scraper`, Sentinel only `/data/sentinel`), it holds `db.json` in memory and owns
|
||||
its atomic writer, it already talks to the scraper's private control API with the
|
||||
shared token, and the admin settings UI lives there. Running it in Sentinel would need
|
||||
a full read-write `/data` mount there and break its "writes only `data/sentinel/`"
|
||||
contract (see [architecture.md §3](architecture.md#3-components)).
|
||||
|
||||
## Settings and status
|
||||
|
||||
Admin only, under Service health (`GET/POST /api/admin/backup`, also included in
|
||||
`GET /api/admin/health` as `backup`):
|
||||
|
||||
- **Back up daily at HH:MM** (container time zone `TZ`) and an enable switch.
|
||||
- **Keep the newest** N backups (`count`, 1–365) or N days (`days`, 1–3650).
|
||||
- **Back up now** starts a run immediately; the page polls every 2 s while it runs.
|
||||
- Status: target folder, number/size of archives, the running phase, last backup
|
||||
(time, trigger, size, duration, file), its error and warnings, next scheduled run.
|
||||
|
||||
Settings and the last 10 results persist atomically in `data/backup.json`. A slot
|
||||
missed while the reader was down runs once at the next start if it is less than a
|
||||
day old; older slots (for example after restoring an old `backup.json`) are skipped.
|
||||
|
||||
```text
|
||||
POST /api/admin/backup {command:"settings", settings:{enabled, time:"HH:MM", retention:{mode:"count"|"days", value}}}
|
||||
POST /api/admin/backup {command:"run"} -> 409 when off or already running
|
||||
```
|
||||
|
||||
## What one run does
|
||||
|
||||
1. Ask the scraper for its state (`GET /api/state`). If the queue runs, pause it
|
||||
(`POST /api/control {action:"pause"}`) and remember that this run paused it.
|
||||
2. Wait until no job is active (the worker stops at its next page/chapter boundary),
|
||||
polling every 2 s for at most 15 minutes. On timeout the run fails, writes nothing
|
||||
and resumes the queue if it paused it. An unreachable scraper does not block the
|
||||
backup; that is recorded as a warning.
|
||||
3. Write `BACKUP/circus-YYYYMMDD-HHMMSS.tar.gz.part`: a gzip'd tar of `DATA` with
|
||||
paths relative to `DATA`, fsync it, read it back to verify, rename it to `.tar.gz`.
|
||||
- `db.json` is serialized from the reader's in-memory state at that moment.
|
||||
- `scraper/queue.json` is stored with `paused: true` (status `paused`), so a
|
||||
restored copy never starts downloading by itself. The live file is not touched.
|
||||
- `scan-cache.json`, `library-index.json`, `catalog.json`, peering, Sentinel state,
|
||||
logs and everything else under `DATA` is copied as found; all of them are
|
||||
written by atomic rename, so each file is complete.
|
||||
- Left out: `runtime*.json`, `*.lock` (`daemon.lock`, `supervisor.lock`), `*.pid`,
|
||||
`*.tmp`, symlinks, and the `BACKUP` folder itself if it lies inside `DATA`.
|
||||
- `.circus-backup.json` at the archive root records when and how it was made.
|
||||
4. Resume the queue only if this run paused it (three attempts; a failure is shown
|
||||
as a warning: resume it under Settings → Scraper).
|
||||
5. After a successful run only, apply retention. It deletes only files named
|
||||
`circus-YYYYMMDD-HHMMSS.tar.gz` directly inside `BACKUP` (age from the name),
|
||||
never the newest one, never other files or subfolders. Stale
|
||||
`circus-*.tar.gz.part` files from an interrupted run are removed at the next run.
|
||||
|
||||
If the reader restarts during a run, `backup.json` still carries the run's
|
||||
"queue paused by backup" marker; the next start resumes the queue and records the
|
||||
run as interrupted. The marker is never part of an archive.
|
||||
|
||||
Measured on a copy of a production data folder (54 MB, 1000 files): 5.8 MB archive in
|
||||
about 5 s including verification. Pauses last until the current chapter is done
|
||||
plus those seconds.
|
||||
|
||||
## Restore
|
||||
|
||||
Restoring replaces the user data with the state of the archive. Reading progress
|
||||
and queue changes made after that backup are lost.
|
||||
|
||||
1. Stop the stack (or at least reader, scraper and Sentinel):
|
||||
`docker compose stop reader scraper sentinel`.
|
||||
2. Move the current data folder aside instead of deleting it:
|
||||
`mv /srv/mangacircus/data /srv/mangacircus/data.before-restore`.
|
||||
3. Unpack the archive into an empty data folder owned by uid 1000:
|
||||
```bash
|
||||
mkdir /srv/mangacircus/data
|
||||
tar -xzf /path/to/circus-YYYYMMDD-HHMMSS.tar.gz -C /srv/mangacircus/data
|
||||
chown -R 1000:1000 /srv/mangacircus/data
|
||||
```
|
||||
(`--same-owner` is the default for root; the archive keeps the original uid/gid.)
|
||||
4. Start the reader first: `docker compose up -d --no-build reader`. Check `/health`,
|
||||
log in, check progress and library. The reader rebuilds a missing or stale library
|
||||
index by itself.
|
||||
5. Start `scraper sentinel`. The restored queue is paused: check it under
|
||||
Settings → Scraper and resume it when it looks right.
|
||||
|
||||
The scraper token is not in `DATA`; Compose keeps it under `.secrets/` (or the
|
||||
deployment's `SCRAPER_TOKEN_PATH`). Restore tests should unpack into a separate
|
||||
folder and run an isolated stack (no second writer against the live library), as
|
||||
a restore-test script does for the host-level archives.
|
||||
@@ -59,6 +59,10 @@ download times by manga/source, and a collapsed recent-check log. Successful
|
||||
no-change nightly checks update this summary and leave the queue. Sources owns
|
||||
the per-hoster request and chapter delay settings.
|
||||
|
||||
Service health also holds the **Backups** card: built-in backup status (last run,
|
||||
size, duration, error), **Back up now**, and the daily schedule and retention form,
|
||||
whose draft survives the page's polling. See [backup.md](backup.md).
|
||||
|
||||
Reading-time collection controls, export and deletion live under Reading time &
|
||||
privacy; reader layout controls stay under Reading. Rescan controls and user
|
||||
administration are separate. Server & library describes existing maintenance,
|
||||
|
||||
Reference in new issue
Block a user