Add built-in backup of the data folder (BL-0044)
The reader owns a scheduled backup into BACKUP (Compose: BACKUP_PATH mounted at /backup, off when unset). A run pauses the scraper queue through its control API, waits (bounded) until no job is active, writes a verified tar.gz of DATA with db.json from memory and scraper/queue.json stored paused, excludes runtime/lock/pid/temp files, resumes only a queue it paused, then applies count- or day-based retention to its own archive names only. Schedule, retention, "Back up now" and last result live in Settings > Service health. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
MaashkinoozheandClaude Opus 5.5 committed
1 parent
fcc50d576c
commit
086f6353f8
15 files changed
+904
-11
No files matched your search
+114
@@ -0,0 +1,114 @@
|
||||
# Built-in backup
|
||||
|
||||
The reader can back up its own `DATA` folder on a schedule set in
|
||||
**Settings → Admin → Service health → Backups**, without stopping any container.
|
||||
The library is not part of it (it is plain files; back it up with the storage that
|
||||
holds it). Implementation: [`backup.js`](../backup.js); tests: `test/backup.test.js`.
|
||||
|
||||
## Turning it on
|
||||
|
||||
| Where | Setting |
|
||||
|---|---|
|
||||
| Native | `BACKUP=/path/to/backups` for `server.js`. Unset = feature off |
|
||||
| Compose | `BACKUP_PATH=/host/folder` in `.env`. The reader mounts it rw at `/backup` and gets `BACKUP=/backup`; without `BACKUP_PATH` the variable stays empty and backups are off |
|
||||
|
||||
The folder must already exist and be writable by uid 1000 (`create_host_path: false`:
|
||||
a missing folder makes `docker compose up` fail instead of silently creating an empty
|
||||
one on the wrong disk). Use another disk or a NAS share, never a folder inside `DATA`.
|
||||
Without `BACKUP_PATH`, Compose still mounts the unused default `./backup`, so that
|
||||
folder must exist too (`node tools/prepare-compose.js` creates it). Upgrading an
|
||||
existing deployment therefore needs either `BACKUP_PATH` in `.env` or an empty
|
||||
`backup/` beside `docker-compose.yml`.
|
||||
|
||||
Archives contain password hashes, sessions and the peering private key: keep the
|
||||
target private.
|
||||
|
||||
### Why the reader owns it
|
||||
|
||||
The reader is the only service that mounts all of `/data` (the scraper sees only
|
||||
`/data/scraper`, Sentinel only `/data/sentinel`), it holds `db.json` in memory and owns
|
||||
its atomic writer, it already talks to the scraper's private control API with the
|
||||
shared token, and the admin settings UI lives there. Running it in Sentinel would need
|
||||
a full read-write `/data` mount there and break its "writes only `data/sentinel/`"
|
||||
contract (see [architecture.md §3](architecture.md#3-components)).
|
||||
|
||||
## Settings and status
|
||||
|
||||
Admin only, under Service health (`GET/POST /api/admin/backup`, also included in
|
||||
`GET /api/admin/health` as `backup`):
|
||||
|
||||
- **Back up daily at HH:MM** (container time zone `TZ`) and an enable switch.
|
||||
- **Keep the newest** N backups (`count`, 1–365) or N days (`days`, 1–3650).
|
||||
- **Back up now** starts a run immediately; the page polls every 2 s while it runs.
|
||||
- Status: target folder, number/size of archives, the running phase, last backup
|
||||
(time, trigger, size, duration, file), its error and warnings, next scheduled run.
|
||||
|
||||
Settings and the last 10 results persist atomically in `data/backup.json`. A slot
|
||||
missed while the reader was down runs once at the next start if it is less than a
|
||||
day old; older slots (for example after restoring an old `backup.json`) are skipped.
|
||||
|
||||
```text
|
||||
POST /api/admin/backup {command:"settings", settings:{enabled, time:"HH:MM", retention:{mode:"count"|"days", value}}}
|
||||
POST /api/admin/backup {command:"run"} -> 409 when off or already running
|
||||
```
|
||||
|
||||
## What one run does
|
||||
|
||||
1. Ask the scraper for its state (`GET /api/state`). If the queue runs, pause it
|
||||
(`POST /api/control {action:"pause"}`) and remember that this run paused it.
|
||||
2. Wait until no job is active (the worker stops at its next page/chapter boundary),
|
||||
polling every 2 s for at most 15 minutes. On timeout the run fails, writes nothing
|
||||
and resumes the queue if it paused it. An unreachable scraper does not block the
|
||||
backup; that is recorded as a warning.
|
||||
3. Write `BACKUP/circus-YYYYMMDD-HHMMSS.tar.gz.part`: a gzip'd tar of `DATA` with
|
||||
paths relative to `DATA`, fsync it, read it back to verify, rename it to `.tar.gz`.
|
||||
- `db.json` is serialized from the reader's in-memory state at that moment.
|
||||
- `scraper/queue.json` is stored with `paused: true` (status `paused`), so a
|
||||
restored copy never starts downloading by itself. The live file is not touched.
|
||||
- `scan-cache.json`, `library-index.json`, `catalog.json`, peering, Sentinel state,
|
||||
logs and everything else under `DATA` is copied as found; all of them are
|
||||
written by atomic rename, so each file is complete.
|
||||
- Left out: `runtime*.json`, `*.lock` (`daemon.lock`, `supervisor.lock`), `*.pid`,
|
||||
`*.tmp`, symlinks, and the `BACKUP` folder itself if it lies inside `DATA`.
|
||||
- `.circus-backup.json` at the archive root records when and how it was made.
|
||||
4. Resume the queue only if this run paused it (three attempts; a failure is shown
|
||||
as a warning: resume it under Settings → Scraper).
|
||||
5. After a successful run only, apply retention. It deletes only files named
|
||||
`circus-YYYYMMDD-HHMMSS.tar.gz` directly inside `BACKUP` (age from the name),
|
||||
never the newest one, never other files or subfolders. Stale
|
||||
`circus-*.tar.gz.part` files from an interrupted run are removed at the next run.
|
||||
|
||||
If the reader restarts during a run, `backup.json` still carries the run's
|
||||
"queue paused by backup" marker; the next start resumes the queue and records the
|
||||
run as interrupted. The marker is never part of an archive.
|
||||
|
||||
Measured on a copy of a production data folder (54 MB, 1000 files): 5.8 MB archive in
|
||||
about 5 s including verification. Pauses last until the current chapter is done
|
||||
plus those seconds.
|
||||
|
||||
## Restore
|
||||
|
||||
Restoring replaces the user data with the state of the archive. Reading progress
|
||||
and queue changes made after that backup are lost.
|
||||
|
||||
1. Stop the stack (or at least reader, scraper and Sentinel):
|
||||
`docker compose stop reader scraper sentinel`.
|
||||
2. Move the current data folder aside instead of deleting it:
|
||||
`mv /srv/mangacircus/data /srv/mangacircus/data.before-restore`.
|
||||
3. Unpack the archive into an empty data folder owned by uid 1000:
|
||||
```bash
|
||||
mkdir /srv/mangacircus/data
|
||||
tar -xzf /path/to/circus-YYYYMMDD-HHMMSS.tar.gz -C /srv/mangacircus/data
|
||||
chown -R 1000:1000 /srv/mangacircus/data
|
||||
```
|
||||
(`--same-owner` is the default for root; the archive keeps the original uid/gid.)
|
||||
4. Start the reader first: `docker compose up -d --no-build reader`. Check `/health`,
|
||||
log in, check progress and library. The reader rebuilds a missing or stale library
|
||||
index by itself.
|
||||
5. Start `scraper sentinel`. The restored queue is paused: check it under
|
||||
Settings → Scraper and resume it when it looks right.
|
||||
|
||||
The scraper token is not in `DATA`; Compose keeps it under `.secrets/` (or the
|
||||
deployment's `SCRAPER_TOKEN_PATH`). Restore tests should unpack into a separate
|
||||
folder and run an isolated stack (no second writer against the live library), as
|
||||
a restore-test script does for the host-level archives.
|
||||
Reference in new issue
Block a user