Add built-in backup of the data folder (BL-0044)

The reader owns a scheduled backup into BACKUP (Compose: BACKUP_PATH mounted at
/backup, off when unset). A run pauses the scraper queue through its control
API, waits (bounded) until no job is active, writes a verified tar.gz of DATA
with db.json from memory and scraper/queue.json stored paused, excludes
runtime/lock/pid/temp files, resumes only a queue it paused, then applies
count- or day-based retention to its own archive names only. Schedule,
retention, "Back up now" and last result live in Settings > Service health.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
MaashkinoozheandClaude Opus 5.5 committed 2026-10-22 18:03:21 +02:00
1 parent fcc50d576c
commit 086f6353f8
15 files changed
+904 -11

No files matched your search

+11 -2
View File
@@ -18,6 +18,7 @@ Related documents:
- [peering.md](peering.md): optional private server-to-server catalog and transfer.
- [discovery.md](discovery.md): reader-owned, opt-in deterministic discovery feed and its limits.
- [source-modules.md](source-modules.md): adding and maintaining first-party hosters inside the single scraper process.
- [backup.md](backup.md): built-in scheduled backup of `data/` and the restore procedure.
## 1. Principles
@@ -67,6 +68,7 @@ survives restarts, so restarting does not itself force another complete walk.
|------|------:|----------------|
| `server.js` | ~350 | HTTP server and router, auth and sessions, JSON storage, API handlers, scan scheduling, notifications and webhooks |
| `library.js` | ~170 | Folder scanner, name parsing, image-size reader, scan cache |
| `backup.js` | ~330 | Built-in `DATA` backup: queue pause/resume, tar.gz writer and verifier, schedule, retention |
| `public/common.js` | ~85 | Browser helpers: `h()` element builder, `api()` fetch wrapper, top navigation with the notification bell |
| `public/index.html` + `library.js` | ~60 | Library overview (cover grid, filter, sort) |
| `public/manga.html` + `manga.js` | ~110 | Manga page: info, follow, chapter list with book separators, versions and votes, read/unread toggle |
@@ -157,7 +159,11 @@ This keeps ids human-readable and derivable without a database. The price: renam
- `data/library-index.json` holds a disposable, versioned index snapshot and shallow
directory stamps. The library folders remain authoritative; a missing or invalid
snapshot triggers a full scan. Rollback-compatible `scan-cache.json` is retained.
- **Backups:** copy `data/db.json`. The atomic writes make a copy at any time consistent.
- `data/backup.json` holds the built-in backup schedule, retention and recent results.
- **Backups:** the reader's built-in backup ([backup.md](backup.md)) archives the whole
`data/` tree into `BACKUP` on a schedule, with `db.json` taken from memory and the
queue paused around the copy. A plain copy of `data/db.json` also works: atomic
writes make it consistent at any time.
## 6. Server
@@ -199,6 +205,7 @@ All requests and responses are JSON. POSTs require `Content-Type: application/js
| `POST /api/admin/rescan` `{full}` | admin | Scan now, and wait for the result |
| `POST /api/admin/user` `{name, password, admin}` | admin | Create a user or reset a password |
| `POST /api/admin/user/delete` `{name}` | admin | Delete a user and their data |
| `GET /api/admin/backup` · `POST /api/admin/backup` `{command:"settings"\|"run"}` | admin | Built-in backup status, schedule/retention, "back up now" ([backup.md](backup.md)) |
### 6.3 Authentication and security
@@ -342,7 +349,8 @@ docker compose up -d --build
- **Volumes:**
- `/library`: read-only in the reader and writable in the scraper. Files must be readable—and scraper targets writable—by uid 1000.
- `/data`: reader state must be writable; the scraper also needs its writable `/data/scraper` state.
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
- `/backup`: reader only, writable target of the built-in backup (`BACKUP_PATH`; off when unset).
- **Configuration:** process wiring uses environment variables (`PORT`, `LIBRARY`, `DATA`, `BACKUP`, `SIGNUP`, `SCAN_INTERVAL`, `FULL_SCAN_INTERVAL`, `COOKIE_SECURE`, `SCRAPER_URL`, and `SCRAPER_TOKEN`). Queue pacing and automatic-update schedule are persisted in `data/scraper/queue.json` and edited through the authenticated admin UI.
- **Without Docker:** `npm run start:all` with Node ≥ 18, or supervise `npm start` and `npm run start:scraper` independently. On the current Windows machine that supervision is already owned by `\MangaCircus Supervisor`.
- **Behind a reverse proxy:** terminate TLS, set `COOKIE_SECURE=1`. A sub-path works.
- **Upgrades:** replace the code and restart. Reader state has no explicit migration;
@@ -366,6 +374,7 @@ docker compose up -d --build
|----------|--------------|-----|
| Index the folder tree in place | Import into a DB / managed storage | Stays readable by any viewer and writable by any tool. Nothing to keep in sync |
| No dependencies, no framework, no build | Express, React, bundler | Minimal code, maximum longevity, runs anywhere Node runs |
| Built-in backup in the reader, queue paused, no container stop | Host job that stops the stack; backup in Sentinel | Only the reader sees all of `/data` and the in-memory `db.json`; the scraper's pause API gives a consistent point without downtime |
| JSON file for user data | SQLite (`node:sqlite` is still experimental in Node 22/24), Postgres | Zero dependencies, trivial backup, enough for the target size. Easy to swap: all access goes through the `db` object |
| Page roles as file-name tags | Page lists in JSON | Visible in any file browser, survives copying, no second source of truth. Unknown tags are harmless |
| Image sizes read on the server | Measure in the browser after load | Layout before load: exact scroll restore, zoom anchoring, auto-detected double pages, no layout jumps |
+114
View File
@@ -0,0 +1,114 @@
# Built-in backup
The reader can back up its own `DATA` folder on a schedule set in
**Settings → Admin → Service health → Backups**, without stopping any container.
The library is not part of it (it is plain files; back it up with the storage that
holds it). Implementation: [`backup.js`](../backup.js); tests: `test/backup.test.js`.
## Turning it on
| Where | Setting |
|---|---|
| Native | `BACKUP=/path/to/backups` for `server.js`. Unset = feature off |
| Compose | `BACKUP_PATH=/host/folder` in `.env`. The reader mounts it rw at `/backup` and gets `BACKUP=/backup`; without `BACKUP_PATH` the variable stays empty and backups are off |
The folder must already exist and be writable by uid 1000 (`create_host_path: false`:
a missing folder makes `docker compose up` fail instead of silently creating an empty
one on the wrong disk). Use another disk or a NAS share, never a folder inside `DATA`.
Without `BACKUP_PATH`, Compose still mounts the unused default `./backup`, so that
folder must exist too (`node tools/prepare-compose.js` creates it). Upgrading an
existing deployment therefore needs either `BACKUP_PATH` in `.env` or an empty
`backup/` beside `docker-compose.yml`.
Archives contain password hashes, sessions and the peering private key: keep the
target private.
### Why the reader owns it
The reader is the only service that mounts all of `/data` (the scraper sees only
`/data/scraper`, Sentinel only `/data/sentinel`), it holds `db.json` in memory and owns
its atomic writer, it already talks to the scraper's private control API with the
shared token, and the admin settings UI lives there. Running it in Sentinel would need
a full read-write `/data` mount there and break its "writes only `data/sentinel/`"
contract (see [architecture.md §3](architecture.md#3-components)).
## Settings and status
Admin only, under Service health (`GET/POST /api/admin/backup`, also included in
`GET /api/admin/health` as `backup`):
- **Back up daily at HH:MM** (container time zone `TZ`) and an enable switch.
- **Keep the newest** N backups (`count`, 1–365) or N days (`days`, 1–3650).
- **Back up now** starts a run immediately; the page polls every 2 s while it runs.
- Status: target folder, number/size of archives, the running phase, last backup
(time, trigger, size, duration, file), its error and warnings, next scheduled run.
Settings and the last 10 results persist atomically in `data/backup.json`. A slot
missed while the reader was down runs once at the next start if it is less than a
day old; older slots (for example after restoring an old `backup.json`) are skipped.
```text
POST /api/admin/backup {command:"settings", settings:{enabled, time:"HH:MM", retention:{mode:"count"|"days", value}}}
POST /api/admin/backup {command:"run"} -> 409 when off or already running
```
## What one run does
1. Ask the scraper for its state (`GET /api/state`). If the queue runs, pause it
(`POST /api/control {action:"pause"}`) and remember that this run paused it.
2. Wait until no job is active (the worker stops at its next page/chapter boundary),
polling every 2 s for at most 15 minutes. On timeout the run fails, writes nothing
and resumes the queue if it paused it. An unreachable scraper does not block the
backup; that is recorded as a warning.
3. Write `BACKUP/circus-YYYYMMDD-HHMMSS.tar.gz.part`: a gzip'd tar of `DATA` with
paths relative to `DATA`, fsync it, read it back to verify, rename it to `.tar.gz`.
- `db.json` is serialized from the reader's in-memory state at that moment.
- `scraper/queue.json` is stored with `paused: true` (status `paused`), so a
restored copy never starts downloading by itself. The live file is not touched.
- `scan-cache.json`, `library-index.json`, `catalog.json`, peering, Sentinel state,
logs and everything else under `DATA` is copied as found; all of them are
written by atomic rename, so each file is complete.
- Left out: `runtime*.json`, `*.lock` (`daemon.lock`, `supervisor.lock`), `*.pid`,
`*.tmp`, symlinks, and the `BACKUP` folder itself if it lies inside `DATA`.
- `.circus-backup.json` at the archive root records when and how it was made.
4. Resume the queue only if this run paused it (three attempts; a failure is shown
as a warning: resume it under Settings → Scraper).
5. After a successful run only, apply retention. It deletes only files named
`circus-YYYYMMDD-HHMMSS.tar.gz` directly inside `BACKUP` (age from the name),
never the newest one, never other files or subfolders. Stale
`circus-*.tar.gz.part` files from an interrupted run are removed at the next run.
If the reader restarts during a run, `backup.json` still carries the run's
"queue paused by backup" marker; the next start resumes the queue and records the
run as interrupted. The marker is never part of an archive.
Measured on a copy of a production data folder (54 MB, 1000 files): 5.8 MB archive in
about 5 s including verification. Pauses last until the current chapter is done
plus those seconds.
## Restore
Restoring replaces the user data with the state of the archive. Reading progress
and queue changes made after that backup are lost.
1. Stop the stack (or at least reader, scraper and Sentinel):
`docker compose stop reader scraper sentinel`.
2. Move the current data folder aside instead of deleting it:
`mv /srv/mangacircus/data /srv/mangacircus/data.before-restore`.
3. Unpack the archive into an empty data folder owned by uid 1000:
```bash
mkdir /srv/mangacircus/data
tar -xzf /path/to/circus-YYYYMMDD-HHMMSS.tar.gz -C /srv/mangacircus/data
chown -R 1000:1000 /srv/mangacircus/data
```
(`--same-owner` is the default for root; the archive keeps the original uid/gid.)
4. Start the reader first: `docker compose up -d --no-build reader`. Check `/health`,
log in, check progress and library. The reader rebuilds a missing or stale library
index by itself.
5. Start `scraper sentinel`. The restored queue is paused: check it under
Settings → Scraper and resume it when it looks right.
The scraper token is not in `DATA`; Compose keeps it under `.secrets/` (or the
deployment's `SCRAPER_TOKEN_PATH`). Restore tests should unpack into a separate
folder and run an isolated stack (no second writer against the live library), as
a restore-test script does for the host-level archives.
+4
View File
@@ -59,6 +59,10 @@ download times by manga/source, and a collapsed recent-check log. Successful
no-change nightly checks update this summary and leave the queue. Sources owns
the per-hoster request and chapter delay settings.
Service health also holds the **Backups** card: built-in backup status (last run,
size, duration, error), **Back up now**, and the daily schedule and retention form,
whose draft survives the page's polling. See [backup.md](backup.md).
Reading-time collection controls, export and deletion live under Reading time &
privacy; reader layout controls stay under Reading. Rescan controls and user
administration are separate. Server & library describes existing maintenance,