MaashkinoozheandClaude Opus 5.5
e9da27e688
The page hotlinked ~100 AniList CDN covers per load and some failed at random. The reader now downloads each catalog cover once (sequential, ~1/s, stops on 401/403/429, images up to 2 MB) into data/greatest-covers/ and serves it from /api/greatest-cover with a year-long browser cache; until a copy exists the route redirects to the catalog URL. The page also accepts the server's relative cover URLs, so owned titles show their library cover again. GREATEST_COVER_WARM=0 turns the downloads off (tests, offline installs). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
202 lines
13 KiB
Markdown
202 lines
13 KiB
Markdown
# Deploying MangaCircus with Docker
|
||
|
||
This guide sets up MangaCircus on one Linux host with Docker Compose: the reader (web app), the managed
|
||
scraper, and optionally HTTPS and server peering. Every step is a command with what you should see. It is
|
||
written so that a person or a coding agent can follow it without other context.
|
||
|
||
## 1. What runs
|
||
|
||
| Service | Port | Role |
|
||
|---|---|---|
|
||
| `reader` | 8080 (published on `READER_BIND`) | Web app: library, reader, accounts, settings. The only writer of users, progress and the catalog (`DATA_PATH`). Reads the library read-only |
|
||
| `scraper` | 8091, internal only | Download queue and source adapters. The only writer of the library (`LIBRARY_PATH`) and of its own state (`SCRAPER_DATA_PATH`). Never publish or proxy this port |
|
||
| `peer-gateway` | 8093, optional (`--profile peering`) | Shares chosen manga with other MangaCircus servers over mutual TLS ([peering.md](peering.md)) |
|
||
|
||
All services run from one image (`node:24-alpine`, no runtime dependencies) as uid/gid 1000. Each container
|
||
has a read-only root file system, no capabilities, rotated logs, a health check and an in-container
|
||
supervisor that restarts a crashed process. The reader works when the scraper is stopped.
|
||
|
||
Data on the host, all bind mounts (no named volumes):
|
||
|
||
| Path (`.env`) | Contents | Back up? |
|
||
|---|---|---|
|
||
| `LIBRARY_PATH` | The manga: `Manga/Book/Chapter/images` ([library-format.md](library-format.md)) | Yes, it is your collection |
|
||
| `DATA_PATH` | `db.json` (users, progress, votes), catalog, discovery feeds, library index cache | Yes (built-in backup, section 8). `discovery-cache/` inside it is safe to delete and may be left out of backups: the next harvest rebuilds it. The same holds for `greatest-covers/` (local copies of the Greatest catalog's cover images, fetched again after the next start) |
|
||
| `SCRAPER_DATA_PATH` | Queue, settings, source ledger, metadata cache | Included in the built-in backup |
|
||
| `BACKUP_PATH` | Built-in backup archives | Keep it on another disk or NAS |
|
||
| `SCRAPER_TOKEN_PATH` | Shared secret between reader and scraper | Regenerate if lost |
|
||
|
||
## 2. Requirements
|
||
|
||
- Linux x86_64 or arm64 with Docker Engine 24+ and the Compose plugin 2.24.4+ (`docker compose version`).
|
||
Docker Desktop on Windows or macOS works for trying it out. Bind mounts are slow there for large libraries.
|
||
- 1–2 CPU cores and 1 GB RAM are enough for a few readers. The first scan of a large library is
|
||
I/O-bound (about 14 minutes for 300,000 pages on a NAS over NFS). Later starts reuse the index cache.
|
||
- Disk: the library plus room for downloads in progress. The image is about 60 MB.
|
||
- Git to fetch the code. Node is **not** needed on the host.
|
||
|
||
## 3. Get the code and prepare folders
|
||
|
||
```bash
|
||
git clone <repository-url> mangacircus && cd mangacircus
|
||
mkdir -p library data/scraper data/peering data/peering-usage backup .secrets
|
||
openssl rand -hex 32 > .secrets/scraper_token && chmod 600 .secrets/scraper_token
|
||
cp .env.example .env
|
||
sudo chown -R 1000:1000 library data backup
|
||
```
|
||
|
||
The containers run as uid 1000: the reader must be able to read the library and write `data/`, and the
|
||
scraper must be able to write the library. On an NFS share with squashed ids, map all users to one account
|
||
that owns the files. Don't use `chmod 777`. Bind mounts never create missing folders: a typo in a path
|
||
fails the start instead of showing an empty library.
|
||
|
||
Already have a library? Point `LIBRARY_PATH` at it (folder format in [library-format.md](library-format.md)).
|
||
Starting empty? Fill it from the web UI's scraper queue (section 6). To try the reader first, create five
|
||
demo manga in an empty library after the build in section 5:
|
||
`docker compose run --rm --no-deps scraper node tools/make-dummy.js /library`.
|
||
|
||
## 4. Configure `.env`
|
||
|
||
| Variable | Default | Meaning |
|
||
|---|---|---|
|
||
| `LIBRARY_PATH` | `./library` | Library folder (absolute path recommended) |
|
||
| `DATA_PATH` | `./data` | Reader data |
|
||
| `SCRAPER_DATA_PATH` | `./data/scraper` | Scraper state |
|
||
| `PEER_DATA_PATH`, `PEER_USAGE_PATH` | `./data/peering`, `./data/peering-usage` | Peering identity and transfer counters (folders must exist even when peering is off) |
|
||
| `SCRAPER_TOKEN_PATH` | `./.secrets/scraper_token` | The secret file from section 3 |
|
||
| `BACKUP_PATH` | unset = backups off | Folder for built-in backups (section 8) |
|
||
| `READER_BIND` / `READER_PORT` | `127.0.0.1` / `8080` | Where the reader listens on the host. `0.0.0.0` = every interface (plain HTTP, LAN only) |
|
||
| `TZ` | `UTC` | Time zone of the containers; affects the nightly update time and backup schedule |
|
||
| `COOKIE_SECURE` | `0` | `1` when readers reach the site over HTTPS |
|
||
| `READING_WINDOWS_JSON`, `READING_TIMEZONE`, `READING_QUIZ_*` | off | Optional server-enforced reading hours ([reading-limiter.md](reading-limiter.md)) |
|
||
| `DISCOVERY_HARVEST_AT` | `03:30` | Daily time (`HH:MM`) of the nightly Discover harvest from AniList, in the container's local time zone (`TZ`). Only users who opted in are harvested; the admin can change the time or switch it off on the Discover page ([discovery.md](discovery.md)) |
|
||
| `DISCOVERY_ANILIST_URL` | AniList's public API | Tests only: points the discovery harvest at a fixture server. Leave unset in production |
|
||
| `GREATEST_COVER_WARM` | `1` | `0` stops the reader from downloading the Greatest catalog's covers (about 100 images, one per second, 10 s after start and once a day for missing ones). Tests and offline installs set `0`; the page then loads covers from the catalog's own URLs |
|
||
| `APP_IMAGE`, `VCS_REF` | `mangacircus:local`, `development` | Image to run; set a pinned registry image to deploy prebuilt releases (section 7) |
|
||
| `COMPOSE_FILE` | `docker-compose.yml` | Extra Compose files, `:`-separated (e.g. `docker-compose.yml:deploy/compose.https.yml`) |
|
||
| `MANGACIRCUS_DOMAIN` | – | Host name for `deploy/compose.https.yml` |
|
||
|
||
Sign-up is closed: the first account becomes the admin, who then adds the other users. Reader settings
|
||
not listed here (scan intervals, scan concurrency) are in the README's environment table.
|
||
|
||
## 5. Build and start
|
||
|
||
```bash
|
||
docker compose config -q # no output = the configuration is valid
|
||
docker compose build
|
||
bash deploy/circus.sh up # reader first, waits for its health check, then the scraper
|
||
bash deploy/circus.sh status
|
||
```
|
||
|
||
Expected:
|
||
|
||
```
|
||
SERVICE STATUS PORTS
|
||
reader Up 1 minute (healthy) 127.0.0.1:8080->8080/tcp
|
||
scraper Up 40 seconds (healthy) 8080/tcp
|
||
{"ok":true,"scan":{"mangas":…,"stale":false,"error":null}, …}
|
||
{"status":"idle","paused":false,"active":0,"queue":0, …}
|
||
```
|
||
|
||
Open `http://127.0.0.1:8080` (or an SSH tunnel: `ssh -L 8080:127.0.0.1:8080 <host>`) and create the admin
|
||
account. The library page fills as soon as the first scan finishes.
|
||
|
||
## 6. Access and adding manga
|
||
|
||
- **On the LAN:** set `READER_BIND=0.0.0.0` (or the host's LAN address), or use `compose.lan.yml` with
|
||
`READER_LAN_BIND`. This is plain HTTP: logins and cookies travel unencrypted, so never forward it to the
|
||
internet.
|
||
- **HTTPS with the bundled Caddy:** point a DNS name at the host, open ports 80 and 443, set
|
||
`MANGACIRCUS_DOMAIN=manga.example.org` and `COMPOSE_FILE=docker-compose.yml:deploy/compose.https.yml`,
|
||
then `bash deploy/circus.sh up`. Caddy fetches the certificate. The reader gets no host port and secure
|
||
cookies.
|
||
- **Your own proxy:** keep `READER_BIND=127.0.0.1` and proxy `http://127.0.0.1:8080`. Alternatively, use
|
||
`compose.vm.yml`, which joins the reader to an external Docker network (`PROXY_NETWORK`, default
|
||
`mangacircus-edge`) and removes the host port. Set `COOKIE_SECURE=1`. All pages use relative URLs, so a
|
||
sub-path such as `/manga/` works.
|
||
- **Private access over a VPN** (WireGuard, Tailscale, …): bind the reader to the VPN address, or keep it on
|
||
loopback and forward from the VPN client.
|
||
- **Adding manga:** admins use **Settings → Admin → Scraper → Inspect & queue** with a series URL from a
|
||
supported source, or copy folders into the library by hand. MangaDex is on by default. WeebCentral and
|
||
MangaPill are unofficial sites that mostly host unlicensed copies, so they start switched off. Switch them
|
||
on per source under **Hoster settings** if you may copy what you download there. While a source is off,
|
||
its queued jobs wait and nightly update checks skip it. Upgraded installations keep the sources their
|
||
queue already used. The reader
|
||
notices new folders within `SCAN_INTERVAL` minutes. Rules for tools that write into the library:
|
||
[scraper-guide.md](scraper-guide.md). Only download what you are allowed to copy; the scraper keeps
|
||
conservative per-host rate limits and stops on 401/403/429 ([scraper/AGENT_RUNBOOK.md](../scraper/AGENT_RUNBOOK.md)).
|
||
|
||
## 7. Operating it
|
||
|
||
`deploy/circus.sh` wraps the safe sequences (run it from any folder; it finds the repository itself):
|
||
|
||
| Task | Command | Notes |
|
||
|---|---|---|
|
||
| Status | `bash deploy/circus.sh status` | services, reader health, queue |
|
||
| Stop / start | `… down` / `… up` | `down` pauses the queue and waits for the running chapter (up to 15 minutes); `up` resumes only what `down` paused |
|
||
| Restart | `… restart` or `… restart reader` | |
|
||
| Logs | `… logs scraper 200` | |
|
||
| Pause / resume downloads | `… pause` / `… resume` | Same as the buttons in Settings → Admin → Scraper |
|
||
| Update to the newest code | `… update` | `git pull --ff-only`, rebuild, pause, recreate, resume |
|
||
| Re-read edited `chapter.json` files | `… rescan` | Full reconciliation without an admin login; admins can also use Settings → Admin → Rescan |
|
||
| Check the newest backup restores | `… restore-test` | Throwaway reader without network; prints `RESTORE TEST OK` |
|
||
|
||
**Deploying prebuilt images** (several hosts, or no building on the server): build once, push to your
|
||
registry and pin the digest:
|
||
|
||
```bash
|
||
docker build --build-arg VCS_REF=$(git rev-parse --short HEAD) -t registry.example.org/mangacircus:$(git rev-parse --short HEAD) .
|
||
docker push registry.example.org/mangacircus:<commit> # note the sha256 digest it prints
|
||
# on the server, in .env: APP_IMAGE=registry.example.org/mangacircus@sha256:<digest>
|
||
docker compose pull && bash deploy/circus.sh restart
|
||
```
|
||
|
||
Pinned digests make rollbacks exact: put the previous `APP_IMAGE` back and restart. Data formats only grow
|
||
(new fields are ignored by older releases), but a rollback does not undo queue or catalog changes made by
|
||
the newer release.
|
||
|
||
**Rules for operators and agents:**
|
||
- Never run two scrapers, or a container and a native `node` process, against the same library or data
|
||
folder.
|
||
- Never run `docker compose down -v`. Stop with `deploy/circus.sh down`.
|
||
- Never publish port 8091, and never print or commit `.secrets/` or `.env`.
|
||
- One-off maintenance tools that write `chapter.json` files are dry runs by default. Run them with
|
||
`--apply --writers-stopped` only after `docker compose stop scraper`, then `deploy/circus.sh rescan`.
|
||
Examples: `tools/backfill-chapter-dates.js` and `tools/release-dates.js`, run inside the image with
|
||
`docker compose run --rm --no-deps scraper node tools/<tool>.js /library …`.
|
||
|
||
## 8. Backups
|
||
|
||
- **Reader and scraper data:** set `BACKUP_PATH` to an existing folder on another disk or NAS (owned by uid
|
||
1000), restart, and choose the schedule and retention under **Settings → Service health → Backups**. A
|
||
run pauses the queue, waits for the current chapter, archives the data folder without stopping anything
|
||
and resumes. Details, archive contents and the restore procedure: [backup.md](backup.md). Check it now
|
||
and then with `deploy/circus.sh restore-test`.
|
||
- **The library:** back it up with your usual file backup (it is plain folders and images), including
|
||
hidden `.mcscraper-*` folders if downloads are in progress.
|
||
- **Restore:** `deploy/circus.sh down`, move the data folder aside, unpack the archive into a fresh data
|
||
folder (`tar -xzf circus-….tar.gz -C data`, owned by uid 1000), `deploy/circus.sh up`. Restored archives
|
||
have the queue paused; resume it after checking.
|
||
|
||
## 9. Optional: server peering
|
||
|
||
Share selected manga with friends' MangaCircus servers: enable the `peering` profile
|
||
(`docker compose --profile peering up -d peer-gateway`) and follow [peering.md](peering.md). The gateway
|
||
needs its own reachable port (`PEER_BIND`, `PEER_PORT`) and uses mutual TLS with pinned certificates.
|
||
|
||
## 10. Troubleshooting
|
||
|
||
| Symptom | Cause / fix |
|
||
|---|---|
|
||
| `docker compose up` fails with "bind source path does not exist" | A folder from `.env` is missing; create it (section 3) |
|
||
| Reader healthy but the library is empty | First scan still running (`docker compose logs reader` shows `[scan] …` when done), or `LIBRARY_PATH` points at the wrong folder |
|
||
| Scraper jobs fail with `EACCES` | The library or `SCRAPER_DATA_PATH` is not writable by uid 1000 |
|
||
| Backups fail with `EACCES` on a NAS | Map the share's users to the uid that owns the folder; the backup probes with a real write |
|
||
| Settings → Scraper says the service is unavailable | Scraper container stopped or unhealthy: `deploy/circus.sh status`, `logs scraper` |
|
||
| Edited `chapter.json` changes don't show | Run `deploy/circus.sh rescan` (in-place edits are picked up by full reconciliations, every 24 hours by default) |
|
||
| Logins fail behind HTTPS | `COOKIE_SECURE=1` needs HTTPS end to end at the browser; with plain HTTP keep it `0` |
|
||
|
||
More background: [architecture.md](architecture.md) (components, API, security model),
|
||
[import.md](import.md) (scanning and caching), [scraper-service.md](scraper-service.md) (queue, adapters,
|
||
recovery) and [resilience.md](resilience.md) (supervisors and incidents).
|